RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 Expanding a single-institution survey to multiple institutions: Lessons learned in research design and deployment Erin R.B. Eldermire, MLS Head, Flower Sprecker University Library Cornell University Ithaca, NY Kristine M. Alpi, MLS, AHIP University Librarian Oregon Health & Sciences University Library Portland, OR Suzanne Fricke, MLS, AHIP Animal Health Sciences Librarian Washington State University Pullman, WA Andrea C. Kepsel, MLS, AHIP Health Sciences Educational Technology Librarian Michigan State University Lansing, MI Erin E. Kerby, MSI Veterinary Medicine Librarian University of Illinois Urbana, IL Hannah F. Norton, MSIS, AHIP Chair, Health Science Campus Library University of Florida Gainesville, FL Jessica R. Page, MLIS Head, Veterinary Medicine Library Ohio State University Columbus, OH ABSTRACT Objective: Creating generalizable knowledge across institutions is a step beyond a successful local research project. The purpose of this article is to share the process and lessons learned from expanding a survey tool developed and piloted at a single veterinary college to its deployment at multiple veterinary colleges in the United States and Canada. Problem: Little guidance exists on expanding a survey developed for a single institution to distribution to health professions students across multiple institutions. RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 Methods: In June 2016, the first author of the survey contacted librarians from veterinary colleges to explore a possible multi-institution study to investigate student behaviors and perceptions around scientific information. Librarians from 29 institutions initially expressed interest. Those at 15 institutions participated in initial planning, and eight elected to distribute the survey. Of these, seven submitted for IRB review at their own institution and one institution facilitated the distribution of the survey under the original institution’s IRB exemption. Findings: The IRB submission process and requirements varied by participating institution. Mean time from submission to approval was 10 days (range: 2-31 days). Several changes were made to the survey based on the recommendations of participating librarians, ranging from simplifying the method of survey distribution to modifying specific questions to make them meaningful across institutions. Because participating institutions did not have synchronized academic calendars, the survey distribution took a staggered approach between institutions based on IRB review and varying institutional processes. Conclusions: Expanding even a simple IRB-exempt survey from one institution to others requires careful consideration of local practices, attention to differences in the IRB process, and ethical considerations for recruiting students where librarians serve as instructors or hold other positions of influence. Attempts to standardize recruitment messaging and survey questions for generalizable results required compromise by the librarian researchers at participating institutions. INTRODUCTION Educators struggle to know if research conducted at one institution will translate to similar results at other institutions with differing curricula and cultures. To develop a generalizable understanding of how veterinary students retrieve, evaluate and apply information from scientific papers, a librarian researcher solicited interest and feasibility of administering a survey, developed and piloted at Cornell University, across multiple institutions. The purpose of this report is to share the process and lessons learned from expanding a survey developed and deployed at a single veterinary college to multiple veterinary colleges in the United States and Canada. Findings from the survey research itself were published in Volume 107, issue 4 of the Journal of the Medical Library Association [Toews, Oct 2019] [1]. LITERATURE REVIEW Little literature exists to inform conducting a multi-institutional survey to assess health professions students’ information behaviors. A search of PubMed in March 2020 RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 for articles with “library” in the affiliation field and the publication type of “Multicenter Study” yielded few studies involving practicing librarians studying populations at multiple institutions. Three of these surveys address the practices of health professionals [2,3] or medical photographers [4]. Each of these have only two authors rather than broader authorship representing the other participating institutions. Perhaps one of the best- known multicenter studies by librarians with authorship from multiple institutions is the Marshall et al [5] study of physicians, residents and nurses on the value of library and information services in patient care. We did not find any multicenter studies by librarians researching students in the health professions. Outside of the librarianship domain, a small number of studies address the implementation of multi-institutional research involving medical students as participants and discuss the varied landscape of Institutional Review Board (IRB) review processes at included institutions. Mealer et al [6] provide a helpful summary of human subjects research and IRBs. Sarpel et al [7] outline experiences in interfacing with seven IRBs for a coordinated research project. A strategic approach to submitting to institutional IRBs is advised [8]. One approach that was successful in Oermann et al’s [9] study across 10 institutions was pursuing a deferral agreement, in which IRBs from different institutions defer to an external IRB’s decision. Whether pursuing novel research or through coordinated survey efforts such as the ACRL Project Outcome for Academic Libraries [10] or Ithaka S+R [11], participation in multi-institutional human research efforts will require librarians at each institution to pursue IRB approval. Regardless of the details, flexibility and communication are paramount as the specifics of a multi-institutional research study may introduce novel situations to navigate [8,12]. Research timeframes can quickly expand and diverge when working across different colleges and curricula [12]. METHODS In May 2016, the first author presented initial findings from a survey piloted at Cornell University at the Medical Library Association Annual Meeting [13]. This survey investigated student behaviors and perceptions about reading scientific information. Through this presentation, she solicited interest from potential collaborators to explore a multi-institutional study using the same survey. On June 17, 2016, the first author sent a message via two email discussion lists that host veterinary librarians (vetlib-g@vt.edu and vetlibs@lists.mlanet.org) to solicit further interest. Librarians from 29 institutions, representing veterinary libraries in Austria, Australia, Canada, Ireland, the United States, and Zimbabwe, expressed interest in participating. Cornell hosted a WebEx [14] meeting to discuss this project on July 7, 2016. During the meeting, librarians at 15 institutions participated in initial planning discussions about timing, the role of librarians in the study, changes needed to survey questions, IRB requirements, promotional strategies, and the use of Qualtrics [15] mailto:vetlib-g@vt.edu mailto:vetlibs@lists.mlanet.org RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 software for survey distribution. The recording was distributed via a OneDrive for Business [16] file. A message bounced back that recipients might not be able to access the files attached, so the recording was resent as a link to a shared Box [17] account. After the meeting, librarians at 17 (59%) institutions decided not to administer the survey. Reasons included poor timing, staff changes, curriculum changes, lack of interest from institutional leaders, and lack of time to devote to such a project. Afterwards, four (14%) additional institutions were unable to continue their collaboration due to leaves (planned and unplanned) or changes in work prioritization. Librarians representing the institutions that remained were all in the United States or Canada. The first author used a consensus-driven approach to make decisions and take next steps. From her initial solicitation message sent to email discussion lists to the date of first survey launch, seven email updates to participating individuals were sent and 105 days elapsed. Between each email update there were a number of individual or follow-up emails or phone calls. Ultimately, librarians at eight institutions (28% of those who initially expressed interest) distributed the survey: librarians at seven institutions submitted for IRB approval or exemption at their own institution and one facilitated the distribution of the survey under the original institution’s IRB exemption. The survey was distributed at participating institutions between September 14 and October 26, 2016. During the survey design and distribution process, participating librarians logged their experience with IRB submission and survey distribution for future reference and comparison. This study reviewing and summarizing those experiences was determined to be exempt by the Michigan State University Institutional Review Board (MSU Study ID STUDY00003868). After the original research was published, participating librarians reviewed their notes and described modifications that needed to be made to the IRB and survey process at their own institutions, and questions and challenges that arose during the process. The narratives were anonymized and composites were created to highlight general themes and challenges. We discussed the possibility of talking with non-participants about their reasons for non-participation, but decided not to pursue this line of investigation due to time constraints. FINDINGS The multi-institutional survey, like the pilot survey, included a 30 short-answer or Likert-type response structure, and an embedded article for participants to read and then respond to additional questions. The survey instrument can be seen in the supplemental files of Eldermire et al [1]. Adapting the survey to be distributed to students across multiple institutions required an iterative process with participation from each institution’s librarian. The adaptations, described below, impacted the survey implementation and research outcomes, and likely contributed to a lower participation rate than the pilot survey. RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 Minor changes to the survey content were made to clarify language to participants in multiple contexts. Examples included adding a drop-down menu for participants to indicate which school they were attending, and contact information of local institutional leads for questions or concerns about the survey. Other changes were more significant. Unlike the pilot survey, no incentives for participation were given for the multi-institutional study. Librarian researchers felt that offering incentives would lead to unneeded logistical complexities as well as funding source challenges. In addition, the embedded article included in the survey was changed to an open-access article so that copyright restrictions or authentication requirements did not add complexity. Although both studies focused on metacognition, the pilot survey used an article on the impact of teaching metacognitive learning strategies on academic performance [18], while the multi-institutional survey article focused on the impact of deliberate practice on the development of clinical skills [19]. The different embedded articles limited comparison between the pilot and multi- institutional surveys. Thus, we did not compare the embedded article analysis between the pilot and multi-institutional surveys. Approval or exemption and project information required by institutional IRBs varied between participating institutions. Three IRBs approved or exempted a customized version of the Cornell University protocol with minor changes. Three others required the local researcher to include additional information, such as timelines for how students would receive the survey invitation, details on data storage, and when and how students would be reminded about the opportunity to participate in the survey. One IRB required a copy of Cornell’s IRB exemption and assurance that IP addresses would not be collected for surveys answered via an anonymous link to move forward in their approval/exemption process. Partnering institutions in multi-state programs, of which there was one, relied on exemption by the primary program institution. The mean time from submission to approval or exemption was 10 days with a range of 2 to 31 days. IRB processes also impacted the distribution method used. The initial plan was to administer the survey using the Qualtrics mailer from the first author’s account, with the ability to send targeted reminders to non-responders. However, we determined that this was not possible without providing student emails to the primary administering institution, which did not meet the confidentiality needs at some institutions Whereas the original survey design broke the survey into two parts, allowing participants to leave the survey and return to it later, the fact that some institutions required distribution of the survey with an anonymous link meant that their participants could not easily leave and return to the survey. Therefore, the survey was adapted to encourage participants to take it in one session. Variance between college administrations also impacted how the survey was distributed to participants. Some institutions did not require direct approval for librarians to contact students about a research study, and the librarian researcher was able to RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 send a survey invitation directly to cohort email lists or through the Qualtrics mailer. Other institutions required permission from Deans of the respective veterinary colleges, did not allow direct emails to student lists from librarians, or restricted the number of times students could receive reminder messages to take the survey. One administration requested that the survey be approved by the American Association of Veterinary Medical Colleges (AAVMC), however, when contacted, the AAVMC declined to do so stating that it was beyond the capacity of their organization. Survey fatigue was a stated concern for at least one institution. Some institutions allowed only one reminder to be sent, others allowed two or more. Due to the variety of processing times for administrative and IRB approval/exemption, as well as different academic calendars at participating institutions, the survey distribution dates at each institution were staggered and did not occur on a particular date. While this worked on a rolling basis, it meant that by the time the survey was released at some institutions, students were already in the midst of the semester and were less likely to participate in the survey due to course load, time restraints, and other similar reasons. We collected data for most of the institutions in the Cornell University instance of Qualtrics. For these, the first author distributed institution-specific results to the coauthor at each institution. Data collected by other institutions in their own instance of Qualtrics were shared with the first author to be aggregated with the group data in the analysis phase. DISCUSSION Conducting this multi-institutional research project took collaboration and patience while the authors iteratively adapted a survey from one institution, sought institutional assurances, and administered it in heterogeneous contexts. Throughout this process we learned lessons we believe will help those considering conducting a multi-institutional study using a survey instrument. First, we advise careful consideration of survey content and platform functionality. When adapting a survey that was created for one institution to be used at multiple institutions, intensive communication between researchers and survey instrument testing may be necessary. Adapting the survey required each researcher to iteratively evaluate the survey with their institutional environment in mind, suggest changes that could be accommodated by all participating institutions, and field test and reassess the updated instrument until all were comfortable with the ultimate product. Second, as research involving human subjects, IRB policies and expectations varied between institutions. Variations can occur in the time elapsed from submission to approval or exemption, as well as in the materials or information required for review. The first author’s expectation that materials from one IRB exemption could be applied to other institutions’ IRB processes with minor edits proved not to be the case, as several RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 other IRBs required much more detailed information. Furthermore, IRB and equivalent review boards in other countries may have varying protections for human subjects or processes for seeking approval or exemption [20]. One size does not fit all. If we were to implement this project again, we would expect variation in IRB processes at each institution. Each IRB is composed of individuals who are operating in a unique institutional environment, so variation and change are to be expected. Additionally, IRB processes have changed since we sought approval for our multi-institutional study. If we were to embark on a similar multi-institutional study at the same institutions now, we would start by examining the IRB process. Since the implementation of this study, the IRB landscape has changed significantly, including the requirement of a single IRB for multi-institutional studies [21]. As this policy trickles down, organizations have adapted. For example, the National Institutes of Health adapted a single IRB policy for multi-site research starting in 2018 [22]. We may also consider a deferral agreement, as suggested by the literature, for a more strategic approach. Third, we found that administrative considerations and expectations varied between institutions. Similar to other institutional variations encountered in this process, administrators such as deans, department heads, and other leaders had differing expectations on how students could learn more about the survey, survey dissemination, and security of participant information. The outcome of such variation translated to differences in how the survey was distributed at each institution, introducing additional variability to the participation and outcomes across institutions. Attempts to standardize recruitment messaging and survey questions for generalizable results required compromise by the researchers at participating institutions. Related, participating librarians varied in faculty status; however, that status was not reported by any participating librarian as influencing the priority that survey distribution received at each institution. Fourth, managing a multi-institutional project required additional layers of adaptability beyond the challenges of project management at a single institution. Typical project management tasks, such as clear and persistent communication from the leader–both with the group and with individuals–and final decision making to move the group forward were essential. Project management across multiple institutions required additional accommodations, such as scheduling discussions across time zones, establishing an online file-sharing platform that all researchers could access and contribute to, and in our specific case, integrating an open access article into the survey to be distributed across institutions rather than negotiating copyright restrictions or institutional links to full-text that would require authentication by participants. Had participants from additional time zones or countries participated, the group may have needed to exercise more flexibility in meeting arrangements, file sharing tools, or other details that we did not encounter in conducting this study. RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 An additional participant protection issue to be addressed would be to consider a system that can be trusted to safely store survey results for a multi-institutional survey. Some institutions specified that they did not want survey responses to be stored at Cornell University. Storing survey data in the cloud with restricted access by certain individuals may be a viable option, but the online survey tools used may introduce other data storage considerations. For example, each of three popular survey tools— Qualtrics, Survey Monkey, and Google Forms—have different access, storage and company policy environments. We suggest careful consideration of the survey tool and implications for data access and storage. Further, the language of our consent process did not permit the data to be made publicly available—this is something we recommend considering in future studies. At the time of our survey, there were significant concerns about survey fatigue and stress among students frequently surveyed for assessment of teaching, assessment of learning, and research. Attitudes and processes surrounding educational research with veterinary students have likely evolved since we pursued this research in 2016. For example, institutions that did not have or were in the process of developing a survey review process for any surveys to be disseminated to the students in 2016 may have set up a formal process, such as a committee established at The Ohio State University for conducting surveys [23]. In the case of one participating institution, there was a plan in 2016 to set up a formal process, but since few requests came in, as of March 2020 the process remains to submit surveys for review and approval by the Associate Dean and Director of Academic Affairs. In planning for another survey, we would attempt to clear review processes well enough in advance so that surveys could be distributed on the same timeline at all institutions, limiting differences in response rates due to timing. In conclusion, expanding even a simple IRB-exempt survey from one institution to others requires careful consideration of local practices, attention to differences in the IRB process, and ethical considerations for recruiting students where librarians serve as instructors or hold other positions of influence. Administering a survey to participants across multiple institutions will require ample time, cooperation, and flexibility amongst the researchers involved. The survey timeframe, delivery method, communication with participants and survey content were all impacted by the process required to deliver the survey to a multi-institutional context. Even though these confounders were a factor in our research, we see great value in conducting studies across institutional boundaries to broaden understanding and conclusions beyond the unique environment of one institution. ACKNOWLEDGEMENTS We thank the anonymous peer reviewers for offering suggestions that strengthened this manuscript. RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 AUTHOR CONTRIBUTIONS Authors include Erin R. B. Eldermire, Kristine M. Alpi, Suzanne Fricke, Andrea C. Kepsel, Erin E. Kerby, Hannah F. Norton and Jessica R. Page. All authors secured IRB exemption/permission from their respective institutions to administer the survey referenced in this manuscript, administered the survey, and contributed substantially to writing the manuscript. In addition: • Erin Eldermire led the effort • Kris Alpi made substantial revisions to the manuscript • Suzanne Fricke made significant suggestions for the inclusion of concepts to the manuscript • Andrea Kepsel sought IRB exemption for this study REFERENCES 1. Eldermire ERB, Fricke S, Alpi KM, Davies E, Kepsel AC, Norton H. Information Seeking and Evaluation: a multiinstitutional survey of veterinary students. J Med Lib Assoc. 2019 May; 107(4):515-526. DOI: https://doi.org/10.5195/jmla.2019.674 2. Ajuwon GA, Olorunsaye JO. Knowledge, access and usage pattern of HINARI by researchers and clinicians in tertiary health institutions in south-west Nigeria. Afr J Med Med Sci. 2013 Mar;42(1):97–106. 3. Cheng GY, Lam LM. Information-seeking behavior of health professionals in Hong Kong: a survey of thirty-seven hospitals. Bull Med Libr Assoc. 1996 Jan;84(1):32–40. 4. Addison FJ, Kehoe MA. An assessment of risk associated with the photography of microbiological materials. J Audiov Media Med. 2004 Sep;27(3):102–6. DOI: https://doi.org/10.1080/01405110400007726 5. Marshall JG, Sollenberger J, Easterby-Gannett S, Morgan LK, Klem ML, Cavanaugh SK, Oliver KB, Thompson CA, Romanosky N, Hunter S. The value of library and information services in patient care: results of a multisite study. J Med Libr Assoc. 2013 Jan;101(1):38–46. DOI: https://doi.org/10.3163/1536-5050.101.1.007 6. Mealer M, Flynn L, Ironside P, Spurlock D. Accelerating multisite research in nursing education: navigating institutional review board regulatory issues. J Nurs Educ. 2017 Feb 1;56(2):65–8. DOI: https://doi.org/10.3928/01484834-20170123-02 7. Sarpel U, Hopkins M, More F, Yavner S, Pusic M, Nick M, Song H, Ellaway R, Kalet A. Medical students as human subjects in educational research - the importance of responder bias. Med Educ Online. 2013 Jan1;18(1):19524. DOI: https://doi.org/10.3402/meo.v18i0.20757 8. North S, Giddens J. Lessons learned in multisite, nursing education research while studying a technology learning innovation. J Nurs Educ. 2013 Sep 13;52(10):567–73. DOI: https://doi.org/10.3928/01484834-20130313-02 https://doi.org/10.5195/jmla.2019.674 https://doi.org/10.1080/01405110400007726 https://doi.org/10.3163/1536-5050.101.1.007 https://doi.org/10.3928/01484834-20170123-02 https://doi.org/10.3402/meo.v18i0.20757 https://doi.org/10.3928/01484834-20130313-02 RESEARCH Hypothesis Vol. 32 No.1 Fall/Winter 2020 9. Oermann MH, Hallmark BF, Haus C, Kardong-Edgren SE, McColgan JK, Rogers N. Conducting multisite research studies in nursing education: brief practice of CPR skills as an exemplar. J Nurs Educ. 2012 Jan;51(1):23–8. DOI: https://doi.org/10.3928/01484834-20111130-05 10. ACRL Project Outcome [Internet]. [cited 2020 Mar 30]. Available from: https://acrl.projectoutcome.org/ 11. Ithaka S+R [Internet]. [cited 2020 Mar 30]. Available from: https://sr.ithaka.org/ 12. Foronda C, Alhusen J. Lessons learned from an international, multisite simulation research study. Nurse Educ. 2016 Aug;41(4):199–203. DOI: https://doi.org/10.1097/nne.0000000000000242 13. Eldermire, E. and Davies, E. (2016, May). Reading behavior of veterinary students: how big (or small) is their picture? Medical Library Association Conference. Toronto, Canada. 14. Webex [Internet]. [cited 2020 Mar 30]. Available from: https://www.webex.com/video-conferencing 15. Qualtrics [Internet]. [cited 2020 Mar 30]. Available from: https://www.qualtrics.com/ 16. Microsoft OneDrive [Internet]. [cited 2020 Mar 30]. Available from: https://products.office.com/en-us/onedrive/onedrive-for-business 17. Box [Internet]. [cited 2020 Mar 30]. Available from: https://www.box.com/home 18. Cook E, Kennedy E, McGuire SY. Effect of Teaching Metacognitive Learning Strategies on Performance in General Chemistry Courses. J Chem Educ. 2013 Aug 13;90(8):961–7. DOI: https://doi.org/10.1021/ed300686h 19. Duvivier RJ, van Dalen J, Muijtjens AM, Moulaert VR, van der Vleuten CP, Scherpbier AJ. The role of deliberate practice in the acquisition of clinical skills. BMC Med Educ. 2011 Dec 6;11:101. DOI: https://doi.org/10.1093/med/9780199366149.003.0005 20. Millum J. Canada’s new ethical guidelines for research with humans: a critique and comparison with the United States. Can Med Assoc J. 2012 Apr 3;184(6):657–61. DOI: https://doi.org/10.1503/cmaj.111217 21. Hodge JG, Gostin LO. Revamping the US Federal Common Rule: Modernizing Human Participant Research Regulations. J Am Med Assoc. 2017 Apr 18;317(15):1521–2. DOI: https://doi.org/10.1001/jama.2017.1633 22. Single IRB policy for multi-site research | grants.nih.gov [Internet]. [cited 2020 Jan 29]. Available from: https://grants.nih.gov/policy/humansubjects/single-irb-policy-multi- site-research.htm 23. Conducting Surveys [Internet]. Office of Academic Affairs, The Ohio State University. 2017 [cited 2020 Mar 30]. Available from: https://oaa.osu.edu/conducting- surveys https://doi.org/10.3928/01484834-20111130-05 https://acrl.projectoutcome.org/ https://sr.ithaka.org/ https://doi.org/10.1097/nne.0000000000000242 https://www.webex.com/video-conferencing https://www.qualtrics.com/ https://products.office.com/en-us/onedrive/onedrive-for-business https://www.box.com/home https://doi.org/10.1021/ed300686h https://doi.org/10.1093/med/9780199366149.003.0005 https://doi.org/10.1503/cmaj.111217 https://doi.org/10.1001/jama.2017.1633 https://grants.nih.gov/policy/humansubjects/single-irb-policy-multi-site-research.htm https://grants.nih.gov/policy/humansubjects/single-irb-policy-multi-site-research.htm https://oaa.osu.edu/conducting-surveys https://oaa.osu.edu/conducting-surveys