Microsoft Word - Evaluation of a Research Internship article.docx Published by University of Minnesota Libraries Publishing Evaluation of a Research Internship in a Rural Healthcare System: The Essentia Health Summer Research Intern Program Catherine Benziger, MD, MPH; Ryan Thiel, MA; Nicole Groth, BS; Vicki Pink, RN; Joseph Doerer, MD; Kate Dean, MBA DOI: https://doi.org/10.24926/jrmc.vXiX.XXX Journal of Regional Medical Campuses, Vol. 3, Issue 2 (2020) z.umn.edu/JRMC All work in JRMC is licensed under CC BY-NC Catherine Benziger, MD, MPH cardiologist in the Division of Heart and Vascular, Essentia Health, Duluth. Joseph Doerer, MD, cardiologist in the Division of Heart and Vascular, Essentia Health, Duluth. Ryan Thiel, MA, is a Research & Evaluation Specialist at the Essentia Institute of Rural Health. Nicole Groth, BS, is a Research Associate at the Essentia Institute of Rural Health. Vicki Pink, now retired, was a nurse at the Minneapolis Heart Institute Abbott Northwestern Hospital. Kate Dean, MBA, is executive director of the Essentia Institute of Rural Health. Corresponding author: Catherine Benziger 407 East Third Street Duluth, MN 55805 Email: Catherine.Benziger@essentiahealth.org Twitter: @DrBengizerHeart All work in JRMC is licensed under CC BY-NC Volume 3, Issue 2 (2020) Journal of Regional Medical Campuses Original Reports Evaluation of a Research Internship in a Rural Healthcare System: The Essentia Health Summer Research Intern Program Catherine Benziger, MD, MPH; Ryan Thiel, MA; Nicole Groth, BS; Vicki Pink, RN; Joseph Doerer, MD; Kate Dean, MBA Abstract Background: Training medical and premedical students in clinical research is challenging with the demands of medical education. Students pursue research opportunities in specialty practice to acquire knowledge and gain experience in the clinical setting. Pairing clinical observation with mentored research informs the value of evidence-based medicine and prepares students for a career in medicine and science. However, programs that offer both research training and clinical observership are rare, particularly in rural areas. Hypothesis: Early exposure to sub-specialty practice in rural areas is important to recruit and train the next generation of physicians to serve this population. The aim of this program was to provide undergraduate and medical students in a largely rural community the opportunity to participate in clinical cardiovascular disease research and active clinical observership. Methods: This report provides a description of the Essentia Health Summer Research Internship Program, a pilot program with the University of Minnesota Medical School aimed to provide mentored clinical research training and clinical observership for medical and premedical students. The program provided an experiential model of clinical patient interaction, problem solving, and access to research that is integrated into clinical experiences as students learn the value of evidence-based medicine. Results: Six medical and 4 premedical students have participated in the internship within the program’s first 2 years. Student- completed weekly journals and surveys reveal enthusiasm for various research and clinical components of the program. Program highlights include the acquisition of research and clinical knowledge through mentorship and research lectures, and observerships throughout the Heart and Vascular Center. Conclusion: The program aims to provide underrepresented rural students training in research methodology through mentorship and observership within specialty practice. Background Preparing medical students to be doctors trained in clinical research is challenging with the current demands of medical education. Preclinical medical students desire the introduction to research and are eager for experience in the hospital setting. Understanding the nature of medical knowledge requires experience with scientific methodology through participation in research. Evidence-based medicine (EBM), the foundation of medical education, has emerged as the accepted philosophy of practice wherein health care practitioners provide therapies based on empirical results instead of tradition or opinion.1 Today, EBM has wide support among medical providers and is considered a requisite foundation for optimal medical care. The question becomes how to train the underlying constructs of EBM. Shadowing in clinical medicine is often passive and observational and may not be the best way to facilitate learning.2 Furthermore, hands-on research experience must have relevance to clinical practice. Until recently, medical schools have taken a standard approach to educating students, following a conventional curriculum, without emphasizing external scholarship or independent research.3,4 By creating an academic environment in which the students and faculty share the scholarly activities, this promotes both curiosity and knowledge of the research.5 Programs that integrate research and active clinical observations are rare but important to help students understand the value of evidence- based practice. The model for this research program was inspired by the Minneapolis Heart Institute Foundation (MHIF) Summer Research Intern (SRI) Program at Abbott Northwestern Hospital in Minneapolis, MN, which started in 2001.6 Two mentors of the Essentia Health Summer Research Intern Program (EHSRI) were former students in the MHIF program, which is why this model was chosen for our program. The EHSRI program is similar to the MHIF program and gives DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 2 Original Reports undergraduate premedical and preclinical medical students exposure to mentored clinical research.6 The MHIF program trained a diverse group of undergraduate and medical students and 52 of 65 alumni (80%) went on to study medicine.6 There is a lack of diversity in cardiovascular specialists, specifically underrepresented minorities and women.7,8 Furthermore, Native Americans and students from rural areas remain underrepresented in medical schools.9-12 There are a growing number of institutions, predominantly in urban areas, that have either active clinical observership programs for premedical students or research-focused programs.6,13-20 However, opportunities for undergraduate and medical students to engage in both clinical research and active clinical observerships remains limited, particularly for students not located in large metropolitan areas.6,13,18-21 Studies suggest that research experience offers students the opportunity to enhance analytical skills, self-directed learning, improved oral and written communication skills, and a better understanding about how to apply knowledge to patient care.3 Other studies have shown that advantages of a research and quality improvement of health services experience, like mentor connection and networking at conferences, can increase the likelihood of a student matching with a future residency program of choice and plays a major role in the student’s professional growth and development during medical school.22 The aim of this program was to provide undergraduate and medical students in a largely rural community the opportunity to participate in clinical cardiovascular disease research and active clinical observership. Methods Essentia Health Research Program Description The Essentia Health Summer Research Intern Program (EHSRI) operates within the Essentia Institute of Rural Health (EIRH) and Essentia Health-Heart and Vascular Center. Located in Duluth, MN, USA, the Essentia Health-Heart and Vascular Center is a tertiary care center which services a large rural area. In the summer of 2018, we completed our first year piloting the 6-week EHSRI with 2 medical students. The program expanded to eight (4 pre-medical and 4 medical) students in the summer of 2019. Students in the program came from 4-year undergraduate programs or an accredited medical school. A prerequisite was having a “pre-medical” focus so most students had recently completed or were currently enrolled in bachelor’s degree majoring in the sciences, applying to medical school, or were current first- year medical students. There were 2 students who applied for the internship in 2018, 19 students in 2019 and 14 students in 2020. Students completed a 4-12-week internship based on availability. Selection criteria for the program included students in excellent academic standing with a strong letter of recommendation and personal statement indicating interest and experience in research. Students received a stipend in alignment to similar summer research programs. The program was run by volunteer faculty including 4 cardiologists and a clinical pharmacist, who guided and mentored students on projects related to their specialty. The EHSRI program is unique in that it recruits students from rural areas, particularly Minnesota and the upper Midwest, and gives them early exposure to specialty practice (cardiology) and clinical research. Due to geographical limitations and lack of academic medical centers in the region, this program was unique in that it provided mentored clinical research at a community hospital that focused on disparities in cardiovascular disease care in rural areas. Early exposure to specialty practice in rural areas is important to recruit and train the next generation of physicians. Furthermore, this patient population is largely rural and has a higher percentage of Native Americans and recruiting students that reflect the population we serve is important.7,8,12,23 Introductory Training A timeline of the summer research experience is outlined in table 1. A critical aspect of the summer research program was orientation. Students were supervised in a step-by-step process to satisfy requirements for patient contact and research (table 2). Prior to orientation, students were expected to complete CITI training, which included approximately 8 hours of online modules that are required by the Essentia Health Institutional Review Board for all research personnel. During orientation, staff provided necessary forms, reference websites (UpToDate, PubMed) and introduction to various cardiology, clinical trials, and Essentia Institute of Rural Health (EIRH) personnel involved in the program. Without supervision, navigating the various components and understanding the roles of various key personnel would be challenging and time consuming. Each student received a 3-ring binder that included a campus map, program orientation guide, face sheets of key personnel, extensive list of abbreviations used in clinical research and cardiology, description of common cardiovascular procedures, PowerPoint slides of cardiac anatomy, common cardiovascular procedures, and a journal for the students to complete weekly about their experience. The orientation guide included guidelines on appropriate dress-code, guidelines for interactions with physicians and staff, daily schedules, research project assignments, and information about protected health information and confidentiality, as well as helpful hints for their final presentation and poster. DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 2 Original Reports Table 1. Timeline for 6-week student summer research program Table 2. Summer Intern Program Orientation Training Components Clinical Experience The clinical observation was a significant part of the overall experience and involved rounding with a physician and cardiovascular department staff on 6 half-days. The students had a checklist of required clinical experiences to complete with the aim of understanding how research can inform clinical practice. The number of patient cases that students experienced are organized by department in Table 3. A schedule was created that included dedicated time for observation of at least 3 ST-elevation myocardial infarction patients in the cardiac catheterization laboratory, implantation of a cardiac device in the electrophysiology laboratory, observation of an echocardiogram, and an exercise nuclear stress test. Students also had the opportunity to interview cardiology patients and round on an inpatient critical care cardiology consult patients. Table 3. Clinical experience Research Experience All research projects were investigator-initiated projects in cardiovascular disease that were approved by the Institutional Review Board. The pilot program provided students the unique opportunity to learn about cardiovascular research while seeing patients in the hospital and clinical setting with various cardiovascular conditions. The research experience required students to read the study protocol and complete a thorough literature search including downloading and becoming familiar with a reference manager, EndNote, prior to data analysis or manuscript preparation. During orientation, the students were given 2 half-day research lectures on study design and basic statistical analysis. The students were provided a dataset with instructional materials on how to conduct simple data analysis to find means, standard deviations, t-tests and ANOVA, and asked to conduct simple analysis. Three members of the research staff at the EIRH volunteered to help guide students in data analysis and interpretation. Students were required to meet with the EIRH informatics team to learn how clinical Electronic Medical Record (EMR) data is stored in large databases (Epic Clarity) and how to communicate effectively with non-clinical informaticists on merging datasets, as well as how to best extract data from the EMR. An overview of REDCap, a secure web application for building and managing online databases, was also provided. The IRB protocols, datasets, and draft manuscripts were stored in a private group using Microsoft SharePoint and only those with IRB approval were allowed access to the group. One cardiologist was responsible for ensuring ongoing project development, meeting biweekly with students, reading the weekly intern journals, and providing general medicine and cardiology-specific career expertise, information, and advise. Presentation At program conclusion, the students were required to submit a PowerPoint presentation to the program director and give a 15-minute presentation about their experience to the division at the weekly journal club. The first component of the presentation was the project background, methods, and preliminary results. Next, students shared highlights of their clinical experiences, including lessons learned. The presentation concluded with discussion amongst heart and vascular, clinical trials, and research staff regarding questions and additional information. The presentation afforded students formal experience presenting and interacting with clinical and research staff, as is often expected in clinical DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 2 Original Reports medicine and demonstrates how research informs clinical practice. Additional Educational Opportunities Weekly attendance at journal club, multidisciplinary heart team meetings, and valve conferences were encouraged. Mandatory attendance at a summer intern-specific lecture on various topics, such as research study design, epidemiology, biostatistics, cardiac anatomy, electrocardiograms, coronary artery disease, heart failure, cardiac imaging, and “Surviving the Wards.” Students also participated in a simulation catheterization lab experience where they experienced the hands-on deployment of cardiac stents, toured the manufacturing facilities where stents and other devices, such as the WATCHMAN are assembled, and participated in an in- depth learning experience reviewing imaging of the heart at Boston Scientific in Maple Grove, MN. Supplemental Program Metrics Students in 2019 completed 2 identical knowledge tests (the first and last week of the program) that evaluated their knowledge of general cardiology principles (8 questions), such as definitions of common cardiology acronyms and identification of common structures on echocardiogram and angiogram, and research methods and analyses (12 questions), including type of study design and regulatory requirements. All students completed weekly reflection journals regarding their confidence, comfort level, knowledge, and understanding of their research project. The journal was submitted to the program director and additional research support staff to gauge progress and triage questions or concerns. Following the conclusion of the program, students completed an online survey detailing their experiences in the program. All survey responses were transcribed in a Microsoft Excel database. An iterative analysis was used in coding the data to achieve rich description and identify emergent themes within the text. Results A total of 10 students completed the internship. Knowledge Evaluation Results from the 2019 pre-and post-test are displayed in Figure 1. The mean pre-test score was 51% (SD = 3.9), while the mean post-test score was 75% (SD = 1.7). Separate cardiology tested scores versus research method scores. Overall, students showed an approximate mean increase of 24% in their knowledge and understanding of the components of heart and vascular research, and individual increases in score ranging from 10% to 45%. Figure 1. Student knowledge of general cardiology principles and research methods Program Evaluation Ten students completed an exit survey that found 40% identified as female, 80% ever-lived in rural areas, and 10% were underrepresented minorities. Through the weekly journal, students self-reported weekly improvement regarding comfortableness of content with a plateau period of 6 weeks to become “very comfortable” with the content of their research project (Figure 2). Additionally, students self- reported a total of 23 manuscripts (M = 2.6, SD = 1.7) completed or in-production at the time of the final survey. Wood W., et al (2018)24 found that about 25% of previous articles discussing similar programs had required students to achieve a specific outcome, for instance, submission of a manuscript or presentation at a regional or national conference. Students in our program were required to give a research presentation locally and were encouraged to submit an abstract to a national conference. Overall, 80% (n=8) had presented at a regional conference and 30% (n=3) at a national conference. Over 50% submitted manuscripts to national peer-reviewed journals. DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 2 Original Reports Figure 2. Weekly student self-rated understanding of project content (*denotes medical students who completed only 6 weeks) Qualitative Assessment Previous studies have noted the difficulty in evaluating outcomes of educational programs until several years after initiation,3 however, students did report (agree, strongly agree) that they were interested in pursuing a career in cardiology (n=9, 90%), research/academic medicine (n=6, 60%), or practicing medicine in a rural area (n=8, 80%). Knowledge acquisition and staff interaction were 2 key themes that emerged from the survey. Additionally, the students suggested earlier assignment of research projects and additional clinical experiences to improve the program. Knowledge Acquisition Students described the knowledge they gained from their clinical observations and leading their research project. One student commented “I appreciate that we were able to see a project through from start to finish. It helped provide a better understanding of all the moving parts that go into a research project.” Another student remarked that “I was very involved in choosing variables and setting up my research study. I feel like I have a better understanding of all the work that goes into research now.” A student elucidated their project dissemination experience, exclaiming “I learned about writing manuscripts, abstracts, posters, and even presenting in front of a room full of physicians.” Students routinely commented on the integral knowledge they acquired and its importance throughout their upcoming career. Staff Interaction Students emphasized the significance of interactions and mentorship from physician mentors and research support staff. A student indicated “The mentors were all so excited and passionate about their research, which rubbed off on me and made this experience a great one.” Another student said “It was wonderful to work with such dedicated people who were always very welcoming and kind. They genuinely wanted me to grow from this experience, and I believe that I have.” This sentiment was echoed by another student who remarked “It genuinely seemed like the providers wanted me to learn and were more than willing to take time to explain things to me in greater detail.” For many students, the interaction between providers and research staff was the highlight of their experience and fostered team-building and personal growth. Suggestions for Program Improvement The final question of the survey asked students “What would you change about the program?” Several students suggested earlier assignment of their research project, saying “Notify the students of what their project will be earlier during orientation week,” and “I would have liked to know my research topic earlier, or even in advance of arriving.” Students also recommended additional clinical experiences, including “I would add more clinical experiences. It would be good to shadow providers in clinic and see their patient interactions,” and “Have a shadowing experience with a clinical cardiologist and follow an interventionalist between procedures. I would love to see more of what their day to day operations look like.” Students offered concrete suggestions to improve the program for future implementation. Discussion We describe the initial experience with the EHSRI program. A total of 10 undergraduate preclinical and medical students completed the program and were eager to engage in clinical and research activities. The EHSRI was adopted from the MHIF summer intern program based in Minneapolis, MN and aimed to offer mentored training for preclinical and medical students from underserved and rural areas in Northern Minnesota the opportunity to gain clinical and research experience in the cardiology specialty. The program successfully recruited students who had lived in rural areas (80%) and female (40%). We aim to increase opportunities for undergraduate and medical students from rural areas to engage in both clinical research and active clinical observerships as most of these internship are located in large metropolitan areas.5,12,17-20 Studies suggest that research experience offers students the opportunity to enhance analytical skills, self-directed learning, improved oral and written communication skills, and a better understanding about how to apply knowledge to patient care.3 Furthermore, we believe students who are exposed to research early in DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 2 Original Reports their training are more likely to understand and practice evidence-based medicine and continue in research in their career. Sustaining an internship program required stakeholder engagement and a dedicated leadership team. Continual evaluation and adaptation of the program with a focus on program outcomes with adequate financial resources to support the students are important. We found that students improved their cardiovascular disease and research knowledge from orientation to completion of internship. We found that it took approximately 6 weeks for students to self-report confidence and understanding of their research project. We would not recommend a summer internship shorter than 6 weeks given this data. Program evaluation was positive. Students valued the staff interaction highly and were eager to have more clinical exposure. All students were able to present their research at a local and most at a national meeting. Adopting future program evaluation using a standardized tool, such as the “scholarly concentration program outcome model” (figure 2 in Havnaer A., et al.)3 could be implemented. This tool could be used to describe in more detail the steps for other health care systems to aid with research program implementation and allow program planners and educators to more rigorously evaluate the scholarly characteristics and outcomes of the program. Evaluation tools, including pre- and post-tests and weekly journals, helps increase the reproducibility of the program at different institutions. Long- term tracking of students to determine if this program influenced their career decision or interest in research are needed; however, due to the long latency period between undergraduate students and completing residency and fellowship, this data will not be available for many years. Limitations Limited advertising and no program website initially limited the number of applicants to those who attended local institutions. Moreover, due to contractual agreements the internship was not available for medical students at all institutions. The start dates for several students were staggered throughout the summer, which made orientation, education, and allocation of resources challenging. The authors recommend a standardized start date for all students. The short duration of the program for medical students limited observational opportunities. Efforts to increase underrepresented minority students in pre-medical and medical school programs are needed to increase minority students in the internship. This program can be adopted for other healthcare systems and other specialties, but we feel it requires dedicated leadership and departmental support and formal mentors. General challenges to conducting research included difficulty merging various datasets, and lack of prerequisite knowledge of data analysis programs and software. Students encountered errors and missing data, which delayed data analysis. Conclusion Undergraduate preclinical and medical students are eager to engage in clinical and research activities. However, students in rural areas face barriers in gaining research and clinical experience due to heavy academic workloads and paucity of academic medical centers. The EHSRI offered mentored training for preclinical and medical students to gain clinical and research experience in the cardiology specialty. Initial enthusiastic feedback demonstrates the program’s potential and success in propelling students to a career in medicine and research. Reproduction of this program in other departments and other community hospitals would help improve access to scholarly activities and clinical exposure for students in rural areas. Acknowledgements The authors wish to thank Christine Leone, Kaitlyn Rikkola, and Brianna Lemke for their assistance creating and implementing the program, and Richard Mullvain, Dr. Wilson Ginete, Dr. Michael Mollerus, Dr. Kalkidan Bishu, and Anne Meyer for providing data and mentoring support. References 1. Sackett DL, Rosenberg WM, Gray JM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. In: British Medical Journal Publishing Group; 1996. 2.Clark CD. Doctoring Undercover: updating the educational tradition of shadowing. Medical education online. 2017;22(1):1-8. 3.Havnaer AG, Chen AJ, Greenberg PB. Scholarly concentration programs and medical student research productivity: a systematic review. Perspect Med Educ. 2017;6(4):216-226. 4.(COFM) CoOFoM. Scholarly Activity within Distributed Medical Education Programs: Reflections and Reccomendations. 2019. 5.Frank JR SL, Sherbino J. CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada;2015. 6.Willenbring BD, McKee KC, Wilson BV, Henry TD. The Minneapolis Heart Institute Foundation Summer Research Internship Program: the benefits of preprofessional experience for prospective physicians. Minnesota medicine. 2008;91(8):47-49. 7.Warnes CA, Fedson SE, Foster E, et al. Working group 2: how to encourage more women to choose a career in cardiology. Journal of the American College of Cardiology. 2004;44(2):238-241. 8.Francis CK, Alpert JS, Clark LT, Ofili EO, Wong RC. Working group 3: how to encourage more minorities to choose a DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 2 Original Reports career in cardiology. Journal of the American College of Cardiology. 2004;44(2):241-245. 9.Metz AM. Medical School Outcomes, Primary Care Specialty Choice, and Practice in Medically Underserved Areas by Physician Alumni of MEDPREP, a Postbaccalaureate Premedical Program for Underrepresented and Disadvantaged Students. Teaching and learning in medicine. 2017;29(3):351-359. 10.Nivet MA, Taylor VS, Butts GC, et al. Diversity in academic medicine no. 1 case for minority faculty development today: Diversity in academic medicine no. 1. Mount Sinai Journal of Medicine: A Journal of Translational and Personalized Medicine: A Journal of Translational and Personalized Medicine. 2008;75(6):491-498. 11.Capers Q, McDougle L, Clinchot DM. Strategies for achieving diversity through medical school admissions. Journal of health care for the poor and underserved. 2018;29(1):9-18. 12.Acosta D, Olsen P. Meeting the needs of regional minority groups: The University of Washington’s programs to increase the American Indian and Alaskan native physician workforce. Academic Medicine. 2006;81(10):863-870. 13.Alexander SF, Lyon LJ, Nevins MA, Ycre LR, Thayer HS. Ten years of orienting college students to careers in medicine. JAMA. 1992;267(24):3330-3331. 14.Chuck JM. Do premedical students know what they are getting into? Western journal of medicine. 1996;164(3):228. 15.Lovecchio K, Dundes L. Premed survival: Understanding the culling process in premedical undergraduate education. Academic Medicine. 2002;77(7):719-724. 16.Davis J, Anderson M, Stankevitz K. Providing premedical students with quality clinical and research experience: The University of Wisconsin Tobacco Science Scholars Program. WMJ: official publication of the State Medical Society of Wisconsin. 2013;112(5):195. 17.Kashou A, Durairajanayagam D, Agarwal A. Insights into an award-winning summer internship program: the first six years. The world journal of men's health. 2016;34(1):9-19. 18.Wagner AK, Stewart PJ. An internship for college students in physical medicine and rehabilitation: effects on awareness, career choice, and disability perceptions. American journal of physical medicine & rehabilitation. 2001;80(6):459-465. 19.Almy TP, Cohen RD, Ham TH, Hornig E, Price J. Health- related experimental learning for college undergraduates. Journal of medical education. 1983;58(5):404-410. 20Gronemeyer SA. The impact of predoctoral pediatric oncology education. Journal of Cancer Education. 2005;20(1):16-22. 21.Patel S, Walsh CM, Udell JA. Exploring medically-related Canadian summer student research programs: a National Cross-sectional Survey Study. BMC medical education. 2019;19(1):140. 22.Chang Y, Ramnanan CJ. A review of literature on medical students and scholarly research: experiences, attitudes, and outcomes. Acad Med. 2015;90(8):1162-1173. 23.Dos Santos LM. Rural Public Health Workforce Training and Development: The Performance of an Undergraduate Internship Programme in a Rural Hospital and Healthcare Centre. International journal of environmental research and public health. 2019;16(7):1259. 24.Wood W, McCollum J, Kukreja P, et al. Graduate medical education scholarly activities initiatives: a systematic review and meta-analysis. BMC Med Educ. 2018;18(1):318.