Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 Librarian and Clinical Faculty Collaborative Use of Formative Assessment in a Longitudinal Evidence-Based Medicine Curriculum Sa’ad Laws, MA, MLIS*a, Mai A. Mahmoud, MBBSb, Moune Jabre, MDc, Amal Khidir, MBBSd, Ziyad Mahfoud, PhDe aCorresponding Author, Health Sciences Library, Weill Cornell Medicine – Qatar, Doha, Qatar, sal2018@qatar-med.cornell.edu bAssociate Professor of Teaching in Medicine/Assistant Dean for Faculty Affairs, Department of Medical Education, Weill Cornell Medicine – Qatar, Doha, Qatar cAssistant Professor of Clinical Obstetrics and Gynecology/Assistant Dean for Academic Affairs, Department of Medical Education, Weill Cornell Medicine – Qatar, Doha, Qatar dAssociate Professor of Pediatrics, Department of Medical Education, Weill Cornell Medicine – Qatar, Doha, Qatar eProfessor of Research in Population Health Sciences, Department of Medical Education, Weill Cornell Medicine – Qatar, Doha, Qatar, Division of Epidemiology, Department of Population Health Sciences, Weill Cornell Medicine, New York, New York Cite as: Laws S, Mahmoud MA, Jabre M, Khidir A, and Mahfoud Z. Librarian and Clinical Faculty Collaborative Use of Formative Assessment in a Longitudinal Evidence-Based Medicine Curriculum. Hypothesis. 2024;36(2). doi:10.18060/28048 Laws, Mahmoud, Jabre, Khidir, Mahfoud. All works in Hypothesis are licensed under a CC BY-NC 4.0 DEED Attribution-NonCommercial 4.0 International. Authors own copyright of their articles appearing in Hypothesis. Readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. For any other use of articles, please contact the copyright owner(s). 1 mailto:sal2018@qatar-med.cornell.edu https://creativecommons.org/licenses/by-nc/4.0/ Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 Background The authors developed a longitudinal curriculum for teaching third- and fourth-year undergraduate medical students evidence-based medicine (EBM). This curriculum involved substantial librarian involvement, the use of formative assessment as a teaching method, and progressive repetition of skills. Methods Students in three clinical clerkships (Medicine, Pediatrics, OB/GYN) completed EBM assignments based on real-world scenarios in this quasi-experimental study. Each clerkship required students to submit an EBM plan, from which they received directed feedback from both the clerkship director and a librarian after submitting preliminary EBM plans. This study tracked student PICO and search submissions to determine if repeated exposure to EBM feedback resulted in improved summative assessments, both in isolation and longitudinally. Results Students’ PICO and searching performance improved between mid-clerkship formative feedback and end-clerkship summative assessment in all three clerkships using rubrics developed by the authors for each clerkship derived from existing literature. When examining student performance sequentially over three clerkships, there was significant improvement between the first and second clerkships, but this did not carry into the third clerkship. Discussion Our findings suggest that the significant inclusion of a librarian and feedback appears to have positive effects on student performance. While it may seem obvious that feedback results in improved outcomes, this method doesn’t appear widespread in medical education. Repetition, while not having a lasting increase in performance, may still be warranted to increase exposure to authentic cases and evidence types. Introduction In medical education, enhancing the ability to make well-informed clinical decisions is crucial to improving patient care. Central to this decision-making process is evidence-based medicine (EBM), a methodology that integrates clinical expertise, patient values, and the best available evidence to guide healthcare decisions. EBM is traditionally described as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”1 The use of EBM requires a holistic understanding of question formation, searching, critical appraisal, and clinical application.2 The acquisition of these skills is a fundamental aspect of healthcare provider education and a lifelong learning skill that has been identified by multiple medical education accreditation organizations at both undergraduate and graduate levels.3,4 While medical education has evolved to include EBM as a fundamental component, there remains a pressing need to ensure that students grasp the theoretical aspects and develop the skills to apply EBM in real-world clinical scenarios. In this context, several studies have examined different methods of teaching EBM to medical trainees. An important area of exploration in these studies is the timing, duration, and intensity of EBM instruction within the undergraduate curriculum. Some studies have focused on implementing EBM instruction early during preclinical years,5,6 while others have opted to delay until clinical-based years.7,8 A common rationale in studies that advocated for delay was implementation at a time when students were interacting with patients in the clinical setting to give more context to EBM learning. In another case, faculty have utilized other approaches, 2 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 such as the use of peer instructors, a capstone course for fourth-year undergraduate medical students, and dedicated seminars.9-11 Although these studies include an array of teaching methods, it is notable that none were conducted in a clinical setting. Both clinical and non-clinical-based instruction demonstrated an improvement in skills and attitudes compared to pre-instruction. Despite the gains noted in several studies, there does not appear to be a consensus about the best methods for instructing trainees in EBM. Ilic and Maloney note that most instructional methods show some level of improvement in competencies; there is no clear benefit of one method over others.12 A notable trend in many studies of EBM in both graduate and undergraduate medical education has been the inclusion of librarians in various roles. While it has been noted that librarians are an underutilized resource for EBM instruction,13 several studies have shown that librarians are adept in instructing students on question formulation and the best methods for searching clinical databases for evidence.14,15 In one example of librarian involvement, Minuti et al. described a flipped-classroom module that used readings, video, and exercise in support of sessions with first-year students that focused on PICO formulation and searching and later with second-year students where additional searching techniques were taught and clinical faculty followed with critical appraisal instruction.16 Minuti et al. noted that librarian involvement was significant for assisting in meeting accreditation requirements and the positive impressions that faculty had after the course. Another notable example of librarian integration in EBM instruction is from a study of trainees’ skills and attitudes in an emergency medicine course.17 This course incorporated significant levels of librarian involvement and emphasis on searching skills. Trainees reported significant improvements in searching-related skills, but they noted that additional instruction is necessary for sustained application and reinforcement. The present study is a further exploration of the benefits of both varied methods in teaching EBM to medical trainees as well as the significant incorporation of librarians within EBM instruction. Both of these areas of inquiry for teaching EBM are unresolved in the current literature and vital for the future development of EBM instruction to medical trainees. With this in mind, the objectives of this study are to evaluate the effectiveness of our EBM curriculum within each clerkship, as well as longitudinally between the first and third clerkships. Methods Study population and characteristics This quasi-experiment focused on EBM instruction within three clinical clerkships: Medicine, Pediatrics, and Obstetrics and Gynecology (OB/GYN). The study was conducted at Weill Cornell Medicine – Qatar (WMC-Q) in Doha, Qatar. WMC-Q follows a four-year, American-based medical curriculum from the main Weill Cornell Medicine campus in New York, USA. These three clerkships were specifically chosen because they are the only three that have explicit EBM curricular elements. Data from 32 students who completed all three clerkships were analyzed in this study. Approval for this study was granted by the WCM-Q institutional review board on February 10, 2022 (21-00026). Because this was a retrospective analysis, the study team requested and was granted a waiver of informed consent of subjects. Before entering these clerkships, students received preliminary EBM instruction during their first year of preclinical education. In this context, students took a course on clinical 3 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 epidemiology and research methods, followed by a 10-hour, face-to-face, course utilizing lectures and labs to emphasize EBM fundamentals, question formulation, searching, and critical appraisal. While the EBM curriculum of the three clerkships in this study followed a similar track, each clerkship employed slight variations to attain their EBM objectives. The distribution and progression of students through the three clerkships is shown in Table 1. It is noteworthy that the EBM instructional model introduced in the Pediatrics Clerkship was implemented during the second clerkship of the academic year. As a result, none of the students included in this study had pediatrics as their initial clerkship. Table 1: Student progression through clerkship with EBM instructions OB/GYN Pediatrics Internal Medicine First Clerkship 13 0 9 Second Clerkship 12 10 10 Third Clerkship 7 22 3 Internal Medicine Clerkship Students in the Internal Medicine Clerkship began the course by completing an asynchronous eLearning module that reviewed EBM fundamentals. The eLearning module in the Internal Medicine Clerkship is a gamified experience, mimicking the structure of the TV game show Jeopardy, where students select from four levels of questions across five categories that increase in difficulty and point value. The module was internally produced using standard web development tools, such as HTML and CSS. The eLearning module’s content included 20 total questions on PICO formation, searching techniques, evidence appraisal, and EBM question type (i.e., Harm, Therapy, Diagnosis, Prognosis). Each category included four questions that escalated in the level of difficulty and point value for correct answers (range 2-8 points). Each question offered feedback for both correct and incorrect answers. The main objective of the eLearning module is to refresh students on EBM fundamentals and highlight potential gaps in EBM comprehension. Students received a final score, with a maximum of 100 points possible, and the module concluded with a leaderboard of the top score for every iteration of the course. All eLearning module data is available to both the librarian and clerkship director. At the midpoint of the clerkship, students were grouped into teams of three to four students by faculty and each team was required to submit a group EBM plan. The content of the EBM plan must be drawn from the clinical experience of at least one of the students in the team. The plan included the case details, PICO, and search strategy. After submission, each EBM plan underwent two reviews. First, the clerkship director reviewed the plan for clinical viability, either approving it or requesting revisions. In parallel, the librarian evaluated each EBM plan’s PICO and search strategy. Additionally, each team met with the librarian to receive feedback on their PICO and search strategy and discussed potential amendments for various aspects of each team’s EBM plan. The summative assessment for EBM in the Internal Medicine Clerkship was a team-based oral presentation. Teams had to provide details about their patient’s case, PICO, search strategy, appraisal of a selected research article, and how they could apply the evidence from their selected article to their patient. After the presentation, fellow students, the librarian, and the director were invited to ask the presenting team questions about their presentation. The librarian assessed the PICO and search strategy, while the director evaluated the clinical and team-based components of each team’s EBM 4 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 presentation. All assessments used a rubric with four levels, from inadequate to exceeds expectation, with each level receiving a score of 0 to 5 (see appendices for detailed rubric). Pediatrics Clerkship Students in the Pediatrics Clerkship started the course with a required short asynchronous eLearning module that reviewed key aspects of EBM, such as searching and constructing a PICO. Students completed four “knowledge check” questions at various points in the tutorials to verify content comprehension. Each question offered feedback for both correct and incorrect answers. All students’ responses were available to both the librarian and clerkship director. At the midpoint of the clerkship, each student submitted an individual EBM plan based on a clinical encounter they had experienced during the clerkship. The EBM plan included the details of their case, PICO, and preliminary search strategy. Each EBM plan was reviewed by the clerkship director for clinical content and relevance. The clerkship director then either approved each plan or asked the student to submit a revised plan before the student could proceed. The librarian reviewed the PICO and search strategy. Both the clerkship director’s and the librarian’s feedback were sent to the student via email. Although no individual meetings were required, students were highly encouraged to set up individual meetings with either the librarian or clerkship director, if they felt this was needed. At the end of the clerkship, students submitted a written EBM project. Student projects introduced their patient’s case, PICO, search strategy, and appraisal of a selected research article, and explained how they could apply the evidence from their article to their patient. The librarian reviewed each EBM project, provided comments on the PICO and search strategy, and then forwarded these to the clerkship director, who completed the grading of clinical and appraisal content. All assessments used a rubric with four levels, from inadequate to exceeds expectation, with each level receiving a score of 0 to 3 (see appendices for detailed rubric). Obstetrics and Gynecology (OB/GYN) Clerkship Students started the OB/GYN Clerkship by completing an asynchronous eLearning module that reviewed EBM fundamentals ranging from question formation, searching, and appraisal, to application. At the start of the eLearning module, students were presented with an OB/GYN-based case. Students then worked through ten questions from three domains, searching/PICO, appraisal, and application, based on the case and reviewed a systematic review relevant to the case. Each question offered feedback for both correct and incorrect answers. Students received a final score with a maximum of ten points. This eLearning module is notable for giving students additional exposure to appraisal and application of systematic reviews, as much of their previous instruction had focused on primary studies, such as randomized controlled trials. At the midpoint of the clerkship, each student submitted an individual EBM plan based on a clinical encounter they had experienced during the clerkship. The EBM plan included the details of their case, PICO, and preliminary search strategy. Each student’s EBM plan was reviewed by the clerkship director for clinical content and either approved or not approved with revisions required. The librarian reviewed the PICO and search strategy. Students met with the librarian to receive feedback on their PICO and search strategy and discussed any general questions they had about their project. At the end of the course, students submitted a write-up project that included a significant section devoted to EBM. Students introduced their patient’s case, PICO, search strategy, and appraisal of a selected research article, and explained how they could apply the evidence from their article to their patient. Students were 5 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 required to incorporate a minimum of five references to support their arguments and all references were suggested to be no older than five years old. All assessments used a rubric with four levels, from not observed to achieved, with each level receiving a score of 0 to 3 (see appendices for detailed rubric). Table 2: Comparison of EBM instruction between clerkships eLearning Format Individual/Group Mid-clerkship Plan Feedback Format Summative Assessment Format Internal Medicine Gamification Group Mandatory In-person Oral presentation Pediatrics Case-based Individual Mandatory Asynchronous text-based Written OB/GYN Case-based Individual Mandatory In-person Written Collaboration between librarians and clinical faculty A notable feature of this EBM curriculum is the significant interaction between the librarian and clerkship directors. In each clerkship, the librarian worked with the clerkship directors to develop eLearning cases and materials. Materials were selected by the librarian and clerkship directors to address noted deficiencies from students in past course iterations. Additionally, the librarian maintained a continuous dialogue with each clerkship director when assessing students’ mid-clerkship EBM plans. For instance, there were several occasions where clinical aspects, such as if the student’s EBM plan utilized appropriate interventions, affected the assessment of the student’s search. During these occasions, the librarian and clerkship director would discuss these and similar aspects of student plans to offer the most useful and appropriate feedback. Finally, each clerkship director made an effort to incorporate the librarian into the summative assessment phase of each EBM project. For instance, in the Internal Medicine Clerkship, the librarian attended EBM presentation sessions and offered feedback about PICO, searching, and other relevant aspects of the presentations. Finally, the librarian and each clerkship director met annually to review the EBM session for that year and discuss revisions, as needed, for the following year’s sessions. Project data Eligible students for this project were limited to students who completed all three clerkships during the July 2020 to June 2021 clerkship cycle. This time frame was selected because it was the first complete cycle of clerkship rotations for an academic year in which the present curriculum was delivered. Since midpoint feedback data was not originally scored using a rubric, the librarian retrospectively reviewed each midpoint plan using the summative rubric of the respective clerkship. The librarian was blinded to the summative score of each student during retrospective scoring. Both midpoint and summative data were then compiled using Excel and included each student’s score for PICO and searching. Data was also included to indicate the sequential order in which each student completed the three clerkships. Statistics For each clerkship, students’ scores on PICO and searching individually, and then a combined PICO plus searching score were summarized using means and standard deviations (SD). To check for improvement in such scores within a clerkship, mid-clerkship scores were 6 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 compared to the scores at the end of the clerkship using the paired t-test. The effect of the interventions in this study made more sense when using mid-clerkship scores as the pre-test scores so that the actual interventions could be better isolated for analysis. As a sensitivity analysis, each of these comparisons were repeated using the Wilcoxon signed rank test. To check if student scores improved over time; that is, from first to second to third clerkship, the Friedman’s test was used to compare the scores over the sequence of clerkships. If differences in scores over time existed, then the paired t-tests were used to test if scores were different between first and second, first and third, or second and third clerkships. There were no adjustments for multiple tests. As a sensitivity analysis, all the pairwise comparisons above were repeated using the Wilcoxon signed rank test. To check on the effect of clerkship type on student scores, we compared the three clerkships, Internal Medicine, OBGYN, and Pediatrics using Friedman’s test with Wilcoxon signed rank test for multiple comparisons if needed. A p-value of 0.05 or less was considered statistically significant. All analyses were done using IBM-SPSS (version 26, Armonk, NY). Results Overall, 32 student records with complete data from all three clerkships were used for analysis. For their first clerkship, 19 students had Medicine and 13 had OBGYN. For their second clerkship, 10 had Medicine, 12 had OB/GYN and 10 had Pediatrics, and for their third clerkship, 3 students had Medicine, 7 had OB/GYN and 22 had Pediatrics. The EBM instructional model for the Pediatrics Clerkship that is described above was introduced during the second clerkship of the academic year used here, thus no students in this study had pediatrics as their first clerkship. Comparing searching and PICO scores over time In this study, we investigated whether the inclusion of a librarian in EBM instruction could lead to improvements in student scores in EBM over time during their clinical clerkships in Internal Medicine, OB/GYN, and Pediatrics. Our analysis focused on three key variables: PICO scores, searching scores, and overall scores. We compared these scores from mid-clerkship feedback (mid-clerkship score) to the end of clerkship summative EBM assessment (end clerkship score). We observed significant increases in all three variables (See Table 3, column 5 for detailed data). Importantly, these improvements were consistent across all clerkships, regardless of whether it was the student’s first, second, or third clerkship. Sensitivity analysis using the nonparametric alternative confirmed the significance of these findings (See Table 3, column X for p-values). We found significant changes in mid-clerkship scores for search (Table 3, paired t-test P=0.001) and overall (paired t-test P=0.018, Table 3) over time but not for PICO (paired t-test P=0.158, Table 3). However, we have not found any significant changes in end clerkship scores over time (column 7 for p-values, Table 3). There were significant increases in student scores on formulating the PICO, search strategy, and total PICO/search, from mid-clerkship feedback to end-of-clerkship summative EBM assessment. These significant increases were observed in all clerkships regardless of whether the clerkship sequence was the student’s first, second, or third (see Table 3). Sensitivity analysis using the nonparametric alternative resulted in the same significance levels (p-values <0.001). There were significant changes over time in mid-clerkship scores on search and 7 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 overall. Students scored significantly higher on the second and third clerkships as compared to the first clerkship (see Table 3). There were no significant changes in end clerkship scores over time. Table 3: Comparing scores within and between the sequence of clerkships Mid-clerkship Score End Clerkship Score Mid- clerkship Score End Clerkship Score Clerkship Order Variable Mean±sd Mean±sd p-value p-value Over Time p-value Over Time First clerkship PICO 1.92±0.38 2.64±0.48 <0.001 0.158 0.383 Second clerkship PICO 2.11±0.40 2.48±0.47 <0.001 Third clerkship PICO 2.00±0.40 2.52±0.43 <0.001 First clerkship SEARCH 1.31±0.28 2.39±0.52 <0.001 0.001* 0.275 Second clerkship SEARCH 1.63±0.51 2.28±0.55 <0.001 Third clerkship SEARCH 1.66±0.41 2.16±0.63 <0.001 First clerkship Overall 3.23±0.49 5.03±0.74 <0.001 0.018* 0.142 Second clerkship Overall 3.73±0.68 4.77±0.75 <0.001 Third clerkship Overall 3.66±0.53 4.67±0.8 <0.001 *Significant at the 5% level Comparing searching and PICO scores at mid-clerkship and end of clerkship In addition to examining student progression over time through the three clerkships, we wanted to examine if student performance improved within each clerkship. We found that the increase in scores on PICO from mid-clerkship feedback to the end of clerkship summative EBM assessment was significantly higher in the first clerkship compared to the second clerkship (paired t-test P=0.005, Table 4) but not the third clerkship (paired t-test P=0.167, Table 4). Additionally, the increase in search score and the overall score was significantly higher in the first clerkship compared to both the second clerkships (Search, paired t-test P=0.005, Overall, paired t-test P=0.001, Table 4) and the third clerkships (Search, paired t-test P=0.001, Overall, paired t-test P=0.003, Table 4). The increase in scores on PICO from mid-clerkship feedback to end-of-clerkship summative EBM assessment was significantly higher in the first clerkship as compared to the second one but not the third one. As for the increase in search score and overall score, they were significantly higher in the first clerkship than both the second and the third clerkships. See Table 4. 8 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 Table 4: Comparing the increase in scores over the sequence of clerkships First Clerkship Second Clerkship Third Clerkship Mean±sd Mean±sd Mean±sd p-value 1 vs. 2 p-value 1 vs. 3 p-value 2 vs. 3 Increase in PICO Score 0.72±0.49 0.38±0.54 0.52±0.56 0.005* 0.167 0.354 Increase in Search Score 1.08±0.56 0.66±0.50 0.50±0.67 0.005* <0.001* 0.245 Increase in Overall Score 1.80±0.78 1.03±0.67 1.02±0.89 <0.001* 0.003* 0.931 *Significant at the 5% level Comparing student performance by clerkship Students’ scores on PICO in the Internal Medicine Clerkship were significantly higher than those in the other two clerkships (p=0.001). On the other hand, there were no significant differences in student search scores between the three clerkships (p=0.327) (see Table 5). Table 5: Comparing the scores between the different types of clerkships regardless of sequence MED OB/GYN PEDS Mean±sd Mean±sd Mean±sd p-value PICO Score 2.77±0.34 2.47±0.51 2.41±0.45 0.001* Search Score 2.28±0.33 2.38±0.71 2.17±0.6 0.327 Overall Score 5.05±0.41 4.84±0.95 4.58±0.79 0.170 *Significant at the 5% level Discussion Our results in this study align with previous studies that indicate the benefits of including librarians in EBM instruction for medical students.18-21 The librarian in this study provided support for question formulation and database searching in addition to providing feedback to clerkship directors that was used in summative assessment. As many studies have noted, the inclusion of librarians in health science instruction can offer immediate and long-term benefits to students. For example, librarian inclusion allows faculty to design instruction that reflects the team-based modeling that many students will encounter when providing care for patients.22 As other studies have noted, the inclusion of librarians and other similar professionals in the instruction of medical trainees has both affective and cognitive benefits.12,22,23,24 Although not the focus of this study, the increased participation of librarians in EBM instruction does provoke two important questions: what level of training is sufficient and needed for this level of inclusion, and what level of burden is required from the librarian due to this level of course support? 9 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 Another major finding of this study is that implementing a curricular design that utilizes formative assessment of student question formation (PICO) and searching had consistent benefits for students between formative and summative assessment. While it may seem obvious to suggest that feedback leads to improvements in summative assessment, it should be noted that this is one of the first reported EBM courses that incorporated formative assessment as a fundamental element of curricular design. The only EBM course that was identified as using feedback was Atwa and Abdelaziz,25 and information on the methods and nature of the feedback to a take-home assignment in this study is unclear. Many of the studies we surveyed here utilized lectures, workshops, or a combination of the two.26 Our results suggest that while there were initial gains between mid-clerkship feedback and final assessments between a student’s first and second clerkship, this progression was not sustained into the third clerkship. One reason for this could be that students had reached a learning plateau after their second clerkship and any subsequent instruction was generally not as effective for most students after this point in the sequence. Alternatively, this result could indicate that students suffered some level of fatigue from the repletion of EBM instruction. With this said our results suggest that students retained the knowledge and skills needed for EBM over time. Our study also supports the idea that providing repetitive, longitudinal instruction in EBM, despite the risk of fatigue, is beneficial for medical trainees. Limitations Our study revealed that the inclusion of a librarian in EBM instruction positively impacted students’ scores, but also has its limitations. EBM assessments in the Internal Medicine Clerkship were given as a group. Because of this, there is the possibility that a student who would have otherwise scored higher or lower in each student’s group was either aided or hindered by their group members. In subsequent studies it would be interesting to see the effect of a control group that could demonstrate how the lack of a librarian’s involvement might impact students’ scores. This study utilized only one scorer (librarian) to assess students. In future iterations of similar studies, it would be beneficial to have multiple scorers trained in advanced search skills, as this would limit potential bias during assessment. This study focused on student searching and PICO formation; however, it would be interesting to also examine how the instruction detailed in this study affects student evidence assessment and clinical application scores. A final limitation of this study involves the use of PICO as an assessment measure. While PICO has been used as a method in the formulation of clinical questions, it does have its limitations and opponents. For instance, studies have cited PICO’s limited utility for non-therapy questions27, ability to effectively translate to database searches28, and non-superiority to alternative methods for formulating questions.29 Conclusion The results of this study suggest that students benefit from varied teaching methods for EBM instruction. Specifically, the use of formative feedback appears to lead to positive outcomes in summative assessments. Our results reiterate and expand on previous studies that indicate that repeated instructional efforts to teach EBM to trainees are beneficial and result in increased knowledge and skills. Finally, our study advocates for greater inclusion of information professionals in EBM instruction, specifically in the areas of PICO/question formation and searching. 10 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 Corresponding Author Sa’ad Laws, Librarian, Education & Research, sal2018@qatar-med.cornell.edu Ethical Declarations Ethical approvals and consent to participate Because the study involved only minimal risk and all study data was deidentified with all links to original identifiers destroyed, a waiver of informed consent was approved by the Weill Cornell Medicine - Qatar institutional review board on February 10, 2022 (IRB Number: 21-00026). The Declaration of Helsinki was followed in conducting the study. All methods were carried out in accordance with relevant guidelines and regulations. Consent for Publication Not Applicable. Data Availability Statement The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request. Competing Interest The authors have no competing interests to declare. Funding Not applicable. Authors’ Contributions Sa’ad Laws assisted in conceptualization, data curation, methodology, and project administration, and is the lead author of the manuscript. Ziyad Mahfoud assisted with conceptualization, study design, formal analysis, and writing the manuscript. Mai A. Mahmoud, Moune Jabre, and Amal Khidir assisted with conceptualization, study design, methodology, and revision of the manuscript. Acknowledgements The authors would like to thank Qatar Foundation for their support of this research. References 1. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: What it is and what it isn’t. BMJ. 1996 Jan 13;312(7023):71–2. doi:10.1136/bmj.312.7023.71 2. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: A quarter century on. Lancet Lond Engl. 2017 Jul 22;390(10092):415–23. doi:10.1016/S0140-6736(16)31592-6 3. Liasion Committee on Medical Education. Standards, publications, & notification forms [Internet]. 2022 [cited 2022 May 16]. Available from: ttps://lcme.org/publications/ 4. ACGME Program Requirements for Graduate Medical Education in Internal Medicine [Internet]. 2020. Available from: 11 mailto: sal2018@qatar-med.cornell.edu ttps://lcme.org/publications/ Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/140_InternalMedicine_2020.pdf 5. Holloway R, Nesbit K, Bordley D, Noyes K. Teaching and evaluating first and second year medical students’ practice of evidence-based medicine. Med Educ. 2004 Aug;38(8):868–78. doi:10.1111/j.1365-2929.2004.01817.x 6. Menard L, Blevins AE, Trujillo DJ, Lazarus KH. Integrating evidence-based medicine skills into a medical school curriculum: A quantitative outcomes assessment. BMJ Evid-Based Med. 2021 Oct;26(5):249–50. doi:10.1136/bmjebm-2020-111391 7. Ismach RB. Teaching evidence-based medicine to medical students. Acad Emerg Med. 2004 Dec;11(12):e6-10. doi:10.1197/j.aem.2004.08.037 8. Aiyer MK, Dorsch JL. The transformation of an EBM curriculum: A 10-year experience. Med Teach. 2008;30(4):377–83. doi:10.1080/01421590701881632 9. Mai DH, Taylor-Fishwick JS, Sherred-Smith W, Pang A, Yaworsky J, Whitty S, et al. Peer-developed modules on basic biostatistics and evidence-based medicine principles for undergraduate medical education. MedEdPORTAL. 2020 Nov 24;16:11026. doi:10.15766/mep_2374-8265.11026 10. Anderson CR, Haydek J, Golub L, Leong T, Smith DT, Liebzeit J, et al. Practical evidence-based medicine at the student-to-physician transition: Effectiveness of an undergraduate medical education capstone course. Med Sci Educ. 2020 Jun;30(2):885–90. doi:10.1007/s40670-020-00970-9 11. Weberschock TB, Ginn TC, Reinhold J, Strametz R, Krug D, Bergold M, et al. Change in knowledge and skills of Year 3 undergraduates in evidence-based medicine seminars. Med Educ. 2005 Jul;39(7):665–71. doi:10.1111/j.1365-2929.2005.02191.x 12. Ilic D, Maloney S. Methods of teaching medical trainees evidence-based medicine: A systematic review. Med Educ. 2014;48(2):124–35. doi:10.1111/medu.12288 13. Boykan R, Jacobson RM. The role of librarians in teaching evidence-based medicine to pediatric residents: A survey of pediatric residency program directors. J Med Libr Assoc. 2017 Oct 2;105(4):355–60. doi:10.5195/jmla.2017.178 14. Ma KSK, Chang HC, Krupat E. Teaching evidence-based medicine with electronic databases for preclinical education. Adv Physiol Educ. 2021 Dec 1;45(4):849–55. doi:10.1152/advan.00057.2021 15. Swanberg SM, Mi M, Engwall K. An integrated, case-based approach to teaching medical students how to locate the best available evidence for clinical care. MedEdPORTAL. 2017 Jan 19;13:10531. doi:10.15766/mep_2374-8265.10531 16. Minuti A, Sorensen K, Schwartz R, King WS, Glassman NR, Habousha RG. Librarians flip for students: Teaching searching skills to medical students using a flipped classroom 12 https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/140_InternalMedicine_2020.pdf https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/140_InternalMedicine_2020.pdf Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 approach. Med Ref Serv Q. 2018 Jun;37(2):119–31. doi:10.1080/02763869.2018.1439184 17. Dorsch JL, Aiyer MK, Meyer LE. Impact of an evidence-based medicine curriculum on medical students’ attitudes and skills. J Med Libr Assoc. 2004 Oct;92(4):397–406 18. Haley J, McCall RC, Zomorodi M, de Saxe Zerdan L, Moreton B, Richardson L. Interprofessional collaboration between health sciences librarians and health professions faculty to implement a book club discussion for incoming students. J Med Libr Assoc. 2019 Jul;107(3):403–10. doi:10.5195/jmla.2019.563 19. Gill KD, Parker R. An untapped resource? Opportunities for faculty-librarian collaboration to enhance drug information resource utilization in pharmacy education. J Med Libr Assoc. 2022 Oct 1;110(4):478–84. doi:10.5195/jmla.2022.1486 20. Muellenbach JM, Houk KM, E Thimons D, Rodriguez B. Integrating information literacy and evidence-based medicine content within a new school of medicine curriculum: Process and outcome. Med Ref Serv Q. 2018;37(2):198–206. doi:10.1080/02763869.2018.1439225 21. Boruff JT, Thomas A. Integrating evidence-based practice and information literacy skills in teaching physical and occupational therapy students. Health Inf Libr J. 2011 Dec;28(4):264–72. doi:10.1111/j.1471-1842.2011.00953.x 22. Brian R, Orlov N, Werner D, Martin SK, Arora VM, Alkureishi M. Evaluating the impact of clinical librarians on clinical questions during inpatient rounds. J Med Libr Assoc. 2018 Apr;106(2):175–83. doi:10.5195/jmla.2018.254 23. Tahmasebi M, Adibi P, Zare-Farashbandi F, Papi A, Rahimi A. The educational role of clinical informationist on improving clinical education among medical students: Based on Kirkpatrick model. J Educ Health Promot. 2020;9:28. doi:10.4103/jehp.jehp_439_19 24. Krom ZR, Batten J, Bautista C. A unique collaborative nursing evidence-based practice initiative using the Iowa model: a clinical nurse specialist, a health science librarian, and a staff nurse’s success story. Clin Nurse Spec CNS. 2010 Apr;24(2):54–9. doi:10.1097/NUR.0b013e3181cf5537 25. Atwa H, Abdelaziz A. Evidence-based medicine (EBM) for undergraduate medical students: A six-step, integrative approach. Med Teach. 2017 Apr;39(sup1):S27–32. doi:10.1080/0142159X.2016.1254750 26. Al Shahrani AS. Development and evaluation of an evidence-based medicine module in the undergraduate medical curriculum. BMC Med Educ. 2020 Aug 6;20(1):256. doi:10.1186/s12909-020-02181-7. 27. Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc. 2006;2006:359-363. 28. Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: A systematic review. J Med Libr 13 Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024 Assoc. 2018;106(4):420-431. doi:10.5195/jmla.2018.345 29. Kloda LA, Boruff JT, Cavalcante AS. A comparison of patient, intervention, comparison, outcome (PICO) to a new, alternative clinical question framework for search skills, search results, and self-efficacy: A randomized controlled trial. J Med Libr Assoc. 2020;108(2):185-194. doi:10.5195/jmla.2020.739 14