




































Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

Librarian and Clinical Faculty Collaborative Use of Formative
Assessment in a Longitudinal Evidence-Based Medicine Curriculum

Sa’ad Laws, MA, MLIS*a, Mai A. Mahmoud, MBBSb, Moune Jabre, MDc, Amal Khidir,
MBBSd, Ziyad Mahfoud, PhDe

aCorresponding Author, Health Sciences Library, Weill Cornell Medicine – Qatar, Doha,
Qatar, sal2018@qatar-med.cornell.edu

bAssociate Professor of Teaching in Medicine/Assistant Dean for Faculty Affairs, Department
of Medical Education, Weill Cornell Medicine – Qatar, Doha, Qatar

cAssistant Professor of Clinical Obstetrics and Gynecology/Assistant Dean for Academic
Affairs, Department of Medical Education, Weill Cornell Medicine – Qatar, Doha, Qatar

dAssociate Professor of Pediatrics, Department of Medical Education, Weill Cornell Medicine
– Qatar, Doha, Qatar

eProfessor of Research in Population Health Sciences, Department of Medical Education,
Weill Cornell Medicine – Qatar, Doha, Qatar, Division of Epidemiology, Department of
Population Health Sciences, Weill Cornell Medicine, New York, New York

Cite as: Laws S, Mahmoud MA, Jabre M, Khidir A, and Mahfoud Z. Librarian and Clinical
Faculty Collaborative Use of Formative Assessment in a Longitudinal Evidence-Based
Medicine Curriculum. Hypothesis. 2024;36(2). doi:10.18060/28048

Laws, Mahmoud, Jabre, Khidir, Mahfoud. All works in Hypothesis are
licensed under a CC BY-NC 4.0 DEED Attribution-NonCommercial 4.0 International.
Authors own copyright of their articles appearing in Hypothesis. Readers may copy articles
without permission of the copyright owner(s), as long as the author(s) are acknowledged in
the copy, and the copy is used for educational, not-for-profit purposes. For any other use of
articles, please contact the copyright owner(s).

1

mailto:sal2018@qatar-med.cornell.edu
https://creativecommons.org/licenses/by-nc/4.0/


Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

Background The authors developed a longitudinal curriculum for teaching third- and
fourth-year undergraduate medical students evidence-based medicine (EBM). This curriculum
involved substantial librarian involvement, the use of formative assessment as a teaching
method, and progressive repetition of skills.

Methods Students in three clinical clerkships (Medicine, Pediatrics, OB/GYN) completed
EBM assignments based on real-world scenarios in this quasi-experimental study. Each
clerkship required students to submit an EBM plan, from which they received directed
feedback from both the clerkship director and a librarian after submitting preliminary EBM
plans. This study tracked student PICO and search submissions to determine if repeated
exposure to EBM feedback resulted in improved summative assessments, both in isolation
and longitudinally.

Results Students’ PICO and searching performance improved between mid-clerkship
formative feedback and end-clerkship summative assessment in all three clerkships using
rubrics developed by the authors for each clerkship derived from existing literature. When
examining student performance sequentially over three clerkships, there was significant
improvement between the first and second clerkships, but this did not carry into the third
clerkship.

Discussion Our findings suggest that the significant inclusion of a librarian and feedback
appears to have positive effects on student performance. While it may seem obvious that
feedback results in improved outcomes, this method doesn’t appear widespread in medical
education. Repetition, while not having a lasting increase in performance, may still be
warranted to increase exposure to authentic cases and evidence types.

Introduction
In medical education, enhancing the ability to make well-informed clinical decisions is crucial
to improving patient care. Central to this decision-making process is evidence-based medicine
(EBM), a methodology that integrates clinical expertise, patient values, and the best available
evidence to guide healthcare decisions. EBM is traditionally described as “the conscientious,
explicit, and judicious use of current best evidence in making decisions about the care of
individual patients.”1 The use of EBM requires a holistic understanding of question formation,
searching, critical appraisal, and clinical application.2 The acquisition of these skills is a
fundamental aspect of healthcare provider education and a lifelong learning skill that has been
identified by multiple medical education accreditation organizations at both undergraduate
and graduate levels.3,4 While medical education has evolved to include EBM as a fundamental
component, there remains a pressing need to ensure that students grasp the theoretical aspects
and develop the skills to apply EBM in real-world clinical scenarios.

In this context, several studies have examined different methods of teaching EBM to medical
trainees. An important area of exploration in these studies is the timing, duration, and
intensity of EBM instruction within the undergraduate curriculum. Some studies have focused
on implementing EBM instruction early during preclinical years,5,6 while others have opted to
delay until clinical-based years.7,8 A common rationale in studies that advocated for delay was
implementation at a time when students were interacting with patients in the clinical setting to
give more context to EBM learning. In another case, faculty have utilized other approaches,

2



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

such as the use of peer instructors, a capstone course for fourth-year undergraduate medical
students, and dedicated seminars.9-11 Although these studies include an array of teaching
methods, it is notable that none were conducted in a clinical setting. Both clinical and
non-clinical-based instruction demonstrated an improvement in skills and attitudes compared
to pre-instruction. Despite the gains noted in several studies, there does not appear to be a
consensus about the best methods for instructing trainees in EBM. Ilic and Maloney note that
most instructional methods show some level of improvement in competencies; there is no
clear benefit of one method over others.12

A notable trend in many studies of EBM in both graduate and undergraduate medical
education has been the inclusion of librarians in various roles. While it has been noted that
librarians are an underutilized resource for EBM instruction,13 several studies have shown that
librarians are adept in instructing students on question formulation and the best methods for
searching clinical databases for evidence.14,15 In one example of librarian involvement, Minuti
et al. described a flipped-classroom module that used readings, video, and exercise in support
of sessions with first-year students that focused on PICO formulation and searching and later
with second-year students where additional searching techniques were taught and clinical
faculty followed with critical appraisal instruction.16 Minuti et al. noted that librarian
involvement was significant for assisting in meeting accreditation requirements and the
positive impressions that faculty had after the course. Another notable example of librarian
integration in EBM instruction is from a study of trainees’ skills and attitudes in an
emergency medicine course.17 This course incorporated significant levels of librarian
involvement and emphasis on searching skills. Trainees reported significant improvements in
searching-related skills, but they noted that additional instruction is necessary for sustained
application and reinforcement.

The present study is a further exploration of the benefits of both varied methods in teaching
EBM to medical trainees as well as the significant incorporation of librarians within EBM
instruction. Both of these areas of inquiry for teaching EBM are unresolved in the current
literature and vital for the future development of EBM instruction to medical trainees. With
this in mind, the objectives of this study are to evaluate the effectiveness of our EBM
curriculum within each clerkship, as well as longitudinally between the first and third
clerkships.

Methods
Study population and characteristics
This quasi-experiment focused on EBM instruction within three clinical clerkships: Medicine,
Pediatrics, and Obstetrics and Gynecology (OB/GYN). The study was conducted at Weill
Cornell Medicine – Qatar (WMC-Q) in Doha, Qatar. WMC-Q follows a four-year,
American-based medical curriculum from the main Weill Cornell Medicine campus in New
York, USA. These three clerkships were specifically chosen because they are the only three
that have explicit EBM curricular elements. Data from 32 students who completed all three
clerkships were analyzed in this study. Approval for this study was granted by the WCM-Q
institutional review board on February 10, 2022 (21-00026). Because this was a retrospective
analysis, the study team requested and was granted a waiver of informed consent of subjects.
Before entering these clerkships, students received preliminary EBM instruction during their
first year of preclinical education. In this context, students took a course on clinical

3



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

epidemiology and research methods, followed by a 10-hour, face-to-face, course utilizing
lectures and labs to emphasize EBM fundamentals, question formulation, searching, and
critical appraisal. While the EBM curriculum of the three clerkships in this study followed a
similar track, each clerkship employed slight variations to attain their EBM objectives. The
distribution and progression of students through the three clerkships is shown in Table 1. It is
noteworthy that the EBM instructional model introduced in the Pediatrics Clerkship was
implemented during the second clerkship of the academic year. As a result, none of the
students included in this study had pediatrics as their initial clerkship.

Table 1: Student progression through clerkship with EBM instructions

OB/GYN Pediatrics Internal Medicine
First Clerkship 13 0 9

Second Clerkship 12 10 10
Third Clerkship 7 22 3

Internal Medicine Clerkship
Students in the Internal Medicine Clerkship began the course by completing an asynchronous
eLearning module that reviewed EBM fundamentals. The eLearning module in the Internal
Medicine Clerkship is a gamified experience, mimicking the structure of the TV game show
Jeopardy, where students select from four levels of questions across five categories that
increase in difficulty and point value. The module was internally produced using standard web
development tools, such as HTML and CSS. The eLearning module’s content included 20
total questions on PICO formation, searching techniques, evidence appraisal, and EBM
question type (i.e., Harm, Therapy, Diagnosis, Prognosis). Each category included four
questions that escalated in the level of difficulty and point value for correct answers (range 2-8
points). Each question offered feedback for both correct and incorrect answers. The main
objective of the eLearning module is to refresh students on EBM fundamentals and highlight
potential gaps in EBM comprehension. Students received a final score, with a maximum of
100 points possible, and the module concluded with a leaderboard of the top score for every
iteration of the course. All eLearning module data is available to both the librarian and
clerkship director.

At the midpoint of the clerkship, students were grouped into teams of three to four students by
faculty and each team was required to submit a group EBM plan. The content of the EBM
plan must be drawn from the clinical experience of at least one of the students in the team.
The plan included the case details, PICO, and search strategy. After submission, each EBM
plan underwent two reviews. First, the clerkship director reviewed the plan for clinical
viability, either approving it or requesting revisions. In parallel, the librarian evaluated each
EBM plan’s PICO and search strategy. Additionally, each team met with the librarian to
receive feedback on their PICO and search strategy and discussed potential amendments for
various aspects of each team’s EBM plan. The summative assessment for EBM in the Internal
Medicine Clerkship was a team-based oral presentation. Teams had to provide details about
their patient’s case, PICO, search strategy, appraisal of a selected research article, and how
they could apply the evidence from their selected article to their patient. After the
presentation, fellow students, the librarian, and the director were invited to ask the presenting
team questions about their presentation. The librarian assessed the PICO and search strategy,
while the director evaluated the clinical and team-based components of each team’s EBM

4



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

presentation. All assessments used a rubric with four levels, from inadequate to exceeds
expectation, with each level receiving a score of 0 to 5 (see appendices for detailed rubric).

Pediatrics Clerkship
Students in the Pediatrics Clerkship started the course with a required short asynchronous
eLearning module that reviewed key aspects of EBM, such as searching and constructing a
PICO. Students completed four “knowledge check” questions at various points in the tutorials
to verify content comprehension. Each question offered feedback for both correct and
incorrect answers. All students’ responses were available to both the librarian and clerkship
director. At the midpoint of the clerkship, each student submitted an individual EBM plan
based on a clinical encounter they had experienced during the clerkship. The EBM plan
included the details of their case, PICO, and preliminary search strategy. Each EBM plan was
reviewed by the clerkship director for clinical content and relevance. The clerkship director
then either approved each plan or asked the student to submit a revised plan before the student
could proceed. The librarian reviewed the PICO and search strategy. Both the clerkship
director’s and the librarian’s feedback were sent to the student via email. Although no
individual meetings were required, students were highly encouraged to set up individual
meetings with either the librarian or clerkship director, if they felt this was needed. At the end
of the clerkship, students submitted a written EBM project. Student projects introduced their
patient’s case, PICO, search strategy, and appraisal of a selected research article, and
explained how they could apply the evidence from their article to their patient. The librarian
reviewed each EBM project, provided comments on the PICO and search strategy, and then
forwarded these to the clerkship director, who completed the grading of clinical and appraisal
content. All assessments used a rubric with four levels, from inadequate to exceeds
expectation, with each level receiving a score of 0 to 3 (see appendices for detailed rubric).

Obstetrics and Gynecology (OB/GYN) Clerkship
Students started the OB/GYN Clerkship by completing an asynchronous eLearning module
that reviewed EBM fundamentals ranging from question formation, searching, and appraisal,
to application. At the start of the eLearning module, students were presented with an
OB/GYN-based case. Students then worked through ten questions from three domains,
searching/PICO, appraisal, and application, based on the case and reviewed a systematic
review relevant to the case. Each question offered feedback for both correct and incorrect
answers. Students received a final score with a maximum of ten points. This eLearning
module is notable for giving students additional exposure to appraisal and application of
systematic reviews, as much of their previous instruction had focused on primary studies,
such as randomized controlled trials.

At the midpoint of the clerkship, each student submitted an individual EBM plan based on a
clinical encounter they had experienced during the clerkship. The EBM plan included the
details of their case, PICO, and preliminary search strategy. Each student’s EBM plan was
reviewed by the clerkship director for clinical content and either approved or not approved
with revisions required. The librarian reviewed the PICO and search strategy. Students met
with the librarian to receive feedback on their PICO and search strategy and discussed any
general questions they had about their project. At the end of the course, students submitted a
write-up project that included a significant section devoted to EBM. Students introduced their
patient’s case, PICO, search strategy, and appraisal of a selected research article, and
explained how they could apply the evidence from their article to their patient. Students were

5



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

required to incorporate a minimum of five references to support their arguments and all
references were suggested to be no older than five years old. All assessments used a rubric
with four levels, from not observed to achieved, with each level receiving a score of 0 to 3
(see appendices for detailed rubric).

Table 2: Comparison of EBM instruction between clerkships

eLearning
Format Individual/Group Mid-clerkship

Plan
Feedback
Format

Summative
Assessment

Format
Internal

Medicine Gamification Group Mandatory In-person
Oral

presentation

Pediatrics Case-based Individual Mandatory
Asynchronous

text-based Written

OB/GYN Case-based Individual Mandatory In-person Written

Collaboration between librarians and clinical faculty
A notable feature of this EBM curriculum is the significant interaction between the librarian
and clerkship directors. In each clerkship, the librarian worked with the clerkship directors to
develop eLearning cases and materials. Materials were selected by the librarian and clerkship
directors to address noted deficiencies from students in past course iterations. Additionally,
the librarian maintained a continuous dialogue with each clerkship director when assessing
students’ mid-clerkship EBM plans. For instance, there were several occasions where clinical
aspects, such as if the student’s EBM plan utilized appropriate interventions, affected the
assessment of the student’s search. During these occasions, the librarian and clerkship
director would discuss these and similar aspects of student plans to offer the most useful and
appropriate feedback. Finally, each clerkship director made an effort to incorporate the
librarian into the summative assessment phase of each EBM project. For instance, in the
Internal Medicine Clerkship, the librarian attended EBM presentation sessions and offered
feedback about PICO, searching, and other relevant aspects of the presentations. Finally, the
librarian and each clerkship director met annually to review the EBM session for that year and
discuss revisions, as needed, for the following year’s sessions.

Project data
Eligible students for this project were limited to students who completed all three clerkships
during the July 2020 to June 2021 clerkship cycle. This time frame was selected because it
was the first complete cycle of clerkship rotations for an academic year in which the present
curriculum was delivered. Since midpoint feedback data was not originally scored using a
rubric, the librarian retrospectively reviewed each midpoint plan using the summative rubric
of the respective clerkship. The librarian was blinded to the summative score of each student
during retrospective scoring. Both midpoint and summative data were then compiled using
Excel and included each student’s score for PICO and searching. Data was also included to
indicate the sequential order in which each student completed the three clerkships.

Statistics
For each clerkship, students’ scores on PICO and searching individually, and then a combined
PICO plus searching score were summarized using means and standard deviations (SD). To
check for improvement in such scores within a clerkship, mid-clerkship scores were

6



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

compared to the scores at the end of the clerkship using the paired t-test. The effect of the
interventions in this study made more sense when using mid-clerkship scores as the pre-test
scores so that the actual interventions could be better isolated for analysis. As a sensitivity
analysis, each of these comparisons were repeated using the Wilcoxon signed rank test. To
check if student scores improved over time; that is, from first to second to third clerkship, the
Friedman’s test was used to compare the scores over the sequence of clerkships. If differences
in scores over time existed, then the paired t-tests were used to test if scores were different
between first and second, first and third, or second and third clerkships. There were no
adjustments for multiple tests. As a sensitivity analysis, all the pairwise comparisons above
were repeated using the Wilcoxon signed rank test. To check on the effect of clerkship type on
student scores, we compared the three clerkships, Internal Medicine, OBGYN, and Pediatrics
using Friedman’s test with Wilcoxon signed rank test for multiple comparisons if needed. A
p-value of 0.05 or less was considered statistically significant. All analyses were done using
IBM-SPSS (version 26, Armonk, NY).

Results
Overall, 32 student records with complete data from all three clerkships were used for
analysis. For their first clerkship, 19 students had Medicine and 13 had OBGYN. For their
second clerkship, 10 had Medicine, 12 had OB/GYN and 10 had Pediatrics, and for their third
clerkship, 3 students had Medicine, 7 had OB/GYN and 22 had Pediatrics. The EBM
instructional model for the Pediatrics Clerkship that is described above was introduced during
the second clerkship of the academic year used here, thus no students in this study had
pediatrics as their first clerkship.

Comparing searching and PICO scores over time
In this study, we investigated whether the inclusion of a librarian in EBM instruction could
lead to improvements in student scores in EBM over time during their clinical clerkships in
Internal Medicine, OB/GYN, and Pediatrics. Our analysis focused on three key variables:
PICO scores, searching scores, and overall scores. We compared these scores from
mid-clerkship feedback (mid-clerkship score) to the end of clerkship summative EBM
assessment (end clerkship score). We observed significant increases in all three variables (See
Table 3, column 5 for detailed data). Importantly, these improvements were consistent across
all clerkships, regardless of whether it was the student’s first, second, or third clerkship.
Sensitivity analysis using the nonparametric alternative confirmed the significance of these
findings (See Table 3, column X for p-values).

We found significant changes in mid-clerkship scores for search (Table 3, paired t-test
P=0.001) and overall (paired t-test P=0.018, Table 3) over time but not for PICO (paired t-test
P=0.158, Table 3). However, we have not found any significant changes in end clerkship
scores over time (column 7 for p-values, Table 3).

There were significant increases in student scores on formulating the PICO, search strategy,
and total PICO/search, from mid-clerkship feedback to end-of-clerkship summative EBM
assessment. These significant increases were observed in all clerkships regardless of whether
the clerkship sequence was the student’s first, second, or third (see Table 3). Sensitivity
analysis using the nonparametric alternative resulted in the same significance levels (p-values
<0.001). There were significant changes over time in mid-clerkship scores on search and

7



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

overall. Students scored significantly higher on the second and third clerkships as compared
to the first clerkship (see Table 3). There were no significant changes in end clerkship scores
over time.

Table 3: Comparing scores within and between the sequence of clerkships

Mid-clerkship
Score

End
Clerkship

Score

Mid-
clerkship

Score

End
Clerkship

Score

Clerkship Order Variable Mean±sd Mean±sd p-value
p-value
Over Time

p-value
Over Time

First clerkship PICO 1.92±0.38 2.64±0.48 <0.001 0.158 0.383
Second clerkship PICO 2.11±0.40 2.48±0.47 <0.001
Third clerkship PICO 2.00±0.40 2.52±0.43 <0.001
First clerkship SEARCH 1.31±0.28 2.39±0.52 <0.001 0.001* 0.275
Second clerkship SEARCH 1.63±0.51 2.28±0.55 <0.001
Third clerkship SEARCH 1.66±0.41 2.16±0.63 <0.001
First clerkship Overall 3.23±0.49 5.03±0.74 <0.001 0.018* 0.142
Second clerkship Overall 3.73±0.68 4.77±0.75 <0.001
Third clerkship Overall 3.66±0.53 4.67±0.8 <0.001
*Significant at the 5% level

Comparing searching and PICO scores at mid-clerkship and end of clerkship
In addition to examining student progression over time through the three clerkships, we
wanted to examine if student performance improved within each clerkship.

We found that the increase in scores on PICO from mid-clerkship feedback to the end of
clerkship summative EBM assessment was significantly higher in the first clerkship compared
to the second clerkship (paired t-test P=0.005, Table 4) but not the third clerkship (paired
t-test P=0.167, Table 4). Additionally, the increase in search score and the overall score was
significantly higher in the first clerkship compared to both the second clerkships (Search,
paired t-test P=0.005, Overall, paired t-test P=0.001, Table 4) and the third clerkships
(Search, paired t-test P=0.001, Overall, paired t-test P=0.003, Table 4).

The increase in scores on PICO from mid-clerkship feedback to end-of-clerkship summative
EBM assessment was significantly higher in the first clerkship as compared to the second one
but not the third one. As for the increase in search score and overall score, they were
significantly higher in the first clerkship than both the second and the third clerkships. See
Table 4.

8



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

Table 4: Comparing the increase in scores over the sequence of clerkships

First
Clerkship

Second
Clerkship

Third
Clerkship

Mean±sd Mean±sd Mean±sd
p-value 1

vs. 2
p-value 1

vs. 3
p-value 2

vs. 3
Increase in PICO
Score 0.72±0.49 0.38±0.54 0.52±0.56 0.005* 0.167 0.354

Increase in Search
Score 1.08±0.56 0.66±0.50 0.50±0.67 0.005* <0.001* 0.245

Increase in Overall
Score 1.80±0.78 1.03±0.67 1.02±0.89 <0.001* 0.003* 0.931
*Significant at the 5% level

Comparing student performance by clerkship
Students’ scores on PICO in the Internal Medicine Clerkship were significantly higher than
those in the other two clerkships (p=0.001). On the other hand, there were no significant
differences in student search scores between the three clerkships (p=0.327) (see Table 5).

Table 5: Comparing the scores between the different types of clerkships regardless of
sequence

MED OB/GYN PEDS
Mean±sd Mean±sd Mean±sd p-value

PICO
Score 2.77±0.34 2.47±0.51 2.41±0.45 0.001*

Search
Score 2.28±0.33 2.38±0.71 2.17±0.6 0.327

Overall
Score 5.05±0.41 4.84±0.95 4.58±0.79 0.170
*Significant at the 5% level

Discussion
Our results in this study align with previous studies that indicate the benefits of including
librarians in EBM instruction for medical students.18-21 The librarian in this study provided
support for question formulation and database searching in addition to providing feedback to
clerkship directors that was used in summative assessment. As many studies have noted, the
inclusion of librarians in health science instruction can offer immediate and long-term benefits
to students. For example, librarian inclusion allows faculty to design instruction that reflects
the team-based modeling that many students will encounter when providing care for
patients.22 As other studies have noted, the inclusion of librarians and other similar
professionals in the instruction of medical trainees has both affective and cognitive
benefits.12,22,23,24 Although not the focus of this study, the increased participation of librarians
in EBM instruction does provoke two important questions: what level of training is sufficient
and needed for this level of inclusion, and what level of burden is required from the librarian
due to this level of course support?

9



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

Another major finding of this study is that implementing a curricular design that utilizes
formative assessment of student question formation (PICO) and searching had consistent
benefits for students between formative and summative assessment. While it may seem
obvious to suggest that feedback leads to improvements in summative assessment, it should
be noted that this is one of the first reported EBM courses that incorporated formative
assessment as a fundamental element of curricular design. The only EBM course that was
identified as using feedback was Atwa and Abdelaziz,25 and information on the methods and
nature of the feedback to a take-home assignment in this study is unclear. Many of the studies
we surveyed here utilized lectures, workshops, or a combination of the two.26

Our results suggest that while there were initial gains between mid-clerkship feedback and
final assessments between a student’s first and second clerkship, this progression was not
sustained into the third clerkship. One reason for this could be that students had reached a
learning plateau after their second clerkship and any subsequent instruction was generally not
as effective for most students after this point in the sequence. Alternatively, this result could
indicate that students suffered some level of fatigue from the repletion of EBM instruction.
With this said our results suggest that students retained the knowledge and skills needed for
EBM over time. Our study also supports the idea that providing repetitive, longitudinal
instruction in EBM, despite the risk of fatigue, is beneficial for medical trainees.

Limitations
Our study revealed that the inclusion of a librarian in EBM instruction positively impacted
students’ scores, but also has its limitations. EBM assessments in the Internal Medicine
Clerkship were given as a group. Because of this, there is the possibility that a student who
would have otherwise scored higher or lower in each student’s group was either aided or
hindered by their group members. In subsequent studies it would be interesting to see the
effect of a control group that could demonstrate how the lack of a librarian’s involvement
might impact students’ scores. This study utilized only one scorer (librarian) to assess
students. In future iterations of similar studies, it would be beneficial to have multiple scorers
trained in advanced search skills, as this would limit potential bias during assessment. This
study focused on student searching and PICO formation; however, it would be interesting to
also examine how the instruction detailed in this study affects student evidence assessment
and clinical application scores. A final limitation of this study involves the use of PICO as an
assessment measure. While PICO has been used as a method in the formulation of clinical
questions, it does have its limitations and opponents. For instance, studies have cited PICO’s
limited utility for non-therapy questions27, ability to effectively translate to database
searches28, and non-superiority to alternative methods for formulating questions.29

Conclusion
The results of this study suggest that students benefit from varied teaching methods for EBM
instruction. Specifically, the use of formative feedback appears to lead to positive outcomes in
summative assessments. Our results reiterate and expand on previous studies that indicate that
repeated instructional efforts to teach EBM to trainees are beneficial and result in increased
knowledge and skills. Finally, our study advocates for greater inclusion of information
professionals in EBM instruction, specifically in the areas of PICO/question formation and
searching.

10



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

Corresponding Author
Sa’ad Laws, Librarian, Education & Research, sal2018@qatar-med.cornell.edu

Ethical Declarations
Ethical approvals and consent to participate
Because the study involved only minimal risk and all study data was deidentified with all
links to original identifiers destroyed, a waiver of informed consent was approved by the Weill
Cornell Medicine - Qatar institutional review board on February 10, 2022 (IRB Number:
21-00026). The Declaration of Helsinki was followed in conducting the study. All methods
were carried out in accordance with relevant guidelines and regulations.
Consent for Publication
Not Applicable.

Data Availability Statement
The datasets used and/or analyzed during the current study are available from the
corresponding author upon reasonable request.
Competing Interest
The authors have no competing interests to declare.
Funding
Not applicable.

Authors’ Contributions
Sa’ad Laws assisted in conceptualization, data curation, methodology, and project
administration, and is the lead author of the manuscript. Ziyad Mahfoud assisted with
conceptualization, study design, formal analysis, and writing the manuscript. Mai A.
Mahmoud, Moune Jabre, and Amal Khidir assisted with conceptualization, study design,
methodology, and revision of the manuscript.

Acknowledgements
The authors would like to thank Qatar Foundation for their support of this research.

References
1. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based
medicine: What it is and what it isn’t. BMJ. 1996 Jan 13;312(7023):71–2.
doi:10.1136/bmj.312.7023.71

2. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: A quarter century on.
Lancet Lond Engl. 2017 Jul 22;390(10092):415–23. doi:10.1016/S0140-6736(16)31592-6

3. Liasion Committee on Medical Education. Standards, publications, & notification forms
[Internet]. 2022 [cited 2022 May 16]. Available from: ttps://lcme.org/publications/

4. ACGME Program Requirements for Graduate Medical Education in Internal Medicine
[Internet]. 2020. Available from:

11

mailto: sal2018@qatar-med.cornell.edu
ttps://lcme.org/publications/ 


Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/140_InternalMedicine_2020.pdf

5. Holloway R, Nesbit K, Bordley D, Noyes K. Teaching and evaluating first and second year
medical students’ practice of evidence-based medicine. Med Educ. 2004 Aug;38(8):868–78.
doi:10.1111/j.1365-2929.2004.01817.x

6. Menard L, Blevins AE, Trujillo DJ, Lazarus KH. Integrating evidence-based medicine
skills into a medical school curriculum: A quantitative outcomes assessment. BMJ
Evid-Based Med. 2021 Oct;26(5):249–50. doi:10.1136/bmjebm-2020-111391

7. Ismach RB. Teaching evidence-based medicine to medical students. Acad Emerg Med.
2004 Dec;11(12):e6-10. doi:10.1197/j.aem.2004.08.037

8. Aiyer MK, Dorsch JL. The transformation of an EBM curriculum: A 10-year experience.
Med Teach. 2008;30(4):377–83. doi:10.1080/01421590701881632

9. Mai DH, Taylor-Fishwick JS, Sherred-Smith W, Pang A, Yaworsky J, Whitty S, et al.
Peer-developed modules on basic biostatistics and evidence-based medicine principles for
undergraduate medical education. MedEdPORTAL. 2020 Nov 24;16:11026.
doi:10.15766/mep_2374-8265.11026

10. Anderson CR, Haydek J, Golub L, Leong T, Smith DT, Liebzeit J, et al. Practical
evidence-based medicine at the student-to-physician transition: Effectiveness of an
undergraduate medical education capstone course. Med Sci Educ. 2020 Jun;30(2):885–90.
doi:10.1007/s40670-020-00970-9

11. Weberschock TB, Ginn TC, Reinhold J, Strametz R, Krug D, Bergold M, et al. Change in
knowledge and skills of Year 3 undergraduates in evidence-based medicine seminars. Med
Educ. 2005 Jul;39(7):665–71. doi:10.1111/j.1365-2929.2005.02191.x

12. Ilic D, Maloney S. Methods of teaching medical trainees evidence-based medicine: A
systematic review. Med Educ. 2014;48(2):124–35. doi:10.1111/medu.12288

13. Boykan R, Jacobson RM. The role of librarians in teaching evidence-based medicine to
pediatric residents: A survey of pediatric residency program directors. J Med Libr Assoc.
2017 Oct 2;105(4):355–60. doi:10.5195/jmla.2017.178

14. Ma KSK, Chang HC, Krupat E. Teaching evidence-based medicine with electronic
databases for preclinical education. Adv Physiol Educ. 2021 Dec 1;45(4):849–55.
doi:10.1152/advan.00057.2021

15. Swanberg SM, Mi M, Engwall K. An integrated, case-based approach to teaching medical
students how to locate the best available evidence for clinical care. MedEdPORTAL. 2017 Jan
19;13:10531. doi:10.15766/mep_2374-8265.10531

16. Minuti A, Sorensen K, Schwartz R, King WS, Glassman NR, Habousha RG. Librarians
flip for students: Teaching searching skills to medical students using a flipped classroom

12

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/140_InternalMedicine_2020.pdf 
https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/140_InternalMedicine_2020.pdf 


Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

approach. Med Ref Serv Q. 2018 Jun;37(2):119–31. doi:10.1080/02763869.2018.1439184

17. Dorsch JL, Aiyer MK, Meyer LE. Impact of an evidence-based medicine curriculum on
medical students’ attitudes and skills. J Med Libr Assoc. 2004 Oct;92(4):397–406

18. Haley J, McCall RC, Zomorodi M, de Saxe Zerdan L, Moreton B, Richardson L.
Interprofessional collaboration between health sciences librarians and health professions
faculty to implement a book club discussion for incoming students. J Med Libr Assoc. 2019
Jul;107(3):403–10. doi:10.5195/jmla.2019.563

19. Gill KD, Parker R. An untapped resource? Opportunities for faculty-librarian
collaboration to enhance drug information resource utilization in pharmacy education. J Med
Libr Assoc. 2022 Oct 1;110(4):478–84. doi:10.5195/jmla.2022.1486

20. Muellenbach JM, Houk KM, E Thimons D, Rodriguez B. Integrating information literacy
and evidence-based medicine content within a new school of medicine curriculum: Process
and outcome. Med Ref Serv Q. 2018;37(2):198–206. doi:10.1080/02763869.2018.1439225

21. Boruff JT, Thomas A. Integrating evidence-based practice and information literacy skills
in teaching physical and occupational therapy students. Health Inf Libr J. 2011
Dec;28(4):264–72. doi:10.1111/j.1471-1842.2011.00953.x

22. Brian R, Orlov N, Werner D, Martin SK, Arora VM, Alkureishi M. Evaluating the impact
of clinical librarians on clinical questions during inpatient rounds. J Med Libr Assoc. 2018
Apr;106(2):175–83. doi:10.5195/jmla.2018.254

23. Tahmasebi M, Adibi P, Zare-Farashbandi F, Papi A, Rahimi A. The educational role of
clinical informationist on improving clinical education among medical students: Based on
Kirkpatrick model. J Educ Health Promot. 2020;9:28. doi:10.4103/jehp.jehp_439_19

24. Krom ZR, Batten J, Bautista C. A unique collaborative nursing evidence-based practice
initiative using the Iowa model: a clinical nurse specialist, a health science librarian, and a
staff nurse’s success story. Clin Nurse Spec CNS. 2010 Apr;24(2):54–9.
doi:10.1097/NUR.0b013e3181cf5537

25. Atwa H, Abdelaziz A. Evidence-based medicine (EBM) for undergraduate medical
students: A six-step, integrative approach. Med Teach. 2017 Apr;39(sup1):S27–32.
doi:10.1080/0142159X.2016.1254750

26. Al Shahrani AS. Development and evaluation of an evidence-based medicine module in
the undergraduate medical curriculum. BMC Med Educ. 2020 Aug 6;20(1):256.
doi:10.1186/s12909-020-02181-7.

27. Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation
for clinical questions. AMIA Annu Symp Proc. 2006;2006:359-363.

28. Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome
(PICO) as a search strategy tool on literature search quality: A systematic review. J Med Libr

13



Peer Reviewed Article Hypothesis, Vol. 36, No. 2, 2024

Assoc. 2018;106(4):420-431. doi:10.5195/jmla.2018.345

29. Kloda LA, Boruff JT, Cavalcante AS. A comparison of patient, intervention, comparison,
outcome (PICO) to a new, alternative clinical question framework for search skills, search
results, and self-efficacy: A randomized controlled trial. J Med Libr Assoc.
2020;108(2):185-194. doi:10.5195/jmla.2020.739

14


