












































Always Room for Mindfulness


2 13

Documenting the Information 
Needs of Students and Faculty 
for ATS Standards Using Student 
Bibliographic Analysis
Ed Hughes, Director of Library Services, Memphis Theological Seminary

ABSTR ACT:  The question asked in this paper was whether student 
bibliographies can be used to assess the usage of a library’s collec-
tion and thus answer the ATS standard of documenting information 
needs. The goal was to analyze ten percent of student bibliographies 
submitted in papers during the past four semesters. This analysis 
showed that 88% of the citations in this sample could also be found 
in our two database vendors, the Digital Theological Library (80%) 
and EBSCO’s eBook Religion Collection (46.4%). Additionally, vendor-
supplied statistics did not always parallel student citations.

The Association of Theological Schools (ATS) Standard 6.5 calls for 
“Documenting that the information needs of the school’s students 
and faculty are met in ways that are appropriate to the school’s 
educational mission, degree programs, and educational modalities” 
(ATS 2020, 10). Because the Memphis Theological Seminary (MTS) 
Library is entirely digital, circulation statistics for print books are no 
longer available. Therefore, we have relied on vendor reports from 
our two major database suppliers, the Digital Theological Library 
and EBSCO’s eBook Religion Collection to supply the equivalent of 
circulation statistics. However, only one of our databases, EBSCO’s 
eBook Religion Collection, can provide title-level statistics, and our 
collection of EBCO ebook titles is much smaller than the Digital 
Theological Library. 

Authors of a previous study noted that their project was time 
consuming: they reviewed 78 student research papers, yielding 
441 bibliographic citations for the purpose of determining quality 
(Heller-Ross 2003). This project was also time-consuming. However, 



2 14    AT L A 2024 PROCEEDINGS

the primary purpose was not to determine quality, although that was 
a factor. Instead, this study sought evidence that students used our 
databases, particularly the Digital Theological Library. This study 
was limited to the past four semesters in the academic years 2023 
and 2024 and covered 127 citations and 21 student research papers. 

To retrieve title-level statistics for the Digital Theological Library, 
the primary challenge was to match our database title holdings with 
what was found in the student bibliographies. This examination of 
student bibliographies found a high level of correspondence (80%) 
between the titles cited in the papers and the titles found in the 
Digital Theological Library. Combined with the matches found in 
EBSCO’s eBook Religion Collection, the resulting coverage was 88.1%. 
By itself, EBSCO’s eBook Religion Collection matched 46.4% of the 
citations. Only 33% of the citations were found in both databases. 
Even fewer were not found in either database, 11.8%.

Although the student bibliographies showed that most of the 
resources they cited could be found in our two largest databases, 
a study of these citations could not unequivocally show that the 
students used our databases for all of their citations. 

A similar statistical uncertainty would also be true for print books 
checked out versus print books cited in student papers. A print title 
found in the local public library would look the same as a print 
title from a seminary library when cited in a student bibliography. 
Seminary circulation statistics for that title might show heavy use, 
but it cannot link the student’s use unequivocally with the copy of 
the title found in the seminary. Thus, when matched, the association 
between titles found in our databases and those found in student 
papers is an inference. 

The conclusions in this study are based on the following: First, if 
a citation appears in the paper, it meets the student’s information 
needs. Second, if it is posted in Canvas, it likely meets the professor’s 
expectations. Last, if the citation also appears as a resource in the 
databases owned by the MTS Library, it is an instance of the library 
fulfilling the students’ and faculty’s informational needs. 

Scrutinizing student bibliographies removes the uncertainty 
that is associated with vendor reports. EBSCO’s eBook Religion 
Collection usage reports of ebook titles do not equate one-for-one 



Listen and Learn Sessions    215

with citations found in student papers. While students do cite the 
highest-accessed ebooks in the Religion Collection, student citation 
counts can be much lower. 

For example, the third edition of Newsom’s Women’s Bible 
Commentary, which is not in the Digital Theological Library, was 
accessed 16 times in EBSCO’s eBook Religion Collection during the 
semesters studied, but it was only cited twice in the papers exam-
ined. Myers and Enns’ 2009 Ambassadors of Reconciliation, found 
only in EBSCO’s eBook Religion Collection, was accessed 20 times 
but only cited twice in student papers. Lastly, Rasell’s 2022 title, 
Way of Abundance, Economic Justice in Scripture and Society, found 
only in EBSCO’s eBook Religion Collection, was accessed 29 times 
but only cited twice.

As might be expected, both the Digital Theological Library and 
EBSCO’s eBook Religion Collection reported an increase in ebook 
usage once MTS book stacks were closed. 

For the Digital Theological Library, ebook download statistics 
increased nearly threefold in the studied period, from an average 
of 150 to nearly 400 per month. Evenly distributed across all our 
approximately 200 students, that would mean two ebook downloads 
per month or eight ebook downloads per semester per student. 

Some additional findings are worth noting. 

In conversation with students, they say that they often buy the 
books they want. They identify a book from our databases as a 
promising resource by downloading it and browsing its contents, 
like the preview option for some books in a Google search. If they 
like it, they buy it. A few of our students report buying all the books 
used for their papers even after finding them online. 

This preference for paper usually comes out in a reference inter-
view. When informed that we no longer circulate print books, they 
want to know if the whole ebook is downloadable. When told that 
it is partially downloadable, they are sometimes disappointed and 
say they will purchase the book.

For other seminaries considering a move to an all-online library, 
it is worth knowing that seminary students may have a stronger 



2 16    AT L A 2024 PROCEEDINGS

affinity for print than other students in higher education. Based on 
personal observations, a comparison of marginalia and underlin-
ing in seminary library print books is much more extensive than in 
other types of academic libraries. Encountering these marginalia for 
the first time is reminiscent of encountering similar kinds of mar-
ginalia found in medieval manuscripts. For some MTS print books 
with several decades of usage, the exegetical student commentary 
constitutes a lengthy and sometimes heated dialogue. Unless stu-
dents print out parts of an ebook, underlining and adding written 
commentary is not possible. Thus, they may prefer to purchase the 
book rather than be restricted to partial downloads. 

A similar phenomenon occurs when digital chapters of ebooks and 
articles are pushed to students struggling with doing their research. 
These forwarded resources may seem especially relevant, but they 
are often missing when examining the student’s final paper.

With fluctuating enrollments, tracking simple download reports 
from vendors may not be useful. However, tracking average down-
loads per student could be a good statistic for documenting that 
information needs are being met, especially if compared with other 
seminary libraries.

EBSCO’s eBook Religion Collection reports are reliable documenta-
tion of browsing behavior by themselves, something the profession 
has long sought by posting “do not shelve” signs in the stacks. Even 
the Digital Theological Library’s count of total downloads gives a 
limited picture of student browsing activity. However, examining 
student bibliographies is better still. They are a record of what the 
students found useful after all the browsing and downloading was 
done.

There are additional reasons for examining student bibliographies. 

This last semester, for instance, we saw a sudden increase in the 
use of peer-reviewed articles in master’s-level student papers. This 
appears to be the result of the Google Chrome “Angular 15” error 
at the end of the semester that prevented students from using the 
Digital Theological Library in either Chrome or Microsoft Edge. 
That this simple error would have that impact was surprising. The 
explanation is that the other database available to students at MTS 
is EBSCO’s eBook Religion Collection, and our collection of ebooks 



Listen and Learn Sessions    217

in the Religion Collection is small. So, when students could not find 
any eBooks in the Digital Theological Library and too few eBooks 
in EBSCO’ eBook Religion Collection, they looked at Atla Religion 
Database with AtlaSerials PLUS and found many relevant articles. 
Consequently, that semester, citations for peer-reviewed articles 
became more abundant in student papers.

An examination of student bibliographies also shows student col-
laboration. This is most evident when several student papers for the 
same class include several or all the same bibliographic citations. 
It also suggests that buying additional copies might be advisable if 
multiple students use one digital copy. 

Opportunities for education about citing were also evident. 
Whether or not they should take the form of individual tutoring 
sessions is open for discussion. 

If the student finds the resource from another location, this might 
mean that the seminary library should buy the title. Four ebook titles 
not owned at the time this study was completed were purchased as 
a result of student citations. 

Examining student papers also provides an opportunity to evalu-
ate databases. In our case, students still use EBSCO’s eBook Religion 
Collection even though it contains few ebooks. Anecdotal evidence 
suggests that they use the Religion Collection in part because they are 
more familiar with it. In this sample, however, students also found 
journal articles in Atla Religion Database with AtlaSerials PLUS that 
cannot be found in the Digital Theological Library.

This study also showed that students are still using Google searches. 
This is evident in the number of open access sources used in their 
papers that were also not in our databases. Although only three such 
open access sources were used, they were peer-reviewed and current.   

For other librarians interested in pursuing this kind of analysis, 
looking at just one class will give a better idea of what is happening 
or not happening in the classroom. For instance, at least one student 
in this sample shows evidence of avoiding digital resources. The 
temptation is to ignore this kind of paper and pick the best papers 
from the class because the quality of the citations is easier to match 
in the database. However, this is a disservice to struggling students. 



2 18    ATL A 2024 PROCEEDINGS

They are more likely to come to us for help than the accomplished 
students, and when they do, it would be good to have some fore-
knowledge of their abilities. 

A hazard of tracking individual titles cited in papers or as part of 
a download report from a vendor is the “hidden citations” that we 
know are there but do not appear in our statistics gathering (Meng 
2024). As an example, in biblical studies, the most cited book should 
be some version of the Bible. However, students often use their 
own Bibles rather than those provided by their seminary, with the 
consequence that the most cited book is generally not provided by 
seminaries and will not show up in our statistics. In a similar vein, 
St. Augustine of Hippo is often not cited in papers that refer to the 
concept of original sin. Many other titles could be mentioned, but 
the point is that title-level statistics will only give a partial view of 
the information needs of students and faculty.

If used long term as a method for documentation, citation analysis 
sample size must be predetermined and adhered to. The sample in 
this study, 127 citations and 21 papers, represents about 20% of the 
papers generated by students and 10% of our student population 
for the period studied. This may or may not be a sufficient sample 
size to meet the criteria in the ATS Standard.

Lastly, students are using artificial intellignce to write their papers. 
A few student papers showed evidence of AI generation and were 
consequently not used for this study. What is important is that none 
of these AI-generated citations can be verified. Shabby citations in 
student papers may turn out to be one of the indicators of AI usage.

REFERENCES

Association of Theological Schools (ATS). 2020. Standards of 
Accreditation. Association of Theological Schools Commission 
on Accrediting.

Heller-Ross, Holly. 2003. “Assessing Outcomes with Nursing Research 
Assignments and Citation Analysis of Student Bibliographies.” 
Reference Librarian (37) 77: 121–136. Preprint. https://soar.suny.
edu/handle/20.500.12648/1189.

https://soar.suny.edu/handle/20.500.12648/1189
https://soar.suny.edu/handle/20.500.12648/1189


Listen and Learn Sessions    219

Middleton, Anne. 2005. “An Attempt to Quantify the Quality of Student 
Bibliographies.” Performance Measurement and Metrics 6 (1): 7–18.

Meng, Xiangyi, Onur Varol, and Albert-László Barabási. 2024. “Hidden 
citations obscure true impact in science.” PNAS Nexus 3 (5): 1–10. 
https://doi.org/10.1093/pnasnexus/pgae155.

https://doi.org/10.1093/pnasnexus/pgae155



