Leiding.indd
Using Citation Checking of
Undergraduate Honors Thesis
Bibliographies to Evaluate Library
Collections
Reba Leiding
This study utilizes citation checking of advanced undergraduate research
papers as a method for evaluating library collections at an institution with
growing undergraduate and graduate research demands. A random sam-
ple of 101 honors thesis bibliographies from the period 1993–2002 was
examined for format, discipline, and local availability rates.The proportion
of journal citations in relation to books increased slightly over the period,
but no other clear trends emerged.The incidence of Web citations began
during the period but did not steadily increase. The study highlighted
specific use patterns and collection weaknesses. Results serve as a
baseline for further study of the library’s undergraduate user population;
further citation studies are encouraged to assess continued use of online
resources as the Internet and electronic technologies evolve.
maller college libraries and
undergraduate libraries at
larger institutions by defini-
tion focus their collection
development activities on materials that
support the undergraduate curriculum.
First-year writing-intensive class as-
signments and term papers for classes
in curriculum majors require collections
that cover a broad range of research topics
but typically are not as comprehensive
as collections at research institutions.
When undergraduate students at larger
institutions undertake more advanced
research, they direct their research needs
to the main library’s stacks or its subject
libraries, where their usage is assimilated
into that of the overall user population.
But when two-year institutions evolve
into four-year institutions, or when
smaller colleges and universities change
their curriculum missions to emphasize
undergraduate research or add advanced-
degree programs, those institutions’
libraries must find ways, to say nothing
of the means, of making the transition to
support more advanced research. Because
in larger institutions the evolving needs
of advanced undergraduate research are
not readily observable, few studies exist
as a guide to libraries in transition on
how to address this new mission. How
can a library assess how adequately it is
responding to such research demands?
An answer may be found by analyzing
bibliographic citations within existing
Reba Leiding is the Assistant to the Dean at James Madison University Libraries; e-mail: leidinrm@jmu.edu.
417
mailto:leidinrm@jmu.edu
418 College & Research Libraries September 2005
examples of advanced undergraduate
research.
Literature Review
The citation is a bibliographical entry in
a footnote, reference list, or bibliography
of a document that contains enough in-
formation (e.g., author, title, publisher, or
journal title) to verify the original item. As
Linda C. Smith’s early review of citation
analysis pointed out, the relationship be-
tween the citing document and the cited
references rests on a number of assump-
tions, two of the most important being (1)
actual use of the cited document by the
citing author and (2) citation of “the best
possible works… of all possible docu-
ments that could be cited.”1 Given these
assumptions of value and actual use, a
citation can be viewed as a relevant arti-
fact of use data. Further, citation checking
of research documents and comparison
of those citations with the availability of
materials in a local collection offers an
unobtrusive and cost-effective method
of evaluating that collection’s ability to
support research.2,3
As Smith’s and Robert N. Broadus’s
literature reviews noted, citation check-
ing is a time-honored and well-studied
method for determining the usefulness of
collections.4,5 It is part of a broad category
known as citation studies, defined by Paul
H. Mosher as “any specific methodologies
that use source citations or references
drawn from the scholarly apparatus of
articles and books as the basis for manipu-
lation, research, and study.”6 Applications
for this research in this broad area have
branched in several directions. Mosher
noted the distinction between studies of
bibliographical citations and the more
“bibliometric” methodologies that ex-
amine the linking of documents through
jointly cited references.7 One branch of
citation-checking studies flourished from
the 1970s onward as libraries responded to
budgetary crises and rising journal prices.
Citations in research literature within a
discipline were checked in order to find
a rationale for canceling subscriptions or
allocating funds. Harry M. Kriz examined
the ratio of books to monographs cited
in master ’s theses for engineering and
found a justification for protecting book
budgets in that field.8 Checking citations
in master’s theses was one of the methods
Christina E. Bolgiano and Mary Kathryn
King used to evaluate journal collections.9
Thomas E. Nisonger checked citations in
selected political science journals to evalu-
ate the subject collection.10 Robin B. Devin
and Martha Kellogg recommended using
citation checking and analysis to develop
a formula for a serial/monograph ratio.11
More recent studies have used citation
checking of local faculty or graduate
student research or of scholarly journals
to evaluate subject collections in biology,
psychology, polymer science, education,
and interdisciplinary studies such as tour-
ism.12–16 A study by Erin T. Smith used a
sample of graduate theses and disserta-
tion bibliographies from 1991 and 2001
to form “snapshots” of graduate research
demands on the library collection. Smith’s
study is similar to the present one in that
it analyzes citations by material type, dis-
cipline, and local availability; she used the
snapshots to gauge the impact of budget
constraints on collection usefulness for
the library’s graduate user population.17
Although numerous studies have
examined research journals or bibliog-
raphies of more advanced researchers,
a smaller body of studies has used cita-
tion checking to gain information about
undergraduate users. Rose Mary Magrill
and Gloriana St. Clair looked at differ-
ences in citation behavior by course level
and in different disciplines.18 Margaret
Sylvia and Marcella Lesher analyzed
journal citations of undergraduate, as well
as graduate, research papers to evaluate
an academic library’s journal collection in
psychology.19 A series of studies by Philip
M. Davis and Davis and Suzanne A. Co-
hen analyzed undergraduate term paper
bibliographies to determine the effect of
the Internet on citation behavior.20–22 One
outcome of Davis and others was the for-
mation of recommendations for research
http:psychology.19
http:disciplines.18
http:population.17
http:ratio.11
http:collection.10
Using Citation Checking of Undergraduate Honors Thesis Bibliographies 419
paper assignments. Karen Hovde used
citation checking to evaluate the impact
of library instruction on student research
skills.23
Even fewer studies have analyzed
undergraduate papers’ bibliographies to
evaluate local collection use. Kathleen E.
Joswick compared local library holdings
of materials cited in freshman composi-
tion papers in order to form recommenda-
tions for building collections that address
undergraduate research needs.24 Joswick
and Jeane Koekkoek Stierman analyzed
journal citations from freshman compo-
sition papers, as well those from faculty
research papers and international citation
pa erns, and found very li le overlap;
their study recommended local use stud-
ies of multiple-user types.25
Background
James Madison University (JMU) is state-
supported university focused primarily
on undergraduate education. The institu-
tion was founded in 1908 as a women’s
normal school but by the 1960s had
become a coeducational institution with
a broader curriculum. Since the 1970s,
when the university took its present
name, the number of students has nearly
tripled and the university currently has an
enrollment of more than 15,500 students.
Although undergraduate education re-
mains the university’s primary mission,
graduate enrollment is increasing along
with the number of graduate programs,
which now include a small number at
the doctoral level. At the same time, the
university is placing greater emphasis on
undergraduate research and encouraging
faculty to include undergraduates in their
research projects. In addition, members
of the teaching faculty have registered
demands for collections to support faculty
research through numerous user satisfac-
tion surveys. This growing emphasis on
advanced research presents a dilemma to
a university library that has long focused
its mission on support of an under-
graduate curriculum. How can librarians
evaluate how adequately current library
collections support research or determine
what specific needs exist?
The present study examines a par-
ticular niche of undergraduate research:
honors thesis bibliographies. Although
the number of students producing an
honors thesis makes up a small percent-
age of the undergraduate population at
JMU (about two percent in 2002), it can
be argued that they are some of the most
intensive users of library resources. Hon-
ors theses are the culmination of a serious
research undertaking. For this reason,
their bibliographies are more likely than
those of shorter undergraduate composi-
tion papers to reliably fulfill those two
assumptions underlying citation analysis
methodology: that the cited material was
actually used by the document’s author,
and that the sources used were the best
possible for the purposes of his or her
research. Because of the long-term nature
of their projects, authors of honors theses
also are less likely to limit their bibliogra-
phies to resources available only within
the university’s library, as is o en the
case with freshman composition papers.
Further, as Louise S. Zipp noted in her
study of graduate thesis citations, student
research can serve as an indicator of re-
search interests of their faculty advisors.26
Thus, a study of theses bibliographies also
offers an efficient way to assess faculty
research needs. Typically, as a ma er of
policy, many academic libraries collect
and house copies of student theses. Even
though students are a fluid population,
these collections constitute a stable and
accessible source of data about evolving
research needs of both faculty and ad-
vanced undergraduate students.
At James Madison University, two
copies of all undergraduate honors theses
have been collected and housed in the
library since the inception of the honors
program in the 1960s. Because theses are
cataloged in-house, their records show
remarkable consistency in detail across
years. All undergraduate theses have
a unique call number, yet all share the
same root call number, which facilitates
http:advisors.26
http:types.25
http:needs.24
http:skills.23
420 College & Research Libraries September 2005
the ability to define the entire population
of records within the online catalog and
move the list into a database for sample
selection.
Methodology
The present study, undertaken in 2003,
examined a sample of ten years’ worth of
thesis bibliographies from 1993 to 2002 to
determine the adequacy of the collection
for undergraduate research. The study
also sought to detect any trends in the
undergraduate research environment,
such as the increased use of journals or
the impact of the Internet on citation
behavior. It was hypothesized that the
growing availability of online indexes and
TABLE 1
Number of Academic Departments
Represented in Study, 1993–2002
Psychology 15
Biology 9
Political Science 9
Integrated Science & Technology 8
History 7
Anthropology and Sociology 6
Business, Economics 6
English 6
Health Sciences 6
Theatre and Dance 6
Philosophy and Religion 5
Art 2
Chemistry 2
Communications Sciences, Disorders 2
Math 2
Media Arts and Design 2
Social Work 2
Speech Communication 2
Computer Science 1
General Studies 1
Kinesiology 1
Music 1
Total 101
abstracts would lead to an increase in the
percentage of journals cited as compared
to monographs. Access to electronic jour-
nals also was increasing throughout the
study’s duration, but because citations
during this early period of electronic use
may not indicate whether a journal article
was accessed in print or via online, no
hypothesis about e-journal use was pro-
posed. In addition, it was hypothesized
that Web citations (that is, citations refer-
ring to Internet pages) would make up an
increasing percentage of citations in the
la er half of the 1990s.
Out of a total of 1,244 honors theses in
the JMU library as of January 2002, 674
theses containing bibliographies were
added to the collection between 1993
and 2002. Call numbers for this popu-
lation were imported into Excel and
sorted by year, and a stratified sample
was chosen using Excel’s random num-
ber generator. A total of 101 thesis bib-
liographies comprised the sample, or
approximately 15 percent of the theses
added in each year. Bibliographies were
checked to determine the total number
of citations and what types of materi-
als were cited, such as books, journals,
newspapers, primary sources (in this
study defined as interviews, original
correspondence, surveys, etc.), Web
citations, and ”other” miscellaneous
sources, including government docu-
ments. Data from the bibliographies
were entered in an Access database.
The total number of citations per bibli-
ography was tracked, but for purposes
of analysis duplicate citations within a
bibliography (that is, subsequent cita-
tions referring to a previously cited
source) were counted only once. This
eliminated skewing in availability rates
that could occur when, for example,
an anthology was cited repeatedly
and reconciled differences in citation
styles that used endnotes rather than
a list of works cited. Citations were
checked against the online catalog to
determine local availability at the time.
In the case of books, ownership was
Using Citation Checking of Undergraduate Honors Thesis Bibliographies 421
determined by the cata-
log date in the record;
because weeding activ-
ity was minimal during
the study years, current
catalog records were
assumed to accurately
reflect holdings for the
period. For journals,
the volume holdings in
the catalog record were
checked to determine
availability of particular
issues when the thesis
was wri en. Availabil-
ity of full-text articles
through aggregator da-
tabases was not checked
for this study, with the
exception of LexisNexis
Academic, which the li-
brary acquired in 1999.27
Those citations not veri-
fied through the online
catalog were searched
in other sources such
as WorldCat or Google
to determine citation
veracity. Data were ana-
lyzed by material type
by year and by disci-
pline to determine if
any significant trends in
composition of bibliog-
raphies emerged. Rates
of local availability were
calculated and analyzed
to evaluate how well lo-
cal collections serve this
research population.
Results
The total number of
citations in the random
sample was 3,564. The
number of unique cita-
tions was 3,407 when
duplicate citations were
eliminated. The shortest
bibliography contained
two citations and the
TA
B
L
E
2
C
om
po
si
ti
on
o
f B
ib
lio
gr
ap
hi
es
, b
y
Y
ea
r
Y
ea
r
To
ta
l
U
ni
qu
e
C
it
at
io
ns
B
oo
ks
C
it
ed
%
B
oo
ks
Jo
ur
na
ls
C
it
ed
%
Jo
ur
na
ls
N
ew
sp
ap
er
s
C
it
ed
%
N
ew
sp
ap
er
s
P
ri
m
ar
y
So
ur
ce
s
C
it
ed
%
P
ri
m
ar
y
So
ur
ce
s
O
th
er
So
ur
ce
s
C
it
ed
%
O
th
er
So
ur
ce
s
W
eb
Si
te
s
% W
eb
Si
te
s
19
93
28
7
10
6
36
.9
%
93
32
.4
%
32
11
.1
%
3
1.
0%
53
18
.5
%
—
—
19
94
25
8
79
30
.6
%
14
5
56
.2
%
—
—
26
10
.1
%
8
3.
1%
—
—
19
95
33
9
16
4
48
.4
%
10
7
31
.6
%
23
6.
8%
3
0.
9%
42
12
.4
%
—
—
19
96
29
6
16
2
54
.7
%
99
33
.4
%
—
—
5
1.
7%
30
10
.1
%
—
—
19
97
26
4
11
8
44
.7
%
11
1
42
.0
%
9
3.
4%
1
0.
4%
20
7.
6%
5
1.
9%
19
98
36
9
14
3
38
.8
%
11
4
30
.9
%
16
4.
3%
20
5.
4%
48
13
.0
%
28
7.
6%
19
99
45
6
12
5
27
.4
%
19
3
42
.3
%
15
3.
3%
1
0.
2%
49
10
.7
%
73
16
.0
%
20
00
47
6
18
9
39
.7
%
17
6
37
.0
%
16
3.
4%
19
4.
0%
27
5.
7%
49
10
.3
%
20
01
40
1
10
3
25
.7
%
20
8
51
.9
%
16
4.
0%
—
—
28
7.
0%
46
11
.5
%
20
02
26
1
49
18
.8
%
16
4
62
.8
%
2
0.
8%
10
3.
8%
16
6.
1%
20
7.
7%
To
ta
ls
/
A
vg
. %
3,
40
7
1,
23
8
36
.3
%
1,
41
0
41
.4
%
12
9
3.
8%
88
2.
6%
32
1
9.
4%
22
1
6.
5%
422 College & Research Libraries September 2005
TABLE 3
Materials in “Other” Category
Material Type
No. of
Times Cited
Percent Owned
by JMU
Government documents, including those from federal,
state, municipal, foreign countries, etc.
88 50.0%
Law texts, court cases, bills, etc. 66 96.9%
Reports, including annual, financial, planning, and
technical reports; discussion, policy, and working papers
56 1.7%
Conference papers and proceedings 24 0%
E-Resources, library 16 100.0%
Brochures, pamphlets 11 0%
Theses 13 76.9%
Videos 8 50.0%
Dissertation, Ph.D. 6 0%
Miscellaneous 33 21.2%
Total 321
longest contained 135, with the median
being 31. Twenty-two academic depart-
ments were represented in the sample.
(See table 1.) Psychology, the largest
department in the university, had the
greatest number with a total of 15; the
list as a whole showed a mix of the
physical sciences, social sciences, and
humanities.
Table 2 shows the composition of bib-
liographies by material format by year.
Looking at the average percentage for all
years combined, it appears that the overall
reliance on books and journals is fairly
evenly divided (36.3% for books as com-
pared to 41.4% for journals). Newspapers
made up a minor part of all citations, at
3.8 percent overall and showed consistent
use near that level even in later years of
the study when local access to newspaper
sources increased with the availability of
Lexis-Nexis. Primary sources likewise
were a minor source overall at 2.6 percent.
The study separated out primary sources
as a material type to highlight any use of
the library’s special collections of local
historical sources; however, no theses
using local primary sources were part of
the sample.
The proportion of “other ” sources
ranged between 3 and 18 percent in the
years examined and overall comprised
9.2 percent of all citations. In table 3,
which also shows local availability of
miscellaneous sources, government
documents made up the largest compo-
nent of the “other” category, accounting
for 27.2 percent of all miscellaneous
sources, with law texts making up an
additional 20.4 percent. The library
had access to 50 percent of government
documents cited and 96.9 percent of law
texts. The remainder of this category
was dominated by grey literature, such
as unpublished technical and financial
reports, conference papers, or brochures,
or with nonprint material.
Web citations accounted for 6.2 per-
cent of citations overall, but no such
citations appeared at all until 1997. In
1999, Web citations amounted to 16 per-
cent of total citations (when the Internet
as a new phenomenon was in fact the
subject of a few theses). Interestingly, the
percentage dropped to around 10 percent
in the next two years and continued to
decrease in the last year of the study to
7.7 percent.
Using Citation Checking of Undergraduate Honors Thesis Bibliographies 423
Effect of Web Citations
Because the inception of the Internet
marked a watershed in the types of sources
available for research, it is informative to
look at the composition of bibliographies
before and a er Web citations began to
appear in the sample bibliographies. (See
table 4.) In the years of the study before
Web citations, books accounted for 45.3
percent of total citations and journal cita-
tions amounted to 36.1 percent. In the
years 1997–2002, Web citations accounted
for nearly 10 percent of all citations, books
dropped to 34.2 percent, and journals
rose to 42.1 percent. If Web citations are
TA
B
L
E
4
C
om
po
si
ti
on
o
f B
ib
lio
gr
ap
hi
es
: P
re
- a
nd
P
os
t I
nt
er
ne
t
C
it
at
io
ns
in
B
ib
B
oo
ks
Jo
ur
na
ls
N
ew
sp
ap
er
s
P
ri
m
ar
y
So
ur
ce
s
“O
th
er
”
R
es
ou
rc
es
W
eb
Si
te
s
To
ta
l 1
99
3–
19
96
1,
26
6
57
3
45
7
57
38
14
1
—
Pe
rc
en
t 1
99
3–
19
96
10
0.
0%
45
.3
%
36
.1
%
4.
5%
3.
0%
11
.1
%
—
To
ta
l 1
99
7–
20
02
2,
29
8
78
7
96
8
82
52
18
8
22
1
Pe
rc
en
t 1
99
7–
20
02
10
0.
0%
34
.2
%
42
.1
%
3.
6%
2.
3%
8.
2%
9.
6%
To
ta
l w
ith
ou
t W
eb
Si
te
s
19
97
–2
00
2
2,
07
7
78
7
96
8
82
52
18
8
—
Pe
rc
en
t w
ith
ou
t
W
eb
S
ite
s
10
0.
0%
37
.9
%
46
.6
%
3.
9%
2.
5%
9.
1%
—
removed from the equation, book citations
increase (38%) but do not reach the level
found before 1997, whereas the percentage
of journal citations increases to 47 per-
cent. Although the data are not sufficient
to argue that Web citations negatively
impact the use of books, the generally ris-
ing percentage of journal citations when
looking at pre- and post-Internet periods
demonstrates that thesis authors are ap-
parently relying more heavily on journals
for their research.
Local Availability
Table 5 shows local availability of cited
references by material type by year. The
percentage of books held by the library
ranged from 53 to 85 percent throughout
the years studied, with the mean at over
65 percent. Whether this percentage can
be termed adequate is a ma er of opinion,
but it is worth noting that the library’s
book budget was static for most of the
period covered by the study. Availability
of journals was even lower overall at 58.2
percent; however, this low percentage is a
bit misleading, and an accurate analysis
needs to factor in the high number of
individual titles cited and the range of
frequency by title. The 1,410 total journal
citations referenced 729 unique journal
titles. The most frequently cited journal
was referenced 22 times; a total of 16
journal titles were cited ten or more times.
(See table 6.) Of these frequently cited
titles, the library had access to over 83
percent. (One frequent citation was a local
newspaper cited heavily in a single bibli-
ography. Without this title, the availability
rate was over 86%.) Over two-thirds of
the journal titles in the list (494) were
cited only once; of these, the library pro-
vided access to 41.5 percent on average.
Newspapers, though not frequently cited
(as to be expected in scholarly research),
displayed the highest rates of local avail-
ability. This can be explained by the fact
that even prior to access to LexisNexis, the
library had extensive back files to the most
frequently cited newspapers, the New York
Times and the Washington Post.
424 College & Research Libraries September 2005
Primary sources
showed the lowest
rates of availability
at 3.4 percent over-
all. Most primary
sources cited in bib-
l i o g r a p h i e s , s u c h
as personally con-
ducted interviews,
surveys, and so on
stemmed from the
student author’s own
research, rather than
use of library ma-
terials. The overall
percentage of mate-
rials in the “other ”
category that were
available locally was
at 45.4%, however,
the availability rate
for government doc-
uments was higher
at 50% and law texts
was 97%, likely due
primarily to Lexis-
Nexis.
Results by Discipline
The data also can
be examined from
the standpoint of
discipline for col-
lection development
p u r p o s e s . T h e s e s
we r e a s s i g n e d t o
one of four broad
disciplines based on
the student’s aca-
demic department,
as found in the thesis
catalog record: busi-
n e s s , h u m a n i t i e s ,
sciences, and social
sciences. (See table
7.) Composition of
bibliographies by
discipline was gen-
erally as expected:
theses in the humani-
ties relied most heav-
TA
B
L
E
5
A
va
ila
bi
lit
y
of
C
it
ed
M
at
er
ia
ls
, b
y
Y
ea
r
Y
ea
r
B
oo
ks
C
it
ed
N
um
be
r
H
el
d
(%
)
Jo
ur
na
ls
C
it
ed
N
um
be
r
H
el
d
(%
)
N
ew
sp
ap
er
s
C
it
ed
N
um
be
r
H
el
d
P
ri
m
ar
y
So
ur
ce
s
C
it
ed
N
um
be
r
H
el
d
O
th
er
S
ou
rc
es
C
it
ed
N
um
be
r
H
el
d
19
93
10
6
65
(6
1.
3)
93
69
(7
4.
2)
32
3
2
(1
00
.0
)
3
0
(0
.0
)
53
47
(8
8.
7)
19
94
79
50
(6
3.
3)
14
5
63
(4
3.
4)
—
—
26
0
(0
.0
)
8
1
(1
2.
5)
19
95
16
4
96
(5
8.
5)
10
7
79
(7
3.
8)
23
20
(8
7.
0)
3
0
(0
.0
)
42
10
(2
3.
8)
19
96
16
2
10
8
(6
6.
7)
99
52
(5
2.
5)
—
—
5
1
(2
0.
0)
30
10
(3
3.
3)
19
97
11
8
84
(7
1.
2)
11
1
77
(6
9.
4)
9
9
(1
00
.0
)
1
1
(1
00
.0
)
20
16
(8
0.
0)
19
98
14
3
78
(5
4.
5)
11
4
74
(6
4.
9)
16
9
(5
6.
3)
20
0
(0
.0
)
48
6
(1
2.
5)
19
99
12
5
68
(5
4.
4)
19
3
12
2
(6
3.
2)
15
14
(9
3.
3)
1
1
(1
00
.0
)
49
30
(6
1.
2)
20
00
18
9
16
2
(8
5.
7)
17
6
96
(5
4.
5)
16
15
(9
3.
8)
19
0
(0
.0
)
27
8
(2
9.
6)
20
01
10
3
73
(7
0.
9)
20
8
12
5
(6
0.
1)
16
13
(8
1.
3)
—
—
28
14
(5
0.
0)
20
02
49
26
(5
3.
1)
16
4
64
(3
9.
0)
2
2
(1
00
.0
)
10
0
(0
.0
)
16
4
(2
5.
0)
To
ta
ls
/A
ve
ra
ge
Pe
rc
en
t
1,
23
8
81
0
(6
5.
4)
1,
41
0
82
1
(5
8.
2)
12
9
11
4
(8
8.
9)
88
3.
4
(2
4.
4)
32
1
14
6
(4
5.
5)
y
re
fe
r t
o
ei
th
er
a
p
hy
si
ca
l i
te
m
is
in
th
e
lib
ra
ry
c
ol
le
ct
io
n
or
e
le
ct
ro
ni
c
ac
ce
ss
o
f f
ul
l-
te
xt
e
-j
ou
rn
al
s
ub
sc
ri
pt
io
ns
. C
ita
tio
ns
a
ls
o
w
er
e
ch
ec
ke
d
ag
ai
ns
t
m
ic
fo
r a
va
ila
bi
lit
y
(1
99
9–
);
c
on
te
nt
fr
om
o
th
er
a
gg
re
ga
to
rs
w
as
n
ot
c
on
si
de
re
d
fo
r t
hi
s
st
ud
y.
“H
el
d”
m
a
L
ex
is
N
ex
is
A
ca
de
N
ot
e:
Using Citation Checking of Undergraduate Honors Thesis Bibliographies 425
TABLE 6
Most Frequently Cited Journals
Journal Title Citations Held
Locally
% Held
Locally
Journal of Personality and Social Psychology 22 22 100.0%
Science 20 20 100.0%
Business Week 18 18 100.0%
Psychological Reports 16 16 100.0%
Casselman Chronicles 15 0 0%
International Journal of Eating Disorders 15 15 100.0%
Nature 15 15 100.0%
Physical Therapy 14 14 100.0%
Economist 13 13 100.0%
Journal of Sex Research 12 11 91.7%
Euromoney 11 11 100.0%
JAMA 11 11 100.0%
Gerontologist 10 10 100.0%
Journal of Quality Technology 10 0 0%
New England Journal of Medicine 10 10 100.0%
Radiology 10 0 0%
Total/Avg. 222 186 83.8%
ily on books (69.3%), whereas theses in
the sciences and social sciences utilized
books to a lesser extent (20.2% and 29.5%,
respectively). Theses in the area of busi-
ness were relatively dependent on books
as well, at 48.4 percent. Predictably, the
highest percentage of journal citations
was in the sciences (58.8%) and second
highest in the social sciences (46.2%),
whereas journal citation use in the hu-
manities was low at 12.6 percent. Use of
Web resources was the highest for busi-
ness at 13.2 percent.
Some particular pa erns emerge when
looking at local availability of materials
in terms of discipline. (See table 8.) The
percentage of books held locally ranged
from 68 to 72.2 percent for all disciplines
except the sciences, where the local avail-
ability was lower at 40.8 percent. Likewise
for journals, the local availability rate was
lowest for the sciences at 44.4 percent,
whereas for the other disciplines the
range was from 64.9 to 70.5 percent. These
figures highlight the historical weak-
nesses in both monograph and journal
collections for the pure sciences, as would
be expected at a smaller institution with
a liberal arts orientation. The high cost of
science materials, particularly journals,
is no doubt a factor, as well. For “other”
resources, the social sciences had the
highest rates of local availability at 65.7
percent, reflecting the use of the govern-
ment documents and local law collections
by such departments as history and politi-
cal science.
Observations
What implications can be seen from an
analysis of thesis citation data? First of all,
books remain tremendously important
for researchers in the humanities and, to
some extent, for business. Although the
former in particular may seem an obvious
point, we should recall that the library ex-
426 College & Research Libraries September 2005
perienced steady state budgets for much the percentage of books cited relative to
of the study period and during some years journals by year in the overall composi-
the size of book budgets eroded in the face tion of bibliographies, but grouping the
of journal inflation. Thus, evidence for the data into pre- and post-Internet periods
importance of book collections serves as does provide some evidence that use of
justification for increased funding. No journals in undergraduate research is in-
overall trend emerges when looking at creasing. Access to journals is particularly
important for undergraduate researchers
TA
B
L
E
7
C
om
po
si
ti
on
o
f B
ib
lio
gr
ap
hi
es
b
y
D
is
ci
pl
in
e
To
ta
l
U
ni
qu
e
C
ita
tio
ns
B
oo
ks
Jo
ur
na
ls
N
ew
sp
ap
er
s
P
ri
m
ar
y
So
ur
ce
s
O
th
er
So
ur
ce
s
W
eb
S
ou
rc
es
D
is
ci
pl
in
e
To
ta
l
%
To
ta
l
%
To
ta
l
%
To
ta
l
%
To
ta
l
%
To
ta
l
%
B
us
in
es
s
34
9
16
9
48
.4
%
97
27
.8
%
8
2.
3%
5
1.
4%
24
6.
9%
46
13
.2
%
H
um
an
iti
es
61
9
42
9
69
.3
%
78
12
.6
%
57
9.
2%
6
1.
0%
27
4.
4%
22
3.
6%
Sc
ie
nc
es
86
2
17
4
20
.2
%
50
7
58
.8
%
4
0.
5%
27
3.
1%
10
4
12
.1
%
46
5.
3%
So
ci
al
Sc
ie
nc
es
1,
57
7
46
6
29
.5
%
72
8
46
.2
%
60
3.
8%
50
3.
2%
16
6
10
.5
%
10
7
6.
8%
To
ta
l
3,
40
7
1,
23
8
1,
41
0
12
9
88
32
1
22
1
in the sciences and social sciences, as
would be expected.
Use of Web citations does not appear to
be a growing trend for advanced under-
graduate research. This would appear to
be a somewhat surprising result given the
impact of the Internet, but it parallels the
findings of Davis and Cohen, and Davis’s
later findings.28–30 This tempering of Web
citation use may reflect the involvement of
faculty advisors in encouraging students
to use more traditional and authoritative
resources. The present study’s findings
can serve as a baseline in future studies
to determine whether Web citations will
increase as the Internet matures and what
impact Internet use will have on the com-
position of future bibliographies.
One clear finding is the importance of
government documents materials, par-
ticularly for social science research. As
noted in table 3, the overall availability
rate for government documents cited
at this library, which is a government
document depository, was 50 percent. The
growing number of government docu-
ments available online and the federal
government’s proposed scaling back of
the document depository program will
reduce the significance of libraries’ physi-
cal depository collections, however. In the
future, access to government documents
will likely depend more on the skills of
researchers—and reference librarians—in
searching and navigating government
Web sites.
A significant proportion of materials
was unavailable locally, primarily in
sources categorized as miscellaneous and
primary, but perhaps most notably in the
large number of journal titles cited only
a few times across the years of the study.
This fact points to the importance of inter-
Using Citation Checking of Undergraduate Honors Thesis Bibliographies 427
TA
B
L
E
8
A
va
ila
bi
lit
y
of
C
it
ed
M
at
er
ia
ls
b
y
D
is
ci
pl
in
e
D
is
ci
pl
in
e
B
oo
ks
Jo
ur
na
ls
N
ew
sp
ap
er
s
P
ri
m
ar
y
So
ur
ce
s
O
th
er
S
ou
rc
es
To
ta
l
C
it
at
io
ns
N
um
be
r
H
el
d
% H
el
d
To
ta
l
C
it
at
io
ns
N
um
be
r
H
el
d
% H
el
d
To
ta
l
C
it
at
io
ns
N
um
be
r
H
el
d
% H
el
d
To
ta
l
C
it
at
io
ns
N
um
be
r
H
el
d
% H
el
d
To
ta
l
C
it
at
io
ns
N
um
be
r
H
el
d
% H
el
d
B
us
in
es
s
16
9
12
2
72
.2
%
97
63
64
.9
%
8
6
75
.0
%
5
0
0.
0%
24
2
8.
3%
H
um
an
iti
es
42
9
30
0
69
.9
%
78
55
70
.5
%
57
52
91
.2
%
6
1
16
.7
%
27
14
51
.9
%
Sc
ie
nc
es
17
4
71
40
.8
%
50
7
22
5
44
.4
%
4
3
75
.0
%
27
0
0.
0%
10
4
21
20
.2
%
So
ci
al
Sc
ie
nc
es
46
6
31
7
68
.0
%
72
8
47
8
65
.7
%
60
53
88
.3
%
50
2
4.
0%
16
6
10
9
65
.7
%
To
ta
l
1,
23
8
81
0
1,
41
0
82
1
12
9
11
4
88
3
32
1
14
6
library loan (ILL) services for libraries in
transition from a curricular to a research
collection. ILL proves more advantageous
for the undergraduate researcher working
on a long-term project than for students
working on shorter writing assignments,
which tend to be researched at the last
minute. Although ILL requests can pro-
vide important information on emerging
needs for journal subscriptions, this study
shows that in most cases it would not be
cost efficient to add the occasionally cited
title to the library collection.
Conclusion
This study found an increasing propor-
tion of journal citations in undergraduate
thesis bibliographies over time, especially
for science and social science disciplines.
The data did not support the hypothesis
that use of Web citations in thesis bibli-
ographies was increasing. The absence
of this trend may be the result of faculty
advisor intervention; faculty and stu-
dents may not have viewed the Web as a
scholarly resource in its early years. Also,
the study did not make clear that use of
Web citations had any singular impact on
the composition of thesis bibliographies
over time. Although the proportion of
journal citations did increase relative to
monographic citations in the la er por-
tion of the period, the data do not point to
specific causes for this shi . It may be due
to a gradually rising number of journal
subscriptions during the study period or
may be the result of the eroding strength
of book collections in a sustained period
of steady state funding, in which case the
student researchers are exhibiting at least
a tendency to rely on accessibility rather
than the best sources for their purposes.
Or the shi may simply be a result of the
particular characteristics of the sample.
At the same time, relatively low local
availability rates occurred for some dis-
ciplines and material types throughout
the study period. Low availability rates
highlight weaknesses in the collection (a
locally useful outcome of the study) but
also provide a positive validation that the
428 College & Research Libraries September 2005
citations under study, in fact, fulfill those
basic assumptions of actual use and qual-
ity of the documents cited. That is, rather
than confining their research to what was
locally available, the study provides evi-
dence that the authors made an effort to
obtain and use the best possible resources
for their projects.
That no other strong trends or pat-
terns emerged from the data over time
demonstrates the complexity of support-
ing academic research needs. Even when
liaison librarians or bibliographers know
the research interests and pa erns of in-
dividual faculty, students, or even depart-
ments, these interests and emphases can
shi in response to numerous interacting
factors. Traditionally, large institutions
have responded to the complexity of
academic research demands by using the
knowledge of subject specialists and main-
taining large “just-in-case” collections.
A smaller academic library with limited
resources must be nimble in assessing and
responding to changing needs.
Locally, results of this study provide a
baseline to track future trends in the use
of Internet citations and online journals.
Repeating the study across disciplines in
future years, or focusing on theses from
particular disciplines or departments,
will provide further assessment of how
well this institution’s library resources are
meeting research needs in a transition-
ing academic environment. Libraries at
other higher education institutions whose
academic mission is shi ing—whether
they are two-year colleges that are mov-
ing toward four-year institutions or col-
leges and smaller universities that are
emphasizing undergraduate research,
adding academic disciplines, or graduate
programs—could employ methodologies
such as those used in this study to evaluate
how well their collections are responding
to changing research demands. Librarians
at these institutions are probably already
aware of existing undergraduate research
populations that serve as bellwethers of
emerging research needs.
Several factors make this a propitious
time to undertake further citation analy-
sis studies of undergraduate research
behavior. The time frame covered in this
study occurs relatively early in the life
of the World Wide Web and e-journal
access. Style manuals by now have in-
corporated formats for citing online ac-
cess, and librarians, faculty, and students
have had time to assimilate them. This
makes the task of tracking use of online
materials in bibliographies more reliable.
The Web continues to mature, offering
an increasing number of authoritative
resources. Citation studies also may
provide information on how effectively
student researchers use online indexes
and emerging link resolver technologies.
Findings from such studies will not only
provide a baseline for future trends stud-
ies as technologies continue to evolve but
also will contribute to what has to date
been a relatively small body of research
on undergraduate library users.
Notes
1. Linda C. Smith, “Citation Analysis,” Library Trends 30 (summer 1981): 83–106.
2. Margaret J. Sylvia, “Citation Analysis as an Unobtrusive Method for Journal Collection
Evaluation Using Psychology Student Research Bibliographies,” Collection Building 17, no. 1 (1998):
20–28.
3. Thomas E. Nisonger, “A Test of Two Citation Checking Techniques for Evaluating Political
Science Collections in University Libraries,” Library Resources & Technical Services 27 (Apr./June
1983): 163–76.
4. Smith, “Citation Analysis.”
5. Robert N. Broadus, “The Applications of Citation Analysis to Library Collection Building,”
in Advances in Librarianship, ed. Melvin J. Voight and Michael H. Harris, pp. 299–335 (New York:
Academic Pr., 1977).
6. Paul H. Mosher, “Quality and Library Collections: New Directions in Research and Practice
in Collection Evaluation,” in Advances in Librarianship, ed. Wesley Simonton, pp. 211–38 (New
Using Citation Checking of Undergraduate Honors Thesis Bibliographies 429
York: Academic Pr., 1984).
7. Ibid., 222.
8. Harry M. Kriz, “Citation Counting and the Future of Engineering Libraries,” Engineering
Education 67 (Apr. 1977): 707–10.
9. Christina E. Bolgiano and Mary Kathryn King, “Profiling a Periodicals Collection,” College
& Research Libraries 39 (Mar. 1978): 99–104.
10. Nisonger, “A Test of Two Citation Checking Techniques.”
11. Robin B. Devin and Martha Kellogg, “The Serial Monograph Ratio in Research Libraries:
Budgeting in Light of Citation Studies,” College & Research Libraries 51 (Jan. 1990): 46–54.
12. Rosalind Walco , “Local Citation Studies: A Shortcut to Local Knowledge,” Science &
Technology Libraries 14, no. 3 (1994): 1–14.
13. Margaret Sylvia and Marcella Lesher, “What Journals Do Psychology Graduate Students
Need? A Citation Analysis of Thesis References,” College & Research Libraries 56 (July 1995):
313–18.
14. Sherri Edwards, “Citation Analysis as a Collection Development Tool: A Bibliometric Study
of Polymer Science Theses and Dissertations,” Serials Review 25, no. 1 (1999): 11–20.
15. Laurel A. Haycock, “Citation Analysis of Education Dissertations for Collection Develop-
ment,” Library Resources & Technical Services 48 (Apr. 2004): 102–6.
16. Juris Dilevko and Keren Dali, “Improving Collection Development and Reference Services
for Interdisciplinary Services through Analysis of Citation Pa erns: An Example Using Tourism
Studies,” College & Research Libraries 65 (May 2004): 216–41.
17. Erin T. Smith, “Assessing Collection Usefulness: An Investigation of Library Ownership
of the Resources Graduate Students Use,” College & Research Libraries 64 (Sept. 2003): 344–55. For
a comment on Smith’s methodology and response, see “Le ers to the Editor,” College & Research
Libraries 65 (Jan. 2004): 6–7.
18. Rose Mary Magrill and Gloriana St. Clair, “Undergraduate Term Paper Citation Pa erns
by Disciplines and Level of Course,” Collection Management 12, no. 3/4 (1990): 25–56.
19. Sylvia, “Citation Analysis as an Unobtrusive Method.”
20. Philip M. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000
Update,” College & Research Libraries 63 (Jan. 2002): 53–60.
21. ———, “Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholar-
ship in a Networked Age,” Portal: Libraries and the Academy 3 (Jan. 2003): 41–51. Available online
at h p://muse.jhu.edu/journals/portal_libraries_and_the _academy/voo3/3.1davis.pdf.
22. ——— and Suzanne A. Cohen, “The Effect of the Web on Undergraduate Citation Behavior
1996–1999,” Journal of the American Society for Information Science and Technology 52, no. 4 (2001):
309–14.
23. Karen Hovde, “Check the Citation: Library Instruction and Student Paper Bibliographies,”
Research Strategies 17 (2000): 3–9.
24. Kathleen E. Joswick, “Library Materials Use by College Freshmen: A Citation Analysis of
Composition Papers,” College and Undergraduate Libraries 1, no. 1 (1994): 43–65.
25. ——— and Jeane Koekkoek Stierman, “The Core List Mirage: A Comparison of the Jour-
nals Frequently Consulted by Faculty and Students,” College & Research Libraries 58 (Jan. 1997):
48–55.
26. Louise S. Zipp, “Thesis and Dissertation Citations as Indicators of Faculty Research Use
of University Library Journal Collections,” Library Resources and Technical Services 40 (Oct. 1996):
335–42.
27. The library had access to two databases containing some full-text articles (Gale Expanded
Academic and General Business File) beginning in 1996, but their full-text content was too un-
stable to be verified reliably as a source of availability. Citations were checked against Lexis-Nexis
Academic as a source for newspapers and law journal articles cited in theses a er 1999. Rarely,
students’ citations indicated that the cited reference was accessed through a database, in which
case the citation was counted as locally held, but in most cases citations did not include any
information about electronic access.
28. Davis and Cohen, “The Effect of the Web on Undergraduate Citation Behavior 1996–
1999.”
29. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update.”
30. ———, “Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholar-
ship in a Networked Age.”
CHOICE - C&RLFull Size Ad
CHOICE.C&RL(Full) 12/23/04 10:59 AM Page 1
1
2
3
4
4 DIFFERENT WAYS
TO PURCHASE MATERIALS FOR
YOUR COLLEGE LIBRARY
Read CHOICE Magazine for hundreds of brief,
critical, expert reviews of new books each month.
Every issue features essays, special features and lists of
forthcoming titles.
Use CHOICE Reviews on Cards.
You’ll get a complete set of reviews and nothing but the
reviews! Each pack is easily divided for quick reference
and portability.
Visit ChoiceReviews.online FREE for 60 days!
Simply click on www.ChoiceReviews.org to gain 24/7
access to over 100,000 reviews for up to 20 authorized
users. Begin your Free Trial today!
Try our ChoiceReviews.online Site License
Edition FREE for 60 days! Now available, our
newest CHOICE option eliminates passwords for quick and
easy campus-wide Internet access to our entire database of
reviews.
The Go
ld
Standa
rd
Among
Academ
ic
Library
Review
Journa
ls.
60 DAYS ONLINE FOR FREE!
To try ChoiceReviews.online Password or ChoiceReviews.online
Site License Edition FREE for 60 days, click on www.ChoiceReviews.org.
To subscribe to CHOICE Magazine or Reviews On Cards,
contact us at www.ala.org/acrl/choice, or call (240) 646-7027.
CHOICE
Meeting the Book Reviewing Needs of
College Libraries and Their Faculty for Over 35 Years!
CHOICE is a publication of the Association of College & Research Libraries,
a Division of the American Library Association.
www.ala.org/acrl/choice
http:www.ChoiceReviews.org