Evidence Based Library and Information Practice
Evidence Based Library and Information Practice 2009, 4:4
Evidence Based Library and Information Practice
Article
It’s All Fun and Games until Someone Learns Something: Assessing the Learning
Outcomes of Two Educational Games
Jennifer McCabe
Assistant Director
James Madison University, East Campus Library
Harrison, VA 22807, USA
Email: mccabeja@jmu.edu
Steven Wise
Vice President of Research and Development
Northwest Evaluation Association
Lake Oswego, OR 97035, USA
Email: Steve.wise@nwea.org
Received: 28 August 2009 Accepted: 2 November 2009
© 2009 McCabe and Wise. This is an Open Access article distributed under the terms of the Creative Commons
Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution,
and reproduction in any medium, provided the original work is properly cited.
Abstract
Objective – To determine whether educational games can be designed that are both
fun and effective in improving information seeking skills.
Methods – Two skills that are known to be particularly difficult for students taking a
required information literacy test were identified. These skills are the ability to
identify citations and the ability to search databases with keywords. Educational
games were designed to address these two skills. The first game, Citation Tic Tac Toe,
placed commonly used bibliographic citations into a tick tac toe style grid. Students
were required to play the Tic Tac Toe game and subsequently given citation
identification exercises. The second game arranged key concepts related to search
phrases in a Magnetic Keyword interface. Students were observed searching
databases before and after playing the Magnetic Keyword game and their pre‐ and
post‐play searches were analyzed.
Results – Students who played the Tic Tac Toe game improved more from pretest to
posttest than students who only took an online tutorial. In addition, students who
played the Magnetic Keyword game demonstrated quicker database searching for
their topics and expressed increased satisfaction with their results.
6
mailto:mccabeja@jmu.edu
mailto:Steve.wise@nwea.org
http://creativecommons.org/licenses/by/2.0
Evidence Based Library and Information Practice 2009, 4:4
7
Conclusions – Games can be created which have measurable educational outcomes
and are fun. It is important, however, to establish the educational objective prior to
beginning game design.
Introduction
Scholars in many disciplines are beginning to
see the value of games and investigate their
design and use as educational tools. In 2005,
the American Library Association hosted
Gaming in Libraries, a symposium on the
relationship between games and library
services. Since then, countless other venues
have sprung up for the discussion of the value
of gaming for educational purposes. Those
interested include all types of educators, game
designers, computer programmers, marketers,
and even librarians. Educators are using
games to teach everything from financial
literacy (WPI Media Relations) to social
responsibility
to surgical
procedures (Sourin 6‐9). Games hold vast
appeal to educators and players in all age
groups.
Fortunately for those interested in exploring
the use of games, profiles of people who play
online games, and the kinds of games that
appeal to different people are plentiful. The
Pew Internet & American Life Project reported
that fifty‐three percent of American adults
over eighteen years old play games, with
twenty one percent of adults playing every
day (Lenhart and Jones 1‐9). An earlier Pew
report, Let the Games Begin: Gaming Technology
and College Students, found that in 2003 seventy
percent of college students reported playing
online games, and one hundred percent of
students surveyed had played a video game at
some point in their lives (Jones 1‐14). Evidence
suggests that players are likely to be educated
and dedicated to game‐play.
Given the plethora of data available about
adult gamers, it seems unwise not to
investigate ways of exploiting the medium for
use in higher education. The challenge of
designing an educational game, that retains
the essential intrinsic motivation of fun, lies in
beginning with the learning objective in mind
and ensuring that the fun objective is
included.
This project originated when two university
administrators at James Madison University
(JMU), Virginia, USA asked the same person
to address two different challenges. In 2004
the Dean of Libraries and Educational
Technologies at JMU began to notice many
students engaged in what appeared to be
gaming activity on the public computers in the
library. Concurrently, an Associate Dean in
the college that includes the health science,
nursing, and social work programs identified
a need to help pre‐professional students
understand the complex issues related to
health literacy. After considerable thought
and discussion the decision was made by the
Dean of the Library to apply for an Institute
for Museum and Library Services (IMLS)
national leadership grant to determine
whether a game‐like environment could meet
the unique challenges of teaching health
literacy and information literacy.
This paper describes the design and testing of
two games specifically created to impart
information literacy skills in measurable ways.
The first game, Citation Tic Tac Toe, was
designed to help undergraduate students
correctly identify a variety of bibliographic
citations. The second game, Magnetic Keyword,
was designed to assist the same student
population translate phrases into keywords in
order to search bibliographic databases more
effectively.
Literature Review
In recent years, the literature on games in
education has grown substantially. Literature
about the relationship between games and
library services is extensive, ranging from
Evidence Based Library and Information Practice 2009, 4:4
discussions on the characteristics of gamers
(Martin and Ewing 209‐225), the process of
designing games (Markey et al. 663‐681), what
distinguishes a game from other active
learning techniques (both electronic and
otherwise), the unique challenges of collecting
games in libraries, to student satisfaction with
games (Smith 1‐12), or how games can be
integrated into library programming at all
levels. When the game creation work
described in this paper began, there was a
notable absence in the literature of models for
assessing games and evidence demonstrating
the effectiveness of games relative to
traditional methods in teaching specific library
related skills. Since that time, there has been
continued research into the efficacy of games
as educational tools.
The previously mentioned lack of evidence
does not mean that games don’t teach.
Indeed, programs similar to that described by
Smith, in which the primary goal was to
counteract the boredom that students felt
while in mandatory classes, often get high
student satisfaction marks. Surely students
are more likely to learn if they are engaged
and have fun in class? But in today’s
educational atmosphere the need for evidence
and accountability is stronger than for fun and
engagement.
A recent addition to the nascent concept of
demonstrating that games are effective is the
study conducted by Hickey, Ingram‐Goble,
and Jameson on the use of a commercial
immersive online game to teach socio‐
scientific inquiry in ecological sciences. They
describe studies wherein sixth‐grade students
used the game Quest Atlantis to learn specific
federally mandated scientific concepts.
Moreover, they describe the assessment of the
use of Quest Atlantis and offer principles for
the design of assessments in virtual
environments. The experience described is
valuable and significant since, as they state,
“Failure to produce convincing evidence of
increased achievements (and models for
obtaining such evidence) is likely to constrain
research funding and broad adoption of
educational games and simulation, and
redirect educational technology resources
toward narrow test‐preparation and tutoring
applications.” (Hickey, Ingram‐Goble, and
Jameson 187‐208).
Constance Steinkuehler and Sean Duncan
conducted a more ambitious study to
determine whether the wildly popular
multiplayer game, World of Warcraft, could be
shown to spontaneously engender “scientific
habits of mind”. They analyzed the
transcripts of nearly two thousand online
discussions carried out by players of World of
Warcraft to determine whether they contained
evidence of eighteen different markers of
scientific thought. They found that the
discourse that playing the game elicited
consisted of “social knowledge construction”
eighty‐six percent of the time ( 530‐543). They
further concluded that the “predominant
epistemological disposition exhibited in the
forum posts was ‘evaluative’ and therefore
appropriate to science” (530‐543).
These examples represent a small selection of
the ways that games are being integrated into
education and studied for impact. It must be
noted that online games themselves are so
varied that no single model of assessing their
impact will work for all activities described as
games. Online games represent novel ways of
interacting, of being in real and virtual worlds,
and of learning. Quantitative methods of
generating scientific evidence can work in
some cases; qualitative methods are more
appropriate for others. This point was
eloquently argued by Steinkuehler, Black &
Clinton, who stated that “the phenomena of
study are ever widening while the range of
methods considered legitimate for their
investigation is increasingly narrowed.” (95‐
100). In keeping with this sentiment, this
paper describes the study of two games by
two different means. The conclusions of each
experiment, while generally in agreement with
each other, illuminate the unique challenges of
using games to address specific learning
objectives. Most interesting, perhaps, are the
questions that the study raises regarding how
to teach the constellation of skills that
contribute to research and information
literacy.
8
Evidence Based Library and Information Practice 2009, 4:4
Context and Aims
James Madison University (JMU) is a
comprehensive university with approximately
eighteen thousand students located in the
central Shenandoah Valley of Virginia, USA.
The University’s mission is to prepare
students to be educated and enlightened
citizens who lead productive and meaningful
lives. As an Association of College and
Research Libraries (ACRL) Best Practice
institution for its information literacy
program, all undergraduate students must
complete an online information literacy
tutorial and pass an information seeking skills
test (ISST) before they can register as second‐
year students. This requirement means that
valuable data is gathered each year about
students’ skills related to the ACRL
Information Literacy Competency Standards
(Association of College and Research
Libraries). Students who do not pass the ISST
may choose to repeat sections of the tutorial or
meet with a librarian for extra assistance with
information seeking. Because of this
requirement, librarians have access to
extensive data as well as anecdotal
information about which specific skills hinder
students’ ability to pass the test. Librarians at
JMU have learned that two skills that students
struggle with are the ability to identify
bibliographic citations and the elements that
comprise them (ACRL Information Literacy
Competency Standard 1.2), and the ability to
break a topic into keywords (ACRL
Information Literacy Competency Standard
2.2.b) (Association of College and Research
Libraries).
In 2005, a grant was received from the
Institute for Museum and Library Services to
create games that teach health literacy and
information literacy. The grant application
stated, “The goals of this project are threefold;
to produce a series of games that help develop
health literacy and information literacy skills,
to evaluate whether traditional information
literacy skills can be acquired through game‐
like experiences, and to determine if this
approach is more appealing and successful
with our students than our current text‐based
approaches.” Staff from the JMU Libraries
and Educational Technologies division
worked to create several games. The first
game was a complex role‐playing game
designed to teach pre‐professional health
students about the challenges of compromised
health literacy. The subsequent games, which
are the subject of this paper, were designed to
address students’ ability to correctly identify
bibliographic citations and to identify
keywords related to search phrases.
Because there is currently no “gold standard”
for assessing the impact of educational games
on explicit learning objectives, the authors
chose to evaluate the two games in different
ways. The first study was a quantitative study
in which players took an achievement test (the
pretest), played the game (Citation Tic Tac Toe),
and repeated the achievement test (the
posttest) to determine whether progress was
made in their ability to identify citations.
Players were compared to a control group
who did not play the game. This study design
was chosen since there are absolute correct
and incorrect answers to citation
identification. The goal of this study was to
determine whether students who played the
Citation Tic Tac Toe game were better at
identifying bibliographic citations by type
than students who took the online tutorial.
The team designed the second game, Magnetic
Keyword, to address a competency that does
not lend itself to definitive correct answers.
The ACRL Information Literacy Competency
Standards address the ability to construct a
search query in standard two, specifically
stating in 2.2, “The information literate
student constructs and implements effectively
designed search strategies.” Outcome 2.2b
states that this competency is shown by
“[Identifying] keywords, synonyms and
related terms for the information needed.”
From a practical perspective, this skill is often
addressed in library instruction sessions by
having students break their search phrase into
its component keywords and brainstorming
lists of broader and narrower terms and
synonyms. Measuring a student’s ability to do
this, and to refine their search based on
9
Evidence Based Library and Information Practice 2009, 4:4
results, is a subjective matter that is influenced
by the search topic, the student’s familiarity
with the database content and interface, and
other variables. Because this competency is
more of an art than a science, a qualitative
method was chosen to evaluate the
effectiveness of the Magnetic Keyword game.
Students were videotaped while playing the
game, and recordings were later analyzed to
determine whether search speed and
satisfaction improved after playing.
Citation Tic Tac Toe was developed and studied
first. It is a standard tic tac toe game with
bibliographic citations in each box. Players
select a square on the grid and see a
bibliographic citation with four choices as to
the type of citation. Choices include book,
journal article, a variety of websites,
government document, book chapter,
dissertation, and newspaper articles. If the
player selects the correct type, an X is awarded
for that box. If the player chooses incorrectly,
the game gives the box an O. When an
incorrect choice is made, the game provides
feedback in the form of a clue as to how to
identify that kind of citation. For example, if
the player sees a citation for a book but makes
the incorrect choice that it is a journal article,
the game generates feedback that says “This is
a book. You can tell this quickly by locating
the city and name of the publisher.” This
feature was designed to help the player learn
the rules of citation identification as they play
the game. Once the player correctly identifies
three citations in a row, they win the game. If
they fail to make the correct choices in three
adjacent boxes, the game wins. Upon
completion of each game, the player can
choose to play again. Figure 1 contains a
screen shot of the feedback screen for the Tic
Tac Toe game.
Fig.1 Screen Shot for Tic Tac Toe Game
The second game was designed to look like a
refrigerator with magnetic words affixed to it,
similar to the popular Magnetic Poetry® toy
sold in bookstores and toy shops. One section
of the screen has a simple search phrase
displayed, and the other section contains a
collection of random words interspersed with
words that correspond to the search phrase.
For example, the search phrase “Do students
who participate in sports get higher grades?”
has the words “students”, “sports” and
“grades” mixed in with other random words
like “heart rate” “traffic” and “practice.” In
most cases there are three correct words mixed
in among 20 distracters. The player drags the
identified keywords below the line on the
refrigerator and clicks on submit when it is
believed the correct keywords have been
10
Evidence Based Library and Information Practice 2009, 4:4
selected. The game then generates feedback
based on whether all of the correct words,
some of the correct words, or none of the
correct words have been chosen. Two points
are received for all correct words, one for
some words, and zero points for no correct
words chosen. To integrate this game with a
course assignment, the player can be assigned
to play until a certain score is achieved or
correctly chosen keywords are associated with
a certain number of search phrases. Figure 2
contains a screen shot for the Magnetic Keyword
game.
Both games were designed using Flash with a
Microsoft Access database running in the
background to manage the game data and
player data.
Methods
The impact of the Tic Tac Toe game was
evaluated by comparing the relative
achievement levels of students who had
played the games with a group that had not
played. The central research question
concerned the degree to which students’
proficiency in identifying citations improved
as the result of playing the game.
Seven sections of Gcom 103 (a mandatory
communications class for first‐year students)
participated in the study during the spring,
2008 semester. These students were chosen
based on their enrollment in the course, in
which the online information literacy tutorial
is required, and their instructors’ willingness
to participate in the study. It was decided that
a true experimental design, in which each
student would be randomly assigned to either
the experimental group (played the game) or
the control group (did not play the game)
would not be appropriate for this study. This
was because it would be instructionally
awkward for some of the students in a section
to have access to the game, while others did
not. Instead, class sections were randomly
assigned, which resulted in three sections
receiving the game, and four sections not
receiving the game.
Fig. 2. Screenshot for Magnetic Keyword Game
A thirty‐five item multiple‐choice test was
developed to assess student proficiency in
identifying citations. A copy of this test is
included as Appendix 1. Each item used four
response options. All students in the study
were asked by their instructors to take the
citation test twice—both at the beginning and
the end of the semester. Thus, although the
same test was used for both the pretest and
posttest, the testing periods were separated by
nearly three months. For comparison
purposes, students were asked to include their
student ID number.
11
Evidence Based Library and Information Practice 2009, 4:4
The Magnetic Keyword game was tested using
Morae™ usability testing software along with
Bailey’s Usability Testing Environment (UTE).
Morae™ usability testing software is a
program that creates video and audio
recordings of users interacting with computer
applications, including the face of the user and
the software performance. The videos can
then be viewed and tagged for later analysis.
Bailey’s UTE is a piece of software that can run
on top of Morae™. It can be pre‐programmed
with specific tasks for the user to complete, as
well as allotting time for the tasks and
providing a venue for immediate user input.
Bailey’s UTE obviates the need for a human‐
mediated usability test, eliminating the
possibility of interference from the mediator.
Both programs were chosen based on their
current use in the library for website usability
testing and because of the researchers’
familiarity and satisfaction with them.
Student testers were recruited for this study
via a notice on the library’s website and
compensated with a gift card to the university
bookstore. Participants were assigned the task
of searching for journal articles on a given
topic in the database Academic Search
Premier. After each search, the user was
asked via Bailey’s UTE if they were satisfied
with their results. After completing four
unique searches, students were asked to play
the Magnetic Keyword game for five minutes.
They were then asked to search for articles on
four new topics in the same database, and
again asked whether they were satisfied. By
analyzing the time that lapsed between the
first round of searching and the second round
(post game play), the researcher could
determine whether their search speed
improved after playing the Magnetic Keyword
game.
Results
Evaluating the Impact of the Tic Tac Toe Game on
Student Achievement
The data was analyzed to evaluate students’
pretest and posttest scores in order to
determine whether the game had a positive
impact on student achievement. However, the
data analysis was complicated by two factors.
First, some students took only the pretest or
the posttest, but not both. Second, a number
of students did not provide their student ID
numbers when they took a test, which
precluded merging the pretest and posttest
data. Our solution for addressing this
problem is described below.
Table 1 shows the basic descriptive statistics
for the overall pretest and posttest data. At
both time points, the reliability estimates
(using coefficient alpha) indicated satisfactory
reliability. In addition, as would be expected,
test performance was found to generally
improve from pretest to posttest, as students
learned more about citation identification.
To accurately compare the experimental and
control groups, only the data from students
who took both the pretest and posttest (and
provided IDs on each occasion) were analyzed
further. The results for these students, shown
in Table 2, showed several interesting
findings. First, the 27 students who played the
game showed a strong, statistically significant
pretest‐posttest gain. The effect size index (d)
indicated that the mean gain was nearly two‐
thirds of a standard deviation. However, the
47 students who did not play the game also
showed significant improvement in citation
Table 1
Descriptive Statistics for the Citations Test
Test N Minimum Maximum Mean Standard Deviation Reliability
Pretest 149 8 31 21.31 4.97 .78
Posttest 134 13 32 22.72 4.64 .77
12
Evidence Based Library and Information Practice 2009, 4:4
identification, albeit to a lesser extent. For
these students, test performance improved
significantly, with an effect size of just under
one third of a standard deviation. Hence, the
overall findings were that (a) both groups
improved in citation identification, (b) the
group playing the game improved, on
average, by a larger amount, and (c) the group
playing the game performed much higher at
posttest.
One problematic aspect of the results shown in
Table 2 is that the experimental and control
groups also differed markedly in their pretest
means. The group that played the game had a
much higher pretest score than the group that
did not play the game. This complicates
interpretations of the results, because the large
differences in posttest performance are to
some extent confounded by the pretest
differences. To address this issue, an analysis
of covariance (ANCOVA) was performed on
the posttest scores, with pretest scores used as
a covariate. The assumptions of the ANCOVA
model (homogeneity of variance and
homogeneity of regression) were found to be
satisfied in the analysis. The results showed
that the adjusted posttest scores were higher
for the group that played the game (mean =
25.27) than the non‐game group (mean =
23.14). This difference was statistically
significant (F(1,71) = 8.69, p = .004) and
corresponded to an effect size of .74 standard
deviations. Thus, even when there was
control for the differences in pretest scores, the
group that played the game substantially
outperformed the group that did not.
Evaluating the Impact of the Magnetic Keyword
Game on Student Search Speed and Satisfaction
Evaluation of the impact of the Magnetic
Keyword game was based on whether students’
speed and self‐reported satisfaction improved
after playing the game. The test was
administered using Bailey’s Usability Testing
Table 2
Test Statistics for Students Who Took Both the Pretest and Posttest
Pretest Posttest
Played Game? Mean SD Mean SD t t Prob.
Effect
Size (d)
Yes (N = 27) 23.78 4.34 26.37 3.73 4.71 < .001 0.65
No (N = 47) 21.13 4.60 22.51 4.34 2.75 .008 0.31
Environment (UTE) to display the questions
and record the answers, and Morae™ was
used to record the search sessions. A
convenience sample of five undergraduate
students completed the test protocol. The
protocol for the test was as follows:
1. Ask the user to search in Academic
Search Complete for articles on the
topic “Does eating cruciferous
vegetables prevent cancer in women?”
Ask whether the user is satisfied with
the results.
2. Ask the user to search in Academic
Search Complete for articles on the
topic “What kind of pollution occurs
in water sources near paper mills?”
Ask whether the user is satisfied with
the results.
3. Ask the user to search in Academic
Search Complete for articles on the
topic “What is the average wage for a
construction worker in the United
States?” Ask whether the user is
satisfied with the results.
4. Ask the user to search in Academic
Search Complete for articles on the
13
Evidence Based Library and Information Practice 2009, 4:4
topic “Is child abuse more likely to
occur in families living below the
poverty line?” Ask whether the user
is satisfied with the results.
5. Ask the user to play the Magnetic
Keyword game to identify the
keywords in at least 4 search phrases.
6. Ask the user to search Academic
Search Complete for the topic “Does
recreational reading improve grades?”
Ask whether the user is satisfied with
the results.
7. Ask the user to search Academic
Search Complete for the topic “Is there
racism in American colleges?” Ask
whether the user is satisfied with the
results.
8. Ask the user to search Academic
Search Complete for the topic “How
prevalent is post‐traumatic stress
among Iraq war veterans?” Ask
whether the user is satisfied with the
results.
9. Ask the user to search Academic
Search Complete for the topic “What
diseases are affected by volcanic
eruptions?” Ask whether the user is
satisfied with the results.
10. Ask whether the user would describe
the game as fun (on a scale of 1‐10
with 1 being the least fun and 10 being
the most fun.
11. Ask whether the user would
recommend the game to a friend (yes
or no).
12. Ask what the user would do to
improve the game (fill in the blank).
The data was analyzed by viewing the video
recordings of the students working through
the test protocol and recording the amount of
time between initiating the searches and
indicating satisfaction with results. The time
each student spent searching in the database
decreased in each player after they played the
Magnetic Keyword game. For the first search,
the average time it took for students to register
satisfaction with their results was 184 seconds.
For the final search, the average time between
searching and satisfaction was 55 seconds.
Figure 3 below represents the average search
time for the eight search tasks described
above.
Discussion
Each experiment described above included
several survey questions to determine whether
the students who participated found the
activity to be fun. Because fun is an intrinsic
motivator, a game that can simultaneously
educate and entertain is a powerful tool.
Students who played the Citation Tic Tac Toe
game described it as “a nifty program” and “a
fun way to practice the information.” Several
student evaluators offered suggestions on how
to make the game more useful, including
increasing the number and types of citations
14 Fig. 3. Average Search Time per Task
Evidence Based Library and Information Practice 2009, 4:4
and creating a pre‐play tutorial. Comments
indicated that students enjoyed the game as it
helped them achieve the goal of correctly
identifying citations, a skill they realize they
must master. Because of the simplicity of the
interface and gameplay, it is unlikely that any
student would play it solely for entertainment.
Comments on the Magnetic Keyword game
indicated that students found it to be
moderately fun, with three players answering
the question related to recommending the
game to a friend affirmatively and two
negatively. In keeping with the sentiments
expressed for the Tic Tac Toe game, one player
wrote“…I think the game has been made as
fun as possible.” Several players indicated
that they would prefer more challenging
search phrases. At the conclusion of one
recording, a student is overheard saying, “Is
that all? I want to search more which was
interpreted as an endorsement of the “replay
value” of the game. As a usability test, the
results indicated that the game was highly
usable, with no student experiencing
navigation or game playing problems. This
game could be further studied by integrating
it into an advanced research and writing
assignment and determining whether it aids
students in their database searching skills in a
“real‐world” situation. The structure of the
game is such that additional search phrase and
distracter data can be added to increase the
replay value or to create a subject specific
game.
It should be noted that the Magnetic Keyword
game is limited because most of the distracter
words were randomly chosen and bear no
relation to the correct keywords. This made it
easier than necessary for players to make the
correct choices. One way to make the game
more challenging while reiterating the
keyword concept would be to include more
word variations instead of random unrelated
words as distracters. For example, in the
search where the phrase is “Do students who
participate in sports get higher grades?” the
distracters could include words like grading,
graduate, graded, study, etc. Carefully chosen
distracters could demonstrate the value of
truncating keywords. An additional limitation
of the research design is the fact that students
likely improved their familiarity and skill with
the Academic Search Premier database during
the course of the test. A more thorough test
could use several different databases with
different interfaces for comparison.
The experience of creating and studying two
different kinds of games demonstrates that
games can prove effective for learning certain
kinds of skills. Reflection on the processes
involved reveals that the most interesting
learning occurs when the game in use is an
analogy for a skill, as opposed to simply using
the game to deliver new content. Future
investigation into the use and study of games
as analogies for specific skills, such as that
conducted by Steinkuehler and Duncan (530‐
543), would reveal the pedagogical value of
commercial games, obviating the need for the
intense design‐assessment cycle in which
educational game designers must currently
engage. Because commercial games are a for‐
profit business, the resources and talent they
attract is unmatched in academia. The future
study of games in higher education should
concentrate on exploring how commercial
games teach and how best to integrate them
into our curricula, rather than on designing
new games.
Conclusions
This research found that it is possible to
design games that are both fun and effective in
imparting educational skills. A game‐like
approach is one that integrates graduated
levels of expertise, new powers, and rewards
with subject‐specific content. The games
described here integrated game‐like
approaches with identification of
bibliographic citations and isolation of
keywords from search phrases. Games that
attempt to teach new skills are easier to design
and measure than games that attempt to
cultivate new behaviors. Reflection on the time
spent developing and analyzing the games
compared to studies of the novel uses of
existing games leads the authors to conclude
15
Evidence Based Library and Information Practice 2009, 4:4
that the latter approach is more efficient and
richer with research possibilities.
In considering how to integrate games into
education, it is wise to heed the adage that
“everything looks like a nail when all you
have is a hammer.” Some situations lend
themselves well to a game approach, while
others may not. Sara deFrietas states that
“…games and simulations can effectively
support learners with skills‐based needs (such
as literacy and numeracy)” ( 249‐264) an
assertion that is confirmed in both the creation
and the study of the Tic Tac Toe game.
Citation identification is a discrete skill, one
which can be learned effectively by using the
game.
As noted earlier, constructing effective search
strategies is more an art than a science. To be
a good searcher one must be familiar with a
variety of databases and their unique thesauri,
the lexicon of the search topic, the conventions
of research of the field, and even the places
where the relevant scholarship is being
conducted. Scholarly literature published in
languages other than English presents unique
challenges to English‐only speakers.
Moreover, there is a certain element of
creativity involved in searching, as a dose of
curiosity and bravery are required to
experiment with word combinations. All of
these are characteristics that develop with
time, experience, failure, and maturity. To
assume that the ability to be an artful searcher
can be deconstructed in the same way as the
identification of citations can is naïve and
potentially offensive to the scholars who
produce the work being searched.
In her introduction to the World of Warcraft
study, Constance Steinkuehler astutely
referred to Henri Poincare’s quote, “Science is
built up of facts, as a house is built of stones;
but an accumulation of facts is no more
science than a heap of stones is a house.”
(Steinkuehler and Duncan 530‐543). The main
challenge librarians face as they attempt to
impart the skills that students need to be
competent researchers lies in guiding them to
make meaning of the information they find.
In approaching the myriad of challenges of
teaching research in an ever more complex
environment, wise librarians will differentiate
between what students must know absolutely
and what skills must be developed
incrementally, and choose approaches
accordingly. Successful games can be
designed that include the intrinsic motivator
of fun, but care should be taken to avoid the
implication that all learning must be fun.
Works Cited
Association of College and Research Libraries.
Information Literacy Competency
Standards for Higher Education.
Chicago: American Library Association,
2000.
de Freitas, Sara, and Martin Oliver. ʺHow Can
Exploratory Learning With Games and
Simulations Within the Curriculum Be
Most Effectively Evaluated?ʺ Computers
& Education 46.3 (2006): 249‐64.
Hickey, Daniel T., Adam Ingram‐Goble, and
Ellen M. Jameson. ʺDesigning
Assessments and Assessing Designs in
Virtual Educational Environments.ʺ
Journal of Science Education and
Technology 18.2 (2009): 187‐208.
Jones, Steven. ʺLet the Games Begin: Gaming
Technology and Entertainment Among
College Students.ʺ Pew Internet and
American Life Project. (2003):1‐14. 1 Dec.
2009
Lenhart, Amanda, Sydney Jones, and
Alexandra Rankin Macgill. ʺPew Internet
Project Data Memo re: Adults and Video
Games.ʺ Pew Internet and American Life
Project. (2008):1‐9.1 Dec. 2009
16
Evidence Based Library and Information Practice 2009, 4:4
17
Markey, Karen, et al. ʺDesigning and Testing a
Web‐Based Board Game for Teaching
Information Literacy Skills and
Concepts.ʺ Library Hi Tech 26.4 (2008):
663‐81.
Martin, Justine, and Robin Ewing. ʺPower Up!
Using Digital Gaming Techniques to
Enhance Library Instruction.ʺ Internet
Reference Services Quarterly 13.2 (2008):
209‐25.
Smith, Felicia A. ʺGames for Teaching
Information Literacy Skills.ʺ Library
Philosophy & Practice 9.2 (2007): 1‐12.
Sourin, A. ʺVirtual Orthopedic Surgery
Training.ʺ IEEE Computer Graphics and
Animations.May/June 20.3 (2000): 6‐9.
Steinkuehler, Constance A, Rebecca W. Black,
and Katherine A. Clinton. ʺResearching
Literacy as Tool, Place, and Way of
Being.ʺ Reading Research Quarterly 40.1
(2005): 95‐100.
Steinkuehler, Constance, and Sean Duncan.
ʺScientific Habits of Mind in Virtual
Worlds.ʺ Journal of Science Education &
Technology 17.6 (2008): 530‐43.
WPI Media Relations. ʺWPI Students Create
Online Game to Simulate Massachusetts
State Budget Process.ʺ News Release May
24, 2003. 1 Dec.2009
Evidence Based Library and Information Practice 2009, 4:4
Appendix
Citation Tic Tac Toe Pre/Post‐test
Citation Test
Identify the type of publication for each citation below. Mark the correct answers on a form
provided. Do not write your name on this sheet or the card, please use your peoplesoft number
only on the form provided.
1. Fraser, K. (Ed.). (2005). Education development and leadership in higher education : Developing an
effective institutional strategy. London ; New York: RoutledgeFalmer.
A‐‐Book
B‐‐ Journal article
C‐‐ Magazine article
D‐‐Chapter or article in a book
2. King, Jacqueline. “Academic success and financial decisions: Helping students make crucial
choices.” Engaging the First‐Year Student. Ed. Robert Feldman, Mahwah, NJ: Lawrence
Erlbaum, 2005. pp. 3‐25.
A‐‐Newspaper article
B‐‐Book
C‐‐Government publication
D‐‐Chapter or article in a book
3. Flea Beetles. (2006, April). Horticulture, 103, 16.
A‐‐Government publication
B‐‐Book
C‐‐Magazine article
D‐‐Chapter or article in a book
4. United States. Presidentʹs Committee on Employment of the Handicapped. (1980). The college
student with a disability: A faculty handbook. Washington, D.C.: U.S. G.P.O.
A‐‐Government publication
B‐‐Magazine article
C‐‐Journal article
D‐‐Chapter or article in a book
5. Ghaderi, A. (2006). Attrition and outcome in self‐help treatment for bulimia nervosa and
binge eating disorder: A constructive replication. Eating Behaviors, 7(4), 300‐308.
A‐‐Magazine article
B‐‐Journal article
C‐‐Book
D‐‐Chapter or article in a book
6. Hartley, J.T., Harker, J.O., & Walsh, D.A. (1980). Contemporary issues and new directions in
adult development of learning and memory. In L.W. Poon (Ed.), Aging in the 1980’s:
Psychological issues (pp. 239‐252).Washington, DC : American Psychological Association.
A‐‐Book
B‐‐Journal article
C‐‐Newspaper article
18
Evidence Based Library and Information Practice 2009, 4:4
D‐‐Chapter or article in a book
7. Bremner, M., Aduddell, K., Bennett, D., & VanGeest, J. (2006). The use of human patient
simulators: Best practices with novice nursing students. Nurse Educator, 31(4), 170‐174.
A‐‐Journal article
B‐‐Book
C‐‐Magazine article
D‐‐Chapter or article in a book
8. United States. Congress. House. Committee on Ways and Means. (1992). Adequacy of the
administrationʹs fiscal year 1993 budget proposals for the internal revenue service.
Washington: U.S. G.P.O;
A‐‐Magazine article
B‐‐Government publication
C‐‐Journal article
D‐‐Chapter or article in a book
9. Girden, E. R. (1996). Evaluating research articles from start to finish. Thousand Oaks, Calif.: Sage
Publications.
A‐‐Journal article
B‐‐Magazine article
C‐‐Book
D‐‐Chapter or article in a book
10. Turman, L.U., Self, P. C., Calacaro, P.V. (2004). Teaching a web‐based course in health
informatics. Reference Services Review, 32(1), 21‐25.
A‐‐Government publication
B‐‐Book
C‐‐Journal article
D‐‐Chapter or article in a book
11. Mayouf, J.S. (1998, December 5). Crisis in sub‐prime mortgage market worries investors.
Philadelphia Enquirer, pp. A31, A14.
A‐‐Journal article
B‐‐Newspaper article
C‐‐Magazine article
D‐‐Chapter or article in a book
12. Condon, John C. Good Neighbors. 2nd ed. Yarmouth, Me.: Intercultural Press, 1997.
A‐‐Journal article
B‐‐Magazine article
C‐‐Book
D‐‐Chapter or article in a book
13. Zhang, G., & Lu, J. (2003). An integrated group decision‐making method dealing with fuzzy
preferences for alternatives and individual judgments for selection criteria. Group Decision &
Negotiation, 12(6), 501‐515.
A‐‐Journal article
B‐‐Book
C‐‐Newspaper article
D‐‐Chapter or article in a book
19
Evidence Based Library and Information Practice 2009, 4:4
14. Block, Robert. ʺKenya, Tanzania Offer Fertile Soil for Intrigue,ʺ Wall Street Journal, August
11, 1998, A12.
A‐‐Magazine or journal article
B‐‐Book
C‐‐Government publication
D‐‐Newspaper article
15. Goulding, Anne. ʺJoking, Being Aggressive and Shutting People Up: The use of Focus Groups
in LIS Reseach.ʺ Education for Information 15.4 (1997): 331. .
A‐‐Book
B‐‐Journal article
C‐‐Government Publication
D‐‐Chapter or article in a book
16. Cameron, Theresa. 2002. Foster care odyssey : A black girlʹs story. Jackson: University Press of
Mississippi.
A‐‐Book
B‐‐Magazine article
C‐‐Journal article
D‐‐Chapter or article in a book
17. Shapiro, S.C.(1979). The SNePS semantic network processing system. In Findler, N.V., editor,
Associative Networks: Representation and Use of Knowledge by Computers, pages 179‐203.
Academic Press, New York
A‐‐Magazine or journal article
B‐‐Newspaper article
C‐‐Book
D‐‐Chapter or article in a book
18. Anderson, Deborah, Martha Finne, and Jeanette Swofford. 1986. Jasonʹs story : Going to a foster
home. Minneapolis, Minn: Dillon Press.
A‐‐Journal article
B‐‐Magazine article
C‐‐Book
D‐‐Chapter or article in a book
19. ADAMHA news on alcohol, drug abuse, and mental health. Rockville, MD; Washington,
D.C.: U.S. Dept. of Health and Human Services, National Institute on Alcohol Abuse and
Alcoholism; Supt. of Docs., U.S. G.P.O., distributor.
A‐‐Magazine article
B‐‐Government publication
C‐‐Book
D‐‐Chapter or article in a book
20. Kant, Immanuel. “What is Enlightenment?” Trials of Modernity: Europe and the Modern
World. Eds. Stacy Burton and Dennis Dworkin. 3rd ed. Boston: Pearson, 2001. 139‐43.
A‐‐Newspaper article
B‐‐Book
C‐‐Government publication
D‐‐Chapter or article in a book
20
Evidence Based Library and Information Practice 2009, 4:4
21. Walker, J. T., Lofton, S. P., Haynie, L., & Martin, T. (2006). The home health nurseʹs role in
geriatric assessment of three dimensions: Depression, delirium, and dementia. Home
Healthcare Nurse, 24(9), 572‐580.
A‐‐Journal article
B‐‐Book
C‐‐Magazine article
D‐‐Chapter or article in a book
22. “French Find.” Choir & Organ, May/June 2006, 3.
A‐‐Magazine article
B‐‐Government publication
C‐‐Book
D‐‐Chapter or article in a book
23. Allen, J. F., Hendler, J., and Tate, A., editors (1990a). Readings in Planning. Morgan Kaufmann,
San Mateo, California.
A‐‐Magazine or journal article
B‐‐Book
C‐‐Government publication
D. Chapter or article in a book
24. Suellentrop, C. ʺ Playing with Our Heads: Why Video Games are Making our Kids Smarter
and More Obedient.” Utne Reader 139 January‐February (2007): 78‐83.
A‐‐Book
B‐‐Magazine article
C‐‐Newspaper article
D‐‐Chapter or article in a book
25. Lubin, J.S. (1980, December 5). On idle: The unemployed shun much mundane work, at least
for a while. San Francisco Chronicle, pp. A1, A25.
A‐‐Journal article
B‐‐Newspaper article
C‐‐Book
D‐‐Chapter or article in a book
26. United States, General Accounting Office, Money Laundering: Regulatory Oversight of Offshore
Private Banking Activities: Report to the Chairman, Washington, D. C.: The Office, 1998.
A‐‐Newspaper article
B‐‐Book
C‐‐Government publication
D‐‐Chapter or article in a book
27. Yan, Xuetong. ʺTerrorism Remains a Significant Threat to Mankind,ʺ China Daily, [Beijing],
18, No. 5351, September 23, 1998, 4.
B‐‐Essay or book chapter
C‐‐Book
D‐‐Government publication
E‐‐Newspaper article
21
Evidence Based Library and Information Practice 2009, 4:4
Web site evaluation questions
28. Embassy of Brazil. (2005). Brazilian Embassy in Washington. Retrieved March 19, 2006, from
http://www.brasilemb.org
A‐‐Commercial web site
B‐‐Organizational web site
C‐‐Academic web site
D‐‐Government web site
29. Spearing, Melissa K. “Schizophrenia.” National Institute of Mental Health. 1999. National
Institute of Mental Health. 13 April 2005 .
A‐‐Commercial web site
B‐‐Organizational web site
C‐‐Academic web site
D‐‐Government web site
30. The Andy Warhol Foundation. The Warhol Foundation: 1997. Online:
http://www.warholfoundation.org/
A‐‐Government web site
B‐‐Commercial web site
C‐‐Educational web site
D‐‐Organizational web site
31. “Generalized Anxiety Disorder.” MedicineNet.Com. 2005. MedicineNet. 21 September 2007
http://www.medicinenet.com/anxiety/article.htm.
A‐‐Government web site
B‐‐Commercial web site
C‐‐Educational web site
D‐‐Organizational web site
32. Hanson, D. J. (2007). Binge drinking. Alcohol Problems and Solutions Retrieved September 19,
2007, from http://www2.potsdam.edu/hansondj/BingeDrinking.html
A‐‐Government web site
B‐‐Commercial web site
C‐‐Educational web site
D‐‐Organizational web site
22
http://www.brasilemb.org/
http://www.medicinenet.com/anxiety/article.htm
Evidence Based Library and Information Practice 2009, 4:4
23
Acknowledgements
This work was supported in part by a National Leadership Grant from the Institute for Museum and
Library Services (grant # LG06‐05‐0150‐05). In the Digital Services department at James Madison
University Greg Brown (browngh@jmu.edu) contributed substantial effort to the design and
programming of Citation Tic Tac Toe and Harper Holsinger (holsinhd@jmu.edu) contributed
significantly to the design and programming of the Magnetic Keyword game. Both games described
in this paper can be viewed and played at http://www.lib.jmu.edu/games/ .
mailto:browngh@jmu.edu
mailto:holsinhd@jmu.edu
http://www.lib.jmu.edu/games/