Lessons Learned:
A Primo Usability Study

Kelsey Brett,
Ashley Lierman,

and Cherie Turner

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

ABSTRACT

The University of Houston Libraries implemented Primo as the primary search option on the library
website in May 2014. In May 2015, the Libraries released a redesigned interface to improve user
experience with the tool. The Libraries took a user-centered approach to redesigning the Primo
interface, conducting a “think-aloud” usability test to gather user feedback and identify needed
improvements. This article describes the method and findings from the usability study, the changes
that were made to the Primo interface as a result, and implications for discovery-system vendor
relations and library instruction.

INTRODUCTION

Index-based discovery systems have become commonplace in academic libraries over the past
several years, and academic libraries have invested a great deal of time and money into
implementing them. Frequently, discovery platforms serve as the primary access point to library
resources, and in some libraries they have even replaced traditional online public access catalogs.
Because of the prominence of these systems in academic libraries and the important function that
they serve, libraries have a vested interest in presenting users with a positive and seamless
experience while using a discovery system to find and access library information. Libraries
commonly conduct user testing on their discovery systems, make local customizations when
possible, and sometimes even change products to present the most user-friendly experience
possible.

University of Houston Libraries has adopted new discovery technologies as they became available
in an effort to provide simplified discovery and access to library resources. As a first step, the
Libraries implemented Innovative Interface’s Encore, a federated search tool, in 2007. When
index-based discovery systems became available, the Libraries saw them as a way to provide an
improved and intuitive search experience. In 2010, the Libraries implemented Serials Solutions’
Summon. After three years and a thorough process of evaluating priorities and investigating
alternatives, the Libraries made the decision to move to Ex Libris’ Primo, which was done in May
of 2014.

The Libraries’ intention was to continually assess and customize Primo to improve functionality
and user experience. The Libraries conducted research and performed user testing, and in May

Kelsey Brett (krbrett@ua.edu) is Discovery Systems Librarian, Ashley Lierman
(arlierman@uh.edu) is Instructional Design Librarian, and Cherie Turner (ckturner2@uh.edu) is
Chemical Sciences Librarian, University of Houston Libraries, Houston, Texas.

mailto:krbrett@ua.edu
mailto:arlierman@uh.edu
mailto:ckturner2@uh.edu

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

2015 a redesigned Primo search results page was released. One of the activities that informed the
Primo redesign was a “think-aloud” usability test that required users to complete a set of two
tasks using Primo. This article will present the method and results of the testing as well as the
customizations that were made to the discovery system as a result. It will also discuss some
broader implications for library discovery and its effect on information literacy instruction.

LITERATURE REVIEW

There is a substantial body of literature discussing usability testing of discovery systems. In the
interest of brevity, we will focus solely on studies and overviews involving Primo implementations,
from which several patterns have emerged.

Multiple studies have indicated that users’ responses to the system are generally positive; even in
testing of very early versions by a development partner users responded positively overall.1
Interestingly, some studies found that in many cases users rated Primo positively in post-testing
surveys even when their task completion rate in the testing had been low.2 Multiple studies also
found evidence that, although users may struggle with Primo initially, the system is learnable over
time. Comeaux found that the time it took users to use facets or locate resources decreased
significantly with each task they performed,3 while other studies saw the use of facets per task
increase for each user over the course of the testing.4

User reactions to facets and other post-limiting functions in Primo were divided. In one of the
earliest studies, Sadeh found that users responded positively to facets,5 and some authors found
users came to use them heavily while searching,6 while others found that facets were generally
underused.7 Multiple studies found that users tended to repeat their searches with slightly
different terms rather than use post-limiting options.8 Thomsett-Scott and Reese, in a survey of
the literature on discovery tools, reported evidence of a trend that users reacted more positively
to post-limiting in earlier discovery studies,9 while the broader literature shows more negative
reactions in more recent studies. This could indicate that shifts in the software, user expectations,
or both may have decreased users’ interest in these options.

A few specific types of usability problems seem common across tests of Primo and other discovery
systems. Across a large number of studies, it has been found that users—especially undergraduate
students—struggle to understand library and academic terminology used in discovery. Some
terminology changes were made after users had difficulty in the earliest usability tests of Primo,10
but users continued to struggle with terms like hold and recall in item records.11 Users also failed
to understand the labels of limiters,12 and they also failed to recognize the internal names of
repositories and collections.13 Literature reviews on discovery systems have found terminology to
be a common stumbling block for searchers across a wide number of individual studies.14

Similarly, users often struggle to understand the scope of options available to them when
searching and the holdings information in item records. Users failed in multiple tests to
distinguish between the article level and the journal level,15 could not interpret bibliographic

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

information sufficiently to determine that they had found the desired item,16 and chose incorrect
options for scoping their searches.17 Many studies found that users were unable to distinguish
between multiple editions of a held item when all item types or editions were listed in the
record.18 In other cases, users had difficulty interpreting locations and holdings information for
physical items.19

Among the needs and desires expressed by and for Primo users in the literature, two in particular
stand out. First, many users expressed a desire for more advanced search options; some wanted
more complexity in certain facets and the ability to search within results,20 while other users
simply wanted an advanced search option to be available.21 Secondly, a large number of studies
indicated that instruction on Primo or other discovery systems was needed for users to search
effectively. In some cases this was the conclusion of the researchers conducting the study,22 while
in other cases users themselves either suggested or requested instruction on the system.23

It is also worth noting that it has been questioned whether usability testing as a whole is a
sufficient mechanism for evaluating discovery-system functionality. Prommann and Zhang found
that usability testing has focused almost exclusively on the technical functioning of the software
and not adequately revealed the ability of discovery systems like Primo to successfully complete
users’ desired tasks.24 They proposed hierarchical task analysis (HTA) as an alternative, to
examine users’ most frequent desires and the capacity of discovery systems to meet them.
Prommann and Zhang acknowledged, however, that as HTA is completed by an expert on the
system rather than by an actual user, some of the valuable information derived from usability
testing (including terms and functions that users do not understand, however well-designed) is
lost in the process; they concluded that a combination of the two systems of testing is ideal to
retain the best of both.

BACKGROUND

At the University of Houston Libraries, the Resource Discovery Systems department (RDS) is
responsible for the maintenance and development of Primo. However, it is important to RDS to
gather feedback and foster buy-in from stakeholders in the Library before making changes to the
system. To that end, RDS works with two committees to assess the system and make
recommendations for its improvement. The Discovery Usability Group and the Discovery Advisory
Group include members from public services, technical services, and systems; each member brings
a unique perspective on discovery. The Discovery Usability Group is charged with assessing the
discovery system through a variety of methods including usability testing, focus groups, and user
interviews. The Discovery Advisory Group reviews results of user testing and makes
recommendations for improvement. All changes to the discovery system are reviewed by the
Groups before they are released for public use.

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

In fall 2014, several months after the Primo implementation, the Discovery Usability Group
conducted a focus group with student workers from the library’s information desk (a dual
reference and circulation desk) to solicit feedback about the functionality of Primo and
suggestions for its improvement. In the meantime, the Discovery Advisory Group was testing
Primo and evaluating Primo sites at peer and aspirational institutions. The groups used the
information collected through the focus group and research on Primo to make recommendations
for improvement. RDS has access to a Primo development sandbox, and many of the
recommended changes were made in the sandbox environment and reviewed by the two groups
prior to public release.

Changes to the search box can be seen in figure 1. Rarely used tabs were replaced with a drop-
down menu to the right of the search box to allow users to limit to “everything,” “books+,” or
“digital library.” To increase visibility, links to “Advanced Search” and “Browse Search” were made
larger and more spacing was added.

Live site:

Development Sandbox:

Figure 1. Search Box in Live Site (Above) and Development Sandbox (Below) at Time of Testing

Changes were also made to create a cleaner and less cluttered search results page (see figure 2).
More white space was added, and the links (or tabs) to “View Online,” “Request,” “Details,” etc.,
were redesigned and renamed for clarity. For example, the “View Online” link was renamed to
“Preview Online” because it opens a box within the search results page that displays the item. The
groups believed “Preview Online” more accurately represents what the link does.

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

Live Site:

Development Sandbox:

Figure 2. Search Results in Live Site (Above) and Development Sandbox (Below) at Time of
Testing

The facets were also redesigned to look cleaner and larger to attract users’ attention (see figure 3).

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

Live Site:

Development Sandbox:

Figure 3. Facets in Live Site and Development Sandbox at Time of Testing

Both groups were happy with the changes to the Primo development sandbox but wanted to test
the effect of the changes on user search behavior before updating the live site. The Discovery
Usability Group conducted a usability test within the development sandbox. The goal of the test
was to find out if users could effectively complete common research tasks using Primo. With that
goal in mind, the group developed a usability test and conducted it during the spring semester of
2015.

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

METHODOLOGY

The Discovery Usability Group developed a usability test using a “think-aloud” methodology,
where users were asked to verbalize their thought process as they completed research tasks
through Primo. Four tasks were designed to mirror tasks that users are likely to complete for class
assignments or for general research. To minimize the testing time, each participant completed two
tasks, with the facilitators alternating between two sets of tasks from one participant to the next.

Test 1

Task 1: You are trying to find an article that was cited in a paper you read recently. You have the
following citation:

Clapp, E., & Edwards, L. (2013). Expanding our vision for the arts in education. Harvard
Educational Review, 83(1), 5–14.

Please find this article using OneSearch [the public-facing name given to the Libraries’ Primo
implementation].

Task 2: You are doing a research project on the effects of video games on early childhood
development. Find a peer-reviewed article on this topic, using OneSearch.

Test 2

Task 1: Recently your friend recommended the book The Lighthouse by P. D. James. Use
OneSearch to find out if you can check this book out from the library.

Task 2: You are writing a paper about the drug cartels’ influence on Mexico’s relationship with the
United States. Find a newspaper article on this topic, using OneSearch.

Two facilitators set up a table with a laptop in the front entrance of the library. They alternated
between the facilitator and note-taker roles. Another group member took on the role of “caller”
and recruited library patrons to participate in the study. The caller set up a table visible to those
passing by with library-branded T-shirts and umbrellas to incentivize participation. The caller
explained what would be expected of the potential participant and went over the informed-
consent document. After signing the form, the participant performed two tasks. After the test the
participant received a library T-shirt or umbrella, and snacks.

The facilitators used Morae Usability Software to record the screen and audio of each test.
Participants were asked for permission to record their sessions, but could opt out. During the
three hour testing period, fifteen library patrons participated in the study, and fourteen sessions
were recorded. Of the fifteen participants, thirteen were undergraduate students (four freshman,
one sophomore, seven juniors, and two seniors), one was a graduate student, and one was a post-
baccalaureate student. The majority of the participants were from the sciences, along with two
students from the College of Business and two from the School of Communications. There were no
participants from the humanities.

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

The facilitators took notes on a rubric (see table 1) that simplified the processes of coding and
reviewing the recordings. After the usability testing, the facilitators reviewed the notes and
recordings, coded them for common themes and breakdowns, and prepared a report of their
findings and design recommendations. The facilitators sent the report, along with audio and
screen recordings, to the Discovery Advisory Group, who reviewed them along with RDS. The
Discovery Advisory Group made additional design recommendations, and RDS used the
information and recommendations to implement additional customizations to the Primo
development sandbox.

Preliminary Questions

ASK: What is your affiliation with the
University of Houston? Year? Major?

ASK: How often do you use the library
website? For what purpose(s)?

Task 1

Describe the steps the participant took to
complete the task S/U

ASK: How did you feel about this task?
What was simple? What was difficult?

ASK: Is there anything that would make
completing this task easier?

Task 2

Describe the steps the participant took to
complete the task S/U

ASK: How did you feel about this task?
What was simple? What was difficult?

ASK: Is there anything that would make
completing this task easier?

Follow-up Question

ASK: What can we do to improve the
overall experience using OneSearch?

Table 1. Task Completion Rubric for Test 1

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

RESULTS

Test 1, Task 1

You are trying to find an article that was cited in a paper you read recently. You have the following
citation:

Clapp, E., & Edwards, L. (2013). Expanding our vision for the arts in education. Harvard
Educational Review, 83(1), 5–14.

Please find this article using OneSearch.

Participant Time on Task Task Completion

1 1m 54s Y

2 4m 13s Y

3 1m 26s Y

4 1m 17s Y

5 1m 26s Y (required assistance)

6 1m 43s Y

7 1m 27s Y

8 1m 5s Y

Table 2. Results for Test 1, Task 1

All eight participants successfully completed this task, although sophistication and efficiency
varied between participants. Some searched by the authors’ last names, which was not specific
enough to return the item in question. Four participants attempted to use advanced search or the
drop-down menu to the right of the search box to pre-filter their results. Two participants viewed
the options in the drop-down menu, which were “everything,” “books+,” and “digital library,” and
left it on the default “everything” search. When prompted, the participants explained that they
were expecting the drop-down to contain title and/or author limiters. Similarly, participants
expected an author limiter in the advanced search.

The citation format seemed to confuse participants, and they tended to search for the piece of
information that was listed first—the authors—rather than the most unique piece of
information—the title. If the first search did not return the correct item in the first few results, the
participant would modify their search by searching for a different element of the citation or
adding another element of the citation to the initial search until the item they were looking for
appeared as one of the first few results. Participant 5 thought they had successfully completed the

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

task, but the facilitator had to point out that the item they chose did not meet the citation exactly,
and on the second try they found the correct item.

Participant 2 worked on the task for more than four minutes, significantly longer than the other
seven participants. They immediately navigated to advanced search and filled out several fields in
the advanced search form with the elements of the citation. If the search did not return their item,
they added more elements until they finally found it. Simply searching the title in the citation
would have returned the item as the first search result. Filling out the advanced search form with
all of the information from the citation does not necessarily increase a user’s chances of finding
the item in a discovery system, though it might do so when searching in an online catalog or
subject database.

The Discovery Advisory and Usability Groups made two recommendations to address some of the
identified issues: include an author search option in the advanced search, and add an “articles+”
option to the drop-down menu on the basic search. RDS implemented both recommendations. The
Discovery Usability Group identified confusion around citations as a common breakdown during
this task. The groups recommended providing instructional information about searching for
known items to address this breakdown; however, RDS is still working on an effective method to
provide this information in a simple and visible way.

Test 1, Task 2

You are doing a research project on the effects of video games on early childhood development. Find
a peer-reviewed article on this topic, using OneSearch.

Participant Time on Task Task Completion

1 3m 44s Y

2 2m 21s Y

3 5m 23s Y (required assistance)

4 2m 5s Y

5 3m 32s Y

6 2m 45s Y

7 3m 8s Y

8 3m 1s Y (required assistance)

Table 3. Results for Test 1, Task 2

All eight participants successfully found an article on this topic, but were less successful in
determining whether the article was peer-reviewed. Only one participant used the “Peer-reviewed

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

Journals” facet without being prompted. Three users noticed the “[Peer-reviewed Journal]” note in
the record information for search results, and used it to determine if the article was peer-reviewed.
One participant went to the full-text of an article, and said it “seemed” like it was peer-reviewed
and considered the task complete. The resource type facets were more heavily used during this
task than the “Peer-reviewed Journals” facet, despite its being promoted to the top of the list of
facets. Two participants used the “Articles” facet, and two participants used the “Reviews” facet,
thinking it limited to peer-reviewed articles. Participants 3 and 8 needed help from the facilitator
to determine whether a source was peer-reviewed. There was an overall misunderstanding of
what peer-reviewed means, which affected participants’ confidence in completing the task.

The design recommendations based on this task included changing the “Peer-reviewed Journals”
facet to “Peer-reviewed Articles” or simply, “Peer-reviewed.” RDS changed the facet to “Peer-
reviewed Articles” to help alleviate confusion. Additionally, the groups recommended emphasizing
the “[Peer-reviewed Journal]” designations within the search results and providing a method for
limiting to peer-reviewed materials before conducting a search. Customization limitations of the
system have prevented RDS from implementing these design recommendations yet. A way to
address the breakdowns caused by misunderstanding terminology also has yet to be identified. It
was disheartening that participants did not use the “Peer-reviewed Journals” facet despite its
being purposefully emphasized on the search results page.

Test 2, Task 1

Recently your friend recommended the book The Lighthouse by P. D. James. Use OneSearch to find
out if you can check this book out from the library.

Participant Time on Task Task Completion

1 1m 7s Y

2 56s Y

3 No recording Y

4 2m 21s Y

5 1m 8s Y

6 2m 14s Y

7 1m 15s Y

Table 4. Results for Test 2, Task 1

All seven participants were able to find this book using Primo, but had difficulty in determining
what to do once they found it. For this task every participant searched by title and found the book
as the first search result. Four users limited to “books+” before searching using the drop-down

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

menu, while the other three remained in the default “everything” search. Only one participant
used the locations tab within the search results to determine availability; the others clicked the
title and went to the item’s catalog record. All participants were able to determine that the book
was available in the library, but there was an overall lack of understanding about how to use the
information in the catalog to check out a book. Participant 1 said that they would write down the
call number, take it to the information desk, and ask how to find it, which was the most
sophisticated response of all seven participants. Participant 4 spent nearly two minutes clicking
through links in the OPAC expecting to find a “Check Out” button and only stopped when the
facilitator stepped in.

A recommended design change based on this task was to have call numbers in Primo and the
online catalog link to a stacks guide or map. This is a feature that may be developed in the future,
but technical limitations prevented RDS from implementing it in time for the release of the
redesigned search interface. Like the previous tasks, some of the breakdowns occurred because of
a lack of understanding of library services. Users easily figured out that there was a copy of the
book in the library, but had little sense of what to do next. None of the participants successfully
located the stacks guide or the request feature that would put the item on hold for them. Steps
should be taken to direct users to these features more effectively.

Test 2, Task 2

You are writing a paper about the drug cartels’ influence on Mexico’s relationship with the United
States. Find a newspaper article on this topic, using OneSearch.

Participant Time on Task Task Completion

1 4m 45s Y (required assistance)

2 59s Y

3 No recording N

4 7m 47s Y

5 2m 52s Y

6 1m 33s Y

7 1m 30s Y

Table 5. Results for Test 2, Task 2

This task was difficult for participants. Two users limited their search initially to “digital library”
using the drop-down menu, thinking it would be a place to find newspaper articles; their searches
returned zero results. Only two users used the “Newspaper Articles” facet without being prompted,
and users did not seem to readily distinguish newspaper articles as a resource type. Participants

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

did not notice the resource type icons without being prompted. Several participants needed to be
reminded that the task was to find a newspaper article, and not any other type of article. With
guidance, most participants were able to complete the task. Participant 4 remained on the task for
almost eight minutes because of their dissatisfaction with the relevancy of the results to the
prompt. Interestingly, they found the “Newspaper Articles” facet and reapplied it after each
modified search, suggesting that they learned to use system features as they went.

One of the recommendations based on this task was to remove “digital library” as an option in the
drop-down menu on the basic search. It was evident that “digital library” did not have the same
meaning to end users as it does to internal users. This recommendation was easily implemented.
Another recommendation was to emphasize the resource type icons within the search results, but
we have not determined a way to do so effectively. One suggestion from the Discovery Usability
Group was to exclude newspaper articles from the search results as a default, but no consensus
was reached on this issue.

LIMITATIONS

The Discovery Usability Group identified limitations to the usability test that should be noted.
Testing was done in a high-traffic portion of the library’s lobby, which is used as study space by a
broad range of students. Participants were recruited from this study space, and we chose not to
screen participants. The fifteen participants in the study did not constitute a representative
sample. Almost all participants were undergraduate students, and no humanities majors
participated. The outcomes might have been different if our participants had included more
experienced researchers or students from a broader range of disciplines. By adding screening
questions or choosing a more neutral location, we would have limited the number of participants
who could complete our testing.

Another limitation was that the participants started the usability test within the Primo interface.
Because Primo is integrated into the Libraries’ website, users would typically begin searching the
system from within the library homepage. The goals of the study required testing of our Primo
development sandbox, which was not yet available to the public, and therefore could not be
accessed in the same way. This gave participants some additional options from the initial search
pages that are not usually available through the main search interface. While testing an active
version of the interface would be preferable, one of our goals was to understand how our
modifications affected user behavior, so testing the unmodified version was not an acceptable
substitute. Additionally, the usability study presented tasks out of context and did not replicate a
true user-searching experience. Despite the limitations, we learned valuable lessons from the
participants in this study.

DISCUSSION

Users successfully completed the tasks in this usability study. Unfortunately, they did not take
advantage of many of the features that can make such tasks easier—particularly facets. This was

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

especially apparent when we asked users to find a peer-reviewed journal article (Test 1, Task 2).
Primo has a facet that will limit a search to only peer-reviewed journal articles, and only one out of
eight participants used this facet during this task. Participants appreciated the pre-search filtering
options, and requested more of them (such as an author search), while post-search facets were
underutilized.

Similarly, participants almost uniformly ignored the links, or tabs, within the search results, which
would provide users with more information, a preview of the full-text, and additional features
such as an email function. Users bypassed these options and clicked on the title instead. The
Discovery Usability Group theorized that users clicked on the title of the item because that
behavior would be successful in a more familiar search interface like Google. The team customized
the configuration so that a title click would open either the full-text of electronic items or the
catalog record for physical items to accommodate users’ instinctive search behaviors. The tabs,
though a prominent feature of the discovery system, have proved to have little value for users.

Throughout the implementation of discovery systems in academic libraries, both research studies
and anecdotal evidence have suggested that users do not find end-user features like facets
valuable; however, discovery system vendors have made no apparent attempt to reimagine the
possibilities for search refinements. Indeed, most of the findings in this study will present few
surprises to anyone familiar with the discovery usability literature, which is itself concerning. As
our literature review has shown, many of the same general usability issues have repeated
throughout studies of Primo since 2008, and most are very similar to usability issues in other,
competitor discovery systems. This raises some concerns about the pace of innovation in the
discovery field, and whether discovery vendors are genuinely taking into account the research
findings about the needs of our users as they refine their products. In a recent article, David
Nelson and Linda Turney identified many issues with discovery facets in their current form that
may be barriers to usage, particularly labeling and library jargon; we join them in urging vendors
and libraries to collaborate more closely for deep analysis of actual facet usage by users, and to
address those factors that have negatively affected facets’ value.25

During our usability study, a common barrier to the successful completion of a task was not the
technology itself but a lack of understanding of the task. Participants had difficulty deciphering a
citation, which may have led to their tendency to search for a journal article by author and not by
title. Many participants struggled with using call numbers, and how to find and check out books in
the library. Peer review also proved to be a difficult or unfamiliar concept for many; when looking
for peer-reviewed articles, some participants clicked on the “Reviews” facet, which limited their
searches to an inappropriate resource type. Additionally, participants did not differentiate
between journal articles and newspaper articles, which may indicate a broader inability to
differentiate between scholarly and nonscholarly resources. This effect may be exaggerated by the
high percentage of science students who participated, as these students may not have frequent
need for newspaper articles. All of these challenges, however, are indicative of a deeper problem

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

with terminology. Regardless of how simple it is to limit a search to peer-reviewed articles, a user
who does not understand what peer review means cannot complete the task with confidence or
certainty.

Librarians struggle with presenting understandable language and avoiding library terminology; as
we discovered, academic language, like “peer-reviewed” and “citation,” presents a similar problem.
These are not issues that can be resolved with a technological solution. Rather, we join previous
authors in suggesting that instruction may be a reasonable way to address many usability issues in
Primo. From our findings and from those in the wider literature, we conclude that general
instruction in information literacy is prerequisite for effective use of this or any research tool,
particularly for undergraduates. Nichols et al. “recommend studying how to effectively provide
instruction on Primo searching and results interpretation,”26 but instruction on the use of a single
tool is of limited utility to students in their academic lives. Instead libraries could bolster
information literacy instruction on key concepts around the production and storage of
information, scholarly communications, and differences in information types. Teaching these
concepts effectively should help to alleviate the most common user issues, including
understanding terminology and different types of information, as well as helping students to
understand key elements of research in general. This is a particularly important point to note for
librarians working as advocates for information literacy instruction, especially in cases where
administrators or faculty may feel that more advanced tools, like discovery systems, should make
instruction obsolete.

CONCLUSION

Several changes were made to the Primo interface in response to breakdowns identified during
the usability study. Resource Discovery Systems first implemented the changes to the Primo
development sandbox. After the Discovery Usability and Advisory Groups agreed on the changes,
they were made available on the live site (see figure 4). The redesigned search results page
became available to the general public between the spring and summer academic sessions of 2015.
In addition to the changes that were made because the usability study, RDS made changes to the
look and feel to make the search results interface more aesthetically pleasing and more in line
with the University of Houston brand.

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

Before (live site):

Figure 4. Primo Interface before Usability Testing

During (development sandbox):

Figure 5. Primo Interface during Usability Testing

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

After (live site):

Figure 6. Primo Interface after Usability Testing

Many larger assertions of this study, encompassing implications for instruction and our needs
from discovery vendors, will require further study to address. The authors intend to continue to
investigate these issues as additional usability testing is conducted and to use the data to support
future vendor relations and instructional curriculum development discussions.

REFERENCES

1. Tamar Sadeh, “User Experience in the Library: A Case Study,” New Library World 109, no. 1/2
(2008): 7–24, doi:10.1108/03074800810845976.

2. Aaron Nichols et al., “Kicking the Tires: A Usability Study of the Primo Discovery Tool,” Journal
of Web Librarianship 8, no. 2 (2014): 172–95, doi: doi:10.1080/19322909.2014.903133; Scott
Hanrath and Miloche Kottman, “Use and Usability of a Discovery Tool in an Academic Library,”
Journal of Web Librarianship 9, no. 1 (2015): 1–21, doi:10.1080/19322909.2014.983259.

3. David J. Comeaux, “Usability Testing of a Web-Scale Discovery System at an Academic Library,”
College & Undergraduate Libraries 19, no. 2–4 (2012): 189–206,
doi:10.1080/10691316.2012.695671.

4. Kylie Jarrett, “FindIt@ Flinders: User Experiences of the Primo Discovery Search Solution,”
Australian Academic & Research Libraries 43, no. 4 (2012): 278–300; Nichols et al., "Kicking
the Tires."

5. Sadeh, “User Experience in the Library.”

http://dx.doi.org/10.1108/03074800810845976
http://dx.doi.org/10.1080/19322909.2014.903133
http://dx.doi.org/10.1080/19322909.2014.983259
http://dx.doi.org/10.1080/10691316.2012.695671

LESSONS LEARNED: A PRIMO USABILITY STUDY | BRETT, LIERMAN, AND TURNER
doi: 10.6017/ital.v35i1.8965

6. Jarrett, “FindIt@ Flinders”; Nichols et al., “Kicking the Tires.”

7. Xi Niu, Tao Zhang, and Hsin-liang Chen, “Study of User Search Activities with Two Discovery
Tools at an Academic Library,” Libraries Faculty and Staff Scholarship and Research 30, no. 5
(2014), doi:10.1080/10447318.2013.873281; Hanrath and Kottman, “Use and Usability of a
Discovery Tool in an Academic Library.”

8. Rice Majors, “Comparative User Experiences of Next-Generation Catalogue Interfaces,” Library
Trends 61, no. 1 (2012): 186–207, doi:10.1353/lib.2012.0029; Niu, Zhang, and Chen, “Study of
User Search Activities with Two Discovery Tools at an Academic Library.”

9. Beth Thomsett-Scott and Patricia E. Reese, “Academic Libraries and Discovery Tools: A Survey
of the Literature,” College & Undergraduate Libraries 19, no. 2–4 (2012): 123–43,
doi:10.1080/10691316.2012.697009.

10. Sadeh, “User Experience in the Library.”

11. Comeaux, “Usability Testing of a Web-Scale Discovery System at an Academic Library.”

12. Jessica Mahoney and Susan Leach-Murray, “Implementation of a Discovery Layer: The
Franklin College Experience,” College & Undergraduate Libraries 19, no. 2–4 (2012): 327–43,
doi:10.1080/10691316.2012.693435.

13. Joy Marie Perrin et al., “Usability Testing for Greater Impact: A Primo Case Study,” Information
Technology & Libraries 33, no. 4 (2014): 57–67.

14. Majors, “Comparative User Experiences of Next-Generation Catalogue Interfaces”; Thomsett-
Scott and Reese, “Academic Libraries and Discovery Tools.”

15. Jarrett, “FindIt@ Flinders”; Mahoney and Leach-Murray, “Implementation of a Discovery
Layer.”

16. Jarrett, “FindIt@ Flinders”; Mahoney and Leach-Murray, “Implementation of a Discovery
Layer”; Nichols et al., “Kicking the Tires."

17. Jarrett, “FindIt@ Flinders”; Mahoney and Leach-Murray, “Implementation of a Discovery
Layer”; Perrin et al., “Usability Testing for Greater Impact : A Primo Case Study.”

18. Jarrett, “FindIt@ Flinders”; Nichols et al., “Kicking the Tires”; Hanrath and Kottman, “Use and
Usability of a Discovery Tool in an Academic Library”; Majors, “Comparative User Experiences
of Next-Generation Catalogue Interfaces.”

19. Comeaux, “Usability Testing of a Web-Scale Discovery System at an Academic Library”;
Thomsett-Scott and Reese, “Academic Libraries and Discovery Tools.”

20. Jarrett, “FindIt@ Flinders.”

21. Mahoney and Leach-Murray, “Implementation of a Discovery Layer”; Perrin et al., “Usability
Testing for Greater Impact.”

http://dx.doi.org/10.1080/10447318.2013.873281
http://dx.doi.org/10.1353/lib.2012.0029
http://dx.doi.org/10.1080/10691316.2012.697009
http://dx.doi.org/10.1080/10691316.2012.693435

INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2016

22. Mahoney and Leach-Murray, “Implementation of a Discovery Layer”; Nichols et al., “Kicking
the Tires”; Niu, Zhang, and Chen, “Study of User Search Activities with Two Discovery Tools at
an Academic Library.”

23. Thomsett-Scott and Reese, “Academic Libraries and Discovery Tools.”

24. Tao Zhang and Merlen Prommann, “Applying Hierarchical Task Analysis Method to Discovery
Layer Evaluation,” Information Technology & Libraries 34, no. 1 (2015): 77–105,
doi:10.6017/ital.v34i1.5600.

25. David Nelson and Linda Turney, “What’s in a Word? Rethinking Facet Headings in a Discovery
Service,” Information Technology & Libraries 34, no. 2 (2015): 76–91,
doi:10.6017/ital.v34i2.5629.

26. Nichols et al., “Kicking the Tires,” 184.

http://dx.doi.org/10.6017/ital.v34i1.5600
http://dx.doi.org/10.6017/ital.v34i2.5629