Review Article


 


A Rapid Review
of the Reporting and Characteristics
of Instruments Measuring Satisfaction with Reference Service in Academic Libraries


 


Heidi Senior


Reference/Instruction
Librarian


Clark Library


University of Portland


Portland, Oregon, United
States of America


Email: senior@up.edu  


 


Tori Ward 


MLIS Graduate 


Syracuse University 


Syracuse, New York, United
States of America 


Email: tjward@syr.edu   


 


Received: 13 Feb. 2019                                                                   Accepted: 6 Nov. 2019


 


 


 2019 Senior and Ward. This is an Open
Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0
International (http://creativecommons.org/licenses/by-nc-sa/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial purposes,
and, if transformed, the resulting work is redistributed under the same or
similar license to this one.




 


 


DOI: 10.18438/eblip29556


 





 


Abstract


 


Objective –
The objective of this review was to examine research instrument
characteristics, and to examine the validity and reliability of research
instruments developed by practicing librarians, which measure the construct of
patron satisfaction with academic library reference services. The authors were
also interested in the extent to which instruments could be reused. 


 


Methods –
Authors searched three major library and information science databases: Library and Information Science
Technology Abstracts (LISTA); Library Science Database (LD); and Library
Literature & Information Science Index. Other databases searched were Current Nursing and Allied Health Literature (CINAHL);
Education Resources Information Center (ERIC); Google Scholar; PubMed; and Web
of Science. The authors identified studies of patron satisfaction with academic
library reference services in which the researcher(s) developed an instrument
to study the satisfaction construct. In this rapid-review study, the studies
were from 2015 and 2016 only. All
retrieved studies were examined for evidence of validity and
reliability as primary indicators of instrument quality, and data was extracted
for country of study, research
design, mode of reference service, data collection method, types of questions,
number of items related to satisfaction, and content of items representing the
satisfaction construct. Instrument reusability was also determined.


 


Results – At the end of the screening stage of the
review, a total of 29 instruments were examined. Nearly all studies were
quantitative or mixed quantitative/qualitative in design. Twenty-six (90%) of the studies
employed surveys alone to gather data. Twelve publications (41%) included a
discussion of any type of validity; five (17%) included discussion of any type
of reliability. Three articles (10%) demonstrated more than one type of
validity evidence. Nine articles (31%) included the instrument in full in an
appendix, and eight instruments (28%) were not appended but were described
adequately so as to be reusable.  


 


Conclusions – This review identified a range of quality in librarians’ research
instruments for evaluating satisfaction with reference services. We encourage
librarians to perform similar reviews to locate the highest-quality instrument
on which to model their own, thereby increasing the rigor of Library and
Information Science (LIS) research in general. This study shows that even a
two-year rapid review is sufficient to locate a large quantity of research
instruments to assist librarians in developing instruments.




 





 


Introduction


 


Reference services are a
primary function of nearly every library. Library staff make themselves
available to patrons through multiple communication modes such as in-person,
chat, phone, and email in order to “recommend, interpret, evaluate, and/or use
information resources to help others to meet particular
information needs” (Reference and User Services Association, 2008). They
might gather statistics relating to the number and type of questions patrons
ask, and perhaps the difficulty of answering those questions according to the
READ Scale (Gerlich & Berard,
2007), but these statistics do not express whether patrons were satisfied with
the answer. To determine if their library’s patrons are satisfied with the
provided service, librarians need to obtain patrons’ opinions directly through
data gathering methods such as surveys or 


 


interviews, known
collectively as research tools or instruments. They might then publish the
results of their study to help fellow librarians develop their own
patron-satisfaction tools. One study found that reference topics represented
9.5% of all library and information sciences research (Koufogiannakis,
Slater, & Crumley, 2004).


 


Systematic instrument review has been a common practice in health
science research and has developed to the extent that standards exist for
specific topic areas, such as the Consensus-based Standards for the Selection
of Health Measurement Instruments (COSMIN) Initiative (2018). This type of
study uses systematic review methodology to identify and analyze the
psychometric characteristics of research instruments. While anthologies of
research instruments produced by librarians and measuring satisfaction with
reference services exist, such as those found in The Reference Assessment
Manual (American Library Association, Evaluation of Reference and Adult
Services Committee, 1995, pp. 255-345), to date it seems that no one has
published a systematic instrument review that would obtain an overall image of
the state of instrument development in this area. We therefore decided to
conduct a review to gain an understanding of the quality of instruments
produced by academic librarians studying patron satisfaction with reference
service. 


 


Literature Review


 


While at the time of our study no reviews of instruments fully using the systematic review
methodology had appeared in LIS literature, we found that researchers had
mentioned instruments and evaluated them to varying extents in articles on
faculty attitudes toward open access publication (Otto, 2016); information
literacy (Beile, 2008; Schilling & Applegate,
2012); information seeking behavior (McKechnie, Chabot, Dalmer,
Julien, & Mabbott, 2016); satisfaction with chat
reference (Lasda Bergman & Holden, 2010); and
assessment of individual research consultations (Fournier & Sikora, 2015). 


 


Of these researchers, only Lasda Bergman &
Holden (2010) followed a systematic review protocol in their research criteria
and search methods, retaining after their final appraisal stage 12 studies
regarding user satisfaction with electronic reference. However, because they
did not present details of each instrument in an evidence table, we were unable
to reproduce their data extraction process. Schilling and Applegate (2012)
identified 27 tools in their survey of academic library literature on student
learning assessment from 2007 to 2012 but did not take a systematic approach
and emphasized each instrument’s content rather than construction and
measurement concerns. Similarly, Fournier and Sikora’s 2015 scoping review
located 20 studies using various methods to assess individual research
consultations but did not review instrument characteristics beyond the type of
assessment method. Beile’s (2008) report covered
widely-known information literacy assessment tools that would provide data
“considered acceptable evidence for program reviews” (p. 1) such as
Standardized Assessment of Information Literacy Skills (SAILS) and Educational
Testing Service’s iSkills, but did not describe a
process for identifying the seven tests and four rubrics included in the paper.
McKechnie et al.’s approach to evaluating research rigor involved using a
checklist that asked whether authors attached or included their instrument – an
element we included in our study – and whether the instrument had undergone
pre-testing, an important component in demonstrating an instrument’s validity
(2016). While conducting a literature review prior to studying effective faculty
outreach messages regarding open access publication, Otto (2016) realized that
the studies reviewed did not accurately reflect faculty understanding due to
flaws in their underlying instruments such as failing to define terms, adapting
previous surveys without updating questions, and inserting inadvertent bias
into survey questions and response options. Although Otto did not report
evidence of the instruments’ validity and reliability specifically, several of
the issues Otto identified might have been resolved had the instruments’
developers paid closer attention to determining their validity.


 


Shortly before completing our manuscript, we learned
of the publication late in 2017 of a systematic review of 22 self-efficacy
scales assessing students’ information literacy skills (Mahmood) from 45
studies published between 1994 and 2015. Because Mahmood’s review was limited
to studies in which authors reported the use of any validity and also any
reliability indicators, it differs from our relatively unrestricted approach.
Mahmood’s study likely omits scales and does not provide a full picture of the
state of instrument development in this area.


We identified two instrument reviews from the field of
education (Gotch & French, 2014; Siddiq, Hatlevik, Olsen, Throndsen, &
Scherer, 2016), the second of which served as a preliminary model for the data
extraction stage of our pilot study. Gotch &
French (2014) reviewed 36 measures published between 1991 and 2012 of classroom
teachers’ assessment literacy, using “evaluation of the content of assessment
literacy measures beyond literature review and solicitation of feedback,”
“internal consistency reliability,” and “internal structure” to demonstrate
validity, and “score stability” (p. 15) to demonstrate reliability of each
instrument. We decided not to use Gotch and French as
a model because the authors did not rigorously follow a systematic approach in
database searching or in presenting their results in evidence tables. The
second systematic instrument review (Siddiq, et al., 2016) covered 38
information and communication technology instruments aimed at primary and
secondary school students, and was a useful framework to emulate because, like
our study, it was descriptive rather than evaluative in design. Furthermore,
the authors carefully documented their search strategy and findings in a way
that adhered closely to systematic review methodology. Like our study, the
authors appeared to be concerned to represent the state of the field and
included instruments whose developers did not address evidence of their
validity or reliability.


 


Our review of librarians’ studies examining
instruments determined that the instrument review methodology is under-used in
librarianship, and that our pilot study identifies a new area of research. By
drawing on similar reviews in education, we demonstrate the usefulness to
librarian-researchers of breaking out of disciplinary compartmentalization for
assistance with promising methodologies. 


 


Research Questions


 


We began this study with a basic question: 


What is the quality of research instruments produced
by librarians? We developed the following more specific questions using patron
satisfaction with reference services in academic libraries as a focus. We
defined reference service as librarians helping others to meet particular
information needs in-person at a desk, roaming, or via consultations; through
virtual methods such as chat and email; or over a telephone.


 


Q1: How did LIS researchers gather data on patron satisfaction with
academic library reference services in the years 2015-2016?


 


Q2: To what extent did the instrument developers document the validity
and reliability of their instruments?


 


Q3: To what extent are the instruments provided in an appendix or described in the publication, to
assist in reuse?


 


Method


 


Selection of Review Type


 


The systematic review is considered the most rigorous
methodology for gathering and synthesizing information based on predetermined
inclusion/exclusion criteria, clear and reproducible search methods, and
quality assessment, with results presented in an evidence table (Xu, Kang,
& Song, 2015; see also McKibbon, 2006; Phelps
& Campbell, 2012). Traditional systematic reviews, however, aim to be
comprehensive in coverage. Because the lead author wanted to accomplish as much
work as possible during a sabbatical, we elected to perform a rapid review,
which follows the systematic review methodology (predetermined
inclusion/exclusion criteria, clear and reproducible search methods, and
quality assessment, with results presented in an evidence table) but is limited
in time (Grant & Booth, 2009, p. 100). See Table 1 for distinctions between
systematic and rapid reviews.  


 


 


Table 1


Differences Between Systematic and Rapid Review
Types


	
  Review Type

  	
  Description

   

  	
  Search

   

  	
  Appraisal

   

  	
  Synthesis

   

  	
  Analysis

   

  
	
  Rapid Review

   

  	
  Assessment of what is already known about a policy or practice issue,
  by using systematic review methods to search and critically appraise existing
  research

   

  	
  Completeness of
  searching determined by time constraints

   

  	
  Time-limited formal quality assessment

   

  	
  Typically, narrative and tabular

   

  	
  Quantities of literature and overall quality/direction of effect of
  literature

   

  
	
  Systematic Review

   

  	
  Seeks to systematically search for, appraise and synthesis research
  evidence, often adhering to guidelines on the conduct of a review

   

  	
  Aims for exhaustive, comprehensive searching

   

  	
  Quality assessment may
  determine inclusion/exclusion

   

  	
  Typically, narrative with tabular accompaniment

   

  	
  What is known; recommendations for practice. What remains unknown; uncertainty
  around findings, recommendations for future research

   

  



Grant & Booth, 2009, p. 95.


 


 


Inclusion Criteria


 


We assembled and agreed upon the following criteria:


 


	Quantitative, qualitative, or mixed-method
     research studies measuring satisfaction with reference service carried out
     in any type of academic library including health science libraries,
     addressing any type of patron. We included studies measuring satisfaction
     with several library services, as long as one question asked about
     reference service.
	Studies published in 2015 or 2016 
	Instruments developed by front-line librarians,
     including adaptations of a standardized instrument such as SERVQUAL or
     SERVPERF
	English, French, or Spanish language



 


Search Strategy


 


EBSCO’s Library and Information Science Technology Abstracts (LISTA) and
ProQuest’s Library Science Database (LD) were the primary sources of studies;
we also searched Library Literature & Information Science Index, CINAHL,
ERIC, Google Scholar, PubMed, and Web of Science. In addition to these
databases, we searched the American Library Association, the Association of
College and Research Libraries, and assessment conference programs that were
published online for the years of interest.


 


When developing our search strategies, we kept in mind the caveat raised
by LIS authors that database thesauri might be incomplete or that subject
headings might not be applied uniformly. VanScoy and
Fontana noted in their 2016 study of reference and information service (RIS)
research that “This method relies on the RIS research articles being correctly
assigned the relevant descriptor in the databases” (p. 96). This warning echoes
that of McKechnie, Baker, Greenwood, & Julien in 2002 who said “Both [EBSCO
and ProQuest] indexes used terms … that were too general to be useful” (p. 123)
and found that indexing terms were incorrectly applied in 28-34% of the
articles they examined, as well as Greifeneder who
warned in 2014 that one of the studies in her literature review might have a
biased retrieval set because it included articles indexed under only two
subject terms rather than searching more widely (Background, para. 8). We
therefore decided to run both subject and keyword searches.


 


After a careful examination of subject terms used in either LISTA or LD,
and heeding past research on effective search strategy, we performed the
following searches:


 


LISTA: (academic AND librar* AND (reference OR
"user satisfaction")) AND (SU (research or surveys or questionnaires)
OR AB (study or survey* or interview* or research*))  


 


LD: all ((academic* AND librar* AND (reference
OR "user satisfaction" OR "customer satisfaction" OR
"customer services"))) AND su(research or surveys or questionnaires) AND ab(study or
survey* or interview* or research*). Note that the LD search is identical to
the LISTA search except for the inclusion of “customer satisfaction” and
“customer services,” which are subject terms not used in the LISTA database.


 


Given the inconsistent application of subject terms in library
literature databases, we note that articles given the subject term “academic
libraries” might not describe undergraduate or community college libraries.
However, we found no additional articles when we re-ran searches with the
subject terms “community college libraries” and “undergraduate libraries.”


 


We then examined abstracts and developed a free-text keyword search that
we adapted for use in all of the databases, in an attempt to find all articles
that might not have had correct subject-term labels: (reference or "research
consultation") AND (satisf* or evaluat* or assess* or improve*) AND (experiment* or
survey* or qualitative or servqual or instrument or investigat* or analysis or questionnaire*) AND
"academic librar*". We also ran a broad
search for librar* AND reference AND satisfaction,
being mindful that the “academic libraries” label might not be uniformly
applied and that some articles might use “college” or “university” instead, or
that various labels might be used to represent different categories of library
patron, or different types of data-gathering instruments. We removed search
terms related to research methodologies to have broad retrieval. 


 


Conforming to our inclusion criteria, we limited results in each
database to journal articles from the years 2015 and 2016 and checked each
database for conference papers as a separate source type. We did not apply
language or geographic location limiters and were prepared to examine articles
in French or Spanish in addition to English, but our searches retrieved only
English-language publications. In preparation for retrieving a large amount of
results, such as within Google Scholar, we determined that we would review the
first 300 items only. Within those 300 results, we ceased reviewing when we
began encountering irrelevant items.


 


When searching PubMed, we applied a search filter provided by COSMIN in
order to better identify all studies containing measurement properties.  In Google Scholar, we utilized the Advanced
Search feature to narrow our results. In
ERIC and CINAHL, we utilized the database thesauri to identify subject terms. We
also hand searched 10 online journals (Journal of Academic Librarianship;
College & Research Libraries; Library & Information Science
Research; portal; Journal of the Medical Library Association;
Journal of Librarianship and Information Science; Reference Services
Review; Medical Reference Services Quarterly; Reference Librarian;
and College and Undergraduate Libraries), adhering to our year
restriction of 2015-2016. To standardize our searches, we created a table in
which both authors’ search strings were input to compare and ensure that we
were staying consistent with our searches and results.


 


All search strategies are provided in
Appendix A.


 


Reviewing Process and Study Evaluation 


 


We compiled citations in a RefWorks database and removed duplicates
using the RefWorks tool. We examined bibliographic information from the
databases, such as title and abstract, to screen for relevant articles. To add
a peer reviewing element to our searches, we kept track of our subsequent
searches on a separate workbook so that each author could observe and be able
to discuss the quality of each search with the other. In those workbooks, we
documented the search conducted, the database in which the search was
conducted, the limiters set in each search, the results of each search, the
citations found from each search (if applicable), and any notes. 


 


Data Extraction 


 


As stated earlier, we used Siddiq et al.’s (2016) extraction sheet as a
model because we aimed to be descriptive rather than evaluative in scope. Following their model, we extracted
the following data: country of study; stated purpose of study; mode of
reference service; age/level of students (if students were part of the targeted
population); size of targeted population; usable responses; sampling strategy;
research design; data collection method; types of questions, other than
demographic (i.e., Likert scale format; presence of open-ended questions);
demographics gathered; technical aspects, such as distribution, availability of
translations, duration of survey period; time allotted to complete survey;
validity indicators; reliability indicators (see below for definitions of validity and reliability); instrument availability in appendix; reusability of
instrument, if not appended;
number of items related to satisfaction; content of items representing the
satisfaction construct. 


 


Definitions


Generally speaking, an instrument is said to provide valid results when
it measures what the instrument’s developer intended it to measure within a
study’s setting and population, and reliable results when the instrument
provides the same score if repeatedly implemented among the same population.
Researchers have further specified various elements that assist in
demonstrating the validity of information obtained via an instrument. We used
these definitions when examining the instruments we gathered. Tables 2 and 3
include the codes we assigned to each element, to make our reporting table more
compact.


 


Table
2  


Definitions
of Validity


	
  Title

  	
  Code

  	
  Definition

  	
  Evidence

  
	
  Face Validity

  	
  V1

   

   

   

  	
  “The instrument appears to
  measure what it claims to measure” (Gay, Mills, & Airasian,
  2006 as quoted in Connaway & Radford, 2017, p.
  82)

  	
  Demonstrated through pre-testing,
  ideally with subjects similar to the target population, and with instrument
  development specialists

  
	
  Content Validity: Item

  	
  V2a

  	
  “…the items of the instrument or test …represent
  measurement in the intended content area” (Connaway
  & Radford, 2017, p. 81).

  	
  Demonstrated through item analysis during
  pre-testing

   

  
	
  Content Validity: Sampling

  	
  V2b

  	
  “…how well the instrument
  samples the total content area” (Connaway &
  Radford, 2017, pp. 81-82)

  	
  Demonstrated through
  discussion of included constructs

  
	
  Construct Validity

  	
  V3

  	
  “…instrument measures the construct in question and
  no other.” (Connaway & Radford, 2017, p. 83)

   

  	
  Demonstrated through factor analysis, other tests of
  dimensionality, to retain convergent (contributing) items and remove divergent
  (non-contributing) ones

  
	
  Intercoder Reliability

  	
  V4

  	
  Degree to which scorers or
  raters agree on evaluating or observing a variable (Connaway
  & Radford, 2017, p. 316).

  	
  Demonstrated through
  percentage agreement among raters

  



We used these definitions
of reliability, and assigned these codes: 


Table
3


Definitions
of Reliability 


	
  Title

  	
  Code

  	
  Definition

  	
  Evidence

  
	
  Internal Consistency

  	
  R1

  	
  How well items on a test
  relate to each other.  (Connaway & Radford, 2017, p. 84)

  	
  Demonstrated through
  Cronbach’s alpha, Kuder-Richardson 20 tests (Catalano, 2016, p. 8).

  
	
  Measurement Reliability

  	
  R2

  	
  “The degree to which an instrument accurately and
  consistently measures whatever it measures” (Connaway
  & Radford, 2017, p. 83).

  	
  Demonstrated through test-retest correlation,
  meaning repeated administration to same group of the whole instrument
  (Catalano, 2016, p. 8) or split-half method, meaning correlation of scores
  obtained from each half of a tested population or from each half of an
  instrument that measures a single construct (Catalano, p. 9; Connaway & Radford, pp. 83-84).

   

  



 


We described an instrument as “resusable” only
when we could answer three questions about the instrument: Are the number of
questions reported? Is the full text of each question provided, and associated
items? Is the format of each question described: scale values, anchor labels
such as “Very Satisfied” and “Very Unsatisfied”? If we felt that we had to
guess as to whether the author fully described an instrument, we labeled it not
replicable. We automatically coded appended instruments as replicable. The most
frequent reason for describing an instrument as “not replicable” was that
authors did not supply the number of questions and items, so that we could not
be sure if they had described the entire instrument.


 


As explained earlier, we restricted the definitions of validity and
reliability to those used by Connaway and Radford
(2017), with occasional details borrowed from Catalano (2016). We decided not
to use the more expansive definitions that Siddiq et al. (2016) employed in
which for example an instrument’s having a basis in theory could be perceived
as evidence that it produced valid results. We developed our own evidence
extraction sheet to avoid obscuring the definition of each of these concepts:
we wanted to focus on precise definitions of validity and reliability, and
Siddiq et al.’s criteria extended beyond those definitions. 


 


After completing the process of acquiring and screening studies, we
jointly read and reviewed six studies in duplicate and compared our extracted
data, to ensure that we agreed. We then separately reviewed the remaining 23
studies and consulted with each other on any confusing elements. After our
subsequent searches we divided responsibility similarly to review the nine
additional articles. When we disagreed, we located more information on the
issue to arrive at a consensus. For example, a disagreement about validity
types might require refreshing our understanding of the definitions. We did not
require a third party to resolve disagreements. We recorded our data in a
shared spreadsheet. 


 


 





Figure 1


Flow chart of
review process




 


Results


 


Through our initial searches in seven databases, we found 2,189 articles
that appeared to be relevant to our study. After removing duplicates, we were
left with 2,108 relevant articles. We further reviewed the article titles and
abstracts and found that 1,770 were not truly relevant to our study. We
assessed the remaining 338 articles for eligibility and rejected 309 articles
because they described instruments measuring satisfaction with only the library
as a whole, or instruments measuring usage of or familiarity with reference
services, or instruments measuring satisfaction with services other than
reference. We extracted data from the final set of 29 studies. Figure 1 shows a
PRISMA flow chart of our process, a standard component of systematic review articles
(Moher, Liberati, Tetzlaff,
& Altman, 2009).


 


We concluded during
the extraction process that certain criteria were more relevant to our
focus on the instrument development process and therefore decided not to report
irrelevant criteria such as “purpose of study,” “age of respondents,” “mode of service,” and “response
rate.” We excluded “time needed to complete the instrument” from this paper
because many authors did not report it. Additional criteria included article
title, journal, age/level of student, type of institution, sampling strategy,
technical aspects (e.g., distribution and survey period), and demographics
gathered. Although we gathered data in these categories, we found that these
criteria did not further our understanding of how librarian researchers report
on instrument development and implementation. Our evidence tables focus on the
following criteria: study; country; research design; data collection method;
types of questions; validity evidence; reliability indicators; whether the instrument
is appended; whether the instrument is mentioned in the abstract as appended;
and replicability of the instrument if
not appended. The list of extraction elements is in Appendix B, and the full
data extraction spreadsheet is available
online (https://docs.google.com/spreadsheets/d/1M7MYhNcKbscrak9CqGpBg0X6z-d6V5qs-IwIE4I92SQ/edit#gid=0).


 


Outside of the data relating to our research questions, the data on
country of study might be of interest to researchers. Ten studies took place in
the United States of America; the second most common country was India with six
studies. The remaining studies took place in China (2), Ghana (1), Jamaica (1),
Malaysia (2), Nigeria (4), Philippines (1), and Taiwan (2).


 


These are the results of our data extraction as they relate to our
research questions:


 


Q1: How did LIS researchers gather data on patron satisfaction with
reference services in the years 2015-2016?


 


Of the 29 studies we gathered, 18 (62%) were solely quantitative in
design and one (3%) solely qualitative. We labeled ten (34%) studies as
combining both quantitative and qualitative designs, but this was usually
because we defined “mixed methods” broadly to allow open-ended questions to be
called qualitative; only two studies (7%) (Jacoby, Ward, Avery, & Marcyk, 2016; Verma & Parang, 2015) were truly mixed
using the more conservative approach as defined by Fidel (2008) in which
qualitative and quantitative methods were used to answer the same research
question. Askew (2015) and Yap and Cajes (2016)
employed quantitative methods to ask students about their satisfaction with
roaming reference service and qualitative methods to ask librarians about their
experience with providing the service; these studies are therefore labeled
“quantitative” for the purposes of this review. 


 


Twenty-six (90%) of the studies employed surveys alone
to gather data. Construction of the surveys varied; the number of items related
to satisfaction ranged from 1 to 16. Eight (29%) of the studies asked only
about overall satisfaction with reference service, while another 10 (34%)
included an item about overall service 


satisfaction as well as other attributes contributing
to satisfaction. Respondents were asked to consider aspects of librarian behaviour such as approachability and responsiveness,
helpfulness, respect for confidentiality, and offering referrals; and aspects
of librarian performance such as ability, accuracy, knowledge, and inspiring
confidence. Five instruments (Blake et al., 2016; Butler & Byrd, 2016;
Huang, Pu, Chen, & Chiu, 2015; Jacoby et al., 2016; Luo & Buer, 2015) asked students to gauge their likeliness to
use, re-use, or recommend the service. Masrek and
Gaskin reported presenting respondents with 10 items about Service quality,
usefulness, and satisfaction (p. 42) but unfortunately did not provide the full
text of the items within their article, making it impossible to determine how
they conceptualized these elements of satisfaction.


 


Researchers commonly used 5-point Likert scales to quantify respondents’
agreement or disagreement with statements; this type of scale occurred in 13
(45%) of the 29 survey instruments. 
Three (10%) of these scales did not have the traditional neutral midpoint.
Two (7%) of the scales offered three positive scores versus two negative (Sivagnanam & Esmail, 2015; Xie & Sun, 2015); the third scale was recoded by its developers to have three negative scores and
two positive (Yan et al., 2015). Similarly, researchers employing 3-point
scales did not always include a midpoint; three studies (Butler & Byrd,
2016; Ekere, Omekwu, & Nwoha, 2016; Yap & Cajes,
2016) offered two positive options and one negative. Two 4-point scales
(Khobragade & Lihitkar, 2016; Yap & Cahes, 2016) were likewise unbalanced, with three positive
and one negative choices. Duan (2016) used 4-point
scales to measure satisfaction with different modes of reference service and
6-point scales to measure satisfaction with reference librarians’ behavior. The
remaining scales ranged in size from two scale points to nine.


 


Most authors used typical labels for scale points, e.g., variations on
“Very Satisfied,” “Somewhat Satisfied,” “Satisfied,” and “Very Dissatisfied,”
“Somewhat Dissatisfied,” and “Dissatisfied,” or related labels such as
“Useful,” and “Adequate,” but some authors labeled scale points differently
from these norms. Duan (2016) provided explanatory
text for each scale point, e.g., “Unsatisfied, because they solved few of my
problems but were not willing to help me again” (p. 164). Sivagnanam
and Esmail (2015) labeled their scale points “Not
Satisfied,” “Not Much Satisfied,” “Particularly Satisfied,” “Fairly Satisfied,”
“Absolutely Satisfied.” Yan, et al. (2015) were not clear, as it seemed they
gave their scale two midpoint labels, “Neutral” and “Not Familiar.” Most
5-point scales had a midpoint labeled “Neutral” (Askew, 2015; Blevins, et al.,
2016; Boyce, 2015; Huang et al, 2015; Mohindra &
Kumar, 2015) or “Neither Agree nor Disagree” (Jacoby et al., 2016; Masrek & Gaskin, 2016). Three authors did not report
the label used for their midpoint (Chen, 2016; Ganaie,
2016; Swoger & Hoffman, 2015).


 


A list of studies with their associated research design, data collection
method, and Likert-scale type is available in Table 4.


 


Table 4


Studies
Included in This Review


	
  Study

  	
  Research Design

  	
  Data Collection Method

  	
  Types of Questions

  
	
  Akor & Alhassan, 2015

  	
  Quant.

  	
  Survey

  	
  4-point scale

  
	
  Askew, 2015

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Blake et al., 2016

  	
  Mixed

  	
  Survey

  	
  4-point scale, open-ended

  
	
  Blevins, DeBerg,
  & Kiscaden, 2016

  	
  Mixed

  	
  Survey

  	
  5-point scale, open-ended

  
	
  Boyce, 2015

  	
  Mixed

  	
  Survey

  	
  Choose from list, 5-point scale, yes/no, open-ended

  
	
  Butler & Byrd, 2016

  	
  Mixed

  	
  Survey

  	
  3-point scale, open-ended

  
	
  Chen, 2016

  	
  Quant.

  	
  Survey (based on SERVQUAL)

  	
  5-point scale

  
	
  Dahan, Taib, Zainudin, & Ismail, 2016

  	
  Quant.

  	
  Survey (based on LIBQUAL)

  	
  9-point scale

  
	
  Duan, 2016

  	
  Quant.

  	
  Survey

  	
  6-point and 4-point scales

  
	
  Ekere, Omekwu, & Nwoha, 2016

  	
  Quant.

  	
  Survey

  	
  3-point scale

  
	
  Ganaie, 2016

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Huang, Pu, Chen, & Chiu, 2015

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Ikolo, 2015

  	
  Quant.

  	
  Survey

  	
  2-point scale

  
	
  Jacoby, Ward, Avery, & Marcyk,
  2016

  	
  Mixed

  	
  Survey, Focus Groups, Interviews

  	
  5-point scale; open-ended

  
	
  Khobragade & Lihitkar,
  2016

  	
  Quant.

  	
  Survey

  	
  4-point scale

  
	
  Kloda & Moore, 2016

  	
  Mixed

  	
  Survey

  	
  3-point scale, open-ended

  
	
  Luo & Buer, 2015

  	
  Mixed

  	
  Survey

  	
  5-point scale, open-ended

  
	
  Masrek & Gaskin, 2016

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Mohindra & Kumar, 2015

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Nicholas et al., 2015

  	
  Mixed

  	
  Survey

  	
  Choose from list, open-ended

  
	
  Sivagnanam & Esmail,
  2015

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Swoger & Hoffman, 2015

  	
  Mixed

  	
  Survey

  	
  5-point scale, open-ended

  
	
  Tiemo & Ateboh,
  2016

  	
  Quant.

  	
  Survey

  	
  4-point scale

  
	
  Verma & Laltlanmawii,
  2016

  	
  Quant.

  	
  Survey

  	
  3-point scale

  
	
  Verma & Parang, 2015

  	
  Mixed

  	
  Surveys, Interviews

  	
  3-point scales

  
	
  Watts & Mahfood,
  2015

  	
  Qual.

  	
  Focus Groups

  	
  Open-ended

  
	
  Xie & Sun, 2015

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Yan, Hu, & Hu, 2015

  	
  Quant.

  	
  Survey

  	
  5-point scale

  
	
  Yap & Cajes,
  2016

  	
  Quant.

  	
  Survey

  	
  3-point and 4-point scales; another scale not specified

  



 


 


Q2: To what extent are the instruments documented or included in the
body of a publication?


 


Nine articles (31%) included the instrument in full in an appendix, and
of the remaining studies we found that eight instruments (28% of the total) were replicable
according to our criteria as described earlier. Detailed information is
provided in Table 5.


 


We noticed that two of the instruments (Duan, 2016; Xie & Sun, 2015)
were translated into Chinese as well as English; both versions were available
to respondents, but the author described the English-language instrument within
the publication. We were unable to determine if any differences might exist
between the two versions.


 


Q3: To what extent are the instruments’ reliability
and validity documented?


 


Twelve publications (41%) included a discussion of any type of validity; five (17%) included discussion of any type
of reliability. Three articles (10%) demonstrated
more than one type of validity evidence. See Table 6 for a complete list.


 


Validity Evidence


 


Face Validity


 


Face validity was the most common type of validity represented, as nine
authors (31%) had pre-tested
their instruments; however, we found that in two cases (7%) (Akor & Alhassan, 2015; Blevins, DeBerg,
& Kiscaden, 2016) only librarian colleagues
participated rather than members of the target population or instrument
development specialists. The pre-testing process with potential respondents
varied; Blake et al. (2016) held campus interview sessions, while Butler and
Byrd (2016) informally polled library student employees. Kloda and Moore (2016)
and three sets of researchers (Askew, 2015; Huang et
al., 2015; Masrek & Gaskin, 2016) presented instrument
drafts to members of their respondent population. Only three studies (10%)
specifically reported pre-testing with a population contrasted with librarians
and therefore presumably instrument development specialists: Chen (2016) met
with “academic experts” (p. 319); Blake et al. worked with “experts from the
university’s Educational Innovation Institute” (p. 227), and Masrek and Gaskin (2016) pre-tested their instrument with
“experts in the faculty” (p. 42).


 


Table 5


Reusability of
Instruments Within Studies


	
  Study

  


  	
  Instrument Appended

  


  	
  Reusability of Instrument

  


 
 	
  Akor & Alhassan, 2015

  


  	
  No

  


  	
  No

  


 
 	
  Askew, 2015

  


  	
  No

  


  	
  Yes

  


 
 	
  Blake et al., 2016

  


  	
  Yes (Online)

  


  	
  Yes

  


 
 	
  Blevins, DeBerg,
  & Kiscaden, 2016

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Boyce, 2015

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Butler & Byrd, 2016

  


  	
  Yes (Online)

  


  	
  Yes

  


 
 	
  Chen, 2016

  


  	
  No

  


  	
  Yes

  


 
 	
  Dahan, Taib, Zainudin,
  & Ismail, 2016

  


  	
  No

  


  	
  Yes

  


 
 	
  Duan, 2016

  


  	
  No

  


  	
  No

  


 
 	
  Ekere, Omekwu, & Nwoha,
  2016

  


  	
  No

  


  	
  Yes

  


 
 	
  Ganaie, 2016

  


  	
  No

  


  	
  No

  


 
 	
  Huang, Pu, Chen, & Chiu,
  2015

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Ikolo, 2015

  


  	
  No

  


  	
  Yes

  


 
 	
  Jacoby, Ward, Avery, & Marcyk, 2016

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Khobragade & Lihitkar, 2016

  


  	
  No

  


  	
  No

  


 
 	
  Kloda & Moore, 2016

  


  	
  No

  


  	
  Yes

  


 
 	
  Luo & Buer,
  2015

  


  	
  No

  


  	
  Yes

  


 
 	
  Masrek & Gaskin, 2016

  


  	
  No

  


  	
  No

  


 
 	
  Mohindra & Kumar, 2015

  


  	
  No

  


  	
  No

  


 
 	
  Nicholas et al., 2015

  


  	
  No

  


  	
  No

  


 
 	
  Sivagnanam & Esmail, 2015

  


  	
  No

  


  	
  No

  


 
 	
  Swoger & Hoffman, 2015

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Tiemo & Ateboh, 2016

  


  	
  No

  


  	
  Yes

  


 
 	
  Verma & Laltlanmawii,
  2016

  


  	
  No

  


  	
  No

  


 
 	
  Verma & Parang, 2015

  


  	
  No

  


  	
  No

  


 
 	
  Watts & Mahfood,
  2015

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Xie & Sun, 2015

  


  	
  Yes

  


  	
  Yes

  


 
 	
  Yan, Hu, & Hu, 2015

  


  	
  No

  


  	
  No

  


 
 	
  Yap & Cajes,
  2016

  


  	
  No

  


  	
  No

  


 




 


 


Table
6


Validitya and Reliabilityb




	
  Study

  	
  Validity Evidence

  	
  Reliability Indicators

  
	
  Akor & Alhassan, 2015

  	
  V1

  	
  Not stated

  
	
  Askew, 2015

  	
  V1

  	
  Not stated

  
	
  Blake et al., 2016

  	
  V1, V2a, V3

  	
  Not stated

  
	
  Blevins, DeBerg,
  & Kiscaden, 2016

  	
  V1

  	
  Not stated

  
	
  Boyce, 2015

  	
  Not stated

  	
  Not stated

  
	
  Butler & Byrd, 2016

  	
  V1

  	
  Not stated

  
	
  Chen, 2016

  	
  V1

  	
  R1

  
	
  Dahan, Taib, Zainudin, & Ismail, 2016

  	
  V3

  	
  R1

  
	
  Duan, 2016

  	
  Not stated

  	
  Not stated

  
	
  Ekere, Omekwu, & Nwoha, 2016

  	
  Not stated

  	
  Not stated

  
	
  Ganaie, 2016

  	
  Not stated

  	
  Not stated

  
	
  Huang, Pu, Chen, & Chiu, 2015

  	
  V1, V3

  	
  R1

  
	
  Ikolo, 2015

  	
  Not stated

  	
  Not stated

  
	
  Jacoby, Ward, Avery, & Marcyk,
  2016

  	
  V4

  	
  Not stated

  
	
  Khobragade & Lihitkar,
  2016

  	
  Not stated

  	
  Not stated

  
	
  Kloda & Moore, 2016

  	
  V1

  	
  Not stated

  
	
  Luo & Buer, 2015

  	
  V2b

  	
  Not stated

  
	
  Masrek & Gaskin, 2016

  	
  V1, V3

  	
  R1

  
	
  Mohindra & Kumar, 2015

  	
  Not stated

  	
  Not stated

  
	
  Nicholas et al., 2015

  	
  Not stated

  	
  Not stated

  
	
  Sivagnanam & Esmail,
  2015

  	
  V1

  	
  Not stated

  
	
  Swoger & Hoffman, 2015

  	
  c

  	
  Not stated

  
	
  Tiemo & Ateboh,
  2016

  	
  Not stated

  	
  Not stated

  
	
  Verma & Laltlanmawii,
  2016

  	
  Not stated

  	
  Not stated

  
	
  Verma & Parang, 2015

  	
  Not stated

  	
  Not stated

  
	
  Watts & Mahfood,
  2015

  	
  Not stated

  	
  Not stated

  
	
  Xie & Sun, 2015

  	
  Not stated

  	
  Not stated

  
	
  Yan, Hu, & Hu, 2015

  	
  V3

  	
  R1

  
	
  Yap & Cajes, 2016

  	
  Not stated

  	
  Not stated

  



aV1 = Face validity; V2a = Content validity
(item); V2b = Content validity (sampling); V3 = Construct validity; V4 =
Intercoder reliability. See Table 2 for full definitions.


bR1 = Internal consistency. See Table 3 for
full definitions.


cIntercoder reliability coefficients not reported.




 


 


Content Validity: Item 


 


Blake et al. (2016) were the sole authors to refer to
item validity as part of their instrument development process, borrowing the
definition from another paper by calling it “internal structure” (Downing,
2003, as cited in Blake et al., p. 227).


 


Content Validity: Sampling


 


Luo and Buer (2015) were the sole researchers
to address sampling validity; their instrument measured variables drawn from
the five areas outlined in the Reference and User Services Association’s (RUSA)
Guidelines for Behavioral Performance of
Reference and Information Service Providers, as well as from past research
on evaluation of reference service. 


 


Construct Validity


 


Five publications (17%)
addressed construct validity as demonstrated by factor analysis and other tests
of dimensionality; three of these are described in the “multiple examples”
section below because they tested construct validity along with other forms of
validity. Two studies (7%)
addressed construct validity alone. 



 


Yan et al. (2015) determined the value of average
variance extracted (AVE) to demonstrate convergent validity of the constructs
in their instrument, and reported that “all of the AVE values range from 0.6727
to 0.8019” (p. 562), and considered these values satisfactory citing Fornell and Larcker’s 1981
publication in which 0.5 is the threshold value for AVE. While not specifically
using the term “divergent validity,” Yan et al. demonstrated that they
identified divergent variables, stating that “Six variables ... are dropped due
to their relatively low factor loadings for its construct” (p. 562).


 


Dahan et al. (2016) used exploratory factor analysis, assessed using
Bartlett’s test for sphericity and the Kaiser-Meyer-Olkin test, and determined
that the analysis was significant (p. 41). They then conducted Varimax testing
with the Kaiser Normalization Rotation method and found “that all Varimax
values are greater than 0.4 and therefore reflect the valid construct of all
items” (p. 41).


 


Intercoder Reliability 


 


Of the ten mixed methods studies (34%), two authors (7%) presented
validity evidence in their reports in the form of inter-rater agreement on
thematic analysis. Two sets of researchers (Jacoby, Ward, Avery, & Marcyk, 2016; Swoger &
Hoffman, 2015) showed evidence of intercoder reliability, as they both
discussed and reviewed their coding process; however, they did not report
reliability coefficients. 


 


Multiple Examples of Validity Evidence


 


Blake et al. (2016) provided evidence of face
validity, item validity, and construct validity within their study; Huang et
al. (2015) demonstrated testing for face validity and construct validity; and Masrek and Gaskin (2016) also showed evidence of a combination
of face validity and construct validity. Blake et al. changed their survey “to
reflect the responses received from librarian reviews and campus interview
sessions,” and consulted instrument development experts who helped them address
content (item) and internal structure (construct) validity components (p. 227).
Huang et al. invited 15 members of the college faculty to participate in their
pre-test, changing the wording of some items based on the faculty’s suggestions
(p. 1181), and tested for convergent and divergent validity using composite
validity and average variance extracted; they determined that convergent
validity was “good” and discriminant validity was “strong” (p. 1185). Masrek and Gaskin showed evidence of a combination of face
validity and construct validity as they pre-tested their instrument with
students who were part of the target population, as well as with experts in the
faculty, and as they analyzed the scales within their instrument for convergent
and discriminant validity (p. 44). We did not find evidence that any of these
researchers looked for convergent and discriminant validity with similar or
different instruments.


 


Reliability Indicators


 


Most (83%) of the studies did not state if they had
tested their instruments for reliability. Five articles (17%) (Chen, 2016; Dahan, et al., 2016; Huang et al., 2015; Masrek & Gaskin, 2016; Yan, et al., 2015) reported
measurement of internal consistency for each component of the satisfaction
construct when the component was measured by multiple scale items. All of the
researchers used Cronbach's ɑ (alpha) test of internal
consistency, in which a value of 0.70 is commonly believed to be a basic
threshold of acceptable level (Nunnally, 1978). Chen (2016) reported ɑ values ranging from 0.7305 to 0.8020, which represented a “satisfactory
level of reliability” (p. 322). Dahan et al. (2016)
reported alpha values ranging from 0.813 to 0.942 (p. 41). Values in Huang et
al.’s study (2015) ranged from .809 to .919 (p. 1184). Masrek
& Gaskin recorded ɑ “well above 0.7” (p. 42), with values ranging from
0.707 to 0.812. Yan, et al. (2015) did not report separate values for each
factor, stating that “Cronbach's alphas of all factors exceed 0.8” (p. 562). 


 


Discussion


 


This rapid review
demonstrates that a less comprehensive
and time-consuming type of systematic review of measurement properties
can be a useful approach to gaining an overview of research by practicing
librarians, as well as pointing to areas for improvement. Our review confirms
some aspects of research studies that other librarian researchers have attended
to and identifies opportunities for further research. This discussion will
place our results within a broader context, followed by recommendations for
improvements in practicing librarians’ instrument design.


 


While solely quantitative study designs continue to be most common in
studies of satisfaction with reference services, approximately one-third of the
studies we located also gathered patron feedback via open-ended questions. For
comparison, VanScoy and Fontana determined in their
study of research approaches to reference and information service that
quantitative studies ranged from 56.65% in 2005 to 83.33% in 2009 (2016, p.
96). In our review, researchers used surveys alone to gather data 86% of the
time, which is higher than the usage of surveys by 50.5% of practitioner
researchers according to Hildreth and Aytac (2007) or
the 62.3% of researchers writing about library instruction, as determined by
Crawford and Feldt (2007, p. 84). However, our results are not surprising given
the quantitative design and measurement goal of the studies we identified. We
found an improvement over McKechnie et al.’s study in which 17.6% of articles
included an appended instrument (2016), with 31% providing this service.


 


Mahmood’s systematic review found that Likert scales “or Likert-type
scoring methods” were used in 15 of 22 scales, and that the “points for scoring
options ranged from 2 to 11” but did not report further detail about the design
of each Likert scale (p. 1044). It could be useful therefore to compare our
results regarding Likert-scale design with studies outside of LIS. Our partial
model for this study, Siddiq et al.’s systematic review of Information and
Communication Technology-literacy assessment instruments, did not include this
information, but Roth, Ogrin, and Schmitz (2016)
reported in their systematic instrument review that of seven instruments
containing Likert scales, three employed 4-point Likert scales, three contained
5-point scales, and two had 7-point scales. These findings indicate that little
agreement exists as to best practices in scale formation. Research on Likert
scale questions suggests that 4-point response scales with a “no opinion”
option avoid the 5-point scale’s potential for central tendency bias
(respondent desire to appear moderate rather than extreme) or social
desirability bias (respondent desire to avoid controversial topics). This
research implies that if a 5-point scale is offered, the midpoint should be
clearly labeled, as otherwise respondents might assign various meanings to the
midpoint such as “don’t know,” “neutral,” or “unsure” (Nadler, Weston, &
Voyles, 2015). 


 


Librarian researchers might not adequately define “satisfaction,” as
only four researchers (14%) developed question items addressing more than one
aspect of this construct. Lasda Bergman and Holden
(2010) identified four components of the satisfaction construct: willingness to
return, positivity of experience, staff quality, and willingness to recommend a
service to a colleague. Luo and Buer’s instrument
(2015) included 10 components to express satisfaction but did not address
another potential component: ethical issues as identified by Kloda and Moore’s
(2016) question item, “The consult reflected a respect for my confidentiality
as a library user.” On their survey measuring satisfaction with digital library
service, including virtual reference, Masrek and
Gaskin (2016) included 24 items representing 6 component factors of
satisfaction, in addition to three items related to overall satisfaction (Masrek & Gaskin, personal communication, April 20,
2017). Instrument developers might consider that responses to a single question
about satisfaction are likely to be positive because “providing tailored
individual help … will always be appreciated, which skews user satisfaction in
survey results” (Fournier & Sikora, 2015, p. 255). When measuring multiple
aspects of the satisfaction construct, a researcher can determine which aspect most
likely detracts or adds to patron satisfaction, and initiate training, other
services, and environmental improvements to address any issues.


 


We were surprised to find that four of the studies we examined (Akor & Alhassan, 2015; Dahan
et al., 2016; Duan, 2016; Xie
& Sun, 2015) contained “double-barreled questions” (Olson, 2008, p. 210) or
“multi-concept” to use Glynn’s phrase (2006, p. 394), which asked respondents
to agree with statements containing two themes combined with “and” such as
“librarians are competent and helpful,” or to rate librarians’ “help and
answers.” Because the researcher doesn’t know which aspect of librarian service
respondents are rating – competence or helpfulness? help or answers? – these
items cannot contribute meaningfully to statistical analysis. Moreover,
respondents will likely take more time to consider each concept, potentially
leading to survey fatigue. Bassili and Scott (1996)
found that “questions took significantly longer to answer when they contained
two themes than when either of their themes was presented alone” (p. 394). If
researchers might design a survey instrument addressing the various components
that make up the satisfaction construct, and thus listing several items to
cover these components, it is important to make the items as simple to answer
as possible, to encourage respondents to complete the survey. Researchers will
usually catch multi-concept questions during a careful pre-testing process.
 


 


Half of the studies we located contained evidence of
instrument validity, while more than three-quarters did not report data on
instrument reliability, which is comparable to results from Mahmood’s (2017)
systematic review of instruments, and results from similar reviews in other
disciplines. While explaining that the study excluded articles without validity
or reliability evidence, Mahmood (2017) stated that “A large number of studies
reported surveys on assessing students’ self-efficacy in IL skills but without
mentioning any reliability and validity of scales” and that “the present
study’s results are consistent with systematic reviews in other areas,”
reporting that between 25% and 50% of studies in three systematic reviews
outside of librarianship included information on validity and reliability of
instruments (p. 1045):


 


For example, the reliability and validity were
reported in only one-third of studies about evaluation methods of continuing
medical education. . . . A study of 11 urbanicity
scales found that psychometric characteristics were not reported for eight
instruments. . . . A recent systematic review in the
area of assessing students’ communication skills found that less than half of
studies reported information on reliability and validity . . . (p. 1045).


 


Our model instrument review researchers Siddiq et al. found that 12 of
30 test developers (40%) reported validation of the test in at least one
publication, and that 24 of the 30 (80%) reported reliability evidence
according to the authors’ criteria (p. 75). The reporting of validity and
reliability evidence can help the reader determine which instrument to use in
replicating a study and could aid in future development of an instrument that
might combine constructs and items identified through a similar review.


 


Recommendations


 


Obtain Training and Refer to Research-Evaluation Checklists


 


Compared to classroom faculty, librarians are frequently at a
disadvantage in designing research projects because they lack coursework in
research methods. Initiatives such as Loyola Marymount University’s Institute
for Research Design in Librarianship, the Medical Library Association’s
Research Training Institute for Health Sciences Librarians, and occasional
professional development opportunities, assist librarians to build their
research knowledge but can’t reach every librarian. For these reasons, we
recommend that librarians become more familiar with existing checklists of
research evaluation (e.g., those provided by Glynn, 2006; and
McKechnie et al, 2016) that can ensure a basic level of structure and rigor,
and further recommend that researchers expand upon these lists as the need for
research guidance becomes apparent. Based on our study, we believe that
checklists for librarians need to include more guidance in instrument design
and in communicating instrument details, e.g., by making sure the target
construct is adequately measured; by addressing validity and reliability; by
designing questions and response items carefully; including the full
instrument; and citing prior instruments.


 


Completely Measure the Construct


 


When designing a research instrument, a researcher needs to determine
which construct to measure, and which items will best represent that construct,
whether it be satisfaction or any other construct. The researcher should keep
in mind that more specific items avoid the problem of confounding variables
which influence the respondent’s answer, or of misinterpretation in which the
respondent’s definition of a construct differs from the researcher’s intended
definition. In the realm of “satisfaction” with a service, many factors could
influence respondents’ opinion of the service being measured, such as librarian
behaviour or performance. It is therefore important
to offer several items, rather than a single question about satisfaction. 


 


Address Validity and Reliability


 


After drafting questions and items, researchers will want to ensure
their instrument has face validity by pre-testing it, with non-librarian
subjects similar to the target respondent population and with experts in
instrument design. These pilot testers should look for bias, for example
avoiding questions such as “How much has this service improved your life?”
which assume a positive response; for clarity and avoiding the use of jargon,
defining terms as Otto (2016) recommended; and for evidence that the question
or item addresses what it is intended to address. If researchers try to address
all variables encompassing the “satisfaction” construct and report this effort
in their paper, that will show evidence of sampling validity. An instrument
with many items could be refined by performing analyses to determine convergent
and divergent items, thus demonstrating construct validity. If the instrument
has been translated into or from a language other than English, developers
should report separate validity and reliability information for each version of
the instrument. 


 


Design Questions and Response Items Carefully


 


We repeat Glynn’s (2006) recommendation that not only questions but also
their “response possibilities” should be “posed clearly enough to be able to
elicit precise answers” (p. 389). Glynn cautions that the Likert scale (i.e.,
strongly agree, agree, no opinion, disagree, strongly disagree) “[lends itself]
to subjectivity and therefore the accuracy of the response is questionable” (p.
394).  Regardless of the scale researchers select, we recommend
employing a 4-, 5-, or 7-point Likert scale. Avoiding 2-point scales allows for
variance in opinion, and avoiding 9-point scales or higher avoids dilution of
opinion. We further recommend that researchers label the scale points in a
uniform fashion but minimally, e.g., “Strongly/Somewhat/Agree” and
“Strongly/Somewhat/Disagree,” rather than offer lengthy definitions of each
point as seen in Duan(2016).


 


Include the Full Instrument


 


Several authorities on research (Connaway
& Radford, 2017; Glynn, 2006; McKechnie et al., 2016) also agree that, to
quote Glynn, “the data collection method must be described in such detail that
it can easily be replicated” (2006, p. 393). Ideally, these authors further
agree, researchers would include their instrument within the body or as an
appendix of their publication, or as an online appendix. We recommend expanding
existing checklists for the evaluation of research in librarianship, e.g.,
Glynn’s Critical Appraisal Tool for Library and Information Research (2006) and
McKechnie et al.’s Research Rigour Tactics (2016), to
remind authors that when they include an appendix containing the instrument,
they should also note its inclusion in their abstract, to increase the
likelihood that future researchers will locate it. If this inclusion is not
possible, then a detailed description of the instrument should be reported in
the body of the paper:


 


	The
     number of questions and items
	The
     full text of each question and associated item
	Question
     format: scale range and endpoint labels, e.g., “Agree” and “Disagree”
	Where
     relevant, the average time needed to complete the instrument 



 


With this information in hand, researchers can readily reproduce the
instrument and use it in their own research. 


 


Cite Prior Instruments


 


We recommend also that authors cite sources if they
base their instrument on previous efforts, demonstrating connections with prior
research and further helping to identify useful instruments. Blevins et al.
(2016) wrote that “three librarians reviewed the existing literature for
similar surveys and developed a set of questions to assess customer service
quality” (p. 287) but did not cite the similar surveys. By citing contributing
studies, librarians uphold the professional value of encouraging their
colleagues’ professional development as stated in the ALA Code of Ethics
(2008). 


 


Further Research


 


As more librarians implement the instrument review methodology,
opportunities for future research will abound. Reviews are needed in other
research areas, for example to evaluate instruments gathering librarian
attitudes toward teaching, collection development, or collaborating with
faculty. While we have presented one model for this methodology, there is ample
room for improvement and refinement of the method; we foresee that specific
standards for instrument review could be developed for librarianship. As
described above, another opportunity for future research would be to examine
concerns of sampling validity, i.e., which items best demonstrate the patron
satisfaction construct. 


 


Limitations


 


As a rapid review
examining two years of librarian research, this study’s results are not necessarily representative of the
body of work on student satisfaction with academic library reference services.
Although we ran keyword as well as subject searches, it is possible that we did
not gather all possible studies presenting librarian-developed instruments due
to inconsistent indexing. It is possible that we have missed relevant articles
due to not manually searching all LIS journals related to our research topic.
 


 


Our descriptive model does not extend to evaluation of
the instrument’s appropriateness in different scenarios such as in-house
research versus research intended for publication. While we generally recommend
designing an instrument that offers questions with several items measuring the
satisfaction construct, it could be appropriate to include a single question
and item addressing satisfaction when service quality assurance is the goal.
For example, Swoger and Hoffman (2015) incorporated a
single question about the usefulness of a specific 


 


type of reference service; in their context the single
question was primarily used for local service evaluation and might have been
appropriate.


 


Conclusion


 


The quality of a research project depends on valid and reliable data
collection methods. In preparation for a study, librarians should search
broadly and attempt to locate the best instrument exemplars on which to model
their own data-gathering method. If researchers do not have time for a
comprehensive systematic review, the present study demonstrates that a rapid
review can reveal a range of research instruments and guide the development of
future instruments. It further demonstrates that the characteristics of
librarian-produced research instruments vary widely, and that the quality of
reporting varies as well. If librarians do not aim to produce high-quality data
collection methods, we need to question our collective findings. By following
the recommendations presented here, future researchers can build more robust
LIS literature.


 


Acknowledgements


 


The authors would like to thank the University of Portland for granting
the lead author a sabbatical, during which much of the work of this study was
accomplished.


 


References


 


Akor, P. U., & Alhassan, J. A. (2015). Evaluation of reference services
in academic libraries: A comparative analysis of three universities in Nigeria.
Journal of Balkan Libraries Union, 3(1), 24-29.


 


American Library Association. Evaluation of Reference and Adult
Services Committee. (1995). The reference assessment manual. Ann Arbor,
MI: Pierian Press.


 


Askew, C. (2015). A mixed methods approach to
assessing roaming reference services. Evidence Based Library and Information
Practice, 10(2), 21-33. https://doi.org/10.18438/B8F60V


 


Bassili, J.
N., & Scott, B. S. (1996). Response latency as a signal
to question problems in survey research. Public Opinion Quarterly, 60(3),
390–399. https://doi.org/10.1086/297760



 


Beile, P. (2008,
March). Information literacy assessment: A review of objective and interpretive
methods and their uses. Paper presented at the Society for Information
Technology & Teacher Education International Conference (SITE), Las Vegas, NV. Retrieved from http://eprints.rclis.org/15861/1/Info%20Lit%20Assessment.pdf


 


Blake, L., Ballance, D., Davies, K., Gaines, J. K.,
Mears, K., Shipman, P., . . . Burchfield, V. (2016). Patron perception and
utilization of an embedded librarian program. Journal of the Medical Library
Association: JMLA, 104(3), 226-230. https://doi.org/10.3163/1536-5050.104.3.008



 


Blevins, A. E., DeBerg, J.,
& Kiscaden, E. (2016). Assessment of service desk quality at an
academic health sciences library. Medical Reference Services Quarterly, 35(3),
285-293. https://doi.org/10.1080/02763869.2016.1189782



 


Boyce, C. M. (2015). Secret shopping as user
experience assessment tool. Public Services Quarterly, 11, 237-253. https://doi.org/10.1080/15228959.2015.1084903



 


Butler, K., & Byrd, J. (2016). Research consultation
assessment: Perceptions of students and librarians. Journal of Academic
Librarianship, 42(1), 83-86. https://doi.org/10.1016/j.acalib.2015.10.011



 


Catalano, A. J. (2016). Streamlining LIS research:
A compendium of tried and true tests, measurements, and other instruments.
Santa Barbara, CA: Libraries Unlimited.


 


Chen, Y. (2016). Applying the DEMATEL approach to identify the focus of
library service quality. The Electronic Library, 34(2), 315-331.  https://doi.org/10.1108/EL-08-2014-0134   


 


Connaway, L. S., &
Radford, M. L. (2017). Research methods in library and information science
(6th. ed.). Santa Barbara, CA: Libraries Unlimited.


 


Crawford, G. A., & Feldt, J. (2007). An analysis of the literature
on instruction in academic libraries. Reference & User Services
Quarterly, 46(3), 77-87.


 


Dahan, S. M., Taib, M. Y., Zainudin,
N. M., & Ismail, F. (2016). Surveying users' perception of academic library
services quality: A case study in Universiti Malaysia
Pahang (UMP) library. Journal of Academic Librarianship, 42(1), 38-43. https://doi.org/10.1016/j.acalib.2015.10.006



 


Duan, X. (2016). How they search, how they feel, and how to serve them?
Information needs and seeking behaviors of Chinese students using academic
libraries. International Information & Library Review, 48(3),
157-168. https://doi.org/10.1080/10572317.2016.1204179


 


Ekere, J. N., Omekwu, C. O., & Nwoha, C. M. (2016). Users’ perception of the facilities,
resources and services of the MTN Digital Library at the University of Nigeria,
Nsukka. Library Philosophy & Practice, 3815, 1-23.


 




Fidel, R. (2008). Are we there yet? Mixed methods research in library
and information science. Library & Information Science Research, 30(4),
265-272. https://doi.org/10.1016/j.lisr.2008.04.001


 


Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation
models with unobservable variables and measurement error. Journal of
Marketing Research, 18(1), 39-50.


 


Fournier, K., & Sikora, L. (2015). Individualized research
consultations in academic libraries: A scoping review of practice and
evaluation methods. Evidence Based Library and Information Practice, 10(4),
247-267. https://doi.org/10.18438/B8ZC7W



 


Ganaie, S. A. (2016). Satisfaction of library and information science students
with the services provided by Allama Iqbal Library of
Kashmir University. International Research: Journal of Library and
Information Science, 6(3), 504-512.


 


Gerlich, B. K., & Berard, G. L. (2007).
Introducing the READ Scale: Qualitative statistics for academic reference
services. Georgia Library Quarterly, 43(4),
7-13.


 


Glynn, L. (2006). A critical appraisal tool for
library and information research. Library Hi Tech, 24(3), 387-399.


 


Gotch, C. M., &
French, B. F. (2014). A systematic review of assessment literacy measures.
Educational Measurement: Issues and Practice, 33(2), 14-18.


 


Grant, M., & Booth, A. (2009). A typology of
reviews: An analysis of 14 review types and associated methodologies. Health
Information and Libraries Journal, 26, 91-108.  


https://doi.org/10.1111/j.1471-1842.2009.00848.x


 


Greifeneder (2014). Trends in information behaviour
research. Paper presented at the Proceedings
of ISIC: The Information Behaviour Conference,
Leeds, England. Retrieved from http://www.informationr.net/ir/19-4/isic/isic13.html


 


Hildreth, C. R., & Aytac,
S. (2007). Recent library practitioner research: A methodological analysis and
critique. Journal of Education for Library and Information Science, 48(3),
236-258.


 


Huang, Y., Pu, Y., Chen, T., & Chiu, P. (2015).
Development and evaluation of the mobile library service system success model. The
Electronic Library, 33(6), 1174-1192. 


 


Ikolo, V. E. (2015). Users satisfaction with library services: A case study
of Delta State University library. International Journal of Information and
Communication Technology Education, 11(2), 80-89.


 


Jacoby, J., Ward, D., Avery, S., & Marcyk, E. (2016). The value of chat reference services: A
pilot study. portal: Libraries and the Academy, 16(1), 109-129.


 


Jordan, J. L. (2015). Additional search strategies may
not be necessary for a rapid systematic review. Evidence Based Library &
Information Practice, 10(2), 150-152. https://doi.org/10.18438/B8FC77 


 


Khobragade, A. D., & Lihitkar,
S. R. (2016). Evaluation of virtual reference service provided by
IIT libraries: A survey. DESIDOC Journal of Library & Information
Technology, 36(1), 23-28. 


 


Kloda, L.A., & Moore, A. J. (2016). Evaluating
reference consults in the academic library. In S. Baughman, S.Hiller,
K. Monroe, & A. Pappalardo (Eds.), Proceedings
of the 2016 Library Assessment Conference Building Effective, Sustainable,
Practical Assessment (pp. 626-633). Washington, DC: Association of
Research Libraries.


 


Koufogiannakis, D.
(2012). The state of systematic reviews in library and information studies.
Evidence Based Library & Information Practice, 7(2), 91-95. 


 


Koufogiannakis, D., Slater, L., & Crumley, E. (2004). A content
analysis of librarianship research. Journal of Information Science, 30(3),
227-239.


 


Lasda Bergman, E. M., & Holden, I. I. (2010). User satisfaction with
electronic reference: A systematic review. Reference Services Review, 38(3),
493-509.  https://doi.org/10.1108/00907321011084789


 


Luo, L., & Buer, V. B.
(2015). Reference service evaluation at an African academic library: The user
perspective. Library Review, 64(8/9), 552-566.


 


Mahmood, K. (2017). Reliability and validity of
self-efficacy scales assessing students’ information literacy skills.
Electronic Library, 35(5), 1035-1051. https://doi.org/10.1108/EL-03-2016-0056


 


Masrek, M. N., &
Gaskin, J. E. (2016). Assessing users satisfaction
with web digital library: The case of Universiti Teknologi MARA. The International Journal of Information
and Learning Technology, 33(1), 36-56.


 


McKechnie, L. E., Baker, L., Greenwood, M., &
Julien, H. (2002). Research method trends in human information literature.
New Review of Information Behaviour Research, 3,
113-125.


 


McKechnie, L., Chabot, R.,
Dalmer, N., Julien, H., & Mabbott, C. (2016, September). Writing
and reading the results: The reporting of research rigour tactics in
information behaviour research as evident in the published proceedings of the
biennial ISIC conferences, 1996 – 2014. Paper presented at the ISIC: The
Information Behaviour Conference, Zadar, Croatia. Retrieved from http://www.informationr.net/ir/21-4/isic/isic1604.html  


 


McKibbon, K. A. (2006). Systematic reviews and librarians. Library Trends, 55(1),
202-215.  https://doi.org/10.1353/lib.2006.0049


 


Moher, D., Liberati, A.,
Tetzlaff, J., & Altman, D. G. (2009). Preferred
reporting items for systematic reviews and meta-analyses: The PRISMA statement.
PLoS Medicine, 6(7),
e1000097.   https://doi.org/10.1371/journal.pmed.1000097


 


Mohindra, R., & Kumar, A. (2015). User satisfaction regarding quality of library services of A.C. Joshi
Library, Panjab University, Chandigarh. DESIDOC Journal of Library &
Information Technology, 35(1), 54-60.


 


Nadler, J. T., Weston, R., & Voyles, E. C. (2015).
Stuck in the middle: The use and interpretation of mid-points in items on
questionnaires. Journal of General Psychology, 142(2), 71-89.  https://doi.org/10.1080/00221309.2014.994590


 


Nicholas, P., Sterling, J., Davis, R., Lewis, J. C., Mckoy-Johnson,
F., Nelson, K., Tugwell, Y., & Tyrell, K. (2015).
“Bringing the library to you!”: The halls of residence librarian program at the
University of the West Indies, Mona Library. New Library World, 116(5/6),
316-335.


 


Nunnally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill.


 


Olson, K. (2008). Double-barreled question. In P. J. Lavrakas
(Ed.), Encyclopedia of survey research methods (p. 210). Thousand Oaks,
CA: SAGE Publications Ltd.  https://doi.org/10.4135/9781412963947.n145


 


Otto, J. J. (2016). A resonant message: Aligning
scholar values and open access objectives in OA policy outreach to faculty and
graduate students. Journal of Librarianship and Scholarly Communication, 4, eP2152.
https://doi.org/10.7710/2162-3309.2152


 


Phelps, S. F., & Campbell, N. (2012). Systematic
reviews in theory and practice for library and information studies. Library
& Information Research, 36(112), 6-15. 


 


Reference and User Services Association. (2008). Definitions of reference. Retrieved from
http://www.ala.org/rusa/guidelines/definitionsreference 


 


Roth,
A., Ogrin, S., & Schmitz, B. (2016). Assessing self-regulated learning in higher education: A systematic
literature review of self-report instruments. Educational Assessment,
Evaluation and Accountability, 28(3), 225-250.


 


Schilling, K., & Applegate, R. (2012). Best
methods for evaluating educational impact: A comparison of the efficacy of
commonly used measures of library instruction. Journal of the Medical
Library Association: JMLA, 100(4), 258-269. https://doi.org/10.3163/1536-5050.100.4.007


 


Siddiq, F., Hatlevik, O. E.,
Olsen, R. V., Throndsen, I., & Scherer, R.
(2016). Taking a future perspective by learning from the past – A systematic
review of assessment instruments that aim to measure primary and secondary
school students' ICT literacy. Educational Research Review, 19, 58-84. https://doi.org/10.1016/j.edurev.2016.05.002


 


Sivagnanam, R., & Esmail, S. M. (2015). User satisfaction and performance of
arts and science college libraries situated in Cuddalore
District: A study. International Journal of Library and Information Studies,
5(5), 145-155.


 


Swoger, B. J. M.,
& Hoffman, K. D. (2015). Taking notes at the reference desk: Assessing and
improving student learning. Reference Services Review, 43(2),
199-214.  


https://doi.org/10.1108/RSR-11-2014-0054


 


Tiemo, P. A., & Ateboh, B. A. (2016). Users’
satisfaction with library information resources and services: A case study
College of Health Sciences Library Niger Delta University, Amassoma,
Nigeria. Journal of Education and Practice, 7(16): 54-59.


 


VanScoy, A., & Fontana, C. (2016). How reference and information service is
studied: Research approaches and methods. Library & Information Science
Research, 38(2), 94-100.  https://doi.org/10.1016/j.lisr.2016.04.002


 


Verma, M. K., & Laltlanmawii,
R. (2016). Use and user’s satisfaction on library resources and services by UG
students of Government Hrangbana College, Aizawl: A study. Journal of Advances in Library and
Information Science, 5(1), 18-23.


 


Verma, M. K., & Parang, B. (2015). Use and user’s satisfaction on
library resources and services by students of School of Physical Sciences,
Mizoram University, Aizawl: A study. Information Studies, 21(2),
193-204. 


 


Watts, J., & Mahfood, S. (2015).
Collaborating with faculty to assess research consultations for graduate
students. Behavioral & Social Sciences Librarian, 34(2), 70-87.  https://doi.org/10.1080/01639269.2015.1042819


 


Xie, J., & Sun, L. (2015). Exploring Chinese students' perspective on
reference services at Chinese academic libraries: A case study approach.
Journal of Academic Librarianship, 41(3), 228-235.  https://doi.org/10.1016/j.acalib.2015.04.002


 


Xu, J., Kang, Q., & Song, Z. (2015). The current state of systematic reviews in library and information
studies. Library & Information Science Research, 37(4), 296-310. https://doi.org/10.1016/j.lisr.2015.11.003



 


Yan, W., Hu, Y., & Hu, C. (2015). Using SEM and
TCM to improve services to libraries: A comparative study on Chinese academic
libraries. Journal of Academic Librarianship, 41(5), 558-566. https://doi.org/10.1016/j.acalib.2015.07.006



 


Yap, J. M., & Cajes, M. L. (2016). Where
are the librarians? The roving reference experience of DLSU libraries. Information
World/Bilgi Dünyası, 17(2), 232-246. Retrieved
from http://bd.org.tr/index.php/bd/article/view/506/575  


 


 


Appendix A


Database Searches


 


CINAHL


	
  Search ID

  	
  Search Terms

  
	
  S5

  	
  librar* AND (reference or consultation or roaming or chat) AND (satisfaction
  or attitudes) AND (survey* or instrument* or questionnaire* or interview* or
  focus group*)

  
	
  S4

  	
  (S1 AND S2 AND S3)

  
	
  S3

  	
  (MH "Research Instruments+")

  
	
  S2

  	
  (MH "Consumer Satisfaction+")

  
	
  S1

  	
  (MH "Library Reference Services")

  



 


ERIC


	
  Search ID

  	
  Search Terms

  
	
  S9

  	
  ((surveys or questionnaires or instruments or
  measures or interviews)) AND (S2 AND S4 AND S7 AND S8)

  
	
  S8

  	
  (surveys or questionnaires or instruments or
  measures or interviews)

  
	
  S7

  	
  DE "Libraries" OR DE "Research
  Libraries" OR DE "Medical Libraries" OR DE "College
  Libraries" OR DE "Academic Libraries" OR DE "Two Year
  Colleges"

  
	
  S6

  	
  ((surveys or questionnaires or instruments or
  measures or interviews) AND (DE "Surveys" OR DE "Attitude
  Measures" OR DE "Interviews" OR DE "Measures (Individuals)"
  OR DE "Questionnaires" OR DE "Research" OR DE "Semi
  Structured Interviews" OR DE "Structured Interviews")) AND (S1
  AND S2 AND S4 AND S5)

  
	
  S5

  	
  (surveys or questionnaires or instruments or
  measures or interviews) AND (DE "Surveys" OR DE "Attitude
  Measures" OR DE "Interviews" OR DE "Measures
  (Individuals)" OR DE "Questionnaires" OR DE
  "Research" OR DE "Semi Structured Interviews" OR DE
  "Structured Interviews")

  
	
  S4

  	
  DE "Attitudes"
  OR DE "Satisfaction" OR DE "Job Satisfaction" OR DE
  "Life Satisfaction" OR DE "Marital Satisfaction" OR DE
  "Participant Satisfaction" OR DE "Student
  Satisfaction" OR DE "User Satisfaction (Information)" OR DE
  "School Attitudes" OR DE "Student Attitudes"

  
	
  S2

  	
  DE "Reference Services" OR DE
  "Library Services"

  
	
  S1

  	
  (DE "Academic Libraries" OR DE
  "College Libraries" OR DE "Research Libraries") AND (DE
  "Reference Services" OR DE "Library Services")

  



 


PubMed –With Applied COSMIN Filter


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  academic library AND reference or consultation or
  roaming or chat) AND (satisfaction or attitudes) AND (survey* or instrument*
  or questionnaire* or interview* or focus group*)

  
	
  S2

  	
  academic librar* AND
  reference or consultation or roaming or chat) AND (satisfaction or attitudes)
  AND (survey* or instrument* or questionnaire* or interview* or focus group*)

  
	
  S3

  	
  (academic librar*) AND
  (reference or consultation or roaming or chat) AND (satisfaction or
  attitudes) AND (survey* or instrument* or questionnaire* or interview* or
  focus group*) 

  
	
  S4

  	
  ((academic library) AND consumer satisfaction) AND
  (survey OR instrument OR questionnaire OR interview OR focus group)

  
	
  S5

  	
  (((library reference services) AND consumer
  satisfaction) AND (((((survey) OR instrument) OR questionnaire) OR interview)
  OR focus group))

  



 


ProQuest Library Science Database (LS) –  Initial Search


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  all((academic*
  AND librar* AND (reference OR "customer
  satisfaction" OR "user satisfaction" OR "customer
  services") )) AND su(research or surveys or
  questionnaires) AND ab(study or survey* or interview* or research*) 

  



 


ProQuest Library Science Database (LS)- Subsequent
search


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  all((reference
  or "research consultation") AND (satisf*
  or evaluat* or assess* or improve*) AND
  (experiment* or survey* or qualitative or servqual
  or instrument or investigat* or analysis or
  questionnaire*) AND "academic librar*") 

  



 


Google Scholar


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  "library reference services" AND
  "consumer satisfaction" AND research instruments"

  
	
  S2

  	
  academic library AND (reference OR consultation OR
  roaming OR chat ) AND (satisfaction or attitudes ) AND ( survey* OR
  instrument* OR questionnaire* OR interview* OR focus group*)

  
	
  S3

  	
  Librar* AND reference AND satisfaction

  
	
  S4

  	
  librar* AND (reference or consultation or roaming or chat) AND (satisfaction
  or attitudes) AND (survey* or instrument* or questionnaire* or interview* or
  focus group*)

  
	
  S5

  	
  librar* AND (reference or consultation or roaming or chat) AND (satisfaction
  or attitudes) AND (survey* or instrument* or questionnaire* or interview* or
  focus group*)

  
	
  S6

  	
  library AND reference AND satisfaction

  
	
  S7

  	
  (academic librar*) AND
  (reference or consultation or roaming or chat) AND (satisfaction or
  attitudes) AND (survey* or instrument* or questionnaire* or interview* or
  focus group*) 

  



 


Library, Information Science & Technology
Abstracts (LISTA) – Initial Search


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  (reference
  or "research consultation" ) AND ( satisf*
  or evaluat* or assess* or improve*) AND (
  experiment* or survey* or qualitative or servqual
  or instrument or investigat* or analysis or
  questionnaire*) AND "academic librar*"

  
	
  S2

  	
  (academic
  AND librar* AND (reference OR "user
  satisfaction")) AND ( SU ( research or surveys or questionnaires ) OR AB
  (study or survey* or interview* or research*) )

  



 


LISTA – Subsequent Search


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  librar* AND reference AND satisfaction

  



 


Web of Science


	
  Search ID

  	
  Search Terms

  
	
  S1

  	
  (academic librar*) AND
  (survey* OR instrument* OR questionnaire* OR interview* OR focus group*) AND
  (satisfaction OR attitudes) AND (reference OR consultation OR roaming OR
  chat)

  
	
  S2

  	
  (library reference services) AND (survey* OR
  instrument* OR questionnaire* OR interview* OR focus group*) AND
  (satisfaction OR attitudes)

  
	
  S3

  	
  (consumer satisfaction) AND (library reference
  services)

  
	
  S4

  	
  (consumer satisfaction) AND (library reference
  services) AND (survey* OR instrument* OR questionnaire* OR interview* OR
  focus group*)

  
	
  S5

  	
  (librar*) AND (consumer
  satisfaction) AND (survey* OR instrument* OR questionnaire* OR interview* OR
  focus group*) AND (reference OR consultation OR roaming OR chat)

  
	
  S6

  	
  (librar*) AND (consumer
  satisfaction) AND (survey* OR instrument* OR questionnaire* OR interview* OR
  focus group*) AND (reference OR consultation OR roaming OR chat)

  



 


 


Appendix B


Data Extraction Elements


APA citation


Article title


Journal


Country


Stated Purpose of Study


Area of Reference Service: General, Desk, Embedded
Librarianship, External, Research Consultation, Virtual Reference (Chat or
Email)


Age/Level of Students -- if students were part of the
targeted population


Targeted Population


Usable Responses


Sampling Strategy


Research Design


Data Collection Method


Types of Questions, other than demographic


Demographics Gathered


Technical Aspects, such as distribution, availability
of translations, duration of survey period


Time Allotted to Complete Survey


Validity Indicators


Reliability Indicators


Instrument Availability


Replicability of Instrument


Comments