ISSN: 2474-3542 Journal homepage: http://journal.calaijol.org Local Collections in Discovery Services: An Inquiry Sharon Q. Yang, Patricia H. Dawson, and Jie Ding Abstract: Many libraries subscribed to discovery services in the hope of boosting the use of their local collections such as print materials, DVDs, or even subscribed ejournals and ebooks that are separate from licensed databases. However, anecdotal evidence shows that the implementation of a discovery service will increase the usage of the vendors’ electronic resources instead. This study aims to find out how well local collections are promoted through EBSCO Discovery Service (EDS), Primo, Summon, and WorldCat Discovery Services (WorldCat) by comparing results of queries in the discovery services versus the number of actual listings from these libraries’ traditional catalogs. Authors conducted three searches based on real-life student assignments in science, history, and sociology in each of the four discovery services. To offset the bias of location configuration, ten libraries with the same discovery tool were selected for each search and total results are averaged. ANOVA and Tukey Honest Significant Difference statistical analyses show differences among the four discovery services investigated. Discovery tools promote local collections in different ways. Some tools provide the possibility to list libraries’ local collections before listing the vendor’s database items, but others are less inclined to do so. To cite this article: Yang, S.Q., Dawson, P.H., & Ding, J. (2019). Local collections in discovery services: An inquiry. International Journal of Librarianship, 4(2), 3-33. To submit your article to this journal: Go to http://ojs.calaijol.org/index.php/ijol/about/submissions http://ojs.calaijol.org/index.php/ijol/about/submissions INTERNATIONAL JOURNAL OF LIBRARIANSHIP, 4(2), 3-33 ISSN:2474-3542 Local Collections in Discovery Services: An Inquiry Sharon Q. Yang, Patricia H. Dawson, Jie Ding Rider University, New Jersey, USA ABSTRACT Many libraries subscribed to discovery services in the hope of boosting the use of their local collections such as print materials, DVDs, or even subscribed ejournals and ebooks that are separate from licensed databases. However, anecdotal evidence shows that the implementation of a discovery service will increase the usage of the vendors’ electronic resources instead. This study aims to find out how well local collections are promoted through EBSCO Discovery Service (EDS), Primo, Summon, and WorldCat Discovery Services (WorldCat) by comparing results of queries in the discovery services versus the number of actual listings from these libraries’ traditional catalogs. Authors conducted three searches based on real-life student assignments in science, history, and sociology in each of the four discovery services. To offset the bias of location configuration, ten libraries with the same discovery tool were selected for each search and total results are averaged. ANOVA and Tukey Honest Significant Difference statistical analyses show differences among the four discovery services investigated. Discovery tools promote local collections in different ways. Some tools provide the possibility to list libraries’ local collections before listing the vendor’s database items, but others are less inclined to do so. Keywords: Discovery Service, Local Collections, EDS, Primo, Summon, WorldCat Discovery Services INTRODUCTION In 2013, Rider University Libraries implemented a discovery service in the hope of boosting the usage of local print collections. Prior to that, the circulation statistics for physical collections decreased steadily over the past five years. The library administration hoped that a discovery service would promote local borrowing, increase circulation of library materials, and attract more students to the library. However, the opposite was true in this case. The circulation statistics did not go up as a result of implementing the discovery service. Comparable to Rider University libraries, many libraries experienced a continued decline of their circulation of print and physical collections. The product descriptions of all the major discovery services market the promotion of local holdings. For instance, for EDS, “[…] customers have the option of influencing the Yang, Dawson and Ding / International Journal of Librarianship 4(2) 4 overall relevance weighting of their catalog and/or institutional repository. This optional setting enables all catalog and institutional repository records to appear higher (or lower) in the search results list relative to other content in the EDS profile” (EBSCO, 2019). Likewise, Primo “[…] can boost local items so that they appear higher on the result list than items coming from Primo Central, all other factors being equal” (Ex Libris, 2017). Similar statements are found about Summon and WorldCat (Ex Libris, 2014; OCLC, 2017). However, there are complaints about possible bias of discovery services. One librarian lamented that, “I have never come across a book from our catalog in all my research using the discovery service[…] As a small library our book circulation stats are already pretty dead, I doubt the discovery service will put any extra nails in the coffin. However it is still very much a possibility” (Regier, 2015). Others voiced the similar concerns (Asher et al., 2013; Parry, 2014). The continued decline of borrowing of local collections may be caused by many factors. What role discovery services have played in this continues to be unknown. Additionally, it has been a mystery about how discovery services presented local collections among the myriad of contenting resources. Nevertheless, a discovery tool determines the quality of users’ research and what materials they use as the evidence shows (Asher et al., 2013; Parry, 2014; Wang et al., 2018). Library collections are developed systematically over the years with carefully selected titles. It is vital that library managers should know how their collections are retrieved and displayed in discovery services as they have a huge impact on users’ research experience. The research described in this paper attempts to address a missing piece in our knowledge about discovery services, namely, to what degree web-scale discovery services promote local collections by examining search results in four major discovery services, EBSCO Discover Service (EDS), Primo, Summon, and WorldCat, with searches conducted in these libraries’ traditional catalogs. LITERATURE REVIEW A review of literature did not reveal any significant research on local collections in discovery services. However, abundant research exists on other aspects of discovery tools. Until now most research on discovery tools focused on usability (Jill, 2018; Power, 2018; Meirose & Lian, 2019; Rigda et al., 2018; Warren, 2017; Woods et al., 2016), user behavior (Cohen & Thorpe, 2015; Dempsey & Valenti, 2016), use pattern changes (Calvert, 2015; Fitzpatrick, 2010; Teolis et al., 2019), and the impact of discovery services on reference and library instruction (Copenhaver & Kochanes, 2016; Walker & Sims, 2012). There seems to be a gap on how discovery services display and promote content of libraries’ local collections. Probably more relevant and closely related to the study reported in this paper is research on changes in use patterns after the implementation of a discovery service. There is an overall increase in the usage of journals and electronic resources in the year after implementing a discovery service (Calvert, 2015; Greiner, 2014; Lawton, 2015; Levine-Clark et al.,2014). An earlier study by Grant Valley State University after they implemented Summon reported a 50% to 150% increase in students’ use of full-text databases and online collections (Fitzpatrick, 2010). Another study reported an increase of usage in e-journals and e-books and a decrease in print collections in Manitoba University Libraries after the implementation of Summon (O’Hara, 2012). Studies on EDS, Primo, and WorldCat produced similar findings that use for Yang, Dawson and Ding / International Journal of Librarianship 4(2) 5 electronic materials increased and print collections decreased (Asher et al., 2013; Calvert, 2015; Greiner, 2011; Lawton, 2015; Levine-Clark et al., 2014; Regier, 2015). Especially noticeable is the finding that a discovery service may influence researchers’ experiences (Asher et al. 2013; Parry 2014; Wang et al., 2018). According to a study by Bucknell University and Illinois Wesleyan University, researchers tend to use what they are given by a search service. For instance, students who used EDS used more journal articles for their assignment. Students who used Google Scholar and the library catalog used more books, while students who searched Summon used more newspaper and magazine articles (Asher et al. 2013). According to Andrew Asher, an assessment librarian at Indiana University at Bloomington, "It’s a logical impossibility to create a querying tool that doesn’t have any form of bias” (Parry, 2014). A more recent study echoed the previous findings that “Web-scale discovery services play an important role in directing users to scholarly contents (Wang et al., 2018). All the above cited studies and findings provide the background that led to the research project in this paper. The research described in this paper is an original idea. RESEARCH METHODS Purpose The purpose of the research is to prove or disprove anecdotal evidence that discovery services promote their own resources over a library’s own resources or vice versa. The four major Web- scale discovery tools, EDS, Primo, Summon, and WorldCat, were selected for investigation. Web-scale is a computer term denoting great processing and high-speed computing power. Web-scale discovery tools are top notch search engines found in most academic libraries. Sample To offset the bias from local configurations, ten libraries were randomly chosen from the customer list of the vendors or on the Internet for each discovery tool, with the exception of EDS libraries where fourteen were selected. This is because in the process of this research four EDS libraries either switched to another discovery tool or blocked guest access. The final sample libraries comprised academic institutions with multiple disciplines with an English language discovery tool regardless of their geographic locations. This resulted in a sample with 16 English speaking, but non-US libraries, in addition of 28 US academic libraries for a total of 44 libraries. Define terms: Physical vs. catalog items Local collections are defined as both physical items located in the library and catalog items which have a bibliographic record in the OPAC. The latter includes electronic collections such as e-books and physical items. It is the total number of items from a catalog regardless of their format. From a systems point of view, any data harvested from the local (traditional) catalog or local data comprise local collections. Discovery services recognize the source of records as from the local catalog or data, but cannot distinguish which are physically housed in the library and which are electronically online as both types are represented by bibliographic records from Yang, Dawson and Ding / International Journal of Librarianship 4(2) 6 the same source. Therefore, both physical items housed in the library and those that are not physical, but have bibliographic records in the local data (such as ebooks and ejournals) are both considered local collections in this research project. Some libraries choose to load electronic books into the catalog, while others made a decision not to do so. Primo and many WorldCat libraries do not have a separate classic catalog. In that case, we used catalog searches within the discovery tool for physical and catalog data. We only examined the top 20 search results in discovery layers because past research shows that 87% of the users will not view search results beyond first 20 (Guan & Jia, 2016) and that most students stop reading beyond first page of search results (Hanneke & O’Brien, 2016). Search statements Three different search statements were used based on real-life student research projects in science, sociology and history classes at Rider University. They include “pesticides AND environment,” an assignment for chemistry class; “Pearl Harbor AND attack” which was the subject for a paper in history classes; and “crime AND poverty,” a sociology assignment. The authors chose three different disciplines because the choice of subject may influence the search outcomes. Data collection The number of a library’s electronic items licensed by the vendor and catalog’s physical items respectively were recorded in the top 20 hits in discovery tool searches. In other words, the authors recorded both catalog items (any items that come from a library catalog records) and physical items (any items that are in house in a local library such as print materials and CD/DVD). To determine the number of physical and electronic items in a library’s local collection, a search was conducted in the local, traditional catalog. This information was compared to the amount of items displayed by the discovery service. If a library did not have a separate classic catalog, the number of local items were determined by using the facet of the discovery tools which was restricted to that library’s holdings. This procedure was used for the three search queries and in the four discovery services. The authors also compared the retrieved physical and catalog items to ascertain if users are seeing the same listing of local and catalog items across all of these interfaces. Theoretically, the same materials owned by the libraries should appear in the discovery services as well as the local catalog. An Excel file was used to record the data for analysis. Analysis of Data To reduce the noise caused by local configuration and differences among individual libraries in the sample, two statistical methods are used to analyze the data, ANOVA (analysis of variance) and Tukey HSD (honestly significant difference) test. Both are commonly used statistical procedures to detect differences among group means. The authors decided on a 95% of confidence level. It is a two-step process to determine if there is a statistically significant difference between the discovery services. “An ANOVA test can tell you if your results are significant overall, but it won’t tell you exactly where those differences lie. After you have run an ANOVA and found significant results, then you can run Tukey’s HSD to find out which specific groups’ means (compared with each other) are different. The test compares all possible https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/hypothesis-testing/anova/ https://www.statisticshowto.datasciencecentral.com/what-is-statistical-significance/ Yang, Dawson and Ding / International Journal of Librarianship 4(2) 7 pairs of means” (Statistics how to, 2019). StatCrunch, a commercial statistical software, is used to assist with data analysis. FINDINGS Demographics Authors decided to use 10 libraries for each discovery service for an estimated total capability to handle local collections. Figure 1 summarizes and describes the libraries in the sample. Figure 1. Geographic Locations of 44 Libraries in the Sample The sample comprises 44 libraries, 28 US academic libraries and 16 foreign academic libraries including those from Canada, UK, Australia, New Zealand, Sweden, Israel, and Italy. All the libraries in the sample use English language in their discovery services. EDS includes14 libraries in the sample while the other three discovery services only have 10 libraries each. This is because half way through the project 3 EDS libraries closed their discovery service to the public, requiring logins, and one EDS library switched to another discovery service. Four academic libraries were randomly chosen to replace these EDS libraries, thus resulting in 14 EDS libraries. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 8 Figure 2. Presence of a Classic Catalog along with a Discovery Service in 40 Libraries in the Sample Figure 2 describes the status of the libraries in the sample regarding the presence of a catalog. For a long time, libraries could not do away with classic catalogs. The discovery services existed side by side with a classic catalog in spite of the original vision for the next generation catalog which disposed favorably towards one user interface and one search across all the library resources. The trend is moving towards one instead of multiple user interfaces. This figure also shows how many libraries in the sample still maintain a discovery service and a separate classic catalog at the same time. The ten Primo libraries in the sample have combined the catalog and discovery service into one user interface, while EDS libraries still need to maintain a classic catalog. WorldCat seems to be in the process of consolidating both into one user interface, with three libraries out of ten that still maintain a separate catalog. Most Summon libraries still have a classic catalog. One user interface and one search for all resources seem to be a better solution from a user’s perspective. For the following three searches, percentages were determined from all data in a group and normalized to show the number within 20 search results. Search Statement One: Pesticides AND Environment Authors searched “pesticides AND environment” in the discovery services of all the 44 libraries in the sample. This topic is based on a student project from an organic chemistry class in Rider University. As this is science related topic, more journal articles from the databases are expected than books from the local collections. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 9 Figure 3. Pesticides AND Environment-Local Collections in the Discovery Top 20 Hits by Percentage & Number of Items Figure 3 displays how many physical and catalog items from the local collections appear in the top 20 search results in a discovery service when searching “pesticides AND environment.” For EDS, 2.5% or equivalent of 0.5 books out of the top 20 search results are physical items and 3% or equivalent of 0.6 books are catalog items, both of which are from the local catalog. In other words, 97.5% or 97% of the top 20 retrieved items in EDS are non-local resources. It is equivalent of 19 or more out of 20 that are journal articles and other electronic materials from databases. For Primo, 27% or equivalent of 5 items out of the 20 top retrieved items are physical items in local library and 48.5% or equivalent of 10 items out of the 20 top retrieved items are pulled from the local data. For Summon it is 31.5% (about 6 books) and 44% (about 9 books) respectively. The search resulted in 29.5 (about 6 books) and 48.5% (about 10 books) out of the top 20 discovery layer retrieved items for WorldCat. For “pesticides and environment,” EDS displays the least of local collections in comparison to the other three web-scale discovery tools. Is it possible that EDS libraries happen to owe the least materials on pesticides and environment? Figure 4 shows this is not the case. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 10 Figure 4. Pesticides AND Environment-Average Number of Physical and Catalog Items on Pesticides and Environment in Local Libraries Figure 4 shows the average number of materials that a library for a given discovery service contains for both physical and catalog items on “pesticides and environment” search. Those numbers came from searching the local catalog or retrieving the local catalog within the discover services. For physical items, EDS Libraries have most items on this subject, an average of 89 item per library, while Primo libraries have the least, an average of 56 items. This presented a striking contrast that EDS displayed the least number of local physical items in Figure 3 (2.5%), but its libraries had the most physical items (89 on average). For catalog items, WorldCat libraries have most items, an average of 220 items per library. It should be noted that it is also one of the discovery services that displays most local catalog items, 48.5%, as shown in Figure 3. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 11 Figure 5. Pesticides AND Environment-The Percentage of Local Physical Items in the Top 20 Discovery Search Per Library For the search “pesticides AND environment,” 0.6% of the total local collections (physical items) made it to the top 20 search results for EDS libraries, 10% for Primo and Summon, and 8% for WorldCat respectively. The statistics in Figure 5 show how much or the percentage of local collections discovery service presented to users in their top display if students search for the science project on “pesticides AND environment”. Primo, Summon, and WorldCat shared a similarity with the exception of EDS which showed a low percentage. Figure 6 illustrates the results for the search “pesticides AND environment.” The results show 0.3% of the total local collections (catalog items) made it to the top 20 search results for EDS libraries, 7% for Primo and Summon, and 4% for WorldCat respectively. Those statistics are similar to those of physical items in Figure 5. Appendix One shows the ANOVA and Tukey results for pesticides and environment. Both physical and catalog results show lower p values, 0.0047 and 0.0003, which are below 0.05, indicating that differences are found among discovery services. The p values in the Tukey table are also below 0.05 when comparing EDS with other discovery services indicating that EDS is statistically different from the other three discovery services with this query. However, no differences are found among Primo, Summon, and WorldCat as their respective p value is larger than 0.05. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 12 Figure 6. Pesticides AND Environment-Percentage of Catalog Items in the Top 20 Discovery Search per Library Per Library Search Statement Two: Pearl Harbor AND Attack This topic is based on a project for a US history class at Rider University. As the search involves a historical event, the retrieved items are expected to be more books from the catalog than journal articles from the databases. Local collections are more appropriate. As expected, the search on “Pearl Harbor AND attack” in each discovery service retrieved more physical and catalog materials from the local collections than the previous search on “pesticides and environment” as shown in Figure 7. EDS still contains the least local collections in its top 20 retrieved items at 19% for physical collections (equivalent to 4 books) and 26% for catalog collections (equivalent to 5 books), while WorldCat had the most with 47% (equivalent to 9 books) and 92% (equivalent to 18 books) out of 20 top retrieved items. Primo and Summon were somewhere in the middle. EDS is consistently low in presenting local collections. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 13 Figure 7. Pearl Harbor AND Attack-Local Collections in the Discovery Top 20 Hits by Percentage and Number of Items Figure 8. Pearl Harbor AND Attack- Average Number of Physical and Catalog Items on Pearl Harbor and attack in Local Libraries Yang, Dawson and Ding / International Journal of Librarianship 4(2) 14 Figure 8 shows how many items libraries have on “Pearl Harbor AND attack” in local collections. These resources are displayed as the average number of books and other physical materials that a library includes for both physical and catalog items on “Pearl Harbor AND attack.” These numbers came from searching the local catalog. For physical items, EDS libraries have the most on this subject, an average of 83 item per library, while Primo and Summon libraries have the least, an average of 44 and 48 items respectively. This presented a somewhat striking contrast that EDS covers the least displayed of local physical items (see Figure 7), an estimated 19% or equivalent to 4 books, but its libraries have the most physical items, 83 on average. For catalog items, libraries with WorldCat have the most material, an average of 1046 items per library. It should be noted that it is also one of the discovery services that displays the most catalog items, 92% or equivalent to 18 books out of top 20 displayed item in WorldCat, as shown in Figure 7. Figure 9. Pearl Harbor AND attack - Number and Percentage of Physical Items in the Top 20 Discovery Search Per Library Figure 9 illustrates how many items and the percentage of total physical collections made it to the top 20 retrieved items in the discovery services when searching “Pearl Harbor AND attack.” Summon displays the most with 27% or equivalent to 13 items, while it is the least for EDS libraries, with 4.3% of the total physical collections on display among the top 20 retrieved items. Figure 10 shows the total number of catalog items per library, and those that actually made it to the top 20 retrieved items in the discovery search. For the search “Pearl Harbor AND attack,” 5% of the total catalog items per EDS library made to the top 20 search results, 11% for Primo libraries, 22% for Summon libraries, and 17% for libraries using WorldCat respectively. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 15 Figure 10. Pearl Harbor and attack-Number and Percentage of Catalog Items in the Top 20 Discovery Search Appendix Two shows ANOVA and Tukey Test for Pearl Harbor and attack. ANOVA has low p values of 0.0049 and 0.0001 for both physical and catalog items, which are smaller than 0.05 indicating that differences are found among discovery services with this query. For physical items, EDS is statistically different from Summon, but not much different than Primo and WorldCat. No differences are found among Primo, Summon and WorldCat. For catalog items, EDS is statistically different from all other three discovery services. Primo is also statistically different from Summon and WorldCat. Search Statement Three: Crime AND Poverty This query is based on a sociology class assignment at Rider University. Both historical and current materials could be useful for this assignment. Figure 11 shows the search result for “crime AND poverty,” a project for a Rider University sociology class. For EDS libraries, about 1% of the top 20 retrieved items are local physical collections (equivalent to 0.2 book out of the 20 items) and 3.5% are local catalog collections (equivalent to 0.7 book out of the 20 items). As in the other searches, EDS retrieved the least local collections among all the discovery services. WorldCat retrieved the most from local collections, 20% physical items (4 books) and 55.5% catalog items (11 books) out of 20 top retrieved items. Primo and Summon are somewhere in the middle. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 16 Figure 11. Crimes AND poverty-Local Collections in the Top 20 Discovery Service Display Figure 12. Crime AND poverty-Average Number of Physical/Catalog Items in Local Collections Yang, Dawson and Ding / International Journal of Librarianship 4(2) 17 Figure 12 describes how many local items a library covers “crime AND poverty” on average. EDS and WorldCat libraries have the largest local collections on “crime and poverty,” while Primo and Summon libraries are in the middle in their collection size on the subject. Figure 13. Crime and poverty-Number and Percentage of Physical Items in the Top 20 Discovery Search Per Library Figure 13 shows the percentage and number of items that made it to the top 20 results in a discovery service search. For libraries with Primo, Summon, and WorldCat, the percentage of the local catalog collections that made the top display are more or less the same, either 3% or 4%. For EDS libraries, the percentage is very small, about 0.1% which is equivalent to 0.2 book. Figure 14 displays the percentage and number of catalog items that made to the top 20 of discovery service searches retrieved from the local collections. For libraries with Primo, Summon, and WorldCat, the percentage is more or less the same, from 3% to 5%, with WorldCat leading the way. EDS is 0.3%, a very small percentage in comparison to the other discovery services. Appendix Three displays the results of ANOVA and Tukey Test. ANOVA shows low p values of 0.0032 and 0.0003, both of which are smaller than 0.05 indicating statistically significant differences exist among discovery services with this query. For physical items, EDS is statistically different from Primo and Summon, but not much different from WorldCat. No differences are found among Primo, Summon, and WorldCat for physical items. For catalog items, EDS is statistically different from Primo, but not from Summon and WorldCat. Primo is statistically different from Summon and WorldCat. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 18 Figure 14. Crime AND poverty-Number and Percentage of Catalog Items in the Top 20 Discovery Search Consistency in listing local collections within the discovery services and the catalog It is ideal to maintain consistency in search results across all user interfaces including discovery search, the search tab for local collections within a discovery interface, the facet labeled “catalog” or “local library”, and the classic catalog when a user uses the same search. It is difficult sometimes for a discovery service to present a consistent listing within its own interface. Figure 15 shows an example of one search that brings up inconsistent results. The search results across the discovery service, WorldCat in this example, are illustrated. The facet for local catalog/collections within the discovery service, and the local catalog for consistency are compared. It is less confusing for users to see the same library items when they do the same search across different user interfaces. When a library does not have a classic catalog, a comparison is made between the discovery search, and the facet for local library. This chart is an example to demonstrate the inconsistencies in retrieval results when searching “Pearl Harbor” AND attack in a WorldCat and a catalog search within the same library. WorldCat allows one to search for items worldwide or just the home library. The left column represents a search conducted within the library and sorted by best match. There were 8 local items found in this search and these resources are marked in blue. The rest of the 12 items retrieved were non local items such as articles from databases. The middle column represents the results found in searching the home library and sorted by that library. Again, eight items found in the first search is also found in this search. However, four items marked in red are different local items found in this facet search and in the catalog search but not in the discovery layer search. Unique local items were found in the facet search and in the catalog search. The catalog search noted Yang, Dawson and Ding / International Journal of Librarianship 4(2) 19 in the right column, does not have the same number of local items found in the other two columns. Figure 15. Example of Inconsistency of Items in Discovery and Catalog Search Are users seeing the same list of items and in the same order across the discovery service, the facet for catalog, and in classic catalog searches? Ideally, users will see the same local items across all the library search engines. It is confusing for the user to see different listings within a discovery service and then in the catalog. The initial discovery search generally interleaves the local items with resources from databases and electronic collections. Preferably, when a user clicks on the facet of a discovery service, he or she can limit the search to the catalog or local collections only within a discovery service and subsequently should see the same local items in the same order as in the initial search in the discovery service interface except intertwined resources from non-local collections. Figure 16 shows the comparison of such a result. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 20 Figure 16. Consistency of Search Results Figure 17. Consistency of Search Results by Discovery Service In Figure 17 “No” means the item is not consistently found or in the same order. Primo libraries do not have a classic catalog and their searches in discovery service show the same list of local items in the same order as those in the facet “Catalog.” Summon also behaves in the same way. EDS and WorldCat show an inconsistency within the discovery layer. Sometimes the local items that appeared in the initial discovery search will show up when limiting to local collections only and sometimes they do not. When compared with the local items retrieved in discovery services, the items retrieved from a catalog search will display a totally different list. This result conflicts with the original vision for the next generation catalog where there is one user interface and one search for all library resources. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 21 Figure 18. Metadata Indexed by Discovery Services for Searching and Relevancy Ranking Metadata may also affect the discovery tool displays. Full text indexing of journal articles has fuller metadata than MARC records with minimum information. Catalog records are at a disadvantage with less metadata in comparison to full text indexing of journal articles. Figure 18 is a summary of metadata indexed by discovery services found on the website of the vendors. Librarians should be aware of the impact of libraries’ configurations of their discovery tools on displaying local collections. This is something that should be taken into consideration when reading this paper. Circulation of local collection decreased for many reasons: increase of electronic resources and decrease of print sources in library collections, the users’ preference for easy access to electronic resources would also impact circulation statistics. Users may prefer electronic resources and journal articles than print materials (Borrego and Anglada, 2016) which may be another factor to consider. DISCUSSIONS To sum up the findings, ANOVA and Tukey Test have verified that EDS is statistically different from Primo, Summon, and WorldCat in presenting local collections in this study. This revelation is consistent across almost all the three searches for both physical and catalog items. Occasionally there are minor differences between Primo, Summon, and WorldCat, but those are not consistent or conclusive. The authors reached the above conclusions with 95% of confidence. The search tab and facets for local collections in both Primo and Summon retrieve Yang, Dawson and Ding / International Journal of Librarianship 4(2) 22 and display results consistently, presenting the same listing in the same order within the discovery service. WorldCat provides consistency with some exceptions. EDS often displays different results. Most Primo libraries can use Primo both as a catalog and discovery interface and WorldCat libraries are in the process of combining both interfaces. However EDS and Summon libraries still need to maintain a classic catalog. A discovery service often influences or determines what materials a user chooses for their research (Asher et al. 2013; Parry 2014; Wang et al., 2018). Librarians are the gate keepers who make important decisions for faculty and students in their information searches. Librarians report circumstantial evidence that vendors’ resources appear before their own library’s items, and our research confirms this to be true of some of the discovery services examined. The discovery vendors claim their products promote libraries’ own local collections before their own resources. For instance, a few Primo libraries arbitrarily display the first ten items from the local collections in the discovery service retrieval within certain customer defined relevancy rules. The practice is open to debate, but the system allows this function. WorldCat and Summon display a decent amount of local holdings. Among the four web-scale discovery services, EDS consistently displays the least number of local collections in spite of its configuration to optimize local collections. One can argue that EDS libraries may not have so many items on these topics but Figure 4 shows that these libraries had more physical materials on the topic of pesticides than other libraries. In spite of having more items, the EDS libraries’ displayed fewer local titles when searching in the discovery service. The choice of subjects may affect how many local collections will make to the top of a discovery search result within a discovery service. Some subjects are better covered by books, while more current topics are best served by journal articles and recent reports. For instance, a historical topic may have many books in the catalog while a science subject should have more current information in journal articles. The findings of this research reflected the fluctuation of location holdings in the top 20 display of a discovery service as subjects changed from science to history and to sociology. All four discovery services showed an increase in the number of local items when searching “Pearl Harbor AND attack. Regardless of the subject matter, EDS remains low in the display of local holdings in the top 20 items. The amount of resources indexed by a discovery service may also impact the information retrieval and display. The top 20 items or the first page in display is a very limited space for competition. The competing resources all have to find ways to the top 20 in order to be viewed and used by students and faculty. The amount of materials a discovery service indexes may affect the opportunity the local collections appear on the top of the search results. The chance of local items is reduced when too many resources are competing to get to the first page of search results. However, discovery services do not publish the number of resources they index nor do they publish the algorithms developed, as this is proprietary information. This area remains unknown. Regardless those two factors, local collections should be on the top display if promoting library collections is a priority. Yang, Dawson and Ding / International Journal of Librarianship 4(2) 23 Limitations Only three topics were tested and thus the scale of this study is limited. Other subject areas should be tested as the choice of a topic may affect information retrievals. There are many variables that cannot be ascertained or measured that influence the circulation of location collections. The algorithms used by the discovery services are proprietary information and the ranking of the results can’t be discerned. In addition, discovery services continuously improve for the better. These findings may become obsolete over time. Lastly, it is not known if librarians are aware that they can work with these vendors to promote their own local holdings over the vendors’ database materials. CONCLUSION This research points out other factors to consider when libraries decide to license a Web-scaled Discovery Service. There are differences the authors found in how a library’s local collection show up in queries amongst the four major discovery services examined. If a library is concerned about its circulation of its own materials because of budget concerns, then testing these Discovery Services and working with the vendors is critical. One factor that stands out from the others that definitely contributed towards the demise of local borrowing: the increase of electronic resources in libraries and their easy access. According to OCLC, by 2020 academic libraries will spend 80% of their material budgets on electronic materials (Burke, 2012). However, currently the print or physical collections in a library are still significant in spite of the trend for more electronic resources. A 2017 survey of collection development in academic libraries shows that print books are still the predominate part of the collections in academic libraries, 60.3% print books vs. 39.7% electronic books (Enis, 2018). When investigating discovery services in which local collections are only one competing force, the question remains, what is the best recommended practice libraries should follow regarding presenting and pushing local collections to users? What is considered to be the proper, good, or fair representation of local collections in the discovery services? If libraries decide to promote local collections, what benchmarks or guidelines should libraries follow to display local collections in discovery services? How much display of local collection on the top of search results is acceptable and not acceptable? Should librarians ask discovery services to place X number of local items in the top result list regardless of their ranking algorithms? Without such benchmarks and thresholds, discovery vendors only have vague ideas. Is it in the best interest of users to place the local items first before those from databases? This study intends to serve as a starting point for more investigation into local collections in discovery tools. Hopefully it will get librarians to start thinking about some of the conventions that we take for granted. For instance, as libraries acquire more and more electronic resources, is the library catalog as a concept and as a facet in a discovery server becoming obsolete? For instance, users may care to know if an item is in print and physically available in the library. So should the facet in a discovery service be labelled “in the library” instead of catalog? Yang, Dawson and Ding / International Journal of Librarianship 4(2) 24 Another crucial question remains. That is, whether libraries should promote local collections at all and what is the rationale for promoting local collections. It seems the dividing line is blurring between local and remote collections. For instance, everything in WorldCat is labeled local as the location is listed as the local library even for journal articles in databases and e-book collections stored remotely with the vendors. Except for print materials physically housed in the library that can be taken out, users cannot tell what is local or remote and probably do not care. The distinction is a tradition and seems to resonate more with librarians than users. Should libraries promote local collections at all? Is it for selfish purposes or in the best interests for users? What are the best interests of users after all? References Asher, A. D., Duke, L. M., & Wilson S. (2013). Paths of discovery: Comparing the search effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and conventional library resources. College & Research Libraries, 74(5), 464-488. Borrego, A. & Lluis A. (2016). Faculty information behaviour in the electronic environment: attitudes towards searching, publishing and libraries. New Library World, 117(3/4), 173- 185. Retrieved December 9, 2019 from https://www.emeraldinsight.com/doi/pdfplus/10.1108/NLW-11-2015-0089 Burke, J. (2012). Web-scale management solution. [Serial Solutions-Webinar]. Retrieved March 10, 2012 from http://www.serialssolutions.com/en/services/intota Calvert, K. (2015). Maximizing academic library Collections: Measuring changes in use patterns owing to EBSCO Discovery Service. College & Research Libraries, 76 (1), 81- 99. Retrieved from https://doi.org/10.5860/crl.76.1.81 Cohen, R. A. & Thorpe A. (2015). Discovering user behavior: Applying usage statistics to shape frontline services. Serials Librarian, 69 (1), 29-46. Retrieved from https://doi.org/ 10.1080/0361526X.2015.1040194 Copenhaver, K. & Koclanes A. (2016). Impact of Web-scale discovery on reference inquiry. Reference Services Review, 44(3), 266-281. https://doi.org/ 10.1108/RSR-11- 2015-0046 Dempsey, M. & Valenti, A.M. (2016). Student use of keywords and limiters in Web-scale discovery searching. Journal of Academic Librarianship, 42(3), 200-206. https://doi.org/ 10.1016/j.acalib.2016.03.002 EBSCO. (2019). How Is Relevance Ranking Determined in EBSCO Discovery Service (EDS)? Retrieved December 6, 2019 from https://help.ebsco.com/interfaces/EBSCO_Discovery_Service /EDS_FAQs/ https://www.emeraldinsight.com/doi/pdfplus/10.1108/NLW-11-2015-0089 http://www.serialssolutions.com/en/services/intota https://doi.org/10.5860/crl.76.1.81 https://doi.org/ https://doi.org/ https://help.ebsco.com/interfaces/EBSCO_Discovery_Service%20/EDS_FAQs/ Yang, Dawson and Ding / International Journal of Librarianship 4(2) 25 relevance_ranking_determined_in_EDS Enis, M. (2018). E-Resources continue academic gains. Library Journal, 143(6), 16-18. Ex Libris. (2014). Summon: Relevance. Retrieved June 27, 2017 from https://knowledge.exlibrisgroup.com/Summon/Product_Documentation/Searching_in_Th e_Summon_Service/Search_Results/Summon%3A_Relevance_Ranking Ex Libris Knowledge Center Ex Libris. (2017). Relevance ranking in Primo. Retrieved June 27, 2017 from http://www.exlibrisgroup.com/category/Relevance-Ranking Fitzpatrick, S. (2010). Summon summons 100th customer. American Libraries, 41(10): 24- 25. Greiner, T. (2011). How does switching to a discovery tool affect circulation? In Declaration of Interdependence: The Proceedings of the ACRL 2011 Conference in Philadelphia, PA, 2011. Retrieved July 25, 2018 from http://www.ala.org/acrl/sites/ala.org.acrl/files/ content/conferences/confsandpreconfs/national/2011/papers/how_does_switching.pdf Guan, J. & Jia, G. (2016). A comparison of discovery tools in Chinese academic libraries. National Library Journal, 108 (6), 71-79. Hanneke, R. & O'Brien, K.K. (2016). Comparison of three Web-scale discovery services for health sciences research. Journal of the Medical Library Association, 104(2), 109-117. Retrieved from http://dx.doi.org/10.3163/1536-5050.104.2.004 Lawton, A. (2015). Use of ESBCO Discovery Tool at one university reveals increased use of electronic collections but decreased use in circulation of print collections. Evidence Based Library & Information Practice, 10(4), 244-246. https://doi.org/ 10.18438/B8J88H Levine-Clark, M., McDonald J., & Price J.S. (2014). The effect of discovery systems on online journal usage: A Longitudinal study. Insights:The UKSG Journal, 27(3), 249- 56. https://doi.org/ 10.1629/2048-7754.153 Meirose, J. & Lian, B. (2019). User testing: Gathering data from first-year medical students as they interact with the EBSCO Discovery Services (EDS). Journal of Electronic Resources in Medical Libraries, 16(1), 1-7. OCLC. 2017. "How Does Relevance Ranking Work in WorldCat Local?" Accessed June 27, 2017. https://www.oclc.org/support/services/WorldCat-local/faq/search.en.html. O’Hara, L. (2012). Collection usage pre- and post-Summon implementation at the University of Manitoba. Evidence Based Library and Information Practice, 7(4), 25-34. https://knowledge.exlibrisgroup.com/Summon/Product_Documentation/Searching_in_Th http://www.exlibrisgroup.com/category/Relevance-Ranking http://dx.doi.org/10.3163/1536-5050.104.2.004 https://doi.org/ https://www.oclc.org/support/services/worldcat-local/faq/search.en.html Yang, Dawson and Ding / International Journal of Librarianship 4(2) 26 Parry, M. (4/25/2014). As researchers turn to Google, libraries navigate the messy world of discovery tools. The Chronicle of Higher Education, 60, (32), 18. Retrieved June 29, 2017 from http://www.chronicle.com/article/As-Researchers-Turn-to-Google/146081 Power, J.A. (2018). EBSCO Information Services usability study on accessibility. Reference Services Review, 46(3), 449-459. Regier, R. (2015). Relevancy ranking in discovery services. A Way of Happening: Library Research Blog. Retrieved June 28, 2017 from https://awayofhappening.wordpress.com/2015/06/24/relevancy-ranking-in-discovery- services/ Ridga, C., Hoogland, M., & Morales, J. (2018). But I just want a book! Is your discovery layer meeting your users’ needs? Journal of Web Librarianship, 12(4), 246-260. Statistics How To. (2019). Retrieved December 9, 2019 from https://www.statisticshowto.datasciencecentral.com/tukey-test-honest-significant-difference/ Teolis, M. G., Stephenson, P.L., Taylor, M.V., & Poletti, E.J. (2019). Change in information professionals’s satisfaction with discovery services. Journal of Hospital Librarianship, 19(4), 321-329. Walker, S. & Sims, L.L. (2012). “Implementing a Discovery Tool at Two HBCUs.” College & Undergraduate Libraries, 19(2-4), 312-326. https://doi.org/ 10.1080/10691316.2012.693370 Wang, X., Cui, Y., Xu, S. (2018). Evaluating the impact of Web-scale discovery Services on scholarly content seeking. Journal of Academic Librarianship, 44(5), 545-552. Warren, R. M. L. (2017). Usability study identifies vocabulary, facets, and education as Primary Primo Discovery System interface problems. Evidence Based Library & Information Practice, 12(3), 177-179. https://doi.org/ 10.18438/B89M14 Woods, J., Gillespie, E., & McManamon, C. (2016). Discovering discovery: Lessons learnt from a usability study at the University of Liverpool. Insights: The UKSG Journal, 29(3), 258-265. https://doi.org/ 10.1629/uksg.320 http://www.chronicle.com/article/As-Researchers-Turn-to-Google/146081 https://awayofhappening.wordpress.com/2015/06/24/relevancy-ranking-in-discovery-%20services/ https://awayofhappening.wordpress.com/2015/06/24/relevancy-ranking-in-discovery-%20services/ https://www.statisticshowto.datasciencecentral.com/tukey-test-honest-significant-difference/ https://doi.org/ Yang, Dawson and Ding / International Journal of Librarianship 4(2) 27 APPENDIX 1 ANOVA and Tukey Test for Pesticides and Environment Phyiscal Items Yang, Dawson and Ding / International Journal of Librarianship 4(2) 28 Catalog/Total Items Yang, Dawson and Ding / International Journal of Librarianship 4(2) 29 APPENDIX 2 ANOVA and Tukey Test for Pearl Harbor and Attack Physical Items Yang, Dawson and Ding / International Journal of Librarianship 4(2) 30 Catalog/Total Items Yang, Dawson and Ding / International Journal of Librarianship 4(2) 31 APPENDIX 3 ANOVA and Tukey Test for Crime and Poverty Physical Items Yang, Dawson and Ding / International Journal of Librarianship 4(2) 32 Catalog/Total Items Yang, Dawson and Ding / International Journal of Librarianship 4(2) 33 About the authors Dr. Sharon Q. Yang received her doctoral degree in library & information science from Columbia University. Now she works as Professor/Systems Librarian at Rider University. Her research interests include library systems, discovery tools, and the semantic web. Patricia Dawson has been an academic librarian for 20 years involved in reference, science liaison areas, instruction and interlibrary loan services. Her research includes instruction methods and assessment, investigating copyright issues and library repositories, and open access issues. Dr. Jie Ding received her Ph. D. degree in Operations Management from the University of Texas at Austin. She has been teaching statistics and operations management for more than twenty years at Rider University. Her research interests are in the areas of stochastic modeling and designing of quality and manufacturing systems. 2--153-714-1-RV-SharonYang_title 2--153-714-1-RV-SharonYang-1-proofread-no space