Using Qualtrics XM to Create a Point-of-Use Survey to Assess the Usability of a Local Implementation of Primo COMMUNICATION Using Qualtrics XM to Create a Point-of-Use Survey to Assess the Usability of a Local Implementation of Primo Matthew Black, Heather Ganshorn, and Justine Wheeler INFORMATION TECHNOLOGY AND LIBRARIES | DECEMBER 2023 https://doi.org/10.5860/ital.v42i4.16475 About the Authors Matthew Black (corresponding author: mblack@ucalgary.ca) is the Director, Systems and Discovery, Libraries & Cultural Resources, University of Calgary. Heather Ganshorn (hganshor@ucalgary.ca) is a science subject librarian, Libraries and Cultural Resources, University of Calgary. Justine Wheeler (jwheeler@ucalgary.ca) is the Assessment Librarian, Libraries and Cultural Resources, University of Calgary. © 2023. Submitted: May 29, 2023. Accepted for Publication: September 12, 2023. Published 18 December 2023. ABSTRACT In 2020, Libraries and Cultural Resources (LCR) at the University of Calgary used Qualtrics XM to design and pilot a point-of-use survey to collect user feedback on the usability of our implementation of Primo, Ex Libris's web-scale discovery service. Over a two-week period, users were presented with the pop-up survey while searching and asked to provide feedback. This article summarizes how we designed and implemented this point-of-use survey and the lessons learned from this project. INTRODUCTION In 2018, Libraries and Cultural Resources (LCR) at the University of Calgary implemented Ex Libris’s Primo, a web-scale discovery service.1 Through an embedded search box on the library webpage (https://www.library.ucaglary.ca), users can use Primo to discover and access resources from LCR’s physical and digital collections. After adopting Primo, LCR wanted to assess usability of the interface and the initial decisions made around user interface display customizations. To do this, in early 2020 LCR’s Primo Working Group piloted a point-of-use intercept survey using Qualtrics XM to collect feedback from users on the success of their search experience. For this, we defined success as users’ perception that they found what they were looking for. As we were unable to find comprehensive guidance on how to set up such a survey, this article will share our process and lessons learned in the hope that others will find it useful. PRIMO AND USABILITY Ex Libris designed Primo “to catch up with user expectations” through implementing “contemporary user experience elements.”2 Since the initial release, many academic libraries have conducted user experience studies on Primo that involved recruiting users and asking them to complete specific common tasks while under observation. Study designers then analyze the results for insights into user experience.3 These studies capture feedback from a sample of users in the context of a usability test environment, which is to some extent artificial. In our study, we aimed to capture authentic user feedback as they searched using Primo, not as they completed predetermined tasks. In the marketing profession, this is known as an intercept survey, an online survey that is triggered during the use of a site or application and can be used to study “natural use of the product.”4 In the library context, Martha Kyrillidou, Terry Plum, and Bruce Thompson used this method, which they referred to as point-of-use surveys, to collect user feedback on networked electronic resources. Importantly, they contended that point-of-use surveys can improve the validity and mailto:mblack@ucalgary.ca mailto:hganshor@ucalgary.ca mailto:jwheeler@ucalgary.ca https://www.library.ucaglary.ca/ INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 2 BLACK, GANSHORN, AND WHEELER response rates of a survey because the survey does not require users to report on “predicted, intended, or remembered use” (which can introduce error) and that proactive interception increases the number of responses and decreases the potential for bias due to nonresponse rates.5 Further, Jane Nichols, Richard Stoddart, and Terry Reese explained how Oregon State University Libraries and Press designed and implemented an in-house intercept survey for collection assessment and noted the in-house method required significant time and support from their developers and that they would investigate whether Qualtrics could support the intercept functionality.6 Using Qualtrics XM to Deliver a Survey In early 2020, LCR formed a Primo Working Group, which includes representative from library systems, research and learning services, public services, and collections. The group meets monthly to review Primo developments and work on improving our search experience. After our initial meetings, the group began to discuss how we collect user feedback on our discovery search interface. A subgroup was formed to come up with a strategy for this assessment. In the subgroup’s initial meeting, we decided to collect user feedback on search experience by asking open-ended questions about how successful users perceived they were with their search and to ask them to identify the elements of the interface that helped or hinder their success. For this, we decided to use Qualtrics XM because it is the University of Calgary’s licensed survey tool and we determined it could be used to deliver the survey as a pop-up. Using Qualtrics XM, we first developed a test survey to understand how to set up the survey flow and how to create the pop-up to display to users. After setting this up, we shared a test version with the Primo Working Group for feedback on the design and testing in our Primo sandbox. Based on this feedback, we finalized a short survey that asked users if they found the resources they were looking for and included a follow-up question about why they feel their search was successful or unsuccessful (see fig. 1). The questions were: 1. What user group do you belong to? If you belong to more than one group, choose the group that had brought you to the Library today: • Academic Staff • Support Staff and Management & Professional Staff • Undergraduate Student • Graduate Student • Continuing Education Student • Community User • Other 2. Did you find what you needed? • Yes (conditionally linked to question 3a) • No (conditionally linked to question 3b) • Somewhat (conditionally linked to question 3a and 3b) 3. • 3a. What helped you find what you needed? • 3b. Why do you think your search was not successful? INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 3 BLACK, GANSHORN, AND WHEELER Figure 1. Finalized survey questions and design in Qualtrics XM. After finalizing the survey, we configured the pop-up. Our plan was to run the survey for a two- week period. During this two-week window when the survey was active, users would be presented with a pop-up during their Primo search session asking if they would like to provide feedback. To trigger the display of the survey in a pop-up, we used Qualtrics XM’s built-in functionality to create a point-of-use intercept, a Website & App Feedback project (see fig. 2). Figure 2. Create a Website & App Feedback project screen. After creating the project, we needed to configure the Intercept and Creative. The Intercept is used to define the conditions for when a user on a site or app is presented with the survey. The Creative is the method for delivering the survey to the users once intercepted on the site. For this project, we decided to use the Responsive Dialog Creative to connect users to our survey (see fig. 3). INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 4 BLACK, GANSHORN, AND WHEELER Figure 3. Feedback collection method selection screen. Within the Creative, we were able to configure and preview the size, style, and text for the dialog and buttons; add images, such as a logo; and set the display animation (see fig. 4). Figure 4. Responsive Dialog Creative configuration screen. We configured the dialog to allow users to accept or decline our request to take the survey and branded it with our logo and colors (see fig. 5). INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 5 BLACK, GANSHORN, AND WHEELER Figure 5. Responsive Dialog Creative with customized message, buttons, colors, and logo. Once we configured the Creative, we needed to set up the Intercept. We set the survey as the target for the Intercept and defined the Targeting logic and Frequency. For the Targeting logic, we used the “If Current URL Starts With” option and used the base URL for our Primo instance (see fig. 6). Figure 6. Intercept Targeting Logic configuration. For the frequency, we configured it to intercept users only when their mouse left the web page because we did not want the survey to intercept users before they had tried searching. This was our best approximation for ensuring the intercept timing was appropriate for the questions we were asking. We determined the other available options would display the survey too early (on load or on focus) and not have achieved a valid timing for the questions. There is also an option to use custom JavaScript code to set the timing, but we decided to just use the out-of-the-box options because we wanted to keep the configuration simple (see fig. 7). INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 6 BLACK, GANSHORN, AND WHEELER Figure 7. Intercept Frequency configuration. To avoid frustrating users by displaying the survey too frequently, we used the “Repeated display prevention for browser cookie” option. Within this configuration, we set the survey to display to 100% of the users who qualify but prevented repeated display for one day. This meant users would not be asked to take the survey more than once a day. To add the intercept to Primo, we copied the JavaScript code snippet that Qualtrics XM generates and added this to the custom.js file in our Primo customization package (see fig. 8). Figure 8. JavaScript code snippet for the Responsive Dialog. Once this code is deployed in Primo, the intercept behavior can be controlled within Qualtrics XM by simply activating or deactivating it within the Website/App Feedback project (see fig 9). Figure 9. Configuration page with the Activation toggle highlighted. INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 7 BLACK, GANSHORN, AND WHEELER This was convenient because we could leave the code in the Primo custom.js file and deactivate or activate in Qualtrics XM as necessary. In addition, when we later updated the target of the intercept to an updated version of the survey we did not have to update the code in Primo. LESSONS LEARNED Proactively intercepting users at point of use can result in a significant number of responses over a short period of time. As mentioned above, we initially ran the survey with the intercept for a two-week period and again, for the same period of time, during two additional semesters. Between these periods, we left the survey active as a passive sidebar that users could choose to access. This passive method collected fewer responses (107) over a six-month period than the point-of-use survey (755) collected over the combined six weeks that the pop-up prompt was active. For those who are interested in running a point-of-use survey, we summarize the lessons we learned below. When Designing a Survey Consider the Following • Keep the survey short and focused on a specific goal. • Assess the survey questions after piloting and iterate—adjust or clarify questions if needed. When Designing the Point-of-Use Intercept Consider the Following • Make sure the point-of-use intercept will not frustrate users by controlling how frequently the survey is presented. • Make sure the timing of the pop-up dialog aligns with your goal(s) by considering: o When do your primary stakeholders most heavily use library resources? o What is the minimum number of responses you aim to collect? o How long do you need to run the survey to achieve this number? o At what point in the users’ search will the survey intercept users to ask for feedback? Will this timing be at the point of use you want feedback on? Provide Users with Options for Timely Support We found that users sometimes confused the feedback survey with an opportunity to report issues or ask for help. Accordingly, at the end of the survey we added a closing message that provided users with options to get immediate help using chat or our FAQs (see fig. 10). Figure 10. Survey closing message. INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 8 BLACK, GANSHORN, AND WHEELER CONCLUSION For academic libraries with access via an institutional license, Qualtrics XM is a flexible and relatively simple tool that can be used to gather point-of-use feedback on user experiences with library discovery services such as Primo. In our case, we analyzed the results for themes and shared these with our Primo Working Group. As a result, the group reviewed the results and came up with actionable items such as updating our display labels for facets, resource availability statements, and access and licensing. This method may also be applicable for collecting feedback on other elements of a library website, vendor platforms, or online collections. ENDNOTES 1 Athena Hoeppner, “The Ins and Outs of Evaluating Web-Scale Discovery Services,” Computers in Libraries, no. 3 (April 2012), https://www.infotoday.com/cilmag/apr12/Hoeppner-Web- Scale-Discovery-Services.shtml. 2 Tamar Sadeh, “User Experience in the Library: A Case Study,” New Library World 109, no. 1/2 (January 11, 2008): 7–24, https://doi.org/10.1108/03074800810845976. 3 Annis Lee Adams and Margot Hanson, “Primo on the Go: A Usability Study of the Primo Mobile Interface,” Journal of Web Librarianship 14, no. 1–2 (April 2, 2020): 1–27, https://doi.org/10.1080/19322909.2020.1784820; Kelsey Renee Brett, Ashley Lierman, and Cherie Turner, “Lessons Learned: A Primo Usability Study,” Information Technology and Libraries 35, no. 1 (April 1, 2016): 7–25, https://doi.org/10.6017/ital.v35i1.8965; David J. Comeaux, “Usability Testing of a Web-Scale Discovery System at an Academic Library,” College & Undergraduate Libraries 19, no. 2–4 (April 2012): 189–206, https://doi.org/10.1080/10691316.2012.695671; Sarah Dahlen, Kenny Garcia, and Kathlene Hanson, “Comparing Apples and Bananas? A/B Testing for Discovery System Optimization,” Library Faculty Publications and Presentations, January 1, 2018, https://digitalcommons.csumb.edu/lib_fac/9; Blake Lee Galbreath, Corey Johnson, and Erin Hvizdak, “Primo New User Interface: Usability Testing and Local Customizations Implemented in Response,” Information Technology and Libraries 37, no. 2 (June 18, 2018): 10–33, https://doi.org/10.6017/ital.v37i2.10191; Scott Hanrath and Miloche Kottman, “Use and Usability of a Discovery Tool in an Academic Library,” Journal of Web Librarianship 9, no. 1 (January 2, 2015): 1–21, https://doi.org/10.1080/19322909.2014.983259; W. Jacobs, Mike DeMars, and J. M. Kimmitt, “A Multi-Campus Usability Testing Study of the New Primo Interface,” College & Undergraduate Libraries 27, no. 1 (January 2, 2020): 1–16, https://doi.org/10.1080/10691316.2019.1695161; Kylie Jarrett, “Findit@Flinders: User Experiences of the Primo Discovery Search Solution,” Australian Academic & Research Libraries 43, no. 4 (December 2012): 278–99; Greta Kliewer et al., “Using Primo for Undergraduate Research: A Usability Study,” Library Hi Tech 34, no. 4 (January 1, 2016): 566–84, https://doi.org/10.1108/LHT-05-2016-0052; Aaron Nichols et al., “Kicking the Tires: A Usability Study of the Primo Discovery Tool,” Journal of Web Librarianship 8, no. 2 (April 3, 2014): 172–95, https://doi.org/10.1080/19322909.2014.903133; Xi Niu, Tao Zhang, and Hsin-liang Chen, “Study of User Search Activities with Two Discovery Tools at an Academic Library,” International Journal of Human-Computer Interaction 30, no. 5 (May 4, 2014): 422–33, https://doi.org/10.1080/10447318.2013.873281; Joy Marie Perrin et al., “Usability Testing for Greater Impact: A Primo Case Study,” Information Technology and Libraries 33, no. 4 (December 18, 2014): 57–66, https://doi.org/10.6017/ital.v33i4.5174; Lynne Porat and Nir Zinger, “Primo New User Interface—Not Just for Undergrads: A Usability Study,” Weave: https://www.infotoday.com/cilmag/apr12/Hoeppner-Web-Scale-Discovery-Services.shtml https://www.infotoday.com/cilmag/apr12/Hoeppner-Web-Scale-Discovery-Services.shtml https://doi.org/10.1108/03074800810845976 https://doi.org/10.1080/19322909.2020.1784820 https://doi.org/10.6017/ital.v35i1.8965 https://doi.org/10.1080/10691316.2012.695671 https://digitalcommons.csumb.edu/lib_fac/9 https://doi.org/10.6017/ital.v37i2.10191 https://doi.org/10.1080/19322909.2014.983259 https://doi.org/10.1080/10691316.2019.1695161 https://doi.org/10.1108/LHT-05-2016-0052 https://doi.org/10.1080/19322909.2014.903133 https://doi.org/10.1080/10447318.2013.873281 https://doi.org/10.6017/ital.v33i4.5174 INFORMATION TECHNOLOGY AND LIBRARIES DECEMBER 2023 USING QUALTRICS XM TO CREATE A “POINT-OF-USE” SURVEY 9 BLACK, GANSHORN, AND WHEELER Journal of Library User Experience 1, no. 9 (2018), https://doi.org/10.3998/weave.12535642.0001.904; Barbara Valentine and Beth West, “Improving Primo Usability and Teachability with Help from the Users,” Journal of Web Librarianship 10, no. 3 (July 2, 2016): 176–96, https://doi.org/10.1080/19322909.2016.1190678. 4 Christian Rohrer, “When to Use Which User-Experience Research Methods,” Nielsen Norman Group, July 17, 2022, https://www.nngroup.com/articles/which-ux-research-methods/. 5 Martha Kyrillidou, Terry Plum, and Bruce Thompson, “Evaluating Usage and Impact of Networked Electronic Resources through Point-of-Use Surveys: A MINES for LibrariesTM Study,” The Serials Librarian 59, no. 2 (July 30, 2010): 485, https://doi.org/10.1080/03615261003674057. 6 Jane Nichols, Richard Stoddart, and Terry Reese, “Nuanced and Timely: Capturing Collections Feedback at Point of Use,” in Too Much Is Not Enough! (Charleston Conference, Against the Grain, 2014), 299, https://doi.org/10.5703/1288284315275. https://doi.org/10.3998/weave.12535642.0001.904 https://doi.org/10.1080/19322909.2016.1190678 https://www.nngroup.com/articles/which-ux-research-methods/ https://doi.org/10.1080/03615261003674057 https://doi.org/10.5703/1288284315275 Abstract Introduction Primo and Usability Using Qualtrics XM to Deliver a Survey Lessons Learned When Designing a Survey Consider the Following When Designing the Point-of-Use Intercept Consider the Following Provide Users with Options for Timely Support Conclusion Endnotes