sept13_b.indd September 2013 417 C&RL News Since 2001, the Harold B. Lee Library (HBLL) at Brigham Young University (BYU) has participated with other libraries, both within and outside of the United States, in the Association of Research Libraries’ (ARL) survey of service quality, LibQUAL+. Analysis of received data is provided by LibQUAL+ team, however, the open-ended comments are not interpreted. While the statistical analysis makes important data available to inform our improvement efforts, the com- ments section often supplies even more useful information. Previously, our library statistician coded survey comments into basic categories. This longitudinal view of the data helps the library understand where progress occurs and where problems persist. Library administrators use the data in their deliberations, and findings are reported in a meeting of the library as a whole. While planning for the fall 2011 survey, HBLL’s assessment unit decided to implement a project to code comments involving library employees from all departments. The coding project undertaken in 2012 provided another perspective on the data and proved to be beneficial, both personally and collectively, to library employees. Working in teams to read and assess the comments allowed them to see and discuss patrons’ concerns. Ultimately, the project had a positive influence on the library as a whole and is still affecting our services today. Library employee involvement The impetus for the LibQUAL+ coding proj- ect was a poster session at the 2010 Library Assessment Conference. Daniel O’Mahony reported on a project he led at Brown Uni- versity using library employees (principally subject librarians and public services librar- ians) to code their LibQUAL+ comments.1 Our assessment unit was interested in learning whether Brown’s process would be replicable. We modeled our project on Brown’s, with modifications based on local circumstances. This idea of using library employees to code LibQUAL+ comments is not unique to BYU or Brown University, however few libraries have used a large group of library employees (more than one- to-four individuals) to do the coding. The library opted to use LibQUAL+ LITE in 2011, and the sample size of respondents increased by 50 percent. More surveys were completed and about one-third more com- ments were received than in previous years. With 942 comments to code, enlisting the help of library employees (as O’Mahoney did) seemed to be a practical way to discover themes and issues that were present. The primary goal of the project was to disseminate the results widely throughout the library so that all employees would gain a better understanding of student percep- tions of library services and resources. A secondary goal was to take advantage of library employees’ creative and innovative ideas for solutions to identified areas of weakness. Finally, this methodology spread the work of coding throughout the library. Allyson Washburn LibQUAL coding cohorts The benefits of involving library employees in LibQUAL+ analysis Allyson Washburn is assessment librarian at Brigham Young University, e-mail: allyson_washburn@byu.edu © 2013 Allyson Washburn C&RL News September 2013 418 The team coding process During the administration of the survey, we began discussing the logistics of coding the comments and the requisite training needed for a successful project. Brown University shared their taxonomy with us, and as we read all 942 comments, it became apparent that with minor editing or substitutions most of Brown’s taxonomy terms were relevant for use in coding our comments. Prior to parsing out the survey according to the self-identified majors of respondents, BYU colleges had to be harmonized with the LibQUAL+ disciplines. Between 75 and 150 comments were then loaded into Excel spreadsheets by discipline. Separate spreadsheets were also constructed con- taining all comments that mentioned online content or the Web page. Although com- ments were separated by the respondents major, the comments reviewed by each group reflected the respondents overall library experiences and not just aspects of their subject area. Spreadsheets were arranged with the four LibQUAL+ dimen- sions (Library as Place, Affect of Service, Information Control, and General) across the top of the sheet with the codes from the taxonomy in columns under each di- mension (see image 1). Coding the comments was a three-step process. First, after reading a comment on the spreadsheet, coders decided the LibQUAL+ category that best fit the com- ment. Second, they consulted the appropri- ate section of the taxon- omy, looked through the codes, read t h e e x p l a - nation, and e x a m i n e d s o m e k e y - w o r d s t h a t m i g h t a p - p e a r i n a c o m m e n t w i t h t h a t c o d e ( s e e image 2). B e f o r e going on to s t e p t h r e e , a n y d i f f e r- e n c e s o f o p i n i o n about what the assigned code should be were dis- cussed and resolved. Third, when team members agreed on the code, they then decided whether the comment was posi- tive, negative, or neutral. Returning to the spreadsheet, the coders entered a P, N, or I in the appropriate code column next to the comment. Because we used an Excel spreadsheet, we had to use a different letter (I) for neutral in order to distinguish be- tween the negative and neutral comments. Teams of library faculty and staff coders were recruited via the library’s e-mail elec- tronic list and personal visits to department chairs and individuals. Eleven teams of three members each and one team of two Image 1. Excel spreadsheet used for coding. View this article online for detailed image. September 2013 419 C&RL News members were organized. The coders were placed on teams based on their discipline and, ultimately, were assigned a sheet of comments from respondents in that disci- pline (with some exceptions). Volunteers were trained during a meeting where they were introduced to their group, learned about the taxonomy, and practiced coding comments. Time was allotted for e a c h t e a m t o practice coding t e n c o m m e n t s f r o m a p r e v i - ous LibQUAL+ survey. Coders were instructed that some com- ments contained i n f o r m a t i o n that could and should be coded in more than one o f t h e d i m e n - s i o n s . A s s e s s - ment unit mem- bers circulated among the teams to answer ques- tions and clarify issues during the training. A t t h e c o n - c l u s i o n o f t h e coding exercise, each team was asked to report their codes for one comment on the spreadsheet. This debriefing was used to help assess and clarify coding misunder- standings before teams worked individually on the 2011 data. Teams received their spreadsheets the day following the training so they could begin coding while the process was still clear in their minds. Each team determined times and places for their work sessions and had a deadline by which their cod- ing needed to be completed. Coders were asked to resolve all differences of opinion that arose using the taxonomy provided. Discussion was encouraged to help team members come to a consensus, but if that was impossible, the majority ruled. Focus group assessment When all groups finished coding, we orga- nized focus groups in order to assess the coders’ experience and to elicit changes and improvements for future LibQUAL+ coding projects. We wanted data about the training, the process, the team ap- proach, and the individual experience. We also wanted suggestions for disseminating the information to the library as a whole. Finally, we hoped to capture suggestions that did not surface in previous discussions for improving the process for the project. Thirty of the 35 coders were able to attend a focus group. Questions were con- structed to guide the discussion and four focus groups were held. For the most part, participants were engaged in the discussion and shared their ideas freely. Recordings Image 2. Taxonomy used in the LibQUAL+ coding project. C&RL News September 2013 420 were transcribed and reviewed by assess- ment personnel. Spreadsheet and taxonomy issues were discussed. However, of particular interest to us were the following themes: Coders liked working in teams. Nearly three-quarters (73%) of participants who commented on working in teams found it a positive experience. They felt it helped them to see perspectives different from their own. One participant described her experi- ence: “People would point out different things. You’d get different perspectives and viewpoints between the three of us. We always ended up agreeing, but being able to discuss it through made it a lot better.” The coders also mentioned that being part of a team helped them meet the deadline for completion because they felt accountable to their team to participate and get the coding done. Working in a team was also helpful for newer employees who had limited or no knowledge of certain areas or services in the library. (These were mainly coders from the library’s IT department who had been employed at the library less than a year.) One coder said, “It helped me as a new employee . . . [to] better understand differ- ent pieces of the library and how they fit together in some of the user perspectives.” Coders also liked working in teams that included members from different parts of the library. They became interested in what other departments were doing. Only one individual found it a negative experience and another saw a disadvantage in the time required for a group to code the comments versus doing it individually. The remaining four had neutral feelings about it. Participants felt that they benefitted personally from coding the comments. A little over half (54%) of the individuals who commented on their experience said that they felt that they personally benefit- ted from the coding experience. Many of these individuals cited specific examples where their perspective changed. A library IT employee commented that “It seems that everyone hates the Web site,” however, he indicated that through the coding process he gathered ideas of how the site could be made less confusing. Several coders remarked that the library had most things the survey respondents thought we were lacking, highlighting the need to discover better methods of informing patrons about the services the library offers. Other coders found that the comments had more helpful information than they had originally thought. Several mentioned that participating in the coding helped them be more effective in helping students. The fol- lowing illustrates a particularly enlightening example for one participant in a group of coders: We were going through our [com- ments]one day, and one of the stu- dents said, ‘I really dislike that I have to go find the books on the shelves’. . . . We were all like, ‘Woah, what do you want? Robots to go get them?’ And . . . we were laughing at it. The very next day, I had a student come up and I was talking to her and…I said, ‘Do you know how to find the call numbers,’ and she said ‘No, my… high school librarian, always got the books for me.’ And then it clicked. . . . That’s what that other student’s com- ment was. They probably don’t know how to find the books. The individuals who did not indicate a sense of personal benefit frequently noted their frustration when working on the proj- ect because they wanted to help or inform patrons who are unaware of services that are already being offered. Most said they would participate in a cod- ing project again. Of those who answered this question, 79 percent responded in the affirmative. These individuals commented that it was beneficial to go through the com- ments and personally see how interactions with library users could be improved. Several coders stated that it was an “eye-opener.” (continues on page 425) September 2013 425 C&RL News erally Funded Scientific Research,” February 22, 2013, accessed June 25, 2013, www.white- house.gov/sites/default/files/microsites/ostp /ostp_public_access_memo_2013.pdf. 16. Barack Obama, “Executive Order— Making Open and Machine Readable the New Default for Government Information,” May 9, 2013, accessed June 25, 2013, www. whitehouse.gov/the-press-office/2013/05/09 /executive-order-making-open-and-machine -readable-new-default-government-. 17. Sylvia Burwell, Steven VanRoekel, Todd Park, and Dominic Mancini, “Memorandum for the Heads of Executive Departments and Agen- cies: Open Data Policy —Managing Information as an Asset,” May 9, 2013, accessed June 25, 2013, www.whitehouse.gov/sites/default/files/omb /memoranda/2013/m-13-13.pdf. 18. Obama, “Making Open and Machine Readable.” 19. Andi Sporkin, “Understanding CHO- RUS,” Association of American Publishers, June 5, 2013, accessed June 25, 2013, http:// publishers.org/press/107/. 20. Prue Adler, Judy Ruttenberg, and Julia Blixrud, “Shared Access Research Ecosystem (SHARE) proposed by ARL, AAU, APLU,” ARL News, June 7, 2013, accessed June 25, 2013, www.arl.org/news/arl-news/2773-shared -access-research-ecosystem-proposed-by -aau-aplu-arl. 21. Ilene Mizrachi, “GenBank: The Nucleo- tide Sequence Database: History,” The NCBI Handbook, McEntyre J, Ostell J, ed. created: October 9, 2002; last update: August 22, 2007, accessed June 25, 2013. www.ncbi.nlm.nih. gov/books/NBK21105/#ch1.History. 22. Kathy Cravedi, “GenBank Celebrates 25 Years of Service with Two-Day Confer- ence; Leading Scientists Will Discuss the DNA Database at April 7-8 Meeting,” NIH News, April 3, 2008, accessed June 25, 2013, www.nih.gov/news/health/apr2008/nlm-03. htm. Others liked the aspect of getting to know library employees from different parts of the library and are now engaged in networking and talking about library issues with them. One coder indicated that it was “really energizing” and all coders enjoyed seeing the positive comments as well. Two coders who said they would not participate again felt that trained students could do the coding or said they would only do it if they felt obligated. One mentioned, “I’ll do it to be a team player, but I would rather not.” Two were neutral about future participation. Conclusion The focus group comments helped us to see where we could improve the process the next time around. Conversations with the library’s Administrative Council helped us determine the methods of sharing the in- formation library-wide. Plans were made to disseminate the findings as widely as possible throughout the library and provide discussion venues for all employees. The comments in their entirety were posted on the library’s internal wiki, along with comments arranged by the discipline of the survey participant. Additionally, a session during the fall li- brary retreat was devoted to working with the comments in groups. The group discussion reaped even more ideas from library person- nel in regards to addressing the concerns of library patrons. Even though we have done a lot of work with the comments, the process is not complete. We will hold a discussion in a library-wide meeting to prioritize problems needing solutions. Thus far the project has served as a unique learning experience for the library and has proved to be of great benefit to all library employees involved. Note 1. Daniel O’Mahony, “Sharing the Wealth: A Process for Engaging a Large Group in Cod- ing LibQUAL+® Survey Comments,” poster presented at the 2010 Library Assessment Conference, Baltimore, Maryland, October 24–27, 2010. (“LibQUAL coding cohorts,” cont. from page 420)