C&RL News June 2021 254 In fall 2018, members of the Teaching and Learning Team at the George Mason University Libraries proposed a focus group study to assess one of the libraries’ student learning outcomes. The outcome, “Students will recognize that the library’s value exceeds traditional information sources in order to transfer their knowledge to experiences beyond the classroom,” proved difficult to assess, but we wanted to produce some data to attempt to measure it. We hoped a focus group study of graduating seniors might provide us with evidence for this outcome. With the support of library administration, we began a two-year research process. We experienced ebbs and flows in our progress as we maintained our regular teaching loads during the semesters, and we experienced a bit of a defeat when we realized we didn’t have sufficient findings for a scholarly article. We emerged from the project with small victories and lessons learned. We’d like to share them with you in the hopes that they will inform your future research projects. Scope When conducting a focus group study, especially on information literacy, it’s important to carefully consider if your question(s) can be answered by stu- dents. Our assessment initially sought to measure our information literacy program’s sixth student learning outcome, which refers to student knowl- edge transfer. Many of our other student learning outcomes were easily measured during library in- struction sessions. Outcome six, however, proved notoriously difficult for our library instructors to measure since it encompasses students’ experiences and thoughts outside of the classroom, particularly after their college careers have ended. Initially, we hoped that the focus group assess- ment would provide us a chance to assess outcome six in a way we were unable to in the classroom. However, after trying to determine a plan for our assessment, we encountered a few difficulties. First, we realized it would be difficult for full-time stu- dents to recognize or articulate answers to questions about post-graduation career choices. Additionally, sometimes students only recognize the extent of the library’s value once they’ve left a particular session/ course or once they graduate. When conducting a similar project, we recommend sticking with mea- surable outcomes and questions that can be easily traced across students’ college careers. Once we realized that outcome six was more of an instructional goal rather than an outcome, our study became more exploratory. We shifted the focus of our questions to cover library services such as space, reference, events, and online services. Unsurprisingly, our broad questions received wide- ranging responses. These responses have been valu- able in certain ways. For example, we collected data pertaining to multiple library departments, which Megan Reichelt, Kayla Gourlay, Valerie Linsinbigler, Ashley Blinstrub, and Maoria J. Kirker Focus groups as an assessment strategy Lessons learned Megan Reichelt is theater and dance librarian, email: mreichel@gmu.edu, Kayla Gourlay is teaching and learning specialist, email: kmorrow7@gmu.edu, Valerie Linsinbigler is online learning specialist, email: vlinsinb@ gmu.edu, Ashley Blinstrub is student success and inclusion librarian, email: ablinstr@gmu.edu, and Maoria J. Kirker is lead, Teaching and Learning Team, email: mkirker@gmu.edu, all at George Mason University © 2021 Megan Reichelt, Kayla Gourlay, Valerie Linsinbigler, Ashley Blinstrub, and Maoria J. Kirker mailto:mreichel%40gmu.edu?subject= mailto:kmorrow7%40gmu.edu?subject= mailto:vlinsinb%40gmu.edu?subject= mailto:vlinsinb%40gmu.edu?subject= mailto:mkirker%40gmu.edu?subject= June 2021 C&RL News255 is beneficial for our colleagues and administration. It’s also helpful to have a broad overview of student perceptions about our services. However, our data is nonspecific and requires further exploration in many areas. As such, it is potentially the starting point for six distinct research projects. So, while we were too specific in our aim to assess outcome six, we were also unrealistic in our hopes to conduct an extensive assessment of all library services. When conducting a similar project, we suggest focuses on one or two research questions. Logistics and communication Our marketing efforts for the focus groups were fairly successful. We used multiple avenues of communication: posting flyers around campus, leveraging relationships with academic units, and reaching out to the registrar’s office for a list of reg- istered seniors. Due to the timing of the request, which preceded the add/drop deadline, not all seniors had registered for the Spring 2019 semes- ter, and we likely missed a particular demographic who waited to register. While we selected this time based on our project schedule, we should have considered the academic calendar. Regardless, we still received a sizable response of 235 interested students. We also did not incorporate a system for di- versifying the participants. We accepted them on a first-come, first-served basis, assuming this would both randomize and diversify the pool. However, this was not the case. Randomization does not guarantee diversity. If randomization was the goal, we could have used a number generator. If diversity was the goal, we could have gathered demographic data as a part of the sign-up system. For example, the majority of our participants were female and many of our male participants either did not RSVP, RSVP’d too late, or did not show up. Depending on the goal of the focus group, it is important to include systems for either randomization or diversification from the beginning. We offered two incentives for the students, the first of which was referenced by almost all of our participants as the reason they decided to attend: a $25 gift card to their choice of Panera Bread, Star- bucks, Chipotle, or Barnes & Noble—all campus vendors. We also advertised that we would provide coffee and snacks during the focus group as a way of helping students feel comfortable and valued. While students’ altruism and desire to help the libraries were contributing factors, extrinsic motivations played a key role in their participation. We intended for the students to receive the gift cards at the end of their focus group session, however, the funding process was delayed, so we asked the students to complete a contact form for when the gift cards became available. All participants received their incentives, but we learned it was im- portant to clarify timelines for financial procedures well in advance to avoid administrative holdups. We held four focus groups at the same time of day over four days. Upon arriving, students signed in and received a pre-survey, which gathered demographic data and quantitative data regarding previous experiences with our two main campus libraries. They also received and signed a consent form. At the beginning of each focus group, a staff member read a formal and scripted introduction to the students detailing the structure of the focus groups and the policies outlined by the informed consent form. Students then introduced themselves by stating their names, majors, and reason for at- tending the focus group. We then proceeded with the questions outlined in the focus group protocol. As the conversation flowed, we encouraged students to adjust their pre-survey answers with different colored ink, if they remembered any new informa- tion based on the discussion. While few students made changes in our study, this could be a useful assessment technique in future studies. There was some confusion among the staff about the length of the focus groups. They were advertised as one hour, but the room was reserved for two, which led to a more leisurely and lengthy discussion on the first day. Because of this, we col- lected more data that day. The following days were tightly moderated to keep time. In the future, clear expectations of the structure of the day should be communicated to all moderators to ensure one group does not dominate the data. Between three and four library staff attended each focus group session, giving us multiple perspec- tives on which to draw. One person led the discus- sion, while the others took notes and occasionally asked follow-up questions. We recorded the video of the focus groups on a laptop, while one or two phones recorded audio. The multiple methods C&RL News June 2021 256 of data collection were essential in transcription. Where one method failed, we could use another. Data analysis Our process for analyzing the transcripts from the focus groups changed multiple times. Creating the transcripts took a significant amount of time be- cause the audio quality from the focus groups was inconsistent. Even with both phone and video re- cordings, the audio volume captured was low, and people talked over one another. This also made it difficult to determine who was speaking in the re- cording. We highly recommend investing in mul- tiple high-quality recording devices to ensure ease of transcription. We used a transcription pedal that made pausing the recordings easier for transcrib- ing, but we only had a short-term free trial for the software, so it became unusable. We planned on partnering with another on-campus organization to have undergraduate research assistants for this process but were unable to secure help. These issues led to the transcription process taking much longer than we had anticipated, especially since this work was often deprioritized under more pressing tasks. Once we completed the transcripts, we used Microsoft Word to perform a round of open coding, using the comments tool to record codes. After we completed the first round of coding, we created a controlled list of themes with definitions. During this process there was also a discussion about the areas of research that remained important to the research project. When we entered our second round of coding, it became clear that the definitions of the combined codes were unclear to some. This led to misunderstanding of the codes and more time spent discussing how to categorize certain quotes. We also found that Microsoft Word was not the best tool for this project, as we had to keep track of multiple different versions of the documents, and we did not know an efficient way to extract similar codes for comparison. We needed a better way to track and organize the codes and quotes. We transferred the documentation from Microsoft Word to Taguette, an open source, web-based quali- tative analysis tool, which is ideal for collaborative projects. This switch required us to re-tag the codes. Taguette made the coding and analysis of our data much easier, as we could easily extract all quotes coded with the same theme. We recommend that before beginning the analysis phase of the project, determine whether the tools you plan to use will meet all your needs. This will prevent backtracking. To publish or not to publish When starting this research project, we not only wanted to find evidence to assess a learning out- come, but we also aspired to publish a peer-re- viewed article. When we completed our analysis, it became clear that our findings were not cohesive, unique, or generalizable. Each of us came to this understanding individually, but it seemed as if no one wanted to be the first to say it. When someone finally suggested our findings weren’t suitable for a peer-reviewed publication, you could sense the collective sigh of relief. So, where did that leave us? We needed to produce an internal report. After all, the results remained important and potentially consequential to our institution. We also recognized that the com- munication of some of the data may be perceived as pointing fingers and laying blame on other units and divisions within the George Mason University Libraries and the greater university community. We tried to focus on solutions and recommendations to address the issues students raised. We also tried to show where our own unit needed improvement. We wrote the report like a scholarly article, however, our interpretation focused on closing the assessment loop. We focused on how the findings could inform future internal studies in the library and how we could better assess the long-term impact of the library on undergraduates. Conclusion Focus groups can be a valuable feedback and assess- ment tool, but they are certainly not without issue. In the future, when we consider our research prob- lem, we will also strongly consider the best format through which to gather this information. In the end, many of the issues we experienced came down to time constraints and lack of experience with this methodology. We did not leave ourselves enough time to research, plan, and test (in addition to our other job duties), and we did not have adequate funding to invest in proper equipment. However, we did implement many successful strategies that we would use again in future. At the end of the day, it’s all part of the learning process.