C&RL News January 2021 32 The introduction of the ACRL Framework for Information Literacy for Higher Educa- tion provided an opportunity for libraries to revisit student learning and instruction practices. At the University of Northern Colorado Librar- ies, we embarked on a process of revising our shared student learning outcomes (SLOs) for all 100-level information literacy credit courses.1 The credit courses, taught by librarians, are offered in conjunction with programs on campus, like the Honors program and Center for Human Enrichment, or as a major require- ment for Criminal Justice, History or Audiology and Speech Language majors. The courses are 1-credit, 8-week classes that introduce students to the research process and focus on the fol- lowing SLOs: SLO 1: Students will be able to develop a research process. SLO 2: Students will be able to demonstrate effective search strategies. SLO 3: Students will be able to evaluate information. SLO 4: Students will be able to develop an argument supported by evidence. Once our SLOs better reflected concepts and practices from the ACRL Framework, we began to develop an assessment plan for our credit course program in order to determine if students were meeting our SLOs. As the impor- tance of assessment in library instruction has gained attention, many have developed SLOs and assessments for individual sessions, but program-wide evaluation can seem daunting. This article covers the steps we took to develop a program-wide assessment plan, including the lessons we learned along the way. The process Assessing student learning outcomes across courses and instructors requires significant buy-in from all who will teach in the library’s instruction program. Instructors must give up some of the freedom they enjoy in the class- room in order to systematically collect useable assessment data, which can provide evidence of learning and information for curriculum changes. Thus, involving all teaching librarians in the development of the assessment plan is essential. Our assessment plan for the shared 100-level SLOs was collaboratively developed by the libraries’ Curriculum Committee, com- prised of all librarians who teach in the infor- mation literacy credit course program. In the first year, we agreed we would use sig- nature assignments to assess the 100-level SLOs. Signature assignments are course-embedded as- signments, activities, projects, or exams that are collaboratively created by instructors to collect evidence for a specific learning outcome. These assignments are then embedded in all sections Lyda Fontes McCartin, Brianne Markowski, and Stephanie Evers Developing an assessment plan for information literacy learning outcomes Process and planning Lyda Fontes McCartin is interim director of the Center for the Enhancement of Teaching and Learning, email: lyda.mccartin@unco.edu, Brianne Markowski is head of Information Literacy and Undergraduate Support, email: brianne.markowski@unco.edu, and Stephanie Evers is information literacy librarian, email: stephanie. evers@unco.edu, at the University of Northern Colorado © 2021 Lyda Fontes McCartin, Brianne Markowski, and Stephanie Evers mailto:lyda.mccartin%40unco.edu?subject= mailto:brianne.markowski%40unco.edu?subject= mailto:stephanie.evers%40unco.edu?subject= mailto:stephanie.evers%40unco.edu?subject= mailto:emilyporterfyke%40gmail.com%20?subject= January 2021 33 C&RL News of a course, regardless of instructor.2 We worked on developing our assessment plan one SLO at time using the following process: Step 1: Develop signature assignment. Each librarian was asked to bring an idea for an as- signment that would assess the SLO to a Curricu- lum Committee meeting. The proposals ranged from assignments that we were already using to assess the SLO in our courses to ideas drawn from the literature. At the meeting, everyone briefly described their proposed assignment to the group. We then discussed the merits of the various assignments as tools for assessing student learning. Discussion prompts include: • What do you like about a proposed as- signment? • Do you feel the assignment would assess the outcome in a meaningful way? • How would the assignment work in your course? After vigorous debate, we agreed upon a sig- nature assignment to pilot. Because the original, proposed assignment often evolved based on our group discussions, a librarian was tasked with revising the assignment to reflect changes we had discussed, as well as developing instruc- tions for students and instructors. Step 2: Pilot signature assignment. The fol- lowing semester, two librarians volunteered to pilot the proposed signature assignment. The librarians revised their course schedules to integrate the assignment as either an in-class activity or take-home assignment. Student work was evaluated by the course instructor for grading purposes and collected for subsequent analysis. At the end of the semester, the librar- ians shared their experiences at a Curriculum Committee meeting by reflecting on the follow- ing questions: • How did the assignment fit in your course schedule? How and when was the assignment introduced? What instructional activities did students engage with before completing the assignment? • Were the assignment instructions clear to students? What revisions may be necessary? • Based on your experience, what would you recommend to other instructors? Piloting allowed us to gather valuable infor- mation on how the proposed signature assign- ment would work in the classroom before em- bedding the assignment in all 100-level courses. Step 3: Determine analysis procedure. While the pilot was underway, we discussed how we would analyze the data (student work) once collected. The analysis procedures included descriptive statistics on the percent of students answering a test question correctly, rubrics, and qualitative analysis. Depending on the an- ticipated complexity of the analysis procedure, one or two librarians were tasked with look- ing for examples of how others had assessed similar learning outcomes and modifying those procedures to fit our needs.3 At the end of the semester, these librarians presented their rec- ommendations to the rest of the Curriculum Committee. Then we tested the recommended analysis procedure on data collected from the pilot using the following questions: • Does this assignment and analysis pro- cedure assess the desired learning outcome? • Does this assignment and analysis proce- dure produce information you feel is useful for improving teaching and learning? • What, if any, changes should we imple- ment to the assessment? Step 4: Set achievement benchmark. Af- ter discussing the results from the pilot, we agreed upon an initial benchmark level of achievement for the SLO. This would be used to measure our success in achieving the SLO once we fully implemented the signature as- signment. For example, we use a standards benchmark for SLO 3—75% of students will correctly identify each source type (primary research, secondary research, non-research/ opinion). There are various types of perfor- mance benchmarks, and it is important to consider what type of benchmark you want C&RL News January 2021 34 to set for your SLOs.4 Step 5: Document the assessment method for the SLO. Two documents help us keep the assessment process on track: the Summative Assessment Overview and the Assessment Plan.5 The Summative Assessment Overview provides instructors with the information they need to embed the signature assignment in their course and collect data for assessment. It also includes a detailed description of the analysis procedures and reporting practices. The Assessment Plan provides a big-picture summary of the assess- ment methods for all program SLOs and docu- ments who is responsible for data collection, analysis, and reporting. It also describes when and how often assessment occurs. In the second year, we implemented the first signature assignment in all sections of the 100-level courses and began the process again for the next SLO. Now in the fifth year, we have successfully embedded signature assignments for each SLO into all sections of 100-level information literacy courses taught at the University of Northern Colorado. At the end of the 2019–2020 academic year, we com- pleted one full cycle of our assessment plan. We continue to meet regularly to analyze data collected from the signature assignments and to discuss how we can use what we’ve learned from the assessment data to improve teaching and student learning. Conclusion Similar to the process we used to develop our SLOs this process for developing a program- wide assessment plan is meant “to be flexible and work within the context of any instruc- tion program.”6 While the process described here was used to assess shared credit course SLOs, the process can be used by librar- ies looking to assess library-wide SLOs for one-shot or embedded information literacy instruction programs. Below are some sug- gestions for successfully implementing this process at your library. Be prepared to rewrite SLOs. During this process, we revised multiple SLOs as we discovered the SLO language did not truly reflect what we wanted to know about stu- dent learning. We recommend reviewing each SLO before you begin developing signature assignments to make sure you are satisfied with the language and that the SLO describes what you want to assess. Then, regularly re- visit the SLO while developing the signature assignment. Sometimes it is the process of creating the assessment that indicates an issue with the learning outcome being assessed. It is also a good idea to review the SLO as you review the pilot data. This is often where we determined that the data we collected for the SLO was not what we really wanted to assess in our program. Do not skip the pilot phase. The pilot phase is critical to project success. The pilot indicates issues with the signature assignment, SLO, and the data collection and analysis process. We piloted each SLO in multiple courses, which allowed us to look at a small sample of data and get feedback from the instructors about the logistics of implementing the assignment. The pilot allows you to make changes to the assess- ment and start over, if needed, before full-scale implementation of the signature assignment. Work on one SLO at a time. When we first began developing our assessment plan, we believed we could develop it in two years by working on one SLO each semester. We underestimated the amount of time it would take to develop, pilot, analyze data, and revise the signature assignment for each SLO. We slowed down our process and focused on one SLO each academic year. This made the process longer, but easier to manage. As we made progress, we were often able to have one signature assignment in pilot phase and another in development phase. A timeline will help keep the process on track, but be prepared to adjust as needed. Set aside time for this process. Our Curricu- lum Committee meets monthly. We were able to accomplish a lot during these meetings, but we also had other business to attend to. At the end of the first year, we held our first, daylong assessment retreat. This gave us more time for in-depth discussions. We have continued (continues on page 40) C&RL News January 2021 40 highlight and share their experiences during the “corona.” While some experiences and solutions were different depending on local financial and personnel resources, infrastructure, and institutional COVID-19 mandates, allowing institutions to do more or less, the prioritiz- ing and commitment to personnel and their user community were unwavering. Of the many strategies reported, three priorities were unmistakable when it came to dealing with corona: the safety of staff and users, the security of library materials, and the security of jobs while keeping employee morale up. I have learned that despite all of the differ- ences due to languages, borders, and nation- state mandated regulations, what unites us, as library and information practitioners, is our enthusiasm for helping our users while supporting ourselves. We have much more in common than linguistic, political, and socio-cultural differences. As the pandemic continues to thrust itself upon us, we have learned to keep the library going to serve our community. Instead of building walls, we have built bridges, leveraging our ingenuity, technology, and collaborative spirit. Notes 1. British Library, “Access to the British Library during Temporary Closure,” Brit- ish Library News, March 19, 2020, https:// www.bl.uk/news/2020/march/access-to -the-british-library-during-temporary-closure (accessed November 16, 2020). 2. British Library, “We’re Back,” British Library News, July 24, 2020, https://www. bl.uk/news/2020/july/we-cant-wait-to-see -you-again (accessed November 16, 2020). 3. World Bank, “Reversals of Fortune,” Poverty and Shared Prosperity 2020, https:// www.worldbank.org/en/publication/poverty -and-shared-prosperity (accessed November 16, 2020). 4. Marcy Oster, “National Library Launch- es Project to Document Coronavirus Impact on Jewish Life,” The Times of Israel, March 30, 2020, https://www.timesofisrael.com /n atio nal-library-launches-pr oject-to -document-coronavirus-impact-on-jewish -life/ (accessed November 20, 2020). 5. “Campaña COVID-19 Bar-Ilan Uni- versity,” 2020, https://www.youtube.com /watch?v=3FVZYh2SgAc (accessed Novem- ber 20, 2020). to hold yearly assessment retreats to analyze data and discuss changes to the curriculum for the following year. Document every decision. Early on we did not take extensive notes during meetings, which we regretted. With all the changes you will make to your SLOs, assessments, and curriculum throughout this process, it is important to document each decision you make. Assign a notetaker for each discus- sion, and keep notes in a central location for easy access. Determine a project manager, but share the work. There are a lot of moving parts to this process, and a project manager is key. Our project manager was the committee chair. This person determined the timeline, set meetings, kept track of each phase, and assigned tasks. While it is important to have a project manager, the work must be shared. For one, it is too much work for one person. More importantly, sharing the work builds buy-in and ownership of the process for all instruction librarians. Developing an assessment plan for shared information literacy learning outcomes is a time-intensive process, but it is possible with teamwork, organization, and a willingness to revisit past decisions. Notes 1. Andrea Falcone and Lyda McCartin, “Be Critical, But Be Flexible: Using the Framework to Facilitate Student Learning Outcome Devel- opment,” C&RL News 79, no. 1 (2018): 16–19. 2. Office of Institutional Research, Plan- (“Developing an assessment plan,” continued from page 34) (continues on page 44) https://www.bl.uk/news/2020/march/access-to -the-british-library-during-temporary-closure https://www.bl.uk/news/2020/march/access-to -the-british-library-during-temporary-closure https://www.bl.uk/news/2020/march/access-to -the-british-library-during-temporary-closure https://www.bl.uk/news/2020/july/we-cant-wait-to-see -you-again https://www.bl.uk/news/2020/july/we-cant-wait-to-see -you-again https://www.bl.uk/news/2020/july/we-cant-wait-to-see -you-again https://www.worldbank.org/en/publication/poverty -and-shared-prosperity https://www.worldbank.org/en/publication/poverty -and-shared-prosperity https://www.worldbank.org/en/publication/poverty -and-shared-prosperity https://www.timesofisrael.com/national-library-launches-project-to-document-coronavirus-impact-on-jewish-life/ https://www.timesofisrael.com/national-library-launches-project-to-document-coronavirus-impact-on-jewish-life/ https://www.timesofisrael.com/national-library-launches-project-to-document-coronavirus-impact-on-jewish-life/ https://www.timesofisrael.com/national-library-launches-project-to-document-coronavirus-impact-on-jewish-life/ https://www.youtube.com/watch?v=3FVZYh2SgAc https://www.youtube.com/watch?v=3FVZYh2SgAc C&RL News January 2021 44 to our changing campus environments made ever more uncertain under the pandemic. Rap- id prototyping is one way to adopt such a shift, but it isn’t the only path forward. There are no perfect sprints, scrums, or development cycles that will reframe academic libraries in a mod- ern light. Agility, risk-tolerance, and think-tank operational attitudes should be our takeaways from industry. We need to learn how to fail faster and share our mistakes publicly as much as we celebrate our successes. If academic libraries can fail faster together, we just might have a shot at emerging from the pandemic as the transformational campus leaders that higher education desperately needs. Notes 1. Alexandra Witze, “Universities Will Never Be the Same After the Coronavirus Crisis—How Virtual Classrooms and Dire Finances Could Alter Academia: Part 1 in a Series on Science After the Pandemic,” Nature, 582 (2020), 162-164, https://www. nature.com/articles/d41586-020-01518-y. 2. Willa Tavernier, “COVID-19 Demon- strates the Value of Open Access,” C&RL News, 81, no. 5 (2020), 226-230. https:// crln.acrl.org/index.php/crlnews/article /view/24414/3225; Ewen Callaway, “Will the Pandemic Permanently Alter Scientific Publishing? The Push for Rapid and Open Publishing Could Take Off—Although Fi- nancial Pressure Lie Ahead: Part 4 in a Series on Science After the Pandemic,” Nature, 582 (2020), 167-168, https://www.nature.com /articles/d41586-020-01520-4. 3. Lisa Peet, “Ithaka Library Directors Sur- vey Reflects a Different Time-and Offers Clues for What May Lie Ahead,” Library Journal, April 16, 2020, https://www.libraryjournal. com/?detailStory=ithaka-library-directors -survey-reflects-different-time-offers-clues -for-what-may-lie-ahead-covid-19. 4. Danielle Cooper, “Project Launch: Can- celing the Big Deal,” Ithaka S+R Blog, May 28, 2020, https://sr.ithaka.org/blog/project -launch-canceling-the-big-deal/. 5. C h r i s t o p h e r C o x , “ C h a n g e d , Changed Utterly,” Inside Higher Ed, June 5, 2020, https://www.insidehighered.com /views/2020/06/05/academic-libraries-will -change-significant-ways-result-pandemic -opinion. 6. Roger C. Schonfeld and Christine Wolff-Eisenberg, “Academic Libraries at a Pivotal Moment,” The Scholarly Kitchen, April 6, 2020, https://scholarlykitchen.sspnet. org/2020/04/06/academic-libraries-pivotal -moment/. 7. John J. Meier and Rebecca K. Miller, “Turning the Revolution into an Evolution: The Case for Design Thinking and Rapid Prototyping in Libraries,” C&RL News, 77, no. 6 (2016), 283-286. 8. Lettie Y. Conrad, “‘Little Deals’ Everywhere: Is Demand-driven Col- lection Development Catching Fire?” T h e S c h o l a r l y K i t c h e n , A u g u s t 1 1 , 2 0 2 0 , h t t p s : / / s c h o l a r l y k i t c h e n . sspnet.org/2020/08/11/little-deals-every w h e r e - d e m a n d - d r i v e n - c o l l e c t i o n -development-catching-fire/?informz=1. (“Developing an assessment plan,” continued from page 40) ning and Effectiveness, “Signature As- signment: Quick Reference Guide,” Pratt Institute, June 17, 2013, https://www.pratt.edu /uploads/signature-assignment-resources.pdf. 3. Brianne Markowski, Lyda F. McCartin, and Rachel Dineen, “Using Signature Assign- ments to Assess Information Literacy Out- comes” (poster presentation, Library Assess- ment Conference, Chicago, IL, forthcoming). 4. Linda A. Suskie, Assessing Student Learning: A Common Sense Guide, 2nd ed. (San Francisco: Jossey-Bass, 2009): 233–52. 5. See Lyda McCartin et al., “Assessment: 100-Level LIB Course Assessment Plan,” Uni- versity of Northern Colorado, 2020, https:// digscholarship.unco.edu/infolit/24. 6. Falcone and McCartin, “Be Critical,” 19. https://www.nature.com/articles/d41586-020-01518-y https://www.nature.com/articles/d41586-020-01518-y https://crln.acrl.org/index.php/crlnews/article /view/24414/3225 https://crln.acrl.org/index.php/crlnews/article /view/24414/3225 https://crln.acrl.org/index.php/crlnews/article /view/24414/3225 https://www.nature.com/articles/d41586-020-01520-4 https://www.nature.com/articles/d41586-020-01520-4 https://www.libraryjournal.com/?detailStory=ithaka-library-directors -survey-reflects-different-time-offers-clues -for-what-may-lie-ahead-covid-19 https://www.libraryjournal.com/?detailStory=ithaka-library-directors -survey-reflects-different-time-offers-clues -for-what-may-lie-ahead-covid-19 https://www.libraryjournal.com/?detailStory=ithaka-library-directors -survey-reflects-different-time-offers-clues -for-what-may-lie-ahead-covid-19 https://www.libraryjournal.com/?detailStory=ithaka-library-directors -survey-reflects-different-time-offers-clues -for-what-may-lie-ahead-covid-19 https://sr.ithaka.org/blog/project-launch-canceling-the-big-deal/ https://sr.ithaka.org/blog/project-launch-canceling-the-big-deal/ https://www.insidehighered.com/views/2020/06/05/academic-libraries-will-change-significant-ways-result-pandemic-opinion https://www.insidehighered.com/views/2020/06/05/academic-libraries-will-change-significant-ways-result-pandemic-opinion https://www.insidehighered.com/views/2020/06/05/academic-libraries-will-change-significant-ways-result-pandemic-opinion https://www.insidehighered.com/views/2020/06/05/academic-libraries-will-change-significant-ways-result-pandemic-opinion https://scholarlykitchen.sspnet.org/2020/04/06/academic-libraries-pivotal-moment/ https://scholarlykitchen.sspnet.org/2020/04/06/academic-libraries-pivotal-moment/ https://scholarlykitchen.sspnet.org/2020/04/06/academic-libraries-pivotal-moment/ https://scholarlykitchen.sspnet.org/2020/08/11/little-deals-everywhere-demand-driven-collection-development-catching-fire/?informz=1 https://scholarlykitchen.sspnet.org/2020/08/11/little-deals-everywhere-demand-driven-collection-development-catching-fire/?informz=1 https://scholarlykitchen.sspnet.org/2020/08/11/little-deals-everywhere-demand-driven-collection-development-catching-fire/?informz=1 https://scholarlykitchen.sspnet.org/2020/08/11/little-deals-everywhere-demand-driven-collection-development-catching-fire/?informz=1 https://www.pratt.edu/uploads/signature-assignment-resources.pdf https://www.pratt.edu/uploads/signature-assignment-resources.pdf https://digscholarship.unco.edu/infolit/24 https://digscholarship.unco.edu/infolit/24