315 Carrots and Sticks: A Qualitative Study of Library Responses to the UK’s Research Excellence Framework (REF) 2021 Open Access Policy Dan DeSanto* This study examines how academic libraries in the UK responded to the Research Ex- cellence Framework (REF) 2021 open access policy. Thirteen information professionals at twelve institutions across the UK took part in semi-structured interviews. Findings from the interviews reveal how libraries created and deployed new infrastructures, workflows, and staffing as well as the methods through which universities commu- nicated the policy’s requirements. The study describes respondents’ experiences of the changes brought about by REF 2021 as well as their thoughts on how the REF 2021 open access policy will affect future REF assessments. Results provide insight for libraries responding to US initiatives such as the August 2022 White House Office of Science and Technology Policy memo directing the open publishing of federally funded research. Introduction Every six to eight years, the United Kingdom’s government council for research investment, UK Research and Innovation (UKRI), undertakes an exercise called the Research Excellence Framework (REF) to assess the research contributions made by each university and reallocate funding based on outcomes. REF guidance outlines the threefold purpose of the REF as: 1. providing accountability for public investment in research and produce evidence of the benefits of this investment 2. providing benchmarking information and establish reputational yardsticks, for use within the higher education sector and for public information 3. informing the selective allocation of funding for research.1 The REF is an assessment of return-on-investment for research undertaken at UK higher education institutions. With approximately £2 billion of total research funding at stake, uni- versities carefully follow the guidance set out by UKRI. The most recent REF, REF 2021, began its assessment period on April 1, 2016, and was completed on March 31, 2021. A university’s REF submission is comprised of three parts: an impact case study (25 percent of overall score); a statement of the university’s research envi- ronment (15 percent of overall score); and research outputs (60 percent of overall score). This study is concerned with only the latter category, in which universities present examples of * Dan DeSanto is a Library Associate Professor at the University of Vermont Libraries; email: ddesanto@uvm.edu. ©2023 Dan DeSanto, Attribution-NonCommercial (https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:ddesanto@uvm.edu https://creativecommons.org/licenses/by-nc/4.0/ 316 College & Research Libraries May 2023 impactful contributions (called “outputs”) made by university researchers.2 By its conclusion, 157 universities had participated in REF 2021. Over 185,000 research outputs were submitted for assessment, representing work from over 76,000 authors.3 The most important change for academic libraries in REF 2021 was an open access (OA) policy stating that articles and conference contributions must be made open, defined as “discoverable, and free to read, download and search within, by anyone with an internet connection,”4 within three months of acceptance to be eligible for inclusion in the REF. This presented an enormous challenge and required that libraries find ways to make open nearly the entirety of their university’s non-monographic publications. The aggressive time frame (publication often fails to happen within three months of acceptance) and the cost of article processing charges (APCs) ruled out publication in OA journals (gold OA) as the primary means for complying with the open access policy. Instead, the policy advanced a process of accepted manuscripts being deposited to institutional repositories (green OA) as the primary means for compliance. Universities quickly needed to implement research management and repository infrastructures able to accommodate and track the vastly increased number of de- posits about to pour in. University libraries facilitated much of the response by setting up or expanding systems to process and track deposits, creating workflows for researcher deposit, and providing the outreach necessary to ensure that researchers understood the policy and deposited their work. Across the UK, libraries were extremely successful. The infrastructure that emerged to support REF 2021’s open access policy has proven to be transformative. Figure 1 looks at the most recent Leiden University Centre for Science and Technology Studies rankings (2016–19) for proportion of university research that was made open access (PP(OA)) and indicates that, globally, the UK holds twenty-four of the top twenty-five places for universities with the highest proportion of research made OA. Going further down the list, forty-six of the top fifty universities worldwide with the highest proportion of open research are in the UK.5 The rankings put in context the degree to which UK universities are making their research open and, as indicated by those interviewed for this study, the marked increase in open research is directly attributable to REF 2021’s open access policy. This study asks questions of repository, communications, and research information professionals at twelve academic libraries who played a significant role in their institutions’ response to the REF open access policy. The study aims to find common experiences that illustrate how libraries found ways forward and created responsive systems that made ac- cessible the majority of research produced on their campuses. Further, it seeks out themes on what respondents considered to be the effects of the open access policy, both intentional and unintentional, and asks them to consider future developments for OA after REF 2021. Literature Review The REF represents one instance of the UK’s increasingly demanding requirements that universities demonstrate the impact of the research they undertake—commonly referred to as the “impact agenda.”6 The impact agenda, and the REF as one of its manifestations, has encountered a good deal of criticism as the commodification of academic work.7,8 John Hol- mwood, one such critic, writes, “The impact agenda …recommends that research should be coproduced with beneficiaries. In consequence, it proposes that research should be aligned Carrots and Sticks 317 with the interests of those beneficiaries and modified in order better to realize them. The in- tention of the impact agenda was to speed up the commercialization of research, or the time from idea to income.”9 This “speeding up” via exercises in researcher accountability has come under scrutiny as numerous authors point to situations where researchers must change the nature of their research to fit a size and shape dictated by an accountability mechanism.10,11,12 The REF, for many, fits this category of a neoliberal assessment exercise that furthers epistemic injustice.13,14 FIGURE 1 Leiden University Centre for Science and Technology Studies Rankings (2016–2019) for Proportion of University Research Made Open Access (PP(OA)) 318 College & Research Libraries May 2023 Extending this criticism to the realm of scholarly communication, the REF 2021 open ac- cess policy can be viewed as incongruous with aims to democratize knowledge or broaden access to information. Instead, it can be viewed as serving the corporatized end of speeding up research-as-product for the purposes of business and economic development.15 In this ac- counting, the open access policy becomes yet another demand that burdens researchers and directs research in ways that benefit corporate entities rather than the public good. Poynder calls the REF “the pressuring and sweating of researchers to increase productivity, for almost exclusively economic reasons” and asks, “How can the OA ethos fit comfortably with this?”16 While criticisms of corporate-driven open access are not unique to the REF, they find a fertile home in the REF’s tying together accountability and access. Whatever the motivation behind the REF 2021 open access policy, it has undeniably made a large amount of scholarly information openly readable that otherwise would not have been. Writers like Martin Eve push back on claims that the UK’s gains in OA are somehow tarnished or should be discounted. Eve agrees that the REF exercise forwards the commodification of research, but differs with writers like Holmwood and Poynder, claiming that “this does not mean that it is the technology of open dissemination of research online—digitalization—that should come in for critique…it seems to me that open access is not to be grouped under the neoliberal rhetoric of ‘there is no alternative.’ There patently is an alternative: we do it already [proprietary publishing]. And for many, many people, it is decidedly worse.”17 The gains made in OA, for Eve, present an improvement on the current publishing system. In this accounting, the benefits of access outweigh the problems of the REF as a catalyst. Studies of the REF 2021 open access policy’s effects on researchers and libraries are much more difficult to find. In 2016, DeGroff outlined potential sticking points and best practices as universities and libraries began setting up systems to respond to REF 2021.18 In the following years, quantitative assessments measured the ways in which the REF’s open access policy was reshaping repositories and expectations for research availability.19,20 At the closing of the REF 2021 cycle, qualitative studies emerged that reflected on the monumental task just completed. The largest of these reflective studies is the Real Time Review done by researchers at Cardiff University and the University of Sheffield. The study surveyed approximately 600 faculty across four UK universities and conducted twenty-one in-depth interviews with REF managers at the same universities. While the study addresses researcher attitudes and knowledge of the REF more broadly, it calls out the open access component in saying, “Importantly, an insight that emerged from the survey data was that the focus on open access and research practices was the most consistently positive and impactful influence of the REF on both researchers’ own work and UK academic culture.”21 While researchers in the Cardiff University/University of Sheffield study were positive about the general idea of increasing access to research after the REF 2021 exercise, research- ers in other studies were decidedly less positive when asked about the specific process of submitting accepted manuscripts to repositories. Carolyn Ten Holter’s 2020 study asks re- searchers about the day-to-day work of submitting research and populating a repository.22 The Ten Holter study finds a lack of enthusiasm for open access stemming from researcher frustrations with deposit processes and poor understanding of how submissions contribute to open scholarship. Repository managers interviewed for the study described their work as cumbersome and technically demanding; however, they remained committed to the benefits of making research open. For both researchers and repository managers, compliance with Carrots and Sticks 319 REF 2021’s open access policy was described as stressful and in danger of becoming an end in itself rather than a means of facilitating open research. Now that REF 2021 has come to a close, reflective work such as Ten Holter’s and the Cardiff and Sheffield study are increas- ingly important as the UK begins to plan for the next REF and governments outside the UK consider open access mandates of their own. The study presented here fits into this nascent body of qualitative research reflecting on and attempting to learn from REF 2021. Methodology Thirteen library and research information professionals at twelve different institutions partici- pated in semi-structured interviews. Participants were invited based on geographic location and university size. Participants represent institutions across England, Scotland, and Wales at universities varying in size from approximately 7,000 to 37,000 students; the largest con- centration clustered at campuses of 15,000—30,000 students.23 Nine participants had primary responsibilities in scholarly communications, three in repository management, and one in research information management. Interviews were conducted between September 2020 and September 2021. The duration ranged from forty-five to ninety minutes, and recordings were made with either Microsoft Teams or Zoom. The semi-structured interviews asked respondents about: 1. their job function in the university library or libraries 2. the infrastructure (both technical and staffing) put in place to make research open access 3. the outreach that occurred to the larger campus 4. green vs. gold routes for open access at their university 5. the library’s role in facilitating compliance with REF 2021’s open access policy 6. the changes in scholarly publishing they attributed to the REF 2021 open access policy. Standard interview questions are attached as an appendix. Respondents answered for as long as they wished, and follow-up questions were asked for the purpose of clarification. Three participants were contacted a second time to clarify initial responses. Interview recordings were loaded into Nvivo qualitative analysis software for transcrip- tion and the development of an initial coding structure. Coding followed an inductive ap- proach and was refined by a second round of coding, done by hand, to develop subcodes. The coding structure informed a thematic analysis of the data.24,25 It should be noted that multiple participants moved to new institutions within the timeframe of the REF exercise; those par- ticipants spoke of their experiences not only at their current institution but also at their prior institution. Before each interview participants were provided with a copy of an institutional review approval stating that their identities and responses would be kept anonymous. Results Infrastructure & Workflows Implementing a CRIS Eleven of the twelve institutions in the study implemented a current research information system (CRIS). Ten institutions implemented a commercial CRIS system (Elsevier’s PURE was used by six universities and Digital Science’s Symplectic by four), and one institution maintained their own. The lone university without a CRIS was in the process of acquiring one. A CRIS serves as a centralized system by which researchers, librarians, and administrators 320 College & Research Libraries May 2023 interact in order to process, manage, and organize the research that takes place at a campus.26 It differs from a repository in that it is not meant to store a large number of full-text documents; rather, it is meant to facilitate and track the full research process from Institutional Review Board (IRB) approval to publication and reporting. Although manuscripts can be deposited through a CRIS, it serves the deposit process only as an intake channel to the repository. Resoundingly, the message from those interviewed was that the CRIS made deposit easier for researchers and measuring compliance easier for managers. Many respondents spoke to the need for simplicity in the deposit process. The idea is that the researcher only engages with one system. They engage with the CRIS. And as many things as possible are fed automatically from external systems. Another librarian stated that deposit through the CRIS should take no longer than two minutes. The CRIS allowed universities to create a comprehensive record of scholarship and report on OA compliance within this body of scholarship. As one librarian noted, the CRIS and REF served each other’s ends of completeness and compliance. So there is an aspect of if your department is assessing what you’ve done…if it’s not in the CRIS, then it’s not been written…. If everybody has to put everything in that system, for the purposes of institutional monitoring, and then we check those systems for REF compliance, then the system serves the REF and the REF serves the system, which works relatively well. Respondents using a CRIS system described it as a vital tool in managing their campus’ research activity and tracking researcher deposits. Because the CRIS touches other parts of the research process, it allows information to be shared between modules to generate a broad picture of scholarship as well as more specific reports on OA compliance. Methods for Depositing Work The majority of universities included in this study relied on researcher deposit as the primary means for populating their repositories. Two universities relied on mediated deposit facili- tated by the library. One university had multiple means of deposit for its different schools. Counter to assumptions of scale, the universities with mediated approaches were not the smallest schools in this study; in fact, two of the three have particularly sizable student and researcher populations. Respondents at researcher deposit universities needed to convey a clear and consistent message regarding deposit expectations, timeline, and workflow. Those interviewed stressed that conveying a simple and concise plan to researchers was essential to their process. We needed to have a simple message …we didn’t want to say, “Here’s how you’re compliant with the REF: you can either do A, B, C, or D.” We just want to say, “There’s one process. Follow this process at all times and you’re fine. If you don’t follow the process, you may not comply.” And our message was “Deposit in PURE Carrots and Sticks 321 within three months of the date of acceptance and we’ll make it open access.” So that was the key, simple message that everyone had to buy into. Our advocacy to researchers is to act on acceptance. It needs to be your workflow that when your paper’s accepted …you get the email from the publisher saying ‘congratulations, we’ve accepted your paper,’ and at that point you need to go into PURE and create a record and upload your accepted manuscript. Just do it. It doesn’t matter if it’s a review article, it doesn’t matter if you are in- vited to comment on something else. It doesn’t matter what it is. Just put it into the repository and then you don’t have to worry about it. Respondents at mediated deposit universities acknowledged the staff time needed to support a mediated approach but pointed to the benefits of accuracy, precision, time saved double-checking deposits, and ease of compliance from researchers: I prefer it that way, because I know that we’re doing it properly, and if you catch them at acceptance, because we had to deposit within three months of acceptance for REF, we knew that we were getting it right. We are not asking the authors really to do very much, but they send us the manu- script and then we can take most of the details from that…. My job is to remove the admin from the researchers as much as possible. While mediated deposit approaches did largely remove the burden for deposit from researchers, some respondents noted that taking the deposit process out of the hands of re- searchers had unintended consequences. It does mean, though, that academics are a bit hands-off so they don’t always engage with the CRIS and with the system because there are other things that you can do, like activities and awards and things like that, which most people just don’t do. So I wouldn’t say it’s not without its problems the way we do it. We would like them to engage more with it. And they don’t because we’re doing so much for them. It seems a valid concern that researcher disengagement in matters of dissemination and access could prove detrimental as funders increase expectations for open research, their poli- cies become more complex, and researchers are asked to demonstrate impact in new ways. Prior to the REF 2021 cycle, many UK libraries harvested publications from databases like Scopus and Web of Knowledge as the primary means for depositing work into their institu- tional repositories. Since the REF policy required work to be deposited within three months of acceptance, this workflow became invalid as a primary means for populating a repository. Many respondents noted, however, that they still use harvesting as a notification mechanism for items that researchers have yet to deposit. 322 College & Research Libraries May 2023 We get weekly reports from Scopus …and so we can sort of nudge authors if it’s not already in PURE and they need to add it to PURE quickly, so that’s how we do our temperature reports. I’ve got to look through all these different [harvested] things but that just kind of works because it helps me to reach out to people and just say, “Hey, here’s a reminder. We have a repository. We have a Publication and Open Access Policy. Just a gentle reminder. Could you please deposit when you get a chance?” …and that has helped I think. A majority of respondents indicated that they will only spend time chasing down work that is still valid (within three months of acceptance) for the REF, although a few stated that they will seek out older non-REF compliant work for the sake of a complete record in the repository. Dealing with the Number of Deposits Institutional repositories needed to adapt in order to process a hugely increased number of deposits. Ten of the twelve institutions in this study maintained an institutional repository. Of the two without a dedicated institutional repository, the CRIS served this purpose and powered a public search interface. The most common repository systems were the University of Southampton’s E-Prints (used by five universities) and the Lyrasis platforms D-Space (used by four) and Fedora (used by one). By design, every university in the UK saw increases in the number of deposits to its re- pository. Respondents highlighted just how dramatic this increase was: It’s a constant stream of things that come through …it’s very much increased. I think we’re depositing between six and seven thousand items a year now. Prior to REF, probably less than a thousand. When I started in the Open Access Team in 2010, we had about 5 percent full text in the repository…. With the REF OA mandate, we’re now at about 90-plus percent. They described needing to rethink previous processes and add additional infrastructure to increase capacity. In addition to the necessary technical infrastructure, libraries were also forced to rethink staffing models and add positions to adequately process the increased num- ber of submissions. Changes to Staffing Most universities in the survey added between one and five new FTE to facilitate compliance with the open access policy. Of two outliers, one added no new positions and another added sixteen. For almost all universities, adding new staffing was essential to accommodate the REF 2021 open access policy. We just knew we that we had to have …more full-time people on this, just seeing the volume of content through. Carrots and Sticks 323 So, it kind of evolved from being just one research support librarian doing every- thing to a research services team covering different areas. [The repository] did see a huge increase; however, it still required a huge increase in resources for the library to process the content …towards the end of 2015 until the end of 2019, so over a four-year period, we went from one to sixteen members of staff. Even with the increased staffing, respondents still largely felt like they lacked enough capacity for the amount of work created by the REF. Those on much smaller staffs especially felt overburdened. We had 1,000 outputs, which isn’t a great deal compared to some institutions which were submitting thousands and thousands of outputs, but …it was just me on my own …managing this …it was a massive job. Thank God it’s over. We’ve never got enough staff. It could definitely do with a bigger team. And actu- ally we have just expanded in 2019. We were really under-resourced before then. So yeah, it can be quite difficult. One librarian noted that new positions were not necessarily granted by the institu- tion; rather, a case had to be made that these new positions were necessary, and this case was made much easier by the ability to point to the financial repercussions of a poor REF submission. Making the business case ended up being able to demonstrate what was going to happen if we didn’t have additional resources. I would like to say resources were offered from the start with “whatever you need is fine,” but it was the negative consequences of not having additional resources that obtained results. The significance of REF required most libraries to add staffing in order to accommodate the demands of increasing deposit numbers and meet the expectations set out by REF 2021. Outreach In addition to processing and managing deposits, libraries also needed to communicate to a wider campus audience the policies, processes, and expectations governing deposits and OA compliance. Starting Early Most respondents noted that their universities began outreach about the new open access policy before the REF process got fully underway in 2016. Those at the few campuses that failed to get an early start discussed feeling behind and needing to catch up. Starting early enabled libraries to iron out processes for deposit and get researchers used to depositing their work. 324 College & Research Libraries May 2023 We started a year early, so we started in 2015 to get people used to the policy be- fore it actually came into force. So, by the time we got to 2016, it wasn’t too bad. People had gotten used to the policy and our systems. By starting early, libraries were able to communicate the workflows researchers had to follow, but respondents also noted a feeling of needing to “get out ahead” of misinformation. We had a bit of a job to do because …people started getting information from outside, and so, as the policy came out, departmental researchers and individual research group leaders started developing their own ideas about [the open access policy]. And…we found that quite lethal. So we embarked on a huge exercise to tell people from one source, from us, what the policy would be and how we were going to implement it, and that worked quite well. So we kind of rushed because of that. Being able to set, from the start, a clear and definitive message around the REF open access policy and university expectations proved crucial to smoothly implementing new workflows and infrastructure. Administrative Communication and Support Establishing a connection to upper-level administration provided access to meetings and events that those in libraries would not have had otherwise. It also tasked administrators with setting expectations and demonstrating that compliance with the open access policy was non-negotiable. I thought it was top down that did it [got them into meetings with faculty]. We wouldn’t have had any inroads into any of it without it being a top-down approach. The only way was through the head of the college …so I went and persuaded her and then she just invited me to all the meetings that happened. It required very strong top down communication and there is a committee which looks after these issues, specifically open access in the university, and it’s chaired by the pro vice chancellor for research…. The message is “You need to be up to speed with this. These are the people you need to contact. Here is the information about how to do that, the help that you need. If you want to ask any questions, here is the help line.” Respondents overwhelmingly indicated that their outreach work relied on good relation- ships with administrators overseeing the university’s REF 2021 response. These relationships helped establish those in the library as the campus experts leading and directing OA compliance. Methods for Direct Outreach Unsurprisingly, issues of scale and capacity largely dictated each university’s approach to con- ducting outreach. Librarians and research managers tasked with communication worked with Carrots and Sticks 325 subject liaisons, held workshops, met with small groups, and spoke one-on-one with faculty. Non-personal outreach included email messages and physical mail drops. Interestingly, two librarians described the helpfulness of including a deposit link directly in their email signatures. Departmental Contacts Respondents from three larger universities described how their libraries created a network of OA contacts within departments and spent significant time training them. In some cases, OA contacts served as departmental advisors tasked with helping researchers deposit their work. In other cases, contacts made deposits on behalf of researchers. In nearly all cases, OA contacts were administrative staff people who required a good deal of training. This is …someone who is probably going to be the poor sod running around chas- ing the academics in the department…the administrative person could be putting things into Symplectic, the CRIS, themselves or they could just be chasing people because they get emailed stuff regarding compliance…. If it’s handled within the department, it does tend to be a lot more effective than an outside body. We established a network of open access contacts in schools and faculties …most schools actually have got a named open access contact, and they’ve got a variety of kind of roles and responsibilities. So some of them do mediated deposit on behalf of the staff in their schools. Some of them are a lot more hands off than that. And they just kind of do a little bit of compliance monitoring and that sort of thing when it’s necessary. But most of our comms go through this, the open access contacts…. So that was one of the big things we did. Actually, it’s probably one of the things that’s had the biggest impact. For those at larger institutions, time spent in “train the trainer” sessions or at help desks specifically dedicated to OA contacts was time well invested. A system of OA contacts less- ened the burden of communicating with faculty about individual deposits and freed up time to tackle higher level reporting, compliance, and outreach needs. Broader OA Education As REF 2021 drew to a close, a few respondents noted a change to their communication plans. After five years of researchers getting used to workflows and expectations, some libraries are taking the opportunity to broaden out their communications around open access and move away from messaging focused mainly on policy and process. I think we tried to switch the message up a bit from being the REF stick to more of the open access carrot…. We basically taught them about the benefits to them: collaboration, raising visibility, rather than, “you’ve got to do this because this is what the university is telling you you’ve got to do.” There was a feeling expressed by some in the study (detailed in the Effects section), as well as the Ten Holter study,27 that researchers’ focus on REF compliance began to overshadow 326 College & Research Libraries May 2023 deeper understandings of open access or open research. This concern precipitated outreach efforts to move faculty beyond a compliance-only mindset. Other Funder-Based Open Access Policies Preceding Open Access Policies Prior to the REF 2021 period beginning in 2016, two large UK funders, UKRI and the Well- come Trust, had already introduced open access policies of their own. These pre-2016 poli- cies tied open access requirements to research grants or awards, but they did not have the aggressive timeline of the REF nor, initially, was compliance strictly enforced. Because the policies were tied to specific funding awards rather than university-wide funding (like the REF), they missed large swaths of faculty whose work did not rely on acquiring research funding. Yet, despite their limitations and checkered compliance, most respondents felt that funder OA policies that preceded REF 2021 did contribute a foundation on which the REF 2021 open access policy built. The big funders such as UK Research and Innovation, you know, they’ve really driven that open access agenda forward. I think they’ve definitely paved the way for something like REF. There’s a certain sort of line that you can trace. It seems inevitable when you look back …. Really, I think in terms of what they [funders] required around open ac- cess, I think that maybe was the testing ground that then allowed them to introduce that the REF policy was going to affect a much wider group of people. So, I think politically it would have been more difficult for them to have that [REF] policy if they hadn’t already had that. While respondents largely indicated that the earlier funder requirements established a framework on which the REF 2021 open access policy could build, many also noted just how dramatically the policy differed from these previous mandates. It was a logical progression, but it was a big step. REF is so, so important and people are so passionate about it in the UK. It’s really so important to them. And we just wouldn’t have had the building blocks in place going in …. I think actually REF is more of a driver than even the [individual] funders. The REF is more encompassing. In fact, one librarian indicated that the REF policy was different enough from the earlier funder requirements that they “probably didn’t matter that much because it [REF 2021] was a whole new thing.” However, most respondents felt that the foundation created by preced- ing funder policies was important, and they expected new funder OA policies, like the most recent from UKRI,28 to push open access boundaries further and create even more aggressive expectations for open research in the next REF cycle. Carrots and Sticks 327 Haves and Have-Nots Open access policies from UKRI, the Wellcome Trust, and other funders did not disappear with the REF. A funded research project might need to meet one set of expectations for the funder and another for REF. Respondents spoke in their interviews about the complexities of managing unaligned OA policies and APC funding while simultaneously building a repository infrastructure for REF 2021. They also described empathizing with researchers who, unlike librarians and information professionals with expertise in scholarly communications, had to fulfill funder requirements with very little familiarity to draw from. One librarian described the burden on funded researchers as a “policy stack” and another as a “minefield,” while others expressed their desire for funder policies to align. In many cases, the repository served double duty as a mechanism for funder compliance when APC funding began to run out: We cannot afford to have current subscriptions the way they are, so we’re kind of seeing green open access as a pressure valve to enable our researchers to comply with their research funder policies. We always run out of money every year, everybody runs out of money. Most years, we would tell somebody we wouldn’t pay for this [APC], actually, because we can make it open access through the green route. And they normally don’t like that. The block grants that we’ve got from the funders like the UKRI one, in particular, it just doesn’t go very far. We’re always running out before the end of the year, and then we’re having to say, “You need to do green open access.” A robust green OA infrastructure, then, not only answered the REF’s call for fast ac- cess to an accepted manuscript but also provided another tool to facilitate compliance with other OA policies. Many respondents noted, however, that this contributes to a dichotomy of “haves” and “have nots” in which repository-based green OA is seen as secondary or lesser- than APC-funded gold OA. What we don’t want, of course, is a situation whereby those that can afford to make their work available through open access with the version of record, which is the one everybody wants, and those that can’t have the green version. It’s almost becoming a two-tiered system where some people, if you’re funded, you can afford to pay for it and if you’re not funded, you’re like second class go- ing the green route because you don’t have any other options. There’s still a bit of a feeling of gold open access as being kind of proper open access and certainly as being more desirable, I guess, for a number of reasons. Possibly calling it ‘gold’ doesn’t help…and green open access is kind of the poor researcher’s alternative. 328 College & Research Libraries May 2023 Many respondents felt that REF 2021 made inroads in countering negative views of green OA by raising its profile and demonstrating that it is legitimate, acceptable, and often the fastest way to make work accessible. Compliance Informing not Enforcing Universities took REF open access compliance very seriously given the possible consequence of having research outputs disqualified. Disqualified quality outputs could jeopardize a uni- versity’s REF submission and result in a loss of revenue to the university. When discussing compliance, respondents made clear that their role was to inform compliance, not to enforce it. Those tasked with enforcing compliance were either university administrators, committees, or departmental administrators. Within the library, we said, “Look, we’re not going to be the compliance checkers. We’ve given you the tools, we’re getting the stuff in, but we’re not going to go around and haven’t got the resources to do this [enforcement].” We weren’t the stick because we’re not really in a position to do it, but we were reporting to senior people in the university…. I think it’s also, realistically, people wouldn’t listen to us when we don’t have the clout, I suppose. I think we were just not in a position to do that, but we have the information to make others do it. Instead, librarians passed information to administrators via dashboards, reports, or spreadsheets. In other cases, departmental OA contacts generated compliance information and served as first-line compliance monitors. One librarian noted that her campus created staff positions specifically for monitoring compliance eighteen to twenty-four months before the REF’s completion. Using Exceptions Some respondents discussed the allowable exceptions in the REF 2021 open access deposit process. The REF 2021 guidance states that 5 percent of outputs included as part of a uni- versity’s complete REF submission are allowed to be non-compliant with the open access policy.29 This small wiggle room is afforded for cases where an output might need to be published in a journal that best fits the research but the journal is not open access and does not allow for the deposit of an accepted manuscript in an institutional repository. In this case, the non-compliant article still could be included in a university’s submission under the 5 percent of exceptions. Respondents stressed that, although these exceptions existed, they did not widely pub- licize them to faculty. Most were cautious with exceptions so that the university would have room later to include a handful of important non-compliant outputs or have space if outputs thought to be compliant proved otherwise. There’s a 5 percent acceptable non-compliance rate …and, you know, we don’t tell our faculty anything about that. Carrots and Sticks 329 The university doesn’t want to get anywhere near that figure because that’s five percent of the entire university return …the motivation is to come nowhere near the five percent cap. Where the publisher absolutely disallows inclusion, [and] there’s going to be an issue with making it open access via green or gold, then yes, where there is no other wiggle room whatsoever and it’s wanted for the REF because it’s that important, that’s where the 5 percent exception will come in. However, two librarians described their universities as consciously using the 5 percent exemption to include as many research outputs as possible, while a librarian at a different university described the practice of trying to max out the 5 percent exception rate as a “dodgy game to be playing.” Effects of the REF 2021 Open Access Policy Increase in UK Research Made OA REF 2021’s most important effect on scholarly communications was the dramatic increase in the amount of UK-based research made openly accessible; however, respondents also noted that the benefits of the infrastructure created to accommodate REF 2021 extend beyond the assessment period and have the potential to grow. The entire landscape has seen positive changes. I have only seen it in regards to content becoming available…. The knock on effect of the open access policies [is] on research data, inclusion, and reproducibility…. It’s a bit of a wedge. It opens up the scholarly publications environment and the scholarly communications environment. It’s hugely increased the interest and infrastructure for green. It means that we have built up a huge infrastructure around repositories and green open access that wasn’t there before. Effectively, yeah, it seems that the REF has been either the carrot or the stick that was required to get things moving…. It’s actually helped institutions, maybe es- pecially smaller institutions, focus themselves to create proper repositories and archive stuff properly. And then, later on, they brought in these mandates which mean that people are more proactively making their research more discoverable, more usable, more readable. The UK’s gains in open access were overwhelmingly viewed as a positive in terms of broadening availability and increasing the impact of research. Those interviewed unani- mously felt that the REF 2021 open access policy achieved its goal of breaking down barriers to research and, in so doing, expanded the reach of UK-based research to a wider and more diverse global audience. Educating Researchers about Open Access Mixed opinions were expressed, however, about whether the REF policy made a difference in researchers’ understandings of open access and scholarly publishing. Some respondents felt 330 College & Research Libraries May 2023 strongly that going through the REF 2021 process had prompted researchers to think more about how their work is disseminated. I think what it’s done is raise the profile of what open access publishing is, whether that’s through a repository or a journal…. We took some of the key questions to our research committees, one of which was around a rights retention strategy and about whether there should be embargoes on the final version of record. And, across the board, our academics all said, “No, we shouldn’t be having to sign over our copyright.” I think that’s partly probably because of REF, because open access just had to be in people’s heads…. I’d suspect we wouldn’t have had the same reaction eight years ago. Other respondents felt similarly but qualified their thoughts by saying that the process of making work comply with REF sometimes overshadowed wider and more thoughtful consid- erations of open research. One librarian, in a very telling statement, said, “When a measure is being used as target, it stops being a good measure.” Similarly, another librarian commented, I think what REF were trying to do is very admirable. What they wanted to do was to try and drive forward the open access agenda by bringing in this policy, this man- date, to be included in REF. I think that’s admirable. I think it did help. It has helped transform the movement, somewhat, but I think also it’s kind of been a little tiny bit detrimental in the way that people think, “Oh, it’s not for REF, then we don’t need to worry about open access” because they kind of feel that they’re just thinking about the policy and they’re not thinking about the bigger or the wider implications relat- ing to open access and the benefits for them and their research and the wider public. There was a tension in responses between those who viewed REF 2021 as an exercise that engaged and educated researchers on issues of access and publication and others who viewed REF 2021 as an exercise that blinkered researchers and encouraged them to think solely about compliance. Two Tracks A small number of librarians described a third effect of REF 2021 as the creation of two “tracks” for research outputs: one track for outputs likely to be included as part of the university’s REF submission and another track for outputs unlikely to be selected. By identifying research outputs as either likely or unlikely to end up as part of the university’s REF submission, researchers could see their departments allocate resources and attention accordingly. One librarian called this “teaching to the test” and clarified by saying, Your head of department will be asked early on in the cycle to highlight outputs that are likely. They’ll pick out across the department that this person is working on this project and it’s likely to be a REF output and …if somebody doesn’t think they’re going to be a ‘good’ research output, they’re not given the time to work on them…. That would probably be pressure on your humanities researcher who is Carrots and Sticks 331 producing that magnum opus to push it into a REF-shaped or REF-timed output or else how could they justify their position? While the “two tracks” problem was identified by a small number of schools, more than one respondent noted that making “for the REF” and “not for the REF” designations early in a project’s development could have a number of important consequences. Such a designation could change the attention a researcher pays to open access, dictate the authors publication choice, affect aspects of the research itself, or, at worst, have implications for a researcher’s career. Conclusion Study Limitations This study represents an in-depth qualitative approach with a small sampling of librarians and information professionals throughout the UK. Further reflective work could extend to a greater number of respondents and analyze response data for trends based on location, institu- tion size, or REF score. Further, the study focuses solely on library support for the “outputs” section of REF 2021 and does not consider the ways in which libraries supported the “impact case studies” or “research environment” sections. Future research could include attention to these sections and incorporate responses from REF administrators. Lessons from REF 2021 The REF 2021 open access policy undeniably accomplished what it set out to do by mak- ing open access a priority for universities and requiring engagement with the scholarly publishing environment. Mandating open access on an aggressive three-month time frame created a green open access infrastructure across the UK that required new technology and workflows to be put in place. And yet, REF 2021 did not incentivize a move to open access; rather, it threatened to penalize those who failed to comply or did so poorly. As one librarian put it, The REF has been the big one, in which case there’s quite a significant differ- ence in the amount of funding that the university can secure. When funding is at stake, then, yes, that’s not a carrot. That’s a very large stick, and it’s an effective one. The metaphor of sticks and carrots came up in multiple interviews and seems to capture a feeling that, while REF 2021’s open access policy was punitive to universities, it got the job done where incentivizing and encouraging had failed. The punitive “stick” approach did, however, present numerous challenges to universities. Respondents expressed in detail the complexities associated with creating and maintaining systems for deposit, outreach, and compliance as well as the added burden placed on researchers and libraries. Going forward, REF 2021 infrastructure will be built on to push the boundaries of open access even further. UKRI’s updated open access policy foreshadows the demands of the next REF, which appear likely to include instituting expectations for immediate open access at time of acceptance and requiring scholarly monographs to be made open.30 This means new pressures placed on libraries, of which respondents were well aware. 332 College & Research Libraries May 2023 Big challenges ahead. That’s quite a scary thing to think about. Trying to imple- ment that here. Just getting people to embrace zero embargo. That’s going to be really difficult. It’s going to be a lot of work educating them [and] engaging with our community and getting that message out there. I think books are definitely going to be the next stage. I think that’s a massive question, though. There’s just so much unanswered about a transition to [OA] books or the scholarly monographs. I think we do know there’s definitely going to be changes and monographs are likely to be included in the next REF. Respondents in the study also identified an increased focus on open data as an expected change in the next REF. These anticipated new REF expectations mean that librarians and research information managers in the UK do not anticipate respite any time soon from ac- commodating REF’s open access demands. Academic librarians outside the UK would do well to take note of the REF 2021 open access policy, the dramatic gains made in the percentage of UK-based research made open, and the staggering amount of work undertaken by institutions to make it happen. Recent developments such as White House Office of Science and Technology Policy’s August memo titled “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research”31 underscore how important it is that US academic librarians stay informed of responses to policy-driven open access in other countries. REF 2021 illustrates that such responses require processes of invention, implementation, reassessment, and communication. Despite the dif- ferences in higher education funding, the lessons from REF 2021 serve as a starting place for libraries in the US to imagine a response to federally-mandated open access and all that will be necessary to accommodate it. Acknowledgements Thank you to those interviewed for this study. Your kindness and generosity are hugely ap- preciated. Carrots and Sticks 333 Appendix. Standard Interview Questions 1) Prior to REF 2021, what was the experience level of your faculty with making their work OA? 2) What structures did your university put in place to deal with number of deposits that were about to pour in? 3) What outreach occurred before and during the REF 2021 assessment period? 4) Do you feel that the REF 2021 open access policy would have been possible without the prior funder requirements from funders like UKRI (formerly RCUK) and the Well- come Trust? 5) How did you account for articles already made OA through an APC (gold OA)? 6) How did you measure OA compliance? 7) What changes in scholarly publishing, if any, have happened or are likely to happen because of the REF 2021 open access policy? Notes 1. United Kingdom Research and Innovation, “What Is the REF? - REF 2021,” Higher Education Funding Council for England, accessed November 2, 2021, https://www.ref.ac.uk/about/what-is-the-ref/. 2. United Kingdom Research and Innovation, “Guidance on Submissions (2019/01) - REF 2021,” Higher Edu- cation Funding Council for England, accessed November 3, 2021, https://www.ref.ac.uk/media/1447/ref-2019_01- guidance-on-submissions.pdf. 3. United Kingdom Research and Innovation, “Submissions Made to REF 2021 - REF 2021,” Higher Education Funding Council for England, accessed November 3, 2021, https://www.ref.ac.uk/news/submissions-made-to- ref-2021/. 4. United Kingdom Research and Innovation, “Guidance on Submissions (2019/01) - REF 2021.” 5. Centre for Science and Technology Studies, “CWTS Leiden Ranking,” Leiden University, accessed No- vember 3, 2021, http://www.leidenranking.com. 6. United Kingdom Research and Innovation Council, “Pathways to Impact,” accessed November 4, 2021, https://www.ukri.org/councils/nerc/guidance-for-applicants/what-to-include-in-your-proposal/pathways-to- impact/. 7. John Holmwood and Chaime Marcuello Servós, “Challenges to Public Universities: Digitalisation, Com- modification and Precarity,” Social Epistemology 33, no. 4 (2019): 309–20, https://doi.org/10.1080/02691728.2019.163 8986. 8. Tina Basi and Mona Sloane, “Impact Is Crippling Higher Education. But It Is Still Part of the Solution,” LSE Impact Blog, London School of Economics and Political Science, April 23, 2018, https://blogs.lse.ac.uk/im- pactofsocialsciences/2018/04/23/impact-is-crippling-higher-education-but-it-is-still-part-of-the-solution/. 9. John Holmwood, “Open Access, ‘Publicity,’ and Democratic Knowledge,” in Reassembling Scholarly Com- munications: Histories, Infrastructures, and Global Politics of Open Access, ed. Martin Paul Eve and Jonathan Gray (Cambridge: MIT Press, 2020), 186, https://doi.org/10.7551/mitpress/11885.001.0001. 10. Alis Oancea, “Research Governance and the Future(s) of Research Assessment,” Palgrave Communications 5, no. 1 (2019): 1–12, https://doi.org/10.1057/s41599-018-0213-6. 11. Julie Rowlands and Susan Wright, “Hunting for Points: The Effects of Research Assessment on Research Practice,” Studies in Higher Education 46, no. 9 (2021): 1801–15, https://doi.org/10.1080/03075079.2019.1706077. 12. Ian James Kidd, Jennifer Chubb, and Joshua Forstenzer, “Epistemic Corruption and the Research Impact Agenda,” Theory and Research in Education 19, no. 2 (2021): 148–67, https://doi.org/10.1177/14778785211029516. 13. Miranda Fricker, Epistemic Injustice: Power and the Ethics of Knowing (Clarendon Press, 2007). 14. Martin Paul Eve and Jonathan Gray, Reassembling Scholarly Communications: Histories, Infrastructures, and Global Politics of Open Access, (Cambridge: MIT Press, 2020), https://direct.mit.edu/books/book/4933/Reassembling- Scholarly-CommunicationsHistories. 15. Holmwood, “Open Access, ‘Publicity,’ and Democratic Knowledge.” 16. Richard Poynder, “Open Access and the Research Excellence Framework: Strange Bedfellows Yoked Together by HEFCE,” The Winnower (2015), https://thewinnower.com/papers/314-open-access-and-the-research- https://www.ref.ac.uk/about/what-is-the-ref/ https://www.ref.ac.uk/media/1447/ref-2019_01-guidance-on-submissions.pdf https://www.ref.ac.uk/media/1447/ref-2019_01-guidance-on-submissions.pdf https://www.ref.ac.uk/news/submissions-made-to-ref-2021/ https://www.ref.ac.uk/news/submissions-made-to-ref-2021/ http://www.leidenranking.com https://www.ukri.org/councils/nerc/guidance-for-applicants/what-to-include-in-your-proposal/pathways-to-impact/ https://www.ukri.org/councils/nerc/guidance-for-applicants/what-to-include-in-your-proposal/pathways-to-impact/ https://doi.org/10.1080/02691728.2019.1638986 https://doi.org/10.1080/02691728.2019.1638986 https://blogs.lse.ac.uk/impactofsocialsciences/2018/04/23/impact-is-crippling-higher-education-but-it-is-still-part-of-the-solution/ https://blogs.lse.ac.uk/impactofsocialsciences/2018/04/23/impact-is-crippling-higher-education-but-it-is-still-part-of-the-solution/ https://doi.org/10.7551/mitpress/11885.001.0001 https://doi.org/10.1057/s41599-018-0213-6 https://doi.org/10.1080/03075079.2019.1706077 https://doi.org/10.1177/14778785211029516 https://direct.mit.edu/books/book/4933/Reassembling-Scholarly-CommunicationsHistories https://direct.mit.edu/books/book/4933/Reassembling-Scholarly-CommunicationsHistories https://thewinnower.com/papers/314-open-access-and-the-research-excellence-framework-strange-bedfellows-yoked-together-by-hefce 334 College & Research Libraries May 2023 excellence-framework-strange-bedfellows-yoked-together-by-hefce. 17. Martin Paul Eve, “Open Access and Neoliberalism,” Social Epistemology Review and Reply Collective 9, no. 1 (2020): 24–25, https://social-epistemology.com/2020/01/21/open-access-and-neoliberalism-martin-paul-eve/. 18. Hannah DeGroff, “Preparing for the Research Excellence Framework: Examples of Open Access Good Practice across the United Kingdom,” The Serials Librarian 71, no. 2 (2016): 96–111, https://doi.org/10.1080/036152 6X.2016.1196634. 19. Drahomira Herrmannova, Nancy Pontika, and Petr Knoth, “Do Authors Deposit on Time? Tracking Open Access Policy Compliance,” in 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL), (2019), 206–16, https:// doi.org/10.1109/JCDL.2019.00037. 20. Abdelghani Maddi, Esther Lardreau, and David Sapinho, “Open Access in Europe: A National and Re- gional Comparison,” Scientometrics 126, no. 4 (2021): 3131–52, https://doi.org/10.1007/s11192-021-03887-1. 21. Netta Weinstein et al., “The Real-Time REF Review: A Pilot Study to Examine the Feasibility of a Lon- gitudinal Evaluation of Perceptions and Attitudes Towards REF 2021,” SocArXiv Papers, (2019), 7, https://doi. org/10.31235/osf.io/78aqu. 22. Carolyn Ten Holter, “The Repository, the Researcher, and the REF: ‘It’s Just Compliance, Compliance, Compliance,’” The Journal of Academic Librarianship 46, no. 1 (January 1, 2020): 102079, https://doi.org/10.1016/j. acalib.2019.102079. 23. Higher Education Statistics Agency, “Where Do HE Students Study?,” accessed October 19, 2021, https:// www.hesa.ac.uk/data-and-analysis/students/where-study#provider. 24. Virginia Braun and Victoria Clarke, “Thematic Analysis,” in APA Handbook of Research Methods in Psychol- ogy, Vol 2: Research Designs: Quantitative, Qualitative, Neuropsychological, and Biological., ed. Harris Cooper et al. (Washington: American Psychological Association, 2012), 57–71, https://doi.org/10.1037/13620-004. 25. Virginia Braun and Victoria Clarke, “Using Thematic Analysis in Psychology,” Qualitative Research in Psychology 3, no. 2 (2006): 77–101, https://doi.org/10.1191/1478088706qp063oa. 26. Nicholas Joint, “Current Research Information Systems, Open Access Repositories and Libraries: AN- TAEUS,” Library Review 57, no. 8 (2008): 570–75, https://doi.org/10.1108/00242530810899559. 27. Ten Holter, “The Repository, the Researcher, and the REF.” 28. United Kingdom Research and Innovation, “UKRI Open Access Policy,” August 6, 2021, https://eprints. bbk.ac.uk/id/eprint/30952/1/2020-02-14-open-access-monographs-and-the-ref-after-ref-2021.markdown. 29. United Kingdom Research and Innovation, “Guidance on Submissions (2019/01) - REF 2021,” 5, Section 231. 30. United Kingdom Research and Innovation, “UKRI Open Access Policy.” 31. Office of Science and Technology Policy, “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research,” Federal Register, August 25, 2022, https://www.whitehouse.gov/wp-content/uploads/2022/08/08- 2022-OSTP-Public-Access-Memo.pdf. https://thewinnower.com/papers/314-open-access-and-the-research-excellence-framework-strange-bedfellows-yoked-together-by-hefce https://social-epistemology.com/2020/01/21/open-access-and-neoliberalism-martin-paul-eve/ https://doi.org/10.1080/0361526X.2016.1196634 https://doi.org/10.1080/0361526X.2016.1196634 https://doi.org/10.1109/JCDL.2019.00037 https://doi.org/10.1109/JCDL.2019.00037 https://doi.org/10.1007/s11192-021-03887-1 https://doi.org/10.31235/osf.io/78aqu https://doi.org/10.31235/osf.io/78aqu https://doi.org/10.1016/j.acalib.2019.102079 https://doi.org/10.1016/j.acalib.2019.102079 https://www.hesa.ac.uk/data-and-analysis/students/where-study#provider https://www.hesa.ac.uk/data-and-analysis/students/where-study#provider https://doi.org/10.1037/13620-004 https://doi.org/10.1191/1478088706qp063oa https://doi.org/10.1108/00242530810899559 https://eprints.bbk.ac.uk/id/eprint/30952/1/2020-02-14-open-access-monographs-and-the-ref-after-ref-2021.markdown https://eprints.bbk.ac.uk/id/eprint/30952/1/2020-02-14-open-access-monographs-and-the-ref-after-ref-2021.markdown https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf