27 A Cross-Institutional Study of eBook Demand- Driven Acquisition (DDA) Use and Efficacy of Eight Large Academic Libraries Kay Downey and Yin Zhang* This study is the first of its kind to analyze and compare demand-driven acquisition (DDA) ebook programs on a large scale by using eight academic libraries. The purpose is to understand which factors contribute to successful collection management prac- tices and sustainability. Study findings also offer insight into weeding practices and suggest that ebooks removed from the discovery pool too soon may impact service to library users. Furthermore, findings based on formula analysis show that return on investment (ROI) for serviceable content is better achieved through a sustained straight DDA model without short-term loans. Introduction In today’s academic environment, the DDA model is widely used among academic libraries for ebook purchasing and is evidence of libraries’ general transition to a “just in time” service model that meets the immediate needs of the user; libraries have been embracing this new acquisi- tion model and are also actively engaged in assessment efforts. A search through library and information science literature in the Web of Science core collection shows there have been 106 articles published on the topic since 2010 alone, with most studies being case studies sharing an individual program’s practices. However, there is a gap in the literature and a general lack of understanding of the best practices when it comes to assessing variables and considerations beyond individual programs. For this study, acquisition and usage data from eight academic institutions were collected to analyze and compare DDA programs. The data collected from each institution detail order and license information, bibliographic data, and usage data for each program. A survey was also used to gather information about the parameters and program variables of each DDA discovery pool. Common to all institutions was the use of ProQuest as the ebook third-party provider and EBSCO’s YBP Library Services as the DDA ebook jobber. Formula logic based on usage data was developed to properly compare ROI under various DDA model scenarios. This study is significant because it is the first large-scale study of its kind; it goes beyond single-program case studies and focuses instead on comparative, cumulative data across eight * Kay Downey is Assistant Dean for Collections & Technical Services in the University Libraries at Kent State University; email: mdowney1@kent.edu. Yin Zhang is Professor in the School of Information at Kent State Uni- versity; email: yzhang4@kent.edu. ©2020 Kay Downey and Yin Zhang, Attribution-NonCommercial (http:// creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:mdowney1@kent.edu mailto:yzhang4@kent.edu http://creativecommons.org/licenses/by-nc/4.0/ http://creativecommons.org/licenses/by-nc/4.0/ 28 College & Research Libraries January 2020 large academic library institutions. Its purpose is to further research on DDA use and efficacy by identifying possible measures of a successful DDA program, as well as potential com- monalities in sale and usage trends that could help in evaluating long-term ROI. Specifically, the present study seeks to answer the following questions regarding the value of DDA ebook collection services: • To what extent do ebooks receive continued use after the initial trigger for purchase or loan? Is a patron’s usage-initiated trigger a random act or an indicator of need that leads to sustained use? • How does ebook use over time affect various DDA program implementation options and associated costs? Answers to these questions will help identify the factors and considerations associated with implementing a DDA program in a large academic library, and commonalities between programs that could help librarians choose the ebook DDA business model that works best for their library. Literature Review As DDA models persist over time, the body of literature about DDA grows as well. Although there is a large volume of published articles that describe DDA programs in academic libraries, most are case studies that describe the various aspects and observations of a single library’s experience in terms of subject analysis, usage, and cost. There are fewer studies, however, that focus on comparative data across various types of ebook acquisition models and none that compare data across individual academic libraries. For the purposes of this study, we reviewed literature that provides an overview of varying DDA models and studies that examine data collected over a number of years to gather information about ebook use over time. In 2014, the National Information Standards Organization (NISO) published Demand- Driven Acquisition of Monographs, which provides recommendations for librarians and ebook providers regarding best practices, standard industry definitions, and consistent methods for implementing DDA programs (www.niso.org/workrooms/dda).1 The NISO document covers all types of libraries and applies to both ebooks and print. The DDA model, if implemented well, allows libraries to provide “just in time access” at the point of need and to provide ac- cess to more content than would be possible under the traditional purchase model. The NISO publication suggests that a “flexible model for DDA that works for publishers, vendors, aggre- gators, and libraries be developed to allow libraries to develop DDA plans that meet differing local collecting and budgetary needs…” (vii).2 The document also emphasizes the value of defining metrics to conduct meaningful program assessment. Analyzing various aspects of both pre- and post-purchase use is essential for such an evaluation. Kent State University Libraries (KSUL) began using the DDA ebook purchasing model in January 2012. Studies about the KSUL program have focused on variables of the DDA model such as log analysis in ebook patron-driven acquisitions,3 comparison of print book and DDA ebook acquisition and use,4 and longitudinal comparisons of KSUL’s existing DDA program against short-term loan business models. Conclusions drawn from the longitudinal study determined that 80 percent of DDA ebooks purchased within one year would have received enough use over time to trigger a purchase. Therefore, had the short-term loan (STL) option been used with the added cost of STL rentals fees, KSUL would have paid substantially more each year for the same content. Other data-driven studies based on ebook usage patterns http://www.niso.org/workrooms/dda A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 29 have also been conducted by the KSUL team to help identify which DDA model is most cost effective,5 which led us to ask if the conclusions drawn from the KSUL DDA are relevant in other university libraries. One noteworthy study from Hong Kong University of Science and Technology (HKUST) Library describes and examines three of their concurrent demand-driven acquisition programs for ebook acquisitions.6 These three programs included an Ebrary DDA, a Wiley Evidence Based Acquisition (EBA) program, and a consortia DDA pilot with JSTOR. Their overall objective was to increase the library’s ebook collection and use. HKUST implemented their Ebrary DDA program in 2012 on the Ebrary platform and managed it through the YBP library services. After the HKUST DDA was in effect for two years, they conducted usage analysis and reviewed the top publishers. Their findings prompted them to start an evidence-based ebook acquisition program with Wiley. Subsequently, HKUST collaborated with consortia libraries to initiate a JSTOR DDA program. They examined usage and subject analysis for each program and reported good use of the Ebrary and Wiley ebooks, while the consortia JSTOR program received comparatively low use for their library. They speculated that per- haps the consortia DDA needed more time or that the subject categories did not align with their research strengths. They concluded that strategic use of DDA programs to supplement standard acquisition models increased the size and use of their ebook collection while also providing ebook content at the point of need. The similarities between HKUST Ebrary program and KSUL’s program has been of interest to researchers at Kent State.7 Like KSUL, HKUST’s analysis of top used publishers, overall usage, and trigger outcomes during a two-year period (2012–2014) closely aligned with KSUL DDA findings. For example, the 2014 KSUL study that compared ebook usage from a DDA discovery pool to print book circulation showed that KSUL usage data and top publishers aligned closely with HKUST outcomes. Both indicated that about 10 percent of the DDA collection received some use and about 5 to 6 percent of the ebooks in the discovery pool were triggered for purchase. A subsequent 2017 KSUL analysis also shows the added value of the longitudinal study, indicating that DDA ebook use over a 5-year duration increased to 16.2 percent for ebooks in the discovery pool, with 7.3 percent of the DDA ebooks receiving enough use to trigger a purchase.8 It would be interesting to know if the HKUST program usage would increase likewise over time. Another study, conducted at Harold B. Lee Library (HBLL) at Brigham Young University, measured the effects of introducing several different types of acquisition models on overall expenditures for monographs and the effect it had on their established approval plan.9 Like HKUST and KSUL, HBLL used YBP and Ebrary to implement their DDA program. HBLL’s DDA program began with a pilot in 2011; thus, they were able to conduct a longitudinal study of the use data over a span of six years from 2011 through 2016. In 2011, HBLL began adding ebooks that matched their approval profile to their discovery pool. The number of available DDA ebook titles grew from 2,595 in 2011 to nearly 41,000 in 2016. In 2013, HBLL opted to incorporate an STL option that, for them, constituted 2 STLs with a purchase on the third use. They based the decision to use STLs upon their ebook use data, which showed that less than 9 percent of all used titles received enough use to warrant a 3-trigger purchase, which is contrary to KSUL findings. In addition to their DDA program, HBLL also moved toward a more comprehensive ebook collections strategy by purchasing publisher ebook packages. They provided access to Elsevier and Wiley ebooks via an EBA model and initiated several 30 College & Research Libraries January 2020 subscription ebook packages. Their study emphasized the expansion of access to many more titles than was previously possible by shifting monograph funds from approval expenditures to various ebook models. In another study conducted at North Carolina State University (NCSU), researchers ex- amined the average length of time it takes for an ebook to trigger and whether that data can be used to predict future use.10 Knowing whether or not titles receive continued use after the initial trigger is useful in determining the type of DDA model that will work best over time. NCSU combined data from vendor reports and their own integrated library system (ILS) to analyze their program’s performance for patterns that may predict trigger activity. Their data showed that, over time, the number of triggered titles increased steadily; however, they were unable to correlate this increase with the number of titles available in the discovery pool. In fact, the percentage of triggered titles actually decreased over time when compared to the size of the discovery pool. Reports from this study indicated that the average time for a title to trigger was seven months for science and technology ebooks and 13 months for humanities and social science ebooks. Comparatively, average trigger time for the KSUL DDA program was about 1.5 years. NCSU also reported that triggered titles received significant continued use after the initial trigger, which is also true for KSUL. Another area of interest in DDA research is the analysis of collaborative collection strat- egies that apply the DDA model in library cooperatives. Consortia such as the Colorado Alliance of Research Libraries, State University System of Florida, ConnectNY, HKUST, and the Orbis Cascade Alliance are just a few examples of collectives that initiated shared DDA programs for consortia.11 The DDA model is complicated and challenging to administer. When cooperatives employ DDA, they must reach a consensus regarding discovery parameters, re- cord configuration, workflow, and cost distribution. Whether or not the collection strategy is weighted more toward ownership or access will also help determine the inclusion of STLs and supplemental ebook subscriptions. Although program differences between cooperatives may influence the cost benefit for certain schools, most literature on the subject reports an overall good ROI based on use in relation to Full Time Enrollment or investment dollars. However, this is only one measure of success; it would be helpful to evaluate other factors in the coop- erative schema such as data that shows which ebooks, after the initial trigger for purchase or loan, receive continued use over time. Reviewing longitudinal usage data within a cooperative schema might also be helpful for identifying commonalities and investment costs over the life of the DDA program. Because of this limitation, the literature review is not contextualized to respond and/or address questions and debates from all the literature available about DDA. The case studies represented here are typical of overall literature that describes evidence of ebook acquisition models. They follow the NISO Standards assessment guidelines when measuring outcomes such as the effect on overall collection expenditures, patterns that may predict trigger activity, subject analysis, and cost per use comparisons with other ebook acqui- sition business models. Literature that analyzes data in terms of a single library’s experience has merit; however, it does not encapsulate the bigger picture of ebook DDA performance commonalities and variances across the library industry. Until now, DDA studies have focused on the single institution, leaving a research gap in assessing true best practices as evidenced by data-driven comparative research. By investigating DDA data and conducting comparative analysis of programs across multiple academic institutions, this study begins to fill the gap identified in the literature review. Research conducted in this study will expand literature of A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 31 DDA programmatic practices, using this comparative approach, to enhance knowledge and understanding of ebook DDA business models. Methods Sample, Data Collection, and Data Processing This study first identified eight large academic libraries with a DDA ebook program offered via Ebrary. The basic profiles of the academic institutions as reported in the Integrated Post- secondary Education Data System (IPEDS) are listed on table 1. Permission was obtained from these institutions to collect data including DDA ebook program information—obtained via an email survey of the participating programs—and ebook reports, available in Ebrary and Ebook Central (EBC), that contain details about an institution’s ebook DDA acquisition and usage. These are reported in the Title and Trigger Reports from both Ebrary and EBC. The title and trigger reports also provide usage data, details of DDA transitions, license and cost details, publisher information, and trigger events. Ebrary documentation was also used to provide current license restrictions and short-term loan rates for all publishers in the sample. For comparison purposes, the selected sample of DDA programs includes both com- monly used DDA-derived business models: Four programs using the DDA model with the STL component and four programs using straight DDA purchases with no short-term loans. The STL model is structured to trigger an ebook purchase only after a set number of STLs have been triggered. The final purchase cost for an ebook in the STL model is the sum of each rental fee plus the list price of the ebook. Comparison and analysis of the sample programs help discover similarities and differences between the programs and the two DDA business models. TABLE 1 IPEDS 2017–2018 Provisional Release Data Institution # 12-Month FTE Total Enrollment Holdings Digital Books Total Materials Services Expenses Total Library Expenses Public or Private School Carnegie Classification 1 22,247 1,094,800 $5,278,470 $13,566,334 Public Doctoral Universities 2 58,904 1,342,200 $19,649,112 $53,188,722 Public Doctoral Universities 3 14,898 1,730,794 $17,645,787 $36,644,375 Private Doctoral Universities 4 28,080 1,526,730 $11,268,057 $21,361,085 Public Doctoral Universities 5 27,079 1,642,337 $10,834,898 $26,528,613 Public Doctoral Universities 6 1,573 135,157 $101,330 $296,371 Private Master’s Colleges & Universities 7 22,889 1,087,218 $ 3,747,052 $10,398,049 Public Master’s Colleges & Universities 8 19,384 661,724 $7,456,693 $15,697,391 Public Doctoral Universities 32 College & Research Libraries January 2020 The collected ebook data reports were first imported into a Microsoft Access database by assigning an institutional ID to each report. For each institution, various data reports were then mapped based on Ebrary ebook IDs shared across reports. This way, an ebook’s acquisi- tion and use could be tracked and analyzed over time. Additionally, inconsistent data were identified, processed, and normalized before analysis was carried out. Measures and DDA Model Scenarios One important note regarding the definition of a trigger for STL and purchase involves the migration from the Ebrary platform to the new EBC platform. The ProQuest Ebrary trigger threshold is 10-10-1-1-1, which means any copy, print, or download may serve as a trigger, while 10-page turns or 10 consecutive minutes of use may also serve as a trigger. The new platform redefines the trigger threshold to five minutes of use or one copy, print, or download. For this study, however, the 10-10-1-1-1 trigger threshold was used because that is the paradigm associ- ated with the historical data used in this study. Another important concept is the definition of a Session. One Session is defined as the number of times a title is opened and incurs any use. An important measure of a successful DDA program is continual usage of an ebook after the initial trigger for purchase or loan. This is used to determine if a patron’s usage-initiated trigger is simply random or an indicator of intentional continuous use. To do that, a cohort- based longitudinal approach was adopted. First, the baseline of the first complete year ebook cohort from the trigger report was identified for each program. Second, usage was tracked over the program duration as reflected in the title reports and delineated by program year. As shown in the program survey of the eight institutions, the DDA programs vary greatly due to many variables. A scenario analysis based on actual ebook use under different DDA business models was conducted to find out the best option for each program. For this com- parison, two straight DDA programs and two STL-based programs were chosen. The choices were based on several considerations and factors to make the programs comparable: (1) the programs were not weeding their DDA pool; (2) the programs had comparable cohort and sample sizes; and (3) the minimum program duration for the cohort was at least three years. Three common scenarios were chosen for comparison. These scenarios were the most com- mon purchase licenses among the eight study programs: Scenario 1 (S1), defined as a direct single user purchase option (SUPO) purchase; Scenario 2 (S2), defined as a one 1-day STL followed by SUPO purchase; and Scenario 3 (S3), defined as three 1-day STLs followed by SUPO purchase. The formula logic for different scenarios was based on actual ebook use data: • Step 1 determined which titles were eligible for STL based on Available License data in the Trigger Report for each title. • Step 2 determined the Raw Trigger Total based on usage. For titles that were STL eligible, the Raw Trigger Total for each title was calculated as follows: N of Views/10 + nCopy + nPrint + nChapter DL + nFull DL. • Step 3 involved calculating the No. of Triggers statistic based on the combination of Step 2 calculations and User Sessions for each Scenario. To do this, the Raw Trigger Total was compared with the corresponding number of user sessions to determine the likely number of triggers that could be applied to each of the different scenarios. For example: 1 trigger = a purchase in S1; 1 STL in S2 and S3 2 triggers = a purchase in S1; 1 STL+ purchase in S2; 2 STLs in S3 A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 33 3 triggers = a purchase in S1; 1 STL+ purchase in S2; 3 STLs in S3 4 or more triggers = a purchase in S1; 1 STL+ purchase in S2; 3 STLs + purchase in S3. • Step 4 determined the cost of each title based on the single user publisher list price for ebooks ineligible for STL, the mapped outcome of STL eligible ebooks from step 3, the publisher’s STL rate, and single user publisher list price. For example, for an ebook published by Princeton University Press, whose 1-day STL rate is 30 percent of list price and the list price is $100, if the ebook had 2 triggers, then □ In S1, it would be purchased at list price at $100 □ In S2, it would fall in 1 STL+ purchase for a cost of $130 □ In S3, it would be 2 STLs for a cost of $60 On the other hand, if the book had 4 or more triggers based on its usage, then □ In S1, it would be purchased at list price at $100 □ In S2, it would fall in 1 STL+ purchase for a cost of $130 □ In S3, it would be 3 STLs plus purchase for a cost of $190 • Finally, in step 5, the total cost and percentage of STL vs. Purchase was calculated under each scenario for each program. The results were summarized in a side-by-side com- parison. Results/Findings Overview of the DDA Programs A preliminary analysis of the sample ebook programs was conducted to obtain some basic program information as reflected in various ebook reports. The results, summarized in table 2, show a wide range of differences in DDA programs among the sample institutions. For TABLE 2 Overall Program Information DDA Programs without STL Inst. ID # of Program Years STL Default Purchase Model # of eBooks Purchased Total Program Cost Avg. Cost/ Title Max. Title Exp. Min. Title Exp. 1 5 years no STL PSUPO 5,701 $578,018 $101 $2,680 $4.99 2 4 years no STL PMUPO 4,170 $549,050 $132 $3,830 $7.49 3 4 years no STL PSUPO 486 $42,854 $88 $347 $2.99 4 5 years no STL PSUPO 2,007 $193,019 $96 $1,785 $4.99 DDA Programs with STL # of Program Years STL Default Purchase Model # of Transactions (STLs & Purchases) Total Program Cost Avg. Transaction Cost Max Transaction Cost Min Transaction Cost 5 3 years 1–3 (1-day) PSUPO 3,267 $103,683 $32 $225 $0.70 6 4 years 1 (7-day) PSUPO 2,656 $170,935 $64 $747 $1.49 7 4 years 3 (1-day) PMUPO 4,170 $164,055 $39 $450 $1.20 8 5 years 3 (1-day) PSUPO 5,300 $153,454 $29 $585 $0.30 34 College & Research Libraries January 2020 example, some institutions started their programs as early as 2009 and some as late as 2014; some programs include STLs while others do not. The size and cost of the program also var- ies between institutions. A survey was sent to the sample institutions to obtain the DDA program information needed to further investigate some of the results and differences among programs that were observed in the preliminary analysis of the ebook reports. The survey included questions about weeding, publisher exclusions, price cap, and general comments about the institution’s own program. The survey results, summarized in table 3, reveal significant variance in the implementation of DDA programs at the sample institutions. Institutions 1 through 4 use the straight DDA model without STLs, while institutions 5 through 8 use a DDA model that includes STLs. Some institutions reported weeding the discovery pool and some did not, and TABLE 3 eBook Program Survey Results Institution ID 1 2 3 4 5 6 7 8 DDA Vendor YBP YBP YBP YBP YBP YBP YBP YBP Academic Complete Subscription NO NO NO NO NO YES YES YES STL 0 0 0 0 1–3 STL: 1 day(s) 1 STL: 7 day(s) 3 STLs: 1 day(s) 3 STLs: 1 day(s) Default Purchase Model PSUPO PMUPO PSUPO PSUPO PSUPO PSUPO PMUPO PSUPO Program Start Date (including pilots) Jan-12 Jul-09 Mar-13 Jun-11 Feb-14 Aug-11 Jul-12 Jun-11 Deduplicates against Print Holdings YES NO YES NO YES YES NO YES Publication Year Limitations 2009 2010 no limit no limit no limit 2014 2010 2010 Weeding (Other Than Publisher Deletes) NO SOME YES YES NO YES YES NO Restricted Content Level YES YES YES YES YES YES YES YES Price Cap $200 $350 no limit no limit $250 $150 $250 $200 Foreign Language Exclusion YES NO NO NO YES YES YES YES Publisher Exclusions YES YES NO YES YES YES YES NO Discovery Pool Automated Automated Selector Mediated Selector Mediated Automated Automated Automated Automated and Selector Mediated A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 35 some discovery pools were automated via the YBP approval plan, while others were entirely selector mediated. Continued eBook Use over Time To address the first research question about whether ebooks receive continued use after the initial trigger, ebook cohort usage data for the complete year was identified and tracked over the program duration. One approach to determine this measure is to examine any comparable ebook cohort usage data for the first complete year over the course of program duration. To do that, the first complete year ebook cohort was identified from the trigger report. Usage was then tracked over the program duration as reflected in all the title reports by program year. Depending on the purchase license and the institution’s DDA model, a triggered title in the cohort may experience only STL(s), a single purchase under a particular license, or a com- bination of both STL(s) and a purchase. In this study, we applied three measures to examine ebook use over time: percent used, user demand, and use intensity. The findings are reported and presented in the following sections. Triggered Ebook Use over Time: Percent of Being Used Ebook use over time was measured by the percentage of ebooks that continue receiving use over program duration. The results of initial ebook cohort’s use over time at the eight programs are shown in figure 1. All ebooks in their respective program’s first-year cohort received use in their trigger year; but overall there was a decline in use of the titles after- ward. Despite this, the results show around 20 percent of ebooks receiving steady usage even in their fifth year. FIGURE 1 Comparable First Cohort Use over Time: Percentage of Being Used 36 College & Research Libraries January 2020 Triggered Ebook Use over Time: User Demand Another measure to gauge use over time is the number of user sessions, which demonstrates the user demand. As shown in figure 2, the first-year cohort received use throughout the pro- gram duration, although it varied by program size, with a decline in user demand over time. FIGURE 2 Comparable First Cohort Used over Time (Total User Sessions): Demand No. of Triggered Titles in the First Comparable Complete Year Cohort FIGURE 3 Comparable First Cohort Use over Time: Use Intensity Average User Sessions per Title A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 37 Triggered Ebook Use over Time: Use Intensity The third measure to gauge use over time was the number of user sessions per used title, which shows the intensity of use. As shown in figure 3, the intensity of use remains strong across all programs for the invested ebooks with at least 3 user sessions per title even into the fifth year. Scenario-Based Comparisons To address the second research question, regarding the effects of ebook use on various DDA program implementation options and associated costs, a scenario-based approach was adopted to effectively and appropriately assess and compare costs. For this study’s comparison, we chose four institutions’ DDA programs, including two straight DDA programs and two STL programs. The choices were based on several factors: 1) the programs are not weeding their DDA pool; 2) the programs have comparable cohort and sample size; and 3) the minimum program duration for the cohort should be at least three years. As detailed in the Methods section, we chose the three most common purchase licenses in the eight study programs for comparison: Scenario 1: Direct SUPO purchase; Scenario 2: A single 1-day STL followed by SUPO purchase; and Scenario 3: Three 1-day STLs followed by SUPO purchase. STL/Purchase Scenario Comparisons Based on Usage The comparisons of the three scenarios based on usage are summarized in table 4, which also provides a detailed breakdown of STLs against purchases for the programs, with the percent- age of purchased ebooks highlighted under each scenario. For institutions 1 and 2, which are under straight purchase models for their ebook pro- grams, more than 90 percent of their initial cohort would have been purchased under S2; about 70 percent of their initial cohort would have been purchased under S3, and any additional triggers of the cohort titles that are in the STL stage may lead to purchase down the road with additional use. It should be noted that more than 20 percent of their initial cohort are not STL- eligible per publisher license. Both institutions also had less than 10 percent of their initial cohort incur light use and stay in 1 STL stage. These findings show that these two institutions’ current ebook DDA straight purchase model makes sense based on their ebook use. On the other hand, for the two programs with STL options in their current practice (in- stitutions 5 and 8), the results are different. Institution 8 has about 50 percent of ebooks in the initial cohort that would have been purchased after going through 3 STLs; additionally, 84 percent of the cohort would have been purchased under S2 and 63 percent under S3. Institu- tion 5 is an outlier, with 22 percent of their initial cohort in 1 STL stage among all programs. Upon closer examination, two potential factors might contribute to the discrepancies, besides the fact that this program has the shortest program duration. First, the program migrated to Ebook Central during its third program year. Second, the use data from the program’s last year were obtained from both Ebrary Portal and Ebook Central, and the two ebook reports from two platforms were integrated to have the full-year usage of the cohort for the last year. Overall, the results show that, for S2 (a one-day STL followed by purchase), the major- ity of the initial cohort would have been purchased over the program duration. The ratio of the purchased titles ranges from 78 to 95 percent, with varied program duration from three to five years. Under S3’s three 1-day STLs and purchase models, results show that, for three programs, nearly two-thirds of the initial cohort would have been purchased over the program 38 College & Research Libraries January 2020 duration, and more than half of the initial cohort would have been purchased for the fourth program with a three-year program duration. Scenario Comparisons Based on Cost Table 5 summarizes the cost comparisons of the three scenarios across the programs with the most cost-effective option highlighted for each program. It should be noted that the expendi- tures are not the actual amount spent by each institution, but strictly scenario comparisons. This is because there are some factors that make it impossible to get the precise numbers. For example, the list price may change over time by publisher, the STL eligibility status may change, STL rates may change over time, or any number of other factors. In addition, some institutions may have negotiated discounted pricing. The comparisons show, with the exception of institution 5, that S1 (direct SUPO purchase) is the most cost-effective DDA model across the board based on the actual use of the initial TABLE 4 Scenario Comparisons Based on Usage Program Cohort Duration STL/Purchase Based on Usage # of eBooks % of the Cohort S2: % of STL vs. Purchase S3: % of STL vs. Purchase Institution 1 5-year program STL 1 STL 41 5% 5% 31% 2 STLs 118 13% 95% 3 STLs 117 13% Purchase 4 STLs and above 433 48% 69% STL not eligible 191 21% TOTAL 900 100% 100% 100% Institution 2 3-year program STL 1 STL 79 9% 9% 28% 2 STLs 101 11% 91% 3 STLs 82 9% Purchase 4 STLs and above 450 49% 72% STL not eligible 213 23% TOTAL 925 100% 100% 100% Institution 5 3-year program STL 1 STL 156 22% 22% 51% 2 STLs 132 18% 78% 3 STLs 80 11% Purchase 4 STLs and above 233 33% 49% STL not eligible 115 16% TOTAL 716 100% 100% 100% Institution 8 4-year program STL 1 STL 101 16% 16% 37% 2 STLs 69 11% 84% 3 STLs 65 10% Purchase 4 STLs and above 305 48% 63% STL not eligible 99 15% TOTAL 639 100% 100% 100% A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 39 cohort over the program duration. For institution 5, the 3-STL plus purchase would make sense among the 3 scenarios. Over time, the difference in costs among the three scenarios would increase because S1 cost is fixed, while titles that are at STL stage in S2 and S3 may continue to be used and could incur additional costs for STLs and even purchases down the road. As part of a closer examination of why institution 5’s program was different, we took a closer look at the 716 ebooks in this cohort and tracked their actual transactions over the program duration. The program’s actual transaction data shows that 20 ebooks in the initial cohort were purchased in the first year, 22 were purchased in the second year, and 68 titles were purchased in the third year with additional costs. The average cost per title owned to the initial cohort is $260.45, and the cost per transaction for the purchases and STLs was $40.01. Based on the evidence and overall trend of all programs in this study, the titles will continue to receive uses and trigger purchases, which will incur additional costs. Discussion The results of this study, in conducting these comparisons, show the value and importance of a longitudinal approach in DDA program assessment. Specifically, this study illustrates the value in examining DDA data over a sustained period of time to assert the bigger picture of ebook use. The data shows that, once an ebook is used, it tends to incur continued use in subsequent years. Therefore, the choice of acquisition model—whether or not the DDA model includes STLs—has consequences on the overall program cost from year to year. This study expands previous studies that have been largely based on individual programs, cross-examining multiple institutions for a more general assessment of the DDA practices. During this process, however, we discovered that the analysis process is data-rich, making it very time-consuming and labor-intensive. Moreover, the complexity of variables between programs made it difficult to compare data across multiple institutions. Some of the specific challenges of data processing, analysis, and comparison were impacted by the following fac- tors: 1) the change in platforms from Ebrary to Ebook Central and the subsequent use report changes; 2) the inconsistency in library practice from one year to the next; 3) publisher changes in price, STL eligibility, STL participation, and STL rate; 4) the time-consuming task of clean- ing publisher name variances; and 5) overall variables across institutions. In addition to these issues, the task of determining the best value and practice is made even more difficult by the fact that each DDA program varies and that each library has its own collection strategy. For example, libraries alter their programs based on ebook purchases, subscription packages, or a combination of both. Some libraries also weed their discovery pools while some do not. Funding shortfalls may impact the duration of the program or may promote cheaper leased content as the only affordable option. Some libraries even have a se- TABLE 5 Scenario Comparisons Based on Cost Institution ID Program Duration Cohort Title Count S1 Cost (Direct SUPO List Price) S2 Cost (1 1-day STL) S3 Cost (3 1-day STLs) Institution 1 5 Years 900 $108,186 $123,348 $130,941 Institution 2 3 Years 925 $114,448 $127,477 $140,486 Institution 5 3 Years 716 $73,526 $71,000 $67,181 Institution 8 4 Years 639 $75,550 $78,941 $85,322 40 College & Research Libraries January 2020 lector mediated DDA program in which the purchase decision is not entirely based on patron use. Nevertheless, the broad descriptive overview of this study illustrates the role DDA plays in the collection practice of each library and highlights the vast diversity between programs. The biggest limitation with this study’s analysis was the difference in weeding protocols. Frequent weeding of ebooks makes it impossible to track use over time; therefore, we were unable to assemble a solid cohort to evaluate usage data. In addition, one institution weeded all titles in the discovery pool annually and started with a new discovery pool each year. This factor limited the number of institutions eligible for the scenario analysis. However, ebook weeding protocols may be an avenue for future research. Some libraries, including KSUL, have reported that it takes more than a year for the average DDA ebook to be discovered; therefore, time in the discovery pool should be a factor when formulating weeding protocols. Ebooks removed from the discovery pool too soon may impact service to library users. Despite the challenges discussed above, this study illustrates a scenario analysis approach that could offer an “apple-to-apple” comparison in DDA program assessment and decision- making. The results from the analysis reveal that the STL business model may make sense initially, but the straight purchase DDA model has a greater return on investment in the long term for owned content. Therefore, for institutions running a sustained DDA program, it may make more sense to adopt the direct purchase DDA model without STLs, particularly if the collection strategy is weighed more toward ROI for ownership of content. Conclusions As the first large-scale longitudinal study of its kind to analyze and compare DDA ebook pro- grams across multiple academic libraries, this study shows that the ROI for owned content is better achieved through a sustained straight DDA model without short-term loans. However, based on our knowledge of the considerable variance in programs, we could also make the case that the best DDA model for any library really depends on the institution’s collection goals and financial means. As new ebook acquisition models like EBA and Access to Own are introduced and adopted, more comparison studies will be needed to identify measures of a successful program. Assessing actual evidence based on use data and employing longitudi- nal formulaic analysis provides a means for determining whether or not the program meets library goals and will help detect patterns that predict cost and return on investment. This study develops and illustrates a methodology that can be used for future studies to further understand factors that contribute to successful collection management practices and sustainability. This study also offers insight into weeding practices and warns against the hastened removal of ebooks. Furthermore, findings based on this formula analysis show that ROI for serviceable content is better achieved through a sustained straight DDA model without short-term loans. Future research with more DDA programs may further validate the findings and uncover additional factors and considerations, such as publisher and academic discipline, which influence the development of best practices in DDA implementation and assessment. A Cross-Institutional Study of eBook Demand-Driven Acquisition (DDA) Use and Efficacy 41 APPENDIX A. DDA Program Survey Questions Initial DDA Program Survey Questions November 2016 We would be requesting the following reports from ProQuest for the time period from the beginning of your DDA program through the present 1. All Purchased Titles: YBP Firm Orders and DDA Orders 2. DDA Trigger Report 3. BookReport2 4. Usage for DDA titles 5. YBP Firm Ordered titles that had no usage We would also like to know if your library 1. used the short-term loan option. If so, when did you start and are you still using STLs? 2. used ProQuest’s Academic Complete ebook lease program at any time. 3. uses YBP to manage DDA invoicing and records delivery. Follow-up Survey Questions April 2017 1. Do you exclude duplicate print holdings from the discovery pool? 2. Do you exclude titles published before a certain year? (For example, at Kent only titles published in print from 2009 forward are allowed into the DDA. Nothing earlier.) 3. Have you weeded the discovery pool? If so, please provide details. 4. Do you restrict the content level? (For example, Kent state loads discovery records only for ebooks that are categorized by YBP as general academic or advanced academic among the categories: Juvenile; Popular; Basic Studies; General Academic; Advanced Academic; Professional.) 5. Do you have a cost limit for ebooks that are loaded in your discovery pool? (For example, Kent excludes ebooks that cost more than $200.) 6. Do you exclude foreign language material? 7. Do you exclude any publishers? If so, please list. 8. Do you have other information about your DDA program that you think would be useful to us? Notes 1. National Information Standards Organization (NISO), Demand Driven Acquisition of Monographs: A Recom- mended Practice of the National Information Standards Organization, available online at https://groups.niso.org/apps/ group_public/download.php/13373/rp-20-2014_DDA.pdf [accessed 11 December 2018]. 2. NISO, Demand Driven Acquisition of Monographs. 3. Cristóbal Urbano et al., “Library Catalog Log Analysis in E-book Patron-Driven Acquisitions (PDA): A Case Study,” College & Research Libraries 76, no. 4 (2015): 412–26. 4. Kay Downey et al., “A Comparative Study of Print Book and DDA Ebook Acquisition and Use,” Technical Services Quarterly 31, no. 2 (2014): 139–60. 5. Yin Zhang and Kay Downey, “Ebook ROI: A Longitudinal Study of Patron-Driven Acquisition Models,” Computers in Libraries 37, no. 5 (2017): 4–8; Yin Zhang et al., “Notes on Operations: A Scenario Analysis of Demand- Driven Acquisition (DDA) of E-Books in Libraries,” Library Resources & Technical Services 59, no. 2 (2015): 84–93. 6. Catherine S.Y. Kwok et al., “Demand-Driven Acquisition at HKUST Library: The New Normal,” Interlend- ing & Document Supply 42, no. 4 (2014): 153–58, https://doi.org/10.1108/ILDS-09-2014-0046. https://groups.niso.org/apps/group_public/download.php/13373/rp-20-2014_DDA.pdf https://groups.niso.org/apps/group_public/download.php/13373/rp-20-2014_DDA.pdf https://faculty180.interfolio.com/public/download.php?key=SDRwNCtxSUpsamxBQ213WS9ucHFuOGtucDh4am5qSzEzU04xUjBjMGJoQWxkSG14WXZObm92YUgrRjU3aTJyMXJTYW1VajZWTjB1dlFIcnRvdjkyT2lENEhCWmpyeC9tYSttdTZhb2YveGM1bW9SY3dzZndGVEZicjhhQzZyTXNyYU1IWkhKaEpKV2doclplbE1JcjhnPT0%3D https://doi.org/10.1108/ILDS-09-2014-0046 https://doi.org/10.1108/ILDS-09-2014-0046 42 College & Research Libraries January 2020 7. Zhang et al., “Notes on Operations.” 8. Zhang and Downey, “Ebook ROI.” 9. Rebecca Schroeder and Rebecca Boughan, “Doing More with Less: Adoption of a Comprehensive E-book Acquisition Strategy to Increase Return on Investment while Containing Costs,” Library Resources & Technical Services 62, no. 1 (2018): 28–36. 10. Shaun R. Bennett, “A Data-driven Approach to Understanding the Demand-driven Acquisition Program at North Carolina State University,” Serials Review 42, no. 3 (2016): 201–09. 11. Beth Denker, “Changing Demand Driven Acquisition Ebook Models: A History and Preliminary Results at the Colorado Alliance of Research Libraries,” Journal of Library Administration 58, no. 3 (2018): 282–92; Valerie L. Boulos et al., “Use-Driven Acquisitions Planning for the SUS Libraries: Here We Go Again,” Works of the FIU Libraries (2016), available online at https://digitalcommons.fiu.edu/glworks/42 [accessed 26 October 2019]; Bart Harloe, Pat Hults, and Adam Traub, “What’s the Use of Use? Return on Investment Strategies for Consortial DDA Programs,” Journal of Library Administration 55, no. 3 (2015): 249–59; Catherine S.Y. Kwok et al., “Demand-Driven Acquisition at HKUST Library”; Hilary Robbeloth, Matthew Ragucci, and Kristina DeShazo, “Evidence-Based Acquisition: A Real Life Account of Managing the Program within the Orbis Cascade Alliance,” Serials Librar- ian 73, no. 3/4 (2017): 240–47, https://doi.org/10.1080/0361526X.2017.1388331. https://faculty180.interfolio.com/public/download.php?key=SDRwNCtxSUpsamxBQ213WS9ucHFuOGtucDh4am5qSzEzU04xUjBjMGJoQWxkSG14WXZObm92YUgrRjU3aTJyMXJTYW1VajZWTjB1dlFIcnRvdjkyT2lENEhCWmpyeC9tYSttdTZhb2YveGM1bW9SY3dzZndGVEZicjhhQzZyTXNyYU1IWkhKaEpKV2doclplbE1JcjhnPT0%3D https://digitalcommons.fiu.edu/glworks/42 https://doi.org/10.1080/0361526X.2017.1388331