ACRL News Issue (B) of College & Research Libraries June 1989 / 475 Output or p erform an ce m easures: T he m aking o f a m a n u a l By V irgin ia T ie f el Chair A C R L A d Hoc C om m ittee on Performance Measures The genesis of the A C R L output measures manual. A ccountability has been one of our society’s m a­ jor concerns in the 1980s, w ith higher education— a n d , w ith in h ig h e r e d u c a tio n , a c a d e m ic libraries—being scrutinized as perhaps never be­ fore. L ibrarians should not be surprised a t this de­ velopm ent, as libraries generally have m uch larger b u d g ets th a n m ost in d iv id u a l acad em ic u n its, th o u g h m ost libraries feel they m ust still p lead “hard-tim es.” L ibrarians are well aw are th a t li­ braries are often seen by academ ic adm inistrators as veritable bottomless pits whose existence has u n ­ questionable value b u t whose im pact on the q u al­ ity of education is difficult to access. O n another level, some state governments are dem anding ac­ countability from th eir educational institutions, w ith one result being th a t librarians in these states are facing the real possibility of “library m easure­ m ent” being established by legislators or govern­ m ental agencies beyond th e p aren t college or uni­ versity. A survey of the professional literature and pro­ grams of recent professional meetings provides am ­ ple evidence of librarians’ response to this chal­ lenge. Taking the lead in addressing this issue, the Association of College and Research Libraries has undertaken a variety of related activities, am ong w hich are plans to publish a m anual on library o u t­ p u t (performance) measures, scheduled for release in 1990. To coincide w ith the publication of the m anual and publicize this im p o rtan t event, a tw o- p a rt program is being planned for the ALA A nnual Conference in 1990. T he program will be jointly sponsored by the University L ibraries, College L i­ braries, and C om m unity and Junior College L i­ braries sections of A CRL, in collaboration w ith ACRL’s ad hoc C om m ittee on Perform ance M ea­ sures. This article will describe th e m anual, how it cam e into being, and some of th e challenges and is­ sues encountered in th e process. The Task Force Recognizing the im portance of th e issues associ­ ated w ith accountability, th e ACRL leadership ini­ tiated action in 1983 by appointing a Task Force on P erform ance Measures for A cadem ic L ibraries. C haired by Robert W . Burns J r . , 1 th e task force was charged to determ ine w h eth er ACRL should issue a m anual on perform ance measures for aca­ dem ic libraries, and, if so, to recom m end a p lan of action to develop such a m anual. Tw o im p o rtan t goals w hich ACRL w an ted to accomplish w ere to stim ulate librarians’ interest in perform ance m ea­ sures and then, if needed, provide practical assist­ ance so th a t they could conduct m eaningful mea- 1 th er m em bers of th e task force w ere Joan C. D urrance (University of M ichigan SLIS), R uth A. F raley (New York O ffice of C o u rt A dm inistra­ tion), Willis M. H u b b ard (G ettysburg College), C harles R. M cClure (Syracuse University SIS), L. Y vonne W u lff (U n iv e rsity of M ic h ig a n ), a n d Douglas L. Zweizig (University of Wisconsin). 476 / C& RL News s u r e m e n t o f th e i r l i b r a r i e s ’ p e r f o r m a n c e . L ibrarians could th en use th e m easurem ent results in planning, internal decision-making, and com­ m unicating w ith institutional adm inistrators. In accomplishing its charge, th e task force was asked to evaluate existing perform ance measures m an ­ uals for th eir applicability to academ ic libraries an d reco m m en d w h a t p e rfo rm an c e m easures- related activities, such as conferences, workshops, an d research projects, should be considered by ACRE. In its w ork, the task force used the definition of perform ance measures found in the Library Data Collection Handbook-. “Counts and combinations of counts w hich enable a lib rary to assess the degree to w hich a program meets its objectives....”2 The task force further specified th a t they viewed p er­ form ance measures as quantitative in n atu re and applicable to th e description of lib rary services (output), resources (input), and internal operation (throughput). The task force noted th a t the term “perform ance m easure” is frequently m isunder­ stood, an d th a t “ activity m easures” or “service measures” m ight m ore accurately denote w h a t the task force h ad in m ind. The task force emphasized the need to differentiate betw een “perform ance measures” and “standards” to prom ote w ider u n ­ derstanding th a t measures are m ean t to provide objective d a ta th a t can assist those responsible for planning, day-to-day m anagem ent, and com m un­ ication. Some examples of specific perform ance measures given were: num ber of requests for infor­ m ation, tim e required to fill requests, num ber of requests filled, num ber of people w aiting in lines, and num ber of documents delivered. After exam ining the issues, th e task force recom ­ m ended th a t ACRL sponsor th e developm ent of a m anual on perform ance measures for academ ic li­ braries. T he task force rep o rted a longstanding need for such a tool, heightened by tighter budgets of the 1980s, to aid academ ic libraries in describing th eir activities quantitatively. Although th e task force identified a vast am ount of literature on p er­ form ance measures, ranging in content from the very simple to th e highly sophisticated, it em pha­ sized th e need for a m anual on perform ance m ea­ sures specific to academ ic libraries. The report rec­ om m ended th a t the measures selected for inclusion in a m anual m eet certain criteria: they should be decision-related; focus on outputs and service to li­ brary clients; be easy to apply, use, and u nder­ stand; inexpensive to adm inister; appropriate to all types and sizes of academ ic libraries; and, gener­ ally, be judged “useful” to library m anagers. W h at follows is a chronological review of how th e m an­ ual on perform ance measures was developed. The Committee F o llo w in g th e re c o m m e n d a tio n o f th e task 2M ary Jo Lynch, Library Data Collection H and­ book (Chicago: ALA, 1981), 178. force, the ACRL Board appointed the ad hoc C om ­ m ittee on Perform ance Measures in 1984, w ith a tw o -p art charge based on th e task force re p o rt.3 In brief, th e com m ittee was to define, describe, and m onitor the w riting of a m anual on perform ance measures for academ ic libraries. The com m ittee was also to w ork w ith other ALA divisions and committees to prom ote and consult on any other work or program s th a t m ight be related to perfor­ m ance measures. The ad hoc com m ittee w as to complete its task by th e 1989 sum m er conference. The com m ittee decided to give first attention to developing a m anual, and chose to focus on selec­ tion of approxim ately 12 perform ance measures (the num ber specified in the com m ittee’s charge) to be included. The com m ittee’s second responsibil­ ity, prom oting interest in and use of perform ance measures, w ould receive attention after th e devel­ opm ent of th e m anual was underw ay. In all of its work th e com m ittee attem pted to involve m any other people. Anyone who attended a com m ittee m eeting o r asked to be placed on its m ailing list re­ ceived all com m ittee-generated papers, including agendas, minutes, drafts of documents, etc. By the end of its work, m ore th an 30 additional people w ere receiving all mailings sent to th e com m ittee members. In addition, as th e com m ittee worked on the m anual, m embers specifically sought out other ACRL committees whose w ork m ight be related to perform ance measures and established com m uni­ cation w ith them . The com m ittee began its w ork w ith extensive reading of the literatu re an d consideration of a num ber of perform ance measures for possible in­ clusion in th e m anual. Special attention was given to tw o handbooks: K a n to r’s O b jective P erfor­ m ance Measures fo r Academ ic and Research L i­ braries (1984) and Zweizig’s O u tp u t Measures fo r Public Libraries (1982).4 T he com m ittee cam e to agree th a t the ACRL m anual should occupy a m id­ dle ground (in complexity) betw een these tw o valu­ able works and th a t measures be w ritten from the perspective of the library user and be term ed “out­ p u t m easures.” The com m ittee identified as goals for ACRL per­ form ance measures th e following: • T o m easure the im pact, efficiency, and effec­ tiveness of library activities. 3The eight members of th e com m ittee are: Mig­ non Adams (Philadelphia College of Pharm acy and Science), Beverlee French (University of Califor­ n ia , D avis), D a v id K aser (In d ia n a U niversity SLIS), P atricia M. Kelley (George W ashington University), L ynn M arko (University of M ichi­ gan ), Jacq u ely n M orris (O ccid en tal C ollege), Jerom e Yavarkovsky (New York State L ibrary), and Virginia Tiefel (Onio State University). 4Paul B. K antor, O bjective Performance M ea­ sures fo r A cadem ic and Research Libraries (W ash­ ington, D .C .: Association of Research Libraries, 1984); Douglas Zweizig and Eleanor Jo Rodger, O u tp u t Measures fo r Public Libraries (Chicago: ALA, 1982). June 1989 / 477 • T o em phasize th a t m easures, not standards, w ere a t issue. • T o dem onstrate/explain library perform ance in m eaningful ways to university adm inistrators. • T o provide measures th a t can be used by heads of units to dem onstrate perform ance levels an d re­ source needs to lib rary adm inistrators. •G e n e ra lly to provide d a ta useful for lib rary planning. The com m ittee th en identified goals specific to th e m anual: • T o present measures th a t are useful for and replicable in all types and sizes of academ ic and re ­ search libraries. • T o present measures th a t are decision-related. • T o present measures th a t are easy to apply and use, inexpensive to adm inister, and user-oriented. • T o present measures th a t are linked to a li­ b ra ry ’s goals and objectives. • T o facilitate use of th e measures for historical comparisons w ith in a lib rary u n it or institution. E ach m em ber th en took a topical area, e. g ., user skills, technical services, an d reference, for m ore concentrated reading an d study. After exam ining their topics, m em bers recom m ended possible o u t­ p u t measures for each keeping in m ind th e com m it­ tee’s resolve th a t th e measures be focused on users. T he com m ittee ranked th e recom m ended o u tp u t measures in priority order an d selected th e tw elve as th e most critical for this first version of th e m a n ­ ual. Those m easures n o t included w ere reco m ­ m ended as “related measures for possible consider­ atio n ,” if additions w ere possible. In considering criteria for th e m an u al, th e com ­ m ittee repeatedly stressed th a t the m anual m ust be applicable to academ ic libraries of all sizes and th a t it should stress m easu rem en t of w h a t a library does, not w h a t librarians do. T he m an u al was to be user-oriented and should not im ply standards, b u t explain how one can m easure organizational p e r­ form ance. T he m an u al should prescribe th e m eth ­ odology for application of specific m easures and describe how statistical d a ta are used in each m ea­ sure. References to relev an t selected lite ra tu re w ere to be associated w ith each m easure, b u t the com m ittee felt strongly th a t care be taken th a t the m an u al not duplicate existing publications. The m anual should em phasize th a t it w ould not be a com prehensive p lanning guide, th a t readers w ould need to consult other sources (cited) to learn about establishing goals and objectives, details of cost analysis, and th e like. In th e end, the m anual, to be accepted an d used, should provide encouragem ent to librarians and give p ractical suggestions about how measures m ight be applied; for exam ple, how a lib rary adm inistrator m ight use inform ation on o u tp u t measures to com m unicate w ith college/u n i­ versity a d m in istra to rs. F in a lly , th e c o m m ittee stressed th a t th e m an u al w ould represent only a first step by w hich a lib rary can m easure its p erfo r­ m ance. The shaping of a m anual In Jan u ary 1987 th e com m ittee presented the ACRL Board w ith a docum ent describing th e 12 o u tp u t measures recom m ended for th e m anual. The measures w ere defined in fairly b road, con­ ceptual term s. T he com m ittee concurred w ith the Board’s subsequent recom m endation th a t th e m ea­ sures should be described in m ore specific term s. The com m ittee also recom m ended th a t, as this was done, the measures be set m ore explicitly in th e context of existing literatu re on o u tp u t measures (to avoid duplication and redundancy). A contract for a specialist w ho w ould refine th e measures and place them in the context of other m anuals was th en p u t o u t for bid. T he proposal subm itted by N ancy V an House, associate professor at th e School of L ib rary and I n ­ form ation Studies, University of C alifornia, Berke­ ley, was selected. V an House’s report was subm it­ ted to th e com m ittee in June 1987 an d w as found by all com m ittee m em bers to have achieved its p u r­ pose very well. T he com m ittee, w orking w ith the rep o rt, refined an d m ade final th e m an u al’s de­ scription. T he m anual w as to include th e follow­ ing: 1) a description of its goals and objectives; 2) a bibliographic essay to provide a fram ew ork for the measures included; 3) a clear description of each m easure, to include inform ation about how to use it, how to obtain d a ta, w h a t to do w ith results, and w h a t skills w ould be needed to adm inister th e m ea­ sures; 4) an extended bibliography; 5) a glossary; and 6) an index. Based on th e c o m m itte e ’s d esc rip tio n o f th e m an u al, ACRL issued a Request for Proposals in Fall 1987. By th e end of 1987 th e com m ittee h ad re v ie w e d resp o n se s a n d s e le c te d N a n c y V a n H ouse’s p ro p o s a l.5 At th e 1988 ALA M idw inter M eeting th e co m m ittee recom m ended th a t th e ACRL Board authorize th e ACRL office to enter into negotiations w ith V an House. Based on a posi­ tive Board response, th e ACRL office negotiated w ith V an H ouse an d cam e to an ag reem en t on term s in early spring. V an House th en began w ork on the first d ra ft of th e m anual. T h e co m m ittee an d V an House agreed th ere should be tw o tests for th e m anual: first, to test the p ro p o sed m easures them selves in selected a c a ­ dem ic lib raries, an d , th e n , w h en th e m easures w ere m ade final, a testing of th e m anual itself to d e te r m in e w h e th e r it co n v ey ed w h a t w as in ­ tended. T he com m ittee reviewed V an House’s out­ line of the m anual and a sum m ary of the measures in sum m er 1988. In the fall, testing of th e measures b e g an . T h e co m m itte e rev iew ed th e m easures again a t th e 1989 M idw inter M eeting. Testing of th e measures continued, w ith th e intent to test the m an u al itself in late spring 1989. T he com m ittee 5N ancy V an House is th e p rim ary au th o r of th e m anual. C harles M cClure and Beth W eil are co­ authors. 478 / C& RL News will review these results a t th e 1989 A nnual C on­ ference. Any needed revisions will be com pleted by th e end of sum m er and th e finished m anual deliv­ ered to the ACRL Executive C om m ittee in O ctober 1989 for action a t the C om m ittee’s fall meeting. The promotion of output measures In late 1986, th e com m ittee tu rn ed some of its a t­ tention to the second p a rt of its charge—“to recom ­ m end program s, policy, an d projects related to perform ance measures for academ ic libraries”— as well as to w ork w ith ALA divisional com m ittees to identify and prom ote activities on topics related to o u tp u t m easures, such as statistical techniques, d a ta collection, and tools useful for im plem enting measures. To help in this prom otion, th e com m it­ tee sent letters in Spring 1987 to th e presidents of all ALA divisions, w ith copies to staff liaisons/execu­ tive directors. The letter asked for identification of any perform ance m easure-related activities w ith in each division. T he response to these calls for infor­ m ation indicated less activity th a n had been antici­ p ated, b u t showed th a t there was some w ork being done in such areas as standards—and m uch in ter­ est in o u tp u t m easures. In d iv id u a l co m m ittee members w ere assigned liaison roles to any group w hich h ad reported possible fu tu re activity related to perform ance measures. To bring m ore visibility to the issue of perfor­ m ance measures, th e com m ittee undertook other approaches as well. An incisive article w ritten by com m ittee m em ber Beverlee French on th e w ork of the com m ittee and its review of th e perform ance measures literatu re appeared in C &R L N ew s in 1987.6 C om m ittee mem bers w orked w ith the U ni­ versity L ibraries Section’s C u rren t Topics Discus­ sion G roup to offer a program at the 1988 M idw in­ t e r M e e tin g .7 C o m m itte e m em b ers also gave individual presentations at other ALA meetings and to other interested groups. Also, as m entioned a t the outset, com m ittee members are collaborat­ ing w ith committees from three ACRL sections in planning a tw o -p art program on output/perform - ance measures a t th e 1990 Annual Conference. An article on perform ance measures in th e Chronicle fo r Higher Education8 m entioned th e com m ittee an d b ro u g h t several in q u iries from lib ra ria n s across th e country w ho w ere eager to use such a m an u al to help address increasing pressure for m easurem ent from adm inistrators and state gov­ ernm ents. 6Beverlee French, “L ib rary Perform ance M ea­ sures,” C & R L N ew s 48 (February 1987): 72-74. 7“W hy M easure Perform ance?” ALA M idw inter Conference, Jan u ary 9, 1987. 8Ju d ith A. T urner, “Academic L ibraries Urged to Study Needs of Users and Set Perform ance Mea­ sures,” Chronicle o f Higher E ducation, Jan u ary 27, 1988, 3. Conclusions W h a t has been learned from the almost five-year w ork of the committee? From my perspective as chair of the com m ittee, there is a greater awareness of the ever-growing dem and for accountability, w hich increases the pressure for m ore m easure­ m ent in libraries. Almost everyone w ith w hom I have talked sees m easurem ent itself as an increas­ ingly im p o rtan t and prom inent issue. Response to this increased scrutiny is some apprehension th a t if lib rarian s do not seize th e in itiativ e, “ m easure­ m en t” will be done by others far less knowledge­ able about libraries, w ith possibly very adverse consequences. There also seems to be a lingering concern on the p a rt of m any librarians about how to m easure and how m uch it will cost, and even u n ­ certainty about the purposes and possible results of m easurem ent. There is clearly a growing interest in how well libraries are m eeting users’ needs, th e role and image of the library, and (in m any institutions) ap p aren t dw indling of traditional adm inistrative understanding and support. One of the continuing challenges the com m ittee faced was to insure th a t o u tp u t measures are u n ­ derstood correctly, especially to be clear th a t they a re n o t “ s ta n d a rd s .” E a rly o n, th e co m m ittee m embers became aw are th a t close association w ith th e issue of standards could become (in one m em ­ ber’s description) “the kiss of d eath” for the m an ­ ual. Any perception th a t such a m anual m ight have th e prescriptive, judgm ental connotation of stan­ dards could highly prejudice any decision to use it. T h e Task Force R eport h a d strongly cautioned ACRL about this issue—and was right on target. T he C om m ittee an d au th o r encountered th e usual problems of com m unication across distances, especially w ith the restrictions of m eeting only at ALA m idw inter and annual conferences (no spe­ cial meetings w ere ever called). Focusing on a com­ plex topic to w hich all m em bers brought different expertise and perspectives, the com m ittee was able to reach consensus on all m ajor issues necessary to produce the m anual. T he spirit of com m on p u r­ pose and belief in the project, com bined w ith a flexible and reasonable attitu d e on the p a rt of all in v o lv e d , e n a b le d th e c o m m itte e to fu lfill its charge and complete a project th a t tru ly reflects th e best of all m em bers’ efforts. T he help of the ACRL staff, especially M ary El­ len Davis, program officer, and JoAn Segal, execu­ tive director of ACRL, was invaluable. The oppor­ tu n ity to w ork w ith a scholar-w riter of th e caliber of N ancy V an H ouse—w ho tru ly cap tu red th e thinking of the com m ittee—was exciting a nd p ro ­ fessionally rew arding. The com m ittee is excited about the m anual and its focus on the user and the user’s perspective. The com m ittee is hopeful th a t the m anual will be well received and widely used. W atch for th e m an u al’s publication in 1990 and th e program s on o u tp u t measures at th e 1990 ALA A nnual Conference. ■ ■