harer.indd The Importance of the Stakeholder in Performance Measurement: Critical Processes and Performance Measures for Assessing and Improving Academic Library Services and Programs John B. Harer and Bryan R. Cole This Delphi study identified the critical processes and performance measures of quality that can serve as a framework for new measures for assessing quality in academic library services and programs. These critical processes and performance measures were developed utilizing the structure and criteria of the Malcolm Baldrige National Quality Award’s 1999 Education Criteria for Performance Excellence. A panel of experts from the library profession was selected to participate in a Delphi panel to determine the importance of a list of critical processes and perfor- mance measures relevant to measuring quality in academic libraries.The processes and performance measures were identified through a review of the literature and in consultation with a review panel of professional librarians. The results of the study showed that a student, faculty, and stakeholder focus was the most important aspect of academic library programs and services for ensuring quality. he library profession has been seeking new perfor- mance measures for mea- suring quality and effective- ness. The authors hypothesized that a study to determine the importance of performance measures adapted from the Malcolm Baldrige National Quality Award’s (MBNQA) Education Criteria for Performance Excellence afforded a strong potential for identifying perfor- mance measures that would advance this inquiry.1 Established by the U.S. Congress in 1987, the Malcolm Baldrige National Quality Award for Performance Excellence is the most sought a er qual- ity management award and recognizes businesses for achievements in quality John B. Harer is an Assistant Professor in the Department of Librarianship at East Carolina University; e-mail: harerj@mail.ecu.edu. Bryan R. Cole is a Professor in the Department of Educational Administration and Human Resource Development at Texas A&M University; e-mail: b-cole@tamu.edu. 149 mailto:b-cole@tamu.edu mailto:harerj@mail.ecu.edu 150 College & Research Libraries and business performance. The Educa- tion Criteria for Performance Excellence were adopted from the business criteria to establish similar high-performance standards and provide the same recog- nition process for educational institu- tions. The purpose of this study was to utilize the expertise of a panel of library professionals to determine the critical processes and performance measures of quality most important to academic libraries. This study was designed within the framework of the Malcolm Baldrige National Quality Award’s (MBNQA) Education Criteria for Performance Excellence because the criteria for the award have been deemed a very effective tool for evaluating educational organiza- tions and because libraries, especially academic libraries, have implemented many aspects of continuous quality im- provement (CQI).2,3 The MBNQA criteria stress the importance of linking perfor- mance measures to critical processes; therefore, both critical processes and performance measures were identified and analyzed by the study. The study identified a significant number of criti- cal processes and performance measures ranked high by the panel and suggested that those measures that incorporated the concept of gauging the needs and expectations of stakeholders were the most important. The Importance of Performance Measurement The use of performance measures crosses the boundaries of many sectors of indus- try. Fundamentally, organizations of all types, whether they are a manufacturer or a hospital or a library, seek ways to demonstrate quality or excellence and effectiveness. The movement toward performance measurement in libraries has been stronger in Europe, especially with the nations of the Commonwealth March 2005 of Great Britain, where performance measurement is mandated by law. The Northumbria International Conference on Performance Measurement in Librar- ies and Information Services has been an early vehicle for scholarly work on the development and use of performance measurement in libraries, having held its fi h conference in July 2003.4 In a keynote speech before the Second Northumbria conference, Rowena Cullen explained the significance of performance measure- ment: “Performance measurement is an essential management tool that may be implemented in a variety of ways…. With this understanding, and with lead- ership and organizational resolve to use measurement as a tool to increase orga- nizational effectiveness, the possibilities are endless.”5 In the United States, the Association of Research Libraries (ARL) has been making efforts to advance the discussion on performance measures. J. C. Blixrud noted in the ARL Newsle er that the use of performance measures is common in libraries today.6 “Performance measures, quality assessment, public ac- countability, benchmarking—these have become common words and phrases in higher education and government literature in the 1990s. The environment in which ARL members and many other libraries operate has changed from one of natural acceptance of value by virtue of function to one in which all units must substantiate their worth.”7 Although many efforts are being made to increase the understanding of the importance of performance mea- sures in libraries and the promotion of performance measures as an effective management tool, serious questions are being raised about the direction of these efforts. For example, J. S. Town has ar- gued that performance measurement has not addressed the issue of performance.8 Instead, libraries have concentrated on what measures to use without significant introspection on whether the measure- ment does what it is designed to do. As he stated, “In my opinion, the gap arises from two main areas: the current data collection methods and structures which both obscure rather than illuminate performance and provide a misleading picture of what performance is or should be; and the implicit model of academic librarianship which informs the choice of current measures.”9 Town described the approaches to performance measurement for libraries within a theoretical context and suggested a total quality manage- ment (TQM, or CQI) framework for employing performance measurement. In this argument, two types of reason- ing from the realm of science can drive the way performance measurement is designed: inductive (observation leads to formation of theory) and hypothetico-de- ductive (an a priori assumption is formed that is used to deduce consequences). The way to the future of defining library performance measurement, according to Town, should be based on three hy- potheses: 1. Total Quality Management pro- vides a ready-made framework for per- formance measurement 2. Digital developments point to a fu- ture for information services that require information strategies 3. The first two hypotheses suggest that libraries in higher education need to think more broadly about themselves, their role and philosophy, as well as their systems of performance measurement.10 J. Winkworth also has noted the shortcomings of the various types of performance measures reported in the literature and suggests that libraries need to establish a better concept of stakeholder.11 In another critical study, A.D. Pra and E. Altman found a lack of correlation between input and output Academic Library Critical Processes 151 measures for U.S. public libraries.12 This study compared input data with output data of twenty-four public library systems that were primarily numerical measures on expenditures, circulation, turnover, and holdings. These and other works suggest that performance measurement in libraries needs greater definition and clarity. The Case for the Importance of Stakeholder Performance Measures Robert Waterman, an associate of Tom Peters and coauthor with him of In Search of Excellence, a best-selling work on qual- ity and successful management practices and considered an early work in the total quality movement, wrote: “What makes top performing companies different, I would urge, is their organizational ar- rangements. Specifically: 1. They are be er organized to meet the needs of their people, so that they at- tract be er people than their competitors do and their people are more greatly motivated to do a superior job, whatever it is they do. 2. They are be er organized to meet the needs of customers so that they are either more innovative in anticipating customer needs, more reliable in meet- ing customer expectations, better able to deliver their product or service more cheaply, or some combination of the above.”13 A review of the literature reveals that there is significant opinion from library professionals promoting measures that clearly address the two concepts Wa- terman suggests are important for the private sector, and within a TQM or CQI framework. A clear and consistent theme in the Northumbria International Conferences has been the importance of determining the needs and expectations of the stakeholders of the library institu- tion. J.C. Crawford went further in a pa- http:libraries.12 http:stakeholder.11 http:measurement.10 152 College & Research Libraries per at the Northumbria Conference.14 He argued that performance issues should come from the needs and experiences of different categories of customers for the purpose of data collection and decision making. These arguments support the efforts of the ARL’s New Measures Ini- tiative, for example. The components of the New Measures Initiative emphasize gauging the stakeholders’ expectations, at the very least, through the LibQUAL+ survey, among other initiatives. Any examination of the Malcolm Bal- drige National Quality Award’s Educa- tion Criteria for Performance Excellence categories will show that the concept of stakeholder is interwoven into the critical processes of each category. The MBNQA award criteria consists of seven categories, two of which address the external stakeholder (Category 3, Student and Stakeholder Focus) and internal stakeholder (Category 5, Faculty and Staff Focus), precisely the concepts Waterman has suggested are key to high- performing organizations. Other critical processes throughout the remaining cat- egories exist that have stakeholder as a component. For example, the Leadership category directs organizations to show how leaders communicate to stakehold- ers and how they listen to their needs and expectations. How the library goes about strategic planning (Category 2) and how the organization determines results (Category 7) also feature the concept of stakeholder as an integral component of the critical processes in the categories. Town went further than the emphasis of stakeholder as the driver for perfor- mance measure design.15 He suggested a more comprehensive TQM/CQI approach for libraries designing and implement- ing performance measures. Town stated: “The main result of drawing on a TQM hypothesis for performance measures is March 2005 the inescapable conclusion that customer satisfaction is the critical measure.” He has argued that the measures suggested by his hypothesis, for example, are: • satisfaction surveys; • designed surveys for improve- ment; • benchmarking; • customer care, involving measures of personal service, materials service; • “Mystery shopper”; • SERVQUAL; • process throughput times.16 Within these contexts, the authors de- termined that a study of new performance measures using the MBNQA criteria would be useful. The Conduct of the Delphi Study of Critical Processes and Performance Measures The Delphi technique for forecasting issues of the future and for gaining con- sensus from a group of experts was used to survey professional librarians drawn from a national pool who had significant knowledge of CQI applications in librar- ies, as either a scholar of library manage- ment or a practitioner who implemented some significant aspect of CQI in a library. It supports informed decision making; gives a way of structuring a large mass of information; provides an expertise in order to achieve informed judgment, decision making, and forecasting; and can be used in discussing issues of both a numerical and a nonquantifiable na- ture. The technique, as a systematic way to draw on the informed judgment of a group of experts, has the capacity to deal with ambiguity and multidimensionality and been widely used to support decision making in the fields of sociology, educa- tion, medicine, and policy making.17–21 At the heart of the Delphi technique is the means for determining informed judgment. In many social science fields, http:times.16 http:design.15 http:Conference.14 Academic Library Critical Processes 153 including education and library science, there is o en a need for judgmental prob- lem solving that can be tested through a consensus of opinion of appropriate ex- perts. A. Rotondi and D. Gustafson noted that the Delphi process has proved to be effective in a variety of fields, problems, and situations.22 W. J. Wilhelm as well as V. Story, L. Hurdley, G. Smith, and J. Saker support the Delphi method for social sci- ence research.23,24 The Delphi technique is a viable approach that enables experts to “deal with a complex problem systemati- cally…. It produces useful information in either the paper-and-pencil mode or the computer mode.”25 It was chosen because the identification of critical processes and performance measures has not lent itself to precise measurement and the theory on the processes and measures for libraries is in a state of fluidity and incomplete. For example, M. Kyrillidou and W. Crowe have described how ARL’s Statistics and Measurement Program has searched for new measures because the program has “expand(ed) beyond measures of ‘input’ (such as collection size, number of staff, expenditures, etc.)” and have gone on to argue that the library community must begin to reach consensus on how best to measure services and programs.26 The library profession is moving toward the use of quantitative and qualitative measures to determine quality and compare institu- tions. Qualitative measures are being promoted as a necessary addition to the assessment of quality in libraries, but no definitive data or theory exists on what qualitative measures are appropriate. This study employed two panels, one to determine an appropriate list of critical processes and performance measures to be evaluated and the other to act as the Delphi experts that ranked each critical process and performance measure on a scale of importance. The Panel Makeup for This Study M.A. Ziglio and E. Ziglio cautioned that the quality of a Delphi study relies on the selection of experts holding accepted credentials and that specific measures should guide the identification and se- lection process.27 The need to identify a specific expertise was recognized prior to selecting the panel. Panelists were chosen on the basis of their published record on performance measures or on the topic of CQI/TQM, especially for li- braries. R.C. Wicklein and J.W. Rojewski have stipulated that the research stan- dard in selecting Delphi panelists rests on expertise.28 The Delphi standard of expertise stands in stark contrast to the scientific research paradigm standard of random samples that establish equitable representation of the population—larger samples constitute a higher-quality study. Previous Delphi researchers have defined experts as “well-informed, leading authorities in their respective fields.”29 L. Westbrook defined experts as those people “whose positions, respon- sibilities, and/or publications indicate expertise in the area.”30 The criteria for panel membership emphasized their knowledge and/or experience with implementation of CQI in libraries, espe- cially academic libraries. Because of the criteria, the panel members came from several different states and institutions, making a Delphi panel appropriate and inexpensive for the data gathering. The first panel consisted of four professional librarians meeting these criteria who had advised the authors early in the study. The second panel consisted of twelve individuals from a cross section of U.S. institutions, including six deans of uni- versity research libraries, three assistant or associate deans of university research libraries, and three practitioner librarians who had developed and implemented a CQI program. Ziglio and Ziglio stipu- http:expertise.28 http:process.27 http:programs.26 http:situations.22 154 College & Research Libraries lated that a small group of ten experts will produce accurate results and K. Brockhoff found that in fact-finding stud- ies, as few as seven panelists can provide optimal results.31,32 Identification of the participants was made using a carefully constructed set of criteria developed by the author with the assistance of the staff of the ARL’s Office of Leadership and Management Services. The most important criterion was substantial evidence of the knowl- edge of CQI in academic libraries. This criterion was determined by a search of the literature for publications wri en by potential panelists on a significant issue for implementing CQI in academic librar- ies. A second criterion was membership or involvement in organizations or pro- grams that emphasize CQI, such as qual- ity award examiners for state or national quality awards, including the Malcolm Baldrige National Quality Award. A third criterion involved evidence that the potential panelist had implemented a CQI/TQM program or service within an academic library se ing. The Study’s Instrumentation There were two phases to the creation and dissemination of the instruments for this study. In the first phase, an in- strument was created to test an initial set of academic library– specific critical processes and performance measures based on the structure of the Malcolm Baldrige National Quality Award Educa- tion Criteria for Performance Excellence. The structure consisted of the seven categories of the criteria and the items associated with each category. A total of eighteen items were part of the MBNQA criteria structure. (See table 1.) The initial set of library critical processes and perfor- mance measures aligned to the MBNQA criteria were derived from the literature, and were refined and revised based on March 2005 the advice of experts on an instrument review panel. A common Delphi method practice is to use a preliminary or first questionnaire as an exploration of the subject and to seek further information for the final instruments to be used.33 Prior to the creation of the critical pro- cesses and performance measures review instrument, the 1999 Malcolm Baldrige National Quality Award’s Education Criteria for Performance Excellence were examined carefully and revised to reflect terminology more appropriate for the library profession and libraries as institutions. Development of the Critical Processes and Performance Measures Within this structure, a series of critical processes and performance measures was devised for each of the eighteen items. A critical process was defined as that method or strategy by which a library addresses a specific function and has been found to produce results that are replicable over time. A performance measure was defined as numerical in- formation on the results of processes, production, and services that quantifies input, output, and factors influencing those processes, production, and services for the purpose of rating or evaluating them for quality and potential for im- provement. It was assumed that no more than two or three critical processes for each item should be in each category in the final instrument and no more than one or two performance measures for each critical process in order to reduce the length of the instrument and encour- age participation and to focus on core processes that were deemed to be high priority. Wording for the specific critical processes was aided by the explanation of each item within the subsequent text of the MNBQA booklet in which the cat- egories and items were published. Revi- sions of the initial wording were made to reflect terminology more appropriate for libraries. This review discovered forty critical processes, distributed across the seven MBNQA categories. Performance measures then were sought for each process identified in the first instrument. An extensive search of the literature was made to determine the state of performance measures for academic libraries and to formulate appropriate Academic Library Critical Processes 155 measures for libraries implementing CQI. Performance measures were sought in known texts on performance measure- ment in libraries, especially works by F.W. Lancaster, P.B. Kantor, D.E. Riggs, and Peter Hernon.34–37 Publications of the ARL were examined for performance measures, as well. Publications that have analyzed and interpreted the Malcolm Baldrige National Quality Award crite- ria for the annual MBNQA awards also TABLE 1 Malcolm Baldrige National Quality Award’s Education Criteria for Performance Excellence, by Category and Items Malcolm Baldrige National Quality Award’s Education Criteria for Performance Excellence: Categories and Items Category 1: Leadership Item 1: Leadership System Item 2: Public Responsibility and Citizenship Category 2: Strategic Planning Item 1: Strategic Development Process Item 2: Organizational Strategy Category 3: Student and Stakeholder Focus Item 1: Knowledge of Student Needs and Expectations Item 2: Student and Stakeholder Satisfaction and Relationship Enhancement Category 4: Information and Analysis Item 1: Selection and Use of Information and Data Item 2: Selection and Use of Comparative Information and Data Item 3: Analysis and Review of Performance Category 5: Faculty and Staff Focus Item 1: Work systems Item 2: Faculty and Staff Education, Training, and Development Item 3: Faculty and Staff Well-being and Satisfaction Category 6: Program and Service Delivery Design and Support Item 1: Program and Service Delivery Design and Delivery Item 2: Support Processes Category 7: Performance Results Item 1: Student Performance Review Item 2: Student and Stakeholder Satisfaction Results Item 3: Faculty and Staff Results Item 4: Organization-specific Results http:instrument.An 156 College & Research Libraries were consulted, especially M.G. Brown.38 Furthermore, numerous professional articles in library journals discussing a specific topic or performance measure were reviewed and measures identified and revised for this study from them (for example, C. Martell, C.T. Townley, J.A. Siggins, H. Ashar and S. Geiger, and C. Russell39–43). This process discovered an initial set of 185 potential performance measures. The Instrument Review Panel Phase From the initial researcher-developed list of processes and performances, a panel of four experts was assembled to review the 185 items and limit them to no more than two critical processes per MBNQA item and two performance measures per critical process. This panel was formed from a list of potential panelists who met the criteria for selecting Delphi panelists as a whole, including a university library dean, a former dean of a graduate school of library science, and two members of the ARL’s Office of Leadership and Manage- ment Services’ executive staff. This panel reviewed the instrument to provide for face validity and clarity and to make rec- ommendations on the critical processes and performance measures to be included. In examining this initial list, the par- ticipants were asked to place the critical processes and performance measures in priority order. The wording of the critical processes and performance measures was revised based on the recommendations of this panel, as well. From this review, the top two critical processes for each item and top two performance measures remained on the compiled list. Where there were dis- crepancies among the panelists, electronic messages were sent to solicit agreement to secure only two critical processes for each item and two performance measures for each critical process. This completed the first phase. March 2005 The Delphi Review Phase For the Delphi review phase, a second instrument was designed to be an iterative questionnaire with anonymous feedback in two iterations. For the Delphi panel, the instrument was structured in the same manner as the instrument from the first phase, using the structure of the seven categories and eighteen items of the MBNQA’s Education Criteria for Performance Excellence (table 1). Four- point, Likert-type scales, however, were added with each critical process and performance measure to enable each par- ticipant to score each item on the instru- ment according to importance. Use of the four-point scale is recommended by L.W. Anderson and others, along with other even-number scales, to avoid an option that permits a “not sure” response.44 The four-point scale limits responses to two degrees of importance or unimportance, for example. The critical processes and performance measures identified through the process of reviewing and revising the first instrument formed the basis for the review by the expert panel. Each critical process and performance measure cor- responded with the appropriate MBNQA category and item, and so processes and measures given lower priorities were not included. The instrument was designed so that for each MBNQA item, the cor- responding critical processes and perfor- mance measures were set in tabular form with the item. Space also was provided for an additional critical process and an additional performance measure in each table when deemed appropriate by any Delphi panel expert. For the second iteration, the instru- ment had the same structure, including each of the MBNQA’s seven categories, eighteen items, and the critical processes and performance measures of the first iteration. Any critical process or perfor- mance measure provided by a participant http:response.44 http:Brown.38 during completion of the first iteration was added in the appropriate table and linked to the appropriate MBNQA item and critical process, respectively. The distribution of the scores of each participant for each critical process and performance measure was added to the table in a column immediately next to the corresponding process or measure, the participants’ own score given for the first iteration listed in a column next to the column with the distribution of scores of all participants, and a final column to record their revisions, if any, for the final iteration. Response Rate Initially, twelve potential panelists were identified by a record or reputation of their expertise, with two dropping out before the instrument was distributed. The response rate for the first round was 100 percent and 90 percent for the second iteration of the second phase. Numerous a empts were made to reach the tenth panelist in the second iteration to secure a 100 percent participation for that round, but without success. Summary of Findings This study began with the hypothesis that the framework of the Malcolm Baldrige National Quality Award’s Education Criteria for Performance Excellence, adapted to the language, functions, and processes within libraries, would lead to the identification of useful and effective performance measures of quality for libraries, especially academic libraries. An important aspect of the hypothesis that must be understood first was that these performance measures needed to be linked to critical processes to be valuable; therefore, the study also hypothesized that the framework of the MBNQA criteria could be used as a means for identifying critical processes as well. Af- Academic Library Critical Processes 157 ter adapting the criteria as explained in this paper and a er an extensive search of the literature, the study was success- ful in identifying both critical processes and performance measures of quality for libraries. The study identified forty-two critical processes in this manner and eighty-two performance measures linked to at least one of these critical processes. (See the appendix for the Delphi review results.) This study also sought to assess these critical processes and performance measures of quality for their utility as effective tools for measuring the quality of library services, functions, and pro- cesses. By subjecting the identified critical processes and performance measures to a Delphi method of review, the study was designed not only to discover these processes and measures but also to deter- mine their level of usefulness. The study resulted in a ranked list of importance of the final set of critical processes and performance measures from a review by a panel of experts employing the Delphi method. Key Findings Regarding the Critical Processes Because there was li le variation in the ratings made by the Delphi panel in its findings regarding the identified critical processes, the results of this study show that the experts consider most of the criti- cal processes “important” at the very least and many consider them very important. The mean and standard deviation were calculated from the responses for each critical process and performance measure. On a rating scale of 1 to 4, with 1 being “very important” and 2 being “impor- tant,” ten critical processes, or nearly 25 percent, were unanimously rated “very important.” Another 10 percent had only one expert out of the ten rating those critical processes as only “important” 158 College & Research Libraries compared to the rating of “very impor- tant” by all the other experts. Only one of the forty-two critical processes was ranked as “of li le importance” (µ = 3.1, s.d. = .78). These results indicate that the experts view almost all of the identified critical processes as significant to the performance measurement process and that these critical processes are highly important to libraries in assessing quality in the results of their programs, services, and functions. Within the Malcolm Baldrige Award criteria’s seven specific categories, the experts were unanimous in their assess- ment that the critical processes related to a customer focus (that is, the student, faculty, and stakeholder focus category) were the most important. (See the ap- pendix.) No other category had this level of unanimity. Clearly, this study suggests that libraries should assess the needs and expectations of students, faculty, and other stakeholders in order to plan for high-quality programs, services, and functions. The preponderance of critical processes with a unanimous rating by the panel is not replicated in the other catego- ries. Gauging the needs and expectations of the library’s customers is the most important aspect for determining high- quality programs, services, and functions, as seen by this panel. In the results found in the other cat- egories, critical processes in the leader- ship category also were considered very important by the experts. Although not all the critical processes received unani- mous ratings by the panel, as was true with the stakeholder category, a third of the nine leadership processes (such as “what employees say about the visibility of senior leaders”) were unanimously rated “very important” by the entire panel and a fourth (such as “existence of a clear plan for encouraging innova- tion”) was rated “very important” by March 2005 all but one panel expert. The results of the remaining categories are less clear. Although the critical processes in these categories have low mean scores with small standard deviations for the most part (indicating li le variation), there is less agreement among panel members as to their importance. The results of this study indicate that the critical processes for gauging student, faculty, and stake- holder needs and expectations, as well as many of those for leadership, methods for gathering information and analysis, and performance results are regarded as very important to developing, maintain- ing, and assessing high-quality library programs, services, and functions. Within the remaining categories, the study has shown several individual critical processes also to be very important to high-quality library programs, although the study is less clear as to the categories as a whole. Key Findings Regarding the Performance Measures When it comes to an assessment of the identified performance measures, vir- tually the same pattern of ratings for that of the critical processes was found. The performance measures brought out more overall variation among the panel of experts, but there is no doubt that the performance measures for Category 3, student, faculty, and stakeholder focus (i.e., a customer focus), also received the highest level of ratings and near- unanimous agreement for most of the performance measures in the category. Furthermore, no other category had as many performance measures with this level of agreement among the panel of experts. When the other categories of perfor- mance measures are examined, for the most part, the pa ern of ratings of im- portance for the critical processes holds true for the categories of performance measures as well. The category for cus- tomer focus is clearly more regarded by the panel for important performance measures than any other category, but the categories for leadership, methods of gathering information and analysis, and performance results follow in the number of measures rated highly by the panel. That is not to say that the experts view performance measures for the categories of strategic planning, faculty and staff focus, and program and service delivery as unimportant but, rather, that there was more variation in the experts’ ratings for the identified measures of these categories. In reality, the panel viewed at least some of the identified performance measures in every category as very important to high-quality library services, programs, and functions. This study shows that when examining those performance measures identified by this research, the experts were in more agree- ment on performance measures for the categories of customer focus (student, faculty, stakeholders), leadership, gather- ing of information and analysis, and per- formance results, generally in that order, but rated performance measures in all the categories as important or very important. The appendix should be examined to get a complete picture of all the performance measures and how they have been rated by the Delphi panel. Within the categories, some perfor- mance measures are qualitative and quantitative in design. Some measures ask for a specific numeric measurement; others describe a process of measure- ment. For example, in the customer focus category, a highly rated performance measure is “A comprehensive system or process exists for tracking student and faculty comments and complaints and what is done to address them” and for the leadership category, a highly rated Academic Library Critical Processes 159 performance measure is “Existence of systems which reinforce the value of continual learning.” In more quantitative measures, such performance measures as “turnover rates” in the faculty and staff category and “purchasing ratios” in the performance results category were included. Measures that were more quantitative in nature were less regarded, as a whole, than those with a qualitative design. In their responses, some panel members added comments that indicated that quantitative measures were of less value to measuring quality than qualitative measures for now and in the future because the panelists believed measuring what was owned is less useful in an age of access to resources globally. An important aspect of this study, how- ever, is that the performance measures, whether quantitative or qualitative, should be used as indicators of how well the respective process is working and as a measure of needed improvement to that process. In this way, libraries can use these processes and measures as a means of individually and collectively under- standing the quality of their operational performance and a ention to customer needs and expectations. Summary This study has identified a large body of critical processes and performance measures that can be used by libraries for the assessment of quality in their programs, services, and functions. By utilizing the Malcolm Baldrige National Quality Award’s Education Criteria, these processes and measures also provide li- braries with the opportunity to develop quality-focused programs, services, and functions, and they can contribute to the development of continuous quality im- provement for use as a management tool. Using these processes and performance measures, library managers can work to 160 College & Research Libraries continuously improve their services and products. This body of processes and measures also may contribute to aiding the library profession in developing its own MBNQA criteria. The results of this study (shown in the appendix) also show the Delphi panel’s expert opinion about the usefulness of all the critical processes and performance measures. This review provides a window to what processes and measures are im- portant for libraries as they seek to build effective programs, services, and func- tions. The study shows that in the judg- ment of this Delphi panel, libraries must gauge the needs and expectations of their customers and must strive to have a strong customer focus in planning for services in order to ensure quality in their efforts. The panel members also noted that processes and measures in the areas of leadership, methods of gathering and analyzing infor- mation, and what results libraries actually achieve also must play a significant role in measuring quality within libraries. The Delphi panel believed that qualitative measures should constitute greater focus in the measurement of quality than quan- titative measures. The study identified specific processes and measures in every category that the panel members saw as very important or important. The critical processes identified by this study must be seen as important for establishing assessment methodology for academic libraries. These critical pro- cesses should begin to provide a frame- work for the profession to determine what should be measured for effective performance. The study results indicate that through this Delphi review, all of these processes are crucial to the perfor- mance of academic libraries. Although they may not represent all the possible critical processes, every critical process in this study has been judged in a sub- stantial way and nearly unanimously by March 2005 the Delphi panel as important to effective library service. The performance measures identified by this study constitute a significant body of potential methods of assessing effec- tiveness. Some of these measures do not appear to be as important as others, but most have been identified as important to gauging effective library services and pro- grams. The study results suggest a broad array of performance measures, from the quantitative to the qualitative. These mea- sures can be adapted in assessment meth- ods and initiatives to provide increased possibilities for institutional assessment. They can form the framework for actual performance measures or provide a lead into an actual measure for evaluating a service or program. The results clearly show that the Delphi panel considers determining stakeholders needs and ex- pectations as the most important aspect of assessment. This appears to suggest that current innovations in assessment, such as ARL New Measures initiatives, are useful and important. These results also clearly indicate the importance of a customer fo- cus in the design of performance measures as well as in programs and services. The study indicates the importance of the assessment of library services and pro- grams, in general, but more important, the need for developing qualitative as well as quantitative performance measures. It has developed a significant body of both critical processes and performance measures that can be useful for these as- sessment efforts. It clearly indicates that all assessment initiatives must emphasize the importance of the stakeholders in the delivery of library services and programs. To measure an effective service or pro- gram, libraries need to identify and then plan for what their customers needs and expectations are, then deliver them before measuring how effective that service de- livery has been. Academic Library Critical Processes 161 Notes 1. National Institute of Standards and Technology. Education Criteria for Performance Excellence (Malcolm Baldrige National Quality Award) (Milwaukee: American Society for Quality, 1999). 2. M. L. Blazey, K. S. Davison, and J. P. Evans. Insights to Performance Excellence in Education, 1999 (Milwaukee: ASQ Quality Pr., 1999). 3. W. D. Penniman, “Quality Rewards and Awards: Quality Has Its Own Reward, But an Award Helps Speed the Process,” Journal of Library Administration 18, no. 1/2 (1993): 127–36. 4. T. K. Watson, “How to Plan and Manage Library Performance Measurement Projects through Continuous Improvement Practice: USQ Library Experience,” in Proceedings of the 2nd Northumbria International Conference on Performance Measurement for Libraries and Information Ser- vices (Newcastle upon Tyne, Eng.: Information North for the Department of Management, Univ. of Northumbria at Newcastle, 1997), 239–59. 5. Rowena Cullen, “Does Performance Measurement Improve Organizational Effectiveness? A Post-modern Analysis,” in Proceedings of the 2nd Northumbria International Conference on Performance Measurement in Libraries and Information Services (Newcastle upon Tyne, Eng.: Information North for the Department of Management, Univ. of Northumbria at Newcastle, 1997), 15. 6. J. C. Blixrud, “Special Issue: Issues in Research Library Measurement,” ARL: A Bimonthly Newsle er of Research Library Issues and Actions 197 (April 1998): 1–2. 7. Ibid., 1. 8. J. S. Town, “Performance or Measurement? in Proceedings of the 2nd Northumbria International Conference on Performance Measurement in Libraries and Information Services (Newcastle upon Tyne, Eng.: Information North for the Department of Management, Univ. of Northumbria at Newcastle, 1997), 89–92. 9. Ibid., 81. 10. Ibid., 83–85. 11. I. Winkworth, “Making Performance Measurement Influential,” in Proceedings of the 2nd Northumbria International Conference on Performance Measurement in Libraries and Information Services (Newcastle upon Tyne, Eng.: Information North for the Department of Management, Univ. of Northumbria at Newcastle, 1997), 93–100. 12. A. D. Pra and E. Altman, “Live by the Numbers, Die by the Numbers: A ributing Too Much Importance to Library Statistics Can Be Hazardous to Your Job,” Library Journal 122 (Apr. 15, 1997): 48–49. 13. R. H. Waterman, “What Makes the Manufacturers of These Products Special? The People, Stupid,” Director 47 (May 1994): 48–52. 14. J. C. Crawford, “The Stakeholder Approach to the Construction of Performance Measures,” in Proceedings of the 2nd Northumbria International Conference on Performance Measurement in Libraries and Information Services (Newcastle upon Tyne, Eng.: Information North for the Department of Management, Univ. of Northumbria at Newcastle, 1997), 177–85. 15. Town, “Performance or Measurement?” 16. Ibid., 84–85. 17. M. Adler and E. Ziglio, Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health (Bristol, Pa.: Jessica Kingsley Publishers, Ltd., 1996). 18. K. A. Calzone, J. Jenkins, and A. Masny, “Core Competencies in Cancer Genetics for Ad- vanced Practice Oncology Nurses,” ONF 29, no. 9 (2002): 1327–33. 19. A. J. Catlin and B. S. Carter, “Creation of a Neonatal End-of-life Palliative Care Protocol,” Journal of Neonatal Nursing 21, no. 4 (2002): 37–49. 20. A. J. McBride, R. Pates, R. Ramadan, and C. McGowan, “Delphi Survey of Experts’ Opin- ions on Strategies Used by Community Pharmacists to Reduce Over-the-counter Drug Misuse,” Addiction 98 (2003): 487–97. 21. D. Sharp, A. Liebenau, N. Stocks, et al., “Locally Developed Guidelines for the A ercare of Deliberate Self-harm Patients in General Practice,” Primary Health Care Research and Development 4 (2003): 21–28. 22. A. Rotondi and D. Gustafson, “Theoretical, Methodological and Practical Issues Arising Out of the Delphi Method,” in Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, ed. M. Adler, and E. Ziglio, 34–55 (Bristol, Pa.: Jessica Kingsley Publishers, Ltd., 1996). 23. W. J. Wilhelm, “Alchemy of the Oracle: The Delphi Technique,” Delta Pi Epsilon Journal 43, no. 1 (2001): 6–26. 24. V. Story, L. Hurdley, G. Smith, and J. Saker, “Methodological and Practical Implications of the Delphi Technique in Marketing Decision-making: A Reassessment,” Marketing Review 1 (2001): 487–504. 162 College & Research Libraries March 2005 25. Wilhelm, “Alchemy of the Oracle,” 14. 26. M. Kyrillidou and W. Crowe, “In Search of New Measures,” Journal of Library Administra- tion 35, no. 4 (2001): 19. 27. M. A. Ziglio and E. Ziglio, Gazing into the Oracle: The Delphi Method and Its Applications to Social Policy and Public Health (London: Jessica Kingsley Publishers, Ltd., 1996). 28. R. C. Wicklein and J. W. Rojewski, “Toward a Unified Curriculum Framework for Technol- ogy Education, Journal of Industrial Teacher Education 36, no. 4 (1999): 38–56. 29. Ibid., 42. 30. L. Westbrook, “Information Access Issues for Interdisciplinary Scholars: Results of a Delphi Study on Women’s Studies Research,” Journal of Academic Librarianship 23, no. 3 (1997): 211–16. 31. Ziglio and Ziglio, Gazing into the Oracle. 32. K. Brockhoff, “The Performance of Forecasting Groups in Computer Dialogue and Face- to-face Discussion,” in The Delphi Method: Techniques and Applications, ed. H. A. Linstone and M. Turoff, 291–321 (Reading, Mass.: Addison-Wesley, 1975). 33. A. L. Delbecq, A. H. Van de Ven, and D. H. Gustafson, Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes (Glenview, Ill.: Sco -Foresman and Co., 1975). 34. F. W. Lancaster, The Measurement and Evaluation of Library Services (Washington, D.C.: Information Resources Pr., 1997). 35. P. B. Kantor, Objective Performance Measures for Academic and Research Libraries (Washington, D.C.: ARL, 1984). 36. D. E. Riggs, Strategic Planning for Library Managers (Phoenix, Ariz.: Oryx, 1984). 37. Peter Hernon, Statistics for Library Decision Making: A Handbook (Norwood, N.J.: Ablex Publishing Corp., 1990). 38. M. G. Brown, Baldrige Award Winning Quality: How to Interpret the Malcolm Baldrige Award Criteria, 3rd ed. (White Plains, N.Y.: Quality Resources, 1993). 39. C. Martel, “The Nature of Authority and Employee Participation in the Management of Academic Libraries,” College and Research Libraries 47 (Mar. 1987): 110–22. 40. Townley, C. T., “Nurturing Library Effectiveness: Leadership for Personnel Development,” Library Administration and Management 3 (winter 1989): 16–20. 41. J. A. Siggins, “Job Satisfaction and Performance in a Changing Environment,” Library Trends 41 (fall 1992): 299–315. 42. H. Ashar and S. Geiger, “Using the Baldrige Criteria to Assess Quality in Libraries,” Library Administration and Management 12 (summer 1998): 147–55. 43. C. Russell, “Using Performance Measurement to Evaluate Teams and Organizational Ef- fectiveness,” Library Administration and Management 12 (summer 1998): 159–65. 44. L. W. Anderson, “A itude Measurement,” in Educational Research, Methodology, and Mea- surement: An International Handbook, ed. J. P. Keeves, 471–83 (New York: Pergamon, 1988). Academic Library Critical Processes 163 APPENDIX Results of the Delphi Expert Panel Review of the Critical Processes and Per- formance Measures with Means and Standard Deviations Appendix description: The appendix has three columns. Column 1 includes the MB- NQA categories (e.g., leadership), MBNQA subcategories (e.g., leadership system), the critical processes in bold type (e.g., 1.1.1, 1.1.2, etc.) followed by their respective performance measures in normal type (e.g., 1.1.1.1, 1.1.2.1, etc.), which were reviewed by the Delphi panel and are arranged in a classified number schemata. Column 2 is the consensus mean for each critical process and performance measure. Column 3 is the standard deviation of the responses of the panel. Number Schemata: Rating Scale: 1. MBNQA Category 1 = very important 1.1: MBNQA Subcategory 2 = important 1.1.1: Critical Process 3 = of li le importance 1.1.1.1: Performance Measure 4 = not important CATEGORY 1: Leadership Subcategory 1: Leadership system Mean S.D. Critical Process 1.1.1. How leaders account for the needs and ex- pectations of all key stakeholders 1.4 1.01 1.1.1.1. What key stakeholders say about the degree to which senior leaders are in touch with the stakeholders’ needs and expectations 1.2 .66 Critical Process 1.1.2. How leaders communicate values and expec- tations and set directions 1.0 0 1.1.2.1. What faculty, staff, and key stakeholders say about visibility of senior leaders in communicating values, expectations, and in setting directions 1.4 .52 1.1.2.2. What employees say about visibility of senior leaders and the degree to which they are in touch with values 1.1 .33 Critical Process 1.1.3. How leaders promote an environment con- ducive to learning 1.1 .33 1.1.3.1. Existence of a clear plan for encouraging innovation and creativity 1.8 .78 1.1.3.2. Existence of a method or system of obtaining improvement ideas from employees 1.7 .66 Critical Process 1.1.4. How leaders communicate shared values, directions, and expectations 1.0 0 1.1.4.1. Extent to which strategies exist for rewarding all categories and functions of employees for behavior consistent with values 1.3 .5 1.1.4.2. Use of a variety of media to communicate quality values to faculty and staff 1.5 .52 Critical Process 1.1.5. How leaders review the leadership system 1.2 .44 164 College & Research Libraries March 2005 1.1.5.1. Existence of a systematic process for evaluating the integration 1.4 .72 of quality values into the management approach 1.1.5.2. Administrator evaluations and feedback mechanisms exist for 1.2 .44 all key administrators Critical Process 1.1.6. How senior leaders share the leadership 1.6 1.63 functions 1.1.6.1. Extent to which leadership opportunities are afforded other 1.0 0 staff Critical Process 1.1.7. How leaders develop leadership throughout 1.0 0 the organization 1.1.7.1. Existence of systems that reinforce the value of continual 1.0 0 learning, for example, reward systems, learning plans Subcategory 2: Public responsibility and citizenship Mean S.D. Critical Process 1.2.1. How the library makes risk factors and legal 1.6 .71 and ethical requirements an integral part of performance improve- ment 1.2.1.1. A systematic process exists to define standards and goals re- 1.7 .83 lated to matters of organizational citizenship and public responsibility 1.2.1.2. Evidence that demonstrates how key goals and standards for 1.8 .78 public responsibility and organizational citizenship are translated into operational policies Critical Process 1.2.2. How leaders communicate and promote 1.8 .33 opportunities for practicing good citizenship such as strengthening community services, the environment, professional associations, etc. 1.2.2.1. Existence of a system for monitoring the extent to which em- 2.0 1.0 ployee behavior is consistent with legal/ethical guidelines 1.2.2.2. Existence of a systematic approach to educate employees 1.5 1.01 regarding legal and ethical behavior 1.2.2.3. Evidence of involvement in service to key communities and 1.7 .66 professional associations CATEGORY 2: Strategic Planning Subcategory 1: Strategic development process Mean S.D. Critical Process 2.1.1. How the library develops its view of the 1.1 .33 future, sets directions, and translates these directions into a clear basis for communicating, deploying, and aligning critical require- ments 2.1.1.1. Evidence that process and technology capabilities/limitations 1.6 .71 are taken into consideration when developing long- and short-term plans and goals 2.1.1.2. Evidence of a systematic process being used to develop the 1.3 .71 library’s view of the future, set directions, and translate these directions into a clear basis for communicating, deploying, and aligning critical requirements. Academic Library Critical Processes 165 2.1.1.3. Evidence that faculty and staff capabilities/needs are identified for the strategic plan to be actualized 1.3 .51 2.1.1.4. Evidence that the library begins planning with a view of where it wants to be and works back to what has to be done today to develop the strategic plan 1.5 .83 Critical Process 2.1.2. How the library gathers input for strategic planning, including how key stakeholder needs and expectations and external and internal factors affecting the library are mea- sured 1.2 .66 2.1.2.1. Evidence that student and other user requirements are thor- oughly identified and that this information is used in developing goals and plans for the library 1.0 0 2.1.2.2. Inputs are obtained from all appropriate levels and functions in the organization prior to developing operational performance improve- ment goals and plans. 1.4 .72 Subcategory 2: Library organizational strategy Mean S.D. Critical Process 2.2.1. How the critical action plan requirements, such as faculty and staff development plans and needs, use learn- ing technologies, key measures and indicators, and resources are spelled out and deployed 1.4 .72 2.2.1.1. Evidence of a well-defined and workable process for approach and deployment of long- and short-term plans in the organization to ensure that quality and customer satisfaction are consistent with goals, objectives, and programs and services 1.6 1.0 2.2.1.2. Evidence that the manner of assigning and deploying resources is consistent with long- and short-term goals and quality services 1.3 .5 Critical Process 2.2.2. How leaders achieve alignment and consis- tency in key learning strategies and key measures 1.5 .52 2.2.2.1. Evidence that key learning strategies and key measures in the strategic planning process are linked to past learning strategies and measures and classified in the strategic plan 2.4 1.01 2.2.2.2. Extent that deployment of plans throughout the organization is part of evaluation 1.7 .97 Critical Process 2.2.3. What faculty and staff resource plans exist to support the overall strategy 1.4 1.01 2.2.3.1. Amount of faculty and staff resource plans that support the overall strategic plan 1.5 1.01 Critical Process 2.2.4. How projected performance relative to past performance and relative to comparable library organizations and benchmark processes are achieved 2.33 .86 2.2.4.1. Evidence to support the extent to which projections of perfor- mance relative to past performance, as well as compared to comparable library organizations, have been accurate 2.7 1.09 2.2.4.2. Comparisons of projections with performance levels attained by the organization(s) against which the library benchmarks. 1.4 .72 166 College & Research Libraries March 2005 CATEGORY 3: Student, Faculty, and Stakeholder Focus Subcategory 1: Knowledge of student and faculty needs and expec- tations Mean S.D. Critical Process 3.1.1. How the library determines the needs and expectations of its current and future students, faculty, and stake- holders to maintain a climate conducive to learning and inquiry for all students, faculty, and stakeholders 1.0 0 3.1.1.1. Identification of the common and unique requirements and expectations for each student and faculty member 1.3 .5 3.1.1.2. Number of contacts with students, faculty, and stakeholders that discuss needs and expectations 1.7 .66 Critical Process 3.1.2. How the library maintains awareness of key general and special needs and expectations of current students, faculty, and stakeholders 1.0 0 3.1.2.1. Evidence that data from satisfied and dissatisfied students, faculty, and key stakeholders is used to design, enhance, or change operations and/or services 1.0 0 3.1.2.2. Use of a systematic process to design operational and/or ser- vice features based upon student, faculty, and stakeholder requirements 1.2 .66 Critical Process 3.1.3. How the library determines and anticipates changing needs and expectations of future students, faculty, and stakeholders 1.0 0 3.1.3.1. Use of a systematic process to evaluate the importance of trends in student, faculty, and stakeholder requirements 1.0 0 3.1.3.2. Evidence exists on the amount of contacts made with future students, faculty, and stakeholders that discuss needs and expectations 2.0 .71 Subcategory 2: Student, faculty, and stakeholder satisfaction and relationship enhancement Mean S.D. Critical Process 3.2.1. How the library provides for effective rela- tionships with key stakeholders to enhance its ability to improve information delivery 1.0 0 3.2.1.1. Use of a planned and systematic approach to evaluate and improve service to students, faculty, and stakeholders for customer relationship management 1.2 .44 3.2.1.2. Performance evaluation of all functions in the organization is partially based upon the degree to which these functions assist student, faculty, and key stakeholder contacts in meeting the identified standards 1.6 1.0 Critical Process 3.2.2. How student, faculty, and stakeholder satisfaction and dissatisfaction is determined for use in improving the library’s ability to improve information delivery and support services 1.0 0 3.2.2.1. Comprehensive system or process exists for tracking student and faculty comments and complaints and what is done to address them 1.0 0 3.2.2.2. Extent to which students, faculty, and key stakeholders in all categories are included in customer satisfaction data 1.1 .33 Academic Library Critical Processes 167 CATEGORY 4: Information and Analysis Subcategory 1: Selection and use of information and data Mean S.D. Critical Process 4.1.1. How information and data are selected and 1.1 .33 managed for use in support of overall library goals, emphasizing action plans and performance measurement 4.1.1.1. Existence of specific criteria for selecting measurement indices 1.5 .53 for programs and services produced for internal and external stakehold- ers 4.1.1.2. Existence of a structure/mechanism for collecting information 1.3 .71 and data for strategic planning Subcategory 2: Selection and use of comparative information and Mean S.D. data Critical Process 4.2.1. How key factors in the selection and use of 1.6 .87 comparative information and data are selected for use in improv- ing performance relating 4.2.1.1. Evidence that a systematic process is used to review and follow 1.6 .87 up comparable comparisons and benchmark studies that are done 4.2.1.2. Evidence that a systematic process is used to evaluate process- 1.7 .97 es for gathering comparable and benchmark data Critical Process 4.2.2. How the library measures where it stands 1.8 .78 relative to other library organizations, compares information, and compares understanding of their own processes and processes of comparable library organizations 4.2.2.1. Evidence that a systematic process for selecting comparable 1.6 .71 institutions for comparison purposes 4.2.2.2. Scope and breadth of data collected on comparable institutions 2.1 .93 and benchmark processes Subcategory 3: Analysis and review of library performance Mean S.D. Critical Process 4.3.1. How information and data from all parts of 1.2 .66 the library are integrated and analyzed to assess performance 4.3.1.1. Evidence that a systematic process is used to review and follow 1.6 .5 up on comparable institutional and benchmark studies 4.3.1.2. Evaluation measurement indices include measures of internal 1.1 .33 customer satisfaction, process, and output quality Critical Process 4.3.2. How the library reviews performance and 1.0 0 capabilities, and uses the review findings to improve performance and capabilities relative to goals and plans 4.3.2.1. Evidence that analysis data have resulted in changes and im- 1.3 .5 provements in types of data collected and reliability of data 4.3.2.2. Existence of a structure or mechanism for review of perfor- 1.7 .83 mance of data- gathering methods and practices CATEGORY 5: Faculty and Staff Focus Subcategory 1: Work systems Mean S.D. Critical Process 5.1.1. How faculty and staff compensation and rec- 1.3 .5 ognition reinforces student achievement and library improvement 168 College & Research Libraries March 2005 5.1.1.1. Existence of a performance measurement and feedback system for all levels of employees, from senior leaders to individual contribu- tors 1.3 .71 5.1.1.2. Extent to which compensation and recognition are based upon the achievement of quality goals 1.7 .83 Critical Process 5.1.2. How consistency between the library’s compensation and recognition system and work structures and processes is defined 1.5 1.01 5.1.2.1. Evidence that a systematic, data-based approach is used to evaluate the effectiveness of the performance measurement, recogni- tion, and any quality-based compensation systems 1.6 .71 5.1.2.2. Existence of data that indicate levels of employee satisfaction with feedback, recognition, and performance-based compensation plans 1.7 .83 Critical Process 5.1.3. How work processes focus on student achievement and needs, and on communication, cooperation, knowledge, and skill sharing 1.2 .44 5.1.3.1. Trends showing compensation and recognition being based upon quality/customer satisfaction and operational results, rather than exclusively on financial results and seniority 1.5 .72 5.1.3.2. Trends showing proportional mix of rewards and recognition given out to both teams and individual employees 1.4 .72 Subcategory 2: Faculty and staff education, training, and develop- ment Mean S.D. Critical Process 5.2.1. How the library structures and encourages an effective education and training approach for faculty and staff 1.1 .33 5.2.1.1. Existence of a structure or mechanism for determining indi- vidual training needs 1.5 .52 5.2.1.2. Evidence that supervisors include education and training needs and suggestions in evaluations 1.2 .44 Critical Process 5.2.2. How the library evaluates the effectiveness of educational and training programs and approaches 1.3 .71 5.2.2.1. Existence of a feedback mechanism for measuring training effectiveness 1.3 .71 5.2.2.2. Amount and objectivity of data collected that indicates the degree to which employees apply knowledge and skills learned in educational courses and training 2.1 1.16 Subcategory 3: Faculty and staff well-being and satisfaction Mean S.D. Critical Process 5.3.1. How the library maintains a safe and healthy work environment 1.7 1.39 5.3.1.1. How employees feel about the work environment in terms of health and safety 1.4 .52 5.3.1.2. Extent to which goals and objectives on health and safety are integrated into work functions 1.5 .72 Critical Process 5.3.2. How the library measures well-being, satis- faction, and motivation of all library and staff 1.5 .72 Academic Library Critical Processes 169 5.3.2.1. Existence of a structure or mechanism for measuring faculty 1.8 .6 and staff well-being and satisfaction 5.3.2.2. Turnover rates 2.6 .5 Critical Process 5.3.3. How the library uses information gathered 1.6 .71 on the work climate, including faculty and staff well-being, satis- faction, and motivation, to improve the work climate 5.3.3.1. Amount of information on faculty and staff well-being and 1.7 .83 satisfaction used in performance improvement 5.3.3.2. Evidence that employee satisfaction problems are analyzed to 1.6 .71 determine their root cause Critical Process 5.3.4. How the library uses information on the 1.8 1.05 work climate to measure the interrelatedness of work climate and library organizational results 5.3.4.1. How data for employee satisfaction indices compare with 2.2 .83 employee satisfaction data from a parent institution (if any) and/or with comparable libraries 5.3.4.2. Existence of a structure or mechanism for use of work climate 2.2 .83 data tied to results CATEGORY 6: Library Program and Service Delivery and Support Management Subcategory 1: Library program and service delivery design and Mean S.D. delivery Critical Process 6.1.1. How the educational and information deliv- 1.4 .72 ery programs and offerings are designed, including formative and summative assessments 6.1.1.1. Evidence to suggest that existing services and programs have 1.1 .33 been designed based upon student and stakeholder requirements 6.1.1.2. Use of a systematic methodology to translate student and stake- 1.1 .33 holder requirements into services and program characteristics Critical Process 6.1.2. How the library’s educational and informa- 1.2 .44 tion delivery offerings are delivered 6.1.2.1. Use of an established and acceptable model for cause analysis 2.1 .78 6.1.2.2. Procedures exist and are used to verify that corrective mea- 1.3 .76 sures/actions produce desired results Critical Process 6.1.3. How information gathered in designing pro- 1.4 .52 grams and services are used to achieve better performance 6.1.3.1. Use of process modeling as a means for identifying opportuni- 1.6 .76 ties for improvement in processes and resulting operations and services 6.1.3.2. Use of comparable institutional data or benchmarks as stimuli 1.8 .6 for identifying opportunities for quality improvement Subcategory 2: Library support processes Mean S.D. Critical Process 6.2.1. How the library designs, implements, man- 1.3 .71 ages, and improves support processes 6.2.1.1. A systematic cause-analysis process is used to diagnose the 1.3 .71 causes of quality programs and process deviation that occur in support departments 170 College & Research Libraries March 2005 6.2.1.2. Systems are in place for measuring performance of support 1.3 .71 functions and feeding data back to the appropriate personnel. CATEGORY 7: Library Performance Results Subcategory 1: Student and clientele performance review Mean S.D. Critical Process 7.1.1. How student performance results are 1.2 .44 measured and linked to mission-related factors and assessment methods 7.1.1.1. Evidence that students’ performance has improved by access to 1.3 .5 library resources 7.1.1.2. Purchasing ratios (e.g., volumes added per faculty or per stu- 3.1 .78 dent) has been improved Subcategory 2: Student and stakeholder satisfaction results Mean S.D. Critical Process 7.2.1. How the library uses student, faculty, and 1.0 0 stakeholder satisfaction results to assess effectiveness of programs and services 7.2.1.1. Extent to which satisfaction results are used in process im- 1.0 0 provement Subcategory 3: Faculty and staff results Mean S.D. Critical Process 7.3.1. How the library uses the results of faculty 1.3 .71 and staff well-being, development, satisfaction, and performance to improve the library’s programs and services 7.3.1.1. Extent to which satisfaction results of faculty and staff are used 1.6 .71 to improve the work climate and to improve processes, programs, and services Subcategory 4: Library-specific results Mean S.D. Critical Process 7.4.1. What measures are unique to the library 1.4 .72 that indicate quality, improvement, or effectiveness 7.4.1.1. Evidence of the specific results that are unique to the library or 1.4 .72 unit being assessed 7.4.1.2. Evidence that a specific result that is unique to the library or 1.5 1.01 unit being assessed has improved from past performance or can be compared to benchmark institutions 7.4.1.3. Evidence of increased usage of library gateway information 1.1 .4