ACRL News Issue (B) of College & Research Libraries S eptem ber 1983 / 271 Bibliographic Instruction Evaluating Bibliographic Instruction: A Handbook E d ito r’s Note: C& RL News is pleased to have the opportunity to reprint (w ith permission) the Con tents, Preface and Introduction to the new hand book published b y the Bibliographic Instruction Section, E v aluating B ibliographic Instruction: A H andbook. The 129-page publication was coordi nated b y the BIS Research C o m m itte e ’s Subcom m ittee on Evaluation, whose m em bers are: M ig non S. A dam s, D avid Carlson, Bonnie G. Gratch, L a r r y H a r d e s ty , D a v id N . K in g , J o h n M a rk T ucker, R ich a rd H u m e W erkin g , and Virginia T ie fe l (c h a ir ). I t is n o w a v a ila b le f r o m A C R L /A L A , 50 E. H uron St., Chicago, IL 60611, a t $13 f o r A C R L m e m b e rs , a n d $17 fo r n o n m em bers. C o n t e n t s “E valuation and Its Uses,” D avid N. King. “E v aluating in Term s of Established Goals and O bjectives,” V irginia Tiefel. “ R ese a rc h D esigns A p p r o p ria te fo r B ib lio graphic In stru ctio n ,” Bonnie G. G ratch. “D a ta-G ath e rin g In stru m en ts,” Mignon S. Ad ams. “D a ta M anagem ent an d Statistical Analysis,” T. M ark Morey and Jacqueline Reihm an. “Significant W orks,” R ichard H um e W erking. “G lossary,” John M ark Tucker. “B ibliography” (Part I, A nnotated List; P a rt II, Basic Textbooks). P r e f a c e Because the rap id grow th of bibliographic in struction is a com paratively recent phenom enon, m any instruction librarians have simply not h ad th e tim e to ev alu ate w h a t they are doing; and m any lack the specialist knowledge about, or expe rience in, program evaluation. This handbook has been w ritte n by librarians (w ith one exception) for librarians both to offer an introduction to basic precepts cf evaluation and to furnish some direc tion to move beyond th e m aterial covered in the handbook. T he subtitle of A H andbook has been chosen be cause the authors wish to stress th a t this is not the definitive w ork in lib rary instruction evaluation. It is an introduction to the topic, intended to give general direction, and especially encouragem ent, to librarians attem p tin g evaluation in this im p o r ta n t area. T he handbook is a collection of chapters on v ari ous aspects of evaluation and each reflects the a u th o r’s approach to th e subject and style of w riting. T here is, th en , diversity in th e w riting; and m any w ho critiqued th e m anuscript noted some inconsis tency in style. But as there was no agreem ent on a uniform style (e.g., betw een m ore form al and in form al styles), th e decision was m ade not to a t te m p t a m ajo r ed itin g of th e m a n u scrip t. I t is hoped th a t readers will not be distracted by this, an d th a t, indeed, they m ight enjoy it. Also, the a u thors invite com m ents and suggestions, all of w hich w ill be retained an d referred to th e authors of a re vised edition. In closing, as chair I w a n t to th an k th e m em bers of th e com m ittee responsible for th e creation of this h a n d b o o k . No fin e r g ro u p of p ro fe ssio n a ls— w o rk in g over a long tim e an d d ista n ce—could have been found. T he contributions of several peo ple beyond th e com m ittee proved invaluable. They are K atherine Branch, W illiam C row e, E van F ä r ber, C onstance Finlay, E lizabeth Frick, M artin Gibson, Jam es Kennedy, Tom Kirk, M aureen Pas- tine, L inda Phillips, R onald Powell, D aniel Ream , Anne Roberts, and N ancy Taylor. BI E v a lu a tio n P ro g ra m T h e n ew ly p u b lish e d E v a lu a tin g B ib lio graphic Instruction: A H andbook (see the In troduction rep rin ted in this issue) will be p re sented for the first tim e to a m ajor gathering of librarians at th e M idwest F ederation of L ib rary Associations C onference in C in c in n ati. T he p r o g r a m , “ L ib r a r y In s tr u c tio n : T h e N ext Step—E v a lu a tio n ,” is scheduled for th e m o rn ing of N ovem ber 4 an d will feature a talk by M ignon A dam s, co o rd in a to r of in fo rm atio n services, SUNY-Oswego, a leading au th o rity on lib ra ry in stru ctio n an d a c o n trib u to r to th e handbook. T he tw o-hour session will be led by V irginia T iefel, d irec to r of lib ra ry user ed u catio n a t Ohio State University, w ho served as chair of th e ACRL com m ittee th a t p rep ared th e book. Several of th e region’s leaders in lib rary user ed ucation will assist in th e question-an d -answ er an d group discussions th a t comprise th e b a l ance of the session. This portion of the M FLA program will be sponsored by th e Ohio L ib rary Association’s Academic and Special L ibraries Division. 272 / C &R L N ew s Steps in the Evaluation Process Deserving of special thanks are M ignon Adams, w ho collected th e m anuscripts and supervised their p rep aratio n for p rin tin g , and G lenda King, who p rep ared the artw o rk for several of th e charts and tables. — Virginia T ief el, Ohio State University L i braries. I n t r o d u c t i o n T he evaluation of library instruction program s has been a subject of much discussion during the past tw o decades. Although th e re seems to have been an increase in the num ber of published evalu ation studies in recent years, it is difficult to tell w h eth er or not this reflects any significant increase in th e use of evaluation by th e m ajority of instruc tion librarians. E valuation is often assumed to be a complex, tim e-consum ing process; and for those u nfam iliar w ith the methods an d tools, it may seem an intim idating prospect. In fact, evaluation is w h a t you make of it. The process m ay indeed take on th e aspect of a sophisti c ated educational research endeavor, if th a t is w h a t you wish. But it is not always necessary to de velop large-scale, com plex projects in order to p r o f it fro m e v a l u a tio n . T h e process itse lf is straightforw ard, easily described in six basic steps. Step 1. Describe the purpose of the evaluation. The first step in any evaluation effort is to m ake sure th a t you fully u n derstand the reasons for eval uating. W ho w ants to know and w hat you hope to learn from th e inform ation you obtain will d eter mine the kinds of inform ation you need to collect and how you can best collect it. C hapter 1 ‚ “E v alu ation and Its Uses,” discusses many of the factors w hich should be considered, and introduces some of the pro m in en t approaches to systematic evalua tion. Step 2. Describe the program in terms of its goals and objectives. Once you have a clear idea of why the evaluation is to be undertaken, it is im portant to develop a description of the program as it cur rently exists. E ducational program s tend to evolve, and statements of program goals and objectives may not fully detail current practice. Be sure to look for any im p lied goals an d objectives th a t might not have been included in a form al state ment. C h ap ter 2, “E valuating in Terms of E stab lished O bjectives,” provides an overview of behav ioral goals and objectives, and explains a taxonomy of educational objectives. Step 3. D eterm ine the criteria to be used for eval uation. W hen the program has been described in sufficient detail, and the goals and objectives are clearly identified, evaluation criteria can be d eter m ined. If goals and objectives have been w ritten w ith care, this step is relatively easy. Just decide w h ic h goals a n d ob jectiv es sh o u ld be stu d ie d w ith in the context of the purpose of the evaluation, and w h a t standards w ould indicate success. But be sure to consider outcomes w hich m ay not have been anticipated by the goals and objectives, or w hich m ay indicate undesirable or counterproduc tive results of instruction. C hapters 1 and 2 include sections discussing the criteria th a t m ight serve as standards. Step 4. Develop the evaluation procedures and overall design of the study. After the criteria have been determ ined, procedures for conducting the evaluation can be developed. If the evaluation is to be a m ajor sum m ative effort, or if you intend to m onitor your program on a continuing basis by means of evaluation, you m ay find it useful to de velop an evaluation plan based upon one of the ap proaches discussed in C hapter 1. An appropriate evaluation design should be selected at this point. C hapter 3 provides an overview of some com monly employed designs. The selection of the design will clarify most of the procedures to be used in the eval uation, w hich instrum ents m ight be adopted, and the kinds of statistical analysis th a t will be neces sary. You m ay find it helpful at this point to set tim e ta b le s, d e te rm in e p ersonnel an d tra in in g needs, and budget for any costs you anticipate. C h a p te r 3, “ Research Designs A p p ro p riate for September 1983 / 273 Evaluating Bibliographic Instruction,” introduces concepts related to experimental designs and de scribes and illustrates their use. Step 5. Develop instruments and collect data. The fifth step in the evaluation process is to develop the instrument or instruments to be used to imple ment the study. The keys to success in this step in volve pretesting the instrument to eliminate any unforeseen errors, and applying the instrum ent systematically. Deviations in the way data are col lected can irreparably compromise the entire eval uation process and render the data useless. Chapter 4, “D ata G athering Instrum ents,” describes a vari ety of instruments which may be used to collect in formation, and offers practical advice on choosing an instrument and/or developing your own tests and questionnaires. Step 6. Analyze the data and report the results. Once the data are collected, they should be inter preted w ithin the context of the criteria established in Step 3. The analysis should point out which cri teria were successfully achieved, as well as identify components of the program th at need improve ment. Chapter 5, “D ata Management and Statisti cal Analysis,” presents the core concepts of statisti cal analysis and describes some of the procedures most useful for the analysis of instructional data. Even if a report to outside clientele is not required, w riting up the result will help in interpreting and assessing the evaluation results. The techniques and methodologies introduced in this handbook may be employed in a number of ways to help understand the effects of your pro gram and improve the quality of instruction. Care ful planning and attention to detail are necessary throughout the process. You will not find the answers to all your evalua tion questions here. Each author provides refer ences for further reading, and an annotated bibli o g ra p h y of suggested re a d in g s is in c lu d e d . Examples of evaluations and some of the more im portant sources for information on the evaluation of bibliographic instruction programs are included in Chapter 6, “Significant W orks.” A glossary is also provided. Even if you follow up on all of the sources mentioned, we encourage you to take ad vantage of the expertise of others on your campus. You need not be an expert to succeed at evalua tion. Evaluation is not an exact science, and crea tivity, resourcefulness, and perseverence yield im pressive results. Your programs and your students will benefit from the effort.— David King, Hous ton Academy o f Medicine, Texas Medical Center Library. ■ ■