Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2010, 5.4 21 Evidence Based Library and Information Practice Article Using ACRL Standards to Assess the Information Literacy of Graduate Students in an Education Program Amy Jo Catalano Assistant Professor of Library Services Hofstra University New York, United States Email: Amy.Catalano@Hofstra.edu Received: 14 July 2010 Accepted: 23 Oct. 2010 2010 Catalano. This is an Open Access article distributed under the terms of the Creative Commons- Attribution-Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc- sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective - This study investigates the information literacy of graduate education students, including those in doctoral cohorts. The Association for Research and College Libraries Information Literacy Standards were used a baseline for measurement. Methods - A survey was sent to all graduate students in the School of Education; it asked a combination of questions measuring students’ perceptions of their information literacy skills and testing their knowledge of information literacy. Results – A total of 172 surveys were returned. The results indicated that while there is a heavy reliance on internet sources, many students were able to determine which sources were reliable and which were not. After attending information instruction sessions, students were more familiar with library services and more inclined to use them. Conclusion - It was determined that a one credit course or multiple sessions of library instruction would better serve graduate students completing capstone projects. Introduction With the increased emphasis within academia on student-centered learning and student learning outcomes assessment by discipline- based and regional accrediting bodies, many accrediting agencies are also beginning to stress the importance of standards-based information literacy skills for students in higher education. This article focuses on the mailto:Amy.Catalano@Hofstra.edu� Evidence Based Library and Information Practice 2010, 5.4 22 assessment of standards-based information literacy skills for graduate students at Hofstra University, a large private institution located on Long Island, New York, USA. Specifically, survey methodology was utilized to determine a baseline for graduate students’ information literacy skills, compared with the Association of College and Research Libraries (ACRL) Standards for Information Literacy (IL) Competency for Higher Education. The Middle States Commission on Higher Education (MSCHE), along with other key accrediting agencies, are proponents of the integration of information literacy as part of the standards of accreditation (Thompson, 2002). Towards that end the MSCHE has adopted the Association of College and Research Libraries (ACRL) Information Literacy Competency Standards, which were developed to measure a learner’s ability to "recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information" (Thompson, 2002, p. 222). In this study, the focus is on education graduate students. These students often complete capstone projects such as dissertations or master’s theses. These future teachers must be information literate themselves if they are to teach the same competencies to their students. In addition, professors in education expect their students to use research-based practices in their teaching as well as research, therefore it is essential that students know how to find and evaluate relevant sources. It is of concern to academic librarians that students who are engaged in the research process are often unaware of library resources, find them difficult to use, or use them ineffectively. Since the ACRL Standards for IL provide us with a framework for assessing the competencies necessary to become an information literate learner both in and out of a student’s academic pursuits, it then follows that we would use these standards to guide us toward the main research question of this study: “Using the ACRL standards for IL as an assessment tool, to what extent are graduate students in education information literate?” Literature Review A review of studies measuring the information literacy, research, or library skills of graduate students revealed that few of the studies use standards as the basis for assessment and even fewer use the ACRL IL standards. Three notable exceptions are studies conducted by Ferguson, Neely, and Sullivan (2006) at the University of Maryland, Emde and Emmett (2007) at the University of Kansas, and Berg and Grant (2003) at San Diego State University. While there is abundant literature on the information literacy skills of pre and in- service teachers, this literature review focuses on skills of graduate education students rather than those of undergraduate education majors, in line with the aims of the study. Oakleaf and Kaske (2009) offered guidance on assessing IL in higher education for academic librarians. They focused on the variety of measures available and how to choose the best one for whichever goals an institution wants to achieve. While they did mention briefly the “Information Literacy Competency Standards for Higher Education,” the authors did not discuss the incorporation of the ACRL IL standards into assessments. A study that reported a positive relationship between graduate education students and information literacy was conducted by Berg and Grant (2003). The research discussed incorporating information literacy into a doctoral program using the ACRL standards for IL, and evaluating the contents of existing courses to identify where the five standards were covered and to what degree of adequacy. Their findings showed that doctoral students in all courses engaged in activities corresponding to each standard. Similarly, Green (2006) discussed a librarian-created education doctoral level course that incorporates information literacy skills into its content. The course is constructed in such a way that students are expected to demonstrate that they are information literate through the searching, identifying, and evaluating of information collected for literature reviews. As students progress in the program they build Evidence Based Library and Information Practice 2010, 5.4 23 on those information literacy skills as they learn to use them in more specialized ways. Emde and Emmett’s (2007) research focused on an assessment tool design based on desired learning outcomes using the ACRL standards as a framework. When designing their assessment tool, they asked “Does the question asked provide a valid measure of what the instructor wants to know about the students’ skills?” (2007, p.212). The authors discerned that the strength of studies such as theirs lies in the formation of questions created to measure real problems that would be encountered in a graduate student’s research. Thus questions used in their assessment tool, which was implemented during a one credit library instruction course for graduate students in Chemistry, used the ACRL standards to create activities or questions that would allow the student to demonstrate the desired learning outcome. Unfortunately, the authors did not enumerate their findings as they corresponded to the Standards. They generalized their findings by whether students passed or failed pre and post tests. Perrett (2004) surveyed graduate students’ information literacy skills using a similar methodology to the study described below. The students first evaluated their own skills and then performed specific tasks in order to compare the two. Perrett found that 47% of the students accurately evaluated their skill level. More specifically, many students over- estimated their internet searching skills. Based on students’ performance they were provided with recommendations for further training to improve their skills; 64% of those students were advised to enroll in a bibliographic instruction course. Earp (2008) studied the information source preferences of education graduate students and concluded that graduate students prefer information that is easily accessible even if it may be unreliable; they prefer electronic access; and they are unaware of many library resources and services such as interlibrary loan. These characteristics of student information seekers, whether graduate or undergraduate, are often shared throughout the disciplines. Unsurprisingly, doctoral students tended to be more diligent in their information seeking. Zaporozhetz (1987) reported that doctoral candidates stated that their faculty advisors expected them to possess advanced bibliographic skills that the advisors themselves did not have. An extension of the issue is illustrated by both Zaporozhetz and Labaree’s (2003) argument that because doctoral students in the field of education are accomplished professionals they may have difficulty admitting that they lack library search skills. Therefore, despite the acknowledged need by doctoral students of possessing advanced library research skills, it is up to the institution and its faculty to ensure that students get this training. Morner (1993), testing the library research skills of doctoral students of science education, concluded that these students were not well equipped for doctoral-level research. Similarly in Alire’s (1984) study of education doctoral students’ attitudes regarding the importance of library usage and the need for bibliographic instruction (BI), she found that over two-thirds of the students thought that knowledge of the library and its resources was important to their academic success. More than half of the students thought that they had deficiencies in library skills and felt that they could benefit from BI. Finally, the majority of the students surveyed stated that a course in library research methodology should be required in programs where dissertations were mandatory and that they would take such a course. Methods The 24 question survey (Appendix A) was modeled after the one created at the University of Maryland, Baltimore County (Ferguson, Neely & Sullivan, 2006) and based on the ACRL Information Literacy Competency Standards for Higher Education. Many of the questions were adapted from the Evidence Based Library and Information Practice 2010, 5.4 24 Neely Test of Relevance, Evaluation, and Information Literacy Attitudes. The first part of the survey posed questions about progress within a program as well as questions relating to the respondents’ occupation and experience using library sources. One question allowed for comments about major sources of frustration during students’ course of study. The second half of the survey presented a combination of questions designed to measure information literacy skills and the students’ perceptions of their information literacy. The author found self-reporting methodology the best way to illustrate the gap between real and perceived information literacy. This is important because, as described above, graduate students in education are often experienced and exceptionally capable performers in their field and reluctant to admit a deficiency in knowledge. The purpose of this study was to inform decision makers in schools of education of common phenomena among education graduate students. The survey was sent to all graduate students enrolled in the School of Education and Health and Human Services (SOEHHS) as well as graduate students in the Speech-Language and Hearing program and the Clinical Psychology program, a total of 1,770 students. The latter two departments were selected because they both have a required 6 hour workshop on the use of library resources whereas SOEHHS does not require their doctoral students to complete such a course. Comparisons of the abilities of those who do participate in such a workshop will be made in the results and discussion sections of this paper. The survey was emailed twice within a two week time period in the spring of 2008. In between each mailing, the directors of doctoral and master’s programs were informed of the study and asked to encourage their students to participate. A total of 172 surveys were returned (9.7% response rate). Of the 172 respondents, 34 were doctoral students while the remainders were graduate students working on either their first or second master’s degree or graduate certificate. Findings and Discussion Overview The level of academic and professional experience of the respondents was noted: 41% of the respondents had completed 5 courses or less while 100% of those in the Education Doctoral program were either school administrators, teachers or were professionals in the field of education. During the course of this study and based on the literature review, it was discovered that Ed.D or Ph.D education students are often very different from other types of Ph.D students in that they often have well-established, full-time, careers as professionals and go to school part-time or during a summer cohort program. This may indicate different skill levels among these students as compared to graduate students in other disciplines (Beile & Boote, 2005). In response to the question, “What is your biggest challenge when looking for information?” 49% stated that writing in and following APA style is their biggest challenge while 45% of respondents said it was the inability to find relevant literature for their research; 30% expressed that they can’t effectively use the online databases, while 27% stated identifying an original research topic is a challenge. Use of Library Services and Experiences The majority (80%) of respondents reported they accessed library resources remotely, and 50% of them accessed these resources at least once a week. When using library resources, whether in person or remotely, 83% accessed online databases while only 53% borrowed books or other materials. Interlibrary loans were used by 29%, but less than 1% used the Research Libraries Program (RLP). The RLP allows students to borrow materials from neighboring academic libraries. The document delivery service was used by 9% of respondents and 36% use the library’s print Evidence Based Library and Information Practice 2010, 5.4 25 journal collection. Librarians came to classrooms for library instruction for 44% of respondents and 87% of those that have had instruction claimed that such instruction helps them to better navigate the library’s resources. While this study reports a relatively low use of interlibrary loan, as doctoral students and faculty learn about the availability and purpose of interlibrary loan, they tend to rely on it to complete their literature reviews. Based on past experiences, some reasons why there is low use of interlibrary loan at the master’s level may include the lack of knowledge of its existence or how to access and use it. Many students are also surprised at how easy it is to place an interlibrary loan request. Other reasons for low use include the fact that libraries have more full-text available and the increasing availability of open access journals. ACRL Standards for Information Literacy Competencies The following section reports the results of the questions pertaining to each of the five ACRL Standards for IL Competencies. No numerical passing score was attributed to this survey, as many of the questions were attitudinal. A discussion of the results follows each standard. Standard 1: “The information literate student determines the nature and extent of the information need.” The survey included 4 questions to measure this standard; the first two related to students comfort levels with information sources. The majority (92%) of students felt equally comfortable finding information from a search engine and their professor, while many (77%) felt comfortable with the library webpage or the library itself (73%). In summary 85% reported that they were either very comfortable or comfortable identifying potential sources of information. Generally this response was confirmed by examining the types of sources students reported to rely on, such as their professors. While this finding doesn’t directly address the performance indicator for Standard 1: “Confers with instructors and participates in class discussions, peer workgroups, and electronic discussions to identify a research topic, or other information need” (ACRL Information Literacy Standard 1.1), it could be inferred that a professor of a course would be consulted by a student when attempting to satisfy an information need presented by that course. However, while reporting on graduate education students, Earp (2008) found that the majority of students surveyed rarely relied on professors, classmates or librarians. The third question asked “Given the topic ‘Students with ADHD in the mainstream classroom’ as the subject of a research project, which of the following steps would you do first to begin your project?” Almost half (44%) of students reported that they would begin their search with the internet using the terms “ADHD” and “mainstream classroom,” whereas 22% said that they would search subject specific databases. Only 14% would formulate questions based on the information need. None of the students would use a reference source on special education. The results of question three strongly indicate the move from traditional reference sources to the internet. Research findings from Martin (2008) concurs with this finding. While students are avoiding reference books, replacing them with internet searches in order to increase knowledge of topic would support the performance indicator “exploring general information sources to increase familiarity with a topic” (ACRL Information Literacy Standard 1.1 ). Conversely, Ferguson, Neely and Sullivan (2006), reported that students consulted a reference source at some stage in their research but rarely in the beginning. Evidence Based Library and Information Practice 2010, 5.4 26 Table 1 Standard 2 n= 161 Standard 2 Q. 1.How frequently do you use the following search strategies Always Often Sometimes Rarely Never Don't know what that is Truncation >1% 3 10 8 6 72 Boolean Operator: NOT 2 6 9 22 21 40 Boolean Operator: OR 4 11.5 18.5 15 12 39 Proximity Operators 2.3 2.3 6.9 8.4 9.1 80 Library of Congress Subject Headings 3 6.9 10.7 14.5 31.3 33.6 ERIC Descriptors 13 20.6 25.3 7.6 13.7 19.8 Cross and Multiple-Field searching (such as by date) 7 26 28 10 14 15 These responses suggest that students rely less on the library, possibly indicating negative experiences with using library sources, such as difficulty navigating complex databases. This result conflicts with the 87% of students who say they perform better after librarian instruction. Standard 2: “The information literate student accesses needed information effectively and efficiently.” The survey included 4 questions used to measure this standard. The first question asked “How frequently do you use the following search strategies: Boolean operators, Truncation, ERIC descriptors and Subject Headings?” A performance indicator demonstrating competency of this standard includes using effective search strategies. For this aspect of the standard of information literacy, many students were deficient. For all search strategies listed on the survey, at least 20% had no knowledge of the term (see Table 1). Another question asked, “During which events do you ask a librarian for assistance?” Most respondents knew when to consult a librarian (ie. advice for where to look for information, or how to use a resource), while 10% did not know when a librarian’s expertise would be helpful. The third question asked, “Where would you go or what would you do to find current information on the following topic: ‘Terrorism on college campuses’?” This question allowed for multiple answers. Nearly 20% said they would go to the internet for the needed information; 14% said they would go to newspaper archives even though the word “current” was emphasized in italics and 12% said they would go to databases. The answers to question one appeared to illustrate that students were deficient in this area. Even though 71% claimed to be comfortable or very comfortable with developing successful search strategies, most students did not demonstrate knowledge of a variety of search tools such as the Library of Congress Subject Headings, truncation, or ERIC descriptors and thesaurus. Similarly, reporting on undergraduate students, Ferguson et al. (2006) indicated that 66% of students surveyed had never used proximity operators or truncation and nearly 74% rarely or infrequently used Library of Congress Subject Headings, ERIC descriptors or other controlled vocabulary. Perrett (2004) reported in her audit of graduate students that less than half of were able to use Boolean operators and only 66% used truncation searching. The lack of knowledge on how to use search tools and strategies may cause in a graduate student to miss out on relevant bodies of work for their research problem. Boote and Beile (2005) Evidence Based Library and Information Practice 2010, 5.4 27 discussed this issue in their article about literature reviews in the education doctoral dissertation. It is essential to the quality of the dissertation that the literature review be complete. Often, without thorough bibliographic training, one cannot do a complete search of all relevant literature. This includes finding literature, retrieving it, and evaluating its value and relevance to the student’s work. One weaknesses of the current study is the formation of questions about search strategies. Many search engines, including Google and some subscription databases, already use Boolean logic. While the student may not have heard of the term “Boolean logic” they may already be implementing it or having it done for them automatically by the database while constructing a search query. This may also apply to truncation, date limiters, and relevance rankings. Saunders (2008) addressed the issue of information retrieval systems and IL standards by concluding that increasingly more and more information retrieval systems (e.g. subscription databases, search engines, or online library catalogs) “include design enhancements meant to improve the precision and recall of the user’s search results by expanding or refining the original query with related terms… [in order to] expand a narrow search” (p. 92). Similarly, other systems provide relevance rankings, relevance feedback, date ranking, as well as thesaurus- enhanced searching. These design enhancements have implications for IL standards as they can enhance the user’s “abilities” to search effectively for the needed information. Further, these enhancements allow instruction librarians to focus on the other aspects of information literacy that comes after information retrieval (such as evaluating research, identifying bias etc.). Those in the education program, as well as those who major in school psychology, tend to rely heavily on the ERIC database. Knowledge of ERIC descriptors and limiters or use of the ERIC thesaurus can be helpful search tools. Ignorance of possible search strategies may cause undue frustration with the research process. Standard 3: “The information literate student evaluates information and its sources critically”. The survey included 3 questions to assess this Standard. The first asked “Can one evaluate an article for bias before reading it?” The majority (76%) of respondents provided answers that demonstrated knowledge of indicators of bias whereas 24% did not know the answer to the question. The next question asked, “Would you find any of the following resources credible or reliable for your research?” Choices were given that ranged from New York Times and CNN to Saturday Night Live’s Weekend Update and the National Enquirer. The large majority of the respondents appeared aware of which sources listed in this question were satirical sources (e.g. Saturday Night Live Weekend update, The Onion, and Jon Stewart’s Daily Show) and which are generally reliable sources (e.g. New York Times and Washington Post). Eight percent thought that these newspapers were never credible. The results of this question were surprising as there appeared to be no consistent or predictable answers. For example, 11% of respondents found that the National Enquirer was at least sometimes credible and 24% said World News Tonight was never credible.(see table 2 for full results). The majority (71%) claim to be comfortable or very comfortable with evaluating information, claims which are supported by the results. However, Ferguson et al. (2006) reported different results for this same question. For example 58% of students, in the Ferguson et al. study, stated that Time or Newsweek was always reliable and that 63% said that the NY Times was reliable. Twelve percent of the Ferguson study found the Saturday Night Live Weekend update credible. This discrepancy may indicate a shift in the perception of credibility from undergraduate to graduate students. Evidence Based Library and Information Practice 2010, 5.4 28 Table 2 Standard 3 n= 157 Standard 3 Q. 2 When would you find any of the following sources reliable or credible ? Always Sometimes Never National Geographic 36.22% 48.82% 14.96% New York Times or Washington Post 25.98% 66.14% 7.87% CNN 24.41% 61.42% 14.17% Time or Newsweek 22.83% 65.35% 11.81% Wikipedia 15.75% 40.16% 44.09% World News Tonight/CBS evening news 14.96% 61.42% 23.62% The Daily News 3.94% 48.82% 47.24% People 3.15% 31.50% 65.35% Jon Stewart's The Daily Show 3.15% 20.47% 76.38% Sports Illustrated 2.36% 41.73% 55.91% Today Show/Good Morning America 2.36% 54.33% 43.31% The Onion (website) 0.00% 18.90% 81.10% Rolling Stone 0.00% 41.73% 58.27% National Enquirer 0.00% 11.02% 88.98% Saturday Night Live's Weekend update 0.00% 11.02% 88.98% The last question that measured this standard asked the respondent to indicate which of seven statements were true when it came to information found on the internet. The majority (90%) said material on the web comes from many varied sources such as business, the government, organizations, or the public, 3% thought the web is more reliable than books and magazines, 2% thought material on the web is factual because the web is monitored by international organizations. Standard 4: ”The information literate student uses information effectively to accomplish a specific purpose”. This standard addresses the ability of the student to organize information and communicate the information for a specific purpose. The survey included two attitudinal type questions corresponding to this standard. This is a difficult standard to assess without demonstration of the competency such as a research paper. An attempt to make an assessment was made by asking students to indicate their comfort levels with specific skills: integrating new information into an existing body of knowledge and organizing information for practical application. The majority (74%) claimed to be very comfortable or comfortable with integrating new information into an existing body of knowledge whereas, 12% were neutral, and 4% percent were uncomfortable or very uncomfortable. Similarly, 77% were very comfortable or comfortable with organizing information for a practical application, whereas 20.5% were neutral, and 2.5% were uncomfortable or very uncomfortable. It is recommended that a method be devised for students to model these skills for future research. Standard 5: “The information literate student… uses information ethically and legally” This standard addresses knowledge of freedom of speech and the use of copyrighted information and was assessed with two questions. The first asked, “What is the best definition of ‘intellectual freedom’?” A range of Evidence Based Library and Information Practice 2010, 5.4 29 responses were given and multiple answers could be selected. “It is the right of every individual to both seek and receive information from all points of view without restriction;” was selected by 68% of respondents, 28% defined intellectual freedom as the encouragement of open and public sharing of ideas; and 3% believed it is to support the bill of rights. However, a small minority (3%) believes it means to prevent cheating by students or to limit access to ideas that some people find objectionable and dangerous. When asked to define intellectual freedom, 94% were able to answer correctly. The second question relating to this standard sought to measure a respondent’s knowledge about plagiarism and the web. When asked if it were true that all information taken from the web must be cited properly only 45.5% answered yes, while 13.5% answered that one may use all text and graphics freely unless they were copyrighted. Similarly, Ferguson et al.’s (2006) respondents also were unsure of when they could use items freely from the internet; 22% either did not know or responded incorrectly about using information from a website. The results of this survey indicate a lack of awareness about several necessary aspects of bibliographic knowledge. On the one hand, the majority of respondents demonstrated knowledge about the reliability of web and other sources, yet there is a heavy reliance on internet sources. Perrett (2004), Martin (2008), and Earp (2008) all confirmed this finding. Students often lack an understanding of the many sources available to them (such as print indexes and subscription databases) and how to use them effectively. Given many students rely on the web rather than library sources, it must be questioned whether students would be more inclined to use library sources more frequently, if they knew their value. Martin (2008) found that even with library instruction undergraduate education students from his study still relied on the web and avoided the databases. One reason cited is the complexity of databases. Earp (2008) also pointed out that since students often rely on full-text of articles readily available, faculty and librarians need to stress “sometimes the best information is still located in print” (p. 84). As previously noted, the study conducted by the University of Maryland (2006) concluded that while students often go to the web first, they perceive they are comfortable using sophisticated research methods and search strategies when they in fact demonstrate a lack of familiarity. This current study supports this conclusion. Comparison between Students Receiving Library Instruction The survey was also administered to students in the Speech, Language and Hearing program and the Psychology doctoral program. The results are intended to be used as a basis for comparison as the students in those programs are required to attend mandatory library instruction sessions. The instruction consisted of two 3 hour sessions that included an orientation to the physical library, the library website, library services, finding books both in the library’s online catalog, as well as catalogs of other libraries (such as Worldcat), navigating the research databases that are specific to Psychology and Speech, and how to use APA citation style. The sessions were taught by a Psychology and Speech subject specialist and allowed for hands-on searching both in the physical library and online. The subject specialist librarian also demonstrated the use of websites specific to the professions of Psychology and Speech. Pre- and post-tests were given during the sessions in order to measure the effectiveness of the instruction. While the sample is too small to be statistically significant the data is worth noting. Fifteen surveys were returned from students in these two programs. Even though these students are required to attend library instruction, 2 of the 15 respondents did not. Some of the most notable differences between the entire pool of respondents and those in the group mandated to take library instruction include: 73% use interlibrary loan as opposed to 29% of the whole group, and nearly 94% knew during which events to go to a librarian while only 89% of the whole group knew. Table 3 Evidence Based Library and Information Practice 2010, 5.4 30 shows that those receiving library instruction, on the whole, were more familiar with terminology surrounding search strategies, with the exception of ERIC limiters (see Table 3). Surprisingly, 18% answered the question incorrectly as to whether information needs to be cited when it is taken from the internet as opposed to 13.5% of the whole group getting that question incorrect. Generally these results suggest those who attended the mandated library instruction session perceived themselves to be more confident about using different sources or identifying different potential uses of information. They also tend to be more knowledgeable about the services the library provides such as reference services, interlibrary loan, and databases that are appropriate for their field of research. It is also more likely that they are taught about specific search strategies using these databases. These students tend to rely less on internet search engines and websites. The entire group demonstrated weaknesses in their understanding of the reliability of Internet resources, identifying bias, and reliance on the web for background information or news. These results suggest at least one library instruction session should be offered to all graduate students who must complete capstone projects. It would also be prudent to offer such information sessions to teachers-in- training, if means of assessing these competencies is not already in place in the teacher education program. This is particularly pertinent as information literacy competency is a requirement of both the Teacher Education Accreditation Council (TEAC) and the National Council for Accreditation of Teacher Education (NCATE) (Birch et al. 2008). Four of the six NCATE standards are parallel to the ACRL standards. As the results of this survey indicate, students often think they are more capable than they are able to demonstrate. Many librarians, teachers, and professors experienced with working with students know that they often go to the internet first for information, and they aren’t quite familiar with sophisticated search terms, strategies, and library jargon. Students often don’t know the full range of library services and some don’t even know what a librarian does. For example, many students confuse e-books or e-libraries for article databases, thus often resulting in fruitless, inefficient, or frustrating searches, further resulting in gaps in their literature reviews. The results of other studies on the subject of information literacy of graduate students such as those completed at the University of Maryland (2006), Zaporozhetz (1987), Boote and Beile (2005), Alire (1984), and Martin (2008). Each study corroborates these results and confirm that regular assignment based bibliographic instruction sessions better facilitates success in graduate studies. Table 3 Comparisons of the Entire Group of Student Surveyed vs. Speech/Psych Students n=15 Comparison of groups of students Whole group Speech/Psych students Don't Know What it is Don't Know What it is Truncation 72% 61% Boolean NOT 40% 20% Boolean OR 39% 23% Library of Congress Subject Headings 80% 55% Proximity Indicators 33.6% 36% ERIC Limitors 19.8% 38% Cross and Multiple-Field Searching 15% 8% Evidence Based Library and Information Practice 2010, 5.4 31 An orientation program for graduate/doctoral students, that incorporates a library resources workshop is another suggestion that has been implemented at several institutions. The experiences of Northwestern University in implementing such a program have been documented by Lightman and Reingold (2005). Limitations of This Study Throughout the course of this research, several limitations and weaknesses of the survey were identified. The return rate of surveys was only 9.7% whereas an optimal rate would have been at least 22%. Further, different results may have been determined had the differences between an online or internet source and database been more clearly defined. It is very possible when students said that they are consulting online sources they are referring to databases. While this study used the terms “search engines” and “websites” to refer to non-library, non-subscription databases, it is impossible to know what students actually understood those terms to mean. An additional limitation is that the use of Boolean operators is somewhat outdated as some search products don’t use them anymore. In the section comparing the Psychology/Speech Language group who received library instruction to the entire sample it was impossible, given the survey tool used, to remove individual respondents and triangulate the data. Thus the responses of the Psychology/Speech Language group are included in the whole group responses, distorting the comparison of the results. However, the results were worth including as they did indicate a difference in the quality of information learned specifically around database specific search strategies. Lastly, while this study strove to measure students’ perception of their abilities as compared to what they could actually demonstrate, this goal would have been better measured with more questions that asked students to apply the IL skills applicable to the standards. The study does not examine information use, but the mechanics of information processing (searching, finding, and evaluating). Data on more complex information processes, such as organization and communication rely on respondents self- reporting, which is not the same as establishing that they can perform these tasks. Therefore, opportunities to demonstrate information processing would have been more illustrative than self-reporting perceived abilities Examples of questions, such as those designed by Emmette and Emde (2007) were based on ACRL guidelines as well as being specific to their subjects’ major, chemistry, are a good model for a future survey. Conclusion and Implications for Future Research This article demonstrates the need to inform library instruction practice based on evidence based research. While professors and librarians may assume that graduate students have limited abilities when their research skills are measured against IL standards, the research has shown that graduate students are far better at evaluating information sources than they may be given credit for. In contrast, they do need advanced bibliographic search skills and strategies that are often not addressed in traditional “one-shot” library instruction sessions. Furthermore, multiple sessions maybe necessary throughout a student’s graduate career, as information needs develop and change. Further suggestions for future research include: define the learning outcomes for the graduate programs as they relate to information literacy and construct more quantifiable ways to assess the acquisition of IL and research skills. Also, individual interviews with students would allow them to exhibit their knowledge of database use and assessments that ask students to demonstrate their ability to construct searches, find full text of articles, use interlibrary loan, and find statistical sources that are appropriate to their research. It is considered best practice to tailor Evidence Based Library and Information Practice 2010, 5.4 32 questions meant to assess information literacy standards to the student’s subject major. Further, employing inferential as opposed to descriptive statistics (as used in this study) would provide an evaluative basis on which to evaluate library instruction specifically created for doctoral students. Future research also needs to be conducted on the differences on information literacy and search abilities of doctoral students completing their dissertations in comparison to students studying at other graduate levels, as the level of research needed to complete a dissertation is far more complex and ongoing than that of a master’s student. Isolation of the abilities of doctoral students would allow for tailored evidence based means of instructing students at this level. References Alire, C. A. (1984). A nationwide survey of education doctoral students’ attitudes regarding the importance of the library and the need for bibliographic instruction (Doctoral dissertation, University of Northern Colorado). Retrieved from Dissertation Abstracts. Association of College and Research Libraries (ACRL). (2000). Information Literacy Competency Standards for Higher Education. Berg, M., & Grant, M. (2003). Information literacy integration in a doctoral program. In D. M. Shinew & S. Walters (Eds.), Information literacy instruction for educators (115-128). New York: Haworth Information Press. Beile, P., & Boote, D. N. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34(6), 3-15. Birch, T., Greenfield, L., Janke, K., Schaeffer, D., & Woods, A. (2008). Partnering with librarians to meet NCATE standards in teacher education. Education, 128(3), 369-379. Brown, C. M. (1999). Information literacy of physical science graduate students in the information age. College & Research Libraries, 60(5), 426-38. Earp, V. (2008). Information source preferences of education graduate students. Behavioral & Social Sciences Librarian, 27(2), 73-91. Emmett, A., & Emde, J. (2007). Assessing information literacy skills using the ACRL standards as a guide. Reference Services Review, 35(2), 210-229. Ferguson, J.E., Neely, T.Y., & Sullivan, K. (2006). A baseline information literacy assessment of biology students. Reference & User Services Quarterly, 46(2), 61-71. Green, R. (2006). Fostering a community of doctoral learners. Journal of Library Administration, 45(1), 169-183. Labaree, D. F. (2003). The peculiar problems of preparing educational researchers. Educational Researcher, 32(1), 13-22. Lightman, H., &. Reingold, R. N. (2005). Martin, J. (2008). The Information Seeking Behavior of Undergraduate Education Majors: Does Library Instruction Play a Role?. Evidence Based Library And Information Practice, 3(4). Retrieved from http://ejournals.library.ualberta.ca/ind ex.php/EBLIP/article/view/1838/3696 Morner, C.J. (1993). A test of library research skills for doctoral students. PhD diss.(Doctoral dissertation, Boston College). Retrieved from Dissertation Abstracts. Oakleaf, M., & Kaske, N. (2009). Guiding questions for assessing information Evidence Based Library and Information Practice 2010, 5.4 33 literacy in higher education. portal: Libraries and the Academy, 9(2), 273-286. Perrett, V. (2004). Graduate information literacy skills: The 2003 ANU audit. The Australian Library Journal, 53(2), 161-171. Saunders, L. (2008). Exploring connections between information retrieval systems and information literacy standards. Library & Information Science Research, 30, 86-93. Singh, A. B. (2005). A report on faculty perceptions of students’ information literacy competencies in journalism and mass communication programs: the ACEJMC survey. College & Research Libraries, 66(4), 294-310. Thompson, G. B. (2002). Information literacy accreditation mandates: What they mean for faculty and librarians. Library Trends, 51(2), 218-241. Zaporozhetz, L. E. (1987). The dissertation literature review: How faculty advisors prepare their doctoral candidates. PhD diss. (Doctoral dissertation, University of Oregon, Eugene). Retrieved from Dissertation Abstracts. Evidence Based Library and Information Practice 2010, 5.4 34 Appendix A Survey: Information Literacy and Library Services for Graduate/Doctoral Students in Education Programs 1. What Program are you currently enrolled in? ____________________________________ 2. How many courses have you completed so far?___________________________________ 3. What is your current occupation? _______________________________________________ 4. For how long? __________________ 5. Do you ever access any library services (such as databases or eBooks) remotely (from home)? Yes/No 6. If yes, approximately how often do you access these resources? a. More than two times per week b. About once per week c. 1 or 2 times per month d. Less than once per month 7. Which Library services have you used so far (check all that apply)? a. Borrow books, videos, or other materials b. Ask Reference Librarians for assistance c. Use course reserves or reference materials d. Use online databases e. Use online course guides or selected Web links f. Get Interlibrary Loan (ILL) materials g. Use microfilm materials h. Use print (paper) journals i. Use Microsoft Office software on library computers j. Use library space to meet with classmates or professors k. Use library space to read or work alone l. Use Document Delivery services m. Get Research Library Program (RLP) card n. Pay fines o. Other (please specify) _____________________________________________ 8. Have any of your classes included an Information Instruction session (when a Librarian comes into your classroom to explain and demonstrate Library resources)? Yes / No 9. After these sessions do you feel as though you can better navigate databases and the library catalog or other research resources? Yes / No 10. What are your biggest challenges to finding information during your graduate/doctoral studies? (select all that apply) a. coming up with an original topic b. finding supporting literature/peer reviewed literature for your topic c. following APA citation and writing style d. effectively using databases Evidence Based Library and Information Practice 2010, 5.4 35 e. Other _________________________________________________________________ 11. What information or skills would be useful for you to learn at this time? _______________________________________________________________________________ 12. What methods do you use to begin research? a. Google b. Databases (which ones?) ___________________________________________________ c. Library catalog d. Reference books such as encyclopedias e. Other ____________________________________________________________________ 13. How comfortable/confident do you feel when seeking information from: (please place a check mark under the appropriate sentiment?) Very Comfortable Comfortable Neutral Uncomfortable Very Uncomfortable a. An internet search engine b. A library web page c. A friend d. A professor or teaching assistant e. A faculty or class website f. The library 14. Please indicate your comfort level with the listed skills Very Comfortable Comfortable Neutral Uncomfortable Very Uncomfortable a. Formulate questions based on information needs b. Identify potential sources of information c. Identify potential sources of information d. Develop successful search strategies e. Access sources of information, including computer- based and other technologies f. Evaluate information g. Organize information for practical application h. Integrate new information into an Evidence Based Library and Information Practice 2010, 5.4 36 existing body of knowledge i. Use information in critical thinking and problem solving 15. Given the topic “Students with ADHD in the mainstream classroom” as the subject of a research project, in what order would you perform the following steps? (use 0 if you would not take a particular step) __ Browse a current printed magazine index __ Browse the most recent issue of an education journal __ Search the Internet using the keywords “ADHD” and “mainstream classroom” __ Look at reference material that provides an overview of violence and teenagers __ Brainstorm the concept, using the terms in the topic __ Formulate questions based on the information needed to begin the research __ Search subject-based and other related databases 16. Other than books and journals, what other types of information are you familiar with or might use for a research project/paper? (Select all that apply by circling your answers.) a. Websites b. Newspapers c. Magazines d. Interviews e. Images/pictures f. Speeches g. Videos/DVDs h. Television/broadcasts i. Television/radio transcripts j. Diaries/letters k. Manuscripts l. Music m. Radio shows/broadcasts n. Dissertations/theses o. Conference proceedings p. Other (internet, journals, scholarly journals) q. None of the above 17. How frequently do you use the following search strategies? Always Often Sometimes Rarely Never Don’t know a. Truncation b. Boolean operator “NOT” c. Boolean operator “OR” d. Proximity operators e. Library of Congress Subject Headings f. ERIC descriptors (in the ERIC database) Evidence Based Library and Information Practice 2010, 5.4 37 g. Cross and multiple-field searching (such as searching by date) 18. Where would you go or what would you do to find current information on the following topic: “Terrorism on college campuses?” (select all that apply by circling) a. Online/Internet b. Reference works such as encyclopedias c. Newspaper archives d. Magazines e. Television news f. Friends/colleagues g. Radio news h. A librarian i. Faculty/professors j. Television/Radio transcripts k. Abstracts and Indexes online (databases) l. Abstracts and indexes in print 19. During which events do you ask a librarian for assistance? (select all that apply) a. When I need advice about where to look for information b. When I don’t know how to use an information source c. When I need help choosing the best information source d. All of the above e. I don’t know when I would consult a librarian for assistance 20. Can one evaluate an article for bias before reading it? a. No. I need to reaed an article to find bias b. Yes. The abstract usually evaluates the article and notes any bias c. Yes. If the article is reporting research, it should be unbiased d. I don’t know e. Yes. The reputation of a journal or publisher may indicate bias 21. When would you find any of the following resources credible or reliable for your research? Always Sometimes Never a. Time or Newsweek b. New York Times or Washington Post c. The Daily News d. Wikipedia e. The Onion (website) f. Rolling Stone g. Sports Illustrated h. People i. National Georgraphic j. National Enquirer k. CNN l. Today Show/Good Morning America m. Saturday Night Live’s Weekend Update n. Jon Stewart’s The Daily Show o. World News Tonite/CBS evening news Evidence Based Library and Information Practice 2010, 5.4 38 22. CIRCLE all that are TRUE: When it comes to information found on the World Wide Web: a. Material on the Web is far more reliable than books and magazines. b. Material on the Web is factual because the Web is monitored by international organizations. c. Material on the Web comes from many varied sources such as business, the government, organizations, or the public. d. Material on the Web is accurate, timely, and appropriate. e. Material from the Web does not have to be cited under any circumstances. f. You may use the text or graphics freely unless they are specifically labeled as being copyrighted. g. You can assume that all of the data or text must be cited properly. 23. What is the best definition of “intellectual freedom” a. The right of every individual to both seek and receive information from all points of view without restriction b. The prevention of cheating by students c. The encouragement of open and public sharing of ideas d. The limiting of access to ideas and information that some people find objectionable or dangerous e. The support of the bill of rights 24. You need to find an editorial on President Franklin Delano Roosevelt’s (1882-1945) New Deal program. How would you be most likely to find one from that time period? a. read the newspaper b. search a print periodical index c. search government documents d. use an encyclopedia / Evidence Based Library and Information Practice/ /