Microsoft Word - Art_1515_ Evidence Based Library and Information Practice 2008, 3:2 3 Evidence Based Library and Information Practice Article An Evaluation of the Five Most Used Evidence Based Bedside Information Tools in Canadian Health Libraries Alison Farrell Public Services Librarian and Instruction Coordinator Health Sciences Library Memorial University of Newfoundland St. John's, Newfoundland, Canada Email: alisonr@mun.ca Received: 13 March 2008 Accepted: 15 May 2008 © 2008 Farrell. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://Creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Objective – This project sought to identify the five most used evidence based bedside information tools used in Canadian health libraries, to examine librarians’ attitudes towards these tools, and to test the comprehensiveness of the tools. Methods – The author developed a definition of evidence based bedside information tools and a list of resources that fit this definition. Participants were respondents to a survey distributed via the CANMEDLIB electronic mail list. The survey sought to identify information from library staff regarding the most frequently used evidence based bedside information tools. Clinical questions were used to measure the comprehensiveness of each resource and the levels of evidence they provided to each question. Results – Survey respondents reported that the five most used evidence based bedside information tools in their libraries were UpToDate, BMJ Clinical Evidence, First Consult, Bandolier and ACP Pier. Librarians were generally satisfied with the ease of use, efficiency and informative nature of these resources. The resource assessment determined that not all of these tools are comprehensive in terms of their ability to answer clinical questions or with regard to the inclusion of levels of evidence. UpToDate was able to provide information for the greatest number of Evidence Based Library and Information Practice 2008, 3:2 4 clinical questions, but it provided a level of evidence only seven percent of the time. ACP Pier was able to provide information on only 50% of the clinical questions, but it provided levels of evidence for all of these. Conclusion – UpToDate and BMJ Clinical Evidence were both rated as easy to use and informative. However, neither product generally includes levels of evidence, so it would be prudent for the practitioner to critically appraise information from these sources before using it in a patient care setting. ACP Pier eliminates the critical appraisal stage, thus reducing the time it takes to go from forming a clinical question to implementing the answer, but survey respondents did not rate it as high in terms of usability. There remains a need for user-friendly, comprehensive resources that provide evidence summaries relying on levels of evidence to support their conclusions. Introduction Evidence based medicine can be defined as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” (Sackett et al. 71). There are four steps to practicing evidence based medicine. First, one must have a question that is phrased in such a way that it can be searched. The next step is searching the literature. Third, one must critically appraise the information found, and lastly, implement the new information in practice (Rosenberg and Donald 1122). Health care professionals often need information in a timely manner at the point of care (i.e., at the patient’s bedside or in the doctor’s office) and may not have the time to follow all four steps. There is too much information, and it takes far too long to read and synthesize that information. Evidence based bedside information tools may be a potential solution for this problem. For the purposes of this study, evidence based bedside information tools are defined as web-based, searchable, patient oriented, and pre-digested forms of information that offer evidence summaries of relevant research that has been critically appraised by scholars in the field and are peer reviewed. These resources should demand little work to provide answers to clinical questions. They provide health professionals with the information they need in a summarized format, thus shortening the time needed to find answers. Good evidence based bedside information tools should be dependable, reliable, and based on appropriate evidence, thus ensuring the health care professional is getting the best information available to make decisions as quickly as possible. Studies have shown that the average physician will spend only two minutes looking for an answer before giving up the quest. (Alper et al., “Answering” 961; Campbell and Ash, “An Evaluation” 435; Ketchell et al. 537). Furthermore, most physicians will look at only a small number of documents to find the answer. In the research conducted by Magrabi and her colleagues, clinicians examined only two documents for an answer to a clinical question in 56.4% of the searches examined in their research (“Clinicians” 299). Further, research has revealed that although physicians encounter many questions in the run of an average workday, they often neglect to look for the answers. Green and Ruff found that residents pursue only 28% of the clinical questions that arise in an Evidence Based Library and Information Practice 2008, 3:2 5 average day (176). Alper et al. showed that while family physicians have an average of six clinical questions per half-day of practice, 70% of those questions go unanswered (“Answering” 960). There are many barriers to finding the answers, such as lack of time, difficulty in phrasing a question, lack of awareness of search resources, forgetting to look, and lack of search skills (Ketchell et al. 538). One potential solution is to turn to a good, reliable evidence based bedside information tool. There are many of these tools available ranging from tools giving short, informative summaries of individual studies to those that synthesize multiple studies into a concise summary. To date, while there have been studies demonstrating the effectiveness of using evidence based bedside information tools for patient care, there have been few that evaluate specific resources. In a United Kingdom study, 60% of the general practitioners who used the ATTRACT system reported that the information gained had changed their practices (Westbrook, Gosling, and Coiera 114). According to Schwartz, 62% of the answers sought in a clinical practice will modify a physician’s opinion, 56% of the answers will change current patient care, and 70% of the answers will affect future patient care (254). Koonce, Giuse, and Todd found that evidence based resources provided answers for general patient care management questions more frequently than they aided complex clinical questions. They also documented that evidence based resources are successful when used at the point of care. Magrabi et al. concluded that general practitioners will use evidence based resources if they are accessible and easy to use in their daily practice (“General Practitioners”). Campbell and Ash performed a user- centered evaluation of five evidence based bedside information tools that resulted in a ranked list. The resources were evaluated based on criteria that practitioners deemed to be most important. The authors argued that the content of a resource is not the only way to measure the usefulness and value of a resource, and that if physicians do not like the interface, design, and organization of a resource they will not use it. They concluded, “A product selected because of excellent content may be rendered useless by a difficult user interface” (“Comparing” 104). Campbell and Ash found that users preferred the interface and design of UpToDate and were able to find more answers to their questions in UpToDate than in the other four resources evaluated. They also found opinions of ACP Pier were mixed, with six percent of the users in their study ranking it as the best of the five resources, and thirteen percent ranking it as the worst (“Comparing” 103-4). The questions used in Campbell and Ash’s study were selected only after they proved to be answerable in all resources tested, and thus their study did not test the comprehensiveness of the resources. Librarians may wish to take both user satisfaction and depth of content into consideration when deciding which resources to purchase. The literature review did not identify any studies that measured the comprehensiveness of resources in terms of their abilities to answer questions, nor in terms of the inclusion of the levels of evidence on which the answers were based. The inclusion of levels of evidence in a resource eliminates the need for the third step, critical appraisal, in the evidence based process. It also allows physicians to quickly implement the newfound knowledge into their practices – an important consideration for physicians with limited time. Alper, White, and Ge found that physicians were able to answer more questions with evidence based resources that synthesized and critically appraised the evidence than Evidence Based Library and Information Practice 2008, 3:2 6 they were with more traditional resources, such as bibliographic databases. As a result, they found that use of resources with a critical appraisal component brought about changes in their clinical practices more frequently (511). Health professionals and librarians have a problem in selecting appropriate evidence based resources. This study examined the most frequently used resources among Canadian health libraries. Findings from this project may assist librarians with purchasing decisions for their libraries and thus aid their clientele, as well. The project began by identifying the five most used tools, the reasons they were being used, and whether or not librarians found them easy to use, efficient, and informative. In addition to librarians’ attitudes towards evidence based bedside information tools, this study tested the comprehensiveness and inclusion of levels of evidence in the five resources. The inclusion of levels of evidence is an important aspect of an evidence based bedside information tool, as it allows the physician to confidently skip the critical appraisal step in evidence based medicine. Methods A survey (Appendix 1) sent to Canadian health librarians via the CANMEDLIB mail list sought to learn the attitudes of librarians towards evidence based bedside information tools. A list of evidence based resources fitting the definition outlined above was developed using the literature, Internet, and previous knowledge. In addition to meeting the definition, the study included only resources available to anyone, either freely or for a fee. This excluded, for example, resources developed for specific organizations, such as the Clinical Information Access Program (CIAP) in Australia (Westbrook 113), and accessible only to employees of that organization. Based on these criteria, the following resources, in alphabetical order, were included for consideration: • ACP Pier • ATTRACT • Bandolier • BestBETs • BMJ Clinical Evidence • DISEASEDEX • DynaMed • First Consult • FPIN Clinical Inquiries • InfoPOEMS/InfoRetriever (now Essential Evidence Plus) • PEPID • UpToDate The survey was sent to all members of the CANMEDLIB mail list. Questions were designed to determine why libraries subscribe to or promote the use of certain products, and the library staff members’ satisfaction with the products in terms of ease of use, efficiency and information provided. After identifying the five most subscribed to or promoted resources, the project developed a test to determine the comprehensiveness of each product. Twenty questions were randomly chosen from the U.K. National Health Service’s Primary Care Question Answering Service. The Primary Care Question Answering Service is a compilation of clinical questions submitted by U.K. health practitioners that have been answered and summarized. This resource was selected because of the range of clinical questions and because the questions had been submitted by practicing clinicians, thus increasing the likelihood of their applicability for this project. The questions were selected with no knowledge of the answers. The librarian researcher searched for the answers to these questions in each of the five most used resources, seeking answers to each question and determining whether the answers included levels of Evidence Based Library and Information Practice 2008, 3:2 7 evidence. All searches were performed by the same librarian. The study did not seek to determine the accuracy or usefulness of the answer provided (e.g., whether it was not detailed enough, too detailed, too long, or too short). Results Survey Participation A survey was distributed in March 2007 to the CANMEDLIB mail group, and a reminder was sent the following week. Respondents had a two-week period to complete the survey. As an incentive to participate in the survey, respondents had the opportunity to compete in a draw for one of two iPod Shuffles. To be entered in the draw, the participant had to include a name and email address, but the inclusion of the name and email address was not necessary to complete the survey. A total of 52 surveys were returned, for a return rate of approximately 10-11%. No statistical analysis was performed on the results, because the low number of surveys returned meant the analysis could not be statistically significant. In light of this, the information gained from this study provides only an indication of trends, rather than a statistically accurate picture. The membership of CANMEDLIB is not limited to librarians, but the majority of respondents (86.3%, n=44) identified themselves as librarians. The majority of respondents (84.3%, n= 43) worked in hospital libraries, academic libraries, or in academic libraries that also served hospitals. Resources Used The evidence based bedside information tools most often subscribed to or promoted by the Canadian health libraries participating in the survey were BMJ Clinical Evidence, UpToDate, ACP Pier, Bandolier, and First Consult. (Figure 1.) Most Used Evidence Based Bedside Information Tools 0 10 20 30 40 50 BM J Clinical Evidence Up ToDate ACP Pier Bandolier First Consult Resource Us ed % L ib r a r ie s S u b sc r ib e d T o o r P r o m o ti n g R e so u r c e Figure 1: Five most-used evidence based bedside information tools in participating Canadian health libraries. Evidence Based Library and Information Practice 2008, 3:2 8 Other resources reported by Canadian libraries responding to the survey included ATTRACT, BestBETs, DISEASEDEX, DynaMed, FPIN Clinical Inquiries, InfoPOEMS/InfoRetriever, and PEPID. Of these, only BestBETs, InfoPOEMS/InfoRetriever, and DynaMed were used by three or more survey respondents. Other products were suggested that did not meet the inclusion criteria outlined in the project methodology, such as eMedicine, Epocrates, MD Consult, TRIP, and e-Therapeutics. It is noteworthy that 22.9% of the respondents were not using any of the listed resources, although it was not clear if these participants used other products they considered to be evidence based bedside information tools. When asked the importance of considerations for purchasing or promoting the selected evidence based bedside information tools, the most common responses were comprehensiveness of the resource, cost, and inclusion of levels of evidence. Actual decisions to purchase or promote products were driven by cost (n=10), demand from physicians or administration (n=14), and the inclusion of products in consortial deals or packages (n=9). Survey participants rated the ease of use, efficiency, and the informative nature of three evidence based bedside information tools used in their institutions. Ratings used a Likert scale, where the possible answers were “Strongly Agree,” “Agree,” “Disagree.” and “Strongly Disagree.” The results showed that librarians were generally satisfied with the evidence based bedside information tools being used in their libraries. Figures 2, 3, and 4 show the results for the five resources that were most used in these Canadian health libraries. Is the resource easy to use? 0 2 4 6 8 10 12 UpToDate BMJ Clinical Evidence ACP Pier Bandolier First Consult Resource Strongly Agree Agree Disagree Strongly Disagree Figure 2: Ease of use of five evidence based bedside information tools. Evidence Based Library and Information Practice 2008, 3:2 9 UpToDate, BMJ Clinical Evidence and Bandolier were perceived as easy to use by all the respondents who used them (n=20, n=16, and n=4 respectively). ACP Pier was the only resource that garnered any negative responses regarding ease of use, with three of twelve respondents disagreeing with the statement that ACP Pier is easy to use. Of all respondents, only one indicated that any of the top five tools was not informative. One of twelve respondents disagreed that ACP Pier is informative. One of four respondents strongly agreed that Bandolier is an informative tool, while half (n=10) of respondents indicated that UpToDate is informative. BMJ Clinical Evidence had the highest percentage of respondents (62.5%, n=10) strongly agreeing with the statement that the resource is informative. Although ACP Pier was the only tool to elicit negative responses to the questions regarding ease of use and informative nature, all tools received at least one negative response regarding efficiency. No respondents strongly disagreed with the statement that the tools were efficient. One out of sixteen respondents disagreed that BMJ Clinical Evidence is an efficient resource, but four out of twelve respondents disagreed that ACP Pier is an efficient resource. Is this resource informative? 0 2 4 6 8 10 12 UpToDate BMJ Clinical Evidence ACP Pier Bandolier First Consult Resource Strongly Agree Agree Disagree Strongly Disagree Figure 3: Informative nature of five evidence based bedside information tools. Evidence Based Library and Information Practice 2008, 3:2 10 Comprehensiveness Testing No resource was able to answer all twenty of the randomly chosen questions from the Primary Care Question Answering Service. UpToDate provided answers for the most questions, answering a total of fourteen. However, only one of those fourteen answers provided a level of evidence. ACP Pier and First Consult each had answers for ten of the twenty questions. ACP Pier provided the levels of evidence for ten questions, and First Consult had levels of evidence for five questions. Bandolier performed most poorly with a total of only four answers for the twenty questions, none of which offered levels of evidence. Table 1 shows the complete results for the test of the comprehensiveness of the resources. Is the resource efficient? 0 2 4 6 8 10 12 UpToDate BMJ Clinical Evidence ACP Pier Bandolier First Consult Resource Strongly Agree Agree Disagree Strongly Disagree Figure 4: Efficiency of five evidence based bedside information tools. Evidence Based Library and Information Practice 2008, 3:2 11 Discussion Because the membership of CANMEDLIB varies as new members subscribe and others unsubscribe, the exact number of people who received the survey is not known. The list moderator indicated that at any given time there are approximately 450 to 500 members of CANMEDLIB (Fahey). Fifty- two responses to the survey were received, giving an approximate response rate of 10% to 11%. Due to the low response rate, results of the survey are not statistically significant, although qualitative information can be gleaned from the responses. Three resources were identified as being in common use in these Canadian health libraries: UpToDate, BMJ Clinical Evidence, and ACP Pier. The number of libraries subscribing to or promoting the use of First Consult and Bandolier, the next most used resources, was low. There is an abundance of evidence based bedside information tools on the market competing for a library’s attention and funding. At present, purchasing decisions in these libraries are based on cost, consortial deals, requests from practitioners, and top down decisions from administrators. However, evaluation of these resources is essential prior to making purchasing decisions. UpToDate is as an easy to use and informative resource, according to results of this study and other user satisfaction surveys (Campbell and Ash “Comparing”). However, the testing completed in this study showed that levels of evidence accompanied only seven percent of the questions answered. This indicates that practitioners could not be confident that their answers were based on good evidence. In contrast to this, although 25% of respondents who listed ACP Pier as one of their choices for evaluating the resources did not agree that it is easy to use, levels of evidence accompanied 100% of the answers found in ACP Pier. This means that if practitioners were to use ACP Pier, they could avoid taking additional time for critical appraisal of the evidence. They could still be confident in applying information from Pier in their practice while decreasing the time needed to progress from forming a clinical question to implementation of the information in their practice. Since survey participants believed ACP Pier to be not as easy to use as UpToDate, practitioners may be less likely to use it. Respondents did, however, rate ACP Pier as an informative resource, with only one respondent disagreeing with this statement. These results show a need for resources to be both easy to use and comprehensive. No matter how comprehensive it may be, a resource is Resource Number (Percent) Questions Answered Number (Percent) Questions Answered and Providing Levels of Evidence UpToDate 14 (70%) 1 (7%) ACP Pier 10 (50%) 10 (100%) First Consult 10 (50%) 5 (50%) BMJ Clinical Evidence 6 (30%) 2 (33%) Bandolier 4 (20%) 0 (0%) Table 1: Test results for comprehensiveness of resources Evidence Based Library and Information Practice 2008, 3:2 12 not useful if it is not easy to use. Conversely, a resource that is easy to use but not comprehensive, may not be as valid for evidence based practice, but it is likely to get used more often due to the high usability. It is clear that there is a high degree of variability in evidence based bedside information tools. Evaluation is difficult due to the varied factors that need to be considered when making purchasing decisions. This was a survey of librarians, not health care practitioners. Practitioners’ views on these resources may be quite different. Previous studies have demonstrated that librarians and health care practitioners place importance on different criteria when evaluating evidence based bedside information tools. (Kupferberg and Jones Hartel) It is not enough to use librarians’ attitudes when judging the appropriateness of a resource that will be used by both librarians and health practitioners. Purchasing decisions should be made within the context of how practitioners will use the resources. It is important to have input from all patrons who may use a resource when deciding on what to purchase. This study provides evidence from a librarians’ perspective, but including the physicians’ viewpoint would have resulted in a more balanced view. Future research is required to facilitate evaluation of these resources. A similar study using practitioners as searchers, rather than librarians, would provide evidence of the use of the resources in practice and more information on their impact and usefulness. Practitioners would also be better qualified to evaluate the quality of the answers provided. This study was limited in that it measured only whether or not the resource gave an answer, rather than the quality or accuracy of the answer. The response rate of 10%-11% is an approximation, not only due to the varied number of CANMEDLIB subscribers, but also due to the fact that people may have forwarded the survey to other colleagues. Some institutions may have more representation than others, depending on how participants interpreted the survey. Due to the small sample size, the results can give only general indications. This was a study of Canadian librarians, thus the results are not representative of all countries. It would be interesting to survey librarians from other countries to identify similarities and differences at an international level. Other potential areas for further study include how to make a comprehensive resource such as ACP Pier more user- friendly and an examination of the usefulness of resources in the PDA environment as meaningful bedside information tools. Conclusion It is evident that both UpToDate and BMJ Clinical Evidence are easy to use and informative. However, the lack of levels of evidence means that these products do not necessarily reduce the time needed to practice evidence based medicine. In contrast, ACP Pier was identified as being the most comprehensive in terms of inclusion of levels of evidence, although respondents did not believe it to be as easy to use. All three resources have significant weaknesses that need to be addressed in terms of providing comprehensive, reliable and user-friendly evidence to practitioners. Although this is a small study with some limitations, librarians making collections development decisions should find it informative. There are different ways to evaluate a product, and all of these must be Evidence Based Library and Information Practice 2008, 3:2 13 acknowledged before making a confident decision about products. This study is a step towards providing evidence for health librarians involved in collection development and acquisition of evidence based bedside information tools. Taking the views of clinicians into consideration when deciding on which products to purchase would further improve the evidence base. Works Cited Alper, Brian S., James J. Stevermer, David S. White, and Bernard G. Ewigman. "Answering Family Physicians' Clinical Questions using Electronic Medical Databases." The Journal of Family Practice 50.11 (Nov. 2001): 960-5. Alper, Brian S., David S. White, and Bin Ge. "Physicians Answer More Clinical Questions and Change Clinical Decisions More often with Synthesized Evidence: A Randomized Trial in Primary Care." Annals of Family Medicine 3.6 (Nov. – Dec. 2005): 507-13. Campbell, Rose, and Joan Ash. "Comparing Bedside Information Tools: A User- Centered, Task-Oriented Approach." AMIA Annual Symposium Proceedings (2005): 101-5. Campbell, Rose, and Joan Ash. "An Evaluation of Five Bedside Information Products Using a User-Centered, Task- Oriented Approach." Journal of the Medical Library Association 94.4 (Oct. 2006): 435-41, e206-7. Fahey, Sue. Personal Interview. 3 Mar. 2007. Green, Michael L., and Tanya R. Ruff. "Why do Residents Fail to Answer their Clinical Questions? A Qualitative Study of Barriers to Practicing Evidence-Based Medicine." Academic Medicine 80.2 (Feb. 2005): 176-82. Ketchell, Debra S., Leilani St. Anna, David Kauff, Barak Gaster, and Diane Timberlake. "PrimeAnswers: A Practical Interface for Answering Primary Care Questions." Journal of the American Medical Informatics Association 12.5 (Sept.-Oct. 2005): 537- 45. Koonce, Taneya Y., Nunzia Bettinsoli Giuse, and Pauline Todd. "Evidence-Based Databases Versus Primary Medical Literature: An in-House Investigation on their Optimal use." Journal of the Medical Library Association 92.4 (Oct. 2004): 407-11. Kupferberg, Natalie, and Lynda Jones Hartel "Evaluation of Five Full-Text Drug Databases by Pharmacy Students, Faculty, and Librarians: Do the Groups Agree?" Journal of the Medical Library Association 92.1 (Jan. 2004): 66-71. Magrabi, Farah, Enrico W. Coiera, Johanna I. Westbrook, A. Sophie Gosling, and Victor Vickland. "General Practitioners' use of Online Evidence during Consultations." International Journal of Medical Informatics 74.1 (Jan. 2005): 1- 12. Magrabi, Farah, Johanna I. Westbrook, Enrico W. Coiera, and A. Sophie Gosling. "Clinicians' Assessments of the Usefulness of Online Evidence to Answer Clinical Questions." Medinfo 11.Pt.1 (2004): 297-300. Rosenberg, William, and Anna Donald. "Evidence Based Medicine: An Approach to Clinical Problem-Solving." BMJ 310.6987 (29 Apr. 1995): 1122-6. Evidence Based Library and Information Practice 2008, 3:2 14 Sackett, David L., William M. C. Rosenberg, J. A. Muir Gray, R. Brian Haynes, and W. Scott Richardson. "Evidence Based Medicine: What It Is and What It Isn't." BMJ 312.7023 (13 Jan. 1996): 71-2. Schwartz, Kendra, Justin Northrup, Nejla Israel, Karen Crowell, Nehman Lauder, and Anne Victoria Neale. "Use of On- Line Evidence-Based Resources at the Point of Care." Family Medicine 35.4 (Apr. 2003): 251-6. Westbrook, Johanna I., A. Sophie Gosling, and Enrico Coiera. "Do Clinicians use Online Evidence to Support Patient Care? A Study of 55,000 Clinicians." Journal of the American Medical Informatics Association 11.2 (Mar.-Apr. 2004): 113-20. Evidence Based Library and Information Practice 2008, 3:2 15 Appendix 1 - Survey The purpose of this survey is to gather information on the selection and use of evidence based bedside information tools. For the purpose of this project I am using the following definition to identify these tools: Evidence Based Bedside Information Tools: web-based, searchable, patient oriented, pre- digested forms of information. They offer evidence summaries of relevant research that has been critically appraised by scholars in the field and are peer reviewed. Please take a few minutes to answer the following questions. Thank you very much for your participation. Survey: 1. Are you a(n): a. Librarian b. Library Staff c. Archivist d. Other (please specify) _______________________________ 2. In what type of library do you work? a. Academic b. Hospital c. Academic library which also serves a hospital d. Government e. Public f. Special g. Other (please specify) _______________________________ 3. Does your library subscribe to any of the following evidence based bedside information tools? Please check all that apply. a. ACP Pier b. ATTRACT c. Bandolier d. Best Bets e. Clinical Evidence f. DiseaseDex g. DynaMed h. First Consult i. FPIN Clinical Inquiries j. InfoRetriever k. Pepid l. UpToDate m. Other (please specify) _______________________________ n. Not applicable – go to question 17. Evidence Based Library and Information Practice 2008, 3:2 16 4. What was the decision process used by your library to decide to purchase this (these) products? (open ended answer) 5. What was most important to you when deciding on a product? a. Easy to use interface b. Levels of evidence c. Cost d. Comprehensiveness e. PDA compatibility f. Other (please specify)__________________________________ 6. Do you perceive these products to be well used? (open ended) 7. Who tends to use these products most often? a. Librarians b. Physicians c. Students d. Other (please specify) ___________________________________ For questions 8-16, please use the scale to indicate your agreement with the following statements: (Please fill in the name of the product you are using to answer each set of questions. Use up to three products (if you have three).) Product One __________________ 8. The product is easy to use. 1 2 3 4 9. The product is informative 1 2 3 4 10. The product is efficient (It can answer clinical questions in under 5 minutes) 1 2 3 4 Product Two __________________ 11. The product is easy to use. 1 2 3 4 12. The product is informative 1 2 3 4 13. The product is efficient (It can answer clinical questions in under 5 minutes) 1 2 3 4 Product Three __________________ 14. The product is easy to use. 1 2 3 4 15. The product is informative 1 2 3 4 16. The product is efficient (It can answer clinical questions in under 5 minutes) 1 2 3 4 Evidence Based Library and Information Practice 2008, 3:2 17 For questions 8-16, the following four point scale is used 1 – strongly agree 2 – agree 3 – disagree 4 – strongly disagree 17. If you wish to be included in a random draw to win one of two iPod shuffles, please leave your name and email address. When analyzing responses, names will be omitted. Again, thank you for taking the time to respond to this survey.