Editorial Evidence Based Library and Information Practice 2009, 4:2 155 Evidence Based Library and Information Practice Evidence Summary E-mailed Evidence Based Summaries Impact Physician Learning More than Practice A Review of: Grad, Roland M., Pierre Pluye, Jay Mercer, Bernard Marlow, Marie Eve Beauchamp, Michael Shulha, Janique Johnson-Lafleur and Sharon Wood-Dauphinee. “Impact of Research- based Synopses Delivered as Daily E-mail: A Prospective Observational Study.” Journal of the American Medical Informatics Association (2008)15.2: 240-5. Reviewed by: Gale G. Hannigan Professor Texas A&M Medical Sciences Library College Station, Texas, United States of America E-mail: g-hannigan@tamu.edu Received: 02 December 2008 Accepted : 18 April 2009 © 2009 Hannigan. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Objective – To determine the use and construct validity of a method to assess the cognitive impact of information derived from daily e-mail evidence based summaries (InfoPOEMs), and to describe the self-reported impact of these InfoPOEMs. Design – Prospective, observational study over a period of 150 days employing a questionnaire and rating scale. Setting – This study was conducted via the Internet between September 8, 2006 and February 4, 2007. Subjects – Canadian Medical Association (CMA) members who received InfoPOEMs via e-mail as of September 2006 were invited to participate. For inclusion in the analyses, a participant was defined as a practising family physician or general practitioner who submitted at least five ratings of InfoPOEMS during the study period (n=1,007). Methods – Volunteers completed a demographic questionnaire and provided informed consent online. Each subsequent InfoPOEM delivered included a link to a “ten-item impact assessment scale” (241). Participants checked “all that apply” of descriptive statements such as: My practice mailto:g-hannigan@tamu.edu http://creativecommons.org/licenses/by/2.0 Evidence Based Library and Information Practice 2009, 4:2 156 was (will be improved); I learned something new; I think this information is potentially harmful. Each combination of selections made was considered a pattern of cognitive impact. College of Family Physicians of Canada received continuing medical education (CME) credit for each InfoPOEM rated. Data were collected by the CMA and forwarded weekly to the investigators who used descriptive statistics, principal component analysis, and multilevel factor analysis to analyze the data. Main Results – 1,007 participants rated an average of 61 InfoPOEMs (ranging from five to 111). A total of 61,493 patterns of cognitive impact were submitted. Eighty- five different patterns were observed, i.e., there were 85 different combinations of the scale’s statements used. Ten patterns accounted for 89.4% of the reports. The top five patterns were: I learned something new (35.2%); No impact (17.1%); This information confirmed I did (will do) the right thing (9.6%); I learned something new AND My practice will be improved (9.4%); and, I was reassured (5.6%). I disagree with this information was checked at least once by 10.3% of the participants, and 8.0% checked I think this information is potentially harmful at least once. Conclusion – The authors applied a cognitive assessment instrument to determine the impact of InfoPOEMs e- mailed to primary care physicians in Canada and found that ten combinations of impact descriptors accounted for 89.4% of the total reports. Most suggested a positive impact on knowledge or practice. Of the total, 17.1% indicated No impact and 1.8% indicated the participant was frustrated as there was not enough information or nothing useful. Commentary Evidence based summaries are a new type of literature designed to assist physicians in their efforts to incorporate scientifically based information into patient care decision making. There needs to be good research into the impact of these summaries on clinical practice, and the authors are to be commended for taking this on and developing and testing an assessment tool. This study would have benefited from a study design flow chart. The response rate of the target population for the study cannot be calculated. As stated under Methods, “all CMA physicians who received InfoPOEMs via e-mail as of September 2006 were eligible to participate” (241). The authors do not provide that number, but do write that 12,800 CMA members had started receiving InfoPOEMs by e-mail in 2005. A total of 1,007 physicians were counted as participants in the study, defined as having submitted at least five ratings. The number of physicians who originally agreed to participate but did not complete at least five ratings was not reported, either. This number would have been informative as a measure of use. Did those physicians use the assessment less than five times because of lack of time or did many of them indicate frustrated…nothing useful or no impact on a few and then quit participating? Knowing the reason for low use of the instrument by people who were initially interested seems key to the purpose of the study. The authors did compare the participants (those with five or more ratings) with Canadian family physicians surveyed in 2004 and noted that “participants were more likely to report utilization of electronic reminder or warning systems in their practice” (241). Very possibly, the study’s convenience sample was biased toward physicians more interested in using electronic updates and completing electronic assessments. Also, family physicians who completed ratings received CME credits, which might indicate selection bias in favour of those who prefer this form of CME or those who participated Evidence Based Library and Information Practice 2009, 4:2 157 primarily for the CME credit. The authors admit that “in the absence of CME credit, use of the method may be less impressive” (244). The study’s sample may or may not be a good indicator of use of evidence based summaries and the assessment tool; the results about impact can only be generalized to those who would use e-mail updates and the assessment tool five or more times. The authors acknowledge the limitations of self-report in determining changes in practice, i.e., did the physician who checked My practice will be improved actually experience an improvement in practice? Readers may wish they had more explanation about the assessment tool itself. For example, why is it that the item No impact “could not be selected simultaneously with any other item” (241). Couldn’t No impact be combined with I was frustrated as there was not enough information or nothing useful? That said, the authors are honing a set of descriptors for assessing the impact of evidence-based summaries and, perhaps, other clinical information. A major point the authors make relates to the value of “just in case” versus “just in time” information. As they point out, e- mailed summaries are a “push” communication, not targeted to individuals who might pay more attention to information that is personally relevant. Databases, and specifically, point-of-care resources that enable one to search for evidence-based summaries at the time of need would most likely result in “greater attention to the message” (244). It is not surprising, then, that physicians reported that they learned something new more often than they reported that the e-mailed information changed practice. “Pushed” as e-mails, evidence based summaries may be more sources of general knowledge than decision-support tools. There is value there, too. Reading e-mailed evidence based summaries may be a more efficient lifelong learning strategy than browsing issues of current journals to keep up with the literature. As evidence based summaries proliferate and are shown, with studies such as this one, to be useful in and have an impact on clinicians’ knowledge – general or specific – perhaps MEDLINE will begin indexing these published summaries as a new publication type and even link them to the original article(s). That way, busy physicians (and librarians) using MEDLINE can find them more easily when patient- specific questions arise, too. Research such as this can provide a valuable measurement tool for clinical librarians and others.