Evidence Based Library and Information Practice 2009, 4:2 174 Evidence Based Library and Information Practice Evidence Summary Clinical Informatics Consult Service Positively Affects Some Clinical Decisions in the ICU A Review of: Mulvaney, Shelagh A., Leonard Bickman, Nunzia B. Giuse, Warren E. Lambert, Nila A. Sathe, and Rebecca N. Jerome." A Randomized Effectiveness Trial of a Clinical Informatics Consult Service: Impact on Evidence-based Decision-making and Knowledge Implementation." Journal of the American Medical Informatics Association 15.2 (2008): 203-11. Reviewed by: Jennifer Kelson Buckinghamshire Hospitals NHS Trust Amersham, United Kingdom Email: jennie.kelson@buckshosp.nhs.uk Received: 20 December 2008 Accepted: 06 May 2009 © 2009 Kelson. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Objective – To determine whether the provision of synthesized research evidence provided by the Clinical Informatics Consult Service (CICS) affects the clinical decision-making of clinicians working in intensive care units (ICUs). Design – Non-blinded randomized control effectiveness trial. Setting – ICUs in United States-based 658 bed university hospital providing tertiary care for adults and children. Subjects – Clinical staff working within one of four ICUs who submitted a request for clinical information during the study period. Methods – Valid requests submitted by clinical staff from the four clinical ICUs (medical, paediatric, trauma, or neonatal) were randomly allocated to receive information from the CICS (CICS provided) or no information (no CICS provided). Pre-consult forms, completed at the time of the request, examined reasons for the request and the clinical actions clinicians thought might be influenced by the search results. Requestors could opt out of the no CICS provided group either before or after the randomization of their request. Responses to requests, supplied within 0.5 to 7 days as agreed with the requestor, included a search strategy and bibliographic references, a targeted list of full-text articles, and a written synthesis and critique of the relevant research. mailto:jennie.kelson@buckshosp.nhs.uk http://creativecommons.org/licenses/by/2.0 Evidence Based Library and Information Practice 2009, 4:2 175 Clinicians within both groups were free to conduct their own searches and reviews. An online evaluation form, emailed to recipients, was used to assess the impact of the information supplied. The evaluation form asked clinicians to record the time spent on their own searches, sources of information consulted including colleagues, the immediate and future impact of the information provided (either from the CICS or their own searches), what influence the information had on their clinical actions, whether there were any barriers to using the information, and quality and overall satisfaction with the results provided by the CICS. Data was analyzed according to the randomized group assignment using standard intention-to- treat analysis for the main outcomes between the two groups. Statistical adjustments were made to control for possible clustering of responses or multiple ratings from individual clinicians. The data was also analyzed on an efficacy basis depending on who provided the search results. The groups were Clinician only, CICS librarian- only, or Both Clinician and CICS librarian. Results from the Clinician only search group were used as a comparison to the remaining groups. This assessment did not take account of the randomization and therefore constitutes a cohort analysis. Results were analysed by one of two methods using statistical software SAS Proc Mixed (v9) for multi-level quantitative data analysis, e.g., analysis of variance, and SPSS (v14) for all other quantitative data analysis, including descriptive statistics. Main Results – The study period was conducted over 19 months: August 2004 to March 2006. During this time, 299 valid requests were received and 226 post consult evaluation forms were returned giving a response rate of 76%. Post consult forms were returned for 108/146 of the CICS provided group and 118/153 of the no CICS provided group. The 24% of requests that had no post consult evaluation were excluded from further analysis. Statistical tests, conducted to check for potential bias relating to missing data, suggested that the missing data had little impact on the findings. Medical and neonatal ICUs accounted for the majority of completed forms (40.3% and 38.1% respectively). The majority of opt-outs (10.2% overall) were from the medical ICU. No significant difference in outcome variables was found between opt-out and other requests when tested using analysis of variance (ANOVA). Evaluation forms were completed by 89 unique clinicians and over half (49) submitted more than one request. The average requests per clinician was 2.96, SD 3.17, range 1-15, and the average number of requests per clinician who submitted more than one request was 4.57, SD 3.55, range 2-15. Total number of responses, mean, standard deviation, and Cohen’s d effect size were reported for the outcome variables based on intention to treat analysis. Results showed no significant difference between the groups on the immediate impact of the information provided, the number of articles read or the frequency with which clinicians consulted colleagues, either formally or informally. The potential future impact of the information was rated higher in the CICS group (p=<0.01) and clinician’s satisfaction levels in the CICS group were also significantly higher (p=<0.001). Only the specific action “different or new treatment” option showed a statistically significant difference between the CICS provided and no CICS provided groups. A significantly greater percentage of clinicians in the CICS provided group reported conducting their own searches (70.2%) compared to the no CICS provided group (36.8%). Where clinicians reported that the information provided, either from their own searches or Evidence Based Library and Information Practice 2009, 4:2 176 CICS, had no impact on patient care decisions, the reasons most commonly cited were that they were already doing what was recommended, there was insufficient evidence to guide their decision, or that the available evidence did not apply to their patient. Analysis of the results on the basis of the intervention actually received showed both potential future impact and satisfaction were significantly higher for both the librarian-only (p<0.001 for both outcomes) and librarian plus clinician search groups (p<0.001 for both outcomes) compared to the clinician only group. In addition, there was a statistically significant difference (p=0.02) of future impact in favour of the librarian plus clinician search group over the librarian-only group. Time taken to complete searches also showed a statistically significant difference between the groups, which is not surprising given that time reported by the librarians included time spent searching and summarizing results, whilst the clinicians recorded time spent searching only. Conclusion – Clinicians reported a higher level of satisfaction with search results provided by the CICS and rated the future impact of the information more highly. The CICS showed a statistically significant impact on some aspects of clinical decision-making, particularly with regard to treatment decisions. Provision of information by the CICS also appeared to encourage clinicians to undertake their own searches. However, the reasons for this are unclear. Commentary In general, the methodology section is thorough and explains the approach taken. Potential biases in the study design and in the analysis of the results were identified and the authors endeavoured to mitigate these by using appropriate statistical techniques (e.g., checking for differences between the groups and using techniques to control for multiple ratings by clinicians). Both SAS and SPSS were used to analyze the results (SAS for multi-level analyses and SPSS for the others). However, the authors have provided little explanation as to why both programs were necessary or where the various packages were used. As this study was a randomized trial, it is a major omission that a discussion of the sample size or power calculation for the study was not included. Underpowered studies, where insufficient subjects are recruited to a study in order to detect a difference between the intervention and control groups, may lead to the erroneous conclusion that the intervention had no effect (type II error). The study reported a number of instances, particularly with regard to actions taken as a result of receiving information, where no statistical difference was found between the CICS provided and no CICS provided groups. Based on the information presented, it is unclear whether this is a valid result, or due to the sample size being too small. Clinicians were asked to self-report on their behaviour and it is conceivable that they under or over-estimated the impact of the information provided. A more objective measurement of clinical actions would improve the reliability and validity of the study. The forms used to record information requests and post- consultation evaluations were also omitted and it is not possible to judge whether, as the authors speculate, question ordering influenced clinician’s responses regarding overall impact of search results. Clinicians’ existing knowledge of and prior training in using health sciences journal literature and databases may have influenced the likelihood they would undertake their own searches and the quality of their retrieved results. However, these aspects were not assessed prior to the study. In addition, follow-up interviews or focus groups could be used to investigate reasons why some clinicians do not use the CICS and to ascertain the reasons why clinicians felt the CICS was useful or not useful. Clinicians who conducted their own searches in addition to those conducted by CICS Evidence Based Library and Information Practice 2009, 4:2 177 may exhibit specific characteristics or personality traits, and this may be worth exploring. The authors identified other limitations of the study, including the randomization of requests by priority level rather than by clinical unit (although this is aspect of the randomization process was not made clear in the methods section) and the impact the expertise of the librarians may have had in relation to the outcomes. A number of previous studies have attempted to ascertain the effect of clinical librarian services; however, this is the first to use a randomized study design. This research is especially timely given the emphasis on impact assessment of healthcare library services within the U.K. However, the results of this study may have limited applicability owing to the ICUs setting and omission of the pre and post evaluation forms. The study results are also potentially inconclusive owing to a possible lack of a power.