Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 1 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems Lauren Shrader1*, Stuart Myerburg2, Eric Larson1 1 Northrop Grumman Corporation, GA 2 Centers for Disease Control and Prevention (CDC), GA Abstract Context: In the United States, immunization recommendations and their associated schedules are developed by the Advisory Committee on Immunization Practices (ACIP). To assist with the translation process and better harmonize the outcomes of existing clinical decision support tools, the Centers for Disease Control and Prevention (CDC) created clinical decision support for immunization (CDSi) resources for each set of ACIP recommendations. These resources are continually updated and refined as new vaccine recommendations and clarifications become available and will be available to health information systems for a coronavirus disease 2019 (COVID-19) vaccine when one becomes available for use in the United States. Objectives: To assess awareness of CDSi resources, whether CDSi resources were being used by immunization-related health information systems, and perceived impact of CDSi resources on stakeholders’ work. Design: Online surveys conducted from 2015–2019 including qualitative and quantitative questions. Participants: The main and technical contact from each of the 64 CDC-funded immunization information system (IIS) awardees, IIS vendors, and electronic health record vendors. Results: Awareness of at least one resource increased from 75% of respondents in 2015 to 100% in 2019. Use of at least one CDSi resource also increased from 47% in 2015 to 78% in 2019. About 80% or more of users of CDSi are somewhat or very highly satisfied with the resources and report a somewhat or very positive impact from using them. Conclusion: As awareness and use of CDSi resources increases, the likelihood that patients receive recommended immunizations at the right time will also increase. Rapid and precise integration of vaccine recommendations into health information systems will be particularly important when a COVID-19 vaccine becomes available to help facilitate vaccine implementation. Keywords: CDSi, clinical decision support, immunizations, guideline-base care, immunization schedule, Immunization Information System *Correspondence: ytl7@cdc.gov DOI: 10.5210/ojphi.v12i1.10602 Copyright ©2020 the author(s) mailto:ytl7@cdc.gov Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 2 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI This is an Open Access article. Authors own copyright of their articles appearing in the Online Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in t he copy and the copy is used for educational, not-for-profit purposes. Introduction In the United States, immunization recommendations and their associated schedules are developed by the Advisory Committee on Immunization Practices (ACIP) [1]. New ACIP schedule changes are communicated through clinical language in peer-reviewed publications, primarily the Centers for Disease Control and Prevention’s (CDC) Morbidity and Mortality Weekly Report (MMWR). ACIP recommendations include age for vaccine administration, number of doses, dosing interval(s), and precautions and contraindications [1]. This same procedure will be used when a coronavirus disease 2019 (COVID-19) vaccine becomes available. Clinical decision support (CDS) helps those involved with health and healthcare make decisions by providing knowledge and individualized patient information. Benefits of clinical decision support are increases in quality of care, fewer errors and adverse events, improved efficiency, and provider and patient satisfaction [2]. To implement ACIP recommendations in clinical settings, technical and clinical subject matter experts must translate the clinical language provided in the ACIP recommendations into technical logic for processing with CDS tools. This is a time- consuming and complex process that in the past often happened independently within different health information systems (HISs), leading to variability in recommendations to providers because there was no centralized approach to how recommendations were received, implemented, and updated. HISs provide healthcare providers with immunization evaluation and forecasting tools designed to automatically determine the recommended immunizations needed when a patient presents for vaccination or routine visit. Examples of HIS include but are not limited to health information exchanges (HIEs), immunization information systems (IISs), electronic health records (EHRs), and pharmacy systems. IISs record all immunization doses administered by participating providers to persons residing within a given jurisdiction [3]. In 2017, approximately 95% of U.S. children under the age of six participated in an IIS [4]. Given this widespread IIS participation, it is important that each patient’s immunization record is consistent and up-to-date within an IIS. To assist with the translation process and better harmonize the outcomes of existing or new CDS tools, CDC initiated the Clinical Decision Support for Immunization (CDSi) project. Beginning in 2010, the CDSi project began to create new clinical decision support resources to support development and maintenance of immunization CDS tools. CDSi was first published in November 2012 and continues to be updated as new ACIP recommendations are published [5]. CDSi resources are intended to make it easier to develop and maintain immunization evaluation and forecasting/CDS engines; increase the accuracy and consistency of immunization evaluation and forecasting; and improve timeliness in accommodating new and updated ACIP recommendations. If stakeholders are aware of and use these tools they will be prepared to quickly incorporate new immunizations into their immunization evaluation and forecasting/CDS engines. During times of a pandemic such as COVID-19, facilitating implementation is key. Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 3 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI CDC understood from the start that the recommendations are complex, the HIS environment is constantly changing, and CDS implementations vary amongst users. Because of this variability, CDSi was designed to be technology-neutral and to serve as guidance for developers, rather than as a specific piece of software. The CDSi resources consist of three main components. The Logic Specification describes the rules required to evaluate and forecast a patient’s immunization(s) against the ACIP recommendations, considering a patient’s immunization history and other relevant medical, behavioral, and environmental observations. It uses defined vocabulary and domain models to build business rules, decision tables, and a processing model which can be implemented by a CDS engine. The Supporting Data is a representation of the ACIP vaccine schedule that describes, by antigen, various factors and their accompanying sets of values to be considered when implementing ACIP recommendations as described in the Logic Specification. Supporting Data can be thought of as a set of configuration files used as input to a CDS engine. It is published both in Microsoft Excel spreadsheet and XML formats. The Test Cases provide a representative set of scenarios and expected outcomes that can be processed against an immunization CDS engine to validate its algorithm. They are published as Excel spreadsheets, in XML, and as part of an online test case management tool. The intended audience for the CDSi resources includes business and technical implementers of immunization CDS engines. These implementers may support any system with an immunization evaluation and forecasting engine. Stakeholders included in this assessment were IIS awardees, IIS vendors, and EHR vendors. The purpose of the CDSi online assessment was to assess stakeholder awareness of CDSi resources, whether CDSi resources were being used by immunization-related HIS stakeholders, and what impact the CDSi resources have had on stakeholders’ work. After they were released, Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 4 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI the resources were promoted by Program Managers Immunization Registry Work Group email distributions, American Immunization Registry Association email distributions, the CDC website, at conferences, through a CDC Immunization Works newsletter article, and via emails from the CDSi project team. The CDSi project has also created infographics, guides, and videos to increase awareness and use of the CDSi resources. These resources are available on the CDC website, distributed at conferences, and used in presentations on CDSi [5]. The goal of the CDSi project evaluation was to understand the impact and use of the resources, as well as inform continued development and support of the CDSi project. In addition to assessing awareness of CDSi resources and whether CDSi resources were being used by stakeholders, we were interested in stakeholder satisfaction with the resources. Evaluating the uptake and use of the CDSi resources helps us understand the preparedness for implementing new vaccines for novel viruses, such as SARS-CoV-2, the virus that causes COVID-19, across IISs. Methods To assess the uptake and impact of CDSi resources on stakeholders’ work, we designed an online survey assessment. Participants CDC funds 64 IIS awardees from public health agencies; the main and technical contacts for each IIS awardee, identified from the CDC published list of main and technical IIS contacts, were invited to participate in the survey. IIS vendors provide IIS systems and support to awardees. Some of these vendors use all or part of CDSi resources, while others develop their own system. IIS and EHR vendors were identified by subject matter experts as well as those who had contacted the CDSi support team with questions. The respondents from the IIS awardees or vendors could have a variety of roles within their organization including, programmatic, technical, and/or clinical. All stakeholders who are candidates for using the CDSi resources are part of the healthcare ecosystem. This ecosystem can change over time, and additional stakeholders may come into play or drop out. However, the main goal of the CDSi project is for healthcare providers to receive accurate and consistent recommendations across any systems they access. The CDSi resources are product-neutral and the guidance is exactly the same for all stakeholders. For organizational reasons, we split respondents out by IIS awardees and vendors, mostly for tracking purposes. However, given the similar responses across respondent types and roles, we combined all responses within our results. The grouping was mainly for follow-up and tracking purposes, since the IIS awardees contact information was known and available, while the vendors’ information was not. Total organizations invited to participate by type: • Round 1 (2015): 64 IIS awardees and 17 vendors • Round 2 (2016): 64 IIS awardees and 19 vendors • Round 3 (2018): 64 IIS awardees and 15 vendors • Round 4 (2019): 64 IIS awardees and 18 vendors Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 5 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI Survey Design The survey was constructed and launched in SurveyMethods online software (SurveyMethods Software, Dallas, Texas). The data were collected and stored through the SurveyMethods survey website. Respondents were asked to provide their name and organization for tracking of response rates, though this was not required. The survey was deployed four times: January 2015 (Round 1), January 2016 (Round 2), January 2018 (Round 3), and January 2019 (Round 4). The questionnaire was reviewed and updated before each round to ensure all survey questions were useful and relevant. The questionnaire was programmed with skip logic so that respondents only were asked questions that were applicable to them. First, all respondents were asked about awareness of the CDSi resources, then those who were aware were asked about use of the resources. The users were then asked questions about each resource that they used. Some examples of questions for the users included how they use and plan to use the resources; if they felt they had detailed knowledge about the resources; how they learned to use the resources; and their perceptions of the available training materials. The survey was estimated to take 15 minutes or less to complete. In the first two rounds, a pre-notification email was sent to respondents one week prior to the survey launch. The pre-notification email was not sent in third and fourth rounds since the respondents were familiar with the survey from past rounds. All respondents were invited via an email sent to each organization. In addition, a broadcast email with a link to the survey was sent to all IIS awardee program managers. One week after the email invitation was sent, a reminder email was sent to those who had not yet responded. The data collection period lasted one month, and a final reminder email was sent a few days before the survey closed. The invitation included a description of the goal of the survey to help each organization identify the person with the correct role to respond. Data analysis was conducted using SAS version 9.4 (Cary, NC) to produce descriptive statistics including frequencies. This project was reviewed by the Human Subjects Coordinator of CDC’s National Center for Immunization and Respiratory Diseases and was determined to be public health practice evaluation; therefore, institutional review board review was not required. Results Response Rate Overall response rates and total organizations responding by type by year were as follows: • Round 1: 75% of stakeholder organizations invited: 52 awardees and 9 vendors • Round 2: 69% of stakeholder organizations invited: 48 awardees and 9 vendors • Round 3: 82% of stakeholder organizations invited: 57 awardees and 8 vendors • Round 4: 78% of stakeholder organizations invited: 55 awardees and 9 vendors Awareness First, all respondents were asked which CDSi resources (Logic Specification, Supporting Data, and Test Cases) they had heard about. Awareness of at least one resource increased from 75% of respondents in Round 1 to 100% in Round 4 (Figure 1). Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 6 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI CDSi Resource Use Those who indicated that they were aware of at least one resource were then asked multiple questions about each resource. First, they were given a brief one-sentence description of each of the resources and asked if they used each resource. In Round 1, 47% of the respondents indicated that they used at least one of the resources; by Round 4, use had continued to increase for each resource, with 78% reporting they used at least one CDSi resource (Figure 1). Figure 1. Awareness and Use of CDSi Resources Abbreviation: CDSi = Clinical Decision Support for immunization. Since there are three different CDSi resources available for stakeholders to use, we looked at how use of all three had changed over time. In Round 1, only 11% of respondents indicated they used all three of the CDSi resources, compared to 57% in Round 4. The percentage of respondents reporting they had either not heard of the resources or not used any of the resources fell from 67% in Round 1 to 22% in Round 4. Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 7 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI Figure 2. CDSi Resources Use from Round 1 to Round 4 Abbreviation: CDSi = Clinical Decision Support for immunization. Overall Satisfaction for CDSi Resource Users Respondents who reported that they used a resource were asked questions about their use of, satisfaction with, and perceptions on impact of the resources. Overall, users of each resource reported high levels of satisfaction and positive impact with the CDSi resources. The satisfaction questions were asked in Round 1 (2015), Round 2 (2016), and Round 4 (2019). The questions were not asked in Round 3 because similarly high levels of satisfaction in Round 1 and 2 were found, and the project team felt that repeating the questions was not a good use of respondents’ time. The questions were added back in Round 4 to continue to monitor satisfaction levels as the number of users increased. Overall satisfaction (“somewhat satisfied” and “very satisfied” responses) ranged from 77% (Logic Specification, Round 2) to 94% (Supporting Data, Round 1). Overall satisfaction increased from Round 2 to Round 4 by more than 10% for the Logic Specification and Test Cases, while Supporting Data stayed about the same, demonstrating that even as use increased, satisfaction with CDSi resources remained high. Overall Impact for CDSi Resource Users The users of each resource were asked about the impact of the resources on their work. Over 80% of respondents reported a positive impact from using the CDSi resources. The impact questions were asked in Round 1 (2015) and Round 2 (2016). Across all of the resources, none of the respondents reported a somewhat negative or very negative impact. At the end of the survey three additional impact questions were asked. Respondents were asked to rate on a scale of 1 (much better) to 7 (much worse) the overall impact of CDSi on their overall process, the process of accommodating new or changed ACIP recommendations, and the impact on developing immunization evaluation and forecasting products. In Round 4 most respondents Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 8 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI reported a positive impact. Between 68% and 72% of respondents rated each question between a 1 and a 3. Figure 3. Overall Impact Results Round 4 Abbreviation: CDSi = Clinical Decision Support for immunization. Discussion Since 2015, awareness and use of the CDSi resources has increased. CDSi resource users are highly satisfied and report a positive impact. About 80% or more of respondents were very or somewhat satisfied with the resources, and >80% of respondents reported a very or somewhat positive impact. Overall, we observed a trend of increasing awareness of all three resources. There was a larger increase in awareness between Round 1 and Round 2. In addition to the promotional activities around the CDSi resources between these rounds, the survey and evaluation activities also may have increased awareness of the CDSi resources between all rounds, especially between Round 1 and Round 2. By Round 4, all respondents reported awareness of at least one of the CDSi resources. This is likely a combination of targeting the correct respondent from each stakeholder who had a role in using the CDSi resources, in addition to increased awareness of CDSi resources in general. Tracking CDSi resources awareness allows the project team to decide if they need additional promotional activities to focus on awareness. At this point, with awareness at high levels, the project team can focus more on promoting use of CDSi resources rather than awareness. Users of the CDSi resources were asked additional questions across the different rounds of the survey. The results of these questions were reviewed and used to help understand the CDSi users and improve the resources. For example, to inform revamping of training materials, users were asked if they felt they had detailed knowledge about the resources, how they learned to use the Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 9 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI resources, and about the available training materials. Questions about how they have used and plan to use the resources were asked to guide development and changes to the resources and new releases. One change made after reviewing feedback from the respondents was adding a yearly flu pre- release. In an effort to improve the time it takes to incorporate new ACIP recommendations, the CDSi project began providing pre-release materials for the yearly flu recommendation in 2019. The pre-release resources are based on the information provided to the CDSi project team by the CDC subject matter experts after an ACIP meeting vote, but prior to the official MMWR publication of the ACIP recommendation. Once the recommendation is published, the pre-release material will be removed, appropriately updated, and released as part of an official CDSi version. This model of CDSi pre-release could be applied to allow implementation development work to begin in advance of the official publication of a COVID-19 vaccine recommendation. Misinterpretation in a CDS engine could lead to healthcare providers under-vaccinating and over- vaccinating many patients. A review of 2014 immunization records from six IIS sites found minimum and maximum age violations and concluded that “minimization of errors reduces wastage, excess cost, and inconvenience for parents and patients.” [6] Kirtland et al. examined live vaccine interval errors among children aged 12 months through six years, finding nearly $1 million in revaccination costs associated with errors across two separate one-year time periods and six IIS sentinel sites [7]. Use of CDSi resources within IIS and EHRs is one way to reduce frequency and cost of vaccine errors. Correct interpretation of guidance is especially important in systems with a high volume of vaccination queries. For example, Epic, an EHR vendor, reported that more than 200 million vaccination queries are sent per year to their vaccination query interface [8]. Resources such as CDSi are key to facilitating correct interpretation of clinical guidance (Logic Specification/Supporting Data) and extensive testing (Test Cases) in systems impacting such a large number of individuals. Given the number of Americans served via HISs, and the extent to which ACIP recommendations inform policy and practice, it is important to understand user perspectives on CDSi resources provided. Utilization-focused evaluations place the emphasis on designing evaluations to ensure their usefulness [9]. This evaluation used a strong utilization-focused approach both in evaluation design, dissemination of findings, and iterative revision of resources informed by the results. There were three key limitations to our survey. One was the sample pool: invited respondents might not include all users of CDSi resources. While we had a known list and current contact information for the IIS awardees, we did not have this information for the IIS and EHR vendors. The vendors were identified by previous contact with the CDSi team with questions or by subject matter expert identification. There may have been others using the CDSi resources who we were unaware of and did not include in our survey. In addition, we may not have identified the correct contact at organizations who use the CDSi resources, which would lead to lower reports of awareness and use. Finally, while we can evaluate uptake and impact of the CDSi resources on the respondents and their organization, we only have anecdotal evidence of impact at the provider and patient level. Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 10 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI Conclusion The CDSi resources provide clarity, consistency, and computability of ongoing childhood, adolescent, and adult immunization evaluation and forecasting to stakeholders. As awareness of the resources increased and use increased, they have become a vital part of the evaluation and forecasting of immunization through CDS engines. Before stakeholders were aware of and using CDSi resources, they faced the time-consuming and complex task of interpreting clinically written ACIP recommendations on their own. This increased the risk of misinterpretation, which could lead to CDS engine outputs that did not match ACIP recommendations, and ultimately provide inaccurate decision support to providers/clinicians. Reducing the barriers of individually interpreting and implementing ACIP recommendations results in more rapid adoption of new vaccines which is critical during pandemic situations, such as COVID-19. We conducted the CDSi online assessment to assess awareness of CDSi resources, whether CDSi resources were being used by stakeholders, and stakeholders’ perceived impact of CDSi resources on their work. Stakeholders have wide and increasing awareness and use of the CDSi resources, coupled with high satisfaction and positive perceived impact. Users of the CDSi resources report a positive impact to develop and maintain immunization forecast and evaluation products, including improved timeliness in accommodating new and updated ACIP recommendations. This leads to more robust immunization clinical decision support engines, with increased accuracy and consistency. Timeliness is critical for IISs preparing to implement new vaccines, such as a potential vaccine for COVID-19. Successful implementation of CDSi tools is likely to facilitate a record of patients’ immunization status that is current, accurate, consistent, and readily available. Ultimately, as awareness and use of CDSi resources increases, the likelihood that patients receive proper immunizations at the right time also increases. This will likely reduce frequency and cost of vaccination errors, which in turn will reduce time, cost, and risks associated with revaccination and/or overvaccination. Our brief online assessment is an example other projects could learn from and implement to evaluate technical guidance, resources, or initiatives. While we found it important to measure awareness and use levels each round, ratings on questions regarding satisfaction and impact were found to be consistently high. Observing this, we edited the set of questions asked of respondents in each round, mindful of respondent burden. A mix of quantitative questions with options for respondents to provide additional comments allowed us to quantify responses as well as obtain deeper understanding when respondents provided clarifying or supporting detail in the additional comments section. These methods and lessons could be applied to evaluate many different types of technical resources or initiatives. Acknowledgements The authors thank Mary Ann Kirkconnell Hall for her help reviewing and editing the manuscript. Clinical Decision Support for Immunization Uptake and Use in Immunization Health Information Systems 11 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e10, 2020 OJPHI Financial Disclosure This work was supported by NCIRD – Support for Immunization Technical Standards and Resources Task Order 0002 #: 200-2015-87992/0002 References 1. Centers for Disease Prevention and Control. ACIP recommendations. Centers for Disease Prevention and Control website. https://www.cdc.gov/vaccines/acip/recommendations.html. Reviewed March 24, 2020. Accessed June 1, 2020. 2. Clinical decision support. Office of the National Coordinator for Health Information Technology (ONC). https://www.healthit.gov/topic/safety/clinical-decision-support. Reviewed April 10, 2018. Accessed June 5, 2020. 3. Centers for Disease Prevention and Control. About immunization information systems. Centers for Disease Prevention and Control website. www.cdc.gov/vaccines/programs/iis/about.html. Reviewed June 7, 2019. Accessed January 7, 2020. 4. Centers for Disease Prevention and Control. 2017 IISAR data participation rates. Centers for Disease Prevention and Control website. https://www.cdc.gov/vaccines/programs/iis/annual- report-iisar/2017-data.html#child. Updated September 22, 2017. Accessed April 21, 2019. 5. Centers for Disease Prevention and Control. Clinical decision support for immunization (CDSi). Centers for Disease Prevention and Control website. https://www.cdc.gov/vaccines/programs/iis/cdsi.html. Reviewed June 7, 2019. Accessed October 17, 2018. 6. Rodgers L, Shaw L, Strikas R, et al. 2018. Frequency and cost of vaccinations administered outside minimum and maximum recommended ages–2014 data from 6 sentinel sites of immunization information systems. J Pediatr. 193, 164-71. doi:https://doi.org/10.1016/j.jpeds.2017.09.057. PubMed. 7. Kirtland KA, Lin X, Kroger AT, Myerburg S, Rodgers L. 2019. Frequency and cost of live vaccines administered too soon after prior live vaccine in children aged 12 months through 6 years, 2014–2017. Vaccine. 37(46), 6868-73. doi:https://doi.org/10.1016/j.vaccine.2019.09.058. PubMed 8. Metroka A, Sull M, Faber G, Aponte A, Bunker N. Electronic health record query-response: building on our success. Presented at the 2018 American Immunization Registry Association National Meeting; August 16, 2018; Salt Lake City, UT. https://repository.immregistries.org/files/resources/5b9bdded9dfb5/c8_- _a_metroka_ehr_qr.pdf. Accessed October 1, 2019. 9. Patton MQ. Utilization-focused Evaluation. 4th ed. Saint Paul, MN: Sage Publications; 2008. https://www.healthit.gov/topic/safety/clinical-decision-support https://doi.org/10.1016/j.jpeds.2017.09.057 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=29249524&dopt=Abstract https://doi.org/10.1016/j.vaccine.2019.09.058 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=31563283&dopt=Abstract