Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Stephen L. Brown1*, Barbara L. Massoudi1, Jamie M. Pina1, Kusuma Madamala2 1. Public Health Informatics Program, RTI International, Research Triangle Park, NC, United States 2. Public Health System Consultant, Portland, OR, United States Abstract Objectives: The Public Health Quality Improvement Exchange (PHQIX) is a free, openly available online community that supports public health practitioners in the rapidly evolving landscape of public health quality improvement (QI). This article’s objective is to describe the user-centered development of PHQIX and its current content and examine how elements of a QI initiative may vary by an organization’s characteristics or QI experience. Methods: PHQIX was developed by taking a user-centered iterative design approach, seeking early and continued input from users to gather requirements for the website. We performed an exploratory analysis of the published QI initiative descriptions, reviewing all QI projects that PHQIX users shared as of January 1, 2018. Results: PHQIX features 193 QI initiatives from a variety of health departments and public health institutes using a wide range of QI methods and tools. Discussion: Submitted QI initiatives focus on many public health domains and favor the PDCA/PDSA cycle; Kaizen; and fishbone diagrams, flowcharts, process maps, and survey methods. Limitations include data coming only from users who represent health departments with sufficient time to complete the PHQIX submission template. Additionally, many initiatives were submitted in part to fulfill a grant requirement, which could skew results. Conclusion: As the field of QI in public health practice evolves, resources targeted to QI practitioners should build on and advance the available resources. Findings from this study will provide insight into QI initiatives being performed and the types of projects that can be expected as organizational experience and collaboration grow. Keywords: informatics, public health, public health accreditation, quality improvement, science of improvement Abbreviations: Plan, Do, Check/Study, Act (PDCA/PDSA); Public Health Quality Improvement Exchange (PHQIX); quality improvement (QI); Expert Panel (EP); User Group (UG) Correspondence: stephenbrown@rti.org* DOI: 10.5210/ojphi.v10i3.9566 Copyright ©2018 the author(s) mailto:stephenbrown@rti.org* Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Introduction Quality improvement (QI) has become an increasingly important activity for health departments as they seek to efficiently improve the health of populations they serve. Public health departments are implementing QI initiatives rapidly, given these initiatives’ potential to streamline processes, reduce costs, improve health outcomes for populations, and implement a culture of quality in organizations [1-3]. Health departments are also moving toward QI to document capacity and performance standards as part of the process of accrediting their organizations [4]. The Public Health Quality Improvement Exchange (PHQIX) is a free, openly available online community that supports public health practitioners in the rapidly evolving landscape of QI in public health [5]. Supported by the Robert Wood Johnson Foundation, this effort has published 193 in-depth descriptions of real-world QI initiatives on its website since it launched in October 2012. PHQIX is unique among resources for public health practitioners because of its exclusive focus on QI in public health, QI project documentation, its community features, and its integrated search capacity that allows users to find examples of recent QI work from other health departments around the country [6]. The primary resource on the PHQIX website is the set of initiative descriptions that present detailed data in a structured format. The descriptions may also include attachments such as diagrams, QI tools and process templates, surveys, policies, and storyboards. Users can browse and search these descriptions through free-text search or by using a set of faceted terms based on a public health QI taxonomy developed specifically for the website. For example, users can search for projects in a variety of topical categories such as immunizations, laboratory services, maternal and child health, environmental health, and administrative areas. Although the QI initiative descriptions are not intended to represent best practices, they help practitioners learn from previous work performed at other health departments and allow them to adopt or adapt elements of the initiatives for their own purposes. Users frequently seek out initiatives that would be practical to implement at their own health departments and are interested in using products of these initiatives (e.g., storyboards, surveys) to increase QI capacity and accreditation readiness at their institutions [7]. The aim of this study is to describe the user-centered development of the tool and its current content and to examine how elements of a QI initiative may vary by an organization’s characteristics or QI experience. The content descriptions include an initiative’s methods (e.g., Lean/Six Sigma, Kaizen), tools (e.g., surveys, process maps), and focus activities (e.g., immunizations, data collection, administrative activities). This is an Open Access article. Authors own copyright of their articles appearing in the Online Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in the copy and the copy is used for educational, not-for-profit purposes. Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Methods PHQIX was developed by taking a user-centered iterative design approach [8-11], which sought early and continued input from users to gather requirements for the website, test early prototypes, and participate in an evaluation of the site [7]. Early in the project, we formed two sets of experts to guide the design and development of the system. Those working in QI in public health became members of our User Group (UG), and national-level experts in QI became members of the Expert Panel (EP). The UG and the EP participated in focus groups; from these groups, we developed storyboards to reflect requirements and design elements for the system. The storyboards were vetted through the EP and were used to guide the development of the initial prototypes, which were tested by stakeholders at public health conferences. These stakeholders were selected because they represented the intended users of the system, and they were identified through the EP. Feedback about the usability of the prototypes was sought using the think-aloud process while stakeholders interacted with the prototype by working through real-life scenarios. This feedback was then incorporated into the next prototype version, which was then tested with users again. A final iteration of the prototype was produced and launched in October 2012 at the American Public Health Association Annual Meeting and Expo. After the rollout of the first operational system in 2012, limited development occurred over the next 2 years, and in 2014, the system went into operations and maintenance mode. Community supporting features that were added during those 2 years included a map of submitters’ locations; QI Spotlight articles, featuring aspects of QI work in public health; video highlights of exemplary projects; Ask An Expert Q&A; the Community Forum threaded discussion list and associated incentives for participation; the monthly newsletter detailing information about the community and the project; and the weekly digest, providing quick snapshots of information about QI happenings. Our exploration of the published QI initiative descriptions consisted of a review of all 193 public health QI projects and the organizational characteristics that registered PHQIX users shared, as of January 1, 2018, approximately 5 years after the website’s initial launch. The users who submitted the QI initiative data were employees of state, local, and tribal health departments who had participated in a public health QI activity. Submissions are subject to a review process by the EP, composed of QI subject matter experts. Upon receiving new submissions, the site’s submission coordinator confirms that all appropriate data fields are properly completed, then distributes the initiative to an expert panel member for review. The panel member then assesses the relevance of the submission as a true QI initiative and determines whether sufficient documentation is provided. Finally, the panel member works directly with the initiative’s submitter to make any necessary clarifications or additions. All accepted submissions are then published on the website and have been included in this study population. The unit of analysis for this study is the QI initiative or project. We conducted an exploratory analysis of the data fields across the QI initiatives, examining what implications an organization’s characteristics have for the type of initiative it will conduct. We sought to determine whether health departments of a particular size, type, or experience with QI would affect the likelihood of them using specific tools or methods or selecting a specific area of services for improvement. The exploration of the initiatives included information about the health department and the population it serves, tools and methods used, project duration, types of partner Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI organizations with which the health department collaborated, and the health department’s level of QI activity. For any QI initiative, it is possible to have used multiple methods, tools, or focus areas. We used the chi-squared test to confirm all statistically significant associations among variables. This paper reports descriptive statistics of these self-assigned attributes and the results of multivariate analyses among QI methods, tools, and health department QI activity level. Results The PHQIX website features 193 QI initiatives from 38 states (Figure 1), which included 159 submissions from local health departments, 28 from state health departments, 2 from tribal health departments, and 4 from public health institutes. Washington State and North Carolina have produced the most published submissions, with 16 initiatives each. Oregon (13), Wisconsin (13), Illinois (12), and Michigan (11) all also published more than 10 submissions. The Midwest and Pacific Northwest regions have had more PHQIX submission activity than other regions, whereas 12 states have not published QI initiatives on PHQIX. The self-reported organizational QI activity level at the submitting organization varies in an ordinal range including formal QI in specific areas (39.9%), informal QI (21.2%), formal agency-wide QI (20.7%), QI culture (11.9%), and QI community (3.6%) [12]. The organizations serve populations ranging from fewer than 24,499 people (5.2%) to more than 1 million (17.1%). The most common population groups are 100,000 to 249,999 (23.3%), more than 1 million (17.1%), and 250,000 to 499,999 (16.2%). The most common submitting organization type is county health department (45.6%), followed by state health department (14.5%), city-county health department (7.8%), and multi-county health department (6.7%). Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Figure 1: Published PHQIX QI initiatives, by state: 2012–2017 The data show that submissions come from a wide variety of public health departments, with smaller health departments submitting fewer initiatives (26.9%), and health departments serving more than 100,000 people submitting more initiatives (73.1%). Health departments with a QI activity level of at least “Formal QI in Specific Areas” submitted the majority (76.2%) of published initiatives, whereas health departments with “Informal QI” submitted only 23.8% of the total number. Table 1 shows that although the Plan, Do, Check/Study, Act (PDCA/PDSA) cycle is the most commonly used QI method or approach (88.1%), health departments also use Kaizen (19.2%), rapid-cycle improvement (17.1%), Lean/Six Sigma (13.5%), and Model for Improvement (13.0%). The most frequently used QI tools reported in Table 2 were brainstorming (67.9%), fishbone diagrams (54.4%), process maps (47.2%), flowcharts (45.6%), surveys (42.5%), root cause analyses (39.4%), and the 5-Whys (32.6%). The most commonly reported partner organizations of the leading organization conducting the QI initiatives were local health departments (20.7%), state health departments (16.1%), community-based organizations (9.9%), and universities (6.2%). QI methods and tools do not vary widely by population served, type of health department, or level of QI activity. PDCA/PDSA methods are used widely across all organization characteristics, but health departments serving larger populations use the Kaizen method more (30.9% for populations greater than 500,000) compared with those serving smaller populations (12.7% for populations fewer than 500,000) (p=.005). Health departments for larger populations are also slightly more Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI likely to use Lean/Six Sigma (17.6% for populations greater than 250,000) than those for smaller populations (8.3% for populations fewer than 250,000). The same is true for rapid-cycle improvement (21.2% for populations greater than 250,000 and 14.6% for populations fewer than 250,000), although these observations did not achieve statistical significance. Table 1: QI methods/approaches for published QI initiatives (N=193)* QI Method/Approach Number of Initiatives Percentage Plan, Do, Check/Study, Act cycle 170 88.1% Kaizen 37 19.2% Rapid-cycle improvement 33 17.1% Lean/Six Sigma 26 13.5% Model for Improvement 25 13.0% Nominal group technique 5 2.6% Business process analysis 4 2.1% Adaptive promising practice 1 0.5% Standardize, Do, Check, Act cycle 1 0.5% Total quality management 1 0.5% *The methods/approaches listed in this table are not mutually exclusive; therefore, the sum of percentages exceeds 100.0%. Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Table 2: QI tools for published QI initiatives (N=193)* QI Tool Number of Initiatives Percentage Brainstorming 131 67.9% Fishbone diagram 105 54.4% Process map 91 47.2% Flowchart 88 45.6% Survey 82 42.5% Root cause analysis 76 39.4% 5-Whys 63 32.6% Cause-and-effect diagram 56 29.0% Prioritization matrix 49 25.4% Affinity diagram 45 23.3% Pareto chart 22 11.4% Run chart 21 10.9% Multi-voting technique 17 8.8% Check sheet 14 7.3% Force-field analysis 13 6.7% Histogram 8 4.1% Strengths-Weaknesses-Opportunities-Threats analysis 8 4.1% Control chart 7 3.6% Interrelationship digraph 6 3.1% Radar chart 6 3.1% Control and influence plot 5 3.6% Tree diagram 4 2.1% Process decision program chart 3 1.6% Specific, Measurable, Achievable, Realistic, Timely (SMART) chart 3 1.6% *The tools listed in this table are not mutually exclusive; therefore, the sum of percentages exceeds 100.0%. Health departments submitting to PHQIX also indicate the types of organizations they partnered with during their QI initiative. Although the most common partner organization types are not Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI significantly affected by submitting organization type or organizational QI activity level, state health departments are more likely to have a partner organization for their QI initiative. State health departments partner with local health departments (p=.02), other state health departments (p=.002), and community-based organizations (p=.001) in 36%, 36%, and 39% of initiatives, respectively. Meanwhile, local health departments partner with the same organization types at rates of 16%, 12%, and 5%, respectively. Table 3 shows the most common focus areas for published initiatives: policies/internal procedures and processes, QI and accreditation readiness, organizational effectiveness, and customer service/satisfaction, all in the Administration category. In addition, the data indicate that organizations with a higher organizational QI activity level reported with higher frequency that they perform initiatives focused on policies/internal procedures and processes and QI and accreditation readiness. Table 3: Most common focus activities for QI initiatives submitted by health departments (N=193) Focus Activity Number of Initiatives Percentage Policies/internal procedures and processes 40 20.7% Organizational effectiveness 37 19.2% QI and accreditation readiness 34 17.6% Customer service/satisfaction 32 16.6% Access to care 25 13.0% Data collection and management/information technology 22 11.4% Environmental health 22 11.4% Communications 20 10.4% Women, Infants, and Children programs 16 8.3% Workforce development 16 8.3% Communicable/infectious diseases 15 7.8% Prenatal care 14 7.3% Capacity development 14 7.3% Performance management 13 6.7% Childhood immunizations: Administration of vaccine to population 12 6.2% Maternal and child health (data collection, epidemiology, and surveillance) 12 6.2% Reportable diseases 10 5.2% Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Collaboration/resource sharing 10 5.2% Maternal and child health home visits 9 4.7% Food safety education 9 4.7% Financial management 8 4.1% Tobacco 8 4.1% As shown in Figure 2, the number of QI initiatives an organization reports performing per year appears to be associated with that organization’s self-reported organizational QI activity level. Figure 2 illustrates that of the organizations reporting informal QI, 83% perform only one to three initiatives per year, whereas only 29% of those reporting a QI community perform the same number of initiatives. Additionally, 0% of organizations reporting informal QI perform 7 to 10 initiatives per year, whereas 43% of those reporting a QI culture do so. Similarly, the percentage of organizations that perform 11 to 20 initiatives annually rises from 0% to 13% for the same groups. The informal QI organization set is also the only one with health departments that perform no QI initiatives per year (10%). Organizations with less formal QI undertook fewer QI initiatives in a given year. The most common duration of QI initiatives submitted to PHQIX, measured from start to finish, is 6–12 months (52.8%), followed by less than 6 months (21.2%) and 12–18 months (7.8%). Figure 2: Annual number of QI initiatives, by organizational QI level: 2012–2017 Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Discussion The PHQIX website and community were designed and developed with engagement of a broad group of public health and QI stakeholders. This ensured that the website would meet the needs of the intended user audience, which is largely composed of local and state public health departments and institutes. In examining those that submitted QI projects to PHQIX, we found that local health departments are much more likely to submit initiatives than state health departments. This is probably because of the significantly higher proportionate number of existing local health departments and because of increased interaction with PHQIX from local health departments that received QI grants from the Robert Wood Johnson Foundation. A higher percentage of submissions were received from staff at health departments working to formalize QI at their health department, which may be the result of increased efforts to perform and document QI work as they strive for accreditation. Conversely, health departments with informal QI perform fewer QI initiatives and may be less inclined to share their work, feeling that it may not meet the standards of QI performed by health departments with more formalized QI. State health departments showed a higher likelihood of collaboration with a partner, which may be caused by many factors, including assisting local health departments, a larger project scope, or more initiative stakeholders. The submitted QI initiatives focus on many different public health domains and favor the PDCA/PDSA cycle; Kaizen; brainstorming; and using fishbone diagrams, flowcharts, process maps, and survey methods. Supporting similar national findings of local and state health departments, the PDCA/PDSA cycle remains the predominant choice for QI method [13,14]. (insert citations of ASTHO and NACCHO Profile data) More recent submissions indicate growing use of the Kaizen method (particularly for organizations that serve larger populations), and future research should monitor this trend. Health departments serving larger populations may be more likely to use such methods as Kaizen and Lean/Six Sigma, perhaps because they have funds budgeted for trainings or seek grants featuring such methods. Although no significant relationships are evident among health departments’ type, capacity, or the methods and tools employed, an organization’s reported QI activity level may have an association with the number of QI initiatives it performs annually and the duration and focus of those initiatives. Health departments seeking to formalize QI will continue to increase the number of initiatives performed annually. . Conclusion As the field of QI in public health practice evolves, resources targeted to QI practitioners should build on and advance the available resources. The increasing number of health departments across the country seeking accreditation will continue to fuel interest in QI information and trainings. As more health departments are likely to increase the formalization of QI at their organizations, future research should continue to monitor the trends of initiatives from organizations with a growing QI culture. Understanding the trajectory of the field of QI in public health is important for practitioners and researchers alike. Findings from this study will provide insight into QI initiatives being performed and the types of projects that can be expected as organizational experience and collaboration grow. As previous studies used QI initiative data to establish a framework to define and assess the impact of QI [15] and to determine which characteristics of QI projects affect whether a given project will achieve its stated goals [16], collecting measures of efficiency or effectiveness could help to expand the usefulness of the database. Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI PHQIX is now being transitioned to the Public Health Accreditation Board [17], which will become the host of the resource. Limitations This study had several limitations: the first is that the data come only from users who represent health departments that could dedicate the time to complete the PHQIX submission template, along with making any necessary revisions. As a result, the PHQIX database represents a snapshot of QI activities in public health, and it is unknown whether that snapshot is representative of the universe of public health QI projects. Many of the initiatives described were submitted in part to fulfill a grant requirement, which could skew the data if only specific types of health departments were eligible for this grant funding or if such grants focused on implementing a specific QI method or tool. Additionally, although a panel of QI experts review the submissions, the accuracy of some of the collected data fields is reliant on the submitter’s understanding of the various options (e.g., health department’s organizational QI activity level) and are therefore subject to errors of self- report. Finally, although the database of QI initiatives featured on PHQIX is substantial, it may not be large enough to infer statistical significance in all observed trends and associations where true differences exist. Acknowledgements We would like to acknowledge the support of the PHQIX User Group, Expert Panel, and all website users for their contributions to PHQIX since its inception in 2012. Financial Disclosure This work was supported by a contract from the Robert Wood Johnson Foundation to RTI International (#716472). Competing Interests No Competing Interests References 1. Riley WJ, Moran JW, Corso LC, Beitsch LM, Bailek R, et al. 2010. Defining Quality Improvement in Public Health. J Public Health Manag Pract. 16(1), 5-7. https://doi.org/10.1097/PHH.0b013e3181bedb49 PubMed 2. Livingood WC, Sabbagh R, Spitzfaden S, Hicks A, Wells L, et al. 2013. A Quality Improvement Evaluation Case Study: Impact on Public Health Outcomes and Agency Culture. Am J Prev Med. 44(5), 445-52. https://doi.org/10.1016/j.amepre.2013.01.011 PubMed 3. Livingood WC, Peden AH, Shah GH, Marshall NA, Gonzalez KM, et al. 2015. Comparison of Practice Based Research Network Based Quality Improvement Technical Assistance and https://doi.org/10.1097/PHH.0b013e3181bedb49 https://doi.org/10.1097/PHH.0b013e3181bedb49 https://doi.org/10.1016/j.amepre.2013.01.011 https://doi.org/10.1016/j.amepre.2013.01.011 Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI Evaluation to Other Ongoing Quality Improvement Efforts for Changes in Agency Culture. BMC Health Serv Res. 15, 300. https://doi.org/10.1186/s12913-015-0956-3 PubMed 4. Riley WJ, Bender K, Lownik E. 2012. Public Health Department Accreditation Implementation: Transforming Public Health Department Performance. Am J Public Health. 102(2), 237-42. https://doi.org/10.2105/AJPH.2011.300375 PubMed 5. The Public Health Quality Improvement Practice eXchange. 2018 [cited 2018 Nov 16]; Available at: http://www.phqix.org/ 6. Pina J, Massoudi BL, Chester K, Koyanagi M. 2018. Synonym-Based Word Frequency Analysis to Support the Development and Presentation of a Public Health Quality Improvement Taxonomy. J Public Health Mgmt Pract. Epub ahead of print. doi: 10.1097/PHH.0000000000000805. 7. Porterfield DS, Marcial LH, Brown S, Throop C, Pina J. 2017. Evaluation of a Quality Improvement Resource for Public Health Practitioners: The Public Health Quality Improvement Practice Exchange. Pub Hlth Reports. 132(2), 140-48. https://doi.org/10.1177/0033354916689609 8. Bernard HR. Handbook of Methods in Cultural Anthropology. Lanham, MD: Altamira Press; 1998. 9. Muller MJ. Participatory Design: The Third Space in HCI. In: Jacko JA, Sears A, editors. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. Mahwah, NJ: Lawrence Erlbaum Associates; 2003. 10. Pilemalm S, Timpka T. 2008. Third Generation Participatory Design in Health Informatics— Making User Participation Applicable to Large-Scale Information System Projects. J Biomed Inform. 41(2), 327-39. https://doi.org/10.1016/j.jbi.2007.09.004 PubMed 11. Stanton NA, Salmon PM, Rafferty LA, Walker GH, Baber C, et al. Human Factors Methods: A Practical Guide for Engineering and Design. 2nd ed. Burlington, VT: Ashgate Publishing Company; 2013. 12. The Phases of a Culture of Quality. 2018 [cited 2018 Nov 16]; Available at: http://qiroadmap.org/the-phases-of-a-culture-of-quality/ 13. Association of State and Territorial Health Officials. ASTHO Profile of State and Territorial Public Health: Volume 4. 2017 [cited 2018 Nov 20]; Available at: http://www.astho.org/Profile/Volume-Four/2016-ASTHO-Profile-of-State-and-Territorial- Public-Health/ 14. National Association of County & City Health Officials. 2016 National Profile of Local Health Departments. 2017 [cited 2018 Nov 20]; Available at: http://nacchoprofilestudy.org/wp- content/uploads/2017/10/ProfileReport_Aug2017_final.pdf https://doi.org/10.1186/s12913-015-0956-3 https://doi.org/10.1186/s12913-015-0956-3 https://doi.org/10.2105/AJPH.2011.300375 https://doi.org/10.2105/AJPH.2011.300375 https://doi.org/10.1177/0033354916689609 https://doi.org/10.1016/j.jbi.2007.09.004 https://doi.org/10.1016/j.jbi.2007.09.004 Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 10(3):e223, 2018 OJPHI 15. McLees AW, Nawaz S, Thomas C, Young A. 2015. Defining and Assessing Quality Improvement Outcomes: A Framework for Public Health. Am J Public Health. 105(Suppl 2), S167-73. https://doi.org/10.2105/AJPH.2014.302533 PubMed 16. Beitsch LM, Carretta H, McKeever J, Pattnaik A, Gillen S. 2013. The Quantitative Story Behind the Quality Improvement Storyboards: A Synthesis of Quality Improvement Projects Conducted by the Multi-State Learning Collaborative. J Public Health Manag Pract. 19(4), 330-40. https://doi.org/10.1097/PHH.0b013e3182629054 PubMed 17. The Public Health Accreditation Board (PHAB). 2018 [cited 2018 Nov 16]; Available at: http://www.phaboard.org/ https://doi.org/10.2105/AJPH.2014.302533 https://doi.org/10.2105/AJPH.2014.302533 https://doi.org/10.1097/PHH.0b013e3182629054 https://doi.org/10.1097/PHH.0b013e3182629054