Lane_518 Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 88 Impact Evaluation of Nurse Advocacy Center for Underserved on Population Health Adrianne J. Lane, EdD, MSN, RN, CNE Professor, College of Health Professions, Northern Kentucky University, lanea6@nku.edu Abstract Purpose: The purpose of this article is to describe a project undertaken to measure the impact of an academic/practice partnership, the Nurse Advocacy Center for Underserved (NACU), on the population health of the region with a goal to design an impact evaluation to measure outcomes of an academic/practice partnership. Sample/Method: The sample included clients, staff, students, faculty, and volunteers from NACU sites. Data was collected through patient survey, focus groups, check-sheet, and satisfaction survey. Analysis was informed by a logic model and the Triple Aim. Findings: Results of focus groups identified themes of advocacy, environment, substance abuse, access, and preventative services. Check-sheet results revealed useful areas for measuring impact on regional health were perception of health and Hepatitis C status. Satisfaction survey results demonstrated that the experience in providing services through NACU was rated as good (10%), very good (50%), and excellent (40%) for non-students and good (15%), very good (23%), excellent (42%) for students. Findings support that NACU impact measures align with the foci areas identified by regional Health Department and greater community. Conclusions: A comprehensive impact evaluation is effective in measuring impact of the academic/practice partnership NACU on the health of the community and region. Dissemination of results will foster similar initiatives that address population health by creating academic/practice Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 89 partnerships that focus on health care disparities among the underserved. Academic/practice partnerships can adapt this logic model evaluation strategy for use in determining impact on similar populations, particularly in areas where access is limited such as rural communities. Population health capacity will be expanded through the dissemination of evidence related to developing and maintaining community and public health partnerships, building public health competence through such partnerships, and improving the health of target populations in urban, suburban, and rural communities. Keywords: health impact assessment, vulnerable populations, population health, program evaluation, health care quality, access, evaluation Impact Evaluation of Nurse Advocacy Center for Underserved on Population Health The need for healthcare is more evident than ever in the underserved population. According to the US Department of Health and Human Services the underserved are those who have “systematically experienced greater obstacles to health based on their racial or ethnic group; religion; socioeconomic status; gender; age; mental health; cognitive, sensory, or physical disability; sexual orientation or gender identity; geographic location; or other characteristics historically linked to discrimination or exclusion” (para 1, 2010). These obstacles lead to gaps in health status between the general population and the underserved; these gaps are called health disparities. With worldwide trends towards increasing urbanization of the population, a widening gap between the rich and the poor, and increasing health disparities between the rich and the poor, a concerted effort by all is required in order to address the health needs of the underserved. This gap is evident in two distinct groups of people: those in rural communities and those who are Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 90 homeless. Addressing the medical issues of both groups of people is a health equity challenge (RHIhub, n.d.; Koh & O’Connell, 2016). The health of the underserved is a barometer for the health of the population as a whole; i.e. the health of a region is only as good as the health of its most vulnerable population. The health of a community directly impacts the overall economic strength of a community (Blue Cross Blue Shield Association, n.d.) https://www.bcbs.com/the-health-of-america/articles/healthy- communities-mean-better-economy. High numbers of underserved in a community burden the health care system by inappropriate use of emergency departments and inability to pay for care. In one study by Truven Health Analytics 71% of all ED visits were avoidable (Rodak, 2013). This adds up to more than $18 billion per year spent in the US on avoidable visits to emergency departments (Choudhry et al., 2007). The purpose of this article is to describe a project undertaken to measure the impact of an academic/practice partnership, the Nurse Advocacy Center for Underserved, on the population health of the region. The goal of the project was to design an impact evaluation that measured outcomes of an academic/practice partnership addressing the needs of the underserved. The academic/practice partnership was between a Nurse Advocacy Center for Underserved (NACU) and a regional Health Department with a goal to build health capacity in the region among the underserved. One of the priorities in achieving improvements in population health is aimed at reducing health disparities in certain groups. This evaluation project benefited the public health community by demonstrating the effectiveness of an academic/practice partnership aimed at addressing health needs in underserved populations. The key objectives of this project were to 1) design a comprehensive evaluation plan based on a logic model, 2) collect and analyze data for identified strategies within the plan, 3) determine the next steps for NACU operations including Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 91 delivery of care, evaluation, advocacy, and sustainability, and 4) disseminate knowledge gained. Results of the impact evaluation provided a basis for the NACU to further its mission to improve health of the underserved. This project resulted in a process for ongoing impact evaluation of the NACU aligned with designated regional health outcomes as identified by the regional Health Department. Such ongoing evaluation provides a basis for seeking grants and funding to support collaborative efforts to address key health initiatives in the region. The focused and collaborative process followed can serve as a model for other communities, both urban and rural, to align outcomes measures and data collection to facilitate addressing regional health care needs. Other academic/practice partnerships can adapt the NACU logic model driven evaluation strategy for use in determining impact on similar populations. Disseminating the results will foster similar initiatives to address population health through creating similar academic/practice partnerships to address the health care disparities among the underserved. Background In 2006, a Nurse Advocacy Center for the Underserved (NACU) was formed by a group of nursing faculty and students at a Midwestern regional university. This university has as a strong mission of community engagement. The university is recognized for its civic engagement and community outreach. It was one of the first universities to be classified as a “Community Engagement” institution by the Carnegie Foundation for the Advancement of Teaching in 2006. A component of the university’s mission is to contribute to the economic, civic, and social vitality of the region. The university identifies community engagement as one of the five goals for achieving their mission. One strategy for working toward such a goal is to apply the talents of Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 92 faculty, staff, and students through research, outreach, partnerships, and other community engagement activities. Aligned with the mission of the university, the College of Health Professions (CHP) has a mission to “promote an innovative environment to facilitate excellence in academics, scholarship, engagement, practice, leadership and lifelong learning within a global context to promote health” (2017). This mission is strongly supported and promoted by College of Health Professions Advisory Board. The District Director of Health for the regional Health Department, is a long- time member of the College Advisory Board. This relationship has been core to the partnership and collaborative opportunities that college has had and continues to have with the Regional Health Department. The mission of the regional health department has changed little since 2014 and today is to “prevent disease, promote wellness and protect against health threats” (2017). To achieve the mission, the health department provides an array of programming focused on, but not limited to, flu vaccines, Hepatitis B vaccines, influenza vaccines, smoking cessation, diabetes management, immunizations, oral care, Affordable Care Act (ACA) enrolment assistance, HIV/AIDS, substance abuse, and maternal-child health. The support of the university, college, and regional health department are at the core of the ongoing services provided by Nurse Advocacy Center for the Underserved (NACU). The impetus for the founding of NACU is the recognition of the important role outreach nursing services play on impacting health disparities of the underserved. This center advocates for quality health care for all persons and has a mission to improve the health of the underserved. Collaboration, teamwork, and partnering are core to addressing these needs throughout region. NACU is a key partner in providing and supporting program foci in the underserved populations of the area. In 2012 NACU was selected as the recipient of a regional health department Award Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 93 of Excellence in Public Health. This award was given in recognition of individuals and organizations within the region that collaborate with the Health Department to promote public health. NACU was nominated by staff in the Clinical Services division of the Health Department. The nomination specifically mentioned NACU efforts to provide flu and Hepatitis B vaccine to the homeless population, reduce unnecessary visits to local emergency rooms and other health promotion activities. NACU has numerous partner sites which include, but are not limited to, homeless shelters, recovery centers, crisis centers, and public housing. Many nursing students have clinical rotations at one or more of the NACU partner sites. Over the years, NACU has been awarded grant money to support the hiring of nurse practitioners on a limited basis at several of the NACU partner locations. In 2008 a church in a nearby area that serves weekly dinners for those who are homeless became a partner of NACU partner. Nursing faculty and students provide care at the established nurse clinic for the adults who come to dinners offered twice a week. In the first year alone, there were 469 visits to the nurse clinic run by the faculty and students at the church. The main health care needs for which the individuals sought care were hypertension, respiratory infections, injuries on the job or from assault, pain, depression/anxiety, and dental problems. Clients are homeless men, women, and children, women in substance abuse rehabilitation, women in shelters for domestic violence, and individuals and families in public housing. NACU has a proven record of stability in structure, collaboration with community partners, and service learning for students that will allow it to meet the opportunities and challenges of the future Central to the NACU model is the delivery of services on site where underserved persons reside, work or congregate. Compared to the general population the underserved have greater barriers to health including lack of insurance, inability to pay, and lack of social support networks Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 94 which result in poorer health status. Gaps in health status of the underserved versus the general population are defined as health disparities. On-site services eliminate some of the barriers and decrease health disparities. Between 2003 and 2014, the number of locations where services have been provided has grown and the number of people with access to a nurse from barely over 100 to several thousand. The organization has continued to grow thanks to committed NACU leadership including the Director, Assistant Director, and faculty and staff of the college, ongoing support and opportunities provided by partnering agencies, particularly the regional health department, and projects led by nursing students. The NACU sites provide health education and health services. Specific services provided depend on the needs of that population and funding. NACU incorporates a model of placing faculty, students, and volunteers where the underserved reside, work, or congregate; this model has proven successful. Students from the BSN, Accelerated BSN, RN-BSN, respiratory therapy, counseling, social work, and public administration have become engaged with the underserved in the region through NACU. Characteristics of individuals who receive services at NACU locations include un-insured, under- insured, homeless, low education level, unemployed, or living in public housing. Since its very beginning in 2006, NACU has collaborated with the regional health department to provide services to the underserved throughout the region. NACU has proven its ability to work with the regional health department and other community partners in addressing community health care needs by placing faculty, students, and collaborators where the underserved live and congregate. Over these years NACU has collected both formative and summative evaluation data through both formal and informal means. Examples of formative assessment conducted include evaluation of processes for volunteer recruitment, grant funding, care delivery, care documentation, and flu vaccine administration. Changes have been made to the way services have Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 95 been offered based on formative assessments; these have included the addition of hours and/or days at specific locations, the addition of services such as those of a nurse practitioner or social worker counseling students and the addition of undergraduate community nursing clinical groups and BSN capstone students. Further, examples of summative assessment have included number of patients per year per site, number of visits per year per site, referrals to providers per year at some sites, number of attendees at health education programs per year at some sites, number of volunteers per year per site, volunteer hours per year, number of health fairs per year, number of foot clinics per year, number of flu vaccines per year, patient satisfaction with services at some sites, and student evaluation of engagement at various points at some sites. This intermittent data collection has been useful to insure good stewardship and determine mechanisms for fine-tuning. An example of such intermittent data collection is that in 2012 data was collected for all locations in which NACU provided services. The results revealed 914 nurse visits, 567 patients seen, approximately 106 students involved, 17 volunteer nurses, 6 paid staff, over 241 volunteer hours, 5 foot care clinics, over 24 health education sessions, 5 major health fairs, 189 flu vaccines, and 32 pneumonia vaccines. The total estimated value of donated services to the community was $61,382 for 2012. Even though much data has been collected intermittently, no formal impact evaluation existed to measure the impact of NACU on the overall regional health needs as outlined by the regional health department. The collection of evaluation data has not been consistent, comprehensive, nor focused toward overall impact evaluation. Impact is an outcome measure associated with long term changes in health behaviors, health related knowledge, higher levels of motivation related to health care, decreased use of resources, improved feelings of health, and the like. (Sharma & Petosa, 2014; Timmreck, 2003). Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 96 Purpose The purpose of this project was to measure the impact of a Nurse Advocacy Center for Underserved (NACU) on the population health of the region. The goal of the project was to design an impact evaluation that measured outcomes of an academic/practice partnership addressing the needs of the underserved. Another goal was that the impact evaluation developed would serve as a model to evaluate the impact of the academic/practice partnership, Nurse Advocacy Center (NACU) and the regional Health Department on building population health capacity. One of the priorities in achieving improvements in population health is aimed at reducing health disparities in certain groups (U.S. Office of Disease Prevention and Health Promotion, n.d.). Method The process for development of an impact evaluation began with the creation of a logic model. A Logic Model was created to guide this impact evaluation (see Figure 1). A Logic Model design works with both process and outcome evaluation and is well accepted as a best practice evaluation strategy for social service programs. Such a model provided a simplified account of the program (problem, intervention, goal), intended outputs (objectives), and intended outcomes (outcomes). The NACU Logic Model was created to coincide with the mission, vision, and goals of the organization. The mission of the Nurse Advocacy Center for the Underserved was to improve the health of the underserved in the region by reducing health disparities. The vision was the elimination of health disparities among the underserved with expansion of services to all underserved of the region. The NACU goals were to 1) Collaborate with other academic centers and community agencies to identify health needs and resources for the underserved, 2) Increase access and decrease barriers to healthcare for the underserved, 3) Provide culturally appropriate primary, secondary, and tertiary interventions through nursing care delivered directly to Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 97 underserved populations in their communities, 4) Advocate for culturally appropriate health care for the underserved, 5) Engage nursing students in the provision of culturally appropriate health care to the underserved, and 6) Continually assess trends and changes in other populations and adapt services as needed. Figure 1. NACU Logic Model Problem •Some adults in this region lack access to healthcare Intervention •Create an Nursing Advocacy Center for the Underserved (NACU) Goal •To provide culturally appropriate healthcare services to the underserved in their community by creating a NACU Objectives •Develop the necessary infrastructure for the NACU •Provide ongoing healthcare services by partnering and collaborating with the community •Secure funding to sustain ongoing services provided through NACU Outcomes •Increase collaboration with community partners to identify health needs and resources of the underserved in the region •Increase access to healthcare for the underserved in the region •Increase numbers of faculty, students, volunteers, and partners in providing healthcare services via NACU •Increase NACU involvement in advocacy activities for culturally appropriate healthcare for the underserved •Increase satisfaction among patients, faculty, students, volunteers, partners of NACU, and key regional health department stakeholders •Stabilize NACU through ongoing funding •Track demographics and service-related result •Disseminate knowledge gained Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 98 The creation of the logic model provided the basis for designing the comprehensive evaluation. This evaluation was an effort to capture services provided to the underserved population. The care provided to the underserved/the most vulnerable is often not captured. A goal of the comprehensive evaluation was to determine the impact of NACU services on the population health of the region. The question posed was ‘Has NACU produced their desired outcomes and addressed other common measures of impact’? This was best addressed via a series of sub-questions, all of which are directly linked to and guided by the NACU Logic Model. Key questions and sub-questions were identified to determine if NACU had an impact on population health in the area served by the regional health department. Examples of such questions included: 1. Has NACU increased collaboration with community partners to identify health needs and resources of the underserved in region? 2. Has NACU increased access to healthcare for the underserved in the region? a. Has NACU affected levels of motivation related to health care in the underserved? b. Has NACU affected improved feelings of health in the underserved? c. Have Emergency Room visits at the partner sites decreased with NACU presence? 3. Has the numbers of faculty, students, volunteers, and partners in providing health care services via NACU increased? 4. Has satisfaction increased among patients, academic partner faculty, academic partner students, volunteers, partners of NACU, and key health department stakeholders? The next step was the operationalization of the logic model which included developing a plan for data collection and identifying the sample. The sample varied depending on the data collection and included patients, stakeholders, staff, faculty, students, and volunteers who engaged in providing services at a NACU site. Measures were taken to protect the rights of all sample Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 99 participants. Institutional Review Board approval was sought and approved: IRB 16-011 Approved Expedited. The next step in this process was identifying and meeting with the key stakeholders. Aligning with the logic model, the data collection tools included a patient survey, focus groups with key stakeholders, a service-results check sheet, and a satisfaction survey. Data was collected at the NACU sites, focus groups were conducted, a service-results check-sheet was developed which would be used on an ongoing basis to collect key impact measures, and a satisfaction survey was distributed. The results of data collection and focus groups were analyzed and a comprehensive analyses was conducted informed by the logic model & Triple Aim: Health, Healthcare, and Costs. The application of the NACU logic model for impact analysis provided a systematic, focused plan for data collection which could (1) determine if the program was being implemented as specified and, if not, how operations differed from those initially planned; (2) identified unintended consequences and unanticipated outcomes; and (3) determined impact of the academic/practice partnership NACU on the community. Community meetings were planned for data collection alignment and futures planning of next steps for NACU including delivery of care, partnerships with regional health department and other community agencies, evaluation, advocacy, and sustainability. The last step of the process was dissemination of knowledge gained. Data Collection The data collection sites for this study included the community partner sites with NACU. These partners have demonstrated ongoing commitment to collaborating with NACU to meet the needs of the underserved within this urban service area. All NACU community partner sites are located in an urban setting. Data collection involved a variety of methods and target groups. Patient data was collected at each of the partner sites using a written survey distributed to patients. Stakeholder data was collected from key stakeholders at each site and from the regional health Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 100 department using focus group format. Satisfaction data was collected from patients, academic partner faculty, academic partner students, volunteers, community partners, and key health department stakeholders using a written survey. Service–related results data was collected from key stakeholders, health department key stakeholders, and the NACU director using a check-sheet format. The service-results check-sheet was developed as a component of this project based on NACU site and focus group data. Further, the ability to collect data as a component of this project allowed an evaluation foundation to be built. This ultimately served as the basis for ongoing comparisons, adjustments to programming, and future planning. For four of the partner sites, survey data was collected in 2012. This project allowed for a second round of data collection. Comparative analysis was conducted for those 4 sites. The patient survey was revised to reflect the implementation of the Affordable Care Act (ACA). The 2012 survey consisted of 26 questions addressing demographics, healthcare coverage, provider history, purpose of contact, health needs, and satisfaction. The revised 2015 survey of this project kept the first 26 questions exactly the same to facilitate analysis. Nine additional items were added that address ACA, perceptions of health and future health needs. For the other four partner sites, the collection of patient survey data serves as a baseline for future data collection. The ability to meet with the key partner site stakeholders and the partner health department representatives using a focus group format promoted engagement in participatory evaluation. The findings of the focus group informed future evaluation as well as served for the basis for developing a service-results check-sheet. The results check-sheet is to be used by both the stakeholders, the partner health department, and the NACU director on an annual basis to collect key measures contributing to impact. Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 101 Capacity building can only be strengthened by the commitment of those involved. Satisfaction is key to commitment. Satisfaction data for each of the constituents (patient, student, faculty, volunteer, and partner) and key health department stakeholders provided a barometer for measuring assurance that NACU is meeting its goals. Approval to conduct the project was obtained from the university Institutional Review Board. A poster was displayed at each site announcing the purpose of the project, requesting participation in the project, and noting that confidentiality would be maintained and assured. For purposes of patient data collection, an eligible participant was defined as one who met the participant criteria and presented as a patient to a NACU nurse at one of the NACU community locations. These patients were un-insured, under-insured, homeless, low education level, unemployed, or living in public housing. For the purposes of tracking outcomes, an ID number was placed on each Health Form. ID numbers were the only identifiers used on the spreadsheets. The participants were asked to complete a NACU Health Form at their visits to a NACU nurse at a community location. Each of the NACU community locations has a separate, private room where patients are seen. If they agreed to participate, the patients completed the form in that private room. Each Health Form took approximately 5 minutes to complete. If the patient had visual or other impairments that make filling out the form difficult, a graduate assistant assisted in reading the form to the patient. No other time or effort was required of participants. When a patient came to a NACU nurse, the nurse explained the services offered and the study. If the patient agreed to participate, the Informed Consent document was given to the patient for completion. Patients were not denied services if they declined to participate in the study. At any time, any participant could decline participation at any time. There was no direct benefit to participants of this study. Indirect benefits included Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 102 improved processes for data collection alignment among multiple agencies with the overall goal to improved healthcare to the underserved and others in the region. For purposes of stakeholder data collection, the principal investigator invited the key contact at each of the community partner sites and the regional health department to a focus group session: a second session was offered to allow attendance by all. The script for the focus groups addressed collaboration, health needs, resources, advocacy, and impact measures. The PI facilitated and recorded the focus group sessions. Findings from the focus group sessions were used both to guide the development of the outcome check-sheet by the PI and to inform the evaluation. All academic partner faculty, academic partner students, volunteers, and site partners providing services at a NACU location during a patient data collection period were asked to complete a satisfaction survey. The service-related results check-sheet was completed by both partner stakeholders and the NACU director. As part of the comprehensive plan, data collected during this project was compared to data collected in 2012. In 2012 four of the five partner sites participated in a research study conducted by NACU. An aim of the 2012 study was to collect data to determine if NACU services were effective at improving access and decreasing barriers to health care. Data collection strategies from clients used in this previous research study were the same as proposed here. Analytic approach The data collected through the various methodologies was analyzed in the following manner. Demographic variables were analyzed through the use of descriptive statistics. Comparative analysis were conducted on the Patient Survey data for years 2012, 2015, and 2016. This included Frequencies, Means, Ranges, Chi Square, and T-Tests. These analyses addressed changes in participant characteristics between the years. Focus Groups were conducted in 2015 to collect Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 103 input from partner stakeholders and the regional health department representatives. Focus group data was analyzed using thematic analysis. The 2012 and 2015 Patient Survey data and 2015 Focus Group results were used to design the service-results check-sheet and inform the evaluation. Service-Results check-sheet data was analyzed using descriptive statistics: Frequencies, Means, and Ranges. Satisfaction Survey data was analyzed using Frequencies, Means, Ranges, and ANOVA. The ANOVA calculation answered whether the group means differed from one another. All analyses were informed by the Logic Model and categorized in terms of the Triple Aim: Health, Health Care, and Costs. Results The key objectives of this project include: 1) design a comprehensive evaluation plan based on the logic model (see Figure 1), 2) collect and analyze data for identified strategies within the plan, 3) determine the next steps for NACU including delivery of care, partnerships with regional Health Department and other community agencies, evaluation, advocacy, and sustainability, and 4) dissemination of knowledge gained. A meeting was held with the key stakeholders in each of the project sites for the NACU and the regional Health Department. All stakeholders were fully engaged. Focus groups were held with the key stakeholders. Data was collected at all NACU sites in 2015 and 2016. The check-sheet was developed based on NACU and focus group data. Data was collected using the newly developed check-sheet in 2016. 2015 and 2016 data was analyzed. 2015 data was compared to 2016 data as well as compared to data collected at all NACU sites in 2012. Satisfaction data was collected at each NACU site. The check-sheet was shared with the stakeholders. The NACU patient registration form was the foundation for completing the impact assessment check-sheet. Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 104 Key client data included demographics, allergies, insurance, tobacco use, alcohol use, recreational drug use, homelessness, living arrangements, employment, barriers to healthcare, and perceptions of personal health. The results demonstrate both similarities and differences among the NACU participants in 2012, 2015, and 2016. In 2016 as compared to both 2012 and 2015, the NACU participants included more females, fewer of Black and Hispanic ethnicity, fewer employed, and many fewer uninsured (see Figure 2). Figure 2. Key Demographics of Participants by Year Results revealed that a number of participant characteristics changed at various levels of significance between 2015 and 2016 and between 2012 and 2015 (see Table 1). Barriers to care increased between 2015 and 2016; however, the barrier of transportation was less of a barrier in 2016 as compared to 2015. Further, participant use of recreational drugs and drinking reduced in 0 10 20 30 40 50 60 70 80 90 100 Fe ma le <3 0 30 -50 >5 0 W hit e Bl ac k Hi sp an ic Ot he r Un ins ure d Em plo ye d HS /G ED Key Demographics 2012 2015 2016 Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 105 the same timeframe, with tobacco use increasing. Between 2012 and 2015, the characteristics of gender, ethnicity, and status of being uninsured also experienced significant change. Results of the stakeholder focus groups identified several themes regarding gaps in services. The themes were awareness of public health services, linking patients to such services, evaluating effectiveness, and lack of documentation on homeless populations. One goal of the focus groups was to serve as the basis for developing a service-related check-sheet. Among the various areas identified as useful for measuring impact on regional health by the regional health department stakeholders were the perception of health as well as Hepatitis C status. The service-related check- sheet was developed based on the focus group input and data was collected using a check-sheet format was collected at each NACU site (See Figure 3). The check-sheet was evaluated as reasonable and informative by the NACU director. Identified variables included age, sex, ethnicity, insured status, income, living arrangements, alcohol use, tobacco use, recreational drug use, occupation, education level, vaccination history, medical history including Hepatitis C, mental health diagnosis, and pain management, barriers to care, perceived health status, referrals, and number of visits to site per year. The check-sheet will serve as the code book for data entry of the variables found on the revised patient registration form. The revised patient registration form informed via the focus groups was implemented without concerns or comments from the NACU site personnel. The regional health department leadership expressed support of the impact assessment via the check-sheet data. Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 106 Table 1 Variables of Significance Two-Year Comparison 2016: 2015 Participant Characteristics p value for significant change Summary No barriers to care <0.0001 Barriers have increased Transportation <0.0001 Transportation remains a barrier but less of a barrier in 2016:2015 Tobacco <0.0001 Less people who do not smoke More smoking 10 or more cigarettes per day. Recreational Drugs <0.0001 Less use overall heroin, meth, cocaine, marijuana, opiates, crack, Xanax Drinks <0.0042 More indicating no drinking 2015: 2012 Participant Characteristics p value Summary Ethnicity <0.0031 More reporting white with fewer indicating black, Hispanic Uninsured <0.0001 More reporting insurance Gender <0.0001 More responding female Results of the satisfaction data collected from academic partner faculty, academic partner students, volunteers, and site partners revealed that the experience in providing services through NACU was rated as good (10%) to very good (50%) to excellent (40%) for the faculty, volunteers and site partners and good (15%) to very good (23%) to excellent (42%) for the students. Comments from the students included that more time should be planned for students to participate more fully at the NACU sites. An unanticipated development was the alignment of the NACU impact assessment with the Regional Gen H goals. A regional Health Collaborative had been meeting for the two previous years to identify the key health strategies to be addressed in a multi-state region. Ultimately, four areas of foci were unveiled. These included healthy behaviors, care delivery, finance and payment, Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 107 and equity. An identified goal related to healthy behaviors is that 70% of the tristate regional residents will report excellent or very good health. One of the NACU impact assessment measures and an item added to the NACU patient registration form is ‘How do you perceive your overall health?’ Other NACU impact measures align with the newly released foci areas identified in our greater community beyond the regional Health Department. Patient Name Date Reason for Visit: Sex Male Female Age DOB (Reported in intervals) Insurance Yes No Type: Barriers to Care Transportation Time Work Cost Other: Income Under $10,000/ year $10,000-$20,000/year $20,000- $30,000/year Over $30,000/year Living arrangements Homeless Yes No If yes, how Long? Home/Apt Street Car Hotel Shelter Rehab Other: Alcohol Use Yes No Drinks per day Tobacco Use Yes No Chew Yes No How much? How often? Cigarettes Yes No Cigarettes per day Rec Drugs Yes No Type? How often? Occupation- Do you currently work? Yes No If yes, type of job: Education Level Less than 8th grade High School grad/GED College Ethnicity White Black Hispanic Other: Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 108 Vaccinations Influenza Pneumonia TDAP Tetanus Hepatitis B Other Medical History Diabetes Hypertension/Heart Disease Respiratory Disease (Asthma, COPD, Bronchitis) Hepatitis C Mental Health Diagnosis Substance Abuse Pain Management List Other: How do you perceive your overall health? Excellent Good Average Poor Extremely Poor Referrals (within last year and ongoing) Date Agency Reason Number of visits to this site/year Figure 3. NACU Impact Evaluation Check-sheet Discussion The results of the project support that all outcomes of the project were achieved: 1) academic- practice partnerships were strengthened, 2) a sustainable impact evaluation plan’ including ongoing comparisons/adjustments to programming and future planning for this academic practice partnership, was created, and 3) a plan for the dissemination of the process for designing the impact evaluation. Further, a logic model, a service-related check-sheet, and an impact evaluation were developed; baseline and/or comparative data were collected and analyzed; and a plan for ongoing evaluation and dissemination was operationalized. Capacity building can best be strengthened by the commitment of those involved. Satisfaction is key to commitment. Satisfaction provided a barometer for measuring assurance that NACU was and continues to meet its goals. A systematic, focused plan for collecting and analyzing data was developed. Results were categorized according to the Triple Aim: Health, Health Care, Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 109 and Costs. Using a comprehensive impact evaluation strategy, provided the ability to determine if the program was being implemented as specified and, if not, how operations differed from those initially planned. Using an impact evaluation strategy allowed for the identification of unintended consequences and unanticipated outcomes. A comprehensive impact evaluation provided the tools needed to determine the impact of the academic/practice partnership NACU on the community and region. This comprehensive evaluation continues to serve as the basis for evaluation, advocacy, and sustainability. A number of limitations of this project were identified. The major limitation of the project was that all participants were via convenience sample. The patients at the NACU sites, the key stakeholder, and the volunteers could choose to participate or to not participate. The focus of the project was a single academic/practice partnership located in an urban area. The PI was the data collector. Another limitation of the project is that the 2012 data was historical data and the newly collected 2015 and 2016 data was matched as best possible. Results of the impact evaluation provided a basis for the NACU and the regional health department to further their missions to improve health of the underserved. The dissemination of the results can foster similar initiatives that address population health by creating academic/practice partnerships that focus on the health care disparities among the underserved. Academic/practice partnerships, similar to the NACU and the regional health department partnership, can be duplicated in other areas of the country. Population health capacity will be expanded through the dissemination of evidence related to developing and maintaining community and public health partnerships, building public health competence through such partnerships, and improving the health of target populations. Other academic/practice partnerships can adapt this logic model evaluation strategy for use in determining impact on similar populations. Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 110 Further, this process serves as model in developing an impact evaluation for programs or projects related to the health of a region or another entity. This project resulted in the creation of a logic model and an impact evaluation, both of which are key in measuring specific outcomes against global outcomes. Developing such a model can be particularly useful in measuring the impact of programs and/or interventions in rural communities with lack of access to needed healthcare services. Rural communities are those with open country and settlements with fewer than 2,500 residents (US Department of Agriculture, n.d.). Those people living in rural areas experience significant health disparities, including increased COPD, higher rates of risky health behaviors, and increased mortality rates (RHIhub, n.d.). This evaluation methodology can be useful for linking rural needs, rural efforts, and rural outcomes with regional and/or state goals and objectives. The process serves as a collaborative pathway for creating effective evaluation between agencies. Using an impact evaluation on an ongoing basis can provide the data necessary for grant funding, resource allocation, and sustainability planning. Conclusions This project benefited the public health community by demonstrating the effectiveness of an academic/practice partnership aimed at addressing health needs in underserved populations. The NACU impact evaluation keenly aligned with designated regional health outcomes as identified by the regional Health Department. Ongoing evaluation can provide data necessary for seeking grants and funding to support collaborative efforts that address key health initiatives in a region. This process can serve as a model for other communities, both urban and rural, to align outcomes measures and data collection to facilitate addressing regional health care needs in a focused and collaborative approach. Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 111 Acknowledgements This project was supported by the AACN/CDC Academic/Public Health Impact Evaluation Projects -- Small Grants, 2016. References Blue Cross Blue Shield Association, (n.d.) Healthy communities mean a better economy. Retrieved from https://www.bcbs.com/the-health-of-america/articles/healthy-communities-mean- better-economy Choudhry, L., Douglass, M., Lewis, J., Olson, C.H., Osterman, R, & Shah, P. (2007). The impact of community health centers & community—affiliated health plans on emergency department use. Retrieved from http://www.communityplans.net/Portals/0/MMC%20 Bibliography/FINAL%20ED%20Report%204.07.pdf Koh, H.K. & O’Connell, J. J. (2016). Improving health care for homeless people. JAMA. 316, 2586 - 2587. http://dx.doi.org/10.1001/jama.2016.18760 Northern Kentucky Independent District Health Department. Mission. (2017). Retrieved from https://nkyhealth.org/about-us/mission-vision-and-values/ Northern Kentucky University College of Health Professions. Mission (2017). Retrieved from https://www.nku.edu/academics/healthprofessions/about.html Rodak, S. (2013). Study: 71% of ED visits unnecessary, avoidable. Becker’s Hospital Review. Retrieved from https://www.beckershospitalreview.com/patient-flow/study-71-of-ed-visits- unnecessary-avoidable.html RHIhub. (n.d.). Rural health disparities. Retrieved from https://www.ruralhealthinfo. org/topics/rural-health-disparities Online Journal of Rural Nursing and Health Care 18 (2) http://dx.doi.org/10.14574/ojrnhc.v18i2.518 112 Sharma, M. & Petosa, R. (2014). Measurement and evaluation for health educators. Burlington, MA: Jones & Bartlett Learning. Timmreck, T. (2003). Planning, program development, and evaluation, (2nd ed.). Sudbury, MA: Jones and Bartlett. United States Department of Agriculture. (n.d.) What is rural? Retrieved from http://www.ers.usda.gov/topics/rural-economy-population/rural-classifications/what-is- rural.aspx United States Department of Health and Human Services, Office of Disease Prevention and Health Promotion (n.d.). Healthy People 2020 [Internet]. Washington, DC. Retrieved from https://www.healthypeople.gov/ United States Department of Health and Human Services National Prevention Council. (2010). Elimination of health disparities. Retrieved from https://www.surgeongeneral.gov/priorities/ prevention/strategy/health-disparities.pdf