733_QOC+comparison+revised+4+11_accepted_setup_4_13_23psf+ar_formatted (1) Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 150 Quality Indicators in Critical Access Hospitals and Small Rural and Urban Hospitals Marianne Baernholdt, PhD, MPH, RN, FAAN1 Nancy Dunton, PhD, FAAN2 Elizabeth M. Grandfield, PhD3 Emily Cramer, PhD4 1Sadie Heath Cabaniss Professor and Dean, School of Nursing, University of Virginia, mb2vy@virginia.edu 2Research Professor, School of Nursing, University of Kansas Medical Center, ndunton@kumc.edu 3Assistant Professor, Department of Methodology and Statistics, Utrecht University, e.m.grandfield@uu.nl 4Research Associate Professor, School of Nursing, University of Kansas Medical Center, ecramer@cmh.edu Abstract Purpose: Critical Access Hospitals (CAHs) have served their rural communities since they were created in the 1990s. CAHs have been exempt from the multiple organizational performance-based approaches that use financial incentives and thus quality indicators targeting CAHs specifically do not exist. Reports have suggested indicators that are appropriate for rural hospitals and CAHs but none have included all types of quality indicators - structures, work processes, outcomes, and quality improvement (QI) activities. Using a subset of the National Database of Nursing Quality Indicators® (NDNQI®) we assessed whether existing indicators for the work environment (structure), patient - and nurse outcomes are appropriate for CAHs. Methods: Using 2017 NDNQI® data on 16 structure and outcome indicators (8 work environment, 6 patient- and 2 nurse outcomes) were extracted. Employing bootstrapped Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 151 confidence intervals CAHs were compared to other small (less than 50 beds) rural and urban hospitals. Findings: 103 CAHs, 63 small rural hospitals, and 91 small urban hospitals had data on at least one indicator (range 87-239 for each indicator). There were no differences between CAHs and other small rural and urban hospitals for 15 of the 16 quality indicators. CAHs had fewer bachelor-prepared nurses compared to urban hospitals. Conclusion: Our results suggest that quality indicators for structure and outcomes currently used in approximately 2000 hospitals are also applicable for CAHs. Whether they apply to all 1350 CAHs and can be used to guide benchmarking and QI in CAHs needs further study. Keywords: Critical Access Hospitals, Quality Indicators, National Database of Nursing Quality Indicators® Quality Indicators in Critical Access Hospitals and Small Rural and Urban Hospitals It has been 20 years since the National Academy of Medicine (NAM, formerly Institute of Medicine (IOM)) released Crossing the Quality Chasm IOM (US) Committee on Quality of Health Care in America (2001). The report made clear that poor quality and safety outcomes were not caused by bad clinicians but rather because of poor systems. Subsequently, multiple organizational performance-based approaches with both financial and accreditive incentives using reports of quality indicators were implemented, targeting first hospitals and later all settings of health care (What Is Pay for Performance in Healthcare?, 2018). Of the approximately 1800 rural hospitals in the United States (American Hospital Association, n.d.), more than 1300 are Critical Access hospitals (CAHs) certified under the Medicare Rural Hospital Flexibility Program (the FLEX program) which was created by Congress in 1997 (Casey et al., 2012). Compared to other hospitals, CAHs have no more than 25 beds, receive cost-based reimbursement from Medicare and have limited reporting requirements (Casey et al., 2015; Lahr et al., 2020; Quick et al., 2019). Thus, quality indicators targeting CAHs Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 152 specifically do not currently exist, making it difficult to ascertain how well CAHs’ perform including their structures, work processes, outcomes, and quality improvement (QI) activities, all typical quality indicators (Agency for Health Research and QualityAHRQ, 2015.; Mitchell & Lang, 2004). Several reports have addressed the lack of quality indicators for all rural hospitals and CAHs specifically. One example is a National Quality Forum (NQF) workgroup that addressed which indicators would be appropriate for all rural hospitals (NQF, 2018). The indicators had to be cross-cutting, resistant to low case-volume and address transitions in care. Further, the authors recommended the indicators should be a mix of process and outcome measures. In general, health care quality measures are classified as either, structure (what needs to be in place in an organization), process (care to improve outcomes) and outcomes (the result of structure, process and patient characteristics) (AHRQ, 2015). Twenty core measures were recommended: nine for hospitals and 11 for ambulatory care. Among the nine hospital outcome measures were hospital acquired infections including catheter-associated urinary tract infections (CAUTI) and Clostridioides difficile (c. diff) infections; falls with injury; readmissions, and patients’ assessments of care using the Hospital Consumer Assessment of Healthcare Provider (HCAPS), which is a survey with 29 questions developed by AHRQ (Centers for Medicare & Medicaid Services, 2021). A process measure was emergency transfer communication. Another example addressing the gap in rural relevant measures is an earlier study focused only on quality indicators appropriate for CAHs. This paper had almost all process indicators related to pneumonia, heart failure, and acute myocardial infarction indicators (Casey et al., 2013). In both examples, the recommendations did not include indicators for structures (e.g., staffing), work processes (e.g., collaborations across departments and between employees), or QI activities (e.g., type and content). Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 153 Simultaneously with these papers, the Medicare Beneficiary Quality Improvement Program (MBQIP) was established in 2011 to assist CAHs with their QI activities. Today, through MBQIP, CAHs voluntary report quality indicators in four domains: patient safety/inpatient, outpatient, patient engagement, and care transitions (Lahr et al., 2021; Swenson & Casey, 2016). In 2019, 94% or 1270 CAHs reported on at least one of nine patient safety/inpatient indicators which included six for hospital-acquired infections (outcomes), transfer from the emergency department (ED) to inpatient unit (process), and a set of antibiotic stewardship elements (process) (Lahr et al., 2021). In 2018 the Federal Office of Rural Health Policy started requiring that CAHs report antibiotic stewardship data to the Center for Disease Control and Preventions National Healthcare Safety Network (NHSN). These data are then included in MBQIP. Except for the antibiotic stewardship which 1077 CAHs reported on, for the other nine indicators the variance in rates was large with only 233-282 CAHs reporting on different types of surgical site infections and 621-789 reporting on other infections. A concern is that some indicators are not appropriate for all CAHs (i.e., surgical infections) and another concern is that volume is too low to report a rate. Therefore, it is difficult to compare CAHs performance with that of other hospitals using data from MBQIP. Looking to other databases with indicators that are appropriate for all hospitals, a prominent example is the National Database of Nursing Quality Indicators® (NDNQI®) which includes indicators addressing structure, process and outcomes at the nursing unit level (Montalvo, 2007). NDNQI® was established in 1998 by the American Nurses Association with 30 hospitals submitting data. By 2014 more than 2000 hospitals submitted data to NDNQI®, the same year Press Ganey acquired NDNQI® (Press Ganey, 2014). NDNQI® provides participating hospitals quarterly benchmark data at the nursing unit level, so they can follow their own improvements and compare themselves to like units across the nation (Duncan et al., 2011). Indicators include structural indicators such as unit type and the work environment (i.e., Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 154 staffing, education, and experience), process indicators including prevention interventions for pressure injury and falls, and both patient and nurse outcomes, for example falls and pressure injury for patients and job enjoyment and job plans (intent-to-stay) for nurses. Numerous studies have used NDNQI® to establish the importance of structures and process for patient and nurse outcomes (Beck et al., 2019; Park et al., 2014; Waters et al., 2015). A few studies have used NDNQI® to compare rural and urban nursing units. Using 2009 NDNQI® data results showed that compared to urban nursing units rural nursing units had less care provided by RNs and fewer RNs with a BSN or higher, rated their work environment and job satisfaction lower, but had higher intent-to-stay, and lower rates of the patient outcome, pressure injury (Baernholdt et al., 2017). In a subsequent study, rural location was not associated with fall rates, but the nurse work environment was: lower ratings of the practice environment, lower staffing, lower percentage of RNs with a Bachelor of Nursing degree, more RNs with less experience, and worse nurse outcomes (job satisfaction and intent- to stay) were all associated with higher fall rates (Baernholdt, Hinton, et al., 2018). In a final longitudinal study using 4 years of NDNQI® data results showed lower pressure ulcer rates were associated with an increase in care interventions (risk and skin assessment on admission and any risk assessment before a pressure ulcer was documented), a 10% increase in RN skill mix (percentage of all nursing care hours provided by RNs), higher job satisfaction and lower turnover intentions were significant in both rural and urban nursing units (Baernholdt et al., 2020). None of the previous studies used CAHs as a comparison group. Currently more than 100 CAHs report one or more quality indicators to NDNQI®. We propose to assess whether NDNQI® indicators are appropriate for CAHs. We compare indicators for the nurse work environment (structure), patient and nurse outcomes between CAHs and other small rural and urban hospitals. Since we are using de-identified data, the study received IRB exemption. Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 155 Methods Data Source This cross-sectional study used 2017 data from NDNQI®. Hospitals submit data on staffing and clinical quality measures each quarter. Participation in measures varies across hospitals and units within hospitals. Data submitted by hospitals are collected according to highly standardized protocols and come from a variety of sources including hospital payroll, patient healthcare records, adverse event (incident) reports, special data collection, and for about 35% of hospitals an annual survey of Registered Nurses (RNs). RN surveys are only included if at least five nurses and 50% of the RNs have responded. Several studies have confirmed the validity and reliability of NDNQI® measures (Bergquist-Beringer et al., 2011; Choi et al., 2014; Choi & Staggs, 2014; Klaus et al., 2013; Waugh & Bergquist-Beringer, 2016). We included CAHs and urban and rural hospitals with 50 beds or less which had submitted data in 2017. Urban hospitals included metropolitan (a core urban area of 50,000 or more population) and rural hospitals included micropolitan (an urban core population of at least 10,000, but less than 50,000) and neither metropolitan nor micropolitan (Office of Management and Budget, 2010). First, we extracted data from the following hospital units for all three hospital types: EDs, medical, surgical and medical-surgical combined (med/surg), and intensive care units (ICUs) and for CAHs we also extracted data from designated critical access units if present. Second, data for all quarterly staffing and patient outcomes were annualized by summing numerator values (e.g., number of patient falls) and denominator values (e.g., number of inpatient days) and calculating rates (e.g. falls/patient days x 1000). Only indicators with data for at least three quarters were included. Third, we created hospital-level scores as CAHs typically have less demarcated units and staff that works in all areas of the hospital (Cramer et al., 2011). The hospital-level scores for the annualized staffing and patient outcome measures were created using a weighted standardization method to account for differences in Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 156 patient acuity and patient volume across unit types and then scores were averaged for the hospital. Standardized scores (z-scores) for each unit were calculated using unit type (e.g., med/surg, EDs, ICUs) mean and standard deviation estimates. For example, the CAUTI rate for each unit was adjusted for unit type by subtracting the average CAUTI rate for units of that type and then divided by the standard deviation of the CAUTI rates for units of that type. The resulting z-score is the difference in standard deviations, of the unit’s CAUTI rate from the average CAUTI rate for units of that type. Then each z-score was weighted by the patient device days for that unit (e.g. patient days). Further, the hospital average was calculated as the average of a hospital’s units weighted z-scores. Finally, z-scores were back-translated to their original metric for reporting of results, by multiplying the z-score by the full sample standard deviation and adding the sample mean. Sample A total of 253 hospitals had data on at least one measure including 103 CAHs, 63 rural hospitals, and 91 urban hospitals (see Table 1 below). The five hospital characteristics included were: 1. Census Division: Northeast, North Central, South, East South Central, West South Central, West North Central, Mountain, and Pacific; 2. Teaching status: Academic Medical Center and other teaching hospitals versus non- teaching hospitals; 3. Ownership: For-Profit, not-for Profit and Government-Federal; 4. American Nurses Credentialing Center (ANCC) Recognition as a Magnet Hospital or a Nursing Pathways Hospital (hospitals recognized for their nursing excellence) (ANCC, n.d.); and 5. Average Daily Census: Number of patients in a hospital for part or all of each day, summed for the month and divided by the number of days in the month. Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 157 Table 1 Hospital Characteristics of CAHs, other Small Rural, and Small Urban Hospital Characteristic Total (n=253) CAHs (n=103) Rural (n=63) Urban (n=91) Freq % Freq % Freq % Freq % Census Division New England 17 6.59 6 5.83 5 7.94 6 6.52 Mid Atlantic 16 6.20 2 1.94 4 6.35 10 10.87 East North Central 82 31.78 37 35.92 23 36.51 22 23.91 West North Central 36 13.95 24 23.30 5 7.94 7 7.61 South Atlantic 30 11.63 7 6.80 8 12.70 15 16.30 East South Central 5 1.94 1 0.97 2 3.17 2 2.17 West South Central 30 11.63 5 4.85 9 14.29 16 17.39 Mountain 17 6.59 7 6.80 5 7.94 5 5.43 Pacific 24 9.30 14 13.59 2 3.17 8 8.70 Teaching Status Teaching 19 7.37 3 2.91 5 7.94 11 11.95 Non-teaching 239 92.64 100 97.09 58 92.06 81 88.04 Ownership Not for Profit 237 91.86 98 95.15 61 96.83 78 84.78 Government 13 5.04 5 4.85 0 0.00 8 8.7 For Profit 8 3.10 0 0.00 2 3.17 6 6.52 ANCC Accreditation (Magnet or Pathway) No Accreditation 219 85.55 94 91.26 57 93.44 78 84.78 Accreditation 37 14.45 9 8.74 4 6.56 14 15.22 Mean SD Mean SD Mean SD Mean SD Daily Census (avg.) 17.52 11.10 8.91 4.37 21.56 8.19 24.40 11.75 Variables We extracted and calculated 16 indicators for the work environment, patient, and nurse outcomes. Work Environment Characteristics of the work environment included eight indicators. RN hours per patient day is a standardized measure of the supply of nursing care. On inpatient units, it represents total monthly productive hours worked by RNs who have direct patient care responsibilities divided by the total number of patient days for the month. In EDs, total productive hours are divided by the total time patients spend in the ED for the month, captured in minutes. RN Education is Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 158 measured as the percentage of direct care RNs on the unit who hold a bachelors or higher degree. There were five variables from the annual RN surveys. Appropriate assignment is RN’s ratings considering their assigned number of patients and the care they require and Influence over schedule is RNs ratings of how much influence they have over the hours/schedules that they work. Both measures have response options ranging from strongly agree (6) to strongly disagree (1). Years of nursing experience is how long a nurse has been on their current nursing unit. The next three variables ask about RNs’ work experience. RNs are asked how often this happens: Treated with dignity: “In my job I am treated with dignity and respect by everyone”; Meaningful contribution: “I have what I need in my job, so I can make a contribution that give meaning to my life”; Recognized and thanked: “I am recognized and thanked for what I do in my job.” Responses range from every day (5) to never (1). Patient Outcomes There were six indicators included for patient outcomes. Four were calculated from administrative data. Total fall rate is a sudden, unintentional descent, with or without injury, that results in the patient coming to rest on the floor or on or against some other surface or on another person. Falls are only counted if they occur on the patient’s unit. For inpatient units, the fall rate is calculated by dividing the number of patient falls by the total number of patient days. Injury fall rate is defined as the number of patient falls that result in an injury (mild, moderate, major or death). Pressure injury rate is a prevalence on a given day in a quarter and is the number of hospital-acquired pressure injury divided by the number of patients assessed for pressure injuries. CAUTI rate is hospital-acquired CAUTI divided by the number of Foley catheter days which are the daily count of patients with a catheter summed across all the days in a month, times 1000. From the RN survey there were two patient outcomes. Quality of care (QOC) where nurses rate QOC on their unit from excellent (4) to poor (1) and missed care Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 159 where nurses respond to a check-list of 16 activities, they may have left undone on their last shift because of time constraints. Nurse Outcomes There were two indicators included from the RN survey for nurse outcomes. Job Enjoyment which has seven items assessing global job enjoyment on a 6-level Likert scale (strongly agree (6) to strongly disagree (1)) and job plans where nurses indicate their job plans for the next year ranging from staying in current position to leaving the profession or retire. For this study it is reported as percentage of nurses who will stay on their unit or in the hospital. Statistical Analysis Due to some data having non-normal distributions, statistical assumptions required for traditional parametric group comparisons (e.g., ANOVA, t-tests) were violated. Therefore, differences across groups were examined using bias corrected bootstrapped confidence intervals in order to evaluate differences between CAHs compared to general hospitals in rural or metropolitan areas (Efron & Tibshirani, 1993). Bootstrapping was performed in SPSS version 26 using the bias corrected bootstrapping with 95% confidence intervals on 1000 samples. Results Study hospitals were found in all regions of the United States, with a plurality of all three hospital types in the East North Central Census region (Illinois, Indiana, Michigan, Ohio, and Wisconsin) (see Table 1). Most of the sample was non-teaching hospitals: CAHs (97.09%), rural hospitals (92.06%), and urban hospitals (88.04%). Similarly, a large majority of all three hospital types were nonprofit facilities (96.83%-84.78%). Small percentages of each hospital type were recognized as ANCC Magnet or Pathways hospitals: CAHs (8.74%), small rural Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 160 hospitals (6.56%), and small urban hospitals (15.22%). The average daily census was 8.91 patients in CAHs, 21.56 in small rural hospitals, and 24.40 in small urban hospitals. The included indicators had rates reported between 87-237 hospitals (see Table 2 below). Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 161 Table 2 Comparisons of Work Environment, Quality and Safety Indicators, and Nurse Outcomes Total CAHS Other Rural Compare to CAHs Urban Compare to CAHs Variable N Mean SD N Mean SD N Mean SD Upper CI Lower CI N Mean SD Upper CI Lower CI Work Environment RN Hours / Patient Day 177 11.07 3.85 66 10.94 4.02 45 11.32 3.75 -10.29 2.85 66 11.01 3.79 -1.17 1.88 % of RNs w/ Bachelors 118 40.52 16.60 41 36.89 15.93 35 37.55 15.82 -15.23 6.61 42 46.55 16.54 23.96 -0.22 Appropriate Assignment 87 4.82 0.43 36 4.95 0.45 18 4.78 .41 -.27 .22 33 4.70 0.39 -0.23 0.41 Influence over schedule 87 3.53 0.46 36 3.55 0.42 18 3.54 .49 -.31 .23 33 3.50 0.50 -0.16 0.48 Years of Experience 87 13.34 3.80 36 13.75 3.73 18 12.76 3.82 -2.09 2.12 33 13.20 3.92 -1.76 2.61 Treated with Dignity 87 4.26 0.30 36 4.28 0.29 18 4.26 0.37 -0.22 0.15 33 4.23 0.26 -0.22 0.16 Meaningful Contributio n 87 4.18 0.29 36 4.24 0.28 18 4.21 0.35 0.49 -0.31 33 4.10 0.26 -0.10 0.24 Recognized and Thanked 87 3.68 0.32 36 3.75 0.31 18 3.67 0.29 -0.23 -0.14 33 3.62 0.34 -0.12 0.29 Patient Outcomes Total Fall Rate* 229 2.73 1.65 96 2.97 1.91 55 2.67 1.42 -1.82 1.14 78 2.48 1.40 -0.60 0.63 Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 162 Injury Fall Rate 229 0.72 0.71 96 0.92 0.86 55 0.68 0.52 -0.18 1.25 78 0.48 0.53 -0.11 0.42 Pressure Injury Rate 191 1.10 2.21 70 0.96 2.04 50 1.38 3.01 -3.48 1.50 71 1.05 1.66 -1.34 0.76 CAUTI Rate 176 0.50 1.19 66 0.48 1.32 47 0.35 0.74 -0.36 1.42 63 0.64 1.32 -1.28 1.05 Quality of Care 87 3.52 0.23 36 3.55 0.20 18 3.49 0.33 -0.20 0.09 33 3.50 0.19 -0.16 0.12 Missed Care 87 1.65 0.69 36 1.52 0.79 18 1.65 0.54 -0.49 -3.48 33 1.80 0.62 -0.48 0.37 Nurse Outcomes Job Enjoyment 87 4.18 0.39 36 4.20 0.42 18 4.19 0.44 -0.39 0.14 33 4.16 3.50 -0.36 0.15 Job Plans (% staying) 87 85.36 10.29 36 85.63 10.98 18 85.46 11.64 4.10 -0.49 33 85.02 8.98 -11.12 3.14 Results in bold were significant at p<.001, SD=standard deviation, * Fall rates for ED units were not included Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 163 Work Environment. Overall, the eight work environment indicators were similar across the 3 hospital types, with the exception of RN education. The 10-point difference in the percent of RNs with a bachelor’s degree in CAHs (36.89%) and urban hospitals (46.55%) was significant. For the other seven work environment indicators, RN hours per patient day ranged from 10.94 in CAHs to 11.32 in other rural hospitals; appropriate assignment was rated 4.95 in CAHs to 4.70 in urban hospitals: years of experience was highest in CAHs at 13.75 years and lowest in other rural hospitals at 12.76 years. Nurses rated whether they felt they were treated with dignity, their work made a meaningful contribution, and were recognized and thanked for their work similarly high, between 3.62 (3 = some days) and 4.28 (4 = most days). Patient Outcomes. The six indicators were similar. Total fall rate ranged from 2.48 in urban hospitals to 2.97 in CAHs, injury fall rate from 0.40 in urban to 0.92 in CAHs, pressure injury rate from 0.96 in CAHs to1.38 in other rural, CAUTI rate from 0.35 in other rural hospitals to 0.64 in urban hospitals; QOC from 3.49 in other rural to 3.55 in CAHs, and missed care from 1.52 in CAHs to 1.80 in urban hospitals (so less than 2 out of 16 tasks were not done). Nurse Outcomes. Job enjoyment was rated from 4.16 in urban to 4.20 in CAHs indicating that RNs had moderate job enjoyment. Most RNs planned to stay in their current job: 85.6% of nurses in CAHs and 85% in urban hospitals planned to stay in their unit or hospital in the next year. Discussion Our study found no differences between CAHs and other small rural and urban hospitals for 15 of the 16 quality indicators across the work environment (structure), patient and nurse outcomes. For the work environment, we found CAHs had fewer bachelor prepared nurses compared to urban hospitals (36.9% vs 46.6), but similar rates to other rural hospitals, the latter Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 164 confirms previous findings (Baernholdt et al., 2014). The difference in CAHs and urban educational levels for RNs is similar (Baernholdt et al., 2017), higher (Jones et al., 2019), or lower (Odahowski et al., 2021; Probst et al., 2019) compared to results from other studies of rural and urban nurses’ educational levels. However, even our study’s urban rate of 46.55% RNs with a BSN is lower than the national 2019 average for all nurses of 59% (Campaign for Action,n.d.). This is noteworthy because research suggest that a higher percentage of BSN prepared nurses in hospitals is associated with improved patient outcomes including mortality, patient’s experience, QOC and fall rates (IOM, 2011; Aiken et al., 2017; Baernholdt, Hinton et al., 2018). In fact, three reports from NAM have maintained the recommendation of increasing the RN workforce to 80% with a bachelor degree (IOM, 2011; Altman et al., 2016; National Academies of Sciences Engineering and Medicine, 2021). The reasons are that BSN prepared nurses are better equipped to handle the increasing complex demands of providing care for our aging populations as BSN nurses have more education in leadership, systems thinking, research, teamwork and collaboration, and health policy (IOM, 2011). Overall strategies to increase educational levels for RNs are warranted and especially in rural areas where the level is significant lower. Several studies have pointed to reasons for this gap and potential solutions. First, rural nurses earn less that their urban counterparts so pursuing further education may be difficult and the subsequent salary increase for more education not worth it (Duffy et al., 2014; Girard et al., 2017). Increasing salary differential and hiring practices such that organizations have BSN as there preferred or required education entry level are predictors of nurses’ willingness to return to school (Warren & Mills, 2009). Second, there are few programs in rural hospitals that provide financial and/or time support for RNs to pursue more education (Milone-Nuzzo, 2015). Organizational incentives such as tuition assistance (reimbursement, forgivable loans for service, paid sabbatical and or educational days) and flexible scheduling are additional predictors of likelihood of nurses returning to school (Warren Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 165 & Mills, 2009). Finally, the lower adoption of broadband in rural areas remains a major problem for rural nurses to access to online learning (Perrin, 2019). The Federal Communications Commission in 2018, launched their 10-year phase 2 plan of the Connect America Fund to expand broadband in the US which should aid in increasing rural communities’ access to broadband (Federal Communications Commission, n.d.). Our study found there were no differences in the six patient outcomes between CAHs and other small rural and urban hospitals. For the 2 indicators for nurse outcomes, we found no differences, and only the aforementioned difference in educational levels between urban small hospitals and CAHs for the eight work environment indicators. Limitations The study has several limitations. First, we used a convenience sample of CAHs and other small rural and urban hospitals from one database. It is possible that these hospitals, including the 103 CAHs are different than other hospitals. Second, while NDNQI® data are of very high quality with standardized and robust processes for data collection (guidelines and training), it is possible that data entry was imperfect. However, a strength is that since NDNQI® data are unrelated to any required reporting data or for reimbursement purposes, the data are less likely to have similar risk of bias as other administrative databases (Waters et al., 2015). Third, we are reporting hospital-level data. This is an advantage for CAHs because they often do not distinguish between units (and acuity) (Cramer et al., 2011), compared to the larger hospitals with 25-50 beds that might have more distinct units according to acuity. Finally, our data are not adjusted for patient characteristics, including acuity. Nevertheless, our findings support that there are existing quality indicators that works well in CAHs to assess quality and safety for benchmark purposes to guide both quality improvement and policy. Conclusion Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 166 Overall, our results suggest that quality indicators for structure and outcomes currently used in approximately 2000 hospitals are also applicable for CAHs. Therefore, administrators and policy makers can consider using NDNQI® indicators for CAH benchmarking. As with any quality indicator, before implementation one has to consider whether the act of measuring increases documentation burden, the utility of the indicator for organizations (e.g. guiding QI activities), clinicians (e.g. informing practice) and patients and families (e.g. aid in deciding where to go for care) (Baernholdt, Dunton, et al., 2018). Future studies should address these issues and whether these indicators apply to all 1300 or so CAHs. References Agency for Health Research and Quality. (2015.). Types of health care quality measures. https://www.ahrq.gov/talkingquality/measures/types.html Aiken, L. H., Sloane, D., Griffiths, P., Rafferty, A. M., Bruyneel, L., McHugh, M., Maier, C. B., Moreno-Casbas, T., Ball, J. E., Ausserhofer, D., Sermeus, W. for the RN4CAST Consortium(2017). Nursing skill mix in European hospitals: Cross-sectional study of the association with mortality, patient ratings, and quality of care. BMJ Quality and Safety, 26(7), https://doi.org/10.1136/bmjqs-2016-005567 Altman, S. H., Butler, A. S., Shern, L., Committee for Assessing Progress on Implementing the Recommendations of the Institute of Medicine Report. The Future of Nursing: Leading Change, Advancing Health, Institute of Medicine, & National Academies of Sciences, Engineering, and Medicine (Eds.). (2016). Assessing Progress on the Institute of Medicine Report The Future of Nursing. National Academies Press (US). https://pubmed.ncbi.nlm.nih.gov/27010049/ American Hospital Association. (n.d.). Fast facts on U.S. hospitals, 2021. https://www.aha.org/system/files/media/file/2021/01/Fast-Facts-2021-table-FY19-data- 14jan21.pdf Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 167 American Nurses Credentialing Center. (n.d.). ANCC magnet recognition program. https://www.nursingworld.org/organizational-programs/magnet/ Baernholdt, M., Dunton, N., Hughes, R. G., Stone, P. W., & White, K. M. (2018). Quality measures: A stakeholder analysis. Journal of Nursing Care Quality, 33(2), 149-156, https://doi.org/10.1097/NCQ.0000000000000292 Baernholdt, M., Hinton, I. D., Yan, G., Xin, W., Cramer, E., & Dunton, N. (2017). A national comparison of rural/urban pressure ulcer and fall rates. Online Journal of Issues in Nursing, 22(2). https://doi.org/10.3912/OJIN.Vol22No02PPT60 Baernholdt, M., Hinton, I. D., Yan, G., Xin, W., Cramer, E., & Dunton, N. (2018). Fall rates in urban and rural nursing units. Journal of Nursing Care Quality, 33(4), 326–333. https://doi.org/10.1097/NCQ.0000000000000319 Baernholdt, M., Kiem-Malpass, J., Hinton, I. D., Yan, G., & Bratt, M. (2014). A comparison of quality of care in critical access hospitals and other small rural hospitals. Online Journal of Rural Nursing and Health Care, 14(2), 3–31. http://dx.doi.org/10.14574/ojrnhc.v14i2.328 Baernholdt, M., Yan, G., Hinton, I. D., Cramer, E., & Dunton, N. (2020). Effect of preventive care interventions on pressure ulcer rates in a national sample of rural and urban nursing units: Longitudinal associations over 4 years. International Journal of Nursing Studies, 105, Article 103455. https://doi.org/10.1016/j.ijnurstu.2019.103455 Beck, S. L., Dunton, N., Berry, P. H., Brant, J. M., Guo, J. W., Potter, C., Spornitz, B., Eaton, J., & Wong, B. (2019). Dissemination and implementation of patient-centered indicators of pain care quality and outcomes. Medical Care, 57(2), 159–166. https://doi.org/10.1097/MLR.0000000000001042 Bergquist-Beringer, S., Gajewski, B., Dunton, N., & Klaus, S. (2011). The reliability of the National Database of Nursing Quality Indicators pressure ulcer indicator. Journal of Nursing Care Quality, 26(4), 292-301. https://doi.org/10.1097/NCQ.0b013e3182169452 Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 168 Campaign for Action. (n.d.). Campaign progress, https://campaignforaction.org/wp- content/uploads/2020/09/dashboard-indicator-updates-Winter2021.pdf Casey, M., Distel, E., Evenson, A., Prasad, S., & Hung, P. (2015). Reporting of healthcare- associated infections by critical access hospitals [Policy Brief #39]. https://www.flexmonitoring.org/sites/flexmonitoring.umn.edu/files/media/hai-reporting.pdf Casey, M. M., Moscovice, I., Klingner, J., & Prasad, S. (2013). Rural relevant quality measures for critical access hospitals. The Journal of Rural Health, 29(2), 159–171. https://doi.org/10.1111/j.1748-0361.2012.00420.x Casey, M. M., Moscovice, I., Klingner J., & Prasad, S. (2012). Relevant quality measures for critical access hospitals [Policy Brief #25] https://www.flexmonitoring.org/publication/relevant-quality-measures-cahs-policy-brief-25 Centers for Medicare & Medicare Services (2021). HCAHPS: Patients' Perspectives of Care Survey, https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment- Instruments/HospitalQualityInits/HospitalHCAHPS#:~:text=The%20HCAHPS%20(Hospital %20Consumer%20Assessment,patients'%20perspectives%20of%20hospital%20care Choi, J., Boyle, D. K., & Dunton, N. (2014). A standardized measure. Western Journal of Nursing Research, 36(1), 105–116. https://doi.org/10.1177/0193945913501723 Choi, J. S., & Staggs, V. S. (2014). Comparability of nurse staffing measures in examining the relationship between RN staffing and unit-acquired pressure ulcers: A unit-level descriptive, correlational study. International Journal of Nursing Studies, 51(10), 1344–1352. https://doi.org/10.1016/j.ijnurstu.2014.02.011 Cramer, M. E., Jones, K. J., & Hertzog, M. (2011). Nurse staffing in critical access hospitals: Structural factors linked to quality care. Journal of Nursing Care Quality, 26(4), 335–343. https://doi.org/10.1097/NCQ.0b013e318210d30a Duffy, M. T., Friesen, M. A., Speroni, K. G., Swengros, D., Shanks, L. A., Waiter, P. A., & Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 169 Sheridan, M. J. (2014). BSN completion barriers, challenges, incentives, and strategies. Journal of Nursing Administration, 44(4), 232–236. https://doi.org/10.1097/NNA.0000000000000054 Duncan, J., Montalvo, I., & Dunton, N. (2011). NDNQI case studies in nursing quality improvement (1st ed.). American Nurses Assocation. Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap (1st ed.). Chapman & Hall. Federal Communications Commission. (n.d.). Connect America Fund Phase II Auction (Auction 903). https://www.fcc.gov/auction/903 Girard, S. A., Hoeksel, R., Vandermause, R., & Eddy, L. (2017). Experiences of RNs who voluntarily withdraw from their rn-to-bsn program. Journal of Nursing Education, 56(5), 260– 265. https://doi.org/10.3928/01484834-20170421-02 Institute of Medicine (US) Committee on Quality of Health Care in America.. (2001). Crossing the quality chasm: A new health system for the 21st century. National Academies Press. Institute of Medicine. (2011). The future of nursing: Leading change, advancing health. National Academies Press. Jones, T. L., Yoder, L. H., & Baernholdt, M. (2019). Variation in academic preparation and progression of nurses across the continuum of care. Nursing Outlook, 67(4), P381-392. https://doi.org/10.1016/j.outlook.2019.02.008 Klaus, S. F., Dunton, N., Gajewski, B., & Potter, C. (2013). Reliability of the nursing care hour measure: A descriptive study. International Journal of Nursing Studies, 50(7), 924–932. https://doi.org/10.1016/j.ijnurstu.2012.07.012 Lahr, M., Chantarat, T., & Moscovice, I. (2020). Hospital compare quality measure results for CAHs, 2018. https://www.flexmonitoring.org/sites/flexmonitoring.umn.edu/files/media/fmt- hospital-compare-national-2018_0.pdf Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 170 Lahr, M., Pick, M., Chantarat, T., Bean, N., & Moscovice Ira. (2021). MBQIP quality measures national annual report-2019. https://www.flexmonitoring.org/publication/mbqip-quality- measures-national-annual-report-2019 Milone-Nuzzo, P. (2015). Improving rural health through support for academic progression. Journal of Nursing Education, 54(10), 539–540. https://doi.org/10.3928/01484834-20150916- 10 Mitchell, P. H., & Lang, N. M. (2004). Framing the problem of measuring and improving healthcare quality: has the Quality Health Outcomes Model been useful? Medical Care, 42(2), II-4-II-11. https://doi.org/10.1097/01.mlr.0000109122.92479.fe Montalvo, I. (2007). The National Database of Nursing Quality Indicators® (NDNQI®). The Online Journal of Issues in Nursing. https://doi.org/10.3912/OJIN.Vol12No03Man02 National Academies of Sciences Engineering and Medicine. (2021). The future of nursing 2020-2030: Charting a path to achieve health equity. The National Academies Press. https://www.nap.edu/download/25982# National Quality Forum. (2018). A core set of rural-relevant measures and measuring and improving access to care: 2018 recommendations from the MAP Rural Health Workgroup. http://www.qualityforum.org/Publications/2018/08/MAP_Rural_Health_Final_Report_- _2018.aspx What is pay for performance in healthcare? New England Journal of Medicine Catalyst (2018, March 1.). https://catalyst.nejm.org/doi/full/10.1056/CAT.18.0245 Odahowski, C. L., Crouch, E. L., Zahnd, W. E., Probst, J. C., McKinney, S. H., & Abshire, D. A. (2021). Rural-urban differences in educational attainment among registered nurses: Implications for achieving an 80% BSN workforce. Journal of Professional Nursing, 37(2), 404–410. https://doi.org/10.1016/j.profnurs.2020.04.008 Office of Management and Budget. (2010). 2010 Standards for delineating metropolitan and Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 171 micropolitan statistical areas. Federal Register. https://www.federalregister.gov/documents/2010/06/28/2010-15605/2010-standards-for- delineating-metropolitan-and-micropolitan-statistical-areas Park, S. H., Boyle, D. K., Bergquist-Beringer, S., Staggs, V. S., & Dunton, N. E. (2014). Concurrent and lagged effects of registered nurse turnover and staffing on unit-acquired pressure ulcers. Health Services Research, 49(4), 1205–1225. https://doi.org/10.1111/1475- 6773.12158 Perrin, A. (2019). Digital gap between rural and nonrural America persists. Pew Research Center. https://www.benton.org/headlines/digital-gap-between-rural-and-nonrural-america- persists-0 Press Ganey. (2014). Press Ganey acquires National Database of Nursing Quality Indicators (NDNQI®). https://www.prnewswire.com/news-releases/press-ganey-acquires-national- database-of-nursing-quality-indicators-ndnqi-262538811.html Probst, J., Eberth, J. M., & Crouch, E. (2019). Structural urbanism contributes to poorer health outcomes for rural America. Health Affairs, 38(12), 1976–1984. https://doi.org/10.1377/hlthaff.2019.00914 Quick, M., Chantarat, T., Norris, A., & Moscovice, I. (2019). Patients’ experiences in CAHs: HCAHPS Results, 2017 [Data Summary Report #27]. https://www.flexmonitoring.org/publication/patients-experiences-cahs-hcahps-results-2017- fmt-data-summary-report-27 Swenson, T., & Casey, M. (2016). MBQIP quality measure trends, 2011-2016.. https://www.flexmonitoring.org/sites/flexmonitoring.umn.edu/files/media/dsr20.pdf Warren, J. I., & Mills, M. E. (2009). Motivating registered nurses to return for an advanced degree. Journal of Continuing Education in Nursing, 40(5), 200–207. https://doi.org/10.3928/00220124-20090422-01 Online Journal of Rural Nursing and Health Care, 2023(1) https://doi.org/10.14574/ojrnhc.v23i1.733 172 Waters, T. M., Daniels, M. J., Bazzoli, G. J., Perencevich, E., Dunton, N., Staggs, V. S., Potter, C., Fareed, N., Liu, M., & Shorr, R. I. (2015). Effect of medicare’s nonpayment for hospital- acquired conditions: Lessons for future policy. JAMA Internal Medicine, 175(3), 347–354. https://doi.org/10.1001/jamainternmed.2014.5486 Waugh, S. M., & Bergquist-Beringer, S. (2016). Inter-rater agreement of pressure ulcer risk and prevention measures in the national database of nursing quality indicators ® (NDNQI). Research in Nursing & Health, 39(3), 164–174. https://doi.org/10.1002/nur.21717