Relationship between compliance to quality indicators, volume, and outcomes in pancreatic cancer resection in a community hospital setting Megan M. Brown a, Roberta E. Redfern a , and Michael D. McPhee b1 aProMedica Research, ProMedica Toledo Hospital, Toledo, OH 43606, and bSurgical Oncology, ProMedica Flower Hospital, Sylvania OH 43560 Objective: To calculate and compare adherence to pancreatic can- cer care quality indicators and related outcomes at three hospitals within our health system and determine their relatedness. Design: Retrospective review of cancer registry data and patient charts from January 1, 2000 - December 31, 2010. Setting: Three hospitals performing pancreatic resections within a single health system. Patients: All patients undergoing resection for the treatment of pancreatic cancer. Main Outcome Measures: Adherence to published quality indi- cators, patient survival, and surgical and institutional variables related to mortality. Results: Complication, morbidity and mortality, and overall sur- vival rates were similar between facilities and were comparable to previously reported values of similar volume facilities. Overall adherence to quality indicators was approximately 73%; compli- ance to indicators in the “outcome” and “efficiency” domains was highest, but adherence was not associated with clinical outcomes. Patient mortality was lowest at the highest volume hospital in our health system; however, surgeon volume was not predictive of outcomes. Conclusion: Use of quality indicators is valuable in determining quality of care and identifying weaknesses, allowing for a targeted approach to improve quality of care and the associated processes. However, adherence to these quality indicators is not necessarily indicative of outcomes. hospitals, community | quality indicators, health care | pancreatic neoplasm Pancreatic cancer is one of the leading causes of death from can-cer in the United States. Unfortunately, the prognosis for pa- tients with pancreatic cancer is typically very poor. Resection of the tumor is the only possible curative treatment, though many patients present with advanced disease which does not allow for surgical inter- vention. While the overall 5 year survival of patients with pancreatic cancer is less than 5%, outcomes are variable and based on a number of factors (1-3). In the climate of pay-for-performance, it becomes increasingly important to identify the variables which contribute to outcomes and can be controlled (4). Current literature suggests that outcomes are dependent on the volume of surgeries performed at the center, and that the volume of resections performed by the particular surgeon may have an impact on patient mortality (5). High volume centers report more favorable outcomes in many procedures, and the variability of care and out- comes has consistently been reported as being greatest in pancreatic cancer (5-8). Though high volume centers report the best outcomes, the ma- jority of pancreatic resections are performed in small, low-volume community hospital settings (6,9). Regionalization of care for pan- creatic cancer has been proposed, though critics assert that the incon- venience and cost to patients, among other factors, make this strategy unfeasible (10). As such, it is necessary to standardize practices and care in centers treating different volumes of patients, the overall goal being better and more consistent outcomes between low, moderate, and high volume centers. In order to standardize care, quality in- dicators have been developed which take into account many factors which may influence outcomes on different levels (11). Evaluation of an institution’s adherence to these indicators may help to identify areas of weakness and reduce variability in quality of care (12-15). Methods The ProMedica Institutional Review Board provided a letter of exemption for this evaluation. The electronic databases within the health system and the cancer registry were queried for all patients undergoing pancreatic cancer resection from January 1, 2000 to De- cember 31, 2010. In total, 91 patient charts from three hospitals in the system were identified and reviewed. Variables collected in- cluded patient demographics (age, gender, comorbid conditions), tu- mor characteristics (location, stage, histology, grade, size), procedu- ral information (date of surgery, surgeon, procedure type, operative time, operative blood loss, lymph nodes examined, margin status), hospital course details (length of stay, consults received, complica- tions), as well as patient outcomes, including readmissions which may have occurred as a result of their surgical care. In order to con- struct a survival curve, date of death was also collected using the United States Death Index. To assess compliance, quality indicators applying to surgical treatment of pancreatic cancer patients were identified from those put forth by Bilimoria et al in 2009 (11). In total, 37 of the 43 high-to- moderate quality indicators applied to pancreatic surgical treatment. Each quality indicator belongs to one of five domains which include structure, process, appropriateness, efficiency, and outcome. Compli- ance with these quality indicators is considered to be reached if the institution is concordant with the quality indicator in at least 90% of the patients undergoing resection; these overall rates of compliance are reported for each facility (11). In addition, each facility’s compli- ance to individual quality indicators was calculated to evaluate differ- ences between each hospital performing pancreatic resections within the health system; these are presented as percent of cases considered compliant to allow for comparisons between each facility. 1To whom correspondence should be sent: Michael.mcpheemd@promedica.org Author contributions: MB, RER, and MDM designed the research protocol; MB collected study data; MB and RER analyzed study data; MDM supervised data analysis; MB, RER, and MDM contributed to drafting of the manuscript and MDM takes responsibility for the paper as a whole. The authors declare no conflict of interest Freely available online through the UTJMS open access option utdr.utoledo.edu/translation/ UTJMS 2015 Vol. 2 3–7 Structure. To evaluate structural compliance, institution volume and surgeon volume were calculated for each year. Patient charts were re- viewed to ensure that all patients were treated in a multidisciplinary effort with a surgeon, medical oncologist, and a radiation oncolo- gist. Hospital services were also evaluated to ensure that the insti- tution was able to provide interventional radiation services, inten- sive care services, endoscopic ultrasonography services, radiation and chemotherapy services, as well as endoscopic retrograde cholan- giopancreatography services. All surgeons underwent verification of certification by the American Board of Surgery or equivalent interna- tional institution. Each facility was also evaluated on their participa- tion in clinical trials. Process. In order to evaluate process compliance, patient charts were reviewed for proper treatment protocol, including a history and phys- ical with thorough preoperative risk assessment, a triple-phase, multi- slice CT or MRI, suspicious adenopathy evaluated by frozen section, use of the College of American Pathologists checklist, macroscopi- cally clear margins, and proper documentation of disease and treat- ment in the patient chart. Appropriateness.Treatment appropriateness compliance was as- sessed by reviewing all charts to ensure that following resection, all patients underwent adjuvant chemotherapy or chemoradiation; if they did not, a reason must be documented. It was also important to ensure that, of the patients resected, none had documented stage IV clinical disease. Efficiency. Elapsed time from diagnosis to treatment was calculated to ensure that not more than two months had passed in order to eval- uate efficiency compliance; time to diagnosis was abstracted from the cancer registry and is defined by the Commission on Cancer as the “first date diagnosis was clinically or histologically established”. Additionally, patient charts were reviewed for operative times and readmissions within 30 days of their surgical procedures. Outcome. Institution margin-negative resection rate, 30-, 60-, and 90-day perioperative mortality, as well as 2-year and 5-year survival rates were calculated to objectively evaluate surgical outcomes. Statistical Methods Survival curves were constructed using the Kaplan Meier method. Student t-tests and analysis of variance (ANOVA) tests were used for mean comparison, while Fisher’s exact tests were used for comparing proportions. P-values less than 0.05 were considered sta- tistically significant. All analyses were completed using R version 2.15.0 (16). Results Patient Characteristics. 91 patients underwent pancreatic resection at one of the three hospitals in our health system offering this treat- ment between the years of 2000 and 2010. The average age at diag- nosis of all patients was 66.6±11.5 years, 52.7% were female. There were no statistically significant differences between the cohorts of patients treated at each of the separate hospitals with regard to age, gender, number of coexisting comorbidities, or tumor characteristics. Demographic and baseline patient characteristics are reported in Ta- ble 1. Adherence. The quality indicators evaluated are stratified into five separate domains: structure, process, appropriateness, efficiency, and outcome. Evaluation of the structure-related quality indicators re- veals that each of our included facilities are adherent to 8 of the 11 (72.7%) defined indicators. Because there is no variability in overall adherence to the majority of domains, the percent of cases in which provided care was concordant with defined indicators is reported in Table 2. Compliance to individual indicators ranges from 0% to 100%; all patients were treated in concordance with all indicators for which the facilities were compliant. Surgical volume and lack of endoscopic ultrasonography services on-site accounted for 2 of the 3 structure-related indicators to which the facility did not adhere. Table 1: Patient demographics and baseline disease characteristics. Patient demographics n=91 Age at diagnosis±SD 66.6±11.5 Women, n (%) 48 (52.7) Comorbidities No comorbid conditions, n (%) 26 (28.6) One comorbid condition, n (%) 24 (26.4) Two comorbid conditions, n (%) 16 (17.6) Three comorbid conditions, n (%) 25 (27.5) Pathology Location of Tumor n (%) Body 3 (3.3) Head 72 (79.1) Tail 12 (13.2) Lesion 3 (3.3) NOS 1 (1.1) AJCC Stage n (%) Stage 0 3 (3.3) Stage I 11 (12.1) Stage III 17 (18.7) Stage IV 6 (6.6) Unknown 4 (4.4) Histology n (%) Adenocarcinoma 32 (35.2) Adenosquamous carcinoma 1 ( 1.1) Duct cell carcinoma 44 (48.4) Leimyosarcoma 1 (1.1) Mucinous carcinoma 3 (3.3) Other 10 (11) Grade n (%) Well-differentiated 15 (16.5) Moderately differentiated 38 (41.8) Poorly differentiated 25 ( 27.5) Unknown 7 (7.7) Table 2: Rates of adherence to quality indicators by domain and over- all survival. Domain Hospital Adherence (%) 1 2 3 Structure 80.5 80.0 78.2 Process 87.0 83.8 91.8 Appropriateness 84.0 80.9 80.0 Efficiency 98.9 98.5 97.5 Outcome 100 100 100 Overall 90.1 88.6 89.5 Survival % 30-day 100 85.3 70 60-day 97.9 79.4 70 2-year 45.3 20.6 20 5-year 22.1 14.7 0 Three of the process-related indicators were excluded for evalu- ation due to unavailability of data or inapplicability of the defined 4 utdr.utoledo.edu/translation/ Brown et al. Table 3: Comparison of survival in pancreatic cancer patients in high and low volume mortality rates(6) to those within our health system. Perioperative Mortalitya 5-yr Overall Survival 5-yr Conditional Survivalb High Low observed High Low observed High Low observed 4.9 10.5 13.2 15.4 12.5 16.5 16.2 14.0 17.7 a 60-day unadjusted perioperative mortality rate b 5 year survival for patients surviving the perioperative period (excludes in-hospital deaths). indicator, as we included only patients who underwent pancreatic resection. Evaluation of documentation within patient charts re- vealed that our institution’s care was concordant with 11 (64.7%), 10 (58.8%), and 14 (82.4%) of the 17 defined indicators in hospitals 1, 2, and 3, respectively. This was the only domain to exhibit variability in overall adherence between facilities. Compliance with process- related indicators ranged from 20.9% to 100% in individual cases; documentation errors and omissions were the primary cause of non- adherence to indicators in this category. Similarly, evaluation of the concordance with indicators in the “appropriateness” domain demon- strates adherence to 1 of the 2 (50%) indicators assessed in each facil- ity; non-adherence was related to documentation errors or omissions. Finally, assessment of “efficiency” and “outcome” domains indicated adherence to all 7 relevant quality indicators in each facility; concor- dance with prescribed care ranged from 95.6% to 100%. Outcomes. Overall 60 day perioperative mortality rate for the en- tire study period was 13.2%, which is comparable to other low-to- moderate volume facilities. Additionally, 5-year overall survival and 5-year conditional survival were comparable to those rates reported for high volume centers (Table 3) (6). Observed annual survival rates as a function of stage of disease were compared to data abstracted from the American College of Surgeons Commission on Cancer; the survival rates of our facilities were comparable to the national data for the study period (Table 4). Furthermore, the complication rate and number of readmissions within 30 days were recorded. The over- all complication rate for all three facilities was 45.1%, ranging from 36.2% to 70.0%, while the 30-day readmission rate was 24.2% over- all. The number of resections occurring each year varied; as few as 5 were performed one year, while 12 were performed the next. Fur- thermore, the 91 resections occurred in 3 separate hospitals; over the study period the three hospitals performed 47, 34, and 10 resections, respectively. Mortality rates varied significantly between the three fa- cilities (Figure 1); however, we found no statistically significant dif- ferences in the number of readmissions or complication rates between facilities. Furthermore, no significant differences were uncovered in Fig. 1: Kaplan-Meier survival curve as a function of facility. the rate of compliance between the three facilities, despite the fact that the volumes of surgeries performed at each hospital were signif- icantly different from one another. While 30-, 60-, and 90-day sur- vival rates were positively associated with increased surgical volume in our three facilities (p = 0.001, p= 0.004, and p= 0.002, respectively, Table 5), these rates were not associated with overall compliance to the quality indicators evaluated. Over the ten year span of data col- lected, no temporal trends were identified with respect to adherence or outcomes In order to determine whether any variables might be associated with 30- and 60- day mortality rates, student t-tests and chi square analyses were completed. The number of pre-existing comorbidities was not associated with 30-day mortality rates (p=0.12), but was sig- nificantly associated with 60-day mortality rates (p=0.007). Further- more, blood loss and surgeon volume were not associated with 30- or 60-day mortality rates (Table 6). However, operative time was signif- icantly associated with 30- and 60-day mortality (Table 6). Surgeries were approximately 100 minutes longer in those patients who were alive after 60 days when compared to those who had passed during that time, suggesting that operative time may be related to 30- and 60-day outcomes. Table 4: Observed (obs) survival rates in our health system compared to national (natl) rates, as reported by the American College of Surgeons Commission on Cancer, during the same time period. Time Stage 0a Stage 1 Stage 2 Stage 3 Stage 4 Overall obs natl obs natl obs natl obs natl obs natl obs natl 1 year 66.7 72.8 73.0 49.5 53.6b 46.2 47.1 42.1 50.0 15.8 55.9b 28.2 2 years 66.7 65.2 55.0 30.9 29.7c 23.7 17.7 17.5 33.0 5.9 32.8c 13.2 3 years 66.7d 61.8 36.0d 23.9 18.9d 15.7 5.9 10.1 0.0 3.9 20.2d 8.9 4 years 66.7 57.9 24.0 20.4 18.9 12.0 5.9 7.4 0.0 3.1 16.5 7.1 5 years 66.7 55.5 12.0 18.6 18.9 10.2 5.9 6.0 0.0 2.8 16.5 6.2 a all values expressed as a percentage b at least one patient censored due to diagnosis date after 10/11/2010 c at least one patient censored due to surgery date after 10/11/20 d at least one patient censored due to surgery date after 10/11/2008 Brown et al. UTJMS 2015 Vol. 2 5 Discussion Our data indicate that our institution adheres well to validated quality indicators for pancreatic cancer surgical care. Of interest, while all three facilities examined adhered very similarly to all pre- scribed quality indicators, the outcomes at each facility were signifi- cantly different from one another. Each facility investigated had very different surgical volumes, such that 30-, 60-, and 90-day mortal- ity rates at our higher volume facilities were significantly better than those at our lower volume facilities. However, all facilities in our health system are considered low to low-moderate volume hospitals in pancreatic resection. Importantly, outcomes at all facilities are on par with reported outcomes at moderate volume hospitals, and many of our reported outcomes meet benchmarks for high volume centers (7). Based on our analysis, adherence to quality indicators is not nec- essarily predictive of outcomes, as our adherence rates did not vary between facilities while the outcomes did significantly. In fact, the hospital with highest overall level of adherence (30 of 37 indicators, 81.1%) exhibited the lowest survival rates. Though these quality in- dicators may help improve processes and quality of care provided, it cannot be expected that adhering perfectly to all indicators will sig- nificantly improve outcomes, in part due to the nature of pancreatic cancer in which cure is quite rare and survival rates are expected to be low. Because our adherence to quality indicators was not strongly as- sociated with outcomes at each facility, other factors that may be pre- dictive of survival were investigated (17-21); surgeon volume, num- ber of pre-existing comorbidities, intraoperative blood loss, and even stage were not predictive of 30-day mortality. However, we did un- cover an interesting trend; operative time was associated with mor- tality, such that patients who died within 90 days postoperatively had significantly shorter operative times than those patients who survived this postoperative period. While our data support previous claims that higher volume sur- gical centers boast better outcomes, we found no evidence to support the hypothesis that surgeons who perform a larger volume of pancre- atic surgeries in our institution can necessarily do the same. When compared to all other surgeons, the surgeon with the largest volume and lowest mean operative times actually had worse outcomes than all other surgeons combined; alternately, the second highest volume surgeon had better outcomes than the collective. This suggests that not only surgeon volume, but other surgical characteristics may be predictive of survival rates. There are a number of lessons which can be learned from our as- sessment going forward. First, we learned that documentation omis- sions and errors are more common than we believed initially, and can be more detrimental than assumed. Efforts to correct this on all levels is being pursued, but it is important to realize that until such an audit is completed, the extent to which documentatin errors occur is likely grossly underestimated. As such, the authors recommend that can- cer centers providing pancreatic cancer care engage in full evaluation of their processes and care provided using the validated indicators published by Bilimoria et al., as described herein (11). Additionally, it must be recognized that outcomes may be due to factors outside of the process of care offered on an institutional level. Surgeon characteristics, particularly operative times, may be sugges- tive of mortality. Our investigation also suggested the possibility that pre-operative assessment and patient screening/selection may have a significant impact on outcomes. It is possible that not screening patients rigorously enough could contribute to variable outcomes in spite of adhering consistently to a number of care-related quality in- dicators. Importantly, during our audit, we recognized that 6 of our pancreatic resections were documented to have occurred in stage IV tumors according to AJCC 6th edition guidelines (22). One of these cases was a clear documentation error, two involved metastases to the spleen and would be considered stage III disease according to current AJCC 7th edition staging guideline (23). Of the remaining three cases in question, one was an emergent procedure, one was an endocrine neoplasm, and the final case involved metastases that were thought to be benign prior to surgery but were determined to be malignant postoperatively. In order to address our institution’s short- comings, we have developed an Oncology Grand Rounds program, with invited pancreatic cancer expert presenters, to better educate our clinicians. Furthermore, to address documentation errors and omis- sions, a preoperative AJCC template has been added to the operative notes, as a reminder to fully document disease staging. Finally, 36% of our patients did not receive adjuvant chemoradiation therapy, with no documented reason. In order to correct for the deficient num- ber of patients receiving postoperative adjuvant chemoradiation ther- apy, our institution recognizes and is considering adopting the cur- rent paradigm shift to neoadjuvant chemoradiation therapy in patients with resectable pancreatic cancer. Evidence suggests that chemora- diation prior to surgery gives better chances of achieving negative margins; more importantly, restaging after this type of treatment will allow physicians to identify patients whose disease has progressed, making surgical resection ineffectual (24). The cost effectiveness of this strategy cannot be understated. Study Limitations The retrospective nature of this evaluation is an inherent weak- ness, as non-adherence rates may be overinflated due to inability to locate documentation; our analysis is also limited by the small sam- ple size available. As such, though we included ten years of data, logistic regression was not possible to investigate some variables as small subgroups lead to unstable statistical estimates. Due to the small sample size available for analysis, our findings may not be gen- eralizable to other settings. Additionally, it may also be due to this small sample size that we could not discern significant differences in the investigation of some variables, owing to type 2 error. Further investigation of how these variables relate to outcomes and quality indicators is warranted. Furthermore, we found that in order to ap- propriately compare all mortality data to previously reported data in the literature, investigators should calculate risk-adjusted mortality. However, this requires detailed data, which were not available in all of our patient charts. We suggest that moving forward, even com- munity hospitals such as ours begin prospectively collecting all data necessary for risk-adjusted mortality calculations. To this end, the decision has been made within our institution to begin participation in the American College of Surgeons’ National Surgical Quality Im- provement Program (NSQIP). Participation on a community level is recommended, as most pancreatic surgeries occur in this setting. Our data indicate that adherence to validated quality indicators in our institution was not directly associated with survival rates, which may call in to question the clinical value of those quality indicators. Though our results do not support clinical integration of all of these indicators, we advocate using these indicators to assess the process of care provided as well as the quality of documentation, as the infor- mation gained by our cancer center during this assessment was found to be of great value. This type of evaluation allows identification of areas of weakness which might not be realized without careful exami- nation, allowing any shortfalls to be addressed appropriately. Further study of the clinical relevance of pancreatic cancer quality indicators is warranted. ACKNOWLEDGMENTS. The authors would like to thank Stephen Wanjiku, MS for assistance in data analysis.- 6 utdr.utoledo.edu/translation/ Brown et al. 1. Langer B (2007) Role of volume outcome data in assuring quality in HPB surgery. HPB (Oxford) 9(5):330-334. 2. Birkmeyer JD, Sun Y, Goldfaden A, Birkmeyer NJ, Stukel TA (2006)Volume and pro- cess of care in high-risk cancer surgery. Cancer 106(11):2476-2481. 3. Riall TS, Nealon WH, Goodwin JS, Townsend CM, Jr., Freeman JL (2008) Outcomes following pancreatic resection: variability among high-volume providers. Surgery 144(2):133-140. 4. Glickman SW, et al. (2009) Alternative pay-for-performance scoring methods: impli- cations for quality improvement and patient outcomes. Med Care 47(10):1062-1068. 5. Halm EA, Lee C, Chassin MR (2001) Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med 137(6):511-520. 6. Bilimoria KY, et al. (2008) Effect of hospital type and volume on lymph node evalu- ation for gastric and pancreatic cancer. Arch Surg 143(7):671-678; discussion 678. 7. Bilimoria KY, et al. (2008) Directing surgical quality improvement initiatives: com- parison of perioperative mortality and long-term survival for cancer surgery. J Clin Oncol 26(28):4626-4633. 8. Finlayson EV, Goodney PP, Birkmeyer JD (2003) Hospital volume and operative mortality in cancer surgery: a national study. Arch Surg 138(7):721-725; discus- sion 726. 9. Birkmeyer JD (2000) High-risk surgery–follow the crowd. JAMA 283(9):1191-1193. 10. Raval MV, Bilimoria KY, Talamonti MS (2010) Quality improvement for pancreatic cancer care: is regionalization a feasible and effective mechanism? Surg Oncol Clin N Am 19(2):371-390. 11. Bilimoria KY, Bentrem DJ, Lillemoe KD, Talamonti MS, Ko CY (2009) Assessment of pancreatic cancer care in the United States based on formally developed quality indicators. J Natl Cancer Inst 101(12):848-859. 12. Gisvold SE, Fasting S (2011) How do we know that we are doing a good job - can we measure the quality of our work? Best Pract Res Clin Anaesthesiol 25(2):109-122. 13. Bilimoria KY, et al. (2001) National assessment of melanoma care using formally developed quality indicators. J Clin Oncol 27(32):5445-5451. 14. Maggard MA, McGory ML, Shekelle PG, Ko CY (2006) Quality indicators in bariatric surgery: improving quality of care. Surg Obes Relat Dis 2(4):423-429; discussion 429-430. 15. McGory ML, Shekelle PG, Ko CY (2006) Development of quality indicators for pa- tients undergoing colorectal cancer surgery. J Natl Cancer Inst 98(22):1623-1633. 16. R Core Team (2012) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 17. Kazanjian KK, et al. (2008) Improved survival following pancreaticoduodenectomy to treat adenocarcinoma of the pancreas: the influence of operative blood loss. Arch Surg 143(12):1166-1171. 18. Schwarz RE, Keny H (2001) Preoperative platelet count predicts survival after resec- tion of periampullary adenocarcinoma. Hepatogastroenterology 48(41):1493-1498. 19. Gooiker GA, et al. (2011) Systematic review and meta-analysis of the volume- outcome relationship in pancreatic surgery. Br J Surg 98(4):485-494. 20. Eppsteiner RW, Csikesz NG, McPhee JT, Tseng JF, Shah SA (2008) Surgeon volume impacts hospital mortality for pancreatic resection. Ann Surg 249(4):635-640. 21. Sohn TA, et al. (2000) Resected adenocarcinoma of the pancreas-616 patients: re- sults, outcomes, and prognostic indicators. J Gastrointest Surg 4(6):567-579. 22. Greene F, et al., eds. AJCC Cancer Staging Handbook. 6th ed. Springer Publishing Company, 2002. 23. Edge B, Byrd D, Comptom C, Fritz A, Greene F, Trotti A, eds. AJCC Cancer Staging Manual. 7th ed. Springer Publishing Company, 2010. 24. Wolff RA, Varadhachary GR, Evans DB (2008) Adjuvant therapy for adenocarci- noma of the pancreas: analysis of reported trials and recommendations for future progress. Ann Surg Oncol 15(10):2773-2786. Brown et al. UTJMS 2015 Vol. 2 7