Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 1 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) Osaro Mgbere, PhD, MS, MPH 1, 2* and Salma Khuwaja, MD, MPH, Dr.PH 1 1 Disease Prevention and Control Division, Houston Health Department, Houston, Texas, USA 2 Institute of Community Health, University of Houston College of Pharmacy, Texas Medical Center, Houston, Texas, USA Abstract Background During the 2009 H1N1 influenza pandemic (pH1N1), the proportion of outpatient visits to emergency departments, clinics and hospitals became elevated especially during the early months of the pandemic due to surges in sick, ‘worried well’ or returning patients seeking care. We determined the prevalence of return visits to a multispecialty clinic during the 2009 H1N1 influenza pandemic and identify subgroups at risk for return visits using model-based recursive partitioning technique. Methods This study was a retrospective analysis of ILI-related medical care visits to multispecialty clinic in Houston, Texas obtained as part of the Houston Health Department Influenza Sentinel Surveillance Project (ISSP) during the 2009 H1N1 pandemic influenza (April 2009 – March 2010). The data comprised of 2680 individuals who made a total of 2960 clinic visits. Return visit was defined as any visit following the index visit after the wash-out phase prior to the study period. We applied nominal logistic regression and recursive partitioning models to determine the independent predictors and the response probabilities of return visits. The sensitivity and specificity of the outcomes probabilities were determined using receiver operating characteristic (ROC) curve. Results Overall, 4.56% (Prob. 0.0%-17.5%) of the cohort had return visits with significant variations observed attributed to age group (76.0%), type of vaccine received by patients (18.4%) and Influenza A (pH1N1) test result (5.6%). Patients in age group 0-4 years were 9 times (aOR: 8.77, 95%CI: 3.39-29.95, p<0.0001) more likely than those who were 50+ years to have return visits. Similarly, patients who received either seasonal flu (aOR: 1.59, 95% CI 1.01-2.50, p=0.047) or pH1N1 (aOR: 1.74, 95%CI: 1.09-2.75, p=0.022) vaccines were about twice more likely to have return visits compared to those with no vaccination history. Model-based recursive partitioning yielded 19 splits with patients in subgroup I (patients of age group 0-4 years, who tested positive for pH1N1, and received both seasonal flu and pH1N1 vaccines) having the highest risk of return visits (Prob.=17.5%). The area under the curve (AUC) for both return and non-return visits was 72.9%, indicating a fairly accurate classification of the two groups. Conclusions Return visits in our cohort were more prevalent among children and young adults, and those that received either seasonal flu or pH1N1 or both vaccines. Understanding the dynamics in care-seeking https://orcid.org/0000-0002-2863-6284 Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 2 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Introduction In the spring of 2009, a novel influenza A (H1N1) virus emerged and was detected first in the United States [1-3] and spread quickly across the United States and the world. From April 12, 2009 to April 10, 2010, CDC estimated there were 60.8 million cases, 274,304 hospitalizations, and 12,469 deaths in the United States due to the (H1N1) pdm09 virus [4,5]. Despite the enormous health consequences, the pandemic provided an important test of our nation’s preparedness activities and our ability to respond and adapt to a large-scale, protracted public health emergency [5]. During the 2009 H1N1 influenza pandemic (pH1N1), the proportion of outpatient visits to emergency departments (ED), clinics and hospitals became elevated around the United States especially during the early months of the pandemic due to surges in sick and ‘worried well’ individuals or returning patients seeking care. This occurred regardless of the availability of “H1N1 Flu Self-Evaluation,” designed to help individuals decide whether to seek medical care or stay home, if they had symptoms consistent with 2009 H1N1 [5]. Surges in patient volumes can compromise the healthcare systems’ ability to deliver care, as revealed by the 2009 H1N1 influenza pandemic [6]. To counter the impact, some healthcare providers and institutions developed novel approaches for emergency care delivery such as triage tents and drive-through examination areas [7,8], to help alleviate surge volumes and potentially prevent transmission of H1N1 influenza. Despite the surges experienced at some facilities, the 2009 H1N1 retrospective summary produced by the US Department of Health and Human Services (HHS) concluded that the ‘‘2009 H1N1 pandemic did not fully test the health care system’s ability to meet a surge in demand for care’’ [5]. The media reports are major sources of health information and they play key roles in health behavior change. Previous researches suggest that intense news media coverage of novel communicable diseases coupled with community attitudes can create public concern, and amplify risk perception [5,8-11]. These have been reported to increase the number of outpatient visits and return visits, influence physicians’ practices and behaviors (example, increase awareness and reporting of communicable diseases), and increase demand for clinical and diagnostic health services [5,11,12]. However, some factors have been identified as motivators for return visits including previous experience with ILI [13], personal beliefs about vaccination [14,15], and previous error in the diagnosis of an illness or the progression of an illness [16-18], and widespread behavior during pandemic would assist policymakers with appropriate resource allocation, and in the design of initiatives aimed at mitigating surges and recurrent utilization of the healthcare system. Keywords: Model-based recursive partitioning, decision tree, subgroup analysis, Influenza-like-illness, H1N1, influenza pandemic, care-seeking behavior, return visit * Correspondence: Osaro.Mgbere@Houstontx.gov DOI: 10.5210/ojphi.v12i1.10576 Copyright ©2020 the author(s) This is an Open Access article. Authors own copyright of their articles appearing in the Online Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in the copy and the copy is used for educational, not-for-profit purposes. mailto:Osaro.Mgbere@Houstontx.gov Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 3 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI report of morbidity and mortality during the pandemic [19]. In contrast, researchers have also noted that up to 73% of patients that showed up at ED for H1N1 influenza fear were the “worried well” and that over 95% of the presenting concerns were minor or nonexistent [20]. The Houston Health Department (HHD) uses several syndromic surveillance systems to monitor influenza-like-illness (ILI) activity in Houston, Texas [21] including the sentinel provider network, which was instrumental in monitoring the novel 2009 H1N1 pandemic in Houston [22]. Although the 2009 pH1N1 influenza has been cataloged by CDC with respect to the timing of the outbreak, geographic distribution, characteristics of cases, and epidemiologic parameters [23], and retrospective summary of events [5], much still remains to be learned from the pandemic, which continue to circulate seasonally in the U.S., and throughout the world [24]. In our most recent study, we explored the dynamics of care-seeking behaviors between the ILI phases (pre-pH1N1, pH1N1 and post-pH1N1) using facility-level data rather than patients’ self-reported survey data and found that 90% of the return visits to clinics occurred during the pH1N1 phase [25]. This finding prompted the need for further research to explore the demographic and clinical factors associated with return visits to clinics during the pH1N1. In recent years, considerable efforts have been dedicated to mathematical modeling studies to complement disease surveillance efforts in the planning of interventions against emerging pandemics including the 2009 H1N1 influenza pandemic [25-34]. The efficient prediction of the expected impact of an emerging pandemic would allow appropriate preparation to be made without diversion of excess resources, and thus, have the potential to reduce pandemic- and non-pandemic related illness and death [32]. Several studies have assessed return visits to emergency departments [16,35-38], but only one known study had assessed return visits for ILI-related conditions over the disease phases using facility-level data from a multispecialty clinic [25]. Illuminating the trend and risk factors for return visits during the pH1N1 may enable clinicians and public health authorities to identify individuals at highest risk for return visit, develop strategies for preventing it and assist policymakers with appropriate resource allocations during future pandemics. Therefore, the objectives of this study were to determine the prevalence of return visits, identify the associated risk factors and subgroups at high risk for return visits during the 2009 H1N1 influenza pandemic using a model-based recursive partitioning technique. Methods Data Source Data used for this study was obtained from the Influenza Sentinel Surveillance Program (ISSP) at the Houston Health Department (HHD), Houston, Texas, and represented a subset of data from a large multispecialty clinic. The ISSP was aimed initially at providing a system to help detect ongoing local ILI activity, monitor trends and morbidity, and provide information that may assist providers in patient care management [22]. When the pH1N1 arose in 2009, the existence of ISSP provided an opportunity for HHD to jump start and continue to work with the local partners to monitor and evaluate the local ILI activities including the impact of the pandemic on related healthcare-seeking behaviors [25]. The multispecialty clinic was chosen because it provided the largest pool of data with the most complete information covering the period of interest. Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 4 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Study Population We used a cohort of 2680 individual patients who received ILI-related medical care at a multispecialty clinic in Houston, Texas during the pH1N1 period (17 April 2009 through 1 March 2010) as defined locally by Mgbere et al. [25]. The patients made a total of 2960 healthcare visits to the clinic for ILI that also served as the qualifying index visits in our cohort, and subsequently, yielded 135 return visits during the period under review. Analytic Measures Dependent Measure The main outcome variable of interest in this study was return visit. Using data from the HHD ISSP, we created a metric to capture aggregate changes in patient visit behavior and care-utilization over time [25]. This metric was designed to reduce bias from an increase in sheer volume of one- time patients, as would be expected during a typical pandemic period. The resulting metric, ‘return visits’ included only non-initial visits (non-index visit) made by a given patient within the pH1N1 phase including those that may have resulted in hospitalization [25]. All return visits for ILI-related conditions prior to and post-pH1N1 period were excluded from the current study since our main objective was to assess the proportion of visits and return visits during the pH1N1 period only. We adopted a 30-day visit-free period prior to study phase to serve as wash-out period that allowed for accurate definition of the index visit, thus avoiding the need to characterize the factors as well as the trajectory of prior visits. If a patient had more than one qualifying visits over the study period, we considered the earliest visit as their index visit [25]. Return visits associated with any other diagnosis outside ILI were also excluded. Healthcare visit to clinic was represented as “0” for non- return visit (index visit) and “1” for return visit. Independent Measures The independent measures used in this study include gender (female, male), age group (0-4, 5-24, 25-49 and 50+ years), vaccine type (seasonal flu, pH1N1 and no history), influenza A (pH1N1) test result (negative, positive) and hospitalization (no, yes). The laboratory-confirmed 2009 pandemic influenza A (H1N1) infection was defined as a positive test result at the local and state public health laboratories or at CDC laboratory using real-time reverse transcription-polymerase chain (rRT-PCR) and viral culture protocols, probes, primers, and reagents approved by CDC [39,40]. The independent associations of these measures with the outcome variable were determined, and subsequently, the independent measures were used as covariates in the prediction and recursive partition model analyses. Data Analysis The demographic and clinical characteristics of the study cohort were summarized using frequencies and proportions, as appropriate. The associations between the independent factors (gender, age group, vaccine type, influenza A (H1N1) test result and hospitalization) and the study outcome (Return visit) were determined using chi-square test (χ2) and the Fisher’s exact test, where expected cell size was <5. Furthermore, we conducted nominal logistic regression model for the outcome variable (return visit) incorporating all the independent factors, except hospitalization Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 5 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI (p>0.05), as predictor variables. This produced the model parameter estimates, standard errors, and the associated hypothesis tests. We also carried out the Effect Likelihood Ratio Tests of the independent factors and produced estimates of the adjusted odds ratios (aORs), 95% confidence intervals (95% CIs) and p-values. The model diagnostics and fit statistics were determined using the -Loglikelihood test, uncertainty coefficient of determination, corrected Akaike information criterion and Bayesian information criterion. Finally, we conducted a recursive partition model (Also refer to as decision tree or classification and regression trees (CART) model) analysis to determine the relationships between the predictors (age group, vaccine type, influenza A (pH1N1) test result) and the dependent factor (Return Visit), forming a tree of decision rules until the desired fit was reached. Although influenza A (pH1N1) test result variable was not statistically significant (p>0.05) in the univariate analysis, its epidemiologic and clinical importance warranted its inclusion in the model. This tree-based method predicts the value of a response variable by forming subgroups of patients within which the response is relatively homogeneous based on the values of a set of predictor variables. The decision tree model was chosen because it has the advantage of being very intuitive and flexible, does not require scaling or normalization of data and the outcome can easily be interpreted by stakeholders. The splitting criteria is based on the likelihood-ratio chi-square (G2) [2*entropy]. A candidate G2 chosen is given by the formula below: G2 test = G 2 parent - (G 2 left + G 2 right) When Partition calculates a G2 or R2 on excluded data for a categorical response, it uses the rate value 0.25/m when it encounters a zero rate in a group with m rows. Otherwise, a missing statistic would be reported, since the logarithm of zero is undefined. The predicted probabilities for the decision tree method used were calculated using the probability statistics. Rate is the proportion of observations at the node for each response level while Prob is the predicted probability for that node of the tree. The method for calculating Prob for the ith response level at a given node is as follows: where the summation is across all response levels, ni is the number of observations at the node for the ith response level, and priori is the prior probability for the i th response level, calculated as: priori = λ*pi+ (1-λ)Pi where pi is the priori from the parent node, Pi is the Probi from the parent node, and λ is a weighting factor currently set at 0.9. The method used for calculating Prob assures that the predicted probabilities are always non-zero. The decision tree model fit was assessed using the following measures: Entropy R-square, generalized R-square, Mean -Log p, Root Mean Square Error (RMSE) and Mean Absolute Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 6 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Deviation. Also, the misclassification rate was used to determine how many observations were correctly and incorrectly classified for each value of the response variable, thus, indicating the model fitness to the data. To avoid the partitioning overfitting the model, we applied the k-fold cross validation method by using part of a data set to estimate model parameters and using the other part to assess the predictive ability of the model. The final model was selected based on the cross-validation R-square. Furthermore, we used the Receiver Operating Characteristic (ROC) curve to determine the “goodness of fit” and measure the sorting efficiency of the model’s fitted probabilities for the response levels. Thus, the true positive y-axis is labeled “Sensitivity” (the probability that a given x value (measure) correctly predicts non-return and return visits) and the false positive x-axis is labeled “1-Specificity” (the probability of incorrect prediction of non-return and return visits based on a given x is 1–sensitivity). The area under the curve (AUC) statistic, defined as the percentage of the space ‘under the curve’ (100% represents perfect classification) was used as the basis for determining the extent to which our model successfully classifies the responses. All tests were two-tailed, with a probability value of α = 0·05 used as the threshold for declaring statistical significance level. All data management and statistical analyses were conducted using JMP Statistical Software version 14.1 (SAS Institute, Cary, North Carolina, USA). Human Subject Protection The data used for this study was originally collected for public health surveillance purposes and therefore, was not considered to be human subjects research in accordance with federal human subjects’ protection regulations. The dataset used contained no individual identifiers, thus, maintaining anonymity of subjects. Hence, this study received an exempt status approval from the Houston Health Department Investigative Review Committee. Results Participants’ Characteristics and Medical Visits Table 1 show the distribution of the characteristics of study subjects and their associations with the ILI-related visits to multispecialty clinics during 2009 H1N1 influenza pandemic period. The overall mean age of the cohort was 22.89±0.35 years, while the mean age of patients who had return visits was 10.70±1.11 years. Majority of the cohort were females (56.1%), of age 5-24 years (40.6%) with approximately 61% of them having no documented vaccination history. Of the 2,960 subjects who had ILI-related medical visits, 84.9% were negative for pH1N1 and only 16.1% of them tested positive for pH1N1 while less than 1% of the reported hospitalizations were related to ILI. We recorded only 4.56% (n=135) of return visits in the cohort following the index visit during the pH1N1 period (Table 1). The number of patients who had return and non-return visits to the multispecialty clinic during the study period by Morbidity and Mortality Weekly Report (MMWR) weeks is depicted in Figure 1. The proportional variations in return visits were mainly associated with patients’ age group (χ2=66.74, p<0.001) and vaccine type (χ2=37.33, p<0.001) administered prior to ILI-related visits and/or diagnoses. About 87.5% of the return visits occurred among Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 7 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI patients who were aged 0-24 years and subsequently decreased significantly with increase age. Patients who received pH1N1 vaccine had 0.71% less return visits (p<0.0001) than those who either received seasonal flu vaccine or had no vaccination history. Gender, pH1N1 test result and hospitalization were not significantly (p>0.05) associated with return visits in our cohort. Table 1: Characteristics of Study Populations’ ILI-Related Visits to Multispecialty Clinic during 2009 pH1N1 Characteristic ILI-Related Visits to Clinic Test Statistics Total N (%) Non-Return Visits n (%) Return Visits n (%) χ2 (df) P-value Overall 2960 (100) 2825 (95.44) 135 (4.56) 384.99 (1) <0.0001**** Gender Female Male 1659 (56.05) 1301 (43.95) 1586 (53.58) 1239 (41.86) 73 (2.47) 62 (2.09) 0.22 (1) 0.636ns Age Group (years) 0 – 4 5 – 24 25 – 49 50+ Mean ± SEM 529 (17.87) 1203 (40.64) 860 (29.05) 368 (12.43) 22.89 ± 0.35 476 (16.08) 1138 (38.45) 847 (28.61) 364 (12.30) 23.47 ± 0.36 53 (1.79) 65 (2.20) 13 (0.44) 4 (0.14) 10.70 ± 1.11 66.74 (3) <0.0001**** Vaccine Type Seasonal Flu pH1N1 No History 590 (19.93) 565 (19.09) 1805 (60.98) 538 (18.18) 534 (18.04) 1753 (59.22) 52 (1.76) 31 (1.05) 52 (1.76) 37.33 (2) <0.0001**** Influenza A (pH1N1) Test Negative Positive 2483 (83.91) 476 (16.09) 2376 (80.30) 448 (15.14) 107 (3.62) 28 (0.95) 2.27 (1) 0.132ns Hospitalization No Yes 2942 (99.39) 18 (0.61) 2809 (94.90) 16 (0.54) 133 (4.49) 2 (0.07) ----f 0.197ns Abbreviations: χ2 (df): Chi-square (degree of freedom); SEM: Standard error of mean. Within a given characteristic, the percentages may not add up to exactly 100 due to rounding. f Fisher’s exact test was used for 2 x 2 table involving cell size less than 5 cases. Significance level: ****=p<0.0001, ns=not significant (p>0.05). Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 8 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Figure 1: Return and Non-Return Visits to Multispecialty Clinic by MMWR Week Return Visits The logistic model parameter estimates, and adjusted odds of patients’ return visits to the multispecialty clinic is presented in Table 2 and indicate that return visits were significantly (p<0.05) associated with age group and vaccine type received by patients. Return visits were generally more likely to occur among young individuals compared to older individuals. For instance, patients who were of age 0-4 and 5-24 years old were about 9 (aOR: 8.768, 95%CI: 3.388-29.945, p<0.0001) and 5 (aOR: 4.884, 95%CI: 1.978-16.246, p<0.001) times more likely to have return visits to the clinic compared to those who were 50 years and above. Similarly, patients who received pH1N1 and seasonal flu vaccinations were 74% (aOR: 1.741, 95%CI: 1.085- 2.750, p=0.022) and 59% (aOR: 1.586, 95%CI: 1.007-2.500, p=0.047) more likely to have return visits to the clinic compared to patients who had no history of vaccinations on record. In contrast, some patients who had no vaccination history on record tended to be 43% (aOR: 0.575, 95%CI: 0.364-0.922, p=0.022) less likely to have return visits to clinic compared to those who received pH1N1 vaccination (Table 2). Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 9 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Table 2: Logistic Model Parameters Estimates and Adjusted Odds Ratios (aOR) for Patients’ Return Visits to Multispecialty Clinic β Level 1 /Level 2 (Ref) aOR 95% Confidence Interval P-Value Lower Upper Gender [L-R χ2 (1) = 0.195, p=0.659ns] € Male Female 0.923 0.647 1.314 0.659ns Female Male 1.083 0.761 1.546 0.659ns Age Group (Years) [L-R χ2 (3) = 40.260, p<.0001****] € 5-24 0-4 0.557 0.364 0.855 0.008** 25-49 0-4 0.176 0.086 0.340 <.0001**** 50+ 0-4 0.114 0.033 0.295 <.0001**** 0-4 5-24 1.795 1.169 2.749 0.008** 25-49 5-24 0.316 0.163 0.570 <.0001**** 50+ 5-24 0.205 0.062 0.506 0.001*** 0-4 25-49 5.674 2.941 11.574 <.0001**** 5-24 25-49 3.160 1.753 6.124 <.0001**** 50+ 25-49 0.647 0.181 1.852 0.435ns 0-4 50+ 8.768 3.388 29.945 <.0001**** 5-24 50+ 4.884 1.978 16.246 0.001*** 25-49 50+ 1.545 0.540 5.535 0.435ns Vaccine Type [L-R χ2 (2) = 6.949, p=0.031*] € pH1N1 No History 1.741 1.085 2.750 0.022* Seasonal Flu No History 1.586 1.007 2.500 0.047* Seasonal Flu pH1N1 0.911 0.550 1.525 0.720ns No History pH1N1 0.575 0.364 0.922 0.022* No History Seasonal Flu 0.631 0.400 0.994 0.047* pH1N1 Seasonal Flu 1.098 0.656 1.818 0.720ns Influenza A (pH1N1) Test [L-R χ2 (1) = 0.467, p=0.495ns] € Positive Negative 1.169 0.739 1.795 0.495 ns Negative Positive 0.856 0.557 1.353 0.495 ns Abbreviations: Ref: Referent, aOR: Adjusted Odds ratio. β Tests and confidence intervals of odds ratios are likelihood ratio based. € Based on effect likelihood ratio test (L-R χ2 (df)). Significance level: *=p<0.05. **=p<0.01, ***=p<0.001, ****=p<0.0001, ns=not significant (p>0.05). Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 10 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Recursive Partition Model The recursive partition model analysis based on the best fit of the independent factors levels produced a decision tree with 19 splits (Figure 2) and variational contributions of the factors as follows: Age group - 3 splits (G^2=68.33, 76.0%), Vaccine type - 11 splits (G^2=16.54, 18.4%), and Influenza A (pH1N1) test result - 5 splits (G^2=5.08, 5.6%), and resultant model entropy coefficient of determination (R2) of 0.0818 and a misclassification rate of 0.0456 (Table 3). Table 4 displays the terminal nodes, the various combinations of independent factors levels and their associated response probabilities and counts for return and non-return visits. The recursive partition model showed significant variations across the terminal nodes as defined by the leaf labels with the probability of return visits to the multispecialty clinic ranging from 0.0% to 17.5%. The highest risk of return visits to the multispecialty clinic during the 2009 pH1N1 was among patients in terminal node TN01, who were of age group 0-4 years, tested positive for pH1N1, and received both seasonal flu and pH1N1 vaccines (Subgroup I, Prob.=17.5%). This was followed by those in terminal node TN02 who were of the same age group (0-4 years), tested positive for pH1N1 and had no history of vaccination (Subgroup II, Prob.=13.3%) and patients in terminal node TN03 who were of age group 5-24 years, tested positive for pH1N1 and received only seasonal flu vaccine (Subgroup III, Prob.=12.2%). The fourth subgroup was those of age group 0- 4 years, who tested negative for pH1N1 and received pH1N1 vaccine (Subgroup IV, Prob.=12.1%). In general, patients who were 50 years and above and tested positive or negative for pH1N1 had a zero probability of return visits to the clinic. The tradeoff between sensitivity and specificity of the probabilities for false-positive and true-positive rates for the clinic visits are depicted in the Receiver Operating Characteristic (ROC) curve in Figure 3. Conceptually, the ROC curve displays the efficiency of a model’s fitted probabilities in sorting out the response levels. The area under the curve (AUC) for both return and non-return was 0.729. The AUC measures discrimination level, that is, the ability of the recursive partitioning model to correctly classify the return and non-return visits. A summary table for the ROC parameters estimates can be accessed in the supplementary materials section (Appendix A). Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 11 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Figure 2: Classification Decision Tree for Patients’ Return Visits to Multispecialty Clinic Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 12 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Table 3: Column Contributions and Recursive Partition Model Fit Term Number of Splits G^2 Plot Proportion Age Group 3 68.329 0.760 Vaccine Type 11 16.537 0.184 Influenza A (pH1N1) Test 5 5.076 0.056 Measure Value Definition Entropy R-Square 0.0818 1-Loglike(model)/Loglike(0) Generalized R-Square 0.0965 (1-(L(0)/L(model))^(2/n))/(1-L(0)^(2/n)) Mean -Log p 0.1702 ∑ -Log(ρ[j])/n RMSE 0.2052 √ ∑(y[j]-ρ[j])2/n Mean Abs Dev 0.0842 ∑ |y[j]-ρ[j]|/n Misclassification Rate 0.0456 ∑ (ρ[j]≠ρMax)/n N 2960 n Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 13 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Table 4: Recursive Partition Model Leaf Report Showing Predicted Response Probabilities and Counts for Patients’ Return and Non-Return Visits to Multispecialty Clinic Terminal Node Leaf Label Response Probability Response Counts Return Visits Non-Return Visits Return Visits Non-Return Visits TN01 ^&Age Group(0-4)&Influenza A (pH1N1) Test(Positive)&Vaccine Type(Seasonal Flu, pH1N1) 0.1753 0.8247 8 37 TN02 ^&Age Group(0-4)&Influenza A (pH1N1) Test(Positive)&Vaccine Type(No History) 0.1332 0.8668 3 19 TN03 ^&Age Group(5-24)^&Influenza A (pH1N1) Test(Positive)&Vaccine Type(Seasonal Flu) 0.1216 0.8784 7 50 TN04 ^&Age Group(0-4)&Influenza A (pH1N1) Test(Negative)^&Vaccine Type(pH1N1) 0.1212 0.8788 6 43 TN05 ^&Age Group(0-4)&Influenza A (pH1N1) Test(Negative)^&Vaccine Type(Seasonal Flu) 0.0936 0.9064 27 261 TN06 ^&Age Group(5-24)^&Influenza A (pH1N1) Test(Negative)&Vaccine Type(pH1N1) 0.0879 0.9121 17 176 TN07 ^&Age Group(25-49)&Influenza A (pH1N1) Test(Positive)&Vaccine Type(pH1N1) 0.0866 0.9134 1 10 TN08 ^&Age Group(0-4)&Influenza A (pH1N1) Test(Negative)&Vaccine Type(No History) 0.0719 0.9281 9 116 TN09 ^&Age Group(25-49)&Influenza A (pH1N1) Test(Negative)&Vaccine Type(Seasonal Flu) 0.0648 0.9352 1 14 TN10 ^&Age Group(5-24)^&Influenza A (pH1N1) Test(Negative)&Vaccine Type(Seasonal Flu) 0.0614 0.9386 11 168 TN11 ^&Age Group(5-24)&Vaccine Type(No History)&Influenza A (pH1N1) Test(Negative) 0.0426 0.9574 22 495 TN12 ^&Age Group(5-24)^&Influenza A (pH1N1) Test(Positive)&Vaccine Type(pH1N1) 0.0355 0.9645 3 82 Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 14 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Terminal Node Leaf Label Response Probability Response Counts Return Visits Non-Return Visits Return Visits Non-Return Visits TN13 ^&Age Group(5-24)&Vaccine Type(No History)&Influenza A (pH1N1) Test(Positive) 0.0292 0.9708 5 167 TN14 ^&Age Group(25-49)&Influenza A (pH1N1) Test(Positive)&Vaccine Type(No History) 0.0173 0.9827 1 58 TN15 ^&Age Group(25-49)&Influenza A (pH1N1) Test(Negative)^&Vaccine Type(No History) 0.0136 0.9864 9 655 TN16 ^&Age Group(50+)&Influenza A (pH1N1) Test(Negative)^&Vaccine Type(No History) 0.0134 0.9866 3 223 TN17 ^&Age Group(50+)&Influenza A (pH1N1) Test(Negative)^&Vaccine Type(pH1N1) 0.0095 0.9905 1 107 TN18 ^&Age Group(25-49)&Influenza A (pH1N1) Test(Negative)^&Vaccine Type(pH1N1) 0.0092 0.9908 1 110 TN19 ^&Age Group(50+)&Influenza A (pH1N1) Test(Negative)&Vaccine Type(Seasonal Flu) 0.0037 0.9963 0 9 TN20 ^&Age Group(50+)&Influenza A (pH1N1) Test(Positive) 0.0015 0.9985 0 25 Leaf label or terminal node (TN) showing the independent factors level subgroups. Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 15 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Figure 3: Receiver Operating Characteristic (ROC) Curve Showing the model’s fitted probabilities for false-positive and true-positive rates [ Discussion Although the 2009 H1N1 pandemic tested some aspects of the nation’s response capabilities, it did not fully address others including testing the health care system’s ability to meet a surge in demand for care [4]. For this reason, acting now on lessons learned is imperative especially as the occurrence does not reduced the risk of a future, severe pandemic. Our study assessed the prevalence of return visits during the 2009 H1N1 pandemic and used model-based recursive partitioning to identify patients’ subgroups that may be at high risk of return visits. The prevalence of return visits in our cohort was 4.56% with significant variations noted across age groups and vaccine types received. This rate is similar to the range of 2.0-5.2% reported for return visits in other studies [16,17,41,42]. The majority of the return visits (87.5%) in our cohort occurred among patients who were of age group 0-24 years. Patients of age group 0-4 years were about 9 times more likely to have return visits compared to patients who were 50 years and above. An earlier Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 16 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI study reported more return visits during the pH1N1 than in the pre- and post pH1N1 phases [25]. Our study noted that patients who received pH1N1 and seasonal flu vaccines were about 2 times more likely to have return visits than those with no history of vaccination. Since the (H1N1)pdm09 virus was very different from circulating H1N1 viruses, vaccination with seasonal flu vaccines offered little cross-protection against (H1N1)pdm09 virus infection [4]. It is possible that some patients may have been vaccinated against seasonal flu and/or pH1N1, but their historical records were not available or reported to the surveillance team, causing the misnomer of the patients being protective. However, the reported vaccine effectiveness (VE) for Influenza A (pH1N1) during the study period was 56% (95%CI: 23-75%) [40,43]. The uptake rates for both seasonal influenza and pH1N1 pandemic vaccines in our cohort prior to their return visits were generally very low (38.5% vs. 23.0%). Although the vaccination patterns in our study mirrored the national trends [44,45], it is possible, in part, to attribute the low uptake to the late availability of pH1N1 vaccine in US in October 2019 [4]. It has been reported that the introduction of a vaccine four months after the pandemic virus arrival limited the use and effectiveness [36]. Despite the low VE, influenza vaccination remains the primary strategy to prevent influenza illness and its complications. Therefore, improvements in both vaccine effectiveness and coverage are needed to help reduce avoidable medical visits. Several factors such as previous experience with ILI [13], personal beliefs about vaccination [14,15], and previous error in the diagnosis of an illness or the progression of an illness [16-18], and widespread report of morbidity and mortality during the pandemic [19], have been identified as motivators for return visits. It is likely that the single most important non-medical cause of surge in health care visits was the media coverage of the pandemic, which created unnecessary public concerns, and amplify the risk perceptions [5,8-11]. Unfortunately, our interpretation is limited by the absence of data on the reasons for the patients’ return visits to the clinic including the roles that media coverage may have played. However, previous research noted that up to 73% of patients that showed up at ED for H1N1 influenza fear were the “worried well” and that over 95% of the presenting concerns were minor or nonexistent [20]. Using the model-based recursive partitioning model, we created a decision tree that classified our cohort into 19 subgroups with the probability of return visits to the multispecialty clinic ranging from 0.0% to 17.5%. The procedure allowed for the automated detection of patient subgroups that are linked to predictive factors by means of a decision tree, and thus, splitting the study population into more homogeneous subgroups [46]. Among the predictor variables, age group was identified as the most important factor influencing patients’ return visits followed by vaccine type and pH1N1 test outcome and their respective interaction effects. Identifying subgroups in this way can help inform decision making thus improving individualized patient care by targeting treatment accordingly [46,47]. Return visits in our cohort were more common among younger patients, which tended to mirror the national trends in US, where approximately 60% of the reported cases of pH1N1 occurred in persons 18 years of age or younger [43]. Our results support the finding that some older adults may be less likely to develop influenza A (H1N1) infection [48], and consequently, leading to a smaller number of return visits among the older individuals’ subgroups. Nearly one-third of people over 60 years old had antibodies against this virus, likely from exposure to an older strain of H1N1 virus earlier in their lives [4]. Nevertheless, given concerns about the potential scope of future pandemic influenza outbreaks, efforts are needed to prepare for rapid increases in health care-seeking behaviors and to develop effective communication strategies that Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 17 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI would encourage behaviors that help slowdown the spread of the virus and minimize unnecessary health care visits to reduce health care surge [49]. Although the proportion of variance explained in the model was about 8.18%, the AUC for both return and non-return visits was 72.9%, indicating a fairly accurate classification of the two groups. The resulting performance statistics derived from the model (sensitivity/specificity) indicate that the proposed model may be useful in practice as a screener for patients at high risk of return visits. Accurate and timely forecasts could aid public health response by informing key preparation and mitigation efforts that could help reduce the overall health and socio-economic impact of pandemics. Limitations and Strength of Study The findings of this study should be interpreted with several important limitations in mind. First, the study was based on data originally collected by HHD for surveillance purposes, and not research. Therefore, the extent to which we can interpret the current findings without supplementary information is limited. For instance, certain important variables of interest such as patients’ reasons for returning to the clinic and/or whether physician recommended the return visits were not available. It is possible that some patients were initially misdiagnosed or mistreated, or experienced progression or some complications in their illness leading to their return visits to the clinic. Also, some patients may have been ill because of another pathogen causing ILI-related symptoms, or that some patients may have been exposed to pH1N1 during the initial clinic visit. Availability of this information would have been beneficial in providing contexts to our results. Second, information on socioeconomic status, race/ethnicity, and health insurance coverage of the patients visiting the clinics were not available for this study. Such data would have enabled identification of more demographic-specific healthcare utilization patterns in the context of pH1N1. However, being a private clinic, we assumed that our cohort represented mostly insured population that may experience different and fewer socioeconomic stressors than uninsured or underinsured populations. In addition, some variables such as vaccination history of the patients were either incomplete or unavailable. Third, while the definition of ILI is well understood, the decision to administer the test is at the discretion of the individual healthcare worker. Individual variation may also have existed between healthcare workers who administered the rapid influenza tests. Fourth, we used data from a single multispecialty clinic because it provided the largest pool of data with the most complete information covering the period of interest. This implies that caution should be exercise in interpreting the outcomes as they may not be representative of the ILI activities in the Houston metropolitan area during the 2009 influenza pandemic. Fifth, the surveillance data did not capture whether patients’ levels of exposure to information about influenza or pH1N1 from traditional or social media sources had any influence on their decision to seek care. It is also possible that the return visits may have been due to clustering effects associated with common factors such as shared living space and similar access to information [25], which may have had a direct impact on both the disease transmissions and beliefs regarding medical care utilization. Finally, we acknowledge that there could be some biases toward predictors with more variance or levels or due to a small change in the learning data with possible direct impact on the structure of the decision tree. However, our model diagnostics generally indicated that our current model was stable. Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 18 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI Despite the constraints that these limitations may have on generalizability, application to practice, and/or utility of our findings, we relied on the best information available at the time to identify the patients’ subgroups that are likely to be at high risk of encountering return visits, if similar pandemic were to occur. The use of actual facility-level data (instead of self-reported survey data) and the segmentation procedure of the recursive partition model allowed for the display of reliable terminal nodes’ discriminatory ability of the associated predictor factors resulting in homogeneous risk strata. In addition, we conducted a series of model diagnostics including cross-validation and sensitivity analyses to support the base-case estimations. Consequently, the tree-based approach applied empowered predictive model with high accuracy and stability, and enhanced ease of interpretation and understanding by the medical community and other stakeholders. Conclusions Return visits accounted for a small proportion of the medical visits, with majority of those being associated with children and young adult patients. Our study helped to empirically identify and rank the subgroups at high risk of return visits, and consequently, patients who could benefit from intense outpatient referral or intervention program to prevent unnecessary return visits. Furthermore, findings from our study could be used by policymakers as part of a decision support system to create awareness and understanding of surges due to recurrent utilization of the healthcare system during pandemics and emergency preparedness. Understanding the dynamics in care-seeking behavior during the 2009 H1N1 pandemic is important to help prepare for future outbreaks of pandemic influenza viruses. This would assist policymakers with appropriate resource allocation, and in the design of policy initiatives aimed at mitigating surges and recurrent utilization of the healthcare system. It is recommended that future studies should take into consideration the limitations identified in the present study including patients’ perspectives on the social and medical factors that may lead to patients’ return visits and health care surge capacity. Summary Points What was already known? • During the 2009 H1N1 influenza pandemic (pH1N1), the proportion of outpatient visits to emergency departments, hospitals and clinics became elevated. • The 2009 pH1N1 did not fully test the health care system’s ability to meet a surge in demand for care. • Considerable efforts in recent years have been dedicated to mathematical modeling studies to complement disease surveillance efforts in the planning of interventions against emerging pandemics. What this study added to our knowledge • This study used model-based recursive partitioning to predict return visits to multispecialty clinic during the 2009 pH1N1. • This study empirically identified and ranked patients at high risk of return visits for 2009 pH1N1 into subgroups that could benefit from intense outpatient referral or intervention program to prevent avoidable return visits. Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 19 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI • This study brings about understanding of the dynamics of care-seeking behaviors during 2009 pH1N1 and the associated predictive factors that could help the medical community prepare for future outbreaks of pandemic influenza virus. Authors’ Contributions OM conceived and designed the study, conducted the data analysis and results interpretation, prepared the initial draft of the manuscript, and participated in the critical review and revision of the article. SK supervised the acquisition of data, participated in results interpretation, and critical review and revision of the article. Both authors read and approved the final version of the article for publication. Acknowledgements We thank the physicians and nurses at the clinics in Houston, TX that participated in the HHD ISSP for their support. The generous contributions and assistance of the Houston Health Department staff at the Bureaus of Epidemiology and Laboratory Services to the ISSP are also greatly appreciated. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Houston Health Department. Funding The authors received no financial support for the research, authorship, and/or publication of this article. Conflict of Interest Disclosure The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Corresponding Author Dr. Osaro Mgbere, Disease Prevention and Control Division, Houston Health Department, 8000 North Stadium Drive, Houston, Texas, 77054, USA. Email: Osaro.Mgbere@Houstontx.gov; Tel: 1-832-393-4593. Appendix A: Supplementary Materials Supplementary materials consist of data provided by the authors that are published to benefit the reader. The contents of this supplementary data are the sole responsibility of the authors. All questions should be directed to the corresponding author Dr. Osaro Mgbere at Osaro.Mgbere@Houstontx.gov. Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 20 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI SUPPLEMENTARY MATERIAL Appendix A: Receiver Operating Characteristic Curve Parameters Estimates for Return and Non-Return Visits to Multispecialty Clinic Probability 1-Specificity Sensitivity Sens-(1-Spec) True Positive True Negative False Positive False Negative . 0.0000 0.0000 0.0000 0 2825 0 135 0.1341 0.0007 0.0148 0.0141 2 2823 2 133 0.1233 0.0131 0.0593 0.0462 8 2788 37 127 0.1173 0.0283 0.1037 0.0754 14 2745 80 121 0.1076 0.1207 0.3037 0.1830 41 2484 341 94 0.0817 0.1274 0.3259 0.1985 44 2465 360 91 0.0798 0.1565 0.3481 0.1917 47 2383 442 88 0.0730 0.1742 0.4000 0.2258 54 2333 492 81 0.0709 0.2152 0.4667 0.2514 63 2217 608 72 0.0693 0.2775 0.5926 0.3151 80 2041 784 55 0.0633 0.3370 0.6741 0.3371* 91 1873 952 44 0.0475 0.3961 0.7111 0.3150 96 1706 1119 39 0.0410 0.5713 0.8741 0.3027 118 1211 1614 17 0.0268 0.5749 0.8815 0.3066 119 1201 1624 16 0.0231 0.6138 0.8889 0.2751 120 1091 1734 15 0.0210 0.6188 0.8963 0.2775 121 1077 1748 14 0.0175 0.6202 0.8963 0.2761 121 1073 1752 14 0.0159 0.6205 0.8963 0.2758 121 1072 1753 14 0.0156 0.6411 0.9037 0.2626 122 1014 1811 13 0.0150 0.6789 0.9111 0.2322 123 907 1918 12 0.0137 0.6821 0.9111 0.2290 123 898 1927 12 0.0134 0.9140 0.9778 0.0638 132 243 2582 3 0.0101 0.9211 0.9778 0.0567 132 223 2602 3 0.0087 1.0000 1.0000 0.0000 135 0 2825 0 0.0087 1.0000 1.0000 0.0000 135 0 2825 0 The row with the highest Sens-(1-Spec) is marked with an asterisk. Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 21 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI References 1. Centers for Disease Control and Prevention (CDC). 2009. Swine influenza A (H1N1) infection in two children – Southern California, March-April 2009. MMWR Morb Mortal Wkly Rep. 58(15), 400-02. PubMed 2. Novel Swine-Origin Influenza A (H1N1) Virus Investigation Team, Dawood FS, Jain S, et al. 2009. Emergence of a novel swine-origin influenza A (H1N1) virus in humans [published correction appears in N Engl J Med. 2009 Jul 2;361(1):102]. N Engl J Med. 360(25), 2605- 15. doi:https://doi.org/10.1056/NEJMoa0903810. PubMed 3. Garten RJ, Davis CT, Russell CA, et al. 2009. Antigenic and genetic characteristics of swine- origin 2009 A(H1N1) influenza viruses circulating in humans. Science. 325(5937), 197-201. doi:https://doi.org/10.1126/science.1176225. PubMed 4. Centers for Disease Prevention and Control. The burden of the influenza A H1N1pdm09 virus since the 2009 pandemic. [Internet]. Atlanta, GA; 2009. Available from: https://www.cdc.gov/flu/pandemic-resources/burden-of-h1n1.html 5. US Department of Health and Human Services. An HHS Retrospective on the 2009 H1N1 Influenza Pandemic to Advance All Hazards Preparedness [Internet]. 2012. Available from: http://www.phe.gov/Preparedness/mcm/h1n1-retrospective/Documents/h1n1- retrospective.pdf 6. McDonnell WM, Nelson DS, Schunk JE. 2012. Should we fear “flu fear” itself? Effects of H1N1 influenza fear on ED use. Am J Emerg Med. 30(2), 275-82. doi:https://doi.org/10.1016/j.ajem.2010.11.027. PubMed 7. Cruz AT, Patel B, DiStefano MC, et al. 2010. Outside the box and into thick air: implementation of an exterior mobile pediatric emergency response team for North American H1N1 (swine) influenza virus in Houston, Texas. Ann Emerg Med. 55(1), 23-31. doi:https://doi.org/10.1016/j.annemergmed.2009.08.003. PubMed 8. Weiss EA, Ngo J, Gilbert GH, Quinn JV. 2010. Drive-through medicine: a novel proposal for rapid evaluation of patients during an influenza pandemic. Ann Emerg Med. 55(3), 268-73. doi:https://doi.org/10.1016/j.annemergmed.2009.11.025. PubMed 9. SteelFisher G, Blendon R, Ross LJ, et al. 2011. Public response to an anthrax attack: reactions to mass prophylaxis in a scenario involving inhalation anthrax from an unidentified source. Biosecur Bioterror. 9(3), 239-50. doi:https://doi.org/10.1089/bsp.2011.0005. PubMed 10. Kasperson RE, Kasperson JX. 1996. The Social Amplification and Attenuation of Risk. Ann Am Acad Pol Soc Sci. 545, 95-105. http://www.jstor.org/stable/1047896. https://doi.org/10.1177/0002716296545001010 11. Bomlitz LJ, Brezis M. 2008. Misrepresentation of health risks by mass media. J Public Health (Oxf). 30(2), 202-04. doi:https://doi.org/10.1093/pubmed/fdn009. PubMed https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19390508&dopt=Abstract https://doi.org/10.1056/NEJMoa0903810 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19423869&dopt=Abstract https://doi.org/10.1126/science.1176225 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19465683&dopt=Abstract https://www.cdc.gov/flu/pandemic-resources/burden-of-h1n1.html http://www.phe.gov/Preparedness/mcm/h1n1-retrospective/Documents/h1n1-retrospective.pdf http://www.phe.gov/Preparedness/mcm/h1n1-retrospective/Documents/h1n1-retrospective.pdf https://doi.org/10.1016/j.ajem.2010.11.027 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21208765&dopt=Abstract https://doi.org/10.1016/j.annemergmed.2009.08.003 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19837479&dopt=Abstract https://doi.org/10.1016/j.annemergmed.2009.11.025 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=20079956&dopt=Abstract https://doi.org/10.1089/bsp.2011.0005 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21819225&dopt=Abstract https://doi.org/10.1177/0002716296545001010 https://doi.org/10.1093/pubmed/fdn009 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=18281322&dopt=Abstract Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 22 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI 12. Keramarou M, Cottrell S, Evans MR, et al. 2011. Two waves of pandemic influenza A(H1N1) 2009 in Wales--the possible impact of media coverage on consultation rates, April-December 2009. Euro Surveill. 16(3), 19772. PubMed 13. Olowokure B, Odedere O, Elliot AJ, et al. 2012. Volume of print media coverage and diagnostic testing for influenza A(H1N1)pdm09 virus during the early phase of the 2009 pandemic. J Clin Virol. 55(1), 75-78. doi:https://doi.org/10.1016/j.jcv.2012.05.013. PubMed 14. Wells CR, Bauch CT. 2012. The impact of personal experiences with infection and vaccination on behaviour-incidence dynamics of seasonal influenza. Epidemics. 4(3), 139-51. doi:https://doi.org/10.1016/j.epidem.2012.06.002. PubMed 15. Wooten KG, Wortley PM, Singleton JA, Euler GL. 2012. Perceptions matter: beliefs about influenza vaccine and vaccination behavior among elderly white, black and Hispanic Americans. Vaccine. 30(48), 6927-34. doi:https://doi.org/10.1016/j.vaccine.2012.08.036. PubMed 16. Lewis TC, Hershenson MB, Singer DC, Butchart AT, Clark S, et al. 2011. Parental Attitudes Associated With Vaccinating Asthmatic Children Against Influenza In A National Poll On Children’s Health. Am J Respir Crit Care Med. 183, A3789. https://www.atsjournals.org/doi/abs/10.1164/ajrccm- conference.2011.183.1_MeetingAbstracts.A3789 17. Burokienė S, Kairienė I, Strička M, et al. 2017. Unscheduled return visits to a pediatric emergency department. Medicina (Kaunas). 53(1), 66-71. doi:https://doi.org/10.1016/j.medici.2017.01.003. PubMed 18. van der Linden MC, Lindeboom R, de Haan R, et al. 2014. Unscheduled return visits to a Dutch inner-city emergency department. Int J Emerg Med. 7, 23. Published Jul 2014. doi:https://doi.org/10.1186/s12245-014-0023-6. PubMed 19. Sauvin G, Freund Y, Saïdi K, Riou B, Hausfater P. 2013. Unscheduled return visits to the emergency department: consequences for triage [published correction appears in Acad Emerg Med. 2013 Mar;20(3):327]. Acad Emerg Med. 20(1), 33-39. doi:https://doi.org/10.1111/acem.12052. PubMed 20. Woo TM. 2010. 2009 H1N1 influenza pandemic. J Pediatr Health Care. 24(4), 258-66. doi:https://doi.org/10.1016/j.pedhc.2010.05.001. PubMed 21. Stokes JW, Banderet LE. 1997. Psychological aspects of chemical defense and warfare. Mil Psychol. 9, 395-415. https://doi.org/10.1207/s15327876mp0904_8 22. Paul S, Mgbere O, Arafat R, Yang B, Santos E. 2017. Modeling and Forecasting Influenza- like Illness (ILI) in Houston, Texas Using Three Surveillance Data Capture Mechanisms. Online J Public Health Inform. 9(2), e187. Published Sep 2017. doi:https://doi.org/10.5210/ojphi.v9i2.8004. PubMed https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21262184&dopt=Abstract https://doi.org/10.1016/j.jcv.2012.05.013 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=22710009&dopt=Abstract https://doi.org/10.1016/j.epidem.2012.06.002 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=22939311&dopt=Abstract https://doi.org/10.1016/j.vaccine.2012.08.036 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=22939908&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=22939908&dopt=Abstract https://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A3789 https://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A3789 https://doi.org/10.1016/j.medici.2017.01.003 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=28233682&dopt=Abstract https://doi.org/10.1186/s12245-014-0023-6 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=25045407&dopt=Abstract https://doi.org/10.1111/acem.12052 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=23570476&dopt=Abstract https://doi.org/10.1016/j.pedhc.2010.05.001 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=20620852&dopt=Abstract https://doi.org/10.1207/s15327876mp0904_8 https://doi.org/10.5210/ojphi.v9i2.8004 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=29026453&dopt=Abstract Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 23 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI 23. Khuwaja S, Mgbere O, Awosika-Olumo A, Momin F, Ngo K. 2011. Using sentinel surveillance system to monitor seasonal and novel H1N1 influenza infection in Houston, Texas: outcome analysis of 2008-2009 flu season. J Community Health. 36(5), 857-63. doi:https://doi.org/10.1007/s10900-011-9386-2. PubMed 24. Flasche S, Hens N, Boëlle PY, et al. 2011. Different transmission patterns in the early stages of the influenza A(H1N1)v pandemic: a comparative analysis of 12 European countries. Epidemics. 3(2), 125-33. doi:https://doi.org/10.1016/j.epidem.2011.03.005. PubMed 25. Mgbere O, Ngo K, Khuwaja S, et al. 2017. Pandemic-related health behavior: repeat episodes of influenza-like illness related to the 2009 H1N1 influenza pandemic. Epidemiol Infect. 145(12), 2611-17. doi:https://doi.org/10.1017/S0950268817001467. PubMed 26. Centers for Disease Control and Prevention. Estimated Influenza Illnesses, Medical visits, Hospitalizations, and Deaths in the United States, 2017–2018 influenza season. [Internet]. Atlanta, GA; 2018. Available from: https://www.cdc.gov/flu/about/burden/2017-2018.htm 27. Presanis AM, De Angelis D, & New York City Swine Flu Investigation Team, et al. 2009. The severity of pandemic H1N1 influenza in the United States, from April to July 2009: a Bayesian analysis. PLoS Med. 6(12), e1000207. doi:https://doi.org/10.1371/journal.pmed.1000207. PubMed 28. Menon DK, Taylor BL, Ridley SA, & Intensive Care Society, UK. 2005. Modelling the impact of an influenza pandemic on critical care services in England. Anaesthesia. 60(10), 952-54. doi:https://doi.org/10.1111/j.1365-2044.2005.04372.x. PubMed 29. Sobieraj JA, Reyes J, Dunemn KN, et al. 2007. Modeling hospital response to mild and severe influenza pandemic scenarios under normal and expanded capacities. Mil Med. 172(5), 486- 90. doi:https://doi.org/10.7205/MILMED.172.5.486. PubMed 30. Chao DL, Halloran ME, Obenchain VJ, Longini IM, Jr. 2010. FluTE, a publicly available stochastic influenza epidemic simulation model. PLOS Comput Biol. 6(1), e1000656. Published Jan 2010. doi:https://doi.org/10.1371/journal.pcbi.1000656. PubMed 31. McVernon J, McCaw JM, Nolan TM. 2010. Modelling strategic use of the national antiviral stockpile during the CONTAIN and SUSTAIN phases of an Australian pandemic influenza response. Aust N Z J Public Health. 34(2), 113-19. doi:https://doi.org/10.1111/j.1753- 6405.2010.00493.x. PubMed 32. Ercole A, Taylor BL, Rhodes A, Menon DK. 2009. Modelling the impact of an influenza A/H1N1 pandemic on critical care demand from early pathogenicity data: the case for sentinel reporting. Anaesthesia. 64(9), 937-41. doi:https://doi.org/10.1111/j.1365-2044.2009.06070.x. PubMed 33. Baker PR, Sun J, Morris J, Dines A. 2011. Epidemiologic modeling with FluSurge for pandemic (H1N1) 2009 outbreak, Queensland, Australia. Emerg Infect Dis. 17(9), 1608-14. doi:https://doi.org/10.3201/eid1709.102012. PubMed https://doi.org/10.1007/s10900-011-9386-2 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21380790&dopt=Abstract https://doi.org/10.1016/j.epidem.2011.03.005 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21624784&dopt=Abstract https://doi.org/10.1017/S0950268817001467 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=28726598&dopt=Abstract https://www.cdc.gov/flu/about/burden/2017-2018.htm https://doi.org/10.1371/journal.pmed.1000207 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19997612&dopt=Abstract https://doi.org/10.1111/j.1365-2044.2005.04372.x https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=16179037&dopt=Abstract https://doi.org/10.7205/MILMED.172.5.486 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=17521095&dopt=Abstract https://doi.org/10.1371/journal.pcbi.1000656 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=20126529&dopt=Abstract https://doi.org/10.1111/j.1753-6405.2010.00493.x https://doi.org/10.1111/j.1753-6405.2010.00493.x https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=23331352&dopt=Abstract https://doi.org/10.1111/j.1365-2044.2009.06070.x https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19645759&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19645759&dopt=Abstract https://doi.org/10.3201/eid1709.102012 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21888785&dopt=Abstract Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 24 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI 34. House T, Baguelin M, Van Hoek AJ, et al. 2011. Modelling the impact of local reactive school closures on critical care provision during an influenza pandemic. Proc Biol Sci. 278(1719), 2753-60. doi:https://doi.org/10.1098/rspb.2010.2688. PubMed 35. Menon DK, Taylor BL, Ridley SA, & Intensive Care Society, UK. 2005. Modelling the impact of an influenza pandemic on critical care services in England. Anaesthesia. 60(10), 952-54. doi:https://doi.org/10.1111/j.1365-2044.2005.04372.x. PubMed 36. Lee VJ, Lye DC, Wilder-Smith A. 2009. Combination strategies for pandemic influenza response - a systematic review of mathematical modeling studies. BMC Med. 7, 76. Published Dec 2009. doi:https://doi.org/10.1186/1741-7015-7-76. PubMed 37. Goldman RD, Ong M, Macpherson A. 2006. Unscheduled return visits to the pediatric emergency department-one-year experience. Pediatr Emerg Care. 22(8), 545-49. doi:https://doi.org/10.1097/01.pec.0000230553.01917.05. PubMed 38. Gallagher RA, Porter S, Monuteaux MC, Stack AM. 2013. Unscheduled return visits to the emergency department: the impact of language. Pediatr Emerg Care. 29(5), 579-83. doi:https://doi.org/10.1097/PEC.0b013e31828e62f4. PubMed 39. Centers for Disease Control and Prevention. Influenza Virus Testing Methods. [Internet]. Atlanta, GA; 2020. Available from: https://www.cdc.gov/flu/professionals/diagnosis/table- testing-methods.htm 40. Gunson R, Maclean A, Davies E, Bennett S, Miller R, et al. 2010. Development of a multiplex real-time RT-PCR that allows universal detection of influenza A viruses and simultaneous typing of influenza A/H1N1/2009 virus. J Virol Methods. 163(2), 258-61. doi:https://doi.org/10.1016/j.jviromet.2009.10.006. PubMed 41. Mintegui Raso S, Benito Fernández J, Vázquez Ronco MA, Ortiz Andrés A, Capapé Zache S, et al. 2000. Los niños que repiten consulta en urgencias de pediatría [Children's unscheduled return visits to an emergency department]. An Esp Pediatr. 52(6), 542-47. PubMed 42. Alessandrini EA, Lavelle JM, Grenfell SM, Jacobstein CR, Shaw KN. 2004. Return visits to a pediatric emergency department. Pediatr Emerg Care. 20(3), 166-71. doi:https://doi.org/10.1097/01.pec.0000117924.65522.a1. PubMed 43. Kelly AM, Chirnside AM, Curry CH. 1993. An analysis of unscheduled return visits to an urban emergency department. N Z Med J. 106(961), 334-36. PubMed 44. Griffin MR, Monto AS, Belongia EA, et al. 2011. Effectiveness of non-adjuvanted pandemic influenza A vaccines for preventing pandemic influenza acute respiratory illness visits in 4 U.S. communities. PLoS One. 6(8), e23085. doi:10.1371/journal.pone.0023085. PubMed 45. Centers for Disease Control and Prevention. Past Seasons Vaccine Effectiveness Estimates. [Internet]. Atlanta, GA; 2019. Available from: https://www.cdc.gov/flu/vaccines-work/past- seasons-estimates.html https://doi.org/10.1098/rspb.2010.2688 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21288945&dopt=Abstract https://doi.org/10.1111/j.1365-2044.2005.04372.x https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=16179037&dopt=Abstract https://doi.org/10.1186/1741-7015-7-76 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=20003249&dopt=Abstract https://doi.org/10.1097/01.pec.0000230553.01917.05 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=16912620&dopt=Abstract https://doi.org/10.1097/PEC.0b013e31828e62f4 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=23603647&dopt=Abstract https://www.cdc.gov/flu/professionals/diagnosis/table-testing-methods.htm https://www.cdc.gov/flu/professionals/diagnosis/table-testing-methods.htm https://doi.org/10.1016/j.jviromet.2009.10.006 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19854220&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=11003963&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=11003963&dopt=Abstract https://doi.org/10.1097/01.pec.0000117924.65522.a1 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=15094574&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=8341474&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=21857999&dopt=Abstract https://www.cdc.gov/flu/vaccines-work/past-seasons-estimates.html https://www.cdc.gov/flu/vaccines-work/past-seasons-estimates.html Model-Based Recursive Partitioning of Patients’ Return Visits to Multispecialty Clinic During the 2009 H1N1 Pandemic Influenza (pH1N1) 25 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 12(1):e4, 2020 OJPHI 46. Seibold H, Zeileis A, Hothorn T. 2016. Model-Based Recursive Partitioning for Subgroup Analyses. Int J Biostat. 12(1), 45-63. doi:https://doi.org/10.1515/ijb-2015-0032. PubMed 47. Mistry D, Stallard N, Underwood M. 2018. A recursive partitioning approach for subgroup identification in individual patient data meta-analysis. Stat Med. 37(9), 1550-61. doi:https://doi.org/10.1002/sim.7609. PubMed 48. Centers for Disease Control and Prevention (CDC). 2009. Serum cross-reactive antibody response to a novel influenza A (H1N1) virus after vaccination with seasonal influenza vaccine. MMWR Morb Mortal Wkly Rep. 58(19), 521-24. PubMed 49. Iuliano AD, Reed C, Guh A, et al. 2009. Notes from the field: outbreak of 2009 pandemic influenza A (H1N1) virus at a large public university in Delaware, April-May 2009. Clin Infect Dis. 49(12), 1811-20. doi:https://doi.org/10.1086/649555. PubMed https://doi.org/10.1515/ijb-2015-0032 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=27227717&dopt=Abstract https://doi.org/10.1002/sim.7609 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=29383818&dopt=Abstract https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19478718&dopt=Abstract https://doi.org/10.1086/649555 https://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=19911964&dopt=Abstract