ISDS Annual Conference Proceedings 2013. This is an Open Access article distributed under the terms of the Creative Commons Attribution- Noncommercial 3.0 Unported License (http://creativecommons.org/licenses/by-nc/3.0/), permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. 27 (page number not for citation purposes) ISDS 2013 Conference Abstracts Validation of Analytic Methods for Combining Evidence Sources in Biosurveillance Howard Burkom*, Yevgeniy Elbert, Liane Ramac-Thomas and Christopher Cuellar Johns Hopkins Applied Physics Laboratory, Laurel, MD, USA Objective This presentation aims to reduce the gap between multivariate an- alytic surveillance tools and public health acceptance and utility. We developed procedures to verify, calibrate, and validate an evidence fusion capability based on a combination of clinical and syndromic indicators and limited knowledge of historical outbreak events. Introduction Recent years’ informatics advances have increased availability of various sources of health-monitoring information to agencies respon- sible for disease surveillance. These sources differ in clinical rele- vance and reliability, and range from streaming statistical indicator evidence to outbreak reports. Information-gathering advances have outpaced the capability to combine the disparate evidence for routine decision support. In view of the need for analytical tools to manage an increasingly complex data environment, a fusion module based on Bayesian networks (BN) was developed in 2011 for the Dept. of Defense (DoD) Electronic Surveillance System for the Early Notifica- tion of Community-Based Epidemics (ESSENCE). In 2012 this mod- ule was expanded with syndromic queries, data-sensitive algorithm selection, and hierarchical fusion network training [1]. Subsequent efforts have produced a full fusion-enabled version of ESSENCE for beta testing, further upgrades, and a software specification for live DoD integration. Beta test reviewers cited the reduced alert burden and the detailed evidence underlying each alert. However, only 39 reported historical events were available for training and calibration of 3 networks designed for fusion of influenza-like-illness, gastro- intestinal, and fever syndrome categories. The current presentation describes advances to formalize the network training, calibrate the component alerting algorithms and decision nodes together for each BN, and implement a validation strategy aimed at both the ESSENCE public health user and machine learning communities. Methods Hierarchical BN training was automated with an optimization method to seek conditional probability table (CPT) entries for each subnetwork that would produce desired results from given combi- nations of algorithm output states. For example, a combination of red alert for laboratory test orders, multiple recent yellow alerts for filled prescriptions, and multiple recent red alerts for ICD-based visits would produce a level of concern above 90% for outbreak investiga- tion using the network, based on results of algorithms applied to the syndromic data streams for each evidence type. The derived CPTs minimize the difference between the desired and computed BN out- puts for all state combinations. For validation in the absence of sufficient reported events, we searched algorithm results from all selected data streams in 3.75 years of data from 289 military facilities to derive ~100 historical events for each fusion type. We adopted a cross-validation strategy by par- titioning these events into 10 sets, deriving the optimal set of BN and algorithm thresholds with each 10% removed, and applying these thresholds to the remaining 10%. Each optimal threshold combination maximized the median odds ratio of the BN outbreak decision node, for alerting during outbreak vs non-outbreak days, over all training events. A single set of thresholds was then used for external validation using the 39 reported events. Results Within each fusion type, the cross-validation strategy produced consistent combinations of algorithm and BN decision thresholds, and the resultant overall threshold choices yielded 88% sensitivity to reported events, a 10-15% improvement over the original demon- stration module. Conclusions We implemented training and validation methods combining meth- ods of the disease surveillance and machine learning communities. The automated BN methods yielded an increase in sensitivity over previous heuristic methods, and the cross-validation supported the BN strategy for prospective use in ESSENCE. Increased epidemi- ologist input is needed for more granular, effective use of data and community acceptance. Extension to other surveillance environments (e.g. civilian state or province) will require expert involvement re- garding both available data and the objectives/constraints of those environments. Keywords Bayesian Network; validation; multivariate; ESSENCE References Burkom H, Elbert Y, Ramac-Thomas L, Cuellar C, Hung V, Refinement of a Population-Based Bayesian Network for Fusion of Health Surveil- lance Data, (2013) Online Journal of Public Health Informatics, 5(1), DOI: http://dx.doi.org/10.5210%2Fojphi.v5i1.4413 *Howard Burkom E-mail: howard.burkom@jhuapl.edu scholcommuser Stamp scholcommuser Rectangle scholcommuser Rectangle scholcommuser Text Box Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 6(1):e175, 2014