2014.ISDS.Abstracts.Final.pdf


ISDS Annual Conference Proceedings 2014. This is an Open Access article distributed under the terms of the Creative Commons Attribution-
Noncommercial 3.0 Unported License (http://creativecommons.org/licenses/by-nc/3.0/), permitting all non-commercial use, distribution, 
and reproduction in any medium, provided the original work is properly cited.

ISDS 2014 Conference Abstracts

Building a Better Syndromic Surveillance System: the 
New York City Experience

Robert Mathes*, Jessica Sell, Anthony W. Tam, Alison Levin-Rector and Ramona Lall

New York City Department of Health and Mental Hygiene, Queens, NY, USA

Objective
To evaluate temporal and spatial aberration detection methods for 

implementation in a local syndromic surveillance system.

Introduction
The New York City (NYC) syndromic surveillance system has 

monitored syndromes from NYC emergency department (ED) visits 
since 2001, using the temporal and spatial scan statistic in SaTScan 
for aberration detection. Since our syndromic system was initiated, 
alternative methods have been proposed for outbreak identification. 
Our goal was to evaluate methods for outbreak detection and apply 
the best performing method(s) to our daily analysis of syndromic data.

Methods
We tested six temporal aberration detection methods: a modified 

C2 algorithm [1], a CUSUM algorithm, the Holt-Winters (HW) 
smoothing method, a generalized linear model (GLM), a temporal 
scan statistic in SaTScan [2], and an autoregressive integrated moving 
average (ARIMA) model. We tested four spatio-temporal methods: a 
generalized linear mixed model (GLMM), a Bayesian model [3], and 
the spatial scan statistic using the Bernoulli probability model and 
space-time permutation scan statistic in SaTScan [2]. We analyzed 
ED visits that occurred in NYC during 2010-2011. To test a method’s 
ability to detect different outbreak types, we added simulated outbreaks 
to NYC syndromic data. A total of 180 datasets were created, each 
containing one spike of varying duration and magnitude, allocated 
either to a single ZIP code, a cluster of ZIP codes, or citywide. To 
compare the performance of these methods, we estimated sensitivity, 
specificity, timeliness, and positive predictive value (PPV) of signals. 
Timeliness was estimated by calculating the earliest day of detection 
as a proportion of the outbreak length before the peak number of 
excess cases (timeliness ranged from 1 to 0, with detection on the 
first day of an outbreak resulting in a timeliness measure of 1). To 
determine how difficult it would be to implement these methods into 
our system, we recorded programming time, computer run time, and 
skills (detailed coding or statistical) needed to program the method.

Results
The temporal scan statistic, our current method for temporal 

outbreak detection, performed the best in regard to sensitivity and 
timeliness of detection (Table 1). Specificity was low however, 
suggesting the method may signal often, even when there is no 
outbreak. All methods had poor PPV, so when a method did signal, 
it was unlikely to be an outbreak. Among the spatial methods, the 
GLMM had the highest sensitivity (Table 2), though it signaled the 
most often. Of note, the methods performed better when the outbreaks 
were of longer duration (>5 days), both in detecting outbreaks at all 
and the timeliness of outbreak detection. The easiest method to code 
was the modified C2 and the most difficult was the Bayesian spatial 
model.

Conclusions
While there was variability in method performance, we feel there 

is no one best method for outbreak detection, as we have to weigh 

issues like signal frequency, specificity, and timeliness. We also 
found certain methods required time and skill to code, which is useful 
when deciding which methods to implement into a daily system. Our 
poor PPV may be explained in that methods are detecting unlabeled 
outbreaks given we used actual syndromic data. Our next step is to 
continue comparing results between these methods, and select the 
most promising methods to run prospectively in our current system.

Table 1. Metrics of the temporal models running at a fixed alert rate of 0.01 
(1 alarm/100 days)

*alert generated at p<0.01

Table 2. Metrics of the spatial models

Keywords
Evaluation; Syndromic surveillance; Aberration detection 
methodologies

References

1. Tokars, J.I., et al., Enhancing time-series detection algorithms for
automated biosurveillance. Emerg Infect Dis, 2009. 15(4): p. 533-9.

2. Kulldorff, M. and I. Information Management Service, SaTScan v9.1:
Software for the spatial and space-time scan statistic http://www.
satscan.org/, 2009.

3. Corberan-Vallet, A. and A.B. Lawson, Conditional predictive inference 
for online surveillance of spatial disease incidence. Stat Med, 2011.
30(26): p. 3095-116.

*Robert Mathes

E-mail: rmathes@health.nyc.gov    

Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * (1):e39, 201