Layout 1


ISDS Annual Conference Proceedings 2012. This is an Open Access article distributed under the terms of the Creative Commons Attribution-
Noncommercial 3.0 Unported License (http://creativecommons.org/licenses/by-nc/3.0/), permitting all non-commercial use, distribution, and
reproduction in any medium, provided the original work is properly cited.

ISDS 2012 Conference Abstracts

Evaluating Biosurveillance System Components using
Multi-Criteria Decision Analysis
Eric Nicholas Generous*1, Alina Deshpande1, Mac Brown1, Lauren Castro1, Kristen
Margevicius1, William Brent Daniel1 and Kirsten Taylor-McCabe2

1Defense Systems Analysis Division, Los Alamos National Laboratory, Los Alamos, NM, USA; 2Bioscience Division, Los Alamos
National Laboratory, Los Alamos, NM, USA

Objective
The use of Multi-Criteria Decision Analysis (MCDA) has tradi-

tionally been limited to the field of operations research, however
many of the tools and methods developed for MCDA can also be ap-
plied to biosurveillance. Our project demonstrates the utility of
MCDA for this purpose by applying it to the evaluation of data
streams for use in an integrated, global biosurveillance system.

Introduction
The evaluation of biosurveillance system components is a com-

plex, multi-objective decision that requires consideration of a variety
of factors. Multi-Criteria Decision Analysis provides a methodology
to assist in the objective analysis of these types of evaluation by cre-
ating a mathematical model that can simulate decisions. This model
can utilize many types of data, both quantitative and qualitative, that
can accurately describe components. The decision-maker can use this
model to determine which of the system components best accomplish
the goals being evaluated. Before MCDA can be utilized effectively,
an evaluation framework needs to be developed. We built a robust
framework that identified unique metrics, surveillance goals, and pri-
orities for metrics. Using this framework, we were able to use MCDA
to assist in the evaluation of data streams and to determine which
types would be of most use within a global biosurveillance system.

Methods
MCDA was implemented using the Logical Decisions® software.

The construction of the evaluation framework was carried out in sev-
eral steps: identification and definition of data streams, metrics and
surveillance goals, and the determination of the relative importance
of each metric to the respective surveillance goal being evaluated.
Sixteen data streams types were defined and identified for evaluation
from a survey we conducted that collected over 200 surveillance
products. A subject matter expert (SME) panel was assembled to help
identify the biosurveillance goals and metrics in which to evaluate
the data streams. To assign values for the metrics, we referenced prop-
erties of data streams used in currently operational systems.

Results
Our survey identified sixteen different classes of data streams: Am-

bulance Records, Clinic/ Health Care Provider Records, ED/ Hospi-
tal Records, Employment/School Records, Established Databases,
Financial Records, Help Lines, Internet Search Queries, Laboraotry
Records, News Aggregators, Official Reports, Police/Fire Depart-
ment Records, Personal Communication, Prediction Markets, Sales,
and Social Media.

Four biosurveillance goals were identified: Early Warning of
Health Threats, Early Detection of Health Events, Situational Aware-
ness, and Consequence Management.  

Eleven metrics were identified: Accessibility, Cost, Credibility,
Flexibility, Integrability, Geographic/Population Coverage, Granu-
larity, Specificity of Detection, Sustainability, Time to Indication, and
Timeliness.

Using the framework, it was possible to use MCDA to rank the
utility of each data stream for each goal.

Conclusions
The results suggest that a “one size fits all” approach does not work

and that there is no ideal data stream that is most useful for each goal.
Data streams that scored more highly for speed tended to rank more
highly when the biosurveillance goal is early warning or early detec-
tion, whereas data streams that scored more highly for data credibil-
ity and geographic/population coverage ranked highly when the goal
was situational awareness or consequence management. However,
there are several data streams that rank consistently within the top 5
for each goal: Internet Search Queries, News Aggregators, Clinic/
Health Care Provider records, ED/ Hospital Records, and Laboratory
Records and may be considered useful for integrated, global biosur-
veillance for infectious disease.

Keywords
evaluation; biosurveillance; multi-criteria decision analysis; data
stream; evaluation framework

Acknowledgments

This project is supported by the Chemical and Biological Technologies
Directorate Joint Science and Technology Office (JSTO), Defense Threat
Reduction Agency (DTRA)

*Eric Nicholas Generous
E-mail: generous@lanl.gov

Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 5(1):e157, 2013