Andrews, D. H. (2017, February 15). Training research and development: Retrospective of a career in 
the defense department. Acquired Wisdom Series, edited by S. Tobias, J. D. Fletcher, & D. Berliner. 
Education Review, 24. http://dx.doi.org/10.14507/er.v24.2213  

Training Research and Development:  

Retrospective of a Career in the Defense Department 

Dee H. Andrews 

Introduction 
I have had the privilege of spending 

a psychologist’s career serving the men and 
women who give service to our nation. 
Starting as an instructional psychologist, I’ve 
also been called upon to learn skills in the 
human factors and industrial/organizational 
psychology domains. I’ve been placed in 
numerous situations as a team member and 
leader where I learned how to work with 
multi-disciplinary teams of engineers, 
physicists, computer scientists, logisticians, 
and contract specialists. During 34 years as a 
Department of Defense (DoD) researcher 
and manager performing training research 

and five years working as a contractor since 
my retirement from the DoD, I can truly say 
it was a rare day when I was not enthused to 
go to work. I hope in this paper to show 
young educational researchers and 
psychologists why I was excited about my 
own career path and suggest reasons why 
they might want to consider a similar path 
for themselves. 

Our military forces must maintain a 
high level of knowledge and skill in 
performing increasingly complex tasks, with 
sophisticated tactics and equipment that 
require innovative approaches to mission 
solutions. Training and education are key 
contributors to mission accomplishment and 
success. New training methods and training 
equipment (e.g., simulators, training devices 
and computer-based instruction) must be 
continually designed, developed, assessed, 
and modified in order for personnel to learn 
and retain skills vital to this end.  

The DoD employs a large cadre of 
scientists of various stripes to perform 
Research and Development (R&D). Some 
scientists are in uniform, some are civil 
servants (my path), some are civilian 
contractors, and some are university 
professors and personnel. They cover many 
domains, including R&D for space vehicles, 
air vehicles, propulsion, materials, sensors, 
directed energy, weapons, and cybersecurity. 

The DoD has committed significant 
research funding to behavioral/human 
factors, which includes educational and 
training research. Some of the broad 
categories of psychological research are: 

http://dx.doi.org/10.14507/er.v24.2213


Acquired Wisdom/Review 2 

A. Cognitive Readiness (“Cognitive
readiness is the mental preparation
(including skills, knowledge, abilities,
motivations, and personal
dispositions) an individual needs to
establish and sustain competent
performance in the complex and
unpredictable environment of
modern military operations,”
Morrison and Fletcher, 2002, p. 1)

B. Controls and Displays (how to
establish the best human system
interface)

C. Design: Tools and Techniques
D. Extreme Environments (operating in

extremes of cold, heat, etc.)
E. Human Factors Psychology and

Human Systems Integration
F. Human Modeling & Simulation
G. Human Performance Measurement
H. Personnel Selection & Classification
I. System Safety, Health Hazards,

Survivability

Training may best fit into the cognitive 
readiness and human factors psychology 
categories; however, it has a place in all the 
behavioral categories above. I’ve always 
been fascinated by the way people get things 
done, especially those that require complex 
functions and tasks. Military personnel have 
many tasks to accomplish. Some are done 
on a regular basis such as maintenance of 
equipment (e.g., aircraft and avionics), and 
some are done only rarely (e.g., 
troubleshooting equipment failures and 
planning operations). However, all these 
functions and tasks require knowledge and 
skills, many of which can only come through 
formal and informal training, and this is why 
I was so attracted to a career of improving 
training in the military.  

There is a section toward the end of 
this paper with 15 lessons learned from my 
career. The lessons learned are best 
understood by first reading the content 
preceding it, but I believe the lessons 
learned will be useful to a psychologist in 
the early stages of their career even if the 
preceding material is not read. Although the 
examples I provide in the article are from 
the military context, many of the principles 

and lessons learned that I included can be 
applied to the civilian world just as well. 

My Educational Psychology Training 
I became interested in the field of 

psychology in high school (I graduated in 
1970) after I came across Sigmund Freud’s 
(1899) Interpretation of Dreams in the library. I 
was fascinated that Freud had developed a 
method for interpreting dreams through his 
psychoanalytic experience. I still look with 
fondness upon the volume even though I 
long ago came to believe that the validity of 
Freud’s methods and conclusions were 
suspect. Nonetheless, the book got me 
thinking about the awesome nature of that 
three pounds of wetware we call the brain.  

I majored in psychology as an 
undergraduate at Brigham Young University 
(BYU), where I graduated in 1976. I took 
the requisite freshman courses, including 
psychology 101. I then decided to go on a 
two-year church mission to England where I 
learned a tremendous amount about 
teaching, learning, and serving God and my 
fellow humans. It was a significant event in 
my life, and I found myself applying what I 
learned to my career on many occasions. 
The mission experience convinced me that I 
wanted to study psychology. I then returned 
to finish my bachelor’s degree at BYU in 
psychology where I had greatly enjoyed 
studying different aspects of the field. I was 
first drawn to experimental psychology and 
enjoyed helping professors develop, conduct 
experiments and analyze experimental data. 
I conducted operant conditioning 
experiments with rats and chicks, and 
became very familiar with the operant 
conditioning Skinner box (Skinner, 1959). I 
had a professor, Paul Robinson, who was 
writing one of the first experimental 
psychology textbooks targeted at 
undergraduates, and he allowed some of us 
to help him edit the text (Robinson, 1976). I 
loved the idea of studying learning in all its 
manifestations. After those experiences I 
was intent on pursuing a career in 
experimental psychology. 

Later in my undergraduate career I 
was exposed to a branch of psychology that 



Retrospective of a career in the defense department  3 

was devoted to improving human learning 
by combining psychological learning 
principles with educational concepts focused 
on human instruction. The field goes by a 
variety of names: instructional design and 
development, instructional science, 
instructional technology, performance 
improvement, instructional systems, and 
others. The field is eclectic in many ways 
and uses a variety of learning and teaching 
methods paired with instructional 
technology to make human learning more 
scientific and systematic. The approach is 
now widespread in training programs of all 
types for schools, universities, corporations, 
government, and the military. Learning 
theory is a fundamental part of the field.  

After working with a number of 
BYU professors, including Drs. David 
Merrill, Harvey Black, and Rex Wadham, I 
decided to pursue graduate training in the 
field. At the time, there were a number of 
good programs in that domain. I chose to 
apply to and attend what was then called the 
Instructional Design and Development 
(IDD) Master’s program at Florida State 
University (FSU). It was highly 
recommended by the professors I knew in 
the program at BYU, and a former alum of 
the BYU graduate program in Instructional 
Science, Dr. Richard Boutwell, was a 
professor in the FSU program. The FSU 
IDD program had some of the best known 
experts in the field including: Drs. Robert 
Gagne, Leslie Briggs, Robert Branson, 
Roger Kaufman, and Ernest Burkman. All 
of the professors had backgrounds in 
education and psychology. As someone 
interested in training, I was especially drawn 
to the fact that of the 14 professors in the 
program, all but one had had experience 
outside of the halls of academia. They 
worked for the government, the military, 
corporations, and research institutions. I was 
impressed by their knowledge about 
designing and conducting effective training. 
In addition, most of the professors had 
degrees in experimental psychology, so we 
received a heavy dose of skills concerning 
research design and statistical analysis all 
aimed at allowing us to develop and use a 

research base to design better instructional 
techniques.  

In the late 1960s and early 1970s the 
FSU IDD program was a leader in 
developing systems and models for 
constructing systematic instruction. The U.S. 
Army funded a significant effort at FSU 
which resulted in the Interservice 
Procedures for Instructional Systems 
Development (IPISD; Branson et al., 1975.)  
The IPISD model, still in use today, has five 
phases that lead to quality instructional 
design and development. The phases are:  

o Analysis
o Design
o Development
o Implementation
o Evaluation

The IPISD model is often referred to as the 
ADDIE model taking the first letter of each 
of the five phases.  

Analysis. In the analysis phase of 
the ADDIE model the instructional 
problem is identified. The instructional 
goals, success metrics, and overall objectives 
are established. Information regarding the 
learner, the “training audience,” and the 
training environment is also identified 
during this phase. It includes learners’ 
preferences, demographics, and existing 
knowledge and skills. 

Design. The design phase of the 
ADDIE model nails down learning 
objectives, instructional methods and 
activities, storyboards, content, subject 
matter knowledge, lesson outlines, and 
media assets. 

Development. The development 
phase of the ADDIE model is where 
instructional designers develop the content 
and learning interactions outlined in the 
design phase. During this phase, content is 
written and graphics, audio, and 
photography are also produced and 
assembled.  



Acquired Wisdom/Review  4 

 Implementation. 
During the implementation 
part of the ADDIE model, 
the instructional designer 
delivers the content and 
materials to Learning 
Management Systems (LMS) 
or directly to the trainer for 
live training events. The 
instructional designer also 
provides training needed by 
trainers, facilitators, subject 
matter experts or instructors.  
 

 Evaluation. During 
the evaluation phase of the 
ADDIE model, the 
instructional designer determines what 
success will look like and how it will be 
measured. Often times, the evaluation 
consists of two phases: formative and 
summative. Formative evaluation is iterative 
and is done throughout the design and 
development processes. This occurs 
throughout the ADDIE process. Summative 
evaluation consists of tests that are done 
after the training materials are delivered. The 
results from these tests help to inform the 
instructional designer and stake holders 
about whether or not the training 
accomplished the original goals outlined in 
the analysis phase (Instructional Design 
Central, 2016).  
 Each of the five phases has sub-
phases which consist of steps that, if 
followed as closely as possible, hold promise 
of producing job incumbents who can do 
their jobs competently. I have used the 
ADDIE model throughout my career and 
found it extremely helpful in the design 
process. A powerful feature of the ADDIE 
model, and just about every other systems 
approach to training design, is that the 
phases and sub-phases are iterative. (Figure 
1) That is, revisions to the instruction are 
made based on data collected during the 
design, development and evaluation process. 
The designer doesn’t have to go in lockstep 
from the first phase to the last, but can 
move forwards and backwards as the 
process proceeds. 
 

 
 
Figure 1.  Iterative nature of the ADDIE 
model (Instructional technology services, 
Texas A&M University. Retrieved from 
itsinfo.tamu.edu)  
 
 I was fascinated by the ADDIE 
model and other models that apply a 
systems approach to instructional 
development. Models for training and 
education have been developed and used by 
universities, companies, research centers, 
and many other organizations. As I 
researched those in existence in the 1970s, it 
seemed that more and more were being 
developed all the time. Some of the models 
were completely original while most them 
were modifications of models that already 
existed, such as the IPISD model developed 
at FSU. Modifications were often made for 
specific challenges or roadblocks that were 
encountered by different organizations 
when designing instruction.  
 I was curious as to why there were 
so many, so I started to make a collection of 
them. I soon found dozens of them. I 
thought it would be helpful to the 
instructional design and development 
community if I were to publish an article 
that related the many models, their origins, 
underpinnings, purposes and uses, and 
documentation. A fellow student, Ludy 
Goodson, and I set about finding and 
analyzing over 70 different models. We 



Retrospective of a career in the defense department                       5 

 
finally settled on 40 models that were 
categorized into four main and 11 sub-
categories. One of our major findings was 
that no model had information that allowed 
us to state whether the model satisfied all 
eleven sub-categories of our taxonomy. In 
many cases we believed the models might 
have addressed all 11 sub-categories, but the 
models’ documentation was deficient. I 
suspected it at the time, and during my 
career I experienced a number of occasions 
when insufficient documentation made it 
difficult to use a particular model. We 
published an article on our analysis in 1980 
(Andrews & Goodson, 1980). 
 I had the great privilege of having 
Dr. Robert M. Gagne serve as my major 
professor at FSU. Dr. Gagne passed away a 
number of years ago, but he is still 
recognized as a worldwide scholar on 
learning science, and his books are still 
widely read. It is from him that I gained a 
greater understanding of learning theory and 
how it could be applied to form the 
foundation of effective instruction. He 
introduced me to the concept of simulation-
based training with a seminal article he 
published in the American Psychologist in 
1954 titled, “Training devices and 
simulators: Some research questions”. 
Despite over 60 years of research on 
simulation-based training since the article 
was published, a number of Gagne’s 
fundamental questions have still not been 
completely answered.  
 Two of his main contributions were 
his five-domain learning taxonomy, and his 
theories about the conditions of learning 
(Gagne, 1985; Gagne, Briggs, & Wager, 
1992). Both drew heavily on learning theory 
and empirical data. The five domains of the 
taxonomy are: 
 

o intellectual skills (discriminations, 
concrete concepts, abstract 
concepts, rule using, and problem 
solving),  

o cognitive strategies (skills by means 
of which learners regulate their own 
learning processes)  

o verbal information (facts and 
knowledge)  

o motor skills 
o attitudes  

 

Gagne and his colleagues set forth a variety 
of recommendations for building 
instructional systems that were based on the 
learning domains and the concept of 
learning conditions.  
 My time at FSU convinced me that I 
wanted to go into the training field. Almost 
every project I worked on after graduate 
school was heavily influenced by what I 
learned from the faculty at the FSU IDD 
department. 
 
Work as a Navy Training Developer and 
Training Researcher 
 I was fortunate to have a number of 
job offers coming out of my Ph.D. program 
in 1979. I wasn’t interested in going the 
professor route although I do enjoy 
teaching. (I’ve taught as an adjunct professor 
for 11 universities since I graduated.) 
Primarily, I wanted to get my hands into 
practical training development and training 
research as soon as I could. Some of the job 
offers were at university instructional 
development centers, others with 
industrial/corporate concerns, but due to 
my interest in all things military I decided to 
go to work for the Navy in a civil service 
job.  
 I went to work as a GS-11 for the 
Naval Training Systems Center (now called 
the Naval Air Warfare Center – Training 
Systems Division) in Orlando, Florida, in 
1979. I started in the Training Analysis and 
Development Division. As the name 
implies, we were tasked with analyzing 
training requirements for new and existing 
training, and developing training system 
designs to meet the requirements. This is 
where I had my first interactions with large 
teams of designers, engineers, logisticians, 
computer programmers, subject matter 
experts, and others. It was a great experience 
learning to work with experts who came at 
the problem from different orientations. I 
learned that while the instructional systems 



Acquired Wisdom/Review 6 

business tries to be as data driven as possible 
in using front-end analysis data, other 
disciplines are not always inclined to trust 
front-end data, whether it be analytically or 
empirically derived, and least of all data 
provided by folks with educational or 
psychological backgrounds like mine. I was 
discouraged about this at first, after putting 
lots of time into what I believed to be 
quality data based front analysis, but I soon 
realized that I would have to adjust to other 
perspectives and expectations if I were to 
get my analyses used in training 
development for actual military jobs.  

Developing training systems that will 
deliver the required training is a complex 
process. That is true in both technical 
development itself and management of 
development. In some cases, we used teams 
of designers from the Navy (civil servants 
and uniformed personnel), sometimes we let 
contracts and had contractor designers 
perform the work, and often it involved a 
combination of in-house and contractor 
personnel. It depended on the size of the 
project and whether the Navy had the 
money to hire contractors for a particular 
job. I discovered that a combination of in-
house and contractor personnel from 
various disciplines made for fascinating team 
dynamics, but we were all able to pull 
together for the basic objective – trained 
and effective personnel. Later in my career 
as I became the team manager for these 
large teams this early experience was vital.  

I spent three years in the Training 
Analysis and Development Division and 
worked on some fascinating projects. I was 
able to use all I learned in graduate school 
about analysis, design, and evaluation. I 
especially called upon what I knew in 
instructional media selection. The military is 
very equipment based. The reader is of 
course familiar with the tools of the trade, 
such as tanks, ships, airplanes, etc. Each of 
these has many primary systems, 
subsystems, and component parts. The 
military has to train personnel to both 
operate and maintain these many different 
systems. Hence, the military’s need for 
simulators and training devices to aid in the 

training. My job was to help determine what 
training requirements had to be met, and, 
just as importantly, which requirements had 
to be trained in formal training settings (the 
military calls these formal settings, “school 
houses”), and which could be met 
informally through on-the-job training.  

A word about the difference 
between “training devices” and 
“simulators”. Often these two terms are 
used interchangeably, but I always tried to 
help people understand that there is a 
significant difference (Andrews, 1988). A 
simulator replicates equipment and 
environments. So, we would build an exact 
replica of an aircraft cockpit, with the 
appropriate switches, buttons, knobs, dials, 
instruments, etc. In addition, we now can 
build out-the-window visual displays that 
have a high degree of fidelity with the real 
world outside the cockpit. We can also 
precisely simulate motion cues via large 
motion platforms, radio calls can be exactly 
replicated, and even odors can be produced 
in the simulated cockpit such as the smell of 
an electrical fire. All of this can be done at 
costs that have declined steadily over the 
years, but still constitute a significant 
amount of money. However, simulating all 
the equipment, environment, and cues 
exactly may not produce the skills and 
knowledge we desire trainees to acquire. On 
quite a few occasions I’ve come across 
simulators that were great at replicating cues 
but didn’t lead to systematic learning. They 
lacked tools to help the trainee and the 
instructor measure training performance and 
advance learning. Use of the ADDIE model 
can lead to the development and 
implementation of these tools for 
simulators. It is when we can effectively 
combine good instructional design and 
training features with good simulation that 
we produce a training device.  

Years ago Semple, Cotton, and 
Sullivan (1981) provided a representative list 
of necessary instructional features:  

Instructional support features 
include … hardware and software 
capabilities that permit instructors to 
manipulate, supplement or otherwise 



Retrospective of a career in the defense department  7 

control student learning experiences. 
The instructional features … are 
freeze; automated demonstrations; 
record and replay; automated cuing 
and coaching; manual and 
programmable sets of initializing 
conditions; manual and 
programmable malfunction control; 
aircrew training device mounted 
audio visual media; automated 
performance measurement; 
automated performance alerts; 
annunciator and repeater 
instruments; closed circuit television; 
automated adaptive training; 
programmed mission scenarios; 
automated controllers; graphic and 
text readouts of controller 
information; computer controlled 
threats; computer managed 
instruction; recorded briefings; 
debriefing aids; and hardcopy 
printouts. (page ii) 

A good set of questions and answers about 
instructional features and simulation in 
general can be found in Best, Galanis, Kerry, 
& Sottilare (2013). The features listed above 
can turn a simulator into a training device. 
Too often simulators lack the above 
capabilities. They represent cockpit and 
visual cues precisely, but lack features that 
enable learning. Oftentimes, the features are 
not included because a proper front end 
analysis, as defined in the ADDIE model, 
was not performed to identify the specific 
tasks to be trained. Consequently, the 
necessary instructional features were left out 
of the design. Sometimes not enough money 
is allocated to design and add the necessary 
instructional features. Too many simulators 
are delivered to training school houses that 
don’t provide what instructors and trainees 
need, and the simulators end up being used 
as “room dividers” and “plant hangers”. I 
exaggerate with those phrases, but poor or 
no front-end analyses and lack of 
instructional features and measures can 
sabotage a simulator that could be used for 
training. So, our job was to argue to the 
simulator design team, with front end 

analysis based facts, about how a training 
device should be designed that would truly 
deliver effective training with instructional 
features.  

I only half-jokingly have said that we 
too often skip a proper front end analysis, 
build the simulator, then go perform a “rear 
end” analysis where we try to figure out 
what functions and tasks the simulator can 
actually be used for to train the trainees. I 
call this “rear end analysis” approach 
“training proctology”. Convincing decision 
makers to allow enough time and resources 
to conduct a proper front end analysis is 
imperative if we want to avoid this 
regrettable approach.  

Throughout this paper I have added 
pictures and graphics of training devices and 
simulators to help illustrate the types of 
systems I worked on while in the Training 
Analysis Division of the Naval Training 
Systems Center. Some of the pictures depict 
actual devices I helped develop, and others 
depict a general type of device that I worked 
on. In some cases, I couldn’t find an actual 
picture of a device that I worked on because 
they are now out of the inventory.  

Maintenance Training 
This instance concerns a MK-92 

Fire Control system on a Navy Frigate. A 
Fire Control system consists of a set of 
equipment, including radar and various 
computer systems that allow ships’ weapons 
to be accurately aimed and fired.  

The maintenance training device we 
developed for the fire control system 
allowed maintenance personnel to learn how 
to perform prescribed maintenance 
functions and troubleshooting. It allowed 
trainees to use both real and simulated test 
equipment such as voltmeters, oscilloscopes, 
ohmmeters, and multi-meters to test various 
parts and pieces of the simulated electronic 
equipment. After our analysis, we 
considered performing all of the training via 
Computer Assisted Instruction (CAI), but 
finally determined that the solution should 
be a combination of CAI with flat panels for 
indicating test points so that the trainees 



Acquired Wisdom/Review  8 

could connect the actual test equipment to 
the test points and take readings.  
 Figures 2 and 3 depict the types of 
electric test equipment used to test many 
types of electronic gear. Figure 4 (next page) 
depicts a training device that allows trainees 
to learn how to test various trouble spots 
while also learning to perform routine 
preventive maintenance functions. It 
measures trainee performance and keeps 
detailed training records for the maintenance 
instructors. 
 
 

 
  
Figure 2:    Actual Voltmeter  
  
 When I first went to work for the military 
there was a thriving line of research that 
sought to determine the effectiveness of 
maintenance training devices. For example, 
Orlansky and String (1981) published a 
report that summarized 12 such studies and 
found that maintenance training devices 
were just as effective for training 
maintenance tasks as using the actual 
equipment. In addition, these devices were 
less expensive to procure than using actual 
equipment for training. Through the years 
since the Orlansky and String report there 
have been a number reports with essentially 
the same finding.  

 Maintenance training research was a 
“hot” research topic from the 1980s to the 
early 1990s, and all of the services’ (Army, 
Navy, Air Force) training research 
laboratories had large research programs. 
Alas, interest and work in this vital area has 
dropped off considerably since then, 
although there are some exceptions. That is 
unfortunate because maintenance is a critical 
component of mission success, and 
important research topics in maintenance 
remain to be addressed.  
 

  
Figure 3:   Simulated test equipment   
 
 An instructional challenge with 
maintenance training. Maintenance on 
complex systems like the Fire Control 
System discussed above can be divided into 
two types: field and depot. Field level 
maintenance is performed at the site where 
the equipment or system is used. Field 
maintenance can be either routine or it can 
consist of troubleshooting a faulty system. 
Field level is most often done by a 
uniformed military member of the training 
audience for whom the training was 
developed. Depot level maintenance is 
performed in a location removed from 
where the system is used for its mission. 
Depot level maintenance is performed either 
in a repair facility run by the military or in a 
factory. Depot level maintenance is 
performed by highly trained and 
experienced personnel, be they military or 
contractors.  
 The true challenge in training field 
level maintenance personnel is 
troubleshooting system faults and failures. 
Routine maintenance is highly 
proceduralized and documented in technical 
manuals, which maintainers will have in 
hand—typically in electronic form. 



Retrospective of a career in the defense department  9 

Maintainers perform routine maintenance 
frequently enough to prevent a steep 
forgetting curve for what they have learned. 
However, fault troubleshooting is done only 
when the fault appears, which will be less 
frequent. Forgetting curves for fault 
troubleshooting skills may be steep because 
they are so rarely required. Steep or not, 
these curves are needed to properly schedule 
refresher, or sustainment, training. As I have 
worked on maintenance training programs, a 
key requirement has always been to teach 
basic principles about how complex systems 
operate. Troubleshooting these systems can 
follow different routes depending on the 
problem. It is usually not possible to teach 
all troubleshooting routes trainees may 
encounter in the field, so it is important that 
they have a fundamental understanding of 
basic electronic and mechanical skills. 
Maintainers go to a basic school to learn 
such principles before they receive advanced 
training for a specific system at a specialized 
school. The basic school usually lasts about 
three to four months. The advanced school 
for a particular system can last for many 
months beyond the basic school.  

Because the time spent in both  
schools is relatively short and considering 
the complexity of many systems,  

instructional designers who are designing 
training for the advanced school are well 
advised to determine just how many  of the 
underlying principles trainees retain after 
they have finished the basic school. It may 
not be extensive since the basic school is 
relatively short. In that case, some of those 
underlying principles may not be re-taught 
in the advanced school so that the trainees 
learn strategies for troubleshooting 
problems they have never seen before when 
they get to the field. These strategies should 
be based on sound fundamentals as opposed 
to what is called “Easter egging” – a 
troubleshooting approach wherein the 
technician in the field isolates the problem 
down to a few possibly faulty components 
and replaces them one at a time to see if that 
fixes the problem. Such a strategy might 
work, but it is inefficient and may actually 
damage the system.  

A key decision designers of training 
devices and other instructional techniques 
must make is how much “lock step” versus 
“free play” to build into the system 
(Andrews & Windmueller, 1986). As the 
name implies, lock step instruction takes the 
student down a fixed path of lessons  

Figure 4:   Virtual Maintenance Trainer     



Acquired Wisdom/Review  10 

without opportunities to deviate. Such  
instruction is good for regularly scheduled 
routine maintenance task training. Free play 
allows the lesson structure to take the 
student where their good actions and errors 
would take them on the actual equipment. 
Such free play systems provide remedial aids 
to the trainee after errors, but the designer 
may structure the lessons so that it allows 
the trainee to go down a wrong path a way 
before correction is provided.  
 

Firefighting Training   

One of the most hazardous tasks 
Navy personnel have to perform is fighting 
fires that break out on ships, submarines, 
and airplanes. What makes the task 
especially dangerous is that these fires are 
often in very tight spaces below deck. The 
space to maneuver firefighting personnel 
and equipment is limited, and the lighting 
may not be operable in those spaces. In 
addition, due to personnel limitations on 
vessels, specialized firefighters are rarely on 
board. The fires are fought by personnel 
who normally have other duties as their 
prime responsibility so that they seldom get 
a chance to practice firefighting for different 
types of fires.  

Traditionally firefighters were 
trained using fire pits filled with 
flammable substances that when 
lighted would allow trainees to get 
a sense of how fire retardant could 
be applied to put out a fire. 
However, the traditional approach 
had three major disadvantages, the 
fire pits polluted the environment, 
the fires were not controllable once 
lit, and trainees were not given 
realistic firefighting venues that 
duplicated the cramped conditions 
they would find aboard ships. A 
solution was to build a re-usable 
structure in which non-polluting 
fires could be simulated using 
substances like propane. Figure 5 
shows such a simulator on which I 
worked. The reader can tell from the many 
burned areas around the openings to the 
structure that some fierce fires have been 

ignited inside. Figure 6 shows trainees 
fighting such a controllable fire in a room 
similar to the interior of a vessel. Instructors 
typically would be in the space with the 
trainees instructing them (loudly and 
physically) by showing them how to man the 
hoses. In addition, instructors in a control 
room turn the flames on and off to control 
the intensity of the fire. They typically have 
cameras that not only allow them to see the 
progress of the fire fight, but also are used 
to record the action for later post training 
debriefs with the trainees.  

 
 

 
 

Figure 5. Firefighting training structure that 
uses clean propane gas  
 

Figure 6. Inside a firefighting training 
structure 

 



Retrospective of a career in the defense department                       11 

 
Firefighter training now includes realistic 
virtual environments. Figure 7 shows an 
example. In this approach, actual fire 
retardant is not dispensed by the trainees, 
but is visually simulated. However, the 
effectiveness of different retardants can be 
simulated for different types of fires.  

Fire fighter training devices can save 
lives, money, reduce pollution, and, with the 
addition of instructional features, they 
improve training effectiveness.  

Figure 7.   Virtual firefighting trainer  
 
Instructional challenges with 

firefighting training. Fidelity is a term 
heard often in the training device and 
simulator arena. Fidelity refers to several 
different issues. Physical fidelity typically 
means that the simulator “looks” like the 
real piece of equipment. For example, all the 
switches, knobs, and dials on a simulated 
fire hose control panel are in the same 
layout and locations as the real control 
panel. Functional fidelity typically means the 
simulator performs like the actual piece of 
equipment performs. So, when I trigger the 
simulated fire hose it sprays out a simulated 
stream or image of stream of fire retardant 
material. Finally, psychological fidelity refers 

to whether the simulator “feels” like the real 
piece of equipment when it’s used. This last 
type of fidelity can be very difficult to get 
right because the expert may not be able to 
articulate what is wrong about the way 
simulator feels to them (Hays & Singer, 
1989).  

As we worked on various training 
solutions for the fire fighter instructional 
system we had to think about the best 
medium for the different tasks to be trained.  

We developed a media selection tool that 
allowed us to consider the main 
characteristics of the task and determine 
where best they could be trained. Our media 
consisted of classroom instruction and 
demonstrations by an instructor, computer 
assisted training, a fire structure where 
different types of fires could be simulated 
and different types of fire retardant could be 
sprayed, and finally a pit fire (even when we 
did the analysis, government authorities 
were becoming increasingly concerned 
about the pollution caused by pit fires.) 
Today one would also have to consider the 
media of virtual reality, as shown in Figure 
7, and augmented reality.  

Our media selection tool helped us 
to make reasonable decisions about which 



Acquired Wisdom/Review  12 

tasks were taught where. After trying the 
training in a formative evaluation setting, we 
then had the information to move some 
tasks to more effective training media within 
our suite of possibilities. These fidelity 
selection decisions attend all instruction, but 
they are especially important in the training 
device and simulator business. The fidelity 
definitions and discussion apply to all the 
training devices and simulators discussed in 
this paper.  

 
Team Training Devices – 
The 14A12 Anti-
submarine team trainer  
 Teamwork plays a 
major role in military 
operations as in most other 
activities. Whether it is a 
two-man machine gun crew 
all the way up to a major 
exercise involving 
thousands of participants, 
teamwork and its attendant 
team training is of major 
concern. Task work is what 
an individual does as part 
of the total team function. Training devices 
have been built for many task work training 
requirements, such as the maintenance 
training device described above. Paris, Salas, 
and Cannon-Bowers (2000) provide a good 
definition of teamwork: 
 Teams are more than collections of 
individuals and teamwork is more than the 
aggregate of their individual behaviors. 
Moreover, one cannot simply label a group 
of individuals a ‘team’ and expect that they 
will perform as a team. Specifically, one may 
conceive a team to be a distinguishable set 
of two or more people who interact 
dynamically, interdependently, and 
adaptively toward a common and valued 
goal/objective/mission, who have each 
been assigned specific roles or functions to 
perform, and who have a limited life-span 
membership (p. 1052). 
 For example, an individual fighting a 
fire must learn how to operate the nozzle of 
the hose dispensing the fire retarding agent. 
That is task training for an individual. 

However, when that individual works in a 
team of firefighters, each individual not only 
has to perform individual tasks, but must 
also learn to perform interdependent, 
coordinated team tasks so that the team 
works as one.  
 I worked with a number of team 
training devices in the Navy. A good 
example was the 14A12 Anti-submarine 
warfare (ASW) team training device shown 
in Figure 8.  

Figure 8.    Device 14A12 Anti-submarine 
warfare training device  
 
 A primary mission for people in 
anti-submarine warfare (ASW) is to protect 
the fleet from underwater threats. ASW is a 
difficult task. Submarines are engineered to 
operate as quietly as possible to reduce the 
chance of detection from sonar as much as 
possible. There are two primary ways that 
sonars can be used. One is to use active 
“pinging” where a pulse of energy is sent 
into the ocean that bounces back to the 
sonar when it hits a hard object like a 
submarine hull. The drawback with active 
signal detection is that the adversary can 
hear the sonar’s ping and locate the sending 
ship just as readily as the sending ship can 
locate the adversary. The other approach is 
to use sonar that passively listens for any 
noise that comes to the sonar. Because that 
approach is stealthier it is often preferred by 
commanders for detecting possible 
adversaries.  



Retrospective of a career in the defense department                       13 

 
 Device 14A12 (Fig. 7) simulated the 
sonar stations for a number of different 
sonars, as well as other equipment used to 
analyze signals and launch weapons of 
different types when necessary. A typical 
sonar team for all the stations usually 
includes two to three dozen people. Training 
scenarios in a team trainer like the 14A12 
can last for several minutes to two to three 
hours depending on the training objective.  
 

 Instructional challenges with 
team training. A major goal of team 
training is to help the team form a shared 
cognition (Cannon-Bowers & Salas, 2001; 
Salas & Fiore, 2004). Shared cognition is 
what allows a team to build a shared mental 
model (Cannon-Bowers, Salas, & Converse, 
1990). Each member of the team should 
have an understanding at some level of the 
tasks that other team members must 
perform if the team is to successfully 
accomplish its mission. Each team member 
should know what pieces of information the 
other team members need from that team 
member, and when they need it. This 
included the teamwork required for 
communication. Team training device 
simulators can be very helpful in allowing 
the team build shared cognition and 
situational awareness. Endsley (1995) 
defines situation awareness as, “knowing 
what’s going on … the perception of the 
elements in the environment within a 
volume of time and space, the 
comprehension of their meaning and the 
projection of their status in the near future” 
(p. 36).  
 The individual members of the team 
must be competent in their own task work, 
and also in the teamwork capabilities that 
require an understanding of what the other 
team members require and are doing (Salas, 
Cooke, & Rosen, 2008). The training 
challenges for teams in the military are 
especially acute because military teams 
seldom stay together very long. For 
example, in the anti-submarine warfare case, 
a team of a dozen or more sonar operators 
and team chiefs may only be together as an 
intact team for a short time. Members may 

only be together for a few weeks or months 
before one or more team members departs 
due to new assignments, retirements or 
separations from the service, combat 
casualties, or one of many other possibilities. 
 Team training devices, like the 
14A12 are located at training sites in the US, 
but many months may pass between a 
team’s visits to the training site, either 
because the ship is deployed or the team 
members are tending to needed equipment 
upgrades and other duties when the ship is 
in port. Even then, there is no guarantee 
that all team members will attend the 
training session together because some 
members may be absent for various reasons. 
Building solid team cognition in the team via 
training is essential and difficult.  
 One potential solution to the 
problem is to embed training capability into 
the actual equipment the team uses on board 
ship (or, aircraft or tank, etc.) so teams can 
conduct operations training so they can have 
the advantage of instructional features and 
performance measurement whether in port, 
at sea, or while preparing for a mission. The 
military services have installed embedded 
training capabilities in some cases, especially 
the Navy, but overall the use of embedded 
training has been far lower than originally 
desired (Alexander et al., 2014).  
 Much team training happens in the 
field, not in the school house. Therefore, the 
instructional tasks normally assigned to a 
full-time instructor fall to leaders within the 
unit. These leaders often lack formal 
instructor training. They can, nevertheless, 
be effective instructors, but receiving 
instructor training improves the chances of 
effectiveness. In some units such training 
for instructors may be given, but often it is 
informal and less effective. Use of 
computer-based training could be helpful to 
these senior personnel in learning 
instructional tasks, but I have rarely seen it 
used.  
 
Ship Handling Training  
 After three years in the Training 
Analysis and Development office, I moved 
to the Human Factors Research and 



Acquired Wisdom/Review  14 

Development Laboratory within the Naval 
Training Systems Center in 1983, and 
received a promotion to the Civil Service 
Grade of GS-13, equivalent to a Major in 
the Air Force or Army, and Lieutenant 
Commander in the Navy. I was excited to 
move to an R&D position because it 
allowed me to devote full time to exploring 
new theories, concepts, and technologies. 
One of the great advantages as a scientist in 
the Department of Defense is the ability to 
modify budgets for laboratory equipment, 
including computer capabilities that a 
scientist can access. Although the scientist 
has to compete within the DoD for funds, 
I’ve always perceived that task to be easier 
than it is for university professors because 
the government scientist is closer to decision 
makers and the source of funding. In my 
career as a government scientist, I seldom 
lacked resources after I made a cogent case 
for how the funding would enhance mission 
effectiveness and success.  
 Another interesting piece of research 
I pursued was developing a training system 
for Navy personnel who were responsible 
for guiding huge vessels safely both into and 
out of harbors and while they are on the 
open water. Improved training was needed 
because high profile accidents had increased. 
These accidents caused significant damage 
to ships (both own ship and other vessels), 
and to piers and other objects in the water. 
There were also some ship groundings. 
Accident investigations revealed that a 
number of the accidents involved a shortfall 
in ship 
handling 
knowledge 
and skills on 
the part of 
the officers 
who were 
“conning” 
the ship 
from its 
bridge. 
These 
officers are 
responsible 
for 

determining the proper course for the ship 
and giving the crew orders for speed and 
direction. These are not easy skills to learn 
because the huge size of many Navy vessels 
makes their responses to commands 
sluggish, ocean conditions are always 
changing, sand bars shift, the behavior of 
other vessels may be erratic, etc. I was 
tasked with analyzing the available training, 
and conducting an R&D program to 
develop alternative training solutions.  
 At the time of the R&D program 
there were several large “full bridge” 
simulators that were being used in the US to 
train mariners in ship handling skills. The 
surface Navy (ships traveling on rather than 
under water) was not making regular use of 
these training devices (Figures 9 and 10).  
 

Figure 9 (top). Bird’s Eye View of a Ship 
Handling Training Device  
 
Figure 10. (bottom) Ship Handling Training 
Device – scene looking out window shown 
in Figure 9.  



Retrospective of a career in the defense department  15 

Junior submarine officers were using a 
device at the Maritime Academy on a regular 
basis. The surface officers received ship 
handling training in relatively small boats 
during their initial training, but they did not 
have access to the modern ship handling 
training devices of the time.  

With the aid of contractor personnel 
who had ship handling expertise, I 
performed a task analysis using the ADDIE 
method to identify necessary ship handling 
skills and the knowledge required to 
perform them. This task analysis was 
validated by Navy ship handling experts. We 
then set about developing a training device 
that was considerably 
smaller than the large ship 
handling simulators then 
available. Not only was the 
goal of the R&D to provide 
effective training, but the 
Navy was also interested in 
developing a more 
affordable solution than the 
available multi-million-dollar 
training device. We 
developed a training device 
that had a footprint small 
enough to fit on a table top. 
Its scenes were driven by a 
computer generator representing the various 
ship controls and displays one would find 
on any ship bridge. We called it a part-task 
trainer because it did not simulate all of the 
controls and displays represented on the full 
task training devices as shown in Figures 7 
and 8 above, but it did simulate tasks that 
the task analysis showed to be those most 
crucial for successful ship handling. We 
conducted a trial to determine if trainees 
learned the necessary skills while using the 
part task training device, and to see if ship 
handling instructors gained enough 
information on the learning state of the 
trainees to make the device worthwhile. The 
answer to both goals was affirmative 
(Hanley & Andrews, 1987).  

The analysis and R&D led the Navy 
to work with a ship handling simulator 
company to build a simulator where Navy 
officers received their initial surface training. 

The Navy then leased simulator time for the 
trainees to train as needed. The lease 
approach allowed the Navy to avoid a multi-
million-dollar buy in any particular year.  

Today, the Navy makes use of 
different simulation approaches for ship 
handling training, including virtual reality 
technology so that the Officer of the Deck 
can train while looking through a virtual 
reality visor (Figure 11) to see the visual cues 
necessary to make ship handling decisions. 
It makes use of the part-task training 
concepts we developed many years earlier 
with R&D on part-task training.  

Figure 11. Virtual reality ship handling trainer 
using head mounted visor 

Instructional challenges with ship 
handling training. Ships are relatively slow 
moving. Teaching relative motion is a 
challenge for trainees who lack an innate 
understanding of the concept.  
Relative motion is the calculation of the 
motion of an object with regard to some 
other moving object. Thus, the motion is 
not calculated with reference to the earth, 
but is the velocity of the object in reference 
to the other moving object as if it were in a 
static state. For example, a person sitting in 
an airplane is at zero velocity relative to the 
airplane, but is moving at the same velocity 
as the airplane with respect to the ground. 
Relative motion is a concept, and its 
calculation occurs with relative velocity, 



Acquired Wisdom/Review  16 

relative speed, or 
relative acceleration 
(which is the change in 
velocity divided by the 
change in time. (Chegg, 
n.d.) 
 Many ship 
collisions occur because 
those who were 
navigating the vessels 
lacked situational 
awareness due to a lack 
of relative motion 
understanding. Ship 
handling simulation 
training devices are 
ideal for teaching the 
concept because the 
scenarios can be run 
over and over until the 
concept is understood 
by the trainee. Once 
understood, the 
concept of relative 
motion is fairly robust 
and not particularly 
prone to decay. Our 
main challenge was to determine how often 
personnel should receive training on relative 
motion to refresh their understanding of the 
concept. Our research showed that training 
should occur at least once a year with 
challenging conditions (e.g., fog, heavy 
winds, shifting currents) in the simulator.  
 Another training challenge is to 
ensure that trainees understand the physical 
effects of underway replenishment 
(UNREP). UNREP involves transferring 
cargo and people between two ships that are 
underway in the open sea. The Navy has to 
do UNREPs because their ships are often at 
sea without a friendly port where they can 
take on supplies. Supply ships must pull 
alongside the receiving ship and cargo is 
passed via cables and hoses strung between 
the ships. Ships cannot stand still for these 
transfers because the ships might be crashed 
together by the waves. They keep underway 
at speed so that those driving the ships can 
control the vessels. As shown in Figure 12,  
Figure 12. Underway replenishment  

 
the ships are only a few yards apart during 
UNREP, and there can be a tremendous 
suction action between the hulls of the ships 
if they get too close. Training personnel to 
understand the various physical effects of 
operating large ships close together is a 
challenge. The hydrodynamic modeling of 
the software that runs the simulator must 
undergo considerable verification and 
validation testing to make sure the simulated 
effects are accurate.  
 
Tank Team Trainer Effectiveness 
Evaluation  
 I was asked to assist the Army with a 
training effectiveness evaluation of a new 
training device that was being designed and 
fielded. The M-1 Abrams Tank Conduct of 
Fire Trainer (Figure 123) was a team trainer 
in that it had training stations for both the 
Tank Commander and the Gunner who 
fired the Abram’s main gun. Both trainees 



Retrospective of a career in the defense department                       17 

 
had the visual displays, 
such as periscopes and 
windows that presented 
simulated terrain views. 
These scenes presented 
imagery of enemy targets, 
along with friendly and 
neutral vehicles. The Army 
provides gunnery training 
that builds from a simple 
set of tank tasks to very 
complex tasks. The 
Conduct of Fire Trainer 
allowed the Commander 
and Gunner to 
communicate as they 
would in a combat 
situation while they looked 
at the simulated scenery 
cues. The instructors had their own 
Instructor/Operator station from which 
they could call up different scenarios, 
monitor team performance with built in 
performance measurement tool, conduct 
replays of the training, freeze the scenarios, 
and make use of other instructional features.  
 We conducted a series of 30 
interviews with sample trainees and 
instructors to get a sense for how well they 
thought learning was occurring from the use 
of the device. Careful notes were kept using 
an interview protocol. In addition, we 
examined pre-training and post-training 
gunnery scores in the simulator. Based on 
our analysis we made a number of 
recommendations about the design of the 
simulator, the performance assessment 
system, and the instructional features. The 
Army incorporated our suggestions into the 
design and use of the training device. The 
COFT is still in use today by the Army. The 
COFT allows the Army to train day and 
night in any weather because any operating 
conditions can be simulated. In addition, 
large amounts of money are saved by not 
using live ammunition and many miles and 
gallons of fuel are saved because a tank does 
not have to be driven anywhere or the gun 
used to fire actual rounds.  
 
 

Figure 13. M-1 Tank Conduct of Fire 
Training Device  
 
 Instructional challenges in tank 
gunnery and command training. Modern 
weapon systems have become both more 
accurate and more lethal through drastic 
upgrades in technology. If a gunner shoots, 
chances are good that they will hit their 
target. Training individual task skills are 
important for gunners and tank gunnery 
simulation can be very helpful. The gunners 
can shoot many more practice rounds, and 
they can shoot them in all kinds of weather 
and terrain conditions. The real training 
challenge comes in deciding how often they 
need to practice in order to maintain their 
proficiency. The Army has “gunnery tables” 
that require gunners to qualify for 
increasingly difficult tasks. Starting with 
hitting one nonmoving target at a close 
range, the trainees must go through a series 
of progressions that end with some very 
difficult tasks like hitting multiple, moving 
targets at considerable range in increasingly 
difficult terrain and visibility conditions. 
How often do trainees and operational 
gunners need to practice these skills once 
they have been achieved?  How should 
practice in a simulator and on a live gunnery 
range best be combined to achieve the 
optimal result?  We can measure 



Acquired Wisdom/Review  18 

performance in a tank simulator very 
accurately, but how can we best measure 
team communication and command and 
control decisions by the crew?   
 
Instructor/Operator Station Design 
Guidelines  
 Many training devices include an 
instructor-controller station for designing 
training scenarios tailored to each trainee, 
operating the training device, and instructing 
the trainee before, during and after training 
sessions. Often the training device will 
provide instructional support features.  
 For example, I found that training 
was not optimal because the 
Instructor/Operator Station (IOS) did not 
allow instructors to teach in the way they 
thought best. I began a program of research 
to develop IOS guidelines that could be 
used in training device design. To do so, I 
worked with a retired Navy Officer, Captain 
John Charles (Charles, 1984) who had 
extensive experience in designing IOSs both 
while in the Navy and later as a contractor. 
We reviewed previous literature on IOS 
design, visited many Navy training device 
locations to watch the IOSs being used, and 
interviewed a great number of instructors 
and operators who used the devices. Taken 
from the technical report that we produced, 
figure 14 shows an IOS for the COFT 
trainer described above and typical of IOSs 
found on many training devices.  
 
Figure 14   Instructor Operator Station  

 A simulation trainer 
instructor/operator station incorporates 
features designed to facilitate and optimize 
the instructor/operator interface. Many 
types of instructional and operating features 
can be and have been implemented. 
Although the majority of these features are 
intended for training elements used by the 
instructor, some of the features are primarily 
used by the IOS operator to support the 
training evolution. Thus the features 
incorporate both instruction and operation 
functions. We identified over forty features 
at the time.  
 
Work for the Army 
 I enjoyed my time working for the 
Navy, and in 1985, after six and one-half 
years with the Navy, I was promoted to the 
Civil Service grade of GS-14 (equivalent to 
military grade of Lt. Colonel) and went to 
work for another R&D organization, the 
U.S. Army Research Institute (ARI) for the 
Behavioral and Social Sciences. I worked for 
the ARI Field Unit in Orlando, Florida. ARI 
is the Army’s premier lab for research and 
development in manpower, personnel, and 
training. It had a number of Field Units 
(now called Research Units) co-located with 
training posts throughout the US.  
 In my two and one-half years at the Field 
Unit, we were mainly focused on developing 
models that would help designers optimize 
the training effectiveness of training devices  
of all types,(Andrews, Singer, Ozkaptan, & 
Hofer (1987; Sticha, Blacksten, Buede,  



Retrospective of a career in the defense department                       19 

 
Singer, & Gilligan, 1990). I was the team  
leader for the effort. Here is brief synopsis 
of this large research effort:  
 The Optimization of Simulation-
Based Training Systems (OSBATS) was 
designed to provide engineers involved in 
the concept formulation process for 
simulators and training devices, a tool with 
which to consider and tradeoff alternative 
features. OSBATS would enable engineers 
to trade-off fidelity and instructional 
features in the context of a training strategy 
to obtain designs that minimize cost for 
desired performance levels. OSBATS 
consists of five modules: a simulator 
decision module, a training device media 
selection module, an instructional features 
module, a fidelity optimization module, and 
a resource allocation module. The heart of 
the OSBATS model resides in the expert 
system rule bases used in the instructional 
features and fidelity optimization modules to 
make decisions about instructional and 
fidelity features to be incorporated in the 
simulator designs. (Singer, 1993, p.1) 
 The engineers and other 
professionals at PEO-STRI used the 
OSBATS models to improve training 
effectiveness. Working in the field of 
operations research and optimization for the 
first time was an enlightening experience for 
me (Radon, n.d.) It taught me much about 
building complex models and providing the 
models with data (both qualitative and 
quantitative) on which the models operated. 
That experience would be helpful to me 
throughout my career.  
 
Work for the Air Force  
 In 1987 I came across a job 
announcement for the position of Technical 
Director at the Operations Research 
Division of the Air Force Human Resources 
Laboratory in Mesa, Arizona. I knew of the 
good reputation of the R&D produced by 
the Laboratory and applied for the job. I 
was excited when offered the job at the GS-
15 level (equivalent to Colonel). The Lab has 
gone through many name changes through 
the years. It is now the Warfighter Readiness 
Research Division of the Air Force Research 

Laboratory (AFRL) at Wright Patterson Air 
Force Base. When I worked for the AFRL, 
our Division had about 100 military and civil 
servants, about 220 on-site contractors, and 
many more contractors around the country. 
The challenge of being the Technical 
Director was particularly interesting because 
of its multi-disciplinary mix of training 
specialists, military subject matter experts, 
engineers, physicists, computer scientists, 
and support staff. Hopefully, the need for all 
these specialties will be apparent as I 
describe the high technology work that the 
Division performed and is still performing.  
 We had good mix of different types 
of psychologists at the lab. They included, 
experimental psychologists, psycho-
perceptual psychologists, industrial-
organizational psychologists, and 
instructional psychologist like me. As 
Technical Director, part of my responsibility 
was to do personnel planning for the Lab, 
deciding what types of positions we needed 
to fill in order to meet our R&D goals. We 
first had to determine what the goals were, 
what types of disciplines we needed (e.g., 
behavioral scientists, engineers, computer 
programmers, etc.), then defend those 
positions to our management chain, and 
then find the right people for those 
positions. I really enjoyed that part of my 
job because we were not only meeting 
immediate needs, but laying the groundwork 
for long range R&D plans.  
 I felt that our job as behavioral 
scientists in the lab was to ask and answer 
the “so what?” questions about our R&D 
efforts. In other words, so what if we could 
invent new simulation and instructional 
technologies, did those innovations make 
any difference in the training effectiveness 
of what we developed? So, I encouraged all 
our scientists and engineers, but especially 
our psychologists, to have that basic 
question in mind as we planned and did our 
research. That was always one of the first 
questions I would ask when I conducted 
program reviews with the scientists.  
 As the Technical Director my main 
lab goals were to: 1. Produce effective 
training techniques and technologies, 2. 



Acquired Wisdom/Review  20 

Perform basic research that would lay the 
foundation for applied research under goal 
number one, and 3. Develop new 
technologies that would reduce the cost of 
training for the Air Force. Our funding 
came from basic and applied R&D funds 
that the lab received from Congress via the 
DoD and Air Force channels, and from 
funding we would get from the specific Air 
Force Commands who were our partners in 
developing better and more affordable 
training techniques and technologies.  
 Our funding allocations were usually 
in the 20-30 million dollar range each year. 
A major responsibility I had was leading the 
planning of the R&D program and then 
justifying and defending that budget through 
our Chain of Command. I enjoyed the 
planning, usually five years or more, and 
then justifying and defending it. It gave me 
an opportunity to work closely with our 
scientists, engineers, and customers so that I 
could explain the program to our superiors 
who often did not have a background in our 
disciplines.  
 The main focus of the Lab when I 
arrived was on pilot and aircrew training 
research. The Air Force, of course, has a 

huge need for quality training for all of its 
pilots, of which there are several thousand if 
we count the pilots of Unmanned Aerial 
Systems (UAS), often referred to as drones. 
The training is quite expensive when done in 
an actual aircraft. For example, the cost to 
fly an F-16 fighter aircraft exceeds $6,000 
per hour, and the cost to train in a large 
multi-aircraft engine aircraft like a bomber 
or transport plane exceeds $15,000 per hour. 
In addition, flying obviously has an element 
of danger, especially for those learning to 
fly. Also, flying pollutes the environment. 
And finally, pilots who are flying training 
missions cannot use all of their systems or 
tactics because of requirements for security. 
For these, and other reasons, pilot training 
turned to simulation early in its history.  
During World War II more than one-half 
million pilots were trained in the Link “Blue 
Box” trainers. (Figure 14).  
 The Blue Box did not have a realistic 
visual display because the technology was 
not available to provide an out the window 
scene. However, it did allow instrument and 
emergency procedure training. There were  
 
Figure 14.   Link “Blue Box” Trainer 



Retrospective of a career in the defense department                       21 

 
none of what today we would consider to be 
instructional features, and the instructor 
pilot was about the only source of 
instruction before, during, and after the 
training flight. The Blue Box did provide 
motion cues through three axes (yaw, pitch, 
roll), which was important for trainees 
because understanding 
control inputs affect 
the aircraft’s 
orientation in the air is 
vital in learning to fly.  
 Earlier I 
mentioned that 
psychologists working 
with simulation 
technology must always 
ask and answer the “so 
what?” question about 
the training 
effectiveness of new 
technology. The “Blue 
Box” is a good 
example of why that 
question was so 
important. It is true 
that while the “Blue 
Box’’ was quite sophisticated for its day, it 
was relatively primitive compared to today’s 
technology. However, for the tasks it was 
designed to train (emergency procedures, 
instrument flying) it was quite effective.  
 The Blue Box could best be 
described as a part-task trainer. The 
instructors of the day did not pretend that it 
could train all required pilot tasks, but it was 
effective at what it was designed for. At our 
lab we invested a good sum in 
understanding the part-task training concept 
better from a psychological point of view. 
To conduct the research, we built a variety 
of part task testbeds. Figure 15 shows an 
example of a part-task training testbed with 
a fighter simulator cockpit and a medium 
level of resolution out the window scene.  
 Note the myriad of button, switches, 
panels, control stick, and throttles the pilot  
must master. It is expensive to functionally 
simulate all of those features in a simulator 
cockpit. That level of fidelity may well be 
justified if the simulator is to be used for 

advanced training including combat skills. In 
the case of the fighter depicted, the Figure  
designers have chosen to provide a high 
level of cockpit physical and functional  
fidelity and a medium level of visual fidelity, 
largely to make the simulator more 
affordable.  

Figure 15. A simulator cockpit with a high 
level of simulation fidelity in the cockpit and 
a medium level of fidelity in the outside 
visual scene. 
 
 Our laboratory was constantly 
developing testbeds. We had a machine 
shop and various software programmers, 
which allowed us to vary the levels of 
fidelity in and out of the cockpit so that we 
could research optimal approaches to 
training. Again, my main task was to lead 
our team in asking the “so what?” question. 
We would bring pilots of various skill levels 
into the lab to test the quality of the training 
that could be provided with different 
training device designs and then seek their 
opinion on the acceptability of different 
simulator characteristics to pilots in the 
training centers and in the field. In addition, 
we would often take training devices at 
various stages of development out to the 
field to continue data collection with 
training and operational pilots.  



Acquired Wisdom/Review  22 

Figure 16.   F-16-part task trainer  
 
 Figure 16 shows another approach 
to a part-task trainer test bed. Note the 
trainer has a small out the window visual, 
and its instrument display is a Cathode Ray 
Tube instead of true instrument displays 
with individual controls like functioning 
buttons. The display does have a simulation 
of Heads Up Display which provides the 
pilot with information like: flight path 
information, navigation information, 
targeting data, and angle of attack 
information showing the aircraft’s relative 
angle to wind flow. We built the part-task 
training testbed to see if emergency 
procedures and other tasks could be taught 
effectively using the technology. For 
example, modern fighters have control 
sticks that have a variety of buttons much 
like a gaming joystick except they are much 
more complicated. The pilots have one hand 
on the control stick and the aircraft throttle, 
which also has various buttons. To operate 
the aircraft, the pilots must develop the skill 
to use the buttons in various combinations 
depending on which aircraft systems, radar, 
weapons systems, and anything else they 
need to operate. There are dozens of 
different control stick and throttle button 
combinations. Pilots learn to use the various 
combinations quickly and automatically 
almost without thinking. It is called “playing  

the piccolo”, because just as effortlessly as a 
concert piccolo player must learn to play 
without really thinking about the notes, so 
must a fighter pilot learn to operate the 
various button functions while in flight, 
especially in combat.  
 Our work with part-task trainers had 
at least two R&D objectives. We wanted to 
both see how effective they were for training 
and we wanted to lower the price as much as 
possible. Large, full mission simulators at 
the time were in the 10s of millions of 
dollars range.  
 Dexter Fletcher, an editor of this 
series, makes an excellent point about 
judging the cost of simulation-based 
training. Cost is only half the issue – return 
from cost is the other. Return can be 
measured in monetary terms, which is not 
easy, but do-able. Return in operational 
effectiveness is harder, but the main issue. 
Consider how expensive our training ranges 
are. Operational costs are astronomical, but 
their operational value is obvious to every 
warfighter, so we pay them. Other forms of 
training are turned down because they “cost 
too much” (although not as much as our 
ranges). Where’s our research to determine 
what they return?  Where’s our research on 
determining ways to assess operational 
return with respect to the cost of training?  
We should be answering the same question 
with respect to personnel selection, 
personnel classification, ergonomic design, 



Retrospective of a career in the defense department                       23 

 
job and performance aids, and other human 
centered investments we should be making. 
Who’s assessing the trade-offs among these 
investments?  (Fletcher, personal 
communication, 2016).  
 After World War II, and with the 
advent of computer technology, engineers 
and psychologists started to develop visual 
systems that could present out the window 
scenes to pilot trainees. By the time, I got 
involved with flight simulation, the field 
could present visual out the window scenes 
to pilots, but they were in black and white, 
and had a variety of resolutions and other 
challenges. In some cases, three dimensional 
large terrain maps were used to present a 
visual image to the trainees. These terrain 
maps resembled the type of simulated 
terrain, with towns, roads, rivers, and other 
features used today by model rail road  

Figure 17. 3-D Terrain board for flight 
simulation visual scenes  

enthusiasts. A closed-circuit television 
camera would be maneuvered over the 
simulated terrain in coordination with the 
pilot’s movements of the simulated aircraft’s 
control stick, and the resulting picture would 
be displayed to the pilots in their out the 
window display. The resulting visual display 
had reasonably good resolution, but the 
terrain the pilot could cover was limited to 
the size of the terrain board which was 
usually about 12 meters long by about 4.5 
meters tall, so the terrain board required a 
large building. Changing the model boards 
was required if the instructors desired the 
trainee pilots to fly over different types of 
terrain. I had familiarity with the terrain 
board from some work I had done with 
Army aviation. A three dimensional terrain 
board is depicted in Figure 17. Note the 
relative size of the man in the bottom right 

corner of the photo.  
 When I arrived, the 
Air Force Research 
Laboratory had an 
impressive history of 
investing in visual 
research and technology 
development intended to 
improve the quality and 
training effectiveness of 
visual displays for 
simulators. For example, 
the lab had developed 
the first color computer 
generated imagery (CGI) 
system. Much of the 
CGI technology so 
prevalent today for 
entertainment has been a 
spin-off from DoD 
investment. Years ago 
the DoD was spinning 
off results from R&D in 
simulation technology to 
the commercial world, 
and now, among other 
things, we are seeing our 
investments in new 
virtual reality technology 

returning to military use. It shows that DoD 
research investment in technology can have 



Acquired Wisdom/Review  24 

a huge multiplier effect when it reaches 
civilian industry and application.  
 In my time as Technical Director the 
capability of visual simulation for flying 
training made huge advances. Simulator 
visual systems have three interrelated 
components: a visual display system, which 
can range anywhere from a head mounted 
display as is seen today in virtual and 
augmented reality; computer generated 
imagery, which calculates the scene and 
sends it to the projectors that drive the 
visual display; and the data base, which may 
represent a real world piece of terrain or a 
scene from the data base developer’s 
imagination. I was involved in making 
resource decisions concerning where the lab 
would invest its visual R&D funds. Which 
innovative technologies were promising?  
What visual research would help us to 
answer questions about the efficacy of new 
technologies? While our lab was certainly 
not alone in this R&D, it is safe to say we 
developed a worldwide reputation for 
quality breakthroughs in a variety of areas. 
Figure 18 shows a scene from a modern 
flight simulator visual system with a level of 
fidelity that was not available when I arrived 
at the lab in 1988. Our lab had much to do 
with helping the field reach this level of 
simulation fidelity.  

Figure 18. Current degree of visual fidelity in 
flight simulation 
 

 Our lab was heavily involved in 
networking technology improvements. 
When fighters go on missions they typically 
fly in four-ship formations. This allows for 
mutual support. A major challenge for those 
advocating simulation-based approaches to 
mission training was producing a four-ship 
simulation complex that was networked 
together. The network had to be incredibly 
fast because the mission demands that the 
data and visual scenes be updated 60 times 
per second (60 Herz). Any time lag in the 
network updates would cause jitter in the 
simulation, which would destroy the realism 
we sought for four-ship mission training. 
Based upon considerable investment by our 
lab and other organizations, we achieved the 
holy grail of a 60 Herz update rate for 
simulation networks. With that technology 
we were able to build an affordable four-
ship simulator testbed for F-16 simulators. 
That proof of concept testbed convinced 
senior leaders that at last full mission 
training could be performed in training 
settings other than just in the air.  
 This capability was important for a 
variety of reasons. Four F-16s flying on a 
training range cost at least $20-25K per 
hour. In addition, every hour the aircraft 
flies means that its total useful life is reduced 
by that hour. Also, flying over a training 

range has training limitations. 
There are classified tactics 
and aircraft sub-systems that 
can’t be flown in open ranges 
because we don’t know who 
might be watching and 
listening. Finally, safety 
limitations with live aircraft 
such as altitude restrictions 
mean that pilots can’t really 
train as they will fight.  
 We conducted studies that 
showed that the simulators 
could be networked over 
very large distances and that 
they could be linked between 

continents and still maintain a secure 
network that updated the data between 
simulators at 60 Hz. We called this type of 
training Distributed Mission Training and 



Retrospective of a career in the defense department                       25 

 
started to include other types of simulators 
in the network such as ground control and 
air control stations to direct the fighters as 
in the real world.  
 All the time our lab was making 
technological breakthroughs, our behavioral 
scientists were conducting research to ask 
and answer the “so what?” questions 
concerning training effectiveness. We 
brought many pilots into our lab and had 
them fly increasingly complex missions to 
measure their progress. We developed new 
instructional features to aid the instructor 
pilots as they conducted pre- and post-
mission briefs and debriefs.  
 Having laid out the case for 
simulation, I hasten to add that we never 
believed that simulation-based training could 
completely replace live training. There are 
important objectives that can only be trained  
in the air effectively. Physiological factors 
such as handling high G loads are an 
example. Our goal, as was the case with the 
OSBATS simulation optimization research, 
was to arrive at an optimal blend of 
simulation and live training depending on 
the training objectives. We always kept that 
goal as a paramount principle in our 
research.  

  

Our lab was heavily involved in the 
expansion of the Distributed Mission 
Training concept. Militaries around the 
world now use the expanded version and it 
is called Live, Virtual, Constructive (LVC) 
training. Live refers to training in actual 
equipment out on a training range. Virtual 
refers to training in a virtual world such as in 
the F-16 four-ship simulator. Constructive 
refers to computer generated forces that act 
in the training scenario as friendly or enemy 
or neutral air, land or sea entities such as 
aircraft, ships or ground vehicles. When all 
three LVC domains are combined, a 
powerful training environment is created. 
 The LVC concept is now used on 
regular basis for training in many militaries. 
It allows coalition forces in many countries  
to train together for missions without ever 
having to be in the same training range. 
Huge sums of money are saved, but, more 
importantly, research showed that the 
training effectiveness of LVC training was as 
high as it was when large forces were 
brought together.  
  
Figure 19. This graphic depicts the LVC 
domains being combined to make a 
synthetic battlefield for training.  

 



Acquired Wisdom/Review  26 

Figure 20 depicts a Combined Air 
Operations Center where the Commander 
and the entire planning, command, and 
control staff are brought together to 
conduct a theater wide mission operation. 
The staff can consist of hundreds of 
different specialists. Militaries have 
conducted large scale command and control 
training exercises since the beginning of 
such operations. Now these large command 
and control centers can be linked to entities 
in ranges, both real and virtual, so that the  
entire command and control chain, down to 
individual operators in aircraft and on the 
ground, can train in a manner very similar to 
real missions.    
 

 
 
Figure 20. Combined Air Operations Center 
 
 I was the Technical Director of our 
Division of the Air Force Research 
Laboratory for 15 years and greatly enjoyed 
leading such a talented team. My last nine 
years were spent as a Senior Scientist (I was 
promoted to be a member of the Senior 
Executive Service Science and Technical 
Cadre with a grade equivalent to a Brigadier 

General). I loved the job because I was 
released from management responsibilities 
and was given the task of performing 
research in any area I chose. I was given a 
small team of support scientists and a 
budget every year. During those years I did 
research in a variety of areas such as:  
 

 Training model development for 
a human factors tool called Manpower 
and Personnel Integration 
(MANPRINT) (Conant, 2014; Fitzgerald 
et al., 2008). MANPRINT is aimed at 
integrating into the system acquisition 
process considerations of human factors. All 
equipment developed for the military is 
made better when human factors such as; 
fatigue, skill requirements, communication 
requirements, human-system integration, 
reach and strength requirements, and 
hearing and considered in the design phase. 

 Accelerating expertise 
development (Hoffman et al. 2014; 
Hoffman, Andrews, & Feltovich, 2012). 
Like all organizations, the military has great 
need to train service members more quickly. 
This project examined the learning literature, 
and provided prescriptions for accelerating 
the attainment of expertise.  

 Training for supervisors to 
mitigate cyber insider threat (Andrews et 
al. 2013; Greitzer et al., 2008). Insider cyber 
theft and sabotage account for about 40% 
of the cyber incidents that occur each year. 
Yet, supervisors are ill-trained to even 
detect, let alone mitigate, the threat. This 
work produced virtual reality training 
modules used to provide training to 
supervisors in this key area.  

 Training to mitigate friendly fire 
by improving combat identification 
(Andrews, Herz, & Wolf, 2010; Wilson, 
Salas, & Andrews, 2010). When a friendly 
soldier or force tragically brings harm to 
another friendly soldier or force, great regret 
follows. This work explored the role that 
better human factors analysis and training 
can and should play in reducing the 
possibility of friendly fire.  



Retrospective of a career in the defense department                       27 

 
 Cyber-friendly fire (Andrews & 
Jabbour, 2011). As I worked in the cyber 
warfare domain area it became clear that 
many of the same principles that applied to 
avoiding friendly fire in the traditional sense 
(e.g., an aircraft accidently shooting a 
friendly tank) could be applied to personnel 
conducting cyber operations. 

     Embedded Training Guidelines. 
(Andrews & Roessingh, 2011). I worked on 
a NATO science panel tasked with 
developing more effective guidelines for 
designing training solutions into an actual 
piece of military weapons system. The goal 
is to allow personnel to conduct training in 
their weapon system. However, to do that, 
certain instructional features and 
performance measurement capabilities must 
be either designed into the system from the 
start or attached (strapped on) to the system 
if it already exists. Our guidelines examined 
the role of embedded systems and gave 
guidelines for their use.  

 Team training (Cooke, Gorman, 
Myers, Duran, & Andrews, 2013; Cooke, 
Pedersen, Connor, Gorman, & Andrews, 
2006). As has been described throughout 
this paper, team training in the military is of 
great importance. Empirical research on the 
topic is developing, but more needs to be 
done. I sponsored a series of studies with 
Nancy Cooke and her lab at Arizona State 
University, examining the role learning 
retention plays in team training and how it 
can be strengthened.  

 Storytelling as a method of 
instruction (Andrews, Hull, & Demeester, 
(Eds.), 2010; Andrews, Hull, & Donahue, 
2010). Through the years I have been struck 
by how much teaching and learning happens 
in the military via stories. Military personnel 
are constantly telling stories about their 
experiences in combat and at other times. It 
is a primary way that knowledge and 
wisdom is passed on. It happens both in 
formal classroom settings and in off-duty 
hours’ venues. This research and resulting 
publications examined what literature there 
was on the topic and proposed a taxonomy 
of storytelling for instruction. The taxonomy 

consists of four distinct, but related, 
categories of storytelling for instruction; 
scenario based, problem-based, narrative 
based training and case based training.  

 Fundamental (ab initio) pilot 
training guides (Fitzgerald et al., 2007). 
Training novice pilots is a complex process. 
The instructor pilots are given a fair amount 
of leeway in judging the performance of 
novices, and therefore there is often an 
inconsistent approach to judging 
performance. This study produced a more 
objective approach based on trainees’ 
behaviors during the training session.  

 
Lessons Learned 
After 40 years on working in the instructional 
psychology field it is interesting to look back 
at major lessons I have learned. These lessons 
are not just about developing effective 
instruction, doing training research, and 
applying training methods and techniques, 
which I to relate below. I’ve also listed some 
lessons on ways to be more effective as a 
developer, researcher, and manager.  

 

 Envision the big picture. I find 
that instructional development, and research 
and development have much more impact 
and are more enjoyable when I, and my 
research team, whatever size it may be, have 
established clear visions and missions. What 
is the goal of the program?  
 For me, a vision sets a picture of 
what I want the world to be like when the 
training or research program are over. It is a 
broad set of achievable goals, that once set 
and agreed to by the team, customers, and 
users, should not be changed without great 
consideration. The vision should cover a 
period of years and an end date should be set. 
It defines the “what” that the team and I will 
be doing to achieve the goals.  
 The mission is “how” we will 
accomplish the “what.” Missions can be 
changed frequently if necessary. We may 
accomplish one mission goal on the way to 
the vision achievement, and then decide for 
the next step to take a different path to move 
towards the “what”.  



Acquired Wisdom/Review  28 

A good example of this vision/mission goal 
process was the path we set in our Air Force 
Research Laboratory for Distributed Mission 
Training. Our vision was a future state where 
every aircrew in the Air Force would be able 
to train in a realistic simulator, in real time, 
with a complete set of instructional features 
and performance measurement as required 
for effective training. Once that vision was 
established and accepted by the team, its 
customers, and users, we stuck to it for a 
number of years. Disruptions in Lab budgets, 
departure of key scientists and engineers, and 
wavering on the part of our Air Force 
customers (military and civilian personnel at 
key decision making levels come and go on a 
regular basis) did not dissuade us from the 
vision. We focused all of our programs 
(missions) on the big vision. We finally 
achieved a major milestone in the Distributed 
Mission Training program when several Four 
Star Generals said they wanted the capability 
for their Commands, and the AF started to 
devote major budget investments to 
implementing the vision. A training 
revolution occurred not just in the Air Force, 
but across the DoD and in allied countries.  
 

 Pay attention to front end 
analysis. The first step in the ISD model 
described earlier is Analysis. ISD Analysis 
consists of three main steps.   
 Needs assessment. Needs assessment is 
conducted when a job performance problem 
has been identified. Needs assessment 
involves a systematic identification of 
solutions to performance problems. The 
assessment determines the root cause of the 
problem, then proposes a solution. The 
problem may be due to inadequate training, 
poor job documentation, poor equipment, 
lack of motivation, or other organizational 
issues.  
 Job analysis. Job analysis is a 
systematic method of listing all the tasks 
necessary to perform a specific job. These 
tasks represent the foundation on which we 
construct performance-based training 
objectives, course content, and evaluation 
instruments. Simply put, the job analysis 
provides a detailed "picture" of the job to be 
trained. The job analysis also provides 

information about entry-level skills and 
possible prerequisites. 
 Target audience analysis. This step 
identifies characteristics that affect trainee 
learning. It includes trainees' educational 
background, previous training experiences, 
relevant work experiences, and motivation 
for training. This information enables 
designers to customize training for the 
intended audience. 
 Training requirements and outcomes.  
Training requirements are the knowledge 
and skills that must be learned. Training 
outcomes are tasks that trainees must 
demonstrate to given standards in order to 
ensure competence on the job. 
 Unfortunately, some customers of 
training developers and supervisors of 
training developers rush to get the front end 
analysis done so the design phase of 
instruction and simulator design can begin. 
It always reminded me of the old Fram oil 
filter commercial where the shop mechanic 
says something like, “You can pay me a little 
extra now for a quality filter, or you can pay 
me a lot more later for a ruined engine.”  
Front end analysis in training development 
is like that. Work hard to convince managers 
and customers that in the long run spending 
the time and resources to do a quality front 
end analysis is fully worthwhile. I have seen 
multi-million-dollar training simulators 
sitting unused because the training they 
deliver is not what the task or job needs
  

 Desperate need for learning 
retention research. After 100 plus years of 
learning research we know a tremendous 
amount about how learners acquire 
knowledge and skills. We don’t know nearly 
as much about the decay rate and retention 
of those knowledge and skills. Having done 
some learning retention research with 
humans, I can say that it is not easy. We 
can’t put the human subjects in a controlled 
setting during the retention period after we 
have measured how well they acquired the 
learning. They have lives and must go about 
their daily business. We can’t control what 
influences they come under during that 
retention period, or even if somehow they 
have practiced with the knowledge and skills 



Retrospective of a career in the defense department                       29 

 
they acquired. In addition, it is difficult to 
get all of the subjects back at the end of the 
retention period. Despite these difficulties, it 
is important to do such studies. I imagine 
that for every 20 learning acquisition studies 
that have been done, we probably don’t 
have more than one retention study. How 
can we judge when sustainment training is 
required if we do not have good retention 
models with valid data?  
 

 Formative evaluation for 
simulation-based training devices. 
Developing effective training simulators is 
no different than developing any other 
training. There is need for formative 
evaluation of effectiveness through different 
phases of the development process. Use 
mock-ups, breadboards, rapid prototyping 
tools, even pencil drawings with subjects 
who share the same entry and learning 
characteristics with the eventual training 
population. Any insight you can get into 
how the test subjects are understanding the 
learning objectives and how they will use the 
eventual training device and training features 
is valuable. Waiting until the device is 
completely finished to check whether its 
training is effective can waste huge amounts 
of time and money if the training is 
ineffective. The earlier the researcher can 
gain insight to the nature of the functions, 
tasks and trainees, the better will be the 
ultimate training system result.  
 

 How much simulation fidelity is 
enough?  In much of our Lab work we 
attempted to ask and answer the question 
that senior decision makers (e.g., Congress, 
the Pentagon, top military leaders) wanted 
to know, “How much simulation is 
enough?”  This question stemmed largely 
from a desire to provide the necessary 
training capabilities at affordable prices. The 
rationale for the question grew from a desire 
to provide just the right amount of fidelity 
and no more. I always felt that question, 
while well intentioned, was short sighted. 
Modern aircraft and their sub-systems are 
constantly changing. New capabilities are 
often added to military aircraft and other 
materiel. What might be enough simulation 

fidelity today, may not be enough in a fairly 
short time. None the less, we heard the 
decision maker’s call for “how much is 
enough” kind of R&D and strove to 
produce findings that might answer the 
question.  
 

Training of instructors for 
simulation based training must be 
sustained throughout a training device 
lifetime. The effectiveness of a training 
device reduces as each new generation of 
instructors appears. New instructor training 
is usually produced for a training device 
when it is first delivered. The manufacturer 
is usually tasked to provide hands on 
training to the instructors for the device. 
That training includes instruction about how 
best to operate the device using all of the 
instructional features and performance 
measurement tools. The organization that 
provides the simulator usually budgets for 
that type of training. However, it rarely 
budgets for a repeat of the training when a 
new generation of instructors takes over or 
the simulator is improved. The result is that 
each succeeding generation of instructors is 
taught by the previous generation in an 
informal manner. Instructor training 
documentation that may have been provided 
when the simulator was first delivered is 
often not used by following generations of 
instructors. Rules of thumb are applied 
more and more so that effectiveness of the 
training device continues to decline. It is 
important to convince the user organization 
that they must budget for instructor training 
as a matter of course for as long as the 
device is used.  

 

Pilot tests of research designs are 
crucial.  I have seen a number of 
researchers, both experienced and otherwise, 
run experiments with their research subjects 
before they have pilot tested their research 
designs. That is typically a mistake. Doing 
behavioral research is fraught with potential 
chance error. Researchers can substantially 
reduce these errors if they first run one or 
more pilot tests. Doing research with 
humans is difficult partly because human 
variability itself can vary over time. Doing 



Acquired Wisdom/Review  30 

pilot studies also consumes time and 
resources, but we need to remember the 
ancient adage repeated by the Fram filter 
mechanic. The more the researcher can do 
to identify research error that comes from a 
faulty design, the better will be the resulting 
research and its findings. 

 

Know what color of R&D funding 
you are seeking. Decide whether you are 
doing basic or applied research – if nothing 
else this distinction matters greatly to 
program managers in charge of funding. 
Basic research is in search of truth, often 
without concern of whether the resulting 
truths can be immediately useful. This type 
of research lays the foundation for theory 
building, which I believe is crucial for 
evaluation. Applied research uses basic 
research findings and resultant theories for 
exploratory research. Exploratory research 
provides a chance to test possible training 
solutions, without necessarily having a 
particular learning problem or user. 
Successful findings and developments can 
then be moved to the advanced exploratory 
phase.  
 A researcher must decide where 
their research falls on the scale described 
above. Government funding for R&D all 
comes in a certain “color”—a funding type 
designation. I’ve seen many researchers try 
to get research funding without attending to 
the type of money allocated for it, and they 
were rarely successful in their quest.  
 

 Mind the Gap. For those who have 
ridden the Underground subway in London 
that phrase will be familiar. The sign is 
posted all over the Underground reminding 
passengers that there is a gap between the 
platform and the train. Ignore that gap at 
your peril. In all instructional R&D there is a 
gap between the work to develop it and its 
successful implementation in the training 
world. A researcher, especially working in the 
applied research area should start early to 
build a bridge to the eventual customer/user. 
On many occasions I’ve seen scientists 
frustrated after producing great R&D, only to 
find customers and/or users not ready for 
the new knowledge or device, or not even 

able to see the need for it. Many R&D 
products are left on the shelf because the gap 
wasn’t minded. My rule is that a researcher 
should start a research project devoting about 
15% of their time to gap minding. As the 
project progresses towards its transition, the 
researcher should steadily increase gap 
minding time while spending less and less 
time on the actual training product or 
research. This transition must be thought 
through carefully or the researcher runs the 
risk of producing a product nobody really 
wants, or producing a product late to the 
customer/user’s need.  
 

 Embrace the bureaucracy!   Any 
system needed to produce viable training, 
training research, and training devices has its 
bureaucratic rules—the systems that provide 
resources for our work, and often the 
requirements for the expenditure of those 
resources. I’ve seen many developers and 
researchers spin their wheels toward progress 
because they didn’t like the rules and tried to 
ignore or short circuit them. With experience, 
a researcher can usually find a path to short 
circuit some rules, but that comes from 
substantial amounts of experience with the 
system. Early on, after a couple of aborted 
attempts to ignore the rules, I determined 
that the energy required to be successful at 
gaming the bureaucracy was not worth the 
time and effort. I finally decided it was easier 
to simply follow the rules, no matter how 
nonsensical they appeared, and spend my 
energy getting the real job done. As you rise 
in management you may eventually agree 
with me.  

 

 Multi-discipline teams. Research 
psychologists and educators need to learn 
how to be valuable members of multi-
disciplinary teams. These teams can include 
people from other behavioral sciences, 
engineers, computer scientists and 
programmers, medical scientists, subject 
matter experts, and a vast variety of other 
disciplines. Learn to form teams, learn from 
people with management skills, learn to listen 
to the concerns from team-mates from other 
disciplines. That last lesson learned came to 
me the hard way. The first multi-disciplinary 



Retrospective of a career in the defense department                       31 

 
team I led did not do well, at least partly 
because I was not truly trying to understand 
the problems team members were relating to 
me. Be flexible in leading and be prepared to 
make mid-course corrections if it appears 
your team is not accomplishing its mission 
toward the larger vision. 

  

 Professional societies. Get 
involved in professional societies that meet 
your interest. I met some of my closest 
professional colleagues and friends by 
volunteering to work on society committees. 
They have been a steady influence on my 
work and a great pool of expertise that I 
have tapped many times. Go to as many 
society meetings as your budget and time 
will allow. It is well worth it.  

 

 Break bad news early. There are 
an infinite number of things that can go 
wrong with a development or research 
project. I have found it is best to break bad 
news early to bosses, customers, funders, 
and users. I have tried to lead my team in 
fixing whatever issue might arise, while at 
the same time letting the various 
stakeholders know that an issue might be 
developing. I have tried the route of keeping 
the bad news to myself for a long time in 
hopes that the problem could be fixed. I 
found that the surprise factor only gets 
bigger as I tried to keep the problems to 
myself. A little bad news early is usually 
better than a lot worse news later on.  

 

 Be serious about managing your 
career. This I learned the hard way. When I 
began my career I had the naïve belief that if 
I did my job to the best of my ability and 
produced good results, the “system” would 
reward me with awards and promotions. It 
didn’t take me long to realize that happens 
on occasion, but it is not the norm no 
matter where you work. Don’t count on 
anyone else to manage your career. It is up 
to you to figure out where you want to go, 
and then what you must do to get there. 
Perhaps it is receiving certain types of 
training, working on committees, moving to 
a new location, being involved in “high 
profile” projects, or getting quality mentors 

in the system. There is certainly nothing 
wrong with a person deciding early on to 
remain a researcher, and not move up the 
management chain. But, I’ve mentored 
many scientists who, in their mid-career 
years, decided they wanted to move up the 
organization. However, because they hadn’t 
done what was necessary earlier in their 
careers to put themselves in competitive 
positions for promotion they were 
disappointed to find that the odds of 
advancement were not what they wished. 
Develop good situational awareness 
concerning your job path and make the 
decisions you need to make. No one should 
be as interested in your career development 
as you are.  

 

 Leave a legacy. One of my favorite 
books is Stephen R. Covey’s The Seven Habits 
of Highly Effective People (2004). In this book 
Covey reported on in-depth research he had 
done with highly effective people. He lays 
out seven habits that he says were the 
hallmark of all successful people.  
 

o Be proactive 
o Begin with the end in mind 
o Put first things first 
o Think Win-Win 
o Seek first to understand, and then seek 

to be understood 
o Synergize 
o Sharpen the saw  
  

In his next book, The 8th Habit: From 
Effectiveness to Greatness he describes how 
“leaving a legacy” should be a major goal for 
all professionals. That is a lesson I have 
learned by having the opportunity to lead 
and mentor a variety of younger colleagues 
as a government science leader and as an 
adjunct faculty member. I had some 
wonderful leaders in my life including my 
father, Dale H. Andrews, Scout leaders, 
religious leaders, and scientists such as 
Robert Gagne. These leaders left a shining 
legacy for me to examine and take 
inspiration. Strive to build your professional 
career so that you too can leave a legacy for 
the many people who you will influence in 
our field.  



Acquired Wisdom/Review  32 

Wrap Up  
 I’ve now been retired from the 
Department of Defense for over five years, 
but continue consulting in DoD training. I 
have worked on training programs for the 
Naval Special Warfare Command, 
conducted a cognitive task analysis of the 
Unmanned Aerial Vehicle (UAS) (drones) 
piloting task, and developed training in 
cyber security. I also teach courses for a 
number of universities online. All of these 
tasks have not only kept my head in the 
training game, but they also allow me to 
work with old friends from DoD days. I still 
love training research and development, and 
intend to stay in it until I can’t do it 
anymore.  

 Would I recommend working for 
the DoD? Working for the DoD is generally 
well supported with resources such as 
funding, lab equipment, subject matter 
expertise, and technical partners. In 
addition, the work has national and 
international importance because of its 
direct effect on the security effectiveness of 
our armed forces. During my career I have 
been pleased to provide our military with 
quality science and technology products to 
help make the training of our personnel and 
allies the finest in the world. We in the 
training and training science business have a 
great responsibility. I highly encourage any 
readers who have been interested in some of 
the work I have related to consider a career 
working for the military of our nation.  

 
References 
 
Alexander, T., Goldberg, S. L., Andrews, D. H., Hourlier, S., Koerhuis, C., Magee, L. F., . . . 

Sottlilare, R. (2014). Improving Human Effectiveness Through Embedded Virtual Simulation.  
    NATO Science and Technology Organization. Technical Report TR-HFM-165.  
Andrews, D. H. (1988). Relationships among simulators, training devices, and learning: A 

behavioral view. Educational Technology, 1(28), 48-53. 
Andrews, D. H. (2000). Distributed mission training. In W. Karwowski (Ed.), International 

encyclopedia of ergonomics and human factors. Philadelphia, PA: Taylor and Francis.  
Andrews, D. H. (2005). Distributed simulation training for teams. In N. Stanton, A. Hedge, K. 

Brookheair, E. Salas & H. Hendrick, (Eds.)  Handbook of Human Factors and Ergonomic 
Methods. London, England: Taylor & Francis.  

Andrews, D. H. & Bell, H. (2000). Modeling and simulation applications to training. In S. Tobias 
and D. Fletcher (Eds.), Training and Retraining handbook. Washington, DC: American 
Psychological Association, Division 15 (Educational Psychology). 

Andrews, D. H., Carroll, L. A., & Bell, H. H. (1995). The future of selective fidelity in training 
devices. Educational Technology, 35(6), 32-36. 

Andrews, D. H, & Craig, S. (Eds.) (2015). Readings in training and simulation. Vol. 2: Research articles 
from 2000-2014. Santa Monica, CA: Human Factors and Ergonomics Society. 

Andrews, D. H., Freeman, J., Andre, T. S., Feeney, J., Carlin, A., Fidopiastis, C. M., & Fitzgerald, P. 
(2013). Training organizational supervisors to detect and prevent cyber insider threats: Two 
approaches. International Conference on Software Testing, Verification and Validation (ICST), EAI 
Endorsed Transactions on Security and Safety, 13(1-6), e4. Available at 
http://www.eudl.eu/journal/sesa 

Andrews, D. H. & Goodson, L.A. (1980). A comparative analysis of models of instructional design. 
Journal of Instructional Development, 3(4).  

Andrews, D. H., Herz, R. P., & Wolf, M. B. (Eds.) (2010). Human factors issues in combat identification. 
London, England: Ashgate Publishers.  

Andrews, D. H., Hull, T., & Demeester, K. (Eds.) (2010). Storytelling as an instructional method: Research 
perspectives. Rotterdam, NL: Sense Publishers.  

Andrews, D. H., & Roessingh, J. J. (2011). Training management in virtual embedded simulations. 
Proceedings of the 22nd International Training Equipment Conference. Cologne, Germany.  

http://www.eudl.eu/journal/sesa


Retrospective of a career in the defense department                       33 

 
Andrews, D. H., Singer, M., Ozkaptan, H., & Hofer, R. (1987). Optimization of training systems 

and subsystems:  Models, methods and data. Proceedings of the National Security Industrial 
Association Conference on Military Personnel and Training. Luxemburg.  

Andrews, D. H., Waag, W. L., & Bell, H. (1992). Technologies for team training. In R. W. Swezey 
& E. Salas (Eds.), Teams: Their training and performance. Englewood Cliffs, NJ: Ablex. 

Andrews, D. H. & Windmueller, H.W. (1986). Lock-step vs. free-play maintenance trainers:  
Definitions and issues. Educational Technology, 7(26), 29-34. 

Best, C., Galanis, G., Kerry, J., & Sottilare, R. (Eds.) (2013). Fundamental Issues in Defense Training and 
Simulation. London: Taylor and Francis.  

Branson, R. K., Rayner, G. T., Lamarr, J., Cox, L., Furman, J. P., King, F. J. & Hannum, W. H. 
(1975). Interservice procedures for instructional systems development. National Technical Information 
Service. ADA 019 486. Retrieved from 
http://www.dtic.mil/dtic/tr/fulltext/u2/a019486.pdf 

Cannon-Bowers, J. A., & Salas, E. (2001). Reflections on shared cognition. Journal of Organizational 
Behavior, 22, 195-202. 

Cannon-Bowers, J. A., Salas, E., & Converse, S. A. (1990). Cognitive psychology and team training: 
Shared mental models in complex systems. Human Factors Society Bulletin, 33(12), 1-4. 

Charles, J. P. (1984). Design guidelines for trainer instructor/operator consoles. Naval Training 
Systems Center, Orlando Fl.,   NAVTRASYSCFN 83-C--0087-1. DTIC Technical Report 
Accession Number AD-A171637.  

Chegg, (n.d.). Relative motion. Retrieved from http://www.chegg.com/homework-
help/definitions/relative-motion-5 

Conant, J. (2014). MANPRINT program focuses on integrating human element. Retrieved from 
http://www.arl.army.mil/www/?article=2470.  

Cooke, N. J., Gorman, J. C., Myers, C. W., Duran, J., & Andrews, D. H. (2013). Retention of team 
coordination skill. In W. Arthur, E. A. Day, W. Bennett, & A. M. Portrey (Eds.), Individual 
and team skill decay: State of the science and implications for practice. New York: Routledge 
Academic. 

Cooke, N. J., Pedersen, H. K., Connor, O., Gorman, J., & Andrews, D. H. (2006). Acquiring team-
level command and control skill for UAV operation. In E. Salas (Series Ed.) & N. Cooke, 
H. Pringle, H. Pedersen, & O. Connor (Eds.), Advances in human performance and cognitive 
engineering research:  Vol 7. Human factors of remotely operated vehicles. (pp. 285-297). 
Oxford:  Elsevier.  

Covey, S. R. (1998). The 7 habits of highly effective people. Provo, UT: Franklin Covey.  
Covey, S. R. (2004). The 8th habit: From effectiveness to greatness. New York: Free Press.  
Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 

37, 32–64. 
Fitzgerald, P. C., Andrews, D. H., Crabtree, M. S., Doyal, J. A., Meador, D. P., &  
 Walsh, W. J. (2008). Incorporating training effects in modeling and simulation  
 software. Proceedings of the Sixth Annual Conference on Systems Engineering Research. Los Angeles, 

CA. 
Fitzgerald, P. C., Andrews, D. H., Crow, B., Diedrichs, R., Karp, M. R., & Stewart, K. (2007, April). 

An investigation of certified flight instructor competencies. Presented at the International 
Symposium on Aviation Psychology, Dayton, OH. 

Freud, S. (1899/1913). The interpretation of dreams. Trans. A.A. Brill. New York: Macmillan.  
Gagne, R. M. (1954). Training devices and simulators: Some research issues.  
 American Psychologist, 9(3), 95-107.  
Gagne, R. M. (1972). The conditions of learning (3rd Ed.). New York. Holt, Rinehart and Winston. 
Gagne, R. M. (1985). The conditions of learning and theory of instruction. 4th ed. New York: Harcourt 

Brace.  

http://www.dtic.mil/dtic/tr/fulltext/u2/a019486.pdf
http://www.chegg.com/homework-help/definitions/relative-motion-5
http://www.chegg.com/homework-help/definitions/relative-motion-5
http://www.arl.army.mil/www/?article=2470


Acquired Wisdom/Review  34 

Gagne, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design. (4th ed.). Orlando, 
Fl., Harcourt, Brace, Jovanovich.  

Greitzer, F. L., Moore, A. P., Cappelli, D. M., Andrews, D. H., Carroll, L. A. &  Hull, T. D. (2008). 
Combating the insider threat. Security and Privacy, 6(1), 61-64. Institute of Electrical and 
Electronic Engineering’s Computer Society.  

Hanley, M. J. & Andrews, D. H. (1987, April). Evaluation of the effectiveness of a table-top simulator device 
as a ship handling trainer. Paper presented at the 1987 Society for Computer 
Simulation - Eastern Simulation Conference. Orlando, FL. 

Hays, R. T., & Singer, M. J. (1989). Simulation fidelity in training and system design: Bridging the gap between 
reality and training. New York: Springer-Verlag. 

Hoffman, R. R., Ward, P., Feltovich, P. J., DiBello, L., Fiore, S. M., & Andrews, D. H. (2014). 
Accelerated learning: Training for high complexity in a complex world. New York: Academic Press.  

Instructional Design Central. (2016). The ADDIE model. Retrieved from: 
http://www.instructionaldesigncentral.com/instructionaldesignmodels 

Instructional Systems Development. (N.D.). Air University. Retrieved from 
http://www.au.af.mil/au/awc/awcgate/doe/isd/paper.htm   

JP 3-05.1 Joint Special Operations Task Force Operations. 26 April 2007 Retrieved from 
http://fas.org/irp/doddir/dod/jp3_05_01.pdf  

Morrison, J. E., & Fletcher, J. D. (2002). Cognitive readiness. Institutes for Defense Analysis P-3735. 
Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/a417618.pdf 

Orlansky, J., & String, J. (1981). Cost-Effectiveness of Maintenance Simulators for Military Training. Final. 
Report. Institute for Defense Analysis, Arlington, Va. Department of Defense, Washington, 
D.C. IDA-P-1568   MDA-903-79-C-0202  

Paris, C. R., Salas, E., & Cannon-Bowers, J. A. (2000). Teamwork in multi-person teams: A review 
and analysis. Ergonomics, 42(8), 1052-1075.  

Robinson, P. W. (1976). Fundamentals of experimental psychology: A comparative approach. Upper Saddle 
River, N.J.: Prentice-Hall.  

Radon. R. (n.d.)  Operations research and optimization: A primer. Retrieved from 
http://www2.isye.gatech.edu/nci-nsf.orart.2002/pdf-files/Rardin.primer.pdf 

Salas, E., Cooke, N. J., & Rosen M.A. (2008, June ). On Teams, Teamwork, and Team 
Performance: Discoveries and Developments. Human Factors, 50(3). 

Salas, E., & Fiore, S. M. (Eds.). (2004). Team cognition: Understanding the factors that drive process and 
performance. Washington, DC: American Psychological Association. 

Semple, C.A., Cotton, J. C., & Sullivan, D. J. (1981). Aircrew training devices: Instructional support features. 
Air Force Human Resources Laboratory. AFHRL TR 80-58. Government accession 
number AD-A096 234. 

Singer, M. J. (1993). The optimization of simulation-based training systems: A review of evaluations and 
validation of rule bases. U.S. Army Research Institute for the Behavioral and Social Sciences. 
Research Report 1653.  

Sticha, P. J., Blacksten, H. R., Buede, D. M., Singer, M. J. & Gilligan, E. L. (1990).                                           
Optimization of simulation-based training systems: Model description, implementation, and evaluation. 
Human Resources Research Organization, Alexandria, Va. DTIC Accession Number ADA 
237266.  

Swezey, R. W., & Andrews, D. H. (Eds.) (2001). Training and simulation: A thirty-year perspective. Santa 
Monica, Ca. Human Factors and Ergonomics Society.  

Wilson, K. A., Salas, E., & Andrews, D. H. (2010) Preventing errors in the heat of battle: Formal 
and informal learning strategies to prevent teamwork breakdowns. In Andrews D. H., Herz 
R. P., & Wolf M. B. (Eds.), Human factors issues in combat identification (pp. 1–28). 
Aldershot, UK: Ashgate. 

 
  

http://www.instructionaldesigncentral.com/instructionaldesignmodels
http://www.au.af.mil/au/awc/awcgate/doe/isd/paper.htm
http://fas.org/irp/doddir/dod/jp3_05_01.pdf
http://www.dtic.mil/dtic/tr/fulltext/u2/a417618.pdf
http://www2.isye.gatech.edu/nci-nsf.orart.2002/pdf-files/Rardin.primer.pdf


Retrospective of a career in the defense department                       35 

 
 
 
 

 
 

 About Acquired Wisdom 
This collection began with an 

invitation to one of the editors, Sigmund 
Tobias, from Norman Shapiro a former 
colleague at the City College of New York 
(CCNY). Shapiro invited retired CCNY 
faculty members to prepare manuscripts 
describing what they learned during their 
College careers that could be of value to 
new appointees and former colleagues. It 
seemed to us that a project describing the 
experiences of internationally known and 
distinguished researchers in Educational 
Psychology and Educational Research 
would be of benefit to many colleagues, 
especially younger ones entering those 
disciplines. We decided to include senior 
scholars in the fields of adult learning and 
training because , although often neglected 
by educational researchers,  their work is 
quite relevant to our fields and graduate 
students could find productive and gainful 
positions in that area.  

Junior faculty and grad students in 
Educational Psychology, Educational 
Research, and related disciplines, could learn 
much from the experiences of senior 
researchers. Doctoral students are exposed 
to courses or seminars about history of the 
discipline as well as the field’s overarching 
purposes and its important contributors. .  

A second audience for this project 
include the practitioners and researchers in 
disciplines represented by the chapter 
authors. This audience could learn from the 

experiences of eminent researchers—how 
their experiences shaped their work, and 
what they see as their major contributions—
and readers might relate their own work to 
that of the scholars. Invitations to potential 
authors were accompanied by Tobias’ 
chapter in this series for illustrative 
purposes. Authors were advised that they 
were free to organize their chapters as they 
saw fit, provided that their manuscripts 
contained these elements: 1) their perceived 
major contributions to the discipline, 2) 
major lessons learned during their careers, 3) 
their opinions about the personal and 4) 
situational factors (institutions and other 
affiliations, colleagues, advisors, and 
advisees) that stimulated their significant 
work. 

We hope that the contributions of 
distinguished researchers receive the wide 
readership they deserve and serves as a 
resource to the future practitioners and 
researchers in these fields. 
  



Acquired Wisdom/Review  36 

 
Acquired Wisdom  

dited by 

          Sigmund Tobias                       J. D. Fletcher                 David C. Berliner 

University at Albany             Institute for Defense Analyses         Arizona State University 

    State University of New York              Alexandria VA          Tempe AZ 

 
Advisory Board Members 

Gustavo Fischman, Arizona State University 
Arthur C. Graesser III, Memphis State University 

Teresa l. McCarty, University of California Los Angeles 
Kevin Welner, Colorado State University 

 

Education Review/Reseñas Educativas/Resenhas Educativas is supported by the edXchange 
initiative’s Scholarly Communications Group at the Mary Lou Fulton Teachers College, 

Arizona State University. Copyright is retained by the first or sole author, who grants right of 
first publication to the Education Review. Readers are free to copy, display, and distribute this 
article, as long as the work is attributed to the author(s) and Education Review, it is 
distributed for non-commercial purposes only, and no alteration or transformation is made 
in the work. More details of this Creative Commons license are available at 
http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by 
the author(s) or Education Review. Education Review is published by the Scholarly 
Communications Group of the Mary Lou Fulton Teachers College, Arizona State University.  
 
Please contribute reviews at http://www.edrev.info/contribute.html. 
 
Connect with Education Review on Facebook (https://www.facebook.com/pages/Education-
Review/178358222192644) and on Twitter @EducReview 

 
 

http://www.edrev.info/contribute.html
https://www.facebook.com/pages/Education-Review/178358222192644
https://www.facebook.com/pages/Education-Review/178358222192644