Developing Training Exemplars for the Requisite Components of Visual Threat Detection.

Full text

Technical Report 1322
Developing Training Exemplars for the Requisite
Components of Visual Threat Detection
Laura Zimmerman and Shane Mueller
Applied Research Associates
James Daniels
Dynamics Research Corporation
Christopher L. Vowels
U. S. Army Research Institute
September 2012
United States Army Research Institute
for the Behavioral and Social Sciences
Approved for public release; distribution is unlimited.

U.S. Army Research Institute
for the Behavioral and Social Sciences
Department of the Army
Deputy Chief of Staff, G1
Authorized and approved for distribution:
BARBARA A. BLACK, Ph.D. MICHELLE SAMS, Ph.D.
Research Program Manager Director
Training and Leader Development
Division
Research accomplished under contract
for the Department of the Army
Applied Research Associates, and
Dynamics Research Corporation
Technical review by
Jennifer Murphy, U.S. Army Research Institute
Thomas R. Graves, U.S. Army Research Institute
NOTICES
DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.
Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, ATTN: DAPE-ARI-ZXM,
6000 6th Street (Bldg 1464 / Mail Stop: 5610), Ft. Belvoir, Virginia 22060.
FINAL DISPOSITION: Destroy this Technical Report when it is no longer needed. Do
not return it to the U.S. Army Research Institute for the Behavioral and Social Sciences.
NOTE: The findings in this Technical Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.

i
REPORT DOCUMENTATION PAGE
1. REPORT DATE (dd-mm-yy)
September 2012
2. REPORT TYPE
Final
3. DATES COVERED (from. . . to)
June 2010 – April 2011
4. TITLE AND SUBTITLE
Developing Training Exemplars for the Requisite Components of Visual
Threat Detection
5a. CONTRACT OR GRANT NUMBER
W74V8H-04-D-0048
5b. PROGRAM ELEMENT NUMBER
622785
6. AUTHOR(S)
Laura Zimmerman, Shane Mueller (Applied Research Associates);
James Daniels (Dynamics Research Corporation); and
Christopher L. Vowels (U.S. Army Research Institute)
5c. PROJECT NUMBER
A790
5d. TASK NUMBER
370
5e. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
8. PERFORMING ORGANIZATION REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)
U. S. Army Research Institute
for the Behavioral & Social Sciences
6000 6TH Street (Bldg. 1464 / Mail Stop 5610)
Fort Belvoir, VA 22060-5610
10. MONITOR ACRONYM ARI
11. MONITOR REPORT NUMBER
Technical Report 1322
12. DISTRIBUTION/AVAILABILITY STATEMENT
Distribution Statement A: Approved for public release; distribution is unlimited.
13. SUPPLEMENTARY NOTES
COR and Subject Matter POC: Christopher L. Vowels
14. ABSTRACT (Maximum 200 words):
In the first stage of this research, a model of visual threat detection, the threat detection loop, was developed and requisite
components of that model identified (Zimmerman, Mueller, Grover, & Vowels, In Preparation). The primary components of
visual threat detection were determined to include dynamic threat monitoring, threat prioritization, and causal reasoning. This
second report describes a final quasi-experiment involving those primary components. Results from all research informed the
development and refinement of a training exemplar that consists of exercises involving each of the components. The exemplar
was improved based on feedback from Soldiers with operational experience and from instructors/trainers with involved in
training threat detection skills.
15. SUBJECT TERMS
threat detection, causal reasoning, irregular warfare, attention, memory
SECURITY CLASSIFICATION OF
19. LIMITATION OF
ABSTRACT
20. NUMBER OF
PAGES
21. RESPONSIBLE PERSON
(Name and Telephone Number)
16. REPORT
Unclassified
17. ABSTRACT
Unclassified
18. THIS PAGE
Unclassified
Unlimited 61
Dorothy Young
703-545-2316
Standard Form 298
Applied Research Associates, Inc.
1750 Commerce Blvd, North
Fairborn, OH 45324-6362

ii

iii
Technical Report 1322
Developing Training Exemplars for the Requisite
Components of Visual Threat Detection
Laura Zimmerman and Shane Mueller
Applied Research Associates
James Daniels
Dynamics Research Corporation
Christopher L. Vowels
U. S. Army Research Institute
Fort Hood Research Unit
Scott B. Shadrick, Chief
U.S. Army Research Institute for the Behavioral and Social Sciences
6000 6th Street, Bldg. 1464
Fort Belvoir, VA 22060
September 2012
Army Project Number Personnel, Performance,
622785A790 and Training Technology
Approved for public release; distribution is unlimited.

iv
ACKNOWLEDGEMENTS
We offer our sincere thanks to the Soldiers who have provided insights and participated
in the research on the present topic. Their efforts continue to assist in our understanding of the
Contemporary Operational Environment. We would also like to thank a number of individuals
in the Fort Hood training community who have repeatedly provided valuable advice and
suggestions during this research.

v
DEVELOPING TRAINING EXEMPLARS FOR THE REQUISITE COMPONENTS OF
VISUAL THREAT DETECTION
EXECUTIVE SUMMARY
Research Requirement:
Visual threat detection continues to be a necessary task carried out by Soldiers in the
operational environment on the majority of (if not every) mission. In this research, the primary
processes of visual threat detection were explored to improve understanding and provide a
substantive model upon which further research could be conducted. As Soldiers are tasked to
operate within and around local populations, it is imperative that they have a solid understanding
of what cues represent likely threats and deserve the appropriate attention and response.
Procedure:
Stimuli were refined from previous research and used in a final experiment. The final
experiment allowed for a test of the threat detection loop which is the model of expert threat
detection developed in the first phases of this research. Soldiers participated in several
computer-based exercises and then took part in semi-structured interviews where they provided
feedback on their experiences involving visual threat detection. Those results were incorporated
with the previous findings and were used in the development of a training exemplar. The
exemplar was refined based on a formative evaluation in which Soldiers and trainers provided
feedback on usability and content.
Findings:
The results revealed that the threat detection loop is a viable conceptualization of
experience-based threat detection. As Soldiers gain experience with the primary components of
threat detection, they develop expertise and are better able to discern what constitutes a valid
threat in operational settings. Both quantitative and qualitative results suggest that experience, as
described by the model, does positively impact the ability to visually search for and make a
determination about threat cues and overall threatening situations. Soldiers and
instructors/trainers provided constructive feedback on the training exemplar; they, likewise,
noted that the exemplar would be useful for newer, less experienced Soldiers as well as for
combat veterans who might need re-training on threat detection skills prior to deployment.
Utilization and Dissemination of Findings:
The model developed for this research can be used to guide further research on visual
threat detection. The materials developed to test the model can also be used to explore future
questions to improve understanding of visual threat detection. The research based training
exemplar can be distributed to operational units to bolster existing training or stand alone as an
example demonstrating the primary components of visual threat detection. The exemplar was
transitioned to a deployed Brigade Combat Team (BCT) to enhance their training in theater as

vi
well as to a training support battalion to augment extant training that battalion offered as part of
their pre-mobilization training regimen.

vii
DEVELOPING TRAINING EXEMPLARS FOR THE REQUISITE COMPONENTS OF
VISUAL THREAT DETECTION
CONTENTS
Page
INTRODUCTION ................................................................................................................... 1
Threat Detection Model ...................................................................................................... 3
Expertise Development ....................................................................................................... 5
EXPERIMENT ........................................................................................................................ 7
Overview ............................................................................................................................. 7
Method ................................................................................................................................ 7
Participants ....................................................................................................................... 7
Materials .......................................................................................................................... 9
Computer-controlled Questionnaire .............................................................................. 9
Computer-controlled Imagery Analysis ........................................................................ 9
Interview Protocol ......................................................................................................... 9
Procedure ......................................................................................................................... 10
Prioritized threat search ................................................................................................ 10
Dynamic threat detection .............................................................................................. 11
Change detection ........................................................................................................... 12
Interviews ...................................................................................................................... 13
Analysis............................................................................................................................. 14
Threat Detection Exercises ............................................................................................ 14
Interviews ....................................................................................................................... 14
Results and Discussion ........................................................................................................ 15
Threat Detection Exercises ............................................................................................... 15
Prioritized Threat Search .............................................................................................. 15
Dynamic threat detection .............................................................................................. 18
Change Detection .......................................................................................................... 20
Interviews: Threat Detection Model ............................................................................ 22
Interviews: Expertise Development ............................................................................. 23
Types of Threats ........................................................................................................... 25
Threat Cues ................................................................................................................... 27
Strategies for Threat Detection ..................................................................................... 28
Threat Detection Tasks ................................................................................................. 29
Threat Detection Skills ................................................................................................. 30
Challenges in detecting threats ..................................................................................... 32
Solutions ....................................................................................................................... 33
SUMMARY OF RESEARCH ................................................................................................. 33

viii
CONTENTS (Continued)
Page
EXEMPLAR TRAINING DEVELOPMENT ......................................................................... 34
Design ............................................................................................................................. 34
Development ................................................................................................................... 37
Formative evaluation ...................................................................................................... 38
SUGGESTIONS FOR METHODS REFINEMENTS AND EXTENSIONS ......................... 39
Dynamic Video Imagery ................................................................................................. 39
Attention-tracking and Eye-tracking Hardware .............................................................. 39
Immersive Training Environments ................................................................................. 40
CONCLUSION ........................................................................................................................ 41
REFERENCES ........................................................................................................................ 43
APPENDIX A. DEMOGRAPHIC QUESTIONS .................................................................. A-1
APPENDIX B. OVERLAY INDICATING LOCATIONS SOLDIERS CLICKED
ON TO IDENTIFY POTENTIAL THREATS .............................................. B-1
APPENDIX C. PHOTO SELECTED FOR THE THREAT DETECTION EXERCISES ..... C-1
APPENDIX D. STORYBOARD EXAMPLE USED IN TRAINING EXEMPLAR
DEVELOPMENT ......................................................................................... D-1
LIST OF TABLES
TABLE 1. JUNIOR ENLISTED AND NCO MEAN AGE, TIME IN SERVICE, AND
NUMBER OF DEPLOYMENTS ..................................................................... 7
TABLE 2. SOLDIER RANK AND NUMBER OF DEPLOYMENTS ............................. 8
TABLE 3. SOLDIER MOS AND DEPLOYMENT STATUS .......................................... 8
TABLE 4. INTERVIEW DATA CATEGORIES .............................................................. 15
TABLE 5. LINEAR REGRESSION OF CLICKS ON PREDETERMINED TARGETS
BY DEPLOYMENT AND TIME PRESSURE ................................................ 17
TABLE 6. LINEAR REGRESSION OF DENSITY BY DEPLOYMENT AND TIME
PRESSURE ....................................................................................................... 18

ix
CONTENTS (Continued)
Page
TABLE 7. MEAN ACCURACY AND TIME FOR THE THREAT SEARCH
EXERCISE ....................................................................................................... 19
TABLE 8. DISTRIBUTION OF TARGET IDENTIFICATIONS IN CHANGE
DETECTION EXERCISE ................................................................................ 21
TABLE 9. NUMBER OF SOLDIERS WHO MADE COMMENTS IN EACH
INTERVIEW DATA CATEGORY ................................................................. 23
TABLE 10. NUMBER OF SOLDIERS WHO DISCUSSED EACH COMMON
TYPE OF THREAT .......................................................................................... 26
TABLE 11. RELATIONSHIP BETWEEN LEARNING OBJECTIVES AND THE
THREAT DETECTION LOOP ........................................................................ 35
LIST OF FIGURES
FIGURE 1. THREAT DETECTION LOOP AND MEASURABLE BEHAVIORS
OF EXPERT THREAT DETECTION SKILL ................................................ 4
FIGURE 2. SCREENSHOT OF PRIORITIZED THREAT SEARCH EXERCISE .......... 11
FIGURE 3. SCREENSHOT OF DYNAMIC THREAT DETECTION EXERCISE ......... 12
FIGURE 4. SCREENSHOT OF CHANGE DETECTION EXERCISE ............................ 13
FIGURE 5. TWO MEASURES OF SEARCH ACCURACY, PLOTTED BY TIME
PRESSURE AND EXPERIENCE ................................................................... 16
FIGURE 6. RELATIONSHIP BETWEEN BIAS TO DETECT THREAT-RELEVANT
VS. THREAT-IRRELEVANT LOCATIONS AND NUMBER OF TIMES
DEPLOYED .................................................................................................... 20
FIGURE 7. PHOTO DISCUSSED BY SOLDIERS THAT REVEALED
EXPERIENCED-BASED REASONING DIFFERENCES ............................ 24
FIGURE 8. PHOTO THAT LED TO DISCUSSION WITH SOLDIERS
CONCERNING THREAT STRATEGIES...................................................... 28
FIGURE 9. PHOTO THAT LED TO DISCUSSION WITH SOLDIERS CONCERNING
THINKING LIKE THE ENEMY AS A THREAT DETECTION SKILL .... 31

x
CONTENTS (Continued)
Page
FIGURE 10. SCREENSHOT OF THREAT SEARCH EXERCISE...................................... 36
FIGURE 11. SCREENSHOT OF REASONING EXERCISE ............................................... 37
FIGURE 12. SCREENSHOT OF THREAT RELEVANCE EXERCISE.............................. . 38

1
DEVELOPING TRAINING EXEMPLARS FOR THE REQUISITE COMPONENTS OF
VISUAL THREAT DETECTION
Introduction
Threat detection is an important skill that United States Army Soldiers must utilize when
working in the Operational Environment (OE). Recognizing that the ability to detect threats is
vital, the Army has focused on developing technologies to assist Soldiers in extending this
capability. While certain technologies, like unmanned aerial vehicles, can be particularly useful
for providing a bird’s eye view of an area of operations, they may not always be available.
Moreover, these technologies are intended to augment, not replace the human ability to detect
threats. Threat detection relies on a number of perceptual skills such as attention management,
pattern matching, and change detection as well as higher-order cognition. Humans must reason
about the threats, incorporating their previous experiences and knowledge about the context to
assess whether a particular cue might indicate a potential threat.
A large body of scientific research provides empirical evidence that can guide our
understanding of how these skills should manifest in the OE with Soldiers of varying experience
levels. Leveraging this research base, several related research projects were conducted to better
understand and, ultimately, improve visual threat detection performance in the OE. By relying
on current psychological theories of visual attention and expertise, military doctrine, and
Soldiers’ threat detection experiences during recent deployments, this research aimed to achieve
a greater understanding of how Soldiers detect threats in the OE. Culmination of that
information resulted in the development of a threat detection model, the Threat Detection Loop,
which is discussed in the next section. The objective was to use this improved understanding of
the threat detection process to create training to enhance Soldiers’ threat detection performance.
The goals of this research were to:
• identify the cues, skills, and strategies required to detect threats in the OE,
• differentiate the cognitive skills employed by experienced versus novice Soldiers, and
• create training exemplars that enhance the cognitive threat detection skills of Soldiers.
This report presents a culminating experiment intended to tie together our research in this
area and provides details about the training exemplar developed to enhance Soldiers’ threat
detection skills. The purpose of the experiment was to test the cognitive and perceptual skills of
Soldiers who had varying levels of experience detecting threats in the OE. Soldiers in this
experiment had experience levels ranging from zero to four deployments. To test their skills,
Soldiers completed a series of computer exercises that examined their performance while doing
(a) prioritized threat search, (b) dynamic threat detection, and (c) change detection. This
research also sought to identify any differences in reasoning about potential threats in the context
of various OE relevant situations. Soldiers participated in interviews and discussed the threat
cues and situational elements depicted in photographs of U.S. troop activities and terrain in Iraq
and Afghanistan.
A previous report documented the procedures and findings of the first research phases
conducted (Zimmerman, Mueller, Grover, & Vowels, In Preparation). The initial step toward

2
understanding threat detection in the OE was to review the scientific literature and gather
information about the cognitive and perceptual processes that factor into threat detection.
Analysis of this research provided insight into the mechanisms that lead to threat detection and
guidance on how to improve threat detection. Three preliminary research phases followed the
literature review. The purpose of these research phases was to investigate the cognitive
processes of Soldiers during threat detection exercises and gather information from experienced
Soldiers about threat detection in the OE. Findings from these research phases formed the
foundation of the current experiment and the training exemplar.
The research in Phase I provided initial understanding of threat detection in the OE.
Soldiers at Fort Hood, TX, participated in interviews and completed questions (Phase 1a) and
police officers on patrol participated in ride-along observations (Phase 1b). This initial research
provided information about threat detection situations and activities in the OE and the case-study
account of patrol activities by police officers provided insight into situations with high threat
likelihood. The results from this research supported the findings highlighted in the literature
review about the importance of threat search, change detection, and attention management when
trying to detect threats. Observations of police officers with varying levels of experience
provided examples of threat detection that occurred in their patrol environments. Those
environments required constant vigilance to detect, assess, and react to potential threats. The
findings illustrated possible differences in threat detection processes of less and more
experienced threat detectors.
In Phase II, Soldiers provided data by completing an imagery analysis exercise,
questionnaires, and in-depth interviews. This allowed for a test and refinement of research
materials and development of a threat detection model. In the imagery analysis exercise,
Soldiers identified areas of potential threats and annotated these areas with reasons why these
areas were a concern. This analysis provided information about the regions of interest, threat
cues, and explanations about the threats. From this, materials were created to measure attention
management, threat search activities, and causal reasoning about threat environments. In the
questionnaires, Soldiers from combat arms, combat service, and combat service support reported
similar levels of concern and difficulty across the different types of threats, with the greatest
concern being for relevant operational threats such as Improvised Explosive Devices (IED) and
suspicious persons and vehicles. During interviews, Soldiers discussed how they gather
information, spot cues and trends, scan environments, observe changes, aggregate information,
and form mental pictures of situations. Their descriptions of the types of threats and cues present
in threat situations feed into research stimuli development and informed the exercises Soldiers
would complete in the experiments that followed. In addition, these data contributed to the
development and refinement of the threat detection loop that represents a threat detection mental
model and illustrates the threat detection process.
Phase III was a preliminary assessment of the refined research materials. These materials
included exercises focused on dynamic threat monitoring/search, threat prioritization, and causal
reasoning. Soldiers completed the exercises and provided feedback about the accuracy and
relevance of the materials. They discussed the photos presented during the exercises and
provided interpretations of the threats present in the photos. They also reasoned about why the
threats were important, rated the degree to which the threats are likely to come to fruition, and

3
described how they would handle similar threats based on their experiences. These findings, as
well as feedback from the Soldiers about the exercises, indicated that the materials developed
would provide a good test of threat detection skill and the photos and exercises were relevant to
current threat detection activities in the OE.
This report presents the findings from the final research phase and consists of a
discussion of the design and development of the training exemplar, including a formative
evaluation and suggestions for refinements and extensions of the research and training. The
report presents information as follows:
• introduction: the remainder of the introduction will discuss the threat detection loop and
research on expertise development,
• research: including discussion of the experiment methods, procedures, and results,
• training design and development: the design includes the process for identifying training
stimuli and content, and the development includes construction of the computer-based
training exemplar and results of two formative evaluations, and
• designs for refinements and extensions: this section suggests the use of eye-tracking
technologies and immersive environments to expand our understanding of threat
detection by examining threat detection performance in more realistic settings such as,
during field-exercises and in Combat Training Centers.
Threat Detection Model
An analysis of the literature and of findings from the initial phases of this research
(Zimmerman et al., In Preparation) led to the refinement of a threat detection loop that represents
the cyclical process engaged in during threat detection. Findings suggested that the primary
components of visual threat detection are (See Figure 1):
• dynamic threat monitoring or maintaining a vigilant search of the environment among
competing visual cues,
• threat prioritization or identifying threat-relevant versus threat-irrelevant cues, and
• causal reasoning or determining why certain cues are present or, for instance, what the
enemy may interpret as a good concealment location.

4
Figure 1. Threat Detection Loop (inner rectangles) and measurable behaviors of expert threat
detection skill (outer rectangles).
Figure 1 depicts steps in the threat detection loop (inner rectangles) and their
corresponding behaviors or abilities (outer rounded rectangles). For instance, to effectively
monitor and search activities (loop) threat detectors need to encode relevant details (a). Based on
an understanding of visual features related to threats, experienced threat detectors should be able
to classify threats accurately and quickly (b). The loop indicates that experienced threat
detectors are likely to see patterns during visual search that match to their prior experiences, and
thus demonstrate better ability to locate and classify high priority threats (i). Their prior
experience should influence their decisions as they determine an appropriate action (d) and
enhance their ability to detect changes and anomalies in their environments (g). Based on
experience, these threat detectors should also demonstrate ability to construct causal stories
about situations (h) and predict how the threats might materialize. This ability to incorporate
information and construct stories should allow them to identify and classify threats in a selective
manner (c), which would influence their decision threshold for taking action. Threat detector
ability to weight the strength of the evidence regarding presence and severity of potential threats
(f) also influences this threshold. Because threat detectors can incorporate their experiences as
they process relevant information to identify threats, this model predicts they will demonstrate
ability to identify appropriate courses of action (e).
The loop consists of several processes that Soldiers develop as they acquire threat
detection knowledge and experience. Because the research conducted in Phases I through III
focused on exercises such as threat prioritization and causal reasoning, the data revealed
informational cues and experiences that are most important for gaining knowledge and
developing the ability necessary for visual threat detection. To identify the processes that form
the threat detection loop, we relied on previous research regarding decision-making models,
vigilance and detection loops, and models of expert decision-making (Boyd, 1987; Klein, 1998;
Klein, Calderwood & Clinton-Cirocco, 1986; Mueller, 2009; Mueller & Weidemann, 2008).
The model allows us to make predictions about Soldiers’ performance as they gain competency

5
and develop toward expert stages. According to the model, proficient Soldiers should be better
able to monitor and search their environment in order to detect changes or anomalies.
One goal of this research was to identify differences in experienced and novice threat
detection processes, thus the model provided a guideline for identifying metrics of expertise.
Each step in the loop is associated with measureable behaviors that indicate the influence of
experience on threat detection. To determine how behaviors contribute to threat detection, it is
possible to measure these behaviors in exercises or laboratory settings. As experience increases,
Soldiers may improve their ability to perform many of the component skills required to detect
threats.
This research examined several of the behaviors identified in this model to detect the
presence of these skills in more experienced Soldiers and to create training to improve threat
detection skills in less experienced Soldiers. Given this model, we expected to find a difference
in the skills of more experienced Soldiers compared to the skills of less experienced Soldiers.
For example, experienced Soldiers should more readily identify threat versus non-threat features
compared to less experienced Soldiers who may not have the experience and subsequent mental
models to understand which features indicate a relevant threat. It is also likely that experienced
Soldiers would make comments about these features that differ from comments made by non-
deployed Soldiers. Soldiers with little experience may not be proficient at identifying cues or
features in the environment that are relevant to detecting threats. They may also have difficulty
distinguishing between cues that are relevant and those that are irrelevant or typically do not
indicate a threat. Because they lack the relevant experience base, they may not be able to explain
why certain cues are threats in specific contexts or hypothesize about what the enemy might do
in similar circumstances. Experienced Soldiers should be able to reason through why features
indicate a threat and describe hypothetical actions and outcomes.
Expertise Development
The observations of police officers in Phase I indicated that the less experienced officer
tended to respond readily to many potential threats without discriminating between cues based
on threat severity or likelihood. Meanwhile, more experienced officers seemed to evaluate
possible threats with little deliberation and judge threat severity by weighing the probability of
effective outcomes. Research shows that, compared to their novice counterparts, domain experts
are better able to perceive and process information, recognize pertinent cues, and match those
present cues to previous experiences in a manner that facilitates successful action (Ericsson &
Charness, 1994; Goodrich, Sterling, & Boer, 2000). While the initial research phases indicated
that experience and knowledge influence and facilitate the threat detection process, that research
did not compare the cognitive skills of Soldiers with varying degrees of threat detection
experience. In order to create effective training that leveraged Soldiers’ experiences, it was
important to determine if differences in cognitive and perceptual strategies and skills exist.
A training program that improves the cognitive skills used to detect threats should
incorporate exercises that promote novice advancement toward expert performance levels.
Dreyfus and Dreyfus (1986) proposed a five-stage model of skill acquisition to describe how
decision-making changes as experience increases. At the novice stage of this model, decision

6
makers recognize objective facts about situations and apply “context-free” rules to the situation.
As decision makers progress through the advanced beginner and competent stages, they begin to
recognize meaningful elements of specific situations and begin to incorporate those elements into
decisions. As expertise develops through the proficient stage, decision makers begin to perceive
situations as a whole, rules become less important, and decision makers become more flexible
and react faster to incoming information. In the final or expert stage, decision makers intuitively
recognize situations and match them to previous experiences. They use mental simulation to
predict events and outcomes of actions and they deal with uncertainty by story-building and
actively seeking information (Dreyfus & Dreyfus, 1986; Ross, Phillips, Klein, & Cohn, 2005).
This model provides a template for how threat detection proficiency may change as Soldiers
acquire experience. The behaviors predicted by the threat detection loop (presented in Figure 1)
are characteristic of decision makers in the proficient and expert stages of the Dreyfus and
Dreyfus model.
Previous research examining assessments made by novice and experienced police officers
of a traffic stop video provided insight into the differences in perceptions and interpretations of a
threatening event (Zimmerman, 2008). In that research, police officers viewed a video taken
from a patrol car dashboard camera in three segments. At the end of each segment, they
provided their assessment of the situation including cues that indicated a threat, interpretations of
the events, and courses of actions they would take. Novice and experienced police officers
tended to focus on the same types of cues, but differed in how they elaborated on their
observations. Consistent with prior research (Klein, Phillips, Battaglia, Wiggins, & Ross, 2002;
Phillips, Klein, & Sieck, 2004), experienced police officers provided descriptions of the event
that were more elaborate, provided and interpreted more cues, and made more interpretations and
predictions about what would occur. This finding is similar to previous research showing that
less and more experienced decision makers use similar sensemaking strategies, but experts show
a deeper understanding of the situation and provide richer explanations (Klein et al., 2002).
Novice officers tended to focus on procedural issues, such as officer safety, and they rarely tried
to ascribe meaning to threatening behaviors or interpret the scene in multiple ways. Experienced
officers tended to consider the context surrounding the situation and make interpretations based
on their prior knowledge of officer, driver, and criminal behavior. They weighed the relevance
and importance of cues within the context of the current situation whereas novice police officers
identified threat cues and based their probable actions on the cues without necessarily
considering the context.
In Zimmerman, et al., (In Preparation), Soldiers with combat experience in the OE
emphasized the importance of experience when searching environments to detect threats. They
reported threat search processes that focused on procedural tasks, such as using intelligence and
typical information gathering techniques, but, unlike the novice police officers, they did not tend
to focus on Soldier safety. Instead, they provided examples from their experiences of techniques
they use to search for threat cues, such as noticing atypical threat indicators and leveraging the
opinions of other Soldiers. Soldiers were able to verbalize the learning process that occurred
during their deployment, particularly how they learned to sort and filter information and
distinguish between threats and non-threats. Their comments also indicated that they could
process threat cues, predict the consequences of those cues, and take preventative actions. They
also recognized that some threat cues were too prevalent or random to spend much energy

7
detecting, such as trash along the side of the road. Soldiers’ comments indicated that they
consider their surroundings when interpreting threat cues, for instance, by noticing trends and
changes and reassessing situations given the new context. The Soldiers in Phase III viewed
photos and not only provided comments about the threat cues present but also were able to
present detailed hypothetical scenarios based on the scenes in the photos and discuss plausible
actions they would take. These comments are characteristic of experience-based processing and
interpretations of situations.
Experiment
Overview
Based on the expert threat detection model (Figure 1) and the results from the previous
experiments, subsets of the stimuli were piloted during the Phase III research (Zimmerman, et
al., In Preparation). These stimuli provided an opportunity to identify useful targets for use in
the training exemplar development. The current research added a change detection exercise to
investigate differences in the ability to detect changes based on experience and to understand the
extent that Soldiers use change detection to identify threats. During the initial phases of this
research, Soldiers indicated that detecting changes is an important component of threat detection,
as represented in the threat detection loop.
Soldiers at Fort Hood, TX, engaged in three computer exercises: a prioritized threat
search, dynamic threat detection, and change detection. Upon completion of those exercises, the
Soldiers viewed photos during interviews and identified threat cues, discussed these cues and
areas of concern, and provided their interpretation of the scenes.
Method
Participants.
Forty-seven U.S. Army Soldiers participated in computer exercises and interviews.
Twenty Soldiers were NCOs (Sergeant; SGT to Master Sergeant; MSG), one was an Officer
(First Lieutenant; 1LT), and 26 were enlisted (Private; Private to Corporal; CPL/ Specialist;
SPC). All Soldiers were male. Table 1 lists the average age, mean time in service, and the mean
number of times deployed for each group.
Table 1
Junior enlisted and NCO mean age (range), time in service (range), and number of
deployments
Rank
Mean Age
Mean Time in Service
% Deployed
Junior Enlisted
23 (18-37)
27 mo. (4-72)
54%
NCO/Officer
29 (23-44)
103 mo. (31-192)
95%

8
Soldiers reported zero to four deployments. Thirty-four Soldiers reported deploying at
least once. Of those who deployed, 83% were combat arms. Table 2 lists the number of Soldiers
in each rank by the number of times they have deployed.
Table 2
Soldier rank and number of deployments
Number of Deployments
Rank
Total
0
1
2
3
4
Private
2
1
1
PV2
3
3
PFC
11
8
3
CPL/SPC
10
7
3
SGT
1
1
SSG
11
1
1
6
3
SFC
7
3
3
1
MSG/1SG
1
1
1LT
1
1
Total
47
13
14
12
7
1
The most common military occupational specialty (MOS) reported by Soldiers was
Infantryman (11B), with 15 deployed and 2 non-deployed Soldiers reporting this MOS (Table 3).
Only one Soldier reported a current MOS different from the MOS held while deployed. This
Soldier reported that his current MOS was Armor Officer (19A), however, his MOS while
deployed was a Motor Transport Operator (88M).
Table 3
Soldier MOS and deployment status
Deployed
MOS
Yes
No
11B Infantryman
15
2
12B Combat Engineer
5
2
13F Field Artillery Forward Observer
3
19A Armor Officer
1
19D Armor Scout
2
1
19K M-1 Crewman
4
2
25N Nodal Network Operator (Signal)
1
25Q Multichannel Transmission Systems Operator-
Maintainer
3
74D Chemical, Biological, Radiological, and Nuclear
Warfare Specialist
1
91A M-1 Tank Mechanic
1
91M M-2/3 Bradley Mechanic
1
1
92F Petroleum Supply Specialist
2
Total
34
13

9
Deployed Soldiers rated how often they went ‘outside the wire,’ or how often they
traveled off their Forward Operating Base (FOB) or similar location, on a scale from 1 (rarely or
never) to 4 (almost every day). Twenty-seven Soldiers reported going outside the FOB almost
every day (M = 3.65, SD = 0.77).
Materials.
Computer-controlled Questionnaire. Materials similar to those used in the previous
experiments were used for the current research (see Zimmerman et al., Appendix A). These
included a computer-controlled demographic and experience questionnaire that contained a
subset of the questions asked in the previous research and additional questions relevant to the
current research (Appendix A). Special-purpose software using the Psychology Experiment
Building Language (PEBL) 0.09 computer experimentation system controlled the computer
exercises (Mueller, 2009). The participant section (above) presents the relevant data from this
questionnaire.
Computer-controlled Imagery Analysis. The three computer-controlled threat detection
exercises presented photos selected from a set of 48 images approved for public release. These
images were retrieved from three main sources, www.DefenseImagery.mil, www.Flickr.com,
and www.defense.gov. Findings from the previous experiments guided the photo selection
process for the final set of 40 images. These images were divided into four groups of 10 and
each group of 10 images was assigned to one of four conditions: time-unlimited prioritized
threat search, time-limited prioritized threat search, single-task threat detection, and dual-task
threat detection. Four counterbalanced conditions were devised using a Latin square design and
participants were assigned to each counterbalanced condition based on the order in which they
participated in the data collection.
Twelve additional images were selected as stimuli for the change detection exercise.
Image-editing software enabled the creation of two versions of each image by seamlessly adding
or removing specific features from the images. Changes included both threat-relevant features
(brickwork, trash, people, vehicles, etc.), and threat-irrelevant features (the length of poles,
ornamental architecture detail, etc.). Soldiers viewed one version of each of the change-
detection images during the initial prioritized threat search exercise and tried to detect changes
on the second version of each image at the end of the sessions.
Interview Protocol. The interviews focused on gathering information about the causal
reasoning used to make threat detection choices during the computer exercise. The set of photos
used in the computer exercises and previous research were used to gather information about
Soldier thought processes and reasoning while detecting threats.

10
Procedure.
The research consisted of two main parts: a computer-controlled exercise and a semi-
structured interview. At the start of each session, following administration of privacy act and
informed consent protocols, Soldiers read a description of their exercises and were told that they
were there to provide information about their threat detection experiences and to identify
common threat detection situations, indicators, and challenges in the OE. Soldiers then
completed the computer exercises at their own pace which typically took 45 minutes to
complete.
Following the demographic questions, Soldiers completed five consecutive computerized
exercises developed to measure attention, search, reasoning, and prioritization strategies for
threat detection. The first two exercises involved a resource-limited threat search process
referred to as the prioritized threat search. The second two exercises involved dynamic threat
detection in which Soldiers engaged in a stimulus detection exercise with distracters. Finally, in
the change detection exercise Soldiers viewed changed imagery which they first saw during the
prioritized threat search exercise and they annotated any changes observed.
Prioritized threat search. The first two computer-controlled exercises following the
demographic questionnaire required Soldiers to search an image for potential threat targets using
a virtual “scope” (Figure 2). Each image contained approximately 10 possible target locations.
The annotations provided by Soldiers in the previous experiments formed the basis for these
target locations. The click locations from previous experiments were not suitable for direct use
because they generally contained too many targets, thus a small number of reasonable and
distinct target locations was selected based on visual examination of the annotated imagery.
When Soldiers clicked the 100-pixel diameter circular scope on one of the target locations, a red
dot (identified target) would display with a 65% probability rate of detection. This probability
allowed participants to conduct repeated searches of the image in locations they suspected of
containing a target. For each image, Soldiers could make a maximum of 10 search clicks. The
first search exercise had no time limit and contained 18 images. The second search exercise also
had 18 images. Soldiers had a maximum of one second per click to search each photo (if they
did not use the click after one second it would become unavailable) and after each click the timer
decremented to the next even remaining second. This scheme encouraged Soldiers to search
steadily throughout the 10second time period rather than click haphazardly when time was about
to elapse. Each condition in the prioritized threat search exercise contained 12 images from the
main set of counterbalanced photos and an additional six images used in the first part of the
change detection exercise. Soldiers did not know that the change-detection images would appear
again later.

11
Figure 2. Screenshot of prioritized threat search exercise.1
Dynamic threat detection. The design of the cued threat detection exercise allowed
experienced Soldiers to use their background knowledge of threats to guide their visual attention.
On each trial, the screen displayed an image overlaid with 75 transparent grey dots (noise)
distributed randomly, but consistently over the image.2
These dots vibrated and obscured the
image. These grey dots obscured the appearance of the target dot and required Soldiers to
maintain constant vigilance in order to detect the target among the field of dots. The target dot
was identical to the noise except that it had a light orange rim around the circle. The Soldiers’
task was to click the mouse button as soon as a target was detected which revealed a mouse
cursor. Once the cursor appeared, they could indicate the location of the target. Each image
presented seven targets in threat-consistent locations and three targets in threat-inconsistent
locations. The threat-consistent locations were determined by examining responses from
previous research with the same or similar imagery.
1 The red bar on the right of image shows remaining search clicks available. Found targets are displayed as
transparent red dots and the virtual threat scope as a transparent green overlay.
2 Each transparent dot had a radius of 12 pixels. Throughout the trial, the location of these dots was jittered with a
bivariate uniform distribution of five pixels. The dot locations updated approximately every 100 milliseconds. A
target dot appeared in the dot field at random delays of between 5 and 10 seconds.

12
Figure 3. Screenshot of dynamic threat detection exercise.3
A second round of the dynamic threat detection exercise required simultaneous
monitoring of changes occurring on a secondary light monitor. The light monitor was located in
the lower right section of the image and had four virtual LED lights that flashed either blue or
red (Figure 3). On each cycle, three lights would illuminate, typically, one would be red and two
would be blue. When the target appeared, three of the four lights would flash red, but would still
update on the same cycle so the three red lights would appear for the same length of time as the
other combinations of blue and red lights. Soldiers needed to monitor both the photo for the
target (grey dot with orange outline) and the light display for three red lights. When they
detected the three red lights, they would click on the screen outside the photo rather than on the
location of the target location in the photo. The purpose of this manipulation was to assess the
degree that threat detection experience affected performance when simultaneously performing
multiple tasks. Research on novice and expert performance shows that novices typically must
consciously attend to the task they are performing while experts do not need to apply as much
conscious thought and attention (Beilock & Carr, 2004; Beilock, Wierenga, & Carr, 2002;
Shiffrin & Schneider, 1977). As a result, experts should be able to devote more attention to the
dual task without a performance decrement while novices should perform poorly under dual task
conditions.
Change detection. The final computer exercise involved “change detection,” in which
Soldiers viewed altered versions of 12 images they had seen during the prioritized threat search
exercise. They viewed half of the images in the untimed version of the exercise and the other
half in the timed version with the photos counterbalanced across Soldiers. In addition, the first
3 At this point, three of the secondary monitor lights were illuminated.

13
version of the change photos shown appeared at random so that half the Soldiers saw the original
image during the exercise, whereas the other half saw the altered version. The changed images
were intermixed randomly with the normal threat search imagery, thus Soldiers had no indication
that these images would later appear as a test of their change detection skill. At test, Soldiers
were told that several things were different about the image compared to the first time they had
seen it (Figure 4). They used the mouse to indicate a location and gave a brief annotation
describing what they believed had changed.
Figure 4. Screenshot of change detection exercise.4
Interviews. After the computer exercise, the research staff interviewed Soldiers about
their threat detection experiences. The interviews lasted approximately 30 minutes. Soldiers
were instructed to look through a set of images (that had been used in the computer tests) and
select one or more that reminded them of a situation they had experienced (in training and/or
operational environments). They identified and described the threats in each photo and provided
explanations about the threats they deemed important. They discussed the threat-related events
that might occur in each photo and discussed possible threat scenarios. Finally, Soldiers
provided feedback about the computerized exercises and made suggestions for computer-based
training exemplar development. During the debriefing, Soldiers could ask questions and provide
feedback about the experiment and the project. They received contact information for mental
health assistance if discussion of their experiences brought up any negative feelings or memories.
4 Participants indicated locations of changes and annotated what they believed the change was.

14
Analysis.
Threat Detection Exercises. The R statistical computing platform Version 2.10.1 (R
Core Development Team, 2009)5
was used to analyze the data from the computerized exercises.
R provided a number of useful add-on modules that enabled image analysis and display. In
addition, for the prioritized threat search exercise, R created images that overlaid transparent red
indicators on top of the original stimuli to indicate the clicked locations of all participants
(Appendix B).
Interviews. To conduct the interview analysis, interview recordings were transcribed.
The Soldiers’ comments were sorted into pre-determined categories, which had been developed
during the previous phases of research (Zimmerman et al., In Preparation). These categories
include types of threats, threat cues, strategies for threat detection, threat detection tasks, skills,
challenges, and training (Table 4). Because Soldiers with different amounts of experience
commented on the same photos, we compared comments made by less and more experienced
Soldiers in each of these categories and compared comments made by multiple Soldiers to the
same photos. Interviews were coded line by line to identify potentially important pieces of data.
We compiled these specific comments according to these categories and kept the coder blind to
each Soldier’s characteristics. After comments were sorted into each broad category, specific
comments were separated out according to finer distinctions. For example, within the category
threat cues, comments could be those related to behavioral, person, or environmental cues.
After all comments were coded according to these categories, we began to identify
elements in the 5-Stage Skill Acquisition Model (Dreyfus & Dreyfus, 1986). Each comment was
represented at a stage in the model so we could ascertain their level of skill development. At that
point, we identified Soldiers’ experience levels and identified comments as belonging to
inexperienced Soldiers (no deployments) or experienced Soldiers (Soldiers with one deployment,
and Soldiers who had between two and four deployments). The number of Soldiers who made
comments in each of the high-level categories were counted and separated according to
experience level. After that, comments were separated within each category according to
experience level so that similarities and differences could be examined within the level of
experience.
In order to identify comments as they related to the threat detection model, we relied on
the same initial coding scheme, although this time solely focusing on the experienced Soldiers.
We then identified each comment according to the threat detection model and sorted the
comments according to each part of the model.
5 R is a commonly used statistical computing tool available for use on all common computing platforms,
http://www.r-project.org/

15
Table 4
Interview data categories
Interview Data Classification
High-level Categories
Types of Threats
Enemy activity, enemy tactics, activity in the
environment, troop activity, crowd behavior
Threat Cues
Behavioral cues, environmental cues,
physical cues (such as wires, garbage),
patterns of cues
Strategies for Threat Detection
Tactics, information gathering, spot cues, spot
trends, ask what-if questions while assessing
situations
Threat Detection Tasks
Scan environment, question subjects, active
evidence search, observe situations and
environments
Threat Detection Skills
Think like the enemy, prioritize information,
aggregate information, form mental picture of
situation, regularly ask what-if questions
Challenges in Detecting Threats
Noticing cues, noticing patterns, enemy
Tactics, Techniques, and Procedures (TTP),
information reliability, night operations
Solutions for Threat Detection
Training/Preparation
Scenario-based training, train consequences,
use Iraqi/Afghan role-players/environments,
think like the enemy, real-world experience,
train with what-if questions
Results and Discussion
Threat Detection Exercises.
Prioritized Threat Search. The prioritized threat search exercise measured performance
with and without time pressure. The number of threats discovered by Soldiers provided a metric
of performance in both conditions. Soldiers found targets in an average of 2.14 clicks while
under time pressure and 2.32 clicks when not under time pressure. A Welch 2-sample t-test
determined these to be significantly different [t(1374) = 2.05, p < .05, Cohen’s d = 0.11],
showing that time pressure reliably reduced the number of targets found. Another way to judge
search accuracy was to compare each Soldier’s click to the other Soldiers’ click locations. For
each click, the number of clicks made by other Soldiers that were within the scope radius of the

16
specified location was counted. This value ranged from 0 to 77 with a mean of 18.4. The
number of clicks within the scope radius indicated whether the location was in a sparse or dense
region. Sparse regions contained fewer than 10 close clicks indicating few Soldiers clicked in
those regions while dense regions contained 10 or more close clicks indicating many Soldiers
clicked in those regions. Approximately 65% of the click locations were in dense regions. For
each Soldier, the number of clicks made in dense regions with and without time pressures were
Figure 5. Two measures of search accuracy, plotted by time pressure and experience.
calculated. Under time pressure, Soldiers made 5.96 dense clicks on average. Under no time
pressure, they made an average of 7.1 dense clicks. This difference was statistically significant
and indicated a medium-sized effect, t(1351) = 8.90, p < .01, Cohen’s d = .48.6
In contrast, there
was no reliable difference between the mean number of sparse clicks in the two conditions
without time pressure (M = 2.90 versus with time pressure (M = 3.07), t (1363) = -1.4, p = .17,
Cohen’s d = .07).
Analysis of Soldier experience focused on the number of deployments. Figure 5 depicts
the relationship between experience and time pressure for clicks in the predetermined threat
regions and clicks in the dense regions. The results show very little difference across experience
levels, but highlight the impact that time pressure has on performance. One explanation for the
relatively larger impact of time pressure on clicks in dense regions is that when under time
pressure Soldiers missed opportunities to search the photo as time elapsed. Similar results were
found when using other measures of performance such as the number of threats identified.
Click density measures were also used; those measures incorporated experience or how well an
individual matched the group. Click density is the most appropriate measure because there were
6 A Cohen’s d of .48 indicates a medium effect size. Effect sizes indicate the strength of the relationship between
two means. When calculating Cohen’s d, a small effect is around .2, medium effect around .5, and large effect .8.

17
no objectively correct threats in the image and thus examining a sparse location could be
considered incorrect. Random clicking on an image suggests an inability to recognize and
indicate relevant threats whereas condensed clicking suggests a focused search of an image for
certain cues.
To determine if the effects of experience on search accuracy were reliable, a linear
regression was computed with linear predictors for number of deployments, time pressure, and a
deployment x time-pressure interaction. When analyzing the number of clicks in which Soldiers
found a target, the number of deployments was unrelated to performance, but time pressure and
the time pressure x deployments interaction were both statistically significant (Table 5). This
interaction shows that the deployment slope is slightly positive (.048 clicks/deployment) when
under time pressure, but slightly negative (-.118 clicks/deployment) when not under time
pressure. This result indicates that Soldiers with more deployments performed slightly better
when under time pressure versus when not under time pressure. However, the results are quite
small overall. Most Soldiers had fewer than two deployments so the effect should be interpreted
with caution.
Table 5
Linear regression of clicks on predetermined targets by deployments and time pressure
Predictor Estimate S.E. t Pr(>|t|)
Intercept 2.362 0.093 25.278 < .001
Deployments -0.035 0.053 -0.655 0.512
Time pressure -0.397 0.132 -3.002 0.003
Deployments x pressure 0.166 0.075 2.194 0.028
Residual standard error: 1.574 on 1372 degrees of freedom
Multiple R-squared: 0.007698, Adjusted R-squared: 0.005529
F-statistic: 3.548 on 3 and 1372 df, p-value: 0.01405
The analysis of clicks in dense regions showed a reliable positive relationship between
the number of deployments and time pressure, but the impact of time pressure was quite large
(by more than one click). Table 6 shows the linear regression results of the model which indicate
only a statistically significant effect of time pressure. Together, these results indicate that time
pressure had an impact on performance. The data also suggest this impact might be larger for
novices and experience reduced or even reversed this effect. Future research could investigate
this effect to understand whether and why time pressure may have potentially less influence on
experienced Soldiers. For instance, more experienced Soldiers are more likely able to sift
through relevant and irrelevant information even in time-pressured situations.

18
Table 6
Linear regression of density by deployments and time pressure
Predictor Estimate S.E. t Pr(>|t|)
Intercept 6.999 0.141 49.64 < .001
Deployments 0.078 0.080 0.97 0.335
Time pressure -1.117 0.199 -5.60 < .001
Deployments x pressure -0.018 0.114 -0.16 0.874
Residual standard error: 2.375 on 1372 degrees of freedom
Multiple R-squared: 0.05566, Adjusted R-squared: 0.05359
F-statistic: 26.95 on 3 and 1372 df, p-value: < 2.2e-16
Dynamic threat detection. The threat search exercise involved searching for targets that
appeared in a dynamic field of transparent dots that jittered throughout the search (Figure 3).
Soldiers responded correctly when they clicked the mouse while the target appeared on the
screen and then indicated the location of the target within 100 pixels of its actual location. While
100-pixel criterion is somewhat arbitrary, Soldiers were able to meet this criterion the majority
of the time. Out of 5,734 total trials in which a target appeared on the image, Soldiers clicked
the mouse to indicate the target was present 3,107 times and 2,598 of these times Soldiers
indicated the target location within 100 pixels. The number of correct and precise trials only
varied from 2,584 to 2,601 when the criterion varied from 50 to 150 pixels, indicating the 100-
pixel criterion was robust.
When analyzing the proportion of correct (clicked mouse when saw target) and precise
(accurately indicated location of target) trials in the single and dual-task search exercises, the
results showed an overall accuracy rate of about 50%. Fewer than half the threat-irrelevant
targets detected and greater than half the threat-relevant targets detected (Table 7). Dual task
trials had higher mean accuracy than single-task trials; however, this finding may stem from
practice effects because the single task was always completed first and the dual-task second. We
intentionally did not counterbalance the order of the single and dual task conditions because the
dual task condition would have been too difficult and the secondary task did not require Soldiers
to have familiarity with the primary task. The photos were different in the single and dual tasks.

19
Table 7
Mean accuracy and time for the threat search exercise
Condition Irrelevant Relevant Monitor
Accuracy
Single 0.446 (.026)* 0.585 (.028) NA
Dual 0.513 (.037) 0.629 (.042) 0.373 (.057)
Time to respond (ms)
Single 4556 (91) 4134 (85) NA
Dual 4513 (106) 4064 (120) 4668 (174)
*Note. Estimate of standard error in parentheses.
To assess whether the differences between conditions were statistically reliable, a model
predicting log (accuracy) based on Condition (single-dual), Relevance, a Condition x Relevance
interaction, and participant enabled within-subject comparisons. We analyzed log (accuracy)
because residuals from the raw accuracy scores were skewed and a normal distribution better
described the residuals from a log (accuracy) model as assumed by the linear regression. This
procedure excluded the trials in which the target appeared as the three red monitor lights (as they
were neither relevant nor irrelevant) although the other trials from the dual task condition were
included. Results of an Analysis of Variance (ANOVA) on this model indicated a statistically
significant effect of threat relevance, F(1,45) = 92, p < .001, η2 = .67, and no reliable effect of
either single versus dual task condition, F(1,45) = 2.1, p = .15, η2 = .044, nor was there a reliable
interaction, F(1,45) = .65, p = .42, η2 = .014.
A second performance measure analyzed was Soldier time-to-respond when they detected
a target. Similar to accuracy, Soldiers responded faster to targets in relevant locations than to
targets in irrelevant locations. Because response time was positively skewed, we performed an
ANOVA on log (reaction time) which again revealed a reliable effect of relevance on response
time, F(1,44) = 6.3, p =.016, η 2 = .12, a reliable effect of the single and dual task conditions,
F(1,44) = 23.7, p <.001, η 2 =.345, and no reliable interaction, F(1,43) = .04, p = .82, η 2 = .001.
Across the two primary measures (accuracy and time-to-respond), there were differences
between threat-relevant and threat-irrelevant target locations which is promising because it
suggests that measuring such differences in a future training system may help provide metrics
about skill level that can be used for assessment and/or feedback. The faster response time in the
dual-task condition compared to single-task condition is counterintuitive. This finding may be a
practice effect or may stem from an overall higher level of arousal induced by the more difficult
dual-task situation. However, the reliable impact of threat relevance indicates that Soldiers were

20
typically able to infer threat-relevant locations and focus their attention in those locations which
degraded their performance on threat-irrelevant trials.
To assess whether Soldiers with more experience differed in their ability to attend to
threat-relevant locations, a threat bias score was calculated for each Soldier by dividing their
mean response time by their mean accuracy. The threat bias score assessed the extent to which
detection is biased toward threat-relevant regions of visual space. A ratio value of 1.0 would
indicate equal performance for both types of targets, values less that 1.0 would indicate a bias
toward threat-relevant locations (i.e., they were identified faster and more often). The natural
logarithm of these values was used to perform a linear regression analysis that compared the log
ratio to number of deployments reported by Soldiers7. A negative threat bias score indicated that
Soldiers were biased toward detecting and identifying targets in threat-relevant regions. Such a
bias is likely to be adaptive and indicates Soldiers are using their limited attentional resources to
monitor the most likely targets, but also indicates that they may miss seeing threats that appear in
unlikely places. The slope (beta = -.31 log-RT units per deployment, t(44) = -.39, p = .69) and
the intercept (-.015) were not significant. This indicates that all Soldiers had a tendency to
identify targets in threat-relevant locations regardless of the number of deployments (Figure 6).
Figure 6. Relationship between bias to detect threat-relevant vs. threat-irrelevant locations and
number of times deployed.
Change Detection. To assess change detection ability, each base image was coded
according to its critical change locations. Coding of the changes in each image indicated the
7 The natural log transformation was used to correct for truncated range between 0 and 1 in cases where mean
accuracy exceeded mean response time.

21
presence, absence, or the change of a feature. In addition, coding differentiated each change as
threat relevant or threat irrelevant. The image set contained 62 coded changes and although each
Soldier saw all 62 images, they responded to only half the images totaling 31 changes per
Soldier. This coding revealed 27 (44%) changes as irrelevant and 35 changes (56%) as relevant.
Results showed several important trends. Across all Soldiers, they annotated 355 targets
but 281, or 79% of these targets did not involve an actual change between the two images. These
281 false alarms were not coded as threat relevant or irrelevant. Of the remaining 74 identified
targets, 71 were in threat-relevant locations, providing evidence that participants noticed changes
related to threats (Table 8). Furthermore, participants were more likely to identify a target when
a feature was absent in the first viewing but present in the second viewing (56 trials) versus when
the target was present in the first viewing, but absent in the second (15 trials).
Table 8
Distribution of target identifications in change detection exercise
Threat status
Uncoded Irrelevant Relevant
Feature present 0 3 53
Feature absent 0 0 15
Feature change 0 0 3
No change 281 0 0
The overall high proportion of false alarms could be evidence that change detection skill
was poor and may even indicate random guessing. However, the proportion of targets actually
found when Soldiers had no warning they should look for changes, indicates an existing ability
to detect change. Although only about 20% of the responses were for actual changes Soldiers
selected about 3% of the image areas as the change zones (ranging from 0.37% to 12.7% across
images). Thus, Soldiers were about 6.7 times more likely to indicate actual changes than if they
had been responding by chance. To assess whether performance differed reliably from chance,
mean accuracy rates (the number change targets identified relative to the total number of
attempts) were compared to the proportion expected by chance. Forty-six Soldiers took part in
the exercise, but two provided no responses. Across the remaining 44 Soldiers, the mean
proportion of attempts that identified a change target was .233 with a 99% confidence interval
between .14 and .32 (three Soldiers did not complete the change exercise because of time
restrictions). The chance proportion of .03 identified change targets lies far outside the region,
t(42) = 5.9, p < .0001, Cohen’s d = 0.90.
Because we were interested in how change detection occurs in an applied setting (as
opposed to in a laboratory), we made use of what is commonly referred to as the incidental
approach (Levin & Simons, 1997). In these situations, participants are not informed that a
change or difference will occur and they are often asked to perform an intermediary task as their
primary task. In Rensink’s review (2002, p. 257) he notes, “The degree of [change] blindness

22
encountered is generally much higher than that found using intentional approaches. However,
some ability to detect change remains.” The results of the change detection exercise used in this
research support that idea. Based on the results, Soldiers may be considered to have had a lower
response threshold or a high false alarm rate. However, when they indicated changes that were
indeed present, they were most likely at threat-relevant locations. Given the severe
consequences of not detecting threat-relevant changes in the operational environment, adopting a
lower threshold is an adaptive strategy.
Interviews: Threat Detection Model.
The analysis presented in this section aimed to link the comments from Soldiers with
deployment experience to the Threat Detection Model proposed by Zimmerman et al., (In
Preparation). These comments provide select examples of behaviors that map onto the loop (see
Figure 1). The next section provided examples specifically relevant to expertise development.
Experienced Soldiers often discussed threat detection exercises that related to their
cognitive and perceptual processes such as encoding details about the situation (see Threat
Detection Loop, Figure 1, box a). For example, Soldiers made the following comments:
• “Just keep an eye out…to keep an eye out for a vehicle and right in here where they’re
sitting.”
• “You have to focus on everything, pay attention to what they’re wearing.”
• “So, I just watched people…I know, okay this time of day, these people are going out and
this time of day, they’re coming back.”
Soldiers also made comments about their information gathering processes that
demonstrate their need to balance making decisions about a threat with the need to gather
information (f).
• “It should be with the local people; they (will) give away everything…they’ll come and
tell you.”
• “Presence patrols; we’d do a lot of them. And a lot of times we’d go to each shop and
talk to people in there and ask them if they noticed anything weird in the area.”
Soldiers with experience often described the challenges related to properly classifying
threat cues and threat irrelevant cues (b) and their ability to detect changes in an environment (g).
They reported:
• “It’s dirt and gravel which is not cool because it’s too easy to mess with dirt and gravel
and make it look the same. Pavement is a little bit easier to notice differences.”
• “I really don’t like looking for IEDs: too many places to put it, to hide it, hard to find, not
enough manpower.”
• “Really, no matter how you looked at it, you could look at almost anyone and decide if
you think they’re suspicious.”

23
• “I mean they changed their tactics every day and every time we changed ours, they’d
change theirs. So it’s hard to…one day it could be a coke can sitting right there and the
next day it could be a tire. You just never know.”
Soldiers also indicated that they often relied on prior experience and knowledge to
determine their search strategies and this, in turn, influenced their ability to detect threats. An
example statement provides illustration of incorporating prior experiences into decisions (d):
• “You’ll see a lot of the areas that they’ve used before. And it’s a big give away, like
they’ll use telephone poles as markers and they’ll use light poles, they’ll use trees,
intersections.”
Interviews: Expertise Development.
This section provides findings from the interviews that represent typical responses from
Soldiers who have never deployed, those who have deployed once, and those who have deployed
two to four times. The purpose of this analysis was to identify differences in causal reasoning
skills between less and more experienced Soldiers as they identified and discussed possible
threats in photographs depicting real-world situations. The results contain information given by
Soldiers in response to the photographs that pertain to the high-level categories (Table 9).
Soldiers in all three groups (0, 1, or 2-4 deployments) were able to identify or discuss types of
threats, threat cues, and threat detection exercises. Soldiers with deployment experience more
often discussed strategies for threat detection, threat detection skills, challenges in detecting
threats, and solutions. The interview analyses explored the content of these differences by
comparing comments made by less and more experienced Soldiers in each category.
Table 9
Number of Soldiers who made comments in each interview data category
Types
of
Threats
Threat
Cues
Strategies
for
Threat
Detection
Threat
Detection
Tasks
Threat
Detection
Skills
Challenges
in
Detecting
Threats
Solutions
Never
Deployed
*N = 13
13 13 8 8 2 4 0
1
Deployment
N = 14 15 15 14 7 7 14 4
2-4
Deployments
N = 19
15 14 13 8 6 10 2
*Note: N = Number of participants who made a comment in each section.

24
Participants provided overall assessments of the photos they viewed. To exemplify these
discussions two Soldiers, a junior enlisted Soldier who had never deployed and an experienced
Soldier who had deployed once, directed their comments on a photograph taken of a cemetery
(Figure 7). The photo they discussed depicted three members of a U.S. force in a cohesive group
and one member who is by himself as they are walking among numerous headstones. During the
interviews, both Soldiers suggested that perhaps the U.S. forces were pursuing someone through
the cemetery or trying to clear the area. The experienced Soldier elaborated on this suggestion,
“or they’re just looking through to clear it and maybe make sure there are no explosives and
nobody hiding in there” and considered additional threats, specifically from snipers.
Figure 7. Photo discussed by Soldiers that revealed experienced-based reasoning differences.
The junior enlisted Soldier focused his comments on the individual separated from the
group and the threat to the lone Soldier stating “That guy by himself is not a very smart idea.
They (the enemy) could be all up over here…basically, right here, this is not right when you’re
clearing anything, you stay together; you don’t ever separate.” This demonstrated the more
procedural and context-free rules to thinking often exhibited by novices (Dreyfus & Dreyfus,
1986). The experienced Soldier was also concerned with the Soldier separated from the group,
but provided suggestions for why he separated from the others:
[T]his guy is on his own and maybe instead of going on his own, he ought to go with
them. Obviously, these guys are clearing around this area…they may need somebody up
there, that way if something does happen they have the high ground already so [that the
lone Soldier] could see around and watch for anything coming up on them.

25
This Soldier was demonstrating his ability to create a causal story about a threat situation (Figure
1, box h) and to hypothesize about possible explanations for why the solitary Soldier was
breaking from procedure and taking what the junior enlisted Soldier perceived as unsafe action,
demonstrating a more proficient skill level.
The junior enlisted Soldier focused on procedural rules discussing how Soldiers should
not enter the cemetery unless they have received permission. The experienced Soldier thought
beyond the situational constraints and discussed why this break in procedure might be a good
strategy, “I’d make sure that we have high ground in case there’s anybody hiding in there as
we’re going through it, to watch for somebody running around...” This difference illustrated
how decision makers incorporate their experience into their assessments of situations (d) to
interpret what they are seeing (Dreyfus & Dreyfus, 1986, Ross, et al., 2005). The experienced
Soldier’s comment also demonstrated that he was able to look at high-priority threat locations
based on knowledge (i) by taking action that reduces the threat.
Types of Threats.
Table 10 lists the types of threats mentioned by Soldiers when viewing the photos and the
number of Soldiers who made explicit reference to these threats. Soldiers identified these threats
spontaneously rather than from a list of threat types, thus similar types are listed separately. For
instance, some Soldiers described bombs and explosives while others made more specific
reference to IEDs and rocket-propelled grenades (RPG), etc. While Soldiers in each deployment
category discussed IEDs most often, followed by snipers, some differences appeared between
those who had deployed compared to those who had not deployed. More Soldiers in the
deployment groups referenced IEDs compared to those who had never deployed. Soldiers who
had never deployed and those who had experienced between two and four deployments both
referenced 10 threats while Soldiers who had experienced one deployment refer to 13 types of
threats. This may be because non-deployed experienced Soldiers relied on threats they learned
about in training or might expect to encounter in a particular situation. Soldiers who have
deployed once have a larger list of possible threats. This may simply be due to changing enemy
TTPs or may be because they have started to expand their knowledge of possible threats, but
have not yet narrowed down the list of potential threats to those that are the most likely. Perhaps
the most experienced Soldiers have narrowed down their list of threats because they are better
able to filter and sort threats according to relevance and likelihood and may indicate they have
more developed schemas and a more proficient skill level (Dreyfus & Dreyfus, 1986).
Soldiers could have discussed any possible types of threats during their interview.
Specific enemy and troop tactics such as those discussed in the cemetery scene (Figure 7) are just
one example. Other differences between less and more experienced Soldiers concerned their
discussion of threats posed by the environment. Less experienced Soldiers often mentioned the
terrain. One Soldier stated, “Just the fact that they’re around the mountains; the visibility around
those mountains is limited. The terrain - it’s easy to blend in and get assaulted or ambushed
from any angle, from the top.” Another typical comment indicated the uncertainty of the
situation, as one Soldier stated “High elevation; you’re at the bottom of the creek bed and people
could be above you. You really don’t know what’s up here. You might have scouts going every
way but, right here, you look like you’re by yourself.”

26
Table 10
Number of Soldiers who discussed each common type of threat*
Soldier Experience
Threat Type
Never deployed
(n = 13)
1 Deployment
(n = 14)
2-4 Deployments
(n = 20)
IED
5 (38%)
10 (71%)
10 (50%)
VBIED
3 (23%)
4 (29%)
0 (0%)
PBIED
1 (8%)
3 (21%)
2 (10%)
Bomb/Explosives
4 (31%)
1 (7%)
6 (30%)
Sniper
4 (31%)
6 (43%)
4 (20%)
RPGs
1 (8%)
2 (14%)
2 (10%)
EFPs
0 (0%)
2 (14%)
1 (5%)
Ambush
2 (15%)
1 (7%)
1 (5%)
Small Arms
0 (0%)
2 (14%)
3 (15%)
Grenade
0 (0%)
2 (14%)
0 (0%)
Mortar
0 (0%)
2 (14%)
0 (0%)
Surveillance
2 (15%)
3 (21%)
0 (0%)
Other
1 (8%)
6 (43%)
1 (5%)
*Note. Soldiers could discuss more than one type of threat.
Three Soldiers who had deployed once made comments about the terrain; one stated, “A
curve in the road was the worst. That’s where they seem to hit you the most because they could
see your vehicle turn from a long ways.” This Soldier’s comment shows that he was able to
identify high-priority threat locations based on his previous knowledge (i). Another Soldier
stated that snipers were a threat and “it’s a linear danger; it’s an open space and there’s more
cover outside than where you’re at. It could be a vast amount of threats; there are a million
different scenarios.” The final Soldier said “Maybe it is a high ground so if there’s a road here to
make sure there’s nobody hiding up there - maybe a sniper or something like that.” These
Soldiers’ comments indicated that they were actively trying to identify the threat and create
stories about the threat situations (h) while still operating with a good deal of uncertainty as they
tried to determine what action they would take.
Finally, two Soldiers who had deployed multiple times commented on the terrain. One
Solider made a statement about how open spaces make troops vulnerable to sniper fire. The
other created a story to explain how he thought terrain was the biggest threat. He stated:
In my opinion, terrain plays the biggest role in everything. It pretty much decides
everything, whether there’s a bunch of kids and maybe the particular enemy in that area
doesn’t like them or doesn’t like to take out children. IED spots and sniper spots, any
advantage point in an ambush and things of that sort.
He then explained how he might take advantage of the terrain if he were the enemy. “I’d just
keep to the right side right there and try to take out part of the road; you could also put one in the
mountain right here. That’s not a lot of damage to the truck but, again, I’d pile a bunch of dirt
and rock on the road and now you have vehicles trapped on either side.” This Soldier

27
demonstrated ability to take the enemy’s perspective to reason about possible threats and
consequences of enemy actions, indicating a higher level of causal reasoning.
Threat Cues.
Less experienced Soldiers tended to discuss specific cues and make general comments
about behavior such as “they look kind of suspicious” or “being by himself, doesn’t look right.”
These Soldiers tended to describe suspicious behaviors in general terms, for instance, “I guess
the biggest thing would be any specific individuals who brought attention to themselves, just
through their actions, just wherever I’m observing at the moment.” These Soldiers also
mentioned environmental cues, specifically rock piles, garbage piles, buildings with windows,
tinted windows, lowered vehicles, and wires.
Soldiers with one or more deployments typically listed threat cues, but also provided
more contextual details and interpretations of the cues and situations. Some of the cues
discussed were:
• someone sitting in a car which might indicate they have a detonation device,
• crowds of people,
• people who act suspicious such as looking at you then running off to hide or trying to
stay hidden,
• piles of junk or garbage,
• animals carrying heavy loads,
• people with bulky or odd clothing,
• anything that seems out of place,
• lights that have gone off,
• telephone poles as markers, spotters,
• damaged road medians or discolored curb,
• wires,
• locals avoiding certain areas, and
• police not doing their jobs.
Many Soldiers in this group also mentioned that any changes they were able to detect
might be a threat cue. For example, one Soldier described a threat “If you’ve been down this
route so many times and it’s always been the same and then all of a sudden there’s a car there
just dead.” These comments indicate that these Soldiers are aware that the ability to detect
change or anomalies in the environment (g) is an important skill.
These experienced Soldiers also reasoned about why specific cues may or may not
indicate a threat. For example, one Soldier stated “There is a possibility of an IED somewhere
around here, but it’s not likely because this is a big puddle. It looks like it just recently rained so
they’re probably not going to put their electronic stuff out because it’d get ruined so they can’t
really use IEDs.” A few Soldiers discussed why lights might be a cue such as “Lights…
sometimes people notice the lights go off when there are things going to happen…Sometimes the
lights will be out and sometimes they will just go off too. So it’s all variable.” These Soldiers
were able to incorporate prior information into their decisions (d) and create causal stories about

28
threat situations (h) by discussing their experiences with light cues and the likelihood that they
are cues to threat. This suggests they are able to classify cues as threats or non-threats (b) based
on the context of the situation.
Strategies for Threat Detection.
Many of the less experienced Soldiers made comments about spotting cues as a strategy
for threat detection. They made comments about maintaining alertness, looking for triggermen
or spotters, and other non-specific comments such as “There are just different areas that you’d
check regularly.” One less experienced Soldier constructed a more detailed narrative about a
situation depicted in Figure 8 and stated that “If we came up here we’d have the guy step away
from the wagon and search the people on the wagon. Have people pull security while you were
doing it. Check the cart, check their bags, and you’d go over and check the motorcycle to see if
anything looks out of place. Go into the house; clear the house. I guess just talk to the people
and see if they know any of them.” He continued to explain “We’d walk by a vehicle, look
around at the people and find out what they’re doing. If they’re moving away, something might
be up. Find out if it’s a suspicious vehicle, deploy your security to look inside and make sure it’s
not trapped, open up the doors, take a look inside the glove compartment, the console,
underneath the seats, and dash. You’d want to pop the hood and trunk to look in there. Look for
any wires that shouldn’t be there.” Finally, this Soldier suggested he would ask locals about any
possible threats in the area. His analysis indicates that perhaps he is moving from the novice
skill set to that of advanced beginner or competent stages because he is processing the
information and searching for plausible solutions within the context of the situation (Dreyfus &
Dreyfus, 1986).
Figure 8. Photo that led to discussion with Soldiers concerning threat strategies.

29
Another strategy recommended by the Soldiers who had one or more deployments was
the use of dismounted patrol. One Soldier stated he would “send your dismounts out to find
them, to look for wires, and to look for stuff” while another mentioned that dismounted patrols
allowed for the detection of a command wire. He said “I actually tripped on a command wire
one day and that’s how we found it. Luckily, we did that dismounted, walked around and looked
for command wire.” Many Soldiers commented that they would look for command wire as a
way to detect IEDs. This statement also echoed the realization that often no cues were present
until the IED detonated. Another mentioned “…I can’t say that we looked for (cues to IEDs)
because we never actually found an IED before we detonated it. Either it just went off on us or a
patroller has already found one and had already blocked traffic.”
Soldiers who had deployed once were interested in using the local population as a
strategy for threat detection. As one Soldier put it, “[Threat detection] should be with the local
people; they give away everything. And usually in populated areas there’ll be no threat. I’ve
never had an issue with the population…they’ll come and tell you because they don’t want
[terrorist activity] there just as much as you don’t want it there.” Leveraging the local
populations was a common theme among many of the Soldiers who had deployed. One Soldier
with multiple deployments relied on an interpreter to detect threats. He stated that “if you have a
good interpreter he’ll say ‘he’s speaking a really odd form of Arabic or he doesn’t sound like a
local.’ So a lot of the interpreters are local, not from the city but from the area, so it’d be like a
southerner saying he speaks like a northerner or western…[information like that] just makes sure
you look a little closer.”
With the exception of the one non-deployed Soldier quoted above, most Soldiers who had
not deployed were unable to provide detailed assessments of strategies to detect threats. Only 8
of 13 Soldiers identified strategies. Those who did tended to mention non-specific strategies.
Soldiers who had deployment experience often gave richer narratives of what strategies they
would take. Additionally, Soldiers who had never deployed did not address topics such as the
lack of strategies, using local populations, or dismounted patrol as strategies for threat detection.
Threat Detection Tasks.
Eight non-deployed Soldiers stated that observing situations and environments was a
main threat detection task. They made comments about checking the area, people, or objects.
One Soldier said “Hopefully, I’d be with other people and I’d say ‘keep an eye on them.’ And
get other people looking around the perimeter, a 360.” Another Soldier said “I’d just pay
attention to the people, the vehicles around the buildings, to see in the windows, or even the top
of the buildings because people get up there and stuff. And I’d pay attention to that a lot.”
Another Soldier commented “It’s just that you have to check people; that is what we were told
[in training]. If the people are in your area and they could potentially be a threat then you should
check them.” These less experienced Soldiers largely described the tasks from a passive point of
view and focused primarily on tasks reinforced through training, as the last Soldier indicated.
While less experienced Soldiers focused on scanning and observing the environment, the
Soldiers who had deployed multiple times often took a more active stance in their threat
detection tasks. One Soldier described searching someone riding a motorcycle and said “We’d

30
stop them and maintain standoff, have him park his motorcycle, get off, and lift his arms up into
a circle. Try and determine if he was a threat or not…we’d just tell him stop and then we’d have
him open his hands and make sure he didn’t have any trigger mechanisms or anything like that in
his hands or running through his sleeve. We’d see if the rest of his coat fell down, like when he
stood up see if his coat fell down as opposed to staying up, bulking, you know.” This Soldier
created stories about the threat situations (h) and incorporated prior knowledge to hypothesize
about what might happen and what actions he might take (d). This Soldier mentally simulated a
situation in which the individual posed a threat (h) “We’d continue standoff and then go check it
out…I’d look at it with optics this way, and then I’d make a big circle around over here, and then
I’d look at it with optics this way, and I’d get that dude on his face. No chance he’s going to
blow me up while he’s laying on his face.” This comment also demonstrated ability to take
action to the better classify the threat (b) and take action to reduce the threat (e). He continued to
suggest what might happen if they were attacked and constructed a rich narrative that included
checking for a secondary device, establishing security, using dismounts to search for command
wire, and maintaining visual contact and communication. His ability to construct a rich
description demonstrates many aspects of the threat detection loop including being able to
encode details about the situation (a), detect changes or anomalies (g), balance his decision with
the need to gather additional information (f), and identify the appropriate course of action (e).
Some Soldiers with one deployment focused on tasks that involved understanding the
overall context of the threat. One Soldier described presence patrols as a task where “you’re just
looking around, checking for things that could be bad, meeting people and you’ll be meeting
high-ups, mean low-down people, everything - just making your presence known.” Another
Soldier thought that one of the most important tasks was to interact with locals to understand the
broader situation. He stated “The biggest thing would be to know all of the rumors going into a
situation; that this guy may or may not know that we know about him. Essentially, whether he
has ties to foreign fighters, al Qaeda, Taliban, or whether he’s like one of the most trustworthy,
most loyal American supporters that the country has.” Another Soldier described the difference
in detecting threats between conducting a presence patrol rather than patrol based on a tip. He
said that during presence patrols “You still look for threats, but you’re really going to treat the
Iraqis, you’d talk to them, just like a regular person. You wouldn’t search them, it’s more of a
trusting type thing.” Soldiers who deployed once focused on how the tasks fit into the broader
context of the situation and explained that knowledge of the local allegiances determined how
they went about threat detection. Often, the Soldiers with no deployment experience focused on
the procedures they would engage in to stay safe without taking into consideration the large
goals of community relationships and intelligence gathering.
Threat Detection Skills.
Only two non-deployed Soldiers commented on skills related to threat detection. One
mentioned that he relied on life experience and the threats encountered in everyday life to
determine threats while the other engaged in thinking like the enemy. He suggested that it would
be easy to hide something in a sack of papers that a donkey was carrying. He said “Yes, it seems
like it’d be real easy to do that, anyway, if I was going to hide something it seems like right at
the bottom of one of those sacks would be perfect.” He mentioned that seeing a scene like this
would raise his suspicion level because “it just seems like it provides (the donkey owner) an

31
opportunity to get close without actually having to go through security points or anything like
that.” While the Soldier was thinking of places the enemy might place an IED, he focused on the
ease of emplacement and the chance that the situation might go unnoticed. These statements
indicated that this Soldier is moving to advanced beginner stages of skill development as he
incorporated situational elements into his threat assessment (Dreyfus & Dreyfus, 1986).
Exercises in thinking like the enemy promote the development of these skills and assist Soldiers
to advance beyond novice stages of skill development.
One Soldier who had deployed multiple times took a different perspective when thinking
like the enemy as he discussed the outcome of the threat (Figure 9) rather than focusing on
emplacing an IED. “They have vehicles and why not plant an IED right here; even if it doesn’t
take out the vehicle it’s going to take out the road, cut the lines in half so vehicles can’t get back
and forth. The vehicle will roll down that hill as easy as hell…and even if the whole road
doesn’t go or the vehicle rolls down then, again, you still have a giant gap and this (Soldier) can’t
cross, that one can’t cross.” This Soldier demonstrated causal reasoning about the threat
situation (h) by suggesting that IED placement could result in multiple outcomes. Another
Soldier considered enemy strategy stating he would be unlikely to drive down a road “because of
all this overturned [dirt], [the enemy] could be…trying to funnel you into a certain side of the
road or whatever.” Both Soldiers were trying to anticipate the intent of the enemy when thinking
about what tactics the enemy would employ in each situation rather than solely focusing on how
one might emplace an IED.
Figure 9. Photo that led to discussion with Soldiers concerning thinking like the enemy as a
threat detection skill.

32
Challenges in detecting threats.
Comments made by two of the four less experienced Soldiers indicated they had
incomplete mental models. One Soldier said it would be difficult for him to determine which
cues to notice “It’s hard for me to say at this moment. I mean, in the moment you’d probably be
able to identify stuff here and there, but talking about it from this point of view I don’t know
exactly what I’d be looking for; just something that sticks out from the rest of them.” The other
Soldier thought it would be difficult to notice behavioral cues and stated “People acting
suspiciously; I don’t really know very much about that. I mean I can sort of tell; just life
experience, but if I could get some training on how to read people a little better it’d be helpful.”
Both statements suggest that while they may know what general cues to look for such as some
unusual or suspicious behavior they lack the mental models to articulate what these cues actually
mean.
Many of the Soldiers who deployed once took a different approach to noticing cues.
They often commented that most changes in the environment were not indicators of an attack
and that because enemy tactics changed often it was hard to distinguish relevant from irrelevant
cues. One Soldier stated “[The enemy is] so creative, [their tactics] change every day so you
can’t really worry about it. All you have to do is your job.” Another Soldier stated “Things
change all the time. I mean you could go down a route six times and have cones in different
areas and all at these random checkpoints and stuff.” He went on to create a narrative about how
enemies do their homework including using camera operators and note takers to watch how the
U.S. forces respond to an initial attack so they can set up a secondary attack as the quick reaction
force (QRF) or medevac to arrive on the scene. This suggests that these Soldiers have ideas of
what cues may indicate attacks, but the realities of the situation require them to look beyond
specific cues that may not be consistently relevant (b). To do this, they need to encode details
about the situation (a), engage in causal reasoning (h), and balance decisions with information
gathering (f).
Soldiers with multiple deployments also said it was difficult to notice cues because the
enemy changed tactics often as a way to reduce their predictability. One Soldier said “I really
don’t like looking for IEDs…too many places to put it, to hide it, hard to find, and not enough
manpower.” Another Soldier stated that “when you’re on patrol you’re concerned about
everything…and, of course you can’t look at everything, you’re going to miss something…”
This same Soldier went on to say “A lot of the younger guys who have never been [deployed],
and some of them have been to classes, then, yes, the only thing they’re going to look for is
stacked up rocks and overturned dirt. Yes, one spot is overturned, they’re going to look at the
small stuff, they’re not going to look at the overall [picture]…that’s what I’m seeing when I look
at this. I’m seeing that anything can be a threat.” These Soldiers understand the role experience
plays in identifying threat and non-threat features (b) as well as appropriately encoding details
about the situation (a). These more experienced Soldiers are able to focus their attention on
relevant cues and reason about what cues are not important or relevant in the moment. They are
better able to organize and prioritize information based on their experience which research
indicates is a characteristic of expertise development (Ericsson & Charness, 2004).

33
Across both groups of Soldiers who had deployed, the reliability of information posed a
specific challenge. A Soldier who deployed multiple times gave a mixed response when asked
whether locals were a good source of intelligence. “Sometimes they were, sometimes they
weren’t. I guess it really depends. I mean a lot of them just looked at you like you’d know they
knew…if they were looking at you like they wanted to stay out of it, they’re not going to tell you
anything. It’s not necessarily that they don’t want to tell us, it’s that they’re scared to tell us.”
Another Soldier stated “You’d go through sectors where we’d just show up and talk to the sheik
and he’d be like ‘There aren’t any IEDs,’ like there really aren’t, or he’d tell you three or four
spots, kind of rat the Taliban out. And then you’d go to other sectors and it’s not as reliable
because that guy is less supportive of American and the Taliban forces are stronger in his area, so
he can’t support more even if he wants to.”
Solutions.
Less experienced Soldiers did not provide many solutions for improving threat detection
performance. Little difference existed between Soldiers who experienced one deployment and
those who had been on multiple deployments. Across both groups, Soldiers stated that constant
alertness and attention to detail were critical to effective threat detection. One Soldier suggested
that foot patrols are an easy way to reduce complacency while increasing mobility and avoiding
attacks. Two Soldiers made comments about the importance of making connections to local
populations. The first Soldier stated “If you’re working well with the locals, you’re being
generous to them, and you’re working hand-in-hand with them, you’ll have no problems in that
area, there’ll be no issues in that area.” The second Soldier stated “Let’s say this is part of a
sector that you’re responsible for, I’d probably know these guys standing here and I would have
a reasonable level of trust, you know what I expect whether it is something that would be a big
threat to me or not.” That Soldier discussed how they might counter an enemy threat “As a
mounted force, if I had other sectors of the city secured, at least the parallel street, to put forces
at the viewpoint of the camera, then kind of flank around to the rear point of the field of focus as
well. That would be ideal; we’re basically taking away their escape route.” Again, these
Soldiers are considering strategies to reduce threat levels rather than simply spotting cues.
Summary of Research
This research examined how less experienced and more experienced Soldiers performed
experimental exercises related to threat detection and responded to situations that contained
potential threats. Some of the key findings suggest that time, threat relevance, and expertise
played a role in Soldiers’ performance. Data in the prioritized threat search exercise
demonstrated that novices were more susceptible to time pressure and the number of
deployments increased accuracy slightly under time pressure. Results in dynamic threat
detection revealed that Soldiers were able to infer threat-relevant locations and appropriately
focus their attention to those areas. Additionally, all Soldiers had a tendency to identify targets
in threat-relevant locations. Finally, the change detection exercise indicated that Soldiers were
able to identify changes related to threats.
An analysis of the interviews revealed differences in the number of times less
experienced and more experienced Soldiers discussed strategies for threat detection, threat

34
detection skills, challenges in detecting threats, and solutions. Further analysis suggested that
there were differences within all of these broad categories, ranging from topics of discussion to
the content of these discussions. Novices typically responded in a rule-based fashion, relying on
things like context-free cues or threat, whereas experts constructed rich narratives and scenarios
that allowed them to describe their assessments of situations in ways that were more complex.
Taken together, these findings suggest ways that less experienced Soldiers differ from
experienced Soldiers. We can use these differences to train less experienced Soldiers to improve
their ability to identify relevant threats, detect relevant changes, and develop causal reasoning
skills applicable to the operational environment.
One limitation of the current research was a lack of clear expert and novice groups.
Soldiers ranged from zero to four deployments. Soldiers fell into groups of relatively even
numbers across deployments when categorized as zero deployments, one deployment, and 2-4
deployments. Using deployments as a proxy measure of experience was logical for this research
and other criteria of expertise, such as years in the military and rank correlated highly with
number of deployments. Conducting research that had two groups with clearly delineated levels
of experience would allow for a clearer analysis of experience differences. The lack of distinct
experience levels may have contributed to a lack of statistical significance in some of the
analyses.
Exemplar Training Development
Design.
Results of the computer exercises demonstrated the utility of these exercises as potential
training methods. In the prioritized threat search exercise, time pressure negatively affected the
performance of Soldiers with zero deployments, but had little or even a positive effect on
Soldiers with one or multiple deployments. Targets in threat-relevant locations were located
faster and with greater accuracy. However, no statistically significant differences were found
based on experience. This finding indicated that Soldiers, even in threat search exercises using
static photos, directed their search to relevant threats rather than to random locations on the
photo making this a useful stimulus for training. Similarly, in the change detection exercise
participants noticed threat-relevant changes rather than threat-irrelevant changes.
The interview analysis provided insight into the reasoning of less and more experienced
Soldiers. As expected, Soldiers with less experience tended to make context-free procedurally-
based statements whereas experts constructed rich narratives to describe their assessments of
situations. This analysis revealed differences in the number of times less experienced and more
experienced Soldiers discussed strategies for threat detection, threat detection skills, challenges
in detecting threats, and solutions. These narratives indicated that less experienced Soldiers did
not have the mental models to draw on to interpret the situations in the photos even if they could
identify relevant threats. Based on this information, the training incorporated reasoning
exercises that prompted Soldiers to deliberate over threat cues and use information about the
situation and environment to reason about likelihood, threat relevance, and severity. This
training focused on helping Soldiers become more able to detect and interpret threats in the

35
context of situations, thus allowing them to broaden their experience base and develop applicable
mental models.
From the research data, four learning objectives were identified. These learning
objectives map onto the expected behaviors of experienced threat detectors identified as part of
the threat detection loop (Figure 1). For instance, the initial exercises focus attention on
encoding details to identify and classify relevant threats. The reasoning exercises increase ability
gather evidence to determine threat importance and impact. The change detection exercises were
developed to increase the perception of changes and improve the ability to determine the
relevance of the change (Table 11).
Table 11
Relationship between learning objectives and the threat detection loop
Learning Objectives
Step in Threat Detection
Loop
Threat Detection Loop
Behaviors
1. Demonstrate an improved
ability to identify relevant
threats in a variety of
situations.
• Monitoring, vigilance,
and search activities
a) Encoding details for
immediate or delayed test
2. Distinguish between threat-
relevant and threat-
irrelevant cues in time-
limited situations.
• Identify/classify threat
or non-threat
b) Ability to classify threat vs.
non-threat features
i) Ability to look at high-
priority threat locations based
on knowledge
3. Recognize the importance
and potential impact of
each detected threat.
• Identify/classify threat
or non-threat
• Gather evidence to ID
threat
c) Ability to adapt decision
selectivity based on
information
f) Ability to balance decision
with information gathering
h) Ability to create causal
stories about threat situations
4. Identify threat-relevant
changes in the
environment.
• Change or anomaly
detection
g) Ability to detect change or
anomaly
To determine which photos to use in the training, 10 instructors and Soldiers from Fort
Hood reviewed the 44 photos and selected the 10 most and 10 least preferred choices to
incorporate into training. The instructors described the most probable threats depicted in 5 of
their top 10 photos along with reasons why those threats would be the most difficult to detect.
Soldiers in the experiment chose some of the same photos as the instructors. Of the photos
selected by Soldiers in the experiment, the instructors also rated many in the top 10. Focus was
given to photos that more than one Soldier commented on because they provided threat cue
locations, annotation, and interview data that would make up the training content. The photos

36
were then reviewed for training applicability, photo quality, and the ability of the photo to
provide training content that met the learning objectives. The interview data was matched to the
final set of 11 photos and then photos were selected that would apply to the threat search,
reasoning, scenario, and change detection exercises (Appendix C).8
To guide the development process, storyboards provided the content and photos that
appear on each screen of the computer-based training (Appendix D). The purpose of the
storyboards was to map out the entire course. Military subject matter experts reviewed the
content and created the questions for the Soldiers to answer and feedback they would receive.
Soldiers also were given feedback about their threat search activities, response time, and
identification of threat relevant cues.
The exercises in the training take a crawl-walk-run approach. The first series of exercises
involve threat searches. Soldiers first view photos with threat cues already marked and click on
these cues to read about what the threat is, why the threat is relevant, and common enemy tactics.
Soldiers then view photos and click on potential threat cues. If the places they click are threats,
annotations pop up, again providing information about those cues. They then engage in a timed
threat search exercise (Figure 10).
Figure 10. Screenshot of Threat Search Exercise.
8 Six instructors selected three of these photos in the top 10 (each of these photos was selected once in the bottom
10) and four selected one photo in the top 10 (none selected it in the bottom 10). One or two instructors chose five
of the photos in the top 10 (with one or two selecting them in the bottom 10). Two photos did not appear in the top
or bottom 10.

37
Following the threat search, Soldiers view photos with questions that require them to
reason about the situations presented and evaluate the threats in context. They engage in critical
thinking about the potential threats and read several possible solutions (Figure 11). They also
read comments made by experienced Soldiers who discussed the photos during the interviews.
Figure 11. Screenshot of Reasoning Exercise.
Finally, Soldiers complete the threat search and reasoning exercises by reading scenarios
that frame the context of the photos and then consider that context to identify threats and
determine relevance (Figure 12). Soldiers also complete two change detection exercises during
the training, one in the context of a threat search and one in the context of causal reasoning.
Development.
The Instructional Systems Design (ISD) process informed the development of the
computer-based training (CBT) materials. The data and storyboards contributed to the
development of the look, content, and functionality of the training product. A Graphical User
Interface (GUI) was created, edited, validated, and verified and then a Sharable Content Object
Reference Model (SCORM) compliant version of the CBT course was developed. When
developing a SCORM compliant CBT course, the major sections of each module, such as
introductions, lessons, and summaries are wrapped into their own Sharable Content Object
(SCO). The SCORM compliancy allows training delivery to a variety of Learning Management
Systems (LMS) that the military currently uses.

38
Figure 12. Screenshot of Threat Relevance Exercise.
While in development, subject matter experts (SME) could view the CBT course on an
internal server to provide comments, request changes, and check the course for functionality.
The evaluation took place throughout the ISD process first with reviews of the storyboards, then
through initial reviews of the introduction slides, and after that, a formative evaluation process.
After the evaluations and final approval of the course, the CBT was placed on an external server
or the client-provided LMS and CD/DVDs were created for distribution.
Formative evaluation.
Three instructors at Fort Hood, TX, completed two evaluations of the training while it
was under development. They reviewed an early draft of the trainer that included the
introduction and the pre-test exercise. Their early assessment included suggestions for
improving the narration, providing feedback at the end of the exercise, and collecting
demographic information. These reviewers also provided feedback about the CBT format and
design. These reviewers conducted a second evaluation of a more advanced version of the
training that included all the content and exercises, but was missing some functionality. The
reviewers answered questions specific to the training product and addressed issues of training
implementation. The following questions were included.
• In terms of being able to interact with the product, are there certain navigation features
that need improvement (because they weren’t easy to use/navigate)?
• Do the scenarios and photos present realistic depictions of frequently encountered
situations in theater? If not, do you have recommendations for scenarios and photos to
include?

39
• Who would be the best audience for this trainer: new Soldiers prior to their first
deployment or as a refresher for Soldiers who are deploying after one or more previous
deployments? Why did you choose that group?
Overall, the responses from the three evaluators were very positive. Some feedback
addressed functioning issues within the trainer that were addressed in the final version. The
reviewers appreciated the sense of urgency that time restrictions within the trainer presented
stating that although it is not the same as being in the actual situation, the restriction modeled
time stress to some extent. Other responses addressed the question of who should use the trainer.
Suggestions included using the trainer for new Soldiers so that they can learn important skills
before deploying, but also using the trainer for Soldiers who need to refresh and sharpen skills
before redeployment. Finally, the reviewers suggested that the trainer in combination with an
instructor would provide students with additional insight to enhancing threat detection skills.
Suggestions for Methods Refinements and Extensions
Our current methodologies used static imagery to both identify and convey threat signals
to Soldiers in the experiments and training and used overt annotation and search behaviors to
collect information regarding threats. This had some practical advantages in that Soldiers could
study imagery closely without time pressure and they could make annotations in detail.
However, new technologies may augment and improve both data collection and training products
in three important ways: 1) use of dynamic video imagery for improved data collection and
training, 2) use of eye-tracking hardware and other means to identify directly the locus of
attention during threat detection behavior, and 3) use of immersive environments to improve
training realism.
Dynamic Video Imagery. The use of static imagery allowed us to use high-resolution
stimuli which contained considerable detail that is often lacking from the video footage of threat
situations. As a result, our methods identified threat cues that were primarily static such as
features of the terrain or persons in the scene that might be a threat. However, our interviews
identified a number of threat cues that are dynamic rather than static. These included subtle
interpersonal cues such as where a person is looking or how he or she is walking, ways that a
vehicle bounces or “drives suspiciously,” and so on. Soldiers can only identify these threat cues
in dynamic video imagery and research might require fundamentally different elicitation
methods to uncover this type of information. To the extent that it is possible to obtain
appropriate video and design proper data collection experiments, training can be further
augmented so that it can cover both static and dynamic threat cues.
Attention-tracking and Eye-tracking Hardware. Several of the research exercises used
behavioral cues (mouse clicks or identification response times) to help identify where Soldiers
were directing their attention. Software or hardware systems could augment these exercises to
allow closer monitoring of visual attention. For example, simply recording the location of the
mouse cursor during a search can give an accurate picture of where a Soldier looks, but an even
more promising method would be to monitor eye movements to identify exact locations of search
activities. Eye-tracking technologies hold particular promise if combined with video imagery
because research can begin to understand more detailed visual search and attention strategies

40
during threat detection. This method would be particularly useful to identify expert threat search
strategies in complex visual environments making this a useful technology for research and for
training.
For example, there is some evidence that eye-tracking technology encourages non-
experts to adopt visual search strategies that are more similar to those of experts. Research by
Sadasivan, Nalanagula, Greenstein, Gramopadhye, and Duchowski (2004) used an overlay of an
expert’s eye movements on an aircraft that required inspection. Participants who received this
type of training demonstrated greater improvement in their visual search performance compared
to those who did not receive training. This feed-forward training allowed participants to rely on
the expert’s search strategy early in the visual search process, thereby adapting to a more
effective search technique. Further research by Nalanagula, Greenstein, and Gramopadhye
(2006) used three types of displays involving eye-tracking to improve performance when
detecting defects in circuit boards. These three displays included a static display representing the
expert’s eye path and general areas of interest, a dynamic display that highlighted the expert’s
areas of interest in the order performed during visual search, and a hybrid display combining
both static and dynamic features. Both the dynamic and hybrid display improved the number of
defects participants were able to identify. Eye-tracking technologies may be used to both
identify search differences between novice and expert threat detectors and to train visual search
strategies and improve skills. With the advent of reasonably cheap eye-tracking hardware,
training facilities can use this technology as a method to improve training and measure skill
level, and researchers can leverage these technologies to understand threat detection strategies.
Immersive Training Environments. A third limitation of the current methodology and
training is that it focused on perceived threats rather than actual threats. The imagery used in the
research stimuli and training did not contain actual attacks or IEDs. While the purpose of the
exercises was not to detect actual threats, but to improve the cognitive skills associated with
threat detection, it may also be valuable to create immersive environments that simulate
sequences of events that result in finding and disrupting threats and dealing with the
consequences of undetected threats. Training threat detection in field environments provides
opportunity for Soldiers to move through situations and scan 360 degrees from a first person
point of view. The current prototype offers the crawl and walk stage of threat detection training
while immersive training environments offers the run stage of learning.
At least three types of immersive environments could emerge from this research: live
training, video exercises, and first-person virtual training. Video-based and live training could
be similar in the scenarios they present and the skills they exercise. Video-based training
provides opportunity to present select scenarios that students can review multiple times and in
conjunction with presentation of knowledge-based training and knowledge checks (i.e., test
questions, and critical thinking exercises). Live training allows students to engage in actual
threat detection exercises both cognitively and physically while providing a safe environment to
refine skills and learn from mistakes. The data collected thus far can contribute to the creation of
training scenarios and feedback guides for use in existing live training such as current Lanes
Training Exercises (LTX), Field Training Exercises (FTX), or similar skill training events.
Video-based training can leverage live scenario materials as a basis for selecting dynamic video
imagery that would drive off-line training via computer or classroom settings. Finally,

41
developing virtual environments would allow for dynamic generation of threats, allowing
situations to replay so students can identify threat cues they missed and engage in after action
reviews etc. Virtual environments also allow for automated feedback and dynamic generation of
content.
Conclusion
The series of research phases conducted to understand Soldier threat detection provides a
better understanding of how Soldiers manage attention, process information, and reason about
events they encounter in the OE. The first three phases of this research provided insight into the
key threats Soldiers consider important and clarified the difficulty Soldiers have in distinguishing
ambiguous cues from certain cues when searching for threats. Data from interviews with
experienced Soldiers enabled the creation of a threat detection loop model and identification of
the behaviors necessary for proficient threat detection. The outcome of the computerized
exercises was stimuli for the final research that included key threat locations and cues in photos
and critical situational factors that influence threat relevance. The aim of the final research phase
was to compare the threat detection performance of less and more experienced Soldiers. This
research demonstrated the cognitive skills Soldiers employ when searching for threat-relevant
cues and changes in situations with and without time pressure. Interviews also provided data that
mapped onto the threat detection loop and contextual details that informed our knowledge of the
threat detection process.
While all of this research provided contextual detail about how Soldiers think about
threat cues, threat detection activities, attention management, and search priorities, the final
research phase contributed empirical evidence of Soldier threat detection abilities. This research
showed that time pressure influences Soldier ability to detect threats with fewer threats being
detected when under time pressure. These results provided some indication that experience
reduces the effects of time pressure, but this effect was small and thus more research is required.
Regardless of experience, Soldiers focused their attention toward detecting relevant threats
compared to irrelevant threats. They were more accurate and responded faster to relevant threats
indicating that targeted searching in relevant locations is a component of threat search exercises.
While experience did not mediate this effect, the interview data indicated that Soldiers with
deployment experience had a deeper understanding of why threats were relevant and under what
circumstances threat relevance would increase or decrease. They were able to construct stories
around the threats, reason about various action choices, and hypothesize about potential
outcomes.
The change detection exercise was one of the more informative concerning Soldier ability
to determine threat relevance. Although results revealed a high false alarm rate when trying to
find changes in the photos presented, when Soldiers accurately identified changes the majority
were threat-relevant changes. This finding adds to the existing change detection literature by
demonstrating Soldier ability to notice changes at rates greater than chance. Previous research
would suggest that the Soldiers should have a difficult time detecting changes. Change blindness
is a phenomenon where individuals are incapable of detecting salient changes in their visual
environment (O’Regan, 1992; Rensink, O’Regan, & Clark, 1997; Simons & Rensink, 2005).
Previous research has demonstrated that observers fail to notice large changes made during eye
movements, across a flicker paradigm, during repeated viewings of the visual scene, and when

42
conversational partners changed during a real-life interaction. Our research demonstrates that at
an applied level Soldiers demonstrate some ability to detect change above chance levels.
Researchers have identified some situations that mitigate change blindness.
Hollingsworth (2004) and Brady, Konkle, Oliva, and Alvarez (2009) suggest that intentional
encoding enables individuals to attend to changes. However, in the current research, Soldiers
received no explicit instructions to suggest they should study the photograph for subsequent
testing. The test to detect changes was a surprise for these participants and their performance
suggests an enhanced ability to detect changes. Further, it should have been difficult for Soldiers
to detect changes because the subsequent testing after the target picture provided significant
interference. Makovski, Shim, and Jiang (2006) tested change detection ability in natural scenes
after presenting blank, visual, or auditory delays and requiring participants either to view
passively the delay screens or attend to the information in the delay. Participants who attended
to the delay information had a significant reduction in their change detection ability. Because
information presented between the target photograph and the changed photograph required
Soldiers to attend to a significant amount of information, it was expected that they would have
difficulty detecting changes during the second presentation of the photos. Further research
should attempt to provide an explanation for this finding.
Experts have been shown to detect changes more often than novices do when the change
is related to their domain of expertise (Jones, Jones, Smith, & Copley 2003; Werner & Thies,
2000). In the current research, we found no differences in change detection performance based
on experience. The data did not provide insights into why no differences existed. It could be
that general training related to detecting threats made Soldiers aware of the potential for change
in their environments and thus they were attuned to looking for changes. It could also be that
threat-relevant cues are emotional stimuli for Soldiers and those cues captured their attention
(Mayer, Muris, Vogel, Nojoredjo, & Merchelbach, 2006; Peira, Golkar, Larsson, & Wiens, 2010;
Vuilleumier, 2005) thus making the presence or absence of such cues salient. It may also be that
the distinction between the experienced and novice groups was not clear in the current research.
Future research should attempt to identify the cause of these findings.
Results of this research provide an increased understanding of threat detection and
directions for future research and training exemplar development. The current training uses
static images and exercises developed during research that also used static images as stimuli to
inform our knowledge of Soldier threat detection. Future research and training should explore
the use of video images and eye tracking to gather data to infer perception and attention
processes and to train threat detection in complex and dynamic environments. Soldiers of
various skill levels tend to focus on relevant threats; however, experience adds context and
reasoning to the threat identification process. Future training should continue to highlight the
causal reasoning and critical thinking components of threat detection. This research indicated
that Soldiers have ability to notice changes, but this ability was limited relative to the high false
alarm rate. Although change detection accuracy was not random, it may be possible to improve
this existing skill with targeted training.

43
References
Beilock, S. L., & Carr, T. H. (2004). From novice to expert performance: Memory, attention
and the control of complex sensori-motor skills. In A. M. Williams, N. J. Hodges, M. A.
Scott, & M. L. J. Court (Eds.), Skill acquisition in sport: Research, theory and practice
(pp. 309-327). New York, NY: Routledge.
Beilock, S. L., Wierenga, S. A., & Carr, T. H. (2002). Expertise, attention, and memory in
sensorimotor skill execution: Impact of novel task constraints on dual-task performance
and episodic memory. The Quarterly Journal of Experimental Psychology: Human
Experimental Psychology, 55, 1211-1240
Boyd, J. (1987). A discourse on winning and losing. Maxwell Air Force Base, AL: Air
University Library Document No. M-U 43947.
Brady, T. F., Konkle, T., Oliva, A., & Alvarez, G. A. (2009). Detecting changes in real-world
objects: The relationship between visual long-term memory and change blindness.
Communicative & Integrative Biology, 2, 1-3.
Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and
expertise in the era of the computer. New York, NY: The Free Press.
Ericsson, K. A., & Charness, N. (1994). Expert performance. American Psychologist, 49, 725–
747.
Goodrich, M. A., Sterling, W. C., & Boer, E. R. (2000). Satisficing revisited. Minds and
Machines, 10, 79–110.
Hollingsworth, A. (2004). Constructing visual representations of natural sciences: The role of
short and long-term visual memory. Journal of Experimental Psychology: Human
Perception and Performance, 30, 519-537.
Jones, B. T., Jones, B. C., Smith, H., & Copley, N. (2003). A flicker paradigm for inducing
change blindness reveals alcohol and cannabis information processing biases in social
users. Addiction, 98, 235–244.
Klein, G. (1998). Sources of Power. Cambridge, MA: MIT Press.
Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid decision making on the fire
ground. Proceedings of the Human Factors Society, 30th Annual Meeting (Vol. 1, pp.
576-580). Dayton, OH: Human Factors Society.
Klein, G., Phillips, J. K., Battaglia, D. A., Wiggins, S. L., & Ross, K. G. (2002). Focus: A
model of sensemaking (Interim Report – Year 1. Prepared under Contract 1435-01-01-
CT-31161 for the U.S. Army Research Institute for the Behavioral and Social Sciences,
Alexandria, VA). Fairborn, OH: Klein Associates.

44
Levin, D. T., & Simons, D. J. (1997). Failure to detect changes to attended objects in motion
pictures. Psychonomic Bulletin & Review, 4, 501-506.
Makovski, T., Shim, W. M., & Jiang, Y. V. (2006). Interference from filled delays on visual
change detection. Journal of Vision, 6, 1459-1470.
Mayer, B., Muris, P., Vogel, L., Nojoredjo, I., & Merckelbach, H. (2006). Fear-relevant change
detection in spider-fearful and non-fearful participants. Journal of Anxiety Disorders, 20,
510–519.
Mueller, S. T. (2009). A Bayesian recognitional decision model. Journal of Cognitive
Engineering and Decision Making, 3, 111-130.
Mueller, S. T., & Weidemann, C. T. (2008). Decision noise: An explanation for observed
violations of signal detection theory. Psychonomic Bulletin and Review, 15, 465-494.
Nalanagula, D., Greenstein, J. S., & Gramopadhye, A. K. (2006). Evaluation of the effect of
feedforward training displays of search strategy on visual search performance.
International Journal of Industrial Ergonomics, 36, 289-300.
O’Regan, J. K. (1992). Solving the “real” mysteries of visual perception: The world as an
outside memory. Canadian Journal of Psychology, 46, 461-488.
Peira, N., Golkar, A., Larsson, M., & Wiens, S. (2010). What you fear will appear: Detection of
schematic spiders in spider fear. Experimental Psychology, 57, 470-475.
Phillips, J. K., Klein, G., & Sieck, W. R. (2004). Expertise in judgment and decision making: A
case for training intuitive decision skills. In D. J. Koehler & N. Harvey (Eds.), Blackwell
handbook of judgment and decision making (pp. 297-315). Victoria, Australia:
Blackwell Publishing.
R Development Core Team (2009). R: A language and environment for statistical computing. R
Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL
http://www.R-project.org
Rensink, R. A. (2002). Change detection. Annual Review of Psychology, 53, 245-277.
Rensink, R. A., O’Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for
attention to perceive changes in scenes. Psychological Science, 8, 368-373.
Ross, K. G., Phillips, J. K., Klein, G., & Cohn (2005). Creating expertise: A framework to
guide technology-based training (Technical Report Contract M67854-04-C-8035, Office
of Naval Research).

45
Sadasivan, S., Nalanagula, D., Greenstein, J., Gramopadhye, A., & Duchowski, A. (2004).
Training novice inspectors to adopt an expert’s search strategy. Proceedings of IIE
Annual Conference, Houston, TX.
Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information
processing: II. Perceptual learning, automatic attending, and a general theory.
Psychological Review, 84, 127-190.
Simons, D. J., & Rensink, R. A. (2005). Change blindness: Past, present and future. Trends in
Cognitive Science, 9, 16-20.
Vuilleumier, P. (2005). How brains beware: Neural mechanisms of emotional attention. Trends
in Cognitive Science, 9, 585-594.
Werner, S., & Thies, B. (2000). Is “change blindness” attenuated by domain-specific expertise?
An expert-novice comparison of change detection in football images. Visual Cognition,
2000, 7, 163–173.
Zimmerman, L. A. (2008). Making sense of human behavior: Explaining how police officers
assess danger during traffic stops. In J. M. Schraagen, L. Militello, T. Ormerod, R.
Lipshitz (Eds.), Naturalistic Decision Making and Macrocognition (pp. 121-140).
Aldershot, U.K.: Ashgate Publishing.
Zimmerman, L. A., Mueller, S. T., Grover, J., & Vowels, C. L. (In Preparation). Determining
the Requisite Components of Visual Threat Detection to Improve Operational
Performance (ARI Technical Report). Fort Belvoir, VA: U.S. Army Research Institute
for the Behavioral and Social Sciences.

A-1
Appendix A
Demographic Questions
Fort Hood - 13-15 December 2010
1. Time in service (years)
2. Current rank
3. Time in current rank (months)
4. Current MOS
5. Age
6. Have you ever deployed? Yes No
7. How many times have you deployed?
8. Location of most recent deployment (city or cities and country)
9. MOS while on most recent deployment
10. How often did you go outside the wire on your most recent deployment?
Never Less than once a month Once a month More than once a month
Once a week More than once a week Everyday
11. Describe some of your duties during your most recent deployment:

B-1
Appendix B
Overlay indicating locations Soldiers clicked on to identify potential threats

C-1
Appendix C
Photos selected for the threat detection exercises
Threat
Search
Reasoning
Scenarios
Pre
/Post
Test

C-2
Change
Detection

D-1
Appendix D
Storyboard example used in training exemplar development
Course: Threat Detection Skills Trainer
Module: Threat Detection TD
TD2-1-2
Lesson: Identifying Threats 2
Segment: 1
Page Title: Identifying Threat Exercise #1 9
Child Page:
Objective:
1) Identify relevant threats in a variety of military patrol contexts
On-Screen Text:
Now it’s your turn. Click on locations that could conceal a threat. When you find relevant threats, a red
dot will appear with information about that threat.
When you are done searching, click the “Finished” button to reveal all the threats. You must click on at
least three threat locations prior to finishing. You have 10 clicks maximum, so evaluate the situation
carefully before choosing
Narration/Closed Captioning: Narrator
Can you find the relevant threats in this photo?
Graphics: (P – photo; G – graphic; F – flash animation; T – table/chart/graph; V – video)
Use photo: Clean 34 from treat-hires folder.
Use information from Threat Photos-Annotations: Slide 06 for threat areas descriptions.
(Place the red dot in the middle of each threat area.)
Audio:
Knowledge Check:
Remedial Screen:
Page ID
Correct Feedback:
Provide the number of clicks the student made and the number of threats found
1st try incorrect:
2nd try incorrect:
Explanatory Information:
When Soldiers click on threat spots, show a red dot in that spot. The spot should remain visible.
Make each red dot
After student finishes, show all red dots and pop-ups. Do not allow them to click finish until they have
made at least three clicks
a pop-up of threat’s description, with ability to click and close
Branching: Back: Next: