Content uploaded by Christopher L. Vowels
Author content
All content in this area was uploaded by Christopher L. Vowels on Jan 07, 2014
Content may be subject to copyright.
Technical Report 1329
Improving Visual Threat Detection: Research
to Validate the Threat Detection Skills Trainer
Laura A. Zimmerman, Drew A. Leins,
Jessica Marcon, Brandon Pearlman
Applied Research Associates
Jacob T. Singer
Texas A&M University-Central Texas
Ron Mueller
Dynamics Research Corporation
Christopher L. Vowels
U.S. Army Research Institute
August 2013
United States Army Research Institute
for the Behavioral and Social Sciences
Approved for public release; distribution is unlimited.
U.S. Army Research Institute
for the Behavioral and Social Sciences
Department of the Army
Deputy Chief of Staff, G1
Authorized and approved for distribution:
MICHELLE SAMS, Ph.D.
Director
Research accomplished under contract
for the Department of the Army by
Applied Research Associates
Technical review by
Steven F. Burnett, U.S. Army Research Institute
Thomas R. Graves, U.S. Army Research Institute
NOTICES
DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.
Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, ATTN: DAPE-ARI-ZXM,
6000 6th Street (Bldg. 1464 / Mail Stop 5610), Ft. Belvoir, VA 22060-5610
FINAL DISPOSITION: Destroy this Technical Report when it is no longer needed. Do
not return it to the U.S. Army Research Institute for the Behavioral and Social Sciences.
NOTE: The findings in this Technical Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized
documentation.
i
REPORT DOCUMENTATION PAGE
1. REPORT DATE (DD-MM-YYYY)
August 2013
2. REPORT TYPE Final
3. DATES COVERED (From - To)
April 2012 – August 2012
4. TITLE AND SUBTITLE
5a. CONTRACT NUMBER
W59JCQ-11-D-0003-0002
Improving Visual Threat Detection:
Research to Validate the Threat Detection Skills Trainer
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
622785
6. AUTHOR(S)
Laura A. Zimmerman, Drew A. Leins, Jessica Marcon, Brandon Pearlman;
Jacob T. Singer;
Ron Mueller;
Christopher L. Vowels
5d. PROJECT NUMBER
A790
5e. TASK NUMBER
370
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
Applied Research Associates Texas A&M - Central Texas Dynamics Research Corp.
1750 Commerce Blvd. North 1001 Leadership Place 60 Frontage Road
Fairborn, OH 45324-6362 Killeen, TX 76549 Andover, MA 01810
8. PERFORMING ORGANIZATION REPORT
NUMBER
9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES)
U.S. Army Research Institute
for the Behavioral and Social Sciences
ATTN: DAPE-ARI
6000 6th Street (Bldg. 1464 / Mail Stop 5610)
Fort Belvoir, VA 22060-5610
10. SPONSOR/MONITOR’S ACRONYM(S)
ARI
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
Technical Report 1329
12. DISTRIBUTION / AVAILABILITY STATEMEN; Distribution Statement A:
Approved for public release; distribution is unlimited.
13. SUPPLEMENTARY NOTES
COR and Subject Matter POC: Christopher L. Vowels
14. ABSTRACT
The purpose of this research was to validate a threat detection skills trainer developed in previous research (Zimmerman,
Mueller, Daniels, & Vowels, 2012; Zimmerman, Mueller, Grover, & Vowels, in preparation). The trainer consists of exercises
intended to improve visual threat detection, including dynamic threat monitoring, threat prioritization, and causal reasoning
(Zimmerman et al., 2012). The current research utilized four tasks to measure the impact of the skills trainer: time and
resource limited threat search; dynamic change detection; situation awareness at the perception and comprehension levels;
and, scenario-based causal reasoning. Findings did not provide evidence that the trainer alone is a viable option for training
visual threat detection, likely due to low statistical power and practice effects. However, all participants showed
improvement on all dependent variables from pre-test to post-test, suggesting that acute, cognitive training could enhance
the skills required for effective threat detection. Future research should tease apart the effects of practice and the effects of
training via the threat detection skills trainer. Additional future experiments should test threat detection in live and simulated
scenarios to increase ecological validity. Longitudinal research would determine if practice and training on visual threat
detection have long-term effects.
15. SUBJECT TERMS
Threat detection, Training, Infantry, Irregular warfare, Asymmetric warfare, Attention
16. SECURITY CLASSIFICATION OF:
17. LIMITATION
OF ABSTRACT
18. NUMBER
OF PAGES
19a. NAME OF RESPONSIBLE PERSON
Dorothy Young
a. REPORT
Unclassified
b. ABSTRACT
Unclassified
c. THIS PAGE
Unclassified
Unlimited
55
19b. TELEPHONE NUMBER (include area
code) 703-545-2316
ii
iii
Technical Report 1329
Improving Visual Threat Detection: Research
to Validate the Threat Detection Skills Trainer
Laura A. Zimmerman, Drew A. Leins,
Jessica Marcon, Brandon Pearlman
Applied Research Associates
Jacob T. Singer
Texas A&M University – Central Texas
Ron Mueller
Dynamics Research Corporation
Christopher L. Vowels
U.S. Army Research Institute
Fort Hood Research Unit
Scott Shadrick, Chief
U.S. Army Research Institute for the Behavioral and Social Sciences
6000 6th Street, Bldg. 1464
Fort Belvoir, Virginia 22060
August 2013
Approved for public release; distribution is unlimited
iv
ACKNOWLEDGEMENT ______
We would like to thank the Soldiers and leaders who participated in this research,
especially for the time and effort they provided.
v
IMPROVING VISUAL THREAT DETECTION: RESEARCH TO VALIDATE THE THREAT
DETECTION SKILLS TRAINER
EXECUTIVE SUMMARY ___________
Research Requirement:
Visual threat detection remains an essential skill for virtually every Soldier in the
operational environment. The purpose of this research was to validate the Threat Detection
Skills Trainer (Zimmerman, Mueller, Daniels, & Vowels, 2012; Zimmerman, Mueller, Grover,
& Vowels, in preparation), developed in previous ARI efforts. This trainer incorporated the
findings from empirical research on visual threat detection, with focus on threat search and
identification, attention management, and reasoning about threats. Providing training that
expands the perceptual and cognitive skills associated with visual threat detection enhances
Soldiers’ overall effectiveness in the operational environment. Training that enhances the ability
to recognize and understand relevant threat cues in operational settings allows Soldiers to carry
out their respective missions in a safer and more effective manner.
Procedure:
To test the effect of the trainer on Soldier skills, Soldiers completed a series of computer-
based tasks. Soldiers completed pre- and post-tests that included limited threat search, dynamic
change tasks, situational awareness tasks, and decision-making exercises. Soldiers were
randomly assigned to a training condition, in which they completed the trainer between the pre-
and post-test, or a control or no training condition, in which they completed a filler task.
Debriefing occurred after the experiment alongside a brief interview regarding realism of the
training tasks. All Soldiers assigned to the control condition had the opportunity to complete the
trainer later, if they chose.
Findings:
Soldiers reported finding the trainer valuable, despite no effect on threat detection
performance between the training and no-training conditions. Results from all four tasks,
quantitative and qualitative, indicated improvement over time. This finding may have resulted
from simple practice effects. These effects appeared to be consistent and robust, with effect
sizes that were sometimes relatively large. Across six statistically significant findings, partial
eta2 ranged from 0.146 to 0.367 (M = 0.25, SD = 0.08). This suggests that, on average, practice
effects (or some other effect of the pre-post manipulation) accounted for 25% of increase in
performance over time. The limitation of significant findings from the primary (training)
manipulation may stem from low statistical power resulting in limited ability to detect subtle
differences between conditions. The overall positive outcome is that all participants improved
on all of the primary skills of visual threat detection with only a 1.5 to 2 hour training session.
vi
Utilization and Dissemination of Findings:
The findings from this research, while insufficient to support and validate the trainer as
an effective method of training or enhancing visual threat detection ability in Soldiers, do not
suggest that the trainer is ineffective. The tasks in the pre- and post-tests were similar to those
presented in the trainer. Across both conditions, improvement occurred, suggesting that these
tasks did have a positive impact on performance. Soldiers with varying levels of experience
strongly expressed that the tool was interesting and challenging, potentially serving as a novel
aid to existing training. Further research should achieve higher statistical power to better
measure the validity and effectiveness of the threat detection skills trainer. The improvement in
performance scores would have to be investigated over a longer time to determine if such acute,
cognitive interventions are effective. Further, such a longitudinal finding could be very
beneficial for operational units who have a large amount of training to accomplish in a limited
training calendar.
vii
IMPROVING VISUAL THREAT DETECTION: RESEARCH TO VALIDATE THE THREAT
DETECTION SKILLS TRAINER
CONTENTS
Page
INTRODUCTION ..................................................................................................................... 1
Threat Detection Skills Trainer (TDST) ................................................................................ 1
Model of Threat Detection ..................................................................................................... 2
Experimental Tasks ................................................................................................................ 4
Limited threat search ......................................................................................................... 4
Dynamic Change Task ...................................................................................................... 5
Situation Awareness Task ................................................................................................. 6
Level 1: Perception ....................................................................................................... 6
Level 2: Comprehension ............................................................................................... 7
Decision-Making Exercises (DMX) ..................................................................................... 7
Summary ............................................................................................................................... 8
Method .................................................................................................................................. 8
Participants ........................................................................................................................ 8
Design ............................................................................................................................... 9
Materials ........................................................................................................................... 9
Computer-controlled stimuli presentation ..................................................................... 9
Military photographs ...................................................................................................... 9
Military videos ............................................................................................................... 9
EOF video stills.............................................................................................................. 10
Filler task ....................................................................................................................... 10
Computer-controlled questionnaire ............................................................................... 10
Procedure .............................................................................................................................. 10
Limited Threat Search .................................................................................................... 11
Dynamic Change Detection ........................................................................................... 11
Situation Awareness....................................................................................................... 11
Decision-Making Exercises (DMX) .............................................................................. 12
Analysis.................................................................................................................................. 12
RESULTS AND DISCUSSION ................................................................................................ 13
Limited Threat Search........................................................................................................... 13
Targets Identification ........................................................................................................ 13
Number of Responses ....................................................................................................... 13
Proportion of targets located per response (Hits Per Click) ............................................. 13
Response time (in milliseconds) ....................................................................................... 14
Dynamic Change Detection .................................................................................................. 14
Accuracy ........................................................................................................................... 14
viii
CONTENTS (Continued)
Page
Distance............................................................................................................................. 15
Threat relevant changes ................................................................................................. 15
Threat irrelevant changes ............................................................................................... 15
Threat relevant/irrelevant comparison ........................................................................... 15
Situation Awareness – Level 1: Perception ......................................................................... 16
Target selection ................................................................................................................. 16
Target vs. foil selection ..................................................................................................... 16
Situation Awareness – Level 2: Comprehension ................................................................. 17
Situation comprehension ................................................................................................... 17
Threat identification .......................................................................................................... 18
Threat-related story building ............................................................................................. 20
Threat detection challenges............................................................................................... 22
Decision-Making Exercises .................................................................................................. 24
Threat descriptions and implications ................................................................................ 24
Explanation of Threat Factors ........................................................................................... 26
Threat Detection and Mitigation Strategies ...................................................................... 27
Threat Detection Challenges ............................................................................................. 28
FURTHER ANALYSIS............................................................................................................. 29
DISCUSSION ............................................................................................................................ 30
REFERENCES .......................................................................................................................... 33
APPENDICES
Appendix A: Situation Assessment Questions ......................................................................... A-1
Appendix B: Decision-Making Exercises (DMX) .................................................................... B-1
Appendix C: Demographic Questions ...................................................................................... C-1
ix
CONTENTS (Continued)
Page
TABLES
Table 1. Relationship between Learning Objectives and the Threat Detection Loop .............. 3
Table 2. Experimental Tasks and Associated Behaviors .......................................................... 4
Table 3. Soldier Rank, Number Deployed, and MOS .............................................................. 9
Table 4. Approximate Time to Complete Each Task ............................................................... 11
Table 5. Video Clip Example 1: The Number of Soldiers Who Discussed Each Type of
Threat by Training Condition .................................................................................... 19
Table 6. Video Clip Example 2: The Number of Soldiers Who Discussed Each type of
Threat by Training Condition .................................................................................... 20
Table 7. The Number of Challenges Identified by Training Condition in the Road
Construction Scenario ................................................................................................. 29
FIGURES
Figure 1. Threat Detection Loop (inner rectangles) and measurable expert threat
detection behaviors (outer rectangles) ...................................................................... 2
Figure 2. Example of image threat search and annotations ...................................................... 4
Figure 3. Example of situation awareness video wireframe ..................................................... 6
Figure 4. VBS2 EOF video still for use in the decision-making exercise ................................ 10
Figure 5. Snapshot of the video that produced elaborative responses ...................................... 18
Figure 6. Snapshot of the video that corresponds to explaining why the elements
present a threat .......................................................................................................... 21
Figure 7. Snapshot of the video that corresponds to responses about challenges to threat
detection ..................................................................................................................... 23
x
1
IMPROVING VISUAL THREAT DETECTION: RESEARCH TO
VALIDATE THE THREAT DETECTION SKILLS TRAINER
Introduction
Previous research identifying the perceptual and cognitive skills of Soldiers during visual
threat detection tasks (Zimmerman, Mueller, Daniels, & Vowels, 2012; Zimmerman, Mueller,
Grover, & Vowels, in preparation) resulted in development of a training exemplar, the “Threat
Detection Skills Trainer,” also referred to as the TDST for the remainder of the document. The
intent of the trainer is to improve Soldier threat detection in dynamic environments. Previous
research enabled us to identify the relevant threat decision skills for the trainer (see the model of
threat detection; Zimmerman et al., in preparation). Even so, it was unclear how using the trainer
may actually enhance Soldiers’ threat detection skills. Thus, the purpose of the current research
was to determine the effectiveness of this trainer and identify how the U.S. Army can use the
trainer to support their need for efficient and effective training of visual threat detection.
Threat Detection Skills Trainer (TDST)
The TDST was designed to improve Soldiers’ ability to (a) identify threats, (b) manage
attention, and (c) engage in causal reasoning about threats. The summative effect of this training
is to develop Soldiers’ mental models through experiential learning in operationally relevant
computerized tasks and to improve threat reasoning ability. As a result, Soldiers will be better
able to recognize threats and react in threat situations. To accomplish this, the trainer
incorporates three sets of exercises requiring Soldiers to interact with digital photos depicting
operational environments where potential threats may exist. In the first set of exercises, Soldiers
view photos marked and annotated with threat cues previously identified by experienced
Soldiers. Soldiers click on marked threat cues to read about the threat, its relevance, and the
common enemy tactics associated with that threat. Soldiers then view photos without pre-
marked threat cues and click on potential cues. If the cues are relevant, an annotation appears
that provides information about that cue. After the threat search task, Soldiers complete a timed
threat search exercise, in which they have limited time to identify as many relevant threat cues as
possible. This first set of exercises enhances ability to identify threats by demonstrating what
threats look like, where they are likely to occur, and the reasons they occur.
In the next set of exercises, Soldiers view photos and answer questions about the content
depicted in the photos. The questions challenge Soldiers to think critically about the scene’s
context, consider potential threats, and generate possible solutions for addressing the threats.
During these exercises, Soldiers review statements made in previous research by experienced
Soldiers who evaluated the threats in the photos.
Following this exercise, Soldiers review written scenarios that provide background and
context to select photos. They read the scenarios and answer questions about potential threats
and the relevance of threats in the photos. These exercises enhance Soldier ability to engage in
causal reasoning about threats by requiring them to think about the greater context in which
potential threats exist and to identify threats based not only on physical features of the
environment, but also on other contextual features.
2
At two points during the previous exercises, Soldiers engage in two change detection
tasks. In those exercises, Soldiers view edited versions of photos that they saw during a previous
exercise. The edited photos contain added or removed threat-relevant and threat-irrelevant items
and Soldiers attempt to identify the changes. This exercise was designed to enhance Soldiers’
ability to manage attention by forcing them to attend to threat-relevant areas of environments and
detect relevant changes.
Model of Threat Detection
Vowels (2010) has argued, to detect, evaluate, and defeat threats in an operational
environment, Soldiers must perceive visual information and comprehend that information within
the context of the operational environment. Only by doing this effectively are they able to make
accurate decisions about potential threats in an operational environment. To comprehend an
environment, Soldiers must effectively search for, identify, and encode threat relevant
information. One outcome of previous research was a model of threat detection that identified a
set of measurable behaviors experienced decision makers would likely exhibit during threat
detection tasks (Zimmerman, et al., 2012; Zimmerman, Mueller, Marcon, Daniels, & Vowels,
2011). The model depicts a cycle of threat detection that includes (a) monitoring and search
activities required to maintain situation awareness when no overt threats are present, (b)
identifying and evaluating potential threats, and c) concluding a threat is present and determining
a course of action (Figure 1). Associated with the model are behaviors that experienced threat
detectors likely engage in as they attempt to detect threats.
Figure 1. Threat Detection Loop (inner rectangles) and measurable expert threat detection behaviors
(outer rectangles). From Determining the Requisite Components of Visual Threat Detection to
Improve Operational Performance (ARI Technical Report) by Zimmerman, L. A., Mueller, S. T.,
3
Grover, J., & Vowels, C. L. (in preparation). Fort Belvoir, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences.
The TDST trainer engages Soldiers in tasks where they monitor environments, engage in
search activities, and identify potential threats. In particular, this training is aimed at enhancing
Soldier ability to encode details about a situation (Refer to Figure 1): (a) encode situational
details, (b) classify threat versus non-threat features, (d) incorporate prior information into
decisions, (g) detect changes or anomalies, (h) create causal stories about threat situations, and
(i) conduct efficient visual search based on knowledge. The training encompasses four learning
objectives (Table 1). These learning objectives map onto the expected behaviors identified as
part of the threat detection loop (Figure 1). For instance, the initial exercises focus on encoding
details to identify and classify relevant threats. The goal of the reasoning tasks is to increase
Soldier ability to gather evidence to determine threat importance and impact. The change
detection tasks focus on perceiving changes and determining the relevance of the change.
Table 1
Relationship between Learning Objectives and the Threat Detection Loop
Learning Objectives
Step in Threat
Detection Loop
Threat Detection
Loop Behaviors
1. Demonstrate an improved
ability to identify relevant
threats in a variety of
situations.
• Monitoring, vigilance, and
search activities
a) Encoding details for immediate
or delayed test
2. Distinguish between threat-
relevant and threat-irrelevant
cues in time-limited situations.
• Identify/classify threat or
non-threat
b) Ability to classify threat vs.
non-threat features
i) Ability to look at high-priority
threat locations based on
knowledge
3. Recognize the importance and
potential impact of each
detected threat.
• Identify/classify threat or
non-threat
• Gather evidence to ID
threat
c) Ability to adapt decision
selectivity based on
information
f) Ability to balance decision with
information gathering
h) Ability to create causal stories
about threat situations
4. Identify threat-relevant
changes in the environment.
• Change or anomaly
detection
g) Ability to detect change or
anomaly
It is possible to objectively identify changes in performance resulting from training,
because these behaviors can be assessed empirically. To determine if the TDST meets the
learning objectives and the overall goal of improving Soldier ability to detect and identify
relevant threats, we conducted an experiment by measuring inexperienced Soldier performance
before and after completing the training. The performance of the experimental group was
compared to a control group of Soldiers who had completed the same before and after exercises
without the training. During the experiment, Soldiers engaged in four tasks, each intended to
measure threat detection behaviors. Table 2 lists the tasks and the behaviors measured in each
task.
4
Table 2
Experimental Tasks and Associated Behaviors
Task
Threat Detection Behavior
Limited threat search
• Look at high-priority threats based on knowledge (i)
•
Classify threat versus non-threat features (b)
Dynamic change task
• Look at high-priority threats based on knowledge (i)
• Classify threat versus non-threat features (b)
• Detect change or anomaly (g)
Situation awareness task
• Encode details for immediate or delayed test (a)
• Look at high-priority threats based on knowledge (i)
• Create causal stories about threat situations (b)
Decision-making exercise
• Classify threat versus non-threat features (b)
• Create causal stories about threat situations (h)
• Incorporate prior knowledge into decisions (d)
Experimental Tasks
Limited threat search.
In previous research (Zimmerman, Mueller, Daniels, & Vowels, 2012; Zimmerman,
Mueller, Grover, & Vowels, in preparation), Soldiers identified and annotated potential threats in
a set of static images. After reading a brief background story for each image, Soldiers viewed
each image and clicked with a computer mouse pointer on areas and items they judged to be the
greatest threat. They rated the likelihood and severity of an actual threat occurring in that
location on a 3-point scale (low, medium, high). After providing ratings, Soldiers annotated the
location with a short description of the potential threat (see Figure 2). Soldiers provided between
three and 1 points of interest per image.
Figure 2. Example of image threat search and annotations.
5
We applied this measurement method during a previous experiment (Zimmerman et al.,
in preparation). During that task, Soldiers searched an image and clicked on locations where
potential threat targets existed. For portions of the task, Soldiers had limited resources (a limit of
10 clicks) and limited time and resources (a maximum of one click per second for 10 seconds).
Findings from that previous research indicated Soldiers with deployment experience performed
slightly better when under time pressure, whereas time pressure had greater negative influence
on Soldiers with no experience.
In the current research, these images provided a measure of performance on a threat
search task. Soldiers clicked on locations in images where threats might exist, using a limited
number of clicks. The closer their clicks were to the locations where experienced Soldiers
clicked in the previous experiment, the more closely we can say their performance matches
someone with experience (proficiency). We hypothesized that the clicks would be closer post-
training compared to pre-training.
Dynamic Change Task.
Soldiers in the current research attempted to respond to threats that appeared in
operationally relevant photos. In the previous research (Zimmerman, et al., 2012) Soldiers
completed a dynamic threat detection task by guiding their visual attention over images obscured
by vibrating transparent dots. Soldiers were required to maintain constant vigilance as they
attempted to detect a target dot that appeared briefly at uneven intervals during the viewing. The
target dot would appear at threat relevant and threat irrelevant locations. Soldiers clicked a
computer mouse button when the target was detected, after which a cursor appeared and Soldiers
indicated the location of the target. The current research deviated from this by incorporating
change detection with threat detection and by updating the stimuli to include threat relevant and
threat irrelevant targets (rather than semi-transparent dots).
In previous research (Zimmerman, et al., 2012), Soldiers engaged in a change detection
task by identifying changes in previously viewed photos. Findings revealed high false alarm
rates; however, when Soldiers did identify actual changes, the majority of these were in threat-
relevant locations. In addition, Solders were more likely to identify the changes when the second
photo contained an object that was absent from the first photo compared to when the object was
removed in the second photo.
The goal of the dynamic change task was to determine if Soldiers focus attention on
threat-relevant locations to detect changes that occur in those locations. We hypothesized that
Soldiers would detect more changes in threat-relevant locations and miss more changes in threat
irrelevant locations after training. We also predicted that reaction time would be quicker when
identifying threats in relevant locations. This task utilized the Flicker paradigm (Rensink,
O’Regan, & Clark, 1997; Scholl, 2000). Soldiers viewed each photo multiple times and in some
viewings, a single object in the photo changed. The photos flashed on the screen and between
flashes, a mask concealed the photo. Soldiers viewed the changed and unchanged photos until
they identified the location of the change.
6
Situation Awareness Task.
The purpose of the situation awareness task was to determine if trained Soldiers are able
to perceive and comprehend elements in an environment more effectively than those who have
not completed the TDST. The ability to perceive and comprehend information constitutes the
first two levels of situation awareness (Endsley, 2000). According to Endsley, situation
awareness occurs at three levels. Level 1 is the ability to perceive objects in an environment,
Level 2 is the ability to comprehend their meaning, and Level 3 is the ability to project their
status into the future. To test both Level 1 and 2 situation awareness, we used video recordings
of Soldier activities in Afghanistan retrieved from military websites. These videos depict various
environments, ranging from dismounted patrols to convoy operations that offer relevant
environmental characteristics. For example, the dismounted patrol environment offers building
structures (both permanent and temporary), individual persons and crowds, cars, potential
surveillance or hiding points, and potentially suspicious behavior. Being able to monitor and
search an environment and subsequently encode details about the situation are precursors to
detecting relevant threats.
Level 1: Perception. To test perception, Soldiers viewed short video segments that
stopped abruptly. At this point, a wireframe drawing of the last frame in the video appeared and
Soldiers placed cutouts of items (e.g., people, vehicles, rocks, windows) from the video into the
wireframe at the approximate location of those items (see Figure 3). The items in the scenes
represented cues that, based on previous research findings, were relevant to identifying and
comprehending threats. In addition to the items from the video (targets), Soldiers could select
items that were not in the video (foils). Each exercise included target items and two foil items.
Analysis included accuracy in item choosing and location.
Figure 3. Example of situation awareness video wireframe.
7
Level 2: Comprehension. After Soldiers identified items in these images, they described
the potential meaning, relevance, and challenges associated with these cues (Appendix A). We
evaluated these descriptions for level of detail and comprehension. We also coded comments
that were characteristic of experienced Soldiers, such as hypotheses about what is occurring in
the scenes, enemy strategies and motivations, and strategies they would use to deal with the
situation.
We hypothesized that Soldiers who completed the TDST would select threat-relevant
items (targets) more often than threat-irrelevant items (foils) and they would be more accurate in
placing target items in correct locations compared to threat-irrelevant locations. They would also
provide descriptions that include hypotheses about the situation and strategies for dealing with
the threats. We also expected that Soldiers who completed the training would provide
descriptions that indicated they comprehend the meaning of the items in the scene, particularly
threat-relevant items.
Decision-Making Exercises (DMX).
The purpose of the DMXs was to determine if training improved Soldier ability to create
causal stories about threat situations and incorporate prior information into their decisions.
Previous findings (Zimmerman, Mueller, Daniels, & Vowels, 2012; Zimmerman, Mueller,
Grover, & Vowels, in preparation), showed that Soldiers with more experience (as quantified by
the number of deployments) provided complex narratives about detecting threats that included
the strategies, skills, and challenges associated with detecting threats. While experienced
Soldiers constructed rich accounts of threat detection situations, novice Soldiers (with no
deployment experience) often gave context-free assessments of the situations, providing
responses that indicated they were aware of important cues that indicated threats, but lacked the
ability to incorporate these cues into meaningful narratives.
We relied on findings from interviews conducted in previous research (Zimmerman, et
al., 2012; Zimmerman, et al., in preparation) and our military subject matter expert to create
engaging and realistic scenarios that reflected current threat detection contexts. The scenarios
included elements of uncertainty, relevant and irrelevant cues, and conflicting goals (Klein,
2004). We hypothesized that Solders who completed the TDST would provide descriptions that
demonstrate more advanced decision characteristics. The Dreyfus and Dreyfus model of skill
acquisition provided metrics for classifying situation assessment and decisions on a range from
‘novice’ to ‘expert’ (Dreyfus & Dreyfus, 1986; Ross, Phillips, Klein, & Cohn, 2005).
Assessments of skill level took into account several factors, such as the strategies, cues, and
challenges that Soldiers discussed.
The goal of the DMX task was to identify any changes in Soldier causal reasoning after
completing the TDST. During the DMXs, Soldiers read a scenario and viewed a photo that
represented their current view of the situation. After they read the scenario, they had two
minutes to provide a solution to the problem presented in the scenario (Appendix B). After
completing their narrative, Soldiers answered several questions aimed at assessing their critical
thinking and causal reasoning skills.
8
Summary
The research objective for this project was to determine how the TDST could impact
threat detection skills. We hypothesized that Soldiers who completed the threat detection skills
training would perform better on the four identified tasks compared to Soldiers who did not
complete the training. We also expected to see improved performance after training compared to
before training. Specifically, we predicted that trained Soldiers would:
• Identify relevant threats faster and more accurately by clicking on threat locations that
reflect the locations previously chosen by experts;
• Detect more changes in threat relevant locations and miss more changes in threat
irrelevant locations;
• Recall more details when asked to describe a current situation from memory and
provide explanations about elements of the situation;
• Identify and be more accurate in placing threat relevant items (targets) in correct
locations compared to threat irrelevant items (foils);
• Provide descriptions of the scene that indicated they comprehend the meaning of the
items in the scene, particularly threat-relevant items; and,
• Provide assessments of situations that include elements indicative of more expert
decision skills.
Method
Participants.
Forty-three U.S. Army Soldiers participated in this research. All Soldiers were male.
The mean age of participants was 21 years (range 19-37). Five Soldiers deployed once, and one
Soldier deployed twice (14% total). Of those Soldiers who reported previous deployment, three
indicated going outside the wire more than once a month, two reported going out once a month,
and one reported going outside less than once a month. The average time in service across all
Soldiers was 17 months (range 9-60 months). The majority of Soldiers had a rank of private first
class (PFC) and ranged from private (PVT) to Corporal/Specialist (CPL/SPC). Table 3 lists the
number of Soldiers in each rank, the number deployed in each rank, and the MOS by rank. The
five Soldiers with MOSs other than 11B had deployed, as did one Soldier with an MOS of
infantryman (11B).
9
Table 3
Soldier Rank, Number Deployed, and MOS
MOS
Rank
Total
Deployed
11B
19K
91B
91M
Private
2
0
2
PV2
1
0
1
PFC
24
1
23
1
CPL/SPC
16
5
12
1
1
2
Note: MOS are 11B = Infantryman, 19K = M1 Armor Crewman, 91B = Wheeled Vehicle
Mechanic, 91M = Bradley Fighting Vehicle System Maintainer.
Design.
This research was a 2 (training: training vs. no training) x 2 (pre/post: pre-test vs. post-
test) mixed design, with training being a between-subjects factor. We randomly assigned
Soldiers to the training condition. Twenty-two Soldiers completed the training.
Materials.
Soldiers completed all tasks and the threat detection training on laptop computers. They
viewed a variety of photos and videos depicting relevant military situations, read content, and
made responses by using a computer mouse to click on the computer screen. They also provided
descriptions and answered questions using the keyboard. Thus, we had written entries for all of
the questions to which Soldiers responded.
Computer-controlled stimuli presentation. We used the Psychology Experiment
Building Language (PEBL) 0.12 computer experimentation system to control the computer
exercises (Mueller, 2009). All stimuli and questions were programmed into PEBL and Soldiers
completed the experiment by viewing and responding in this program. The PEBL recorded all
responses for later analysis.
Military photographs. We chose photos for this experiment from among a set of photos
retrieved during research conducted to develop the TDST (Zimmerman, et al., 2012). These
images were retrieved from three main sources, www.DefenseImagery.mil, www.Flickr.com,
and www.defense.gov. All images were approved for public release. These photos appeared in
the limited threat search task, dynamic change detection task, and decision-making exercises.
Military videos. The U.S. Army Research Institute retrieved a random selection of
military videos depicting recent operations in Afghanistan and Iraq from the Defense Imagery
Management Operations Center (DIMOC). All videos were approved for public release. We
reviewed these videos and selected relevant scenes depicting threat detection challenges. The
average length of the edited video clips was 12 seconds (range 7s to 13s).
10
EOF video stills. In addition to the military videos depicting live events, we also used
still snapshots of simulated videos depicting challenging decision points in military situations.
The U.S. Army Research Institute recently produced a series of Virtual Battle Space 2 (VBS2)
escalation of force (EOF) exercises (Johnston, Leibrecht, Topolski, Vowels, & Singer, in
preparation). Soldiers viewed snapshots of these exercises to complete two decision-making
exercises. Figure 4 shows a screen shot of an EOF exercise.
Figure 4. VBS2 EOF video still for use in the decision-making exercise.
Filler task. Soldiers who did not complete the training engaged in a filler task for
approximately 30 minutes. The filler task included games such as word find, Sudoku, and
crossword puzzles. Soldiers could choose which games they wanted to play. Soldiers completed
the paper-based games using pen or pencil. They could take the games with them when the
experiment was complete.
Computer-controlled questionnaire. Soldiers completed a demographic questionnaire as
a part of the computerized tasks. The questionnaire included demographic questions along with
questions about education and military experience (Appendix C).
Procedure.
This experiment lasted approximately two hours per Soldier. The first five minutes of the
experiment, involved briefing Soldiers on the nature of the tasks and then collecting informed
consent. Soldiers then completed a practice exercise to ensure they understood the tasks.
Following this, they participated in the pre-test by completing the four tasks. After a 5-minute
break, the Soldiers in the training condition completed the threat detection skills training while
Soldiers in the no training condition engaged in the filler task. The training took approximately
11
30-35 minutes to complete. After another 5-minute break, all Soldiers completed the post-test
(see Table 4). The four tasks included in the pre- and post-tests were counterbalanced to control
for order and sequence effects. The stimuli in each task (e.g., photos) were also counterbalanced
when applicable. Following the post-test, Soldiers completed a brief demographic questionnaire
and provided any final comments. Finally, a researcher debriefed the Soldiers individually or in
groups as they finished the experiment. Soldiers had the opportunity to ask questions during the
debriefing session.
Table 4
Approximate Time to Complete Each Task
Task
Time to complete
Informed Consent and Briefing
5 minutes
Pre-test (including practice exercise)
30-35 minutes
Break
5 minutes
Training or Filler Task
30-35 minutes
Break
5 minutes
Post-Test (including demographics and debrief)
30-35 minutes
Limited Threat Search. Soldiers monitored static scenes and identified possible threats
by clicking on items in the scenes. They were limited to a maximum of 10 clicks and 15 seconds
per photo. The limited resources (i.e., clicks and time) required them to adopt a more
conservative selection criterion while the limited time added time pressure. The purpose of these
constraints was to force Soldiers to focus their attention on threat relevant locations and choose
only high-priority threats. Soldiers completed this task by viewing a series of photos and
clicking with a computer mouse on the photos in threat relevant locations that they deemed a
high-priority. We evaluated Soldier performance by comparing their responses to the responses
of experienced Soldiers in previous research.
Dependent variables recorded in this task included the locations chosen, weighted by the
order in which they were selected, and the time profile of the searches. Independent variables
included threat relevance and threat location based on previous research data.
Dynamic Change Detection. Soldiers viewed the changed and unchanged photos until
they identified the location of the change or until 20 seconds elapsed, at which time they were
required to identify a location where they thought a change had occurred. Each altered and
unaltered photo appeared several times in a row. Once Soldiers detected the change, they
indicated where they saw the change by clicking that area on the photo. Once they clicked their
chosen spot, the location of the actual change appeared. The purpose of task was to determine if
Soldiers noticed changes in threat relevant areas faster and more often than changes in threat
irrelevant areas.
Dependent variables recorded in this task included change detection accuracy, frequency
of detecting threats in relevant versus irrelevant locations, and response latency. Independent
variables included manipulation of change locations based on relevancy.
12
Situation Awareness. Soldiers viewed videos of U.S. Soldiers conducting vehicle and
foot patrols in Afghanistan. The videos stopped abruptly during the action and a wireframe of
the last frame of the video appeared alongside several items. Each video lasted an average of 11
seconds (range 7 to 12s). Soldiers then placed the items from the video into the wireframe at the
location they remembered seeing the item. They could select from among items that were in the
video (targets) or not in the video (foils). Following identification, they described what was
happening in the scene and answered questions about the relevant cues, threats, and challenges.
This task required Soldiers to perceive information from the scene and comprehend the meaning
of that information.
Dependent variables in this task were item identification accuracy, item placement
accuracy, and completeness of description. Independent variables included whether the item was
a target or foil.
Decision-Making Exercises (DMX). Soldiers read short scenarios about challenging
patrol situations. Each scenario included a photo that depicted the last moment in the scenario.
After reading a scenario, Soldiers had two minutes to describe potential threats in the scene and
explain the meaning of the threats. They then answered questions about the threat-relevant cues
and challenges presented in the scenarios. This task tested Soldiers’ causal reasoning and critical
thinking skills.
Dependent variables for this task included elaboration of answers to include the meaning
and implication of threats, why cues were relevant, strategies used to detect and mitigate threats,
and challenges when attempting to detect threats. We evaluated answers according to the
Dreyfus and Dreyfus 5-stage model of skill acquisition by examining Soldier explanation of
cues, strategies, and challenges along criteria matching novice, advanced beginner, competent,
proficient, and expert (Dreyfus & Dreyfus, 1986).
Analysis.
The primary purpose of the research was to evaluate the effectiveness of the TDST by
measuring several metrics of performance. We used qualitative and quantitative analysis
techniques to provide empirical results with both theoretical and applied value. We used the
statistical program SPSS to analyze the quantitative data. We conducted repeated measures
analyses of variance (ANOVA) to analyze data gathered from all the tasks. For all ANOVAs,
equality of variance was assessed using Box’s M statistic. There were no observed violations
variance-covariance equality, all ps > .026.1 No statistically significant main effects of training
were observed across tests (only one p < .04, all other ps > .065); therefore, with one exception,
we do not report main effects of training.
In addition, we conducted qualitative analysis of the data gathered in the situation
awareness task and the DMX task. The purpose of the qualitative analysis was to identify
1 Because Box’s M is a sensitive test, p-values > .001 do not represent a significant difference (Tabachnick & Fidell,
2001).
13
characteristics in Soldier responses that resembled decision makers experienced in military threat
detection (Zimmerman, et al., 2012). Two researchers rated select segments of the situation
awareness and DMX data for each qualitative question. Average agreement was 79% (range
43% to 100%). After coding a sample of data, the researchers identified disagreements and
resolved them through discussion. If agreement was low, the researchers refined the rating
criteria to improve consistency across coders and then coded an additional sample of data to
assess agreement. After achieving consistent levels of agreement, each researcher coded half the
remaining data. The researchers occasionally conferred with each other to discuss difficult
instances and maintain consistent coding practices.
Results and Discussion
Limited Threat Search
Targets Identification.
We calculated the number of targets identified by adding the frequency of clicks made by
each Soldier on targets that experienced Soldiers identified in previous research as threat relevant
(Zimmerman, et al., in preparation). A repeated measures ANOVA using pre/post as a within-
subjects variable and training as a between-subjects variable revealed no interaction effect of
training x pre/post on the number of threat relevant targets located, F (1, 41) = .372, p = .545,
partial eta2 = .009. However, a main effect of pre/post on number of threat relevant targets
located was observed, F (1, 41) = 16.143, p < .001, partial eta2 = .283.2 The number of threat
relevant targets located increased from pre-test (M = 36.42, SD = 9.94) to post-test (M = 42.19,
SD = 9.85). Soldiers’ ability to locate threats in the images improved over time regardless of the
training intervention, which indicates a practice effect over time.
Number of Responses.
No interaction effect of training x pre/post on the total number of responses (clicks) was
observed, F (1, 41) = 2.257, p = .141, partial eta2 = .052. However, a main effect of pre/post on
the total number of searches (clicks) was observed, F (1, 41) = 23.766, p < .001, partial eta2 =
.367. The number of total searches (clicks) increased from pre-test (M = 141.19, SD = 19.42) to
post-test (M = 152.07, SD = 15.88). This finding suggests that participants were likely to
respond more often by clicking on threat locations over time, irrespective of the training
intervention.
Proportion of Targets Located Per Response (Hits Per Click).
No interaction effect of training x pre/post on the proportion of targets found per response
(click) was observed, F (1, 41) = 1.897, p = .176, partial eta2 = .044. A possible main effect of
pre/post on the proportion of targets found per response was observed, F (1, 41) = 4.789, p =
2 Because we conducted four distinct analyses within this test, we compared all statistics to a Bonferroni corrected
alpha of .0125.
14
.034, partial eta2 = .105. However, this effect failed to meet Bonferroni’s corrected alpha of
.0125 (correcting for four planned comparisons). Despite failing to meet a more stringent test of
statistical significance, the number of targets found per response (clicks) increased from pre-test
(M = .26, SD = .06) to post-test (M = .28, SD = .08). Soldiers tended to identify threats more
often during the post-test and they tended to find more targets per click. This might indicate that
they became more efficient at detecting relevant threats over time, but this claim should be
balanced against the size of the difference observed (M = .02) and the probability of error (p =
.034).
Response time (in milliseconds).
To obtain response time, we calculated the time that elapsed between the start of each
trial and each response within that trial. A repeated measures ANOVA using pre/post (pre- vs.
post-test) as a within-subjects variable and training (training vs. no training) as a between-
subjects variable revealed no interaction effect of training x pre/post on response time, F (1, 41)
= 1.573, p = .217, partial eta2 = .037. However, a main effect of pre/post on response time was
observed, F (1, 41) = 18.283, p < .001, partial eta2 = .308. The time participants took between
search clicks decreased from pre-test (M = 707.43, SD = 214.12) to post-test (M = 602.55, SD =
195.70). This result supports an increase in efficiency for detecting threats over time: Soldiers
searched more quickly, more often, and found more targets during post-test trials than during
pre-test trials. A possible main effect of training was observed, t(41) = 2.106, p = .041, Cohen’s
d = 0.64; Mean Trained = 543.5, SD = 168.8; Mean Untrained = 664.4, SD = 206.6. There was
no difference between groups pre-training: t(41) = .913, p = .366. This suggests that Soldiers in
the training condition responded quicker than did Soldiers in the control condition during the
limited threat search. However, this effect failed to meet Bonferroni’s corrected alpha of .016
(correcting for three planned comparisons of training group by pre/post -test), and so should be
interpreted with caution. That is, any increased in efficiency observed was likely driven by
practice effects rather than training.
Findings from the limited threat search analysis indicate that training did not influence
performance beyond the benefits seemingly gained through practice alone. Important to note is
that training did not appear to harm performance. The tasks in the pre- and post-test conditions
were similar to the tasks in the training condition, thus these results may indicate that simply
practicing threat detection using tasks such as these are enough to improve performance.
Dynamic Change Detection
Accuracy.
We considered Soldier responses accurate when they clicked within 50 pixels of the
change location in the photo.3 Accuracy was the proportion of changes identified across 20
3 In Zimmerman et al., (2012), Soldiers were 83% accurate when clicking on target locations when the accepted hit
radius was 50 pixels. This rate improved marginally to 84% when the accepted hit radius increased to either 100 or
150 pixels. A 50-pixel radius provided a slightly more stringent test of accuracy without sacrificing validity.
15
trials. A repeated measures ANOVA using pre/post (pre- vs. post-test) as a within-subjects
variable and training (training vs. no training) as a between-subjects variable revealed no
interaction effect of training x pre/post on accuracy, F (1, 38) = 1.49, p = .229, partial eta2 =
.038. However, a main effect of pre/post on accuracy was observed, F (1, 38) = 10.04, p = .003,
partial eta2 = .209; post hoc t (39) = 3.092, p = .004, d = 0.44.4 The number of changes
identified increased from pre-test (M = .472, SD = .175) to post-test (M = .546, SD = .159). This
finding indicates that Soldier ability to detect differences between pairs of photos improved with
time, regardless of training intervention. However, this measure of accuracy may have been
somewhat approximate because it was measured using a 50-pixel threshold and did not account
for trials in which a Soldier may have identified the change location but simply did not click
close enough to the location on screen, in which case it was scored as inaccurate. To assess this,
we analyzed the distance between response location and change location.
Distance.
To conduct a more sensitive analysis of response accuracy, we calculated the distance
between response location and the center of the change location. A repeated measures ANOVA
using pre/post (pre- vs. post-test) as a within-subjects variable and training (training vs. no
training) as a between-subjects variable revealed no interaction effect of training x pre/post on
the distance between response location and change location, F (1, 38) = 0.243, p = .625, partial
eta2 = .006. However, a main effect of pre/post on distance was observed, F (1, 38) = 6.49, p =
.015, partial eta2 = .146; post hoc t (39) = 2.55, p = .015, d = 0.415. The distance between the
response location and the change location decreased from pre-test (M = 155, SD = 51) to post-
test (M = 134, SD = 51). This test supported the accuracy results reported above: Soldier ability
to identify the location of changes improved over time, regardless of training intervention.
We conducted additional analysis to understand if training influenced Soldier ability to
spot threat-relevant changes more than the ability to spot threat-irrelevant changes.
Threat relevant changes. Analysis of only threat relevant changes revealed no
interaction effect of training x pre/post on accuracy, F (1, 38) = .029, p = .867, partial eta2 =
.001.
Threat irrelevant changes. Analysis of only threat irrelevant changes revealed no
interaction effect of training x pre/post on accuracy, F (1, 38) = 1.994, p = .166, partial eta2 =
.050.
Threat relevant/irrelevant comparison. We analyzed the interaction of training, pre/post
testing, and the threat relevance of the change in each pair of photos by calculating difference
scores at each test time (pre vs. post) and using the scores as the dependent variable in a repeated
measures ANOVA. Analysis of response time difference (RTIrrelevant – RTRelevant) revealed no
interaction between training x pre/post, F (1, 38) = .484, p = .491, partial eta2 = .013. Analysis
4 Because we conducted two distinct analyses (accuracy and distance) within this test, we compared all statistics to a
Bonferroni corrected alpha of .025.
16
of accuracy difference (ACCRelevant – ACCIrrelevant) also revealed no interaction between training x
pre/post, F (1, 38) = .149, p = .702, partial eta2 = .004. These findings indicate that threat
relevance did not influence reaction time or accuracy. Overall, the only statistically significant
effects observed in the change detection task might indicate practice effects, as Soldier ability to
detect changes across pairs of photos improved over time, but not with training.
These results mirror those found in the limited threat search task by again indicating that
practicing the skills required to accomplish the task was enough to improve performance, with
no benefit exclusively from training. Again, training did not appear to harm performance; it
simply did not have an observable effect on performance.
Situation Awareness – Level 1: Perception
Target selection.
To assess threat detection perception, we analyzed the number of target and foil items
selected in the wireframe images. Repeated measures ANOVAs with pre/post (pre- vs. post-test)
as a within-subject factor and training (training vs. no training) as a between-subjects factor
revealed no interaction effect of training x pre/post on the number of targets (vs. foils) selected,
F (1, 31) = .001, p = .977, partial eta2 < .001. No main effect of pre/post on the number of
targets selected was observed, F (1, 31) = .917, p = .346, partial eta2 = .029.5 No interaction
effect of training x pre/post on the number of foils (vs. targets) selected was observed, F (1, 31)
= .771, p = .387, partial eta2 = .024. However, a main effect of pre/post on the number of foils
selected was observed, F (1, 31) = 8.228, p = .007, partial eta2 = .210. The number of foils
selected decreased from pre-test (M = 3.48, SD = 3.4) to post-test (M = 2.39, SD = 3.0).
Collapsing across training conditions, two paired samples t-tests revealed a difference in the
number of targets selected versus the number foils selected: a pre-test, more targets (M = 6.79,
SD = 2.27) than foils (M = 3.48, SD = 3.41) were selected, t(32) = 6.94, p < .001, d = 1.31;
similarly, at post-test, more targets (M = 6.42, SD = 2.45) than foils (M = 2.39, SD = 2.97) were
selected, t(32) = 8.40, p < .001, d = 1.48. These findings indicate that Soldiers were no more
likely to select targets over time but they did appear to discriminate by selecting fewer foils over
time. This discrimination does not appear to be a function of training.
Target vs. foil selection.
Given the difference between selecting targets versus foils over time, we calculated a
difference score for this factor and included it as a dependent variable in a repeated measures
ANOVA. Analyzing the difference between the number of targets selected versus the number of
foils selected revealed no interaction effect of training x pre/post, F (1, 31) = .974, p = .331,
partial eta2 = .030. However, a main effect of pre/post on target vs. foil selection was observed,
F (1, 31) = 4.281, p = .047, partial eta2 = .121. However, this effect failed to meet Bonferroni’s
corrected alpha of .025. Despite failing to meet a more stringent test of statistical significance,
5 Because we conducted two distinct analyses within this test, we compared all statistics to a Bonferroni corrected
alpha of .025.
17
the difference in the number of targets (vs. foils) selected appears to have increased from pre-test
(M = 13.30, SD = 2.73) to post-test (M = 14.03, SD = 2.76). This finding further indicates that
while the number of targets selected remained relatively stable over time, the number of foils
selected decreased over time (a 5% increase in targets selected vs. a 31% decrease in foils
selected).
As observed in the previous tasks, practice effects seemed to account for observed
difference in performance. This is not to say that training was ineffective. It did not harm
performance; however, its effects are at least matched by those of practice alone.
Situation Awareness - Level 2: Comprehension.
To understand Soldier comprehension, we conducted both quantitative and qualitative
analysis of Solder responses to four questions (Appendix A). We expected that Soldiers who
completed training would provide more elaborate descriptions and would provide details that
indicated they comprehended the meaning and implications of the threat cues. Across all
questions, Soldiers gave relatively brief responses.
Situation comprehension.
Researchers rated answers to the first question “Describe what is happening in the scene”
as literal or implied. Responses received literal ratings when Soldiers described what they
viewed in the videos with no interpretation of what they saw. Researchers rated responses as
implied when Soldiers provided interpretations of what they viewed in the videos. In previous
research, more experienced Soldiers constructed rich, causal stories about a threat situation
(Zimmerman, et al., 2012). We expected training to enhance Soldiers’ ability to create causal
stories about the threat environment. For example, one video clip showed Soldiers walking
down a road lined with high grass and then walking down a road in a line with foreign personnel
(for example, the Afghan National Army). Typical literal responses to this question included:
• “Foot patrol” or “patrols,”
• “A foot patrol is walking next to a wall,” and
• “Soldiers moving.”
In such examples, Soldiers described what occurred in the video clip without providing
any interpretation. This is consistent with more novice performance, where such responses are
generally fact-driven. In general, Soldiers with more experience tend to create stories around
situations. Soldiers’ who responded with elaborative descriptions often included implied or
story-telling components:
• “EOD under escort on a move to a possible IED location,”
• “Training the Afghan army,” and
• “Marines conduct a presence patrol.”
These examples illustrate Soldier ability to reason from what they saw to interpretations
based on cues and the expectations that have about the cues. We expected that with training,
18
Soldiers would provide answers that indicate better comprehension of the scenario. Repeated
measures ANOVA with pre/post (pre- vs. post-test) as a within-subjects factor and training
(training vs. no training) as a between-subjects factor revealed no interaction effect on the
number of literal vs. implied responses F (1, 32) = .13, p = .72, partial eta2 < .01. Soldiers
descriptions did not vary with training, F (1, 32) = .49, p = .49, partial eta2 = .02 or pre/post, F
(1, 32) = 2.09, p = .16, partial eta2 = .06.
Threat identification.
The majority of Soldiers responded to the second question, “Which element(s) in the
scene are the biggest threat?” by listing the threats as asked. This is consistent with novice or
advanced beginner expected performance (Dreyfus & Dreyfus, 1986; Zimmerman et al., 2012).
A few Soldiers also provided cursory explanations about the implications of these threats (e.g.,
could be an improvised explosive device [IED]). Across all videos, Soldiers listed threats and
added additional explanation in 33 instances. Training did not appear to influence Soldier’s
propensity to elaborate beyond threat identification. One scenario produced the most elaboration
(8 instances, see Figure 5). In this video, a group of Soldiers walked past a building, looked at
the ground, and indicated footprints. The terrain was rocky, with small hills that included grass
and dry brush. Soldiers’ hypotheses included:
• “The box right next to the tree [is a threat] because it could be an IED, the actual tree
[could be a threat] because somebody could be hiding in it,”
• “The weeds and brush can hide IED’s,” and
• “The brush and ditches can be used to conceal an IED.”
Figure 5. Snapshot of the video that produced elaborative responses.
19
More elaborative responses to this video may have been due to unique features of the
situation. The cues present in the scenario could have been particularly strong (e.g., Soldiers in
the video are looking at the ground; they must be looking for threats) making it easier for
Soldiers to give additional information. The majority of Soldiers described threat cues without
further explanation. For scenario described above, these responses included:
• “The unsettled dirt,”
• “Grassy area,” and
• “Weapons not held properly.”
We hypothesized that training would influence Soldier ability to detect relevant cues.
Soldier identification of the biggest threat in each video yielded two measures of threat cue
selection: the type of cues and the number of cues identified. On average, Soldiers identified 1.5
cues per video clip and these cues ranged from seven to 21 cue categories per video clip.
Zimmerman et al., (2012) found that Soldiers with more deployments identified a greater variety
of threats compared to Soldiers who had never deployed. Given the relatively low level of
experience among Soldiers in this reseearch, we did not expect to see significant differences in
cue identification with and without training. However, we expected small differences in the
kinds of cues selected by trained Soldiers compared with untrained Soldiers. In general, Soldiers
did not identify different cues as a function of training (Table 5).
Table 5
Video Clip Example 1: The Number of Soldiers Who Discussed Each Type of Threat by
Training Condition
Threat Cue
Pre-test
No Training
(n* = 11)
Pre-test Training
(n = 6)
Post-test
No Training
(n = 7)
Post-test
Training
(n = 13)
Building
8
1
4
5
Grass
6
4
3
6
Corn
0
1
2
1
Hole in wall
1
1
1
0
Wall
1
0
2
1
Ditch
0
0
0
1
IEDs
0
0
1
2
Enemy fire
1
0
0
0
*Note: n indicates the number Soldiers per condition. Soldiers could discuss more than one type of
threat.
Regardless of training, Soldiers primarily identified the building and the grass as the
greatest threats. Differences among the relative percentages of threats could be due to variation
in the number of Soldiers who responded in each condition. In another video example, Soldiers
identified the threats in a video that depicted U.S. military and foreign military personnel
walking down a road. A motorcycle drove past the forces while they walked down the middle of
the road. Again, Soldiers identified similar cues regardless of training (Table 6).
20
Table 6
Video Clip Example 2: The Number of Soldiers Who Discussed Each Type of Threat by
Training Condition
Threat Cue
Pre-test
No Training
(n* = 7)
Pre-test
Training
(n = 14)
Post-test
No Training
(n = 10)
Post-test
Training
(n = 5)
IED
0
0
1
0
Vehicle
1
7
5
2
Man on bike
4
5
3
3
White van
1
0
1
0
Hole in ground
1
0
1
0
People
0
0
2
1
Moped
3
4
0
0
VBIED
0
2
1
0
Fresh dirt
0
1
0
0
Light pole
0
1
0
0
Stack of Boxes
0
1
0
0
*Note: n indicates the number Soldiers per condition. Soldiers could discuss more than one type of
threat.
Across training conditions, Soldiers identified vehicles and the man on the bike as the
two greatest threats. After training, Soldiers appear to identify fewer threats; however, this could
be due to a small number of Soldiers responding in each condition. An alternative explanation is
that with training, participants narrow their choices to threat relevant cues rather than identifying
all possible threats (Zimmerman et al., 2012). Such a finding may be scenario specific, as this
finding did not appear to generalize across scenarios.
If training helped Soldiers focus on more relevant cues, we would expect a difference in
the number of relevant cues reported. Repeated measures ANOVA with pre/post (pre- vs. post-
test) as a within-subject factor and training (training vs. no training) as a between-subjects factor
revealed no interaction effect on the number of reported cues, F (1, 33) = .25, p = .62, partial
eta2 = .01. There was no training effect on the number of reported cues, F (1, 33) = .21, p = .65,
partial eta2 = .01, and the number of cues reported did not differ from pre- to post-training, F (1,
33) = 1.03, p = .32, partial eta2 = .03.
Threat-related story building.
When responding to the question “What makes you believe that these elements are a
threat,” Soldiers tended to provide general explanations, often restating the cues provided in
response to the second question (“Which element(s) in the scene is the biggest threat”), or
matching the cue to an associated threat with no additional explanation. Researchers classified
responses into three categories: 1) general cue or no explanation, 2) simple cue/threat match
with no elaboration, 3) simple cue/threat match with elaboration. For example, Soldiers watched
a video that depicted military personnel walking down a street (see Figure 6). The street
21
contained numerous vehicles, trees, power lines, and buildings. Three participants restated the
cues named in in response to the previous question:
• “Everything on the street,”
• “They are hostile,” and
• “They are just sitting around.”
These Soldiers did not explain why they thought specific cues were threats, instead they
provided details about the cues or provided general non-specific answers cues (e.g., everything is
a threat, or people are hostile). This type of explanation is typical of novices who often struggle
to explain the relevance of the cues. The majority of responses to this video were simple
cue/threat matches with no elaboration. This indicated that Soldiers were able to identify the
threats associated to relevant cues, but not extend this thinking to make predictions or strategize.
Simple cue/threat matches are characteristic of advanced beginners, who are able to make
relevant distinctions between a cue and an outcome as they build their scripts of situation (e.g.,
Dreyfus & Dreyfus, 1986). Typical answers included:
• “Any of the [cars] could be VBIEDs,”
• “IED can be hidden in the engine compartment of the blue truck and the small wall
can provide cover for the IED,” and
• “Snipers can be in the tower and bombs could be in vehicles, on people, or on the side
of the road.”
Figure 6. Snapshot of the video that corresponds to explaining why the elements present a
threat.
22
Some Soldiers demonstrated the ability to think beyond simple cue/threat matches and
elaborated on their answers by including enemy strategy, challenges, and story-building.
Soldiers demonstrated thinking characteristic of competent skill levels (Dreyfus & Dreyfus,
1986). Examples included:
• “Open road implies a location for VBIEDs and IEDs, concealment hides enemy
combatants setting up an ambush,”
• “There are so many cars that [this situation] could easily be a set up for a complex
attack,” and
• “They stopped there for a reason, maybe that’s what the enemy wants.”
These responses indicated that Soldiers were hypothesizing about the enemy activities, in
addition to matching the cue to a threat.
Despite qualitative evidence that Soldiers responded in ways that represent differing
levels of skill development, training had no effect on Soldier explanations of why the elements
were threats, F (1, 32) = .84, p = .36, partial eta2 = .03. There was no training by pre/post
interaction, F (1, 32) = .14, p = .77, partial eta2 < .01, nor was there a main pre/post effect, F (1,
32) = 3.10, p = .09, partial eta2 = .09.
Threat detection challenges.
Soldiers answered the question, “What are the challenges to threat detection in this scene
(or what makes it difficult)?” with varying levels of detail. Previous research showed that
Soldiers with less experience had more difficulty discussing challenges to threat detection
(Zimmerman et al., 2012). Given the experience level represented in this sample, we expected
Soldiers to have difficulty discussing the challenges to threat detection. Researchers categorized
responses into three categories: (a) decontextualized challenge, (b) situational challenge, but
general or procedural, (c) creating a story around the challenge. Overall, Soldiers’ provided
general or procedural challenges, either in the context of the video situation or in general. The
following response examples correspond to a video clip showing a Soldier asking a local civilian
to open the hood of his car, followed by the Soldier looking in the car’s engine compartment
(Figure 7).
23
Figure 7. Snapshot of the video that corresponds to responses about challenges to threat
detection.
Responses categorized as decontextualized challenge reflected procedural or general
challenges that could apply to all scenarios. These responses reflect novice performance that is
“context-free” and objectively factual, consistent with Dreyfus and Dreyfus’s (1986) model of
skill acquisition. These responses included:
• “Possible suspect,”
• “The terrain, and not knowing what to look for,” and
• “Undetectable.”
Soldiers often provided descriptions of challenges specific to the situation but these
challenges tended to be procedural in nature. These responses were specific to the situation and
indicated the Soldiers were able to identify relevant challenges (e.g., safely inspecting the car,
dividing attention between important cues, and identifying IEDs in the vehicle). As experience
increases, Soldiers should begin to differentiate between general, context-free threats, and those
that are specific to the immediate situation. Responses in this category included:
• “The Soldier cannot safely check the car himself,”
• “Keeping an eye both on the civilian and car,” and
• “The vast amount of places in the vehicle that an IED could be placed.”
The final category included responses that included stories about challenges that went
beyond the typical situational script. In one example, the Soldier said, “can’t really see the
24
windows in the building next to [the Soldier], the car can have hidden wires, the [local civilian]
has not been searched, which could be hard to get the man to do.” In this example, the Soldier
identified two challenges that fit into the previous category: 1) being unable to see in the
windows and the car having hidden wires and 2) the possibility that the man was not searched.
Projecting beyond these challenges, the Soldier also considered future difficulties associated with
gaining consent to search the man. This Soldier reasoned beyond the man not being searched (a
procedural concern) and discussed how interacting with the civilian and the search could be
problematic. Although the Soldier did not describe why this was a challenge, the comment
reflects thinking beyond the immediate challenge and incorporates contextual elements.
An example from another video clip included, “the smoke coming from the bushes takes
[Soldiers’] attention away from the car and the trash pile.” Here, the Soldier demonstrated that
he was considering enemy strategy (e.g., the smoke diverts Soldier attention from the real threat).
This response was specific to the immediate situation and accounted for relevant contextual
information that could increase the challenges to threat detection.
Although Soldiers’ responses reflected different levels of expertise, repeated measures
ANOVA with pre/post (pre- vs. post-test) as a within-subject factor and training (training vs. no
training) as a between-subjects factor revealed no interaction effect on descriptions of challenges
F (1, 32) = .10, p = .75, partial eta2 < .01. Training did not affect Soldier ability to create a story
about the challenge, F (1, 32) = 2.80, p = .10, partial eta2 = .08 and there was no change from
pre- to post-training, F (1, 32) = 1.33, p = .26, partial eta2 = .04.
Decision-Making Exercises
Soldiers read two decision-making exercises (DMX) pre-test and two DMXs post-test.
They answered four questions after reading each scenario (Appendix B). From these answers, we
conducted qualitative and quantitative analyses. Below we present the results from each
question.
Threat descriptions and implications.
After reading the DMX scenario, Soldiers answered a question designed to uncover the
threats, cues, and interpreted meanings of these threats. The first question asked Soldiers to
“Describe any potential threats present in this scene and explain the meaning and implications of
these threats. You have two minutes to write a response.” Researchers coded responses
according to a data category scheme developed by Zimmerman et al. (2012). In that previous
analysis, researchers classified interview data into seven categories: types of threats, threat cues,
strategies for threat detection, threat detection tasks, threat detection skills, challenges in threat
detection, and solutions for threat detection preparation/training. Zimmerman et al. found that
Soldiers differed in their ability to produce data for each category based on their experience
levels. Soldiers with a greater number of deployments more often discussed strategies for threat
detection, threat detection skills, challenges in threat detection, and solutions, compared to
Soldiers who had never deployed. All Soldiers, regardless of deployment experience, discussed
types of threats, threat cues, and threat detection tasks. Using this coding scheme, we
25
determined if training changed Soldier’s response characteristics to match more closely the
responses of experienced Soldiers in the previous research.
In the pre-test, no training condition, Soldiers identified 37 types of threats and 57 threat
cues, 3 strategies for threat detection, 2 tasks, and 2 challenges across all four scenarios. They
did not identify any skills, or solutions. In the pre-test, training condition, Soldiers identified 36
types of threats, 88 threat cues, 2 strategies, and 7 challenges across all scenarios. They did not
identify any skills, tasks, or solutions. Threat types included IEDs, ambush, Taliban, suicide
bombers, and weapons. Soldiers often reported threat types in combination with threat cues.
Common threat cues included people, parked cars, and men acting strange or running towards
Soldiers. For example:
• “There are many people out on this road moving rocks (cue) which can hide IEDs
(threat). Many of these people (cue) don’t look like a threat but [it is] possible, could
be another ambush (threat) waiting for the right [moment] to set off IEDs. The
mountains (cue) in front could possibly have snipers (threat) as well.”
• “If [the area is] very empty (cue), an IED (threat) may be placed and it is in the
middle of the city [which is] probably usually busy (cue). Car (cue) parked there
could be a vehicle borne IED (threat) and the men in the video could be Taliban
(threat) and have IEDs strapped on to them (threat).”
The strategies Soldiers reported included:
• “Those could be insurgents planting IEDs into the road, who are using the
Commander's Emergency Response Program (CERP) as an excuse to be “working.”
I would not believe it is legitimate work until I see records of it.”
• “…I would stop and investigate and also see if there was any construction work to be
done at this time.”
Challenges Soldiers reported included not knowing a suspicious person’s intentions,
protecting innocent bystanders, and not knowing if other Soldiers are aware of the threat.
In the post-test, no training condition, Soldiers identified 23 types of threats, 48 threat
cues, and 5 challenges across all scenarios. They did not identify any strategies, skills, tasks, or
solutions. In the post-test, training condition, Soldiers identified 37 types of threats, 57 threat
cues across all scenarios. They did not identify any strategies, tasks, skills, challenges or
solutions. Common threat types included IEDs, suicide bomber, ambush, snipers. Consistent
with the findings in the pre-test condition, Soldiers often reported threat cues paired with threat
types. Common threat cues included cars, buildings, running man, wheelbarrows, and disturbed
dirt.
Based on previous research (Zimmerman et al., 2012), we expected that training might
increase the rate at which Soldiers described strategies, challenges, skills, and solutions. While
Soldiers in the training group identified the same number of threats post-test, they provided
fewer threat cues and did not describe any strategies or challenges. Soldiers in the no training
group identified fewer threat types compared to the training group and compared to the training
26
group post-test. Fatigue may have affected performance during the post-test. The lack of
changes in the training group may indicate that the training was too subtle have an effect on
open-ended question responding.
Explanation of Threat Factors.
Soldiers responded to the question, “List three to five factors that make this situation
dangerous and provide a brief explanation of why each is dangerous.” The majority of Soldiers
(M = 17) listed three factors per clip, thus we evaluated the first three factors reported and
evaluated differences across conditions. No Soldiers listed five factors, and an average of less
than one (0.75) Soldier listed four factors per clip. On average, five Soldiers listed one factor per
clip and six Soldiers listed two factors per clip. Soldiers tended to identify threat cues in the
photographs that accompanied each DMX rather than threats described in the written scenario.
While many of the cues were the same, very few Soldiers related the cues directly to the scenario
content. Of the 134 responses to this question, only eight responses (6%) directly related the
cues to the situation presented in the scenario. No differences emerged across pre- and post-test
conditions, and we could not draw any reliable conclusions given the small sample size.
Not all Soldiers provided explanations about why each factor was dangerous. For
example, in the Road Construction scenario (Appendix B), the picture shows men working on a
rocky mountainside moving rocks in wheelbarrows across a roadway. The written scenario
depicts a situation where security patrols have increased because of several small-arms attacks
and attempted ambushes. One afternoon, the unit discovers local civilians working on the road,
but they have no record of scheduled work in the area. Examples of response identifying cues
but not providing explanations included:
• “Concealment, terrain, deception,”
• “Mound of dirt,” and
• “Large number of workers, large amount of rubble, unscheduled construction.”
Soldiers who provided explanations about why the cues they identified were dangerous
indicated:
• “They might be terrorists. Nice pile of rubble to put IEDs in. Good place for an
ambush. Might have weapons hidden in the carts.”
• “Digging on the side of the road can hide explosives. Work being done on a curve in
the road [makes it] harder to see buried explosives. High elevation on both sides
[makes this] easy ambush spot.”
• “Wheelbarrows could be hiding weapons for an ambush. Holes on the road side
[where they could have] planted IED’s. All military age males [are] fit to fight. The
terrain is good for sniper attacks.”
Repeated measures ANOVAs assessing the effects of training and pre/post tests on
whether Soldiers provided explanations revealed no interactions, F (1, 29) = .00, p = .95, partial
eta2 < .01. Training significantly influenced whether Soldiers gave an explanation about why the
cues indicated danger, F (1, 29) = 5.41, p = .03, partial eta2 = .16. We expected that Soldiers
27
would provide more detailed explanations after training; however, untrained participants (M =
.80, SD = .02) provided explanations more often than did trained participants (M = .56, SD = .02)
across pre/post tests. We do not have an explanation for this finding. The participants were
randomly assigned to the training conditions and the effect was across the pre/post condition,
indicating that training did not necessarily influence the results. There were no significant
effects for the pre/post condition and the training by pre/post interaction was not significant. The
observed power was low (equal to .05), which could have influenced the ability to detect a
significant effect.
Threat Detection and Mitigation Strategies.
To understand how Soldiers think about threat response strategies, we asked them to
“Name the one potential threat you are most concerned about. What strategies would you use to
detect and mitigate that threat?” Researchers identified the first named threat and reviewed the
data to determine if Soldiers reported different threats depending on training. There were no
consistent differences in the threats discussed across the training and the no training groups. The
majority of the threats named were the threat cues, such as “the man running,” “the car parked
too close to the checkpoint,” and “men with wheelbarrows.” In 35% of the responses (n = 33),
Soldiers named the possible threat associated with the threat cues, such as IEDs, ambushes, and
snipers.
We identified whether or not Soldiers identified strategies to detect or mitigate the threat.
For example, the Afghan National Police (ANP) Incident DMX scenario (Appendix B) includes
a picture showing an old woman, a truck with two Afghan nationals, and a building located
behind the truck. The scenario describes a situation where the Soldier is manning a post with a
member of the ANP who is acting erratically and threatens to call the Taliban. In this scenario,
Soldiers’ responses where they only identified the threat included:
• “Old woman,”
• “The one threat that I am most worried about is him waving his gun around, not
caring about his muzzle awareness. He could possibly shoot me or anyone around the
post and start a shooting spree and cause other members of the ANP to start shooting
as well,” and
• “Ambush from houses.”
Examples of responses in which Soldiers’ discussed strategies include:
• “I'd clear everyone out of there and take control of that machine gun,”
• “That old lady being a possible suicide bomber would be my top priority. I would
have a female Soldier come and search her to make sure she’s clear.” and
• “Most concerned about the angry ANP. [I would] ask to have him replaced, removed
from the area. Keep a close watch on him and observe for any continued erratic
behavior.”
We expected that trained Soldiers should be better able to strategize about threat
detection or mitigation, given that Soldiers with more experience are more apt to provide threat
28
detection strategies when describing threat situations (Zimmerman et al., 2012). Repeated
measures ANOVA with pre/post (pre- vs. post-test) as a within-subject factor and training
(training vs. no training) as a between-subjects factor revealed no interaction effect on reporting
of strategies, F (1, 29) = .29, p = .59, partial eta2 = .01. Observed power was low (equal to .08),
which makes it difficult to detect an effect. Training had a marginally significant effect on
whether Soldiers offered a strategy to detect or mitigate the threats, F (1, 29) = 3.59, p = .07,
partial eta2 = .11. Trained Soldiers (M = .68, SD = .02) provided strategies to mitigate the threat
more often than did untrained Soldiers (M = .44, SD = .02). There was a significant effect from
pre-test to post-test on whether Soldiers offered a strategy, F(1, 29) = 11.78, p < .01, partial eta2
= .29, though it was not in the expected direction. Soldiers offered strategies less often during
the post-test (M = .43, SD = .01) than they did during the pre-test (M = .69, SD = .01). This
finding could indicate fatigue from pre- to post-test, though the main effect of training indicates
that the training group provided more strategies across test sessions. Greater power might clarify
if an interaction exists and if it is driven by the training condition post-test.
Threat Detection Challenges.
To identify Soldiers’ understanding of challenges in threat situations, Soldiers responded
to the request to “List the top three challenges to detecting threats in this situation and state why
they are challenges.” Researchers identified the first three named challenges to determine
whether Soldiers identified different challenges after training. Soldiers appeared to identify
similar challenges both pre- and post-training. Table 7 depicts the challenges Soldiers’ indicated
for the Road Construction DMX scenario (Appendix B). Across all conditions, Soldiers reported
that determining the work legitimacy was an important challenge. They also discussed the
mountain/terrain/location in which the scenario occurred and the wheelbarrows or identification
of the wheelbarrow contents. Other challenges appear in each condition, such as losing hearts
and minds of the local population and the absence of any apparent weapons. The training group
seemed to list a greater variety of challenges post-test in this scenario; however, not all the
scenarios reflected this pattern. For instance, in the Convoy Ops DMX scenario, Soldiers in the
no training group reported a greater variety of challenges, and reported challenges more often
than did the training group, both pre- and post-test.
29
Table 7
The Number of Challenges Identified by Training Condition in the Road Construction
Scenario
Challenge Type
Pre-test
No Training
(n* = 5)
Pre-test
Training
(n = 10)
Post-test
No Training
(n = 8)
Post-test
Training
(n = 15)
Work legitimacy
1
3
1
2
Wheelbarrow/identification of
contents
1 1 2 1
Unknown enemy
1
1
Mountains/terrain/location
1
1
1
4
Men/workers
1
3
Rubble
1
Line of sight
1
Hiding places
1
Losing hearts and minds
1
Construction
1
Dirt
1
Potential spotters
1
Can’t see in holes
1
Tools
1
No weapons apparent
1
Number of men
1
Unidentified personnel
1
*Note: n indicates the number of challenges identified.
Further Analysis
The primary objective of this research was to determine if the TDST had an effect on
identified skills such as target identification, detecting relevant changes, and causal reasoning.
To determine if prior training influenced performance during this research, we asked Solders to
describe any training they received that improved their ability to detect threats (Appendix C,
question 12). From those responses, 16 participants responded that they had received training
relevant to threat detection, while 32 responded that they had not received such training.
Because this was a small and disproportionate sample, inferential statistical analysis would be
suspect. However, a descriptive analysis did yield suggestive findings concerning whether
previous training influenced performance.
Soldiers who reported receiving threat detection training in the Army, prior to the
experiment, responded similarly on the dependent variables to Soldiers who had not received
previous training. Army training on threat detection can take several days to weeks to complete.
Moreover, some experience can only be gained through the conduct of actual operations.
Soldiers in our experiment spent only between 1.5 to 2 hours completing the experimental tasks.
For half the participants, this included the TDST. Analysis of Soldier responses to the question
about previous training indicated the difficult nature of training threat detection skills. Though
30
the skills are paramount to obtain, especially for the operational environments Soldiers find
themselves in training such skills and perhaps, even maintaining them requires consistent
application and practices.
Discussion
The quantitative findings across experimental tasks indicate that training was no more
effective at improving performance than was the practice that resulted from completing the pre-
test and post-test. Practice effects appeared to be consistent and robust. In each task, Soldiers’
performance improved over time, and the effect sizes were relatively large in some cases (e.g.,
partial eta2 > .30). Thus, to the extent that the TDST promotes practicing the skills required for
detecting threats, it improves performance. However, to be more certain of this potential
improvement, future research would have to tease apart the potential impact of different
experimental designs as well as the influence of different measures of performance.
A longitudinal research program examining the longer-term effects of completing the
trainer could be worth pursuing. In the current research, the results showed an improvement
from pre- to post-test on almost all dependent variables of interest. This indicates that we
accurately identified the skills requisite for threat detection (Vowels, 2010; Zimmerman, et al.,
2011; Zimmerman, et al., in preparation; Zimmerman, et al., 2012). If testing can demonstrate
an improvement in the skills necessary for threat detection in two hours or less using an acute,
cognitive training program, it is worth investigating the effects that might occur over several
weeks to months of training. A potential methodology that has demonstrated recent success is
retrieval-based or test-enhanced learning. Results from previous research have consistently
shown the enhancement of short-term retention and long-term retention of material (Roediger &
Butler, 2011; Roediger, Agarwal, McDaniel, & McDermott, 2011). Though typically examined
using grade school populations, it is possible to expand the procedure to a military population.
Such testing could provide a legitimate examination of training to enhance retrieval-based
memory and recognitional decision-making in an applied setting (U.S. military). See also
Kornell and Rhodes (2013) concerning the influence of feedback in test-enhanced learning.
In the present research, qualitative analysis focused on contextual differences in Soldiers
responses. Practice effects were noted during some of the analyses as the details provided during
the post-test tasks tended to decrease relative to pre-test tasks. In addition, analysis occasionally
revealed differences in the training groups, but in the opposite direction than anticipated.
Though assigned randomly to conditions, Soldiers in the no-training group, relative to the
training group, tended to provide explanations with more details and offered a greater number of
strategies for overcoming threats. This occurred across the pre- and post-tests. Overall, any
differences in answer content were minor across the conditions. This could be due to the nature
of the tasks or to fatigue. Requiring Soldiers to write answers on multiple trials, and across
similar tasks, might have reduced their motivation to respond at length.
Limitations of this experiment were the predicted small effect of training and a small
sample size. A larger sample size would increase the likelihood of observing small effects.
However, it is important to have realistic expectations about the magnitude of the impact of the
trainer on performance. Given the present results, we have the benefit of not overestimating how
31
impactful the trainer might be. Another limitation in this experiment is that the experimental test
tasks were similar to the tasks used in the trainer, thus, it is not surprising that practice effects
seemed to “wash out” any effects of training. Perhaps a third comparison group of Soldiers who
had less opportunity for practice would have benefited this research. For practical purposes,
however, this third group was not feasible in the present experiment. Another solution for future
research is to create tasks that do not resemble the tasks in the trainer, but provide the same
measures required for the hypotheses.
Future research should tease apart the effects of practice and the effects of training via the
TDST on threat detection performance. For example, protocols could incorporate differing
amounts of practice or multiple training sessions over time. Future experiments should test
threat detection in real-world scenarios. Placing participants in live-action scenarios might
mitigate practice as a confound because the critical tasks will likely be significantly different
from those experienced in the trainer. This would remove the writing requirement over multiple
trials and may reduce the effects of fatigue. Finally, live scenarios provide the opportunity to
examine the effect of the trainer in a more ecologically valid context. Future research might also
benefit from testing Soldiers with different levels of training and experience. The present
experiment tested Soldiers with very little training and experience. It may be interesting to
explore whether more or less trained Soldiers benefit to differing degrees from completing the
TDST.
The Army continues to operate in asymmetric warfare environments, which are marked
by high uncertainty, time pressure, and high-stakes decisions. Hybrid strategies and tactics,
involving a mix of standard and irregular warfare methods and technologies, are assumed to
influence operations for the foreseeable future (TRADOC Pamphlet 525-3-0, The U.S. Army
Capstone Concept). Given that, even organizations that largely field technologies for the
irregular battlefield, also understand the best weapon to defeat asymmetric threats is a well
educated and well trained Army Soldier (Bosker, 2012). As such, the TDST acts as a
supplemental resource that directly provides needed training and practice for the Army’s most
crucial asymmetric deterrent, the human Soldier.
32
33
References
Bosker, A. J. (2012, December). JIEDDO focusing on current fight, preparing for future threats.
Retrieved from https://www.jieddo.mil/news_story.aspx?ID=1499
Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuitive
expertise in the era of the computer. New York: The Free Press.
Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical review. In
M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement.
Mahwah, NJ: LEA.
Johnston, J., Leibrecht, B. C., Topolski, R., Vowels, C. L., & Singer, J. T. (in preparation). Unit-
focused escalation of force training kit. (ARI Research Product). Fort Belvoir, VA: U.S.
Army Research Institute for the Behavioral and Social Sciences.
Klein, G. (2004). The power of intuition. New York: A currency Book/Doubleday.
Kornell, N., & Rhodes, M.G. (2013). Feedback reduces the metacognitive benefit of tests.
Journal of Experimental Psychology: Applied, 19, 1-13.
Mueller, S. T. (2009). A Bayesian recognitional decision model. Journal of Cognitive
Engineering and Decision Making, 3, 111-130.
Rensink, R. A., O’Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for
attention to perceive changes in scenes. Psychological Science, 8 (5), 368–373.
Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced
learning in the classroom: Long-term improvements from quizzing. Journal of
Experimental Psychology: Applied, 17, 382-395.
Roediger, H. L., & Butler, A. C., (2011). The critical role of retrieval practice in long-term
retention. Trends in Cognitive Science,15, 20-27.
Ross, K. G., Phillips, J. K., Klein, G., & Cohn, J. (2005). Creating expertise: A framework to
guide technology-based training (Final Report prepared under Contract M67854-04-C-
8035 for Marine Corps System Command/Program Manager for Training Systems under
the sponsorship of the Office of Naval Research MARCORSYSCOM/PMTRASYS,
Orlando, FL). Fairborn, OH: Klein Associates Inc.
Scholl, B. J. (2000). Attenuated change blindness for exogenously attended items in a flicker
paradigm. Visual Cognition, 7, 377-396.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics (4th ed.). Needham
Heights, MA: Allyn & Bacon.
34
U.S. Department of Army (2012, December). The U.S. Army Capstone Concept. Washington
DC: Author.
Vowels, C. L. (2010). Asymmetric attention: Visualizing the uncertain threat. (ARI Research
Report 1916). Arlington, VA: U.S. Army Research Institute for the Behavioral and
Social Sciences.
Zimmerman, L. A., Mueller, S. T., Daniels, J., & Vowels, C. L. (2012). Developing training
exemplars for the requisite components of visual threat detection (ARI Technical Report
1322). Fort Belvoir, VA: U.S. Army Research Institute for the Behavioral and Social
Sciences.
Zimmerman, L. A., Mueller, S. T., Grover, J., & Vowels, C. L. (in preparation). Determining the
requisite components of visual threat detection to improve operational performance (ARI
Technical Report). Fort Belvoir, VA: U.S. Army Research Institute for the Behavioral
and Social Sciences.
Zimmerman, L. A., Mueller, S. T., Marcon, J. L., Daniels, J. D., & Vowels, C. L. (2011).
Improving soldier threat detection skills in the operational environment.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC).
Orlando, FL.
A-1
Appendix A
Situation Assessment Questions
Following the drag-and-drop exercise, participants will answer the following questions:
1. Describe what is happening in the scene.
2. Which element(s) in the scene is the biggest threat?
3. What makes you believe that these elements are a threat?
4. What are the challenges to threat detection in this scene (or what makes threat
detection difficult)?
A-2
B-1
Appendix B
Decision-Making Exercises (DMX)
After reading each of the DMXs, participants answered the following questions:
1. Describe any potential threats present in this scene and explain the meaning and
implications of these threats. You have two minutes to write a response.
2. List three to five factors that make this situation dangerous and provide a brief
explanation of why each is dangerous.
3. Name the one potential threat you are most concerned about. What strategies would
you use to detect and mitigate that threat?
4. List the top three challenges to detecting threats in this situation and state why they
are challenges.
These questions followed the last sentence in each of the below DMXs:
B-2
DMX Title: Road Construction
You are Bravo Co 1/5 CAV. Your company mission is to conduct COIN operations, with the
frequent use of CERP funds to build community infrastructure and win the hearts and minds of
the local populace. When you first arrived at your AOR, there were very few incidents and the
community generally supported a U.S. military presence. In recent months, sentiment has
changed. There have been multiple IED incidents in the past two weeks along route Grover, the
major supply route for U.S. forces in your sector, with several small-arms attacks and attempted
ambushes on supply convoys after IEDs pinned them in place. This has resulted in increased
security patrols along with a big push to engage with community members, in part to win back
their approval, and in part to determine why the U.S. is losing support.
For this reason, your unit goes out on frequent vehicle and foot patrols with the goal of
interacting with the village locals. On this afternoon, your convoy is driving into the village so
the platoon leader can do a KLE with the village elder. As your unit conducted movement along
route Grover, you came upon several individuals doing work along the road. You do not have
any record of any scheduled work in this area, but with the recent upswing in new CERP efforts,
this might be legitimate work. You are stopped on the road while the squad leader attempts to
determine if any work is scheduled for this area.
This photo shows what you are viewing at the point where your unit has stopped.
B-3
Title: Iraq Elections
You have temporarily been assigned to the 2nd Recon Division, U.S. Marine Corps for upcoming
security at various checkpoints throughout the city of Baghdad, Iraq. The government of Iraq
designated these checkpoints as election booths. The military put added security measures into
effect because terrorists have publicly vowed to increase violent activities as Election Day
approaches. The threat of sectarian violence also increases as one or both of the parties on the
ballot sway favor.
Your ROE are to stop and screen all traffic between election hours up through the date of
election. The authorization of deadly force is approved when peaceful options have been
exhausted and or for immediate protection of the security forces. You are allowed to apprehend
any or all personnel that are acting suspiciously, appear threatening, or are perpetrating violence
against other citizens of Iraq.
It is currently late morning and your unit is doing initial security checks of each checkpoint in
your AO. You are approaching the third checkpoint of the day. The other two checkpoints were
secure and operations to set up election booths were progressing without incident. The personnel
operating the checkpoints were known members of Iraqi security forces trained by U.S. Soldiers.
As your unit approaches the third checkpoint, there appears to be no evidence of the election
booth supplies and, unlike the first two checkpoints, no activities are underway to set up the
election booths. You have not yet identified the checkpoint personnel.
B-4
This photo shows your current view of the checkpoint as your unit approaches.
Title: ANP Incident
You are at a joint compound, manning a post with a member of the Afghan National Police
(ANP). You have worked with this the ANP member several times before. On this occasion, the
ANP member is acting erratically and arguing with everyone. He was repeatedly asking you to
exchange some U.S. dollars for rupees. You kept refusing and finally asked the ANP member
stop asking you. The ANP member became angry and began acting carelessly with his weapon,
pointing it in the sky and walking away from the post. You finally called the ANP member’s
sergeant to report the ANP member’s behavior. The ANP sergeant ordered the ANP member to
return to his post. The ANP member returned and began threatening to call the Taliban. You are
now at your post watching as the ANP member’s erratic actions continue.
B-5
Title: Convoy Operation
You are the gunner in your truck and the lowest-ranking member of the patrol. Your mounted
patrol takes a break at a location that you consider relatively safe. You are in the turret on watch
during the break. The other vehicle occupants are outside the vehicle. Your vehicle is
positioned best to cover a slight draw approaching your position.
A local national male of military age appears in the draw and rapidly approaches your position.
You begin using EOF procedures - yelling at him to stop, waving at him to stop, and then
pointing your weapon toward him. You are unsure where the rest of your patrol is while you
continue to monitor the man, though from your vantage point you know they are not in your field
of fire and they are nearby. The man continues to approach at a high rate of speed and closes to
within 20 meters. You cannot tell what his intentions are. You shout for your team leader, but
he does not hear you.
This photo shows what you are viewing at the point where your unit has stopped.
B-6
C-1
Appendix C
Demographic Questions
1. Time in service (years)
2. Current rank
3. Time in current rank (months)
4. Current MOS
5. Age
6. Have you ever deployed? Yes No
7. If yes, how many times have you deployed?
8. Location of most recent deployment (city or cities and country)
9. MOS while on your most recent deployment:
10. How often did you go outside the wire on your most recent deployment?
Never Less than once a month Once a month More than once a month
Once a week More than once a week Everyday
11. Describe some of your duties during your most recent deployment and indicate if you
conducted dismounted patrols (present patrols and/or clearing operations) and route
clearance:
12. Describe any training you have received that improved your ability to detect threats and
indicate approximate date of training month/year:
