Content uploaded by Spencer K. Lynn
Author content
All content in this area was uploaded by Spencer K. Lynn on Apr 28, 2021
Content may be subject to copyright.
Modeling Perceptual Judgement in Believable Agents:
A Signal Detection Approach
Spencer K. Lynn (slynn@cra.com)
Human Effectiveness Division, Charles River Analytics, Inc.
625 Mt. Auburn St., Cambridge, MA 02138
Taylor Curley (taylor.curley@gatech.edu)
School of Psychology, Georgia Institute of Technology
J.S. Coon Bldg., 654 Cherry Street, Atlanta, Georgia 30332
Peter Weyhrauch (pweyhrauch@cra.com)
Human Effectiveness Division, Charles River Analytics, Inc.
625 Mt. Auburn St., Cambridge, MA 02138
Abstract
Computer modeling of Warfighter performance is an increasingly
important element for the Department of Defense in developing and
evaluating tactics, techniques, and procedures (TTPs) as well as
military acquisition strategies. To do this modeling, human
performance researchers are working to integrate models of complex
cognitive and physical systems, as well as the processes that
moderate them, in a unified framework. To better model visual
detection and identification processes in realistic performance
situations, we have constructed an agent-based model of visual
perception based on signal detection theory that can be moderated by
exigent processes, such as stress and fatigue.
Keywords: Perceptual Judgement, Signal Detection, Believable
Agents, Solder Simulation.
Introduction
A Warfighter in a combat environment is expected to
continuously search his or her visual field to maintain
situational awareness. Misidentification of relevant stimuli,
such as failure to detect an enemy combatant or incorrect
identification of a friend as an enemy, has costly results for
the Warfighter and associated team members. Furthermore,
Warfighters are often trained to sustain their attention over
long periods of time, but by-products of situational
demands, such as operational stress and fatigue, can
significantly impact performance related to visual attention
processes (Staal, 2004; Janelle & Hatfield, 2008). Thus, it is
of paramount importance to understand perceptual judgment
processes in individual Warfighters when confronted with
moderators of operational performance.
Methods
The DREEMS Project
To better understand the dynamics of Warfighter
performance, Charles River Analytics has introduced the
Dynamic Representation for Evaluating the Effect of
Moderators and Stress on Performance (DREEMS) project.
Using a modeling language called Hap (Loyall et al., 1991),
DREEMS models individual Warfighter performance
through the use of situational awareness modeling
(SAMPLE; Zacharias et al., 1996). The architecture models
agent behavior as the cumulative result of an information
processing module feeding into a situation assessor, which
then guides an agent’s decision-making via behavior trees.
Moderating variables can exert influence over these
modules at any point in the architecture.
Signal Detection Approach
Using a signal detection theory (SDT) approach to
cognition provides a robust method of exploring a wide
variety of behaviors, particularly for those in which
individuals encounter perceptual uncertainty and behavioral
risk (Lynn & Barrett, 2014). Importantly, the parameters of
SDT mechanisms have been shown to be sensitive to
moderating variables, such as emotion on field of view
(Schmitz et al., 2009). Here, we follow previous research
illustrating how perceptual decision parameters in a
perceptual SDT model are moderated by differences in
individual state (e.g., Lynn et al., 2012) and extend this
approach to modeling visual threat detection in a battlefield
environment.
Model Specifications
We model perception as a set of underlying receptors that
correspond to different regions in the Field of Regard
(FOR), or visual field, a span of 135°. These receptors
respond to signals from the visual environment. When a
signal is presented in the visual field, the ability of the agent
to discern the signal from background noise is a function of
the location of the signal in the FOR: Perceptual sensitivity
to discriminate signal from noise decreases toward the
periphery of the FOR (Fig. 1).
Figure 1. An agent’s FOR (Field of Regard). Past the FOV
(Field of Vision; 75°-105°), receptor sensitivity declines
Lynn, S.K., T. Curley, and P. Weyhrauch (2018). Modeling Perceptual Judgement in Believable Agents: A Signal Detection Approach.
Presented at the joint annual meetings of the Society for Mathematical Psychology and the International Conference on Cognitive Modelling,
21-24 July, Madison, Wisconsin, USA.
with increasing distance from the fovea.
After a signal is detected in the visual field, the agent
makes a decision about the identity of the signal, such as if
it is a threat or not. The criterion defining the perceptual
judgement between noise and objects-of-interest in the
environment (e.g. threat vs. non-threat) is given by SDT’s
utility function: u(x) = αhP[CD] + αmP[MD] + (1-α)aP[FA]
+ (1-α) jP[CR], where P[…] is the probability of each of the
four possible outcomes, correct detection (CD), missed
detection (MD), false alarm (FA), or correct rejection (CR);
α is the base rate probability of encountering a signal; 1−α
is the probability of encountering noise; and h, m, a, and j
represent the payoffs (benefits or costs) for hits, misses,
false alarms, and correct rejections, respectively. Thus, the
expected utility of adopting a decision threshold at a
particular signal value, x, is defined by the probabilities of
four outcomes, the base rate, and the payoffs. The optimal
decision threshold is found at the point of the highest utility
(Fig. 2).
Figure 2. A signals-approach to threat detection. Blue and
green Gaussians represent probability densities defining
what threats and non-threats look like, respectively, with
notional mean appearance of each category depicted by the
combatant and journalist. When perceptual uncertainty
exists (depicted as overlap of the Gaussians), mistakes
cannot be eliminated, but exposure to them can be
optimized. The maximum of a utility function (black curve)
locates this optimum: the decision criterion that will
maximize net benefits over a series of decisions.
Integrating Moderating Variables
Under our model, how effective an agent is at perceptual
decision making is dependent on the agent’s situation
awareness, defined as the accuracy of its signal transduction
at the receptors and its estimates of the CD, FA, etc.
probabilities; the base rate; and the payoffs. In order to
simulate the influence of operational variables on perceptual
decision making in Warfighters, we applied a function that
affects the accuracy of these parameters as function of
moderating variables, such as fatigue.
For example, distortion of the receptor transduction (or
another parameter, such as the base rate) can be modeled as
a simple sum of the incoming signal value, x, plus the
influence of behavioral moderators: x* = x+Mu(ba), where
Mu(ba) is the square of either a single or a collection of
behavioral moderators. The combined influence of several
moderators can be determined by a number different
methods, including only using the value of the moderator
with the most influence. For this project, we have employed
a logarithmic combination method (e.g. Moors, 2009): ba =
0.1
∗
log2 (∑sgn(bi)
∗
210*|bi), where bi is an individual
moderator in a set of behavioral moderators acting upon an
agent. Any moderator variable or collection of variables,
then, can affect the perceived value of a signal or effect how
that signal is judged by influencing the agent’s estimate of
the optimal threshold location – a pattern consistent with
cognitive affective research (Lynn et al., 2012).
Acknowledgments
This material is based upon work supported by the US
Army Command Center, Aberdeen Proving Ground, Natick
Contracting Division ACC-APG-NCD under Contract No.
W911QY-17-C-0009. Any opinions, findings and
conclusions or recommendations expressed in this material
are those of the author(s) and do not necessarily reflect the
views of the US Army Command Center, Aberdeen Proving
Ground, Natick Contracting Division ACC-APG-NCD.
References
Bates, J., et al. (1994). The role of emotion in believable
agents. Communications of the ACM, 37, 122–125.
Janelle, C. M., & Hatfield, B. D. (2008). Visual attention
and brain processes that underlie expert performance:
Implications for sport and military psychology. Military
Psychology, 20, S39.
Loyall, A. B., & Bates, J. (1991). Hap: A reactive, adaptive
architecture for agents. Carnegie Mellon University
Department of Computer Science Technical Report
CMU-CS-91-147.
Lynn, S. K., & Barrett, L. F. (2014). “Utilizing” signal
detection theory. Psychological Science, 25, 1663–1673.
Lynn, S. K., Zhang, X., & Barrett, L. F. (2012). Affective
state influences perception by affecting decision
parameters underlying bias and sensitivity. Emotion, 12,
726.
Moors, A. (2009). Theories of emotion causation: A review.
Cognition and Emotion, 23, 625–662.
Schmitz, T. W., De Rosa, E., & Anderson, A. K. (2009).
Opposing influences of affective state valence on visual
cortical encoding. Journal of Neuroscience, 29, 7199-
7207.
Staal, M. A. (2004). Stress, cognition, and human
performance: A literature review and conceptual
framework. NASA Technical Report NASA/TM-2004-
212824.
Zacharias, G. L., Miao, A. X., Illgen, C., Yara, J. M., &
Siouris, G. M. (1996). SAMPLE: Situation awareness
model for pilot in-the-loop evaluation. In Proceedings of
the 1st annual conference on situation awareness in the
tactical air environment.
Modeling Perceptual Judgement in
Believable Agents:
A Signal Detection Approach
Spencer K. Lynn (slynn@cra.com)1
Taylor Curley (taylor.curley@gatech.edu)2
Peter Weyhrauch (pweyhrauch@cra.com)1
1Human Effectiveness Division, Charles River Analytics,
Inc., 625 Mt. Auburn St., Cambridge, MA 02138
2School of Psychology, Georgia Institute of Technology, J.S.
Coon Bldg., 654 Cherry Street, Atlanta, Georgia 30332
The DREEMS Project
• A Warfighter in acombat environment is expected to continuously
search his or her visual field to maintain situational awareness.
•Misidentification of relevant stimuli, such as failure to detect an
enemy combatant or incorrect identification of afriend as an enemy,
has costly results for the Warfighter and associated team members.
•By-products of situational demands, such as stress and fatigue, can
significantly impact operational performance via moderation of
perceptual processes (e.g., Staal 2004).
•It is of paramount importance to understand perceptual judgment
processes in individual Warfighters when confronted with moderators
of operational performance.
•To better model visual detection and identification processes in
realistic performance situations, we have constructed an agent-based
model of visual perception based on signal detection theory that can
be moderated by exigent processes, such as stress and fatigue.
Background
Internal and external factors affect perceptual judgements, and it is rare
for these moderators to have their effects in isolation.
In the current model, effective decision-making depends on the agent’s
situational awareness, which we define as the agent’s accuracy of signal
transduction at the sensory receptors and it’s estimates of the signal
parameters.For example:
•We simulate disrupted transduction as the simple sum of the incoming
signal value, x,and the combined influence of one or more behavioral
moderators, Mu(ba):x* =x+Mu(ba).
•To simulate the combined influence of moderators, we employ a
logarithmic combination method:ba= 0.1
∗
log2(∑sgn(bi)
∗
210*|bi|)
where biis asingle behavioral moderator (Moors 2009).
Integrating Moderators
We model perception as an array of receptors that correspond to
different regions in the Field of Regard (FOR;Figure 2). For asignal in
the FOR, an agent’s ability to correctly categorize the signal as athreat
or not is afunction of location in the FOR and the agent’s internal state.
The agent makes adecision about the identity of the signal (as a
threat or not).The criterion defining this perceptual judgement is given
by the signal detection theory (SDT) utility function (Figure 3).
Model Architecture
Charles River Analytics has introduced the Dynamic Representation
for Evaluating the Effect of Moderators on Stress (DREEMS) project
•DREEMS models Warfighter performance through the use of
situational awareness modeling (e.g., SAMPLE;Zacharias et al.
1996). See Figure 1.
•Agent performance is represented in behavior trees using alanguage
called Hap (Loyall &Bates 1991).
•DREEMS models behavior generation as the cumulative result of an
information processing module feeding into asituation module, which
then guide’s an agent’s behavior via goals and behaviors in Hap.
The DREEMS Project
This material is based upon work supported by the US Army Command Center,
Aberdeen Proving Ground, Natick Contracting Division ACC-APG-NCD under
Contract No. W911QY-17-C-0009.Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the US Army Command Center, Aberdeen Proving
Ground, Natick Contracting Division ACC-APG-NCD.
Acknowledgments
Figure 1. Elements of aSAMPLE-based agent.
Figure 2.The Field of Regard.Past the FOV (Field of Vision;75°-105°),
receptor sensitivity declines with increasing distance from the fovea.
Figure 3. The signals approach to threat detection.Blue and green
Gaussians represent probability densities defining what threats and
non-threats look like, respectively, with notional mean appearance of
each category depicted by the combatant and journalist.When
perceptual uncertainty exists (depicted as overlap of the Gaussians),
mistakes cannot be eliminated, but exposure to them can be optimized.
The maximum of autility function (black curve) locates this optimum:
the decision criterion that will maximize net benefits over aseries of
decisions (after Lynn &Barrett 2014).
Loyall, A. B., and Bates, J. 1991.Hap: A Reactive, Adaptive Architecture for Agents.
Tec hn ic a l Report CMU-CS-91-147.Department of Computer Science.
Carnegie Mellon University.
Lynn, S. K., Zhang, X., & Barrett, L. F. (2012). Affective state influences perception
by affecting decision parameters underlying bias and sensitivity.Emotion, 12,
726.
Lynn, S. K., & Barrett, L. F. (2014). “Utilizing” signal detection theory.Psychological
Science 25:1663-1673..
Moors, A. (2009). Theories of emotion causation: A review.Cognition and Emotion,
23,625–662.
Staal, M. A. (2004). Stress, cognition, and human performance: A literature review
and conceptual framework.NASA Tec h nic al Report NASA/TM-2004-212824
Zacharias, G. L., Miao, A. X., Illgen, C., Yar a, J. M., & Siouris, G. M. (1996,June).
SAMPLE:Situation awareness model for pilot in-the-loop evaluation.
In Proceedings of the 1st Annual Conference on Situation Awareness in the
Tactical Air Environment.
References
Modeling perceptual judgement with the SDT utility function provides
parameters through which physical or psychological factors can moderate
perception (e.g., Lynn et al.2012).
These parameters include the signal value (x-axis in Fig. 3), the
similarity of threats and non-threats (Gaussians in Fig. 3), the relative
base rate of threats to non-threats, and the benefits and costs of correct
vs incorrect signal categorization.In SDT, these parameters underlie
performance metrics of perceptual sensitivity and response bias.
In our model, moderators affect the accuracy of the agent’s estimates
of these parameters, which are building blocks of situation awareness.
Moderating Perception
•Incorporate additional findings from published research.
•Identify knowledge gaps re:moderation of Warfighter-relevant tasks.
•Run studies to fill gaps.
Future Directions
Lynn, S.K., T. Curley, and P. Weyhrauch (2018). Modeling Perceptual Judgement in Believable Agents: A Signal Detection Approach. Presented at the joint
annual meetings of the Society for Mathematical Psychology and the International Conference on Cognitive Modelling, 21-24 July, Madison, Wisconsin, USA.
