Conference PaperPDF Available

Modeling Perceptual Judgement in Believable Agents: A Signal Detection Approach

Authors:

Abstract

A Warfighter in a combat environment is expected to continuously search his or her visual field to maintain situational awareness. Misidentification of relevant stimuli, such as failure to detect an enemy combatant or incorrect identification of a friend as an enemy, has costly results for the Warfighter and associated team members. By-products of situational demands, such as stress and fatigue, can significantly impact operational performance via moderation of perceptual processes. It is of paramount importance to understand perceptual judgment processes in individual Warfighters when confronted with moderators of operational performance. To better model visual detection and identification processes in realistic performance situations, we have constructed an agent-based model of visual perception based on signal detection theory that can be moderated by exigent processes, such as stress and fatigue. Charles River Analytics has introduced the Dynamic Representation for Evaluating the Effect of Moderators on Stress (DREEMS) project. DREEMS models Warfighter performance through the use of situational awareness modeling. Agent performance is represented in behavior trees using a language called Hap (Loyall & Bates 1991). DREEMS models behavior generation as the cumulative result of an information processing module feeding into a situation module, which then guide’s an agent’s behavior via goals and behaviors in Hap.
Modeling Perceptual Judgement in Believable Agents:
A Signal Detection Approach
Spencer K. Lynn (slynn@cra.com)
Human Effectiveness Division, Charles River Analytics, Inc.
625 Mt. Auburn St., Cambridge, MA 02138
Taylor Curley (taylor.curley@gatech.edu)
School of Psychology, Georgia Institute of Technology
J.S. Coon Bldg., 654 Cherry Street, Atlanta, Georgia 30332
Peter Weyhrauch (pweyhrauch@cra.com)
Human Effectiveness Division, Charles River Analytics, Inc.
625 Mt. Auburn St., Cambridge, MA 02138
Abstract
Computer modeling of Warfighter performance is an increasingly
important element for the Department of Defense in developing and
evaluating tactics, techniques, and procedures (TTPs) as well as
military acquisition strategies. To do this modeling, human
performance researchers are working to integrate models of complex
cognitive and physical systems, as well as the processes that
moderate them, in a unified framework. To better model visual
detection and identification processes in realistic performance
situations, we have constructed an agent-based model of visual
perception based on signal detection theory that can be moderated by
exigent processes, such as stress and fatigue.
Keywords: Perceptual Judgement, Signal Detection, Believable
Agents, Solder Simulation.
Introduction
A Warfighter in a combat environment is expected to
continuously search his or her visual field to maintain
situational awareness. Misidentification of relevant stimuli,
such as failure to detect an enemy combatant or incorrect
identification of a friend as an enemy, has costly results for
the Warfighter and associated team members. Furthermore,
Warfighters are often trained to sustain their attention over
long periods of time, but by-products of situational
demands, such as operational stress and fatigue, can
significantly impact performance related to visual attention
processes (Staal, 2004; Janelle & Hatfield, 2008). Thus, it is
of paramount importance to understand perceptual judgment
processes in individual Warfighters when confronted with
moderators of operational performance.
Methods
The DREEMS Project
To better understand the dynamics of Warfighter
performance, Charles River Analytics has introduced the
Dynamic Representation for Evaluating the Effect of
Moderators and Stress on Performance (DREEMS) project.
Using a modeling language called Hap (Loyall et al., 1991),
DREEMS models individual Warfighter performance
through the use of situational awareness modeling
(SAMPLE; Zacharias et al., 1996). The architecture models
agent behavior as the cumulative result of an information
processing module feeding into a situation assessor, which
then guides an agent’s decision-making via behavior trees.
Moderating variables can exert influence over these
modules at any point in the architecture.
Signal Detection Approach
Using a signal detection theory (SDT) approach to
cognition provides a robust method of exploring a wide
variety of behaviors, particularly for those in which
individuals encounter perceptual uncertainty and behavioral
risk (Lynn & Barrett, 2014). Importantly, the parameters of
SDT mechanisms have been shown to be sensitive to
moderating variables, such as emotion on field of view
(Schmitz et al., 2009). Here, we follow previous research
illustrating how perceptual decision parameters in a
perceptual SDT model are moderated by differences in
individual state (e.g., Lynn et al., 2012) and extend this
approach to modeling visual threat detection in a battlefield
environment.
Model Specifications
We model perception as a set of underlying receptors that
correspond to different regions in the Field of Regard
(FOR), or visual field, a span of 135°. These receptors
respond to signals from the visual environment. When a
signal is presented in the visual field, the ability of the agent
to discern the signal from background noise is a function of
the location of the signal in the FOR: Perceptual sensitivity
to discriminate signal from noise decreases toward the
periphery of the FOR (Fig. 1).
Figure 1. An agent’s FOR (Field of Regard). Past the FOV
(Field of Vision; 75°-105°), receptor sensitivity declines
Lynn, S.K., T. Curley, and P. Weyhrauch (2018). Modeling Perceptual Judgement in Believable Agents: A Signal Detection Approach.
Presented at the joint annual meetings of the Society for Mathematical Psychology and the International Conference on Cognitive Modelling,
21-24 July, Madison, Wisconsin, USA.
with increasing distance from the fovea.
After a signal is detected in the visual field, the agent
makes a decision about the identity of the signal, such as if
it is a threat or not. The criterion defining the perceptual
judgement between noise and objects-of-interest in the
environment (e.g. threat vs. non-threat) is given by SDT’s
utility function: u(x) = αhP[CD] + αmP[MD] + (1-α)aP[FA]
+ (1-α) jP[CR], where P[] is the probability of each of the
four possible outcomes, correct detection (CD), missed
detection (MD), false alarm (FA), or correct rejection (CR);
α is the base rate probability of encountering a signal; 1α
is the probability of encountering noise; and h, m, a, and j
represent the payoffs (benefits or costs) for hits, misses,
false alarms, and correct rejections, respectively. Thus, the
expected utility of adopting a decision threshold at a
particular signal value, x, is defined by the probabilities of
four outcomes, the base rate, and the payoffs. The optimal
decision threshold is found at the point of the highest utility
(Fig. 2).
Figure 2. A signals-approach to threat detection. Blue and
green Gaussians represent probability densities defining
what threats and non-threats look like, respectively, with
notional mean appearance of each category depicted by the
combatant and journalist. When perceptual uncertainty
exists (depicted as overlap of the Gaussians), mistakes
cannot be eliminated, but exposure to them can be
optimized. The maximum of a utility function (black curve)
locates this optimum: the decision criterion that will
maximize net benefits over a series of decisions.
Integrating Moderating Variables
Under our model, how effective an agent is at perceptual
decision making is dependent on the agent’s situation
awareness, defined as the accuracy of its signal transduction
at the receptors and its estimates of the CD, FA, etc.
probabilities; the base rate; and the payoffs. In order to
simulate the influence of operational variables on perceptual
decision making in Warfighters, we applied a function that
affects the accuracy of these parameters as function of
moderating variables, such as fatigue.
For example, distortion of the receptor transduction (or
another parameter, such as the base rate) can be modeled as
a simple sum of the incoming signal value, x, plus the
influence of behavioral moderators: x* = x+Mu(ba), where
Mu(ba) is the square of either a single or a collection of
behavioral moderators. The combined influence of several
moderators can be determined by a number different
methods, including only using the value of the moderator
with the most influence. For this project, we have employed
a logarithmic combination method (e.g. Moors, 2009): ba =
0.1
log2 (sgn(bi)
210*|bi), where bi is an individual
moderator in a set of behavioral moderators acting upon an
agent. Any moderator variable or collection of variables,
then, can affect the perceived value of a signal or effect how
that signal is judged by influencing the agent’s estimate of
the optimal threshold location a pattern consistent with
cognitive affective research (Lynn et al., 2012).
Acknowledgments
This material is based upon work supported by the US
Army Command Center, Aberdeen Proving Ground, Natick
Contracting Division ACC-APG-NCD under Contract No.
W911QY-17-C-0009. Any opinions, findings and
conclusions or recommendations expressed in this material
are those of the author(s) and do not necessarily reflect the
views of the US Army Command Center, Aberdeen Proving
Ground, Natick Contracting Division ACC-APG-NCD.
References
Bates, J., et al. (1994). The role of emotion in believable
agents. Communications of the ACM, 37, 122125.
Janelle, C. M., & Hatfield, B. D. (2008). Visual attention
and brain processes that underlie expert performance:
Implications for sport and military psychology. Military
Psychology, 20, S39.
Loyall, A. B., & Bates, J. (1991). Hap: A reactive, adaptive
architecture for agents. Carnegie Mellon University
Department of Computer Science Technical Report
CMU-CS-91-147.
Lynn, S. K., & Barrett, L. F. (2014). Utilizing signal
detection theory. Psychological Science, 25, 16631673.
Lynn, S. K., Zhang, X., & Barrett, L. F. (2012). Affective
state influences perception by affecting decision
parameters underlying bias and sensitivity. Emotion, 12,
726.
Moors, A. (2009). Theories of emotion causation: A review.
Cognition and Emotion, 23, 625662.
Schmitz, T. W., De Rosa, E., & Anderson, A. K. (2009).
Opposing influences of affective state valence on visual
cortical encoding. Journal of Neuroscience, 29, 7199-
7207.
Staal, M. A. (2004). Stress, cognition, and human
performance: A literature review and conceptual
framework. NASA Technical Report NASA/TM-2004-
212824.
Zacharias, G. L., Miao, A. X., Illgen, C., Yara, J. M., &
Siouris, G. M. (1996). SAMPLE: Situation awareness
model for pilot in-the-loop evaluation. In Proceedings of
the 1st annual conference on situation awareness in the
tactical air environment.
Modeling Perceptual Judgement in
Believable Agents:
A Signal Detection Approach
Spencer K. Lynn (slynn@cra.com)1
Taylor Curley (taylor.curley@gatech.edu)2
Peter Weyhrauch (pweyhrauch@cra.com)1
1Human Effectiveness Division, Charles River Analytics,
Inc., 625 Mt. Auburn St., Cambridge, MA 02138
2School of Psychology, Georgia Institute of Technology, J.S.
Coon Bldg., 654 Cherry Street, Atlanta, Georgia 30332
The DREEMS Project
A Warfighter in acombat environment is expected to continuously
search his or her visual field to maintain situational awareness.
Misidentification of relevant stimuli, such as failure to detect an
enemy combatant or incorrect identification of afriend as an enemy,
has costly results for the Warfighter and associated team members.
By-products of situational demands, such as stress and fatigue, can
significantly impact operational performance via moderation of
perceptual processes (e.g., Staal 2004).
It is of paramount importance to understand perceptual judgment
processes in individual Warfighters when confronted with moderators
of operational performance.
To better model visual detection and identification processes in
realistic performance situations, we have constructed an agent-based
model of visual perception based on signal detection theory that can
be moderated by exigent processes, such as stress and fatigue.
Background
Internal and external factors affect perceptual judgements, and it is rare
for these moderators to have their effects in isolation.
In the current model, effective decision-making depends on the agent’s
situational awareness, which we define as the agent’s accuracy of signal
transduction at the sensory receptors and its estimates of the signal
parameters.For example:
We simulate disrupted transduction as the simple sum of the incoming
signal value, x,and the combined influence of one or more behavioral
moderators, Mu(ba):x* =x+Mu(ba).
To simulate the combined influence of moderators, we employ a
logarithmic combination method:ba= 0.1
log2(∑sgn(bi)
210*|bi|)
where biis asingle behavioral moderator (Moors 2009).
Integrating Moderators
We model perception as an array of receptors that correspond to
different regions in the Field of Regard (FOR;Figure 2). For asignal in
the FOR, an agent’s ability to correctly categorize the signal as athreat
or not is afunction of location in the FOR and the agent’s internal state.
The agent makes adecision about the identity of the signal (as a
threat or not).The criterion defining this perceptual judgement is given
by the signal detection theory (SDT) utility function (Figure 3).
Model Architecture
Charles River Analytics has introduced the Dynamic Representation
for Evaluating the Effect of Moderators on Stress (DREEMS) project
DREEMS models Warfighter performance through the use of
situational awareness modeling (e.g., SAMPLE;Zacharias et al.
1996). See Figure 1.
Agent performance is represented in behavior trees using alanguage
called Hap (Loyall &Bates 1991).
DREEMS models behavior generation as the cumulative result of an
information processing module feeding into asituation module, which
then guide’s an agent’s behavior via goals and behaviors in Hap.
The DREEMS Project
This material is based upon work supported by the US Army Command Center,
Aberdeen Proving Ground, Natick Contracting Division ACC-APG-NCD under
Contract No. W911QY-17-C-0009.Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the US Army Command Center, Aberdeen Proving
Ground, Natick Contracting Division ACC-APG-NCD.
Acknowledgments
Figure 1. Elements of aSAMPLE-based agent.
Figure 2.The Field of Regard.Past the FOV (Field of Vision;75°-105°),
receptor sensitivity declines with increasing distance from the fovea.
Figure 3. The signals approach to threat detection.Blue and green
Gaussians represent probability densities defining what threats and
non-threats look like, respectively, with notional mean appearance of
each category depicted by the combatant and journalist.When
perceptual uncertainty exists (depicted as overlap of the Gaussians),
mistakes cannot be eliminated, but exposure to them can be optimized.
The maximum of autility function (black curve) locates this optimum:
the decision criterion that will maximize net benefits over aseries of
decisions (after Lynn &Barrett 2014).
Loyall, A. B., and Bates, J. 1991.Hap: A Reactive, Adaptive Architecture for Agents.
Tec hn ic a l Report CMU-CS-91-147.Department of Computer Science.
Carnegie Mellon University.
Lynn, S. K., Zhang, X., & Barrett, L. F. (2012). Affective state influences perception
by affecting decision parameters underlying bias and sensitivity.Emotion, 12,
726.
Lynn, S. K., & Barrett, L. F. (2014). “Utilizingsignal detection theory.Psychological
Science 25:1663-1673..
Moors, A. (2009). Theories of emotion causation: A review.Cognition and Emotion,
23,625662.
Staal, M. A. (2004). Stress, cognition, and human performance: A literature review
and conceptual framework.NASA Tec h nic al Report NASA/TM-2004-212824
Zacharias, G. L., Miao, A. X., Illgen, C., Yar a, J. M., & Siouris, G. M. (1996,June).
SAMPLE:Situation awareness model for pilot in-the-loop evaluation.
In Proceedings of the 1st Annual Conference on Situation Awareness in the
Tactical Air Environment.
References
Modeling perceptual judgement with the SDT utility function provides
parameters through which physical or psychological factors can moderate
perception (e.g., Lynn et al.2012).
These parameters include the signal value (x-axis in Fig. 3), the
similarity of threats and non-threats (Gaussians in Fig. 3), the relative
base rate of threats to non-threats, and the benefits and costs of correct
vs incorrect signal categorization.In SDT, these parameters underlie
performance metrics of perceptual sensitivity and response bias.
In our model, moderators affect the accuracy of the agent’s estimates
of these parameters, which are building blocks of situation awareness.
Moderating Perception
Incorporate additional findings from published research.
Identify knowledge gaps re:moderation of Warfighter-relevant tasks.
Run studies to fill gaps.
Future Directions
Lynn, S.K., T. Curley, and P. Weyhrauch (2018). Modeling Perceptual Judgement in Believable Agents: A Signal Detection Approach. Presented at the joint
annual meetings of the Society for Mathematical Psychology and the International Conference on Cognitive Modelling, 21-24 July, Madison, Wisconsin, USA.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
What do inferring what a person is thinking or feeling, judging a defendant's guilt, and navigating a dimly lit room have in common? They involve perceptual uncertainty (e.g., a scowling face might indicate anger or concentration, for which different responses are appropriate) and behavioral risk (e.g., a cost to making the wrong response). Signal detection theory describes these types of decisions. In this tutorial, we show how incorporating the economic concept of utility allows signal detection theory to serve as a model of optimal decision making, going beyond its common use as an analytic method. This utility approach to signal detection theory clarifies otherwise enigmatic influences of perceptual uncertainty on measures of decision-making performance (accuracy and optimality) and on behavior (an inverse relationship between bias magnitude and sensitivity optimizes utility). A "utilized" signal detection theory offers the possibility of expanding the phenomena that can be understood within a decision-making framework.
Article
Full-text available
Studies of the effect of affect on perception often show consistent directional effects of a person's affective state on perception. Unpleasant emotions have been associated with a "locally focused" style of stimulus evaluation, and positive emotions with a "globally focused" style. Typically, however, studies of affect and perception have not been conducted under the conditions of perceptual uncertainty and behavioral risk inherent to perceptual judgments outside the laboratory. We investigated the influence of perceivers' experienced affect (valence and arousal) on the utility of social threat perception by combining signal detection theory and behavioral economics. We compared 3 perceptual decision environments that systematically differed with respect to factors that underlie uncertainty and risk: the base rate of threat, the costs of incorrect identification threat, and the perceptual similarity of threats and nonthreats. We found that no single affective state yielded the best performance on the threat perception task across the 3 environments. Unpleasant valence promoted calibration of response bias to base rate and costs, high arousal promoted calibration of perceptual sensitivity to perceptual similarity, and low arousal was associated with an optimal adjustment of bias to sensitivity. However, the strength of these associations was conditional upon the difficulty of attaining optimal bias and high sensitivity, such that the effect of the perceiver's affective state on perception differed with the cause and/or level of uncertainty and risk.
Article
Advances in methodology have allowed sport psychologists to broaden their understanding of expert performance through inclusion of physiological assessments of expertise (e.g., eye movements and bioelectric signals such as the electroencephalogram [EEG]). Innovations linking physiology, basic cognitive processes, and performance have illustrated the degree to which these techniques can converge on a finer-grained understanding of factors driving learning and performance at exceptional levels. A recent focus of this research has centered on the coupling between visual search patterns and other psychophysiological indices of attention and arousal (such as the spectral characteristics of the EEG), particularly among expert and nonexpert performers. Within this context, we describe results of recent investigations that have employed psychophysiological assessment of emotion/attention interactions and their influence on motor task performance in increasingly applied ecologically valid environments either simulated in the laboratory or monitored in actual competitive settings. Current theoretical, empirical, and applied questions in sport psychology are discussed while considering critical issues pertaining to current and future military initiatives.
Article
Positive and negative emotional states are thought to have originated from fundamentally opposing approach and avoidance behaviors. Furthermore, affective valence has been hypothesized to exert opposing biases in cognitive control. Here we examined with functional magnetic resonance imaging whether the opposing influences of positive and negative states extend to perceptual encoding in the visual cortices. Based on prior behavioral research, we hypothesized that positive states would broaden and negative states would narrow visual field of view (FOV). Positive, neutral, and negative states were induced on alternating blocks. To index FOV, observers then viewed brief presentations (300 ms) of face/place concentric center/surround stimuli on interleaved blocks. Central faces were attended, rendering the place surrounds unattended. As face and place information was presented at different visual eccentricities, our physiological metric of FOV was a valence-dependent modulation of place processing in the parahippocampal place area (PPA). Consistent with our hypotheses, positive affective states increased and negative states decreased PPA response to novel places as well as adaptation to repeated places. Individual differences in self-reported positive and negative affect correlated inversely with PPA encoding of peripheral places, as well as with activation in the mesocortical prefrontal cortex and amygdala. Psychophysiological interaction analyses further demonstrated that valence-dependent responses in the PPA arose from opponent coupling with extrafoveal regions of the primary visual cortex during positive and negative states. These findings collectively suggest that affective valence differentially biases gating of early visual inputs, fundamentally altering the scope of perceptual encoding.
Article
Artificial intelligence researchers attempting to create engaging, apparently living creatures may find important insight in the work of artists who have explored the idea of believable character. In particular, appropriately timed and clearly expressed emotion is a central requirement for believable characters. We discuss these ideas and suggest how they may apply to believable interactive characters, which we call "believable agents." This work was supported in part by Fujitsu Laboratories and Mitsubishi Electric Research Laboratories. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of any other parties. Keywords: artificial intelligence, emotion, believable agents, art, animation, believable characters. 1 BELIEVABILITY 1 1 Believability There is a notion in the Arts of "believable character." It does not mean an honest or reliable character, but one th...
Hap: A reactive, adaptive architecture for agents
  • A B Loyall
  • J Bates
Loyall, A. B., & Bates, J. (1991). Hap: A reactive, adaptive architecture for agents. Carnegie Mellon University Department of Computer Science Technical Report CMU-CS-91-147.
SAMPLE: Situation awareness model for pilot in-the-loop evaluation
  • G L Zacharias
  • A X Miao
  • C Illgen
  • J M Yara
  • G M Siouris
Zacharias, G. L., Miao, A. X., Illgen, C., Yara, J. M., & Siouris, G. M. (1996). SAMPLE: Situation awareness model for pilot in-the-loop evaluation. In Proceedings of the 1st annual conference on situation awareness in the tactical air environment.