Modeling Theory of Mind and Cognitive Appraisal with
David V. Pynadath1, Mei Si2, and Stacy C. Marsella1
1Institute for Creative Technologies, University of Southern California
12015 Waterfront Drive, Playa Vista, CA 90094-2536 USA
2Cognitive Science Department, Rensselaer Polytechnic Institute
110 8th Street, Troy, NY 12180 USA
firstname.lastname@example.org, email@example.com, firstname.lastname@example.org
April 7, 2011
Agent-based simulation of human social behavior has become increasingly important as a basic research tool to
further our understanding of social behavior, as well as to create virtual social worlds used to both entertain and
educate. A key factor in human social interaction is our beliefs about others as intentional agents, a Theory of Mind.
How we act depends not only on the immediate effect of our actions but also on how we believe others will react. In
this paper, we discuss PsychSim, an implemented multiagent-based simulation tool for modeling social interaction
and influence. While typical approaches to such modeling have used first-order logic, PsychSim agents have their
own decision-theoretic models of the world, including beliefs about their environment and recursive models of other
agents. Using these quantitative models of uncertainty and preferences, we have translated existing psychological
theories into a decision-theoretic semantics that allow the agents to reason about degrees of believability in a novel
way. We demonstrate the expressiveness of PsychSim’s decision-theoretic implementation of Theory of Mind by
presenting its use as the foundation for a domain-independent model of appraisal theory, the leading psychological
theory of emotion. The model of appraisal within PsychSim demonstrates the key role of a Theory of Mind capacity
in appraisal and social emotions, as well as arguing for a uniform process for emotion and cognition.
behavior and a range of applications. For example, computational models of psychological or sociological theories
promise to transform how theories of human behavior are formulated and evaluated . In addition, computational
models of human social interaction have also become increasingly important as a means to create simulated social
environments used in a variety of training and entertainment applications . For example, many serious games have
We argue that such models of social interaction must address the fact that people interact within a complex social
framework. A central factor in social interaction is the beliefs we have about each other, a Theory of Mind .
Our choice of action is influenced by how we believe others will feel and react. Whether we believe what we are
told depends not only on the content of the communication but also on our model of the communicator. How we
emotionally react to another’s action is influenced by our beliefs as well, for example whether we believe he or she
intended to cause harm . The central goal of our research is to bring such Theory of Mind capacities to the design
of computational models of social interaction both as a basic research tool and as a framework for virtual character
Unfortunately, traditional artificial intelligence techniques are ill-suited for modeling Theory of Mind. Representa-
tions using first-order logic are often insensitive to the distinctions among conflicting goals that people must balance in
a social interaction. For example, psychological research has identified a range of goals that motivate classroom bullies
(e.g., peer approval, sadism, tangible rewards) . Different bullies may share the same goals, but the relative priori-
ties that they place on them will lead to variations in their behavior. Resolving the ambiguity among equally possible,
but unequally plausible or preferred, options requires a quantitative model of uncertainty and preference. Unfortu-
nately, more quantitative frameworks, like decision theory and game theory, face their own difficulties in modeling
human psychology. Game-theoretic frameworks typically rely on concepts of equilibria that people rarely achieve
in an unstructured social setting like a classroom. Decision-theoretic frameworks typically rely on assumptions of
rationality that people violate.
We have developed a social simulation framework, PsychSim [20, 27], that operationalizes existing psychological
theories as boundedly rational computations to generate more plausibly human behavior. PsychSim allows a user to
quickly construct a social scenario where a diverse set of entities, groups or individuals, interact and communicate.
Each entity has its own preferences, relationships (e.g., friendship, hostility, authority) with other entities, private
beliefs, and mental models about other entities. The simulation tool generates the behavior for these entities and
provides explanations of the result in terms of each entity’s preferences and beliefs. The richness of the entity models
allows one to explore the potential consequences of minor variations on the scenario.
A central aspect of the PsychSim design is that agents have fully specified decision-theoretic models of others.
Such quantitative recursive models give PsychSim a powerful mechanism to model a range of factors in a principled
way. For instance, we exploit this recursive modeling to allow agents to form complex attributions about others,
send messages that include the beliefs and preferences of others, and use their observations of another’s behavior to
influence their model of that other.
In operationalizing psychological theories within PsychSim, we have taken a strong architectural stance. We
assume that decision-theoretic agents that incorporate a Theory of Mind provide a uniform, sufficient computational
core for modeling the factors relevant to human social interaction. While the sufficiency of our framework remains an
open question, such a strong stance yields the benefit of uniform processes and representations that cover a range of
phenomena. Our stance thus provides subsequent computational benefits, such as optimization and reuse of the core
algorithms that provide the agent’s decision-making and belief revision capacities.
More significantly, this uniformity begins to reveal common elements across apparently disparate psychological
phenomena that typically have different methodological histories. To illustrate such common elements, we have
demonstrated how a range of human psychological and social phenomena can be modeled within our framework,
including wishful thinking , influence factors , childhood aggression  and emotion .
In this article, we discuss two of those models. First, we use a model of childhood aggression to motivate the
discussion of the overall framework as well as to demonstrate its expressiveness. Second, in keeping with the theme
of this volume, we go into considerable detail on how PsychSim’s decision-theoretic agents with a Theory of Mind
provide a particularly effective basis for a computational model of emotion.
Computational models of emotion have largely been based on appraisal theory [5, 7, 18, 22, 6, 23, 28, 47], a leading
psychological theory of emotion. Appraisal theory argues that a person’s subjective assessment of their relationship
to the environment determines his or her emotional responses [8, 13, 14, 23, 29, 30, 41, 42]. This assessment occurs
along several dimensions, such as motivational congruence, accountability, novelty and control. For example, an
event that leads to a bad outcome for a person (motivationally incongruent) and is believed to be caused by others
(accountability) is likely to elicit an anger response. On the other hand, if the event is believed to be caused by the
person himself/herself, he/she is more likely to feel guilt or regret .
We approach the task of incorporating appraisal into the existing PsychSim multiagent framework as a form of
thought experiment: Can we leverage the existing processes and representations in PsychSim to model appraisal? The
motivations for this thought experiment are three-fold. First, we seek to demonstrate overlap between the theoretical
model of appraisal theory and decision-theoretic, social agents of PsychSim. Specifically, we are interested in whether
appraisal offers a possible blueprint, or requirements specification, for intelligent social agents by showing that an
existing framework not predesigned with emotion or appraisal in mind has, in fact, appraisal-like processes already
Conversely, we seek to illustrate the critical role that subjective beliefs about others plays in allowing agents to
model social emotions. Because the agent’s representations and decision-making processes are rooted in a Theory
of Mind capacity, incorporating and maintaining beliefs about others, the appraisal process inherits this social frame,
allowing the agent to appraise events from its own perspective as well as others. Thus, in keeping with the tenets of
social appraisal , the behaviors, thoughts and emotions of the other can also be appraised and thereby influence
Finally, we seek a design that is elegant, by reusing architectural features to realize new capabilities such as emo-
tion. Alternative approaches for creating embodied conversational agents and virtual agents often integrate separate
modules for emotion, decision-making, dialogue, etc., which leads to sophisticated but complex architectures .
The work here is an alternative minimalist agenda for agent design. In particular, based on the core theory of mind
reasoning processes, appraisal can be derived with few extensions.
We begin the paper with a demonstration of PsychSim’s application to a childhood aggression scenario. We then
discuss how PsychSim can represent appraisal theory and present a preliminary assessment of its implementation.
2 The Agent Models
This section describes PsychSim’s underlying architecture, using a school bully scenario for illustration. The agents
represent different people and groups in the school setting. The user can analyze the simulated behavior of the students
to explore the causes and cures for school violence. One agent represents a bully, and another represents the student
who is the target of the bully’s violence. A third agent represents the group of onlookers, who encourage the bully’s
exploits by, for example, laughing at the victim as he is beaten up. A final agent represents the class’s teacher trying
to maintain control of the classroom, for example by doling out punishment in response to the violence. We embed
PsychSim’s agents within a decision-theoretic framework for quantitative modeling of multiple agents. Each agent
maintains its independent beliefs about the world, has its own goals and it owns policies for achieving those goals.
2.1 Model of the World
Each agent model starts with a representation of its current state and the Markovian process by which that state evolves
over time in response to the actions performed.
Each agent model includes several features representing its “true” state. This state consists of objective facts about
the world, some of which may be hidden from the agent itself. For our example bully domain, we included such
state features as power(agent), to represent the strength of an agent. trust(truster,trustee) represents
the degree of trust that the agent truster has in another agent trustee’s messages. support(supporter,
supportee) is the strength of support that an agent supporter has for another agent supportee. We represent
the state as a vector, ? st, where each component corresponds to one of these state features and has a value in the range
Agents have a set of actions that they can choose to change the world. An action consists of an action type (e.g.,
punish), an agent performing the action (i.e., the actor), and possibly another agent who is the object of the action.
For example, the action laugh(onlooker, victim) represents the laughter of the onlooker directed at the
2.1.3 World Dynamics
The state of the world changes in response to the actions performed by the agents. We model these dynamics using a
transition probability function, T(? si,? a,? sf), to capture the possibly uncertain effects of these actions on the subsequent
Pr(? st+1= ? sf|? st= ? si,? at= ? a) = T(? si,? a,? sf)
For example, the bully’s attack on the victim impacts the power of the bully, the power of the victim, etc. The
distribution over the bully’s and victim’s changes in power is a function of the relative powers of the two—e.g., the
larger the power gap that the bully enjoys over the victim, the more likely the victim is to suffer a big loss in power.
tions to the scenario . We believe PsychSim has a range of innovative applications, including computational social
science and the modeling of social training environments.
As an illustration of such applications, and PsychSim’s expressiveness, we provide a computational model of
appraisal for POMDP-based agents, implemented in the Thespian framework for interactive narrative. The focus is
on five key appraisal dimensions for virtual agents: motivational relevance, motivational congruence, accountability,
control and novelty. The approach argues that appraisal is an integral part of a social agent’s cognitive processes.
All of these capabilities of the appraisal model derive from the basic PsychSim cognitive components as laid out
in Section 2. We were thus able to leverage the decision-theoretic Theory of Mind as implemented in our agents to
realize appraisal theory, even though modeling appraisal was not an original intention of the agents’ design. The reuse
of architectural features therefore provides, not only a novel computational model of emotion, but also a demonstration
of a tight relationship between emotion and cognition, suggesting a uniform cognitive structure for emotion and cog-
nition. In addition, by demonstrating how a Theory of Mind capacity is critical to deriving appraisals, and in particular
modeling appraisals critical to social emotions like anger, this work argues for the critical role for Theory of Mind in
modeling social interaction generally.
This work was sponsored by the U.S. Army Research, Development, and Engineering Command (RDECOM), and the
content does not necessarily reflect the position or the policy of the Government, and no official endorsement should
 R. P. Abelson, E. Aronson, W. J. McGuire, T. Newcomb, M. Rosenberg, and P. H. Tannenbaum, editors. Theories
of Cognitive Consistency: A Sourcebook. Rand McNally, Chicago, IL, 1968.
 R. Aylett, J. Dias, and A. Paiva. An affectively-driven planner for synthetic characters. In Proceedings of the
International Conference on Automated Planning and Scheduling, 2006.
 R. Cialdini. Influence: Science and Practice. Allyn and Bacon, Boston, MA, 2001.
 F. de Rosis, C. Castelfranchi, V. Carofiglio, and G. Grassano. Can computers deliberately deceive? a simulation
tool and its application to Turing’s imitation game. Computational Intelligence, 19(3):253–263, 2003.
 J. Dias and A. Paiva. Feeling and reasoning: A computational model for emotional characters. In Proceedings
of the Portuguese Conference on Artificial Intelligence, pages 127–140, 2005.
 M. S. El Nasr, J. Yen, and T. Ioerger. Flame: Fuzzy logic adaptive model of emotions. Autonomous Agents and
Multi-Agent Systems, 3(3):219–257, 2000.
 C. Elliott. The affective reasoner: A process model of emotions in a multi-agent system. PhD thesis, Northwestern
University Institute for the Learning Sciences, 1992.
 N. Frijda. The Emotions. Cambridge University Press, 1987.
 P. J. Gmytrasiewicz and E. H. Durfee. A rigorous, operational formalization of recursive modeling. In Proceed-
ings of the International Conference on Multi-Agent Systems, page 125132, 1995.
 J. Gratch and S. Marsella. A domain-independent framework for modeling emotion. Cognitive Systems Research,
 J. Y. Ito, D. V. Pynadath, and S. C. Marsella. A decision-theoretic approach to evaluating posterior probabilities
of mental models. In AAAI Workshop on Plan, Activity, and Intent Recognition, 2007.
 J. Y. Ito, D. V. Pynadath, L. Sonenberg, and S. C. Marsella. Wishful thinking in effective decision making. In
Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1527–1528,
 R. S. Lazarus. Emotion & Adaptation. Oxford University Press., New York, 1991.
 N. Lazzaro. Why we play games: four keys to more emotion in player experiences. In Game Developers
 H. Leventhal and K. R. Scherer. The relationship of emotion and cognition: A functional approach to a semantic
controversy. Cognition and Emotion., 1:3–28, 1987.
 A. Manstead and A. Fischer. Social appraisal: The social world as object of and influence on appraisal processes.
In . T. J. K. R. Scherer, A. Schorr, editor, Appraisal Processes in Emotion: Theory, Research, Application, pages
221–232. Oxford University Press, New York, 2001.
 W. Mao and J. Gratch. Social causality and responsibility: Modeling and evaluation. In Proceedings of the
International Conference on Virtual Agents, Kos, Greece, 2005.
 R. Marinier, J. Laird, and R. Lewis. A computational unification of cognitive behavior and emotion. Journal of
Cognitive Systems Research, 2009.
 S. C. Marsella and J. Gratch. Ema: A model of emotional dynamics. Cognitive Systems Research., 10(1):70–90,
 S. C. Marsella, D. V. Pynadath, and S. J. Read. PsychSim: Agent-based modeling of social interactions and
influence. In Proceedings of the International Conference on Cognitive Modeling, pages 243–248, 2004.
 R. McAlinden, A. Gordon, H. C. Lane, and D. Pynadath. UrbanSim: A game-based simulation for counterinsur-
gency and stability-focused operations. In Proceedings of the AIED Workshop on Intelligent Educational Games,
 D. Moffat and N. Frijda. Where there’s a will there’s an agent. In Proceedings of the ECAI Workshop on Agent
Theories, Architectures, and Languages, Amsterdam, The Netherlands, 1995.
 A. Ortony, G. L. Clore, and A. Collins. The Cognitive Structure of Emotions. Cambridge. UK: Cambridge
University Press, 1998.
 A. Paiva, J. Dias, D. Sobral, R. Aylett, P. Sobreperez, S. Woods, and C. Zoll. Caring for agents and agents
that care: Building empathic relations with synthetic agents. In Proceedings of the International Conference on
Autonomous Agents and Multi-Agent Systems, pages 194–201, 2004.
 R. Petty and J. Cacioppo. Communication and persuasion: Central and peripheral routes to attitude change.
Springer, New York, NY, 1986.
 D. V. Pynadath and S. C. Marsella. Fitting and compilation of multiagent models through piecewise linear
functions. In Proceedings of the International Conference on Autonomous Agents and Multi Agent Systems,
pages 1197–1204, 2004.
 D. V. Pynadath and S. C. Marsella. PsychSim: Modeling theory of mind with decision-theoretic agents. In
Proceedings of the International Joint Conference on Artificial Intelligence, pages 1181–1186, 2005.
 W. S. Reilly and J. Bates. Building emotional agents. Technical Report CMU-CS-92-143, Carnegie Mellon
 I. Roseman. Cognitive determinants of emotion: A structural theory. Review of Personality and Social Psychol-
ogy, 2:11–36, 1984.
 K. Scherer. Appraisal considered as a process of multilevel sequential checking. In K. Scherer, A. Schorr,
and T. Johnstone, editors, Appraisal Processes in Emotion: Theory, Methods. Oxford University Press., Oxford,
 D. Schwartz. Subtypes of victims and aggressors in children’s peer groups. Journal of Abnormal Child Psychol-
ogy, 28:181–192, 2000.
 K. G. Shaver. The Attribution Theory of Blame: Causality, Responsibility and Blameworthiness. Springer-Verlag,
 M. Si, S. C. Marsella, and D. V. Pynadath. Thespian: An architecture for interactive pedagogical drama. In
Proceedings of the Conference on Artificial Intelligence in Education, 2005.
 M. Si, S. C. Marsella, and D. V. Pynadath. Thespian: Using multi-agent fitting to craft interactive drama. In
Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems, pages 21–28,
 M. Si, S. C. Marsella, and D. V. Pynadath. Thespian: Modeling socially normative behavior in a decision-
theoretic framework. In Proceedings of the Conference on Intelligent Virtual Agents, 2006.
 M. Si, S. C. Marsella, and D. V. Pynadath. Proactive authoring for interactive drama: An author’s assistant. In
Proceedings of the Conference on Intelligent Virtual Agents, Paris, France, 2007.
 M. Si, S. C. Marsella, and D. V. Pynadath. Directorial control in a decision-theoretic framework for interactive
narrative. In Proceedings of the International Conference on Interactive Digital Storytelling, pages 221–233,
 M. Si, S. C. Marsella, and D. V. Pynadath. Evaluating directorial control in a character-centric interactive narra-
tive framework. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems,
pages 1289–1296, 2010.
 M. Si, S. C. Marsella, and D. V. Pynadath. Modeling appraisal in theory of mind reasoning. Journal of Au-
tonomous Agents and Multi-Agent Systems, 20(1):14–31, 2010.
 R. D. Smallwood and E. J. Sondik. The optimal control of partially observable Markov processes over a finite
horizon. Operations Research, 21:1071–1088, 1973.
 C. A. Smith and P. C. Ellsworth. Patterns of appraisal and emotion related to taking an exam. Personality and
Social Psychology, 52:475–488, 1987.
 C. A. Smith and R. S. Lazarus. Emotion and adaptation. In L. A. Pervin, editor, Handbook of personality:Theory
and research. Guilford, New York, 1990.
 J. G. Stacy Marsella and P. Petta. Computational models of emotion. In K. Scherer, T. Bnziger, and E. Roesch,
editors, Ablueprintforanaffectivelycompetentagent: Cross-fertilizationbetweenEmotionPsychology, Affective
Neuroscience, and Affective Computing. Oxford University Press, Oxford, 2010.
 W. Swartout, R. Hill, J. Gratch, W. Johnson, C. Kyriakakis, C. LaBore, R. Lindheim, S. C. Marsella, D. Miraglia,
B. Moore, J. Morie, J. Rickel, M. Thi´ ubaux, L. Tuch, R. Whitney, and J. Douglas. Toward the holodeck:
Integrating graphics, sound, character and story. In Proceedings of the International Conference on Autonomous
Agents, pages 409–416, 2001.
 J. Taylor, J. Carletta, and C. Mellish. Requirements for belief models in cooperative dialogue. User Modelling
and User-Adapted Interaction, 6:23–68, 1996.
 D. R. Traum, W. Swartout, S. C. Marsella, and J. Gratch. Fight, flight, or negotiate: Believable strategies for
conversing under crisis. In Proceedings of the Conference on Intelligent Virtual Agents, 2005.
 J. Velasquez. Modeling emotions and other motivations in synthetic agents. In Proceedings of the National
Conference on Artificial Intelligence, 1997.
 B. Weiner. The Judgment of Responsibility: A Foundation for a Theory of Social Conduct. The Guilford Press,
 A. Whiten, editor. Natural Theories of Mind. Basil Blackwell, Oxford, UK, 1991.