Stacy Marsella

Northeastern University | NEU

Research in procedural content generation (PCG) has recently heralded two major methodologies: machine learning (PCGML) and declarative programming. The former shows promise by automating the specification of quality criteria through latent patterns in data, while the latter offers significant advantages for authorial control. In this paper we prop...

Virtual humans and social robots frequently generate behaviors that human observers naturally see as expressing emotion. In this review article, we highlight that these expressions can have important benefits for human–machine interaction. We first summarize the psychological findings on how emotional expressions achieve important social functions...

Studying mental models has recently received more attention, aiming to understand the cognitive aspects of human-computer interaction. However, there is not enough research on the elicitation of mental models in complex dynamic systems. We present Thought Bubbles as an approach for eliciting mental models and an avenue for understanding players' me...

We stumble upon new and repeating information daily. As information comes from many sources, social media continues to play a predominant role in disseminating information, ultimately impacting individuals’ perceptions and behaviors. A prime example of this impact was observed during the COVID-19 pandemic, in which social media use was influencing...

Human social interaction hinges on the ability to interpret and predict the actions of others. The most valuable explanatory variable of these actions, more important than environmental or social factors, is the one that we do not have direct access to: the mind. This lack of access leaves us to impute the mental states—beliefs, desires, emotions,...

To assist game developers in crafting game NPCs, we present EvolvingBehavior, a novel tool for genetic programming to evolve behavior trees in Unreal Engine 4. In an initial evaluation, we compare evolved behavior to hand-crafted trees designed by our researchers, and to randomly-grown trees, in a 3D survival game. We find that EvolvingBehavior is...

More than 30 years after its initial publication, this new edition of The Cognitive Structure of Emotions refines and updates Ortony, Clore, and Collins's OCC model of emotions. Starting from a three-way classification of construals of the world––events, the attribution of responsibility for events, and objects––the authors propose a systematic acc...

During crises like COVID-19, individuals are inundated with conflicting and time-sensitive information that drives a need for rapid assessment of the trustworthiness and reliability of information sources and platforms. This parallels evolutions in information infrastructures, ranging from social media to government data platforms. Distinct from cu...

During crises like COVID-19, individuals are inundated with conflicting and time-sensitive information that drives a need for rapid assessment of the trustworthiness and reliability of information sources and platforms. This parallels evolutions in information infrastructures, ranging from social media to government data platforms. Distinct from cu...

Unhealthy eating behavior is a major public health issue with serious repercussions on an individual’s health. One potential solution to overcome this problem, and help people change their eating behavior, is to develop conversational systems able to recommend healthy recipes. One challenge for such systems is to deliver personalized recommendation...

Understanding decision-making in dynamic and complex settings is a challenge yet essential for preventing, mitigating, and responding to adverse events (e.g., disasters, financial crises). Simulation games have shown promise to advance our understanding of decision-making in such settings. However, an open question remains on how we extract useful...

People often encounter highly stressful, emotion-evoking situations. Modeling and predicting people's behavior in such situations, how they cope, is a critical research topic. To that end, we propose a computational model of coping that casts Lazarus' theory of coping into a Partially Observable Markov Decision Process (POMDP) framework. This inclu...

Older adults are at greater risk of experiencing negative physical and psychological impacts of the novel coronavirus 2019 (COVID-19) pandemic. Our ongoing study is assessing COVID-19 information exposure in adults aged 55 and above compared to other age groups living in Massachusetts and Georgia. This work investigates the potential association be...

This paper aims at advancing the fundamental understanding of the affordances of Augmented Reality (AR) as a workplace-based learning and training technology in supporting manual or semi-automated manufacturing tasks that involve both complex manipulation and reasoning. Between-subject laboratory experiments involving 20 participants are conducted...

Computer aided interactive drama has been widely applied for entertainment and pedagogy. Most existing approaches for authoring interactive drama use either story-centric or character-centric processes. In this work, we present a new framework that integrates both types of processes to support authoring. This framework uses a multi-agent system to...

Game AI designers must manage complex interactions between the AI character, the game world, and the player, while achieving their design visions. Computational co-creativity tools can aid them, but first, AI and HCI researchers must gather requirements and determine design heuristics to build effective co-creative tools. In this work, we present a...

Social signal processing and virtual social interaction technologies have allowed the creation of social skills training applications, and initial studies have shown that such solutions can lead to positive training outcomes and could complement traditional teaching methods by providing cheap, accessible, safe tools for training social skills. Howe...

As autonomous machines, such as automated vehicles (AVs) and robots, become pervasive in society, they will inevitably face moral dilemmas where they must make decisions that risk injuring humans. However, prior research has framed these dilemmas in starkly simple terms, i.e., framing decisions as life and death and neglecting the influence of risk...

Appraisal theories of emotion argue that emotions arise from a process of comparing individual needs and concerns to external demands. That is, emotions cannot be explained by solely focusing on the environment or by solely focusing on the individual. Rather, they reflect an individual’s subjective assessment of their relation to the environment. A...

One potential solution to help people change their eating behavior is to develop conversational systems able to recommend healthy recipes. Beyond the intrinsic quality of the recommendations themselves , various factors might also influence users' perception of a recommendation. Two of these factors are the conversational skills of the system and u...

One of the main challenges for conversational agents is to select the optimal dialogue policy based on the state of the interaction. This challenge becomes even harder when the conversational agent not only has to achieve a specific task, but also aims at building rapport. Although some work already tried to tackle this challenge using a Reinforcem...

Unhealthy eating behavior is a serious public health issue with massive repercussions on an individual's health. One potential solution to this problem is to help people change their eating behavior by developing systems able to recommend healthy recipes that can influence eating behavior. One challenge for such systems is to deliver healthy recomm...

Agent-based simulations are widely used for modeling human behavior in various contexts. However, such simulations may oversimplify human decision-making. We propose the use of Gamettes to extract rich data on human decision-making and help in improving the human behavioral aspects of models underlying agent-based simulations. We show how Gamettes...

The United States has experienced prolonged severe shortages of vital medications over the past two decades. The causes underlying the severity and prolongation of these shortages are complex, in part due to the complexity of the underlying supply chain networks, which involve supplier-buyer interactions across multiple entities with competitive an...

Causal inference and generalizability both matter. Historically, systematic designs emphasize causal inference, while representative designs focus on generalizability. Here, we suggest a transformative synthesis – Systematic Representative Design (SRD) – concurrently enhancing both causal inference and “built-in” generalizability by leveraging toda...

Systematic Representative Design (SRD), enabled by today’s technologies leverages many of the strengths of past designs into a new synthesis affording the capacity for both causal inference and generalizability to everyday life (GEL). In doing so, it could help better integrate past and ongoing empirical research findings in psychology. Generally,...

The use of metaphoric gestures by speakers has long been known to influence thought in the viewer. What is less clear is the extent to which the expression of multiple metaphors in a single gesture reliably affect viewer interpretation. Additionally, gestures which express only one metaphor are not sufficient to explain the broad array of metaphori...

UBeBot allows a mobile user to create a 3D avatar of themselves using a photo, as well as dress and style the avatar. Users then record their voice, allowing the avatar to act our the content of the utterance, including lip sync, facial expressions, gestures and other body language. This animated performance is generated automatically by analyzing...

It is commonly assumed that a person’s emotional state can be readily inferred from his or her facial movements, typically called emotional expressions or facial expressions. This assumption influences legal judgments, policy decisions, national security protocols, and educational practices; guides the diagnosis and treatment of psychiatric illness...

Significance Autonomous machines that act on our behalf—such as robots, drones, and autonomous vehicles—are quickly becoming a reality. These machines will face situations where individual interest conflicts with collective interest, and it is critical we understand if people will cooperate when acting through them. Here we show, in the increasingl...

The growing epidemic of drug shortages in the United States causes challenges for providers all across thecritical health care infrastructure and demonstrates the lack of resiliency within drug delivery supply chains.With many drugs having no acceptable substitute, drug shortages directly translate to a public health andsafety risk. One of the unde...

In this paper we highlight the different challenges in modeling communicative gestures for Embodied Conversational Agents (ECAs). We describe models whose aim is to capture and understand the specific characteristics of communicative gestures in order to envision how an automatic communicative gesture production mechanism could be built. The work i...

Bullying is a pressing societal problem. As such, it is important to gain a better understanding of the mechanisms involved in bullying and of resilience factors which might protect victims. Moreover, it is necessary to provide tools that can train potential victims to strengthen their resilience. To facilitate both of these goals, the current stud...

There has been growing interest on agents that represent people’s interests or act on their behalf such as automated negotiators, self-driving cars, or drones. Even though people will interact often with others via these agent representatives, little is known about whether people’s behavior changes when acting through these agents, when compared to...

While applications of virtual agents in training and pedagogy have largely concentrated on positive valenced environments and interactions, human-human interactions certainly also involve a fair share of negativity that is worth exploring in virtual environments. Further, in natural human interaction as well as in virtual spaces, physical actions a...

Interactive narrative (IN) has increasingly been used for social skill training. However, extensive content creation is needed to provide learners with flexibility to replay scenarios with sufficient variety to achieve proficiency. Such flexibility requires considerable content creation appropriate for social skills training. The goal of our work i...

Understanding a person’s mental state is a key challenge to the design of Artificial General Intelligence (AGI) that can interact with people. A range of technologies have been developed to infer a user’s emotional state from facial expressions. Such bottom-up approaches confront several problems, including that there are significant individual and...

Human interactions are characterized by explicit as well as implicit channels of communication. While the explicit channel transmits overt messages, the implicit ones transmit hidden messages about the communicator (e.g., his/her intentions and attitudes). There is a growing consensus that providing a computer with the ability to manipulate implici...

Guilt and envy play an important role in social interaction. Guilt occurs when individuals cause harm to others or break social norms. Envy occurs when individuals compare themselves unfavorably to others and desire to benefit from the others' advantage. In both cases, these emotions motivate people to act and change the status quo: following guilt...

To be perceived as believable partners in human-machine interactions, virtual agents have to express adequate social attitudes. The social attitude expressed by an agent should reflect the social situation of the interaction. The agent ought to take into account its role and its social relation toward its interactants when deciding how to react in...

It has long been recognized that facial expressions, body posture/gestures and vocal parameters play an important role in human communication and the implicit signalling of emotion. Recent advances in low cost computer vision and behavioral sensing technologies can now be applied to the process of making meaningful inferences as to user state when...

In this research, we investigate how grittier individuals might incur some costs by persisting when they could move on. Grittier participants were found to be less willing to give up when failing even though they were likely to incur a cost for their persistence. First, grittier participants are more willing to risk failing to complete a task by pe...

We describe an authoring framework for developing virtual humans on mobile applications. The framework abstracts many elements needed for virtual human generation and interaction, such as the rapid development of nonverbal behavior, lip syncing to speech, dialogue management, access to speech transcription services, and access to mobile sensors suc...

Gestures during spoken dialog play a central role in human communication. As a consequence, models of gesture generation are a key challenge in research on virtual humans, embodied agents capable of faceto- face interaction with people. Machine learning approaches to gesture generation must take into account the conceptual content in utterances, ph...

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.

Our method automatically generates realistic nonverbal performances for virtual characters to accompany spo- ken utterances. It analyses the acoustic, syntactic, se- mantic and rhetorical properties of the utterance text and audio signal to generate nonverbal behavior such as such as head movements, eye saccades, and novel gesture animations based...

There has been rapid growth of cross-disciplinary research employing computational methods to understand human behavior, as well as facilitate interaction between people and machines, with work on computational models of emotion becoming an important component. Modeling appraisal theory in an agent provides an interesting perspective on the relatio...

Recent advances in scanning technology have enabled the widespread capture of 3D character models based on human subjects. However, in order to generate a recognizable 3D avatar, the movement and behavior of the human subject should be captured and replicated as well. We present a method of generating a 3D model from a scan, as well as a method to...

Computer aided interactive narrative allows people to participate actively in a dynamically unfolding story, by playing a character or by exerting directorial control. Because of its potential for providing interesting stories as well as allowing user interaction, interactive narrative has been recognized as a promising tool for providing both educ...

This work presents a hierarchical framework that generates continuous gesture animation performance for virtual characters. As opposed to approaches that focus more on realizing individual gesture, the focus of this work is on the relation between gestures as part of an overall gesture performance. Following Calbris’ work [3], our approach is to st...