Mei Si

Rensselaer Polytechnic Institute | RPI · Department of Cognitive Science

This paper proposes an acoustic model for predicting the acoustical room characteristics from a running binaural signal. This is accomplished via training a convolutional neural network on a precedence effect model to extract the spatial locations of the direct sound source and its early reflections. The precedence effect model extends and modifies...

Efficient evaluation of dialogue agents is a major problem in conversational AI, with current research still relying largely on human studies for method validation. Recently, there has been a trend toward the use of automatic self-play and bot-bot evaluation as an approximation for human ratings of conversational systems. Such methods promise to al...

Recent advances in large-scale language modeling and generation have enabled the creation of dialogue agents that exhibit human-like responses in a wide range of conversational scenarios spanning a diverse set of tasks, from general chit-chat to focused goal-oriented discourse. While these agents excel at generating high-quality responses that are...

No PDF available ABSTRACT Dialogue systems have become a popular research medium as recent advances in task-oriented and open-domain systems combined with deep learning technologies have increased the potential for practical applications across many disciplines. One such vein of applications involves multi-modal dialogue systems deployed in interac...

No PDF available ABSTRACT A new model architecture is presented to predict room acoustical parameters from a running binaural signal. For this purpose, a deep neural network architecture is combined with a precedence effect model to extract the spatial and temporal locations of the direct signal and early reflections. The precedence effect model bu...

No PDF available ABSTRACT Learning to speak a second language involves a cycle of observation, mimicry, and feedback. A student observes a teacher, attempts to copy the teacher's performance, and then the teacher provides feedback on how the student performed. When the students' access to feedback is limited, so is their ability reinforce their lea...

Emotion plays a vital role in social interaction. Often, the speaker’s attitude is as essential as, if not more important than, his/her words for communication purposes. In this paper, we present experiments for using conversational context to help text-based emotion detection. We used data from the Dialogue Emotion Recognition Challenge – EmotionX...

This project aims to help people explore, understand, and rediscover the many-to-many relationships of content within library archives using multi-modal storytelling. This project builds upon an existing multi-modal storytelling system which is designed to help people explore large knowledge graphs by actively constructing narratives using informat...

Objectives Previous studies have shown that brief mindfulness trainings can have significant analgesic effects. However, the effects of the various components of mindfulness on pain analgesia are not well understood. The objective of this study was to examine the effects of two components of mindfulness interventions—attention and acceptance—on pai...

As the information age changes the world around us, humans are interacting with ever-increasing amounts of data. While it can be difficult to remember details from data, presenting it in a coordinated manner can aid in its retention and recall. In this work, we investigate whether placing a greater degree of emphasis on elements of coordination in...

Making analogies is an important rhetorical device. Analogies can not only help explain new concepts, but also make the expressions more interesting and creative. In our previous work, we explored an automated approach for making analogies using data from knowledge graphs where concepts are connected by their relationships. The algorithm computes h...

With the fast development of natural language processing and web technologies in recent years, the amount of knowledge/data available for everyone to explore has grown exponentially. This project is aimed at creating an interactive and proactive presentation agent. In particular, narrative and storytelling techniques are used for engaging the audie...

Narrative and dialogue are effective ways for engaging people and helping people organize and memorize information. In this work, we present an automated narration system that takes structured information from the Internet and tailors the presentation to a user using storytelling techniques. It is aimed at presenting the information as an interesti...

Storytelling has always been an effective and intuitive method of exchanging information. In today’s world of large, open, structured data, storytelling can benefit the ways in which people explore and consume such information. In this work, we investigate this potential. In particular, methods for creating multiple interweaving storylines are expl...

Facial expressions are important cues of people's emotions, attitudes, and intentions. Smiling is one of the most common facial expressions and is often associated with a welcoming, positive attitude. In social interactions, people sometimes fake their smiles for this effect, and phony fake smiles can often be detected by others. In this work, we e...

With the fast development of internet technology, people can have easy access to a massive amount of information. The goal of this project is to provide a personal assistant for helping people explore large network of information by using narrative technologies. We propose an automated narration system that takes structured information and tailors...

Background: With the increasing availability of Smartphones and wearable tracking devices, it is now feasible and affordable to apply such mobile devices to delivering mindfulness-based stress reduction (MBSR) and intermittent fasting (IF) to lower blood pressure, as traditional MBSR and IF incur the burden of commuting to the intervention sites fo...

Agent-based simulation of people’s behaviors and minds has become increasingly popular in recent years. It provides a research platform to simulate and compare alternative psychological and social theories, as well as to create virtual characters that can interact with people or among each other to provide pedagogical or entertainment effects. In t...

Second language acquisition after the students have learned their first language is a unique process. One major difference between learning a foreign language and one’s mother tongue is that second language learning is often facilitated with digital media, and in particular, through interacting with computers. This project is aimed at leveraging co...

Computer aided interactive narrative allows people to participate actively in a dynamically unfolding story, by playing a character or by exerting directorial control. Because of its potential for providing interesting stories as well as allowing user interaction, interactive narrative has been recognized as a promising tool for providing both educ...

The Ninth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE) was held October 14-18, 2013, at Northeastern University in Boston, Massachusetts. Workshops were held on the two days prior to the start of the main conference, giving attendees a chance to hold in-depth discussions on topics that complement t...

Young men (18-24) who have sex with men (YMSM) are at high risk for contracting HIV. Most existing HIV prevention interventions focus on changing intervening cognitive and deliberative processes or outcomes (e.g.,beliefs, norms, self-efficacy, intentions) to change behavior. Many MSM, however, guided by contextual cues in emotionally arousing scena...

Alas, there are akratic persons. We know this from the human case, and our knowledge is nothing new, since for instance Plato analyzed rather long ago a phenomenon all human persons, at one point or another, experience: (1) Jones knows that he ought not to - say - drink to the point of passing out, (2) earnestly desires that he not imbibe to this p...

Communication between robots and humans is a challenge in complex built environments. In this project, we are exploring how previous achievements in this area apply to human/robot communication within immersive virtual displays. In our concrete example, we host human-scale robots that interact with humans in our Collaborative-Research Augmented Imm...

Risky decisions (e.g., having unprotected sex; using methamphetamine) are often context dependent, automatic, and affectively based, but the contextual triggers that contribute to those risks may not be well understood by the individual him or herself. Virtual game simulations, designed to capture real-life situations, for example for PTSD sufferer...

Storytelling is interactive - storytellers seldom recite the same content, instead, they adapt the narrative to their audience. Storytellers both introduce new topics from time to time, and elaborate on the topics the audience has previously shown interest on. In our previous work, we created a digital storytelling system that simulates this proces...

Social facilitation refers to the phenomenon that the presence of others can improve one's performance in simple tasks. This work investigates whether/how we can achieve the same effect using virtual characters. In particular, we want to study the difference among using a digital character, a cognitive robot, and a real person to achieve this effec...

A number of studies have demonstrated that short-term meditation intervention can lead to greater tolerance and lower pain or distress ratings of experimentally induced pain. However, few attempts have been made to examine the effects of short-term mindfulness-based intervention on the tolerance and distress of pain, when delivered in a therapist-f...

Storytelling, when happening face to face, is a highly interactive process. A good storyteller attracts the audience by continuously observing their responses and adjusting the storytelling accordingly. The goal of this project is to simulate this process in digital storytellers. We created an automatic storytelling system that periodically estimat...

In this paper, we explored the use of features that represent body posture and movement for automatically detecting people's emotions in non-acted standing scenarios. We focused on four emotions that are often observed when people are playing video games: triumph, frustration, defeat, and concentration. The dataset consists of recordings of the rot...

Virtual reality has long been used for training simulations in fields from medicine to welding to vehicular operation, but simulations involving more complex cognitive skills present new design challenges. Foreign language learning, for example, is increasingly vital in the global economy, but computer-assisted education is still in its early stage...

A storyteller builds a narrative that captivates the audience, immersing them in the story. Storytelling is an interactive process. Though the listeners cannot affect what happens in the story, a good narrator observes the audience's responses and adjusts his/her storytelling accordingly. We present an automated storytelling agent that is aimed at...

Agent-based simulation of human social behavior has become increasingly important as a basic research tool to further our understanding of social behavior, as well as to create virtual social worlds used to both entertain and educate. A key factor in human social interaction is our beliefs about others as intentional agents, a Theory of Mind. How w...

This book constitutes the refereed proceedings of the 4th International Conference on Interactive Digital Storytelling, ICIDS 2011, held in Vancouver, Canada, in November/December 2011. The 17 full papers, 14 short papers and 16 poster papers were carefully reviewed and selected from 72 paper and poster submissions. In addition, the volume includes...

In recent years, there has been an increasing interest in exploring virtual environments and computer aided interactive stories as tools in developing health promotion and disease prevention interventions. Applications have been developed to address a range of health related conditions, including stress [1], risky behaviors [2] and post-traumatic s...

In order for the author to create his/her intended effects using interactive narratives, the user has to be able to understand his/her experience as designed by the author. In this paper, we argue that a key desideratum for interactive narrative frameworks is to model the characters' motivational consistency during the interaction. This work report...

Computer aided interactive narrative has received increasing attention in recent years. Automated directorial control that manages the development of the story in the face of user interaction is an important aspect of interactive narrative design. Most existing approaches lack an explicit model of the user. This limits the approaches’ ability of pr...

Cognitive appraisal theories, which link human emotional experience to their interpretations of events happening in the environment, are leading approaches to model emotions. In this paper, we investigate the computational modeling of appraisal in a multi-agent decision-theoretic framework using POMDP based agents. We illustrate how five key apprai...

Cognitive appraisal theories, which link human emotional experience to their interpretations of events happening in the environment, are leading approaches to model emotions. Cognitive appraisal theories have often been used both for simulating “real emotions” in virtual characters and for predicting the human user’s emotional experience to facilit...

Computer aided interactive drama has been widely applied for entertainment and pedagogy. Most existing approaches for authoring interactive drama use either story-centric or character-centric processes. In this work, we present a new framework that integrates both types of processes to support authoring. This framework uses a multi-agent system to...

Computer-based systems for interactive dramas allow the user to participate actively in the unfolding of a story in a virtual world. Various approaches have been explored for facilitating the human author in creating computer-based in- teractive dramas. Most of these approaches can be catego- rized as either story-centric or character-centric desig...

Interactive drama allows people to participate actively in a dynamically unfolding story, by playing a character or by exerting directorial control. One of the central challenges faced in the design of interactive dramas is how to ensure that the author’s goals for the user’s narrative experience are achieved in the face of the user’s actions in th...

Interactive drama allows people to participate actively in a dynamically unfolding story, by playing a character or by exerting di- rectorial control. One of the central challenges faced in the design of interactive dramas is how to ensure that the author's goals for the user's narrative experience are achieved in the face of the user's actions in...

To facilitate lifelike conversations with the human players in interactive dramas, virtual characters should follow similar conversational norms as those that govern human-human conversations. In this paper, we present a model of conversational norms in a decision-theoretic framework. This model is employed in the Thespian interactive drama system....

To facilitate lifelike conversations with the human players in interactive dramas, virtual characters should follow similar conversational norms as those that govern human-human conversations. In this paper, we present a model of conversational norms in a decision-theoretic framework. This model is employed in the Thespian interactive drama system....

There has been a growing interest in designing multi-agent based interactive dramas. A key research challenge faced in the design of these systems is to support open-ended user interaction while ensuring dramatic user experiences and consistent character personalities. Autonomous agents with reactive and planning abilities are well suited for reali...

Interactive drama is increasingly being used as a pedagogical tool in a wide variety of computer-based learning environments. However, the effort re- quired to build interactive dramas is quite significant. We built Thespian, an archi- tecture that supports faster development of IPDs, open-ended interaction, encoding of pedagogical goals and quanti...

Effective coordination of robots, agents and people promises to improve the safety, robustness and quality with which shared goals are achieved by harnessing the highly heterogeneous entities' diverse capabilities. Proxy-based integration architectures are emerging as a standard method for coordinating teams of heterogeneous entities. Such architec...

Effective coordination of robots, agents and people promises to improve the safety, robustness and quality with which shared goals are achieved by harnessing the highly heterogeneous entities' diverse capabilities. Proxy-based integration architectures are emerging as a standard method for coordinating teams of heterogeneous entities. Such architec...

Combining the unique capabilities of robots, agents and people (RAPs) promises to improve the safety, efficiency, reliability and cost at which some goals can be achieved, while allowing the achievement of other goals not previously achievable.

Social norms are shared rules of behavior that facilitate social interaction. Although norms are com-monly followed, other factors such as personality traits or more pressing goals may nevertheless lead to behavior that violates norms. To facilitate life-like social interaction with users in interactive dramas, the virtual characters ideally should...

The goal of the Tactical Language Training (TLT) system is to rapidly teach people how to speak in a foreign language and behave in a foreign culture. TLT combines an intelligent tutoring system with a 3D game where learners get to practice their skills in simulated social situations. In the tutoring system, the learners interact with a tutoring ag...