Shaun J. Canavan

University of South Florida | USF · Department of Computer Science & Engineering

In this paper we present a method for recognizing emotions using video and audio data captured from a mobile phone. A mobile application is also presented that captures audio and video data, which were used to predict emotion with a convolutional neural network. We show results of our deep network on images taken from the BP4D+ [11] database, and a...

Facial expression is central to human experience. Its efficient and valid measurement is a challenge that automated facial image analysis seeks to address. Most publically available databases are limited to 2D static images or video of posed facial behavior. Because posed and un-posed (aka "spontaneous") facial expressions differ along several dime...

Facial expression Recognition is a growing and important field that has applications in fields such as medicine, security, education, and entertainment. While there have been encouraging approaches that have shown accurate results on a wide variety of datasets, in many cases it is still a difficult problem to explain the results. To enable deployme...

Despite the promising performance of automated pain assessment methods, current methods suffer from performance generalization due to the lack of relatively large, diverse, and annotated pain datasets. Further, the majority of current methods do not allow responsible interaction between the model and user, and do not take different internal and ext...

p>Modeling of human emotion is a challenging problem that can require multiple signals types, as well as contextual information that has been obtained over time. Considering this, in this paper we present our approach, based on physiological signals, to the Emotion Physiology and Experience Collaboration (EPIC) challenge at Affective Computing & In...

p>Modeling of human emotion is a challenging problem that can require multiple signals types, as well as contextual information that has been obtained over time. Considering this, in this paper we present our approach, based on physiological signals, to the Emotion Physiology and Experience Collaboration (EPIC) challenge at Affective Computing & In...

Ethical affective computing (AC) requires maximizing the benefits to users while minimizing its harm to obtain trust from users. This requires responsible development and deployment to ensure fairness, bias mitigation, privacy preservation, and accountability. To obtain this, we require methodologies that can quantify, visualize, analyze, and mine...

It is important that interactive Affective Brain-Computer Interface (aBCI) applications support some degree of emotional intelligence. Many of the previous works that have explored the use of the Electroencephalogram (EEG) for the purpose of emotion recognition have focused on using data coming from the entire brain. However, the emotional activity...

In this paper we describe our approach to the arousal and valence track of the 3rd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). We extracted facial features using OpenFace and used them to train a multiple output random forest regressor. Our approach performed comparable to the baseline approach.

The quantified measurement of facial expressiveness is crucial to analyze human affective behavior at scale. Unfortunately, methods for expressiveness quantification at the video frame-level are largely unexplored, unlike the study of discrete expression. In this work, we propose an algorithm that quantifies facial expressiveness using a bounded, c...

We present an approach utilizing Topological Data Analysis to study the structure of face poses used in affective computing, i.e., the process of recognizing human emotion. The approach uses a conditional comparison of different emotions, both respective and irrespective of time, with multiple topological distance metrics, dimension reduction techn...

We present an approach utilizing Topological Data Analysis to study the structure of face poses used in affective computing, i.e., the process of recognizing human emotion. The approach uses a conditional comparison of different emotions, both respective and irrespective of time, with multiple topological distance metrics, dimension reduction techn...

In this paper, we propose a multimodal approach to emotion recognition using physiological signals by showing how these signals can be combined and used to accurately identify a wide range of emotions such as happiness, sadness, and pain. The proposed approach combines multiple signal types such as blood pressure, respiration, and pulse rate into o...

Currently, diagnosis of Autism Spectrum Disorder (ASD) is a lengthy, subjective process and machine learning has been shown to be able to accurately classify ASD, which can help take some of the subjectivity out of the diagnosis. Considering this, we propose a machine learning-based approach to classification of ASD, across age, that make use of su...

In this work, we propose to use 3D facial landmarks for the task of subject identification, over a range of expressed emotion. Landmarks are detected, using a Temporal Deformable Shape Model and used to train a Support Vector Machine (SVM), Random Forest (RF), and Long Short-term Memory (LSTM) neural network for subject identification. As we are in...

In this work, we address the importance of affect in automated pain assessment and the implications in real-world settings. To achieve this, we curate a new physiological dataset by merging the publicly available bioVid pain and emotion datasets. We then investigate pain level recognition on this dataset simulating participants' naturalistic affect...

The quantification of visual affect data (e.g. face images) is essential to build and monitor automated affect modeling systems efficiently. Considering this, this work proposes quantified facial Temporal-expressiveness Dynamics (TED) to quantify the expressiveness of human faces. The proposed algorithm leverages multimodal facial features by incor...

Detecting action units is an important task in face analysis, especially in facial expression recognition. This is due, in part, to the idea that expressions can be decomposed into multiple action units. In this paper we investigate the impact of action unit occurrence patterns on detection of action units. To facilitate this investigation, we revi...

In this paper, we propose a method for pain recognition by fusing physiological signals (heart rate, respiration, blood pressure, and electrodermal activity) and facial action units. We provide experimental validation that the fusion of these signals results in a positive impact to the accuracy of pain recognition, compared to using only one modali...

Landmark localization is an important first step towards geometric based vision research including subject identification. Considering this, we propose to use 3D facial landmarks for the task of subject identification, over a range of expressed emotion. Landmarks are detected, using a Temporal Deformable Shape Model and used to train a Support Vect...

To fully understand the complexities of human emotion, the integration of multiple physical features from different modalities can be advantageous. Considering this, we present an analysis of 3D facial data, action units, and physiological data as it relates to their impact on emotion recognition. We analyze each modality independently, as well as...

In this paper, we propose to detect forged videos, of faces, in online videos. To facilitate this detection, we propose to use smaller (fewer parameters to learn) convolutional neural networks (CNN), for a data-driven approach to forged video detection. To validate our approach, we investigate the FaceForensics public dataset detailing both frame-b...

In this paper, we propose to detect facial action units (AU) using 3D facial landmarks. Specifically, we train a 2D convolutional neural network (CNN) on 3D facial landmarks, tracked using a shape index-based statistical shape model, for binary and multi-class AU detection. We show that the proposed approach is able to accurately model AU occurrenc...

People with autism spectrum disorder (ASD) display impairments in social interaction and communication skills, as well as restricted interests and repetitive behaviors, which greatly affect daily life functioning. Current identification of ASD involves a lengthy process that requires an experienced clinician to assess multiple domains of functionin...

In this work, we present the synthesis of physiological and motion data to classify, detect and estimate affective state ahead of time (i.e. predict). We use raw physiological and motion signals to predict the next values of the signal following a temporal modeling scheme. The physiological signals are synthesized using a one-dimensional convolutio...

In this paper, we propose a new representation of human emotion through the fusion of physiological signals. Using the variance of these signals, the proposed method increases the effect of signals that contribute to the recognition accuracy, while decreasing the effect of those that do not. The new representation is a powerful approach to recogniz...

This paper explores the identification of smartphone users when certain samples collected while the subject felt happy, upset or stressed were absent or present. We employ data from 19 subjects using the StudentLife dataset, a dataset collected by researchers at Dartmouth College that was originally collected to correlate behaviors characterized by...

In this paper, we propose an approach, for sign language recognition, that makes use of a virtual reality headset to create an immersive environment. We show how features from data acquired by the Leap Motion controller, using an egocentric view, can be used to automatically recognize a user signed gesture. The Leap features are used along with a r...

In this paper, we propose a deformable synthesis model that can be used to synthesize data to train deep neural networks for the task of emotion recognition. This model is created through the use of 3D facial landmarks, which are then projected to the 2D image plane for training a deep network. We show that this model can accurately recognize a ran...

We propose an approach to the OMG-Empathy Challenge for predicting self-annotated, continuous values of valence within the range [-1,1]. We propose the fusion of hand-crafted and deep features, extracted from both actor and listener data, to predict these valence levels. The hand-crafted features include image level fusion, facial landmarks, and sp...

A new way to recognize human emotions in a real-time setting is presented. This is facilitated by the development of a new method for fusing emotion-based physiological data into one signal. These new signals retain the important variance found in different physiological data. This is facilitated by normalizing the signals and weighting the importa...

In 1997 Rosalind Picard introduced fundamental concepts of affect recognition [1]. Since this time, multimodal interfaces such as Brain-computer interfaces (BCIs), RGB and depth cameras, physiological wearables, multimodal facial data and physiological data have been used to study human emotion. Much of the work in this field focuses on a single mo...

Facial expression is central to human experience. Its efficiency and valid measurement are challenges that automated facial image analysis seeks to address. Most publically available databases are limited to 2D static images or video of posed facial behavior. Because posed and un-posed (aka “spontaneous”) facial expressions differ along several dim...

True immersion of a player within a game can only occur when the world simulated looks and behaves as close to reality as possible. This implies that the game must correctly read and understand, among other things, the player's focus, attitude toward the objects/persons in focus, gestures, and speech. In this paper, we proposed a novel system that...

In this paper, we propose a novel method for detecting and tracking landmark facial features on purely geometric 3D and 4D range models. Our proposed method involves fitting a new multi-frame constrained 3D temporal deformable shape model (TDSM) to range data sequences. We consider this a temporal based deformable model as we concatenate consecutiv...

This paper presents a novel dynamic curvature based approach (dynamic shape-index based approach) for 3D face analysis. This method is inspired by the idea of 2D dynamic texture and 3D surface descriptors. The dynamic texture (DT) based approaches [30][31][32] encode and model the local texture features in the temporal axis, and have achieved great...

This paper presents a novel dynamic curvature based approach (dynamic shape-index based approach) for 3D face analysis. This method is inspired by the idea of 2D dynamic texture and 3D surface descriptors. The dynamic texture (DT) based approaches [30][31][32] encode and model the local texture features in the temporal axis, and have achieved great...

3D facial representations have been widely used for face recognition. There has been intensive research on geometric matching and similarity measurement on 3D range data and 3D geometric meshes of individual faces. However, little investigation has been done on geometric measurement for 3D sketch models. In this paper, we study the 3D face recognit...

In this paper, we present a vision-based human-computer interaction system, which integrates control components using multiple gestures, including eye gaze, head pose, hand pointing, and mouth motions. To track head, eye, and mouth movements, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough loc...

Understanding how humans recognize face sketches drawn by artists is of significant value to both criminal investigators and researchers in computer vision, face biometrics and cognitive psychology. However, large scale experimental studies of hand-drawn face sketches are still very limited in terms of the number of artists, the number of sketches,...

Hand pointing has been an intuitive gesture for human interaction with computers. Big challenges are still posted for accurate estimation of finger pointing direction in a 3D space. In this paper, we present a novel hand pointing estimation system based on two regular cameras, which includes hand region detection, hand finger estimation, two views'...

A video sequence of a head moving across a large pose angle contains much richer information than a single-view image, and hence has greater potential for identification purposes. This paper explores and evaluates the use of a multi-frame fusion method to improve face recognition in the presence of strong shadow. The dataset includes videos of 257...

The past decade has witnessed a significant progress in biometric technologies, to a large degree, due to the availability of a wide variety of public databases that enable benchmark performance evaluations. In this paper, we describe a new database that includes: (i) rotating head videos of 259 subjects; (ii) 250 hand-drawn face sketches of 50 sub...

Dynamic modeling of facial appearances and sight directions are demanded for HCI and multimedia applications. Traditional approaches for face tracking and eye tracking from 2D videos do not involve explicit facial modeling. In this paper, we propose to use an explicit 3D model to model the dynamic facial appearance as well as the eye shape to estim...

This paper presents a face recognition study that implicitly utilizes the 3D information in 2D video sequences through multi-sample fusion. The approach is based on the hypothesis that continuous and coherent intensity variations in video frames caused by a rotating head can provide information similar to that of explicit shapes or range images. Th...