Shaun J. Canavan

Shaun J. Canavan
University of South Florida | USF · Department of Computer Science & Engineering

PhD

About

60
Publications
16,149
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
1,401
Citations
Citations since 2017
44 Research Items
1166 Citations
2017201820192020202120222023050100150200250
2017201820192020202120222023050100150200250
2017201820192020202120222023050100150200250
2017201820192020202120222023050100150200250
Introduction
Assistant Professor at the University of South Florida. Member of the Computer Vision and Pattern Recognition group. Research interests include HCI, Affective Computing, Computer Vision, Deep Learning, Biometrics, Image Processing, and VR/AR.
Additional affiliations
August 2017 - present
University of South Florida
Position
  • Professor (Assistant)
Description
  • Conduct research on affective computing, biometrics, HCI, VR/AR, image processing and computer vision
August 2017 - present
University of South Florida
Position
  • Professor (Assistant)
Description
  • Teach Affective Computing and Analysis of Algorithms.
August 2015 - May 2017
Binghamton University
Position
  • Professor (Assistant)
Description
  • Taught incoming freshman how to conduct research in real lab setting.
Education
June 2008 - May 2015
Binghamton University
Field of study
  • Computer Science

Publications

Publications (60)
Conference Paper
Full-text available
In this paper we present a method for recognizing emotions using video and audio data captured from a mobile phone. A mobile application is also presented that captures audio and video data, which were used to predict emotion with a convolutional neural network. We show results of our deep network on images taken from the BP4D+ [11] database, and a...
Conference Paper
Full-text available
Facial expression is central to human experience. Its efficient and valid measurement is a challenge that automated facial image analysis seeks to address. Most publically available databases are limited to 2D static images or video of posed facial behavior. Because posed and un-posed (aka "spontaneous") facial expressions differ along several dime...
Article
It is important that interactive Affective Brain-Computer Interface (aBCI) applications support some degree of emotional intelligence. Many of the previous works that have explored the use of the Electroencephalogram (EEG) for the purpose of emotion recognition have focused on using data coming from the entire brain. However, the emotional activity...
Preprint
In this paper we describe our approach to the arousal and valence track of the 3rd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). We extracted facial features using OpenFace and used them to train a multiple output random forest regressor. Our approach performed comparable to the baseline approach.
Preprint
Full-text available
The quantified measurement of facial expressiveness is crucial to analyze human affective behavior at scale. Unfortunately, methods for expressiveness quantification at the video frame-level are largely unexplored, unlike the study of discrete expression. In this work, we propose an algorithm that quantifies facial expressiveness using a bounded, c...
Article
We present an approach utilizing Topological Data Analysis to study the structure of face poses used in affective computing, i.e., the process of recognizing human emotion. The approach uses a conditional comparison of different emotions, both respective and irrespective of time, with multiple topological distance metrics, dimension reduction techn...
Preprint
Full-text available
We present an approach utilizing Topological Data Analysis to study the structure of face poses used in affective computing, i.e., the process of recognizing human emotion. The approach uses a conditional comparison of different emotions, both respective and irrespective of time, with multiple topological distance metrics, dimension reduction techn...
Chapter
In this paper, we propose a multimodal approach to emotion recognition using physiological signals by showing how these signals can be combined and used to accurately identify a wide range of emotions such as happiness, sadness, and pain. The proposed approach combines multiple signal types such as blood pressure, respiration, and pulse rate into o...
Chapter
Currently, diagnosis of Autism Spectrum Disorder (ASD) is a lengthy, subjective process and machine learning has been shown to be able to accurately classify ASD, which can help take some of the subjectivity out of the diagnosis. Considering this, we propose a machine learning-based approach to classification of ASD, across age, that make use of su...
Chapter
In this work, we propose to use 3D facial landmarks for the task of subject identification, over a range of expressed emotion. Landmarks are detected, using a Temporal Deformable Shape Model and used to train a Support Vector Machine (SVM), Random Forest (RF), and Long Short-term Memory (LSTM) neural network for subject identification. As we are in...
Preprint
Full-text available
In this work, we address the importance of affect in automated pain assessment and the implications in real-world settings. To achieve this, we curate a new physiological dataset by merging the publicly available bioVid pain and emotion datasets. We then investigate pain level recognition on this dataset simulating participants' naturalistic affect...
Preprint
The quantification of visual affect data (e.g. face images) is essential to build and monitor automated affect modeling systems efficiently. Considering this, this work proposes quantified facial Temporal-expressiveness Dynamics (TED) to quantify the expressiveness of human faces. The proposed algorithm leverages multimodal facial features by incor...
Preprint
Detecting action units is an important task in face analysis, especially in facial expression recognition. This is due, in part, to the idea that expressions can be decomposed into multiple action units. In this paper we investigate the impact of action unit occurrence patterns on detection of action units. To facilitate this investigation, we revi...
Conference Paper
In this paper, we propose a method for pain recognition by fusing physiological signals (heart rate, respiration, blood pressure, and electrodermal activity) and facial action units. We provide experimental validation that the fusion of these signals results in a positive impact to the accuracy of pain recognition, compared to using only one modali...
Preprint
Landmark localization is an important first step towards geometric based vision research including subject identification. Considering this, we propose to use 3D facial landmarks for the task of subject identification, over a range of expressed emotion. Landmarks are detected, using a Temporal Deformable Shape Model and used to train a Support Vect...
Preprint
To fully understand the complexities of human emotion, the integration of multiple physical features from different modalities can be advantageous. Considering this, we present an analysis of 3D facial data, action units, and physiological data as it relates to their impact on emotion recognition. We analyze each modality independently, as well as...
Preprint
In this paper, we propose to detect forged videos, of faces, in online videos. To facilitate this detection, we propose to use smaller (fewer parameters to learn) convolutional neural networks (CNN), for a data-driven approach to forged video detection. To validate our approach, we investigate the FaceForensics public dataset detailing both frame-b...
Preprint
In this paper, we propose to detect facial action units (AU) using 3D facial landmarks. Specifically, we train a 2D convolutional neural network (CNN) on 3D facial landmarks, tracked using a shape index-based statistical shape model, for binary and multi-class AU detection. We show that the proposed approach is able to accurately model AU occurrenc...
Article
People with autism spectrum disorder (ASD) display impairments in social interaction and communication skills, as well as restricted interests and repetitive behaviors, which greatly affect daily life functioning. Current identification of ASD involves a lengthy process that requires an experienced clinician to assess multiple domains of functionin...
Conference Paper
Full-text available
In this work, we present the synthesis of physiological and motion data to classify, detect and estimate affective state ahead of time (i.e. predict). We use raw physiological and motion signals to predict the next values of the signal following a temporal modeling scheme. The physiological signals are synthesized using a one-dimensional convolutio...
Conference Paper
Full-text available
In this paper, we propose a new representation of human emotion through the fusion of physiological signals. Using the variance of these signals, the proposed method increases the effect of signals that contribute to the recognition accuracy, while decreasing the effect of those that do not. The new representation is a powerful approach to recogniz...
Preprint
Full-text available
This paper explores the identification of smartphone users when certain samples collected while the subject felt happy, upset or stressed were absent or present. We employ data from 19 subjects using the StudentLife dataset, a dataset collected by researchers at Dartmouth College that was originally collected to correlate behaviors characterized by...
Conference Paper
Full-text available
In this paper, we propose an approach, for sign language recognition, that makes use of a virtual reality headset to create an immersive environment. We show how features from data acquired by the Leap Motion controller, using an egocentric view, can be used to automatically recognize a user signed gesture. The Leap features are used along with a r...
Conference Paper
Full-text available
In this paper, we propose a deformable synthesis model that can be used to synthesize data to train deep neural networks for the task of emotion recognition. This model is created through the use of 3D facial landmarks, which are then projected to the 2D image plane for training a deep network. We show that this model can accurately recognize a ran...
Conference Paper
Full-text available
We propose an approach to the OMG-Empathy Challenge for predicting self-annotated, continuous values of valence within the range [-1,1]. We propose the fusion of hand-crafted and deep features, extracted from both actor and listener data, to predict these valence levels. The hand-crafted features include image level fusion, facial landmarks, and sp...
Conference Paper
A new way to recognize human emotions in a real-time setting is presented. This is facilitated by the development of a new method for fusing emotion-based physiological data into one signal. These new signals retain the important variance found in different physiological data. This is facilitated by normalizing the signals and weighting the importa...
Conference Paper
In 1997 Rosalind Picard introduced fundamental concepts of affect recognition [1]. Since this time, multimodal interfaces such as Brain-computer interfaces (BCIs), RGB and depth cameras, physiological wearables, multimodal facial data and physiological data have been used to study human emotion. Much of the work in this field focuses on a single mo...
Article
Full-text available
Facial expression is central to human experience. Its efficiency and valid measurement are challenges that automated facial image analysis seeks to address. Most publically available databases are limited to 2D static images or video of posed facial behavior. Because posed and un-posed (aka “spontaneous”) facial expressions differ along several dim...
Article
Full-text available
True immersion of a player within a game can only occur when the world simulated looks and behaves as close to reality as possible. This implies that the game must correctly read and understand, among other things, the player's focus, attitude toward the objects/persons in focus, gestures, and speech. In this paper, we proposed a novel system that...
Conference Paper
Full-text available
In this paper, we propose a novel method for detecting and tracking landmark facial features on purely geometric 3D and 4D range models. Our proposed method involves fitting a new multi-frame constrained 3D temporal deformable shape model (TDSM) to range data sequences. We consider this a temporal based deformable model as we concatenate consecutiv...
Conference Paper
Full-text available
This paper presents a novel dynamic curvature based approach (dynamic shape-index based approach) for 3D face analysis. This method is inspired by the idea of 2D dynamic texture and 3D surface descriptors. The dynamic texture (DT) based approaches [30][31][32] encode and model the local texture features in the temporal axis, and have achieved great...
Conference Paper
Full-text available
This paper presents a novel dynamic curvature based approach (dynamic shape-index based approach) for 3D face analysis. This method is inspired by the idea of 2D dynamic texture and 3D surface descriptors. The dynamic texture (DT) based approaches [30][31][32] encode and model the local texture features in the temporal axis, and have achieved great...
Conference Paper
Full-text available
3D facial representations have been widely used for face recognition. There has been intensive research on geometric matching and similarity measurement on 3D range data and 3D geometric meshes of individual faces. However, little investigation has been done on geometric measurement for 3D sketch models. In this paper, we study the 3D face recognit...
Article
Full-text available
In this paper, we present a vision-based human-computer interaction system, which integrates control components using multiple gestures, including eye gaze, head pose, hand pointing, and mouth motions. To track head, eye, and mouth movements, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough loc...
Conference Paper
Full-text available
Understanding how humans recognize face sketches drawn by artists is of significant value to both criminal investigators and researchers in computer vision, face biometrics and cognitive psychology. However, large scale experimental studies of hand-drawn face sketches are still very limited in terms of the number of artists, the number of sketches,...
Conference Paper
Full-text available
Hand pointing has been an intuitive gesture for human interaction with computers. Big challenges are still posted for accurate estimation of finger pointing direction in a 3D space. In this paper, we present a novel hand pointing estimation system based on two regular cameras, which includes hand region detection, hand finger estimation, two views'...
Conference Paper
Full-text available
A video sequence of a head moving across a large pose angle contains much richer information than a single-view image, and hence has greater potential for identification purposes. This paper explores and evaluates the use of a multi-frame fusion method to improve face recognition in the presence of strong shadow. The dataset includes videos of 257...
Conference Paper
Full-text available
The past decade has witnessed a significant progress in biometric technologies, to a large degree, due to the availability of a wide variety of public databases that enable benchmark performance evaluations. In this paper, we describe a new database that includes: (i) rotating head videos of 259 subjects; (ii) 250 hand-drawn face sketches of 50 sub...
Conference Paper
Full-text available
Dynamic modeling of facial appearances and sight directions are demanded for HCI and multimedia applications. Traditional approaches for face tracking and eye tracking from 2D videos do not involve explicit facial modeling. In this paper, we propose to use an explicit 3D model to model the dynamic facial appearance as well as the eye shape to estim...
Conference Paper
Full-text available
This paper presents a face recognition study that implicitly utilizes the 3D information in 2D video sequences through multi-sample fusion. The approach is based on the hypothesis that continuous and coherent intensity variations in video frames caused by a rotating head can provide information similar to that of explicit shapes or range images. Th...

Network

Cited By

Projects

Projects (7)
Project
Participation in the OmG-Empathy challenge - https://www2.informatik.uni-hamburg.de/wtm/omgchallenges/omg_empathy.html
Project
Real-time emotion recognition feedback in AR using Meta 2.
Project
Recognize emotion by fusing multiple physiological signals that include heart rate, respiration, blood pressure, EDA, and EEG.