Giuseppe Boccignone

University of Milan | UNIMI · Department of Computer Science

We draw on a simulationist approach to the analysis of facially displayed emotions - e.g., in the course of a face-to-face interaction between an expresser and an observer. At the heart of such perspective lies the enactment of the perceived emotion in the observer. We propose a novel probabilistic framework based on a deep latent representation of...

Attention supports our urge to forage on social cues. Under certain circumstances, we spend the majority of time scrutinising people, markedly their eyes and faces, and spotting persons that are talking. To account for such behaviour, this paper develops a computational model for the deployment of gaze within a multimodal landscape, namely a conver...

Finding the underlying principles of social attention in humans seems to be essential for the design of the interaction between natural and artificial agents. Here, we focus on the computational modeling of gaze dynamics as exhibited by humans when perceiving socially relevant multimodal information. The audio-visual landscape of social interaction...

Remote photoplethysmography (rPPG) aspires to automatically estimate heart rate (HR) variability from videos in realistic environments. A number of effective methods relying on data-driven, model-based and statistical approaches have emerged in the past two decades. They exhibit increasing ability to estimate the blood volume pulse (BVP) signal upo...

The respiration rate (RR) is one of the physiological signals deserving monitoring for assessing human health and emotional states. However, traditional devices, such as the respiration belt to be worn around the chest, are not always a feasible solution (e.g., telemedicine, device discomfort). Recently, novel approaches have been proposed aiming a...

A core endeavour in current affective computing and social signal processing research is the construction of datasets embedding suitable ground truths to foster machine learning methods. This practice brings up hitherto overlooked intricacies. In this paper, we consider causal factors potentially arising when human raters evaluate the affect fluctu...

A principled approach to the analysis of eye movements for behavioural biometrics is laid down. The approach grounds in foraging theory, which provides a sound basis to capture the uniqueness of individual eye movement behaviour. We propose a composite Ornstein-Uhlenbeck process for quantifying the exploration/exploitation signature characterising...

In this study we propose an approach to assess the fear of heights through a 3D virtual reality environment. We show that an immersive scenario provides a suitable infrastructure to such purpose, when supported by related behavioural and physiological measurements. Our approach is grounded in the principled framework of constructed emotions. This a...

We present a simple, yet general method to detect fake videos displaying human subjects, generated via Deep Learning techniques. The method relies on gauging the complexity of heart rate dynamics as derived from the facial video streams through remote photoplethysmography (rPPG). Features analyzed have a clear semantics as to such physiological beh...

Symmetries, invariances and conservation equations have always been an invaluable guide in Science to model natural phenomena through simple yet effective relations. For instance, in computer vision, translation equivariance is typically a built-in property of neural architectures that are used to solve visual tasks; networks with computational lay...

Every year, in Europe alone, hundreds of workers die by falling from high height. This number could be greatly reduced by means of better training and quick detection of individuals with issues toward work at height. Workers proving to be less suited for the job can be subject to more intensive training or recruited for different positions. Unfortu...

Code used in the following papers: ----- 1) Boccignone, G., & Ferraro, M. (2004). Modelling gaze shift as a constrained random walk. Physica A: Statistical Mechanics and its Applications, 331(1-2), 207-218. ----- 2) Boccignone, G., & Ferraro, M. (2013). Feed and fly control of visual scanpaths for foveation image processing. Annals of telecommun...

Recent debates in the literature discuss commonalities between Attention-Deficit/Hyperactivity Disorder (ADHD) and Autism Spectrum Disorder (ASD) at multiple levels of putative causal networks. This debate requires systematic comparisons between these disorders that have been studied in isolation in the past, employing potential markers of each dis...

This paper presents a comprehensive framework for studying methods of pulse rate estimation relying on remote photoplethysmography (rPPG). There has been a remarkable development of rPPG techniques in recent years, and the publication of several surveys too, yet a sound assessment of their performance has been overlooked at best, whether not undeve...

Autism Spectrum Disorder (ASD) and Attention-Deficit/Hyperactivity Disorder (ADHD) represent two common neurodevelopmental disorders with considerable co-occurrence. Their comorbidity (ASD + ADHD) has been included in the latest diagnostic guidelines (DSM-V, 2013). The present study focuses on social visual attention that i) is a main aspect of soc...

Social interaction in individuals with Autism Spectrum Disorder (ASD) is characterized by qualitative impairments that highly impact quality of life. Bayesian theories in ASD frame an understanding of underlying mechanisms suggesting atypicalities in the evaluation of probabilistic links within the perceptual environment of the affected individual....

By and large, current visual attention models mostly rely, when considering static stimuli, on the following procedure. Given an image, a saliency map is computed, which, in turn, might serve the purpose of predicting a sequence of gaze shifts, namely a scanpath instantiating the dynamics of visual attention deployment. The temporal pattern of atte...

When automatic facial expression recognition is applied to video sequences of speaking subjects, the recognition accuracy has been noted to be lower than with video sequences of still subjects. This effect known as the speaking effect arises during spontaneous conversations, and along with the affective expressions the speech articulation process i...

In this Chapter we consider eye movements and, in particular, the resulting sequence of gaze shifts to be the observable outcome of a stochastic process. Crucially, we show that, under such assumption, a wide variety of tools become available for analyses and modelling beyond conventional statistical methods. Such tools encompass random walk analys...

By and large, current visual attention models mostly rely, when considering static stimuli, on the following procedure. Given an image , a saliency map is computed, which, in turn, might serve the purpose of predicting a sequence of gaze shifts, namely a scanpath instantiating the dynamics of visual attention deployment. The temporal pattern of att...

Despite the popularity that saliency models have gained in the computer vision community, they are most often conceived, exploited and benchmarked without taking heed of a number of problems and subtle issues they bring about. When saliency maps are used as proxies for the likelihood of fixating a location in a viewed scene, one such issue is the t...

Authors' pre-print version of an article that will be published in the Proceedings of the International Conference of Image analysis and Processing, September 2019, Trento, Italy: Image Analysis and Processing ICIAP 2019 - Lecture Notes in Computer Science, Springer Berlin / Heidelberg, 2019

We discuss a preliminary investigation on the feasibility of inferring traits of social participation from the observable behaviour of individuals involved in dyadic interactions. Trait inference relies on a stochastic model of the dynamics occurring in the individual core affect state-space. Results obtained on a publicly available interaction dat...

We address the deployment of perceptual attention to social interactions as displayed in conversational clips, when relying on multimodal information (audio and video). A probabilistic modelling framework is proposed that goes beyond the classic saliency paradigm while integrating multiple information cues. Attentional allocation is determined not...

We present a probabilistic generative model for tracking by prediction the dynamics of affective spacial expressions in videos. The model relies on Bayesian filter sampling of facial landmarks conditioned on motor action parameter dynamics; namely, trajectories shaped by an autoregressive Gaussian Process Latent Variable state-space. The analysis-b...

We address the deployment of perceptual attention to social interactions as displayed in conversational clips, when relying on multi-modal information (audio and video). A probabilistic modelling framework is proposed that goes beyond the classic saliency paradigm while integrating multiple information cues. Attentional allocation is determined not...

When humans are immersed in realistic, ecological situations that involve other humans, attention deployment strives for monitoring the behavior, intentions and emotions of others even in the absence of a given external task. Under such circumstances, the internal goal of the perceiver is to control attention so to maximize the implicit reward in f...

Understanding human gaze behaviour in social context, as along a face-to-face interaction, remains an open research issue which is strictly related to personality traits. In the effort to bridge the gap between available data and models, typical approaches focus on the analysis of spatial and temporal preferences of gaze deployment over specific re...

Understanding human gaze behaviour in social context, as along a face-to-face interaction, remains an open research issue which is strictly related to personality traits. In the effort to bridge the gap between available data and models, typical approaches focus on the analysis of spatial and temporal preferences of gaze deployment over specific re...

We present AMHUSE (A Multimodal dataset for HUmour SEnsing) along with a novel web-based annotation tool named DANTE (Di-mensional ANnotation Tool for Emotions). The dataset is the result of an experiment concerning amusement elicitation, involving 36 subjects in order to record the reactions in presence of 3 amusing and 1 neutral video stimuli. Ga...

In this note, we address the problem of simulating electromyographic signals arising from muscles involved in facial expressions - markedly those conveying affective information -, by relying solely on facial landmarks detected on video sequences. We propose a method that uses the framework of Gaussian Process regression to predict the facial elect...

In this note, we address the problem of simulating electromyographic signals arising from muscles involved in facial expressions -markedly those conveying affective information-, by relying solely on facial landmarks detected on video sequences. We propose a method that uses the framework of Gaussian Process regression to predict the facial electro...

The poster shows preliminary results on a complex Visual Search Paradigm

On Going Analysis on a Visual Search Task, proposed variables of Intra-subject Variability.

Science urges philosophy to be more empirical and philosophy urges science to be more reflective. This markedly occurred along the “discovery of the artificial” (CORDESCHI 2002): in the early days of Cybernetics and Artificial Intelligence (AI) researchers aimed at making machines more cognizant while setting up a framework to better understand hum...

In this paper a number of problems are considered which are related to the modelling of eye guidance under visual attention in a natural setting. From a crude discussion of a variety of available models spelled in probabilistic terms, it appears that current approaches in computational vision are hitherto far from achieving the goal of an active ob...

In this research we have analyzed functional magnetic resonance imaging (fMRI) signals of different networks in the brain under resting state condition. To such end, the dynamics of signal variation, have been conceived as a stochastic motion, namely it has been modelled through a generalized Langevin stochastic differential equation, which combine...

Within a Music Information Retrieval perspective, the goal of the study presented here is to investigate the impact on sound features of the musician's affective intention, namely when trying to intentionally convey emotional contents via expressiveness. A preliminary experiment has been performed involving 10 tuba players. The recordings have been...

Within a Music Information Retrieval perspective, the goal of the study presented here is to investigate the impact on sound features of the musician's affective intention, namely when trying to intentionally convey emotional contents via expressiveness. A preliminary experiment has been performed involving $10$ tuba players. The recordings have be...

In this research we have analyzed functional magnetic resonance imaging (fMRI) signals of different networks in the brain under resting state condition. To such end, the dynamics of signal variation, have been conceived as a stochastic motion, namely it has been modelled through a generalized Langevin stochastic differential equation, which combine...

In this paper we shall consider the problem of deploying attention to subsets of the video streams for collating the most relevant data and information of interest related to a given task. We formalize this monitoring problem as a foraging problem. We propose a probabilistic framework to model observer's attentive behavior as the behavior of a fora...

This note gives a preliminary account of the transcoding or rechanneling problem between different stimuli as it is of interest for the natural interaction or affective computing fields. By the consideration of a simple example, namely the color response of an affective lamp to a sensed facial expression, we frame the problem within an information-...

Complex systems are to be seen as typically having multiple levels of organization. For instance, in the behavioural and cognitive sciences, there has been a long lasting trend, promoted by the seminal work of David Marr, putting focus on three distinct levels of analysis: the computational level, accounting for the What and Why issues, the algorit...

Contemporary developments in neuroscience and psychology suggest that scientists are likely to deal with a multiplicity of levels, where each of the different levels entails laws of behavior appropriate to that level (Berntson et al., 2012). Also, gathering and modeling data at the different levels of analysis is not sufficient: the integration of...

In this article we address the issue of adopting a local sparse coding representation (Histogram of Sparse Codes), in a part-based framework for inferring the locations of facial land-marks. The rationale behind this approach is that unsupervised learning of sparse code dictionaries from face data can be an effective approach to cope with such a ch...

Understanding the mental state of other people is an important skill for intelligent agents and robots to operate within social environments. However, the mental processes involved in ‘mind-reading’ are complex. One explanation of such processes is Simulation Theory — it is supported by a large body of neuropsychological research. Yet, determining...

We introduce a model of attentional eye guidance based on the rationale that the deployment of gaze is to be considered in the context of a general action-perception loop relying on two strictly intertwined processes: sensory processing, depending on current gaze position, identifies sources of information that are most valuable under the given tas...

Decoding mental states from the pattern of neural activity or overt behavior is an intensely pursued goal. Here we applied machine learning to detect expertise from the oculomotor behavior of novice and expert billiard players during free viewing of a filmed billiard match with no specific task, and in a dynamic trajectory prediction task involving...

Visual attention guides our gaze to relevant parts of the viewed scene, yet the moment-to-moment relocation of gaze can be different among observers even though the same locations are taken into account. Surprisingly, the variability of eye movements has been so far overlooked by the great majority of computational models of visual attention. In th...

Code implementing the model discussed in the paper Ecological Sampling of Gaze Shifts by Giuseppe Boccignone and Mario Ferraro, IEEE TRANSACTIONS ON CYBERNETICS, VOL. 44, NO. 2, FEBRUARY 2014 Abstract—Visual attention guides our gaze to relevant parts of the viewed scene, yet the moment-to-moment relocation of gaze can be different among observer...

The code is a simple Demo of the Ecological Sampling (ES) method, which generates gaze shifts on video clips (frame sequences). It is a baseline implementation of the Ecological Sampling model described in Boccignone & Ferraro [1], a stochastic model of eye guidance The gaze shift mechanism is conceived as an active random sampling that the "fora...

This note introduces a visual attention model of text localization in real-world scenes. The core of the model built upon the proto-object concept is discussed. It is shown how such dynamic mid-level representation of the scene can be derived in the framework of an action-perception loop engaging salience, text information value computation, and ey...

Braitenberg's neuroanatomical research was carried out in the endeavour to identify the network structures specific to a given part of the brain. But, more generally, his work concerns the functional interpretation of brain structures. Yet, since computers came into play in the 1950s, Braitenberg was well aware of their potential for bridging the g...

Is any unified theory of brain function possible? Following a line of thought dating back to the early cybernetics (see, e.g., Cordeschi, 2002), Clark (in press) has proposed the action-oriented Hierarchical Predictive Coding (HPC) as the account to be pursued in the effort of gaining the “Grand Unified Theory of the Mind”—or “painting the big pict...

Foveation-based processing and communication systems can exploit a more efficient representation of images and videos by removing or reducing visual information redundancy, provided that the sequence of foveation points, the visual scanpath, can be determined. However, one point that is neglected by the great majority of foveation models is the “no...

The ability to predict, given an image or a video, where a human might fixate elements of a viewed scene has long been of interest in the vision community. However, one point that is not addressed by the great majority of computational models is the variability exhibited by different observers when viewing the same scene, or even by the same subjec...

Heart rate variability (HRV) is an important measure of sympathetic and parasympathetic functions of the autonomic nervous system and a key indicator of cardiovascular condition. This paper proposes a novel method to investigate HRV, namely by modelling it as a linear combination of Gaussians. Results show that three Gaussians are enough to describ...

The ability to predict, given an image or a video, where a human might fixate elements of a viewed scene has long been of interest in the vision community. In this note we propose a different view of the gaze-shift mechanism as that of a motor system implementation of an active random sampling strategy that the Human Visual System has evolved in or...