On mapping emotional states and implicit gestures to sonification output from the 'Intangible Musical Instrument'

Conference Paper · July 2016with 73 Reads
DOI: 10.1145/2948910.2948950
Conference: 3rd International Symposium on Movement and Computing
Abstract
Sonification is an interdisciplinary field of research, aiming at generating sound from data based on systematic, objective and reproducible transformations. Towards this direction, expressive gestures play an important role in music performances facilitating the artistic perception by the audience. Moreover, emotions are linked with music, as sound has the ability to evoke emotions. In this vein, a combinatory approach which aims at gesture and emotion sonification in the context of music composition and performance is presented here. The added value of the proposed system is that both gesture and emotion are able to continuously manipulate the reproduced sound in real-time.

Do you want to read the rest of this conference paper?

Request full-text
Request Full-text Paper PDF
  • ... One possible way to model inspiration is to use an observer's environment or culture as an inspiring source. Research in electroencephalogram-based affective computing (i.e., reading brain waves) suggests that computers may soon be able to perceive an observer's emotional state beyond those of their human counterparts (Volioti et al. 2016). Alternatively, inspiration could be modeled using sentiment analysis in a variety of online domains. ...
    Conference Paper
    Full-text available
    A common framework is helpful for effective evaluation , collaboration, and incremental development of creative systems. The Hierarchical Bayesian Program Learning (HBPL) framework was recently shown to be highly effective at learning human-level concepts, achieving new standards of performance in one-shot classification, parsing, and generation of handwritten characters. We argue that the HBPL framework is well-suited for modeling creative artefacts in general, one reason being that it allows explicit modeling of intention , structure, and substructure. Furthermore, the major challenge to the HBPL framework, namely how joint distributions should be factored, focuses system design-ers' attention on the philosophical debates that occur among artists themselves, suggesting that the HBPL framework might also serve as a more precise scaffolding for such debates. We demonstrate the framework's efficacy using lyrical composition as a specific example. In addition to being able to generate novel artefacts , we illustrate how HBPL models can be used to incorporate creative knowledge in broader applications including recommendation systems.
  • Conference Paper
    Emotion plays an important role in the interaction between humans as emotion is fundamental to human experience, influencing cognition, perception, learning communication, and even rational decision-making. Therefore, studying emotion is indispensable. This paper aims at finding the relationships between EEG signals and human emotions based on emotion recognition experiments that are conducted using the commercial Emotiv EPOC headset to record EEG signals while participants are watching emotional movies. Alpha, beta, delta and theta bands filtered from the recorded EEG signals are used to train and evaluate classifiers with different learning techniques including Support Vector Machine, k-Nearest Neighbour, Naïve Bayes and AdaBoost.M1. Our experimental results show that we can use the Emotiv headset for emotion recognition and that the AdaBoost.M1 technique and the theta band provide the highest recognition rates.
  • Conference Paper
    There is a growing interest in ‘unlocking’ the motor skills of expert musicians. Motivated by this need, the main objective of this paper is to present a new way of modeling expressive gesture variations in musical performance. For this purpose, the 3D gesture recognition engine ‘x2Gesture’ (eXpert eXpressive Gesture) has been developed, inspired by the Gesture Variation Follower, which is initially designed and developed at IRCAM in Paris and then extended at Goldsmiths College in London. x2Gesture supports both learning of musical gestures and live performing, through gesture sonification, as a unified user experience. The deeper understanding of the expressive gestural variations permits to define the confidence bounds of the expert’s gestures, which are used during the decoding phase of the recognition. The first experiments show promising results in terms of recognition accuracy and temporal alignment between template and performed gesture, which leads to a better fluidity and immediacy and thus gesture sonification.
  • Article
    Full-text available
    This article presents a gesture recognition/adaptation system for human--computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a Sequential Monte Carlo inference technique. Contrary to standard template-based methods based on dynamic programming, such as Dynamic Time Warping, the algorithm has an adaptation process that tracks gesture variation in real time. The method continuously updates, during execution of the gesture, the estimated parameters and recognition results, which offers key advantages for continuous human--machine interaction. The technique is evaluated in several different ways: Recognition and early recognition are evaluated on 2D onscreen pen gestures; adaptation is assessed on synthetic data; and both early recognition and adaptation are evaluated in a user study involving 3D free-space gestures. The method is robust to noise, and successfully adapts to parameter variation. Moreover, it performs recognition as well as or better than nonadapting offline template-based methods.
  • Article
    Full-text available
    Expert musical performance is rich with movements that facilitate performance accuracy and expressive communication. As in sports or rehabilitation, these movements can be sonified for analysis or to provide realtime feedback to the performer. Expressive move-ment is different however in that movements are not strictly goal-oriented and highly idiosyncratic. Drawing upon insights from the literature, this paper argues that for expressive movement in mu-sic, sonifications should be evaluated based upon their capacity to convey information that is relevant to visual perception and the relationship of movement, performer and music. Benefits of the synchronous presentation of sonification and music are identified, and examples of this display type are provided.
  • Book
    Music's ability to express and arouse emotions is a mystery that has fascinated both experts and laymen at least since ancient Greece. The predecessor to this book, Motion and Emotion (OUP, 2001) was critically and commercially successful and stimulated much further work in this area. In the years since the publication of that book, empirical research in this area has blossomed, and the successor to Music and Emotion reflects the considerable activity in this area. The Handbook of Music and Emotion offers an 'up-to-date' account of this vibrant domain. It provides comprehensive coverage of the many approaches that may be said to define the field of music and emotion, in all its breadth and depth. The first section offers multi-disciplinary perspectives on musical emotions from philosophy, musicology, psychology, neurobiology, anthropology, and sociology. The second section features methodologically-oriented chapters on the measurement of emotions via different channels (e.g., self report, psychophysiology, neuroimaging). Sections three and four address how emotion enters into different aspects of musical behavior, both the making of music and its consumption. Section five covers developmental, personality, and social factors. Section six describes the most important applications involving the relationship between music and emotion. In a final commentary, the editors comment on the history of the field, summarize the current state of affairs, as well as propose future directions for the field. The only book of its kind, the Handbook of Music and Emotion will fascinate music psychologists, musicologists, music educators, philosophers, and others with an interest in music and emotion (e.g. in marketing, health, engineering, film, and the game industry). It will be a valuable resource for established researchers in the field, a developmental aid for early-career researchers and postgraduate research students, and a compendium to assist students at various levels. In addition, as with its predecessor, it will also interest from practicing musicians and lay readers fascinated by music and emotion.
  • Article
    Full-text available
    Numerous studies have investigated whether music can reliably convey emotions to listeners, and-if so-what musical parameters might carry this information. Far less attention has been devoted to the actual contents of the communicative process. The goal of this article is thus to consider what types of emotional content are possible to convey in music. I will argue that the content is mainly constrained by the type of coding involved, and that distinct types of content are related to different types of coding. Based on these premises, I suggest a conceptualization in terms of "multiple layers" of musical expression of emotions. The "core" layer is constituted by iconically-coded basic emotions. I attempt to clarify the meaning of this concept, dispel the myths that surround it, and provide examples of how it can be heuristic in explaining findings in this domain. However, I also propose that this "core" layer may be extended, qualified, and even modified by additional layers of expression that involve intrinsic and associative coding. These layers enable listeners to perceive more complex emotions-though the expressions are less cross-culturally invariant and more dependent on the social context and/or the individual listener. This multiple-layer conceptualization of expression in music can help to explain both similarities and differences between vocal and musical expression of emotions.
  • Article
    Full-text available
    We propose to use real-time EEG signal to classify happy and unhappy emotions elicited by pictures and classical music. We use PSD as a feature and SVM as a classifier. The average accuracies of subject-dependent model and subject-independent model are approximately 75.62% and 65.12%, respectively. Considering each pair of channels, temporal pair of channels (T7 and T8) gives a better result than the other area. Considering different frequency bands, high-frequency bands (Beta and Gamma) give a better result than low-frequency bands. Considering different time durations for emotion elicitation, that result from 30 seconds does not have significant difference compared with the result from 60 seconds. From all of these results, we implement real-time EEG-based happiness detection system using only one pair of channels. Furthermore, we develop games based on the happiness detection system to help user recognize and control the happiness.
  • Article
    The study of music perception has focused almost exclusively on sound, ignoring the role of seeing the performer's body movements. Whilst anecdotes frequently refer to the importance of the performer's movements, there is scant psychological evidence to support this finding. The closest equivalent work in visual event perception research has shown that covert mental dispositions (for instance, an intention to deceive an observer) are specified in body movements, and therefore provide important information for the observer.. With these findings in mind, this article investigates the information conveyed by the movements of a musical performer when s/he is asked to play-the same piece in three different expressive manners. These performance manners are presented to observers in three modes: vision alone, sound alone and sound and vision together to investigate the relative contributions of the different perceptual modes. The results reveal that not only is vision a useful source of information about manner, but that it actually more clearly specifies manner than the other modes. These findings emphasise the need to consider visual as well as sound information in psychological enquiries into music perception.
  • Article
    Full-text available
    A growing number of affective computing researches recently developed a computer system that can recognize an emotional state of the human user to establish affective human-computer interactions. Various measures have been used to estimate emotional states, including self-report, startle response, behavioral response, autonomic measurement, and neurophysiologic measurement. Among them, inferring emotional states from electroencephalography (EEG) has received considerable attention as EEG could directly reflect emotional states with relatively low costs and simplicity. Yet, EEG-based emotional state estimation requires well-designed computational methods to extract information from complex and noisy multichannel EEG data. In this paper, we review the computational methods that have been developed to deduct EEG indices of emotion, to extract emotion-related features, or to classify EEG signals into one of many emotional states. We also propose using sequential Bayesian inference to estimate the continuous emotional state in real time. We present current challenges for building an EEG-based emotion recognition system and suggest some future directions.
  • Article
    Full-text available
    Factor-analytic evidence has led most psychologists to describe affect as a set of dimensions, such as displeasure, distress, depression, excitement, and so on, with each dimension varying independently of the others. However, there is other evidence that rather than being independent, these affective dimensions are interrelated in a highly systematic fashion. The evidence suggests that these interrelationships can be represented by a spatial model in which affective concepts fall in a circle in the following order: pleasure (0), excitement (45), arousal (90), distress (135), displeasure (180), depression (225), sleepiness (270), and relaxation (315). This model was offered both as a way psychologists can represent the structure of affective experience, as assessed through self-report, and as a representation of the cognitive structure that laymen utilize in conceptualizing affect. Supportive evidence was obtained by scaling 28 emotion-denoting adjectives in 4 different ways: R. T. Ross's (1938) technique for a circular ordering of variables, a multidimensional scaling procedure based on perceived similarity among the terms, a unidimensional scaling on hypothesized pleasure–displeasure and degree-of-arousal dimensions, and a principal-components analysis of 343 Ss' self-reports of their current affective states. (70 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
  • Article
    Full-text available
    This paper considers the issues involved in the design of electronic and computer interfaces, specifically mapping - the designed link between an instrument's playing interface and its sound source. It defines the problem area, reviews the literature, and gives examples of specific system mappings. A general model is presented, with the aim of providing a framework for future discussions on what makes an effective mapping. Several guidelines for mapping strategies are given, based on existing work.
  • Article
    Full-text available
    This paper presents a quantitative exploratory analysis of expressive movements of several clarinetists playing standard concert solo repertoire. It builds on our previous work involving the analysis of clarinetists' ancillary gestures by focusing on the timing of various performance manners, the relation that ancillary gestures have to the musical score, the different styles of expressive movements among performers, and the perception of these movements by the audience.
  • Chapter
    Full-text available
    This chapter presents a general framework for gesture-controlled audio processing. The gesture parameters are assumed to be multi-dimensional temporal profiles obtained from movement or sound capture systems. The analysis is based on machine learning techniques, comparing the incoming dataflow with stored templates. The mapping procedures between the gesture and the audio processing include a specific method we called temporal mapping. In this case, the temporal evolution of the gesture input is taken into account in the mapping process. We describe an example of a possible use of the framework that we experimented with in various contexts, including music and dance performances, music pedagogy and installations.
  • Article
    Full-text available
    Researchers have wondered how the brain creates emotions since the early days of psychological science. With a surge of studies in affective neuroscience in recent decades, scientists are poised to answer this question. In this target article, we present a meta-analytic summary of the neuroimaging literature on human emotion. We compare the locationist approach (i.e., the hypothesis that discrete emotion categories consistently and specifically correspond to distinct brain regions) with the psychological constructionist approach (i.e., the hypothesis that discrete emotion categories are constructed of more general brain networks not specific to those categories) to better understand the brain basis of emotion. We review both locationist and psychological constructionist hypotheses of brain-emotion correspondence and report meta-analytic findings bearing on these hypotheses. Overall, we found little evidence that discrete emotion categories can be consistently and specifically localized to distinct brain regions. Instead, we found evidence that is consistent with a psychological constructionist approach to the mind: A set of interacting brain regions commonly involved in basic psychological operations of both an emotional and non-emotional nature are active during emotion experience and perception across a range of discrete emotion categories.
  • Article
    Full-text available
    Since emotions play an important role in the daily life of human beings, the need and importance of automatic emotion recognition has grown with increasing role of human computer interface applications. Emotion recognition could be done from the text, speech, facial expression or gesture. In this paper, we concentrate on recognition of “inner” emotions from electroencephalogram (EEG) signals. We propose real-time fractal dimension based algorithm of quantification of basic emotions using Arousal-Valence emotion model. Two emotion induction experiments with music stimuli and sound stimuli from International Affective Digitized Sounds (IADS) database were proposed and implemented. Finally, the real-time algorithm was proposed, implemented and tested to recognize six emotions such as fear, frustrated, sad, happy, pleasant and satisfied. Real-time applications were proposed and implemented in 3D virtual environments. The user emotions are recognized and visualized in real time on his/her avatar adding one more so-called “emotion dimension” to human computer interfaces. An EEG-enabled music therapy site was proposed and implemented. The music played to the patients helps them deal with problems such as pain and depression. An EEG-based web-enable music player which can display the music according to the user’s current emotion states was designed and implemented.
  • Article
    Full-text available
    A new method for analysing nonlinear and non-stationary data has been developed. The key part of the method is the `empirical mode decomposition' method with which any complicated data set can be decomposed into a finite and often small number of 'intrinsic mode functions' that admit well-behaved Hilbert transforms. This decomposition method is adaptive, and, therefore, highly efficient. Since the decomposition is based on the local characteristic time scale of the data, it is applicable to nonlinear and non-stationary processes. With the Hilbert transform, the 'instrinic mode functions' yield instantaneous frequencies as functions of time that give sharp identifications of imbedded structures. The final presentation of the results is an energy-frequency-time distribution, designated as the Hilbert spectrum. In this method, the main conceptual innovations are the introduction of `intrinsic mode functions' based on local properties of the signal, which make the instantaneous frequency meaningful; and th
  • Article
    Full-text available
    Research indicates that people value music primarily because of the emotions it evokes. Yet, the notion of musical emotions remains controversial, and researchers have so far been unable to offer a satisfactory account of such emotions. We argue that the study of musical emotions has suffered from a neglect of underlying mechanisms. Specifically, researchers have studied musical emotions without regard to how they were evoked, or have assumed that the emotions must be based on the "default" mechanism for emotion induction, a cognitive appraisal. Here, we present a novel theoretical framework featuring six additional mechanisms through which music listening may induce emotions: (1) brain stem reflexes, (2) evaluative conditioning, (3) emotional contagion, (4) visual imagery, (5) episodic memory, and (6) musical expectancy. We propose that these mechanisms differ regarding such characteristics as their information focus, ontogenetic development, key brain regions, cultural impact, induction speed, degree of volitional influence, modularity, and dependence on musical structure. By synthesizing theory and findings from different domains, we are able to provide the first set of hypotheses that can help researchers to distinguish among the mechanisms. We show that failure to control for the underlying mechanism may lead to inconsistent or non-interpretable findings. Thus, we argue that the new framework may guide future research and help to resolve previous disagreements in the field. We conclude that music evokes emotions through mechanisms that are not unique to music, and that the study of musical emotions could benefit the emotion field as a whole by providing novel paradigms for emotion induction.
  • Article
    Multisensory integrated expressive environments is a framework for mixed reality applications in the performing arts such as interactive dance, music, or video installations. MIEEs address the expressive aspects of nonverbal human communication. We present the multilayer conceptual framework of MIEEs, algorithms for expressive content analysis and processing, and MIEEs-based art applications.
  • Article
    Full-text available
    Two channels have been distinguished in human interaction: one transmits explicit messages, which may be about anything or nothing; the other transmits implicit messages about the speakers themselves. Both linguistics and technology have invested enormous efforts in understanding the first, explicit channel, but the second is not as well understood. Understanding the other party's emotions is one of the key tasks associated with the second, implicit channel. To tackle that task, signal processing and analysis techniques have to be developed, while, at the same time, consolidating psychological and linguistic analyses of emotion. This article examines basic issues in those areas. It is motivated by the PKYSTA project, in which we aim to develop a hybrid system capable of using information from faces and voices to recognize people's emotions
  • Sonification of Emotions II: Live music in a pediatric hospital
    • Constanza Preti
    • Emery Schubert
    Constanza Preti and Emery Schubert. 2011. Sonification of Emotions II: Live music in a pediatric hospital. In Proceedings of the 17th International Conference on Auditory Display (ICAD2011), 20-23.
  • Erdal Yilmaz, Fabien Moutarde and Athanasios Manitsaris
    • Christina Volioti
    • Edgar Hemery
    • Sotiris Manitsaris
    • Vicky Tsekouropoulou
    Christina Volioti, Edgar Hemery, Sotiris Manitsaris, Vicky Tsekouropoulou, Erdal Yilmaz, Fabien Moutarde and Athanasios Manitsaris. 2015. Music Gestural Skills Development Engaging Teachers, Learners and Expert Performers. Procedia Manufacturing, 3, 1543-15.
  • Musical gestures and embodied cognition
    • Marc Leman
    Marc Leman. 2012. Musical gestures and embodied cognition. In Proceedings of the Journées d'informatique musicale (JIM), 5-7.
  • Integrating Emotional Data into Music Performance: Two Audio Environments for the Emotional Imaging Composer
    • R Michael
    • Ian Winters
    • Marcelo M Hattwick
    • Wanderley
    Michael R. Winters, Ian Hattwick and Marcelo M. Wanderley. 2013. Integrating Emotional Data into Music Performance: Two Audio Environments for the Emotional Imaging Composer, In Proceedings of the 3rd International Conference on Music and Emotion (ICME3).