Roddy Cowie

• Queen's University Belfast

Affective computing develops systems, which recognize or influence aspects of human life related to emotion, including feelings and attitudes. Significant potential for both good and harm makes it ethically sensitive, and trying to strike sound balances is challenging. Common images of the issues invite oversimplification and offer a limited unders...

Computational research underscores the complex abilities underlying humor. Two decades of work have achieved substantial progress in some areas, notably systems that make jokes; detecting and generating laughter; and using irony in interactions. Sophisticated evaluations clarify both strengths and limitations. The achievements illuminate specific a...

The ninth Audio-Visual Emotion Challenge and workshop AVEC 2019 was held in conjunction with ACM Multimedia'19. This year, the AVEC series addressed major novelties with three distinct tasks: State-of-Mind Sub-challenge (SoMS), Detecting Depression with Artificial Intelligence Sub-challenge (DDS), and Cross-cultural Emotion Sub-challenge (CES). The...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) 'State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition' is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly...

Representing computationally everyday emotional states is a challenging task and, arguably, one of the most fundamental for affective computing. Standard practice in emotion annotation is to ask humans to assign a value of intensity or a class value to each emotional behavior they observe. Psychological theories and evidence from multiple disciplin...

The eighth Audio-Visual Emotion Challenge and workshop AVEC 2018 was held in conjunction with ACM Multimedia'18. This year, the AVEC series addressed major novelties with three distinct sub-challenges: bipolar disorder classification, cross-cultural dimensional emotion recognition, and emotional label generation from individual ratings. The Bipolar...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2018) "Bipolar disorder, and cross-cultural affect recognition'' is the eighth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly under the same conditions...

The seventh Audio-Visual Emotion Challenge and workshop AVEC 2017 was held in conjunction with ACM Multimedia'17. This year, the AVEC series addresses two distinct sub-challenges: emotion recognition and depression detection. The Affect Sub-Challenge is based on a novel dataset of human-human interactions recorded 'in-the-wild', whereas the Depress...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2017) "Real-life depression, and affect" will be the seventh competition event aimed at comparison of multimedia processing and machine learning methods for automatic audiovisual depression and emotion analysis, with all participants competing under strictly the same conditions. The goal of the...

Background: The aptitude to infer the shape of 3-D structures, such as internal organs from 2-D monitor displays, in image guided endoscopic and laparoscopic procedures varies. We sought both to validate a computer-generated task Pictorial Surface Orientation (PicSOr), which assesses this aptitude, and to identify norm referenced scores. Methods:...

Automatic emotion recognition in realistic domains is a challenging task given the subtle expressive behaviors that occur during human interactions. The challenges start with noisy emotional descriptors provided by multiple evaluators, which are characterized by low interevaluator agreement. Studies have suggested that evaluators are more consisten...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2016) "Depression, Mood and Emotion" will be the sixth competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological depression and emotion analysis, with all participants competing under strictly the same conditions. Th...

The sixth Audio-Visual Emotion Challenge and workshop AVEC 2016 was held in conjunction ACM Multimedia'16. This year the AVEC series addresses two distinct sub-challenges, multi-modal emotion recognition and audio-visual depression detection. Both sub-challenges are in a way a return to AVEC's past editions: the emotion sub-challenge is based on th...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2016) "Depression, Mood and Emotion" will be the sixth competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological depression and emotion analysis, with all participants competing under strictly the same conditions. Th...

We present the first Audio-Visual+ Emotion recognition Challenge and workshop (AV+EC 2015) aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological emotion analysis. This is the 5th event in the AVEC series, but the very first Challenge that bridges across audio, video and physiological...

The fifth Audio-Visual Emotion Challenge and workshop AVEC 2015 was held in conjunction ACM Multimedia'15. Like the previous editions of AVEC, the workshop/challenge addresses the detection of affective signals represented in audio-visual data in terms of high-level continuous dimensions. A major novelty was further introduced this year by the incl...

We present the first Audio-Visual+ Emotion recognition Challenge and workshop (AV+EC 2015) aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological emotion analysis. This is the 5th event in the AVEC series, but the very first Challenge that bridges across audio, video and physiological...

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence, arousal and dominance. In addition to structured self-report questionnaires, psychologists and psychiatrists use in their evalua...

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence, arousal and dominance. In addition to structured self-report questionnaires, psychologists and psychiatrists use in their evalua...

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone, since it has to be adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior...

Recording and annotating a multimodal database of natural expressivity is a task that requires careful planning and implementation, before even starting to apply feature extraction and recognition algorithms. Requirements and characteristics of such databases are inherently different than those of acted behaviour, both in terms of unconstrained exp...

The third Audio-Visual Emotion Challenge and workshop AVEC 2013 will be held in conjunction ACM Multimedia'13. Like the 2012 edition of AVEC, the workshop/challenge addresses the interpretation of social signals represented in both audio and video in terms of the high-level continuous dimensions arousal and valence, but importantly this year the da...

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence and arousal. In addition, psychologists and psychiatrists take the observation of expressive facial and vocal cues into account w...

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone: it is adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates...

GTrace is a program which allows users to create a 'trace' that specifies how they see emotion rising and falling over time. They respond by using a mouse to move a cursor in a 1-D window which appears beside the material to be rated. Users can select from a range of pre-specified scales, or form their own. This version of the program is compatible...

SEMAINE has created a large audiovisual database as part of an iterative approach to building Sensitive Artificial Listener (SAL) agents that can engage a person in a sustained, emotionally coloured conversation. Data used to build the agents came from interactions between users and an 'operator' simulating a SAL agent, in different configurations:...

Hearing loss has a wide range of repercussions which are easily overlooked. Lipreading is surrounded by psychological issues, involving phenomenology, individual differences, and its interaction with other processes which compete for the same mental resources. Profound loss affects speech production as well as speech reception. The changes which oc...

Classical structure from motion schemes tend not to differentiate explicitly among viewpoints. However, intuitively it seems easier to answer specific questions about objects’ geometry from some viewpoints that others. Correspondingly, people seem to recognise “bad” viewpoints, and may seek out “good” ones. An experiment tested this intuition. Subj...

Computational research suggests that visual mechanisms may be discussed in terms of the postulates which underlie them. This provides a way of reformulating a long standing debate between experimentalists and ‘Cartesians’, who attempt to reconstruct the workings of vision by invoking mathematics and the physics of the visible world. Cartesians tend...

Social Signal Processing (SSP) as an emerging research area can draw on material from many disciplines, but it needs effective ways to organise the material. We propose a framework that integrates concepts, drawn primarily from psychology, but with input from other disciplines, in a way that indicates how they relate to SSP. We identify seven core...

We present the second Audio-Visual Emotion recognition Challenge and workshop (AVEC 2012), which aims to bring together researchers from the audio and video analysis communities around the topic of emotion recognition. The goal of the challenge is to recognise four continuously valued affective dimensions: arousal, expectancy, power, and valence. T...

The second international Audio/Visual Emotion Challenge and Workshop 2012 (AVEC 2012) is introduced shortly. 34 teams from 12 countries signed up for the Challenge. The SEMAINE database serves for prediction of four-dimensional continuous affect in audio and video. For the eligible participants, final scores for the Fully-Continuous Sub-Challenge r...

This paper tries to achieve a balanced view of the ethical issues raised by emotion-oriented technology as it is, rather than as it might be imagined. A high proportion of applications seem ethically neutral. Uses in entertainment and allied areas do no great harm or good. Empowering professions may do either, but regulatory systems already exist....

The ILHAIRE project seeks to scientifically analyse laughter in sufficient detail to allow the modelling of human laughter and subsequent generation and synthesis of laughter in avatars suitable for human machine interaction. As part of the process an incremental database is required providing different types of data to aid in modelling and synthes...

Computational research with continuous representations depends on obtaining continuous representations from human labellers. The main method used for that purpose is tracing. Tracing raises a range of challenging issues, both psychological and statistical. Naive assumptions about these issues are easy to make, and can lead to inappropriate requirem...

Conversations do not only consist of spoken words but they also consist of non-verbal vocalisations. Since there is no standard to define and to classify (possible) non-speech sounds the annotations for these vocalisations differ very much for various corpora of conversational speech. There seems to be agreement in the six inspected corpora that he...

The Audio/Visual Emotion Challenge and Workshop ( http://sspnet.eu/avec2011 ) is the first competition event aimed at comparison of automatic audio, visual, and audiovisual emotion analysis. The goal of the challenge is to provide a common benchmark test set for individual multimodal information processing and to bring together the audio and video...

In the literature, politeness has been researched within many disciplines. Although Brown and Levinson's theory of politeness (1978, 1987) is often cited, it is primarily a linguistic theory and has been criticized for its lack of generalizability to all cultures. Consequently, there is a need for a more comprehensive approach to understand and exp...

Databases are the empirical foundation of emotion-oriented computing, and creating them poses problems that are central to the field. There is now a substantial history of work on these problems, and the first chapter in the section reviews it. The two central chapters separate out the two main challenges, collecting appropriate material and develo...

The HUMAINE Database is grounded in HUMAINE’s core emphasis on considering emotion in a broad sense – ‘pervasive emotion’ – and engaging with the way it colours action and interaction. The aim of the database is to provide a resource to which the community can go to see and hear the forms that emotion takes in everyday action and interaction, and t...

This chapter deals with the task of defining and describing emotion. What do people mean when they identify emotion as a key domain for computing? How are “emotions” related to, and differentiated from, other affective phenomena? The chapter considers the definitions of emotions (and other affective states) formulated by scientists and those that a...

Labelling emotion databases is not a purely technical matter. It is bound up with theoretical issues. Different issues affect labelling of emotional content, labelling of the signs that convey emotion, and labelling of the relevant context. Linked to these are representational issues, involving time course, consensus and divergence, and connections...

The chapter reviews methods of obtaining records that show signs of emotion. Concern with authenticity is central to the task. Converging lines of argument indicate that even sophisticated acting does not reproduce emotion as it appears in everyday action and interaction. Acting is the appropriate source for some kinds of material, and work on that...

Developing databases for emotion-oriented computing raises specific and complex issues at multiple levels, from the practicalities of recording to conceptual issues in psychology. Whether it is developing databases or using them, research in emotion-oriented computing needs to think about these issues rather than reflexively importing habits derive...

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and non-verbal interaction capabilities. The work is motivated by the aim to provide technology with competences in interpreting and producing the emotional and non-verbal behaviours required to sustain a conversational di...

The SAL system embodies a new kind of human-computer interaction, where a person and a computer carry out a fluent, emotionally coloured conversation. Because that kind of capability is new, evaluating systems that have it is a new challenge. This paper outlines techniques that have been developed to evaluate SAL interactions, and uses the case to...

Deafness is due to a disorder of the ears, but its repercussions spread throughout a person's life. Other people who are closely involved are often affected too. The central concern of this book is one repercussion of hearing loss: that is, the deterioration of speech which can follow when someone loses his or her hearing. We describe research on t...

The network of excellence HUMAINE is currently making a co-ordinated, interdisciplinary effort to develop a consistent view on emotion-oriented computing. This overview paper proposes a "map" of the research area, distinguishing core technologies from application-oriented and psychologically oriented work. First results from the on-going research i...

Despite major advances within the affective computing research field, modelling, analysing, interpreting and responding to naturalistic human affective behaviour still remains as a challenge for automated systems as emotions are complex constructs with fuzzy boundaries and with substantial individual variations in expression and experience. Thus, a...

This demonstration aims to showcase the recently completed SEMAINE system. The SEMAINE system is a publicly available, fully autonomous Sensitive Artificial Listeners (SAL) system that consists of virtual dialog partners based on audiovisual analysis and synthesis (see http://semaine.opendfki.de/wiki). The system runs in real-time, and combines inc...

The Audio/Visual Emotion Challenge and Workshop (AVEC 2011) is the first competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and audiovisual emotion analysis, with all participants competing under strictly the same conditions. This paper first describes the challenge participation...

In this chapter we consider certain long-term ethical issues which are peculiar or special to emotion-oriented technology and which make the topic particularly charged ethically, both for the lay public and for those working in the area. We identify four such issues. First, it is far from clear whether technologies made by humans are conceivably ca...

Technology always faces questions about what it should and should not do, but they are unusually difficult to answer in the case of emotion-oriented computing – partly because they affect particularly sensitive areas of human life, and partly because of uncertainties surrounding the ways that machines might affect those areas. Those difficulties ca...

The exploration of how we react to the world and interact with it and each other remains one of the greatest scientific challenges. Latest research trends in cognitive sciences argue that our common view of intelligence is too narrow, ignoring a crucial range of abilities that matter immensely for how people do in life. This range of abilities is c...

We developed a cooperative time-sensitive task to study vocal expression of politeness and efficiency. Sixteen dyads completed 20 trials of the 'Maze Task', where one participant (the 'navigator') gave oral instructions (mainly 'up', 'down', left', 'right') for the other (the 'pilot') to follow. For half of the trials, navigators were instructed to...

We have recorded a new corpus of emotionally coloured conversations. Users were recorded while holding conversations with an operator who adopts in sequence four roles designed to evoke emotional reactions. The operator and the user are seated in separate rooms; they see each other through teleprompter screens, and hear each other through speakers....

What will it be like to admit Artificial Companions into our society? How will they change our relations with each other? How important will they be in the emotional and practical lives of their owners – since we know that people became emotionally dependent even on simple devices like the Tamagotchi? How much social life might they have in contact...

For many applications of emotion recognition, such as virtual agents, the system must select responses while the user is speaking. This requires reliable on-line recognition of the user's affect. However most emotion recognition systems are based on turnwise processing. We present a novel approach to on-line emotion recognition from speech using Lo...

Sensitive Artificial Listeners (SAL) are virtual dialogue partners based on audiovisual analysis and synthesis. Despite their very limited verbal understanding, they intend to engage the user in a conversation by paying attention to the user's emotions and nonverbal expressions. The SAL characters have their own emotionally defined personality, and...

Databases are fundamental to affective computing. Directly or indirectly, they provide a large proportion of the information about human affective functioning that is used by affective systems. Information may be drawn from them in many ways – by machine learning, by direct use of extracts, or by a combination of human judgment and machine measurem...

Understanding the nature of emotion is the kind of challenge that attracts people immediately. This article aims to highlight a challenge that has less instant appeal but which is none the less fascinating once it is accepted. Roughly speaking, the challenge is to develop good ways of describing what happens in the parts of people's lives where emo...

A decade ago, perceiving emotion was generally equated with taking a sample (a still photograph or a few seconds of speech) that unquestionably signified an archetypal emotional state, and attaching the appropriate label. Computational research has shifted that paradigm in multiple ways. Concern with realism is key. Emotion generally colours ongoin...

Dimensional descriptions are used in computational research on music and emotion, but the approach has potentially much more power and subtlety than many applications use. Two main kinds of refinement are considered-description of moment-by-moment change, and the use of more than the two or three best known dimensions.

Studies of the processing of social signals and behaviour tend to focus intuitively on a few variables, without a framework to guide selection. Here, we attempt to provide a broad overview of the relevant variables, describing both signs and what they signify. Those are matched by systematic consideration of how the variables relate. Variables inte...

There clearly are important issues associated with the distinction between acted and naturalistic data, but focusing on acting may not be the best way to articulate them. An alternative is to focus on differences of structure which are often (but not always) associated with the distinction. Several such differences relate to the way signs are distr...

This volume collects the contributions presented at the ACII 2009 Doctoral Consortium, the event aimed at gathering PhD students with the goal of sharing ideas about the theories behind affective computing; its development; and its application. Published papers have been selected out a large number of high quality submissions covering a wide spectr...

In today's affective databases speech turns are often labelled on a continuous scale for emotional dimensions such as valence or arousal to better express the diversity of human affect. How- ever, applications like virtual agents usually map the detected emotional user state to rough classes in order to reduce the mul- tiplicity of emotion dependen...

This volume collects the contributions presented at the ACII 2009 Doctoral Consortium, the event aimed at gathering PhD students with the goal of sharing ideas about the theories behind affective computing; its development; and its application. Published papers have been selected out a large number of high quality submissions covering a wide spectr...

One of the motives for studying faces and gestures is the role that they play in spontaneous, socially rich interaction between humans. If computers are to interact with humans in that mode (or to analyse what they are doing in it), methods of interpreting the non-verbal signals that they use are critical. It is becoming clear that developing those...

This paper describes work in the recently started project SEMAINE, which aims to build a set of Sensitive Artificial Listeners – conversational agents designed to sustain an interaction with a human user despite limited verbal skills, through robust recognition and generation of non-verbal behaviour in real-time, both when the agent is speaking and...

The aim of the paper is to document and share an induction technique (The Sensitive Artificial Listener) that generates data that can be both tractable and reasonably naturalistic. The technique focuses on conversation between a human and an agent that either is or appears to be a machine. It is designed to capture a broad spectrum of emotional sta...

The HUMAINE project is concerned with developing interfaces that will register and respond to emotion, particularly pervasive emotion (forms of feeling, expression and action that colour most of human life). The HUMAINE Database provides naturalistic clips which record that kind of material, in mul- tiple modalities, and labelling techniques that a...

In everyday life, speech is part of a multichannel system involved in conveying emotion. Understanding how it operates in that context requires suitable data, consisting of multimodal records of emotion drawn from everyday life. This paper reflects the experience of two teams active in collecting and labelling data of this type. It sets out the cor...

The network of excellence HUMAINE is currently making a co-ordinated, interdisciplinary eort to develop a consistent view on emotion-oriented computing. This overview paper proposes a "map" of the research area, distinguishing core technologies from application- oriented and psychologically oriented work. First results from the on- going research i...

There has been rapid development in conceptions of the kind of database that is needed for emotion research. Familiar archetypes are still influential, but the state of the art has moved beyond them. There is concern to capture emotion as it occurs in action and interaction ('pervasive emotion') as well as in short episodes dominated by emotion, an...

There are multiple reasons to expect that recognising the verbal content of emotional speech will be a difficult problem, and recognition rates reported in the literature are in fact low. Including information about prosody improves recognition rate for emotions simulated by actors, but its relevance to the freer patterns of spontaneous speech is u...