Nuno N. Correia

Tallinn University | TLU · School of Digital Technologies

There is an identified lack of visual feedback in electronic music performances. Live visuals have been used to fill in this gap. However, there is a scarcity of studies that analyze the effectiveness of live visuals in conveying feedback. In this paper, we aim to study the contribution of live visuals to the understanding of electronic music perfo...

INTRODUCTION: We propose AudioVisual User Interfaces (AVUI), a novel type of UI linking interaction, sound and image. It extends the concept of Graphical User Interface (GUI) by adding interconnected sound and image. OBJECTIVES: We aim to situate AVUIs in relation to identified relevant fields: Human-Computer Interaction (HCI), sonic interaction de...

In this paper, we investigate how to augment a dance performance using a multisensory approach in a way that communicates the dancing process as an embodied experience. We collaborated with a dancer and a media artist over an 8-week residency to prepare and present a multisensory dance performance and a spin-off installation. We present related wor...

The purpose of this paper is to deepen the understanding of interconnections among actors and artifacts in a music performance. To this end, this work presents a case study with four participants. Each participant created a piece using Puffin, a screen score system that allows for real-time generation of scores to be played by traditional instrumen...

This paper proposes a dynamic VR environment design approach to enhance the way a user experiences a computer generated character’s movement. We transferred concepts and practices from performance studies to create a dynamic VR environment, in which design elements are interconnected and changing with the character’s movement. Our focus was on spac...

We identify problems regarding teaching in higher education, particularly in computing subjects: short attention spans; lack of focus on cutting-edge topics; lack of attention to relevant issues for students; and disconnect from research. We propose that co-organising hackathons with students can assist in addressing these issues. We present a lite...

This paper investigates how the concept of score has been used in the NIME community. To this end, we performed a systematic literature review of the NIME proceedings, analyzing papers in which scores play a central role. We analyzed the score not as an object per se but in relation to the users and the interactive system(s). In other words, we pri...

INTRODUCTION: Technological advances have led to newer instruments and forms of audiovisual performance. Simultaneously, there has been a resurgence of analogue equipment in sound synthesis. However, such instruments for video synthesis are yet to be well established.OBJECTIVES: The primary objective of this research is to propose design guidelines...

In this paper, we propose to consider the sonic interactions that occurs in a dance performance from an ecological perspective. In particular, we suggest using the conceptual models of artefact ecology and design space. As a case study, we present a work developed during a two weeks artistic residency in collaboration between a sound designer, one...

In this paper, we introduce the concept of VR Open Scores: aleatoric score-based virtual scenarios where an aleatoric score is embedded in a virtual environment. This idea builds upon the notion of graphic scores and composed instrument , and apply them in a new context. Our proposal also explores possible parallels between open meaning in interact...

INTRODUCTION: We present an audience study investigating the impact of different technologies to create visuals in dance performances. OBJECTIVES: We investigated four conditions: motion capture, sensors, camera image, and minimal interaction; and four variables: how much did the audience perceive a connection between the body and the visuals; the...

This paper presents ARCAA (Actors, Role, Context, Activity, Artefacts), a framework that supports designers in understanding the artefact ecology in the music performance scenario, in particular, allowing to frame the role of the different actors. The ARCAA framework is the result of the combination of two different areas of HCI: artefact ecology c...

There has been an increased interest in HCI research regarding the possibilities of interactive technology applied to the field of dance performance, particularly contemporary dance. This has produced numerous strategies to capture data from the moving bodies of the dancers and to map that data into different types of display formats. In this paper...

In this paper, we present Penguin, a system for live scoring, and Studio I, a piece composed for the system and an accordion. The system is shaped in two modules, one that generates a musical stream in real-time and the other that manages a live scoring process. Penguin is designed to be used in interactive performances alongside traditional instru...

This paper proposes the idea that a score for a Digital Musical In- strument (DMI) can be approached from a design perspective. In particular, we focus on the concepts of affordances and constraints. The paper describes the use of scores in a piece called XXV composed for Chimney, a DMI and Cello. Both the piece and the system are detailed with a f...

The combined use of sound and image has a rich history, from audiovisual artworks to research exploring the potential of data visualization and sonification. However, we lack standard tools or guidelines for audiovisual (AV) interaction design, particularly for live performance. We propose the AVUI (AudioVisual User Interface), where sound and imag...

We have identified potential for tablets to be used as stand-alone tools for audiovisual performance, and not simply as controllers, due to their portability, expressive capabilities of multi-touch, and processing power. To explore this potential, we have developed AV Zones (AudioVisual Zones), an iPad app for audiovisual performance. In a prelimin...

There is growing interest in the application of crossmodal perception to interface design. However, most research has focused on task performance measures and often ignored user experience and engagement. We present an examination of crossmodal congruence in terms of performance and engagement in the context of a memory task of audio, visual, and a...

This paper compares three net art projects combining sound and animation by the author and Andre Carrilho (under the name Video Jack): AVOL, Master and Margarita and AV Clash. They follow a development path aiming to create projects enabling integrated audiovisual creativity that are flexible, intuitive, playful to use and engaging to experience. T...

We present a user-centered approach for prototyping tools for performance with procedural sound and graphics, based on a hackathon. We also present the resulting prototypes. These prototypes respond to a challenge originating from earlier stages of the research: to combine ease-of-use with expressiveness and visibility of interaction in tools for a...

There is a growing interest in audiovisual performance and composition. In this paper, we would like to investigate if the tools for audiovisual performance and composition have caught up with this growing interest and the practices in the field. In particular, we are interested in tools that use computer-­generated graphics. To address these issue...

The present study aims to address the following research question: how to create a tool for audiovisual performance, allowing for real-time usage of shared online visual resources, which can be customizable, and used across a variety of different hardware platforms? To address this issue, we have developed AVVX (AudioVisual Vector eXchange), a nove...

This paper introduces a novel collaborative environment (PESI) in which performers are not only free to move and interact with each other but where their social interactions contribute to the sonic outcome. PESI system is designed for co-located collaboration and provides embodied and spatial opportunities for musical exploration. To evaluate PESI...

This dissertation studies four projects combining visuals, sound and interactivity by the author and André Carrilho (under the name Video Jack). The aim of the study is to create web-based interactive audiovisual art projects for integrated and simultaneous manipulation of sound and motion graphics, in a way that is coherent, flexible, easy to use,...

This paper presents the research project The Notion of Participative and Enacting Sonic Interaction (PESI). PESI aims to extend the engagement of performers in collective music practices using embodied approaches in physical and social interaction. In its design, the mobile phone functions as a tangible and expressive musical instrument, together w...

This paper presents AVVX (AudioVisual Vector eXchange), a tool for live visuals and audiovisual performances developed by Nuno N. Correia. AVVX aims to answer the following question: how to design a tool for live visuals, allowing for the quick development of graphic materials, with open standards, and using the Internet as a resource for content e...

Audio-Visual On Line (AVOL) is an interactive audio-visual project for the Web, installation and performance by Video Jack (a duo composed of the author and André Carrilho). AVOL was one of the four winners of a competition by the Portuguese Ministry of Culture to develop artworks for their net art portal. Further to the launch of this portal, AVOL...

Master and Margarita is an audiovisual work by Portuguese new media art collective Video Jack that adapts Mikhail Bulgakov’s Russian modernist novel. This article studies those aspects of intermediality that resulted from this particular adaptation. Video Jack’s project, in contrast to other similar audiovisual artworks, does not aim to follow an a...

This paper compares two multimodal net art projects, AVOL and AV Clash, by the author and André Carrilho (under the name Video Jack). Their objective is to create projects enabling integrated audiovisual expression that are flexible, easy to use, playful and engaging to experience. The projects are contextualized with related works. The methodology...

This paper presents an evaluation of new media art project AV Clash. AV Clash is a Web-based artistic project that allows the creation of audiovisual compositions, consisting of combinations of animations with sounds retrieved from online database Freesound.org. The evaluation is based on the answers to an online questionnaire. It has an experience...

This paper presents Master and Margarita, an interactive audiovisual project for web and performance adapting Mikhail Bulgakov's novel of the same name. It aims to address two main research questions. First: how to integrate music and motion graphics in an interactive audiovisual project for the web and performance, in a way that is versatile, easy...

AVOL (AudioVisual OnLine) is an interactive audiovisual project for the Web, installation and performance by Video Jack (a collective composed of the author and André Carrilho). AVOL was one of the four winners of a competition by the Portuguese Ministry of Culture to develop artworks for their net art portal. Further to the launch of this portal,...

AVOL is an interactive audiovisual project for the Web by Video Jack (Nuno N. Correia and André Carrilho). Its main objective is to allow for an integrated musical and visual expression, in a way that is playful to use and engaging to experience. In this paper, the project is presented and contextualized with related works. Its design is then discu...

In this paper, the project AV Clash will be presented. AV Clash is a Web-based tool for integrated audiovisual expression, created by Video Jack (the author and André Carrilho, with the assistance of Gokce Taskan). In AV Clash, users can manipulate seven “objects” that represent sounds, incorporating audio-reactive animations and graphical user int...

Heat Seeker is an audio-visual project, which has been released in different formats: performance, video, and website. In this paper, it is contextualized with similar projects that combine music and visuals. The motivation and aims behind Heat Seeker are then presented. The main objective is to combine visuals with sound in an electronic music per...

This paper presents Master and Margarita, an interactive audiovisual project adapting Mikhail Bulgakov’s novel of the same name. Initially it was presented as performance, and later it was released as a website. The project aims to address two main research questions. First: how to integrate music and motion graphics in an interactive audiovisual p...

The use of multimedia with performance arts has a rich history. This paper proposes an innovative approach for the use of a Multimedia Application with music performances, through the integration of interactive digital animation. The Application and its development are described. The interface, which is part of the visual experience produced by the...

O multimédia é uma actividade produtiva situada na convergência do sector das Tecnologias de Informação e da Comunicação (TIC) com o sector da produção de conteúdos, actividade esta em expansão e com potenciais efeitos indutores a outras actividades económicas. A convergência das TIC com o sector de fornecimento de conteúdos abre caminho ao apareci...

We are presently at a turning point in mobile communications – the third generation (“3G” or UMTS) is imminent. The implications of UMTS for the Portuguese multimedia entertainment sector could be particularly relevant - mobile communications in Portugal have a penetration rate above the European average. Portugal is already a laboratory for testin...

The use of multimedia with performance arts has a rich history. This paper proposes an innovative approach for the use of a multimedia application with performance arts, through the integration of interactive and generative digital animation. The application and its development are described. The interface, which is part of the visual experience pr...