Sergi Jordà

Pompeu Fabra University | UPF · Music Technology Group

This paper explores the attributes necessary to determine the creative merit of intermediate artifacts produced during a computational co-creative process (CCC) in which a human and an artificial intelligence system collaborate in the generative phase of a creative project. In an active listening experiment, subjects with diverse musical training (...

This paper presents a real-time drum generation system capable of accompanying a human instrumentalist. The drum generation model is a transformer encoder trained to predict a short drum pattern given a reduced rhythmic representation. We demonstrate that with certain design considerations, the short drum pattern generator can be used as a real-tim...

Video gaming is now available as a fully immersive experience that creates responsive inputs and outputs concerning the user, and some experimental developers have integrated the use of the voice, brain, or muscles as input controls. The use of physiological signal equipment can provide valuable information regarding the emotion of a player or pati...

The conception and design of new musical interfaces is a multidisciplinary area that tightly relates technology and artistic creation. In this paper, the author first exposes some of the questions he has posed himself during more than a decade experience as a performer, composer, interface and software designer, and educator. Finally, he illustrate...

This paper reports on the design and evaluation of drum rhythm spaces as interactive bi-dimensional maps used for the visualisation, retrieval and generation of drum patterns. We carry out two experiments exploring human processing of polyphonic drum patterns concluding with a list of descriptors that significantly influence similarity sensations....

This paper presents two experiments carried out to find rhythm descriptors that allow the organization of drum patterns in spaces resembling subjects similarity sensations. We revisit rhythm spaces published by Alf Gabrielsson in 1973, based on subject similarity ratings of drum rhythms from an early drum machine, and construct a new rhythm space b...

Concatenative synthesis is a sample-based approach to sound creation used frequently in speech synthesis and, increasingly, in musical contexts. Unit selection, a key component, is the process by which sounds are chosen from the corpus of samples. With their ability to match target units as well as preserve continuity, Hidden Markov Models are ofte...

‘Real-Time’ is one of the most frequent concepts used referring to digital media. Despite it is commonly related to immediate feedback, it is not clear when it happens and what its characteristics are, especially in current digital interfaces. Although real-time has been written about very extensively in the context of Human Computer Interaction an...

In this article, we summarize recent research examining concatenative synthesis and its application and relevance in the composition and production of styles of electronic dance music. We introduce the conceptual underpinnings of concatenative synthesis and describe key works and systematic approaches in the literature. Our system, RhythmCAT, is pr...

In this paper we present an interactive two-dimensional representation of musical chord progressions, integrated into a computer program that generates house music harmonic loops in MIDI format, based on a user's input. Our aim is to encapsulate relevant tonal information and display it in ways that are easy to understand for novices and untrained...

This paper first introduces two previous software-based music instruments designed by the author, and analyses the crucial importance of the visual feedback introduced by their interfaces. A quick taxonomy and analysis of the visual components in current trends of interactive music software is then proposed, before introducing the reacTable*, a new...

Interaction designers often use machine learning tools to generate intuitive mappings between complex inputs and outputs. These tools are usually trained live, which is not always feasible or practical. We combine RepoVizz, an online repository and visualizer for multimodal data, with a suite of Interactive Machine Learning tools, to demonstrate a...

This article presents a video-based field study of the Reactable, a tabletop tangible user interface (TUI) for music performance, in a hands-on science centre. The goal was to investigate visitors’ social interactions in a public setting. We describe liminality and cross-group interaction, both synchronous with fluid transitions and overlaps in use...

In this paper we study key estimation in electronic dance music, an umbrella term referring to a variety of electronic music subgenres intended for dancing at nightclubs and raves. We start by defining notions of tonality and key before outlining the basic architecture of a template-based key estimation method. Then, we report on the tonal characte...

This paper explores the potential benefits of using similarity-based two-dimensional classifications and their corresponding GUIs, for drum samples retrieval in a creativity-oriented context. Preliminary user studies with professional electronic music producers point up the frustration and laboriousness of finding suitable drum samples in the incre...

In this paper we evaluate b-Reactable, a digital music instrument that combines implicit physiology-based interaction through EEG and ECG, and explicit gestural interaction for sound generation and control. This multimodality is embodied in tangible objects named physiopucks, which are driven by biosignals. We hypothesize that multimodality increas...

Measuring rhythm similarity is relevant for the analysis and generation of music. Existing similarity metrics tend to consider our perception of rhythms as being in time without discriminating the importance of some regions over others. In a previously reported experiment we ob- served that measures of similarity may differ given the presence or ab...

This paper is inspired in the ideas of rhythmical variation and evolution, which are connected to similarity and contrast. Two experiments on rhythm similarity are presented that examine the possible relations between objective metrics and human similarity ratings. We wanted to test the possible differences in similarity ratings when a beat was ind...

Shareable interfaces, those that can be interacted simultaneously by several users, are a common tool used both in CSCW research and in real world applications. They tend however to lack a capability that has been traditionally relevant to the usefulness of computing systems: multi-tasking. In this paper we explain why a combination of the multi-us...

This article presents and discusses a selection of interactive works that use different representations of time-images in which past and present merge into a single space of representation. These cinematic experiences generate singular relationships of temporal perception and consciousness in relation to cinema, connected to the participant's physi...

In this paper we propose citizen-driven, public data visualization as a tool to support social and civic purposes in public spaces. We argue for the potential of this approach, motivating it with recent trends and developments in the areas of information visualization, urban computing, and urban screens, and we layout a transdisciplinary research a...

Co-located tabletop tangible user interfaces (TUIs) for music performance are known for promoting multi-player collaboration with a shared interface, yet it is still unclear how to best support the awareness of the workspace in terms of understanding individual actions and the other group members actions, in parallel. In this paper, we investigate...

The field of physiology-based interaction and monitoring is developing at a fast pace. Emerging applications like fatigue monitoring often use sound to convey complex dynamics of biological signals and to provide an alternative, non-visual information channel. Most Physiology-to-Sound mappings in such auditory displays do not allow customization by...

The maturation process of the NIME field has brought a growing interest in teaching the design and implementation of Digital Music Instruments (DMIs) as well as in finding objective evaluation methods to assess the suitability of these outcomes. In this paper we propose a methodology for teaching NIME design and a set of tools meant to inform the d...

There has been little research on how interactions with tabletop and Tangible User Interfaces (TUIs) by groups of users change over time. In this article, we investigate the challenges and opportunities of a tabletop tangible interface based on constructive building blocks. We describe a long-term lab study of groups of expert musicians improvising...

In this paper we introduce a new method for 6DoF marker tracking, specially designed for Microsoft SecondLight or any camera-based tabletop interface that is able to see objects through the surface. Our method is based on topological region adjacency for the identification of the markers, which are fitted into a squared shape for properly track the...

In this paper we introduce a new method for 6DoF marker tracking, specially designed for Microsoft SecondLight or any camera-based tabletop interface that is able to see objects through the surface. Our method is based on topological region adjacency for the identification of the markers, which are fitted into a squared shape for properly track the...

Over the last few decades there has been steady growth in research that addresses the real-time sonification of electroencephalographic (EEG) data. Diverse application areas include medical data screening, Brain Computer Interfaces (BCI), neurofeedback, affective computing and applications in the arts. The present paper presents an overview and cri...

Public displays and projections are becoming increasingly available in various informal urban settings. However, their potential impact on informing and engaging citizens on relevant issues has still been largely unexplored. In this paper, we show that visualizations displayed in public settings are able to increase social awareness and discourse b...

Multi-touch technologies have been increasing its popularity in the last decade. Nowadays we can find a plethora of devices that includes this technology: tablets, smartphones or even in desktop computers. In parallel tangible tabletop surfaces started to appear in the global market with devices such as Reactable. What all this devices have in comm...

While the HCI community has been putting a lot of effort on creating physical interfaces for collaboration, studying multi-user interaction dynamics and creating specific applications to support (and test) this kind of phenomena, it has not addressed the problems implied in having multiple applications sharing the same interactive space. Having an...

There is little evaluation of musical tabletops for music performance, and current approaches tend to have little consideration of social interaction. However, in collaborative settings, social aspects such as coordination, communication, or musical engagement between collaborators are fundamental for a successful performance. After an overview of...

This position paper summarises some themes encountered when analysing video data in the context of music performance with interactive tabletops. It presents methodological approaches and coding schemes used for a set of experiments on musical tabletops and collaboration. Finally, it outlines an initial taxonomy based on the outcomes of the projects...

This study assesses the potential of the Reactable, a musical tangible user interface, to help in the acquisition of social interaction abilities in children with Autistic Spectrum Condition (ASC). With this purpose, nine children with ASC participated in the research; the sample being its own control group, and a simple subject design was develope...

Creativity is a relevant characteristic for people's development as it facilitates the generation of new ideas and innovation processes. Although technology has played an important role on creativity stimulation, it still needs to be explored for a better understanding and support in the context of information and communication technologies. In thi...

We are proud to present the proceedings of TEI 2012 --- the sixth international conference on tangible, embedded, and embodied interaction design. Research in this area stretches across the full spectrum of fields involved in the design of interactive systems, from software, electronics, and mechanics, to form, aesthetics, and social impact. The pa...

Physiological Computing has been applied in different disciplines, and is becoming popular and widespread in Human-Computer In-teraction, due to device miniaturization and improvements in real-time processing. However, most of the studies on physiology-based interfaces focus on single-user systems, while their use in Computer-Supported Collaborativ...

With the advent of tabletop interaction, collaborative activities are better supported than they are on single-user PCs because there exists a physical shareable space, and interaction with digital data is more embodied and social. In sound and music computing, collaborative music making has traditionally been done using interconnected networks, bu...

In this paper we give an analysis of the literature on a set of problems that can arise when undertaking the interaction design of multi-touch applications for collaborative real-time music activities, which are designed for multi-touch technologies (e.g. smartphones, tablets, interactive tabletops, among others). Each problem is described, and a c...

Tap. Slide. Swipe. Shake. Tangible user interfaces have some scientists toying around with stuff you can really put your hands on.

In this paper we present the Reactable, a new electronic musical instrument with a simple and intuitive tabletop interface that turns music into a tangible and visual experience, enabling musicians to experiment with sound, change its structure, control its parameters and be creative in a direct, refreshing and unseen way.

Tangible interactions on horizontal surfaces are increasingly relevant for collaborative applications, embodied interaction, musical performance, and interaction with 3D information. This unique studio opportunity introduces approaches to developing applications on four related platforms: the Reactable: a musical tabletop, and its companion fiducia...

Since commercial musical recordings became available about a century ago and until very recently, they had always been distributed by means of a physical support. Nowadays that record companies have ultimately started to distribute music over the Internet, "music" does not use physical space anymore, and we can now store amazingly large music colle...

The Reactable is a new electronic musical instrument with a simple and intuitive design, which enables musicians to experiment with sound, change its structure, control its parameters in a direct, refreshing and unseen way. It uses a luminous surface interface, where the musician controls the system by manipulating tangible objects.

This paper presents SongExplorer, a system for the ex- ploration of large music collections on tabletop interfaces. SongExplorer addresses the problem of finding new inter- esting songs on large music databases, from an interaction design perspective. Using high level descriptors of musi- cal songs, SongExplorer creates a coherent 2D map based on s...

In recent years we have seen a proliferation of musical tables. Believing that this is not just the result of a tabletop trend, in this paper I first discuss the reasons that make tabletop interfaces promising musical interfaces. After that, I present the reactable, a musical instrument based on a tabletop interface that exemplifies several of thes...

This paper introduces TurTan, a tangible programming language for creative exploration inspired by Logo, which uses a tabletop interface with tangible objects. The aim of this project is to design a toy language for programming entertainment and creative purposes. Along this paper we also discuss some interesting technical issues we have found duri...

This paper explores one of the application domains in which tangible and tabletop interfaces have currently shown more positive results, studying and unveiling the essential reasons that turn live music performance and tabletop interaction into promising and exiting fields of multi-disciplinary research and experimentation. The paper is structured...

‘When asked what musical instrument they play, few computer musicians respond spontaneously with “I play the computer.” Why not?’(Wessel and Wright 2002). Actually, most computer music performers still seem shyly reluctant to consider the computer as a regular musical instrument, but nonetheless, the computer is finally reaching the point of feelin...

In recent years we have seen a proliferation of musical tables. Believing that this is not just the result of a tabletop trend, in this paper we first discuss several of the reasons for which live music performance and HCI in general, and musical instruments and tabletop interfaces in particular, can lead to a fertile two-way cross-pollination that...

The reacTable* is a novel multi-user electro-acoustic musical instrument with a tabletop tangible user interface. In this paper we will focus on the various collaborative as- pects of this new instrument as well as on some of the re- lated technical details such as the networking infrastruc- ture. The instrument can be played both in local and remo...

This paper introduces the scoreTable*, a tangible interactive music score editor which started as a simple application for demoing "traditional" approaches to music creation, using the reacTable* technology, and which has evolved into an independent research project on its own. After a brief discussion on the role of pitch in music, we present a br...

Dreams in High Fidelity is a painting that evolves. It was designed and rendered with the Electric Sheep screen-saver, a cyborg mind composed of 30,000 computers and people mediated by a genetic algorithm. Physically it consists of a small computer ...

This paper describes reacTIVision: a camera based two dimensional fiducial (marker) tracking system developed for the reacTable*, a table based tangible musical instrument. Key features of reacTIVision include: (1) the ability to track a large number of fiducials with faster than real-time performance and (2) fiducial size may be varied depending o...

In this paper we study the potential and the challenges posed by multi-user instruments, as tools that can facilitate interaction and responsiveness not only between performers and their instrument but also between performers as well. Several previous studies and taxonomies are mentioned, after what different paradigms exposed with examples based o...

This paper describes the reacTable*, a novel multi-user electro-acoustic music instrument with a tabletop tangible user interface, which is being developed at the MTG in Barcelona. We first introduce the reacTable* project and some of the concepts behind its design and we then dis-cuss interaction and performance scenarios.

We present an Internet browser plug-in for interactive real-time sound synthesis. The plug-in runs the pure data (Pd) sound synthesis engine within the browser, and allows flexible and transparent sound generation and control. Pd is a general sound synthesis language that is based on the MAX graphical programming paradigm, which allows constructing...

Music instruments are used to play and to produce music, transforming the actions of one or more performers into sound. This article explores some instrument design issues in three distinct parts. The first section attempts to define what music instruments are, how traditional instruments function and what they can do, and what future instruments c...

This article reflects the current state of the reacTable* project, an electronic music instrument with a tangible table-based interface, which is currently under development at the Audiovisual Institute at the Universitat Pompeu Fabra. In this paper we are focussing on the issue of Dynamic Patching, which is a particular and unique aspect of the so...

When envisaging new digital instruments, designers do not have to limit themselves to their sonic capabilities (which can be absolutely any), not even to their algorithmic power; they must be also especially careful about the instruments' conceptual capabilities, to the ways instruments impose or suggest to their players new ways of thinking, new w...

This article explores some music instruments generic properties such as the diversity, the variability or the reproducibility of their musical output, the linearity or non-linearity of their behavior, and tries to figure out how these aspects can bias the relation between the instrument and its player, and how they may relate to more commonly studi...

In this paper we present an Internet browser plug-in for real-time sound synthesis. The plug-in uses the Pure Data (Pd) sound synthesis engine, and allows flexible and trans- parent sound generation and control embedded into web- pages. Pd is a general sound synthesis language that is based on the MAX graphical programming paradigm. It can be used...

New digital musical instruments designed for professional and trained musicians can be quite complex and challenging, offering as a counterpart a great amount of creative freedom and control possibilities to their players. On the other hand, instruments designed for amateur musicians or for audiences in interactive sound installations, tend to be q...

This paper first introduces two previous software-based music instruments designed by the author, and analyses the crucial importance of the visual feedback introduced by their interfaces. A quick taxonomy and analysis of the visual components in current trends of interactive music software is then proposed, before introducing the reacTable*, a new...

This paper exposes some topics and questions about music improvisation using computers that the author has considered during more than a decade of experience as a performer, composer, software designer and educator. After a quick review of several of his previous interactive music systems, some of those issues are then confronted in FMOL, a program...

This paper first introduces two previous software-based music instruments designed by the author, and analyses the crucial importance of the visual feedback introduced by their interfaces. A quick taxonomy and analysis of the visual components in current trends of interactive music software is then proposed, before introducing the reacTable, a new...

An abstract is not available.

In this paper the authors propose a new architecture and new features for collaborative music composition. These design principles have been applied, without any loss of generality, to a system that has already been extensively tested on-line for the last three years, and which has allowed composers from around the world to participate in the colle...

In this paper the authors present an overview of recent work on ongoing projects at the Audiovisual Institute from the Pompeu Fabra University in Barcelona, focused on Internet collaborative virtual environments for music applications. Although presenting different strategies, they all put a special emphasis in performance and composition/productio...

In this paper we present Afasia, an interactive multimedia performance based in Homer's Odyssey [2]. Afasia is a one-man digital theater play in which a lone performer fitted with a sensor-suit conducts, like Homer, the whole show by himself, controlling 2D animations, DVD video and conducting the music mechanically performed by a robot quartet. Af...

multidisciplinary area that tightly relates technology and artistic creation. In this paper, the author first exposes some of the questions he has posed himself during more than a decade experience as a performer, composer, interface and software designer, and educator. Finally, he illustrates these topics with some examples of his work.

In this paper the authors propose a new architecture and new features for collaborative music composition. These design principles have been applied, without any loss of generality, to a system that has already been extensively tested on-line for the last three years, and which has allowed composers from around the world to participate in the colle...