Skills and Expertise
Jan 2003 - Jul 2005
- Institut für Neuroinformatics
- Zürich, Switzerland
- Invited Researcher
Research Items (163)
- Sep 2018
Technology could represent an ally in easing the formulation of a bond between music therapist and client. So far, both scholars and therapists have claimed that the implementation of technology could be paramount in transforming the client’s therapeutic process, beyond representing a new and effective methodology for session analysis. This article proposes the implementation of musical technologies within the daily practice of music therapy, the latter understood as the appropriate field for the application of interactive systems technology, which track the user’s movements and transform them into audiovisual and haptic feedback. The article presents an exploratory review of the use of digital instruments, based on motion tracking, in order to develop a new basis for music therapy procedures, and discusses how specific features of digital musical instruments could benefit music therapy sessions. All discussions are set within the framework of embodied cognition. We provide a discussion on how multisensory integration may be used in the treatment of clients with severe cognitive and motor difficulties. We advocate the need for an interdisciplinary approach to the practice of music therapy, and propose future lines of research concerned with the design of multimodal and empowerment-based technologies.Abbreviation: American Association of Pediatrics (AAP); Autism Spectrum Disorder (ASD); Digital Musical Interfaces (DMIs); e-motion, colour and music (e-mocomu); Motion composer (MC); Parkinson’ s disease (PD); Soundbeam (SB) KEYWORDS: DMI, empowerment, sensorimotor maps, multiple trajectories, therapeutic bond
This article discusses a new algorithmic model of Désordre, György Ligeti's first étude for piano. The model is able to accurately reconstruct the original score and automatically simulate neighboring variants. The algorithm is deterministic, that is, each set of parameter values corresponds to one and only one result. The basic strategies used to model the highly entropic aspects of the composition were the formulation of primitive versions of the secondary voices and the decomposition of rhythmic patterns at independent subjacent and explicit levels. The first parameter controls the pitch-set used by the right hand and by way of its left-hand complement. The NSP parameter (the nth sum of different primes) determines the basic rhythmic relationships and ensures the asymmetry of rhythmic patterns. Computer simulations are further controlled by specifying units of time and pitch allowing for other, uncommonly used temperaments and divisions of the whole note. Several neighboring variants of the original piece are discussed, such as simulations of octatonic, whole-tone and chromatic scales being used as a pitch-set of one of the pianist's hands. The piece was also simulated using other asymmetrical rhythmic relationships, as well as a temperament based on quarter-tones.
The general aim of this research is to analyze the emergent timbre (McAdams, 2013; Rossetti, Teixeira & Manzolli, 2017; Rossetti & Manzolli, 2017) in live-electronic music performance. We believe that the emergent characteristics of the timbre result from the interactions between the different overlapped sound layers such as the tape sounds, the dynamic part of real-time processing, and the instrumental sounds which can be or not expressed by extended techniques. The specific goal is to understand the temporal and spectral activities of these emergent sound structures in terms of Volume and Spectral Liveness. The Volume can help us to perform a formal segmentation of the analyzed pieces, and the Spectral Liveness determines the flux qualities of the achieved emergent sonorities.
In this article, an analysis of the piece Desdobramentos do contínuo for violoncello and live-electronics is addressed concerning instrumental extended techniques, electroacoustic tape sounds, real-time processing, and their interaction. This is part of a broad research about the computer-aided musical analysis of electroacoustic music. The objective of the analysis of this piece is to understand the spectral activity of the emergent sound structures, in terms of which events produce huge timbre variations, and to identify timbre subtle nuances that are not perceptible on a first listen of the work. We conclude comparing the analyses results to the compositional hypotheses presented in the initial sections.
- Jan 2018
- Music Technology with Swing
The SELFHOOD installation was conceived aiming to instigate a reflection on the self through a practical and interactive experience. A representation of each participant is created in a form of a cloud of points and a sound drone, suggesting their selves. The dynamics of the visitors’ movements is sonified in such way that colours and sound textures are fused in a surrounding hexaphonic system. CromaCrono≈, the system for immersive improvisation that produces digitally synthesized sounds in real time, is described. Philosophical concepts concerning notions of the Self are presented. We propose that the notion of Presence can be induced by virtual and/or physical sources of stimulation governed by a number of principles that underlie human experience, creativity, and discovery. The methodological point of view is that the notion of Presence indicates that there are essential inputs for the construction of self-referral agents.
As part of a research study of an audio descriptors model for the analysis of live-electronic music, this article addresses Ressonâncias (2015) for piano and electronic sounds by Jônatas Manzolli. As a theoretical basis, we approach the Fourier Series and Transform, an operation that serves as a basis for the electroacoustic processes employed in the composition and for the analyses performed from descriptors. The hypotheses to be verified are related to the malleability of the electroacoustic processes employed in the piece, which can deliver specific results for different performances, besides the existence of a tendency of greater spectral similarity in the compositional process, where the level of variation between two performances is higher in the initial sections and lower in the final sections. As a methodology, we analyze the spectral features of the fixed tape sounds and compare the timbre produced (the amalgam of instrumental and electroacoustic sounds) in two different performances from the data obtained from the analysis. We conclude by discussing two aspects: (1) the level of sonic malleability reached by the composer in structuring his creative process, and (2) the analysis that emphasizes the impossibility of traditional music notation to represent the plentitude of the emerging sound resulting from the interaction between instrumental and electroacoustic sounds. In other words, the applicability of the analytical model studied is relevant in the context of current live-electronic music, mainly when it addresses the dynamic fusion of the instrumental part with tape sounds and/or real-time treatments.
- Sep 2017
- International Symposium on Computer Music Multidisciplinary Research
We discuss in this paper strategies of dynamic mapping applied to the design of augmented instruments. The proposed method is based on a feedback architecture that allows adjustment of mapping functions through pattern detection from the sonic response of an instrument. We applied this method to design Metaflute, an augmented instrument based on a hybrid system of gestural capture. We used a set of eight phase vocoder modules to process flute samples in real-time with performer movements detected by coupled sensors. A set of audio features was extracted from the sonic response of each synthesis module within a certain period and sent to a K-nearest neighbor algorithm (k-NN). Each synthesis module has its mapping functions modified based on patterns found by k-NN algorithm. This procedure was repeated iteratively so that the system adjusts itself in relation to its previous states. Preliminary results show a gradual differentiation of mapping functions for each synthesis module, allowing them to perform a different response in relation to data arising from the gestural interface.
This article proposes an audio descriptors model for the analysis of live electroacoustic music. In this context, an analysis of the work Desdobramentos do contínuo for violoncello and live electronics is addressed, concerning both tape (deferred time) sounds and live electronics (instrument sound and real-time processing). For this analysis, audio descriptors such as spectral flux, energy mean, centroid, and loudness were employed. The objective was to define which events produce huge timbre variations and to identify timbre subtle nuances which are not perceptible on a first listen of the work. We conclude comparing the analysis results to the compositional hypotheses presented in sections 2 and 3.
This paper presents a Case Study on interactive musical technology and discusses how it enhances the Creative Empowerment gained through the proprioception awareness of the participants within an interactive environment. We propose the concepts of Creative Empowerment (CE) and Sensorimotor Maps (SM) (Partesotti, 2016) in order to explain this thesis. Our viewpoint is that this kind of technology could be applied both in the therapeutic and educative environment and it would bring benefits in the learning process of people of any age and gender. In recent literature studies shown the effectiveness of the technological environment applied to educational and therapeutic setting (Camurri et al., 2010, Kontogeorgakopoulos, Wechsler & Keay-Bright 2013, Peñalba, 2015). Digital Musical Interfaces (DMIs) are instruments designed for various contexts, in particular for musical and artistic applications, and can be used by non-experts and experts (Miranda & Wanderley, 2006). Our study also connects the DMI technology to the Mixed Interactive Reality (MIR) (Partesotti, 2016), which is based on the proprioceptive interaction of the user and that integrates diverse perceptual modalities (2016). The Case Study uses e-mocomu (e-motion, colors and music), a prototype technology developed to enable users to control sounds and colors by means of their movements, and which integrates diverse perceptual modalities that permits their creative expression. The technology has been applied to a case study showing the potentialities that this interactive instrument offers within the learning filed. Furthermore taking into account concepts for the Sensorimotor Contingency Theory (Nöe & O’Regan, 2000) and the MIR technology, the Case Study verifies how e-mocomu might integrate the following sensorimotor contingencies: visual, auditory and proprioceptive. The Case Study engaged 17 participants. Each participant followed a set of tasks for gestural and audio-visual analysis, and eventually improvised with e-mocomu for an undetermined time. Every performance was videotaped; furthermore we gathered the length and the acceleration of users’ movements in time, during their improvisation session in order to track their SM and their EC. Before and after the test we collected additional information with the SAM test (Self-Assessment Manikin) (Lang, 1980) in order to assess a former emotional evaluation. We compare participant’s Arousal and Valence to data from their performances: time of the improvisation sessions, acceleration curves and analysis of their movements based on the video annotations The results outline a change both in arousal and in valence. In particular a correlation between the change in valence and the duration in time of the users’ performance was found. As the results have shown, the duration of the creative performance is connected to the CE. Hence, the case study highlights how a MIR technology that arouses the CE, gained through a proprioceptive awareness, would bound to benefit the learning process of the participant both in education and in a motor rehabilitation context
Trata-se do segundo Diário de Bordo relacionado a uma série de composições com o objetivo centrado na criação de obras a partir da análise de conteúdo espectral. Utilizamos tal metodologia para elaborar a obra “O azeite, a lua e o rio – numa ilha rodeada de ouro, com água até o joelho”, para flauta; clarinete e violoncelo (2013). Para caracterizar as sonoridades desejadas, desenvolveu-se um ambiente de programação em Pure Data, em conjunto com uma biblioteca de análise, via Descritores de Áudio, denominada PDescriptors. Com o auxilio do computador, realizamos diversos experimentos para avaliar uma variedade de configurações instrumentais relacionadas a sonoridades “rugosas”, “lisas” e com intensidades sonoras às quais variam de pianissimo a fortissimo. A pesquisa ratifica a aplicação de novas tecnologias computacionais tanto no processo criativo quanto em análise musical. Palavras-chave: Composição musical; Composição musical assistida por computador; Timbre musical; Descritores de áudio; PureData.
Trata-se do relato de um processo criativo musical no qual utilizou-se estratégias composicionais vinculadas a análises de conteúdo espectral com descritores de áudio. Tal procedimento foi o suporte para a elaboração da obra "Lana Tai - no dia em que nasceu uma aquarela", para orquestra de cordas. A partir de um banco de dados sonoros, realizaram-se diversas experimentações com o objetivo de testar configurações orquestrais elaboradas com técnicas estendidas de execução instrumental. Tais experimentos foram realizados com suporte do ambiente computacional Pure Data (PD) com o qual utilizou-se a biblioteca de descritores de áudio PDescriptors. Os resultados, as reflexões e as questões levantadas durante toda a elaboração, criação e execução de "Lana Tai" suscitam um novo aporte no desenvolvimento tanto da pesquisa em processos criativos quanto na criação de novas obras inéditas.
Este artigo apresenta a implementação computacional e analisa o comportamento dos visitantes da instalação audiovisual Caminho das águas. Baseados em um método de análise desenvolvido durante a pesquisa que inclui análises de questionários, análises estatísticas sobre os dados de movimento e análises estésicas sobre o resultado audiovisual gerado pelos visitantes, observamos a ocorrência de padrões gestuais e de correlações entre o processo interativo e a formação artística dos visitantes.
The paper presents a new OpenMusic library that implements a Genetic Programming method of Symbolic Regression on sets of input data-points and seeks for a Common Lisp function (S-expression) that can be used either to create mathematical models that could potentially help to understand the mathematical behavior of the input data or to generate parameters in computer-aided composition. Stressing that a number of issues must still be addressed to improve the proposed library, the paper presents some of the strategies to do this and to make Symbolic Regression a practical tool in computer-assisted music composition and analysis.
- Mar 2015
- Virtual Reality (VR), 2015 iEEE
The meaningful representation and exploration of big data constitutes a challenge for many scientific fields. In recent years, auditory displays have been effectively employed to address this problem. The coupling of sonification with visualization techniques in mul- timodal displays can lead to the implementation of powerful tools for the understanding of complex datasets. In this study, we applied sonification techniques to a complex dataset from neuroscience. To do so, we used BrainX3, a novel immersive technology for the ex- ploration of large brain networks. We conducted an experiment to assess whether the addition of an auditory layer would result in better user performance of brain region identification at different spatial resolutions.
- Jan 2015
Focusing on the interactivity that a robotic interface establishes between the virtual and the real world, some sensory systems and mobile robotic platforms were developed for the AURAL project, a robotic evolutionary environment for sound production. From the AURAL perspective, human and robots are agents of a complex system and the sonification is the emergent propriety produced by their interaction and behavior. One way to characterize types of interactions is by looking at ways in which systems can be coupled together to interact. The representation of the interaction between a person and a dynamic system as a simple feedback loop faces the role of information looping through both a person and a system. Two different sonification paradigms were applied in AURAL environment. In the first case, the sonification is generated by an evolutionary mapping of the robot trajectories into sound events. In the second case, the sound production is the result of a generative process. As such the sonification here is not seen as an isolated aspect of AURAL, but as a representation of the synergetic capacity of the agents to collaborate and produce a complex product. A comparison between the results obtained with both approaches is presented. The structure/novelty tradeoff has been approached.
New technologies emphasize interaction and novel music interfaces and alternative forms and modes of interactive media have been realized. The digital era is promoting interaction between audiovisuals, music and improvisation in virtual and mixed reality spaces. Have we “stones” in our hands - the computer technology? As the primitive man was stimulated by the images and sounds from the stones, we might be stimulated, by mixing the virtual and real worlds, to understand the integration of the brain and the body with such yet unpolished technological tools. This article presents a conceptual discussion articulating interactive music composition and computer creativity. It is also introduce the idea of using interactive media as laboratory to study creativity and to mediate the creation of artworks.
We propose a search method, namely Query-by-Multiple-Examples, that is able to search, within an audio sample database, for a particular sonic characteristic. The charac-teristic is learned on-the-fly by means of multiple exam-ples provided by a human user, thus avoiding ambiguities due to manual labelling. We evaluate four variations of the proposed method using ground truth provided by three musicians. It is shown that, for queries based on sonic characteristics, the query modelling process yields more correct results than if several single-example queries were executed in parallel using the same input data.
Digital Musical Instruments (DMIs) have diculties establishing themselves after their creation. A huge number of DMIs is presented every year and few of them actually remain in use. Several causes could explain this reality, among them the lack of a proper instrumental technique, inadequacy of the traditional musical notation and the non-existence of a repertoire dedicated to the instrument. In this paper we present the case study of Entoa, the first written music for Intonaspacio, a DMI we designed in our research project. We propose a process of composition that considers the design of the instrument as a required step in order to start defining an instrumental technique. We present an overview of the instrument and strategies for mapping data from sensors to sound processing, in order to accomplish an expressive performance.
This article focuses on techniques to increase expressivity and intuitiveness of an interactive process in the context of audiovisual installations based on motion capture devices. We propose a series of strategies for mapping and data parameterization obtained from the analysis of visitors' gestures. Information retrieved from the analysis of gestures is interpreted in terms of its meaningful characteristics using de-scriptors of gesture segmentation, physical-domain characterisitcs (dis-tance, velocity, acceleration and angular relations) and Laban's e↵ort analysis. E↵ort allows us to di↵erentiate quality of gestural movements aiming to increase responsiveness of the system for the data acquired in real-time from visitor's movements, thereby intensifying the interactive experience. By using simple techniques derived from physical descritors we have ecient data control and optimized computer processing. These techniques are being implemented through a framework of MaxMSP abstractions that determine constraints to retrieve information from the data streaming provided by Motion Capture systems. We present the resources developed and ongoing improvements of this framework for mapping.
This chapter analyzes the similarities and differences of meaning in language and music, with a special focus on the neural underpinning of meaning. In particular, factors (e.g., emotion) that are internal to an agent are differentiated from factors that arise from the interaction with the external environment and other agents (e.g., sociality and discourse). This “world axis” (from internal to external worlds) is complemented by three other axes: the “affective–propositional axis,” the “sensorimotor–symbolic axis,” and the “structure axis” (from small- to large-scale structure). Common structure–function relationships in music and language and their neuronal substrate are addressed, with emphasis on expectation and prediction. A special focus has been put on how the factors discourse or narrative relate to emotion and appraisal. Neurocinematics is studied for its focus on large-scale structure where music and language strongly interact. Published in the Strungmann Forum Reports Series.
Music can be defined as organized sound material in time. This chapter explores the links between the development of ideas about music and those driven by the concept of the embodied mind. Music composition has evolved from symbolic notated pitches to converge onto the expression of sound filigrees driven by new techniques of instrumental practice and composition associated with the development of new interfaces for musical expression. The notion of the organization of sound material in time adds new dimensions to musical information and its symbolic representations. To illustrate our point of view, a number of music systems are presented that have been realized as exhibitions and performances. We consided these synthetic music compositions as experiments in situated aesthetics. These examples follow the philosophy that a theory of mind, including one of creativity and aesthetics, will be critically dependent on its realization as a real-world artifact because only in this way can such a theory of an open and interactive system as the mind be fully validated. Examples considered include RoBoser, a real-world composition system that was developed in the context of a theory of mind and brain called distributed adaptive control (DAC), and “ADA: intelligent space,” where the process of music expression was transported from a robot arena to a large-scale interactive space that established communication with its visitors through multi-modal composition. Subsequently, re(per)curso, a mixed reality hybrid human–machine performance, is analyzed for ways of integrating the development of music and narrative. Finally, the chapter concludes with an examination of how multimodal control structures driven by brain–computer interfaces (BCI) can give rise to a brain orchestra that controls complex sound production without the use of physical interfaces. All these examples show that the production of sound material in time that is appreciated by human observers does not need to depend on the symbolically notated pitches of a single human composer but can emerge from the interaction between machines, driven by simple rules and their environment. Published in the Strungmann Forum Reports Series.
In electronic music, it is often useful to build loops from discrete events, such as playing notes or triggering digital e�ects. This process generally requires using a visual interface, as well as pre-de�ning tempo and time quantization. We present a novel digital musical instrument capable of looping events without using visual interfaces or explicit knowledge about tempo or time quantization. The instrument is built based on a prediction algorithm that detects repetitive patterns over time, allowing the construction of rhythmic layers in real-time performances. It has been used in musical performances, where it showed to be adequate in contexts that allow improvisation.
We present a new framework to study the interaction between musicians, technology and various capabilities that extend the expressive possibilities of live performance with electronics. We emphasize that extended techniques, the development of new interfaces and performance related to multimodal processes lead to the expansion of the musical universe, revitalize the viewer primary condition of the listeners enabling their interaction with other senses. Understanding musical performance under this new perspective extends the expressive resources for new music, especially when we join composition with the multiple facets of mediated performance.
Apresentamos neste artigo a instalação audiovisual interativa Abstrações, descrevendo sua proposta artística, implementação técnica e levantando apontamentos preliminares sobre a efetividade dos processos de interação com o público visitante observados durante a sua implementação.
- Sep 2012
- Conference on Interdisciplinary Musicology
Systematic musicology has long been investigating the relationship between musical styles of many composers. In fact, musical ideas are often invented and reinvented by means of cognitive processes that are, most of the time, unconscious. Many structural elements are transmitted from one composer to another, from one musical piece to another, contributing to the cycle of genesis, development and demise of musical styles In our research, we address this issue with the aid of computer systems that simulate real-world processes. Interactive musical systems are the ideal tool to achieve this goal. The processes in question are particularly related to the way humans process music information, from the perception of sound and musical structures, their storage in memory and subsequent use in the creation of new works.
Resumo: Neste artigo apresentamos um sistema computacional bioinspirado que utiliza Algoritmos Genéticos Interativos (AGI) para controle de diversidade dentro de uma população. Utilizamos AGIs como base conceitual para o comportamento dinâmico da instalação audiovisual interativa Cerejeira. A utilização de AGI apresentada neste trabalho destina-se ao controle interativo de uma instalação audiovisual onde os delimitadores são utilizados para controle dos processos de síntese sonora evolutiva desta instalação. Conforme explicado no artigo, as regras para controle do processo reprodutivo são definidas por delimitadores ambientais e o desenvolvimento genético é associado a regras de fitness que determinam o processo de seleção. Palavras-chave: composição, sonologia, algoritmo genético interativo, computação evolutiva, instalação multimídia Cerejeira: environmental delimiters to diversity control using interactive genetic algorithms Abstract: We present here a bio-inspired computing system that uses Interactive Genetic Algorithms (IGA) to genotype diversity control in an evolutionary population. We use IGAs as a conceptual basis for the dynamic activity of the, here presented, interactive audiovisual installation named Cerejeira. The usage of IGA presented in this article aims the interactive control of this installation, where delimiters are used to control the evolutionary sound synthesis computational model. The rules to control the process of reproduction are defined through environmental delimiters and the population genetic development is associated to rules of fitness given by the selection process.
- Aug 2012
- XXII Congresso da Associação Nacional de Pesquisa e Pós-Graduação em Música
This research is based on theories such as Markov chains, a Stochastic process presented by Iannis Xenakis in his book Formalized Music: thought and mathematics in composition; and Logistic Map, a specific case of Dynamic Systems applied to Computer Assisted Composition (CAC). The discussion is guided by the preliminary results obtained during the composition of two pieces: PARTITA and 7, generated from the implementation of the these two formal models in Pure Data (PD).
This article focuses on the interdisciplinary research involving Computer Music and Generative Visual Art. We describe the implementation of two interactive artistic systems based on principles of Gestural Data (WILSON, 2002) retrieval and self-organization (MORONI, 2003), to control the Evolutionary Sound Synthesis method (ESSynth). The frst implementation uses, as gestural data, image mapping of handmade drawings. The second one uses gestural data from dynamic body movements of dance. The resulting computer output is generated by an interactive system implemented in Pure Data (PD). This system uses principles of Evolutionary Computation (EC), which yields the generation of a synthetic adaptive population of sound objects. Considering that music could be seen as "organised sound" the contribution of our study is to develop a system that aims to generate "self-organised sound"; a method that uses evolutionary computation to bridge between gesture, sound and music. Resumo: Este artigo trata da pesquisa interdisciplinar envolvendo Música Computacional e Arte Visual Generativa. Descrevemos aqui a implementação de duas instalações artísticas interativos baseadas nos princípios de coleta de dados gestuais (WILSON 2002) e na auto-organização (MORONI 2003) para controlar um modelo de síntese evolutiva sonora (ESSynth). A primeira implementação utiliza como dados gestuais o mapeamento por visão computacional de imagens de desenhos feitos a mão. A segunda implementação usa dados estuais de movimentos dinâmicos de coreografa. O resultado computacional são dados gerados por um sistema interativo implementado em PureData (PD). Este sistema utiliza princípios de Computação Evolutiva que permite a geração de uma população sintética de objetos sonoros dinâmicos. Considerando que música pode ser defnida como "som organizado" a contribuição deste estudo é desenvolver um sistema computacional que gera "som auto-organizado", que se trata de um método evolutivo de computação evolutiva que intersecciona: gestos, som e música.
- Oct 2011
- 4th International Conference on Image and Signal Processing (CISP'11)
- International Congress on Image and Signal Processing
This paper aims to present an overview on Abstrações, an audiovisual installation controlled by video motion parameters. This article focuses on an implemented state machine model, the interaction between video motion recognition, the audio and video processing and the states of its implementation.
- Sep 2011
- 13º Simpósio Brasileiro de Computação Musical
The article presents concepts and methods developed to compose the rhythmic structure of "Projections on Symmetries and Self-Similarities", a piece for 9 percussionists. It mainly describes the mathematical modeling used in a compositional platform to compose this piece and also comments on its further application in a series of musical compositions.
This article presents RUBRICAS (2010), an interactive work involving musicians, dancers, actors and video artists. The aim of this work was to create a performance anchored in the notion of textural discourse and in the use of rubrics for commenting, stimulating and broadening the scope of interaction within the performance agents. This article evaluates the potential of collective actions, improvisation and technological support in multimodal performances.
Este artigo foi reproduzido do original final entregue pelo autor, sem edições, correções ou considerações feitas pelo comitê técnico. A AES Brasil não se responsabiliza pelo conteúdo. Outros artigos podem ser adquiridos através da Audio www.aes.org. Informações sobre a seção Brasileira podem ser obtidas em www.aesbrasil.org. Todos os direitos são reservados. Não é permitida a reprodução total ou parcial deste artigo sem autorização expressa da AES Brasil. ABSTRACT In this work we propose a Linear Downmix from Surround to Stereo Systems, which is capable to keep some psychoacoustics effects present in the Surround System also in the Stereo System from de downmix improvement from 5.1-to two-channel stereophonic system [3,4]. This can be achieved through linear control of signal amplitudes distributed among the surrounding speakers and two low pass filters for the rear speakers. Formally this control can be viewed as a solution of a nonhomogeneous linear equation system of two equations with five parameters. So at least a non-trivial solution always exists. Finally we present an implementation of our model in Pure Data and we discuss the design of the psychoacoustic aspects of our setup with the computer implementation.
The eXperience Induction Machine (XIM) is one of the most advanced mixed-reality spaces available today. XIM is an immersive space that consists of physical sensors and effectors and which is conceptualized as a general-purpose infrastructure for research in the field of psychology and human–artifact interaction. In this chapter, we set out the epistemological rational behind XIM by putting the installation in the context of psychological research. The design and implementation of XIM are based on principles and technologies of neuromorphic control. We give a detailed description of the hardware infrastructure and software architecture, including the logic of the overall behavioral control. To illustrate the approach toward psychological experimentation, we discuss a number of practical applications of XIM. These include the so-called, persistent virtual community, the application in the research of the relationship between human experience and multi-modal stimulation, and an investigation of a mixed-reality social interaction paradigm. KeywordsMixed-reality–Psychology–Human–computer interaction–Research methods–Multi-user interaction–Biomorphic engineering
The creation of an artwork named RePartitura is discussed here under principles of Evolutionary Computation (EC) and the triadic model of thought: Abduction, Induction and Deduction, as conceived by Charles S. Peirce. RePartitura uses a custom-designed algorithm to map image features from a collection of drawings and an Evolutionary Sound Synthesis (ESSynth) computational model that dynamically creates sound objects. The output of this process is an immersive computer generated sonic landscape, i.e. a synthesized Soundscape. The computer generative paradigm used here comes from the EC methodology where the drawings are interpreted as a population of individuals as they all have in common the characteristic of being similar but never identical. The set of specific features of each drawing is named as genotype. Interaction between different genotypes and sound features produces a population of evolving sounds. The evolutionary behavior of this sonic process entails the self-organization of a Soundscape, made of a population of complex, never-repeating sound objects, in constant transformation, but always maintaining an overall perceptual self-similarity in order to keep its cognitive identity that can be recognize for any listener. In this article we present this generative and evolutionary system and describe the topics that permeates from its conceptual creation to its computational implementation. We underline the concept of self-organization in the generation of soundscapes and its relationship with computer evolutionary creation, abductive reasoning and musical meaning for the computational modeling of synthesized soundscapes.
Apresentamos os aspectos gerais de nossa pesquisa, que envolve o estudo da relação entre instrumentos musicais, técnica instrumental e repertório. Novas interfaces musicais poderão ser especialmente beneficiadas pela pesquisa nesta área, na medida em que possam usufruir da experiência acumulada nos séculos de existência dos instrumentos tradicionais. Em especial, técnicas e modos de execução podem servir como elemento unificador dessas práticas.
- Apr 2010
- Applications of Evolutionary Computation, EvoApplications 2010: EvoCOMNET, EvoENVIRONMENT, EvoFIN, EvoMUSART, and EvoTRANSLOG, Istanbul, Turkey, April 7-9, 2010, Proceedings, Part II
A new approach is presented which integrates evolutionary computation and real world devices such as mobile robots and an omnidirectional vision system. Starting with an evolutionary composition system named JaVOX, a hybrid environment named AURAL evolved. In the AURAL, the behavior of mobile robots in an arena is applied as a compositional strategy. It uses trajectories produced by mobile robots to modify the fitness function of a real time sonic composition. The model is described, its evolutionary design and how the interaction between the real world devices was implemented. This research is oriented towards the study of automatic and semi-automatic processes of artistic production in the sound domain.
A new approach which integrates an interactive evolutionary interface for sound production, real world devices, such as mobile robots, and an omnidirectional vision system, is presented. Starting with an evolutionary composition system named VOX POPULI and later JaVOX, a hybrid environment named AURAL has evolved. It uses curves drawn in a GUI and trajectories produced by mobile robots to modify the fitness function of a real time sonic composition. An arena was constructed to allow the interaction between robots and to control the variations of sonic parameters producing real time MIDI data. The navigation supervisor system, the sonification environment and graphic results of real time experiments are described.
- Oct 2009
A new approach which integrates an omnidirectional vision system, real world devices, such as mobile robots, and an interactive evolutionary interface for sound production, is presented. Starting with an evolutionary composition system named VOX POPULI and later JaVOX, a hybrid environment named AURAL has evolved. It uses curves drawn in a GUI and trajectories produced by mobile robots to modify the fitness function of a real time sonic composition. An arena was constructed to allow the interaction between robots and to control the variations of sonic parameters producing real time MIDI data. The navigation supervisor system, the sonification environment and graphic results of real time experiments are briefly described.
In Generative Art produced with computational support, gestural data has been increasingly considered an important source of information that is, at the same time, intuitively created and conveying the artistic meaning of the resulting artwork. In the same way, the concept of self-organization has been used in the study of human creativity and art production. This article describes the implementation of two interactive multi-modal artworks that use these two principles to compute adaptive sonifications. As gestural data, the first implementation uses the mapping of a handmade drawing collection. The second one uses the retrieval of body action movements performed by a dancer. The resulting soundscapes are created by a dynamic system implemented in PD (PureData). It uses principles of Evolutionary Computation (EC), which yields to the creation of a synthetic adaptive population of sound objects based on the retrieved gestural data. That emulates the biological evolution of individuals undergoing the processes of selection and reproduction. The overall sound is constantly generated by all individuals within the population. This is the system output, which can be described as a self-organized synthetic soundscape engendered by the initial artistic generated gestural information.
Virtual and mixed reality environments (VMRE) often imply full-body human-computer interaction scenarios. We used a public multimodal mixed reality in-stallation, the Synthetic Oracle, and a between-groups design to study the ef-fects of implicit (e.g., passively walking) or explicit (e.g., pointing) interaction modes on the users' emotional and engagement experiences, and we assessed it using questionnaires. Additionally, real-time arm motion data was used to cat-egorize the user behavior and to provide interaction possibilities for the explicit interaction group. The results show that the online behavior classification corre-sponded well to the users' interaction mode. In addition, contrary to the ex-plicit interaction, the engagement ratings from implicit users were positively cor-related with a valence but were uncorrelated with arousal ratings. Interestingly, arousal levels were correlated with different behaviors displayed by the visitors depending on the interaction mode. Hence, this study confirms that the activity level and behavior of users modulates their experience, and that in turn, the interaction mode modulates their behavior. Thus, these results show the impor-tance of the selected interaction mode when designing users' experiences in VMRE.
Virtual and mixed reality environments (VMRE) often imply full-body human-computer interaction scenarios. We used a public multimodal mixed reality installation, the Synthetic Oracle, and a between-groups design to study the effects of implicit (e.g., passively walking) or explicit (e.g., pointing) interaction modes on the users' emotional and engagement experiences, and we assessed it using questionnaires. Additionally, real-time arm motion data was used to categorize the user behavior and to provide interaction possibilities for the explicit interaction group. The results show that the online behavior classification corresponded well to the users' interaction mode. In addition, contrary to the explicit interaction, the engagement ratings from implicit users were positively correlated with a valence but were uncorrelated with arousal ratings. Interestingly, arousal levels were correlated with different behaviors displayed by the visitors depending on the interaction mode. Hence, this study confirms that the activity level and behavior of users modulates their experience, and that in turn, the interaction mode modulates their behavior. Thus, these results show the importance of the selected interaction mode when designing users' experiences in VMRE.
Resumo: O gesto, sempre presente na criação da obra artística, seja esta plástica ou musical, vem sendo recentemente explorado por ferramentas tecnológicas que permitem seu interfaceamento multimodal. Este trabalho trata de um mapeamento sinestésico de gestos, formadores dos desenhos conceituais, em objetos sonoros. A imagem de um desenho é aqui vista não como um fim, mas como a representação de uma forma no decorrer do tempo. Esta, por sua vez, é o registro de um movimento contendo uma intenção expressiva. O som resultante é aqui composto por objetos sonoros que são unidades formantes de um sistema sônico maior, auto-organizado em uma paisagem sonora dinâmica. Palavras-chave: som, imagem, sinestesia, processo criativo, computação musical, síntese evolutiva, sonologia.
Part of the University of Campinas (UNICAMP, www.unicamp.br), the Interdisciplinary Nucleus of Sound Communication (NICS, www.nics.unicamp.br) was founded in 1983 by the composer Raul do Valle and collaborators. Since then, NICS has maintainined a longstanding list of participations in the academic and artistic musical scenario, specially the ones concerning the application of mathematical models and computing technology in sound synthesis methods and electro-acoustic music composition. Here we describe the major activities that NICS has been recently engaged with, specially in the areas of interactive performance, computer music, robotics, evolutionary computation applied to musical creation and, recently, physiologic signals acquisition for the retrieval of cognitive and emotive musical features.
- Jan 2009
- Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, HRI 2009, La Jolla, California, USA, March 9-13, 2009
This study aims to provide a platform for exploring robotic navigation in line with evolutionary computation of sound control data. Real world devices, two mobile robots and an omnidirectional vision system are integrated to sonify trajectories of robots in real time.
Two implementations of an Evolutionary Sound Synthesis method using the Interaural Time Difference (ITD) and psychoacoustic descriptors are presented here as a way to develop criteria for fitness evaluation. We also explore a relationship between adaptive sound evolution and three soundscape characteristics: keysounds, key-signals and sound-marks. Sonic Localization Field is defined using a sound attenuation factor and ITD azimuth angle, respectively (Ii, Li). These pairs are used to build Spatial Sound Genotypes (SSG) and they are extracted from a waveform population set. An explanation on how our model was initially written in MATLAB is followed by a recent Pure Data (Pd) implementation. It also elucidates the development and use of: parametric scores, a triplet of psychoacoustic descriptors and the correspondent graphical user interface.
This report describes continuaMENTE, an interactive audiovisual piece composed for tape, texts, video, interactive percussion and live electronics. It was commissioned by the Itaú Cultural Foundation, São Paulo and created at the Interdisciplinary Nucleus for Sound Studies (NICS), Unicamp, Brazil. First performed on August 2007 in the exposition "Memória do Futuro" (Memory of the Future), continuaMENTE integrated several materials with real-time sounds and music's gestures produced by three percussionists. The use of three interfaces: interactive mallets, gloves and a carpet, and the musiciansáctions generated complex sound textures on a MIDI controlled piano. This report describes the conceptual view, used resources, sonic strategies and graphic notation, and the performance rules related to this work.
Music is well known for affecting human emotional states, yet the relationship between specific musical parameters and emotional responses is still not clear. With the advent of new human-computer interaction (HCI) technologies, it is now possible to derive emotion-related information from physiological data and use it as an input to interactive music systems. This raises the question of how musical parameters are mapped to emotional states. We assess this question using both verbal and physiological responses. While most of the work on musical interfaces is based on explicit HCI, e.g. involving gestures, we study the potential of implicit interaction based on human emotional states. Our results show that a significant correlation exists between electrodermal activity, heart rate, heart rate variability and the subjective evaluation of well-defined musical parameters. Hence this demonstrates the feasibility of automated music composition based on physiological feedback. Providing implicit musical HCI will be highly relevant for a number of applications including music therapy, automatic generation of music for interactive virtual story telling and games, music for video games and physiologically-based musical instruments.
We synchronize cameras and analog lighting with high speed projectors. Radiometric compensation allows displaying flexible blue screens on arbitrary real world surfaces. A fast temporal multiplexing of coded projection and flash illumination enables ...
This paper describes the OmniEye, an omnidirectional vision system developed to track mobile robots in AURAL, i. e., in a computational structured environment. AURAL aims to control the interaction between visual, sound and robotic information in a research for automatic and semi-automatic processes of artistic production. Different convex mirrors can be used to achieve an omnidirectional system. The use of a spherical mirror in this case introduces distortions in the image. A toolbox for the calibration of central omnidirectional cameras was used to obtain a first estimation for the imaging function. Then, a genetic algorithm was applied to adjust the coefficients of the imaging function. Experimental results and the application of the OmniEye for translating robotic paths into sound events in the AURAL environment are described.
Resumo: Este artigo apresenta aspectos teóricos sobre a interpretação de obras que envolvem elementos sonoros, visuais e o espaço físico. A relação entre a natureza e o significado da música é apresentada no contexto da interação com outros domínios diferentes do sonoro. Discute-se que tanto intérpretes como espectadores ainda não estão familiarizados com a diversidade de elementos e expressões num contexto multisensorial. Duas composições que abordam este tema são apresentadas. Na primeira desenvolve-se a interação entre som e luz e na segunda entre o som e o tato. Finalmente, discute-se que em obras desta natureza amplia-se a dimensão da interpretação através de uma ação co-criativa. Palavras-chave: multimodal, sinestesia, instalação sonora.
This paper describes the OmniEye, an omnidirectional vision system developed to track mobile robots in AURAL, a computational structured environment. AURAL aims to control the interaction between visual, sound and robotic information in a research for automatic and semi-automatic processes of artistic production. Different convex mirrors can be used to achieve an omnidirectional system. The use of a spherical mirror in this case introduces distortions in the image. A toolbox for the calibration of central omnidirectional cameras was used to obtain a first estimation for the imaging function. On a second step, a genetic algorithm was applied to adjust the coefficients of the imaging function. Experimental results and the application of the OmniEye for translating robotic paths into sound events in the AURAL environment are described.
An ongoing research on expressivity of interactive composition and performances is presented. This study is based on three frameworks: psychology of anticipation, improvisation understood as a musical development process and multimodality used to enhance expressiveness and immersion among composer(s), interpret(s) and audience.
The boundaries of art are subjective, but the impetus for art is often associated with creativity, regarded with wonder and admiration along human history. Most interesting activities and their products are a result of creativity. The main goal of our approach is to explore new creative ways of editing and producing videos, using evolutionary algorithms. A creative evolutionary system makes use of evolutionary computation operators and properties and is designed to aid our own creative processes, and to generate results to problems that traditionally required creative people to solve. Our system is able to generate new videos or to help a user in doing so. New video sequences are combined and selected, based on their characteristics represented as video annotations, either by defining criteria or by interactively performing selections in the evolving population of video clips, in forms that can reflect editing styles. With evolving video, the clips can be explored through emergent narratives and aesthetics in ways that may reveal or inspire creativity in digital art.
A new way to control sound spatial dispersion using the ESSynth Method is introduced here. The Interaural Time Difference (ITD) is used as genotype of an evolutionary control of sound spatialization. Sound intensity and the ITD azimuth angle are used to define spatial dispersion and spatial similarity. Experimental results where crossover and mutation rates were used to create spatial sonic trajectories are discussed.
Until recently, the sonification of Virtual Environments had often been reduced to its simplest expression. Too often soundscapes and background music are predetermined, repetitive and somewhat predictable. Yet, there is room for more complex and interesting sonification schemes that can improve the sensation of presence in a Virtual Environment. In this paper we propose a system that automatically generates original background music in real-time called VR-RoBoser. As a test case we present the application of VR-RoBoser to a dynamic avatar that explores its environment. We show that the musical events are directly and continuously generated and influenced by the behavior of the avatar in three-dimensional virtual space, generating a context dependent sonification.
It is generally admitted that music is a powerful carrier of emotions (4, 21), and that audition can play an important role in enhancing the sensation of presence in Virtual Environments (5, 22). In mixed-reality environments and interactive multi-media systems such as Massively Multiplayer Online Games (MMORPG), the improvement of the user's perception of immersion is crucial. Nonetheless, the sonification of those environments is often reduced to its simplest expression, namely a set of prerecorded sound tracks. Background music many times relies on repetitive, predetermined and somewhat predictable musical material. Hence, there is a need for a sonification scheme that can generate context sensitive, adaptive, rich and consistent music in real-time. In this paper we introduce a framework for the sonification of spatial behavior of multiple human and synthetic characters in a Mixed-Reality environment. Previously we have used RoBoser (1) to sonify different interactive installation including the interaction between humans and a large-scale accessible space called Ada (2) Here we are investigating the applicability of the RoBoser framework to the sonification of the continuous and dynamic interaction between individuals populating a mixed-reality space. We propose a semantic layer that maps sensor data into intuitive parameters for the control of music generation, and show that the musical events are directly influenced by the spatial behavior of human and synthetic characters in the space, thus creating a behavior-dependant sonification that enhance the user's perception of immersion.
This work presents an interactive device to control an adaptive tuning and synthesis system. The gestural controller is based on the theremin concept in which only an antenna is used as a proximity sensor. This interactive process is guided by sensorial consonance curves and adaptive tuning related to psychoacoustical studies. We used an algorithm to calculate the dissonance values according to amplitudes and frequencies of a given sound spectrum. The theoretical background is presented followed by interactive composition strategies and sound results.
Here is presented an approach for interactive soundscape design based on Interaural Time Difference (ITD) cues and Evolutionary Computation (EC). We define a Sonic Localization Field (SLF) where the pair of parameters: sound intensity and ITD azimuth localization angle are the Spatial Sound Genotypes (SSG) that control the adaptive sonic evolution. A Pure data (Pd) software application working with a parametric score is proposed as a way of dynamically guiding the automatic generation of soundscapes.
Future mixed reality systems will need to support large numbers of simultaneous, nonexpert users at reasonable per-user costs if the systems are to be widely deployed within society in the short to medium term. We have constructed a prototype of such a system, an interactive entertainment space called Ada that was designed to behave like a simple organism. Using Ada we conducted two studies: the first assessing the ef- fect of varying the operating parameters of the space on the collective behavior and attitudes of its users, and the second assessing the relationships among user demograph- ics, behavior, and attitudes. Our results showed that small changes in the ambient set- tings of the environment have a significant effect on both user attitudes and behavior, and that the changes in user attitudes do not necessarily correspond to the environ- mental changes. We also found that individual user opinions are affected by demo- graphics and reflected in overt behavior. Using these results, we propose some tenta- tive guidelines for the design of future shared mixed reality spaces.
Apresentamos neste artigo aspectos introdutórios de nossa pesquisa de Doutorado, que visa a desenvolver um modelo computacional para o estudo da "affordance musical" sob o ponto de vista da técnica instrumental. Nosso objetivo é investigar possíveis relações existentes entre as técnicas instrumentais tradicionais e as técnicas aplicadas a novas interfaces musicais. Palavras-Chave: affordance, performance, interface musical, técnica instrumental.
- Jul 2006
- Smart Graphics, 6th International Symposium, SG 2006, Vancouver, Canada, July 23-25, 2006, Proceedings
Computational creativity is certainly interesting and potentially important. ArTVox, a Java programmed evolutionary environment, arose from the attempt to emulate computational creativity applied to artistic production, in visual and sound domains, by using interactive genetic algorithms. Objects inspired in Kandinsky’s artworks are being programmed in Shape, an auxiliary Java environment, to be inserted in ArTVox. Today, ArTVox creates and evolves visual compositions of geometric primitives that, by their turn, guide the sound production in another evolutionary environment, JaVox, integrated to ArTVox.
We propose a new multimedia authoring paradigm based on evolutionary computation, video annotation, and cinematic rules. New clips are produced in an evolving population through genetic transformations influenced by user choices, and regulated by cinematic techniques like montage and video editing. The evolutionary mechanisms, through the fitness function will condition how video sequences are retrieved and assembled, based on the video annotations. The system uses several descriptors, as genetic information, coded in an XML document following the MPEG-7 standard. With evolving video, the clips can be explored and discovered through emergent narratives and aesthetics in ways that inspire creativity and learning about the topics that are presented.
- Apr 2006
- Applications of Evolutionary Computing, EvoWorkshops 2006: EvoBIO, EvoCOMNET, EvoHOT, EvoIASP, EvoINTERACTION, EvoMUSART, and EvoSTOC, Budapest, Hungary, April 10-12, 2006, Proceedings
We present a theoretical evolutionary musical accompaniment generating system capable of evolving to different organized sounds according to an external performer. We present a new approach for implementing the fitness functions.
Generating sounds for music composition with the desired timbral characteristics has been a challenge ever since the dawn of electroacoustic music. Timbre is a remarkably complex phenomenon that has puzzled researchers for a long time. Actually, the nature of musical signals is not fully understood yet. In this paper, we present a sound synthesis technique that uses Kohonen's one-dimensional self-organizing map to generate neuronal-sounds to respond to a fixed and predefined set of stimulus-sounds, producing timbral variants with the desired characteristics. The self-organizing algorithm provides maintenance of topology so that the intended aesthetical result is properly achieved by avoiding the formal definition of the timbral attributes. To evaluate the obtained results we propose crossing a mathematical/subjective spectral distance from the neuronal-sounds to the stimulus-sounds with the method of timbral classification using Kohonen's two-dimensional self-organizing map.
This paper presents a new model for measuring similarity in a general Rhythm Space. Similarity is measured by establishing a comparison between subsequences of a given rhythm. We introduce the hierarchical subdivision of rhythm sequences in several levels, and compute a Distance Matrix for each level using the "block distance". The information about the similarity of the rhythmic substructures is retrieved from the matrices and coded into a Similarity Coefficient Vector (SCV). We also present possibilities for the reduction to single values of similarity derived from the SCV. In addition, two applications of the formal model are presented, showing the potential for development using this approach.
We present a novel paradigm for the interactive composition and performance of music called Roboser consisting of a real-world device (i.e., a robot), its control software, and a composition engine that produces streams of MIDI data in real time. To analyze the properties of this framework, we present the application of Roboser to a learning mobile robot, called EmotoBot, that is controlled by the Distributed Adaptive Control (DAC) architecture. The EmotoBot composition is based on the generation of real-time sound events that express sensory, behavioral, and the internal states of the robot's control model. We show that EmotoBot produces a complex set of sonic layers and quantify its ability to generate complex emergent sonic structures. We subsequently describe further applications of the Roboser framework to other interactive systems, including a large-scale interactive exhibition called Ada. Our results show the potential of the Roboser paradigm to define the central-processing stage of interactive composition systems. Moreover, Roboser provides a general framework for transforming information from real-world systems into complex sonic structures and as such constitutes a real-world composition system.
- Aug 2005
- Artificial Immune Systems: 4th International Conference, ICARIS 2005, Banff, Alberta, Canada, August 14-17, 2005, Proceedings
- International Conferences on Artificial Immune Systems
Computer generated sounds for music applications have many facets, of which timbre design is of groundbreaking significance. Timbre is a remarkable and rather complex phenomenon that has puzzled researchers for a long time. Actually, the nature of musical signals is not fully understood yet. In this paper, we present a sound synthesis method using an artificial immune network for data clustering, denoted aiNet. Sounds produced by the method are referred to as immunological sounds. Basically, antibody-sounds are generated to recognize a fixed and predefined set of antigen-sounds, thus producing timbral variants with the desired characteristics. The aiNet algorithm provides maintenance of diversity and an adaptive number of resultant antibody-sounds (memory cells), so that the intended aesthetical result is properly achieved by avoiding the formal definition of the timbral attributes. The initial set of antibody-sounds may be randomly generated vectors, sinusoidal waves with random frequency, or a set of loaded waveforms. To evaluate the obtained results we propose an affinity measure based on the average spectral distance from the memory cells to the antigen-sounds. With the validation of the affinity criterion, the experimental procedure is outlined, and the results are depicted and analyzed.
RESUMO A finalidade desta proposta é a aplicação de inteligência artificial (IA) e técnicas bio-inspiradas na composição e improvisação. A composição e a improvisação musicais são processos criativos que podem ser descritos em termos de uma busca estética no espaço de estruturas candidatas; neste caso, o objetivo final é criar estruturas musicais esteticamente agradáveis. Esta busca pode ser executada em níveis estruturais diferentes, desde padrões rítmicos ou melódicos macroscópicos até as dimensões timbrais microscópicas do som musical. Pode-se ou não incluir o usuário no sistema, criando diversos níveis de interatividade. Ou ainda, críticos artificiais de música poderiam ser usados (ou até co-evoluidos com o material musical), extraindo características da arte-final e as classificando de acordo com determinados critérios. No caso particular da composição musical, uma vez que o espaço da busca e os objetivos são definidos, é necessário uma técnica para atingir o resultado final. Quando nós consideramos improvisação musical, não há nenhum objetivo explícito e certamente nada de específico é esperado como resultado final, mas é o trajeto através do espaço da busca que é de interesse. Muitas abordagens foram propostas para o uso de IA e de técnicas bio-inspiradas na música, variando de autômatos celulares, computação evolutiva, enxames de partículas, redes neurais artificiais, sistemas imunes artificiais, vida artificial, entre outras. Este trabalho se destina a explorar a auto-organização como paradigma de composição e improvisação dentro do contexto da música computacional através da aplicação de algoritmos bio-inspirados como candidatos apropriados para abordar os vários problemas encontrados durante o processo subjetivo da experiência musical. INTRODUÇÃO Música computacional é um campo em constante crescimento em parte porque permite ao compositor grande flexibilidade na manipulação sonora ao procurar pelo resultado desejado. No exemplo particular da composição musical, uma vez que o espaço de busca e os objetivos são definidos, uma técnica para conseguir o produto final é requerida. Dentro da estrutura deste trabalho, quando consideramos improvisação musical, não há objetivo final e nada como um resultado final. É o trajeto através do espaço da busca que é de interesse. Muitas abordagens diferentes foram propostas para atender as exigências dos processos de composição e improvisação, isto é criar música esteticamente interessante, com resultados que variam do inesperado ao indesejado, dependendo de um número vasto dos fatores e da própria metodologia. As técnicas tradicionais de síntese sonora apresentam limitações especialmente devido ao fato de não levarem em consideração a natureza dinâmica e/ou subjetiva da música, usando processos que são demasiadamente simples ou não especificamente desenvolvidos para a manipulação de sons musicais. Sons musicais são notavelmente complexos e difíceis de gerar, pois eles pertencem a uma classe distinta de sons que apresentam determinadas características. Tais sons têm geralmente espectros dinâmicos, isto é cada freqüência parcial tem uma evolução temporal do envelope única. Eles são ligeiramente inarmônicos e as parciais possuem uma flutuação de alta freqüência e baixa-amplitude de natureza estocástica. Os parciais possuem assincronia de ataque, isto é, parciais mais altas atacam depois dos parciais mais baixos. Nossos ouvidos são altamente seletivos e freqüentemente rejeitam sons que são muito matematicamente perfeitos e estáveis.
This paper and the accompanying video describe Ada, an immersive 160m space for human-machine interaction. Ada is conceived as an artificial organism that plays with its visitors, with multi-modal sensors consisting of pressure-sensitive floor tiles, microphones for localising and recognising sounds, and pan-tilt video cameras called gazers. Her output modalities include a 360 video projection screen capable of showing realtime computer graphics combined with multiple live video streams, RGB neon lights in the floor tiles, pan-tilt lights and an automatic neural music composition system. Visitors provided input to the space primarily by moving around, clapping their hands and speaking. Ada was presented at the Swiss national exhibition Expo.02 in Neuchtel from May 15 to October 20, 2002, and attracted 553,700 visitors. In this paper we give a brief overview of the Ada exhibit, and describe the accompanying video.
We conceived Ada: Intelligent Space exhibit as an artificial organism, integrating a large number of sensory modalities, and let it interact with visitors using a multitude of effector systems. Ada used a language of sound and light to communicate its moods, emotions, and behaviors. Here we describe the mechanisms behind Ada's sound communication, its real-time performance, and its interpretation by human subjects.
Buildings are usually conceived as expressing a static functionality where users are required to adapt to its pre-defined properties. This view of architecture is outdated. Buildings can be conceived and constructed as perceiving, acting and adapting entities not unlike biological systems. Although several existing projects have aimed at developing "smart" rooms or buildings, in most of these attempts information technology is used in a strictly local way. "Ada: Intelligent space" goes beyond this utilitarian perspective and aims to develop a space that can dynamically change its overall functionality and quality through an active dialog with its visitors.
This paper describes the basic system architecture of Visual Roboser that transforms image sequences into sonic structures. It is an extension of the Roboser Project and it uses images captured from the environment in real time to drive the production of sonic structures. It is based on an architecture that integrates a neural simulator and two JAVA modules: one for image processing and another for sound generation. Possible applications of Visual Roboser include real time composition, multimedia live performances, sonic installations and intelligent interactive rooms such as our latest work name "Ada: the Intelligent Space" (http://www.ada-exhibition.ch).
While much is now known about the operation and organisation of the brain at the neuronal and microcircuit level, we are still some way from understanding it as a complete system from the lowest to the highest levels of description. One way to gain such an integrative understanding of neural systems is to construct them. We have built the largest neuromorphic system yet known, an interactive space called 'Ada' that is able to interact with many people simultaneously using a wide variety of sensory and behavioural modalities. 'She' received 553,700 visitors over 5 months during the Swiss Expo.02 in 2002. In this paper we present the broad motivations, design and technologies behind Ada, and discuss the construction and analysis of the system.
Ada is an entertainment exhibit that is able to interact with many people simultaneously, using a language of light and sound. "She" received 553,700 visitors over 5 months during the Swiss Expo.02 in 2002. In this paper we present the broad motivations, design and technologies behind Ada, and a first overview of the outcomes of the exhibit.
- Dec 2002
- Creative Evolutionary Systems
This chapter introduces a new system, Vox Populi, based on evolutionary computation, for composing music in real time. A population of chords is properly codified according to the MIDI protocol and evolves by the application of genetic algorithms. A fitness criterion is defined to indicate the best chord in each generation, and this chord is selected as the next element in the sequence to be played. Each new generated chord is a new sound palette that a musician can use to continue the music evolution. Graphic controls provide user-friendly manipulation of the fitness and of the sound attributes. Evolutionary computation is used to stimulate the user with novel sounds, and it allows the user to respond. Associating the dynamic behavior of genetic algorithms with these tools for real-time interaction, Vox Populi becomes a musical instrument. But unlike a traditional instrument, Vox Populi is able to create its own sound raw material and to provide choice criteria simultaneously. All these features enhance the user's music capabilities and mark this system as state of the art in computer music. Next, a general description of the main components of the computational environment and melodic, harmonic, and voice range criteria for musical fitness are defined. The chapter also explains the genetic encoding of notes and the evolutionary cycle for chord production.