Article

Drumkit simulator from everyday desktop objects

Springer Nature
Multimedia Tools and Applications
Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

In this paper, an augmented reality application for drumkit simulation is presented. The system is capable of classifying any percussive sounds produced by the user from an everyday desktop environment, e.g. clapping, snapping, stroking different objects with a pencil, etc., recognizing up to six different classes of drum hits. These different types of user-generated sounds will subsequently be associated to predefined drumkit sounds, resulting in a natural and intuitive audio interface for drummers and percussionists, which only requires a computer with a built-in microphone. A set of audio features and classification techniques are evaluated for the implementation of the aforementioned system.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Due to these advantages, AR technology, which has become widespread in the educational fields mentioned above, has also been introduced in music education, and related studies have been increasing. Upon reviewing the literature, it is seen that some of the studies using AR technology in the field of music are related to the development of applications for fun and game-based learning (Herrero et al., 2015;Zhou et al., 2004) and instrument design (Chouvatut & Jindaluang, 2013;Zhaparov & Assanov, 2014). However, throughout the last decade, research using this new technology has been linked to the theoretical and instrumental components of music education. ...
Article
Full-text available
This study aims to examine the studies scanned in Web of Science and Scopus databases between 2006-2020 on the use of augmented reality applications in music education in terms of their descriptive features, methodological features, and outcomes, and to reveal the trends in this field. The criterion sample approach was utilized in this qualitative study, and 35 selected studies were reviewed using the publication classification form and analyzed using the content analysis method. The descriptive and methodological feature data were translated into frequency values using the SPSS 22 program and then interpreted using tables and charts for easy comprehension. Furthermore, the advantages and limitations of AR applications utilized in music education, which were the research's outputs, were coded, organized into relevant categories, and interpreted. The results showed that most studies focused on piano and guitar education for beginners; no studies were undertaken on wind instruments or voice training. Studies, particularly in recent years, have focused on user experience research. Furthermore, it was found that AR applications have the advantages of increasing and facilitating student learning performance, making learning exciting and fun, and providing motivation. Some challenges during use, owing to technical issues and limited field content in the programs, resulted in limited improvements in music education.
Conference Paper
Full-text available
Hyper-instruments extend traditional acoustic instruments with sensing technologies that capture digitally subtle and sophisticated aspects of human performance. They leverage the long training and skills of performers while simultaneously providing rich possibilities for digital control. Many existing hyper-instruments suffer from being one of a kind instruments that require invasive modifications to the underlying acoustic instrument. In this paper we focus on the pitched percussion family and describe a non-invasive sensing approach for extending them to hyper-instruments. Our primary concern is to retain the technical integrity of the acoustic instrument and sound production methods while being able to intuitively interface the computer. This is accomplished by utilizing the Kinect sensor to track the position of the mallets without any modification to the instrument which enables easy and cheap replication of the proposed hyper-instrument extensions. In addition we describe two approaches to higher-level gesture control that remove the need for additional control devices such as foot pedals and fader boxes that are frequently used in electro-acoustic performance. This gesture control integrates more organically with the natural flow of playing the instrument providing user selectable control over filter parameters, synthesis, sampling, sequencing, and improvisation using a commercially available low-cost sensing apparatus.
Article
Full-text available
This paper presents a framework called Music via Motion (MvM) designed for the transdomain mapping between physical movements of the performer(s) and multimedia events, translating activities from one creative domain to another-for example, from physical gesture to audio output. With a brief background of this domain and prototype designs, the paper describes a number of inter- and multidisciplinary collaborative works for interactive multimedia performances. These include a virtual musical instrument interface, exploring video-based tracking technology to provide an intuitive and nonintrusive musical interface, and sensor-based augmented instrument designs. The paper also describes a distributed multimedia-mapping server which allows multiplatform and multisensory integrations and presents a sample application which integrates a real-time face tracking system. Ongoing developments and plausible future explorations on stage augmentation with virtual and augmented realities as well as gesture analysis on the correlations of musical gesture and physical gesture are also discussed.
Article
Full-text available
Disembodied voices" is an interactive environment de-signed for an expressive, gesture-based musical performance. The motion sensor Kinect, placed in front of the performer, provides the computer with the 3D space coordinates of the two hands. The application is designed according to the metaphor of the choir director: the performer, through gestures, is able to run a score and to produce a real-time expressive interpretation. The software, developed by the authors, interprets the gestural data and controls articulated events to be sung and expressively performed by a vir-tual choir. Hence the name of the application: you follow the conductor's gestures, hear the voices but don't see any singer. The system also provides a display of motion data, a visualization of the part of the score performed at that time, and a representation of the musical result processed by the compositional algorithm.
Article
Full-text available
Conductors play an important role in an ensemble. They are essential to an ensemble functioning as a single coherent unit, being responsible for controlling tempo and dynamics, whilst also channeling the emotional intensity of a piece. Traditional conducting requires visual cues and is challenged by contemporary composers using distributed performance techniques, where the performers' view of the conductor may be obscured. This paper proposes an interface to enhance coordination in distributed performance. The interface is a non-intrusive system for communicating conductor information via haptic feedback. Data is collected using sensors and mapped into haptic feedback and transmitted via a wireless connection. To verify its effectiveness an experiment is designed where participants attempt to discern the characteristics of beat and time signature from a vibrating motor. The paper concludes with the latest findings and discusses future directions and impacts for the system.
Conference Paper
Full-text available
This paper presents the results of a study of conductor gesture and examines the effect of that gesture on musician action within a tertiary music rehearsal and pedagogical context. This study follows experimental work on conductor gesture and attempts to complement the results of experimental studies by using a corpus of naturally-occurring video recorded data of conducting classes. The aim is to determine whether the physical actions (gestures) of conductors can be seen to be systematically related to musician action (playing of instruments) in such naturally occurring data. In addition, we explore how quantitative analytical techniques, more familiar to experimental approaches, can be used as the basis for a framework for examining conductor gesture in data collected using ethnographic research approaches. This study concludes that conductor gesture has a significant effect on musician action, and suggests some salient features of conductor gesture.
Chapter
Full-text available
Four audio feature sets are evaluated in their ability to differentiate five audio classes: popular music, classical music, speech, noise and crowd noise. The feature sets include low-level signal properties, mel-frequency spectral coefficients, and two new sets based on perceptual models of hearing. The temporal behavior of the features is analyzed and parameterized and these parameters are included as additional features. Using a standard Gaussian framework for classification, results show that the temporal behavior of features is important for automatic audio classification. In addition, classification is better, on average, if based on features from models of auditory perception rather than on standard features.
Article
Full-text available
We present an exhaustive review of research on automatic classification of sounds from musical instruments. Two different but complementary approaches are examined, the perceptual approach and the taxonomic approach. The former is targeted to derive perceptual similarity functions in order to use them for timbre clustering and for searching and retrieving sounds by timbral similarity. The latter is targeted to derive indexes for labeling sounds after culture- or user-biased taxonomies. We review the relevant features that have been used in the two areas and then we present and discuss different techniques for similarity-based clustering of sounds and for classification into pre-defined instrumental categories.
Article
Full-text available
As the increase of the sales of Wii game consoles, the Wii Remote is able to used as a common input device for a com-puter system. Some software is developed for acquiring the data from the Wii Remote for further processing. This paper presents a Wii-based gestural interface for computer-based conducting systems. It employs the infrared camera in a Wii Remote to capture the conducting gestures of a conduc-tor who is holding an infrared baton, and then represents the gestures using visual and aural representations after data analysis and gesture classification/following. The interface is intended to be used for pedagogy purposes. So it accepts standard conducting gestures, uses an infrared baton simi-lar to a real baton and displays the trajectory of a gesture on the screen which can be compared to the corresponding diagram shown in a textbook. In addition, aural feedback is also supported in this interface. It plays a MIDI note to represent a certain beat in a conducting gesture.
Conference Paper
Full-text available
Music is appreciated by people from all walks of life. Music fits into our daily schedules in many ways, from casual listening to cul-tural events or movies. At a deeper level, the mechanics of music are not usually known to most lay people and learning the compo-nents of music theory can be a lengthy and difficult process. We present a new paradigm of social musical exploration and creation system using the physical body as an interface. We have created a physical mixed reality interactive game which enables people from all walks of life to interact in a physical space and learn fundamen-tals of music theory through experimentation. The initial prototype teaches pitch, time signature and dynamics in music. Initial player studies were conducted to refine the prototype to improve the us-ability, playability, and to ensure that the learning objectives are ac-complished. We provide an evaluation of the research project and assess the usefulness of the system in the classroom setting as well as an interactive museum setting. Future plans for development are discussed in the conclusion of the paper to provide for the future development direction.
Article
Full-text available
In this paper we describe a system which allows users to use their full-body for controlling in real-time the generation of an expressive audio-visual feedback. The system extracts expressive motion features from the user's full-body movements and gestures. The values of these motion features are mapped both onto acoustic parameters for the real-time expressive rendering of a piece of music, and onto real-time generated visual feedback projected on a screen in front of the user.
Article
Full-text available
Recognition of musical instruments in multi-instrumental, poly-phonic music is a difficult challenge which is yet far from being solved. Successful instrument recognition techniques in solos (monophonic or polyphonic recordings of single instruments) can help to deal with this task. We introduce an instrument recognition process in solo re-cordings of a set of instruments (bassoon, clarinet, flute, guitar, piano, cello and violin), which yields a high recognition rate. A large and very diverse solo database (108 different solos, all by different performers) is used in order to encompass the different sound possibilities of each instrument and evaluate the generali-zation abilities of the classification process. First we bring classification results using a very extensive collection of features (62 different feature types), and then use our GDE feature selection algorithm to select a smaller feature set with a relatively short computation time, which allows us to perform instrument recognition in solos in real-time, with only a slight decrease in recognition rate. We demonstrate that our real-time solo classifier can also be useful for instrument recognition in duet performances, and im-proved using simple "source reduction".
Article
Full-text available
The UBS Virtual Maestro is an interactive conducting system designed by Immersion Music to simulate the experience of orchestral conducting for the general public attending a classical music concert. The system utilizes the Wii Remote, which users hold and move like a conducting baton to affect the tempo and dynamics of an orchestral video/audio recording. The accelerometer data from the Wii Remote is used to control playback speed and volume in real-time. The system is housed in a UBS-branded kiosk that has toured classical performing arts venues throughout the United States and Europe in 2007 and 2008. In this paper we share our experiences in designing this standalone system for thousands of users, and lessons that we learned from the project.
Conference Paper
Full-text available
We present a comparative evaluation of automatic classification of a sound database containing more than six hundred drum sounds (kick, snare, hihat, toms and cymbals). A preliminary set of fifty descriptors has been refined with the help of different techniques and some final reduced sets including around twenty features have been selected as the most relevant. We have then tested different classification techniques (instance-based, statistical-based, and tree-based) using ten-fold cross-validation. Three levels of taxonomic classification have been tested: membranes versus plates (super-category level), kick vs. snare vs. hihat vs. toms vs. cymbals (basic level), and some basic classes (kick and snare) plus some sub-classes -i.e. ride, crash, open-hihat, closed hihat, high-tom, medium-tom, low-tom- (sub-category level). Very high hit-rates have been achieved (99%, 97%, and 90% respectively) with several of the tested techniques.
Conference Paper
Full-text available
This paper introduces a tool known as the Haptic Drum Kit, which employs four computer-controlled vibrotactile devices, one attached to each wrist and ankle. In the applications discussed here, haptic pulses are used to guide the playing, on a drum kit, of rhythmic patterns that require multi-limb co-ordination. The immediate aim is to foster rhythm skills and multi-limb coordination. A broader aim is to systematically develop skills in recognizing, identifying, memorizing, retaining, analyzing, reproducing, and composing polyphonic rhythms. We consider the implications of three different theories for this approach: the work of the music educator Dalcroze (1865-1950 (1); the entrainment theory of human rhythm perception and production (2,3); and sensory motor contingency theory (4). In this paper we report on a design study; and identify and discuss a variety of emerging design issues. The study demonstrates that beginning drummers are able to learn intricate drum patterns from haptic stimuli alone.
Conference Paper
Full-text available
In this paper, we investigate the impact of machine learn- ing algorithms in the development of automatic music clas- sification models aiming to capture genres distinctions. The study of genres as bodies of musical items aggregated according to subjective and local criteria requires corre- sponding inductive models of such a notion. This process can be thus modeled as an example-driven learning task. We investigated the impact of different musical features on the inductive accuracy by first creating a medium-sized collection of examples for widely recognized genres and then evaluating the performances of different learning al- gorithms. In this work, features are derived from the MIDI transcriptions of the song collection.
Article
Full-text available
In tackling data mining and pattern recognition tasks, finding a compact but effective set of features has often been found to be a crucial step in the overall problem-solving process. In this paper, we present an empirical study on feature analysis for recognition of classical instrument, using machine learning techniques to select and evaluate features extracted from a number of different feature schemes. It is revealed that there is significant redundancy between and within feature schemes commonly used in practice. Our results suggest that further feature analysis research is necessary in order to optimize feature selection and achieve better results for the instrument recognition problem.
Conference Paper
Full-text available
Various kinds of feature sets have been proposed to represent characteristics of musical instruments. While those feature sets have been chosen in a rather heuristic way, in this study, we demonstrate that the log-power spectrum suffices to represent characteristics that are essential to identifying instruments. For efficient encoding of instrument characteristics, we then reduce the number of features by applying the well-known dimension reduction techniques: principal component analysis (PCA) and linear discriminant analysis (LDA). For the classification of eight instruments, the features obtained by applying PCA-LDA to the log-power spectrum performed very well in comparison to existing methods with a recognition rate of 91% with as few as ten features.
Conference Paper
Full-text available
Recent efforts in audio indexing and retrieval in music databases mostly focus on melody. If this is appropriate for polyphonic music signals, specific approaches are needed for systems dealing with percussive audio signals such as those produced by drums, tabla or djembe. Most studies of drum signal transcription focus on sounds taken in isolation. In this paper, we propose several methods for drum loop transcription where the drums signals dataset reflects the variability encountered in modern audio recordings (real and natural drum kits, audio effects, simultaneous instruments, etc.). The approaches described are based on hidden Markov models (HMM) and support vector machines (SVM). Promising results are obtained with a 83.9% correct recognition rate for a simplified taxonomy.
Conference Paper
Full-text available
A set of features is evaluated for musical instrument recognition out of monophonic musical signals. Aiming to achieve a compact representation, the adopted features regard only spectral characteristics of sound and are limited in number. On top of these descriptors, various classification methods are implemented and tested. Over a dataset of 1007 tones from 27 musical instruments and without employing any hierarchical structure, quadratic discriminant analysis shows the lowest error rate (7.19% for the individual instrument and 3.13% for instrument families), outperforming all the other classification methods (canonical discriminant analysis, support vector machines, nearest neighbours). The most relevant features are demonstrated to be the inharmonicity, the spectral centroid and the energy contained in the first partial
Article
Full-text available
We address the issue of automatically extracting rhythm descriptors from audio signals, to be eventually used in content-based musical applications such as in the context of MPEG7. Our aim is to approach the comprehension of auditory scenes in raw polyphonic audio signals without preliminary source separation.
Article
Interactive music video games are a readily available, mainstream technology but they are not generally seen as educative tools. Nor are they established within school teaching and learning environments. This study investigated children's use of these games from a music education perspective. Nine children, aged 9–11 years, and two specialist music teachers each participated in an individual semi-structured interview. The child participants were drawn from four metropolitan schools and the teachers were from different metropolitan schools. Results indicate that the games may help to develop some music skills and knowledge and that the games are of high interest and importance to students. The music teachers acknowledged the potential of interactive music video games to be incorporated into traditional music curriculum but they each expressed a belief that ideally the technology needs further development, including a greater capacity to compose and create using the programs. These findings suggest that, based on constructivist learning theories, there may be a place for these games in music education.
Article
Acoustic signal classification consists of extracting the features from a sound, and of using these features to identify classes the sound is liable to fit.. Different types of noise coming from different vehicles mix in the environment and identifying a particular vehicle is a challenging one. Feature Extraction is done to identify the characteristic of the vehicle. The characteristic of each vehicle will be used to detect its presence and classify its type. Six different features of the vehicle acoustic signals are calculated and then further utilized as input to the classification system. These features include Signal Energy, Energy Entropy, Zero-Crossing Rate, Spectral Roll-Off, Spectral Centroid and Spectral Flux. All these features are extracted from each and every acoustic signal of the vehicles.
Article
Mobile phones offer an attractive platform for interactive music performance. We provide a theoretical analysis of the sensor capabilities via a design space and show concrete examples of how different sensors can facilitate interactive performance on these devices. These sensors include cameras, microphones, accelerometers, magnetometers and multitouch screens. The interactivity through sensors in turn informs aspects of live performance as well as composition though persistence, scoring, and mapping to musical notes or abstract sounds.
Article
We present the first multimedia system to conduct a realistic electronic orchestra. Users can control tempo, dynamics, and instrument emphasis of the orchestra through natural conducting gestures with an infrared baton. Using gesture recognition and tempo adjustment algorithms, the system plays back an audio and video recording of an actual orchestra that follows the users conducting in real time. A major achievement of this system is its ability to vary playback speed in real time while avoiding audio artifacts such as pitch changes. The system has been deployed as an exhibit and has become a major attraction of a large Vienna-based music exhibition center.
Article
Recent evidence suggests that music-based movement (MbM) therapy may be a promising intervention to improve gait and gait-related activities in Parkinson's disease (PD) patients, because it naturally combines cognitive movement strategies, cueing techniques, balance exercises and physical activity while focussing on the enjoyment of moving on music instead of the current mobility limitations of the patient. A meta-analysis of RCTs on the efficacy of MbM-therapy, including individual rhythmic music training and partnered dance classes, was performed. Identified studies (K = 6) were evaluated on methodological quality, and summary effect sizes (SES) were calculated. Studies were generally small (total N= 168). Significant homogeneous SESs were found for the Berg Balance Scale, Timed Up and Go test and stride length (SESs: 4.1,2.2,0.11; P-values <0.01; I ² 0,0,7%, respectively). A sensitivity analysis on type of MbM-therapy (dance- or gait-related interventions) revealed a significant improvement in walking velocity for gait-related MbM-therapy, but not for dance-related MbM-therapy. No significant effects were found for UPDRS-motor score, Freezing of Gait and Quality of Life. Overall, MbM-therapy appears promising for the improvement of gait and gait-related activities in PD. Future studies should incorporate larger groups and focus on long-term compliance and follow-up.
Conference Paper
Although we can sense someone's vocalizations with our ears, nose, and haptic sense, speech is invisible to us without the help of technical aids. In this paper, we present three interactive artworks which explore the question: "if we could see our speech, what might it look like?" The artworks we present are concerned with the aesthetic implications of making the human voice visible, and were created with a particular emphasis on interaction designs that support the perception of tight spatio-temporal relationships between sound, image, and the body. We coin the term in-situ speech visualization to describe a variety of augmented-reality techniques by which graphic representations of speech can be made to appear coincident with their apparent point of origination.
Conference Paper
In this paper we describe MoBoogie, an application that allows users to manipulate and arrange music through movement. MoBoogie is designed to foster experiences in creative expression for children and potentially adults. The application responds to users' movements by changing variables in a continuous stream of music loops. Results from this study suggest that the creative expressions arose in the joint space of movement and music, and did not primarily have to be in one form or the other. This allowed users with limited experience in dance and music making to be creative in such forms of expression.
Conference Paper
In this paper we present the Reactable, a new electronic musical instrument with a simple and intuitive tabletop interface that turns music into a tangible and visual experience, enabling musicians to experiment with sound, change its structure, control its parameters and be creative in a direct, refreshing and unseen way.
Conference Paper
This paper deals with the understanding of four musical time patterns and three tempos that are generated by a human conductor of robot orchestra or an operator of computer-based music play system using the hand gesture recognition. We use only a stereo vision camera with no extra special devices such as sensor glove, 3D motion capture system, infra-red camera, electronic baton and so on. We propose a simple and reliable vision-based hand gesture recognition using the conducting feature point (CFP), the motion-direction code, and the motion history matching. The proposed hand gesture recognition system operates as follows: First, it extracts the human hand region by segmenting the depth information generated by stereo matching of image sequences. Next, it follows the motion of the center of the gravity(COG) of the extracted hand region and generates the gesture features such as CFP and the direction-code. Finally, we obtain the current timing pattern of the music's beat and tempo by the proposed hand gesture recognition using either CFP tracking or motion histogram matching. The experimental results show that the musical time pattern and tempo recognition rate are over 86% on the test data set when the motion histogram matching is used.
Conference Paper
Using tangible interaction in interactive educational systems can benefit learning. This can be supported by relying on experientially originating schemata in the interaction design of learning systems. This paper presents the design and evaluation of MoSo Tangibles, a set of interactive, physical artifacts with which children manipulate the pitch, volume and tempo of ongoing tones, in order to structure their understanding of these abstract sound concepts in terms of multiple different concrete body-based concepts. The results indicate that MoSo provided children with a physical handle to reason about the targeted abstract concepts.
Conference Paper
Rhythm is an important aspect of music development of children. The drum is the instrument that is often played by children and helps them acquire rhythmic skills of music. There are, however, other sources and instruments that are used to create the effects of rhythm. Nevertheless, current education neglects the exploration activities of sound and its relationship with rhythm in elementary school. Based on the positive learning effects of digital game-based learning and learning companion, findings based on this research developed a mobile rhythm learning system that make the rhythm’s learning anywhere and anytime possible. This digital learning companion leads the learner to explore a variety of sound material in the daily life. The learner can use the sound material as background music to the rhythm games, thereby enhancing rhythm learning by this system. In addition, the learner could collaboratively ensemble a rhythm game with their peers through Bluetooth connection.
Conference Paper
In this paper we present the results of a comparative study that explores the potential benefits of using embodied interaction to help children, aged 7 to 10, learn abstract concepts related to musical sounds. Forty children learned to create musical sound sequences using an interactive sound making environment. Half the children used a version of the system that instantiated a body-based metaphor in the mapping layer connecting body movements to output sounds. The remaining children used a version of the same environment that did not instantiate a metaphor in the mapping layer. In general, children were able to more accurately demonstrate sound sequences in the embodied metaphor based system version. However, we observed that children often resorted to spatial rather than body-based metaphors and that the mapping must be easily discoverable as well as metaphorical to provide benefit.
Article
In this paper, the problem of the design of a simple and efficient music-speech discriminator for large audio data sets in which advanced music playing techniques are taught and voice and music are intrinsically interleaved is addressed. In the process, a number of features used in speech-music discrimination are defined and evaluated over the available data set. Specifically, the data set contains pieces of classical music played with different and unspecified instruments (or even lyrics) and the voice of a teacher (a top music performer) or even the overlapped voice of the translator and other persons. After an initial test of the performance of the features implemented, a selection process is started, which takes into account the type of classifier selected beforehand, to achieve good discrimination performance and computational efficiency, as shown in the experiments. The discrimination application has been defined and tested on a large data set supplied by Fundacion Albeniz, containing a large variety of classical music pieces played with different instrument, which include comments and speeches of famous performers.
Article
Cepstral coefficients based on a constant Q transform have been calculated for 28 short (1-2 s) oboe sounds and 52 short saxophone sounds. These were used as features in a pattern analysis to determine for each of these sounds comprising the test set whether it belongs to the oboe or to the sax class. The training set consisted of longer sounds of 1 min or more for each of the instruments. A k-means algorithm was used to calculate clusters for the training data, and Gaussian probability density functions were formed from the mean and variance of each of the clusters. Each member of the test set was then analyzed to determine the probability that it belonged to each of the two classes; and a Bayes decision rule was invoked to assign it to one of the classes. Results have been extremely good and are compared to a human perception experiment identifying a subset of these same sounds.
Conference Paper
Several features were compared with regard to recognition performance in a musical instrument recognition system. Both mel-frequency and linear prediction cepstral and delta cepstral coefficients were calculated. Linear prediction analysis was carried out both on a uniform and a warped frequency scale, and reflection coefficients were also used as features. The performance of earlier described features relating to the temporal development, modulation properties, brightness, and spectral synchronity of sounds was also analysed. The data base consisted of 5286 acoustic and synthetic solo tones from 29 different Western orchestral instruments, out of which 16 instruments were included in the test set. The best performance for solo tone recognition, 35% for individual instruments and 77% for families, was obtained with a feature set consisting of two sets of mel-frequency cepstral coefficients and a subset of the other analysed features. The confusions made by the system were analysed and compared to results reported in a human perception experiment
Article
An electronic orchestra with a complex performance database and MIDI (Musical Instrument Digital Interface) controllers is presented. This system responds to the gestures of a conductor through a CCD camera and a sensor glove. The processing of musical information is examined, and the system and its operation are described. It consists of a gesticulation system and the MIDI controllers. The human interface is also discussed. The first professional conductor who tried the system could direct it without special skills or knowledge of either MIDI or a computer. Tests revealed feedback effects on the behavior of the users through their auditory sense, which in turn suggested some ways to determine the parameters of tempo prediction and compensation. The system can play in concert with human performers and even as a part of a symphony orchestra
Conducting a virtual ensemble with a kinect device
  • A Rosa-Pujazón
  • I Barbancho
  • L J Tardón
  • A M Barbancho
Gyroscope-Based Conducting Gesture Recognition
  • A Höofer
  • A Hadjakos
  • M Mühlhäuser
Drum-hitting gesture recognition and prediction system using kinect
  • A Rosa-Pujazón
  • I Barbancho
  • L J Tardón
  • A M Barbancho
Development of a mobile rhythm learning system based on digital game-based learning companion. Edutainment Technologies. Educ Game and Virtual Reality/Augmented Real Appl
  • C Wang
  • A Lai
A study of wii/kinect controller as musical controllers
  • Y Qin
Orchestra: Wireless sensor system for augmented performances & fusion with kinect
  • T Todoroff
  • J Leroy
  • C Picard-Limpens