Rebecca Fiebrink's research while affiliated with University for the Creative Arts and other places

Publications (79)

Article
In this article, we present research on customizing a variational autoencoder (VAE) neural network to learn models and play with musical rhythms encoded within a latent space. The system uses a data structure that is capable of encoding rhythms in simple and compound meter and can learn models from little training data. To facilitate the exploratio...
Article
This paper describes the process of developing a software tool for digital artistic exploration of 3D human figures. Previously available software for modeling mesh-based 3D human figures restricts user output based on normative assumptions about the form that a body might take, particularly in terms of gender, race, and disability status, which ar...
Book
Full-text available
These are the proceedings of the research conference "Designing with Artificial Intelligence", which took place in September 2020. The proceedings cover theoretical papers on learning and teaching machines to do creative tasks, practical research on how machines can be used to create design and art and essays on the discourse of the creative nature...
Article
Full-text available
To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces...
Preprint
The authors present a visual instrument developed as part of the creation of the artwork Learning to See. The artwork explores bias in artificial neural networks and provides mechanisms for the manipulation of specifically trained for real-world representations. The exploration of these representations acts as a metaphor for the process of developi...
Preprint
We introduce a method which allows users to creatively explore and navigate the vast latent spaces of deep generative models. Specifically, our method enables users to \textit{discover} and \textit{design} \textit{trajectories} in these high dimensional spaces, to construct stories, and produce time-based media such as videos---\textit{with meaning...
Article
Substantial research has explored the experience of immersion in digital games, but it is unclear whether this phenomenon extends to other game genres such as board games. Immersion is a concept widely discussed in the board game community by both developers and players and yet there is relatively little research in the area. This paper presents a...
Article
This article aims to lay a foundation for the research and practice of machine learning education for creative practitioners. It begins by arguing that it is important to teach machine learning to creative practitioners and to conduct research about this teaching, drawing on related work in creative machine learning, creative computing education, a...
Conference Paper
The authors present a visual instrument developed as part of the creation of the artwork Learning to See. The artwork explores bias in artificial neural networks and provides mechanisms for the manipulation of specifically trained-for real-world representations. The exploration of these representations acts as a metaphor for the process of developi...
Article
The connection between art and technology is much tighter than is commonly recognized. The emergence of aesthetic computing in the early 2000s has brought renewed focus on this relationship. In this article, we articulate how art and Human–Computer Interaction (HCI) are compatible with each other and actually essential to advance each other in this...
Conference Paper
In this paper, we explore the potential for everyday Twitter users to design and use soundscape sonifications as an alternative, “calm” modality for staying informed of Twitter activity. We first present the results of a survey assessing how 100 Twitter users currently use and change audio notifications. We then present a study in which 9 frequent...
Conference Paper
Current Machine Learning (ML) models can make predictions that are as good as or better than those made by people. The rapid adoption of this technology puts it at the forefront of systems that impact the lives of many, yet the consequences of this adoption are not fully understood. Therefore, work at the intersection of people's needs and ML syste...
Conference Paper
Digital music technology plays a central role in all areas of the music ecosystem. For both, music consumers and music producers, intelligent user interfaces are a means to improve access to sound and music. The second workshop on Intelligent Music Interfaces for Listening and Creation (MILC) provides a forum for the latest developments and trends...
Article
Full-text available
In this article, we describe methods and consequences for giving audience members interactive control over the real-time sonification of performer movement data in electronic music performance. We first briefly describe how to technically implement a musical performance in which each audience member can interactively construct and change their own...
Chapter
Rebecca Fiebrink is a Senior Lecturer at Goldsmiths, University of London, where she designs new ways for humans to interact with computers in creative practice. As a computer scientist and musician, much of her work focuses on applications of machine learning to music, addressing research questions such as: ‘How can machine learning algorithms hel...
Article
The growing importance of machine learning creates challenging questions for computing education.
Article
Full-text available
Machine learning offers great potential to developers and end users in the creative industries. For example, it can support new sensor-based interactions, procedural content generation and end-user product customisation. However, designing machine learning toolkits for adoption by creative developers is still a nascent effort. This work focuses on...
Article
Machine learning is one of the most important and successful techniques in contemporary computer science. Although it can be applied to myriad problems of human interest, research in machine learning is often framed in an impersonal way, as merely algorithms being applied to model data. However, this viewpoint hides considerable human work of tunin...
Article
Full-text available
The 11th Summer Workshop on Multimodal Interfaces eNTERFACE 2015 was hosted by the Numediart Institute of Creative Technologies of the University of Mons from August 10th to September 2015. During the four weeks, students and researchers from all over the world came together in the Numediart Institute of the University of Mons to work on eight sele...
Chapter
Machine learning is the capacity of a computational system to learn structure from data in order to make predictions on new data. This chapter draws on music, machine learning, and human-computer interaction to elucidate an understanding of machine learning algorithms as creative tools for music and the sonic arts. It motivates a new understanding...
Chapter
In this chapter, I describe how supervised learning algorithms can be used to build new digital musical instruments. Rather than merely serving as methods for inferring mathematical relationships from data, I show how these algorithms can be understood as valuable design tools that support embodied, real-time, creative practices. Through this discu...
Article
Computer music research realizes a vision of performance by means of computational expression, linking body and space to sound and imagery through eclectic forms of sensing and interaction. This vision could dramatically influence computer science education, simultaneously modernizing the field and drawing in diverse new participants. In this artic...
Conference Paper
Enabled by artificial intelligence techniques, we are witnessing the rise of a new paradigm of computational creativity support: mixed-initiative creative interfaces put human and computer in a tight interactive loop where each suggests, produces, evaluates, modifies, and selects creative outputs in response to the other. This paradigm could broade...
Conference Paper
Full-text available
User interaction with intelligent systems need not be limited to interaction where pre-trained software has intelligence " baked in. " End-user training, including interactive machine learning (IML) approaches, can enable users to create and customise systems themselves. We propose that the user experience of these users is worth considering. Furth...
Chapter
We draw on our experiences with the Princeton Laptop Orchestra to discuss novel uses of the laptop’s native physical inputs for flexible and expressive control. We argue that instruments designed using these built-in inputs offer benefits over custom standalone controllers, particularly in certain group performance settings; creatively thinking abo...
Chapter
While several researchers have grappled with the problem of comparing musical devices across performance, installation, and related contexts, no methodology yet exists for producing holistic, informative visualizations for these devices. Drawing on existing research in performance interaction, human-computer interaction, and design space analysis,...
Article
Full-text available
Machine learning is the capacity of a computational system to learn structures from datasets in order to make predictions on newly seen data. Such an approach offers a significant advantage in music scenarios in which musicians can teach the system to learn an idiosyncratic style, or can break the rules to explore the system's capacity in unexpecte...
Conference Paper
Full-text available
Machine learning is one of the most important and successful techniques in contemporary computer science. It involves the statistical inference of models (such as classifiers) from data. It is often conceived in a very impersonal way, with algorithms working autonomously on passively collected data. However, this viewpoint hides considerable human...
Article
Full-text available
Skeletal data acquisition generates a huge amount of high-dimensionality data. In many fields where motion capture techniques are now used, practitioners would greatly benefit from high-level representations of these motion sequences. However meaningful motion data dimensionality reduction is not a trivial task and the selection of the best set of...
Conference Paper
Full-text available
Research on students' learning in computing typically investigates how to enable individuals to develop concepts and skills, yet many forms of computing education, from peer instruction to robotics competitions, involve group work in which understanding may not be entirely locatable within individuals' minds. We need theories and methods that allow...
Conference Paper
Full-text available
We have applied interactive machine learning (IML) to the creation and customisation of gesturally controlled musical interfaces in six workshops with people with learning and physical disabilities. Our observations and discussions with participants demonstrate the utility of IML as a tool for participatory design of accessible interfaces. This wor...
Conference Paper
A sonification is a rendering of audio in response to data, and is used in instances where visual representations of data are impossible, difficult, or unwanted. Designing sonifications often requires knowledge in multiple areas as well as an understanding of how the end users will use the system. This makes it an ideal candidate for end-user devel...
Article
The Digital Fauvel is an interactive facsimile edition of the Roman de Fauvel as preserved in the manuscript Paris, Bibliothèque Nationale, Ms. fr.146. It enables a variety of user interactions with the manuscript using a large multitouch tabletop computer, including viewing and navigating high-resolution scans, viewing superimposed translations of...
Article
Interactive end-user training of machine learning systems has received significant attention as a tool for personalizing recognizers. However, most research limits end users to training a fixed set of application-defined concepts. This paper considers additional challenges that arise in end-user support for defining the number and nature of concept...
Article
We present Gesture Mapper, an application for digital musical instrument designers to rapidly prototype mappings from performer gestures to sound synthesis parameters. Prior work [2] has shown that using interactive supervised learning to generate mappings from user-generated examples can be more efficient and effective than users writing mapping f...
Article
Teaching artistic skills to children presents a unique challenge: High-level creative and social elements of an artistic discipline are often the most engaging and the most likely to sustain student enthusiasm, but these skills rely on low-level sensorimotor capabilities, and in some cases rote knowledge, which are often tedious to develop. We hypo...
Conference Paper
Full-text available
We present a method for automatic feature extraction and cross-modal mapping using deep learning. Our system uses stacked autoencoders to learn a layered feature representation of the data. Feature vectors from two (or more) different domains are mapped to each other, effectively creating a cross-modal mapping. Our system can either run fully unsup...
Conference Paper
Finger rehabilitation is crucial to a patient's full recovery following finger surgery. In order to avoid contact-induced infection while simultaneously improving patient motivation and reducing barriers to therapist customization of the rehabilitation system, we design the Virtual Therapist: a Phantom robot-based haptic system with a user-friendly...
Article
Full-text available
A weekly seminar consisting of seven composers and one computer scientist was convened for the purpose of exploring questions surrounding how technology can support aspects of the computer music composition process. The composers were introduced to an existing interactive software system for creating new musical interfaces and compositions, which t...
Conference Paper
Full-text available
In this paper we discuss how the band 000000Swan uses machine learning to parse complex sensor data and create intricate artistic systems for live performance. Using the Wekinator software for interactive machine learning, we have created discrete and continuous models for controlling audio and visual environments using human gestures sensed by a c...
Conference Paper
Model evaluation plays a special role in interactive machine learning (IML) systems in which users rely on their assessment of a model's performance in order to determine how to improve it. A better understanding of what model criteria are important to users can therefore inform the design of user interfaces for model evaluation as well as the choi...
Article
Full-text available
In this paper we discuss how the band 000000Swan uses machine learning to parse complex sensor data and create intricate artistic systems for live performance. Using the Wekinator software for interactive machine learning, we have created discrete and continuous models for controlling audio and visual environments using human gestures sensed by a c...
Article
Full-text available
Using the Wekinator software tool for real-time, interactive machine learning [3] and the K-Bow commercial sensor bow [5], we have constructed a real-time cello bow articulation classification system. This system is capable of outputting articulation labels (e.g., "legato," "marcato," "spiccato") in real-time as a cellist performs. These labels, wh...
Conference Paper
My work concerns the design of interfaces for effective interaction with machine learning algorithms in real-time application domains. I am interested in supporting human interaction throughout the entire supervised learning process, including the generation of training examples. In my dissertation research, I seek to better understand how new mach...
Article
Full-text available
We propose a demonstration of The Wekinator, our soft-ware system that enables the application of machine-learning based music information retrieval techniques to real-time musical performance, and which emphasizes a richer human-computer interaction in the design of ma-chine learning systems.
Conference Paper
Multi-touch interactions are a promising means of control for interactive tabletops. However, a lack of precision and tactile feedback makes multi-touch controls a poor fit for tasks where precision and feedback are crucial. We present an approach that offers precise control and tactile feedback for tabletop systems through the integration of dynam...
Article
Full-text available
Supervised learning methods have long been used to allow musical interface designers to generate new mappings by example. We propose a method for harnessing machine learning algorithms within a radically interactive paradigm, in which the designer may repeatedly generate examples, train a learner, evaluate outcomes, and modify parameters in real-ti...
Article
Full-text available
We describe our tool for interactively creating musical controller mappings using a "play-along" paradigm, in which a user pretends to play along with a musical score in real-time using an arbitrary input control modality. As the user "performs," a supervised machine learning system builds a training dataset from the user's gestures and the synthes...
Conference Paper
Full-text available
We explore the potential for and implications of musical (or proto-musical) social interaction and collaboration using currently available technologies embedded into mobile phones. The dynamics of this particular brand of social intercourse and the emergence of an associated aesthetic is described. The clichéd concept of a global village is made a...
Conference Paper
Full-text available
Computational support of creativity is a core concern of our daily work, as researchers and musicians working in computer music. We are enthusiastic about the prospect of attending the Computational Creativity Support workshop at CHI 2009, both to share our work on laptop orchestras and real-time machine learning in music performance, and to explor...
Conference Paper
Full-text available
In this paper, we discuss our recent additions of audio analysis and machine learning infrastructure to the ChucK music programming language, wherein we provide a complementary system prototyping framework for MIR researchers and lower the barriers to applying many MIR algorithms in live music performance. The new language capabilities preserve Chu...
Conference Paper
Full-text available
Machine learning techniques such as classification have proven to be vital tools in both music information retrieval and music performance, where they are useful for leveraging data to learn and model relationships between low-level features and high-level musical concepts. Explicitly supporting feature extraction and classification in a computer m...
Conference Paper
Hierarchical taxonomies of classes arise in the analysis of many types of musical information, including genre, as a means of organizing overlapping categories at vary- ing levels of generality. However, incorporating hierarchi- cal structure into conventional machine learning systems presents a challenge: the use of independent binary classi- fier...
Conference Paper
Full-text available
We draw on our experiences with the Princeton Laptop Orchestra to discuss novel uses of the laptop's native physical inputs for flexible and expressive control. We argue that instruments designed using these built-in inputs offer benefits over custom standalone controllers, particularly in certain group performance settings; creatively thinking abo...
Conference Paper
Full-text available
In this paper, we present a new programming model for performing audio analysis, spectral processing, and feature extraction in the ChucK programming language. The solution unifies analysis and synthesis in the same high-level, strongly-timed, and concurrent environment, extending and fully integrating with the existing language framework. In parti...
Conference Paper
Full-text available
Previous work has employed an approach to the evaluation of wrapper feature selection methods that may overstate their ability to improve classification accuracy, because of a phenomenon akin to overfitting. This paper discusses this phenomenon in the context of recent work in machine learning, demonstrates that previous work in MIR has indeed exag...
Conference Paper
Full-text available
While several researchers have grappled with the problem of comparing musical devices across performance, installa- tion, and related contexts, no methodology yet exists for producing holistic, informative visualizations for these de- vices. Drawing on existing research in performance inter- action, human-computer interaction, and design space anal...
Conference Paper
Full-text available
This paper describes the use of the Autonomous Classifi- cation Engine (ACE) to classify beatboxing (vocal per- cussion) sounds. A set of unvoiced percussion sounds belonging to five classes (bass drum, open hihat, closed hihat and two types of snare drum) were recorded and manually segmented. ACE was used to compare various classification techniqu...
Conference Paper
Full-text available
This paper presents ACE (Autonomous Classification Engine), a framework for using and optimizing classifi- ers. Given a set of feature vectors, ACE experiments with a variety of classifiers, classifier parameters, classifier ensembles and dimensionality reduction techniques in order to arrive at a good configuration for the problem at hand. In addi...
Conference Paper
Full-text available
Music classification continues to be an important com- ponent of music information retrieval research. An un- derutilized tool for improving the performance of classi- fiers is feature weighting. A major reason for its unpopu- larity, despite its benefits, is the potentially infinite cal- culation time it requires to achieve optimal results. Ge- ne...
Article
Full-text available
This paper describes ACE, a framework for automati-cally finding effective classification methodologies for arbitrary supervised classification problems. ACE per-forms experiments with both individual classifiers and classifier ensembles in order to find the approaches best suited to particular problems. A special emphasis is placed on classifier e...
Article
I propose a final project for IFT6080 that investigates the classification of music in a user- and context-specific way, with the goal of facilitating automatic generation of playlists using a personalized, dynamic, context-dependent notion of musical similarity. This project builds on previous work in music classification and music similarity asse...
Article
This paper discusses The Box, a playful, real-time musical controller whose physical design incorporates useful visual and tactual feedback. The motivation behind the design, implementation details, musical mapping strategy, and perfor- mance evaluation are each provided in turn.
Article
We discuss our work creating the Wekinator software for end-user interactive machine learning, and we outline five key findings pertaining to our observations of its use in music composition and performance.

Citations

... Based on recent research on Machine Learning (ML) and autoencoders for generative visualization (e.g., [8,14]), and in the use of ML for embodied interaction design (e.g., [38]), we identify potential in using ML to create interactive visuals for dance performance, from a corpus of body maps. Aly et al. 's research on appropriating biosensors for artistic purposes, identifying that "biosensing decodes inner structures of the performer's body as a control variable" [2], suggest that biosignal sensors can be useful to achieve our aim of revealing inner processes of dancers. ...
... Although most games research focuses on a limited canon of digital games [68], the arguments reviewed above can be largely applied to analog games, which can provide, e.g., immersive experiences [49] similarly to digital games [75]. However, each medium has advantages and drawbacks. ...
... Somewhat surprisingly, immersive tendencies were also shown to significantly predict increases in the use of tabletop games, suggesting that, contrary to its frequently mistaken characterization as an affordance unique to electronic media, board and card games are also quite capable of providing immersion for players (cf. Farkas et al., 2020). ...
... Tigas proposed an open-ended mapping process (entitled "latent mapping"), leveraging an ML algorithm trained on a corpus of unlabelled gestural data [33]. Plant et al. proposed a new tool based on interactive ML (InteractML) to "make embodied interaction design faster, adaptable and accessible to developers of varying experience Figure 1: Diagram with the four stages of the research, over a six-month period and background" [37]. In InteractML "the user provides training examples (movement data), classifes these examples and can iteratively edit" those examples [38]. ...
... Such user-centered approaches to machine learning have become popular to rapidly build custom gesture recognition and sonifcation systems. Examples include Fiebrink's Wekinator [30], which is widely used in music performance and pedagogy, the Gesture Recognition Toolkit [36], and more recently the RapidMix API [9] and InteractML [23]. While these tools are dedicated to facilitating the design of movement-based interactions, they focus on rapid prototyping rather than live coding and improvisation. ...
... Just as previous developments in computing have triggered changes in computing education, machine learning is now acting as a catalyst for change throughout the education system both in K-12 and higher education. The focus of computing education has shifted before, new shifts will come in the future, and it has been suggested that the next frontier in computer science education research is how to teach artificial intelligence [15], [82], [83]. ...
... A model building component in iML framework interacts with Oracles by issuing queries for additional training data or feedback against its intermediate results. iML based methods mainly aspires to build robust Huang et al. [2011], , Carlini and Wagner [2017], Szegedy et al. [2013a], Sun et al. [2018], Brendel et al. [2017], Guo et al. [2019], Cheng et al. [2019a], Guo et al. [2018], Porter et al. [2013], Ma et al. [2019], , Slack et al. [2020], Emamjomeh-Zadeh and Kempe [2017] models by trading-off accuracy for trust Teso and Kersting [2019], Gutzwiller and Reeder [2017], , Ribeiro et al. [2016], Mozina [2018], Gutzwiller and Reeder [2017], Holzinger et al. [2018], Turchetta et al. [2019], Berkenkamp et al. [2016], Sui et al. [2018], Van Den Elzen and Van Wijk [2011], Liu et al. [2017b], Zhao et al. [2018], Mühlbacher et al. [2014] and low resource learning Ambati [2011], Frazier and Riedl [2019], Porter et al. [2013], Preuveneers et al. [2020], Holzinger et al. [2017], Fails and Olsen Jr [2003], Amershi et al. [2011], Tegen et al. [2020], Amershi et al. [2012], Dzyuba et al. [2014], , Jain et al. [2020], Fiebrink and Cook [2010], Gillian and Paradiso [2014], Schedel et al. [2011], Diaz et al. [2019], Arendt et al. [2018Arendt et al. [ , 2017. ...
... Relating the algorithms to real-world experiences, as we describe in this paper alleviates the comprehension difficulties. There is hardly any published research that delves into the problem of teaching machine learning effectively to any population [11]. The functioning of the human brain has inspired the development of neural networks and deep learning frameworks. ...