Francois Pachet

• PhD. HDR
• Managing Director at Spotify

During 2015 and early 2016, the cultural application of Computational Creativity research and practice took a big leap forward, with a project where multiple computational systems were used to provide advice and material for a new musical theatre production. Billed as the world's first 'computer musical... conceived by computer and substantially cr...

The behavior of users of music streaming services is investigated from the point of view of the temporal dimension of individual songs. Specifically, the main object of the analysis is the point in time within a song at which users stop listening and start streaming another song (“skip”). The main contribution of this study is the ascertainment of...

This chapter reflects on about 10 years of research in AI- assisted music composition, in particular during the Flow Machines project. We reflect on the motivations for such a project, its background, its main results and impact, both technological and musical, several years after its completion. We conclude with a proposal for new categories of ne...

In addition to traditional tasks such as prediction, classification and translation, deep learning is receiving growing attention as an approach for music generation, as witnessed by recent research groups such as Magenta at Google and CTRL (Creator Technology Research Lab) at Spotify. The motivation is in using the capacity of deep learning archit...

We now present a preliminary analysis and summary of the various systems surveyed, following our proposed five dimensions referential, through various tables. This provides material for an analysis of the relations between the different dimensions and the corresponding design decisions.

The first dimension, the objective, is the nature of the musical content to be generated.

We are now reaching the core of this book. This chapter will analyze in depth how to apply the architectures presented in Chapter 5 to learn and generate music. We will first start with a naive, straightforward strategy, using the basic prediction task of a neural network to generate an accompaniment for a melody.

Deep networks are a natural evolution of neural networks, themselves being an evolution of the Perceptron, proposed by Rosenblatt in 1957 [165].

The second dimension of our analysis, the representation, is about the way the musical content is represented. The choice of representation and its encoding is tightly connected to the configuration of the input and the output of the architecture, i.e. the number of input and output variables as well as their corresponding types.

We now revisit some design decision issues raised through our analysis and discuss related prospects.

In our analysis, we consider five main dimensions to characterize different ways of applying deep learning techniques to generate musical content. This typology is aimed at helping the analysis of the various perspectives (and elements) leading to the design of different deep learning-based music generation systems.

We address the issue of editing musical performance data, in particular MIDI files representing human musical performances. Editing such sequences raises specific issues due to the ambiguous nature of musical objects. The first source of ambiguity is that musicians naturally produce many deviations from the metrical frame. These deviations may be i...

The behavior of users of music streaming services is investigated from the point of view of the temporal dimension of individual songs; specifically, the main object of the analysis is the point in time within a song at which users stop listening and start streaming another song ("skip"). The main contribution of this study is the ascertainment of...

Research applying machine learning to music modeling and generation typically proposes model architectures, training methods and datasets, and gauges system performance using quantitative measures like sequence likelihoods and/or qualitative listening tests. Rarely does such work explicitly question and analyse its usefulness for and impact on real...

Popular songs have arguably a huge impact on society. It is therefore legitimate to investigate the nature of the creative act underlying popular song composition. Ethnographic experiments in song composition are difficult to conduct. This chapter describes an experiment addressing the role of feedback in the lead sheet composition process. To what...

This book is a survey and analysis of how deep learning can be used to generate musical content. The authors offer a comprehensive presentation of the foundations of deep learning techniques for music generation. They also develop a conceptual framework used to classify and analyze various types of architecture, encoding models, generation strategi...

In addition to traditional tasks such as prediction, classification and translation, deep learning is receiving growing attention as an approach for music generation, as witnessed by recent research groups such as Magenta at Google and CTRL (Creator Technology Research Lab) at Spotify. The motivation is in using the capacity of deep learning archit...

We aim at enforcing hard constraints to impose a global structure on sequences generated from Markov models. In this report, we study the complexity of sampling Markov sequences under two classes of constraints: Binary Equalities and Grammar Membership Constraints. First, we give a sketch of proof of #P-completeness for binary equalities and identi...

This book is a survey and an analysis of different ways of using deep learning (deep artificial neural networks) to generate musical content. At first, we propose a methodology based on four dimensions for our analysis: - objective - What musical content is to be generated? (e.g., melody, accompaniment...); - representation - What are the informati...

https://arxiv.org/abs/1612.01010

VAEs (Variational AutoEncoders) have proved to be powerful in the context of density modeling and have been used in a variety of contexts for creative purposes. In many settings, the data we model possesses continuous attributes that we would like to take into account at generation time. We propose in this paper GLSR-VAE, a Geodesic Latent Space Re...

Machine-learning techniques have been recently used with spectacular results to generate artefacts such as music or text. However, these techniques are still unable to capture and generate artefacts that are convincingly structured. In this paper we present an approach to generate structured musical sequences. We introduce a mechanism for sampling...

The composition of polyphonic chorale music in the style of J.S Bach has represented a major challenge in automatic music composition over the last decades. The art of Bach chorales composition involves combining four-part harmony with characteristic rhythmic patterns and typical melodic movements to produce musical phrases which begin, evolve and...

Research in collaborative music learning is subject to unresolved problems demanding new technological solutions. One such problem poses the suppression of the accompaniment in a live recording of a performance during practice, which can be for the purposes of self-assessment or further machine-aided analysis. Being able to separate a solo from the...

Jazz guitar solos are improvised melody lines played on one instrument on top of a chordal accompaniment (comping). As the improvisation happens spontaneously, a reference score is non-existent, only a lead sheet. There are situations, however, when one would like to have the original melody lines in the form of notated music, see the Real Book. Th...

In the context of contemporary monophonic music, expression can be seen as the difference between a musical performance and its symbolic representation, i.e. a musical score. In this paper, we show how Maximum Entropy (MaxEnt) models can be used to generate musical expression in order to mimic a human performance. As a training corpus, we had a pro...

We introduce a Maximum Entropy model able to capture the statistics of melodies in music. The model can be used to generate new melodies that emulate the style of the musical corpus which was used to train it. Instead of using the $n-$body interactions of $(n-1)-$order Markov models, traditionally used in automatic music generation, we use a $k-$ne...

We introduce a Maximum Entropy model able to capture the statistics of melodies in music. The model can be used to generate new melodies that emulate the style of the musical corpus which was used to train it. Instead of using the $n-$body interactions of $(n-1)-$order Markov models, traditionally used in automatic music generation, we use a $k-$ne...

Most works in automatic music generation have addressed so far specific tasks. Such a reductionist approach has been extremely successful and some of these tasks have been solved once and for all. However, few works have addressed the issue of generating automatically fully fledged music material, of human-level quality. In this article, we report...

Children's Creative Music-Making with Reflexive Interactive Technology discusses pioneering experiments conducted with young children using a new generation of music software for improvising and composing. Using artificial intelligence techniques, this software captures the children’s musical style and interactively reflects it in its responses. Th...

Modeling polyphonic music is a particularly challenging task because of the intricate interplay between melody and harmony. A good model should satisfy three requirements: statistical accuracy (capturing faithfully the statistics of correlations at various ranges, horizontally and vertically), flexibility (coping with arbitrary user constraints), a...

Recent applications of constraint programming to entertainment , e.g., music or video, call for global constraints describing the structure of temporal sequences. A typical constraint approach is to model each temporal event in the sequence with one variable, and to state constraints on these indexed variables. However, this approach hampers the st...

We present FlowComposer, a web application that helps users compose musical lead sheets, i.e. melodies with chord labels. Flow-Composer integrates a constrained-based lead sheet generation tool in which the user retains full control over the generation process. Users specify the style of the lead sheet by selecting a corpus of existing lead sheets....

This chapter introduces the vision and the technical challenges of the Flow Machines project. Flow Machines aim at fostering creativity in artistic domains such as music and literature. We first observe that typically, great artists do not output just single artefacts but develop novel, individual styles. Style mirrors an individual’s uniqueness; s...

Like most human productions, language is the product of cultural evolution, and as such exhibits high levels of complexity. A natural representation of language is written text, an expression of language by letters or other marks. Preceded by proto-writing systems of ideographic and/or early mnemonic symbols, so-called true writing, in which the co...

Plagiarism is usually studied from an analysis viewpoint: how to detect that a text contains copies of another one. In this chapter we study plagiarism from the generation viewpoint: how to generate a text with a guarantee of non-plagiarism. More precisely, we address the problem of Markov sequence generation with forbidden k-gram constraints. This...

The generation of musical material in a given style has been the subject of many studies with the increased sophistication of artificial intelligence models of musical style. In this paper we address a question of primary importance for artificial intelligence and music psychology: can such systems generate music that users indeed consider as corre...

This book collects research contributions concerning quantitative approaches to characterize originality and universality in language. The target audience comprises researchers and experts in the field but the book may also be beneficial for graduate students. Creativity might be considered as a morphogenetic process combining universal features w...

Sampling random sequences from a statistical model, subject to hard constraints, is generally a difficult task. In this paper, we show that for Markov models and a set of Regular global constraints and unary constraints, we can perform perfect sampling. This is achieved by defining a factor graph, composed of binary factors that combine a Markov ch...

We address the problem of generating all possible palindromes from a corpus of Ngrams. Palin-dromes are texts that read the same both ways. Short palindromes (" race car ") usually carry precise , significant meanings. Long palindromes are often less meaningful, but even harder to generate. The palindrome generation problem has never been addressed...

Many natural phenomena exhibit power law spectra. In particular, so-called 1=f� noise series with � close to 1 (also called pink noise) occur in sound, music and countless human artifacts or natural events, from the fluctuations of the flood levels of the Nile to movements of the stock market. As a consequence, many generative models for 1=f noise...

Lead sheets are music scores consisting of a melody and a chord grid, routinely used in many genres of popu lar music. With the increase of online and portable mus ic applications, the need for easily embeddable, adapt able and extensible lead sheet editing tools is pressing . We introduce LeadsheetJS, a Javascript library for vis ualiz- ing,...

The Comic Strip Game is a system allowing users to create dialogues for speechless cartoon strips during shared, online content creation sessions. This paper describes the results of a protocol providing each participant with implicit feedback and inspiration from other participants. We observed the behaviour of subjects and investigate the impact...

To which extent peer feedback can affect the quality of a music composition? How does musical experience influence the quality of a feedback during the song composition process? To answer these questions we designed and conducted an experiment in which participants compose short songs using an online lead sheet editor, are given the possibility to...

This chapter deals with the issue of learning how to improvize. Traditional MOOCs provide jazz students with comprehensive theoretical and motivate students to practice intensively on their own. However, without a view of one's progress, and without feedback, individual practice is a long and winding road along which many students get lost. Indeed,...

The tremendous success of rock music in the second half of the 20th century has boosted the sophistication of production and mixing techniques for this music genre. However, there is no unified theory of mixing from the viewpoint of sound engineering. In this paper, we highlight relationships between loudness and spectrum in individual tracks, esta...

An apparatus and method for machine-updating of prototypes—for example, during design of digital objects, or content-management—has input means (15) for inputting descriptive classifiers (tags) that a user assigns to prototypes and a tag model generator (20) that uses machine learning techniques to produce a model of the association between the ass...

The tremendous success of rock music in the second half of the 20th century has boosted the sophistication of production and mixing techniques for this music genre. However, there is no unified theory of mixing from the viewpoint of sound engineering. In this paper, we highlight relationships between loudness and spectrum in individual tracks, esta...

An animal-machine audio interaction system includes a sound monitor for monitoring the sounds made by one or more animals, a sound segmenter for identifying coherent sound segments within the sounds made by the animal(s), a sound analyzer for analyzing and assigning a category to each sound segment, an output sound selector for selecting an output...

We introduce the problem of generating musical leadsheets, i.e. a melody with chord labels, in the style of an arbitrary composer, that satisfy arbitrary user constraints. The problem is justified by the very nature of musical creativity, as many composers create music precisely by imitating a given style to which they add their own constraints. We...

Markov processes are widely used to generate sequences that imitate a given style, using random walk. Random walk generates sequences by iteratively con-catenating states to prefixes of length equal or less than the given Markov order. However, at higher orders, Markov chains tend to replicate chunks of the corpus with a size possibly higher than t...

Markov processes are widely used to generate sequences that imitate a given style, using random walk. Random walk generates sequences by iteratively con-catenating states to prefixes of length equal or less than the given Markov order. However, at higher orders, Markov chains tend to replicate chunks of the corpus with a size possibly higher than t...

Jazz music is a genre that consists mainly of improvising over known tunes, represented as a lead sheet. This study addresses the question ‘to what extent does a lead sheet carry information about its composer?’ Primarily, this study considers chord progressions alone, and secondarily melodic and temporal information combined with various multiple...

Markov processes are widely used to generate sequences that imitate a given style, using random walk. Random walk generates sequences by iteratively concatenating states to prefixes of length equal or less than the given Markov order}. However, at higher orders, Markov chains tend to replicate chunks of the corpus with a size possibly higher than t...

We address the problem of automatically harmonizing a leadsheet in the style of any arranger. We model the arranging style as a Markov model estimated from a corpus of non-annotated MIDI files. We consider a vertical approach to harmonization, in which chords are all taken from the ar-ranger corpus. We show that standard Markov models, using variou...

Jazz standards are songs representative of a body of musical knowledge shared by most professional jazz musicians. As such, the corpus of jazz standards constitutes a unique opportunity to study a musical genre with a " closed-world " approach, since most jazz composers are no longer in activity today. Although many scores for jazz standards can be...

A new type of Markovian sequence generator and generation method generates a Markovian sequence having controllable properties, notably properties that satisfy at least one control criterion which is a computable requirement holding on items in the sequence. The Markovian sequence is generated chunkwise, each chunk containing a plurality of items i...

When they improvise, musicians typically alternate between several playing modes on their instruments. Guitarists in particular, alternate between modes such as octave playing, mixed chords and bass, chord comping, solo melodies, walking bass, etc. Robust musical interactive systems call for a precise detection of these playing modes in real-time....

Markov processes are increasingly used to generate finite-length sequences that imitate a given style. However, Markov processes are notoriously difficult to control. Recently, Markov constraints have been introduced to give users some control on generated sequences. Markov constraints reformulate finite-length Markov sequence generation in the fra...

Markov processes are increasingly used to generate finite-length sequences that imitate a given style. However, Markov processes are notoriously difficult to control. Recently, Markov constraints have been introduced to give users some control on generated sequences. Markov constraints reformulate finite-length Markov sequence generation in the fra...

Loop pedals are real-time samplers that playback audio played previously by a musician. Such pedals are routinely used for music practice or outdoor “busking”. However, loop pedals always playback the same material, which can make performances monotonous and boring both to the musician and the audience, preventing their widespread uptake in profess...

Meta-data (tags) for an audiovisual file can be generated by producing an initial estimate of the tags and then revising the estimate (notably to expand it and/or render it more precise) based on the assumption that the relationships which hold between the different tags for a set of manually-tagged training examples will also hold for the tags of...

We address the issue of generating texts in the style of an existing author, that also satisfy structural constraints imposed by the genre of the text. We focus on song lyrics, for which structural constraints are well-defined: rhyme and meter. Although Markov processes are known to be suitable for representing style, they are difficult to control...

Virtuosos are human beings who exhibit exceptional performance in their field of activity. In particular, virtuosos are interesting for creativity studies because they are exceptional problem-solvers. However, virtuosity is an under-studied field of human behavior. Little is known about the processes involved to become a virtuoso, and in what they...

AI for creativity and innovation

Markov chains are a well known tool to model temporal properties of many phenomena, from text structure to fluctuations in economics. Because they are easy to generate, Markovian sequences, i.e. temporal sequences having the Markov property, are also used for content generation applications such as text or music generation that imitate a given styl...

A crucial step in the understanding of vocal behavior of birds is to be able to classify calls in the repertoire into meaningful types. Methods developed to this aim are limited either because of human subjectivity or because of methodological issues. The present study investigated whether a feature generation system could categorize vocalizations...

Many systems use Markov models to generate finite-length sequences that imitate a given style. These systems often need to enforce specific control constraints on the sequences to generate. Unfortunately, control constraints are not compatible with Markov models, as they induce long-range dependencies that violate the Markov hypothesis of limited m...

In the field of songbird research, many studies have shown the role of male songs in territorial defense and courtship. Calling, another important acoustic communication signal, has received much less attention, however, because calls are assumed to contain less information about the emitter than songs do. Birdcall repertoire is diverse, and the ro...

We propose a system, the Continuator, that bridges the gap between two classes of traditionally incompatible musical systems: (1) interactive musical systems, limited in their ability to generate stylistically consistent material, and (2) music imitation systems, which are fundamentally not interactive. Our purpose is to allow musicians to extend t...

Feature generation has been proposed recently to generate feature sets automatically, as opposed to human-designed feature sets. This technique has shown promising results in many areas of supervised classification, in particular in the audio domain. However, feature generation is usually performed blindly, with genetic algorithms. As a result sear...

The problem of modeling improvisation has received a lot of attention recently, thanks to progresses in machine learning, statistical modeling, and to the increase in computation power of laptops. The Continuator (Pachet, 2003) was the first real time interactive systems to allow users to create musical dialogs using style learning techniques. The...

A team of researchers proposed a novel approach to music composition, called description-based design that attempted to remove the need for the user to understand anything technical related to the target objects. The researchers focused on creating simple musical objects, such as unaccompanied melodies to demonstrate the approach. The general descr...

In this work, we address to the problem of making the machine listen and react to the musician in an improvisation situation with the purpose of generating high-quality music.

This paper addresses the problem of automatically extracting perceptive information from acoustic signals, in a supervised classification context. Global labels, i.e., atomic information describing a music title in its entirety, such as its genre, mood, main instruments, or type of vocals, are entered by humans. Classifiers are trained to map audio...

We present a feature generation system designed to create audio features for supervised classification tasks. The main contribution to feature generation studies is the notion of analytical features (AFs), a construct designed to support the representation of knowledge about audio signal processing. We describe the most important aspects of AFs, in...

In this work, we address to the problem of making the machine listen and react to the musician in an improvisation situation with the purpose of generating high-quality music.

This report summarises the discussion and experimental work produced by the authors at the 2009 symposium Computational Creativity: An Interdisciplinary Approach, Dagstuhl Leibniz-Zentrum für Informatik. It outlines the motivation for using computational techniques to stimulate human creativity, briefly summarising its historical context and predec...

A research project was conducted at CSL to address the issues of reflexive interactions between man and machine. Reflexive interactions allow users to create objects of interest without being an expert. Researchers of London Neuroscience institute demonstrated a tickling robot arm that can be remotely controlled by a button. Reflexive interactions...

This article provides an introduction to Charmed, a tangible interactive media artwork that explores aspects of daily life in urban environments. The article discusses the conceptual, aesthetic, and technical dimensions of the work, our creative ...

We propose an algorithm for exploiting statistical properties of large-scale metadata databases about music titles to answer musicological queries. We introduce two inference schemes called "direct" and "inverse" inference, based on an efficient implementation of a kernel regression approach. We describe an evaluation experiment conducted on a larg...

In this study we analyzed the possible context-specific and individual-specific features of dog barks using a new machine-learning algorithm. A pool containing more than 6,000 barks, which were recorded in six different communicative situations was used as the sound sample. The algorithm's task was to learn which acoustic features of the barks, whi...

The “bag-of-frames” approach (BOF) to audio pattern recognition models signals as the long-term statistical distribution of their local spectral features, a prototypical implementation of which being Gaussian Mixture Models of Mel-Frequency Cepstrum Coefficients. This approach is the most predominant paradigm to extract high-level descriptions from...

We describe a large-scale experiment aiming at validating the hypothesis that the popularity of music titles can be predicted from global acoustic or human features. We use a 32.000 title database with 632 manually-entered labels per title including 3 related to the popularity of the title. Our experiment uses two audio feature sets, as well as the...

Is music a form of knowledge? Probably not, even if music is undoubtedly an important part of our cultural heritage. Music is not a type of knowledge, at least in first approximation, because music has no consensual, shared meaning. One of the main reasons why music has no meaning, as opposed to text or even pictures, is that music is not referenti...

This work aims to evaluate the effectiveness of EDS as a tool to automatically extract descriptors for real-world problems, such as melody extraction, chord recognition, and sound classification, comparing its performance and development time to traditional approaches. Each of these problems constitutes a case study, and along with the comparative...

The "bag-of-frames" approach (BOF) to audio pattern recognition represents signals as the long-term statistical distribution of their local spectral features. This approach has proved nearly optimal for simulating the auditory perception of natural and human environments (or soundscapes), and is also the most predominent paradigm to extract high-le...

Many works in audio and image signal analysis are based on the use of "features" to represent characteristics of sounds or images. Features are used in various ways, for instance as inputs to classifiers to categorize automatically objects, e.g. for audio scene description. Most, if not all, approaches focus on the development of clever classifiers...