Gabriel Sargent's research while affiliated with IRISA - Institut de Recherche en Informatique et Systèmes Aléatoires and other places

Music structure estimation has recently emerged as a central topic within the field of Music Information Retrieval. Indeed, as music is a highly structured information stream, knowledge of how a music piece is organized represents a key challenge to enhance the management and exploitation of large music collections. This article focuses on the benefits that can be expected from a regularity constraint on the structural segmentation of popular music pieces. Specifically here, we study how a constraint which favors structural segments of comparable size provides a better conditioning of the boundary estimation process. Firstly, we propose a formulation of the structural segmentation task as an optimization process which separates the contribution from the audio features and the one from the constraint. We illustrate how the corresponding cost function can be minimized using a Viterbi algorithm. We present briefly its implementation and results in three systems designed for and submitted to the MIREX 2010, 2011 and 2012 evaluation campaigns. Then, we explore the benefits of the regularity constraint as an efficient mean for combining the outputs of a selection of systems presented at MIREX between 2010 and 2015, yielding a level of performance competitive to that of the state-of-the-art on the “MIREX10” dataset (100 JPop songs from the RWC database).

At a large timescale, music pieces can be described as the succession of structural segments which form the global organization of the piece. The present article proposes a model called "System & Contrast", which aims at describing the inner organization of such structural segments in terms of : (i) a carrier system, i.e. a sequence of morphological elements forming a matrix network of self-deducible syntagmatic relationships and (ii) a contrast, i.e. a substitutive element, usually the last one, which partly departs from the logic of the system. The S&C model applies at several timescales and to a wide variety of musical dimensions in a very polymorphous way, therefore offering an efficient meta-description of mid-level musical content.

Video summarization has been a core problem to manage the growing amount of content in multimedia databases. An efficient video summary should display an overview of the video content and most existing approaches fulfill this goal. However, such an overview does not allow the user to reach all details of interest selectively and progressively. This paper proposes a novel scalable summary generation approach based on the On-Line Analytical Processing data cube. Such a structure integrates tools like the drill down operation allowing to browse efficiently multiple descriptions of a dataset according to increased levels of detail. We adapt this model to video summary generation by expressing a video within a cross-media feature space and by performing clusterings according to particular subspaces. Consensus clustering is used to guide the subspace selection strategy at small dimensions, as the novelty brought by the least consensual subspaces is interesting for the refinements of a summary. Our approach is designed for weakly-structured contents such as cultural documentaries. We perform its evaluation on a corpus of cultural archives provided by the French Audiovisual National Institute (INA) using information retrieval metrics handling single and multiple reference annotations. The performances obtained overall improved results compared to two baseline systems performing random and arbitrary segmentations, showing a better balance between Precision and Recall.

Video summarization has been a core problem to manage the growing amount of content in multimedia databases. An efficient video summary should display an overview of the video content and most of existing approaches fulfil this goal. However the information does not allow user to get all details of interest selectively and progressively. This paper proposes a scalable video summarization approach which provides multiple views and levels of details. Our method relies on the usage of cross media space and consensus clustering method. A video document is modelled as a data cube where the level of details is refined over nonconsensual features of the space. The method is designed for weakly structured content such as cultural documentaries and was tested on the INA corpus of cultural archives.

Today, technologies for information storage and transmission allow the creation and development of huge databases of multimedia content. Tools are needed to facilitate their access and browsing. In this context, this article focuses on the segmentation of a particular category of multimedia content, audio-visual musical streams, into music pieces. This category includes concert audio-video recordings, and sequences of music videos such as the ones found in musical TV channels. Current approaches consist in supervised clustering in a few audio classes (music, speech, noise), and, to our knowledge, no consistent evaluation has been performed yet in the case of audio-visual musical streams. In this paper, we aim at estimating the temporal boundaries of music pieces relying on the assumed homogeneity of their musical and visual properties. We consider an unsupervised approach based on the generalized likelihood ratio to evaluate the presence of statistical breakdowns of MFCCs, Chroma vectors, dominant Hue and Lightness over time. An evaluation of this approach on 15 manually annotated concert streams shows the advantage of combining tonal content features to timbral ones, and a modest impact from the joint use of visual features in boundary estimation.

Interest has been steadily growing in semantic audio and music information retrieval for the description of music structure, i.e. the global organization of music pieces in terms of large-scale structural units. This article presents a detailed methodology for the semiotic description of music structure, based on concepts and criteria which are formulated as generically as possible. We sum up the essential principles and practices developed during an annotation effort deployed by our research group (Metiss) on audio data, in the context of the Quaero project, which has led to the public release of over 380 annotations of pop songs from three different data sets. The paper also includes a few case studies and a concise.

Recent progress in information and communication technologies makes it easier to access large collections of digitized music. New representations and algorithms must be developed in order to get a representative overview of these collections, and to browse their content efficiently. It is therefore necessary to characterize music pieces through relevant macroscopic descriptions. In this thesis, we focus on the estimation of the structure of music pieces : the goal is to produce for each piece a description of its organization by means of a sequence of a few dozen structural segments, each of them defined by its boundaries (starting time and ending time) and a label reflecting its audio content.The notion of music structure corresponds to a wide range of meanings depending on the musical properties and the temporal scale under consideration. We introduce an annotation methodology based on the concept of "semiotic structure" which covers a large variety of musical styles. Structural segments are determined through the analysis of their similarities within the music piece, the coherence of their inner organization ("system-contrast" model) and their contextual relationship. A corpus of 383 pieces has been annotated according to this methodology and released to the scientific community.In terms of algorithmic contributions, this thesis concentrates in the first place on the estimation of structural boundaries. We formulate the segmentation process as the optimization of a cost function which is composed of two terms. The first one corresponds to the characterization of structural segments by means of audio criteria. The second one relies on the regularity of the target structure with respect to a "structural pulsation period". In this context, we compare several regularity constraints and study the combination of audio criteria through fusion.Secondly, we consider the estimation of structural labels as a probabilistic finite-state automaton selection process : in this scope, we propose an auto-adaptive criterion for model selection, applied to a description of the tonal content. We also propose a labeling method derived from the system-contrast model.We evaluate several systems for structural segmentation of music based on these approaches in the context of national and international evaluation campaigns (Quaero, MIREX). Additional diagnostic is finally presented to complement this work.

Music structure refers to the description of the long term organization of a music piece through a sequence of struc-tural segments. A structural segment can be defined by its structural borders (a start time, an end time) and a label reflecting the similarity of its music content compared to the other segments'. Its duration is typically around 16 s and more. This document presents the music structure estimation system submitted to MIREX's structural segmentation task in 2012. It is composed of three steps : feature extraction, structural border estimation and segment labeling. First, the system produces a sequence of chroma vectors [6] ex-pressed at the snap scale [1] (section 1). This sequence is used to calculate a segmentation criterion based on a morphological model of the structural segments [2] (sec-tion 2.1). The structural border estimation is performed by searching the segmentation with lowest cost, which com-bines this criterion and a regularity constraint (section 2.2). The segments are then labeled by clustering according to their similarity, through the minimization of an adaptive model selection criterion (section 3).

Music structure description, i.e. the task of representing the high-level organization of music pieces in a concise, generic and reproducible way, is currently a scientific challenge both algorithmically and conceptually. In this paper, we focus on semiotic structure, i.e. the description of similarities and internal relationships within a music piece, as a low-rate stream of arbitrary symbols from a limited alphabet and we address methodological ques-tions related to annotation. We formulate the labeling task as a blind demodulation problem, whose goal is to identify a minimal set of semi-otic codewords, whose realizations within the music piece are subject to a number of connotative variations viewed as modulations. The determination of labels is achieved by combining morphological, paradigmatic and syntagmatic considerations relying respectively on (i) a morphological model of semiotic blocks in order to de-fine their individual properties, (ii) the support of proto-typical structural patterns to guide the comparison be-tween blocks and (iii) a methodology for the determina-tion of distinctive features across semiotic classes. Specific notations are introduced to account for unresolv-able semiotic ambiguities, which are occasional but must be considered as inherent to the music matter itself. A set of 500 music pieces labeled in accordance with the pro-posed concepts and annotation conventions is being re-leased with this article.