Content uploaded by Katerina El Raheb
Author content
All content in this area was uploaded by Katerina El Raheb on Jun 05, 2018
Content may be subject to copyright.
A preview of the PDF is not available
BalOnSe: Ballet Ontology for Annotating and Searching Video performances
Abstract
In this paper we present BalOnSe (named after the ballet step balance), an ontology-based web interface that allows the user to annotate classical ballet videos, with a hierarchical domain specific vocabulary and provides an archival system for videos of dance. The interface integrates a hierarchical vocabulary based on classical ballet syllabus terminology (Ballet.owl) implemented as an OWL-2 ontology. BalOnSe supports the search and browsing of the multimedia content using metadata (title, dancer featured, etc.), and also implements the functionality of "searching by movement concepts", i.e., filtering the videos that are associated with particular required terms of the vocabulary, based on previous submitted annotations. In the paper, we present the ballet.owl ontology, and its structure, explaining the conceptual modeling decisions. We highlight the main functionality of the system and finally, we present how the manual ontology guided annotation allows the user to search the content through the vocabularies and also view statistics in the form of tag clouds.
... In recent years, there have been several attempts to capture and model movement expressions in performing arts. For example, Raheb et al. proposed an ontology that allows users to annotate classical ballet videos with a domain-specific vocabulary implementing the functionality of "motion concept search, " through which a user searches for specific movements performed within each video [15]. Kannan et al. focus on modeling dance video objects at multiple levels of detail by implementing the Dance Video Semantic Model (DVSM) [23]: This vocabulary attempts to combine and interconnect the underlying semantics between songs and movements. ...
... On the one hand, the ontology must be able to express the modeling of kinesthetic processes. On the other hand, it should also make the cultural aspect visible, which such movement is intended to express [15]. On that, we argue that the cultural aspects of movement, style, technique, or influence do not need to be rendered explicitly in the ontology, rather, they can and should emerge from the interplay of the domain ontology with a cultural one, whereupon a computational method is applied. ...
... In our example, the following data table provides the scoping: It indicates that swords that are held with two hands are a trait of the Japanese Muromachi culture if single-edged (line 2) and of the Anglo-Saxon cultures if double-edged (line 3). 15 1 VALUES (?weaponType ?featureValue ?ident) { 2 ( data:SingleEdgedSword :TwoHandedGrip wd:Q334845 ) # Q334845 is Muromachi Japan At this stage, one does not yet know what such weapons and styles are or if they even exist. Applying the rule in Listing 8, scoped using the data table of Listing 9, attempts to detect cultural traits and to trace them back to the evidence that warrants them. ...
Investigating the intangible nature of a cultural domain can take multiple forms, addressing for example the aesthetic, epistemic and social dimensions of its phenomenology. The context of Southern Chinese martial arts is of particular significance as it carries immaterial components of all these aspects: the technical and stylistic framework of a martial art system; the imagery associated to movements; and the transmission of knowledge orally, practically or through influence, are but examples of intangible characteristics that can and should be captured, not unlike cultural artifacts. The latter case– the one of formalizing cultural influence through its various forms of evidence– is emblematic as well as largely untrodden ground. A previous attempt at detecting cultural influence computationally was made in the context of Roman archaeology, though the binding of that early effort with the domain model was tight; also, there has hardly been any prior dedicated effort to model the martial arts domain through ontologies.
In this paper, we present the realization of the full cycle of a computational approach to investigating cultural contact in Southern Chinese martial arts. The entire approach is predicated upon the usage of standards and techniques of the Semantic Web and formal knowledge. Starting from a modular domain ontology, which models martial arts independently of the goal of capturing cultural influence, we perform knowledge extraction from archival material from the Hong Kong Martial Arts Living Archive and generate a dataset of the results modeled after said ontology. Then, we combine the resulting knowledge base with a rule model that represents ways to infer knowledge of potential contact between cultures based on the evidence present in the knowledge base. The results offer an insight into how an inference-based computational model can be applied to detect interesting facts even in the as-yet underexplored domain of intangible cultural heritage. The implemented workflow shows that the full-cycle employment of semantic technologies can offer the ground truth required for largely different approaches, such as statistical and machine learning ones, to operate.
... They mainly exploited classical computer vision approaches to recognize critical elements like the dancer's posture[24, 91, 94, 98], movements made during a dance[21, 97, 140], and gestures of a dancer[140] from image and video data. These type of analyses is entailed to develop real-life applications like tutoring systems[20, 48, 71], retrieval systems[7], heritage preservation[91], dance video archival and annotation [33,45,46]. Other than these technical analyses, any form of dance possesses rich domain knowledge, which is also a subject to explore. ...
... People tried to develop automatic or semi-automatic annotation tools for different dance domains, which tend to ease manual effort. For example, Raheb et al. published two works [45,46] focusing on Ballet dance aiming for dance archival and annotation system. The author proposed an user interface named BalOnSe maintaining a Ballet constrained vocabulary, which acts as an annotation system and browsing facility using dance metadata. ...
This article presents a literature review on the domain of dance research that exploits ontology artifacts to manage their domain knowledge. Any dance form around the world is rich in knowledge because of its historical and geographical diversity, movement rules, and interpretive aspects. Researchers found various approaches to manage this knowledge base, among which the ontology development process is preferred by most people owing to its superficial and easy-to-manage characteristics. However, the heterogenic use of ontology in different dance research aspects demands an organized study to understand its contributions to the domain. Our survey approach towards this objective starts with a systematic literature selection and further grouping them into four categories based on ontology involvement. Second, we discuss each group of articles by their contributions and the level of ontology involvement. Third, a novel evaluation framework is proposed, which assesses each selected article based on nineteen attributes from ontology quality, development, and applications perspectives. We rank each article into three qualitative measures, i.e., High(H), Medium(M), and Low(L), for our attribute set based on our understanding. Finally, We comprehensively analyze the outcomes of our qualitative assessment to present the current research status and their limitations in the candidate domain. This review aspires to be a cornerstone resource, enlightening researchers about the current landscape and future prospects of ontological involvement in dance research.
... These systems may also allow manual annotations to be added along the video timeline, creating benchmark datasets for machine learning research, enriching dance video collections with expert insights, and stimulating discussions within the dance community through shared movement analysis. Some systems, e.g., BalOnSe, facilitate advanced searches using domain-specific vocabulary and metadata such as titles and featured dancers (e.g., [37]). While these systems are valuable for the dance community and research, they are not necessarily designed for learners to look up unfamiliar dance moves. ...
... (1) The concept of timeline-based video annotation has been previously proposed in the context of annotation frameworks for generating reliable ground-truth datasets for dance [37]. ...
Analyzing dance moves and routines is a foundational step in learning dance. Videos are often utilized at this step, and advancements in machine learning, particularly in human-movement recognition, could further assist dance learners. We developed and evaluated a Wizard-of-Oz prototype of a video comprehension tool that offers automatic in-situ dance move identification functionality. Our system design was informed by an interview study involving 12 dancers to understand the challenges they face when trying to comprehend complex dance videos and taking notes. Subsequently, we conducted a within-subject study with 8 Cuban salsa dancers to identify the benefits of our system compared to an existing traditional feature-based search system. We found that the quality of notes taken by participants improved when using our tool, and they reported a lower workload. Based on participants' interactions with our system, we offer recommendations on how an AI-powered span-search feature can enhance dance video comprehension tools.
... In [26], the author uses the Motion Bank System PieceMaker manual annotator to explore the necessity of dance annotation in the choreography. For cultural heritage material or choreographs, other such tools [13,29] also have been developed. ...
... In 2003, as stated in the Convention, there was "no instrument for the safeguarding of intangible cultural heritage" and the safeguarding measures identified both at national (art. [11][12][13][14] and international (art. [16][17][18] levels were primarily based on their recognition, inventorying, and promotion. ...
Motivated by the need to leverage technologies to enhance the preservation, accessibility, quantitative data analysis, and valorization of Dance Heritage, this work introduces DanXe, a framework based on Artificial Intelligence and Extended Reality for the digitization, automatic analysis, and immersive manipulations of tangible and intangible cultural heritage material. Our contribution offers a framework to define the Digital Twin of documentary assets associated with dance heritage, enriching the storytelling of its traditions, performers, and specific languages. The implications of the proposed framework extend beyond the realm of dance, impacting various creative industries and cultural heritage preservation efforts. The approach is validated through a specific case study, linked to the artistic legacy of the dancer and choreographer Rudolf Nureyev intending to narrate his artistic impact. Moreover, we contribute with the description of an effective strategy for the comprehensive organization and analysis of all materials related to the legacy of the dancer.
... For instance, El Raheb and Ioannidis (2013) transferred the concepts of dance choreography into a DanceOWL dataset, drawing from the structure of notation scores to produce a machine-operable vocabulary. This ontology was then applied to annotate videos of classical ballet performances, which allows movement search via domain concepts (El Raheb et al. 2016). In a more CHfocused effort, Mallik, Chaudhury, and Ghosh (2011) devised an ontology for Indian classical dances, providing knowledge-level descriptors about specific hand gestures, facial expressions, and body postures applicable to enrich a multimedia heritage data system. ...
Traditional martial arts are treasures of humanity’s knowledge and critical carriers of sociocultural memories throughout history. However, such treasured practices have encountered various challenges in knowledge transmission and now feature many entries on the UNESCO list of intangible cultural heritage. In tackling the urgency of knowledge preservation through digital means, this project employs an ontology-based approach to model the conceptual realm of traditional martial arts. Accordingly, it creates the Martial Art Ontology (MAon), a comprehensive domain ontology with an annotated data resource incorporating entities and relations from embodied, epistemic, and sociocultural facets. MAon underlines the significance of embodied qualities and addresses relevant dimensions, such as kinesthetics, techniques, mnemonics, and tactics, along with stylistic, interpretative, and ideological components. It features scholarly terminology developed through literature analysis, interviews with masters, and expert validations. The instantiation of MAon is realized through annotating three archetypal Southern Chinese styles, offering exhaustive descriptions concerning techniques, forms, principles, and form sets, amongst others. In summary, the reported approach encodes the manifold of martial arts into a structured vocabulary and an interlinked data resource, accessible to both human-reading and machine-operating applications. By applying it to manifest a range of knowledge concepts, we demonstrate the potential of ontology-based datafication to create coherent representations for intangible cultural entities and to enable an interoperable data infrastructure across multimodal cultural archives.
... Dance analysis is one of the growing research fields in the twentieth-century. People exploited multiple western dances like ballet [El Raheb et al. 2016;Raheb et al. 2017], salsa [Karavarsamis et al. 2016] as well as Indian classical dances like Kathak [Saha et al. 2021], Bharatanatyam [Bhuyan et al. 2022;Jadhav et al. 2012;Mallick et al. 2019] in recent times. Over the years, people tried to analyze dancer posture [Mallick et al. 2019], foot movements [Castro-Méndez et al. 2022], expression [Asahina et al. 2015], hand gesture [Saha et al. 2013] as well as dance guided audio [Saha et al. 2021]. ...
... Aside from sharing the same sensing technology and development space, this work was quite distinct from Elemental Agency. As noted in [13], Ballet is built upon a very reduced and precise set of gestures that are standardized internationally. In this way, the development of movement to sound relationships required the composer/designer to develop mappings, sound qualities and musical structure that catered to a specific set of gestures and phrases. ...
This paper presents composition and system design across two contrasting dance/media pieces, created contempor-neously, having very different affordances and constraints. In each case, the structure of the music and its sonic details were driven by the movements of the dancers through the use of electromyogram and motion sensing, machine learning and adaptive transitions between sections. The first piece was a collaboration with the National Ballet School of Canada in which twenty-one dancers from different countries arrived in time for a single rehearsal before public performances. The second piece was developed over the course of multiple sessions with a team of media artists and dancers, using existing cultural metaphors related to the elements in order to create a shared focus on gestural language , providing a shared perspective on non-human agency that resided "above" the level of movement, sound or light. The first piece was much more top-down due to real-world constraints as well as affordances of the genre, while the second was designed as a bottom-up approach from its beginnings. In each case there were important elements of fixed structuring as well as emergent gestures across sound/movement/light that resulted from the interactions of the dancer collective and the emergent agencies of the interactive systems employed.
Recent technologies have extended opportunities for online dance learning by overcoming the limitations of space and time. However, dance teachers report that student–teacher interaction is more likely to be challenging in a distant and asynchronous learning environment than in a conventional dance class, such as a dance studio. To address this issue, we introduce DancingInside, an online dance learning system that encourages a beginner to learn dance by providing timely and sufficient feedback based on Teacher-AI cooperation. The proposed system incorporates an AI-based tutor agent (AI tutor, in short) that uses a 2D pose estimation approach to quantitatively estimate the similarity between a learner's and teacher's performance. We conducted a two-week user study with 11 students and 4 teachers. Our qualitative study results highlight that the AI tutor in DancingInside could support the reflection on a learner's practice and help the performance improvement with multimodal feedback resources. The interview results also reveal that the human teacher's role is essential in complementing the AI feedback. We discuss our design and suggest potential implications for future AI-supported cooperative dance learning systems.
In this paper, we discuss the key elements of a semantic dance-move representation model based on rule-based extractions of logical descriptions from existing Labanotation scores. This is part of a larger effort on representing and analyzing dance movement based on choreological approaches and notation systems. The main goal is to develop a Knowledge-Based System that provides functionality (a) to search by movement concepts and characteristics in a meaningful way for dance practitioners, who may not necessarily be specialists in notation or analysis, and (b) to link different manifestations of movement recordings, especially Labanotation scores. We use examples to highlight the primary and abstract representation model and outline the main challenges in interpreting and segmenting a Labanotation score to transform it in a semi- Automated way into a sequence of meaningful recognizable movement concepts. We are not aiming to develop an alternative notation system, but to construct a model and methodology to access existing scores (in digital form) and exploit the underlying information about movement for further computational analysis. We take into account some existing choreological approaches, which use an analogy between dance structure analysis and morphological language studies, and identify multiple levels of describing dance and movement. Finally, we discuss limitations of our approach as well as potential uses of the "search by movement" idea and outline some theoretical observations that emerged during this work.
In this paper, we present a Knowledge Based System for describing and storing dances that takes advantage of the expressivity of Description Logics. We propose exploiting the tools of the Semantic Web Technologies in representing and archiving dance choreographies by developing a Dance Ontology in OWL-2. Description Logics allow us to express complex relations and inference rules for the domain of dance movement, while Reasoning capabilities make it easy to extract new knowledge from existing knowledge. Furthermore, we can search within the ontology based on the steps and movements of dances by writing SPARQL queries. The constructing elements of the ontology and their relationships to construct the dance model are based on the semantics of the Labanotation system, a widely applied language that uses symbols to denote dance choreographies.
This paper discusses the evaluation of a video annotator that supports multimodal annotation and is applied to contemporary dance as a creation tool. The Creation-Tool was conceived and designed to assist the creative processes of choreographers and dance performers, functioning as a digital notebook for personal annotations. The prototype, developed for Tablet PCs, allows video annotation in real-time, using a live video stream, or postevent, using a pre-recorded video stream. The tool also allows different video annotation modalities, such as annotation marks, text, audio, ink strokes and hyperlinks. In addition, the system enables different modes of annotation and video visualization. The development followed an iterative design process involving two choreographers, and a usability study was carried out, involving international dance performers participating in a contemporary dance "residence - workshop".
We present a collaborative tool for choreographers and dancers, the Choreographer's Notebook, which allows multimodal annotation of rehearsal videos during the dance production process. The dance production process is a creative process in which exploration and expression are paramount and we describe issues we have observed over four different dance production cycles. The Choreographer's Notebook addresses the recurring problem of scarcity of rehearsal time and space, which is a limiting factor for exploration and expression. The tool is a web-based video application that enables choreographers to provide multimodal, context-sensitive instructions and feedback to the dancers outside of formal rehearsal time. The novelty of this tool is its ability to provide a multimodal collaborative video-based workflow. This article also presents a thorough ethnographic investigation of the dance production process and results from the use of the Choreographer's Notebook as an artifact within that process.
The empirical investigation of human gesture stands at the center of multiple research disciplines, and various gesture annotation schemes exist, with varying degrees of precision and required annotation effort. We present a gesture annotation scheme for the specific purpose of automatically generating and animating character-specific hand/arm gestures, but with potential general value. We focus on how to capture temporal structure and locational information with relatively little annotation effort. The scheme is evaluated in terms of how accurately it captures the original gestures by re-creating those gestures on an animated character using the annotated data. This paper presents our scheme in detail and compares it to other
In this paper, we present an approach for semantic annotation of movement in the videos that is based on ontology model and semantic concept classifiers. The video movement ontology (VMO) is defined by using OWL (Ontology Web Language), and included both schema and data. In fact, ontology concepts and their relationships that construct the movement model are based on the semantics of the Benesh Movement Notation (BMN), which can describe any form of dance or human movement. We have exploited the knowledge embedded into the ontology by using Semantic Web Rules Language (SWRL). In our approach, SWRL rules are used to perform rule-based reasoning over both concepts and concept instances, and to improve the quality of movement annotation in the video. Additionally, we can search within the VMO by writing SPARQL queries. According to the authors' knowledge, this is the first time the ontology concept is used to annotate with BMN the video human movements.
This article considers a system for the semiautomatic annotation of an audio-visual media of dance domain, DMAR (Dance Media Annotation, authoring and Retrieval system). We present the architecture of the system, a manual annotation tool, a semiautomatic authoring tool and a search engine for the choreographers, dancers and students to demonstrate how the dance media can be semantically annotated and how this information can be used for the retrieval of the dance media objects. This article then outlines the underlying XML Schema based content description structures of DMAR and discusses the merits and demerits of our approach of evolving semantic network as the basis for the audio-visual content description. Further, this article proposes a quality metric, fidelity to evaluate the expressive power of the dance annotations. Finally, evaluation results are presented to depict the performance of the dance video queries in terms of precision and recall.