Sotiris Manitsaris

• Doctor of Philosophy
• Deputy Director - Centre for Robotics at Mines Paris, PSL University

We address the problem of systematizing the authoring of digital dictionaries for craft education from ethnographic studies and recordings. First, we present guidelines for the collection of ethnographic data using digital audio and video and identify terms that are central in the description of crafting actions, products, tools, and materials. Sec...

A roadmap is proposed that defines a systematic approach for craft preservation and its evaluation. The proposed roadmap aims to deepen craft understanding so that blueprints of appropriate tools that support craft documentation, education, and training can be designed while achieving preservation through the stimulation and diversification of prac...

When willing to experience and learn a craft, “learning by doing” has been proven to be the most effective approach. Traditionally apprentices spend entire years close to the master, observing, imitating him/her, interacting with him/her, and receiving guidelines. Inspired by this natural process actively involving both the master and the apprentic...

A roadmap is proposed that defines a systematic approach for craft preservation and its evaluation. The proposed roadmap aims at deepening craft understanding, so blueprints of appropriate tools that support craft documentation, education, and training can be designed while achieving preservation through the stimulation and diversification of pract...

The analysis of human movements has been extensively studied due to its wide variety of practical applications. Nevertheless, the state-of-the-art still faces scientific challenges while modeling human movements. Firstly, new models that account for the stochasticity of human movement and the physical structure of the human body are required to acc...

Human movement analysis is a key area of research in robotics, biomechanics, and data science. It encompasses tracking, posture estimation, and movement synthesis. While numerous methodologies have evolved over time, a systematic and quantitative evaluation of these approaches using verifiable ground truth data of three-dimensional human movement i...

Human movement analysis is a key area of research in robotics, biomechanics, and data science. It encompasses tracking, posture estimation, and movement synthesis. While numerous methodologies have evolved over time, a systematic and quantitative evaluation of these approaches using verifiable ground truth data of three-dimensional human movement i...

A protocol for the representation of traditional crafts and the tools to implement this are proposed. The proposed protocol is a method for the systematic collection and organization of digital assets and knowledge, their representation into a formal model, and their utilization for research, education, and preservation. A set of digital tools acco...

The high prevalence of work-related musculoskeletal disorders (WMSDs) could be addressed by optimizing Human-Robot Collaboration (HRC) frameworks for manufacturing applications. In this context, this paper proposes two hypotheses for ergonomically effective task delegation and HRC. The first hypothesis states that it is possible to quantify ergonom...

In this paper, a representation based on digital assets and semantic annotations is established for Traditional Craft instances, in a way that captures their socio-historic context and preserves both their tangible and intangible Cultural Heritage dimensions. These meaningful and documented experiential presentations are delivered to the target aud...

This work regards the representation of handicrafts for craft training and demonstration in the environment of an ethnographic heritage museum. The craft of mastic cultivation is chosen as a use case. This paper presents the process of representation and presentation of this craft, following an articulated pipeline approach for data collection, ann...

Traditional crafts exhibit tangible and intangible dimensions. Intangible dimensions include the practitioner’s gestural know-how in craft practice and have received smaller attention than tangible dimensions in digitization projects. This work presents the process of representation and presentation of the glasswork and is exemplified in the re-cre...

La vision par ordinateur est la seule technologie permettant la reconnaissance des gestes musicaux des doigts, sans interférence entre le musicien et son instrument, et à un faible coût. Le système PianOrasis propose deux approches pour la reconnaissance : a) la reconnaissance statique, ou reconnaissance des doigtés, et b) la reconnaissance dynamiq...

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth...

The context of this thesis is the collaboration between humans and machines in various industrial real-world situations. I propose collaboration mechanisms that are based on Human-centred Artificial Intelligence, which I define as methods and concepts of machine learning and pattern recognition on signals recorded from the human body. I am interest...

In industry, ergonomists apply heuristic methods to determine workers’ exposure to ergonomic risks; however, current methods are limited to evaluating postures or measuring the duration and frequency of professional tasks. The work described here aims to deepen ergonomic analysis by using joint angles computed from inertial sensors to model the dyn...

During an eight-hour shift, an industrial worker will inevitably cycle through specific postures. Those postures can cause microtrauma on the musculoskeletal system that accumulates, which in turn can lead to chronic injury. To assess how problematic a posture is, the rapid upper limb assessment (RULA) scoring system is widely employed by the indus...

Human-centered artificial intelligence is increasingly deployed in professional workplaces in Industry 4.0 to address various challenges related to the collaboration between the operators and the machines, the augmentation of their capabilities, or the improvement of the quality of their work and life in general. Intelligent systems and autonomous...

The goal of this work is to build the basis for a smartphone application that provides functionalities for recording human motion data, train machine learning algorithms and recognize professional gestures. First, we take advantage of the new mobile phone cameras, either infrared or stereoscopic, to record RGB-D data. Then, a bottom-up pose estimat...

Currently, biomechanics analyses of the upper human body are mostly kinematic i.e., they are concerned with the positions, velocities, and accelerations of the joints on the human body with little consideration on the forces required to produces them. Tough kinetic analysis can give insight to the torques required by the muscles to generate motion...

Movement-based interactive systems, and gesture recognition in general, can contribute to the human learning of motor skills. This paper presents the design and initial evaluation of a sensori-motor learning mechanism that provides on-line feedforward to the user when s/he repeats a gesture. This mechanism is based on real time statistical comparis...

We reconstruct 3D deformable object through time, in the context of a live pottery making process where the crafter molds the object. Because the object suffers from heavy hand interaction, and is being deformed, classical techniques cannot be applied. We use particle energy optimization to estimate the object profile and benefit of the object radi...

Human–robot collaboration in industrial context requires a smooth, natural and efficient coordination between robot and human operators. The approach we propose to achieve this goal is to use online recognition of technical gestures. In this paper, we present together, and analyze, parameterize and evaluate much more thoroughly, three findings prev...

Manual laborers from the industry sector are often subject to critical physical strain that lead to work-related musculoskeletal disorders. Lifting, poor posture and repetitive movements are among the causes of these disorders. In order to prevent them, several rules and methods have been established to identify ergonomic risks that the worker migh...

The piano sonatas of Beethoven represent part of the Intangible Cultural Heritage. The aims of this research were to further explore this intangibility by placing emphasis on defining emotional normative ratings for the “Waldstein” (Op. 53) and “Tempest” (Op. 31) Sonatas of Beethoven. To this end, a musicological analysis was conducted on these par...

This article describes a prototype natural user interface, named the Intangible Musical Instrument, which aims to facilitate access to knowledge of performers that constitutes musical Intangible Cultural Heritage using off-the-shelf motion capturing that is easily accessed by the public at large. This prototype is able to capture, model, and recogn...

Intangible Cultural Heritage (ICH) creations include, amongst other, music, dance, singing, theatre, human skills and craftsmanship. These cultural expressions are usually transmitted orally and/or using gestures and are modified over a period of time, through a process of collective recreation. As the world becomes more interconnected and many dif...

This paper presents a methodological framework for the use of gesture recognition technologies in the learning/mastery of the gestural skills required in wheel-throwing pottery. In the case of self-instruction or training, learners face difficulties due to the absence of the teacher/expert and the consequent lack of guidance. Motion capture technol...

The tangible cultural heritage is the result of high level know-how, acquired and transmitted over centuries within local communities through the association of gestural intelligence and the creativity of the human spirit. However, this know-how is currently threatened by globalization and industrialization, due in particular to the devaluation of...

Intangible cultural heritage (ICH) is a relatively recent term coined to represent living cultural expressions and practices, which are recognised by communities as distinct aspects of identity. The safeguarding of ICH has become a topic of international concern primarily through the work of United Nations Educational, Scientific and Cultural Organ...

In this paper, we focus on two challenges to enable human-robot collaboration in factories. The first challenge is to evaluate the acceptability of an operator to work with a robot on a new collaborative task. Comparing physical and virtual situation, we highlight notions related to acceptability which can be evaluated using virtual reality. This w...

There is a growing interest in ‘unlocking’ the motor skills of expert musicians. Motivated by this need, the main objective of this paper is to present a new way of modeling expressive gesture variations in musical performance. For this purpose, the 3D gesture recognition engine ‘x2Gesture’ (eXpert eXpressive Gesture) has been developed, inspired b...

In this paper, we introduce the i-Treasures Intangible Cultural Heritage (ICH) dataset, a freely available collection of multimodal data captured from different forms of rare ICH. More specifically, the dataset contains video, audio, depth, motion capture data and other modalities, such as EEG or ultrasound data. It also includes (manual) annotatio...

Sonification is an interdisciplinary field of research, aiming at generating sound from data based on systematic, objective and reproducible transformations. Towards this direction, expressive gestures play an important role in music performances facilitating the artistic perception by the audience. Moreover, emotions are linked with music, as soun...

This paper presents a unified framework computer vision approach for finger gesture early recognition and interaction that can be applied on sequences of either RGB or depth images without any supervised skeleton extraction. Either RGB or time-of-flight cameras can be used to capture finger motions. The hand detection is based on a skin color model...

Creating a digital metaphor of the "in person transmission" of manual-crafting motor skills, is an extremely complicated and challenging task. We are aiming to achieve the above by creating a mixed reality environment, supported by an interactive system for sensorimotor learning that relies on pathing techniques. The gestural instruction of a perso...

Enabling Human-Robot collaboration (HRC) requires robot with the capacity to understand its environment and actions performed by persons interacting with it. In this paper we are dealing with industrial collaborative robots on assembly line in automotive factories. These robots have to work with operators on common tasks. We are working on technica...

We present a musical instrument, named the Embodied Musical Instrument (EMI) which allows musicians to perform free gestures with the upper--body including hands and fingers thanks to 3D vision sensors, arranged around the tabletop. 3D interactive spaces delimit the boundaries in which the player performs metaphorical gestures in order to play with...

In this paper, we present good practices of applying and extending Random Decision Forests (RDFs) for the 3D modelling of scenes where humans interact with moving, deformable and revolving objects in a professional context. We apply our method to two use-cases; the first is in the industrial context of the luxury leather good production while the s...

This article presents a platform for learning theoretical knowledge and practical motor skills of musical gestures by combining functionalities of Learning Management Systems (LMS) and Serious Gaming (SG). The teacher designs his/her educational scenario that can be articulated by both theoretical and practical activities. The learner accesses onli...

Thanks to improvements of the last decades in the robotic field, small Unmanned Aerial Vehicle (sUAV) have become efficient remote sensors for infantrymen on the battlefield. Their increasing mobility and autonomy allow acquiring essential strategic information in a constantly evolving environment. However, due to the hostile nature of the battlefi...

A large variety of musical instruments, either acoustical or digital, are based on a keyboard scheme. Keyboard instruments can produce sounds through acoustic means but they are increasingly used to control digital sound synthesis processes with nowadays music. Interestingly, with all the different possibilities of sonic outcomes, the input remains...

We present a framework and preliminary experimental results for technical gestures recognition using a RGB-D camera. We have studied a collaborative task between a robot and an operator: the assembly of a motor hoses. The goal is to enable the robot to understand which task has just been executed by a human operator in order to anticipate on his ac...

This paper presents a methodology for modelling expert gestural performances in wheel-throwing pottery. The approach is based on building an operational model that describes how expert gestures are performed, taking also into consideration relationships between different parts of the body. This model is estimated using state-space estimation method...

In the context of using an Unmanned Aerial Vehicle (UAV) in hostile environments, gestures allow to free the operator of bulky control interfaces. Since a navigation plan is defined before the mission, only a few commands have to be activated during the mission. This allows a gestural symbolic interaction that maps commands to a set of gestures. Ne...

This paper proposes a marker-less computer vision methodology for the simultaneous recognition of complex finger musical gestures performed in space without any tangible musical instrument. Image analysis techniques are applied in order to detect and identify the fingertips on a video. Scale and rotation invariance techniques are also applied. The...

The acquisition of gestural know-how in manual professions constitutes a real challenge since it passes from master to learner, through a many years long « in person » transmission. However this binding transmission is not always possible for practical reasons; the learner must train himself alone, by using traditional Knowledge Management tools su...

This research has been conducted in the context of the ArtiMuse project that aims at the modeling and renewal of rare gestural knowledge and skills involved in the traditional craftsmanship and more precisely in the art of wheel-throwing pottery. These knowledge and skills constitute intangible cultural heritage and refer to the fruit of diverse ex...

The paper presents a methodology for offline statistical analysis of expert technical gestures applied in pottery interaction. The technical gestures are described using rotations of each segment of the upper-part of the body, including hands and head. The motion capture is based on a suit with inertial sensors. The results confirm the initial hypo...

We present a framework and preliminary experimental results for real-time recognition of human operator actions. The goal is, for a collaborative industrial robot operating on same assembly-line as workers, to allow adaptation of its behavior and speed for smooth human-robot cooperation. To this end, it is necessary for the robot to monitor and und...

Cultural expression is not limited to architecture, monuments or collections of artifacts. It also includes fragile intangible live expressions, which involve knowledge and skills such as music, dance, singing, theatre, human skills and craftsmanship. These manifestations of human intelligence and creativeness constitute our Intangible Cultural Her...

The aim of this research is to compare the RGB, RG normalized, HSV and YCrCb colour spaces, which are used for skin detection in finger gesture recognition. Two comparison methods have been used: a) the first method calculates the deviation of the pixels identified as finger gesture and it is based on statistics by calculating how much the distribu...

The article describes a system that uses real time measurements of the vocal tract to drive a voice-replacement system for post-laryngectomy patients. Based on a thermoformed acquisition helmet, miniature ultrasound machine, and video camera, and incorporating Hidden Markov Model speech recognition, the device has been tested on three speakers, one...

This paper proposes a methodology for the real-time finger gesture following and control of mechatronic systems based on computer vision and machine learning techniques. The goal of this research is to develop a human-machine interface that could be able to control a mechatronic system by performing finger gestures in space or on a surface without...

A computer vision methodology for the recognition of finger gestures performed on a music instrument has been recently developed and implemented in the PianOrasis system. PianOrasis recognises the gestures of all the five fingers simultaneously, but not in real-time. In this paper, an optimisation of the above methodology is presented, implying the...

Intégralité des actes de cette conférence disponible au lien suivant: http://www.issp2011.uqam.ca/upload/files/proceedings.pdf

This thesis presents a prototype computer vision system for the gesture recognition in the interaction between the pianist and the musical instrument. Computer vision is the only technology that permits the recognition of the gestures, without interference between the pianist and his instrument, and at a low cost. The system proposes two approaches...

This paper presents a method based on computer vision, which allows the gesture recognition of a pianist's right hand. The choice of computer vision as a processing method was made due to the lack of a satisfying fingering detection system addressing musicians. On the other hand, hidden Markov models (HMM) are an advisable mathematic theory managin...

This paper presents a computer vision method, which allows pianist fingering retrieval. Computer vision can lead to gesture recognition in music interaction, while the pianist feel and move freely. The method has been developed keeping in mind the specific position of the pianist's hand. It can be applied for classical musical instruments too. Spat...

Voice, text or music recognition systems have quite a lot of elements in common with gesture recognition systems. They recognize "something" moving in orbit in space and in time. Hidden Markov Models (HMM) is an advisable mathematic theory managing stochastic processes such as human voice and gestures. This paper proposes a system retrieving music...

The evolution in Internet and Multimedia Technology and the continuously increasing demand for enhanced remote and mobile services render as essential the adaptation of educational material in these requirements. The main topic addressed in this paper is to propose an alternative process for students attending courses. It refers to multiple types o...

ICT can play significant role in the educational process as they bring important changes to the traditional way of teaching, using multimedia and internet technology. This paper presents the evaluation of the application of a Learning Management System, called ASDL (Asynchronous & Synchronous Distance Learning), in Greek private high-schools. ASDL...

This paper presents the specifications of an open-source learning management system using ICT, which was implemented on a Greek high school. This effort began because of our interest to see the impact of the use of multimedia and Internet resources having e-learning content, in a classroom. Greece lags far behind in use of Internet towards other Eu...

Cette thèse présente un système prototype de vision par ordinateur pour la reconnaissance des gestes dans l'’interaction entre le pianiste et l'’instrument. La vision par ordinateur est la seule technologie permettant la reconnaissance des gestes, sans interférence entre le pianiste et son instrument, et à un faible coût. Le système propose deux ap...