No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Posture-based and Action-based Graphs for Boxing Skill Visualization
Abstract
Automatic evaluation of sports skills has been an active research area. However, most of the existing research focuses on low-level features such as movement speed and strength. In this work, we propose a framework for automatic motion analysis and visualization, which allows us to evaluate high-level skills such as the richness of actions, the flexibility of transitions and the unpredictability of action patterns. The core of our framework is the construction and visualization of the posture-based graph that focuses on the standard postures for launching and ending actions, as well as the action-based graph that focuses on the preference of actions and their transition probability. We further propose two numerical indices, the Connectivity Index and the Action Strategy Index, to assess skill level according to the graph. We demonstrate our framework with motions captured from different boxers. Experimental results demonstrate that our system can effectively visualize the strengths and weaknesses of the boxers.
... The student wore a VR-Head Mounted Display (HMD) and could see the results directly on the screen. Moreover, Shen et al. [36] focused their work on the construction and visualization of the posture-based graph that focuses on the standard postures for launching and ending actions. They propose two numerical indices, the Connectivity Index and the Action Strategy Index, to measure skill level and the strengths and weaknesses of the boxers. ...
... The proposed system has the potential to provide support in assessing posture after sports injuries, particularly in martial arts, such as karate, where posture is fundamental to performing the sport correctly [24,25,36], and to monitor martial arts athletes after injuries to support the restoration of their movements and position. The superposition of 3D bone models reconstructed from medical imaging develops a more physiologically relevant environment. ...
Rehabilitation is a vast field of research. Virtual and Augmented Reality represent rapidly emerging technologies that have the potential to support physicians in several medical activities, e.g., diagnosis, surgical training, and rehabilitation, and can also help sports experts analyze athlete movements and performance. In this study, we present the implementation of a hybrid system for the real-time visualization of 3D virtual models of bone segments and other anatomical components on a subject performing critical karate shots and stances. The project is composed of an economic markerless motion tracking device, Microsoft Kinect Azure, that recognizes the subject movements and the position of anatomical joints; an augmented reality headset, Microsoft HoloLens 2, on which the user can visualize the 3D reconstruction of bones and anatomical information; and a terminal computer with a code implemented in Unity Platform. The 3D reconstructed bones are overlapped with the athlete, tracked by the Kinect in real-time, and correctly displayed on the headset. The findings suggest that this system could be a promising technology to monitor martial arts athletes after injuries to support the restoration of their movements and position to rejoin official competitions.
... Early methods formulate in-between motions as motion planning problem [Wang et al. 2015[Wang et al. , 2013Ye and Liu 2010], which requires solving complex optimizations and are prohibitively slow for real-time applications. Data-driven methods have also been developed [Kovar et al. 2008;Min and Chai 2012;Shen et al. 2017]. However, to handle arbitrary in-between motions and target frames, the size of needed data in memory grows exponentially [Harvey et al. 2020]. ...
... Alternatively, data-driven methods can avoid slow optimizations by searching in structured data, e.g. motion graphs [Kovar et al. 2008;Min and Chai 2012;Shen et al. 2017]. However, since the control or constraints can be diverse, the size of needed data in memory to cover all situations grows exponentially [Harvey et al. 2020], leading to unaffordable space complexity. ...
Real-time in-between motion generation is universally required in games and highly desirable in existing animation pipelines. Its core challenge lies in the need to satisfy three critical conditions simultaneously: quality, controllability and speed , which renders any methods that need offline computation (or post-processing) or cannot incorporate (often unpredictable) user control undesirable. To this end, we propose a new real-time transition method to address the aforementioned challenges. Our approach consists of two key components: motion manifold and conditional transitioning. The former learns the important low-level motion features and their dynamics; while the latter synthesizes transitions conditioned on a target frame and the desired transition duration. We first learn a motion manifold that explicitly models the intrinsic transition stochasticity in human motions via a multi-modal mapping mechanism. Then, during generation, we design a transition model which is essentially a sampling strategy to sample from the learned manifold, based on the target frame and the aimed transition duration. We validate our method on different datasets in tasks where no post-processing or offline computation is allowed. Through exhaustive evaluation and comparison, we show that our method is able to generate high-quality motions measured under multiple metrics. Our method is also robust under various target frames (with extreme cases).
... Early methods formulate in-between motions as motion planning problem [Wang et al. 2015[Wang et al. , 2013Ye and Liu 2010], which requires solving complex optimizations and are prohibitively slow for real-time applications. Data-driven methods have also been developed [Kovar et al. 2008;Min and Chai 2012;Shen et al. 2017]. However, to handle arbitrary in-between motions and target frames, the size of needed data in memory grows exponentially [Harvey et al. 2020]. ...
... Alternatively, data-driven methods can avoid slow optimizations by searching in structured data, e.g. motion graphs [Kovar et al. 2008;Min and Chai 2012;Shen et al. 2017]. However, since the control or constraints can be diverse, the size of needed data in memory to cover all situations grows exponentially [Harvey et al. 2020], leading to unaffordable space complexity. ...
Real-time in-between motion generation is universally required in games and highly desirable in existing animation pipelines. Its core challenge lies in the need to satisfy three critical conditions simultaneously: quality, controllability and speed, which renders any methods that need offline computation (or post-processing) or cannot incorporate (often unpredictable) user control undesirable. To this end, we propose a new real-time transition method to address the aforementioned challenges. Our approach consists of two key components: motion manifold and conditional transitioning. The former learns the important low-level motion features and their dynamics; while the latter synthesizes transitions conditioned on a target frame and the desired transition duration. We first learn a motion manifold that explicitly models the intrinsic transition stochasticity in human motions via a multi-modal mapping mechanism. Then, during generation, we design a transition model which is essentially a sampling strategy to sample from the learned manifold, based on the target frame and the aimed transition duration. We validate our method on different datasets in tasks where no post-processing or offline computation is allowed. Through exhaustive evaluation and comparison, we show that our method is able to generate high-quality motions measured under multiple metrics. Our method is also robust under various target frames (with extreme cases).
... In [13], the posture-based graph method was applied to analyze the movements, while the shadow boxing motions of the boxer were captured using an optical motion capture system. Visualization of movements was one of the main objectives of the study. ...
... Here, a very interesting approach to movement recognition is applied-tracking poses when moving. As the authors write in [13], "the posture-based graph focuses on evaluating the common postures that are used to start and end actions. In such a graph, the nodes represent similar postures and the edges represent similar actions". ...
This work aimed to study the automation of measuring the speed of punches of boxers during shadow boxing using inertial measurement units (IMUs) based on an artificial neural network (ANN). In boxing, for the effective development of an athlete, constant control of the punch speed
is required. However, even when using modern means of measuring kinematic parameters, it is necessary to record the circumstances under which the punch was performed: The type of punch (jab, cross, hook, or uppercut) and the type of activity (shadow boxing, single punch, or series of punches).
Therefore, to eliminate errors and accelerate the process, that is, automate measurements, the use of an ANN in the form of a multilayer perceptron (MLP) is proposed. During the experiments, IMUs
were installed on the boxers’ wrists. The input parameters of the ANN were the absolute acceleration and angular velocity. The experiment was conducted for three groups of boxers with different levels of training. The developed model showed a high level of punch recognition for all groups, and it can be concluded that the use of the ANN significantly accelerates the collection of data on the kinetic characteristics of boxers’ punches and allows this process to be automated.
... In [13], the posture-based graph method was applied to analyze the movements, while the shadow boxing motions of the boxer were captured using an optical motion capture system. Visualization of movements was one of the main objectives of the study. ...
... Here, a very interesting approach to movement recognition is applied-tracking poses when moving. As the authors write in [13], "the posture-based graph focuses on evaluating the common postures that are used to start and end actions. In such a graph, the nodes represent similar postures and the edges represent similar actions". ...
This work aimed to study the automation of measuring the speed of punches of boxers during shadow boxing using inertial measurement units (IMUs) based on an artificial neural network (ANN). In boxing, for the effective development of an athlete, constant control of the punch speed is required. However, even when using modern means of measuring kinematic parameters, it is necessary to record the circumstances under which the punch was performed: The type of punch (jab, cross, hook, or uppercut) and the type of activity (shadow boxing, single punch, or series of punches). Therefore, to eliminate errors and accelerate the process, that is, automate measurements, the use of an ANN in the form of a multilayer perceptron (MLP) is proposed. During the experiments, IMUs were installed on the boxers’ wrists. The input parameters of the ANN were the absolute acceleration and angular velocity. The experiment was conducted for three groups of boxers with different levels of training. The developed model showed a high level of punch recognition for all groups, and it can be concluded that the use of the ANN significantly accelerates the collection of data on the kinetic characteristics of boxers’ punches and allows this process to be automated.
... Recent work has shown that hand tremor can be modelled and reduced by deep learning as a denoising process [20], but the use of frequency is still yet to be considered. Similarly, frequency information is likely to be useful for sports motion analysis and visualisation [21]. In addition, the users can extend our system for multi-class classification tasks by minor modifications on the final fully connected layer. ...
Early prediction is clinically considered one of the essential parts of cerebral palsy (CP) treatment. We propose to implement a low-cost and interpretable classification system for supporting CP prediction based on General Movement Assessment (GMA). We design a Pytorch-based attention-informed graph convolutional network to early identify infants at risk of CP from skeletal data extracted from RGB videos. We also design a frequency-binning module for learning the CP movements in the frequency domain while filtering noise. Our system only requires consumer-grade RGB videos for training to support interactive-time CP prediction by providing an interpretable CP classification result.
... Recent work has shown that hand tremor can be modelled and reduced by deep learning as a denoising process [20], but the use of frequency is still yet to be considered. Similarly, frequency information is likely to be useful for sports motion analysis and visualisation [21]. In addition, the users can extend our system for multi-class classification tasks by minor modifications on the final fully connected layer. ...
Early prediction is clinically considered one of the essential parts of cerebral palsy (CP) treatment. We propose to implement a low-cost and interpretable classification system for supporting CP prediction based on General Movement Assessment (GMA). We design a Pytorch-based attention-informed graph convolutional network to early identify infants at risk of CP from skeletal data extracted from RGB videos. We also design a frequency-binning module for learning the CP movements in the frequency domain while filtering noise. Our system only requires consumer-grade RGB videos for training to support interactive-time CP prediction by providing an interpretable CP classification result.
... Motion capture technology has been widely applied within sports science and the healthcare sectors. Shen et al. [Shen et al. 2017] proposed a visualization framework for evaluating the skills level of the player in sports such as boxing. In the healthcare sector, optical MOCAP data has been used for Diagnosing Musculoskeletal and Neurological Disorder [Rueangsirarak et al. 2018]. ...
With the rapid increase in individuals participating in resistance training activities, the number of injuries pertaining to these activities has also grown just as aggressively. Diagnosing the causes of injuries and discomfort requires a large amount of resources from highly experienced physiotherapists. In this paper, we propose a new framework to analyse and visualize movement patterns during performance of four major compound lifts. The analysis generated will be used to efficiently determine whether the exercises are being performed correctly, ensuring anatomy remains within its functional range of motion, in order to prevent strain or discomfort that may lead to injury.
... In this section, we will first review existing examples of AR in various industries. While Virtual Reality (VR) and interactive computer graphics have been used for teaching and learning, such as partner dancing [3], visualizing wrestling [4], [5] and boxing [6], [7] skills, in the last two decades, more attention has been paid on vision-based frameworks which make use of cameras and sensors. By capturing the information from the surrounding using cameras and sensors, useful feedback can be provided to the user, such as posture monitoring [8] and interacting with virtual objects using body movement [9], [10]. ...
Connecting network cables to network switches is a time-consuming and inefficient task, and requires extensive documentation and preparation beforehand to ensure no service faults are encountered by the users. In this paper, a new AR smartphone application that overlays network switch information over the user’s vision is designed and developed for real working environment to increase user’s efficiency in working with a network switch. Specifically, the prototype of the AR App is developed on the Android platform using both the Unity game engine and Vuforia AR library and connecting to the network switch to retrieve network information through telnet. By using the camera on the smartphone for capturing the visual information from the working environment, i.e. the network switch in this App, the network switch information such as speed, types, etc. will be overlaid on each port on the smartphone screen. A user study was conducted to evaluate the effectiveness of the AR App to assist users in performing network tasks. In particular, participants were tasked with connecting switchports to a patch panel to match up corresponding configurations. After three tests, it was found that the times for completion and mistakes made were reduced in the final test when compared to the first. This highlights the positive effects of the application in improving the user’s efficiency.
... Shen et al. focus on motion analysis and visualisation. The proposed system enables the high-level analysis of motion quality based on the connectivity and variety of motion in a database, supporting applications in sports training and rehabilitation [3] . Ronan Boulic received the Ph.D. degree in computer science from University of Rennes, France, in 1986 and the Habilitation degree in computer science from University of Grenoble, France, in 1995. ...
The understanding of human motion is important in many areas such as sports, dance, and animation. In this paper, we propose a method for visualizing the manifold of human motions. A motion manifold is defined by a set of motions in a specific motion form. Our method visualizes the ranges of time-varying positions and orientations of a body part by generating volumetric shapes for representing them. It selects representative keyposes from the keyposes of all input motions to visualize the range of keyposes at each key timing. A geometrical volume that contains the trajectories from all input motions is generated for each body part. In addition, a geometrical volume that contains the orientations from all input motions is generated for a sample point on the trajectory. The user can understand the motion manifold by visualizing these motion volumes. In this paper, we present some experimental examples for a tennis shot form.
Evaluating potential musculoskeletal disorders risks in real workstations is challenging as the environment is cluttered, which makes it difficult to accurately assess workers' postures. Being marker-free and calibration-free, Microsoft Kinect is a promising device although it may be sensitive to occlusions. We propose and evaluate a RULA ergonomic assessment in real work conditions using recently published occlusion-resistant Kinect skeleton data correction. First, we compared postures estimated with this method to ground-truth data, in standardized laboratory conditions. Second, we compared RULA scores to those provided by two professional experts, in a non-laboratory cluttered workplace condition. The results show that the corrected Kinect data can provide more accurate RULA grand scores, even under sub-optimal conditions induced by the workplace environment. This study opens new perspectives in musculoskeletal risk assessment as it provides the ergonomists with 30 Hz continuous information that could be analyzed offline and in a real-time framework.
Motion analysis and visualization are crucial in sports science for sports training and performance evaluation. While primitive computational methods have been proposed for simple analysis such as postures and movements, few can evaluate the high-level quality of sports players such as their skill levels and strategies. We propose a visualization tool to help visualizing boxers' motions and assess their skill levels. Our system automatically builds a graph-based representation from motion capture data and reduces the dimension of the graph onto a 3D space so that it can be easily visualized and understood. In particular, our system allows easy understanding of the boxer's boxing behaviours, preferred actions, potential strength and weakness. We demonstrate the effectiveness of our system on different boxers' motions. Our system not only serves as a tool for visualization, it also provides intuitive motion analysis that can be further used beyond sports science.
Being marker-free and calibration free, Microsoft Kinect is nowadays widely used in many motion-based applications, such as user training for complex industrial tasks and ergonomics pose evaluation. The major problem of Kinect is the placement requirement to obtain accurate poses, as well as its weakness against occlusions. To improve the robustness of Kinect in interactive motion-based applications, real-time data-driven pose reconstruction has been proposed. The idea is to utilize a database of accurately captured human poses as a prior to optimize the Kinect recognized ones, in order to estimate the true poses performed by the user. The key research problem is to identify the most relevant poses in the database for accurate and efficient reconstruction. In this paper, we propose a new pose reconstruction method based on modeling the pose database with a structure called Filtered Pose Graph, which indicates the intrinsic correspondence between poses. Such a graph not only speeds up the database poses selection process, but also improves the relevance of the selected poses for higher quality reconstruction.We apply the proposed method in a challenging environment of industrial context that involves sub-optimal Kinect placement and a large amount of occlusion. Experimental results show that our real-time system reconstructs Kinect poses more accurately than existing methods.
Depth sensor based 3D human motion estimation hardware such as Kinect has made interactive applications more popular recently. However, it is still challenging to accurately recognize postures from a single depth camera due to the inherently noisy data derived from depth images and self-occluding action performed by the user. In this paper, we propose a new real-time probabilistic framework to enhance the accuracy of live captured postures that belong to one of the action classes in the database. We adopt the Gaussian Process model as a prior to leverage the position data obtained from Kinect and marker-based motion capture system. We also incorporate a temporal consistency term into the optimization framework to constrain the velocity variations between successive frames. To ensure that the reconstructed posture resembles the accurate parts of the observed posture, we embed a set of joint reliability measurements into the optimization framework. A major drawback of Gaussian Process is its cubic learning complexity when dealing with a large database due to the inverse of a covariance matrix. To solve the problem, we propose a new method based on a local mixture of Gaussian Processes, in which Gaussian Processes are defined in local regions of the state space. Due to the significantly decreased sample size in each local Gaussian Process, the learning time is greatly reduced. At the same time, the prediction speed is enhanced as the weighted mean prediction for a given sample is determined by the nearby local models only. Our system also allows incrementally updating a specific local Gaussian Process in real time, which enhances the likelihood of adapting to run-time postures that are different from those in the database. Experimental results demonstrate that our system can generate high quality postures even under severe self-occlusion situations, which is beneficial for real-time applications such as motion-based gaming and sport training.
Movement sequences are essential to dance and expressive movement practice; yet, they remain underexplored in movement and computing research, where the focus on short gestures prevails. We propose a method for movement sequence analysis based on motion trajectory synthesis with Hidden Markov Models. The method uses Hidden Markov Regression for jointly synthesizing motion feature trajectories and their associated variances, that serves as basis for investigating performers' consistency across executions of a movement sequence. We illustrate the method with a use-case in Tai Chi performance, and we further extend the approach to cross-modal analysis of vocalized movements.
Creating realistic human movement is a time consuming and labour intensive task. The major difficulty is that the user has to edit individual joints while maintaining an overall realistic and collision free posture. Previous research suggests the use of data‐driven inverse kinematics, such that one can focus on the control of a few joints, while the system automatically composes a natural posture. However, as a common problem of kinematics synthesis, penetration of body parts is difficult to avoid in complex movements. In this paper, we propose a new data‐driven inverse kinematics framework that conserves the topology of the synthesizing postures. Our system monitors and regulates the topology changes using the Gauss Linking Integral (GUI), such that penetration can be efficiently prevented. As a result, complex motions with tight body movements, as well as those involving interaction with external objects, can be simulated with minimal manual intervention. Experimental results show that using our system, the user can create high quality human motion in real‐time by controlling a few joints using a mouse or a multi‐touch screen. The movement generated is both realistic and penetration free. Our system is best applied for interactive motion design in computer animations and games.
The recent advancement of motion recognition using Microsoft Kinect stimulates many new ideas in motion capture and virtual reality applications. Utilizing a pattern recognition algorithm, Kinect can determine the positions of different body parts from the user. However, due to the use of a single-depth camera, recognition accuracy drops significantly when the parts are occluded. This hugely limits the usability of applications that involve interaction with external objects, such as sport training or exercising systems. The problem becomes more critical when Kinect incorrectly perceives body parts. This is because applications have limited information about the recognition correctness, and using those parts to synthesize body postures would result in serious visual artifacts. In this paper, we propose a new method to reconstruct valid movement from incomplete and noisy postures captured by Kinect. We first design a set of measurements that objectively evaluates the degree of reliability on each tracked body part. By incorporating the reliability estimation into a motion database query during run time, we obtain a set of similar postures that are kinematically valid. These postures are used to construct a latent space, which is known as the natural posture space in our system, with local principle component analysis. We finally apply frame-based optimization in the space to synthesize a new posture that closely resembles the true user posture while satisfying kinematic constraints. Experimental results show that our method can significantly improve the quality of the recognized posture under severely occluded environments, such as a person exercising with a basketball or moving in a small room.
This paper proposes a new methodology for synthesizing animations of multiple characters, allowing them to intelligently compete with one another in dense environments, while still satisfying requirements set by an animator. To achieve these two conflicting objectives simultaneously, our method separately evaluates the competition and collaboration of the interactions, integrating the scores to select an action that maximizes both criteria. We extend the idea of min-max search, normally used for strategic games such as chess. Using our method, animators can efficiently produce scenes of dense character interactions such as those in collective sports or martial arts. The method is especially effective for producing animations along story lines, where the characters must follow multiple objectives, while still accommodating geometric and kinematic constraints from the environment.
We present a technique to automatically distill a motion-motif graph from an arbitrary collection of motion capture data. Motion motifs represent clusters of similar motions and together with their encompassing motion graph they lend understandable structure to the contents and connectivity of large motion datasets. They can be used in support of motion compression, the removal of redundant motions, and the creation of blend spaces. This paper develops a string-based motif-finding algorithm which allows for a user-controlled compromise between motif length and the number of motions in a motif. It allows for time warps within motifs and assigns the majority of the input data to relevant motifs. Results are demonstrated for large datasets (more than 100,000 frames) with computation times of tens of minutes.
Realistic and directable humanlike characters are an ongoing goal in animation. Motion graph data structures hold much promise for achieving this goal; however, the quality of the results obtainable from a motion graph may not be easy to predict from its input motion clips. This article describes a method for using task-based metrics to evaluate the capability of a motion graph to create the set of animations required by a particular application. We examine this capability for typical motion graphs across a range of tasks and environments. We find that motion graph capability degrades rapidly with increases in the complexity of the target environment or required tasks, and that addressing deficiencies in a brute-force manner tends to lead to large, unwieldy motion graphs. The results of this method can be used to evaluate the extent to which a motion graph will fulfill the requirements of a particular application, lessening the risk of the data structure performing poorly at an inopportune moment. The method can also be used to characterize the deficiencies of motion graphs whose performance will not be sufficient, and to evaluate the relative effectiveness of different options for improving those motion graphs.
Many compelling applications would immediately become feasible if novice users had the ability to synthesize high quality human motion based only on a simple sketch or a few easily specified constraints. We approach this problem by representing the desired motion as an interpolation of two time-scaled paths through a motion graph. The graph is constructed to support interpolation and pruned for efficient search. We use an anytime version of A to find a globally optimal solution in this graph that satisfies the user's specification. This solution is not subject to local minima and avoids the inefficient motions that are sometimes seen with locally optimal search algorithms and traditional motion graphs. Our approach retains the natural transitions of motion graphs and the ability to synthesize physically realistic variations provided by interpolation. We demonstrate the power of this approach by synthesizing optimal or near optimal motions that include a variety of behaviors in a single motion.
We introduce Gaussian process dynamical models (GPDM) for nonlinear time series analysis, with applications to learning models of human pose and motion from high-dimensionalmotion capture data. A GPDM is a latent variable model. It comprises a low-dimensional latent space with associated dynamics, and a map from the latent space to an observation space. We marginalize out the model parameters in closed-form, using Gaussian process priors for both the dynamics and the observation mappings. This results in a non-parametric model for dynamical systems that accounts for uncertainty in the model. We demonstrate the approach, and compare four learning algorithms on human motion capture data in which each pose is 50-dimensional. Despite the use of small data sets, the GPDM learns an effective representation of the nonlinear dynamics in these spaces.
This paper describes a model, based on a Markov process model, of daily human behavior in an intelligent house where human behavior is observed with small motion detectors. The number of sensor states is reduced to a few dozen by a vector quantization method, and transitions within this reduced set of states are observed. Then, the state transition probability and the transition duration time distribution are used as the templates of daily human activity. The validity of those templates is evaluated by detecting unusual human behavior in three sets of different human behavior data. Successful detection of unusual behavior without any a priori knowledge shows the effectiveness of probabilistic human behavior description in the intelligent house.
Real-time control of three-dimensional avatars is an important problem in the context of computer games and virtual environments. Avatar animation and control is difficult, however, because a large repertoire of avatar behaviors must be made available, and the user must be able to select from this set of behaviors, possibly with a low-dimensional input device. One appealing approach to obtaining a rich set of avatar behaviors is to collect an extended, unlabeled sequence of motion data appropriate to the application. In this paper, we show that such a motion database can be preprocessed for flexibility in behavior and efficient search and exploited for real-time avatar control. Flexibility is created by identifying plausible transitions between motion segments, and efficient search through the resulting graph structure is obtained through clustering. Three interface techniques are demonstrated for controlling avatar motion using this data structure: the user selects from a set of available choices, sketches a path through an environment, or acts out a desired motion in front of a video camera. We demonstrate the flexibility of the approach through four different applications and compare the avatar motion to directly recorded human motion.
We present a framework to synthesize character movements based on high level parameters, such that the produced movements respect the manifold of human motion, trained on a large motion capture dataset. The learned motion manifold, which is represented by the hidden units of a convolutional autoencoder, represents motion data in sparse components which can be combined to produce a wide range of complex movements. To map from high level parameters to the motion manifold, we stack a deep feedforward neural network on top of the trained autoencoder. This network is trained to produce realistic motion sequences from parameters such as a curve over the terrain that the character should follow, or a target location for punching and kicking. The feedforward control network and the motion manifold are trained independently, allowing the user to easily switch between feedforward networks according to the desired interface, without re-training the motion manifold. Once motion is generated it can be edited by performing optimization in the space of the motion manifold. This allows for imposing kinematic constraints, or transforming the style of the motion, while ensuring the edited motion remains natural. As a result, the system can produce smooth, high quality motion sequences without any manual pre-processing of the training data.
The behavioral structure of human movements is imposed by multiple sources, such as rules, regulations, choreography, habits, and emotion. Our goal is to identify the behavioral structure in a specific application domain and create a novel sequence of movements that abide by structure-building rules. To do so, we exploit the ideas from formal language, such as rewriting rules and grammar parsing, and adapted those ideas to synthesize the three-dimensional animation of multiple characters. The structured motion synthesis using motion grammars is formulated in two layers. The upper layer is a symbolic description that relates the semantics of each individual's movements and the interaction among them. The lower layer provides spatial and temporal contexts to the animation. Our multi-level MCMC (Markov Chain Monte Carlo) algorithm deals with the syntax, semantics, and spatiotemporal context of human motion to produce highly-structured, animated scenes. The power and effectiveness of motion grammars are demonstrated in animating basketball games from drawings on a tactic board. Our system allows the user to position players and draw out tactical plans, which are animated automatically in virtual environments with three-dimensional, full-body characters.
This paper presents a novel solution for realtime generation of stylistic human motion that automatically transforms unlabeled, heterogeneous motion data into new styles. The key idea of our approach is an online learning algorithm that automatically constructs a series of local mixtures of autoregressive models (MAR) to capture the complex relationships between styles of motion. We construct local MAR models on the fly by searching for the closest examples of each input pose in the database. Once the model parameters are estimated from the training data, the model adapts the current pose with simple linear transformations. In addition, we introduce an efficient local regression model to predict the timings of synthesized poses in the output style. We demonstrate the power of our approach by transferring stylistic human motion for a wide variety of actions, including walking, running, punching, kicking, jumping and transitions between those behaviors. Our method achieves superior performance in a comparison against alternative methods. We have also performed experiments to evaluate the generalization ability of our data-driven model as well as the key components of our system.
This paper introduces a new generative statistical model that allows for human motion analysis and synthesis at both semantic and kinematic levels. Our key idea is to decouple complex variations of human movements into finite structural variations and continuous style variations and encode them with a concatenation of morphable functional models. This allows us to model not only a rich repertoire of behaviors but also an infinite number of style variations within the same action. Our models are appealing for motion analysis and synthesis because they are highly structured, contact aware, and semantic embedding. We have constructed a compact generative motion model from a huge and heterogeneous motion database (about two hours mocap data and more than 15 different actions). We have demonstrated the power and effectiveness of our models by exploring a wide variety of applications, ranging from automatic motion segmentation, recognition, and annotation, and online/offline motion synthesis at both kinematics and behavior levels to semantic motion editing. We show the superiority of our model by comparing it with alternative methods.
In this paper, we propose a new method to efficiently synthesize character motions that involve close contacts such as wearing a T-shirt, passing the arms through the strings of a knapsack, or piggy-back carrying an injured person. We introduce the concept of topology coordinates, in which the topological relationships of the segments are embedded into the attributes. As a result, the computation for collision avoidance can be greatly reduced for complex motions that require tangling the segments of the body. Our method can be combinedly used with other prevalent frame-based optimization techniques such as inverse kinematics.
Presents a system that can synthesize novel motion sequences from a database of motion capture examples. This is achieved through learning a statistical model from the captured data which enables the realistic synthesis of new movements by sampling the original captured sequences. New movements are synthesized by specifying the start and end keyframes. The statistical model identifies segments of the original motion capture data to generate novel motion sequences between the keyframes. The advantage of this approach is that it combines the flexibility of keyframe animation with the realism of motion capture data.
In this paper, we introduce a new virtual baseball batting training system called NICEMEET VR, which enable batters to face a variety of baseball pitchers. The pitching motions are obtained using the video-based motion capture system called KROPS, which enable users to capture the 3D human motion from a single view video sequences. The batter holds a bat with a reflector attached to the sweet spot, and stands in front of the screen where the pitching motions are projected. No real baseball is actually thrown. Instead, a 3D virtual ball is rendered over the screen. The batter swings the bat toward the ball when it reaches the home-base. The trajectory of the swung bat is analyzed using dual supply photoelectric sensors behind the screen, that cast and also detect the infra red beam reflected by the bat. Using our system, the player can practice batting against any pitcher, such as top professional major league pitchers, or famous pitchers in the past by capturing their motion from videos.
In this paper, we present an example-based motion synthesis tech- nique that generates continuous streams of high-fidelity, control- lable motion for interactive applications, such as video games. Our method uses a new data structure called a parametric motion graph to describe valid ways of generating linear blend transitions be- tween motion clips dynamically generated through parametric syn- thesis in realtime. Our system specifically uses blending-based parametric synthesis to accurately generate any motion clip from an entire space of motions by blending together examples from that space. The key to our technique is using sampling methods to iden- tify and represent good transitions between these spaces of motion parameterized by a continuously valued parameter. This approach allows parametric motion graphs to be constructed with little user effort. Because parametric motion graphs organize all motions of a particular type, such as reaching to different locations on a shelf, us- ing a single, parameterized graph node, they are highly structured, facilitating fast decision-making for interactive character control. We have successfully created interactive characters that perform se- quences of requested actions, such as cartwheeling or punching. CR Categories: I.3.7 (Computer Graphics): Three-Dimensional Graphics and Realism—Animation
This paper proposes a methodology that allows users to control character's motion interactively but continuously. Inspired by the work of Gleicher et al. (GSKJ03), we propose a semi-automatic method to build fat graphs where a node corresponds to a pose and its incoming and outgoing edges represent the motion segments starting from and ending at similar poses. A group of edges is built into a fat edge that parameterizes similar motion segments into a blendable form. Employing the existing motion transition and blending methods, our run-time system allows users to control a character interactively in continuous parameter spaces with conventional input devices such as joysticks and the mice. The capability of the proposed methodology is demonstrated through several applications. Although our method has some limitations on motion repertories and qualities, it can be adapted to a number of real-world applications including video games and virtual reality applications.
This paper presents an inverse kinematics system based on a learned model of human poses. Given a set of constraints, our system can produce the most likely pose satisfying those constraints, in realtime. Training the model on different input data leads to different styles of IK. The model is represented as a probability distribution over the space of all possible poses. This means that our IK system can generate any pose, but prefers poses that are most similar to the space of poses in the training data. We represent the probability with a novel model called a Scaled Gaussian Process Latent Variable Model. The parameters of the model are all learned automatically; no manual tuning is required for the learning component of the system. We additionally describe a novel procedure for interpolating between styles.
There are many applications that demand large quantities of natural looking motion. It is difficult to synthesize motion that looks natural, particularly when it is people who must move. In this paper, we present a framework that generates human motions by cutting and pasting motion capture data. Selecting a collection of clips that yields an acceptable motion is a combinatorial problem that we manage as a randomized search of a hierarchy of graphs. This approach can generate motion sequences that satisfy a variety of constraints automatically. The motions are smooth and human-looking. They are generated in real time so that we can author complex motions interactively. The algorithm generates multiple motions that satisfy a given set of constraints, allowing a variety of choices for the animator. It can easily synthesize multiple motions that interact with each other using constraints. This framework allows the extensive re-use of motion capture data for new purposes.
A computer simulation model of human airborne movement is described. The body is modelled as 11 rigid linked segments with 17 degrees of freedom which are chosen with a view to modelling twisting somersaults. The accuracy of the model is evaluated by comparing the simulation values of the angles describing somersault, tilt and twist with the corresponding values obtained from film data of nine twisting somersaults. The maximum deviations between simulation and film are found to be 0.04 revolutions for somersault, seven degrees for tilt and 0.12 revolutions for twist. It is shown that anthropometric measurement errors, from which segmental inertia parameters are calculated, have a small effect on a simulation, whereas film digitization errors can account for a substantial part of the deviation between simulation and film values.
I used a computer simulation model of aerial movement to investigate the techniques for producing and controlling rotations of the human body during free flight. I found that the rotational motion can change from a twisting somersault to a nontwisting somersault by flexing at the hips at a suitable time. Twist may be produced in the aerial phase by means of asymmetrical movements of arms or hips, which result in a tilting of the longitudinal axis away from the plane perpendicular to the angular momentum vector. Asymmetrical movements may also remove the tilt and stop the twist. Elite performances of twisting somersaults are characterized by a large contribution from aerial twisting techniques. A progression of movements is presented for learning a double somersault with one and a half twists in the second somersault.
The Gaussian process latent variable model (GPLVM) is a novel unsupervised approach to nonlinear low dimensional embedding proposed by Lawrence (2005). This paper presents the development of a framework for the implementation of the GPLVM for fault detection. A series of experiments have been carried out comparing and combining the GPLVM to the conventional and widely used linear dimension reduction technique of principal component analysis (PCA). The inclusion of the GPLVM for the visualisation and data analysis, led to a considerable improvement in the classification results
This tutorial provides an overview of the basic theory of hidden
Markov models (HMMs) as originated by L.E. Baum and T. Petrie (1966) and
gives practical details on methods of implementation of the theory along
with a description of selected applications of the theory to distinct
problems in speech recognition. Results from a number of original
sources are combined to provide a single source of acquiring the
background required to pursue further this area of research. The author
first reviews the theory of discrete Markov chains and shows how the
concept of hidden states, where the observation is a probabilistic
function of the state, can be used effectively. The theory is
illustrated with two simple examples, namely coin-tossing, and the
classic balls-in-urns system. Three fundamental problems of HMMs are
noted and several practical techniques for solving these problems are
given. The various types of HMMs that have been studied, including
ergodic as well as left-right models, are described
In this paper we present a novel method for creating realistic, controllable motion. Given a corpus of motion capture data, we automatically construct a directed graph called a motion graph that encapsulates connections among the database. The motion graph consists both of pieces of original motion and automatically generated transitions. Motion can be generated simply by building walks on the graph. We present a general framework for extracting particular graph walks that meet a user's specifications. We then show how this framework can be applied to the specific problem of generating different styles of locomotion along arbitrary paths.
We approach the problem of stylistic motion synthesis by learning motion patterns from a highly varied set of motion capture sequences. Each sequence may have a distinct choreography, performed in a distinct style. Learning identifies common choreographic elements across sequences, the different styles in which each element is performed, and a small number of stylistic degrees of freedom which span the many variations in the dataset. The learned model can synthesize novel motion data in any interpolation or extrapolation of styles. For example, it can convert novice ballet motions into the more graceful modern dance of an expert. The model can also be driven by video, by scripts, or even by noise to generate new choreography and synthesize virtual motion-capture in many styles. In Proceedings of SIGGRAPH 2000, July 23-28, 2000. New Orleans, Louisiana, USA. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole o...
Realtime style trans62 fer for unlabeled heterogeneous human motion119:1-119:10. doi:10.1145/2766999 Construction and optimal search of inter65 polated motion graphs Interaction patches 68 for multi-character animation
- S Xia
- C Wang
- J Chai
- J Hodgins
- A Safonova
- Jk Hodgins
- Shum
- Hph
- T Komura
- M Shiraishi
- S Yamazaki
60
doi:10.1145/1230100.1230123.
61
[22] Xia, S, Wang, C, Chai, J, Hodgins, J. Realtime style trans62
fer for unlabeled heterogeneous human motion. ACM Trans Graph
63
2015;34(4):119:1-119:10. doi:10.1145/2766999.
64
[23] Safonova, A, Hodgins, JK. Construction and optimal search of inter65
polated motion graphs. ACM Trans Graph 2007;26(3). doi:10.1145/
66
1276377.1276510.
67
[24] Shum, HPH, Komura, T, Shiraishi, M, Yamazaki, S. Interaction patches
68
for multi-character animation. ACM Trans Graph 2008;27(5):114:169
114:8. doi:10.1145/1409060.1409067.
70
[25] Shum, HPH, Komura, T, Yamazaki, S. Simulating multiple character
71
interactions with collaborative and adversarial goals. IEEE Transactions
72
on Visualization and Computer Graphics 2012;18(5):741-752. doi:10.
A finite state machine based on topology
- Esl Ho
- T Komura
Ho, ESL, Komura, T. A finite state machine based on topology co-75
- Ogy Coordinates
ogy Coordinates. Computer Graphics Forum 2009;doi:10.1111/j.
79
- Ditions
ditions. Applied Ergonomics 2016;doi:10.1016/j.apergo.2016.
86
10.015.
87
Topology aware 96 data-driven inverse kinematics
- Esl Ho
- Hph Shum
- Cheung
- Ym
- Pc Yuen
Ho, ESL, Shum, HPH, Cheung, Ym, Yuen, PC. Topology aware
96
data-driven inverse kinematics. Comp Graph Forum 2013;32(7):61-70.
97
A data-132 driven approach to quantifying natural human motion
- L Ren
- A Patrick
- A A Efros
- J K Hodgins
- J M Rehg
Ren, L, Patrick, A, Efros, AA, Hodgins, JK, Rehg, JM. A data-132
driven approach to quantifying natural human motion. ACM Trans Graph 133
2005;24(3):1090-1097. doi:10.1145/1073204.1073316.
Real handball goalkeeper vs. virtual handball thrower
P, et al. Real handball goalkeeper vs. virtual handball thrower. Presence:
1805964.1805970. doi:10.1145/1805964.1805970.
Motion synthesis and editing in low-dimensional 38 spaces. Computer Animation and Virtual Worlds (Special Issue: CASA 39
- H J Shin
- J Lee
Shin, HJ, Lee, J. Motion synthesis and editing in low-dimensional
38
spaces. Computer Animation and Virtual Worlds (Special Issue: CASA
39
2006) 2006;17(3-4):219 -227.
Motion synthesis from annota-49 tions
- O Arikan
- Forsyth
- Da
- J F O'brien
Arikan, O, Forsyth, DA, O'Brien, JF. Motion synthesis from annota-49
tions. ACM Trans Graph 2003;22(3):402-408. doi:10.1145/882262.
50
882284.
Interaction patches 68 for multi-character animation
- Hph Shum
- T Komura
- M Shiraishi
- S Yamazaki
Shum, HPH, Komura, T, Shiraishi, M, Yamazaki, S. Interaction patches
68
for multi-character animation. ACM Trans Graph 2008;27(5):114:1-69
114:8. doi:10.1145/1409060.1409067.
ordinates for wrestling games
ordinates for wrestling games. Computer Animation and Virtual Worlds
76
2011;22(5):435-443. doi:10.1002/cav.376.
77
ISBN 978-3-642-16957-1
- Heidelberg Berlin
Berlin, Heidelberg: Springer-Verlag. ISBN 978-3-642-16957-1; 2010, p.
91
23-34. doi:10.1007/978-3-642-16958-8_3.
92