No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Automatic 3D Posing from 2D Hand-Drawn Sketches
Abstract
Inferring the 3D pose of a character from a drawing is a non-trivial and under-constrained problem. Solving it may help automate various parts of an animation production pipeline such as pre-visualisation. In this paper, a novel way of inferring the 3D pose from a monocular 2D sketch is proposed. The proposed method does not make any external assumptions about the model, allowing it to be used on different types of characters. The 3D pose inference is formulated as an optimisation problem and a parallel variation of the Particle Swarm Optimisation algorithm called PARAC-LOAPSO is utilised for searching the minimum. Testing in isolation as well as part of a larger scene, the presented method is evaluated by posing a lamp and a horse character. The results show that this method is robust and is able to be extended to various types of models.
... Drawing the correct scale of objects without absolute references (for cinematic production) or 3D references (for CGI productions) is another possible mistake. Performing the storyboarding process in 3D can resolve these issues and provide an interactive pre-visualization of the storyboard [7]. ...
... The proposed method can potentially simplify the process of shooting CGI, resulting in a novel workflow based on the interactive collaboration of human creativity with automated intelligence. In [7], the authors investigate a novel way to automatically pose 3D models from 2D sketches: the pose inference is formulated as an optimization problem, and a parallel variation of the Particle Swarm Optimisation algorithm has been proposed to search for the minimum of its objective function. The scope of this research was to pose models for pre-visualization, providing a direct link between the storyboarding and the pose layout phases of a 3D animation pipeline. ...
Creatives in the animation and film industries constantly explore new and innovative tools and methods to enhance their creative process, especially in pre-production. As realistic, real-time rendering techniques have emerged in recent years, 3D game engines, modeling, and animation tools have been exploited to support storyboarding and movie prototyping. This research proposes a 3D storyboarding tool to improve existing storytelling approaches. A novel storyboarding pipeline is proposed, which can automatically generate a storyboard including camera details and a textual description of the events occurring in each scene. Users create storyboards by selecting actors, performing available actions, positioning the camera in the 3D scene, and taking screenshots to save a vignette of the storyboard; the corresponding description is generated based on the actors’ actions and their status. A software implementation of the proposed pipeline has also been developed in the guise of a 3D desktop application aimed at expert and novice storyboarders. The system has been tested to evaluate its usability. Preliminary results confirm that the users have appreciated the application.
... Using drawings and minimal user input, a database approach example leverages a set of mocap animation sequences to use as a knowledge base (Jain et al. 2009). Finally, an example of the generative approach uses global optimization and generates new poses every iteration, until the correct pose is found (Gouvatsos et al. 2014). ...
... Using drawings and minimal user input, a database approach example leverages a set of mocap animation sequences to use as a knowledge base (Jain et al. 2009). Finally, an example of the generative approach uses global optimization and generates new poses every iteration, until the correct pose is found (Gouvatsos et al. 2014). ...
he line of action is a conceptual tool often used by cartoonists and illustrators to help make their figures more consistent and more dramatic. We often see the expression of characters--may it be the dynamism of a super hero, or the elegance of a fashion model--well captured and amplified by a single aesthetic line. Usually this line is laid down in early stages of the drawing and used to describe the body's principal shape. By focusing on this simple abstraction, the person drawing can quickly adjust and refine the overall pose of his or her character from a given viewpoint. In this paper, we propose a mathematical definition of the line of action (LOA), which allows us to automatically align a 3D virtual character to a user specified LOA by solving an optimization problem. We generalize this framework to other types of lines found in the drawing literature, such as secondary lines used to place arms. Finally, we show a wide range of poses and animations that were rapidly created using our system.
The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human-inhabited environment. Because of many potentially important applications, “looking at people” is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D approaches. Where appropriate, systems are reviewed. We conclude with some thoughts about future directions.
Virtual beings are playing a remarkable role in today's public entertainment, while ordinary users are still treated as audiences due to the lack of appropriate expertise, equipment, and computer skills. In this paper, we present a fast and intuitive storyboarding interface, which enables users to sketch-out 3D virtual humans, 2D/3D animations, and character intercommunication. We devised an intuitive "stick figure→fleshing-out→skin mapping" graphical animation pipeline, which realises the whole process of key framing, 3D pose reconstruction, virtual human modelling, motion path/timing control, and the final animation synthesis by almost pure 2D sketching. A "creative model-based method" is developed, which emulates a human perception process, to generate the 3D human bodies of variational sizes, shapes, and fat distributions. Meanwhile, our current system also supports the sketch-based crowd animation and the storyboarding of the 3D multiple character intercommunication. This system has been formally tested by various users on Tablet PC. After minimal training, even a beginner can create vivid virtual humans and animate them within minutes.
The skills required to create compelling three-dimensional animation using computer software are quite different from those required to create compelling hand animation with pencil and paper. The three-dimensional medium has several advantages over the traditional medium---it is easy to relight the scene, render it from different view-points, and add physical simulations. In this work, we propose a method to leverage the talent of traditionally trained hand animators to create three-dimensional animation of human motion, while allowing them to work in the medium that is familiar to them. The input to our algorithm is a set of hand-animated frames. Our key insight is to use motion capture data as a source of domain knowledge and 'lift' the two-dimensional animation to three dimensions, while maintaining the unique style of the input animation. A motion capture clip is projected to two dimensions. First, time alignment is done to match the timing of the hand-drawn frames and then, the limbs are aligned to better match the pose in the hand-drawn frames. Finally the motion is reconstructed in three dimensions. We demonstrate our algorithm on a variety of hand animated motion sequences on different characters, including ballet, a stylized sneaky walk, and a sequence of jumping jacks.
In this paper we address the problem of human body pose estimation from still images. A multi-view set of images of a person sitting at a table is acquired and the pose estimated. Reliable and efficient pose estimation from still images represents an important part of more complex algorithms, such as tracking human body pose in a video sequence, where it can be used to automatically initialise the tracker on the first frame. The quality of the initialisation influences the performance of the tracker in the subsequent frames. We formulate the body pose estimation as an analysis-by-synthesis optimisation algorithm, where a generic 3D human body model is used to illustrate the pose and the silhouettes extracted from the images are used as constraints. A simple test with gradient descent optimisation run from randomly selected initial positions in the search space shows that a more powerful optimisation method is required. We investigate the suitability of the Particle Swarm Optimisation (PSO) for solving this problem and compare its performance with an equivalent algorithm using Simulated Annealing (SA). Our tests show that the PSO outperforms the SA in terms of accuracy and consistency of the results, as well as speed of convergence.
An articulated figure is often modeled as a set of rigid segments connected with joints. Its configuration can be altered by varying the joint angles. Although it is straightforward to compute figure configurations given joint angles (forward kinematics), it is not so to find the joint angles for a desired configuration (inverse kinematics). Since the inverse kinematics problem is of special importance to an animator wishing to set a figure to a posture satisfying a set of positioning constraints, researchers have proposed many approaches. But when we try to follow these approaches in an interactive animation system where the object to operate on is as highly articulated as a realistic human figure, they fail in either generality or performance, and so a new approach is fostered. Our approach is based on nonlinear programming techniques. It has been used for several years in the spatial constraint system in the Jack TM human figure simulation software developed at the Compute...
Generating highly expressive and caricatured poses can be difficult in 3D computer animation because artists must interact with characters indirectly through complex character rigs. Furthermore, since caricatured poses often involve large bends and twists, artifacts arise with traditional skinning algorithms that are not designed to blend large, disparate rotations and cannot represent extremely large rotations. To overcome these problems, we introduce a differential blending algorithm that can successfully encode and blend large transformations, overcoming the inherent limitation of previous skeletal representations. Based on this blending method, we illustrate a sketch-based interface that supports curved bones and implements the line-of-action concept from hand-drawn animation to create expressive poses in 3D animation. By interpolating stored differential transformations across temporal keyframes, our system also generates caricatured animation. We present a detailed technical analysis of our differential blending algorithm and show several posing and animation results created using our system to demonstrate the utility of our method in practice.
This work presents Sketch2Scene, a framework that automatically turns a freehand sketch drawing inferring multiple scene objects to semantically valid, well arranged scenes of 3D models. Unlike the existing works on sketch-based search and composition of 3D models, which typically process individual sketched objects one by one, our technique performs co-retrieval and co-placement of 3D relevant models by jointly processing the sketched objects. This is enabled by summarizing functional and spatial relationships among models in a large collection of 3D scenes as structural groups. Our technique greatly reduces the amount of user intervention needed for sketch-based modeling of 3D scenes and fits well into the traditional production pipeline involving concept design followed by 3D modeling. A pilot study indicates that it is promising to use our technique as an alternative but more efficient tool of standard 3D modeling for 3D scene construction.
This paper presents an empirical comparison of several shape representations in order to search a database of training examples (silhouettes) for the task of human pose estimation. In particular, we compare the Discrete Cosine Transform (DCT), Lipschitz embeddings and the Histogram of Shape Con- texts that has previously demonstrated some success in this task. Our results suggest that a simple linear transformation of the image (such as the DCT) is as effective as the more complex, non-linear methods.
Animated virtual humans created by skilled artists play a remarkable role in today’s public entertainment. However, ordinary
users are still treated as audiences due to the lack of appropriate expertise, equipment, and computer skills. We developed
a new method and a novel sketching interface, which enable anyone who can draw to “sketch-out” 3D virtual humans and animation.
We devised a “Stick Figure→ Fleshing-out→Skin Mapping” graphical pipeline, which decomposes the complexity of figure drawing
and considerably boosts the modelling and animation efficiency. We developed a gesture-based method for 3D pose reconstruction
from 2D stick figure drawings. We investigated a “Creative Model-based Method”, which performs a human perception process
to transfer users’ 2D freehand sketches into 3D human bodies of various body sizes, shapes and fat distributions. Our current
system supports character animation in various forms including articulated figure animation, 3D mesh model animation, and
2D contour/NPR animation with personalised drawing styles. Moreover, this interface also supports sketch-based crowd animation
and 2D storyboarding of 3D multiple character interactions. A preliminary user study was conducted to support the overall
system design. Our system has been formally tested by various users on Tablet PC. After minimal training, even a beginner
can create vivid virtual humans and animate them within minutes.
Medical imaging has produced a large number of volumetric images capturing biological structures in 3D. Computer-based understanding of these structures can often benefit from the knowledge of shape components, particularly rod-like and plate-like parts, in such volumes. Previously, skeletons have been a common tool for identifying these shape components in a solid object. However, obtaining skeletons of a grayscale volume poses new challenges due to the lack of a clear boundary between object and background. In this paper, we present a new skeletonization algorithm on grayscale volumes typical to medical imaging (e.g., MRI, CT and EM scans), for the purpose of identifying shape components. Our algorithm does not require an explicit segmentation of the volume into object and background, and is capable of producing skeletal curves and surfaces that lie centered at rod-shaped and plate-shaped parts in the grayscale volume. Our method is demonstrated on both synthetic and medical data.
Particle swarm optimization has become a common heuristic technique in the optimization community, with many researchers exploring the concepts, issues, and applications of the algorithm. In spite of this attention, there has as yet been no standard definition representing exactly what is involved in modern implementations of the technique. A standard is defined here which is designed to be a straightforward extension of the original algorithm while taking into account more recent developments that can be expected to improve performance on standard measures. This standard algorithm is intended for use both as a baseline for performance testing of improvements to the technique, as well as to represent PSO to the wider optimization community
Learning to track 3d human motion from silhouettes
- Agarwal A
- Triggs B
AGARWAL A., TRIGGS B.: Learning to track 3d human
motion from silhouettes. In In International Conference on Machine Learning (2004), pp. 9-16. 4
Recovering 3d human pose from monocular images. Pattern Analysis and Machine Intelligence
- Agarwal A
- Triggs B
AGARWAL A., TRIGGS B.: Recovering 3d human pose
from monocular images. Pattern Analysis and Machine Intelligence, IEEE Transactions on 28, 1 (Jan 2006), 44-58. doi:
10.1109/TPAMI.2006.21. 4
Interactive skeleton techniques for enhancing motion dynamics in key frame animation
- Burtnyk N
- M Wein
BURTNYK N., WEIN M.: Interactive skeleton techniques for enhancing motion dynamics in key frame animation.
Commun. ACM 19, 10 (Oct. 1976), 564-569. URL: http:
//doi.acm.org/10.1145/360349.360357, doi:10.
1145/360349.360357. 2
A sketching interface for articulated figure animation
- Davis J Agrawala
- M Chuang
- E Popović
- Z Salesin D
[DAC * 03] DAVIS J., AGRAWALA M., CHUANG E., POPOVIĆ
Z., SALESIN D.: A sketching interface for articulated figure animation.
In Proceedings of the 2003 ACM SIG-GRAPH/Eurographics Symposium on Computer Animation
(Aire-la-Ville, Switzerland, Switzerland, 2003), SCA '03, Eurographics Association, pp. 320-328. URL: http://dl.acm.
org/citation.cfm?id=846276.846322. 1
Particle swarm optimization with soft search space partitioning for video-based markerless pose tracking
- P Fleischmann
- Austvoll I
- Kwolek B
FLEISCHMANN P., AUSTVOLL I., KWOLEK B.:
Particle swarm optimization with soft search space partitioning for video-based markerless pose tracking.
In
Advanced Concepts for Intelligent Vision Systems, Blanc-Talon J., Philips W., Popescu D., Scheunders P., Zemcik
P., (Eds.), vol. 7517 of Lecture Notes in Computer Science.
Tangible and modular input device for character articulation
- Jacobson A
- D Panozzo
- O Glauser
- C Pradalier
- O Hilleges
- Sorkine-Horning O
[JPG * 14] JACOBSON A., PANOZZO D., GLAUSER O.,
PRADALIER C., HILLEGES O., SORKINE-HORNING O.:
Tangible and modular input device for character articulation. ACM Transactions on Graphics (proceedings of ACM
SIGGRAPH) 33, 4 (2014). 2
SCA '03, Eurographics Association
- Li Y Gleicher
- M Xu Y.-Q
- Shum H.-Y
LI Y., GLEICHER M., XU Y.-Q., SHUM H.-Y.: Stylizing motion with drawings. In Proceedings of the 2003 ACM
SIGGRAPH/Eurographics Symposium on Computer Animation
(Aire-la-Ville, Switzerland, Switzerland, 2003), SCA '03, Eurographics Association, pp. 309-319. URL: http://dl.acm.
org/citation.cfm?id=846276.846320. 1
Lifesketch -a framework for sketch-based modelling and animation of 3d objects
- Yang R Wãijnsche
YANG R., WÃIJNSCHE B. C.: Lifesketch -a framework
for sketch-based modelling and animation of 3d objects. In in
âȂŸProceedings of the Australasian User Interface Conference
(AUIC 2010 (2010), p. 2010. 1