On-site real-time 3D match move for MR-based previsualization with relighting.
ABSTRACT We are developing a previsualization method called MR-PreViz, which utilizes mixed reality technology for filmmaking [Tenmoku et al. 2006]. To determine camera-work at the shooting site, estimating camera position and posture is required. In this paper, we introduce a method for on-site real-time 3D match move and relighting for MR-PreViz. To realize the match move, we developed a computer vision-based camera tracking method using natural feature tracking. This method is based on details about the site captured in advance. The method can automatically construct a feature landmark database (LMDB) using a fiducial marker. Moreover, the result of the method enables MR-PreViz to design lighting for the site using a relighting method. To add lighting effects to the real objects, the relighting method uses reflectance properties of the real objects and LMDB.
- [Show abstract] [Hide abstract]
ABSTRACT: In order to rehearse for a play or a scene from a movie, it is generally required that the actors are physically present at the same time in the same place. In this paper we present an example and experience of a full body motion shared virtual environment (SVE) for rehearsal. The system allows actors and directors to meet in an SVE in order to rehearse scenes for a play or a movie, that is, to perform some dialogue and blocking (positions, movements, and displacements of actors in the scene) rehearsal through a full body interactive virtual reality (VR) system. The system combines immersive VR rendering techniques as well as network capabilities together with full body tracking. Two actors and a director rehearsed from separate locations. One actor and the director were in London (located in separate rooms) while the second actor was in Barcelona. The Barcelona actor used a wide field-of-view head-tracked head-mounted display, and wore a body suit for real-time motion capture and display. The London actor was in a Cave system, with head and partial body tracking. Each actor was presented to the other as an avatar in the shared virtual environment, and the director could see the whole scenario on a desktop display, and intervene by voice commands. A video stream in a window displayed in the virtual environment also represented the director. The London participant was a professional actor, who afterward commented on the utility of the system for acting rehearsal. It was concluded that full body tracking and corresponding real-time display of all the actors' movements would be a critical requirement, and that blocking was possible down to the level of detail of gestures. Details of the implementation, actors, and director experiences are provided.Presence Teleoperators & Virtual Environments 04/2012; 21(2):229-243. · 1.04 Impact Factor
On-Site Real-Time 3D Match Move for MR-based Previsualization with Relighting
Ryosuke Ichikari∗, Kaori Kikuchi, Wataru Toishita, Ryuhei Tenmoku, Fumihisa Shibata, Hideyuki Tamura
Figure 1: Our method (left: Structure-from-motion, middle: Match move, right: Relighting)
We are developing a previsualization method called MR-PreViz,
which utilizes mixed reality technology for filmmaking [Tenmoku
et al. 2006]. To determine camera-work at the shooting site, esti-
mating camera position and posture is required. In this paper, we
introduce a method for on-site real-time 3D match move and re-
lighting for MR-PreViz. To realize the match move, we developed
a computer vision-based camera tracking method using natural fea-
ture tracking. This method is based on details about the site cap-
tured in advance. The method can automatically construct a feature
landmark database (LMDB) using a fiducial marker. Moreover, the
result of the method enables MR-PreViz to design lighting for the
site using a relighting method. To add lighting effects to the real
objects, the relighting method uses reflectance properties of the real
objects and LMDB.
2Flow of On-Site Match Move and Relighting
out using specialized equipment as shown in the following steps.
Step 1 Preparation of CG Action Data
This step collects CG environment data and action data be-
fore making MR-PreViz movies. The action data are sepa-
rately recorded, and their timings and positions are adjusted
for building one-on-one fighting action. Moreover, we devel-
oped a method of adjusting the action displacement to fit into
the motion area derived from the actual site.
Step 2 Gathering Geometrical and Photometrical Information of
the Shooting Site (shown in the left image of Figure 1)
This step gathers geometrical information including the 3D
positions of the feature points and reflectance property of the
surfaces at the site for MR-PreViz shooting. This process is
automatically realized by utilizing a fiducial marker.
Step 3 Compositing MR-PreViz Image Using Real-Time 3D
Match Move and Relighting (shown in the middle/right im-
age of Figure 1)
The geometrical information is used for real-time match move
during the MR-PreViz shooting. The lighting condition of the
MR-PreViz movie can be changed using the geometry and re-
flectance property gathered in Step 2.
3Gathering Geometrical and Photometrical
Information of the Shooting Site
In preparation for the MR-PreViz shooting, geometrical and photo-
metrical information of the site is gathered by pre-shooting using
markers. Geometry of the site is estimated using a Structure-From-
Motion technique. In particular, the positions of feature points in
3D space are first estimated by using epipolar geometry on several
frames in the video sequence. Secondly, 6DOFs of the camera and
positions of new feature points are simultaneously calculated by
tracking the feature points. Finally, the coordinates of the 3D points
are transformed into world coordinates by recognizing a fiducial
marker. The data of the feature points are entered into our LMDB.
For the relighting, the relationship between pixel value and illu-
minance is obtained as a reflectance property. Illuminance is au-
tomatically calculated under several lighting conditions by using a
reference marker where the reflectance property of the marker is
Real-Time 3D Match Move and Relighting
Match move is realized by correlating the 2D feature points in the
images with the 3D points in the LMDB.
After compositing the MR-PreViz movie, lighting conditions of the
MR-PreViz images are intentionally changed in according to the
method described in [Ichikari et al. 2009]. In this case, the method
uses the geometry and the reflectance property of the site captured
in Step 2. Thus, our method realizes the interactive design capabil-
ities of camera-work and lighting at the site in real-time for MR-
This research is partly supported by the CREST Program of JST.
ICHIKARI, R., HATANO, R., OHSHIMA, T., SHIBATA, F., AND TAMURA,
H. 2009. Designing cinematic lighting by relighting in mr-based pre-
visualization. In SIGGRAPH ASIA Posters.
TENMOKU, R., ICHIKARI, R., SHIBATA, F., KIMURA, A., AND TAMURA,
H. 2006. Design and prototype implementation of mr pre-visualization
workflow. In DVD-ROM Proc. of the Int’l Workshop on Mixed Reality
Technology for Filmmaking, 1–7.
Copyright is held by the author / owner(s).
SIGGRAPH 2010, Los Angeles, USA, August 3 ‒ 7, 2010.