On-Site Real-Time 3D Match Move for MR-based Previsualization with Relighting
Ryosuke Ichikari∗, Kaori Kikuchi, Wataru Toishita, Ryuhei Tenmoku, Fumihisa Shibata, Hideyuki Tamura
Figure 1: Our method (left: Structure-from-motion, middle: Match move, right: Relighting)
We are developing a previsualization method called MR-PreViz,
which utilizes mixed reality technology for filmmaking [Tenmoku
et al. 2006]. To determine camera-work at the shooting site, esti-
mating camera position and posture is required. In this paper, we
introduce a method for on-site real-time 3D match move and re-
lighting for MR-PreViz. To realize the match move, we developed
a computer vision-based camera tracking method using natural fea-
ture tracking. This method is based on details about the site cap-
tured in advance. The method can automatically construct a feature
landmark database (LMDB) using a fiducial marker. Moreover, the
result of the method enables MR-PreViz to design lighting for the
site using a relighting method. To add lighting effects to the real
objects, the relighting method uses reflectance properties of the real
objects and LMDB.
2Flow of On-Site Match Move and Relighting
out using specialized equipment as shown in the following steps.
Step 1 Preparation of CG Action Data
This step collects CG environment data and action data be-
fore making MR-PreViz movies. The action data are sepa-
rately recorded, and their timings and positions are adjusted
for building one-on-one fighting action. Moreover, we devel-
oped a method of adjusting the action displacement to fit into
the motion area derived from the actual site.
Step 2 Gathering Geometrical and Photometrical Information of
the Shooting Site (shown in the left image of Figure 1)
This step gathers geometrical information including the 3D
positions of the feature points and reflectance property of the
surfaces at the site for MR-PreViz shooting. This process is
automatically realized by utilizing a fiducial marker.
Step 3 Compositing MR-PreViz Image Using Real-Time 3D
Match Move and Relighting (shown in the middle/right im-
age of Figure 1)
The geometrical information is used for real-time match move
during the MR-PreViz shooting. The lighting condition of the
MR-PreViz movie can be changed using the geometry and re-
flectance property gathered in Step 2.
3Gathering Geometrical and Photometrical
Information of the Shooting Site
In preparation for the MR-PreViz shooting, geometrical and photo-
metrical information of the site is gathered by pre-shooting using
markers. Geometry of the site is estimated using a Structure-From-
Motion technique. In particular, the positions of feature points in
3D space are first estimated by using epipolar geometry on several
frames in the video sequence. Secondly, 6DOFs of the camera and
positions of new feature points are simultaneously calculated by
tracking the feature points. Finally, the coordinates of the 3D points
are transformed into world coordinates by recognizing a fiducial
marker. The data of the feature points are entered into our LMDB.
For the relighting, the relationship between pixel value and illu-
minance is obtained as a reflectance property. Illuminance is au-
tomatically calculated under several lighting conditions by using a
reference marker where the reflectance property of the marker is
Real-Time 3D Match Move and Relighting
Match move is realized by correlating the 2D feature points in the
images with the 3D points in the LMDB.
After compositing the MR-PreViz movie, lighting conditions of the
MR-PreViz images are intentionally changed in according to the
method described in [Ichikari et al. 2009]. In this case, the method
uses the geometry and the reflectance property of the site captured
in Step 2. Thus, our method realizes the interactive design capabil-
ities of camera-work and lighting at the site in real-time for MR-
This research is partly supported by the CREST Program of JST.
ICHIKARI, R., HATANO, R., OHSHIMA, T., SHIBATA, F., AND TAMURA,
H. 2009. Designing cinematic lighting by relighting in mr-based pre-
visualization. In SIGGRAPH ASIA Posters.
TENMOKU, R., ICHIKARI, R., SHIBATA, F., KIMURA, A., AND TAMURA,
H. 2006. Design and prototype implementation of mr pre-visualization
workflow. In DVD-ROM Proc. of the Int’l Workshop on Mixed Reality
Technology for Filmmaking, 1–7.
Copyright is held by the author / owner(s).
SIGGRAPH 2010, Los Angeles, USA, August 3 ‒ 7, 2010.