3D video stabilization for humanoid eyes using vision and inertial sensors inspired by human VOR

Abstract

This paper proposes a new 3D video stabilization for humanoid eyes using vision and inertial sensors. Most existing video stabilization methods based on 2D motion model work well where scenes are far from the camera or can be modeled as a plane. But these 2D video stabilizations may fail to stabilize unstable videos because a humanoid robot operates in the real 3D world. And so far 3D video stabilization approaches cannot be directly applied to humanoid robot eyes because they use an off-line optimization method. In order to solve above problems, we propose 3D video stabilization based on a spatially-varying warping technique that uses sparse reconstructed 3D data. And we integrate the proposed 3D video stabilization into a hardware stabilization device for humanoid eyes inspired by human VOR in order to compensate large and rapid unwanted motions. The experimental results showed that the proposed 3D video stabilization system can be a potential candidate for humanoid eyes.

Full text

Page 1

Abstract—This paper proposes a new 3D video stabilization
for humanoid eyes using vision and inertial sensors. Most
existing video stabilization methods based on 2D motion model
work well where scenes are far from the camera or can be
modeled as a plane. But these 2D video stabilizations may fail to
stabilize unstable videos because a humanoid robot operates in
the real 3D world. And so far 3D video stabilization approaches
cannot be directly applied to humanoid robot eyes because they
use an off-line optimization method. In order to solve above
problems, we propose 3D video stabilization based on a
spatially-varying warping technique
reconstructed 3D data. And we integrate the proposed 3D video
stabilization into a hardware stabilization device for humanoid
eyes inspired by human VOR in order to compensate large and
rapid unwanted motions. The experimental results showed that
the proposed 3D video stabilization system can be a potential
candidate for humanoid eyes.

Index Terms – 3D video stabilization, Spatially-varying
warping, Vestibulo-Ocular Reflex (VOR)
that uses sparse
I. INTRODUCTION
ECENTLY, humanoid robots are able to walk and run.
When they move, a video acquired from their eyes on
head is not stable, that is shaky or jiggling. Therefore a remote
operator who controls the humanoid robot is unpleasant to see
that video. Furthermore, this unstable video might degrade
the performance of vision processing such as visual tracking,
detection, recognition, and surveillance due to unwanted
shakes or jiggles. If an unstable video from humanoid robot
eyes can be stabilized, these performance above-mentioned
will be surely improved.
Video stabilization (or digital image stabilization) is the
technique that removes undesired motions caused by
unintentional shakes [1]. Here, undesired motions are
considered to have higher frequency than desired motions [2].
For example, when a humanoid robot runs, a shaky motion
caused by running is an undesired motion.
There have been many developments on video stabilization
in consumer electronics devices, such as digital cameras and

Manuscript received July 23, 2010. This research was supported by the
MKE(The Ministry of Knowledge Economy), Korea, under the Human
Resources Development Program for Convergence Robot Specialists support
program supervised by the NIPA(National IT Industry Promotion Agency)
(NIPA-2010-C7000-1001-0007).
Yeon Geol Ryu and Myung Jin Chung are with the Division of Electrical
Engineering, Korea Advanced Institute of Science and Technology (KAIST),
Daejeon, Korea, 305-701 (e-mail:
mjchung@ee.kaist.ac.kr).
Hyun Chul Roh is with the Robotics Program, KAIST, Daejeon, Korea,
305-701 (e-mail: rohs@cheonji.kaist.ac.kr).
ygryu@cheonji.kaist.ac.kr,
camcorders [3-5]. But these video stabilization techniques in
electronics devices are hard to be applied to humanoid vision
system. Large motion is difficult to be compensated because
these techniques have small baseline of an active lens or a
vision sensor.
2D motion based video stabilization technology is already
mature. If scenes (or objects) are far from the camera or can
be modeled as a plane, it works well. But video stabilization
based on 2D motion model has a fundamental limitation that
is not able to exactly represent the real 3D world where
humanoid robots operate. It may fail to stabilize long video
due to error accumulation caused by the lack of the camera
motion model [6]. Therefore, we have to adopt more
advanced camera motion model such as 3D motion model to
be applied to humanoid robot vision system.
Several 3D camera motion model based video stabilization
papers that showed very surprising experiment results have
been recently published in computer vision community [7-9].
By the way, so far 3D video stabilization approaches cannot
be directly applied to humanoid robot vision system because
these approaches cannot process in real-time. Unfortunately,
they use an optimization method that is batch process in
off-line.
There were some researches for robot eyes. But they focused
on only hardware stabilization or compensation rather than
software video stabilization. Hardware stabilization cannot
clearly remove unwanted motion due to inaccurate sensing
and actuating, and a delay from a sensor to a actuator.
In this paper, we propose 3D video stabilization for
humanoid robot eyes using vision and inertial sensors
inspired by human VOR to overcome the limitations that are a
small baseline problem, a weak 2D motion model, and an
off-line processing above mentioned. Our main contribution
in 3D video stabilization is a spatially-varying warping
technique that uses sparse reconstructed 3D data to generate
stabilized image sequences. Furthermore, we integrated the
proposed 3D video stabilization into a hardware stabilization
device for robotic eyes in order to overcome the problem of
large and rapid motions.
The rest of this paper is organized as follows. Section II
describes the related works including video stabilization on
consumer electronics devices, 2D and 3D motion model
based video stabilization, and robot eyes. The proposed 3D
video stabilization for humanoid robot eyes, which is
described in section III. In section IV, various results of the
experiments are presented, while conclusions and future work
are drawn in section V.
3D Video Stabilization for Humanoid Eyes Using Vision and Inertial
Sensors Inspired by Human VOR
Yeon Geol Ryu, Hyun Chul Roh, and Myung Jin Chung, Member, IEEE
R

Page 2

II. RELATED WORK
A. Video stabilization on consumer electronics devices
There are two kinds of video stabilization techniques on
consumer electronics devices such as still cameras and
camcorders. One is optical stabilization and the other is active
sensor stabilization [10]. Optical stabilization uses a floating
lens element to compensate for pan and tilt but not roll
rotation. Active sensor stabilization uses a moving image
sensor to compensate for 2D translations. However, when a
camera moves a lot, these two stabilizations fail to stabilize
due to their small baseline. A floating lens and moving image
sensor can move within very small distance.
B. 2D motion model based video stabilization
The research of 2D motion model (translation, similarity,
affine, and projective) based video stabilization has already
been much progress. Many researches showed their excellent
experimental results. And there are open video stabilization
tools [11-13]. This 2D video stabilization generally consists
of 3 steps: motion estimation, motion filtering (or smoothing),
image composition (or image warping). The 2D based
algorithm can be distinguished by the methods used in these
three steps. First, in the motion estimation step, the accuracy
of estimated motion parameters ultimately determines the
effectiveness of the stabilization process. Second, motion
filtering step removes unwanted motions from the total
motion that is estimated in the first step. Many approaches
have been proposed. For example, Gaussian filtering [2],
parabolic fitting [14], and regularization [15] have been
proposed as an off-line processing, while motion vector
integration [16] and Kalman filter [17-18] are able to process
in real-time. In this paper, the Kalman filter is adopted to
process in real-time. Lastly, image composition step warps
(or transforms) an unstable input image into a stabilized
image using correction motion parameters that are filtered out
in the second step.
If scenes (or objects) are far from the camera, 2D based
video stabilization works well. But it has a fundamental
limitation that is not able to exactly represent real 3D camera
motions (three translations and three rotations). It sometimes
fails to stabilize long video due to error accumulation caused
by the lack of camera motion model [7].
C. 3D motion model based video stabilization
To overcome the limitations of 2D motion model based
video stabilization, some researchers have been proposed 3D
motion model based video stabilization [7-9]. And 3D video
stabilization papers that show very surprising experiment
results have been recently published in computer vision
community. Brandon M. et al. used light field camera array to
get accurate dense depth map images and directly computed a
sequence of relative poses between virtual (or stabile) camera
and the camera array employing a space-time optimization
[7]. Feng Liu et al. used the fact that small shifts in viewpoint
can be faked by a carefully constructed content-preserving
warp even though the result is not physically accurate [8].
Guofeng Z. et al used sparse reconstructed 3D data instead of
recovering dense depth map to make optimization processing
efficient and approximate geometry representation and
analyzed the resulting warping errors [9].
However, unfortunately, so far 3D video stabilization
approaches in published papers cannot be processed in
real-time because they use an optimization method that is
batch process in off-line.
D. Robotic eyes
There have been several researches on the design and
implementation of robotic eyes. For example, Alexander L. et
al. adopted pneumatic actuators [19], Thomas V. et al. used
piezo actuators [20], and Giorgio C. et al. applied tendon
driven actuators [21-22] to control the eye movements. But
they focused on only hardware stabilization or compensation
rather than software video stabilization. Hardware
stabilization cannot clearly remove unwanted motion due to
inaccurate sensing and actuating, and a hardware delay from a
sensor to an actuator.

III. 3D VIDEO STABILIZATION FOR HUMANOID EYES
In this section, we describe procedure of the proposed 3D
video stabilization for humanoid eyes shown in figure 1. Our
procedure follows the framework of general 2D motion
model based video stabilization: motion estimation, motion
filtering, and image composition. In motion estimation step,
stereo cameras are used to estimate 3D camera motions (three
translational and three rotational motions). In motion filtering
step, Kalman filter is adopted to filter out unwanted motion in
real-time. In image composition step, we propose a
spatially-varying warping
reconstructed 3D data and optimal constant depth based
pre-warping technique to fill pixels in Delauray Triangulation
(DT)-undefined region. Furthermore, we integrate the
proposed 3D video stabilization into a hardware stabilization
device for robotic eyes in order to compensate large and rapid
motions that the only software video stabilization cannot
cover.
technique using sparse
A. 3D camera motion estimation
In order to estimate the camera (or humanoid eyes) motions
that are three translations and three rotations, we adopt a
general motion estimation technique based on 3D object
points and their 2D projections as the following equation [23-
24].

=
∑
R t
S

n
n-1
R
and
between (n-1)th camera and nth camera, respectively.
are the kth 3D object point reconstructed at the
(n-1)th image and the kth corresponding 2D projection at
(n-1)th image, respectively. K is a camera intrinsic matrix.
[]
feature
2
n
n-1
n
n-1n,kn-1,k
|
k
| argmin-|
∈
Rtx K Rt X (1)
n
n-1
t
are three translations and three rotations
n-1,k
X

and
n,k
x

Page 3

Stereo correspondence points between stereo images and
motion correspondence points between inter-frames are
found by using a Zero mean Normalized Cross-Correlation
(ZNCC) matching and KLT tracker [25], respectively. In
order to robustly estimate 3D motion, the RANSAC
algorithm [26] is adopted to remove the outliers among the
corresponding points set.
B. 3D camera motion filtering
In this step, the unwanted high frequency motions can be
separated from the original (or unstable) 3D camera motions
(three translational and three rotational motions estimated in
motion estimation step) by low-pass filters, smoothing, or
fitting methods. We adopt Kalman filter as a low-pass filter in
order to filter out the six unwanted motion parameters in
real-time [27]. Correction motion parameters between an
unstable camera motion and a stabilized camera (or a virtual
camera) motion shown in figure 2 can be calculated as
follows.

1,
cuf
=
R R R
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎡
⎜
⎢
⎜
⎥⎢
⎜
⎥⎢
−
⎜
⎣
⎜
⎜
⎡⎤
=⎣⎦
⎝
cucf
−
=−
tt R t (2)
,
cos(
sin(
)
)
sin(
cos(
)0
0
1
cos()0
1
0
sin()1
0
0
00
)00 cos(
sin(
)
)
sin(
cos(
)
00sin()cos())
cos(
sin(
)
)
sin(
cos(
)0
0
1
)
00
zzyy
uzzxx
yyxx
T
uxyz
zz
fzz
where
ttt
θ
θ
θ
θ
θθ
θ
θ
θ
θθθ
θ
θ
θ
θ
⎡
⎢
⎢
⎢
⎣
⎤
⎥
⎥
⎥
⎦
−
⎡
⎢
⎢
⎢
⎣
⎤
⎥
⎥
⎥
⎦
⎡
⎢
⎢
⎢
⎣
⎤
⎥
⎥
⎥
⎦
=−
−
⎡⎤
⎦
=⎣
⎡
⎢
⎤
⎥
−
=⎢
⎢⎣
⎦
R
t
R
cos()0
1
0
sin()1
0
0
00
00 cos(
sin(
)
)
sin(
cos(
)
sin( ) cos())
yy
xx
yyxx
T
fxyz
ttt
θθ
θ
θ
θ
θθθ
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎠
⎤⎡
⎥⎢
⎥⎢
⎥⎢
⎦
⎤
⎥
⎥
⎥
⎦
−
⎣
t

Here,
c
R and
ct are rotational motion and translational
correction motion, respectively. And ,
are filtered motion parameters of the original 3D camera
motions which are ,, ,,
xyzx
t t t θ θ and
, ,t θ θ and
,,
xyzxy
t t
z θ
,
y
z θ , respectively.
C. 2D image composition
In 2D motion model based video stabilization, a stabilized
image can be directly and easily generated by warping (or
transforming) an unstable
estimated-filtered motion parameters. But generating
stabilized image is much more difficult issue in 3D video
stabilization because there is no direct connection between
2D image motion and 3D camera motion. There are two
approaches for 2D image composition (or view warping) in
3D video stabilization. One approach is dense-depth-map
based image composition [7]. Brandon M. et al. used light
field camera array (five cameras) to get accurate dense depth
map images. However, Getting an accurate dense depth map
is also challenging issue and time-consuming in computation.
Therefore, it is impossible to apply this approach for
humanoid robot eyes. The
sparse-reconstructed 3D data based image composition [8-9].
Although this method cannot generate accurate stabilized
image like the first approach, but this is much faster than the
first one.
In this paper, we adopt the latter approach and propose a
novel warping technique that uses sparse reconstructed 3D
data in order for fast processing, and we do not use
optimization method that processes in off-line. Guofeng Z. et
al proposed two 2D image composition (or view warping)
methods that are based on 1) slant planar imposter and 2)
optimal constant depth. These two methods showed good
performance in a variety of environment [9]. But their
methods might show worse results under the scene with
various depths because they assume that all 3D points are on
the one plane: slant plane or constant depth. In order to
improve their method, we propose a novel spatially-varying
warping technique that uses sparse-reconstructed 3D data and
image through the
other approach is

Fig. 1. Flowchart of the proposed 3D video stabilization for humanoid
robot eyes

Fig. 2. Correction motion,
and virtual stabilized camera
c
R and
ct , between real unstable camera

Page 4

DT.
1) Spatially-varying warping technique
We now start a detailed description about our proposed
warping method. A corresponding point (or pixel) in a
stabilized image to point (or pixel) in an unstable image can
be calculated as a following equation.


j
⇔
x
(
jc
xK R K x

Here,
su
j
x
)
1
/
su
jcj
Z
−
=+
t
(3)
u
jx and
s
jx mean the jth unstable and stable point (or
pixel coordinates), respectively.
(2) in motion filtering step.
j
Z is a z-coordinate (or depth) of
X corresponding to
c
R and
ct are from equation
j
u
jx and is calculated in motion
estimation step. As a result, the sets of
are the sparse-reconstructed 3D data, where J is the number of
reconstructed 3D feature points.
After getting correspondence pairs,
unstable and stabilized images, DT is created from the point
sets,
u
jx and
j
Z , j=1,2,…,J,
s
j
u
j
⇔
xx , between
s
jx , j=1,2,..,J, shown in figure 3 [28]. We can find the
corresponding triangulation of the point sets,
because we already knew the correspondence pairs between
unstable and stabilized image through equation (3). And then,
the kth similarity transformation,
triangles in unstable and stabilized images shown in figure 3.
This process is repeated to all triangle pairs in order to get
similarity transformation for pixels in every triangle. Finally,
we can warp unstable input image into stabilized image
through spatially-varying similarity transformation set.
2) Pre-warping for DT-undefined region
There are always pixels (or image points) being not inside
any triangles (shown in figure 3) because we do not use a
dense depth map image but sparse-reconstructed 3D data that
are sets of
j
Z , j=1,2,...,J. These image points are
called points in DT-undefined region and do not have a
similarity transformation there. In order to warp pixels in this
undefined region, F. Liu et al. used infinite homographies and
u
jx , j=1,2,..,J,
k T , is calculated between
u
jx and
general homographies [8]. Their final result with the latter
method was better than the former. But there might be a little
image distortion because general homographies do not use
3D information. In this paper, the optimal constant depth
based pre-warping technique is adopted in order to warp
pixels in the DT-undefined region as equation (4) [9].

(
ic
xK R K x
(
)
)
1
.
/
su
icCont
Z
−
=+
t
(4)
)

(
1
11
.minmax
2
Cont
where ZZZ
−
−−
=+

Here,
DT-undefined region of the stabilized and unstable image,
respectively, shown in figure 3.
minimum and maximum depth (or z-coordinate) among
sparse-reconstructed 3D data, respectively.
s
ix and
u
ix are the ith point (or pixel coordinates) in the
min
Z
and
max
Z
mean the
.
Cont
Z
is a
harmonic mean of
min
Z
and
max
Z
.
D. Hardware of robotic eyes inspired by human VOR
When a humanoid robot walks or runs, a camera on its head
sometimes moves, largely and rapidly. The only vision based
video stabilization sometimes fails to stabilize a video which
has large and rapid camera motions due to a motion blur and
computation time in feature tracking. In order to remove these
unwanted motions, we adopt a hardware stabilization device
for humanoid robot eyes inspired by human VOR shown in
figure 4. VOR is a reflex eye movement that stabilizes eyes
on the in the direction opposite of the head movement in order
to preserve the image on the center of the visual field [27]. For
example, when the head moves to the left, the eyes move to
the right, and vice versa. And this is one of the fastest reflex in
human.
To follow a human VOR function, an IMU is used to sense
three rotational motions (roll, pitch, and yaw) like human
ears and three DC motors are used as actuators to rotate stereo
cameras like human occular muscles shown in figure 4. And
An unstable image A stabilized image

The kth triangle DT-undefined region

Fig. 3. Concept of spatially-varying warping based on similarity
transformation with Delaunay Triangulation


Fig. 4. A hardware stabilization device for humanoid robot eyes

Page 5

Kalman filter is adopted to find out unwanted three rotational
motions from the original three rotational motions which are
sensed by the IMU. And the only unwanted three rotational
motions are compensted by three actuators .

IV. EXPERIMENTAL RESULTS
To assess the performance of video stabilization, several
measures were used, for
Transformation Fidelity (ITF). But, “comparison to other
stabilized video” is widely regarded as the best assessment
method for video stabilization, recently. As a result, the
assessment is sometimes difficult and somewhat subjective.
We now discuss the results of our method on a variety of
conditions, as well as comparisons to other 3D video
stabilization approaches (view warping with slant planar
imposter or optimal constant depth) proposed by Guofeng Z.
et al [9] (as in the supplementary material). It was very
difficult to find better things between our results and theirs.
But we found that our result was a little better when there
were significant depth variations in the scene because they
assumed that every 3D point be on a plane such as slant plane
or constant depth.
The results of 3D video stabilization based on a
spatially-varying warping with pre-warping are shown in
figure 5. A stabilized image by the 3D video stabilization
based on spatially-varying warping without pre-warping has a
example, Inter-frame
wide of undefined regions (black region) because there is no
similarity transformation in these regions shown in figure
5(b). In order to solve this problem, the proposed 3D video
stabilization adopted an optimal constant depth based
pre-warping technique. The final result is shown in figure
5(c). In this paper, the unavoidable unfilled region induced by
image warping was not cared for. We simply trimmed the
edge of stabilized image to hide an unfilled region that
irritates users, shown in figure 5(c). There have been
researches on reducing unfilled region [2, 6].
The results of the proposed 3D stabilization for humanoid
eyes are shown in figure 6. We equipped another extra camera
on the hardware stabilization device to show how the image
(or camera) motion is rapid and large. An unstable image
from the extra camera without any hardware and software
stabilization is shown figure 6(a). We can see a large image
motion, and a blurred image due to rapid motion. We
discovered that the only vision based video stabilization
cannot cover such rapid and large motions. A stabilized
image by the only hardware stabilization system for
humanoid eyes without software video stabilization is shown
in figure 6(b). There were still unwanted motions due to the
delay of hardware from IMU sensor to actuators. A final
stabilized image by both a hardware stabilization device for
humanoid eyes and a software 3D stabilization based on
spatially-varying warping with pre-warping is shown in
figure 6(c).

(a) (b) (c)
Fig. 5. Results of 3D video stabilization based on spatially-varying warping with pre-warping. (a) is an unstable input image. (b) is a stabilized image by
the 3D video stabilization based on spatially-varying warping without pre-warping.(c) is the stabilized image by the proposed 3D video stabilization based
on spatially-varying warping with pre-warping.


(a) (b) (c)
Fig. 6. Results of the proposed 3D video stabilization for humanoid eyes. (a) is an unstable image without any hardware and software stabilization. (b) is
a stabilized image by the only hardware stabilization system for humanoid eyes without software video stabilization. (c) is a final stabilized image by both
a hardware stabilization system for humanoid eyes and a software 3D stabilization based on spatially-varying warping with pre-warping.