Low-cost implementation of bird's-eye view system for camera-on-vehicle

Abstract

Many papers concentrate on the 3*3 perspective transformation matrix calculation of bird's view system, but few discussed for the whole system implementation. In this paper, a low-cost bird's-eye view system is proposed, which adopts an elaborate software/hardware cooperative system structure. It can be applied to all kinds of vision-based system in vehicles directly as a ready-made module.

Full text

7.1-4
Abstract--Many papers concentrate on the 3*3 perspective
transformation matrix calculation of bird’s view system, but few
discussed for the whole system implementation. In this paper, a
low-cost bird’s-eye view system is proposed, which adopts an
elaborate software/hardware cooperative system structure. It can
be applied to all kinds of vision-based system in vehicles directly
as a ready-made module.
I. INTRODUCTION
In general, camera-on-vehicle has serious perspective effect
as Fig. 1(a). Because of the perspective effect, the driver can’t
feel distance correctly and advanced image processing or
analysis becomes difficult also. Consequently, as Fig. 1(b),
perspective transformation is necessary. The raw images have
to be transferred into bird’s-eye view as Fig. 1(c).
The bird’s-eye view image is the perspective projection of
original image as in Fig. 1. The relationship between pixel (x,
y) of bird’s-eye view image and pixel (u, v) of original image
can be formulated by a 3*3 homography matrix as in
11 12 13
21 22 23
31 32 33
'
'
x aaau
yaaav
waaaw
⎡⎤⎡ ⎤⎡⎤
⎢⎥⎢ ⎥⎢⎥
=
⎢⎥⎢ ⎥⎢⎥
⎢⎥⎢ ⎥⎢⎥
′
⎣⎦⎣ ⎦⎣⎦
(1)
where x=x '/w ' and y=y '/w ', as in [1].
Many researches have been done to calculate the
transformation matrix. For example, [2] used camera height,
focal length and mounting angle, and [3] used vanishing point
of lanes to calculate it. [4] proposed a vision-based method
which used 3 patterns to get the matrix.
However, all previous papers only discussed for the
transformation matrix. Actually, even though we got the
matrix, a real-time implemented by hardware is also needed.
So this paper addresses the implementation of bird’s-eye view
II. PROPOSED SYSTEM
As Fig. 2, the proposed system includes 3 modules. Module
I calculates transformation matrix by software. Module II
generates the look-up tables (LUTs) of address mapping as an
This work was supported by Hynix Semiconductor, and also
sponsored by Seoul R&BD Program (10560) and ETRI SoC Industry
Development Center, Human Resource Development Project for IT
SoC Architect. The IDEC provide research facilities for this study.
initialization of system. Module III transfers input original
images into bird’s-eye view images by hardware.
A. Perspective transformation matrix calculation
Because the calculation of transformation matrix is not the
emphasis of this paper, we adopt the algorithm of [4] to get the
matrix directly. And because the matrix only needs to be
calculated once at the beginning of system initialization,
module I is implemented by software, but not by hardware.
B. Initialization
After getting the transformation matrix, straightforward
approach of the bird’s-eye view system includes 3 steps of
perspective mapping, shearing and enlarging, an in Fig. 3.
To reduce complexity of implementation, we proposed the
3 optimized algorithms based on the straightforward approach.
(1) Real number multiplication and division are substituted
by using LUT when calculates coordinates by (1). So, as in
Fig.1, forward and backward mapping LUTs are used.
(2) Three steps are combined into one-step transformation,
as shown in Fig. 4. Suppose that the inverse perspective
transformation matrix, shearing matrix and enlarging matrix
are H1, H2 and H3, respectively, as formula (2) shows,
H1=
11 12 13
21 22 23
31 32 33
aaa
aaa
aaa
⎡
⎤
⎢
⎥
⎢
⎥
⎢
⎥
⎣
⎦
H2=
1
2
10
10
001
s
s
⎡
⎤
⎢
⎥
⎢
⎥
⎢
⎥
⎣
⎦
H3=
1
2
00
00
001
e
e
⎡⎤
⎢⎥
⎢⎥
⎢⎥
⎣⎦
(2)
then combined matrix Hcom can be expressed as follows:
Low-cost Implementation of Bird’s-eye View System for
Camera-on-vehicle
Lin-Bo Luo, In-Sung Koh, Kyeong-Yuk Min, Jun Wang, Jong-Wha Chong, Member, IEEE
Shearing
Perspective
mapping Enlarging
(a)Input image;(b)Bird’s-eye view image;(c)Sheared image;(d)Output image
Fig. 3. Straightforward transformation procedure
Y
N
I. Perspective transformation
matrix calculation
Bilinea
r
interpolation
Bird’s-eye view image
Forward mapping
LUT
Backward mapping
LUT
II. Initialization
Ring
buffer
LUT generator
Initialize?
Control
-Counter
-LUT access
-Address generation
Bilinea
r
interpolation
…
FIFO
Image data
Memory buses
Address & Internal data
Input image
III. Transformation
ASIC/FPGA
Fig. 2. Structure of proposed bird’s-eye view system
G
round
p
lane
(c)Bird’s-eye
view image
(u,
v
) (x,y)
(a) Perspective
view image
Bird’s-eye view
(Virtual view)
Perspective
view
Ө
h
(
b
Perspective trans
f
ormatio
n
Fig. 1. Illustration of perspective transformation in a parking lot scene
978-1-4244-4316-1/10/$25.00 ©2010 IEEE

Hcom = H1 H2 H3=
1 11 2 12 2 1 11 12 13
1 21 2 22 2 1 21 22 23
1 31 2 32 2 1 31 32 33
()()
()()
()()
ea Sa e Sa a a
ea Sa e Sa a a
ea Sa e Sa a a
++
⎡⎤
⎢⎥
++
⎢⎥
⎢⎥
++
⎣⎦
(3)
The combination can be done by software and it is
calculated once. This scheme can save hardware cost largely.
(3) Picture size needed for perspective transformation
should be calculated in advance. Suppose image size is m*n,
then the one-step of Fig.4 only needs the last (m-1-v0) rows of
image in Fig. 4 (a). If we pre-evaluate the value of v0, the first
v0 rows of input image should not be saved and processed, and
so memory size and computational cost can be saved.
C. Pipelined hardware structure
As in the module III of Fig. 2, the proposed pipelined
hardware structure includes 4 blocks of ring buffer, control,
interpolation and FIFO.
The ring buffer buffers input pixels to get pixels of a 2*2
widows at a time for bilinear interpolation.
Control unit consists of counter, LUT access and address
generation. Counter counts input pixels. Control unit starts to
work when input reaches the pixel (u0,v0) of Fig. 4(a).
LUT access adopts window operation as in [5]. The 2*1
forward mapping (FWD) and n*1 backward mapping (BWD)
are used here, as shown in Fig. 5.
Fox example, along with the raster-scanning order of input
image, the steps of LUT accessing are the following: ①
finding the forward mapping of inputs (ui, vi) and (ui+1,vi) by
accessing FWD LUT, that is, (xi, yi) and (xi+1, yi) in output
coordinates system as in Fig. 5(b), ② selecting the points
which x values are between xi with xi+1 and y values are equal
to roof(yi), i.e., (xi′, yi′),…,( xi+n-1′, yi′) in the blue dashed
rectangle of Fig. 5(b), ③ finding their backward mapping
(ui′,vi′),…,( ui+n-1′,vi′) in the coordinates system of input image
by accessing BWD LUT as in Fig. 5(a).
After backward mapping, the points (ui′, vi′),…,( ui+n-1′, vi′)
are usually located space area among the real input pixels as in
Fig. 5(a), so bilinear interpolations are used to improve output
image quality as in Fig. 6. For interpolation, the coordinates of
backward mapping (ui′, vi′) are divided into two parts of
integer and offset. Integer part is used to determine the
location of 2*2 window for bilinear interpolation, whereas the
offset is used to be the weight of bilinear interpolation. And to
reduce required system frequency, n interpolations is operated
in parallel as in Fig. 2.
Finally, a FIFO is used to transfer parallel data to sequential
again and synchronize output pixels with input frequency.
III. EXPERIMENT
Our experiment conditions are as follows: CMOS rear
camera with 120o horizontal angle, camera height h=95cm and
mounting angle Ө=45o; display screen is 640*480 pixels.
Thanks to the algorithms described in the above, the memory
size reduced and computational cost saved as compared with
the straightforward algorithm is showed in Table 1.
TABLE 1. COST SAVING BY THE PROPOSED ALGORITHMS
Proposed
al
g
orithms LUT One-step
transformatio
n
Pre-evaluation
Memory or cost
reduction
Real number
multiplication
and division free
Memory size ca.67%↓,
computational cost
ca. 67%↓
Memory size
ca. 40%↓
IV. CONCLUSION
This paper proposed a real-time bird’s-eye view system,
which performs the perspective transformation by a low-cost
pipelined hardware structure. The module can be used in
automobile parking assist system directly and be applied to
other vision-based automobile system.
REFERENCES
[1] G. Wolberg, Digital Image Warping, IEEE computer society press: Los
Alamitos, 1990, pp. 52-56
[2] M. Bertozzi and A. Broggi, “GOLD: A parallel real-time stereo vision
system for generic obstacle and lane detection,” IEEE Trans. Image
Processing, vol. 7, pp. 62-81, 1998.
[3] C. Zhaoxue and S. Pengfei, “Efficient method for camera calibration in
traffic scenes,” Electron. Lett., vol. 40, pp. 368–369, 2004.
[4] H. Kano, et al,“Precise top view image generation without global metric
information,” IEICE Trans. Inf. & Syst., vol. E91-D, pp. 1893-1898, 2008.
[5] S.Oh and G.Kim, “FP GA-based fast image warping with data-
parallelization schemes,” IEEE Trans. On Consumer Electron., vol. 54,
pp. 2053-2059, Nov. 2008
Perspective
transformatio
(u
0 ,v0
)
(a) Original image; (b) Bird’s-eye view image
Fig. 4. Combined one-step perspective transformation
(b)Coordinates system of
bird’s-eye view image
(a) Coordinates system
of input image
Location of
input pixel
Forward mapping
of input pixel
Location of
output pixel
Backward mapping
of output pixel
v Backward mapping
Forward
mapping
u
(xi+n-1′,yi′)
(xi′,yi′)
(ui+n-1′,vi′)
(xi,yi) (xi+1,yi)
x
(ui′,vi′)
(ui+1,vi)
(ui,vi)
y
Fig. 5. 2*1 forward mapping and n*1 backward mapping of LUT access
Input
coordinates
(ui,vi) ui′
vi′
Floor(ui
′
)
Floor(vi
′
)
Offset ai
(ui′-floor(ui′)) Mult. Add
Mult. Add
Mult.
Subtract
Add FIFO
Output pixels
Vi,j-1
Vi-1,j-1
Vi-1,j
Vi,j
Subtract
Subtract
Counter
L
ine
buffer
(m+2)
Dela
y
Dela
y
Input image
Bilinear
interpolation
Offset
b
i
(vi′-floor(vi′))
Image data
Memory buses
Address & Internal data
FWM
LUT
BWM
LUT
LUT access
2*1 n*1
FWM BWM
Fig. 6. Proposed hardware structure of LUT access and interpolation