Page 1

Point Pattern Matching Based Visual Global

Localization using Ceiling Lights

Hyun Chul Roh, Chang Hun Sung, Min Tae Kang, and Myung Jin Chung

Robotics Program,

Korea Advanced Institute of Science and Technology (KAIST),

Gwahangno, Yuseong-gu, Daejeon, 305-701, Republic of Korea,

{rohs, sungch, kmt876}@rr.kaist.ac.kr, mjchung@ee.kaist.ac.kr

Abstract— This paper proposes a new system for point pattern

matching based visual global localization using spot lightings in

ceiling. The proposed algorithm is suitable for system that

demands high localization accuracy and fast update rate such as

a service guide robot in the exhibition or fair. A single camera

looking upward direction is mounted on the head of the mobile

robot, and spot lighting positions are detected by thresholding

and identification process. For detecting more spot lightings, we

choose wide FOV lens, and inevitably there is serious image

distortion. But by applying correction calculation only for the

positions of spot lightings not whole image pixels, we can decrease

the processing time. And then using point pattern matching and

least square estimation, finally we can get the precise position and

orientation of the mobile

demonstrate the accuracy and update rate of the proposed

algorithm in real environments.

robot. Experimental results

Keywords-localization; ceiling lights; pattern matching; indoor

mobile robot; thresholding; identification

I.

INTRODUCTION

To navigate a mobile robot autonomously, some

fundamental techniques are required. These are Localization,

Path Planning, Path Following and Collision Avoidance.

Among these techniques, Localization is the most important

and basic technique. Determining robot’s absolute position and

orientation is actually called Robot Localization [2].

But this localization of a mobile robot is comparatively a

difficult task. If a mobile robot doesn’t know its position and

orientation, it cannot process a path planning and even

autonomous navigation. For this localization technique, many

kind of sensors which are Light Detection And Ranging

(LIDAR), Vision Sensor, Ultrasonic Wave Sensor and RF-

based sensor are used [2]. LIDAR has been applied for very

wide range of mobile robot localization in indoor [3] and

outdoor [4] environment. Long distance, fast updates rate, wide

angular range and good accuracy, such as it has many

advantages. However, due to the high cost, heavy weight, size

and safety problem, this sensor has limitations in practical

perspective.

Recently, in order to overcome the limitations of using

LIDAR sensor, many researches have been progressed to use

vision sensor for localization. Furthermore, this vision sensor

has a lot of strengths such as small size, light, fast update rate

and relatively low cost. Also, the proportion of the indoor

mobile robot is larger than the outdoor one and every indoor

environment, there are ceiling lights. Because of these reasons,

much of work which are based on ceiling vision researched for

several years.

Aider et al. [5] proposed pattern tracking based localization

method using natural landmarks. In this research, the robot was

able to localize itself and to reproduce learned trajectories

defined by a set of ceiling images representing the visual

memory.

Jeong et al. [6] suggested a fast and robust CV-SLAM

(Ceiling Vision – based Simultaneous Localization and

Mapping) technique using a single ceiling vision sensor. This

method solved the rotation and affine transform problems using

3D gradient orientation estimation method and multi-view

description of landmarks.

Nguyen et al. [7] introduced a method of self-localization

by constructing a natural ceiling landmark system based on the

invariance of ceiling features on sequence images. And the

mobile robot was composed of a compact mainboard and a

fish-eye camera.

Based on the natural features on the ceiling, a visual

positioning technique was proposed by Xu et al. [8]. The initial

orientation and position of the mobile robot in the world frame

was estimated with a specified block on the ceiling via

perspective-n-point-based positioning method.

In this paper, we propose point pattern matching based

accurate and fast visual global localization system using ceiling

lights in indoor environment that is non-repetitive pattern of

spot lights in the ceiling. Our main contribution in localization

doesn’t needed prior information or data and can calculate

robot’s position and orientation from only a current frame.

Proceedings of 2011 8th Asian Control Conference (ASCC)

Kaohsiung, Taiwan, May 15-18, 2011

MoA8.4

- 281 -

Page 2

Figure 1. Flowchart of the proposed localization using ceiling lights

The rest of this paper is organized as follows. Section II

describes the spot lighting detection procedure which is

constituted image thresholding and spot lighting identification.

In section III, the proposed point pattern matching based

localization is presented, while experimental environment and

results are analyzed in section IV. And conclusions and further

work are drawn in section V.

II.

SPOT LIGHTING DETECTION

In this section, we describe the procedure of image

thresholding and robust identification method [9] for spot

lighting detection shown in the front of figure 1.

A. Thresholding

The segmentation problem is transformed into a simple

thresholding one on the images. The threshold is determined

based on the distribution of the images of the background

pixels.

Let I be a window image of size W

pixel located at ( , )

x y with the intensity value

data pixel, will be denoted by (( , )

1,,

yH

? ?

. The data set is then defined to be a set of data

pixels as

H

?

pixels. Each

z

, called a

1,

x

? ?

xy

x y ;

xy

z ) for

,

W

,

??

??

, ;

x y z

1,,,1,,

xy

xW yH

? ???

??

(1)

The data set

such that the rectangle of the candidate spot is center position

of the highest intensity pixel. The residual of the xy th data

( )( , )

f x y , denoted by

difference between the original and the estimated intensity of

the xyth data pixel given by

( )( , )

x y is extracted from a small region

s

f

pixel with respect to

s

xy r

, is the

( )( , )

x y

s

xyxy

rzf

??

(2)

Let ? be the residual mean, ? be the standard deviation of

the residuals of the background pixels and the background

region

B

? , given by

*

xy

( , ;

x y z

)

1

xyB

B

r

?

??

?

?

?

(3)

2

2*

xy

( , ;

x y z

)

1

()

xyB

B

r

??

??

??

?

?

(4)

Where

thresholdT , according to Niblack's method [10], the image is

then segmented into a binary image so that the candidate spot

lighting region is to be white with value one and the

background region to be black with value zero as follows:

B

?

is the area size of

B

?

. For a given

*

xy

*

xy

1,/

( , )

Z x y

1,/

rT

rT

? ?

? ?

?

?

??

??

??

??

(5)

B. Visual Perception Based Identification

In order to identify the real region-spot lightings from the

candidates found in the previous part, the properties of the

human visual perception have to be considered. In this part, we

first present human perception model and, based on it,

formulate a measurement index level. The final identification

procedure is then followed.

(undistorted)

Thresholding?

Spot Lighting Identification?

Undistortion Lighting

Position?

Point Pattern Matching?

Final Localization?

Binary Image?

d

Q

(distorted)?

Q

PQ

?

Matched Pair ?

Robot Pose ?[ , , ]

x y ?

Image Grab?

Bitmap array?

- 282 -

Page 3

Figure 2. Result of the proposed thresholding

1) Visual Perception Model

The fovea is always focused on the object of interest by the

accommodation ability of the eye. The typical simulation of the

retina consists of object region, object-background region, and

surround-background region with different luminance stimuli

[11]. The regions are arranged in the observation field as

concentric ones, with the object in the middle followed by the

object-background and the surround-background.

The object-background is the close neighbor-hood of the

object that has a strong influence on the perception of the

object. The luminance stimulus of each region can be

simplified to the mean of the gray intensity of the

corresponding region in the image and will be denoted by

oI for the object and

We have the following measurement index level:

bI for the object-background, respectively.

0.33

1.97/ 0.72

o

A

b

?

I

II

L

?

?

(6)

Where A is the area of a spot lighting.

2) Identification

For each candidate region-spot lighting, we first locate a

WH

?

window such that the minimal bounding rectangle of

the candidate is centered within the window. Let I? be the

located window image. Next, by approximating the image

surface of I? except the pixels belonging to some candidate

region- spot lighting detected. The image I? is then subtracted

from

f

( ) s

, giving a residual imageR . The object-background

region is obtained by dilating the candidate spot lighting region

with

/ 2 / 2

wh

??

?

structuring element [12] and then by

excluding the candidate spot lighting region from the dilation

result, where w? is the width of the minimal bounding

rectangle and h? is the height of the rectangle. Finally, we

compute the level of the candidate region- spot lighting from

Eq. (6) using

oI ,

bI and A given by

()

1

( )

R p

o

o

p

o

I

??

?

?

?

(7)

()

1

( )

R p

b

b

p

b

I

??

?

?

?

(8)

o

A ? ?

(9)

Figure 3. Position of the spot lighting identification,

d

Q

Identify applicable sponsor/s here. (sponsors)

- 283 -

Page 4

III. POINT PATTERN MATCHING AND LOCALIZATION

In order to find the mobile robot’s position and orientation

from the identified spot lightings in the previous section,

undistortion and point pattern matching [13] have to be

processed. In this section, we first describe fast undistortion

method and the final localization for point pattern matching

procedure is then followed.

1) Spot Lighting Point Undistortion

For the localization using ceiling vision, the more features

extracted, the better accuracy going to get. And for more

features, a specific lens which has wide field of view has to be

considered. But if you choose this wide angle lens, it is

inevitable there must be a serious distortion in the image. For

this reason, image correction process is essential.

Assume that there’s no distortion in the lens,

[ , , ]T

X Y Z

?

in the 3D space is mapped to normalized

coordinate

_

n u

X

P

by central projection. And this given by

x

X Z

y

Y Z

?

??

_

_

_

/

/

n u

n u

n u

?

?

?

?

?

?

?

?

?

??

X

(10)

_

n u

X

is distorted by the nonlinear(radial & tangential)

properties of the lens. And distorted pixel,

_

n d

X

is given by

246

_

123

__

246

_

12

?

?

3

22

n u

x

y

4__5_

22

n u

5__4_

1

1

2

2

(

(

2

2

)

)

n d

x

y

n dn u

n d

k x

k x

n un u

n un u

k r

k r

?

?

k r

k r

k r

k r

y

y

k r

k r

??

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

? ??

?

?

?

?

?

??

?

?

XX

(11)

Where

22

n u

x

2

n u

__

ry

??

. And Eq. (11) is transformed to

pixel-coordinate of

_

p d

X

by camera as follows:

__

___

0

01011

p dxxxn d

x

y

p d p dyy n d

x

y

ffc

cf

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

??

??

??

??

??

?

?

?

?

?

??

X

(12)

Where

xf and

are principle point.

Using above mentioned distortion model, we can easily get

the corrected image. But calculating the whole image pixels

takes a lot of time. In the previous section, we already get the

yf

are focal length, ? is skew coefficient,

xc and

yc

positions of spot lightings,

just get the undistorted positions of spot lighting, Q , don’t

have to calculate the whole pixels. And the number of these

positions is much smaller than the number of whole image

pixels. Then we can have some advantages in terms of the

processing time.

d

Q (subscript d: distorted). If we

2) Point Pattern Matching

Two points patterns P and Q consists of m feature

points extracted from the target model map, where P ?

12

,,},{

m

p pp

?

extracted from the spot lightings in the previous section, where

,,{},

m

q qQq

?

?

, and Q consists of n feature points

12

.

Each point is described with the coordinates (x,y)Tthat is

{(,) |1,, }

ii

A matching is to find a correspondence between a point

P and a certain point

T

pp

Pxyim

?? ?

and

{(,) |1, , }

n

aa

T

qq

Qxya

?? ?

.

ip and

aq inQ; that makes this corresponding

G

pair consistent under a registration

( , , , )

xy

t t s

?. The registration

( , , , )

xy

parameters:

t t s

G

?

is an affine transformation composed of four

t t s ? , where s is a scaling factor, ? the

,

xy

t t

, x y directions, respectively. Let

registration maps straight lines into straight lines and maps a

triangle into a similar triangle. It is a global transformation

since the overall Fast algorithm for point pattern matching

geometric relationships between points do not change. It maps

a point

(,)T

pp

xpy

?

, , ,

yx

rotation angle, and

the translations along the

( , , , )

xy

rt t s ??

. The

to a point

(,)T

qq

xqy

?

as follows

? ?

cos

sin

sin

cos

qxp

r

qyp

x

y

t

t

x

y

q G P

s

s

s

s

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

????

?

(13)

Figure 4. Matching between Model and Target Point Pattern

- 284 -

Page 5

Figure 5. Proposed Undistortion method and matching result

3) Localization

In the previous part, matched pair P

Using camera intrinsic matrix K and height of the ceiling,

positions of spot lights Qin pixel-coordinate are reconstructed

to

real

Q

Q

?

is calculated.

in real-coordinate as follows:

1

{| 1,, }

n

reala

Q K q Z a

?

??? ?

(14)

By Least Square Estimation Method [14] from

constructed by real 3-D coordinates, we can get final

localization results which are position (

orientation

robot

?

real

P

to

real

Q

,)

robot

x

robot

y

and

.

We want to find the transformation parameters

( , , ,

xy

t t s ? ) given the minimum value of the sum of squared

coordinates differences ( , , , )

xy

S t t s ? of these corresponding

points.

1

i

?

difference between

real

P

( , , , )

xy

G t t s ?

real

Q

cos

sin

y

t

ss

?

?

??

?

?

?

? ??

?

??

???

???

?

, , , )

y

s t t s

(

k

T

ixi

e e

???

, where

and the transformed point

ie is the coordinates

sin

cos

cos

sn

i

1

1

0

0

ii

ii

ii

ii

pq

x

i

pq

x

aa

y

pii

aa

x

y

x

y

t

e

t

t

x

y

y

c rq

ss

xs

s

??

?

?

?

?

?

?

? ?

?

? ?

? ?

? ?

?

?

?

?

?

?

? ?

?

? ?

?

??

?

?

?

?

?

?

?

?

??

?

?

(15)

IV. EXPERIMENTAL RESULTS

The proposed algorithm was applied to a 1.5m height

service robot. We used a single NTSC 1/3 inch CMOS Gray

scale camera, and FOV of lens is 120 degree, and 2GHz CPU

Laptop. 640? 480 resolution for input image was used and

thresholding sub-image size was set to be 64?64. We have

tested our system in the specific place where 1st floor of T-city

building in Incheon, Korea. The test was carried out in two

steps. The first test was for the static (stopped) condition

measurement where already known position and rotate by

90degree to analyze position, orientation error. The second test

was loop closing that follows rectangle-shaped path with

approx.0.15m/s speed and corner points of rectangle are

already known.

All experiment was accomplished in real time and robot

pose update rate was approximately 10~12frame/sec that

means whole localization processing time from grab image is

less than 100ms with maximum 11 spot lighting target model.

Error

()

X mm

50.487

()

Y mm

60.732

( )

? ?

0.618

Table 1. Average static error of our Algorithm

- 285 -