Red-eye detection and correction using inpainting in digital photographs

Abstract

When we take pictures with flash, red-eye effect often appears in photographs. Flash light passing through pupil is reflected on the blood vessels, and arrives at a camera lens. This phenomenon makes red-eyes in photographs. Several algorithms have been proposed for removal of red-eyes in digital photographs. This paper proposes a red-eye removal algorithm using inpainting and eye-metric information, which is largely composed of two parts: red-eye detection and red-eye correction. For red-eye detection, face regions are detected first. Next, red-eye regions are segmented in the face regions using multi-cues such as redness, shape, and color information. By region growing, we select regions, which are to be completed with iris texture by an exemplar-based inpainting method. Then, for red-eye correction, pupils are painted with the appropriate radii calculated from the iris size and size ratio. Experimental results with a large number of test photographs with red-eye effect show that the proposed algorithm is effective and the corrected eyes look more natural than those processed by the conventional algorithms.

Full text

Page 1

IEEE Transactions on Consumer Electronics, Vol. 55, No. 3, AUGUST 2009
Contributed Paper
Manuscript received July 11, 2009 0098 3063/09/$20.00 © 2009 IEEE
1006
Red-Eye Detection and Correction Using Inpainting
in Digital Photographs
Seunghwan Yoo and Rae-Hong Park, Senior Member, IEEE

Abstract — When we take pictures with flash, red-eye
effect often appears in photographs. Flash light passing
through pupil is reflected on the blood vessels, and arrives at
a camera lens. This phenomenon makes red-eyes in
photographs. Several algorithms have been proposed for
removal of red-eyes in digital photographs. This paper
proposes a red-eye removal algorithm using inpainting and
eye-metric information, which is largely composed of two
parts: red-eye detection and red-eye correction. For red-eye
detection, face regions are detected first. Next, red-eye
regions are segmented in the face regions using multi-cues
such as redness, shape, and color information. By region
growing, we select regions, which are to be completed with
iris texture by an exemplar-based inpainting method. Then,
for red-eye correction, pupils are painted with the
appropriate radii calculated from the iris size and size ratio.
Experimental results with a large number of test
photographs with red-eye effect show that the proposed
algorithm is effective and the corrected eyes look more
natural than those processed by the conventional algorithms.
1
Index Terms — Red-eye, Red-eye detection, Red-eye
correction, Inpainting, Biometric information, Pupil size.
I. INTRODUCTION
When taking pictures in dark lighting conditions, people
use flash light to make objects visible. However, flash light
causes red-eye effect. Flash light passing through enlarged
pupils is reflected on the blood vessels inside eyes, and arrives
at a camera lens. This flash light makes eyes in the photograph
red.
Red-eye removal methods are categorized into two classes:
physics-based and software-based. A physics-based method is
to prevent red-eye effect. Red-eye effect can be avoided by
changing the lighting condition or by increasing the distance
between a camera lens and flash. A software-based method
post-processes digital photographs using algorithms to remove
existing red-eye in them. Algorithms for red-eye removal have
been proposed [1–8] and commercial softwares have been also
developed.
Generally, a red-eye removal algorithm is divided into two
parts: red-eye detection and red-eye correction. It is difficult
to detect red-eyes correctly because red-eye regions are
usually quite small and there are numerous areas that have
similar features to red-eye regions. Many commercial software

S. Yoo and R.-H. Park are with the Department of Electronic Engineering,
Sogang University, C.P.O. Box 1142, Seoul 100-611, Korea (e-mail: {adri00,
rhpark}@sogang.ac.kr).

for red-eye removal let users select red-eyes manually to
overcome the difficulty in detecting the red-eye region. On the
other hand, red-eye correction algorithms are relatively simple.
Many algorithms simply reduce red-component of red-eye
areas. Instead, various algorithms are proposed for delineating
natural boundaries of corrected red-eyes [1, 5, 7, 8].
This paper proposes a red-eye removal algorithm using
inpainting and eye-metric information. For red-eye detection,
a face detection stage is included [9]. We limit the candidate
region by using face detection, and detect red-eye using
multiple cues–redness, shape, and color information. After
detecting red-eyes, we expand the red-eye regions by region
growing and morphology operations so that they become fit to
our correction algorithm. In correction part, red-eyes are filled
with textures of surrounding iris by an exemplar-based
inpainting method [10, 11]. Finally, pupils, whose size is
computed along with eye size information, are painted and
highlight of eye is added for natural appearance. The proposed
algorithm produces better correction results because it
removes enlarged part of pupil in darkness and paints the
pupil of appropriate size.
The rest of the paper is structured as follows. Section II
explains the cause of red-eye effect and reviews conventional
methods for red-eye removal. Two parts of the proposed
algorithm, red-eye detection and red-eye correction, are
presented in Sections III and IV, respectively. Experimental
results and discussions are described in Section V and finally
conclusions are given in Section VI.
II. RED-EYE EFFECT
This section explains the cause of red-eye effect first and
describes the methods of handling red-eye effect.
A. Cause of Red-Eye Effect
Red-eye effect results from flash light of a camera. When it
is dark, pupils of eyes are enlarged. Through the enlarged
pupils, flash light from a camera passes the eye lens and is
reflected back. This reflected light is blood-color since it is
reflected from the blood vessels inside an eye.
Fig. 1 describes the geometric situation in which red-eye
effect occurs. The light reflected from the eye composes the
red-eye beacon. If a camera lens which receives the light is
located inside the red-eye beacon, pupils in the picture look
red. In other words, red-eye effect occurs if
2/x <
where x is the distance between a camera lens and flash light,
and rc is the radius of the red-eye beacon at the camera
position.
cr

(1)
A u t h o r i z e d l i c e n s e d u s e l i m i t e d t o : I E E E X p l o r e . D o w n l o a d e d o n M a y 1 3 , 2 0 1 0 a t 1 1 : 4 5 : 4 9 U T C f r o m I E E E X p l o r e . R e s t r i c t i o n s a p p l y .
www.DownloadPaper.ir
www.DownloadPaper.ir

Page 2

S. Yoo and R.-H. Park: Red-Eye Detection and Correction Using Inpainting in Digital Photographs
www.DownloadPaper.ir
www.DownloadPaper.ir
1007
From the similar triangles in Fig. 1, the radius of red-eye
beacon at camera position, rc is expressed as
lrr/
×=
pcpc
l

(2)
where lp, lc, and rp denote the diameter of eye sphere, the
distance between an eye and a camera, and radius of pupil,
respectively. Let lp be a constant, then rc is proportional to lc
and rp. From (1) and (2), it is noted that red-eye effect can
occur more frequently when x is small and lc and rp are large.
B. Methods for Red-Eye Removal
Red-eye effect will never occur if we do not use flash light
when we take a picture. Another light source can be used to
control brightness instead of flash light. But it is impractical to
carry another lighting device and most of people use flash
light when it is dark.
Methods to remove red-eye effect are divided into two
ways. First one is to prevent the red-eye by avoiding the
situation physically that satisfies the inequality in (1). We can
increase the distance between flash and lens (x) so that the
lens is located outside the red-eye beacon. However, the size
of a camera should be large enough. Or we can also make the
size of pupil (rp) small by using pre-exposure flash. Camera
will consume much power and people feel annoyed when
seeing the pre-exposure flash.
Second, red-eye effect can be removed by software
algorithms. Algorithms for red-eye removal are researched by
many corporations and laboratories [1–8]. Some image editing
software tools offer the function of red-eye removal and some
companies developed the software for red-eye removal to
apply to their products.

C. Conventional Algorithms for Red-Eye Removal
In general, red-eye removal algorithms are composed of
two parts: red-eye detection and red-eye correction. We
review the conventional algorithms for red-eye detection and
red-eye correction in this subsection.
1) Red-Eye Detection
Most of red-eye removal algorithms largely focused on red-
eye detection than on red-eye correction. Because red-eyes are
usually small and share similarities with many small red things,
it is complex and difficult to find them reliably. Many
algorithms have been proposed for red-eye detection [1–8].
In order to find red-eyes, some algorithms take advantage
of face detection. Gaubatz and Ulichney detected red-eyes
using color variation, redness, redness variation and glint in
the previously-detected face area [1]. Schettini et al. searched
red-eyes using redness and size information in the face region
[2]. Ioffe first found red-eye candidates using trained detector
and then excluded false alarms outside the face region [3].
In some research, learning is used for red-eye detection.
Ioffe trained the red-eye, red-eye outline, and face detector by
boosting and classified red-eye [3]. Luo et al. used Haar-like
rectangle features to train the red-eye detector and made a
classifier [4].
Willamowski and Csurka presented a probabilistic approach
[5]. Rather than identifying the red-eye region, probability is
assigned to each pixel depending on the possibility that the
pixel corresponds to the red-eye region. Thus, the detector
reduces the false negative possibility. Volken et al. detected
red-eye using sclera and skin color information around eye [6].
They calculated the possibility to be red-eye using the
distribution of sclera and skin color and found red-eye using
size and shape information.
2) Red-Eye Correction
Most of algorithms for red-eye correction basically perform
the same steps, while algorithms for red-eye detection are
different. Most red-eye correction algorithms desaturate red
color component from red-eyes [1–8]. Of course, they
distinguished their algorithms from other algorithms by
proposing the method that makes corrected red-eyes more
natural. Gaubatz and Ulichney desaturated red color in
proportion to redness in order to soften the boundary of red-
eye [1]. Willamowski and Csurka removed redness according
to the possibility of red-eye [5]. Zhang et al. corrected red-eye
by adaptively calculating the factor for red-eye region [7].
Ulichney and Gaubatz did perceptual study in order to find the
most natural luminance for corrected eyes [8].

III. PROPOSED RED-EYE DETECTION
This section describes a proposed red-eye detection
algorithm. As shown in Fig. 2, the proposed red-eye detection
algorithm is composed of three modules: face detection, red-
eye detection, and region-growing. When an image, Iin, is
given as input, faces are detected and eye regions, Rf are
defined. Then, red-eye regions, Rr, are searched inside Rf.
Red-eye regions are expanded to
method for the next step of red-eye correction.
r
R′ by a region-growing
A. Face Detection
A face detection algorithm is used to limit the candidate
region for red-eye detection. In this paper, Viola and Jones’
algorithm is used to detect faces [9]. If we found the face

Fig. 2. Geometry of red-eye reflection [8].

Fig. 1. Block diagram of the proposed red-eye detection algorithm.
A u t h o r i z e d l i c e n s e d u s e l i m i t e d t o : I E E E X p l o r e . D o w n l o a d e d o n M a y 1 3 , 2 0 1 0 a t 1 1 : 4 5 : 4 9 U T C f r o m I E E E X p l o r e . R e s t r i c t i o n s a p p l y .

Page 3

IEEE Transactions on Consumer Electronics, Vol. 55, No. 3, AUGUST 2009
1008
regions, then we restrict the search region more narrowly.
Eyes are located in specific area of the face region by the
attribute of the face detection algorithm we used. When we
divide the face region into 20 cells, in four rows and five
columns, we observe that two eyes are mostly placed in the
second-row (from second-column cell to fourth-column cell),
as shown in Fig. 3. Thus, we define the three cells as
candidate regions for eyes (Rf). Fig. 4 shows examples of face
detection, where green boxes denote face regions and blue
boxes represent the eye regions. Most of faces are detected
unless the face is occluded by other objects. And if face
regions are found correctly, eyes are detected in the specific
area.


B. Red-Eye Detection
Red-eyes are detected by using features of red-eye extracted
from the eye region. Fig. 5 shows the block diagram of the
proposed red-eye detection process and Fig. 6 shows an
example of the red-eye detection process. Fig. 6(a) is the
detected eye region and Figs. 6(b)–(f) are the results of each
stage.

1) Redness detection
Redness is the primary feature for red-eye detection. We
use Gaubatz and Ulichney’s measure for redness [1]. Redness
is defined as
=
),(
rkyxByxG
y
(
xR
yx redness
++
),
),
2
(
),(
2
2

(3)
where R, G, and B denote red, green, and blue components,
respectively, and kr is a constant to avoid zero denominator.
The pixel at (x, y) is considered red if
redness(x, y) >
1r τ

(4)
where the value of redness threshold,
experimentally. Fig. 6(b) shows the result of redness detection
from an eye region in Fig. 6(a).
1r τ
is adjusted
2) Skin color removal
Since human skin often contains red component much,
redness is calculated high in skin region. So there are many
false alarms in skin region. These false alarms can be reduced
by eliminating skin color region. Fig. 6(c) shows the result of
skin color removal from Fig. 6(b)
We use
r
CY
′′
color space for skin color removal, since it
shows the best performance of skin color detection. YCgCr
and
r
CY
′′
spaces are proposed for face detection [12, 13],
and RGB color space is converted to
gC
gC
r
′
gC
′
CY
as follows
⎥
⎦
⎥
⎥
⎤
⎢
⎣
⎢
⎢
⎡
+
⎥
⎦
⎥
⎥
⎤
⎢
⎣
⎢
⎢
⎡
⎥
⎦
⎥
⎥
⎤
⎢
⎣
⎢
⎢
⎡
−−
−−=
⎥
⎦
⎥
⎥
⎤
⎢
⎣
⎢
⎢
⎡
128
128
16

071 . 0 368. 0 439 . 0
121 . 0438 . 0 317. 0
098. 0 504. 0257 . 0
B
G
R
C
C
Y
r
g

(5)
48 30 sin 30 cos
−⋅+⋅
′
g
=
′
g
=
oo
r
CCC

80
+
30 cos30 sin
⋅+⋅−
′
r
oo
rg
CCC
.
(6)
Skin color is in the range of
125
≤
C
140
≤
′
g
?
217136
≤
′
r
≤
C

(7)
in
r
′
gC
′
CY
color space [13].

Fig. 3. Eye candidate region.




Fig. 4. Examples of face detection.

Fig. 5. Block diagram of the proposed red-eye detection.



(a) (b)


(c) (d)


(e) (f)
Fig. 6. An example of red-eye detection process of the proposed
algorithm. (a) eye candidate region, (b) redness detection, (c) skin
color removal, (d) closing operation, (e) labeling, (f) shape filtering.
A u t h o r i z e d l i c e n s e d u s e l i m i t e d t o : I E E E X p l o r e . D o w n l o a d e d o n M a y 1 3 , 2 0 1 0 a t 1 1 : 4 5 : 4 9 U T C f r o m I E E E X p l o r e . R e s t r i c t i o n s a p p l y .
www.DownloadPaper.ir
www.DownloadPaper.ir

Page 4

S. Yoo and R.-H. Park: Red-Eye Detection and Correction Using Inpainting in Digital Photographs
www.DownloadPaper.ir
www.DownloadPaper.ir
1009
Fig. 7 shows the results of skin color removal in various
color spaces. The detected skin regions in RGB, normalized
RGB, YCbCr, HSV, and
CY
′
r
′
C
gC
Y
color spaces are shown in
′′
color space gives the
Figs. 7(a)–7(e), respectively.
rgC
most differentiable results, in which the red-eye regions are
well separated from the skin region.

3) Closing operation
Since previous two steps are pixelwise, result regions are
scattered and difficult to handle. Using closing morphology
operation, neighboring regions are merged and tiny holes in
the regions are filled. The regions become simple and easy to
handle by closing operation, as shown in Fig. 6(d).

4) Labeling
Then, connected regions are labeled by sequential
connected components algorithm with 4-connectivity [14].
Using labeling, we distinguish each connected component and
take advantage of the information of size and shape of each
component. Fig. 6(e) shows the result of the labeling, in which
each component region has different color.
5) Shape filtering
In order to reduce false alarms, three features (region’s
width and height, ratio of width to height, compactness) are
used, defined as follows:
WkwWk
w
max
iw
maxmin
≤≤
,
HkhHk
hih
min
≤≤

(8)
2
2
1
≤≤
i
i
h
w

(9)
2
ii
i
hw
a
⋅
≥

(10)
where
connected component of i-th label, with W and H signifying
the width and height of the face region, to which the
component belongs.
,
hw
kk
constants signifying the range of possible pupil size relative to
the width and height of a face region. The constants are
selected as
50/ 1
==
hw
kk
area of the connected component of i-th label.
The size filter described by (8) gives high performance, since
it filters false alarms using the size information of a face, to
which each region belongs. We calculate slimness, or ratio of
width to height, by (9). Since shape of pupil is circular, we
remove slim regions, whose ratio of width to height is too large
or too small. We also remove a non-compact region, shaped like
alphabet ‘L’ or ‘C’, by (10). Most of the false alarms are
removed by shape filtering as shown in Fig. 6(f).

i w and
ih denote the width and height of the
,
minmin max
h
max
w
and ,
kk
are the
,
min min
. 12 / 1
=
max
h
max
w
=
kk

ia is
C. Region-Growing
We use a region-growing algorithm so that the region fits to
the next process, inpainting. In the previous red-eye detection
stage, some red-eye regions may be excluded by the threshold,
r τ , in (4), if redness is weak. Red-eye regions need to be
grown to cover all the regions which appear red in comparison
to the original color of iris. If not, results of inpainting contain
red pixels in iris region and become unnatural.
Examining the boundary of red-eye regions detected in
previous steps, the area that satisfies the following conditions
is to be included in the red-eye region
1
redness(x, y) >
2
r τ

(11)
lum(x, y) >
l τ
(12)
125
<
′g
C
?
140
>
′g
C
?
136
<
′r
C
?
217
>
′r
C

(13)
where
for red-eye region, respectively. Note that (13) represents non-
skin region.
Fig. 8 shows the pseudo-code of the region growing method.
For region growing, we first filter the redness and luminance
as described in (11) and (12). Redness threshold,
smaller than
1
to include even weak red pixels. Specular
reflectance area can be included in the red-eye regions by (12).
On the contrary, we exclude skin region from red-eye region.
Skin region in eye regions may be detected as red-eye since it
is reddish.
Through iterations, the red-eye region is grown, and if there
is no update and iteration is more than 10 times, the iteration
terminates. For the first two iterations, the condition in (13) is
not used because the red-eye area may contain parts that
overlap with skin region. Finally, the closing operation
smoothen the boundary of the region. Fig. 9 shows an
example of region growing.
2
r τ
and
l τ are the thresholds for redness and luminance
,
2
r τ
is
r τ

(a) (b) (c)

(d) (e)
Fig. 7. Skin color detection in various color spaces. (a) RGB, (b) rgb,
(c) YCbCr, (d) HSV, (e)
.
r
CY
′′

gC
A u t h o r i z e d l i c e n s e d u s e l i m i t e d t o : I E E E X p l o r e . D o w n l o a d e d o n M a y 1 3 , 2 0 1 0 a t 1 1 : 4 5 : 4 9 U T C f r o m I E E E X p l o r e . R e s t r i c t i o n s a p p l y .

Page 5

IEEE Transactions on Consumer Electronics, Vol. 55, No. 3, AUGUST 2009
1010
Fig. 8. Pseudo-code for region growing.


(a) (b) (c)
Fig. 9. Region growing of red-eye. (a) original image, (b) detected red-eye
region, (c) result of region growing.


Fig. 10. Block diagram of the proposed red-eye correction algorithm.


Fig. 11. Block diagram of iris detection.

IV. PROPOSED RED-EYE CORRECTION
The proposed correction algorithm uses a medical research
and digital inpainting algorithm to correct the red-eye region
while other algorithm simply desaturates the red color
components. As shown in Fig. 10, we can calculate the
adequate pupil size, dpupil by using the information on eye size
and the measure of iris diameter, diris. After red-eye
regions,
r
R′ , are filled with iris texture by an inpainting
method, pupil is painted with black color with the pupil size,
dpupil, producing the corrected image, Iout.

A. Iris Detection
Size of iris depends on person and scale of an input image.
To get the size of iris, we process as shown in Fig. 11. Since it
is difficult to detect iris directly, we make use of the result of
red-eye detection in Section III. By going through outside the
boundary of red-eye, we can find the boundary between iris
and sclera. Each block in Fig. 11 is described in the following.

1) Canny edge detection
Canny edge detector is applied to the red-eye region mask
in order to detect the boundary of the red-eye region. Since
the mask is binary, clear boundary can be obtained. Fig. 12
shows an example. From the red-eye mask in Fig. 12(a), its
boundary is obtained as in Fig. 12(b).

2) Hough transformation
Detected red-eye regions often do not have circular shape. We
cannot get the accurate center position and radius of the iris from
the crooked shape of detected region. Thus, we detect the most
similar circle with the red-eye region by using Hough
transformation. Fig. 12(c) shows the result of Hough
transformation. Since it detects not only boundary of iris but also
false alarms, we remove false alarms by limiting the size of
detected circles. Even though there is no false edge, there are
only a small number of circle candidates because eyes are usually
small. If the threshold of transformation is not appropriate, circles
might not be detected or many false alarms may exist.

3) Sclera detection
At the boundary between iris and sclera, we can find edges
by using gradient operation, and obtain the radius of iris.
From both sides of iris regions, the pixels with large
horizontal gradient are searched vertically, where Sobel masks
are used. We decide that a pixel is in the boundary if
g
τ>
,
,
gvG
g
τ>
,
where gR,v, gG,v, and gB,v denote gradient values in red, green,
and blue components, respectively, and
gvR
,
gvB
g
τ>
,

(14)
g
τ signifies the fixed
threshold for gradient values.
BEGIN Region-Growing
Initialize:
iter = 0
IF
yx,(
yx),(
END IF
DO
x
∀
,(
IF iter < 3
IF
r
R
∈
∈
)

r R
′

rr
RRyxNy
∈∉∈
pp ,), ( ),()

2
),(
r
yx
∈
redness
x(
END IF
ELSE
IF
OR NOT skin-color
yx,(
END IF
END IF
iter++
WHILE (iter<10 OR updated
END DO
END
τ
>

OR lum(x,y)
l τ
>

i R y),
2
),(
r
yx redness
τ
>
OR lum(x,y)
l τ
>

i R
∈
)

r R′ )


(a) (b) (c)
Fig. 12. Example of iris detection. (a) red-eye region, (b) Canny edge
detection, (c) circle detection by Hough transformation.

A u t h o r i z e d l i c e n s e d u s e l i m i t e d t o : I E E E X p l o r e . D o w n l o a d e d o n M a y 1 3 , 2 0 1 0 a t 1 1 : 4 5 : 4 9 U T C f r o m I E E E X p l o r e . R e s t r i c t i o n s a p p l y .
www.DownloadPaper.ir
www.DownloadPaper.ir