Page 1

An Illuminance-Reflectance Nonlinear Video Enhancement Model

for Homeland Security Applications

Li Tao, Richard Tompkins, and Vijayan K. Asari

Department of Electrical and Computer Engineering

Old Dominion University, Norfolk, VA 23529

Email: {ltaox001, rtomp002, vasari} @odu.edu

ABSTRACT

enhancement algorithm is proposed for improving the

visual quality of digital video streams captured by

surveillance camera under insufficient and/or non-uniform

lighting conditions. The paper presents computational

methods for estimation of scene illuminance and

reflectance, adaptive dynamic range compression of

illuminance, and adaptive enhancement for mid-tone

frequency components. The images are processed in a

similar way as human eyes sensing a scene. The algorithm

demonstrates high quality of enhanced images, robust

performance and fast processing speed. Compared with

Retinex and multi-scale retinex with color restoration

(MSRCR), the proposed method shows a better balance

between luminance enhancement

enhancement as well as a more consistent and reliable

color rendition without introducing incorrect colors. This

is an effective technique for image enhancement with

simple computational procedures, which makes real-time

enhancement for homeland

successfully realized. The application of this image

enhancement technique to the FRGC images yields

improved face recognition results.

A illuminance-reflectance model based video stream

and contrast

security application

1. Introduction

When human eyes sense a scene, the large dynamic

range of the radiance from the scene can be well perceived

through a series of adaptive mechanisms for brightness

perception and processing. Some major dynamic

compression mechanisms are based on the lateral

processing at the retinal level, and some occur in the early

visual cortex [1]. Unfortunately, current cameras and

image display devices do not have similar sophisticated

features. Saturation and underexposures are common in

images due to limited dynamic range of the imaging and

display equipment. This problem becomes more common

and severe when insufficient or non-uniform illumination

occurs.

However, the conventional image enhancement

techniques, like the global brightness and contrast

enhancement, gamma

compression and histogram equalization, are generally

incapable of providing satisfactory enhancement results

for those images having shadows which are created due to

the problem discussed in last paragraph. In order to

improve the quality of images captured under complex

lighting conditions, some advanced image enhancement

techniques have been developed, such as Retinex [2],

multi-scale retinex (MSR) [3,4], luma dependent nonlinear

enhancement (LDNE) [5], and adaptive histogram

equalization [6]. Among them, Retinex based methods are

effective in dynamic range compression and local contrast

enhancement, which are derived from Land’s “Retinex”

theory [7] that deals with human visual perception of

lightness and color. However, both MSR and standard

Retinex may produce strong “halo” effect and incorrect

colors on enhanced images due to their separate nonlinear

processing of the three spectral bands, which will be

avoided in the proposed algorithm. In addition, both

methods are also computationally inefficient with the

three-band processing. Therefore, it has to rely on

hardware implementation to achieve real time processing

of video sequences [8].

In this paper, we present a new image enhancement

algorithm based on illuminance perception and processing

to achieve dynamic range compression while retaining or

even enhancing visually important features. Due to its

simplicity of our algorithm, real time processing of color

video streams has been implemented in software on

personal computer for homeland security applications.

Our algorithm is based on some assumptions about

image formation and human vision behavior. First, the

image intensity I(x, y) can be simplified and formulated as

a product [9]:

adjustment, logarithmic

( , )

I x y

( , ) ( , )

L x y R x y

?

(1)

where R(x, y) is the reflectance and L(x, y) is the

illuminance at each point (x, y). Second, the luminance L

Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)

0-7695-2479-6/05 $20.00 © 2005 IEEE

Page 2

is assumed to be containing in the low frequency

component of the image while the reflectance R mainly

includes the high frequency component of the image. This

assumption can be easily understood if considering that R

generally varies much faster than L does in most parts of a

image with a few exceptions, like shadow boundaries

where large L changes exist. In addition, in a real world

scene, the dynamic range of the illumination variation can

be several orders larger than the dynamic range of the

reflectance. Therefore, compressing the dynamic range of

the illuminance is an effective way for image enhancement.

Finally, because human eyes responds to local changes in

contrast rather than to global brightness levels, it is

possible to keep the visually important features

represented by reflectance while compressing the image’s

dynamic range mainly induced by illuminance.

2. Algorithm

The proposed image enhancement algorithm is

composed of four major steps: (1) illuminance

estimation and reflectance extraction; (2) adaptive

dynamic range compression of illuminance; (3) adaptive

mid-tone frequency components enhancement; (4) image

restoration, which combines illuminance and reflectance to

recover the intensity image and then performing color

recovery.

2.1. Illuminance Estimation

Accurate estimation of illuminance of a scene from an

image is a difficult task. Many techniques have developed

to deal with this problem [5,8,9]. In our algorithm, the

low-pass filtered result of the intensity image through

2D discrete Gaussian filtering is used as the estimated

illuminance by:

? ?

??

00

mn

where L is the illuminance, I is the original intensity image

and G is the 2D Gaussian function with size M ? N. G is

defined as:

?

x

?

?

?

?

where q is determined for normalizing the Gaussian

function by

?

2

c

q e dxdy

?

??

and c is the scale to determines the size of the

neighborhood. In our algorithm, c = 2~5. Then,

illuminance is normalized as:

??

???

11

),(),(),(

MN

ynxmGnmIyxL

(2)

?

22

2

( , )

G x y

.

y

c

q e

??

?

?

?

??

(3)

?

22

1

xy

?

?

?

?

?

?

?

?

??

?

(4)

( , )

L x y

( , )

L x y ?

255

n

(5)

For color images, the intensity image I(x,y) is obtained

using the following method:

?

),,( max),(

yxryxI

?

? )

y

,( ),,(

xbyxg

(6)

where r, g and b are the RGB components of color

images in RGB color space. We process on the intensity

image instead of processing all color bands separately

because the latter way is more time consuming and not

found significantly superior to intensity processing in

terms of the quality of enhanced images. This method is in

fact the definition of the value component in HSV color

space. We didn’t use NTSC to get the intensity image

because NTSC generally produces inconsistent or shifted

colors like the red colors in the enhanced image after

linear color restoration compared with the original image.

Once the illuminance L is obtained using Eq. (2), the

reflectance R is computed using Eq. (1). One example

showing the results of L and R from an image is provided

in Figs. 2(b) and 2(c). It can be observed that the

illuminance L comprises the mid-tone and low-frequency

information of the image. This is not the real illuminance

as defined in physics but an approximated illuminance that

contains both the illuminance and the mid-tone and

low-frequency components of reflectance. The visually

important features (high frequency reflectance) and a

small part of illumination information are contained in the

reflectance in which the major illumination effect is

removed. Therefore, important image features will be kept

after the dynamic range compression of illuminance.

Based on those observations, reflectance and illuminance

were also regarded as details and base [9].

2.2 Adaptive Dynamic Range Compression of

Illuminance

The dynamic range compression of illuminance realized in

our algorithm uses the proposed Windowed Inverse

Sigmoid (WIS) function. The sigmoid function is defined

as:

1

( )

1

e?

?

This function can also be used as the intensity transfer

function for dynamic range compression by performing

the computational steps described by Eqs. (8) - (10).

av

f v

?

(7)

maxminmin

'[ ()()]()

nn

LL f vf vf v

???

(8)

11

''ln1

'

n

n

L

aL

?

?

?

?

?

''

?

?

?

??

(9)

min

v

,

maxmin

n

n enh

L

Lv

v

?

(10)

where Eq. (8) is used for linearly mapping the input range

[0 1] (normalized illuminance) to the magnitude range

Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)

0-7695-2479-6/05 $20.00 © 2005 IEEE

Page 3

[f(vmin) f(vmax)] for WIS. Eq. (9) is the inverse sigmoid

function. Eq. 10 is applied to normalize the output

illuminance to [0 1]. Parameters vmax, and vmin are used to

change the line shape of the transfer function.

Fig.1. WIS is used for Intensity transformation. Fig.1. WIS is used for Intensity transformation.

A set of the curve shapes of WIS transfer function is

provided in Fig. 1 with a = 1 and vmax = 3. Those curves

are produced by Eqs. (8) - (10) with different parameters:

vmin= -6 for the dotted red curve; vmin = -3 for the dashed

red curve; and vmin = -4.5 for the solid red curve. Identity

transfer function is also provided for comparison.

Obviously, the inverse sigmoid function can also be used

to pull down the intensity of the overly-lighted pixels.

The value of vmax is always set to 3 for all images while

the value for vmin is image dependent and determined by

the global mean Im of the intensity image I to obtain

self-adaptive control over the dynamic range compression.

Based on the image enhancement experiments, the value

for vmin is determined by Imas:

min

6 70

70

3 6

? ?

70 150

80

3 150

m

m

m

m

for I

I

forI

for I

?

??

?

?

?

?

??

??

?

???

?

(11)

(a) (b) (c)

(d) (e) (f)

Fig.2. (a) original image; (b) estimated illuminance; (c) reflectance;

(d) dynamic range compressed illuminance; (e) mid-tone

frequency components enhancement from (d); (f) enhanced image. frequency components enhancement from (d); (f) enhanced image.

Fig.2. (a) original image; (b) estimated illuminance; (c) reflectance;

(d) dynamic range compressed illuminance; (e) mid-tone

2.3. Adaptive Mid-tone Frequency Components

Enhancement

It has been noted that the illuminance also contains the

mid-tone and low frequency components of reflectance

which has been degraded during the dynamic range

compression. For the original image with low contrast or

slow-varying reflectance map, the degradation of mid-tone

frequency features may be rather obvious in the output

images. Therefore, a center-surround type of contrast

enhancement method is utilized to compensate this

degradation. This mid-tone frequencies enhancement is

carried out as defined in the following two equations:

(,)

,,

' ( ,

x y

) ( ,

x y

)E x y

n enh n enh

LL

?

(12)

where the exponent is defined by:

( , )

( , )

I x y

( , )

E x y

( , )

R x y

P

P

conv

I x y

?

?

?

?

?

? ??

(13)

L’n,enh(x, y) is the illuminance after mid-tone frequencies

enhancement and R(x, y) is the ratio between I(x, y) and its

low-pass version Iconv(x, y) which is computed through the

same operations as in Eqs. (2) – (4) with a larger scale c

(10 ? 10, lower cut-off frequency). P is determined by the

global standard deviation ? of the input intensity image I(x,

y) as:

2 30

for

?

?

?

?

?

0.03 2.93080

1/280

P for

for

?

?

?

?

?

?

? ????

(14)

According to this definition, P and ? have a linear

relationship within the range ??(30, 80]. This relationship

is determined based on huge number of image

enhancement experiments. Here, the global standard

deviation ? of I(x, y) is considered as an indication of the

contrast level of the original intensity image. It should be

noted that P can also be changed by users to manually

adjust the contrast enhancement process.

The mid-tone frequency components enhancement

process defined in Eqs. (12) and (13) is actually an

intensity transformation process and can be understood

using Fig. 3.

00.10.20.3 0.40.5

In'(x,y)

0.60.7 0.80.91

0

0.2

0.4

0.6

0.8

1

In'(x,y)E(x,y)

E=2

E=3

E=1/2

E=1/3

E=1

Fig.3. Intensity transformation for mid-tone frequencies

enhancement for illuminance enhancement for illuminance

Fig.3. Intensity transformation for mid-tone frequencies

Since Ln,enh is normalized to 1 as defined before,

Ln,enh (x, y)E(x, y) will be larger than Ln,enh (x, y) if E(x, y) is

less than 1 (e.g. the center pixel is brighter than

Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)

0-7695-2479-6/05 $20.00 © 2005 IEEE

Page 4

surrounding pixels leading to R(x, y) < 1), and vise versa.

In this way, the contrast of the compressed illuminance

image can be improved. Here, the ratio R(x, y) is obtained

from the original intensity image I(x, y) and its low-pass

filtered result Iconv (x, y), since the reflectance’s mid-tone

and low frequency information has been lost and degraded

in the estimated illuminance L.

The parameters of P and vmin are introduced to this

algorithm to increase the flexibility of this algorithm.

Meanwhile, they can also improve the robustness of the

enhancement performance to produce consistent good

results for images captured under various types of lighting

conditions.

2.4. Image Restoration and Color Recovery

Following the dynamic range compression and contrast

enhancement, the final illuminance L’n,enh and reflectance

R are combined using Eq. (1) to produce an intensity

image I’ with compressed dynamic range (see Fig. 2(f)).

For color images, a linear color restoration process based

on the chromatic information of the original image is

applied to I’ to recover the RGB color bands (r’, g’, b’) as:

''

''

rrgg

II

'

'

III

I

bb

???

(15)

so the color information (hue and saturation) in the

original image is preserved in the enhanced image.

3. Algorithm Implementation for Real-time

Homeland Security Application

A software package named IESuite has been developed

and implemented on desktop/laptop PCs for real-time

image enhancement using C++ in Windows XP

environment on a PC with a 3.2GHz Intel Pentium 4

processor and 1GB DDR SDRAM memory. Sony

DCR-HC85 digital video camera is used to capture the

video stream with a frame size of 360 ? 240 pixels. A

processing speed of 12 frames per second has been

achieved. Fig. 4 shows a screen capture of the interface of

the video enhancement program. The digital video stream

is transferred from video camera into computer via IEEE

1394 ports. In order to achieve optimized computing speed

for real-time applications, the large scale convolutions are

computed in frequency domain with Fast Fourier

Transform (FFT) while small scale convolutions are still

performed in spatial domain.

The video stream enhancement can be controlled

with playback commands, and to ‘snapshot’ any enhanced

frame is permitted. The enhanced digital video stream can

be recorded by multi-clips at anytime when the user is

interested in the scene, and all the clips finally are

automatically saved into one AVI video file with a

selected encoder/compressor. In addition, the recording

rate can be manually set.

Fig.4. Screen capture of real-time digital video enhancement by

our developed enhancement software package ‘IESuite’. our developed enhancement software package ‘IESuite’.

Fig.4. Screen capture of real-time digital video enhancement by

4. Results and Discussion

In this section, several important features of the

algorithm will be first discussed with experimental results.

Then the performance and robustness of the algorithm will

be evaluated by comparing with MSRCR and Retinex.

Finally, a statistic analysis method will be introduced to

illustrate the image enhancement effects of the algorithm

4.1. Adaptive Dynamic Range Compression

The curve shape of WIS can directly affect the

dynamic range compression of the illuminance. From Fig.

1, it can be easily understood how the curve shape can be

changed by adjusting the value for vmin which can be

manually tuned or image dependent for adaptive control.

The effect of vmin is illustrated in Fig. 5 with a sample

image treated with various vmin values, Fig. 5(c) is set by

Eq. (11). Obviously, the vmin value set by Eq. (11)

produces better result (Fig.5(c)) than manual adjustment

results (Figs. 5(b) and 5(d)) without over or under

enhancement of brightness.

(a) (b) (c) (d)

Fig.5. Image enhancement with different Fig.5. Image enhancement with different ? ?min

enhanced image with enhanced image with ? ?min

= -3; (c) enhanced image with ? ?min

set by Eq. (11); (d) enhanced image with set by Eq. (11); (d) enhanced image with ? ?min

Note: This sample image is provided by [10]

min: (a) original image; (b) : (a) original image; (b)

min = -3; (c) enhanced image with

min = -4.9, = -4.9,

min = -7. = -7.

4.2. Adaptive Mid-tone Frequency Enhancement

The parameter P in Eq. (14) is used to tune the

enhancement for mid-tone frequency components to

improve the image contrast which is poor in the original

image or has been significantly degraded due to dynamic

range compression. The effect of P on image enhancement

is illustrated in Fig. 6 with the sample image processed

Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)

0-7695-2479-6/05 $20.00 © 2005 IEEE

Page 5

with various P values. Fig.6(c) is determined by Eq. (14),

which produces a more appropriate result compared to the

other two enhanced images with manually set P values.

For example, the wood grains are well enhanced without

producing severe halo effect, that is obvious in Fig.6(d).

(a) (b) (c) (d)

Fig.6. Image enhancement with different Fig.6. Image enhancement with different P P: (a) original image; (b)

enhanced image with enhanced image with P P = 1/2; (c) enhanced image with

= 1/2; (c) enhanced image with P P = 1, set

by Eq. (14); (d) enhanced image with by Eq. (14); (d) enhanced image with P P = 2.

: (a) original image; (b)

= 1, set

= 2.

4.3. Algorithm Robustness Evaluation

To test the performance and robustness of the proposed

algorithm, it is applied to images twice: the first

enhancement processing is carried out on the original

image, then the second enhancement processing is

performed on the output image from the first enhancement.

An example of this process is presented in Fig. 7

accompanied by the results produced by MSRCR and

Retinex. Retinex is realized using the Matlab? code

provided in Reference [2] by Frankle and McCann. A

commercial digital image

PhotoFlair® version 2.0 (TruView Imaging Company) is

used to implement MSRCR algorithm. It can be observed

that the images processed by our method demonstrate a

higher visual quality than those processed by MSRCR and

Retinex. Our method yields better color accuracy and

better balance between the luminance and contrast across

the whole image due to its adaptiveness and flexibility

involved in the processing. Obviously, after twice

enhancement, the proposed algorithm produces the

minimal change from the first enhancement result while

image quality is degraded in images produced by MSRCR

and Retinex, in which incorrect lightness, color rendition,

halo effect, and image noise become much more visible

after the second enhancement.

processing software

4.5. Statistical Analysis

The statistical properties of images, image mean and the

zonal standard deviation, are used to assess the visual

quality of images in terms of brightness and contrast

which are directly associated with those statistical

parameters. The local brightness is measured by the image

local mean while the local contrast is evaluated by taking

the regional standard deviations. As shown in Fig. 8(a), we

plotted the local mean and local standard deviations of all

blocks (block size: 55 ? 55 pixels) of a sample image (Fig.

7(a)) before and after image enhancement. It can be seen

that in some regions of the sample image, the luminance

enhancement is dramatic (eg. the shadows of the wall and

the lady) but in some regions it is almost the same as the

original luminance (eg. the regions of the window). This is

exactly matched with the scheme of our nonlinear

luminance enhancement stage, which can adaptively

compress the dynamic range. For the local contrast

enhancement, the effect is obvious in the regions where

the luminance distribution is relatively uniform in the

original image.

(a)

(b)

(c)

(d)

Fig.7. Robustness evaluation: (a) original image, (b) MSRCR, (c)

Retinex and (d) the proposed algorithm. Left column: enhanced

results from the original images (enhanced once), Right column:

enhanced results from the left column images (enhanced twice). enhanced results from the left column images (enhanced twice).

Note: This sample image is provided by [10]

Fig.7. Robustness evaluation: (a) original image, (b) MSRCR, (c)

Retinex and (d) the proposed algorithm. Left column: enhanced

results from the original images (enhanced once), Right column:

(a) (b)

Fig.8. Statistical characteristics analysis of the image [Fig. 7(a)]

before and after image enhancement. (a): red curve and blue curve

are the local mean of each block of the original and the enhanced

image, respectively; (b): green curve and black curve are the local

standard deviation of each block of the original and enhanced

image, respectively. image, respectively.

Fig.8. Statistical characteristics analysis of the image [Fig. 7(a)]

before and after image enhancement. (a): red curve and blue curve

are the local mean of each block of the original and the enhanced

image, respectively; (b): green curve and black curve are the local

standard deviation of each block of the original and enhanced

Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)

0-7695-2479-6/05 $20.00 © 2005 IEEE