An illuminancereflectance nonlinear video enhancement model for homeland security applications
ABSTRACT A illuminancereflectance model based video stream enhancement algorithm is proposed for improving the visual quality of digital video streams captured by surveillance camera under insufficient and/or nonuniform lighting conditions. The paper presents computational methods for estimation of scene illuminance and reflectance, adaptive dynamic range compression of illuminance, and adaptive enhancement for midtone frequency components. The images are processed in a similar way as human eyes sensing a scene. The algorithm demonstrates high quality of enhanced images, robust performance and fast processing speed. Compared with Retinex and multiscale retinex with color restoration (MSRCR), the proposed method shows a better balance between luminance enhancement and contrast enhancement as well as a more consistent and reliable color rendition without introducing incorrect colors. This is an effective technique for image enhancement with simple computational procedures, which makes realtime enhancement for homeland security application successfully realized. The application of this image enhancement technique to the FRGC images yields improved face recognition results

Conference Paper: An automatic waveletbased nonlinear image enhancement technique for aerial imagery
[Show abstract] [Hide abstract]
ABSTRACT: Recently we proposed a waveletbased dynamic range compression algorithm to improve the visual quality of digital images captured in the high dynamic range scenes with nonuniform lighting conditions. The fast image enhancement algorithm which provides dynamic range compression preserving the local contrast and tonal rendition is a very good candidate in aerial imagery applications such as image interpretation for defense and security tasks. This algorithm can further be applied to video streaming for aviation safety. In this paper the latest version of the proposed algorithm which is able to enhance aerial images so that the enhanced images are better then direct human observation, is presented. The results obtained by applying the algorithm to numerous aerial images show strong robustness and high image quality.Recent Advances in Space Technologies, 2009. RAST '09. 4th International Conference on; 07/2009 
Conference Paper: Research of Algorithm of Mining Frequent Closed Pattern about Distributed Data Stream
[Show abstract] [Hide abstract]
ABSTRACT: New algorithm of mining frequent closed pattern about distributed data stream was proposed. The algorithm bases on the threelayer structure model. First, the leaflayer nodes overlap the collecting itemsets. Second, the middle Alayer nodes expurgate the infrequent items of overlapped items again and again, and then insert them into prefixsubtree. Third, the toplayer nodes and the middlelayer ones answer the clients' request online in feedback way. Analysis and experimental results indicate: This algorithm really resolves the heavy workload between layers and has favorable application prospect.Information Engineering and Computer Science (ICIECS), 2010 2nd International Conference on; 01/2010  SourceAvailable from: Ahmet Gungor Pakfiliz[Show abstract] [Hide abstract]
ABSTRACT: Recently, we proposed an enhancement technique for uniformly and nonuniformly illuminated dark images that provides high color accuracy and better balance between the luminance and the contrast in images to improve the visual representations of digital images. In this paper we define an improved version of the proposed algorithm to enhance aerial images in order to reduce the gap between direct observation of a scene and its recorded image.
Page 1
An IlluminanceReflectance Nonlinear Video Enhancement Model
for Homeland Security Applications
Li Tao, Richard Tompkins, and Vijayan K. Asari
Department of Electrical and Computer Engineering
Old Dominion University, Norfolk, VA 23529
Email: {ltaox001, rtomp002, vasari} @odu.edu
ABSTRACT
enhancement algorithm is proposed for improving the
visual quality of digital video streams captured by
surveillance camera under insufficient and/or nonuniform
lighting conditions. The paper presents computational
methods for estimation of scene illuminance and
reflectance, adaptive dynamic range compression of
illuminance, and adaptive enhancement for midtone
frequency components. The images are processed in a
similar way as human eyes sensing a scene. The algorithm
demonstrates high quality of enhanced images, robust
performance and fast processing speed. Compared with
Retinex and multiscale retinex with color restoration
(MSRCR), the proposed method shows a better balance
between luminance enhancement
enhancement as well as a more consistent and reliable
color rendition without introducing incorrect colors. This
is an effective technique for image enhancement with
simple computational procedures, which makes realtime
enhancement for homeland
successfully realized. The application of this image
enhancement technique to the FRGC images yields
improved face recognition results.
A illuminancereflectance model based video stream
and contrast
security application
1. Introduction
When human eyes sense a scene, the large dynamic
range of the radiance from the scene can be well perceived
through a series of adaptive mechanisms for brightness
perception and processing. Some major dynamic
compression mechanisms are based on the lateral
processing at the retinal level, and some occur in the early
visual cortex [1]. Unfortunately, current cameras and
image display devices do not have similar sophisticated
features. Saturation and underexposures are common in
images due to limited dynamic range of the imaging and
display equipment. This problem becomes more common
and severe when insufficient or nonuniform illumination
occurs.
However, the conventional image enhancement
techniques, like the global brightness and contrast
enhancement, gamma
compression and histogram equalization, are generally
incapable of providing satisfactory enhancement results
for those images having shadows which are created due to
the problem discussed in last paragraph. In order to
improve the quality of images captured under complex
lighting conditions, some advanced image enhancement
techniques have been developed, such as Retinex [2],
multiscale retinex (MSR) [3,4], luma dependent nonlinear
enhancement (LDNE) [5], and adaptive histogram
equalization [6]. Among them, Retinex based methods are
effective in dynamic range compression and local contrast
enhancement, which are derived from Land’s “Retinex”
theory [7] that deals with human visual perception of
lightness and color. However, both MSR and standard
Retinex may produce strong “halo” effect and incorrect
colors on enhanced images due to their separate nonlinear
processing of the three spectral bands, which will be
avoided in the proposed algorithm. In addition, both
methods are also computationally inefficient with the
threeband processing. Therefore, it has to rely on
hardware implementation to achieve real time processing
of video sequences [8].
In this paper, we present a new image enhancement
algorithm based on illuminance perception and processing
to achieve dynamic range compression while retaining or
even enhancing visually important features. Due to its
simplicity of our algorithm, real time processing of color
video streams has been implemented in software on
personal computer for homeland security applications.
Our algorithm is based on some assumptions about
image formation and human vision behavior. First, the
image intensity I(x, y) can be simplified and formulated as
a product [9]:
adjustment, logarithmic
( , )
I x y
( , ) ( , )
L x y R x y
?
(1)
where R(x, y) is the reflectance and L(x, y) is the
illuminance at each point (x, y). Second, the luminance L
Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)
0769524796/05 $20.00 © 2005 IEEE
Page 2
is assumed to be containing in the low frequency
component of the image while the reflectance R mainly
includes the high frequency component of the image. This
assumption can be easily understood if considering that R
generally varies much faster than L does in most parts of a
image with a few exceptions, like shadow boundaries
where large L changes exist. In addition, in a real world
scene, the dynamic range of the illumination variation can
be several orders larger than the dynamic range of the
reflectance. Therefore, compressing the dynamic range of
the illuminance is an effective way for image enhancement.
Finally, because human eyes responds to local changes in
contrast rather than to global brightness levels, it is
possible to keep the visually important features
represented by reflectance while compressing the image’s
dynamic range mainly induced by illuminance.
2. Algorithm
The proposed image enhancement algorithm is
composed of four major steps: (1) illuminance
estimation and reflectance extraction; (2) adaptive
dynamic range compression of illuminance; (3) adaptive
midtone frequency components enhancement; (4) image
restoration, which combines illuminance and reflectance to
recover the intensity image and then performing color
recovery.
2.1. Illuminance Estimation
Accurate estimation of illuminance of a scene from an
image is a difficult task. Many techniques have developed
to deal with this problem [5,8,9]. In our algorithm, the
lowpass filtered result of the intensity image through
2D discrete Gaussian filtering is used as the estimated
illuminance by:
? ?
??
00
mn
where L is the illuminance, I is the original intensity image
and G is the 2D Gaussian function with size M ? N. G is
defined as:
?
x
?
?
?
?
where q is determined for normalizing the Gaussian
function by
?
2
c
q edxdy
?
??
and c is the scale to determines the size of the
neighborhood. In our algorithm, c = 2~5. Then,
illuminance is normalized as:
??
???
11
),(),(),(
MN
ynxmGnmIyxL
(2)
?
22
2
( , )
G x y
.
y
c
q e
??
?
?
?
??
(3)
?
22
1
xy
?
?
?
?
?
?
?
?
??
?
(4)
( , )
L x y
( , )
L x y ?
255
n
(5)
For color images, the intensity image I(x,y) is obtained
using the following method:
?
),,( max),(
yxryxI
?
? )
y
,(), ,(
xbyxg
(6)
where r, g and b are the RGB components of color
images in RGB color space. We process on the intensity
image instead of processing all color bands separately
because the latter way is more time consuming and not
found significantly superior to intensity processing in
terms of the quality of enhanced images. This method is in
fact the definition of the value component in HSV color
space. We didn’t use NTSC to get the intensity image
because NTSC generally produces inconsistent or shifted
colors like the red colors in the enhanced image after
linear color restoration compared with the original image.
Once the illuminance L is obtained using Eq. (2), the
reflectance R is computed using Eq. (1). One example
showing the results of L and R from an image is provided
in Figs. 2(b) and 2(c). It can be observed that the
illuminance L comprises the midtone and lowfrequency
information of the image. This is not the real illuminance
as defined in physics but an approximated illuminance that
contains both the illuminance and the midtone and
lowfrequency components of reflectance. The visually
important features (high frequency reflectance) and a
small part of illumination information are contained in the
reflectance in which the major illumination effect is
removed. Therefore, important image features will be kept
after the dynamic range compression of illuminance.
Based on those observations, reflectance and illuminance
were also regarded as details and base [9].
2.2 Adaptive Dynamic Range Compression of
Illuminance
The dynamic range compression of illuminance realized in
our algorithm uses the proposed Windowed Inverse
Sigmoid (WIS) function. The sigmoid function is defined
as:
1
( )
1
e?
?
This function can also be used as the intensity transfer
function for dynamic range compression by performing
the computational steps described by Eqs. (8)  (10).
av
f v
?
(7)
maxmin min
'[ ()( )]()
nn
L L f vf v f v
???
(8)
11
''ln1
'
n
n
L
aL
?
?
?
?
?
''
?
?
?
??
(9)
min
v
,
maxmin
n
n enh
L
Lv
v
?
(10)
where Eq. (8) is used for linearly mapping the input range
[0 1] (normalized illuminance) to the magnitude range
Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)
0769524796/05 $20.00 © 2005 IEEE
Page 3
[f(vmin) f(vmax)] for WIS. Eq. (9) is the inverse sigmoid
function. Eq. 10 is applied to normalize the output
illuminance to [0 1]. Parameters vmax, and vmin are used to
change the line shape of the transfer function.
Fig.1. WIS is used for Intensity transformation. Fig.1. WIS is used for Intensity transformation.
A set of the curve shapes of WIS transfer function is
provided in Fig. 1 with a = 1 and vmax = 3. Those curves
are produced by Eqs. (8)  (10) with different parameters:
vmin= 6 for the dotted red curve; vmin = 3 for the dashed
red curve; and vmin = 4.5 for the solid red curve. Identity
transfer function is also provided for comparison.
Obviously, the inverse sigmoid function can also be used
to pull down the intensity of the overlylighted pixels.
The value of vmax is always set to 3 for all images while
the value for vmin is image dependent and determined by
the global mean Im of the intensity image I to obtain
selfadaptive control over the dynamic range compression.
Based on the image enhancement experiments, the value
for vmin is determined by Imas:
min
670
70
3 6
? ?
70 150
80
3 150
m
m
m
m
for I
I
forI
for I
?
??
?
?
?
?
??
??
?
???
?
(11)
(a) (b) (c)
(d) (e) (f)
Fig.2. (a) original image; (b) estimated illuminance; (c) reflectance;
(d) dynamic range compressed illuminance; (e) midtone
frequency components enhancement from (d); (f) enhanced image. frequency components enhancement from (d); (f) enhanced image.
Fig.2. (a) original image; (b) estimated illuminance; (c) reflectance;
(d) dynamic range compressed illuminance; (e) midtone
2.3. Adaptive Midtone Frequency Components
Enhancement
It has been noted that the illuminance also contains the
midtone and low frequency components of reflectance
which has been degraded during the dynamic range
compression. For the original image with low contrast or
slowvarying reflectance map, the degradation of midtone
frequency features may be rather obvious in the output
images. Therefore, a centersurround type of contrast
enhancement method is utilized to compensate this
degradation. This midtone frequencies enhancement is
carried out as defined in the following two equations:
(,)
,,
'( ,
x y
) ( ,
x y
)E x y
n enhn enh
LL
?
(12)
where the exponent is defined by:
( , )
( , )
I x y
( , )
E x y
( , )
R x y
P
P
conv
I x y
?
?
?
?
?
? ??
(13)
L’n,enh(x, y) is the illuminance after midtone frequencies
enhancement and R(x, y) is the ratio between I(x, y) and its
lowpass version Iconv(x, y) which is computed through the
same operations as in Eqs. (2) – (4) with a larger scale c
(10 ? 10, lower cutoff frequency). P is determined by the
global standard deviation ? of the input intensity image I(x,
y) as:
230
for
?
?
?
?
?
0.03 2.930 80
1/2 80
Pfor
for
?
?
?
?
?
?
? ????
(14)
According to this definition, P and ? have a linear
relationship within the range ??(30, 80]. This relationship
is determined based on huge number of image
enhancement experiments. Here, the global standard
deviation ? of I(x, y) is considered as an indication of the
contrast level of the original intensity image. It should be
noted that P can also be changed by users to manually
adjust the contrast enhancement process.
The midtone frequency components enhancement
process defined in Eqs. (12) and (13) is actually an
intensity transformation process and can be understood
using Fig. 3.
0 0.10.2 0.30.40.5
In'(x,y)
0.60.7 0.80.91
0
0.2
0.4
0.6
0.8
1
In'(x,y)E(x,y)
E=2
E=3
E=1/2
E=1/3
E=1
Fig.3. Intensity transformation for midtone frequencies
enhancement for illuminance enhancement for illuminance
Fig.3. Intensity transformation for midtone frequencies
Since Ln,enh is normalized to 1 as defined before,
Ln,enh (x, y)E(x, y) will be larger than Ln,enh (x, y) if E(x, y) is
less than 1 (e.g. the center pixel is brighter than
Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)
0769524796/05 $20.00 © 2005 IEEE
Page 4
surrounding pixels leading to R(x, y) < 1), and vise versa.
In this way, the contrast of the compressed illuminance
image can be improved. Here, the ratio R(x, y) is obtained
from the original intensity image I(x, y) and its lowpass
filtered result Iconv (x, y), since the reflectance’s midtone
and low frequency information has been lost and degraded
in the estimated illuminance L.
The parameters of P and vmin are introduced to this
algorithm to increase the flexibility of this algorithm.
Meanwhile, they can also improve the robustness of the
enhancement performance to produce consistent good
results for images captured under various types of lighting
conditions.
2.4. Image Restoration and Color Recovery
Following the dynamic range compression and contrast
enhancement, the final illuminance L’n,enh and reflectance
R are combined using Eq. (1) to produce an intensity
image I’ with compressed dynamic range (see Fig. 2(f)).
For color images, a linear color restoration process based
on the chromatic information of the original image is
applied to I’ to recover the RGB color bands (r’, g’, b’) as:
''
''
rrgg
II
'
'
III
I
bb
???
(15)
so the color information (hue and saturation) in the
original image is preserved in the enhanced image.
3. Algorithm Implementation for Realtime
Homeland Security Application
A software package named IESuite has been developed
and implemented on desktop/laptop PCs for realtime
image enhancement using C++ in Windows XP
environment on a PC with a 3.2GHz Intel Pentium 4
processor and 1GB DDR SDRAM memory. Sony
DCRHC85 digital video camera is used to capture the
video stream with a frame size of 360 ? 240 pixels. A
processing speed of 12 frames per second has been
achieved. Fig. 4 shows a screen capture of the interface of
the video enhancement program. The digital video stream
is transferred from video camera into computer via IEEE
1394 ports. In order to achieve optimized computing speed
for realtime applications, the large scale convolutions are
computed in frequency domain with Fast Fourier
Transform (FFT) while small scale convolutions are still
performed in spatial domain.
The video stream enhancement can be controlled
with playback commands, and to ‘snapshot’ any enhanced
frame is permitted. The enhanced digital video stream can
be recorded by multiclips at anytime when the user is
interested in the scene, and all the clips finally are
automatically saved into one AVI video file with a
selected encoder/compressor. In addition, the recording
rate can be manually set.
Fig.4. Screen capture of realtime digital video enhancement by
our developed enhancement software package ‘IESuite’. our developed enhancement software package ‘IESuite’.
Fig.4. Screen capture of realtime digital video enhancement by
4. Results and Discussion
In this section, several important features of the
algorithm will be first discussed with experimental results.
Then the performance and robustness of the algorithm will
be evaluated by comparing with MSRCR and Retinex.
Finally, a statistic analysis method will be introduced to
illustrate the image enhancement effects of the algorithm
4.1. Adaptive Dynamic Range Compression
The curve shape of WIS can directly affect the
dynamic range compression of the illuminance. From Fig.
1, it can be easily understood how the curve shape can be
changed by adjusting the value for vmin which can be
manually tuned or image dependent for adaptive control.
The effect of vmin is illustrated in Fig. 5 with a sample
image treated with various vmin values, Fig. 5(c) is set by
Eq. (11). Obviously, the vmin value set by Eq. (11)
produces better result (Fig.5(c)) than manual adjustment
results (Figs. 5(b) and 5(d)) without over or under
enhancement of brightness.
(a) (b) (c) (d)
Fig.5. Image enhancement with different Fig.5. Image enhancement with different ? ?min
enhanced image with enhanced image with ? ?min
= 3; (c) enhanced image with ? ?min
set by Eq. (11); (d) enhanced image with set by Eq. (11); (d) enhanced image with ? ?min
Note: This sample image is provided by [10]
min: (a) original image; (b) : (a) original image; (b)
min = 3; (c) enhanced image with
min = 4.9, = 4.9,
min = 7. = 7.
4.2. Adaptive Midtone Frequency Enhancement
The parameter P in Eq. (14) is used to tune the
enhancement for midtone frequency components to
improve the image contrast which is poor in the original
image or has been significantly degraded due to dynamic
range compression. The effect of P on image enhancement
is illustrated in Fig. 6 with the sample image processed
Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)
0769524796/05 $20.00 © 2005 IEEE
Page 5
with various P values. Fig.6(c) is determined by Eq. (14),
which produces a more appropriate result compared to the
other two enhanced images with manually set P values.
For example, the wood grains are well enhanced without
producing severe halo effect, that is obvious in Fig.6(d).
(a) (b) (c) (d)
Fig.6. Image enhancement with different Fig.6. Image enhancement with different P P: (a) original image; (b)
enhanced image with enhanced image with P P = 1/2; (c) enhanced image with
= 1/2; (c) enhanced image with P P = 1, set
by Eq. (14); (d) enhanced image with by Eq. (14); (d) enhanced image with P P = 2.
: (a) original image; (b)
= 1, set
= 2.
4.3. Algorithm Robustness Evaluation
To test the performance and robustness of the proposed
algorithm, it is applied to images twice: the first
enhancement processing is carried out on the original
image, then the second enhancement processing is
performed on the output image from the first enhancement.
An example of this process is presented in Fig. 7
accompanied by the results produced by MSRCR and
Retinex. Retinex is realized using the Matlab? code
provided in Reference [2] by Frankle and McCann. A
commercial digital image
PhotoFlair® version 2.0 (TruView Imaging Company) is
used to implement MSRCR algorithm. It can be observed
that the images processed by our method demonstrate a
higher visual quality than those processed by MSRCR and
Retinex. Our method yields better color accuracy and
better balance between the luminance and contrast across
the whole image due to its adaptiveness and flexibility
involved in the processing. Obviously, after twice
enhancement, the proposed algorithm produces the
minimal change from the first enhancement result while
image quality is degraded in images produced by MSRCR
and Retinex, in which incorrect lightness, color rendition,
halo effect, and image noise become much more visible
after the second enhancement.
processing software
4.5. Statistical Analysis
The statistical properties of images, image mean and the
zonal standard deviation, are used to assess the visual
quality of images in terms of brightness and contrast
which are directly associated with those statistical
parameters. The local brightness is measured by the image
local mean while the local contrast is evaluated by taking
the regional standard deviations. As shown in Fig. 8(a), we
plotted the local mean and local standard deviations of all
blocks (block size: 55 ? 55 pixels) of a sample image (Fig.
7(a)) before and after image enhancement. It can be seen
that in some regions of the sample image, the luminance
enhancement is dramatic (eg. the shadows of the wall and
the lady) but in some regions it is almost the same as the
original luminance (eg. the regions of the window). This is
exactly matched with the scheme of our nonlinear
luminance enhancement stage, which can adaptively
compress the dynamic range. For the local contrast
enhancement, the effect is obvious in the regions where
the luminance distribution is relatively uniform in the
original image.
(a)
(b)
(c)
(d)
Fig.7. Robustness evaluation: (a) original image, (b) MSRCR, (c)
Retinex and (d) the proposed algorithm. Left column: enhanced
results from the original images (enhanced once), Right column:
enhanced results from the left column images (enhanced twice). enhanced results from the left column images (enhanced twice).
Note: This sample image is provided by [10]
Fig.7. Robustness evaluation: (a) original image, (b) MSRCR, (c)
Retinex and (d) the proposed algorithm. Left column: enhanced
results from the original images (enhanced once), Right column:
(a) (b)
Fig.8. Statistical characteristics analysis of the image [Fig. 7(a)]
before and after image enhancement. (a): red curve and blue curve
are the local mean of each block of the original and the enhanced
image, respectively; (b): green curve and black curve are the local
standard deviation of each block of the original and enhanced
image, respectively. image, respectively.
Fig.8. Statistical characteristics analysis of the image [Fig. 7(a)]
before and after image enhancement. (a): red curve and blue curve
are the local mean of each block of the original and the enhanced
image, respectively; (b): green curve and black curve are the local
standard deviation of each block of the original and enhanced
Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop (AIPR05)
0769524796/05 $20.00 © 2005 IEEE