An Effective Surveillance System Using Thermal Camera

Abstract

Thermography, or thermal visualization is a type of infrared visualization. Thermographic cameras are used in many heavy factories like metal recycling factories, wafer production factories and etc for monitoring the temperature conditions of the machines. Besides, thermographic camera can be used to detect trespassers in environment with poor lighting condition, whereby, the conventional digital cameras are less applicable in. In this paper, we proposed two simple and fast detection algorithms into a cost effective thermal imaging surveillance system. This surveillance system not only used in monitoring the functioning of different machinery and electrical equipments in a factory site, it can also used for detecting the trespassers in poor lighting condition. Experimental results show that the proposed surveillance system achieves high accuracy in monitoring machines conditions and detecting trespassers.

Full text

An Effective Surveillance System Using Thermal Camera
Wai Kit Wong, Poi Ngee Tan, Chu Kiong Loo and Way Soong Lim
Faculty of Engineering and Technology, Multimedia University, 75450 Jln Ayer Keroh Lama, Malaysia.
Email: wkwong@mmu.edu.my, poingee24@yahoo.com, ckloo@mmu.edu.my, wslim@mmu.edu.my.
Abstract- Thermography, or thermal visualization is a type of
infrared visualization. Thermographic cameras are used in
many heavy factories like metal recycling factories, wafer
production factories and etc for monitoring the temperature
conditions of the machines. Besides, thermographic camera
can be used to detect trespassers in environment with poor
lighting condition, whereby, the conventional digital cameras
are less applicable in. In this paper, we proposed two simple
and fast detection algorithms into a cost effective thermal
imaging surveillance system. This surveillance system not
only used in monitoring the functioning of different
machinery and electrical equipments in a factory site, it can
also used for detecting the trespassers in poor lighting
condition. Experimental results show that the proposed
surveillance system achieves high accuracy in monitoring
machines conditions and detecting trespassers.
Keywords- Thermal imaging system, Machine condition
monitoring, Surveillance System, Image Processing &
Understanding.
I. INTRODUCTION
In an automated factory, there are various type of
automated machines that need monitoring. When there is
any malfunctioning of machines, extra heat will be
generated and can be picked up by thermal camera.
Thermal camera will generate an image to indicate the
condition of the machine. This enables the operator to
decide on the on/off switch. Any malfunctioned machines
detected will proceed to further repairmen action. This is
so-called thermal imaging monitoring.
The use of thermal imaging monitoring is much
convenient compared to conventional maintenance method;
the operator needs to perform some hands on job to
measure the functioning machines frequently, which
required more man power and longer maintenance time.
With the aids of thermal imaging monitoring, the operator
can maintain and monitor the machines by just observing
the thermal images on the machines captured routinely and
display on a monitor, even from a remote location. So, this
can reduce hands on workload, man power, maintenance
time and improve safety, since some overheat devices
cannot see through eyes, but can be read from thermal
images, hence the use of thermal imaging monitoring can
prevent accident happen too.
One problem encountered in most surveillance systems
is the change in ambient light, especially in an outdoor
environment where the lighting condition is varies
naturally. This makes the conventional digital color images
analysis task in smart surveillance very difficult. One
common approach to alleviate this problem is to train the
system to compensate for any change in the illumination
[1]. However, this is generally not enough for trespasser
detection in dark. In recent time, thermal camera has been
used for imaging objects in the dark. The camera uses
infra-red (IR) sensors that captures IR radiation coming
from different objects in the surrounding and forms IR
image [2].
Since IR radiation from an object is due to the thermal
radiation, and not the light reflected from the object, such
camera can be conveniently used for trespasser detection in
night vision too.
In this paper, two simple and fast detection algorithms
are embedded into a cost effective thermal imaging
surveillance system. This surveillance system is not only
used for monitoring the functioning condition of different
machines in a factory site, but can also use for detecting
the trespassers in a poor lighting condition. Experimental
results show that the proposed surveillance system
achieves high accuracy in monitoring machines conditions
and detecting trespassers. The paper is organized in the
following way: Section II will be briefly comments on the
thermal imaging surveillance system. Section III presents
the proposed algorithms for machine condition monitoring
and trespasser detection, section IV reports some
experimental results. Finally in section V, we draw some
conclusion and envision future developments.
II. THERMAL IMAGING SURVEILLANCE SYSTEM
MODEL
The thermal imaging surveillance system model proposed
in this paper is shown in Fig.1. The system is very simple,
it required a fine resolution thermal camera and a
laptop/PC with Matlab ver 7.0 programming.
Fig. 1. Thermal Imaging Surveillance System Model
1. Thermal Camera
The thermal camera used in this paper is a low cost and
fine resolution model: AXT100 manufactured by ANN
ARBOR SENSOR SYSTEM [3]. The thermal camera cost
around USD $5000 and it can capture thermal image with
resolution up to 256 X 248 pixels. The price of the thermal
camera is considered cheap as compare to some other
brand of thermal cameras (FLIR [4], SATIR [5], FLUKE
[6], etc) with the same resolution output. Besides, AXT100
also has some advanced signal processing features such as
linear or logarithmic scaling and false color, atmospheric
correction, faster scaling and sampling rate. The digital
control is also accomplish through the 10/100 Ethernet port
connected to a laptop or PC via an embedded firmware,
named Internalweb, which can be interface with Matlab [3].
Therefore, it is best outfitted for machine condition
monitoring and surveillance system.
2. Laptop/PC
A laptop or PC can be used for image processor, either
place on site or in a monitoring room. Matlab ver 7.0
programming is chosen to be used because it has user
friendly interface software with the AXT100 thermal
Capture thermal
image
Thermal
camera
Laptop/
PC
Process images and
signal alarm

camera, it can partitioned the images captured by the
thermal camera easily according to each single machine to
be monitored, process captured thermal images smoothly
with the algorithm we programmed in, store the suspected
images into a compact file easily (with time and date
recorded, 9 suspected images combined into 1 single image
in sub-plot form) and alarm operator with self recorded
sound (e.g. : “machine A overheat”, “machine B and
machine C overheat”, “Trespasser intruding”, etc)
III. ALGORITHM FOR MACHINE CONDITION
MONITORING AND TRESPASSER DETECTION
In this paper, we propose two simple and effective
algorithms for machine condition monitoring and
trespasser detection for the thermal imaging surveillance
system
a) Algorithm for machine condition monitoring
The algorithm for machine condition monitoring is
summarized as below:
Step 1: Acquire image from thermal camera into
laptop.
Step 2: Partition the region of interest (ROI) of the
image into n sections horizontally where n =
number of machines.
Step 3: Define ):,:( )max()min()max()min( nnnnn yyxxm =be the
range of the partition, separate the ROI of the
image into
):,
)(
1( maxmin
max
max
)1max( yy
n
xn
xm nn += − (1)
An example of partitioning of ROI for machine
condition monitoring system is shown in Fig. 2
with )229:1,83:1(
1=m,
)229:1,166:84(
2=m,)229:1,249:167(
3=m
Fig. 2: Partitioning of ROI for machine condition monitoring system.
Step 4: Calculate the sum of RGB elements for each pixel
(x, y) using:
),(),(),(),( yxyxyxyx BGRT ++= (2)
where ),( yx
Ris the value of red element of pixel
(x,y); ),( yx
Gis the value of green element of pixel
(x,y) and ),( yx
B is the value of blue element of
pixel(x,y). Pixel with higher temperature will give
higher ),( yx
T value. For example, at a reference point
as shown in Fig.3, 620111255254
)262,40( =++=T.
Fig. 3: Calculating value of T of reference point
Step 5: Define an overheat threshold value, ),( rr yx
T .
),( rr yx
Tis a value with sum of ),( yx
R,),( yx
G and
),( yx
B equal to a predefined overheat color tone
value, For example, if machines exceed 55.7°C is
consider overheat, then ),( rr yx
T is the total sum of
),( yx
R,),( yx
G and ),( yx
B for the color tone value as
shown in reference point in Fig. 3.
Step 6: Compare ),( yx
T with ),( rr yx
T in each machine’s
section. If ),( yx
T ≥),( rr yx
T , then overheat take place
at pixel (x,y), else if ),( yx
T < ),( rr yx
T , then no
overheat take place at pixel (x,y). A variable h is
used to gather the number of pixels in concern:
⎪
⎩
⎪
⎨
⎧
=→<
=→≥
0
1
),(
),(
),(
),(
i
yx
yx
i
yx
yx
hTT
hTT
for
rr
rr (3)
where i as the sequence number of scanned pixel.
Step 7: Define the minimum overheat size of a machine, S.
If total overheated pixels in a section are more
than S, then the machine in that particular section
is said to be overheated, else the machine in that
section is consider function in normal condition.
⎪
⎪
⎩
⎪
⎪
⎨
⎧
→<
→≥
∑
∑
=
=
overheatnotMachineSh
overheatMachineSh
for
pixeltotal
i
i
pixeltotal
i
i
1
1 (4)
b) Algorithm for trespasser detection
The algorithm for trespasser detection is summarizing
as below:

Step 1: Adjust the thermal camera detection range to
30˚C to 40˚C so that object with human body
temperature range can be detected.
Step 2: Capture images continuously from thermal
camera into laptop and names it as x
Pwhere
x = 1, 2, 3…is the discrete time instant.
Step 3: Divide each image captured from thermal
camera into (m x n) regions. Each region
consists of equal number of pixels.
Step 4: Define a matrix, M with size of (m x n) to
represent the characteristic of each
corresponding region.
Step 5: Define a threshold value Q. Q is the threshold
value of the difference between sum of R, G,
B value for a particular current image pixel to
previous image pixel.
Step 6: Define a variable, h for counting the number
of pixels exceeding Q. Initially, h is set to 0.
Step 7: Define H as a minimum number of pixels with
difference exceeding Q.
Step 8: Compare current taken image x
Pwith previous
taken image 1−x
P. For each corresponding
region, find out the difference between a
particular current image and previous image
pixels’ sum of R, G, B value. If the difference
between sum of R, G, B value for a particular
current image pixel to previous image pixel
≥ Q, then h = h + 1. If h ≥ H, mark a “1”
into the corresponding element of M, else if h
< H, mark a “0” into the corresponding
element of M. An example is shown in Fig. 4.
Step 9: Let F be the number of different elements
which align vertically and continuously.
Some examples of calculation of F are shown
below.
Examples:
E.g. (1)
F = 4 in this example.
E.g. (2)
F = 3 in this example because only 3
different elements are aligned vertically and
continuously.
E.g. (3)
If there are more than 2 groups of vertically and
continuously different elements, then we will take
the largest number. In this case, F = 3
Step 10: Define G as minimum regions that a human being
will appear on screen. If F ≥ G, then alarm
unknown trespasser detected.
Fig. 4: Partitioning of ROI for trespasser detection surveillance system.
IV. EXPERIMENTAL RESULT
In this section, we briefly illustrate the application of
the proposed thermal imaging surveillance system for
machine monitoring and trespasser detection. As for
machine condition monitoring part, we select a three
machines case for studies. The Thermal images for the
functioning machines are collected at the Applied
Mechanics Lab in Faculty of Engineering and Technology,
Multimedia University.
Fig. 5: Case studies of machines for monitoring captured in Applied
Mechanical Lab (Digital Color Form).
Fig. 6: Case studies for machines for monitoring captured in Applied
Mechanics Lab (Thermal image, all machines are functioning in overheat
condition)
An image captured by using digital camera on the site is
shown in Fig 5. A thermal image is also captured by using
thermal camera on the site based on three machines are

functioning in overheat condition, as shown in Fig 6.
Machine A (leftmost) and Machine C (rightmost) are vibro
test machines with same model and same specs, where as
Machine B (center) is a fatigue test machine with cooling
system. The motors of machine A, B and C are considered
to be overheated when it reaches 90˚C. Hence we set the
temperature measurement range on thermal camera from
80˚C to 90˚C. The temperature level display on captured
thermal images are with different color tones ranging from
black, brown, dark red, red, orange, yellow, light yellow to
white represents each step size of temperature range
display on the thermal camera. The minimum size of the
machines appear in the image is approximately1400 pixels.
Hence, we set S = 1000.
The possible machines condition can be divided into 8
major classes, namely: All the machines function properly
(none of the machines overheat); Machine A overheat;
Machine B overheat; Machine C overheat; Machine A and
B overheat; Machine B and C overheat; Machine A and C
overheat; Machine A, B and C overheat.
The algorithm for machine condition monitoring was
evaluated with respect to the thermal images captured live
and displayed on monitor screen as interpreted by an
operator (human observer) the overall description of which
could be called the “Operator Perceived Activity” (OPA)
[7]. The operator will comments on the images captured by
the thermal camera, whether the observed particular
machine is overheated or not and compare with that
detected by the surveillance captured thermal images.
From the total of 10,000 captured images, 9635 images
were tracked perfectly (machines conditions agreed by
both observer and surveillance system), that is an accuracy
of 96.35%.
As for trespasser detection part, the temperature range
of the thermal camera is switched to human body
temperature range, i.e. from 30˚C to 40˚C. temperature
level display on the captured thermal images are with
different color tones ranging from black, brown, dark
red ,red, orange, yellow, light yellow to white represents
each step size of temperature range. The reason that we
select 30˚C to 40˚C range and not a smaller range (35˚C to
40˚C) is because sometimes the trespasser is wearing thick
cover (e.g. cap, thick jacket and thick jean) so that
temperature released is a little bit lower than normal. So,
we extend the range to 5˚C more below normal range,
which is 30˚C to 40˚C. However, if a bigger temperature
range is used, let say, 20˚C to 40˚C, then more noise and
distortion are included into the captured thermal images.
This is because the temperature range (20˚C to 40˚C) is fall
into the room temperature range, in which undesired IR
signals (sunlight, fluorescent tube light, etc) are also
absorbed into the captured thermal images. The
performance can be seen and by comparing Fig 7 and Fig 8.
We can see that there are a lot of noise and distortions
appears in Fig 8 due to undesired IR signals. Therefore, we
can conclude that the optimal temperature range to
measure human being trespasser in the surveillance system
is 30˚C to 40˚C.
.
Fig. 7: Thermal image with temperature range 30˚C to 40˚C used for
detecting trespasser.
Fig. 8: Thermal image with temperature range 20˚C to 40˚C used for
detecting trespasser.
In algorithm for trespasser detection, there are three
parameters need to be optimized which are Q, H and G,
where Q is the threshold value of the difference between
sum of R, G, B value for a particular current image pixel to
previous image pixel, H is minimum number of pixels with
difference exceeding Q, and G is minimum regions that a
human being will appear on screen.
Since the image captured by thermal camera is in RGB
form therefore the difference of sum of RGB values
between a particular current image pixel to previous image
pixel is in between 0 to 765. For Q parameter, 1000 sample
images (with or without human being) are used to test for
every difference point with step size of 15. The accuracy
vs difference of sum of RGB values is plotted as Fig 9.
From the plot, the optimum Q value is 345 with highest
accuracy of 95.30%.
Fig. 9: Accuracy vs difference of sum of RGB values

We partition the captured thermal image into 20 regions
(m =5, n = 4) with each region consists of equal number of
pixels (3038). As for H value, we tried the algorithm with
pets (hamster, cat, dog) and human, moving toward and
away from the captured region. 1000 sample images are
captured. By using the sample images, we repeated the
simulation with H = 10, 20, 30 …100% of number of
pixels difference to total pixels in one region ratio. The
graph Accuracy vs. Number of Pixel Difference to Total
Pixels in One Region Ratio is plotted in Fig. 10. From the
plot, the optimum H value is 50% of total pixels in a
region, with the highest accuracy of 97%.
Fig. 10: Accuracy vs. Number of Pixel Difference to Total Pixels in One
Region
As for G value, we tested the algorithm with human
moving toward and away from the captured region with
minimum regions that a human being will appear on screen,
G = 1, 2, 3, 4 and 5. The graph of accuracy vs minimum
regions that a human being will appear on screen G is
shown in Fig. 11. From the graph, the optimum G value is
3 with highest accuracy of 93.5%.
Fig. 11: Accuracy vs minimum regions that a human being will appear on
screen
For testing the trespasser detection performance of our
surveillance system, a total of 1000 thermal images are
captured as samples. This includes thermal images with a
single trespasser, more than one trespasser without
trespasser and animals (cats, birds etc which are not
counted as trespasser). The OPA [7] is used and the
operator will comments on the images captured, whether
there is any trespasser or not and compare with that
detected result of the surveillance system. From the total of
1000 samples images for evaluation, 838 were tracked
perfectly (trespasser-or-not condition agree by both
observer and surveillance system), that is with an accuracy
of 83.8%. The surveillance system are also function in a
fast way whereby the routine time required to capture in a
thermal image, detect the machines condition/trespasser
until the signal alarm or not is only 2.5 seconds.
V. CONCLUSION
In this paper, we proposed two simple and fast detection
algorithms, and embedded them into a cost effective
thermal imaging surveillance system. This surveillance
system is not only used for monitoring the functioning
condition of different machines in a factory site, but can
also use for detecting the trespassers in a poor lighting
condition. The experimental results show that the proposed
surveillance system achieves high accuracy in monitoring
machines conditions and detecting trespassers. In future, an
automatic power supply control system will be added to
the machine condition monitoring system. When the
monitoring system detected any of the machines
overheated, the automatic power supply control system
will cut off the power supply of the respective machine(s).
This enhancement can reduce hands on workload, man
power and maintenance time.
REFERENCES
[1] C.lu, and M.S Drew, “Automatic Compensation for camera Settings
for Images Taken Under Different Illuminants”, technical paper,
School of Computer Science, Simon Fraser University, Vancouver,
British, Columbia, Canada, p.p.1-5, 2007
[2] Thermographic camera
Retrieved August 18, 2008, from Wikipedia, the free encyclopaedia
Web Site: http://en.wikipedia.org/wiki/Thermal_camera
[3] AXT100 Fixed-Mount Thermal Infrared Imaging Camera
Retrieved July 07, 2008, from Ann Arbor Sensor Systems, LLC
Web Site: http://www.aas2.com/products/axt100/
[4] http://www.flirthemography.com
[5] Retrieved August 19, 2008, from Guangzhou SAT Infrared
Technology Co. LTD(SATIR)
Web Site: http://www.sat.com.cn/english/index.php
[6] Fluke Thermal Imaging
Retrieved July 30, 2008 from Pyrometer Systems Ltd.
Web Site: http://www.pyrometer.co.uk/id9.html
[7] J. Owens, A. Hunter and E. Fletcher, “A Fast Model–Free
Morphology–Based Object Tracking Algorithm”, British Machine
Vision Conference, p.p. 767-776, 2002