High Performance Adaptive Fidelity Algorithms for Multi-Modality Optic Nerve Head Image Fusion.

Full text

Page 1

High Performance Adaptive Fidelity Algorithms
for Multi-Modality Optic Nerve Head Image Fusion
Hua Cao & Nathan Brener & Bahram Khoobehi &
S. Sitharama Iyengar
Received: 4 May 2010 /Revised: 5 May 2010 /Accepted: 5 May 2010
# Springer Science+Business Media, LLC 2010
Abstract A high performance adaptive fidelity approach
for multi-modality Optic Nerve Head (ONH) image fusion
is presented. The new image fusion method, which consists
of the Adaptive Fidelity Exploratory Algorithm (AFEA)
and the Heuristic Optimization Algorithm (HOA), is
reliable and time efficient. It has achieved an optimal
fusion result by giving the visualization of fundus image
with a maximum angiogram overlay. Control points are
detected at the vessel bifurcations using the AFEA. Shape
similarity criteria are used to match the control points that
represent same salient features of different images. HOA
adjusts the initial good-guess of control points at the sub-
pixel level in order to maximize the objective function
Mutual-Pixel-Count (MPC). In addition, the performance of
the AFEA and HOA algorithms was compared to the
Centerline Control Point Detection Algorithm, Root Mean
Square Error (RMSE) minimization objective function
employed by the traditional Iterative Closest Point (ICP)
algorithm, Genetic Algorithm, and some other existing
image fusion approaches. The evaluation results strengthen
the AFEA and HOA algorithms in terms of novelty,
automation, accuracy, and efficiency.
Keywords Biomedical imaging.Feature extraction.
Heuristic optimization.Image fusion.Image registration
1 Introduction
Multi-modalityimagefusion,whichusuallyrequiresintensive
computational effort, is a very challenging problem
because of the possible vast content change and non-
uniform distributed intensities of the involved images [1,
2]. In practical clinical applications, automation of image
fusion techniques will reduce the number of images
reviewed by the physicians and speed up the patient care
routine [3]. In the eye clinics, comparison of angiogram
grayscale and fundus true color Optic Nerve Head (ONH)
images (Fig. 1) is often required in order to identify
dynamic aspects of the circulation and evaluate various
ONH vascular disorders. By locking the multi-modality
ONH images into one single volume, the proposed new
algorithms allow ophthalmologists to match the same eye
over time and get a sense of disease progression.
Ultimately, the new approach allows pinpointing the
surgical tools to increase accuracy and overall speed of
the surgery.
1.1 Image Fusion Overview
The most widely used methods for image fusion are
feature-based and area-based. The feature-based method
extracts and matches the common structures (features)
from separate images. The feature refers to the salient
structures, such as the central line of vessels and the
vessel bifurcation points in the ONH network. Mutual
Information (MI) [4–6] incorporated with the gradient
optimization is a frequently used optimization measure-
ment in area-based non-rigid image registration and
fusion. The proposed approach employed the MI concept
and simplified it to Mutual-Pixel-Count (MPC). MPC
measures the overlapping pixels of the ONH vasculature.
H. Cao (*):N. Brener:S. Iyengar
Computer Science Department, Louisiana State University,
Baton Rouge, LA 70802, USA
e-mail: hcao@csc.lsu.edu
B. Khoobehi
Department of Ophthalmology, LSU Eye Center,
New Orleans, LA 70112, USA
J Sign Process Syst
DOI 10.1007/s11265-010-0496-3

Page 2

If the images are geometrically aligned, MPC represents
the maximal pixel alignment. The feature-based method
for feature/control point detection and the area-based
method for the optimization fusion are integrated in the
proposed approach.
The traditional Iterative Closest Point (ICP) method
[7] is often used to reconstruct free-form curves and
surfaces from different raw scans. The ICP algorithm
firstly gives the initial estimation of the transformation,
and then iteratively repeats the transformation estimation
by minimizing the RMSE objective function until certain
termination criteria are met. The initial estimation and
iterative adjustment routine of the new algorithms pre-
sented in this paper is an enhancement over the traditional
ICP. The comparative estimation will be discussed in
Section 4.
The novel ONH image fusion approach made two new
contributions to the medical image fusion area. The new
contributions are the Adaptive Fidelity Exploratory Algo-
rithm (AFEA) for control point detection and Heuristic
Optimization Algorithm (HOA) for adjustment of the
control points’ initial good-guess. Firstly, the image
segmentation extracts ONH vasculature edges using Canny
Edge Detector [8]. Secondly, the control points are detected
at the vessel bifurcations using the AFEA. Thirdly, shape
similarity criteria are used to match the control points that
represent same salient features of different images. Finally,
HOA iteratively adjusts the initial good-guess of control
points at the sub-pixel level in order to maximize the
objective function Mutual-Pixel-Count (MPC). The itera-
tion can be terminated either when MPC reaches the
maximal or when the maximum allowable loop count is
reached.
1.2 Affine Transformation Model
A 2D affine model (Eq. 1) is applied by solving the
Gaussian matrix to get P∈{a1, a2, a3, a4, b1, b2}. Affine
model’s advantage lies in that it can measure lost
information such as skew, translation, rotation, shearing
and scaling that maps finite points to finite points and
parallel lines to parallel lines [9].
2
¼
000x2
y2
x3
y3
100
000x3
y3
u1
v1
u2
v2
u3
v3
6666664
where (x1, y1), (x2, y2) and (x3, y3) are the three control points
from the input image; (u1, v1), (u2, v2) and (u3, v3) are the
corresponding control points from the reference image.
3
7777775
x1
0
x2
y1
0
y2
1
0
1
0
x1
0
0
y1
0
0
1
0
1
0
1
2
6666664
3
7777775
a1
a2
b1
a3
a4
b2
2
6666664
3
7777775
ð1Þ
1.3 Subjects’ Image Acquisition
The ONH and overlying vessels were imaged with a
Topcon TRC-50EX fundus camera (Fig. 2) attached to a
hyperspectral imaging system [10]. The primates were
cynomolgus monkeys of 4–4.5 years of age and 2.5–3 kg
body weight with normal eyes. The experimental monkey
was anesthetized with the intramuscular ketamine (7–
10 mg/kg), xylazine(0.6–1 mg /kg), and intravenous
pentobarbital (25–30 mg/kg) [11]. ONH images were
obtained with inspiration of pure oxygen and room air at
the controlled intraocular pressures of 10 mm Hg sustained
for up to 10 min after the eyes were dilated.
1.4 Image Binarization and Edge Extraction
In the ONH network, objects of interest are the ONH
vessels, i.e. arteries and veins. Firstly, both of the reference
and input images are binarized (Fig. 3 (a) and (b)) using a
clustering threshold developed by Otsu [12]. Otsu’s thresh-
olding, one of the most referenced thresholding methods,
Figure 2 The optical diagram of an ONH hyperspectral imager with a
fundus camera. The region of interest is imaged with a fundus camera
(FC).The intermediate image (IM) is formed at the slit (S) of an
imaging spectrograph (IS). The output spectrum is focused on a CCD
camera (C). Dotted lines are the light collection path. The spectro-
graph and camera are translated following the y axis. Motion is
controlled to create a 1:1 ratio between adjacent pixels in the x
direction and lines in the y direction [10].



(a) (b)
Figure 1 Optic Nerve Head Images. a input angiogram grayscale
image; b reference fundus true color image.
J Sign Process Syst

Page 3

gives satisfactory results when the numbers of pixels in
each class are close to each other [13]. The threshold is a
normalized intensity value that lies in the range [0, 1]. If the
input intensity is less than Otsu’s threshold, the output
binary pixel is marked as 0 (black); otherwise, the output
pixel is marked as 1 (white).
The Canny operator finds edges by looking for local
maxima of the gradient of the input image (Fig. 3 (c) and
(d)). Canny’s method detects the edges at the zero-crossings
of the second directional derivative of the image. It
performs the zero-crossings of
?
It uses two thresholds for detecting strong and weak
edges. The zero-crossings are corresponding to the first
directional-derivative’s maxima and minima in the direction
of the gradient. Each pixel’s edge gradient is computed and
compared with the gradients of its neighbors along with the
d2G ? I
dn2
ðÞ
¼d
dG
dn
?? I
??
dn
ð2Þ
gradient direction. The gradient magnitude at Px+1, y+1and
Px-1, y-1(Fig. 4) can be calculated as:
?
þuy? ux
Pxþ1;yþ1: G Pxþ1;yþ1
?¼u
uyG x þ 1;y þ 1
ðÞ
uy
G x;y þ 1
ðÞð3Þ
Px?1;y?1: G Px?1;y?1
??¼u
uyG x ? 1;y ? 1
þuy? ux
uy
ðÞ
G x;y ? 1
ðÞð4Þ
Canny operator is less likely than the others to be “fooled”
by noise, and more likely to detect true weak edges [8].
2 Control Point Detection
2.1 Problem Formulation
In the ONH network, features are the vessel centerline or
vasculature bifurcations, also known as the control points.
As required by the 2D affine transformation model, three
control points are identified at the reference image and three
corresponding ones are located at the input image (Eq. 5).
?
where, CP1, CP2and CP3are the control points from the
reference image and each of them represent one feature of
the ONH vessels; CP
control points from the input image and each represent
same feature as CP1, CP2, and CP3respectively.
Control points detection is an essential step of an image
fusion approach. Good control point selection will ensure
fused image generated at a short running time. Bad control
point selection will significantly increase the computation
cost, or even cause the image fusion fail. Some particular
vessel abnormalities make images not necessarily matching
the ONH structures. Even when structure and function
correspond, the abnormality still happens if inconsistence
exists between structural and functional changes. Further
CP1;CP2;CP3
ð Þ , CP
0
1;CP
0
2;CP
0
3
?
ð5Þ
0
1, CP
0
2and CP
0
3are the corresponding
Z
Y
X
Y
X
Z
(a) (b)
(c)(d)
(e)(f)
Figure 3 a Binarized input image from the grayscale angiogram
image; b Binarized reference image from the true color fundus image;
c Canny edges of the reference image; d Canny edges of the input
image; e 3D shaded surface plot of the reference image; f 3D shaded
surface plot of the input image. The X-Yaxis of the 3D shaded surface
plot corresponds to the original image size. The height Z axis is a
single-valued function defined over a geometrically rectangular grid. Z
specifies the color data as well as surface height, so that color is
proportional to surface height with range of [0, 1]. It can be observed
from (e) and (f) that all ONH salient features are preserved in the
Canny edges.
Figure 4 Canny Edge Detection—Localization of Maxima.
J Sign Process Syst

Page 4

more, angiogram grayscale images usually have higher
resolution and are rich in information, whereas fundus color
images have lower resolution and are indeed abstract with
some details or even missing some small vessels. Practi-
cally, those situations will create difficulties in extracting
the control points because the delineation of vein bound-
aries may not be precise. The AFEA algorithm presented in
this paper is able to conquer these difficulties and give an
initial good-guess of the control points.
2.2 Eight-Connectivity Chain Code
In the AFEA algorithm, edge pixels are processed by using an
eight-connectivity chain code (Table 1). An edge curve can
be represented by an integer sequence based on the position
of the current edge pixel Nito their eight neighbors at the 2D
spatial domain:
Ni2 1;2;3;4;5;6;7;8
where, numerical digits 1–8 correspond to different angles.
The ONH vessel edges detected through image segmen-
tation present the chain codes in such rule that:
fg
ð6Þ
1) If southeast “5” neighbor is detected, neither east “3”
neighbor nor south “1” neighbor should be detected;
2) If northwest “6” neighbor is detected, neither west “4”
neighbor nor north “2” neighbor should be detected;
3) If southwest “7” neighbor is detected, neither west “4”
neighbor nor south “1” neighbor should be detected;
4) If northeast “8” neighbor is detected, neither east “3”
neighbor nor north “2” neighbor should be detected.
2.3 Adaptive Fidelity Exploratory Algorithm (AFEA)
Generally speaking, there are two broad groupsof vasculature
controlpointprocessmethods.Thefirstgroupiscalled“pixel-
processing approach” [14]. It uses matched filtering, seg-
mentations, thinning, and bifurcation identification by
processing every pixel and imposing numerous operations
at each pixel. The “pixel-processing approach” scales poorly
with large image size and can hardly meet short computation
deadlines.The secondgroupis called“exploratoryalgorithms”
[15, 16]. The AFEA algorithm can be classified into the
second group. In the ONH network, the control points are
selected at the vessel bifurcations on Canny edges. A vessel
tracking is efficiently implemented on Canny edges without
traveling at every pixel.
The AFEA traces the image contour by locating an
initial point and exploiting the local neighbors. Firstly, the
entire image is split into two equal size blocks of West and
East. Given the vessel bifurcation feature of ONH network,
splitting images into two blocks is able to improve the
efficiency of vessel tracking and accuracy of bifurcation
locating. Secondly, the edge pixels are processed from west
to east and from north to south at the West Block (Fig. 5).
(a) An example for the step count calculation on West Block.
(b) An example for the step count calculation on East Block.
Figure 5 Examplesforthestepcountcalculation.Bluepixel—edge;red
pixel—control point candidate; yellow pixel—local direction change.
Table 1 Chain code representation.
Chain codeDirectionAngle
1
2
3
4
5
6
7
8
South
North
East
West
Southeast
Northwest
Southwest
Northeast
270o
90o
0o
180o
315o
135o
225o
45o
J Sign Process Syst

Page 5

Edge pixels at the East Block are processed in the opposite
direction (Fig. 5). The initial points are located at the
northwest corner and northeast corner on Canny edges at
the West Block and East Block, respectively. The eight-
connectivity chain code rule prevents the ambiguity to
choose the next to-be-processed pixel, because there is
always at most one connected neighbor, if there is any, next
to the current pixel. However, if there is no next edge pixel,
a scanning will be triggered by following the same direction
of the vessel tracking from the current point till it finds an
unprocessed edge pixel. When every edge pixel of the
entire ONH network has been processed, the tracking will
be terminated.
At the West Block, as long as the edge pixels are
heading toward East, no matter toward North East, East
or South East, the current direction is marked as “East”
and the step count is incremented by 1. If the direction
starts to change, i.e. change from East to West, the step
count needs to be compared to a rollback threshold. If
smaller, roll back the most recent change of the step
count; otherwise, keep the new step direction. The
rollback threshold is used to determine whether or not
a direction change is local. When a rollback is triggered,
it means the direction change is local, and thus it should
not be considered as a real vessel bifurcation. The local
direction change pixels (the yellow pixel on Fig. 5 (a)),
which usually come from the curved part on the ONH
vessel edges, do not represent the real vessel bifurcations.
The (X, Y) coordinate of a non-local changing pixel is a
possible control point’s coordinate that the algorithm will
determine later. If the step count is equal to a maximum
allowed step count threshold, and the previous direction’s
step count is greater than or equal to the maximum
allowed step count threshold, the most recent possible
control point is determined as a true control point
candidate (Figs. 6–7).
2.4 Outlier Control Point Detection
Rather than only preserving the real vessel bifurcations,
the edge detection process could create extra edge
bifurcations due to the gray level property of the vessels.
These extra edge bifurcations are the outlier control
points (Fig. 7 (a)). They are not representing the true
bifurcation features of the ONH vasculature. Removing
outliers are important because the mis-detected control
points could cause mis-matched control point pairs
between the reference and input images and furthermore
cause fusion failure.
In the AFEA algorithm, outlier control points can be
removed by detecting the opposite global direction
change pixels. In the West Block, the direction change
sign for control point identification is from heading
eastward to westward. If there is any detected direction
change pixel having the opposite direction change, i.e.
from heading westward to eastward, this direction change
pixel is identified as an outlier control point. In the East
Block, the direction change pixel will be marked as an
outlier if its direction change sign is from heading
eastward to westward.
2.5 Control Point Pair Matching
The detected control points need to be matched as one
pair per match using a certain matching criterion. The
Shape Similarity Criteria [17–19], which are able to
identify an unknown shape by matching it to the most
similar shape model, are employed to match the control
points into three pairs as required by the 2D affine
transformation. The shape similarity based matching
process takes the following procedure. Firstly, the control
points are divided into a certain number of groups. The
image having less number of control points is taken as the
grouping base. Suppose image I1has n control points, and
image I2has m control points, and m<n, then m will be
the group number determining that how many groups of
control points it will have. Secondly, each control point in
I1is combined with each control point in I2, and there are
totally m×n control point pairs. The distance |d| between
each control point pair is then calculated within each
group using Eq. 7.
d j j ¼
where (x1, y1) is the coordinate of one control point; (x2,
y2) is the coordinate of the other control point. Thirdly,
inside each group, the pair with minimum |d| is chosen.
The assumption it uses the minimum distance as matching
criteria is based on the fact that the two images do not
have huge rotation, shearing or translation, and thus the
same features on each image are close to each other. If two
or more control points in one image match the same
control point in the other image, the true match will
always has smaller |d| than the false match based on the
Shape Similarity Criteria assumption. Finally, the method
selects the three smallest distance |d| control point pairs as
the final (x, y) and (u, v) for the 2D affine transformation.
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
x1? x2
ðÞ2þ y1? y2
ðÞ2
q
ð7Þ
3 Heuristic Optimization
3.1 Problem Formulation
An initial good-guess of the control points improves the
probability of convergence and the quality of the fused
image. The initial guess, which is close to the optimal
J Sign Process Syst

Page 6

result, is not an accurate solution because the Canny
operator detects edges at the fuzzy ONH vasculature.
Figure 8 (a) are the fused image based on the initial guess,
where the fundus image does not have a maximum
angiogram overlay. Therefore, an optimization procedure
is required to adjust the initial good-guess control points
in order to achieve the optimal fusion result. Such process
can be formulated as a heuristic problem of optimizing an
objective function that maximizes the Mutual-Pixel-Count
between the reference and input images. The algorithm
finds the optimal solution by continuously refining the
transformation parameters in an ordered way. During the
iteration, the reference image’s control points’ (u, v)
coordinates are fixed. Only input image’s control points’
(x, y) coordinates are subject to adjustment. The fused
image is assumed to be optimal when the objective
function is maximized.
3.2 Objective Function: Mutual-Pixel-Count (MPC)
Mutual-Pixel-Count measures the ONH vasculature over-
lapping for corresponding pixels in both images. When the
vasculature pixel’s transformed (u, v) coordinates on the
input image correspond to the vasculature pixel’s coordi-
(a) Angiogram grayscale image’s control point candidate selection.
(b) Control point candidate CP1 selected at the West Block
(c) Control point candidate CP2 selected at the East Block
CP1
CP2
CP3
CP1
CP2
(d) Control Point candidate CP3 selected at the East Block
CP3
Figure 6 Angiogram grayscale image’s control point selection; Black pixel-edge; red pixel—control point candidate.
J Sign Process Syst

Page 7

nates on the reference image, MPC is incremented by 1.
MPC is assumed be maximized when the image pair is
perfectly geometrically aligned by the transformation. The
problem can be mathematically formulated as maximization
of the following objective function:
fmpcx;y;u;v
ð Þ ¼
X
u;v2ROI
Iref x;y
ð Þ¼0and Iinput u;v
ð Þ¼0
Iinput Txu;v
ðÞ;Tyu;v
ðÞ
??
ð8Þ
where,
Tx¼ a1u þ a2v þ b1
ð9Þ
Ty¼ a3u þ a4v þ b2
The objective function fMPC denotes the Mutual-Pixel-
Count. Tx(Eq. 9) and Ty(Eq. 10) are the transformations for
u and v coordinates of the input image. The Region-of-
Interest (ROI) is the ONH vasculature region where MPC is
calculated on. Parameters {a1, a2, a3, a4, b1, b2} is
obtained using Eq. 1.
After the image segmentation, the binary images of the
input and the reference images are obtained, i.e. Iinputand
Iref(Fig. 3 (a) and (b)). Only black pixels from both images
contribute to MPC. The ideal case is that all zero (black)
pixels of the input image are mapped onto zero pixels of the
reference image. This calculation can be formulated in
pseudo codes as follows:
ð10Þ
if B x;y
ð Þ ¼ B xu;yv
thenMPC ¼ MPC þ 1;
ðÞ
end if
where B is a binary 2D map. In this binary map, 0 denotes
the vessel pixel (black) and 1 denotes the background pixel
(white). For the affine transformation T and all pixels (ui,
vi), where
ui;vi
ð Þ 2 ROI; xu;yv
ð Þ ¼ T u;v
ðÞð11Þ
3.3 Heuristic Optimization Algorithm (HOA)
To solve the optimization problem, a global optimization
scheme (e.g. the brute force exhaustive search technique) can
guarantee the successful outcome of the global maxima but
with the tradeoff on excess computation cost. In the real
scenario,the searchdomainrange has tobenarroweddownin
order toaccelerate the execution.A local optimizationscheme
is usually applied to reduce the computation cost. However,
local optimization can be attracted to local maxima [20].
The HOA algorithm does not guarantee that the
objective function always reaches the global maxima. The
reason is not because the heuristic method is inefficient, but
the fact that the features and objects on the reference and
input images are not identical in most cases.
In HOA, the initial movement direction is randomly
determined. Suppose ∇M is the changed volume of fMPC.
MPCprevis the previous fMPCprior to the movement, and
MPC is the current fMPCafter the movement, then:
8
>
>
rM ¼ MPC ? MPCprev
> 0 :Keepmovingthecoordinate
towardthesamedirection:
? 0 :
Stepð1Þ ? Stopuorvmovementin
thatsamedirection;
Stepð2Þ ? Movethecoordinate
towardanewrandomdirection
>
>
>
>
>
>
>
>
>
>
>
>
<
:
ð12Þ
This process can be formulated in pseudo codes as follows:
Assign startPixel(u, v ) to currentPixel;
Initialize nextMPC to 0;
For all four neighbors of currentPixel
Locate a neighbor (ui, vi) of currentPixel in an ordered way;
// i is the number of visited neighbors
if ( fMPC(ui, vi) > nextMPC )
Assign fMPC(ui, vi) to nextMPC;
Assign (ui, vi) to currentPixel;
else
break;
end if
end for loop
Coordinates’ adjustment is iteratively implemented until
one of the following convergence criteria is reached:
1. Predefined maximum number of loops is reached.
2. The updated MPC is smaller than ε, i.e.
fMPCnþ1x;y;u;v
ð
where,
ε is a very small non-negative threshold; fMPCnþ1x;y;u;v
is the updated MPC; and fMPCnx;y;u;v
MPC.
Þ ? fMPCnx;y;u;v
ðÞ
?? ??"
ð13Þ
ðÞ
ðÞ is the current
3.4 Implementation
The experimental monkey was anesthetized repeatedly
every 30 min as required to maintain the animal in deep,
stage IV anesthesia. However, the anesthetics procedure
cannot be simply repeated on humans only for the eye
examination purpose. The clinical goal of the ONH image
fusion (Figs. 8 and 9) is to achieve a maximum angiogram
overlay on the fundus image. In all cases, fused images are
generated following binarization, edge extraction, control
point detection using the AFEA algorithm, and optimiza-
tion using the HOA algorithm.
J Sign Process Syst

Page 8

(a) Fundus color image’s control point candidate selection
(b) Control Point candidate CP1 selected at the West Block
CP1 CP4

CP2
CP3
Outlier
CP5
(c) Control Point candidate CP2 selected at the West Block
(d) Control Point candidate CP3 selected at the West Block
(e) Control Point candidate CP4 selected at the East Block
(f) Control Point candidate CP5 selected at the East Block
Figure 7 Fundus color image’s control point selection. Black pixels are edges; red pixels are control point candidates; the blue pixel on (a) is the
outlier.
J Sign Process Syst

Page 9

4 Qualitative Evaluation
The performance of the proposed AEFA and HOA
algorithms were compared to the results of Centerline
Control Point Detection Algorithm, Root Mean Square
Error (RMSE) minimization objective function employed
by the traditional Iterative Closest Point (ICP) algorithm,
and Genetic Algorithm. In addition, some other existing
image fusion approaches were discussed in terms of
running time. The time series, objective function fMPC,
and the visual fusion results have been extensively used in
this evaluation to measure and compare the performance of
different fusion methods. The evaluation results strengthen
the AFEA and HOA algorithms in terms of novelty,
automation, accuracy, and efficiency.
4.1 Genetic Optimization Algorithm
Genetic Algorithm (GA) is a well-known global optimization
technique which has been employed to solve numerous
optimization problems, including biology, chemistry, medical
physics, and medical image processing [21, 22]. Crossover
and mutation are the two frequently used GA operations. In
this comparative GA analysis, there are totally 3 groups of
data populations, and each group stands for one control point
from the input image. The population size remains same for
each generation. Each individual is an unsigned char array.
Each element in the array is a random number Sn.
Sn2 0;1;2;3
where, 0, 1, 2, and 3 stand for the possible moving direction
toward East, West, North and South, respectively. The initial
3 individuals are the coordinates of the initial guess of the
control points ((x1, y1), (x2, y2), (x3, y3)). Each individual’s
fitness is estimated by the same objective function fMPC. Half
of the total individuals (parents+children) with higher fMPC
are selected. The offspring generation is iteratively produced
till the termination condition is reached. 40 simulations with
the final GA-based fused images were carried out. The
average rate of running time and the objective function fMPC
showed that the HOA achieved a better optimization with a
higher fMPCthan GA and lower running time consumption
(Table 2) (Fig. 10).
fg
ð14Þ
4.2 Centerline Control Point Detection Algorithm
For evaluating and comparing the proposed AFEA algo-
rithm to the Centerline Control Point Detection Algorithm
Figure 9 fMPCincreasing during optimization (Y-axis is the fMPC; X-
axis is the loop count).
(e) Final Fused Image
(a) Fused result from the initial
good-guess of control points
(b)
(d)(c)
Figure 8 Fused image improvement during the iteration. fMPC(a)–(e)
=30372, 30888, 30914, 31134, 32277.
J Sign Process Syst

Page 10

reported by Laliberte in literature [23], the same ONH
image pairs were fused by applying both of the HOA
algorithm and the Genetic Algorithm. In Laliberte’s
algorithm, the vessel thin image is firstly created using a
thinning algorithm that guarantees a one-pixel width.
Secondly, pixels with three or four neighbors were
identified as the control points. Thirdly, control points were
matched by utilizing the following criteria:
1) Control point pair located inside a distance threshold;
2) With same number of neighbors;
3) Distance between angle is less than a threshold σ;
4) Eliminate the one without matches.
The initial control point’s objective function fMPC is
7134. They are lower than the proposed AFEA algorithm’s
result of 30372. The final fMPCafter applying the HOA is
29720, which is apparently lower than the final fMPCby
applying the AFEA algorithm, i.e. 32277 (Table 3).
Therefore, the AFEA algorithm has provided a better
performance than the Centerline Control Point Detection
Algorithm in terms of fMPC(Fig. 11).
Furthermore, Laliberte’s centerline control point detec-
tion algorithm has 10 threshold parameters of which 7 are
dependent on the image resolution and 3 left main free.
More threshold parameters the algorithm has, more
human’s intervention is required when parameters need
adjustment, and hence less automation level the program is.
In the AFEA and HOA algorithms, there are totally 9
adjustable threshold parameters (Table 4). Among them, 5
are dependent on the image resolution/size and 4 are left
main free.
4.3 RMSE Minimization—ICP’s Objective Function
Root Mean Square Error (RMSE) between the coordinates
of the reference image and the transformed image has been
used by the Iterative Closest Point algorithm as the
objective function. RMSE is defined as:
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
i¼0
RMSE ¼
X
N
xture? xreverse
ðÞ2þ yture? yreverse
ðÞ2
!
=N
v
u
u
t
ð15Þ
where N is the number of common pixels existing in both
images. The xtrueand ytrueare the true coordinates from the
reference image. The xreverse and yreverse are the reverse
transformed coordinates by applying the reverse transfor-
mation of the known 2D affine polynomials. Figure 12
displays 6 selected intermediate fused images during RMSE
minimization. It can be observed that the fused image’s
quality is not necessarily getting improved when RMSE is
becoming smaller. There is no clear relationship between
the quality of the fused image and the value of RMSE that
can follow a certain pattern. Therefore, RMSE is not an
ideal objective function when applying to multi-modality
ONH image fusion.
(b)(a)
(d)(c)
Figure 10 Fused image generated by GA: a fMPC=31227; b fMPC=
32007; c fMPC=32185; d fMPC=32220.
Table 2 Comparison of GA and HOA’s average performance.
AlgorithmfMPC
Running time
GA
HOA
31830
32277
7.3 minutes
1 minute
(a)(b)
Figure 11 Fused image by applying the Centerline algorithm. a: by
HOA with fMPC=29720;b: by GA algorithm with fMPC=29337.
Table 3 Comparison of centerline control point detection algorithm
with the AFEA algorithm.
Objective function fMPC
Centerline algorithmAFEA algorithm
Initial fMPC
GA final fMPC(Average)
HOA final fMPC
7143
29337
29720
30372
31861
32277
J Sign Process Syst

Page 11

4.4 Other Existing Data Fusion Approaches
In the manual fusion approach, the ophthalmologist
identifies control points at vessel bifurcations, which are
common to both images that are to be registered. The
control points placed by the experts seemed appropriate.
However, the fusion result might not be optimal at many
cases. The disadvantage of human-interactive approach
includes, but not limited to inaccuracy in the placement of
control points, inconsistency of the fusion results, and the
significantly increased interaction time during manual
adjustment of the control points. The average time for
human manual fusion is about 35 min, including initial
control point selection, manual adjustment of control
points’ coordinates, and evaluation of the fusion result
after each adjustment.
Ma proposed a uniform spatial sub-sampling approach,
vector quantization algorithm, and stratified sampling with
centroid refinements strategy in [24]. Among these three
approaches, stratified sampling with centroid refinements
strategy achieved shortest average running time of 11 min
for a satisfactory fusion result. Matsopoulos and Mour-
avliansky proposed an automatic retinal image fusion
scheme using global optimization techniques in [21]. They
reported an average execution time of 4.5 min. Airborne
Underwater Geophysical Signals (AUG Signals), Toronto,
Canada, has developed an image registration software
capable of automatically registering temporal hyperspectral,
polarimetric Synthetic Aperture Radar (SAR) or multi-
sensor images [25]. AUG Signals’ software running time is
from 3–7 min. With no more than 1 min running time, the
AEFA and HOA algorithms, therefore, has significant
advantage compared to other existing image fusion
approaches in terms of running time (Table 5).
(a) (b)
(c) (d)
(e)(f)
Figure 12 Fused image using RMSE objective function. RMSE (a)–
(f)=119.51, 99.54, 71.34, 64.76, 57.01, 49.30.
Table 4 Threshold parameters of AFEA and HOA algorithms.
ParametersAnnotation
(1) Comparing Edge
(2) Otsu’s threshold
(3) Adjusted Otsu’s Threshold
(4) Vasculature Extraction Threshold
(5) Black Area Percentage
(6) Max Steps on Gray Image
(7) Max Steps on Color Image
(8) Roll Back Threshold
(9) Maximum loop count
ROI’s X-axis for estimating fMPC
Image binarization [12]
Image binarization
Making either reference or input image transparent, so that one can be overlaid on the other
Determining whether parameter (2) or (3) should be used
A maximum allowed step count threshold on the grayscale image for the control point identification
A maximum allowed step count threshold on the color image for the control point identification
Determining whether or not a direction change pixel is a final control point candidate
Determining when the HOA stops
Table 5 Running time comparison.
MethodsRunning time
Manual approach
Uniform spatial sub-sampling
Vector quantization algorithm
Stratified sampling
Matsopoulos’s method
AUG Signals
AFEA and HOA algorithms
35 minutes
24 minutes
19 minutes
11 minutes
4.5 minutes
3 minutes
< 1 minute
J Sign Process Syst

Page 12

5 Conclusions
This study has made two new and unique contributions to
the multi-modality medical image fusion area in terms of
novelty, efficiency, and accuracy. The first contribution is
the new automated control point detection AFEA algo-
rithm. The evaluation study with the Centerline Control
Point Selection Algorithm shows the advantage of the
AFEA. The second contribution is the HOA algorithm for
the initial guess of control points’ optimization. Building
initially on existing work, such as binarization and edge
extraction, the proposed approach is in fact a series of
algorithms designed to work together to solve the image
fusion automation problem. The experiments have also
highlighted areas of potential improvement of clinical tools
for retinopathy diagnosis. In future research, it would be of
interest to expand the AFEA and HOA algorithms for
human or animals’ 3D eye, brain, or body image fusion for
widespread use.
Acknowledgment
Ning for their support and help during this research. This work is
funded by BCVC programs.
The authors are grateful to Dr. Thompson and Dr.
References
1. Wong, A., & Orchard, J. (2009). Robust multimodal registration
using local phase-coherence representations. Journal of Signal
Processing Systems, 54(1–3), 89–100.
2. Yang, G., Stewart, C., Sofka, M., & Tsai, C. (2007). Registration
of challenging image pairs: initialization, estimation, and decision.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
29(11), 1973–1989.
3. Woo, J., Hong, B., Hu, C., Shung, K., Kuo, C., & Slomka, P.
(2009). Non-rigid ultrasound image registration based on intensity
and local phase information. Journal of Signal Processing
Systems, 54(1–3), 33–43.
4. Loeckx, D., Slagmolen, P., Maes, F., Vandermeulen, D., &
Suetens, P. (2010). Nonrigid image registration using conditional
mutual information. IEEE Transactions on Medical Imaging, 29
(1), 19–29.
5. Estevez, P., Tesmer, M., Perez, C., & Zurada, J. (2009).
Normalized mutual information feature selection. IEEE Trans-
actions on Neural Networks, 20(2), 189–201.
6. Lin, F., & Olivo, M. (2009). Guest editors’ comments; the state-
of-the-art technologies for medical and biological imaging.
Journal of Signal Processing Systems, 54(1–3), 1–6.
7. Stewart, C., Tsai, C., & Roysam, B. (2003). The dual-bootstrap
iterative closest point algorithm with application to retinal image
registration. IEEE Transactions on Medical Imaging, 22(11),
1379–1394.
8. Canny, J. (1986). A computational approach to edge detection.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
8, 679–698.
9. Chen, X., Yang, J., Zhang, J., & Waibel, A. (2002). Automatic
detection of signs with affine transformation. 6th IEEE Workshop
on Applications of Computer Vision, 32–36.
10. Khoobehi, B., Beach, J., & Kawano, H. (2004). Hyperspectral
imaging for measurement of oxygen saturation in the optic nerve
head. Investigative Ophthalmology & Visual Science, 45, 1464–
1472.
11. Beach, J., Ning, J., & Khoobehi, B. (2007). Oxygen saturation in
optic neve head structures by hyperspectral image analysis.
Current Eye Research, 32, 161–170.
12. Otsu, N. (1979). A threshold selection method from gray-level
histograms. IEEE Transactions on Systems, Man, and Cybernet-
ics, 9, 62–66.
13. Sezgin, M., & Sankur, B. (2004). Survey over image thresholding
techniques and quantitative performance evaluation. Journal of
Electronic Imaging, 13, 146–165.
14. Chaudhuri, S., Chatterjee, S., & Katz, N. (1989). Detection of
blood vessels in retinal images using two-dimensional matched
filters. IEEE Transactions on Medical Imaging, 8, 263–269.
15. Can, A., Shen, H., Turner, J. N., Tanenbaum, H. L., & Roysam,
B. (1999). Rapid automated tracing and feature extraction from
retinal fundus images using direct exploratory algorithms. IEEE
Transactions on Information Technology in Biomedicine, 3, 125–
138.
16. Poli, R., & Valli, G. (1996). An algorithm for real-time vessel
enhancement and detection. Computer Methods and Programs in
Biomedicine, 52, 1–22.
17. Antania, S., Leeb, D., Longa, R., & Thoma, G. (2004). Evaluation
of shape similarity measurement methods for spine x-ray images.
Journal of Visual Communication and Image Representation, 15,
285–302.
18. Arkin, E. M., Chew, L. P., Huttenlocher, D. P., Kedem, K., &
Mitchell, J. (1991). An efficiently computable metric for compar-
ing polygonal shapes. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 13(3), 209–216.
19. Avis, D., & ElGindy, H. (1983). A combinatorial approach to
polygon similarity. IEEE Transactions on Information Theory, IT-
2, 148–150.
20. Zhu, Y. (2007). Mutual information-based registration of temporal
and stereo retinal images using constrained optimization. Com-
puter Methods and Programs in Biomedicine, 86, 210–215.
21. Matsopoulos, G., Mouravliansky, N., & Delibasis, K. (1999).
Automatic retinal image registration scheme using global optimi-
zation techniques. IEEE Transactions on Information Technology
in Biomedicine, 3, 47–60.
22. Doldberg, D. (1989). “Genetic algorithms in optimization”;
search and machine learning. MA: Addison-Wesley.
23. Laliberte, F., & Gagnon, L. (2003). Registration and fusion of
retinal images—An evaluation study. IEEE Transactions on
Medical Imaging, 22(5), 661–673.
24. Ma, B. (2001). Parametric and nonparametric approaches
for multisensor data fusion. PhD dissertation; University of
Michigan.
25. Airborne Underwater Geophysical Signals (AUG Signals); Auto-
matic image registration—temporal, multi-sensor and multi-layer
registration. Toronto, Canada.
J Sign Process Syst

Page 13

Hua Cao received the B.E. degree of Management Information
Systems from University of Finance and Economics, China in 2000,
the M.S. degree of Systems Science from Louisiana State University,
USA in 2003, and the Ph.D. degree of Computer Science from
Louisiana State University, USA in 2008. She is a Senior Research
Scientist in the Computer Science Department and Ophthalmology
Department at Louisiana State University. Dr. Cao has more than 10
research publications including one book, one book chapter, three
journals and six conference proceedings. Her research interests
include, biomedical imaging, feature detection, data registration,
image fusion, artificial intelligence, and route planning.
Nathan Brener received the B.A. degree of Physics from Brandeis
University in 1965 and the Ph.D. degree of Physics from Louisiana
State University in 1971. He is a faculty member of Computer Science
Department, Louisiana State University. He has approximately 35
years experience in the development of fast algorithms for high
performance computing. Dr. Brener has more than 50 research
publications in refereed journals and has made numerous presentations
at scientific meetings. His research interests include predictive
intelligence, artificial intelligence, route planning algorithms, image
processing, and parallel processing.
Bahram Khoobehi received the Ph.D. degree in physics from North
Texas State University, Denton. He is currently a Professor of
Ophthalmology, Louisiana State University Health Sciences Center,
Department of Ophthalmology, New Orleans, and also an Adjunct
Associate Professor of Biomedical Engineering, at Tulane University
School of Engineering, New Orleans. His research interests include
hyperspectral imaging to measure oxygen saturation in the retina and
optic nerve head; targeting dye and drug delivery systems to the
retina; and, retinal blood flow.
S. Sitharama Iyengar (M’88-SM’89-F’95) received the M.S. degree
from the Indian Institute of Science, Bangalore, in 1970 and the Ph.D.
degree from Mississippi State University in 1974. He is the chairman
and Roy Paul Daniels Chaired Professor of Computer Science at
Louisiana State University, Baton Rouge, and is also the Satish
Dhawan Chaired Professor at the Indian Institute of Science. His
publications include 13 books (Prentice-Hall, CRC Press, IEEE
Computer Society Press, John Wiley & Sons, etc.) and more than
280 research papers. He is the founder and editor-in-chief of the
International Journal of Distributed Sensor Networks. He has been
involved with research in high-performance algorithms, data struc-
tures, sensor fusion, data mining, and intelligent systems. Dr. Iyengar
was awarded the Distinguished Alumnus Award by the Indian Institute
of Science in March 2003. He has served as an associate editor for the
IEEE and as a guest editor for the IEEE Transactions on Knowledge
and Data Engineering, the IEEE Transactions on Systems, Man, and
Cybernetics, and the IEEE Transactions on Software Engineering. He
is a fellow of the IEEE, the ACM, and the AAAS.
J Sign Process Syst