A Novel Approach for Automatic Follow-Up of Detected Lung Nodules

Abstract

Our long term research goal is to develop an image-based approach for early diagnosis of lung nodules that may lead to lung cancer. This paper focuses on monitoring the progress of detected lung nodules in successive chest low dose CT (LDCT) scans of a patient using non-rigid registration. In this paper, we propose a new methodology for 3D LDCT data registration. The registration methodology is non-rigid and involves two steps: global alignment of one scan (target data) to another scan (reference data) using the learned prior appearance model followed by local alignments in order to correct for intricate deformations. From two subsequent chest scans, visual appearance of the chest images, after equalizing their signals, are modeled with a Markov-Gibbs random field with pairwise interaction. Our approach is based on finding the affine transformation to register one data set (target data) to another data set (reference data) by maximizing a special Gibbs energy function using a gradient descent algorithm. To get accurate appearance model, we developed a new approach to an automatically select the most important cliques that describe the visual appearance of LDCT data. To handle local deformations, we propose a new approach based on deforming each voxel over evolving closed and equi-spaced surfaces (iso-surfaces) to closely match the prototype. The evolution of the iso-surfaces is guided by an exponential speed function in the directions minimizing distances between corresponding pixel pairs on the iso-surfaces on both data sets. Our preliminary results on 10 patients show that the proper registration could lead to precise identification of the progress of the detected lung nodules.

Full text

Page 1

A NOVEL APPROACH FOR AUTOMATIC FOLLOW–UP OF DETECTED LUNG NODULES
Ayman El-Baz1, Georgy Gimel’farb2, Robert Falk3, and Mohamed A. El-Ghar4
1Bioengineering Department, University of Louisville, Louisville, KY, USA.
2Department of Computer Science, University of Auckland, Auckland, New Zealand.
3Director, Medical Imaging Division, Jewish Hospital, Louisville, KY, USA.
4Urology and Nephrology Department, University of Mansoura, Mansoura, Egypt.
ABSTRACT
Our long term research goal is to develop an image-based approach
forearly diagnosisof lung nodulesthat may lead to lung cancer. This
paper focuses on monitoring the progress of detected lung nodules
in successive chest low dose CT (LDCT) scans of a patient using
non-rigid registration. In this paper, we propose a new methodol-
ogy for 3D LDCT data registration. The registration methodology
is non-rigid and involves two steps: global alignment of one scan
(target data) to another scan (reference data) using the learned prior
appearance model followed by local alignments in order to correct
for intricate deformations. From two subsequent chest scans, vi-
sual appearance of the chest images, after equalizing their signals,
are modeled with a Markov-Gibbs random field with pairwise inter-
action. Our approach is based on finding the affine transformation
to register one data set (target data) to another data set (reference
data) by maximizing a special Gibbs energy function using a gra-
dient descent algorithm. To get accurate appearance model, we de-
veloped a new approach to an automatically select the most impor-
tant cliques that describe the visual appearance of LDCT data. To
handle local deformations, we propose a new approach based on de-
forming each voxel over evolving closed and equi-spaced surfaces
(iso-surfaces) to closely match the prototype. The evolution of the
iso-surfaces is guided by an exponential speed function in the direc-
tions minimizing distances between corresponding pixel pairs on the
iso-surfaces on both data sets. Our preliminary results on 10 patients
show that the proper registration could lead to precise identification
of the progress of the detected lung nodules.
Index Terms— Lung cancer, Low dose computed tomography,
non-rigid registration, rigid registration.
1. INTRODUCTION
Because lung cancer is the most common cause of cancer deaths,
fast and accurate analysis of pulmonary nodules is of major impor-
tance for medical computer-aided diagnostic systems (CAD). In [1]
we introduced a fully automatic nodule detection algorithm showing
the accuracy up to 93.3% on the experimental database containing
200 real LDCT chest data sets with 36,000 2D slices. In [2,3] we
introduced an accurate segmentation approach to segment the de-
tected lung nodules from LDCT images. Below, we focus in the
next CAD stage, namely, on accurate registration of the detected
lung nodules for subsequent volumetric measurements to monitor
how the detected lung nodules change over the time.
We use a three-step procedure to separate the nodules from their
background: (i) an initial LDCT slice is segmented with algorithms
introduced in [4] to isolate lung tissues from surrounding structures
in the chest cavity as shown in Fig. 1, (ii) registration of two succes-
sive CT scan data sets which are taken at two different times, and
(a)(b)(c)
Fig. 1. Step 1 [4] of our segmentation: an LDCT slice (a) with
isolated lungs (b), and (c) the normalized segmented lung image.
(iii) the nodules in the isolated lung regions are segmented by evolv-
ing deformable boundaries under forces that depend on the learned
current and prior appearance models. At Step1 each LDCT slice is
modelled as a bi-modal sample from a simple Markov–Gibbs ran-
dom field of interdependent region labels and conditionally indepen-
dent voxel intensities (gray levels). This step is necessary for more
accurate separation of nodules from the lung tissues at Step 3 be-
cause voxels of both the nodules and other chest structures around
the lungs are of quite similar intensity. In this paper we will focus
on the second step.
Previous work. Tracking the temporal behavior of a nodule is
a complicated task because of the change in the patient’s position
at each data acquisition, as well as the effects of heart beats and
respiration of the patient. In order to get accurate measurements
about the progress of lung nodules over the time, all this motion
should be compensated by registering CT data sets taken at different
time periods with each other. In the literature, many methods have
been described for the medical image registration problem (see [5]),
and also for the compensation of the lung motion (see [6]). Below,
we will give some examples of previous work on CT lung images
registration.
For the follow-up of small nodules, Brown et al. [7] developed a
patient-specific model with 81% success in 27 nodules. Ko et al. [8]
used the center mass point of the structures and applied rigid and
affine image registration techniques with 96% success in 58 nodules
of 10 patients.
To account for the non-rigid motion and deformation of the lung,
Woods et al. [9] developed an objective function using an anisotropic
smoothness constraint and a model based on continuum mechan-
ics. Wood’s algorithm required the detection and registration of
feature points as explained in [10], and then interpolating the dis-
placement by the model of continuum mechanics. In Wood’s study,
the difference of the estimated and real volumes was calculated to
be 1.6%. In 2003, Dougherty et al. [11] developed an optical flow
method, a model-based motion estimation technique for estimating
first a global parametric transformation and then local deformations.
This method allowed the alignment of serial CT images with a 95%
correlation. Another optical flow analysis approach was developed
V - 5011-4244-1437-7/07/$20.00 ©2007 IEEEICIP 2007

Page 2

by Naqa et al. [12], where the optical flow analysis was combined
with the information of a spirometer, a device measuring the airflow
into and out of the lungs to track the breathing motion automatically.
The spirometry approach used in Naqa’s study was based on the re-
construction techniques for 4D CT during free breathing proposed
by [13].
There are also studies using CT lung images for pulmonary regis-
tration. For this purpose, Zhang et al. [14] used a standard lung atlas
to analyze the pulmonary structures in CT images. This atlas is reg-
istered to new images by a 3D surface based registration technique
composed of global transformation and local elastic transformations.
Li et al. [15] used feature points for correspondence and landmark &
intensity based registration algorithms to warp a template image to
the rest of the lung volumes.
2. METHODS
The objective of the proposed image analysis approach is to follow
up the detected lung nodules from LDCT images. To achieve this
goal, an image analysis system consisting of three steps is proposed.
These steps are: 1) segmentation of lung from LDCT images, 2) a
non-rigid registration approach to align two successive LDCT scans
and to correct the motion artifacts caused by breathing and patient
motion, and (3) segmentation of the lung nodules. In this paper we
will focus on the second step and the first and third steps are shown
in details in [2–4].
3. GLOBAL ALIGNMENT
Before describing the mathematical detail of the proposed global
alignment approach, we define the following notations:
Basic notation. We denote Q = {0,...,Q − 1}; R = [(x,y,z) :
x = 0,...,X − 1;y = 0,...,Y − 1;z = 0,...,Z − 1], and
Rp ⊂ R a finite set of scalar image signals (e.g. gray levels), a 3D
arithmetic lattice supporting digital LDCT data g : R → Q, and its
arbitrary-shaped part occupied by the prototype, respectively. A fi-
nite set N = {(ξ1,η1,ζ1),...,(ξn,ηn,ζn)} of (x,y,z)-coordinate
offsets defines neighbors {((x + ξ,y + η,z + ζ),(x − ξ,y −
η,z − ζ)) : (ξ,η,ζ) ∈ N} ∧ Rp interacting with each pixel
(x,y,z) ∈ Rp. The set N yields a neighborhood graph on Rp
to specify translation invariant pairwise interactions with n families
Cξ,η,ζof cliques cξ,η,ζ(x,y,z) = ((x,y,z),(x + ξ,y + η,z + ζ))
(see Fig. 2). Interaction strengths are given by a vector VT=
?VT
transposition.
ξ,η,ζ: (ξ,η,ζ) ∈ N?
of potentials VT
ξ,η,ζ =
?Vξ,η,ζ(q,q?) :
(q,q?) ∈ Q2?depending on signal co-occurrences; here T indicates
Fig. 2. Pairwise pixel interaction MGRF model.
3.1. Data Normalization
To account for monotone (order-preserving) changes of signals (e.g.
due to different illumination or sensor characteristics), the LDCT
images are equalized using the cumulative empirical probability dis-
tributions of their signals.
3.2. MGRF based appearance model
Generic MGRF with multiple pairwise interaction (Fig. 2) [4,16],
the Gibbs probability P(g) ∝ exp(E(g)) of an object g aligned
with the prototype g◦on Rpis specified with the Gibbs energy
E(g) = |Rp|VTF(g)
(1)
where FT(g) is the vector of scaled empirical probability distribu-
tions of signal co-occurrences over each clique family: FT(g) =
[ρξ,η,ζFT
ative size of the family and Fξ,η,ζ(g) = [fξ,η,ζ(q,q?|g) : (q,q?) ∈
Q2]T; here, fξ,η,ζ(q,q?|g) =
bilities of signal co-occurrences, and Cξ,η,ζ;q,q?(g) ⊆ Cξ,η,ζ is a
subfamily of the cliques cξ,η,ζ(x,y,z) supporting the co-occurrence
(gx,y,z = q, gx+ξ,y+η,z+ζ = q?) in g. The co-occurrence distribu-
tions and the Gibbs energy for the object are determined over Rp,
i.e. within the prototype boundary after an object is affinely aligned
with the prototype. To account for the affine transformation, the ini-
tial image is resampled to the back-projected Rpby interpolation.
The appearance model consists of the neighborhood N and the
potential V to be learned from the prototype.
ξ,η,ζ(g) : (ξ,η,ζ) ∈ N] where ρξ,η,ζ =
|Cξ,η,ζ|
|Rp|
is the rel-
|Cξ,η,ζ;q,q?(g)|
|Cξ,η,ζ|
are empirical proba-
Learning the potentials: The MLE of V is proportional in the
first approximation to the scaled centered empirical co-occurrence
distributions for the prototype [16]:
?
where U is the vector with unit components. The common scaling
factor λ is also computed analytically; it is approximately equal to
Q2if Q ? 1 and ρξ,η,ζ ≈ 1 for all (ξ,η,ζ) ∈ N. In our case it can
be set to λ = 1 because the registration uses only relative potential
values and energies.
Vξ,η,ζ = λρξ,η,ζ
Fξ,η,ζ(g◦) −
1
Q2U
?
; (ξ,η,ζ) ∈ N
Learning the characteristic neighbors: To find the charac-
teristic neighborhood set N, the relative energies Eξ,η,ζ(g◦) =
ρξ,η,ζVT
ances of the corresponding empirical co-occurrence distributions,
are compared for a large number of possible candidates. To automat-
ically select the characteristic neighbors, we consider an empirical
probability distribution of the energies as a mixture of a large “non-
characteristic” low-energy component and a considerably smaller
characteristic high-energy component: P(E) = πPlo(E) + (1 −
π)Phi(E). Both the components Plo(E), Phi(E) are of arbitrary
shape and thus are approximated with linear combinations of posi-
tive and negative discrete Gaussians (efficient EM-based algorithms
introduced in [4,16] are used for both the approximation and esti-
mation of π).
Appearance-based registration: The object g is affinely trans-
formed to (locally) maximize its relative energy E(ga) = VTF(ga)
under the learned appearance model [N,V]. Here, ga is the part
of the object image reduced to Rp by the affine transformation
a = [a11,...,a23]: x?= a11x + a12y + a13z + a14; y?=
a21x + a22y + a23z + a24; z?= a31x + a32y + a33z + a34.
The initial transformation is a pure translation with a11 = a22 = 1;
ξ,η,ζFξ,η,ζ(g◦) for the clique families, i.e. the scaled vari-
V - 502

Page 3

a12 = a21 = 0, ensuring the most “energetic” overlap between the
object and prototype. Then the gradient search for the local energy
maximum closest to the initialization selects the 12 parameters a.
4. LOCAL MOTION MODEL
To handle local deformations, we propose a new approach based
on deforming the object over evolving closed and equi-spaced con-
tours/surfaces (iso-contours/surfaces) to closely match the proto-
type. The evolution of the iso-contours/surfaces is guided by an
exponential speed function in the directions minimizing distances
between corresponding pixel pairs on the iso-contours/surfaces on
both images. The normalized cross correlation is used to find the
correspondent points between these iso-contours/surfaces.
The first step of our approach is to generate the distance map in-
side the object using fast marching level sets [17]. The second step
is to use this distance map to generate iso-surfaces (Fig. 3). Note
that the number of iso-contours/surfaces, which is not necessarily
the same for both images, depends on the accuracy and the speed
required by the user. The third step consists in finding the corre-
spondences between the iso-surfaces using normalized cross corre-
lation. The final step is the evolution of the iso-surfaces; here, our
goal is to deform the iso-surfaces in the first data set (target image)
to match the iso-surfaces in the second data set (reference image).
Before stating the evolution equation, let us define the following:
(a) Reference data (b) Target data
Fig. 3. Equal spaced generated surfaces.
Fig. 4. Evolution scenario.
• bh
on the reference data, pk = (xk,yk,zk) forming a circularly
connected chain of line segments (p1,p2), ..., (pK−1,pK),
(pK,p1).
• bγ
γ on the target data, pn = (xn,yn,zn) forming a circularly
connected chain of line segments (p1,p2), ..., (pN−1,pN),
(pN,p1).
g1= [ph
k: k = 1,...,K] – K control points on surface h
g2= [pγ
n: n = 1,...,N] – N control points on surface
• S(Ph
surface h in image g1and its corresponding point on surface γ
in image g2,
k,Pγ
n) denotes the Euclidean distance between a point on
• S(Pγ
on surface γ in image g1and its nearset point on surface γ −1
in image g1
n,Pγ−1
n
) denotes the Euclidean distance between a point
• ν(.) is the propagation speed function .
The evolution bτ → bτ+1of the deformable boundary b in dis-
crete time, τ = 0,1,..., is specified by the system of difference
equations pγ
ν(Pγ
and un,τ is the unit vector along the ray between two correspondant
points. The propagation speed function is selected so as to satisfy the
following conditions: ν(Pγ
ν(Pγ
The latter condition, known as the smoothness constraint, prevents
the current point from cross-passing the closest neighbor surfaces
as shown in Fig. 4.Note that the function ν(Pγ
exp?β(Pγ
?
n,τ+1= pγ
n,τ+ ν(Pγ
n,τ)un,τ; n = 1,...,N, where
n,τ) is a propagation speed function for the control point Pγ
n,τ
n,τ) = 0 if S(Ph
k,Pγ
k,Pγ
n,τ),S(Pγ
n,τ) = 0, otherwise
n,τ,Pγ+1
n,τ) = min?S(Ph
n,τ),S(Pγ
n,τ,Pγ−1
n,τ)?.
n,τ) = −1 +
n,τ)S(Ph
k,Pγ
n,τ)?; satisfies the above conditions, where
S(Ph
β(Pγ
n,τ) is the propagation term such as, at each surface point
min?
β(Pγ
n,τ) =
ln
k,Pγ
n,τ),S(Pγ
n,τ,Pγ−1
n,τ),S(Pγ
n,τ,Pγ+1
n,τ)?
+1
?
S(Ph
k,Pγ
n,τ)
.
5. EXPERIMENTAL RESULTS AND CONCLUSIONS
The proposed non-registration approach is tested on clinical datasets
collected from 10 patients. Each patient has five scans, and the pe-
riod between each two successive CT scans is three months. This
preliminary clinical database collected by low dose CT scan proto-
col with the following scanning parameters: slice thickness of 2.5
mm reconstructed every 1.5 mm, scanning pitch 1.5, pitch 1 mm,
KV 140, MA 100, and F.O.V 36 cm. Figure 5(d) shows checker-
board visualization between the data sets shown in Fig. 5(a) and the
aligned data set shown in Fig. 5(c) to demonstrate the effect of the
motion of the lung tissues. It can be seen that the connectivity at the
edges of the lung, between the two volumes is not smooth when us-
ing the global model only, this is due to the local deformation which
comes from breathing and heart beats. Figure 5 shows the results
after applying the local deformation model. It shows the connec-
tivity at the edges of lung region, and between the two volumes is
smoother when using the proposed local deformation model.
After the two volumes of different time instants are registered,
the task is to find out if the nodules are growing or not. For this pur-
pose, after registration, we are segmenting the lung nodules using
our previous approach [2,3]. Once the nodules are segmented in the
original and registered image sequences, the volumes of the nodules
are calculated using the Δx, Δy, and Δz values from the scanner. In
our studies, these values are 0.7mm, 0.7mm, and 2.5mm respec-
tively. Figure 6 shows the detected changes in two detected lung
nodules for two different patients over one year.
In this paper, we introduced a new approach for the non-rigid reg-
istration of spiral LDCT images. The proposed algorithm consist
of two steps: global alignment of one scan (target data) to another
scan (reference data) using the learned prior appearance model fol-
lowed by local alignments in order to correct for intricate deforma-
tions. The preliminary results on 10 patients show that the proper
registration could lead to precise identification of the progress of the
detected lung nodules.
V - 503

Page 4

(a)(b)(c)
(d) (e)(f)
Fig. 5. (a) Reference data, (b) target data, (c) transformed data using
12 degree of freedom affine transformation, (d) checkerboard visual-
izationtoshowthemotioneffectofthelungtissues, (e)ournon-rigid
regeneration results, and (g) checkerboard visualization to show the
quality of the proposed local deformation model.
Fig. 6. The results of the proposed follow up registration algorithm
for two patients over one year
6. REFERENCES
[1] A. A. Farag, A. El-Baz, and G. Gimelfarb, “Quantitative Nod-
ule Detection in Low Dose Chest CT Scans: New Template
Modeling and Evaluation for CAD System Design,” Proc. of
International Conference on Medical Image Computing and
Computer-Assisted Intervention (MICCAI), Palm Springs, Cal-
ifornia, USA, October 26-29, 2005, pp. 720-728.
[2] A. Farag, A. El-Baz, G. Gimel’farb, R. Falk, M. Abou El-
Ghar, T. Eldiasty, S. Elshazly, ”Appearance Models for Ro-
bust Segmentation of Pulmonary Nodules in 3D LDCT Chest
Images,” Proc. of International Conference on Medical Im-
age Computing and Computer–Assisted Intervention (MIC-
CAI’06), Copenhagen, Denmark, October 1–6, 2006, pp. 662-
670.
[3] A. El-Baz, A. A. Farag, G. L. Gimel’farb, R. Falk, M. Abou
El-Ghar, T. Eldiasty, A Framework for Automatic Segmen-
tation of Lung Nodules from Low Dose Chest CT Scans,
Proc. of IEEE International Conference on Pattern Recognition
(ICPR06), Hong Kong, August 2024, 2006, pp. 611-614.
[4] A. A. Farag, A. El-Baz, and G. Gimel’farb, “Precise Segmen-
tation of Multi-modal Images,” IEEE Transactions on Image
Processing, vol. 15, no. 4, pp. 952-968, April 2006.
[5] J. B. A. Maintz and M. A. Viergever, “A Survey of Medical
Image Registration,” Medical Image Analysis, vol.2, pp.1–36,
1998.
[6] J. Ko and D. Naidich, “Computer-aided diagnosis and the eval-
uation of lung disease,” Journal of thoracic imaging, pp. 19–
136, 2004.
[7] M. Brown, M. McNitt-Gray, N. Mankovich et al., “Method for
segmenting chest CT image data using an anatomical model:
preliminary results,” IEEE Trans Med Imaging, vol. 20, pp.
242-1250, 2001.
[8] J. Ko, and M. Betke, “Chest CT: automated nodule detection
and assessment of change over timepreliminary experience,”
Radiology, vol. 218, pp. 267–273, 2001.
[9] K. Woods, L. Fan, C. W. Chen, and Y. Wang, “Model Sup-
ported Image Registration and Warping for Change Detec-
tion in Computer-aided Diagnosis,” Applied Imagery Pattern
Recognition(AIPR)AnnualWorkshops, WashingtonDC,2000.
[10] L. Fan and C. W. Chen, “An integrated approach to 3D warp-
ing and registration from lung images,” Proceedings of SPIE
Conf.DevelopmentsinX-RayTomographyII,Denver, CO,July
1999.
[11] L.Dougherty, J.AsmuthandW.Gefter, “AlignmentofCTlung
volumes with an optical flow method,” Academic radiology,
vol. 10, pp. 249–254, 2003.
[12] I. Naqa, D. Low, J. Deasy et al., “Automated breathing mo-
tion tracking for 4D computed tomography,” Nuclear Science
Symposium Conference Record, vol. 5,pp. 3219–3222, 2003.
[13] D. Low et al., “A method for the reconstruction of four-
dimensional synchronized CT scans acquired during free
breathing,” Medical Physics, vol. 30, No. 6, 2003.
[14] L. Zhang and J. Reinhardt, “3D pulmonary CT image regis-
tration with a standard lung atlas,” Proc. SPIE Conf. Medical
Imaging, 4322, pp. 67-77, 2000.
[15] B. Li, G. E. Christensen, J. Dill, E. A. Hoffman, and J. M.
Reinhardt, “3-D inter-subject warping and registration of pul-
monary CT images for a human lung model,” Proc. SPIE Conf.
Medical Imaging, vol. 4683, pp. 324–335, San Diego, CA,
2002.
[16] A. El-Baz, A. Ali, A.A. Farag, G. L.Gimel’farb, “A Novel Ap-
proach for Image Alignment Using a Markov-Gibbs Appear-
ance Model,” Proc. of International Conference on Medical
Image Computing and Computer-Assisted Intervention (MIC-
CAI’06), Copenhagen, Denmark, October 1–6, 2006, pp. 734–
741.
[17] J. A. Sethian, “Fast marching level set method for monoton-
ically advancing fronts,” Proc. Nat. Acad. Sci., USA, vol. 93,
pp. 1591–1595, Feb. 1996.
V - 504