# A Novel Approach for Automatic Follow-Up of Detected Lung Nodules

**ABSTRACT** Our long term research goal is to develop an image-based approach for early diagnosis of lung nodules that may lead to lung cancer. This paper focuses on monitoring the progress of detected lung nodules in successive chest low dose CT (LDCT) scans of a patient using non-rigid registration. In this paper, we propose a new methodology for 3D LDCT data registration. The registration methodology is non-rigid and involves two steps: global alignment of one scan (target data) to another scan (reference data) using the learned prior appearance model followed by local alignments in order to correct for intricate deformations. From two subsequent chest scans, visual appearance of the chest images, after equalizing their signals, are modeled with a Markov-Gibbs random field with pairwise interaction. Our approach is based on finding the affine transformation to register one data set (target data) to another data set (reference data) by maximizing a special Gibbs energy function using a gradient descent algorithm. To get accurate appearance model, we developed a new approach to an automatically select the most important cliques that describe the visual appearance of LDCT data. To handle local deformations, we propose a new approach based on deforming each voxel over evolving closed and equi-spaced surfaces (iso-surfaces) to closely match the prototype. The evolution of the iso-surfaces is guided by an exponential speed function in the directions minimizing distances between corresponding pixel pairs on the iso-surfaces on both data sets. Our preliminary results on 10 patients show that the proper registration could lead to precise identification of the progress of the detected lung nodules.

**0**

**0**

**·**

**0**Bookmarks

**·**

**58**Views

Page 1

A NOVEL APPROACH FOR AUTOMATIC FOLLOW–UP OF DETECTED LUNG NODULES

Ayman El-Baz1, Georgy Gimel’farb2, Robert Falk3, and Mohamed A. El-Ghar4

1Bioengineering Department, University of Louisville, Louisville, KY, USA.

2Department of Computer Science, University of Auckland, Auckland, New Zealand.

3Director, Medical Imaging Division, Jewish Hospital, Louisville, KY, USA.

4Urology and Nephrology Department, University of Mansoura, Mansoura, Egypt.

ABSTRACT

Our long term research goal is to develop an image-based approach

forearly diagnosisof lung nodulesthat may lead to lung cancer. This

paper focuses on monitoring the progress of detected lung nodules

in successive chest low dose CT (LDCT) scans of a patient using

non-rigid registration. In this paper, we propose a new methodol-

ogy for 3D LDCT data registration. The registration methodology

is non-rigid and involves two steps: global alignment of one scan

(target data) to another scan (reference data) using the learned prior

appearance model followed by local alignments in order to correct

for intricate deformations. From two subsequent chest scans, vi-

sual appearance of the chest images, after equalizing their signals,

are modeled with a Markov-Gibbs random field with pairwise inter-

action. Our approach is based on finding the affine transformation

to register one data set (target data) to another data set (reference

data) by maximizing a special Gibbs energy function using a gra-

dient descent algorithm. To get accurate appearance model, we de-

veloped a new approach to an automatically select the most impor-

tant cliques that describe the visual appearance of LDCT data. To

handle local deformations, we propose a new approach based on de-

forming each voxel over evolving closed and equi-spaced surfaces

(iso-surfaces) to closely match the prototype. The evolution of the

iso-surfaces is guided by an exponential speed function in the direc-

tions minimizing distances between corresponding pixel pairs on the

iso-surfaces on both data sets. Our preliminary results on 10 patients

show that the proper registration could lead to precise identification

of the progress of the detected lung nodules.

Index Terms— Lung cancer, Low dose computed tomography,

non-rigid registration, rigid registration.

1. INTRODUCTION

Because lung cancer is the most common cause of cancer deaths,

fast and accurate analysis of pulmonary nodules is of major impor-

tance for medical computer-aided diagnostic systems (CAD). In [1]

we introduced a fully automatic nodule detection algorithm showing

the accuracy up to 93.3% on the experimental database containing

200 real LDCT chest data sets with 36,000 2D slices. In [2,3] we

introduced an accurate segmentation approach to segment the de-

tected lung nodules from LDCT images. Below, we focus in the

next CAD stage, namely, on accurate registration of the detected

lung nodules for subsequent volumetric measurements to monitor

how the detected lung nodules change over the time.

We use a three-step procedure to separate the nodules from their

background: (i) an initial LDCT slice is segmented with algorithms

introduced in [4] to isolate lung tissues from surrounding structures

in the chest cavity as shown in Fig. 1, (ii) registration of two succes-

sive CT scan data sets which are taken at two different times, and

(a)(b)(c)

Fig. 1. Step 1 [4] of our segmentation: an LDCT slice (a) with

isolated lungs (b), and (c) the normalized segmented lung image.

(iii) the nodules in the isolated lung regions are segmented by evolv-

ing deformable boundaries under forces that depend on the learned

current and prior appearance models. At Step1 each LDCT slice is

modelled as a bi-modal sample from a simple Markov–Gibbs ran-

dom field of interdependent region labels and conditionally indepen-

dent voxel intensities (gray levels). This step is necessary for more

accurate separation of nodules from the lung tissues at Step 3 be-

cause voxels of both the nodules and other chest structures around

the lungs are of quite similar intensity. In this paper we will focus

on the second step.

Previous work. Tracking the temporal behavior of a nodule is

a complicated task because of the change in the patient’s position

at each data acquisition, as well as the effects of heart beats and

respiration of the patient. In order to get accurate measurements

about the progress of lung nodules over the time, all this motion

should be compensated by registering CT data sets taken at different

time periods with each other. In the literature, many methods have

been described for the medical image registration problem (see [5]),

and also for the compensation of the lung motion (see [6]). Below,

we will give some examples of previous work on CT lung images

registration.

For the follow-up of small nodules, Brown et al. [7] developed a

patient-specific model with 81% success in 27 nodules. Ko et al. [8]

used the center mass point of the structures and applied rigid and

affine image registration techniques with 96% success in 58 nodules

of 10 patients.

To account for the non-rigid motion and deformation of the lung,

Woods et al. [9] developed an objective function using an anisotropic

smoothness constraint and a model based on continuum mechan-

ics. Wood’s algorithm required the detection and registration of

feature points as explained in [10], and then interpolating the dis-

placement by the model of continuum mechanics. In Wood’s study,

the difference of the estimated and real volumes was calculated to

be 1.6%. In 2003, Dougherty et al. [11] developed an optical flow

method, a model-based motion estimation technique for estimating

first a global parametric transformation and then local deformations.

This method allowed the alignment of serial CT images with a 95%

correlation. Another optical flow analysis approach was developed

V - 5011-4244-1437-7/07/$20.00 ©2007 IEEEICIP 2007

Page 2

by Naqa et al. [12], where the optical flow analysis was combined

with the information of a spirometer, a device measuring the airflow

into and out of the lungs to track the breathing motion automatically.

The spirometry approach used in Naqa’s study was based on the re-

construction techniques for 4D CT during free breathing proposed

by [13].

There are also studies using CT lung images for pulmonary regis-

tration. For this purpose, Zhang et al. [14] used a standard lung atlas

to analyze the pulmonary structures in CT images. This atlas is reg-

istered to new images by a 3D surface based registration technique

composed of global transformation and local elastic transformations.

Li et al. [15] used feature points for correspondence and landmark &

intensity based registration algorithms to warp a template image to

the rest of the lung volumes.

2. METHODS

The objective of the proposed image analysis approach is to follow

up the detected lung nodules from LDCT images. To achieve this

goal, an image analysis system consisting of three steps is proposed.

These steps are: 1) segmentation of lung from LDCT images, 2) a

non-rigid registration approach to align two successive LDCT scans

and to correct the motion artifacts caused by breathing and patient

motion, and (3) segmentation of the lung nodules. In this paper we

will focus on the second step and the first and third steps are shown

in details in [2–4].

3. GLOBAL ALIGNMENT

Before describing the mathematical detail of the proposed global

alignment approach, we define the following notations:

Basic notation. We denote Q = {0,...,Q − 1}; R = [(x,y,z) :

x = 0,...,X − 1;y = 0,...,Y − 1;z = 0,...,Z − 1], and

Rp ⊂ R a finite set of scalar image signals (e.g. gray levels), a 3D

arithmetic lattice supporting digital LDCT data g : R → Q, and its

arbitrary-shaped part occupied by the prototype, respectively. A fi-

nite set N = {(ξ1,η1,ζ1),...,(ξn,ηn,ζn)} of (x,y,z)-coordinate

offsets defines neighbors {((x + ξ,y + η,z + ζ),(x − ξ,y −

η,z − ζ)) : (ξ,η,ζ) ∈ N} ∧ Rp interacting with each pixel

(x,y,z) ∈ Rp. The set N yields a neighborhood graph on Rp

to specify translation invariant pairwise interactions with n families

Cξ,η,ζof cliques cξ,η,ζ(x,y,z) = ((x,y,z),(x + ξ,y + η,z + ζ))

(see Fig. 2). Interaction strengths are given by a vector VT=

?VT

transposition.

ξ,η,ζ: (ξ,η,ζ) ∈ N?

of potentials VT

ξ,η,ζ =

?Vξ,η,ζ(q,q?) :

(q,q?) ∈ Q2?depending on signal co-occurrences; here T indicates

Fig. 2. Pairwise pixel interaction MGRF model.

3.1. Data Normalization

To account for monotone (order-preserving) changes of signals (e.g.

due to different illumination or sensor characteristics), the LDCT

images are equalized using the cumulative empirical probability dis-

tributions of their signals.

3.2. MGRF based appearance model

Generic MGRF with multiple pairwise interaction (Fig. 2) [4,16],

the Gibbs probability P(g) ∝ exp(E(g)) of an object g aligned

with the prototype g◦on Rpis specified with the Gibbs energy

E(g) = |Rp|VTF(g)

(1)

where FT(g) is the vector of scaled empirical probability distribu-

tions of signal co-occurrences over each clique family: FT(g) =

[ρξ,η,ζFT

ative size of the family and Fξ,η,ζ(g) = [fξ,η,ζ(q,q?|g) : (q,q?) ∈

Q2]T; here, fξ,η,ζ(q,q?|g) =

bilities of signal co-occurrences, and Cξ,η,ζ;q,q?(g) ⊆ Cξ,η,ζ is a

subfamily of the cliques cξ,η,ζ(x,y,z) supporting the co-occurrence

(gx,y,z = q, gx+ξ,y+η,z+ζ = q?) in g. The co-occurrence distribu-

tions and the Gibbs energy for the object are determined over Rp,

i.e. within the prototype boundary after an object is affinely aligned

with the prototype. To account for the affine transformation, the ini-

tial image is resampled to the back-projected Rpby interpolation.

The appearance model consists of the neighborhood N and the

potential V to be learned from the prototype.

ξ,η,ζ(g) : (ξ,η,ζ) ∈ N] where ρξ,η,ζ =

|Cξ,η,ζ|

|Rp|

is the rel-

|Cξ,η,ζ;q,q?(g)|

|Cξ,η,ζ|

are empirical proba-

Learning the potentials: The MLE of V is proportional in the

first approximation to the scaled centered empirical co-occurrence

distributions for the prototype [16]:

?

where U is the vector with unit components. The common scaling

factor λ is also computed analytically; it is approximately equal to

Q2if Q ? 1 and ρξ,η,ζ ≈ 1 for all (ξ,η,ζ) ∈ N. In our case it can

be set to λ = 1 because the registration uses only relative potential

values and energies.

Vξ,η,ζ = λρξ,η,ζ

Fξ,η,ζ(g◦) −

1

Q2U

?

; (ξ,η,ζ) ∈ N

Learning the characteristic neighbors: To find the charac-

teristic neighborhood set N, the relative energies Eξ,η,ζ(g◦) =

ρξ,η,ζVT

ances of the corresponding empirical co-occurrence distributions,

are compared for a large number of possible candidates. To automat-

ically select the characteristic neighbors, we consider an empirical

probability distribution of the energies as a mixture of a large “non-

characteristic” low-energy component and a considerably smaller

characteristic high-energy component: P(E) = πPlo(E) + (1 −

π)Phi(E). Both the components Plo(E), Phi(E) are of arbitrary

shape and thus are approximated with linear combinations of posi-

tive and negative discrete Gaussians (efficient EM-based algorithms

introduced in [4,16] are used for both the approximation and esti-

mation of π).

Appearance-based registration: The object g is affinely trans-

formed to (locally) maximize its relative energy E(ga) = VTF(ga)

under the learned appearance model [N,V]. Here, ga is the part

of the object image reduced to Rp by the affine transformation

a = [a11,...,a23]: x?= a11x + a12y + a13z + a14; y?=

a21x + a22y + a23z + a24; z?= a31x + a32y + a33z + a34.

The initial transformation is a pure translation with a11 = a22 = 1;

ξ,η,ζFξ,η,ζ(g◦) for the clique families, i.e. the scaled vari-

V - 502

Page 3

a12 = a21 = 0, ensuring the most “energetic” overlap between the

object and prototype. Then the gradient search for the local energy

maximum closest to the initialization selects the 12 parameters a.

4. LOCAL MOTION MODEL

To handle local deformations, we propose a new approach based

on deforming the object over evolving closed and equi-spaced con-

tours/surfaces (iso-contours/surfaces) to closely match the proto-

type. The evolution of the iso-contours/surfaces is guided by an

exponential speed function in the directions minimizing distances

between corresponding pixel pairs on the iso-contours/surfaces on

both images. The normalized cross correlation is used to find the

correspondent points between these iso-contours/surfaces.

The first step of our approach is to generate the distance map in-

side the object using fast marching level sets [17]. The second step

is to use this distance map to generate iso-surfaces (Fig. 3). Note

that the number of iso-contours/surfaces, which is not necessarily

the same for both images, depends on the accuracy and the speed

required by the user. The third step consists in finding the corre-

spondences between the iso-surfaces using normalized cross corre-

lation. The final step is the evolution of the iso-surfaces; here, our

goal is to deform the iso-surfaces in the first data set (target image)

to match the iso-surfaces in the second data set (reference image).

Before stating the evolution equation, let us define the following:

(a) Reference data (b) Target data

Fig. 3. Equal spaced generated surfaces.

Fig. 4. Evolution scenario.

• bh

on the reference data, pk = (xk,yk,zk) forming a circularly

connected chain of line segments (p1,p2), ..., (pK−1,pK),

(pK,p1).

• bγ

γ on the target data, pn = (xn,yn,zn) forming a circularly

connected chain of line segments (p1,p2), ..., (pN−1,pN),

(pN,p1).

g1= [ph

k: k = 1,...,K] – K control points on surface h

g2= [pγ

n: n = 1,...,N] – N control points on surface

• S(Ph

surface h in image g1and its corresponding point on surface γ

in image g2,

k,Pγ

n) denotes the Euclidean distance between a point on

• S(Pγ

on surface γ in image g1and its nearset point on surface γ −1

in image g1

n,Pγ−1

n

) denotes the Euclidean distance between a point

• ν(.) is the propagation speed function .

The evolution bτ → bτ+1of the deformable boundary b in dis-

crete time, τ = 0,1,..., is specified by the system of difference

equations pγ

ν(Pγ

and un,τ is the unit vector along the ray between two correspondant

points. The propagation speed function is selected so as to satisfy the

following conditions: ν(Pγ

ν(Pγ

The latter condition, known as the smoothness constraint, prevents

the current point from cross-passing the closest neighbor surfaces

as shown in Fig. 4.Note that the function ν(Pγ

exp?β(Pγ

?

n,τ+1= pγ

n,τ+ ν(Pγ

n,τ)un,τ; n = 1,...,N, where

n,τ) is a propagation speed function for the control point Pγ

n,τ

n,τ) = 0 if S(Ph

k,Pγ

k,Pγ

n,τ),S(Pγ

n,τ) = 0, otherwise

n,τ,Pγ+1

n,τ) = min?S(Ph

n,τ),S(Pγ

n,τ,Pγ−1

n,τ)?.

n,τ) = −1 +

n,τ)S(Ph

k,Pγ

n,τ)?; satisfies the above conditions, where

S(Ph

β(Pγ

n,τ) is the propagation term such as, at each surface point

min?

β(Pγ

n,τ) =

ln

k,Pγ

n,τ),S(Pγ

n,τ,Pγ−1

n,τ),S(Pγ

n,τ,Pγ+1

n,τ)?

+1

?

S(Ph

k,Pγ

n,τ)

.

5. EXPERIMENTAL RESULTS AND CONCLUSIONS

The proposed non-registration approach is tested on clinical datasets

collected from 10 patients. Each patient has five scans, and the pe-

riod between each two successive CT scans is three months. This

preliminary clinical database collected by low dose CT scan proto-

col with the following scanning parameters: slice thickness of 2.5

mm reconstructed every 1.5 mm, scanning pitch 1.5, pitch 1 mm,

KV 140, MA 100, and F.O.V 36 cm. Figure 5(d) shows checker-

board visualization between the data sets shown in Fig. 5(a) and the

aligned data set shown in Fig. 5(c) to demonstrate the effect of the

motion of the lung tissues. It can be seen that the connectivity at the

edges of the lung, between the two volumes is not smooth when us-

ing the global model only, this is due to the local deformation which

comes from breathing and heart beats. Figure 5 shows the results

after applying the local deformation model. It shows the connec-

tivity at the edges of lung region, and between the two volumes is

smoother when using the proposed local deformation model.

After the two volumes of different time instants are registered,

the task is to find out if the nodules are growing or not. For this pur-

pose, after registration, we are segmenting the lung nodules using

our previous approach [2,3]. Once the nodules are segmented in the

original and registered image sequences, the volumes of the nodules

are calculated using the Δx, Δy, and Δz values from the scanner. In

our studies, these values are 0.7mm, 0.7mm, and 2.5mm respec-

tively. Figure 6 shows the detected changes in two detected lung

nodules for two different patients over one year.

In this paper, we introduced a new approach for the non-rigid reg-

istration of spiral LDCT images. The proposed algorithm consist

of two steps: global alignment of one scan (target data) to another

scan (reference data) using the learned prior appearance model fol-

lowed by local alignments in order to correct for intricate deforma-

tions. The preliminary results on 10 patients show that the proper

registration could lead to precise identification of the progress of the

detected lung nodules.

V - 503

Page 4

(a)(b)(c)

(d) (e)(f)

Fig. 5. (a) Reference data, (b) target data, (c) transformed data using

12 degree of freedom affine transformation, (d) checkerboard visual-

izationtoshowthemotioneffectofthelungtissues, (e)ournon-rigid

regeneration results, and (g) checkerboard visualization to show the

quality of the proposed local deformation model.

Fig. 6. The results of the proposed follow up registration algorithm

for two patients over one year

6. REFERENCES

[1] A. A. Farag, A. El-Baz, and G. Gimelfarb, “Quantitative Nod-

ule Detection in Low Dose Chest CT Scans: New Template

Modeling and Evaluation for CAD System Design,” Proc. of

International Conference on Medical Image Computing and

Computer-Assisted Intervention (MICCAI), Palm Springs, Cal-

ifornia, USA, October 26-29, 2005, pp. 720-728.

[2] A. Farag, A. El-Baz, G. Gimel’farb, R. Falk, M. Abou El-

Ghar, T. Eldiasty, S. Elshazly, ”Appearance Models for Ro-

bust Segmentation of Pulmonary Nodules in 3D LDCT Chest

Images,” Proc. of International Conference on Medical Im-

age Computing and Computer–Assisted Intervention (MIC-

CAI’06), Copenhagen, Denmark, October 1–6, 2006, pp. 662-

670.

[3] A. El-Baz, A. A. Farag, G. L. Gimel’farb, R. Falk, M. Abou

El-Ghar, T. Eldiasty, A Framework for Automatic Segmen-

tation of Lung Nodules from Low Dose Chest CT Scans,

Proc. of IEEE International Conference on Pattern Recognition

(ICPR06), Hong Kong, August 2024, 2006, pp. 611-614.

[4] A. A. Farag, A. El-Baz, and G. Gimel’farb, “Precise Segmen-

tation of Multi-modal Images,” IEEE Transactions on Image

Processing, vol. 15, no. 4, pp. 952-968, April 2006.

[5] J. B. A. Maintz and M. A. Viergever, “A Survey of Medical

Image Registration,” Medical Image Analysis, vol.2, pp.1–36,

1998.

[6] J. Ko and D. Naidich, “Computer-aided diagnosis and the eval-

uation of lung disease,” Journal of thoracic imaging, pp. 19–

136, 2004.

[7] M. Brown, M. McNitt-Gray, N. Mankovich et al., “Method for

segmenting chest CT image data using an anatomical model:

preliminary results,” IEEE Trans Med Imaging, vol. 20, pp.

242-1250, 2001.

[8] J. Ko, and M. Betke, “Chest CT: automated nodule detection

and assessment of change over timepreliminary experience,”

Radiology, vol. 218, pp. 267–273, 2001.

[9] K. Woods, L. Fan, C. W. Chen, and Y. Wang, “Model Sup-

ported Image Registration and Warping for Change Detec-

tion in Computer-aided Diagnosis,” Applied Imagery Pattern

Recognition(AIPR)AnnualWorkshops, WashingtonDC,2000.

[10] L. Fan and C. W. Chen, “An integrated approach to 3D warp-

ing and registration from lung images,” Proceedings of SPIE

Conf.DevelopmentsinX-RayTomographyII,Denver, CO,July

1999.

[11] L.Dougherty, J.AsmuthandW.Gefter, “AlignmentofCTlung

volumes with an optical flow method,” Academic radiology,

vol. 10, pp. 249–254, 2003.

[12] I. Naqa, D. Low, J. Deasy et al., “Automated breathing mo-

tion tracking for 4D computed tomography,” Nuclear Science

Symposium Conference Record, vol. 5,pp. 3219–3222, 2003.

[13] D. Low et al., “A method for the reconstruction of four-

dimensional synchronized CT scans acquired during free

breathing,” Medical Physics, vol. 30, No. 6, 2003.

[14] L. Zhang and J. Reinhardt, “3D pulmonary CT image regis-

tration with a standard lung atlas,” Proc. SPIE Conf. Medical

Imaging, 4322, pp. 67-77, 2000.

[15] B. Li, G. E. Christensen, J. Dill, E. A. Hoffman, and J. M.

Reinhardt, “3-D inter-subject warping and registration of pul-

monary CT images for a human lung model,” Proc. SPIE Conf.

Medical Imaging, vol. 4683, pp. 324–335, San Diego, CA,

2002.

[16] A. El-Baz, A. Ali, A.A. Farag, G. L.Gimel’farb, “A Novel Ap-

proach for Image Alignment Using a Markov-Gibbs Appear-

ance Model,” Proc. of International Conference on Medical

Image Computing and Computer-Assisted Intervention (MIC-

CAI’06), Copenhagen, Denmark, October 1–6, 2006, pp. 734–

741.

[17] J. A. Sethian, “Fast marching level set method for monoton-

ically advancing fronts,” Proc. Nat. Acad. Sci., USA, vol. 93,

pp. 1591–1595, Feb. 1996.

V - 504