# A fast inverse consistent deformable image registration method based on symmetric optical flow computation.

**ABSTRACT** Deformable image registration is widely used in various radiation therapy applications including daily treatment planning adaptation to map planned tissue or dose to changing anatomy. In this work, a simple and efficient inverse consistency deformable registration method is proposed with aims of higher registration accuracy and faster convergence speed. Instead of registering image I to a second image J, the two images are symmetrically deformed toward one another in multiple passes, until both deformed images are matched and correct registration is therefore achieved. In each pass, a delta motion field is computed by minimizing a symmetric optical flow system cost function using modified optical flow algorithms. The images are then further deformed with the delta motion field in the positive and negative directions respectively, and then used for the next pass. The magnitude of the delta motion field is forced to be less than 0.4 voxel for every pass in order to guarantee smoothness and invertibility for the two overall motion fields that are accumulating the delta motion fields in both positive and negative directions, respectively. The final motion fields to register the original images I and J, in either direction, are calculated by inverting one overall motion field and combining the inversion result with the other overall motion field. The final motion fields are inversely consistent and this is ensured by the symmetric way that registration is carried out. The proposed method is demonstrated with phantom images, artificially deformed patient images and 4D-CT images. Our results suggest that the proposed method is able to improve the overall accuracy (reducing registration error by 30% or more, compared to the original and inversely inconsistent optical flow algorithms), reduce the inverse consistency error (by 95% or more) and increase the convergence rate (by 100% or more). The overall computation speed may slightly decrease, or increase in most cases because the new method converges faster. Compared to previously reported inverse consistency algorithms, the proposed method is simpler, easier to implement and more efficient.

**1**Bookmark

**·**

**99**Views

- [Show abstract] [Hide abstract]

**ABSTRACT:**Respiration-induced kinematics of thoracic tumors suggests a simple analogy with elasticity, where a strain tensor is used to characterize the volume of interests. The application of the biomechanical framework allows for the objective determination of tumor characteristics.Medical Physics 07/2014; 41(7):073503. · 3.01 Impact Factor -
##### Article: Development and evaluation of an articulated registration algorithm for human skeleton registration

[Show abstract] [Hide abstract]

**ABSTRACT:**Accurate registration over multiple scans is necessary to assess treatment response of bone diseases (e.g. metastatic bone lesions). This study aimed to develop and evaluate an articulated registration algorithm for the whole-body skeleton registration in human patients. In articulated registration, whole-body skeletons are registered by auto-segmenting into individual bones using atlas-based segmentation, and then rigidly aligning them. Sixteen patients (weight = 80–117 kg, height = 168–191 cm) with advanced prostate cancer underwent the pre- and mid-treatment PET/CT scans over a course of cancer therapy. Skeletons were extracted from the CT images by thresholding (HU>150). Skeletons were registered using the articulated, rigid, and deformable registration algorithms to account for position and postural variability between scans. The inter-observers agreement in the atlas creation, the agreement between the manually and atlas-based segmented bones, and the registration performances of all three registration algorithms were all assessed using the Dice similarity index—DSIobserved, DSIatlas, and DSIregister. Hausdorff distance (dHausdorff) of the registered skeletons was also used for registration evaluation. Nearly negligible inter-observers variability was found in the bone atlases creation as the DSIobserver was 96 ± 2%. Atlas-based and manual segmented bones were in excellent agreement with DSIatlas of 90 ± 3%. Articulated (DSIregsiter = 75 ± 2%, dHausdorff = 0.37 ± 0.08 cm) and deformable registration algorithms (DSIregister = 77 ± 3%, dHausdorff = 0.34 ± 0.08 cm) considerably outperformed the rigid registration algorithm (DSIregsiter = 59 ± 9%, dHausdorff = 0.69 ± 0.20 cm) in the skeleton registration as the rigid registration algorithm failed to capture the skeleton flexibility in the joints. Despite superior skeleton registration performance, deformable registration algorithm failed to preserve the local rigidity of bones as over 60% of the skeletons were deformed. Articulated registration is superior to rigid and deformable registrations by capturing global flexibility while preserving local rigidity inherent in skeleton registration. Therefore, articulated registration can be employed to accurately register the whole-body human skeletons, and it enables the treatment response assessment of various bone diseases.Physics in Medicine and Biology 03/2014; 59(6):1485. · 2.92 Impact Factor - SourceAvailable from: Wafa Moualhi[Show abstract] [Hide abstract]

**ABSTRACT:**The motivation of this work is to register a tumor brain magnetic resonance (MR) image with a normal brain atlas. A normal brain atlas is deformed in order to take account of the presence of a large space occupying tumor. The method use a priori model of tumor growth assuming that the tumor grows in a radial way from a starting point. First, an affine transformation is used in order to bring the patient image and the brain atlas in a global correspondence. Second, the seeding of a synthetic tumor into the brain atlas provides a template for the lesion. Finally, the seeded atlas is deformed combining a method derived from optical flow principles and a model for tumor growth (MTG). Results show that an automatic segmentation method of brain structures in the presence of large deformation can be provided.7th International Conference on Machine Vision ICMV 2014, Milan, Italy; 11/2014

Page 1

A fast inverse consistent deformable image registration method based on symmetric optical

flow computation

This content has been downloaded from IOPscience. Please scroll down to see the full text.

Download details:

IP Address: 221.130.23.146

This content was downloaded on 04/10/2013 at 19:47

Please note that terms and conditions apply.

2008 Phys. Med. Biol. 53 6143

(http://iopscience.iop.org/0031-9155/53/21/017)

View the table of contents for this issue, or go to the journal homepage for more

Home Search CollectionsJournalsAbout Contact us My IOPscience

Page 2

IOP PUBLISHING

PHYSICS IN MEDICINE AND BIOLOGY

Phys. Med. Biol. 53 (2008) 6143–6165

doi:10.1088/0031-9155/53/21/017

A fast inverse consistent deformable image registration

method based on symmetric optical flow computation

Deshan Yang1, Hua Li, Daniel A Low, Joseph O Deasy

and Issam El Naqa

Department of Radiation Oncology, School of Medicine, Washington University in St. Louis,

4921 Parkview Place, LL, St. Louis, MO 63110, USA

Received 10 June 2008, in final form 16 September 2008

Published 14 October 2008

Online at stacks.iop.org/PMB/53/6143

Abstract

Deformable image registration is widely used in various radiation therapy

applications including daily treatment planning adaptation to map planned

tissue or dose to changing anatomy. In this work, a simple and efficient inverse

consistency deformable registration method is proposed with aims of higher

registrationaccuracyandfasterconvergencespeed. Insteadofregisteringimage

I to a second image J, the two images are symmetrically deformed toward one

anotherinmultiplepasses, untilbothdeformedimagesarematchedandcorrect

registrationisthereforeachieved. Ineachpass,adeltamotionfieldiscomputed

by minimizing a symmetric optical flow system cost function using modified

optical flow algorithms. The images are then further deformed with the delta

motion field in the positive and negative directions respectively, and then used

for the next pass. The magnitude of the delta motion field is forced to be less

than 0.4 voxel for every pass in order to guarantee smoothness and invertibility

for the two overall motion fields that are accumulating the delta motion fields

in both positive and negative directions, respectively. The final motion fields

to register the original images I and J, in either direction, are calculated by

inverting one overall motion field and combining the inversion result with the

other overall motion field. The final motion fields are inversely consistent

and this is ensured by the symmetric way that registration is carried out. The

proposed method is demonstrated with phantom images, artificially deformed

patient images and 4D-CT images. Our results suggest that the proposed

method is able to improve the overall accuracy (reducing registration error by

30% or more, compared to the original and inversely inconsistent optical flow

algorithms),reducetheinverseconsistencyerror(by95%ormore)andincrease

the convergence rate (by 100% or more). The overall computation speed may

slightly decrease, or increase in most cases because the new method converges

1Author to whom any correspondence should be addressed.

0031-9155/08/216143+23$30.00© 2008 Institute of Physics and Engineering in MedicinePrinted in the UK6143

Page 3

6144 D Yang et al

faster. Compared to previously reported inverse consistency algorithms, the

proposed method is simpler, easier to implement and more efficient.

(Some figures in this article are in colour only in the electronic version)

1. Introduction

It has been witnessed in the recent years that anatomical image information (kVCT, daily

megavoltage-CTorcone-beam-CT,MRI,etc)andfunctionalimageinformation(PET,SPECT,

fMRI, etc) were increasingly adopted into patient radiation treatment management. Image

registration is a procedure to transform different image datasets into a common coordinate

system so that corresponding points of the images are matched and the complementary

information from the different images can be analyzed for different diagnostic and therapeutic

purposes. Kessler provided a comprehensive review of image registration for radiation

therapy (Kessler 2006). Image registration algorithms can be generally grouped into rigid

registration and deformable (non-rigid) registration according to the type of transformation

thatanalgorithmapplies. Whilerigidregistrationonlyappliesrigid(oraffine)transformations

with limited number of free parameters (up to 12), deformable registration uses much larger

number of free parameters (up to three times the total number of voxels in an image) in order

to describe non-rigid tissue deformation in 3D space.

Deformable image registration can be computed based on features extracted from the

images, e.g. points (Kessler et al 1991), lines (Balter et al 1992) and surfaces (van Herk

and Kooy 1994) or based on metrics directly derived from the image intensity values,

e.g. mean square error (MSE) (Thirion 1998) for images from the same modality, mutual

information (MI) (Viola and Wells 1995) and cross-correlation (Kim and Fessler 2004) for

images from different modalities. Mean-squared-error (MSE) based CT to CT deformable

image registration is especially important for radiation therapy applications, including patient

response monitoring, treatment adaptation, dose tracking and patient motion modeling, etc

(Lu et al 2004, Wang et al 2005, Sarrut et al 2007). This paper focuses on such algorithms.

Regardless of the image registration algorithm, registration accuracy is always one of the

mostimportantaspectsthataffectstheclinicalapplicabilityofthealgorithm. Formostmedical

images, registration results often cannot be validated on a voxel-by-voxel basis because there

is no such ground truth available. While landmark matching and structure volume matching

are often used for results validation, they are not voxel by voxel based and the overall accuracy

of such a validation is quite limited because landmarks or structures can only cover limited

regions of the entire image.

Inverse consistency, which means that the registration results are consistent from

registering the images in the forward direction (from image 1 to image 2) or in the reverse

direction (from image 2 to image 1), is often considered as one of the more feasible ways

for measuring image registration accuracy (Christensen et al 2006) for any registration

algorithms. This is based on the fact that results by an accurate registration algorithm must

be inversely consistent. Therefore, inverse consistency is always desirable for any deformable

registration algorithm in addition to its accuracy. For image-guided and adaptive radiation

therapy (ART) applications, such inverse consistency is not only desirable but also practically

useful. Information such as treatment planning contours, etc is defined on the treatment

planning CT, while daily doses, contours, etc are referenced to the daily images. Inverse

consistency registrations can provide voxel mapping in both directions so that information can

be consistently mapped from one image to the other image.

Page 4

A fast inverse consistent deformable image registration method 6145

Table 1. Notations used in this paper.

I

J

Id

?

x

V, U

?Vn

I ◦ V

V2◦ V1

V−1

The moving image, or the first image

The fixed image, or the second image

The difference image, Id= J − I

The image domain

Coordinates of image positions in ?

The ‘pull-back’ displacement motion vector fields

The delta motion field

=I(x−V(x)), the image I deformed by V

=V1(x−V2(x)) + V2(x), the composition of two motion fields

The inverted vector field of V

Computationspeedisalsoveryimportantforimage-guidedandadaptiveradiationtherapy

(ART). For example, registration needs to be computed quickly and accurately between the

treatment fractionations. In the future, such a computation may need to be completed online

while a patient is on the treatment table. Computation speed is also demanded by 4D-CT

based respiratory motion estimation because of the large amount of 4D image data.

Because of these reasons, we propose a new and efficient inversely consistent deformable

registration method in this paper. The new method uses a simplified system cost function

and solves registration in a symmetric way.

intensity gradients) from both images are symmetrically used in the computation, both

registrationaccuracyandconvergencespeedareimprovedintheproposedmethodcomparedto

asymmetrical inverse-inconsistency algorithms. Because the system cost function is simpler,

the overall computation speed is improved compared to other inverse consistency algorithms.

Because image information (intensity and

1.1. Optical flow deformable image registration and inverse consistency

1.1.1. Optical flow.

modality deformable image registration. They are based on image intensity and gradient

information. For two images I and J to be registered, let I be the moving image and J be the

fixed image, a displacement motion vector field V registers I to J so that

Optical flow algorithms are among the most used algorithms for single

J = I ◦ V ≡ I(x − V(x)),

(1)

where ◦ is the composition operator, V is the motion field and x is the spatial coordinate. The

motion field V is the displacement vector field instead of the transformation vector field. V

is often referred to as the deformation field or the optical flow field. Description of notations

used in this paper is in table 1.

V normally cannot be resolved by only using equation (1) because the system is

underdetermined. Other constraints, such as global smoothness, are often enforced in order

to successfully compute V. With the additional global smoothness constraints, the system cost

function could be written as

?

where R is the smoothness constraint (also known as regularization constraint) function,

? is the image domain and α is a constant. Many optical flow algorithms use R(V) =

tr((∇V)T(∇V)), where tr( ) is the matrix trace operator. If |V| is small, equation (2) could

E1=

?

((J − I ◦ V)2+ α2R(V))d?,

(2)

Page 5

6146 D Yang et al

(a) (b)

Figure 1. Illustration of asymmetric registration and inverse consistency error. Point A (in

image I) and B (in image J) are matching points. V is computed by registering I to J. U is computed

by registering image J to image I. (a) After imperfect asymmetric registrations, point A moves to

point A?and point B moves to point B?. (b) Using U, A?will be moved to A??. Similarly, B??is

B?moved by using V. The distance from A to A??, and from B to B??, are the inverse consistency

errors.

be expressed using a Taylor expansion of the first-order terms in the following differential

form:

?

where Id= J − I, ∇ is the gradient operator, · is the vector inner product operator.

V could be solved by minimizing E1with many numerical methods, either iteratively or

analytically. Barronetal(1994)havereviewedmanypublishedopticalflowalgorithms(Barron

etal1994,McCaneetal2001)andsummarizedthealgorithmsintofourcategories: differential

techniques, region-based matching, energy-based methods and phase-based techniques. This

paper uses the Horn–Schunck (HS) algorithm (Horn and Schunck 1981) and the demons

diffusionalgorithm(Thirion1998). Thesetwoalgorithmsbelongtothedifferentialtechniques

where the differential form of the system cost equation is solved using the image intensity and

gradient. Such differential optical flow algorithms are often referred to as small-motion-model

algorithms because they only work if |V| is sufficiently small so that the Taylor expansion

series can be applied.

E1=

?

((Id+ ∇I · V)2+ α2R(V))d?,

(3)

1.1.2. Registration in the inverse direction.

to image I in the backward direction, the second motion field U needs to be computed so

that

Traditionally, if image J needs to be registered

I = J ◦ U.

(4)

A similar system equation could be written as

?

Even if V has already been computed, U has to be computed independently because there

is unfortunately no direct dependence among the solutions of V and U. This is illustrated in

figure 1(a).

E2=

?

((I − J + ∇J · U)2+ α2R(U))d?.

(5)

1.1.3. Inverse consistency.

consistent so that registration could start with either image and the results are consistent.

Inverse consistency could be written as

It is desirable for many applications that V and U are inversely

V ◦ U = 0 = U ◦ V,

(6)

where the composition operator ◦ between two motion fields is defined in table 1.

Page 6

A fast inverse consistent deformable image registration method 6147

The inverse consistency error (ICE) could be then defined as

ICE1= |V ◦ U|

ICE2= |U ◦ V|.

(7)

(8)

If V and U are inversely consistent, ICE1and ICE2will be both 0. Otherwise, ICE1and

ICE2will not be 0 and may not be the same, as illustrated in figure 1(b). A combined inverse

consistency error term ICE can be defined as

ICE = (ICE1+ ICE2)/2.

(9)

1.2. Previous inverse consistency methods

It is generally difficult to have V and U consistent if the image registration computations for

bothdirectionsarecarriedoutseparatelyorwithoutexplicitconstraintsforinverseconsistency.

Therefore, most inverse consistency registration algorithms perform computations for both

directions simultaneously and explicitly constrain V and U to be, or closely to be, inversely

consistent.

Christensen and Johnson (2001) seem to be among the earliest groups to consider inverse

consistencyfordeformableimageregistration. Intheiralgorithm,VandUweresimultaneously

computed by minimizing the symmetric system cost equation (10), which contained the

similarityconstraint,inverseconsistencyconstraintandadiffeomorphismregularityconstraint.

Diffeomorphism refers to the continuous differentiability of the motion field as discussed

below.

?

+(V − U−1)2+ (V−1− U)2+ |L(V)|2+ |L(U)|2)d?,

where the linear elastic operator L = −a2∇2− b∇ + c, a, b and c are constants. U and V

were parameterized with Fourier sequences and solved iteratively. Both V and U needed to

be inverted to obtain V−1and U−1in every iteration. An inversion procedure, to be further

discussed later, was performed iteratively or analytically, for a displacement motion vector

field V, by minimizing |V−1◦ V| or |V−1◦ V|.

Alvarez et al (2007) proposed an algorithm based on the system cost equation (11). The

algorithm does not explicitly invert the forward and reverse motion fields during the iterations.

Instead, the inverse consistency error is computed and minimized per iteration.

?

+αER(J,U) + βES(V,U) + βES(U,V))d?,

E =

?((I ◦ V − J)2+ (J ◦ U − V)2

(10)

E =

?((I ◦ V − J)2+ (J ◦ U − V)2+ αER(I,V)

(11)

where the regularization constraint ER(I,V) = tr((∇V)TD(∇I)∇V), α and β are constants,

theinverseconsistencyconstraintES(V,U) = |U◦V|2andD (∇I)isaregularizedprojection

matrix in the direction perpendicular to ∇I.

Cachier and Rey (2000) analyzed the reasons why results of unidirectional registrations

are asymmetric and pointed out that inversely inconsistant approaches penalized the image

expansion more than the shrinkage. They proposed an inverse-invariant type system cost

equation given in equation (12) and two finite element implementations to solve the new cost

function, depending on where motion field inversion is being computed or not. Registration

Page 7

6148 D Yang et al

does not need to be performed simultaneously for both forward and reverse directions in this

method

?

Leow et al (2005) reported an approach to model the backward motion field by a function

oftheforwardmotionfield,thereforeinverseconsistencyregistrationcanbecomputedwithout

computing the inversion of the motion fields. They used a symmetric system cost function,

similar to equation (10), with the V−1and U−1replaced by the functions of V and U.

Diffeomorphism algorithms (Dupuis et al 1998, Christensen et al 1996, Trouve 1998) are

closely related to inverse consistency. Diffeomorphism means continuous, differentiable and

invertible. These algorithms are often referred to as large-motion-model algorithms because

the regularization term in the system cost function is different and the algorithms can compute

smooth and continuous large motions. Dupuis et al (1998) showed theoretically that the

solution for the diffeomorphism system cost equation is unique, smooth, differentiable and

invertible. It should be understood that the invertibility is not equal to inverse consistency and

diffeomorphism algorithms are not inversely consistent by default.

There are a few inverse consistency algorithms proposed under the diffeomorphism

framework. Joshi et al (2004) proposed a method to construct a template image from multiple

images for brain mapping. The major computation of this algorithm is done in the Fourier

frequency domain. The system cost equation is given by

??

where N is the total number of images,ˆI is the shape average image, which is updated during

the iterations, Viis the motion field to deform image i toˆI,viis the velocity vector field for

image i and Vi(x) =?1

and the system cost equation reduces to

?

Similar algorithms have also been proposed by Avants and Gee (2004) and by Beg and

Khan (2007). These algorithms are all based on the idea that both images are deformed toward

the ‘mean shape’ image in order to achieve better registration. Such an idea is quite similar

to the basic concept of the method investigated in this paper. We will further compare our

method to these algorithms in the later sections.

The goal of computing image registration with inverse consistency is to improve the

registration accuracy and to provide consistent motion fields for both registration directions.

Better accuracy has been achieved by adding additional inverse consistency constraints and

using symmetric system cost functions. However, solving the more complicated registration

problem is usually much slower.

E =

?

?

(1 + |dV|)(I ◦ V − J)2+ α

?

1 +

1

|dV|

?

?dV?2

?

d?.

(12)

E =

N

?

i=1

?

(Ii◦ Vi−ˆI)2d? +

?1

t=0

?

?

?Lvi?2d?dt

?

,

(13)

t=0vi(Vi((x,t),t))dt.

If the number of images is 2, then this method becomes an inverse consistency method

E =1

2

?

(I ◦ V − J ◦ U)2d? +

?1

t=0

?

?

?Lv1?2d?dt +

?1

t=0

?

?

?Lv2?2d?dt.

(14)

2. Methods and materials

2.1. Overview

Our method is similar to the inverse consistency diffeomorphism algorithms (Joshi et al 2004,

Avants and Gee 2004, Beg and Khan2007), however, focuses onsimplicityand computational

Page 8

A fast inverse consistent deformable image registration method 6149

Figure 2. Demonstration of the proposed inversely consistent registration method. Matching

points A and B are in image I and image J, respectively. After n passes, A is moved to point A?

and B is moved to point B?. A?and B?are in close proximity, but are not perfectly registered. Vn

and Unare the overall motion fields. The delta motion field ?Vnand ?Unare computed for each

pass.

efficiency. As illustrated in figure 2, I and J are symmetrically deformed pass-by-pass toward

each other. Inand Jndenote I and J deformed after pass n. Registration is achieved on Inand

Jnwhen Inand Jnmatches.

At pass n, a delta motion field ?Vn, is computed by minimizing a symmetric optical flow

system cost equation (to be discussed in the following section) using modified optical flow

algorithms. The two overall motion fields, Vnfor image I and Unfor image J, are updated by

accumulating ?Vnand −?Vnas

Vn= ?Vn◦ Vn−1

Un= (−?Vn) ◦ Un−1

and Inand Jnare then updated as

(15)

(16)

In= I ◦ Vn

Jn= J ◦ Un.

(17)

(18)

The two new deformed images Inand Jnwill be used for the next pass.

Initially, V0= U0= 0. Because ?Vnis a ‘pull-back’ motion field (defined on the voxel

grid of Inand Jn, instead of the voxel grid of I and J), the Vn?= −Unfor pass numbers n > 1,

therefore Vnand Unare updated individually. The magnitude of ?Vnis forced to be less than

0.4 voxel size in order to ensure the smoothness and invertibility of Vnand Unas discussed

below. If the registration direction is reversed, it can be shown that Inand Jnwill be swapped,

and consequently Vnand Unwill be swapped.

The final motion fields, VIJwhich registers I to J, and UJIwhich registers J to I, are

calculated as

VIJ= U−1

UJI= V−1

n

◦ Vn

◦ Un

(19)

n

(20)

from the last Vnand Un. It can be shown that VIJand UJIare inversely consistent to each other.

If the registration direction is reversed, Vnand Unsimply swap. VIJand UJIwill also be simply

swapped. Because VIJand UJIare inversely consistent, the final motion fields computed in the

forward and the backward registration directions are inversely consistent. Both VIJand UJI

can be computed in one step regardless of the registration direction.

2.2. Symmetric optical flow system cost equation

At pass n, we compute the delta motion fields ?Vnand ?Unto achieve further image

registration between the current deformed In−1and Jn−1. In−1is deformed using ?Vnand

Page 9

6150 D Yang et al

generates Inaccording to

In= In−1◦ ?Vn= In−1− ∇In−1· ?Vn

Jn−1is deformed using ?Unand generates Jnaccording to

Jn= Jn−1◦ ?Un= Jn−1− ∇Jn−1· ?Un

?Vnand ?Unare solved by minimizing the following new system cost equation:

?

To simplify the new equation, we add another hard constraint on ?Vnand ?Un

(21)

(22)

E =

?

((Jn− In)2+ β2R(?Vn) + β2R(?Un))d?.

(23)

?Vn+ ?Un= 0(24)

and we select the smoothness regularity function R( ) so that

R(?Vn) = R(−?Vn)

(25)

and let α2= 2β2, then the system cost equation could be rewritten in the following differential

form using Taylor expansion:

?

which is simplified into

?

where IS= In−1+ Jn−1and Id= Jn−1− In−1.

One can see that equation (27) has exactly the same form as equation (3). This means

that the intermediate deformation fields ?Vncould be solved with the same algorithms that

solve equation (3) while having ?Un= −?Vn. Most regularization functions, including the

ordinary optical flow global smoothness function R(V) = tr((∇V)T(∇V)), are good choices

for equation (25).

E =

?

((Jn−1− In−1+ (∇Jn−1+ ∇In−1) · ?Vn)2+ α2R(Vn))d?

(26)

E =

?

((Id+ ∇Is· ?Vn)2+ α2R(?Vn))d?,

(27)

2.3. Solving the system cost equation

2.3.1. Case 1: Horn–Schunck (HS) optical flow algorithm.

(HS) (Horn and Schunck 1981) algorithm solves equation (3) using the following iterative

solution:

Vk+1=¯Vk−(Id+¯Vk· ∇I)∇I

α2+ |∇I|2

where Vkis the motion field at iteration k,¯Vkis Vkaveraged for each pixel in the neighborhood

of that pixel.

To solve the new system cost equation (27), the iterative equation is modified slightly to

The original Horn–Schunck

,

(28)

Vk+1=¯Vk−(Id+¯Vk· ∇IS)∇IS

α2+ |∇IS|2

,

(29)

where IS= In−1+ Jn−1and Id= Jn−1− In−1.

After all iterations are finished, the last Vk+1is ?Vn, the desired solution for equation (27).

Page 10

A fast inverse consistent deformable image registration method 6151

2.3.2. Case II: using demons algorithm.

solves equation (3) using this iterative solution

?

where Gσis Gaussian lowpass filter with a window width σ, k is the iteration number.

To solve equation (27), we replace the gradient terms with ∇In−1+ ∇Jn−1. We also do

not have to use multiple iterations because we are already applying multiple passes at the In

and Jnlevels. In this way, the equation can be reduced to

?

|∇Is|2+ I2

The original demons algorithm (Thirion 1998)

Vk+1=

Vk−

(I ◦ Vk− J)∇J

|∇J|2+ (I ◦ Vk− J)2

?

∗ Gσ,

(30)

?Vn=

Id∇Is

d

?

∗ Gσ.

(31)

2.4. Inversion of Vnand Un

2.4.1. Guarantee of invertibility.

fields can be computed. To be invertible, Vnand Unshould be smooth, without folding and

the determinant of the Jacobian matrix should be strictly positive defined (Leow et al 2005).

However, neither the HS algorithm nor the demons algorithm guarantees the invertibility for

Vnand Un. Therefore, the following small-step multiple pass approach is used to ensure it.

ThestrategyistocomputeVnandUninsmallerincrementalsteps. If?Vnisdiffeomorphic,

then Vnand Un, which are accumulating ?Vnand −?Vn, will also be diffeomorphic. Such

an approach has been reported previously (Cootes et al 2004, Rueckert et al 2006). Rueckert

et al reported that the maximal displacement of the control points for their cubic B-Spline

algorithm needs to be less than 0.40 of the spacing of the control points for the motion field to

be diffeomorphic. Such a conclusion could be indirectly applied to the optical flow algorithms

by treating voxels in each image as B-Spline control points. If ?Vnis less than 0.4 voxel size,

?Vnwill be diffeomorphic and Vnand Unwill also be diffeomorphic.

We used the following ad hoc step after every pass to explicitly reduce the magnitude of

?Vnto 0.4 voxel if it is greater than 0.4 voxel

??Vn

0.4 × ?Vn/|?Vn|

There are alternative approaches to guarantee diffeomorphism for ?Vn. Vercauteren

et al (2007) reported using exp(?Vn) to replace ?Vn. The term exp(?Vn) is approximated

by composing ?Vn/n for n times. For example, exp(?Vn) = (?Vn/16) ◦ (?Vn/16) ◦ ··· ◦

(?Vn/16). Because ?Vn/n is diffeomorphic if n is large, exp(?Vn) will be diffeomorphic.

One problem of this approach is that exp(?Vn) ?= ?Vn, neither in direction nor in magnitude,

therefore exp(?Vn) is a rough but diffeomorphic approximation of ?Vn.

Neither our magnitude limiting procedure nor the method of using exp(?Vn) is perfect.

If ?Vnis accurate, then errors will be introduced by either method. Such errors have to be

recovered in the next pass, and could slow down the overall convergence. Our method is,

however, simpler to implement and more computationally efficient.

Smoothness of ?Vnafter the magnitude limiting procedure should not be a concern

because ?Vnis discrete and the largest possible magnitude difference of ?Vnbetween two

adjacent voxels is 0.4 × 2 = 0.8. However, if more smoothness is desired, ?Vncan be

smoothed by a Gaussian lowpass filter as Gσ(?Vn) → ?Vn, where σ is the window size. The

maximal magnitude of ?Vnwill still be less than 0.4 after such a lowpass filtering step and

the smoothed ?Vnwill still be diffeomorphic.

Vnand Unmust be invertible so that the final motion

?Vn⇐

|?Vn| ? 0.4

|?Vn| > 0.4.

(32)

Page 11

6152 D Yang et al

2.4.2. Motion field inversion.

way, as used in many diffeomorphism algorithms, is to integrate (accumulate) the inverse of

the delta motion fields during the passes. This means to integrate the inverse of ?Vnand

?Un(?Un= −?Vn). Because magnitude of ?Vnis small, (?Vn)−1can be approximated as

−?Vn. Integration of (?Vn)−1and (?Un)−1is slightly different from computation of Vnand

Unby accumulating ?Vnand ?Unbecause (?Vn)−1and (?Un)−1are the push-forward motion

fields while ?Vnand ?Unare the pull-back motion fields. However, the way to approximate

(?Vn)−1by −?Vndoes not work very well with multigrid approaches because a small ?Vnin

the coarse image resolution stage equals to larger motion in the finer resolution stage and will

have larger approximation errors. To reduce such error in the finer resolution stage, the spatial

step (maximal |?Vn|) in the coarse stage must be very small, much smaller than 0.4 voxel.

Using very small spatial steps contradicts the idea of using the multigrid approach since the

multigrid approach is applied to improve computation speed.

Methods todirectlycompute theinversemotionfieldhave been reported(Christensenand

Johnson 2001, Cachier and Rey 2000). Ashburner reported a fast method based on the idea of

tetrahedral and affine transformation inversion (Ashburner et al 2000). We used this method

in this work because it is computationally efficient and accurate. The method is already

implemented in the statistical parametric mapping (SPM) (Friston 2006) version 5 package.

We evaluated the code from SPM and demonstrated that it worked well for all our tested cases

with averaged error <0.05 pixel and maximal error <0.1 pixel.

Vnand Uncan be inverted by a few different ways. The easiest

2.5. The entire procedure

The entire inverse consistency method can be described in following pseudo code:

(1) Let the pass number n = 0 and V0= U0= 0.

(2) Compute the deformed image Inand Jnaccording to equations (17) and (18).

(3) Use one of the modified optical flow algorithms to perform registration between Inand Jn

and compute ?Vn+1.

(4) Limit the magnitude of ?Vn+1according to equation (32), and optionally smooth ?Vn+1

with a Gaussian lowpass filter.

(5) Let n = n + 1, update the overall motion fields Vnand Unaccording to equation (15) and

(16) and optionally smooth Vnand Unwith another Gaussian lowpass filter.

(6) If the results have not converged and n is less than the maximal step number allowed, and

then go back to step 2 for the next pass. Convergence is determined by checking whether

the maximal magnitude of ?Vnis less than a user set value, for example, 0.01 voxel.

(7) Otherwise, compute the final deformation fields according to equations (19) and (20).

The entire procedure is similar to a regular asymmetric registration procedure, with

additional steps 4 and 7. An important difference is that the computation needs to be carried

out for both images to update Inand Jn, Vnand Un, while a regular asymmetric procedure often

only needs to compute similar variables for one image. Optional smoothing in step 4 helps

to smooth ?Vnafter the magnitude of ?Vnis limited. Optional smoothing in step 5 helps to

diffuse the motion from high contrast regions into neighborhoods with low contrast regions

(Thirion 1998).

2.6. Implementation

The proposed method is implemented primarily in MATLAB with image processing toolbox.

The motion field inversion procedure is implemented in C/C++. Besides the multiple passes

Page 12

A fast inverse consistent deformable image registration method 6153

(a) (b)(c)

Figure 3. Yosemite images: (a) the Yosemite 0 image, used as moving image, (b) the Yosemite 1

image, used as fixed image, overlaid with ground truth motion field, (c) the difference image.

approach, we also used the multigrid approach to sequentially carry out the registration in

multiple down-sampled image resolution stages. We used five stages for the pig lung image

set, and four stages for Yosemite, liver and kidney and patient lung image sets. The number

of stages is selected to ensure capturing of the largest possible image deformations. We used

eight passes for each stage. For the HS algorithm, we used five iterations for each pass with

α = 0.2. For the demons algorithm, we used σ = 2 pixels and did not use multiple iterations.

Before the two images were registered, their intensities were always normalized to [0, 1]

by dividing by the common maximal intensity value. The Laplacian pyramid down-sampling

filter (Burt and Adelson 1983) was used to half-sample the images in the multigrid approach.

Bilinear (for 2D images) or trilinear (for 3D images) interpolation was used for situations

where interpolation is needed. The differential mask [−1 8 0 −8 1]/12 was used for all

gradient computations.

3. Evaluation

3.1. Image data sets

We used three 2D image data sets and one 3D image data set to test the new method. All three

2D image datasets are accompanied by ground truth motion fields. The 3D image data did

not have ground truth deformation fields, therefore manually selected landmarks were used

for accuracy validation purposes.

3.1.1. Yosemite sequence—2D.

sequence, which was originally generated at SRI (Barron et al 1994). The Yosemite image

sequenceiswidelyusedforthevalidationofdeformableimageregistrationalgorithms. Ground

truth motion field is known. Maximal magnitude of motion is 5.19 pixels. Figure 3 shows

both images and their difference. Evaluation using this image dataset would make it possible

to directly compare our method to other reported deformable registration algorithms.

We used the first two images from the Yosemite 2D image

3.1.2. CT images of pig lung phantom—2D.

phantom (Yang et al 2007b), with pixel size of 0.2441 × 0.2441 mm. The CT slice was

deformed according to a synthesized motion field. The original CT slice is used as the moving

image. The generated one is used as the fixed image. The synthesized motion field is the

ground truth. The maximal motion was 25 pixels. Figure 4 shows the original and the

deformed CT slice, the difference image and the optical flow vector field. We also deformed

We used a cropped CT slice of a pig lung

Page 13

6154D Yang et al

(a)(b)

(c) (d)

Figure 4. Pig lung CT images: (a) the original slice, (b) the generated image, (c) the difference

image, (d) the generated image with ground truth motion field vectors. The difference image is

limited between [−0.5, 0.5] in order to show the difference better.

the CT slice with a similar synthesized motion field with maximal motion magnitude of

5 pixels. The second generated image is used for convergence analysis.

3.1.3. Patient upper abdominal CT scan—2D.

abdominal CT scan, which contains liver and kidney, with pixel size 0.9766 × 0.9766 mm.

This CT scan has much lower image contrast than the pig lung CT scan. It is good to test

for low image contrast situations. We deformed this CT scan with a synthesized motion field.

The maximal motion was 10 pixels. Figure 5 shows the original and the deformed CT slice,

the difference and the synthesized motion field. For convergence analysis, we used another

similar motion field of maximal magnitude of 3 pixels to deform the CT slice.

We used a transverse slice of patient upper

3.1.4. Patient 4D-CT images.

Thisdatasetwasusedinmulti-institutionalevaluationstudyofdeformableregistration(Brock

2007). The first volume represents an exhalation phase and the second one represents an

inhalation phase. Dimensions are 512 × 512 × 152 for both phases, with voxel size of

0.9766 × 0.9766 × 2.5 mm. For this study, we first cropped the 3D-CT volumes to delete

almost everything but lung and then half-sampled every 2D transverse slice. The volume

dimensions became 122 × 150 × 110, and voxel size became 1.9532 × 1.9532 × 2.5 mm.

Figure 6 shows the coronal slices of both volumes, the difference image and the checkerboard

image. In total, 38 corresponding landmarks have been manually selected by physicians on

the original 512 × 512 × 152 volumes. Among the 38 landmarks, 17 were in the right lung,

17 were in the left lung, 2 for heart and 2 for aorta. The landmark selection procedure was

reported in Brock (2007).

We used two 3D-CT volumes from a patient 4D-CT dataset.

3.2. Evaluation procedures

We performed several quantitative comparisons to evaluate the proposed inverse consistency

method against the corresponding asymmetric optical flow algorithms. We always used the

same parameters, multigrid and multiple pass settings in order to make the comparison as fair

as possible.

Page 14

A fast inverse consistent deformable image registration method6155

(a) (b)

(c) (d)

Figure 5. The liver and kidney CT images: (a) the original slice, (b) the generated image, (c) the

difference, (d) the generated image overlaid with motion field vectors. The difference image is

scaled and limited between [−0.2, 0.2] in order to visualize the differences better.

(a)(b)(c)

Figure 6. Patient 4D-CT images. (a) The exhaled phase—the moving image, (b) the inhaled

phase—the fixed image, (c) the difference.

3.2.1. Accuracy validation.

error vector field Verris computed as

After the resulting motion field is computed, the displacement

Verr= V − Vgt,

(33)

Page 15

6156 D Yang et al

where V is the computed field and Vgtis the ground truth field. Mean, standard deviation and

maximal values are computed for |Verr|. For the 2D images, the mean and standard deviation

for absolute angular error Aerrare also computed. Aerris defined as

Aerr= |tan−1(Vy/Vx) − tan−1(Vy,gt/Vx,gt)|,

where tan−1is the inverse tangent function, Vyand Vxare the x and y component of V, and

Vy,gtand Vx,gtare the x and y component of Vgt. The ground truth motion fields are available

for the 2D images; Verrand Aerrare computed for the entire image.

Forthe4D-CTimagedataset,Verrisonlycomputedforthelandmarkpoints. Ifalandmark

point is not located in the center of a voxel, liner interpolation was used to compute V for

the landmark point. We did not compute angular errors for this image dataset. Instead, we

computed the mean, max and standard deviation for Verr,x, Verr,yand Verr,zwhere Verr,x, Verr,y

andVerr,zarethex,yandzdirectionalcomponentsofVerr. Directionsofx,yandzfor3Dimage

volumes are also called as LR (left–right), AP (anterior–posterior) and SI (superior–inferior)

directions.

MSE(meansquareerror)andMI(mutualinformation)betweenthedeformedimageIand

thefixedimageJarewidelyusedinmanypublishedpapersasindirectmeasurementsofimage

registration accuracy. We avoided to use them in this work because (1) they are not accurate

measurements of registration accuracy since better MSE or MI does not directly translates into

higher accuracy, (2) MI is not useful for single modality image registration analysis.

(34)

3.2.2. Inverseconsistencyevaluation.

in both forward and backward directions. For the proposed inverse consistency method, we

only performed registration once in the forward direction and the motion fields for both

registration directions were computed from the results of a single computation step. Inverse

consistency error ICE is computed according to equation (9) using the motion fields of both

directions.

Fortheasymmetricalgorithms,wecarriedregistration

3.2.3. Convergence analysis.

converging to the ground truth and converging to a stable solution. Note that converging

to a stable solution, does not necessarily mean that the stable solution is the known ground

truth as these problems are non-convex. We only used 2D image sets for such convergence

study, because computation cost will be too expensive for the 4D-CT data. Registrations for

convergence analysis were performed without using the multigrid approach so that the native

convergence ability of the methods can be evaluated.

We evaluated the algorithms for two types of convergence,

3.2.4. Analysis of accuracy improvement.

consistency optical flow algorithm is more accurate than the inverse-inconsistency version

of the HS and the demons algorithms (see section 4).

could contribute for this accuracy improvement: (1) using spatial gradients of both images;

(2) the symmetric formation that deforms both images to the middle. To test our conjecture,

we performed experimental simulations as follows. We used the sum of the gradients of

both images to replace the single image gradient in the original optical flow algorithms, and

performed registration in asymmetric way as the original asymmetric optical flow algorithms.

This means using equation (27) to solve V as defined in equation (1). Results of such modified

asymmetric optical flow algorithms were compared to the results by the original algorithms

and the proposed inverse consistency algorithms. The same multigrid, multiple pass and

smoothing parameters were used for all three algorithms to ensure that the comparisons were

fair.

We have observed that the proposed inverse

We conjecture that two things