Simultaneous correspondence and nonrigid 3D reconstruction of the coronary tree from single Xray images.
ABSTRACT We present a novel approach to simultaneously reconstruct the D structure of a nonrigid coronary tree and estimate point correspondences between an input Xray image and a reference 3D shape. At the core of our approach lies an optimization scheme that iteratively fits a generative D model of increasing complexity and guides the matching process. As a result, and in contrast to existing approaches that assume rigidity or quasirigidity of the structure, our method is able to retrieve large nonlinear deformations even when the input data is corrupted by the presence of noise and partial occlusions. We extensively evaluate our approach under synthetic and real data and demonstrate a remarkable improvement compared to stateoftheart.

Article: Accurate coronary centerline extraction, caliber estimation and catheter detection in angiographies.
[Show abstract] [Hide abstract]
ABSTRACT: Segmentation of coronary arteries in XRay angiography is a fundamental tool to evaluate arterial diseases and choose proper coronary treatment. The accurate segmentation of coronary arteries has become an important topic for the registration of different modalities which allows physicians rapid access to different medical imaging information from Computed Tomography (CT) scans or Magnetic Resonance Imaging (MRI). In this paper, we propose an accurate fully automatic algorithm based on Graphcuts for vessel centerline extraction, caliber estimation, and catheter detection. Vesselness, geodesic paths, and a new multiscale edgeness map are combined to customize the Graphcuts approach to the segmentation of tubular structures, by means of a global optimization of the Graphcuts energy function. Moreover, a novel supervised learning methodology that integrates local and contextual information is proposed for automatic catheter detection. We evaluate the method performance on three datasets coming from different imaging systems. The method performs as good as the expert observer w.r.t. centerline detection and caliber estimation. Moreover, the method discriminates between arteries and catheter with an accuracy of 96.5%, sensitivity of 72%, and precision of 97.4%.IEEE transactions on information technology in biomedicine: a publication of the IEEE Engineering in Medicine and Biology Society 09/2012; · 1.69 Impact Factor  SourceAvailable from: Coert T Metz[Show abstract] [Hide abstract]
ABSTRACT: A method for registering preoperative 3D+t coronary CTA with intraoperative monoplane 2D+t Xray angiography images is proposed to improve image guidance during minimally invasive coronary interventions. The method uses a patientspecific dynamic coronary model, which is derived from the CTA scan by centerline extraction and motion estimation. The dynamic coronary model is registered with the 2D+t Xray sequence, considering multiple Xray time points concurrently, while taking breathing induced motion into account. Evaluation was performed on 26 datasets of 17 patients by comparing projected model centerlines with manually annotated centerlines in the Xray images. The proposed 3D+t/2D+t registration method performed better than a 3D/2D registration method with respect to the accuracy and especially the robustness of the registration. Registration with a median error of 1.47 mm was achieved.IEEE transactions on medical imaging. 02/2013;
Page 1
Simultaneous Correspondence and NonRigid 3D Reconstruction of the
Coronary Tree from Single Xray Images
Eduard Serradell1
Adriana Romero2,4
Rub´ en Leta3
Carlo Gatta2,4
Francesc MorenoNoguer1
1Institut de Rob` otica i Inform` atica Industrial, CSICUPC, Barcelona, Spain
2Centre de Visi´ o per Computador, UAB, Barcelona, Spain
3Hospital de la Santa Creu i Sant Pau, Barcelona, Spain
4Departament de Matem` atica Aplicada i An` alisi, UB, Barcelona, Spain
{eserradell,fmoreno}@iri.upc.edu, aromero@cvc.uab.es, rleta@hsp.santpau.es, carlo.gatta@ub.edu
Abstract
We present a novel approach to simultaneously recon
struct the 3D structure of a nonrigid coronary tree and es
timate point correspondences between an input Xray im
age and a reference 3D shape. At the core of our approach
lies an optimization scheme that iteratively fits a generative
3D model of increasing complexity and guides the matching
process. As a result, and in contrast to existing approaches
that assume rigidity or quasirigidity of the structure, our
method is able to retrieve large nonlinear deformations
even when the input data is corrupted by the presence of
noise and partial occlusions. We extensively evaluate our
approach under synthetic and real data and demonstrate a
remarkable improvement compared to stateoftheart.
1. Introduction
Recovering the 3D structure of a nonrigid coronary tree
from single Xray images is a highly ambiguous problem
since many different 3D configurations can virtually have
the same projection. As shown in Fig. 1 the problem be
comes even more challenging because Xray images are of
ten affected by a series of artifacts such as noise, blurring,
partial occlusions and vessel discontinuity. Thus, solving
this problem requires from prior knowledge about the type
of deformations the structure can undergo.
Standard approaches within medical imaging assume a
reference 3D scan of the tree is known and that defor
mations in the input image are negligible. This reduces
the shape recovering task to a rigid 3Dto2D registra
tion [9, 11, 16]. There exist a recent attempt of addressing
the nonrigidity nature of the problem, although it has only
been shown effective for relatively small deformations [8].
We may find other related areas in computer vision that
essentially solve the same problem but in a different con
text, for instance, the techniques for 3D nonrigid surface
Figure 1. Recovering the structure of a nonrigid coronary tree.
Top: Given an input Xray image (left), and a reference struc
ture (shown in red at the topright image) we are able to retrieve
the 3D configuration of the coronary tree in the input image (yel
low). Bottom: Results on synthetic data, for which we know the
ground truth and allows us to evaluate the method under noise, oc
clusions (blue dots), and different levels of deformation. The left
figure depicts the 2D results and the figure on the right represents
the ground truth (black), the prior (red) and our solution (yellow).
Note that even when the prior significantly differs from the ground
truth, our approach yields very accurate results.
reconstruction [5, 14, 17] and articulated human pose esti
mation from monocular images [1, 18, 21, 27]. In these ap
proaches, though, it is often easy to obtain large amounts of
training data and build detailed parametric models for spe
cific deformations or 2Dto3D mappings that directly link
2D observations with 3D configurations.
producing these detailed models and mappings is beyond
our possibilities, because Xray images are harmful for the
patient and, besides one single reference 3D scan of the
coronary tree, no further prior knowledge can be used.
We therefore propose a novel approach that, given solely
Unfortunately,
Page 2
one single Xray image and a reference 3D configuration,
simultaneously recovers the 3D structure of the coronary
tree in the input image and establishes matches with the ref
erence shape. As shown in Fig. 1, our method can recover
the 3D structure in the input image even when it highly dif
fers from that of the reference configuration. In addition we
can handle large amounts of noise and occlusions.
The key contributions that make this possible are
twofold. First we use a generative model that progressively
increases its complexity and allows a coarse to fine fitting
while 3Dto2D matches are estimated. Second, we take
advantage of a recursive parameterization of the coronary
tree that introduces dependencies between all the nodes of
the tree, and diffuses the local constraints to the whole
structure.Both the parameterization and the generative
model are then integrated within a Kalmanbased optimiza
tion framework. In the results section we will show that the
overall methodology has significant advantages when com
pared to stateofthe art approaches.
2. Related Work
Recovering the 3D structure of the coronary tree from
single vascular images involves dealing with many differ
ent issues. Besides the inherent ambiguity of the monocular
nonrigid reconstruction, the problem is further accentuated
due to the presence of noise in the images and partial oc
clusions between different branches of the tree. This com
plexity has been traditionally alleviated by considering the
vascular system as a rigid structure [9, 11, 16] and using
multiple views [26, 29]. To the best of our knowledge, [8]
is the only approach in the medical imaging literature that
considers the nonrigid nature of the problem. They intro
duce 3D priors and inextensibility constraints into a steepest
descent scheme to solve for the shape. Yet, their optimiza
tion procedure is only effective under relatively simple de
formations as those occurring in the liver artery.
On the other hand, our approach has similarities with
the techniques to reconstruct nonrigid 3D surfaces and es
timate articulated pose from monocular images. Among the
former, it has been shown that 3D shape can be retrieved by
imposing local inextensibility and constraints introduced by
a set of 3Dto2D correspondences between the input im
age and a reference shape [5, 14, 17, 19]. In essence we
will also use the same kind of assumptions, although we
will need from additional constraints since in our context
the 3Dto2D correspondences are unknown and have to be
resolved simultaneously with the shape. Moreover, local
distance constraints are much less restrictive when dealing
with points linked through a treelike structure than when
dealing with neighboring points on a surface. In addition,
many of these approaches impose strong shape priors based
on previously acquired training data [17, 19] while in our
approach accurate training data is hard to obtain and we
have to rely on very weak shape priors.
Since the coronary tree may be regarded as an articu
lated structure, one might think in applying the techniques
of articulated pose estimation to our problem [1, 21, 27].
These approaches rely on large amounts of data for learn
ingamappingfrom2Dimageobservationsto3Dposes, and
have the advantage of not requiring to solve the 2Dto3D
correspondence problem. Yet, as said above, while obtain
ing sufficient training data is feasible for applications such
as human pose estimation [2, 23], it becomes prohibitive in
our framework, as the number of Xray images that may be
captured for each patient is limited.
Recent works suggest introducing similar constraints as
those used for nonrigid shape recovery into the formula
tion of articulated pose estimation problems [18, 22, 24].
This allows fitting more detailed parametric 3D models [22]
and reducing the dependency of articulated pose estimation
techniques on the training data [18]. However, reducing the
dependency on training data has the drawback of increasing
the sensitivity to artifacts into the input data.
Drawing particular inspiration on these approaches, our
method also combines tools from the techniques for articu
latedpose estimationand shape recovery. However, inorder
totackleproblemswithmuchlargeramountsofimagenoise
and occlusions, we propose using a generative 3D model
that progressively increases its complexity and adaptability
while establishing correspondences and detecting and re
jecting outlier points. In addition, we represent the articu
lated structure using a recursive parametrization that, as we
will show in the results section, yields remarkable improved
results when compared against [18].
3. Algorithm Overview
The focus of this paper is on retrieving the 3D structure
of the coronary tree while establishing 3Dto2D point cor
respondences between an input Xray image and a reference
3D scan. However, the overall algorithm requires additional
tasks as we next detail:
1. Feature extraction: Our input data is an Xray im
age and a volumetric 3D Computed Tomography (CT)
scan. In a preprocessing step we segment the vessel re
gions on both sets of data and extract points of interest.
2. Generative model for the coronary tree: We repre
sent the 3D feature points as a tree, parameterized
by the joint angles and distances between consecutive
points. Since we cannot explicitly compute deforma
tion modes from training data, we estimate them by
performing a Probabilistic Principal Component Anal
ysis (PPCA) over a set of synthetic samples for which
we randomized the values of the joint angles. The flex
ibility of the model will be controlled by the number of
components of the PPCA and by the magnitude of the
noise used to generate the deformed samples.
Page 3
(a)(b)(c)(d)
(e) (f)(g) (h)
Figure 2. Feature extraction. Top: Extraction of the 3D skeleton from the 3D Computed Tomography. (a) Two levels of thresholding.
(b) Vessel segmentation. (c) 3D skeletonization. (d) Feature point orientations. Bottom: 2D Feature extraction from the Xray images. (e)
Original image. (f) Vesselness segmentation. (g) Extracted feature points (h) Estimated feature orientations.
3. Nonrigid reconstruction and matching:
Kalmanbased approach, we iteratively solve for the
3Dto2D correspondences and progressively fit the
deformation modes onto the 3D points.
Using a
In the following sections we discuss each one of these
constituent pieces.
4. Feature Extraction
We next briefly describe the steps we perform to seg
ment the vessel regions and extract the 3D and 2D features
from the raw 3D scans and from the Xray images respec
tively. Note that this task is by itself an active area of re
search within medical imaging [4, 7, 20]. Yet, we design
relatively simple solutions, partially based on the popular
vessel enhancement filter proposed in [6], which although
not being free of error, demonstrate the robustness of our
algorithm for 3D nonrigid reconstruction.
4.1. 3D Features
The raw data of the volumetric 3D CT scanner is com
posed of a stack of grayscale image slices. The CT volume
is initially segmented using the fuzzy connectedness tree al
gorithm [28], and as shown in Fig. 2a, a number of regions
corresponding to different physiological structures are gen
erated. Fig. 2b shows the vessellike formations. The ves
sel centerlines are then accurately detected based on a local
steepest gradient ascent of the vesselness measure [6]. Fi
nally, we homogeneously distribute an arbitrary number of
nodes along the skeleton, and enforce connection using a
minimum spanning tree approach [15](Fig. 2c).
In order to perform the 3Dto2D matching in a subse
quent step, besides using the 3D position, we describe each
node based on a unitary vector that indicates the local artery
orientation, which is computed from the eigenvectors of the
Hessian matrix at the appropriate scale. Although we could
have used more elaborated descriptors, we found the po
sition and orientation to be discriminative enough in our
context of 3D heart scans, where despite nonrigid defor
mations, the global orientation of the heart remains quite
stable throughout the cardiac cycle. In Fig. 2d we depict the
orientation assigned to each node of the tree.
4.2. 2D Features
A reliable and robust detection of the vessel centerlines
in the 2D Xray images is more challenging than in the 3D
case. Occlusions produced by vessel crossings, reduced lo
cal contrast in the diaphragm area, discontinuities due to
lack of the contrastagent (or stenosis), are some of the ar
tifacts we may find in these images. Even though a Digi
tal Subtraction Angiography technique can help to reduce
some of these problems, its practical use is limited to se
quences in which the relative pose of the Xray equipment
does not change with respect to the patient, which is not our
case. In addition, while in practice the 3D feature extrac
tion can be performed offline in a preoperative stage, 2D
features must be computed in realtime and without manual
intervention. As a consequence, we will only extract points
of interest and will not attempt to solve the connectivity tree
of the vascular structure.
The extraction of points of interest is performed by ap
plying the multiscale vesselness filtering proposed in [6].
We then perform a nonmaxima suppression on the vessel
ness map, followed by a thresholding. For each detected
location, a ridge traversal algorithm, similar to the one pro
posed in [3] is applied, providing the centerlines of arteries.
In order to select a specific number of keypoints and ensure
they are homogeneously distributed, we further apply a k
means clustering algorithm, were we set k to the number of
desired keypoints. Finally, each of the keypoints is again
described by the vessel orientation at the point, computed
from the eigenvectors of the Hessian matrix. Fig. 2(g,h)
Page 4
show the final keypoints and their respective orientations.
5. Nonrigid Reconstruction Algorithm
In this section we discuss the core elements of our ap
proach. After formalizing the problem, we introduce the re
cursive model we use to represent the deformable coronary
tree, as its particular representation will play a decisive role
for being robust to high levels of occlusions and allowing
to retrieve large deformations. We then describe the weak
priors we use to build a probabilistic generative model of
the tree, and finally we describe the iterative algorithm we
propose that combines all the previous ingredients to solve
for the shape and assign correspondences.
5.1. Problem formulation
Let Mref= {xref
we have extracted from the 3D reference scan and U =
{u1,...,unf} the nffeature points extracted from the 2D
Xray image, corresponding to a projection of M, a non
rigidly deformed version of Mref. Our goal is to retrieve
both as many 3Dto2D correspondences {xref,u} as pos
sible, and the 3D configuration of the deformed structure
M = {x1,...,xnm}. Note that since in practice the key
points are obtained from uniformly sampling the segmented
3D scans and Xray images, it may not exist a perfect one
toone match between the sets Mrefand U. However, in
the synthetic results section we will show that our algorithm
tolerateslargeamounts ofnoiseandocclusions, whichcom
pensates for all these inaccuracies in the matches.
1
,...,xref
nm} be the nmmodel points
5.2. Recursive 3D Model Parameterization
Since we know the links between the 3D points of M,
we can represent the structure of the deformable model by
the vector:
m = [x⊤
1,ρ⊤
2,...,ρ⊤
nm]⊤
(1)
where x1are the 3D coordinates of the root node, and ρj=
[rij,θij,φij]⊤are the spherical coordinates of the vector
joining the ith and jth nodes. Thus, the 3D position of a
node xkmay be recursively written as:
xk= x1+
?
i,j∈Ak
rijcosθijsinφij
rijsinθijsinφij
rijcosφij
where Akcontains all the ancestors of the kth node. Ob
serve that using this formulation, when the 3D position of
a node is updated, the location of all its ancestors is also
updated. This is a remarkable novel contribution of our for
mulation, as it naturally introduces constraints that go be
yond local neighborhoods. In addition, the propagation of
the error using this recursive parameterization is very well
suitedtodealwithstructureslikethecoronarytree, inwhich
the root node remains almost at constant position while the
terminal nodes are usually highly deformed.
Figure 3. Building weak deformation priors. Since we do not
explicitly use training data, we build a weak prior of the tree struc
ture by assigning gaussian noise to its nodes (left). Note that since
our parameterization is recursive, we assign larger noise values to
the terminal nodes of the tree. Using this prior, we then produce
multiple random shape samples (right) and fit a PPCA model.
5.3. Introducing Synthetic Deformation Priors
Although we do not explicitly use training data besides
the reference 3D scan, we synthetically define weak priors
on the feasible deformations, which will be used within the
optimization scheme to progressively fit the 3D model to
the input data.
Given the 3D parameterization mrefof the reference
shape, we compute the weak priors by first generat
ing multiple samples {si} from the normal distribution
N(mref,Σm), where Σmis a 3nm× 3nmdiagonal co
variance matrix generated by concatenating the covariances
of x1and the vectors {ρj}j=2,...,nm. As shown in Fig. 3
these covariances are set to relatively large values in order
to deal with different types of deformations. Yet, in order to
avoid completely random shapes with no physical meaning
we slightly smooth the resulting sampled shapes. In addi
tion, the variance associated to the internode lengths rij
is set to a very small value, as we assume the vessels do
not stretch. Note that although real vessels may stretch, we
represent the tree by a sufficiently dense set of points that
makes the inextensibility assumption locally correct.
We then learn a lowdimensional deformation model of
the coronary tree, by applying Probabilistic Principal Com
ponent Analysis [25] over the set of deformed samples {si}.
By doing this, we can approximate the pose parameters of
Eq. 1 as a weighted sum of a mean structure m0and nq
deformation modes Q = [q1,...qnq] :
m = m0+
nq
?
i=1
αiqi= m0+ Qα
(2)
where α = [α1,...,αnq]⊤are unknown modal weights
that define the current structure, and whose covariance is
defined by a nq× nqmatrix Σα.
5.4. Iterative Fitting and 3Dto2D Matching
Representing the coronary tree by means of the modal
weights of Eq. 2 allows rewriting the problem as that of
estimating the parameters α∗such that the following repro
jection error is minimized
Page 5
InitializationIteration #1Iteration #2 Iteration #3Rigid Registration
Figure 4. Iterative Fitting and 3Dto2D Matching. We plot the 3D results (top) and 2D results (bottom) of the fitting process at different
iterations and the matches we establish. The leftmost figures show the initial prior (red) and the ground truth structure. The next 3 images
show the fitting process of our approach (yellow) for different iterations. The figures on the right show what might be obtained using just
a rigid registration, which is a standard solution to this kind of problems. Obviously the rigid approach yields large errors, specially on the
terminal nodes of the coronary structure.
α∗= argmin
α
nm
?
i=1
?Proj(xi;α) − Match(xi,U)?
(3)
where Match(xi,U) returns the match uj ∈ U for a 3D
point xi, and Proj(xi;α) is the perspective projection of
the 3D point xigiven the modal weights α.
Note that we assume we know the projection matrix of
the imaging system, including both the pose of the patient
with respect to the CT and Xray equipment, and the intrin
sic parameters of the detector. In practice, though, there is
always some translation in the position of the patient with
respect to the global reference system. In addition, the heart
deformation during the cardiac cycle and the global heart
shift produced by the breathing introduce additional trans
lation and small rotation effects. In order to address these
issues, we have considered the translations and small rota
tions as part of the nonrigid model.
We next turn to the algorithmic steps to minimize Eq. 3.
We achieve this by alternatively solving correspondences
and fitting the 3D model, which we have initialized with the
reference 3D model provided by the 3D scanner.
Establishing 3Dto2D Correspondences. Let us assume
that as input of an iteration we have α and Σα, the modal
weights and their covariance matrix estimated at the pre
vious iteration, the set M of 3D model points deformed
according the weights α, and the set U of 2D features ex
tracted from the Xray image. We then compute 3Dto2D
matches with the following steps:
1. We project the 3D model points onto the image, con
sidering the current configuration of modal weights.
We denote these projections by V = {v1,...,vnm}
where vi= Proj(xi;α).
2. For each vi we establish an uncertainty region sur
rounding it, by propagating the covariance Σαof the
modal weights to the image plane. This region will be
a Gaussian centered at each viand with covariance:
Σi
v= J(xi)ΣαJ(xi)⊤
where J(xi) is the 2 × nqJacobian of the projection
equationProj(xi;α)withrespecttothemodalweights
α, evaluated at the 3D point xi.
3. Given the set {vi,Σi
points uj∈ U as a standard Optimal Assignment Prob
lem using the Hungarian algorithm [12]. For this pur
pose, for each potential match {vi,uj} we introduce
a cost defined as a linear combination of the Maha
lanobis distance between the two points and their sim
ilarity in the orientation computed in Section 4:
v} we solve the matching with the
Cij= λ1Mah(vi,uj) + λ2Angle(vi,uj)
where Mah(vi,uj) = (vi− uj)(Σi
and Angle(vi,uj) is the difference in the orientations
between the points of ujand xi, the 3D model point
projected on vi. Note that since the orientation vector
of xiis originally computed in the 3D space, we need
to project it on the image plane to compare it with the
orientation of vi. The terms λ1and λ2are constant
scale factors used to give similar orders of magnitude
toeach ofthecomponents ofthecostfunction. Inprac
tice, although the Hungarian algorithm is already an
efficient technique for determining the optimal match
ing, we further reduce its complexity by only consid
ering those costs Cijwhich are below a certain thresh
old. This also prevents from fitting the shape to outlier
correspondences or mismatches.
v)−1(vi− uj)⊤,
Page 6
Deformation Level [σθ,σφ]Image Noise [σn] Occlusions (%)
RMS Error 3D [voxels]
RMS Error 2D [pix]
Deformation Samples
σθ = σφ= 0.1
Error 3D: 5.8 [vox]
σθ = σφ= 0.3
Error 3D: 14.7 [vox]
σθ = σφ= 0.5
Error 3D: 28.8 [vox]
Figure 5. Synthetic experiments. RMS error distribution for 3D reconstruction (top) and 2D reprojection (bottom) for each experiment.
The whiskers denote min. and max. errors, the box spans from first to third quartile and the interbox lines show the mean RMS error.
Updating Modal Weights. Given the set {xi,ui}i=1,...,nc
of estimated correspondences we then use Kalman filter
equations to update the modal weights α and its covari
ance matrix Σα. Since we simultaneously use all of the
estimated matches, we define an extended 3ncvector of 3D
points ˆ x = [x⊤
locations ˆ u = [u⊤
Jacobian matrixˆJ = [J(x1)⊤,...,J(xnc)⊤]⊤. We then
update α and Σαas
α+
=
α + K(ˆ u − Proj(ˆ x;α))
(I − KˆJ)Σα
1,...,x⊤
1,...,u⊤
nc]⊤, an extended 2ncvector of 2D
nc]⊤, and an extended 2nc× nq
Σα+
=
where K is the Kalman gain and I the identity matrix.
Iterating and Increasing Flexibility. The matching and
modal weight updating processes are iteratively repeated
until the convergence of Eq. 3.
In order to adapt the reference model to highly deformed
structures, such as those shown in Fig. 1, the number of
modes we use is increased at each iteration. This allows to
progressively fit the structure, starting from the most rigid
parts up to the more deformed ones. In addition, using
more rigid structures at early stages yields robustness to
mismatches, preventing to adapt the model towards outlier
2D features. In practice, for a coronary tree with nmpoints,
and thus with a maximum number of 3nmmodes, at itera
tion number niterwe used nq= 3nm/(10 − niter) modes.
We found the number of iterations to converge to be always
≤ 5. For instance, Fig. 4 shows an example in which con
vergence was achieved after 3 iterations.
6. Results
We now present the results on both synthetic and real
data. In the synthetic results we compare our approach (de
noted ArtDeform) to [18]1, which is a representative exam
ple of the stateoftheart in articulated pose recovery that
has been shown successful in recovering human and hand
pose. We also compare it to the solution obtained from
a 3Dto3D rigid transformation between the reference 3D
shape and the ground truth shape associated to the input im
age, which would represent the best solution that a rigid
2Dto3D registration algorithm could obtain. For this pur
pose, we use a standard technique for absolute orientation
estimation [10]. Note that both in [18] and [10], the corre
spondencesareassumedtobeknown, whileinourapproach
we simultaneously estimate them with the shape.
6.1. Synthetic Experiments
In this section we extensively evaluate the performance
of each algorithm against noise in the correspondences, dif
ferent levels of deformation and partial occlusions.
generated random shapes of 50 nodes within a volume of
300×300×300 voxels, such as the treelike structure shown
in Fig. 1, and simulated the deformations undergone in
the coronary tree by applying increasing levels of noise
{σθ,σφ} to the joint angles. We then projected each 3D
shape on a 512×512 image and added gaussian noise of
standard deviation σnto the 2D correspondences. In addi
tion, a percentage poof the projected points was randomly
removed in order to simulate partial occlusions. Given the
original reference shape and the set of projected points of
the deformed shape, we then performed the reconstruction
with each of the algorithms.
Three different types of experiments were performed.
We initially evaluated the amount of deformation each al
gorithm was able to recover by sweeping the variances of
We
1We thank Dr. Mathieu Salzmann for kindly testing the data of our
experiments on the algorithm proposed in [18].
Page 7
Left Coronary ArteryRight Coronary Artery
2D Reprojection Error
Rigid Reg.ArtDeform
5
10
15
20
25
30
Reprojection Error [pix]
Figure 6. Reconstruction results on real vessel structures. Left: Several samples of the LCA and RCA reconstructions. Upper row:
initial model (red) and the final nonrigid tree (yellow) overlaid on the input Xray images. We also plot the results of the 2D segmentation
(blue), which, as can be seen, contains certain amounts of false positives. Bottom row: Initial and retrieved 3D shapes. Note that despite
the segmentation suffers from occlusions and false positive regions our approach yields an accurate solution. Right: 2D reprojection error,
which is used to quantitatively assess the quality of the real results, because the 3D ground truth is not available.
the joint angles within the range [σθ,σφ] ∈ [0 − 0.5] rad,
and randomly setting σn ∈ [1 − 3] and po ∈ [5 − 20]%.
To give significance to the levels of deformation and recon
struction errors, Fig. 5right depicts different deformations
of the model corresponding to specific values of {σθ,σφ}.
In a second experiment we analyzed the robustness to im
age noise by synthetically introducing random noise of
σn∈ [0−10], and setting {σθ,σφ} ∈ [0.15−0.3] and po∈
[5 − 20]%. Finally, we evaluated the effect of occlusions in
an interval po∈ [0 − 50]%, with {σθ,σφ} ∈ [0.15 − 0.3]
and σn∈ [1 − 3].
For each set of parameters we performed 50 trials. The
graphs on Fig. 5 depict the mean 2D reprojection error, ex
pressed in pixels, and the mean 3D reconstruction error, ex
pressed in voxels. Observe that our algorithm consistently
outperforms [18] in all experiments. This difference is spe
cially remarkable when dealing with occlusions, for which
we obtain reconstruction errors below 5 voxels even when
a 50% of the model is occluded. Observe in the top right
graph of Fig. 5 that these amounts of error correspond to
very good approximations. It is fair to mention, though,
that [18] is a general algorithm easily adaptable to different
domains, fromarticulatedstructurestodeformablesurfaces,
while our algorithm is specifically designed to handle tree
like and articulated structures. In Fig. 5 we also plot the
results that would be recovered using a rigid registration
and, as expected, the errors are significantly larger. Note
that the error values for this case are scaled by a factor 1/2
for displaying purposes.
6.2. Real Data
We also evaluated our approach on real CT data and X
ray data collected during ordinary preoperative diagnosis
and percutaneous intervention. We collected CT data of 7
patients, using a Philips Brilliance iCT, at the 75% of the
heart cycle, with slice thickness 0.67 or 0.8mm, and pixel
resolution between 0.38×0.38 and 0.45×0.45 mm. We col
lected a total of 17 Xray sequences, 10 of Left Coronary
Artery (LCA) and 7 of Right Coronary Artery (RCA), us
ing a single plane Philips INTEGRIS Allura Flat Detec
tor. Image and camera calibration was performed using the
catheter width and the geometrical information on the C
Arm position. For each sequence, one image in which the
contrast liquid was sufficiently visible was selected.
For each of pair CT scan/Xray image, we then extracted
3D and 2D features as described in Sect. 4. In all experi
ments, we represented the segmented CT volume as a tree
with 75nodes, and extracted 500 feature points from each
Xray image. Starting with the initial tree of the CT scan,
we then iteratively fit the model and established 3Dto2D
matches. In all experiments we achieved convergence in
less than 5 iterations, taking about 8 seconds per iteration.
Note that this represents in fact a significantly faster algo
rithm compared to competing methods. For instance, [8]
reports computation times of about 7 minutes per image.
The 2D registration and 3D reconstruction results of a
few sample experiments are depicted in Fig. 6. Observe that
even when the segmentation contains false positive regions,
or does not detect some branches of the coronary tree our
approach is able to provide an accurate solution.
Sincethe3Dgroundtruthdoesnotexistforthedeformed
artery tree, we quantitatively evaluated the performance of
our algorithm based on the 2D reprojection error with re
spect to ground truth centerlines manually annotated by an
expert physician. This error is shown in the bar plot of
Fig. 6right, which summarizes the results for all the 17
experiments. Observe that our nonrigid approach clearly
outperforms a method that rigidly registers the original CT
Page 8
scan. In fact, considering an average calibrated pixel res
olution of 0.22 mm, the median error of our method is of
about 1.9 mm. This compares very well with the 1 mm er
ror reported in [11], especially considering that they restrict
their evaluation to Xray images acquired at systole and di
astole time instants, where the coronary tree deformation is
minimal compared to other cardiac cycle instants we con
template.
7. Conclusion
We have presented a novel approach to estimate the 3D
structure of the coronary tree from single Xray images. In
order to handle the large amounts of deformation, noise
and occlusions present in this kind of images we have in
troduced a generative model based on a recursive parame
terization that progressively increases its complexity. We
have integrated this model within a Kalman filter frame
work which, making use of very weak priors on the struc
ture, iteratively guides the matching process while recover
ing coarsetofine levels of deformation.
The formulation we propose is fairly general, and al
lows integrating additional features. As part of future work,
we consider exploiting motion coherence for tracking heart
beat sequences in real time. Moreover, we believe that the
inextensibility constraints between neighboring nodes may
be relaxed, thus allowing to handle stretchable structures.
Acknowledgments
This work has been partially funded by the project La
Marat´ o de TV3 082131; by the Spanish Ministry of Science
and Innovation under projects DPI200806022, DPI2010
17112, Consolider Ingenio 2010 CSD200700018; and by
the EU project GARNICS FP7247947. The work of C.
Gatta is supported by a Beatriu de Pinos Fellowship.
References
[1] A. Agarwal and B. Triggs. 3d human pose from silhouettes
by relevance vector regression. In CVPR, 2004.
[2] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers,
and J. Davis. Scape: shape completion and animation of
people. ACM SIGGRAPH, 24:408–416, 2005.
[3] C.Bauer, H.Bischof. Edge Based Tube Detection for Coro
naryArteryCenterlineExtraction. MICCAIWorkshop, 2008.
[4] F. Benmansour, L. Cohen, M. Law, and A. Chung. Tubular
anisotropy for 2d vessels segmentation. In CVPR, 2009.
[5] A. Ecker, A. D. Jepson, and K. N. Kutulakos. Semidefinite
programming heuristics for surface reconstruction ambigui
ties. In ECCV, 2008.
[6] A. Frangi, W. Niessen, K. Vincken, and M. Viergever. Mul
tiscale vessel enhancement filtering. In MICCAI, 1998.
[7] G.Gonzalez, E.Turetken, F.Fleuret, P.Fua. Delineating trees
in noisy 2d images and 3d imagestacks. In CVPR, 2010.
[8] M. Groher, D. Zikic, and N. Navab. Deformable 2d3d reg
istration of vascular structures in a one view scenario. IEEE
Trans. on Medical Imaging, 2009.
[9] J. Hipwell, G. Penney, R. McLaughlin, K. Rhode, P. Sum
mers, T. Cox, J. Byrne, J. Noble, and D. Hawkes. Inten
sity based 2d3d registration of cerebral angiograms. IEEE
Trans. on Medical Imaging, 2003.
[10] B. Horn, H. Hilden, and S. Negahdaripour. Closedform
solution of absolute orientation using orthonormal matrices.
Journal Optical Society of America, 1988.
[11] C. Metz et al. Patient specific 4d coronary models from ecg
gated cta data for intraoperative dynamic alignment of cta
with xray images. In MICCAI, 2009.
[12] J. Munkres. Algorithms for the assignment and transporta
tion problems. J. Soc. Ind. and Applied Mathematics, 1957.
[13] A. Myronenko and X. Song. Point set registration: Coherent
point drift. IEEE Trans. PAMI, 2010.
[14] M. Perriollat, R. Hartley, and A. Bartoli.
templatebased reconstruction of inextensible surfaces. In
BMVC, 2008.
[15] R. Prim. Shortest connection networks and some generaliza
tions. Bell System Technical Journal, 1957.
[16] D. Ruijters, B. M. ter Haar Romeny, and P. Suetens.
Vesselnessbased 2d3d registration of the coronary arteries.
Int. J. Comp. Ass. Radiology and Surgery, 2009.
[17] M. Salzmann, F. MorenoNoguer, V. Lepetit, and P. Fua.
Closedform solution to nonrigid 3d surface registration. In
ECCV, 2008.
[18] M. Salzmann and R. Urtasun. Combining discriminative and
generativemethodsfor3ddeformablesurfaceandarticulated
pose reconstruction. In CVPR, 2010.
[19] J. Sanchez, J. Ostlund, P. Fua, and F. MorenoNoguer. Si
multaneous pose, correspondence and nonrigid shape. In
CVPR, 2010.
[20] M. Schaap et al. Standardized evaluation methodology and
reference database for evaluating coronary artery centerline
extraction algorithms. Medical Image Analysis, 2009.
[21] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estima
tion with parametersensitive hashing. In ICCV, 2003.
[22] L. Sigal, A. O. Balan, and M. J. Black. Combined discrim
inative and generative articulated pose and nonrigid shape
estimation. In NIPS, 2007.
[23] L. Sigal and M. J. Black. Humaneva: Synchronized video
and motion capture dataset for evaluation of articulated hu
man motion. In Technical Report, Brown University, 2006.
[24] C. Sminchisescu, A. Kanaujia, and D. Metaxas. Learning
joint topdown and bottomup processes for 3d visual infer
ence. In CVPR, 2006.
[25] M. Tipping and C. Bishop. Probabilistic principal compo
nent analysis. Journal of the Royal Statistical Society, 1999.
[26] G.A. Turgeon et al. 2d3d registration of coronary an
giogramsforcardiacprocedureplanningandguidance. Med
ical Physics, 2005.
[27] R. Urtasun and T. Darrell. Local probabilistic regression for
activityindependent human pose inference. In CVPR, 2008.
[28] C. Wang and O. Smedby
andskeletonizationbasedoncompetingfuzzyconnectedness
tree. In MICCAI, 2007.
[29] J. Yao and R. Taylor.Assessing accuracy factors in de
formable 2d/3d medical image registration using a statistical
pelvis model. In ICCV, 2003.
Monocular
Coronary artery segmentation
View other sources
Hide other sources
 Available from Carlo Gatta · Jun 5, 2014
 Available from upc.edu