Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images.

Full text

Page 1

Simultaneous Correspondence and Non-Rigid 3D Reconstruction of the
Coronary Tree from Single X-ray Images
Eduard Serradell1
Adriana Romero2,4
Rub´ en Leta3
Carlo Gatta2,4
Francesc Moreno-Noguer1
1Institut de Rob` otica i Inform` atica Industrial, CSIC-UPC, Barcelona, Spain
2Centre de Visi´ o per Computador, UAB, Barcelona, Spain
3Hospital de la Santa Creu i Sant Pau, Barcelona, Spain
4Departament de Matem` atica Aplicada i An` alisi, UB, Barcelona, Spain
{eserradell,fmoreno}@iri.upc.edu, aromero@cvc.uab.es, rleta@hsp.santpau.es, carlo.gatta@ub.edu
Abstract
We present a novel approach to simultaneously recon-
struct the 3D structure of a non-rigid coronary tree and es-
timate point correspondences between an input X-ray im-
age and a reference 3D shape. At the core of our approach
lies an optimization scheme that iteratively fits a generative
3D model of increasing complexity and guides the matching
process. As a result, and in contrast to existing approaches
that assume rigidity or quasi-rigidity of the structure, our
method is able to retrieve large non-linear deformations
even when the input data is corrupted by the presence of
noise and partial occlusions. We extensively evaluate our
approach under synthetic and real data and demonstrate a
remarkable improvement compared to state-of-the-art.
1. Introduction
Recovering the 3D structure of a non-rigid coronary tree
from single X-ray images is a highly ambiguous problem
since many different 3D configurations can virtually have
the same projection. As shown in Fig. 1 the problem be-
comes even more challenging because X-ray images are of-
ten affected by a series of artifacts such as noise, blurring,
partial occlusions and vessel discontinuity. Thus, solving
this problem requires from prior knowledge about the type
of deformations the structure can undergo.
Standard approaches within medical imaging assume a
reference 3D scan of the tree is known and that defor-
mations in the input image are negligible. This reduces
the shape recovering task to a rigid 3D-to-2D registra-
tion [9, 11, 16]. There exist a recent attempt of addressing
the non-rigidity nature of the problem, although it has only
been shown effective for relatively small deformations [8].
We may find other related areas in computer vision that
essentially solve the same problem but in a different con-
text, for instance, the techniques for 3D non-rigid surface
Figure 1. Recovering the structure of a non-rigid coronary tree.
Top: Given an input X-ray image (left), and a reference struc-
ture (shown in red at the top-right image) we are able to retrieve
the 3D configuration of the coronary tree in the input image (yel-
low). Bottom: Results on synthetic data, for which we know the
ground truth and allows us to evaluate the method under noise, oc-
clusions (blue dots), and different levels of deformation. The left
figure depicts the 2D results and the figure on the right represents
the ground truth (black), the prior (red) and our solution (yellow).
Note that even when the prior significantly differs from the ground
truth, our approach yields very accurate results.
reconstruction [5, 14, 17] and articulated human pose esti-
mation from monocular images [1, 18, 21, 27]. In these ap-
proaches, though, it is often easy to obtain large amounts of
training data and build detailed parametric models for spe-
cific deformations or 2D-to-3D mappings that directly link
2D observations with 3D configurations.
producing these detailed models and mappings is beyond
our possibilities, because X-ray images are harmful for the
patient and, besides one single reference 3D scan of the
coronary tree, no further prior knowledge can be used.
We therefore propose a novel approach that, given solely
Unfortunately,

Page 2

one single X-ray image and a reference 3D configuration,
simultaneously recovers the 3D structure of the coronary
tree in the input image and establishes matches with the ref-
erence shape. As shown in Fig. 1, our method can recover
the 3D structure in the input image even when it highly dif-
fers from that of the reference configuration. In addition we
can handle large amounts of noise and occlusions.
The key contributions that make this possible are
twofold. First we use a generative model that progressively
increases its complexity and allows a coarse to fine fitting
while 3D-to-2D matches are estimated. Second, we take
advantage of a recursive parameterization of the coronary
tree that introduces dependencies between all the nodes of
the tree, and diffuses the local constraints to the whole
structure.Both the parameterization and the generative
model are then integrated within a Kalman-based optimiza-
tion framework. In the results section we will show that the
overall methodology has significant advantages when com-
pared to state-of-the art approaches.
2. Related Work
Recovering the 3D structure of the coronary tree from
single vascular images involves dealing with many differ-
ent issues. Besides the inherent ambiguity of the monocular
non-rigid reconstruction, the problem is further accentuated
due to the presence of noise in the images and partial oc-
clusions between different branches of the tree. This com-
plexity has been traditionally alleviated by considering the
vascular system as a rigid structure [9, 11, 16] and using
multiple views [26, 29]. To the best of our knowledge, [8]
is the only approach in the medical imaging literature that
considers the non-rigid nature of the problem. They intro-
duce 3D priors and inextensibility constraints into a steepest
descent scheme to solve for the shape. Yet, their optimiza-
tion procedure is only effective under relatively simple de-
formations as those occurring in the liver artery.
On the other hand, our approach has similarities with
the techniques to reconstruct non-rigid 3D surfaces and es-
timate articulated pose from monocular images. Among the
former, it has been shown that 3D shape can be retrieved by
imposing local inextensibility and constraints introduced by
a set of 3D-to-2D correspondences between the input im-
age and a reference shape [5, 14, 17, 19]. In essence we
will also use the same kind of assumptions, although we
will need from additional constraints since in our context
the 3D-to-2D correspondences are unknown and have to be
resolved simultaneously with the shape. Moreover, local
distance constraints are much less restrictive when dealing
with points linked through a tree-like structure than when
dealing with neighboring points on a surface. In addition,
many of these approaches impose strong shape priors based
on previously acquired training data [17, 19] while in our
approach accurate training data is hard to obtain and we
have to rely on very weak shape priors.
Since the coronary tree may be regarded as an articu-
lated structure, one might think in applying the techniques
of articulated pose estimation to our problem [1, 21, 27].
These approaches rely on large amounts of data for learn-
ingamappingfrom2Dimageobservationsto3Dposes, and
have the advantage of not requiring to solve the 2D-to-3D
correspondence problem. Yet, as said above, while obtain-
ing sufficient training data is feasible for applications such
as human pose estimation [2, 23], it becomes prohibitive in
our framework, as the number of X-ray images that may be
captured for each patient is limited.
Recent works suggest introducing similar constraints as
those used for non-rigid shape recovery into the formula-
tion of articulated pose estimation problems [18, 22, 24].
This allows fitting more detailed parametric 3D models [22]
and reducing the dependency of articulated pose estimation
techniques on the training data [18]. However, reducing the
dependency on training data has the drawback of increasing
the sensitivity to artifacts into the input data.
Drawing particular inspiration on these approaches, our
method also combines tools from the techniques for articu-
latedpose estimationand shape recovery. However, inorder
totackleproblemswithmuchlargeramountsofimagenoise
and occlusions, we propose using a generative 3D model
that progressively increases its complexity and adaptability
while establishing correspondences and detecting and re-
jecting outlier points. In addition, we represent the articu-
lated structure using a recursive parametrization that, as we
will show in the results section, yields remarkable improved
results when compared against [18].
3. Algorithm Overview
The focus of this paper is on retrieving the 3D structure
of the coronary tree while establishing 3D-to-2D point cor-
respondences between an input X-ray image and a reference
3D scan. However, the overall algorithm requires additional
tasks as we next detail:
1. Feature extraction: Our input data is an X-ray im-
age and a volumetric 3D Computed Tomography (CT)
scan. In a preprocessing step we segment the vessel re-
gions on both sets of data and extract points of interest.
2. Generative model for the coronary tree: We repre-
sent the 3D feature points as a tree, parameterized
by the joint angles and distances between consecutive
points. Since we cannot explicitly compute deforma-
tion modes from training data, we estimate them by
performing a Probabilistic Principal Component Anal-
ysis (PPCA) over a set of synthetic samples for which
we randomized the values of the joint angles. The flex-
ibility of the model will be controlled by the number of
components of the PPCA and by the magnitude of the
noise used to generate the deformed samples.

Page 3

(a)(b)(c)(d)
(e) (f)(g) (h)
Figure 2. Feature extraction. Top: Extraction of the 3D skeleton from the 3D Computed Tomography. (a) Two levels of thresholding.
(b) Vessel segmentation. (c) 3D skeletonization. (d) Feature point orientations. Bottom: 2D Feature extraction from the X-ray images. (e)
Original image. (f) Vesselness segmentation. (g) Extracted feature points (h) Estimated feature orientations.
3. Non-rigid reconstruction and matching:
Kalman-based approach, we iteratively solve for the
3D-to-2D correspondences and progressively fit the
deformation modes onto the 3D points.
Using a
In the following sections we discuss each one of these
constituent pieces.
4. Feature Extraction
We next briefly describe the steps we perform to seg-
ment the vessel regions and extract the 3D and 2D features
from the raw 3D scans and from the X-ray images respec-
tively. Note that this task is by itself an active area of re-
search within medical imaging [4, 7, 20]. Yet, we design
relatively simple solutions, partially based on the popular
vessel enhancement filter proposed in [6], which although
not being free of error, demonstrate the robustness of our
algorithm for 3D non-rigid reconstruction.
4.1. 3D Features
The raw data of the volumetric 3D CT scanner is com-
posed of a stack of grayscale image slices. The CT volume
is initially segmented using the fuzzy connectedness tree al-
gorithm [28], and as shown in Fig. 2a, a number of regions
corresponding to different physiological structures are gen-
erated. Fig. 2b shows the vessel-like formations. The ves-
sel centerlines are then accurately detected based on a local
steepest gradient ascent of the vesselness measure [6]. Fi-
nally, we homogeneously distribute an arbitrary number of
nodes along the skeleton, and enforce connection using a
minimum spanning tree approach [15](Fig. 2c).
In order to perform the 3D-to-2D matching in a subse-
quent step, besides using the 3D position, we describe each
node based on a unitary vector that indicates the local artery
orientation, which is computed from the eigenvectors of the
Hessian matrix at the appropriate scale. Although we could
have used more elaborated descriptors, we found the po-
sition and orientation to be discriminative enough in our
context of 3D heart scans, where despite non-rigid defor-
mations, the global orientation of the heart remains quite
stable throughout the cardiac cycle. In Fig. 2d we depict the
orientation assigned to each node of the tree.
4.2. 2D Features
A reliable and robust detection of the vessel centerlines
in the 2D X-ray images is more challenging than in the 3D
case. Occlusions produced by vessel crossings, reduced lo-
cal contrast in the diaphragm area, discontinuities due to
lack of the contrast-agent (or stenosis), are some of the ar-
tifacts we may find in these images. Even though a Digi-
tal Subtraction Angiography technique can help to reduce
some of these problems, its practical use is limited to se-
quences in which the relative pose of the X-ray equipment
does not change with respect to the patient, which is not our
case. In addition, while in practice the 3D feature extrac-
tion can be performed off-line in a pre-operative stage, 2D
features must be computed in real-time and without manual
intervention. As a consequence, we will only extract points
of interest and will not attempt to solve the connectivity tree
of the vascular structure.
The extraction of points of interest is performed by ap-
plying the multi-scale vesselness filtering proposed in [6].
We then perform a non-maxima suppression on the vessel-
ness map, followed by a thresholding. For each detected
location, a ridge traversal algorithm, similar to the one pro-
posed in [3] is applied, providing the centerlines of arteries.
In order to select a specific number of keypoints and ensure
they are homogeneously distributed, we further apply a k-
means clustering algorithm, were we set k to the number of
desired keypoints. Finally, each of the keypoints is again
described by the vessel orientation at the point, computed
from the eigenvectors of the Hessian matrix. Fig. 2(g,h)

Page 4

show the final keypoints and their respective orientations.
5. Non-rigid Reconstruction Algorithm
In this section we discuss the core elements of our ap-
proach. After formalizing the problem, we introduce the re-
cursive model we use to represent the deformable coronary
tree, as its particular representation will play a decisive role
for being robust to high levels of occlusions and allowing
to retrieve large deformations. We then describe the weak
priors we use to build a probabilistic generative model of
the tree, and finally we describe the iterative algorithm we
propose that combines all the previous ingredients to solve
for the shape and assign correspondences.
5.1. Problem formulation
Let Mref= {xref
we have extracted from the 3D reference scan and U =
{u1,...,unf} the nffeature points extracted from the 2D
X-ray image, corresponding to a projection of M, a non-
rigidly deformed version of Mref. Our goal is to retrieve
both as many 3D-to-2D correspondences {xref,u} as pos-
sible, and the 3D configuration of the deformed structure
M = {x1,...,xnm}. Note that since in practice the key-
points are obtained from uniformly sampling the segmented
3D scans and X-ray images, it may not exist a perfect one-
to-one match between the sets Mrefand U. However, in
the synthetic results section we will show that our algorithm
tolerateslargeamounts ofnoiseandocclusions, whichcom-
pensates for all these inaccuracies in the matches.
1
,...,xref
nm} be the nmmodel points
5.2. Recursive 3D Model Parameterization
Since we know the links between the 3D points of M,
we can represent the structure of the deformable model by
the vector:
m = [x⊤
1,ρ⊤
2,...,ρ⊤
nm]⊤
(1)
where x1are the 3D coordinates of the root node, and ρj=
[rij,θij,φij]⊤are the spherical coordinates of the vector
joining the i-th and j-th nodes. Thus, the 3D position of a
node xkmay be recursively written as:
xk= x1+
?
i,j∈Ak


rijcosθijsinφij
rijsinθijsinφij
rijcosφij


where Akcontains all the ancestors of the k-th node. Ob-
serve that using this formulation, when the 3D position of
a node is updated, the location of all its ancestors is also
updated. This is a remarkable novel contribution of our for-
mulation, as it naturally introduces constraints that go be-
yond local neighborhoods. In addition, the propagation of
the error using this recursive parameterization is very well
suitedtodealwithstructureslikethecoronarytree, inwhich
the root node remains almost at constant position while the
terminal nodes are usually highly deformed.
Figure 3. Building weak deformation priors. Since we do not
explicitly use training data, we build a weak prior of the tree struc-
ture by assigning gaussian noise to its nodes (left). Note that since
our parameterization is recursive, we assign larger noise values to
the terminal nodes of the tree. Using this prior, we then produce
multiple random shape samples (right) and fit a PPCA model.
5.3. Introducing Synthetic Deformation Priors
Although we do not explicitly use training data besides
the reference 3D scan, we synthetically define weak priors
on the feasible deformations, which will be used within the
optimization scheme to progressively fit the 3D model to
the input data.
Given the 3D parameterization mrefof the reference
shape, we compute the weak priors by first generat-
ing multiple samples {si} from the normal distribution
N(mref,Σm), where Σmis a 3nm× 3nmdiagonal co-
variance matrix generated by concatenating the covariances
of x1and the vectors {ρj}j=2,...,nm. As shown in Fig. 3
these covariances are set to relatively large values in order
to deal with different types of deformations. Yet, in order to
avoid completely random shapes with no physical meaning
we slightly smooth the resulting sampled shapes. In addi-
tion, the variance associated to the inter-node lengths rij
is set to a very small value, as we assume the vessels do
not stretch. Note that although real vessels may stretch, we
represent the tree by a sufficiently dense set of points that
makes the inextensibility assumption locally correct.
We then learn a low-dimensional deformation model of
the coronary tree, by applying Probabilistic Principal Com-
ponent Analysis [25] over the set of deformed samples {si}.
By doing this, we can approximate the pose parameters of
Eq. 1 as a weighted sum of a mean structure m0and nq
deformation modes Q = [q1,...qnq] :
m = m0+
nq
?
i=1
αiqi= m0+ Qα
(2)
where α = [α1,...,αnq]⊤are unknown modal weights
that define the current structure, and whose covariance is
defined by a nq× nqmatrix Σα.
5.4. Iterative Fitting and 3D-to-2D Matching
Representing the coronary tree by means of the modal
weights of Eq. 2 allows rewriting the problem as that of
estimating the parameters α∗such that the following repro-
jection error is minimized

Page 5

InitializationIteration #1Iteration #2 Iteration #3Rigid Registration
Figure 4. Iterative Fitting and 3D-to-2D Matching. We plot the 3D results (top) and 2D results (bottom) of the fitting process at different
iterations and the matches we establish. The left-most figures show the initial prior (red) and the ground truth structure. The next 3 images
show the fitting process of our approach (yellow) for different iterations. The figures on the right show what might be obtained using just
a rigid registration, which is a standard solution to this kind of problems. Obviously the rigid approach yields large errors, specially on the
terminal nodes of the coronary structure.
α∗= argmin
α
nm
?
i=1
?Proj(xi;α) − Match(xi,U)?
(3)
where Match(xi,U) returns the match uj ∈ U for a 3D
point xi, and Proj(xi;α) is the perspective projection of
the 3D point xigiven the modal weights α.
Note that we assume we know the projection matrix of
the imaging system, including both the pose of the patient
with respect to the CT and X-ray equipment, and the intrin-
sic parameters of the detector. In practice, though, there is
always some translation in the position of the patient with
respect to the global reference system. In addition, the heart
deformation during the cardiac cycle and the global heart
shift produced by the breathing introduce additional trans-
lation and small rotation effects. In order to address these
issues, we have considered the translations and small rota-
tions as part of the non-rigid model.
We next turn to the algorithmic steps to minimize Eq. 3.
We achieve this by alternatively solving correspondences
and fitting the 3D model, which we have initialized with the
reference 3D model provided by the 3D scanner.
Establishing 3D-to-2D Correspondences. Let us assume
that as input of an iteration we have α and Σα, the modal
weights and their covariance matrix estimated at the pre-
vious iteration, the set M of 3D model points deformed
according the weights α, and the set U of 2D features ex-
tracted from the X-ray image. We then compute 3D-to-2D
matches with the following steps:
1. We project the 3D model points onto the image, con-
sidering the current configuration of modal weights.
We denote these projections by V = {v1,...,vnm}
where vi= Proj(xi;α).
2. For each vi we establish an uncertainty region sur-
rounding it, by propagating the covariance Σαof the
modal weights to the image plane. This region will be
a Gaussian centered at each viand with covariance:
Σi
v= J(xi)ΣαJ(xi)⊤
where J(xi) is the 2 × nqJacobian of the projection
equationProj(xi;α)withrespecttothemodalweights
α, evaluated at the 3D point xi.
3. Given the set {vi,Σi
points uj∈ U as a standard Optimal Assignment Prob-
lem using the Hungarian algorithm [12]. For this pur-
pose, for each potential match {vi,uj} we introduce
a cost defined as a linear combination of the Maha-
lanobis distance between the two points and their sim-
ilarity in the orientation computed in Section 4:
v} we solve the matching with the
Cij= λ1Mah(vi,uj) + λ2Angle(vi,uj)
where Mah(vi,uj) = (vi− uj)(Σi
and Angle(vi,uj) is the difference in the orientations
between the points of ujand xi, the 3D model point
projected on vi. Note that since the orientation vector
of xiis originally computed in the 3D space, we need
to project it on the image plane to compare it with the
orientation of vi. The terms λ1and λ2are constant
scale factors used to give similar orders of magnitude
toeach ofthecomponents ofthecostfunction. Inprac-
tice, although the Hungarian algorithm is already an
efficient technique for determining the optimal match-
ing, we further reduce its complexity by only consid-
ering those costs Cijwhich are below a certain thresh-
old. This also prevents from fitting the shape to outlier
correspondences or mismatches.
v)−1(vi− uj)⊤,

Page 6

Deformation Level [σθ,σφ]Image Noise [σn] Occlusions (%)
RMS Error 3D [voxels]
RMS Error 2D [pix]
Deformation Samples
σθ = σφ= 0.1
Error 3D: 5.8 [vox]
σθ = σφ= 0.3
Error 3D: 14.7 [vox]
σθ = σφ= 0.5
Error 3D: 28.8 [vox]
Figure 5. Synthetic experiments. RMS error distribution for 3D reconstruction (top) and 2D reprojection (bottom) for each experiment.
The whiskers denote min. and max. errors, the box spans from first to third quartile and the inter-box lines show the mean RMS error.
Updating Modal Weights. Given the set {xi,ui}i=1,...,nc
of estimated correspondences we then use Kalman filter
equations to update the modal weights α and its covari-
ance matrix Σα. Since we simultaneously use all of the
estimated matches, we define an extended 3ncvector of 3D
points ˆ x = [x⊤
locations ˆ u = [u⊤
Jacobian matrixˆJ = [J(x1)⊤,...,J(xnc)⊤]⊤. We then
update α and Σαas
α+
=
α + K(ˆ u − Proj(ˆ x;α))
(I − KˆJ)Σα
1,...,x⊤
1,...,u⊤
nc]⊤, an extended 2ncvector of 2D
nc]⊤, and an extended 2nc× nq
Σα+
=
where K is the Kalman gain and I the identity matrix.
Iterating and Increasing Flexibility. The matching and
modal weight updating processes are iteratively repeated
until the convergence of Eq. 3.
In order to adapt the reference model to highly deformed
structures, such as those shown in Fig. 1, the number of
modes we use is increased at each iteration. This allows to
progressively fit the structure, starting from the most rigid
parts up to the more deformed ones. In addition, using
more rigid structures at early stages yields robustness to
mismatches, preventing to adapt the model towards outlier
2D features. In practice, for a coronary tree with nmpoints,
and thus with a maximum number of 3nmmodes, at itera-
tion number niterwe used nq= 3nm/(10 − niter) modes.
We found the number of iterations to converge to be always
≤ 5. For instance, Fig. 4 shows an example in which con-
vergence was achieved after 3 iterations.
6. Results
We now present the results on both synthetic and real
data. In the synthetic results we compare our approach (de-
noted ArtDeform) to [18]1, which is a representative exam-
ple of the state-of-the-art in articulated pose recovery that
has been shown successful in recovering human and hand
pose. We also compare it to the solution obtained from
a 3D-to-3D rigid transformation between the reference 3D
shape and the ground truth shape associated to the input im-
age, which would represent the best solution that a rigid
2D-to-3D registration algorithm could obtain. For this pur-
pose, we use a standard technique for absolute orientation
estimation [10]. Note that both in [18] and [10], the corre-
spondencesareassumedtobeknown, whileinourapproach
we simultaneously estimate them with the shape.
6.1. Synthetic Experiments
In this section we extensively evaluate the performance
of each algorithm against noise in the correspondences, dif-
ferent levels of deformation and partial occlusions.
generated random shapes of 50 nodes within a volume of
300×300×300 voxels, such as the tree-like structure shown
in Fig. 1, and simulated the deformations undergone in
the coronary tree by applying increasing levels of noise
{σθ,σφ} to the joint angles. We then projected each 3D
shape on a 512×512 image and added gaussian noise of
standard deviation σnto the 2D correspondences. In addi-
tion, a percentage poof the projected points was randomly
removed in order to simulate partial occlusions. Given the
original reference shape and the set of projected points of
the deformed shape, we then performed the reconstruction
with each of the algorithms.
Three different types of experiments were performed.
We initially evaluated the amount of deformation each al-
gorithm was able to recover by sweeping the variances of
We
1We thank Dr. Mathieu Salzmann for kindly testing the data of our
experiments on the algorithm proposed in [18].

Page 7

Left Coronary ArteryRight Coronary Artery
2D Reprojection Error
Rigid Reg.ArtDeform
5
10
15
20
25
30
Reprojection Error [pix]
Figure 6. Reconstruction results on real vessel structures. Left: Several samples of the LCA and RCA reconstructions. Upper row:
initial model (red) and the final non-rigid tree (yellow) overlaid on the input X-ray images. We also plot the results of the 2D segmentation
(blue), which, as can be seen, contains certain amounts of false positives. Bottom row: Initial and retrieved 3D shapes. Note that despite
the segmentation suffers from occlusions and false positive regions our approach yields an accurate solution. Right: 2D reprojection error,
which is used to quantitatively assess the quality of the real results, because the 3D ground truth is not available.
the joint angles within the range [σθ,σφ] ∈ [0 − 0.5] rad,
and randomly setting σn ∈ [1 − 3] and po ∈ [5 − 20]%.
To give significance to the levels of deformation and recon-
struction errors, Fig. 5-right depicts different deformations
of the model corresponding to specific values of {σθ,σφ}.
In a second experiment we analyzed the robustness to im-
age noise by synthetically introducing random noise of
σn∈ [0−10], and setting {σθ,σφ} ∈ [0.15−0.3] and po∈
[5 − 20]%. Finally, we evaluated the effect of occlusions in
an interval po∈ [0 − 50]%, with {σθ,σφ} ∈ [0.15 − 0.3]
and σn∈ [1 − 3].
For each set of parameters we performed 50 trials. The
graphs on Fig. 5 depict the mean 2D reprojection error, ex-
pressed in pixels, and the mean 3D reconstruction error, ex-
pressed in voxels. Observe that our algorithm consistently
outperforms [18] in all experiments. This difference is spe-
cially remarkable when dealing with occlusions, for which
we obtain reconstruction errors below 5 voxels even when
a 50% of the model is occluded. Observe in the top right
graph of Fig. 5 that these amounts of error correspond to
very good approximations. It is fair to mention, though,
that [18] is a general algorithm easily adaptable to different
domains, fromarticulatedstructurestodeformablesurfaces,
while our algorithm is specifically designed to handle tree-
like and articulated structures. In Fig. 5 we also plot the
results that would be recovered using a rigid registration
and, as expected, the errors are significantly larger. Note
that the error values for this case are scaled by a factor 1/2
for displaying purposes.
6.2. Real Data
We also evaluated our approach on real CT data and X-
ray data collected during ordinary pre-operative diagnosis
and percutaneous intervention. We collected CT data of 7
patients, using a Philips Brilliance iCT, at the 75% of the
heart cycle, with slice thickness 0.67 or 0.8mm, and pixel
resolution between 0.38×0.38 and 0.45×0.45 mm. We col-
lected a total of 17 X-ray sequences, 10 of Left Coronary
Artery (LCA) and 7 of Right Coronary Artery (RCA), us-
ing a single plane Philips INTEGRIS Allura Flat Detec-
tor. Image and camera calibration was performed using the
catheter width and the geometrical information on the C-
Arm position. For each sequence, one image in which the
contrast liquid was sufficiently visible was selected.
For each of pair CT scan/X-ray image, we then extracted
3D and 2D features as described in Sect. 4. In all experi-
ments, we represented the segmented CT volume as a tree
with 75-nodes, and extracted 500 feature points from each
X-ray image. Starting with the initial tree of the CT scan,
we then iteratively fit the model and established 3D-to-2D
matches. In all experiments we achieved convergence in
less than 5 iterations, taking about 8 seconds per iteration.
Note that this represents in fact a significantly faster algo-
rithm compared to competing methods. For instance, [8]
reports computation times of about 7 minutes per image.
The 2D registration and 3D reconstruction results of a
few sample experiments are depicted in Fig. 6. Observe that
even when the segmentation contains false positive regions,
or does not detect some branches of the coronary tree our
approach is able to provide an accurate solution.
Sincethe3Dgroundtruthdoesnotexistforthedeformed
artery tree, we quantitatively evaluated the performance of
our algorithm based on the 2D reprojection error with re-
spect to ground truth centerlines manually annotated by an
expert physician. This error is shown in the bar plot of
Fig. 6-right, which summarizes the results for all the 17
experiments. Observe that our non-rigid approach clearly
outperforms a method that rigidly registers the original CT

Page 8

scan. In fact, considering an average calibrated pixel res-
olution of 0.22 mm, the median error of our method is of
about 1.9 mm. This compares very well with the 1 mm er-
ror reported in [11], especially considering that they restrict
their evaluation to X-ray images acquired at systole and di-
astole time instants, where the coronary tree deformation is
minimal compared to other cardiac cycle instants we con-
template.
7. Conclusion
We have presented a novel approach to estimate the 3D
structure of the coronary tree from single X-ray images. In
order to handle the large amounts of deformation, noise
and occlusions present in this kind of images we have in-
troduced a generative model based on a recursive parame-
terization that progressively increases its complexity. We
have integrated this model within a Kalman filter frame-
work which, making use of very weak priors on the struc-
ture, iteratively guides the matching process while recover-
ing coarse-to-fine levels of deformation.
The formulation we propose is fairly general, and al-
lows integrating additional features. As part of future work,
we consider exploiting motion coherence for tracking heart
beat sequences in real time. Moreover, we believe that the
inextensibility constraints between neighboring nodes may
be relaxed, thus allowing to handle stretchable structures.
Acknowledgments
This work has been partially funded by the project La
Marat´ o de TV3 082131; by the Spanish Ministry of Science
and Innovation under projects DPI2008-06022, DPI2010-
17112, Consolider Ingenio 2010 CSD2007-00018; and by
the EU project GARNICS FP7-247947. The work of C.
Gatta is supported by a Beatriu de Pinos Fellowship.
References
[1] A. Agarwal and B. Triggs. 3d human pose from silhouettes
by relevance vector regression. In CVPR, 2004.
[2] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers,
and J. Davis. Scape: shape completion and animation of
people. ACM SIGGRAPH, 24:408–416, 2005.
[3] C.Bauer, H.Bischof. Edge Based Tube Detection for Coro-
naryArteryCenterlineExtraction. MICCAIWorkshop, 2008.
[4] F. Benmansour, L. Cohen, M. Law, and A. Chung. Tubular
anisotropy for 2d vessels segmentation. In CVPR, 2009.
[5] A. Ecker, A. D. Jepson, and K. N. Kutulakos. Semidefinite
programming heuristics for surface reconstruction ambigui-
ties. In ECCV, 2008.
[6] A. Frangi, W. Niessen, K. Vincken, and M. Viergever. Mul-
tiscale vessel enhancement filtering. In MICCAI, 1998.
[7] G.Gonzalez, E.Turetken, F.Fleuret, P.Fua. Delineating trees
in noisy 2d images and 3d image-stacks. In CVPR, 2010.
[8] M. Groher, D. Zikic, and N. Navab. Deformable 2d-3d reg-
istration of vascular structures in a one view scenario. IEEE
Trans. on Medical Imaging, 2009.
[9] J. Hipwell, G. Penney, R. McLaughlin, K. Rhode, P. Sum-
mers, T. Cox, J. Byrne, J. Noble, and D. Hawkes. Inten-
sity based 2d-3d registration of cerebral angiograms. IEEE
Trans. on Medical Imaging, 2003.
[10] B. Horn, H. Hilden, and S. Negahdaripour. Closed-form
solution of absolute orientation using orthonormal matrices.
Journal Optical Society of America, 1988.
[11] C. Metz et al. Patient specific 4d coronary models from ecg-
gated cta data for intra-operative dynamic alignment of cta
with x-ray images. In MICCAI, 2009.
[12] J. Munkres. Algorithms for the assignment and transporta-
tion problems. J. Soc. Ind. and Applied Mathematics, 1957.
[13] A. Myronenko and X. Song. Point set registration: Coherent
point drift. IEEE Trans. PAMI, 2010.
[14] M. Perriollat, R. Hartley, and A. Bartoli.
template-based reconstruction of inextensible surfaces. In
BMVC, 2008.
[15] R. Prim. Shortest connection networks and some generaliza-
tions. Bell System Technical Journal, 1957.
[16] D. Ruijters, B. M. ter Haar Romeny, and P. Suetens.
Vesselness-based 2d3d registration of the coronary arteries.
Int. J. Comp. Ass. Radiology and Surgery, 2009.
[17] M. Salzmann, F. Moreno-Noguer, V. Lepetit, and P. Fua.
Closed-form solution to non-rigid 3d surface registration. In
ECCV, 2008.
[18] M. Salzmann and R. Urtasun. Combining discriminative and
generativemethodsfor3ddeformablesurfaceandarticulated
pose reconstruction. In CVPR, 2010.
[19] J. Sanchez, J. Ostlund, P. Fua, and F. Moreno-Noguer. Si-
multaneous pose, correspondence and non-rigid shape. In
CVPR, 2010.
[20] M. Schaap et al. Standardized evaluation methodology and
reference database for evaluating coronary artery centerline
extraction algorithms. Medical Image Analysis, 2009.
[21] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estima-
tion with parameter-sensitive hashing. In ICCV, 2003.
[22] L. Sigal, A. O. Balan, and M. J. Black. Combined discrim-
inative and generative articulated pose and non-rigid shape
estimation. In NIPS, 2007.
[23] L. Sigal and M. J. Black. Humaneva: Synchronized video
and motion capture dataset for evaluation of articulated hu-
man motion. In Technical Report, Brown University, 2006.
[24] C. Sminchisescu, A. Kanaujia, and D. Metaxas. Learning
joint top-down and bottom-up processes for 3d visual infer-
ence. In CVPR, 2006.
[25] M. Tipping and C. Bishop. Probabilistic principal compo-
nent analysis. Journal of the Royal Statistical Society, 1999.
[26] G.A. Turgeon et al. 2d-3d registration of coronary an-
giogramsforcardiacprocedureplanningandguidance. Med-
ical Physics, 2005.
[27] R. Urtasun and T. Darrell. Local probabilistic regression for
activity-independent human pose inference. In CVPR, 2008.
[28] C. Wang and O. Smedby
andskeletonizationbasedoncompetingfuzzyconnectedness
tree. In MICCAI, 2007.
[29] J. Yao and R. Taylor.Assessing accuracy factors in de-
formable 2d/3d medical image registration using a statistical
pelvis model. In ICCV, 2003.
Monocular
Coronary artery segmentation