3D reconstruction of a femoral shape using a parametric model and two 2D fluoroscopic images

Abstract

In medical diagnostic imaging, the X-ray CT scanner and the MRI system have been widely used to examine 3D shapes and internal structures of living organisms and bones. However, these apparatuses are generally large and very expensive. Since an appointment is also required before examination, these systems are not suitable for urgent fracture diagnosis in emergency treatment. However, X-ray/fluoroscopy has been widely used as traditional medical diagnosis. Therefore, the realization of the reconstruction of precise 3D shapes of living organisms or bones from a few conventional 2D fluoroscopic images might be very useful in practice, in terms of cost, labor, and radiation exposure. The present paper proposes a method by which to estimate a patient-specific 3D shape of a femur from only two fluoroscopic images using a parametric femoral model. First, we develop a parametric femoral model by the statistical analysis of 3D femoral shapes created from CT images of 56 patients. Then, the position and shape parameters of the parametric model are estimated from two 2D fluoroscopic images using a distance map constructed by the Level Set Method. Experiments using synthesized images, fluoroscopic images of a phantom femur, and in vivo images for hip prosthesis patients are successfully carried out, and it is verified that the proposed system has practical applications.

Full text

Page 1

3D reconstruction of a femoral shape using a parametric model
and two 2D fluoroscopic images
Ryo Kurazumea,*, Kaori Nakamuraa, Toshiyuki Okadab, Yoshinobu Satoc, Nobuhiko Suganoc,
Tsuyoshi Koyamad, Yumi Iwashitaa, Tsutomu Hasegawaa
aGraduate School of Information Science and Electrical Engineering, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan
bOsaka University Hospital, Medical Center for Translational Research, 2-15 Yamadaoka, Suita-shi, Osaka 565-0871, Japan
cGraduate School of Medicine, Osaka University, 2–2 Yamadaoka, Suita-shi, Osaka, 565-0871, Japan
dDepartment of Orthopaedic Surgery, National Hospital Organization Osaka-Minami Medical Center, 2-1 Kidohigashimachi, Kawachinagano, Osaka 586-8521, Japan
a r t i c l ei n f o
Article history:
Received 18 September 2007
Accepted 15 August 2008
Available online 12 September 2008
Keywords:
Fluoroscopic image
Parametric femoral model
Registration
Medical image diagnosis
a b s t r a c t
In medical diagnostic imaging, the X-ray CT scanner and the MRI system have been widely used to exam-
ine 3D shapes and internal structures of living organisms and bones. However, these apparatuses are gen-
erally large and very expensive. Since an appointment is also required before examination, these systems
are not suitable for urgent fracture diagnosis in emergency treatment. However, X-ray/fluoroscopy has
been widely used as traditional medical diagnosis. Therefore, the realization of the reconstruction of pre-
cise 3D shapes of living organisms or bones from a few conventional 2D fluoroscopic images might be
very useful in practice, in terms of cost, labor, and radiation exposure. The present paper proposes a
method by which to estimate a patient-specific 3D shape of a femur from only two fluoroscopic images
using a parametric femoral model. First, we develop a parametric femoral model by the statistical anal-
ysis of 3D femoral shapes created from CT images of 56 patients. Then, the position and shape parameters
of the parametric model are estimated from two 2D fluoroscopic images using a distance map con-
structed by the Level Set Method. Experiments using synthesized images, fluoroscopic images of a phan-
tom femur, and in vivo images for hip prosthesis patients are successfully carried out, and it is verified
that the proposed system has practical applications.
? 2008 Elsevier Inc. All rights reserved.
1. Introduction
In medical diagnostic imaging, the X-ray Computed Tomogra-
phy (CT) scanner and the Magnetic Resonance Imaging (MRI) sys-
tem have been widely used to examine the 3D shape or internal
structure of living organisms and bones. However, these appara-
tuses are generally large and very expensive, and thus, they are
usually installed in large medical institutions rather than small
local clinics. Since an appointment is also required before examina-
tion, these systems are not suitable for urgent fracture diagnosis in
emergency treatment.
Meanwhile, X-ray/fluoroscopy has been widely used as tradi-
tional medical diagnosis. Recently, digital fluoroscopy has been
developed and widely used in many hospitals. The cost of this fluo-
roscopic inspection system is much lower than that of CT or MRI
systems and the system can be dealt with more conveniently. Fur-
thermore, the risk of radiation exposure is also lower than that of
the CT inspection system.
From the aboveconsiderations,the realization of the reconstruc-
tion of precise 3D shapes of living organisms or bones from a few
conventional 2D fluoroscopic images might be very useful in prac-
tice, in terms of cost, labor, and radiation exposure. In particular,
there is a strong demand from surgeons for 3D computer-aided sur-
gery without laborious CT imaging for simple surgeries such as arti-
ficial joint replacement or fracture treatment. A practical 3D
diagnostic system usingcommon 2Dfluoroscopic imagesis desired.
However, 3D shape reconstruction from a 2D image is a funda-
mentally ill-posed problem, and so a sufficient number of images
must be obtained, or several constraint conditions for the 3D shape
must be determined. However, the shapes of bones have inherent
and universal patterns, and thus, by modelling such inherent pat-
terns, 3D shape reconstruction from a few 2D images is possible.
In the present paper, a technique by which to estimate the
patient-specific 3D shape of a femur from only two fluoroscopic
images is proposed. The proposed technique utilizes a parametric
femoral model constructed by statistical analysis of 3D femoral
shapes created from CT images of 56 patients. The position/orien-
tation and shape parameters of the parametric model are then esti-
mated from two 2D fluoroscopic images by solving the 2D/3D
registration problem using a distance map constructed by the Level
Set Method.
The 2D/3D registration problem is well established in image
processing, especially for texture mapping in Computer Graphics
1077-3142/$ - see front matter ? 2008 Elsevier Inc. All rights reserved.
doi:10.1016/j.cviu.2008.08.012
* Corresponding author. Fax: +81 92 802 3607.
E-mail address: kurazume@is.kyushu-u.ac.jp (R. Kurazume).
Computer Vision and Image Understanding 113 (2009) 202–211
Contents lists available at ScienceDirect
Computer Vision and Image Understanding
journal homepage: www.elsevier.com/locate/cviu

Page 2

or Augmented Reality. For a rigid object, (1) feature-based
techniques [1–3], (2) image-based techniques using texture, reflec-
tance, brightness, and shading [4,5,6], and (3) silhouette-based
techniques [7–10], have been proposed. In particular, in surgical
navigation systems, Digitally Reconstructed Radiographs (DRRs)
[11,12] are widely used in 2D/3D registration for the fluoros-
copy-guided surgery. However, the construction of DRR is time-
consuming and special techniques such as the use of graphics
hardwares is indispensable for quick medical diagnosis. Since tex-
ture or shadow are usually unavailable in fluoroscopic images, fast
2D/3D registration technique using simple and robust features
such as silhouette and contour lines is desirable.
In 2D/3D registration of a non-rigid object such as soft tissues in
medical imaging, similarity measure [13,14], mutual information
[15], affine [16,17], geometric hashing [18], and displacement-
field-based transformation [19] have been proposed and tested.
In addition, the 3D shape estimation of a parameterized object,
such as the shape reconstruction of mathematical plaster models
with unknown parameters using a laser range finder [20], or the
comparison of multiple cross-section images of a 3D model and a
3D parametric model [21], has also been studied. However, these
studies assumed the use of a sufficient number of images or a pre-
cise 3D shape taken by a laser range finder, and only a few studies
have examined 3D non-rigid shape reconstruction from only a few
2D images [22–24]. Zheng et al. [24] proposed a similar approach
with our method for estimating a femoral shape from fluoroscopic
images. Their technique is based on an active shape model and
conventional ICP method. They introduced experiments for 11
cadaveric femurs using three fluoroscopic images including the
one taken from the longitudinal direction of the femoral shaft.
However, detailed discussion using in vivo images taken by clinical
setting and calibration technique for fluoroscopic imaging was not
presented.
This paper propose the 3D reconstruction technique of a femo-
ral shape using two in vivo fluoroscopic images of patients. A dis-
tance map constructed by the Level Set Method is utilized for fast
and robust 2D/3D registration with M-estimator. The calibration
technique for fluoroscopic imaging using two kinds of calibration
markers is also presented. This paper is organized as follows. Sec-
tion 2 describes the statistical shape model of the femur proposed
by Okada, et al. [25,26] at first. Then, we introduces the proposed
3D reconstruction technique of a femoral shape using the statisti-
cal shape model and two 2D fluoroscopic images. In Section 3,
experiments using synthesized images, fluoroscopic images of a
phantom femur, and in vivo images for hip prosthesis patients,
are successfully carried out, and it is verified that the proposed sys-
tem has practical applications. Finally, Section 4 presents our
conclusion.
2. 3D parametric femoral model
In this section, we introduce the 3D reconstruction technique of
a femoral shape using the statistical shape model and two 2D fluo-
roscopic images.
2.1. Overview of the proposed algorithm
The procedure of the proposed technique is summarized as
follows:
(1) First, we develop a 3D parametric femoral model by the sta-
tistical analysis of 3D femoral shapes created from CT
images of 56 patients. With this model, a general 3D shape
of the femur is expressed by the average shape and several
shape parameters. Thus, the 3D shape estimation of the
patient’s femur from two 2D fluoroscopic images can be
divided into two procedures, the determination of optimum
position and orientation of the 3D model in two 2D fluoro-
scopic images (Step 2) and the estimation of optimum shape
parameters (Step 3).
(2) The position and orientation of the parametric model in two
2D fluoroscopic images are determined by the 2D/3D regis-
tration technique using a distance map constructed by the
Level Set Method.
(3) The optimum shape parameters of the parametric model are
estimated by comparing the silhouette contour of the para-
metric model and two 2D fluoroscopic images using a dis-
tance map obtained in Step 2.
(4) Step 2 and 3 are repeated until the residual error between
the silhouette contour of the parametric shape model and
two 2D fluoroscopic images becomes less than a threshold
value.
Each of the above steps is explained in details in the following
sections.
2.2. 3D parametric femoral model
We utilize the statistical shape model of the femur proposed by
Okada et al. [25,26]. In this technique, a number of 3D femoral
shapes created from CT images are analyzed statistically, and the
parametric femoral model [27,28], which consists of the average
3D shape and several shape parameters, is created. With this para-
metric femoral model, a general 3D shape of the femur is expressed
by the average shape and several shape parameters.
The concrete procedure for creating a parametric 3D femoral
model is as follows:
1. Surface models of femurs are created from CT images by man-
ual segmentation and Marching Cubes.
2. Local coordinate axes of the surface models are determined by
applying the principal component analysis (PCA) to the set of
3D positions of the node points in each surface model. The cen-
ter of gravity is defined as the origin of the local coordinate sys-
tem. The Z axis is determined as the axis corresponding to the
largest eigenvalue, which is toward the longitudinal direction
of the femoral shaft from the hip to knee. The region up to
35% of total length of the femur along the Z axis from the fem-
oral head top defined as the node point having the minimum Z
coordinate is extracted as a proximal femur, where the total
length is defined as the difference of the maximum and mini-
mum Z coordinates in the femur surface model, and the value
of 35% is determined so as to cover a region anatomically
regarded as a proximal femur.
3. One of the femoral model is selected as the reference model and
displacement vector fields to all other models described by the
thin-plate spline are calculated using a point-based non-rigid
registration algorithm[29]. Although the approximation accu-
racy of the resulted parametric model may somewhat depend
on the selected reference model, we currently selected it based
on visual assessment so that it is not largely deviated from the
average shape.The reference model is decimated using an algo-
rithm available in the visualization toolkit so that the number of
nodes of the surface model becomes 1500. 1500 points were
used because the proximal femur shape can be represented in
a reasonable accuracy using 1500 points while computational
cost for the non-rigid registration is still acceptable, for exam-
ple, less than 10 min for each case.The reference model is
non-rigidly registered with each of other models so as to find
the corresponding 1500 points on each model. The non-rigid
registration algorithm [29] generate 3D displacement vector
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
203

Page 3

field from the reference to each dataset. The 1500 points of each
model are determined by combining those of the reference
model and the generated displacement vector field. These pro-
cesses do not guarantee physically-meaningful correspon-
dences. Using these correspondences, however, each surface
model can be represented as a 1500 ? 3 dimensional shape vec-
tor, that is, a fixed-dimensional vector. For our current purpose,
correspondences are acceptable if they are plausible even
though there is not some guarantee.
4. Given n shape vectors, the average shape vector is given by their
average. PCA is applied to a set of the shape vectors subtracted
by the average shape vector to obtain the eigenvectors whose
coefficients correspond to the shape parameters.
The parametric femoral model used in the following experi-
ments was created using CT images of 56 patients. By applying
PCA to 56 samples of 3D femoral shapes, we extracted the most
significant 50 principal components (p1;p2;...;p50), standard devi-
ation(r1;r2;...;r50),and corresponding
(v1;v2;...;v50). With the obtained parametric femoral model, the
general 3D shape of a femur is expressed as
principalvectors
x0¼ x þ ðp1?r1? v1Þ þ ðp2?r2? v2Þ þ ???
where x is the surface point of the average shape and x0is the sur-
face point of the general shape. Therefore, the general 3D shape of a
femur is expressed by the parametric femoral model with
ð1Þ
? average
determined)
? several (up to 50) shape parameters (estimated)
3Dshapeandseveralprincipalvectors(pre-
Fig. 1 shows the contribution ratio of the shape parameters for
the statistical femoral model.
2.3. Reconstruction of 3D femoral shape from two 2D fluoroscopic
images
In this section, we introduce the 2D/3D registration algorithm
and the estimation procedure of the optimum shape parameters
using two fluoroscopic images.
This 2D/3D registration algorithm utilizes the contour lines of
the silhouette of the 2D image and the projected contour lines of
the 3D model. The optimum position of the 3D model is deter-
mined such that the contour lines coincide with each other on
the 2D image plane. In commonly used approaches such as the
ICP algorithm, the error metric is usually defined as the sum of
the distances between the points on the 2D contour lines and their
nearest points on the projected contour lines of the 3D model.
However, the nearest point search is a laborious task and is time
consuming even for the kd tree-based algorithm [30].
In the present approach, the 2D distance map [7] is utilized.
First, the 2D distance map from the contour lines is created on
the 2D image using the Fast Marching Method [31,32] or raster
scan algorithms using local operators [33,34]. Once the 2D distance
map is created, the error metric is obtained directly from the 2D
distance map as the value at the points on the projected contour
lines of the 3D model. Using the course-to-fine strategy called
‘‘Distance Band” [7], a 2D distance map can be constructed quite
rapidly using the Fast Marching Method.
When 2D/3D registration and estimation of the shape parame-
ters are performed at the same time, the depth from the view point
and the scale of the 3D model cannot be distinguished. Therefore,
the proposed algorithm utilizes two fluoroscopic images taken
from two viewpoints at different positions. In addition, we assume
that the 3D femoral parametric model is constituted by a large
number of small triangle patches of approximately the same size.
In the following sections, we explain Step 2 and Step 3 shown in
Section 2.1. In both steps, 2D distance maps from the extracted
contour lines of patient’s femur in fluoroscopic images are created
at first. Then the error between the contour lines of the femur and
the projected contour lines of the 3D model is minimized by read-
ing the distance values of the 2D distance maps.
2.3.1. Registration of 2D fluoroscopic images and the 3D parametric
model
A brief description of the registration procedure of the 2D fluo-
roscopic images and the 3D parametric model is given as follows:
1. Extract the contour lines of the femur in the fluoroscopic
images using an active contour model such as snakes or the
Level Set Method [31].
2. Construct a 2D distance map from the extracted contour lines
using the Fast Marching Method [31,32]. Fig. 2 shows an exam-
ple of the constructed 2D distance map of a femoral image.
3. Place the parametric femoral model at an arbitrary initial posi-
tion which is determined manually and calculate the 2D projec-
tion image of the 3D model.
4. Extract contour lines of the projected image and corresponding
3D patches of the 3D model. This procedure can be executed
quite rapidly by the OpenGL hardware accelerator (Appendix
A.1).
5(a). Apply the force calculated from the 2D distance map at the
projected contour points directly to the corresponding 3D
patch. Details are presented in Section 2.4.
6(a). Using the robust M-estimator, which is a robust estimation
technique, the total force and moment around the center
of gravity (COG) is calculated.
0
0.2
0.4
0.6
0.8
1
10 20 30 40 50
Number of parameters
Contribution
0
93%
Fig. 1. Contribution of parametric model.
+20 pixels
Boundary
Contour line in distance map
+40 pixels
Fig. 2. 2D distance map from contour in the femoral image.
204
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211

Page 4

7(a). Steps 3 to 6(a) of the procedure are repeated for two images
captured from the different viewpoints sequentially, and the
total force and moment are calculated.
8(a). Update the position of the 3D parametric model according to
the total force and moment.
9(a). Repeat Steps 3 to 8(a) until the magnitude of the total force
and moment becomes less than the pre-defined threshold
value.
2.3.2. Estimation of the optimum shape parameters
The estimation procedure of the optimum shape parameters of
the 3D parametric femoral model is shown in this section. This pro-
cedure also uses the 2D distance map from the contour line of the
femur in the fluoroscopic image, which has already been con-
structed, as described in Section 2.3.1. Therefore, from Steps 1 to
4 are same as Section 2.3.1 and we can skip these procedures by
using the obtained 2D distance map in Step 4 of the above proce-
dure, After Step 4, the optimum shape parameters are estimated as
follows:
5(b). Calculate the error E, which is defined as the sum of the val-
ues of the 2D distance map at the projected contour line of
the 3D parametric model.
6(b). Find the optimum shape parameters that minimize the error
E at the current position using the conjugate gradient
method.
7(b). Reconstruct the 3D shape according the obtained shape
parameters using Eq. (1).
8(b). Repeat Steps 3 to 7(b) until the error E becomes less than the
pre-defined threshold value.
2.4. 2D/3D registration using the robust M-estimator
After obtaining the distance map on the 2D fluoroscopic image
and the list of the triangular patches of the 3D model correspond-
ing to the contour points, the force fiis applied to all of the trian-
gular patches of the contour points (Figs. 3 and 4), as explained in
Step 5(a).
fi¼ Di
rDi
jrDij
ð2Þ
where Di is the value of the distance map at the contour point,
which corresponds to the triangular patch i, and rDiis the gradient
of Di.
In Step 6(a), the total force and moment around the center of
gravity (COG) is calculated by the following equations:
X
M ¼
i
F ¼
i
wðfiÞð3Þ
X
wðri? fiÞð4Þ
where riis a vector from the COG to triangular patch i and wðzÞ is a
particular estimate function. In practical situations, the contour of
the femur is occasionally occluded or blurred, or the 2D image is
corrupted by noise. In such cases, the obtained boundary does not
coincide with the projected contour of the 3D model and the correct
distance value cannot be obtained. To deal with this problem, we
introduce the robust M-estimator in order to disregard contour
points with large errors. In our implementation, we utilized a fol-
lowing function using a Lorentzian function with a variance r2
and a vector z.
wðzÞ ¼
1
1þ j zj2=r2z
Let us consider the force fiand the moment ri? fias an error zi
and the sum of the error as
X
where EðPÞ is a scalar function of a vector P which is the position of
the 3D parametric model, and qðzÞ is a particular estimate function,
which is defined as
ð5Þ
EðPÞ ¼
i
qðziÞð6Þ
oqðzÞ
oz
¼ wðzÞ
The position P that minimizes the error EðPÞ is obtained as the
following equation:
ð7Þ
oE
oP¼
X
i
oqðziÞ
ozi
ozi
oP¼ 0
ð8Þ
Here, we define the weight function wðzÞ as the following equation
in order to evaluate the error term:
wðzÞz ¼ wðzÞ ¼oqðzÞ
From the above equation, we obtain the following weighted least
squares method:
X
In our implementation, the optimum position, which minimizes the
error EðPÞ, is obtained by the steepest gradient method, as shown in
Step 9(a).
oz
ð9Þ
oE
oP¼
i
wðziÞziozi
oP¼ 0
ð10Þ
3. Experiments
3.1. Simulation using DRRs
First, we conducted the experiments using Digitally Recon-
structed Radiographs (DRRs) in order to evaluate the fundamental
fi
Boundary
Fig. 3. Force fiis applied to 3D triangular patch i on the contour line.
F
Image plane
Focal point
Projected
contour point
Object
Steepest descent
direction
v
a
b
fi
x
z
y
fi
ri
Center of gravity
fi
M
fi
Center of gravity
Fig. 4. Calculation of the total force and moment around the center of gravity
(COG).
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
205

Page 5

performance of the proposed method. In the experiment, the esti-
mation accuracy for 10 femoral models is examined using two
reconstructed fluoroscopic images. Among 10 models, five models
(model data 1–5) are used for the construction of the 3D paramet-
ric model and five models (test data 1–5) are not used.
We determine the directions of the fluoroscopic images as
shown in Fig. 5 considering the possible direction in actual radio-
graphic examination. Under this condition, two view directions
meet at right angles at the main axis of the femur AP. Examples
of the reconstructed fluoroscopic images are shown in Fig. 6.
All of the 3D femoral shapes used in the experiments were
reconstructed precisely by the CT scanner beforehand, and the
optimized shape parameters, which minimize the distance errors
between surface points, were determined by comparing the 3D ac-
tual shape and the 3D parametric model and searching all possible
candidates.
First, we chose up to 10 principal components and estimated
the position and optimum shape parameters of the femur on the
fluoroscopic images. In this experiment, the position estimation
of the femur and the optimum parameter estimation were
repeated alternately and independently. An example of the exper-
imental results for test data 4 is shown in Fig. 7, which illustrates
the average shape, the actual shape, and the estimated shape.
Fig. 8 indicates the average error between the estimated shape
and the actual shape. The average error is defined as the average of
the minimum distance from the surface point of the estimated
shape to the triangle patches of the actual shape. In this figure,
‘‘AS” on the horizontal axis indicates the average error when the
position is estimated but all of the shape parameters are fixed to
their initial values (all of the parameters are set to ‘‘0”). This figure
shows that the average error gradually decreases as the number of
estimated shape parameters increases. However, the average error
converges when the number of shape parameters is approximately
five and no significant difference is observed, even if the number of
the shape parameters increases.
In addition, Fig. 9 shows the types of errors defined below when
the number of estimated shape parameters is five. Table 1 also
indicates the average of the error, the standard deviation, the max-
imum value, and the minimum value for the 10 models.
Average error 1 The average error between the average model
and the actual shape.
Average error 2 The average error between the 3D optimized
estimated shape and the 3D actual shape obtained by compar-
ing the actual shape and the parametric model.
Average error 3 The average error between the estimated shape
and the actual shape obtained by comparing the two 2D fluoro-
scopic images and the 3D parametric model (proposed method).
The experimental results show that Average error 3 between
the estimated shape and the actual shape is less than 1.1 mm at
Ap
Fig. 5. Directions of DRRs for simulation.
Fig. 6. Examples of DRRs for simulation experiments.
Fig. 7. Actual and estimated femoral model with and without shape parameter
estimation.
0
0.5
1
1.5
2
2.5
AS2468 10
Number of estimated parameters
Average error [mm]
test data 1~5
model data 1~5
AS: Average shape
Fig. 8. Average error and number of estimated shape parameters.
0
1
2
3
No.1
No.2
No.3
No.4
No.5
0
1
2
3
No.1
No.2
No.3No.4
No.5
average error 1average error 2average error 3
test data
model data
Fig. 9. Average error after shape parameter estimation (number of principal
components: 5).
206
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211