Page 1
3D reconstruction of a femoral shape using a parametric model
and two 2D fluoroscopic images
Ryo Kurazumea,*, Kaori Nakamuraa, Toshiyuki Okadab, Yoshinobu Satoc, Nobuhiko Suganoc,
Tsuyoshi Koyamad, Yumi Iwashitaa, Tsutomu Hasegawaa
aGraduate School of Information Science and Electrical Engineering, Kyushu University, 744 Motooka, Nishiku, Fukuoka 8190395, Japan
bOsaka University Hospital, Medical Center for Translational Research, 215 Yamadaoka, Suitashi, Osaka 5650871, Japan
cGraduate School of Medicine, Osaka University, 2–2 Yamadaoka, Suitashi, Osaka, 5650871, Japan
dDepartment of Orthopaedic Surgery, National Hospital Organization OsakaMinami Medical Center, 21 Kidohigashimachi, Kawachinagano, Osaka 5868521, Japan
a r t i c l ei n f o
Article history:
Received 18 September 2007
Accepted 15 August 2008
Available online 12 September 2008
Keywords:
Fluoroscopic image
Parametric femoral model
Registration
Medical image diagnosis
a b s t r a c t
In medical diagnostic imaging, the Xray CT scanner and the MRI system have been widely used to exam
ine 3D shapes and internal structures of living organisms and bones. However, these apparatuses are gen
erally large and very expensive. Since an appointment is also required before examination, these systems
are not suitable for urgent fracture diagnosis in emergency treatment. However, Xray/fluoroscopy has
been widely used as traditional medical diagnosis. Therefore, the realization of the reconstruction of pre
cise 3D shapes of living organisms or bones from a few conventional 2D fluoroscopic images might be
very useful in practice, in terms of cost, labor, and radiation exposure. The present paper proposes a
method by which to estimate a patientspecific 3D shape of a femur from only two fluoroscopic images
using a parametric femoral model. First, we develop a parametric femoral model by the statistical anal
ysis of 3D femoral shapes created from CT images of 56 patients. Then, the position and shape parameters
of the parametric model are estimated from two 2D fluoroscopic images using a distance map con
structed by the Level Set Method. Experiments using synthesized images, fluoroscopic images of a phan
tom femur, and in vivo images for hip prosthesis patients are successfully carried out, and it is verified
that the proposed system has practical applications.
? 2008 Elsevier Inc. All rights reserved.
1. Introduction
In medical diagnostic imaging, the Xray Computed Tomogra
phy (CT) scanner and the Magnetic Resonance Imaging (MRI) sys
tem have been widely used to examine the 3D shape or internal
structure of living organisms and bones. However, these appara
tuses are generally large and very expensive, and thus, they are
usually installed in large medical institutions rather than small
local clinics. Since an appointment is also required before examina
tion, these systems are not suitable for urgent fracture diagnosis in
emergency treatment.
Meanwhile, Xray/fluoroscopy has been widely used as tradi
tional medical diagnosis. Recently, digital fluoroscopy has been
developed and widely used in many hospitals. The cost of this fluo
roscopic inspection system is much lower than that of CT or MRI
systems and the system can be dealt with more conveniently. Fur
thermore, the risk of radiation exposure is also lower than that of
the CT inspection system.
From the aboveconsiderations,the realization of the reconstruc
tion of precise 3D shapes of living organisms or bones from a few
conventional 2D fluoroscopic images might be very useful in prac
tice, in terms of cost, labor, and radiation exposure. In particular,
there is a strong demand from surgeons for 3D computeraided sur
gery without laborious CT imaging for simple surgeries such as arti
ficial joint replacement or fracture treatment. A practical 3D
diagnostic system usingcommon 2Dfluoroscopic imagesis desired.
However, 3D shape reconstruction from a 2D image is a funda
mentally illposed problem, and so a sufficient number of images
must be obtained, or several constraint conditions for the 3D shape
must be determined. However, the shapes of bones have inherent
and universal patterns, and thus, by modelling such inherent pat
terns, 3D shape reconstruction from a few 2D images is possible.
In the present paper, a technique by which to estimate the
patientspecific 3D shape of a femur from only two fluoroscopic
images is proposed. The proposed technique utilizes a parametric
femoral model constructed by statistical analysis of 3D femoral
shapes created from CT images of 56 patients. The position/orien
tation and shape parameters of the parametric model are then esti
mated from two 2D fluoroscopic images by solving the 2D/3D
registration problem using a distance map constructed by the Level
Set Method.
The 2D/3D registration problem is well established in image
processing, especially for texture mapping in Computer Graphics
10773142/$  see front matter ? 2008 Elsevier Inc. All rights reserved.
doi:10.1016/j.cviu.2008.08.012
* Corresponding author. Fax: +81 92 802 3607.
Email address: kurazume@is.kyushuu.ac.jp (R. Kurazume).
Computer Vision and Image Understanding 113 (2009) 202–211
Contents lists available at ScienceDirect
Computer Vision and Image Understanding
journal homepage: www.elsevier.com/locate/cviu
Page 2
or Augmented Reality. For a rigid object, (1) featurebased
techniques [1–3], (2) imagebased techniques using texture, reflec
tance, brightness, and shading [4,5,6], and (3) silhouettebased
techniques [7–10], have been proposed. In particular, in surgical
navigation systems, Digitally Reconstructed Radiographs (DRRs)
[11,12] are widely used in 2D/3D registration for the fluoros
copyguided surgery. However, the construction of DRR is time
consuming and special techniques such as the use of graphics
hardwares is indispensable for quick medical diagnosis. Since tex
ture or shadow are usually unavailable in fluoroscopic images, fast
2D/3D registration technique using simple and robust features
such as silhouette and contour lines is desirable.
In 2D/3D registration of a nonrigid object such as soft tissues in
medical imaging, similarity measure [13,14], mutual information
[15], affine [16,17], geometric hashing [18], and displacement
fieldbased transformation [19] have been proposed and tested.
In addition, the 3D shape estimation of a parameterized object,
such as the shape reconstruction of mathematical plaster models
with unknown parameters using a laser range finder [20], or the
comparison of multiple crosssection images of a 3D model and a
3D parametric model [21], has also been studied. However, these
studies assumed the use of a sufficient number of images or a pre
cise 3D shape taken by a laser range finder, and only a few studies
have examined 3D nonrigid shape reconstruction from only a few
2D images [22–24]. Zheng et al. [24] proposed a similar approach
with our method for estimating a femoral shape from fluoroscopic
images. Their technique is based on an active shape model and
conventional ICP method. They introduced experiments for 11
cadaveric femurs using three fluoroscopic images including the
one taken from the longitudinal direction of the femoral shaft.
However, detailed discussion using in vivo images taken by clinical
setting and calibration technique for fluoroscopic imaging was not
presented.
This paper propose the 3D reconstruction technique of a femo
ral shape using two in vivo fluoroscopic images of patients. A dis
tance map constructed by the Level Set Method is utilized for fast
and robust 2D/3D registration with Mestimator. The calibration
technique for fluoroscopic imaging using two kinds of calibration
markers is also presented. This paper is organized as follows. Sec
tion 2 describes the statistical shape model of the femur proposed
by Okada, et al. [25,26] at first. Then, we introduces the proposed
3D reconstruction technique of a femoral shape using the statisti
cal shape model and two 2D fluoroscopic images. In Section 3,
experiments using synthesized images, fluoroscopic images of a
phantom femur, and in vivo images for hip prosthesis patients,
are successfully carried out, and it is verified that the proposed sys
tem has practical applications. Finally, Section 4 presents our
conclusion.
2. 3D parametric femoral model
In this section, we introduce the 3D reconstruction technique of
a femoral shape using the statistical shape model and two 2D fluo
roscopic images.
2.1. Overview of the proposed algorithm
The procedure of the proposed technique is summarized as
follows:
(1) First, we develop a 3D parametric femoral model by the sta
tistical analysis of 3D femoral shapes created from CT
images of 56 patients. With this model, a general 3D shape
of the femur is expressed by the average shape and several
shape parameters. Thus, the 3D shape estimation of the
patient’s femur from two 2D fluoroscopic images can be
divided into two procedures, the determination of optimum
position and orientation of the 3D model in two 2D fluoro
scopic images (Step 2) and the estimation of optimum shape
parameters (Step 3).
(2) The position and orientation of the parametric model in two
2D fluoroscopic images are determined by the 2D/3D regis
tration technique using a distance map constructed by the
Level Set Method.
(3) The optimum shape parameters of the parametric model are
estimated by comparing the silhouette contour of the para
metric model and two 2D fluoroscopic images using a dis
tance map obtained in Step 2.
(4) Step 2 and 3 are repeated until the residual error between
the silhouette contour of the parametric shape model and
two 2D fluoroscopic images becomes less than a threshold
value.
Each of the above steps is explained in details in the following
sections.
2.2. 3D parametric femoral model
We utilize the statistical shape model of the femur proposed by
Okada et al. [25,26]. In this technique, a number of 3D femoral
shapes created from CT images are analyzed statistically, and the
parametric femoral model [27,28], which consists of the average
3D shape and several shape parameters, is created. With this para
metric femoral model, a general 3D shape of the femur is expressed
by the average shape and several shape parameters.
The concrete procedure for creating a parametric 3D femoral
model is as follows:
1. Surface models of femurs are created from CT images by man
ual segmentation and Marching Cubes.
2. Local coordinate axes of the surface models are determined by
applying the principal component analysis (PCA) to the set of
3D positions of the node points in each surface model. The cen
ter of gravity is defined as the origin of the local coordinate sys
tem. The Z axis is determined as the axis corresponding to the
largest eigenvalue, which is toward the longitudinal direction
of the femoral shaft from the hip to knee. The region up to
35% of total length of the femur along the Z axis from the fem
oral head top defined as the node point having the minimum Z
coordinate is extracted as a proximal femur, where the total
length is defined as the difference of the maximum and mini
mum Z coordinates in the femur surface model, and the value
of 35% is determined so as to cover a region anatomically
regarded as a proximal femur.
3. One of the femoral model is selected as the reference model and
displacement vector fields to all other models described by the
thinplate spline are calculated using a pointbased nonrigid
registration algorithm[29]. Although the approximation accu
racy of the resulted parametric model may somewhat depend
on the selected reference model, we currently selected it based
on visual assessment so that it is not largely deviated from the
average shape.The reference model is decimated using an algo
rithm available in the visualization toolkit so that the number of
nodes of the surface model becomes 1500. 1500 points were
used because the proximal femur shape can be represented in
a reasonable accuracy using 1500 points while computational
cost for the nonrigid registration is still acceptable, for exam
ple, less than 10 min for each case.The reference model is
nonrigidly registered with each of other models so as to find
the corresponding 1500 points on each model. The nonrigid
registration algorithm [29] generate 3D displacement vector
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
203
Page 3
field from the reference to each dataset. The 1500 points of each
model are determined by combining those of the reference
model and the generated displacement vector field. These pro
cesses do not guarantee physicallymeaningful correspon
dences. Using these correspondences, however, each surface
model can be represented as a 1500 ? 3 dimensional shape vec
tor, that is, a fixeddimensional vector. For our current purpose,
correspondences are acceptable if they are plausible even
though there is not some guarantee.
4. Given n shape vectors, the average shape vector is given by their
average. PCA is applied to a set of the shape vectors subtracted
by the average shape vector to obtain the eigenvectors whose
coefficients correspond to the shape parameters.
The parametric femoral model used in the following experi
ments was created using CT images of 56 patients. By applying
PCA to 56 samples of 3D femoral shapes, we extracted the most
significant 50 principal components (p1;p2;...;p50), standard devi
ation(r1;r2;...;r50),andcorresponding
(v1;v2;...;v50). With the obtained parametric femoral model, the
general 3D shape of a femur is expressed as
principal vectors
x0¼ x þ ðp1?r1? v1Þ þ ðp2?r2? v2Þ þ ???
where x is the surface point of the average shape and x0is the sur
face point of the general shape. Therefore, the general 3D shape of a
femur is expressed by the parametric femoral model with
ð1Þ
? average
determined)
? several (up to 50) shape parameters (estimated)
3Dshapeandseveralprincipalvectors(pre
Fig. 1 shows the contribution ratio of the shape parameters for
the statistical femoral model.
2.3. Reconstruction of 3D femoral shape from two 2D fluoroscopic
images
In this section, we introduce the 2D/3D registration algorithm
and the estimation procedure of the optimum shape parameters
using two fluoroscopic images.
This 2D/3D registration algorithm utilizes the contour lines of
the silhouette of the 2D image and the projected contour lines of
the 3D model. The optimum position of the 3D model is deter
mined such that the contour lines coincide with each other on
the 2D image plane. In commonly used approaches such as the
ICP algorithm, the error metric is usually defined as the sum of
the distances between the points on the 2D contour lines and their
nearest points on the projected contour lines of the 3D model.
However, the nearest point search is a laborious task and is time
consuming even for the kd treebased algorithm [30].
In the present approach, the 2D distance map [7] is utilized.
First, the 2D distance map from the contour lines is created on
the 2D image using the Fast Marching Method [31,32] or raster
scan algorithms using local operators [33,34]. Once the 2D distance
map is created, the error metric is obtained directly from the 2D
distance map as the value at the points on the projected contour
lines of the 3D model. Using the coursetofine strategy called
‘‘Distance Band” [7], a 2D distance map can be constructed quite
rapidly using the Fast Marching Method.
When 2D/3D registration and estimation of the shape parame
ters are performed at the same time, the depth from the view point
and the scale of the 3D model cannot be distinguished. Therefore,
the proposed algorithm utilizes two fluoroscopic images taken
from two viewpoints at different positions. In addition, we assume
that the 3D femoral parametric model is constituted by a large
number of small triangle patches of approximately the same size.
In the following sections, we explain Step 2 and Step 3 shown in
Section 2.1. In both steps, 2D distance maps from the extracted
contour lines of patient’s femur in fluoroscopic images are created
at first. Then the error between the contour lines of the femur and
the projected contour lines of the 3D model is minimized by read
ing the distance values of the 2D distance maps.
2.3.1. Registration of 2D fluoroscopic images and the 3D parametric
model
A brief description of the registration procedure of the 2D fluo
roscopic images and the 3D parametric model is given as follows:
1. Extract the contour lines of the femur in the fluoroscopic
images using an active contour model such as snakes or the
Level Set Method [31].
2. Construct a 2D distance map from the extracted contour lines
using the Fast Marching Method [31,32]. Fig. 2 shows an exam
ple of the constructed 2D distance map of a femoral image.
3. Place the parametric femoral model at an arbitrary initial posi
tion which is determined manually and calculate the 2D projec
tion image of the 3D model.
4. Extract contour lines of the projected image and corresponding
3D patches of the 3D model. This procedure can be executed
quite rapidly by the OpenGL hardware accelerator (Appendix
A.1).
5(a). Apply the force calculated from the 2D distance map at the
projected contour points directly to the corresponding 3D
patch. Details are presented in Section 2.4.
6(a). Using the robust Mestimator, which is a robust estimation
technique, the total force and moment around the center
of gravity (COG) is calculated.
0
0.2
0.4
0.6
0.8
1
10 20 30 40 50
Number of parameters
Contribution
0
93%
Fig. 1. Contribution of parametric model.
+20 pixels
Boundary
Contour line in distance map
+40 pixels
Fig. 2. 2D distance map from contour in the femoral image.
204
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
Page 4
7(a). Steps 3 to 6(a) of the procedure are repeated for two images
captured from the different viewpoints sequentially, and the
total force and moment are calculated.
8(a). Update the position of the 3D parametric model according to
the total force and moment.
9(a). Repeat Steps 3 to 8(a) until the magnitude of the total force
and moment becomes less than the predefined threshold
value.
2.3.2. Estimation of the optimum shape parameters
The estimation procedure of the optimum shape parameters of
the 3D parametric femoral model is shown in this section. This pro
cedure also uses the 2D distance map from the contour line of the
femur in the fluoroscopic image, which has already been con
structed, as described in Section 2.3.1. Therefore, from Steps 1 to
4 are same as Section 2.3.1 and we can skip these procedures by
using the obtained 2D distance map in Step 4 of the above proce
dure, After Step 4, the optimum shape parameters are estimated as
follows:
5(b). Calculate the error E, which is defined as the sum of the val
ues of the 2D distance map at the projected contour line of
the 3D parametric model.
6(b). Find the optimum shape parameters that minimize the error
E at the current position using the conjugate gradient
method.
7(b). Reconstruct the 3D shape according the obtained shape
parameters using Eq. (1).
8(b). Repeat Steps 3 to 7(b) until the error E becomes less than the
predefined threshold value.
2.4. 2D/3D registration using the robust Mestimator
After obtaining the distance map on the 2D fluoroscopic image
and the list of the triangular patches of the 3D model correspond
ing to the contour points, the force fiis applied to all of the trian
gular patches of the contour points (Figs. 3 and 4), as explained in
Step 5(a).
fi¼ Di
rDi
jrDij
ð2Þ
where Di is the value of the distance map at the contour point,
which corresponds to the triangular patch i, and rDiis the gradient
of Di.
In Step 6(a), the total force and moment around the center of
gravity (COG) is calculated by the following equations:
X
M ¼
i
F ¼
i
wðfiÞð3Þ
X
wðri? fiÞð4Þ
where riis a vector from the COG to triangular patch i and wðzÞ is a
particular estimate function. In practical situations, the contour of
the femur is occasionally occluded or blurred, or the 2D image is
corrupted by noise. In such cases, the obtained boundary does not
coincide with the projected contour of the 3D model and the correct
distance value cannot be obtained. To deal with this problem, we
introduce the robust Mestimator in order to disregard contour
points with large errors. In our implementation, we utilized a fol
lowing function using a Lorentzian function with a variance r2
and a vector z.
wðzÞ ¼
1
1þ j zj2=r2z
Let us consider the force fiand the moment ri? fias an error zi
and the sum of the error as
X
where EðPÞ is a scalar function of a vector P which is the position of
the 3D parametric model, and qðzÞ is a particular estimate function,
which is defined as
ð5Þ
EðPÞ ¼
i
qðziÞð6Þ
oqðzÞ
oz
¼ wðzÞ
The position P that minimizes the error EðPÞ is obtained as the
following equation:
ð7Þ
oE
oP¼
X
i
oqðziÞ
ozi
ozi
oP¼ 0
ð8Þ
Here, we define the weight function wðzÞ as the following equation
in order to evaluate the error term:
wðzÞz ¼ wðzÞ ¼oqðzÞ
From the above equation, we obtain the following weighted least
squares method:
X
In our implementation, the optimum position, which minimizes the
error EðPÞ, is obtained by the steepest gradient method, as shown in
Step 9(a).
oz
ð9Þ
oE
oP¼
i
wðziÞziozi
oP¼ 0
ð10Þ
3. Experiments
3.1. Simulation using DRRs
First, we conducted the experiments using Digitally Recon
structed Radiographs (DRRs) in order to evaluate the fundamental
fi
Boundary
Fig. 3. Force fiis applied to 3D triangular patch i on the contour line.
F
Image plane
Focal point
Projected
contour point
Object
Steepest descent
direction
v
a
b
fi
x
z
y
fi
ri
Center of gravity
fi
M
fi
Center of gravity
Fig. 4. Calculation of the total force and moment around the center of gravity
(COG).
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
205
Page 5
performance of the proposed method. In the experiment, the esti
mation accuracy for 10 femoral models is examined using two
reconstructed fluoroscopic images. Among 10 models, five models
(model data 1–5) are used for the construction of the 3D paramet
ric model and five models (test data 1–5) are not used.
We determine the directions of the fluoroscopic images as
shown in Fig. 5 considering the possible direction in actual radio
graphic examination. Under this condition, two view directions
meet at right angles at the main axis of the femur AP. Examples
of the reconstructed fluoroscopic images are shown in Fig. 6.
All of the 3D femoral shapes used in the experiments were
reconstructed precisely by the CT scanner beforehand, and the
optimized shape parameters, which minimize the distance errors
between surface points, were determined by comparing the 3D ac
tual shape and the 3D parametric model and searching all possible
candidates.
First, we chose up to 10 principal components and estimated
the position and optimum shape parameters of the femur on the
fluoroscopic images. In this experiment, the position estimation
of the femur and the optimum parameter estimation were
repeated alternately and independently. An example of the exper
imental results for test data 4 is shown in Fig. 7, which illustrates
the average shape, the actual shape, and the estimated shape.
Fig. 8 indicates the average error between the estimated shape
and the actual shape. The average error is defined as the average of
the minimum distance from the surface point of the estimated
shape to the triangle patches of the actual shape. In this figure,
‘‘AS” on the horizontal axis indicates the average error when the
position is estimated but all of the shape parameters are fixed to
their initial values (all of the parameters are set to ‘‘0”). This figure
shows that the average error gradually decreases as the number of
estimated shape parameters increases. However, the average error
converges when the number of shape parameters is approximately
five and no significant difference is observed, even if the number of
the shape parameters increases.
In addition, Fig. 9 shows the types of errors defined below when
the number of estimated shape parameters is five. Table 1 also
indicates the average of the error, the standard deviation, the max
imum value, and the minimum value for the 10 models.
Average error 1 The average error between the average model
and the actual shape.
Average error 2 The average error between the 3D optimized
estimated shape and the 3D actual shape obtained by compar
ing the actual shape and the parametric model.
Average error 3 The average error between the estimated shape
and the actual shape obtained by comparing the two 2D fluoro
scopic images and the 3D parametric model (proposed method).
The experimental results show that Average error 3 between
the estimated shape and the actual shape is less than 1.1 mm at
Ap
Fig. 5. Directions of DRRs for simulation.
Fig. 6. Examples of DRRs for simulation experiments.
Fig. 7. Actual and estimated femoral model with and without shape parameter
estimation.
0
0.5
1
1.5
2
2.5
AS 2468 10
Number of estimated parameters
Average error [mm]
test data 1~5
model data 1~5
AS: Average shape
Fig. 8. Average error and number of estimated shape parameters.
0
1
2
3
No.1
No.2
No.3
No.4
No.5
0
1
2
3
No.1
No.2
No.3No.4
No.5
average error 1average error 2 average error 3
test data
model data
Fig. 9. Average error after shape parameter estimation (number of principal
components: 5).
206
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
Page 6
worst, and it is verified that the 3D shape can be estimated using
two 2D fluoroscopic images with the same accuracy in case that
the 3D shapes are compared directly. Moreover, it is confirmed
that there is no significant difference between the models that
were used for the construction of the parametric femoral model
and those that were not.
Note that in some cases in Fig. 9, Average error 3 is smaller than
Average error 2. In the calculation of Average error 2, the origin and
the direction of the local coordinate systems of the estimated and
actual shapes coincide with each other and relative position of
these shapes is fixed. On the other hand, for Average error 3, rela
tive position of these shapes is also optimized by 2D/3D registra
tion proposed in Section 2.3, and Average error 3 becomes
smaller than Average error 2.
3.2. Experiments using the phantom femur
We conducted experiments using a dry femur bone and fluoro
scopic images. In these experiments, a special fluoroscopic imaging
apparatus (Siemens, Siremobil ISOC) was used for the fluoroscopic
photography from various directions around the phantom femur.
First, we captured images of calibration markers with nine glass
bubbles (left of Fig. 10) at 50 positions around the markers from 0
to 190? using the fluoroscopic apparatus. The 3D positions of the
markers were also measured precisely by the CT scanner. Next,
the intrinsic and extrinsic parameters of the fluoroscopic apparatus
were calibrated by Tsai’s method [35].
After calibration, we replaced the markers with the dry femur
bone and captured 50 images at the same positions. In addition,
the precise 3D shape of the dry bone was measured by the CT scan
ner. Next, we chose two fluoroscopic images from 50 images, as
mentioned below, and estimated the position and the optimum
parameters in fluoroscopic images using the proposed techniques.
Examples of the fluoroscopic images are shown in Fig. 11.
Fig. 12 shows one example of the pair (No. 4 and 24) of fluoro
scopic images, which were captured from directions crossing at
right angles. The average errors of the estimated femoral shape
are shown in Fig. 13 and Table 2 for various numbers of shape
parameters used for the estimation. The estimation process and
the estimated 3D shape when the number of the estimated shape
parameters is 10 are shown in Figs. 14 and 15. The calculation time
by a 3.2GHz Pentium IV, is approximately one minute, including
the contour detection by the Level Set Method and the shape
parameter estimation.
Finally, the average errors for various pairs of fluoroscopic
images are shown in Fig. 16 when the number of the estimated
shape parameters is 10. As a result of a series of experiments using
the phantom femur, we concluded that the 3D shape can be esti
mated with an average error of less than 1.2 mm if we choose
the proper images captured from directions that intersect approx
imately at right angles.
3.3. In vivo experiments
Next, we conducted in vivo experiments for hip prosthesis pa
tients. Fluoroscopic images of four patients were taken in clinical
practice using the fluoroscopic imaging apparatus (Siemens, Sire
Table 1
Comparison of average errors (mm)
AverageSTD. Maximum Minimum
Average error 1
Average error 2
Average error 3
(Test data)
(Model data)
1.69
0.90
0.54
0.13
2.52
1.06
0.90
0.60
0.91
0.81
0.15
0.07
1.08
0.90
0.66
0.71
Calibration markers
Femoral dry bone
Fig. 10. Fluoroscopic images of the calibration markers and the phantom femur.
No.1
0 deg.
No.25
95 deg.
(approx.)
No.50
190 deg.
FemurMarkers
Fig. 11. Measured fluoroscopic images.
No.4No.24
Fig. 12. Examples of fluoroscopic images (Nos. 4 and 24).
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
207
Page 7
mobil ISOC), and shapes measured by the CT scanner and esti
mated by parameter estimation were compared.
3.3.1. Calibration procedure
First, we measured the internal and external parameters of the
fluoroscopic apparatus. The internal parameters consists of five
parameters, that is, focus length, 1st order radial lens distortion,
center pixel x and y, and scale factor. On the other hand, the exter
nal parameters contain six parameters, that is, the position and ori
entation of a radiation source in 3D space. After the calibration
process, we can get correct fluoroscopic images of known scale
without lens distortion, and the relative positions of two view
points. These parameters are all requisite for the proposed shape
estimation procedure. A noncoplanar marker and a coplanar mar
ker, shown in Fig. 17, are used for the calibration of external and
internal parameters, respectively. These markers are constructed
of acrylonitrile butadiene styrene (ABS), which indicated the high
est transmission of Xrays in preliminary experimentation. The
noncoplanar marker contains nine small stainless steel spheres
and the coplanar marker contains 16 small stainless steel disks.
The calibration procedure using these markers is as follows:
Step 1. The noncoplanar marker is captured by Xray CT and the
relative positions of the stainless steel spheres are measured.
Step 2. A fluoroscopic image of the noncoplanar marker is cap
tured, as shown in Fig. 18, and the internal parameters of
the fluoroscopic imaging apparatus are estimated by Tsai’s
method.
Step 3. The coplanar marker is placed under the patient’s hip and
two fluoroscopic images of the femur are captured from
two directions, as shown in Figs. 19 and 20.
0
0.5
1
1.5
2
2.5
123456789 10
Average error [mm]
Number of parameters
3D CT image
(Average error 2)
two radiographs
(Average error 3, Proposed method)
AS
AS : Average shape
Fig. 13. Comparison of estimation errors using CT image and two radiographs.
Table 2
Average of estimation errors of the phantom femur (mm)
Number of parameters AverageSTD.Max.
0
1
2
3
4
5
6
7
8
9
10
1.544
1.568
1.344
1.103
1.068
1.020
0.986
1.009
0.929
0.931
0.889
1.009
1.021
1.111
0.868
0.869
0.814
0.817
0.785
0.743
0.738
0.709
5.381
5.486
6.653
4.820
4.910
4.776
4.770
4.174
3.813
3.709
3.758
Fig. 14. Process of shape and position estimation of the femur.
Fig. 15. Actual femoral shape and estimated femoral model with and without shape
parameter estimation.
Image No.
24
4143444
4
14
24
34
44
1.246
[mm]
0.992 1.128
1.0500.980
1.143
1.504
1.091
1.223
1.377
( 15 deg.)
∼
∼
( 53 deg.)
∼
( 91 deg.)
∼
( 129 deg.)
∼
( 167 deg.)







Fig. 16. Estimation errors for various pairs of fluoroscopic images.
ABS
Stainless steel ball
90 mm
Noncoplanar marker for
external parameter calibration
ABS
150 mm
Stainless steel disk
50 mm
Coplanar marker for
internal parameter calibration
Fig. 17. Noncoplanar and coplanar markers.
208
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
Page 8
Step 4. The external parameters of two fluoroscopic images are
estimated using the internal parameters obtained in Step
2 and the projection image of stainless steel disks by Tsai’s
method. After this calibration process, the relative posi
tions of two viewpoints of fluoroscopic images are precisely
determined.
3.3.2. Results
First, we manually extracted the contour of the femur (Fig. 21)
in the fluoroscopic images (Fig. 20). Image size and resolution are
512 ? 512 pixels and 0.454 mm pixel. We then estimated the
position and 10 shape parameters from the silhouette of the femur.
The precise 3D shapes of patients’ femurs were measured precisely
by CT scanner.
The average errors and standard deviations of the estimated
femoral shapes are shown in Fig. 22. In these figures, ‘‘A” in the
horizontal axis shows the case in which only the position is esti
mated without parameter estimation.
The experimental results show that the average error between
the estimated shape and the actual shape is approximately from
0.8 to 1.1 mm for the in vivo experiments. One example of average,
actual, and estimated shapes for Case 4 is shown in Fig. 23, and the
distribution of average error is shown in Fig. 24. In Fig. 24, dark re
gions indicate less error, and the brightness of each point is propor
tional to its average error. From this figure, we verified that the
errors in the femoral head and lesser trochanter are reduced.
Fig. 18. Example fluoroscopic image of noncoplanar marker.
Xray apparatus
Coplanar marker
Fig. 19. Xray photography of coplanar marker.
Fig. 20. Two fluoroscopic femoral images of hip prosthesis patients.
Fig. 21. Extracted contours of the femur in fluoroscopic images.
0
AS
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
12
345678910 1112
Number of parameters
AS
12
3456789101112
Number of parameters
Average error [mm]
case1
case2
case3
case4
AS : Average shape
0
0.2
0.4
0.6
0.8
1.0
1.2
S.D. [mm]
case1
case2
case3
case4
AS : Average shape
Fig. 22. Average error and standard deviation for the number of estimated shape
parameters for the femurs of four patients.
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
209
Page 9
4. Conclusions
We proposed a method by which to estimate the 3D shape of
the in vivo femur from only two fluoroscopic images using a para
metric femoral model. Although the precise 3D shape of the femur
is usually measured using a CT scanner or an MRI system, the pro
posed method enables a precise 3D shape to be estimated using
only two fluoroscopic images taken by an inexpensive fluoroscopic
inspection apparatus. Thus, the cost of the inspection system can
be dramatically reduced and the 3D imagebased medical diagno
sis becomes available even in small clinics.
In vivo experiments revealed the average error between the
estimated shape and the actual shape to be 0.8–1.1 mm, and it
was verified that the 3D shape can be estimated using two 2D fluo
roscopic images taken from different view points with the same
accuracy as in the case of 3D shapes being compared directly.
In the future, optimum conditions, such as the optimum num
ber and directions of the fluoroscopic images, will be investigated,
and clinical experiments in fluoroscopic image diagnosis will be
performed.
Appendix A
A.1. Fast extraction of the projected contour line of the 3D parametric
femoral model
In Step (4) in Section 2.3.1, the contour detection and identifica
tion triangular patches on the 3D model corresponding to points on
the contour line are computationally expensive and time consum
ing. In our implementation, we utilize the highspeed rendering
function of the OpenGL hardware accelerator, and thus these pro
cedures are executed quite rapidly.
The detailed algorithm is as follows. Initially, we assigned dif
ferent colors to all of the triangular patches in the 3D model and
draw the projected image of the 3D model on the image buffer
using the OpenGL hardware accelerator. The contour points of
the 3D model are detected by raster scanning of the image buffer.
By reading the colors of the detected contour points, we can iden
tify the corresponding triangular patches on the 3D geometric
model.
References
[1] I. Stamos, P.K. Allen, Integration of range and image sensing for photorealistic
3D modeling, in: Proceedings of the 2000 IEEE International Conference on
Robotics and Automation, 2000, pp. 1435–1440.
[2] I. Stamos, P.K. Allen, Automatic registration of 2D with 3D imagery in urban
environments, in: Proceedings of the International Conference on Computer
Vision, 2001, pp. 731–737.
[3] L. Liu, I. Stamos, Automatic 3D to 2D registration for the photorealistic
rendering of urban scenes, in: IEEE International Conference on Robotics and
Automation, 2005.
[4] R. Kurazume, K. Noshino, Z. Zhang, K. Ikeuchi, Simultaneous 2D images and 3D
geometric model registration for texture mapping utilizing reflectance
attribute, in: Proceedings of Fifth Asian Conference on Computer Vision
(ACCV), 2002, pp. 99–106.
[5] M.D. Elstrom, P.W. Smith, Stereobased registration of multisensor imagery
for enhanced visualization of remote environments, in: Proceedings of the
1999 IEEE International Conference on Robotics and Automation, 1999, 1948–
1953.
[6] K. Umeda, G. Godin, M. Rioux, Registration of range and color images using
gradientconstraintsandrangeintensity
Seventeenth International Conference on Pattern Recognition, 2004, 12–15.
[7] Y. Iwashita, R. Kurazume, K. Hara, T. Hasegawa, Fast alignment of 3D
geometrical models and 2D color images using 2D distance maps, in:
Proceedings The Fifth International Conference on 3D Digital Imaging and
Modeling (3DIM), 2005, pp. 164–171.
[8] Q. Delamarre, O. Faugeras, 3D articulated models and multiview tracking with
silhouettes, in: Proceedings of the International Conference on Computer
Vision, vol. 2, 1999, pp. 716–721.
[9] K. Matsushita, T. Kaneko, Efficient and handy texture mapping on 3D surfaces,
In Comput. Graphics Forum 18 (1999) 349–358.
[10] P.J. Neugebauer, K. Klein, Texturing 3D models of real world objects from
multiple unregistered photographic views, In Comput. Graphics Forum. 18
(1999) 245–256.
[11] G. Penny, J. Weese, J. Little, P. Desmedt, D. Hill, D. Hawkes, A comparison of
similarity measures for use in 2D/3D medical image registration, IEEE Trans.
Med. Imaging 17 (4) (1998) 586–595.
[12] L. Zollei, E. Grimson, A. Norbash, W. Wells, 2D–3D rigid registration of Xray
fluoroscopy and CT images using mutual information and sparsely sampled
histogram estimators, in: Proceedings of Computer Vision and Pattern
Recognition, 2001, pp. 696–703.
[13] C.V. Stewart, C.L. Tsai, A. Perera, A viewbased approach to registration:
theory and application to vascular image regisrtaion, in: International
Conference on Information Processing in Medical Imaging (IPMI), 2003, pp.
475–486.
[14] A. Guimond, A. Roche, M. Ayache, J. Meunier, Threedimensional multimodal
brain warping using the demons algorithm and adaptive intensity corrections,
IEEE Trans. Med. Imaging 20 (1) (2001) 58–69.
images,in:Proceedingsof
Fig. 23. Actual and estimated femoral model with and without shape parameter
estimation for Case 4.
Fig. 24. Error distribution indicated by brightness (higher brightness indicates
larger error.
210
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
Page 10
[15] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, P. Suetens, Multimodality
image registration by maximization of mutual infomation, IEEE Trans. Med.
Imaging 16 (2) (1997) 187–198.
[16] C.V. Stewart, C.L. Tsai, A. Perera, Rigid and affine registration of smooth
surfaces using differential properties, in: Proceedings of Third European
Conference on Computer Vision (ECCV’94), 1994, pp. 397–406.
[17] C.R. Meyer, J.L. Boes, B. Kim, P.H. Bland, K.R. Zasadny, P.V. Kison, K. Korak, K.A.
Fery, R.L. Wahl, Demonstration of accuracy and clinical versatility of mutual
information for automatic multimodality image fusion using affine and thin
plate spline warped geometric deformations, Med. Image Anal. 1 (3) (1997)
195–206.
[18] A. Gueziec, X. Pennec, N. Ayache, Medical image registration using geometric
hashing, IEEE Comput. Sci. Eng. Spec. Issue Geometric Hashing 4 (4) (1997) 29–
41.
[19] P.R. Andresen, M. Nielsen, Nonrigid registration by geometry constrained
diffusion, Med. Image Comput. Comput. Assist. Interv. (MICCAI’ 99) (1999)
533–543.
[20] T. Masuda, Y. Hirota, K. Ikeuchi, K. Nishino, Simultaneous determination of
registration and deformation parameters among 3D range images, in: Fifth
International Conference on 3D Digital Imaging and Modeling, 2005, 369–
376.
[21] C.S.K. Chan, D.C. Barratt, P.J. Edwards, G.P. Penney, M. Slomczykowski, T.J.
Charter, D.J. Hawkes, Cadaver validation of the use of ultrasound for 3D model
instantiation of bony anatomy in image guided orthopaedic surgery, in:
Lecture Notes in Computer Science, 3217 (Proceedings Seventh International
Conference on Medical Image Computing and Computer Assisted Intervention,
Part II (MICCAI 2004), St.Malo, France), 2004, pp. 397–404.
[22] D. Terzopoulos, D. Metaxas, Dynamic 3D models with local and global
deformations: deformable superquadrics, IEEE Trans. Pattern Anal. Mach.
Intell. 13 (7) (1991) 703–714.
[23] K. Nakamura, R. Kurazume, T. Okada, Y. Sato, N. Sugano, T. Hasegawa, 3D
reconstruction of a femoral shape using a parametric model and two 2D
radiographs, in:Proceedings of
Understanding, 2006, pp. 78–83.
[24] G. Zheng, M. Ballester, M. Styner, L. Nolte, Reconstruction of patientspecific
3D bone surface from 2D calibrated fluoroscopic images and point distribution
MeetingonImage Recognitionand
model, in: Proceedings of Medical Image Computing and ComputerAssisted
Intervention (MICCAI’06), 2006, pp. 25–32.
[25] Keita Yokota, Toshiyuki Okada, Masahiko Nakamoto, Yoshinobu Sato,
MasatoshiHori, J. Masumoto,Hironobu
Construction of conditional statistical atlases of the liver based on spatial
normalization using surrounding structures, Int. J. Comput. Assist. Radiol. Surg.
(2006) 39–40.
[26] Toshiyuki Okada, Ryuji Shimada, Yoshinobu Sato, Masatoshi Hori, Keita
Yokota, Masahiko Nakamoto, YenWei Chen, Hironobu Nakamura, Shinichi
Tamura, Automated segmentation of the liver from 3D CT images using
probabilistic atlas and multilevel statistical shape model, in: Proceedings of
the Tenth International Conference on Medical Image Computing and
Computer Assisted Intervention (Proc. MICCAI 2007), 2007, pp. 86–93.
[27] T.F. Cootes TF, C.J. Cooper, C.J. Taylor, J. Graham, Active shape models—their
training and application, Comput. Vis. Image Understanding 61 (1) (1995) 38–
59.
[28] T. Okada, M. Nakamoto, Y. Sato, N. Sugano, H. Yoshikawa, S. Tamura, T. Asaka,
Y.W. Chen, Effects of surface correspondence methods in statistical shape
modelling of the proximal femur on approximation accuracy, in: Proceedings
of the Twenth International Congress and Exhibition Computer Assisted
Radiology and Surgery CARS, 2006, p. 016.
[29] Haili Chui, Anand Rangarajan, A new point matching algorithm for nonrigid
registration, Comput. Vis. Image Understanding 89 (23) (2003) 114–141.
[30] M. Greenspan, M. Yurick, Approximate kd tree search for efficient ICP, in:
Proceedings of 3D Digital Imaging and Modeling 3DIM, 2003, pp. 442–448.
[31] J. Sethian, Level Set Methods and Fast Marching Methods, second ed.,
Cambridge University Press, UK, 1999.
[32] J. Sethian, A fast marching level set method for monotonically advancing fronts,
in: Proceedings of the National Academy of Science 93, 1996, pp. 1591–1595.
[33] G. Borgefors, Distance transformations in arbitrary dimensions, Comput. Vis.
Graphics Image Process. 27 (3) (1984) 321–345.
[34] S.W. Shih, Y.T. Wu, Fast Euclidean distance transformation in two scans using a
3 3 neighborhood, Comput. Vis. Image Understanding 93 (2) (2004) 195–205.
[35] R.Y. Tsai, An Efficient and Accurate Camera Calibration Technique for 3D
Machine Vision, in: Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition, 1986, pp. 364–374.
Nakamura, ShinichiTamura,
R. Kurazume et al./Computer Vision and Image Understanding 113 (2009) 202–211
211