Content uploaded by Peter Mountney
Author content
All content in this area was uploaded by Peter Mountney
Content may be subject to copyright.
G.-Z. Yang et al. (Eds.): MICCAI 2009, Part I, LNCS 5761, pp. 483–490, 2009.
© Springer-Verlag Berlin Heidelberg 2009
Optical Biopsy Mapping for Minimally Invasive Cancer
Screening
Peter Mountney1,2, Stamatia Giannarou2, Daniel Elson2,
and Guang-Zhong Yang1,2
1 Department of Computing
2 Institute of Biomedical Engineering Imperial College, London SW7 2BZ, UK
{peter.mountney,stamatia.giannarou03,ds.elson,
g.z.yang}@imperial.ac.uk
Abstract. The quest for providing tissue characterization and functional map-
ping during minimally invasive surgery (MIS) has motivated the development
of new surgical tools that extend the current functional capabilities of MIS.
Miniaturized optical probes can be inserted into the instrument channel of stan-
dard endoscopes to reveal tissue cellular and subcellular microstructures, allow-
ing excision-free optical biopsy. One of the limitations of such a point based
imaging and tissue characterization technique is the difficulty of tracking
probed sites in vivo. This prohibits large area surveillance and integrated func-
tional mapping. The purpose of this paper is to present an image-based tracking
framework by combining a semi model-based instrument tracking method with
vision-based simultaneous localization and mapping. This allows the mapping
of all spatio-temporally tracked biopsy sites, which can then be re-projected
back onto the endoscopic video to provide a live augmented view in vivo, thus
facilitating re-targeting and serial examination of potential lesions. The pro-
posed method has been validated on phantom data with known ground truth and
the accuracy derived demonstrates the strength and clinical value of the
technique. The method facilitates a move from the current point based optical
biopsy towards large area multi-scale image integration in a routine clinical
environment.
1 Introduction
With recent advances in biophotonics and surgical instrumentation, there is an in-
creasing demand to bring cellular and molecular imaging modalities to an in vivo, in
situ setting to allow for real-time tissue characterization, functional assessment and
intra-operative guidance. Miniaturization of the confocal laser scanning microscope,
for example, has led to imaging probes that can be inserted into the instrument chan-
nel of a standard endoscope to visualize cellular and subcellular microstructures to
provide ‘optical biopsy’ without excision of tissue. Following the application of a
contrast agent, this can allow for the detection of colorectal adenomas, disruption in
the pit pattern of the colon, angiogenesis, and neoplasia in Barrett’s esophagus [1]. It
has also been used without a contrast agent to detect malignant disruption of the bron-
chial basement membrane using elastin autofluorescence [2]. Other techniques that
enable microscopic detection and characterization of tissue include Optical Coherence
484 P. Mountney et al.
Tomography (OCT), two photon excited fluorescence and high magnification endo-
scopy [3]. There have also been successful clinical trials of techniques that acquire
detailed spectroscopic information for cancer detection, for example using the time-
or wavelength-resolved fluorescence or Raman properties.
For in vivo applications, all of these techniques suffer from the limitation of only
providing a small, localized probe region whilst the organs of interest may require a
large surface area to be surveyed. Technically, the main difficulty of tracking the
optical biopsy sites is that these probes leave no marks on the tissue. Furthermore,
the optical biopsy sites move in and out of the view in a standard endoscope image as
the examination progresses and may deform as a result of respiration or tissue-
instrument interaction. Current approaches to long-term tissue-instrument tracking
assume the use of rigid laparoscopes and availability of optical markers [4]. Structure
from motion has been used to reconstruct 3D tissue models, but it suffers from drift
and does not work well when revisiting biopsy sites [5]. For extending the effective
field-of-view of the endoscopic image, image mosaicing [6] and dynamic view ex-
pansion [7] have been used to reconstruct enlarged field-of-views, although these
techniques tend not to explicitly deal with motion parallax.
In practice, optical probes are typically introduced through the instrument channel
while holding the endoscope stationary. Since the probe needs to be placed in contact
with the tissue when the optical biopsy takes place, tracking the tip of the probe en-
ables the localization of the biopsy site. To this end, it is necessary to take into ac-
count scale, rotation and illumination changes when tracking the tool. Current ap-
proaches to needle and surgical instrument tracking may be applicable [8, 9], but a
combined approach by integrating probe tracking with a 3D probabilistic map built in
situ using only white light endoscopic images with no additional fiducials can ensure
robustness and practical clinical use. This work proposes an image-based tracking
system based on SLAM (Simultaneous Localization and Mapping) for optical probes.
This will allow for subsequent localization and contextual analysis of microstructures
or guiding real tissue biopsy. The main contribution of this paper is to combine
SLAM with probe tracking to create a 3D model of the tissue surface and spatio-
temporally tracked optical biopsy sites. These biopsy sites are subsequently re-
projected back onto the image plane to provide a live augmented view in vivo, thus
facilitating re-targeting and serial examination. The proposed method has been
Fig. 1. (a) A typical microconfocal fluorescence image showing the microstructure of a sample,
(b) the relative configuration of a confocal fluorescence probe when inserted through the in-
strument channel of a standard endoscope, and (c) a typical endoscopic white light image of the
bronchus used for navigation
Optical Biopsy Mapping for Minimally Invasive Cancer Screening 485
validated on phantom data with known ground truth. The method will facilitate a shift
from the current point based optical biopsy towards multi-scale image integration in a
routine clinical environment.
2 Methods
2.1 Probabilistic Mapping
The first step of the proposed tracking framework is to establish a probabilistic map-
ping of the environment. Previous work on SLAM based approaches has shown the
ability to generate 3D tissue models and recover the relative pose of the endoscope
[10]. A long-term map is generated making it more resilient to drift and error accumu-
lation over time, and thus is well suited to returning to previously targeted areas. In
this work, a vision based sequential approach has been used. This is based on an
Extended Kalman Filter (EKF) framework with state vector x containing the posi-
tion(, , )
xyz
ccc , orientation 123 4
(, , , )
qqq q
cccc , translational velocity (,, )
xyz
vvv and angu-
lar velocity (, , )
xyz
ωωω of the endoscope. In addition, the state vector also stores 3D
locations of salient features in the map(, , )
xyz
yyy . A constant velocity, constant angu-
lar velocity motion model is used to predict the endoscope’s motion with Gaussian
noise. Accompanying the state vector is the covariance matrix which stores the uncer-
tainty of the endoscope and feature locations in 3D. In this sequential map building
approach, new features are added to the map on the fly by feature matching con-
strained by epipolar geometry to estimate their 3D positions relative to the endoscopic
camera.
2.2 Biopsy Site Estimation
The initial position of the biopsy site in the image plane is estimated through probe
tracking. In this work, no marker was attached and no changes were made to the col-
our of the imaging probe. The technique exploits the fact that the camera is relatively
static when the biopsy is taken. The segmentation of the tool is achieved by combin-
ing background subtraction and color segmentation in the HSV space. In this study, a
simple background subtraction technique is used based on inter-frame difference.
Foreground/background models are learnt and updated over time. The background
model is initialized with the first frame of the video sequence. For the extraction of
foreground objects, the current frame is subtracted from the background model and
any significant difference is labeled as foreground. If no foreground object is identi-
fied, the current frame becomes the background model. On the saturation plane, the
shaft of the probe is highlighted in dark grey on a bright background. Therefore, fore-
ground pixels (Fig. 2 (a)) are used as seeds for region growing in the saturation color
plane (Fig. 2 (b)) to segment the probe shaft as shown in Fig. 2(c).
In order to identify the tip of the probe, the centroid of the shaft is extracted. The
tangentials of the shaft are detected at the global maxima of the Hough transform and
the axis of the shaft is computed as the eigenvector corresponding to the smallest
eigenvalue of the moment of inertia. The localization of the tip of the probe is per-
formed with respect to a reference point located at the intersection of the shaft and the
o
486 P. Mountney et al.
Fig. 2. Probe tracking and biopsy site estimation within the image plane; (a) background sub-
traction, (b) color saturation distribution within the image, (c) segmented tool regions, and (d)
the model fitted tool (centroid -red dot, the reference point - green dot, vanishing tangential
lines - cyan, radius at the center of mass and at the reference point – yellow).
distal tip. The 3D position of the tip is estimated using a semi-model based approach
assuming rigidity and incorporating prior knowledge of the width of the probe and the
relationship between the reference point and the tip. The position of the reference
point and the orientation of the shaft are estimated in 3D and the prior model enables
the localization of the tip of the probe c
b relative to the camera.
2.3 Global Biopsy Mapping
Following the steps in Section 2.2, the position of biopsy site c
b is estimated in the
camera coordinate system. It is transformed into the world coordinate system using:
wwcw
bCbc=+ (1)
where w
bis the biopsy site in the world coordinate system, w
Cand w
care the orienta-
tion and position of the camera in the global SLAM coordinate system. Although the
3D position of the biopsy site is now defined, this position is never directly observed
or measured again. There are two reasons for this; the actual site on the tissue is usu-
ally occluded by the probe when the biopsy is taken, and there may not be any salient
features at or around the biopsy site to be tracked. In this case, 2D tracking would fail.
However, the strength of the proposed probabilistic map is that the position of the
biopsy site can be updated without directly measuring it. This is made possible by the
co-variance matrix which models the uncertainty of all the biopsy positions. The ith
biopsy site w
i
bis inserted into the state vector and the co-variance matrix P is up-
dated. The co-variance matrix is updated with the partial derivatives /
iv
bx∂∂ of the
biopsy site with respect to the camera position, as well as the measurement model
/
ii
bh∂∂ and measurement noise Ras shown in Eq (2).
where xis the position of the endoscope and i
yis the ith feature in the map.
The position and uncertainty of the biopsy sites are correlated to the camera posi-
tion and the rest of the features in the map. Fig. 3 illustrates this sequential map build-
ing demonstrating how the camera, features and the biopsy sites are correlated and
w
i
x
by
z
⎛⎞
⎟
⎜⎟
⎜⎟
⎜⎟
⎜
=⎟
⎜⎟
⎜⎟
⎟⎜ ⎟
⎜⎟
⎜
⎝⎠
,
1
111 1
1
T
i
xx xy xx
v
T
i
yx yy yx
v
ww wwTwwT
ii iiiii
xx xy xx
vv vvii
b
PP P
x
b
PP P P
x
bb bbbb
PP P R
xx xxhh
⎡⎤
∂
⎢⎥
⎢⎥
∂
⎢⎥
⎢⎥
∂
⎢⎥
=⎢⎥
∂
⎢⎥
⎢⎥
∂∂ ∂∂∂∂
⎢⎥
+
⎢⎥
⎢⎥∂∂ ∂∂∂∂
⎣⎦
(2)
Optical Biopsy Mapping for Minimally Invasive Cancer Screening 487
Fig. 3. (a-d) Schematic representation of sequential probabilistic mapping updates. The cam-
era’s position c is shown in red with the uncertainty represented by an ellipse, features y1, y2
and y3 represented in dark gray, the biopsy site b shown in green and the tissue shown in light
gray. (a) c measures y1 with low uncertainty, (b) c is navigated to a new position with growing
uncertainty. Features y2 and y3 are measured and biopsy b is taken. (c) c is navigated close to
y1 and positional uncertainty increases. (d) Feature y1 is measured and the position estimate of
c is improved. Resulting in an improved estimate of b, as it is correlated to c.
temporally updated. At the time when the biopsy site is observed, the uncertainty of
the camera’s position may be high, as illustrated in Fig. 3 (b) but the relative position
of the biopsy site to surrounding features is well defined. Over time, the camera will
re-measure these surrounding features in the map as in Fig. 3 (d) and the position
estimation of the camera will improve, thus reducing the uncertainty. Therefore, the
position estimation of the biopsy site will also improve as it is correlated to the posi-
tion of the camera and will not drift away in the global map. To facilitate real-time
examination, the biopsy sites 1
{...}
ww
i
bb are visualized in this study by re-projecting
the 3D points into the camera plane based on the intrinsic camera parameters and the
estimated camera position from SLAM. This provides an augmented view of the bi-
opsy sites for the operator.
2.4 Experimental Set-Up
The proposed approach has been validated on a silicon phantom of the airway coated
with acrylic paint to provide realistic texture and internal reflections. Sponge cell
structures were attached to the internal surface to enable optical biopsies to be taken
using a confocal fluorescence endoscope system (Cellvizio, Mauna Kea Technolo-
gies, Paris). Validation was performed by measuring the accuracy of biopsy sites in
the image space as the endoscope navigated through the phantom. The ground truth
data used for comparison was collected using an optical tracking device (Northern
Digital Inc, Ontario, Canada) and an experienced observer. To obtain the ground truth
position of the camera, a rigid stereo laparoscope fitted with four optical markers was
used. The position l
c and orientation l
C of the center of the left camera relative to
the optical markers were acquired using standard hand-eye calibration [11]. This
enabled the position of the camera to be calculated in the world coordinate system w
c
and w
C.To obtain the ground truth of the 3D biopsy site positions, the experienced
observer manually identified the sites on the stereo images at the time when the bi-
opsy was taken. By using the camera’s intrinsic and extrinsic parameters, the 3D
position c
bof the biopsy site was obtained relative to the camera, and its position in
488 P. Mountney et al.
the world coordinate system w
b was determined as *
wwcw
bCbc=+. At each sub-
sequent frame, the biopsy site w
b was projected into the ground truth camera position
(/)
cc
oxxz
xx fkbb=− and (/)
cc
oyyz
yy fkbb=− where x
fk and y
fk are the focal length
and o
xand o
yare the principal point. To validate the proposed probe tracking ap-
proach, the probe was mounted in a rigid sheath. This evaluation step was combined
with manually defined image coordinates of the probe’s location.
3 Results
The proposed algorithm was validated on a two minute long stereo laparoscopic video
sequence consisting of navigation to four different areas, including six biopsies and
re-targeting previously taken biopsies.
Quantitative analysis of the position of the biopsy sites in the image plane is shown
in Table 1. The average visual angle error for the position of the biopsy sites ranges
from 1.18° to 3.86°. Figs. 4 (d-e) show the estimated biopsy site position and ground
truth position of site three over a short sequence before the site goes out of view.
Accuracy of the biopsy position estimation is affected by the proximity of the camera
to the site where close proximity leads to a magnification of the error. Fig. 5 illus-
trates the results of the augmented biopsy sites at different stages of the procedure
where changes in illumination, scale and view point are experienced. Fig. 5 demon-
strates the practical value and clinical relevance of the proposed method; the entire
procedure is represented where six biopsies are taken and added to the global map,
including the associated biopsy images of the sponge cell structures.
a) b) c)
d) e) f)
Fig. 4. (a-c) Probe tracking: Ground truth (red) and estimated (green) position of probe at (a)
site six and (b) site three. (c) Ground truth (red) and tracked probe position (green) during
navigation between biopsy sites. (d-f) Augmented biopsy site three: (d-e) the X and Y pro-
jected pixel error showing the site being tracked (f) the ground truth projected position (red)
and the estimated position (green) for a short section of the procedure.
Optical Biopsy Mapping for Minimally Invasive Cancer Screening 489
Table 1. Average error of biopsy site estimation and probe tracking for phantom experiments
Probe tracking Augmented biopsy sites
Biopsy sites Average visual
angle error
Percent of
FOV
Visual angle
error
Percent of
FOV
1 1.85° 4.89% 2.34° 5.37%
2 1.33° 3.59% 3.06° 7.58%
3 0.87° 2.29% 2.22° 5.59%
4 0.86° 2.23% 1.18° 2.99%
5 0.81° 1.75% 2.06° 4.61%
6 3.22° 8.88% 3.86° 10.09%
Fig. 5. (a-d) Biopsy site position (green spheres). The spheres are 2mm in diameter and appear
in different sizes when they are projected onto the image under perspective projection; (e)
shows the six biopsy sites with corresponding micro-confocal fluorescence endoscope images.
Detailed quantitative analysis of the probe tracking when the biopsies are taken is
shown in Table 1. The tracking errors range from 0.81° to 3.22° of the visual angle
and an example error distribution is illustrated in Fig. 4 (a-b). Quantitative analysis of
the probe tracking on the whole sequence gave an average visual angle error of 2.87°.
The sensitivity and specificity were 0.9706 and 0.9892, respectively. As expected, the
accuracy deteriorates when the probe is introduced and removed from the scene as a
part of the shaft is occluded, or when the probe is very close to the camera.
4 Conclusion
In this paper we have proposed a novel approach for microconfocal optical biopsy
tracking which can be used to augment intra-operative navigation and retargeting of
previously examined tissue regions. The system has been validated with a detailed
490 P. Mountney et al.
phantom experiment and we have demonstrated that this approach can accurately
project the location of biopsy sites, thus enabling its practical clinical use. The pro-
posed method requires no prior information of the tissue geometry and can operate
consistently in a sparse feature environment. The proposed method is robust to small
local deformation and rigid global motion. Modeling large scale nonlinear tissue
deformation, however, is not trivial and will be addressed in future work.
Acknowledgments. We gratefully acknowledge support from the EPSRC and the
Technology Strategy Board grants DT/F003064/1 and DT/E011101/1.
References
1. Meining, A., Bajbouj, M., von Delius, S., Prinz, C.: Confocal Laser Scanning Microscopy
for in vivo Histopathology of the Gastrointestinal Tract. Arab Journal of Gastroenterol-
ogy 8, 1–4 (2007)
2. Thiberville, L., Moreno-Swirc, S., Vercauteren, T., Peltier, E., Cavé, C., Bourg Heckly, G.:
In Vivo Imaging of the Bronchial Wall Microstructure Using Fibered Confocal Fluores-
cence Microscopy. American Journal of Respiratory and Critical Care Medicine 175, 22–
31 (2007)
3. Van Dam, J.: Novel methods of enhanced endoscopic imaging. GUT 52(4) (2003)
4. Wengert, C., Cattin, P.C., Duff, J.M., Székely, G.: Markerless Endoscopic Registration and
Referencing. In: Larsen, R., Nielsen, M., Sporring, J. (eds.) MICCAI 2006. LNCS,
vol. 4190, pp. 816–823. Springer, Heidelberg (2006)
5. Wu, C.-H., Sun, Y.-N., Chen, Y.-C., Chang, C.-C.: Endoscopic Feature Tracking and
Scale-Invariant Estimation of Soft-Tissue Structures. IEICE Transactions on Information
and Systems 2, 351–360 (2008)
6. Atasoy, S., Noonan, D.P., Benhimane, S., Navab, N., Yang, G.-Z.: A global approach for
automatic fibroscopic video mosaicing in minimally invasive diagnosis. In: Metaxas, D.,
Axel, L., Fichtinger, G., Székely, G. (eds.) MICCAI 2008, Part I. LNCS, vol. 5241, pp.
850–857. Springer, Heidelberg (2008)
7. Lerotic, M., Chung, A.J., Clark, J., Valibeik, S., Yang, G.-Z.: Dynamic View Expansion
for Enhanced Navigation in Natural Orifice Transluminal Endoscopic Surgery. In:
Metaxas, D., Axel, L., Fichtinger, G., Székely, G. (eds.) MICCAI 2008, Part II. LNCS,
vol. 5242, pp. 467–475. Springer, Heidelberg (2008)
8. Wengert, C., Bossard, L., Häberling, A., Baur, C., Székely, G., Cattin, P.C.: Endoscopic
Navigation for Minimally Invasive Suturing. In: Ayache, N., Ourselin, S., Maeder, A.
(eds.) MICCAI 2007, Part II. LNCS, vol. 4792, pp. 620–627. Springer, Heidelberg (2007)
9. Krupa, A., Gangloff, J., Doignon, C., de Mathelin, M.F., Morel, G., Leroy, J., Soler, L.,
Marescaux, J.: Autonomous 3-D positioning of surgical instruments in robotized laparo-
scopic surgery using visual servoing. IEEE Transactions on Robotics and Automation 19,
842–853 (2003)
10. Mountney, P., Stoyanov, D., Davison, A.J., Yang, G.-Z.: Simultaneous Stereoscope Local-
ization and Soft-Tissue Mapping for Minimal Invasive Surgery. In: Larsen, R., Nielsen,
M., Sporring, J. (eds.) MICCAI 2006. LNCS, vol. 4190, pp. 347–354. Springer, Heidel-
berg (2007)
11. Tsai, R., Lenz, R.: Real Time Versatile Robotic Hand/Eye Calibration using 3D Machine
Vision. In: Proc. ICRA 1988, pp. 554–561 (1988)
