Realtime Organ Tracking for Endoscopic Augmented Reality Visualization Using Miniature Wireless Magnetic Tracker.
ABSTRACT Organ motion is one of the problems on augmented reality (AR) visualization for endoscopic surgical navigation system. However,
the conventional optical and magnetic trackers are not suitable for tracking of internal organ motion. Recently, a wireless
magnetic tracker, which is called the Calypso 4-D localization system has been developed. Since the sensor of the Calypso
system is miniature and implantable, position of the internal organ can be measured directly. This paper describes AR system
using the Calypso system and preliminary experiments to evaluate the AR system. We evaluated distortion error caused by the
surgical instruments and misalignment error of superimposition. Results of the experiments shows potential feasibility and
usefulness of AR visualization of moving organ using the Calypso system.
- SourceAvailable from: Xiongbiao Luo[Show abstract] [Hide abstract]
ABSTRACT: PURPOSE : Flexible fiber-optic bronchoscopy is a widespread medical procedure for the diagnosis and treatment of lung diseases. Navigation systems are needed to track the flexible endoscope within the bronchial tree. Electromagnetic (EM) tracking is currently the only technology used clinically for this purpose. The registration between EM tracking and patient anatomy may become inaccurate due to breathing motion, so the addition of image-based tracking has been proposed as a hybrid EM-image-based system. METHODS : When EM tracking is used as an initialization for image registration, small changes in the initialization may lead to different local minima and noise is amplified by hybrid tracking. The tracking output is modeled as continuous and uses splines for interpolation, thus smoothness is greatly improved. The bronchoscope pose relative to computed tomography data is interpolated using Catmull-Rom splines for position and spherical linear interpolation (SLERP) for orientation. RESULTS : The hybrid method was evaluated using ground truth poses manually selected by experts, where mean inter-expert agreement was determined as 1.26 mm. Using four dynamic phantom data sets, the accuracy was 4.91 mm, which is equivalent to previous methods. Compared to state-of-art methods, inter-frame smoothness was improved from 2.77-3.72 to 1.24 mm. CONCLUSIONS : Hybrid image and electromagnetic endoscope guidance provides a more realistic and physically plausible solution with significantly less jitter. This quantitative result is confirmed by visual comparison of real and virtual video, where the virtual video output is much more consistent and robust, with fewer occasions of tracking loss or unexpected movement compared with previous methods.International Journal of Computer Assisted Radiology and Surgery 03/2013; · 1.36 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: Augmented reality (AR), a useful visualization technique, is reviewed based literatures. The AR research methods and applications are surveyed since AR was first developed over forty years ago. Recent and future AR researches are proposed which could help researchers of decide which topics should be developed when they are beginning their own researches in the field.10/2009: pages 311-337;
- [Show abstract] [Hide abstract]
ABSTRACT: We present an image-guided intervention system based on tracked 3D elasticity imaging (EI) to provide a novel interventional modality for registration with pre-operative CT. The system can be integrated in both laparoscopic and robotic partial nephrectomies scenarios, where this new use of EI makes exact intra-operative execution of pre-operative planning possible. Quick acquisition and registration of 3D-B-Mode and 3D-EI volume data allows intra-operative registration with CT and thus with pre-defined target and critical regions (e.g. tumors and vasculature). Their real-time location information is then overlaid onto a tracked endoscopic video stream to help the surgeon avoid vessel damage and still completely resect tumors including safety boundaries. The presented system promises to increase the success rate for partial nephrectomies and potentially for a wide range of other laparoscopic and robotic soft tissue interventions. This is enabled by the three components of robust real-time elastography, fast 3D-EI/CT registration, and intra-operative tracking. With high quality, robust strain imaging (through a combination of parallelized 2D-EI, optimal frame pair selection, and optimized palpation motions), kidney tumors that were previously unregistrable or sometimes even considered isoechoic with conventional B-mode ultrasound can now be imaged reliably in interventional settings. Furthermore, this allows the transformation of planning CT data of kidney ROIs to the intra-operative setting with a markerless mutual-information-based registration, using EM sensors for intraoperative motion tracking. Overall, we present a complete procedure and its development, including new phantom models - both ex vivo and synthetic - to validate image-guided technology and training, tracked elasticity imaging, real-time EI frame selection, registration of CT with EI, and finally a real-time, distributed software architecture. Together, the system allows the surgeon to concentrate on intervention completion with less time pressure.Proc SPIE 03/2010;
Realtime Organ Tracking for Endoscopic
Augmented Reality Visualization Using
Miniature Wireless Magnetic Tracker
Masahiko Nakamoto1, Osamu Ukimura2, Inderbir S. Gill3, Arul Mahadevan3,
Tsuneharu Miki2, Makoto Hashizume4, and Yoshinobu Sato1
1Division of Image Analysis, Graduate School of Medicine, Osaka University, Japan
2Department of Urology, Kyoto Prefectural University of Medicine, Kyoto, Japan
3Cleveland Clinic, Cleveland, OH, USA
4Graduate School of Medical Sciences, Kyushu University, Japan
Abstract. Organ motion is one of the problems on augmented reality
(AR) visualization for endoscopic surgical navigation system. However,
the conventional optical and magnetic trackers are not suitable for track-
ing of internal organ motion. Recently, a wireless magnetic tracker, which
is called the Calypso 4-D localization system has been developed. Since
the sensor of the Calypso system is miniature and implantable, position
of the internal organ can be measured directly. This paper describes AR
system using the Calypso system and preliminary experiments to evalu-
ate the AR system. We evaluated distortion error caused by the surgical
instruments and misalignment error of superimposition. Results of the
experiments shows potential feasibility and usefulness of AR visualiza-
tion of moving organ using the Calypso system.
Laparoscopic partial nephrectomy is a minimally invasive technique to remove
a renal tumor. In order to remove the tumor completely and preserve healthy
tissues as much as possible, augmented reality (AR) visualization of the CT im-
age where the resection line is depicted would be useful because the boundary
between healthy tissues and the tumor is not clear by endoscopic observation .
However, the surgeon holds up the kidney and changes its direction during resec-
tion of the tumor to confirm the resection line from several viewpoints. Therefore,
realtime tracking of the kidney motion is required to superimpose the CT image
onto live endoscopic images accurately.
Magnetic trackers are suitable to track objects inside the body. Conventional
wired magnetic trackers have been used to track flexible tubular instruments like
a catheter or a bronchoscope . However, they have not been employed to
track organ motion because it is necessary to pass the wire through the body.
Recently, a wireless magnetic tracker, which is called the Calypso 4-D localiza-
tion system (Calypso Medical Technologies, Inc., Seattle, WA, USA), has been
developed and applied to tracking of the prostate motion during the radiation
T. Dohi, I. Sakuma, and H. Liao (Eds.): MIAR 2008, LNCS 5128, pp. 359–366, 2008.
c ? Springer-Verlag Berlin Heidelberg 2008
360M. Nakamoto et al.
therapy . Three miniature magnetic sensors, which are called “transpon-
ders”, are implanted into the prostate beforehand, and then their 3D positions
are measured in realtime. Although the measurement and gating methods for
internal organ motion caused by respiration were proposed , these
methods assume periodical and regular motion. Since organ motion caused by
surgical operation is not periodical and regular unlike the organ motion caused
by respiration, the Calypso system is suitable for this purpose.
Our objective is to evaluate feasibility of endoscopic AR system using the
Calypso system. We evaluate the Calypso system from two viewpoints: (1) Dis-
tortion error caused by metallic surgical instruments. (2) Accuracy of superimpo-
sition. The conventional magnetic trackers suffer from magnetic field distortion
caused by metallic objects. Since the kidney is held by metallic surgical instru-
ments during surgical operation, effects of metallic surgical instruments would
not be negligible. Therefore, magnitude of the distortion error is an important
factor for clinical application of the Calypso system. We also demonstrate su-
perimposition of the CT image onto live endoscopic images, and thus accuracy
and feasibility of the endoscopic AR system are evaluated.
We demonstrated AR visualization for laparoscopic partial nephrectomy
(Fig. 1) . In order to assist determination of the resection line, the resec-
tion margin was visualized by colored areas. The kidney is divided to tumor,
resection margin, resection and healthy tissue areas, and these areas are colored
with red, yellow, green and blue (Fig. 1 left). The width of the resection margin
and resection areas is 5 mm. If the surgeon resects the tumor along the green
resection area, the resection margin of 5 mm or more can be kept. In order to
maintain accurate superimposition, the kidney needed to keep staying at the
same position and orientation as when the registration was performed because
the kidney was not tracked. Since the kidney is moved by surgical operation
practically, realtime tracking of the kidney is required.
Fig.1. Augmented reality visualization for laparoscopic partial nephrectomy. Left: 3D
kidney model reconstructed from CT image. Middle: original laparoscopic image. Right:
Realtime Organ Tracking for Endoscopic Augmented Reality Visualization 361
The AR system consists of the Calypso 4D Localization System, Polaris (North-
ern Digital Inc., Waterloo, Ontario, Canada), laparoscope (OTV-S5, Olympus,
Japan) and PC. The Calypso system consists of the miniature implantable elec-
tromagnetic transponder, which is 8.5 mm in length (Fig. 2(a)), 4D electromag-
netic array, which is a source of AC electromagnetic energy (Fig. 2(b)), and
console. The Calypso system measures the position of the transponder in 3.3
frames per second. Measurement volume is 150 × 150 × 270 mm. Bias error
and reproducibility are less than 0.4 mm and 0.6 mm, respectively . Three
transponders are attached to the organ to track position and orientation of a
target organ. The Polaris is also employed to track the laparoscope.
Integration of the coordinate systems of both trackers and registration of
the CT image are required to render superimposed images. Firstly, the matrix
TMT→OT, which transforms the coordinate system from the Calypso system
frame to the Polaris frame, is obtained by point matching algorithm described
as the following:
TMT→OT = argmin
where riand piare positions of the transponder measured by the Calypso sys-
tem and the Polaris, respectively. Secondly, the registration matrix between the
CT and Polaris frames at time t, TIMG→OT,t, is obtained by point matching
algorithm described as the following:
where S is a diagonal matrix of which diagonal factors represent the voxel sizes
of the CT image and qi is a position of the transponder in the CT image.
(a) Transponder is around
8.5 mm in length.
(b) Appearance of Calypso system.
Fig.2. Calypso 4D localization system
362M. Nakamoto et al.
Distance from transponder (mm)
Fig.3. Distortion error caused by surgical instruments
Finally, arbitrary position qt = (x,y,z,1)tin the CT image is projected to
t= (sx?,sy?,s,1)tin the laparoscopic image by the following equation:
where A and Text are intrinsic and extrinsic camera parameters obtained by
camera calibration, and Tcam,tis a rigid transformation matrix measured by the
Because the Calypso system is not connected to the PC in the current system,
position data measured by the Calypso system is imported to the PC offline and
then the superimposed images are rendered after the acquisition of positions and
3.2Experimental Conditions and Methods
We evaluated effects of metallic surgical instruments on the position measured
by the Calypso system. We employed the laparoscope, forceps (CLICKline Bowel
grasper, KARL Storz, Germany) and scissors (CLICKline Sciccors, KARL Storz,
Germany), which were commonly used in laparoscopic surgery, as surgical instru-
ments. The transponder and the surgical instrument were fixed on the operating
table which was made of non-metallic materials. The position of the transpon-
der was measured around 50 times while changing the distance between the
transponder and the instrument from 0 to 150 mm. We performed the mea-
surements for each instrument and performed them for all three instruments.
Powers of the laparoscope and the light source were turned off except when
the distance was 0 mm. The powers of them were turned on and off when the
Realtime Organ Tracking for Endoscopic Augmented Reality Visualization363
(a) Cow kidney with transponders.
(b) Arrangement of instruments.
Fig.4. Experimental setup for augmented reality visualization
distance was 0 mm. Distortion error caused by the surgical instruments is defined
as the difference of positions between with and without the surgical instruments.
We employed a cow kidney to demonstrate AR visualization. In order to
simulate laparoscopic partial nephrectomy, a pseudo tumor was embedded in
the kidney, and three transponders were attached around the pseudo tumor
(Fig. 4(a)). The CT image of the kidney was acquired, and then the kidney and
the pseudo tumor were segmented. The kidney and the laparoscope with optical
markers were fixed on the operating table (Fig. 4(b)). A part of the kidney was
grasped with the forceps and moved up and down, and then the pseudo tumor
was resected with the scissors and lifted up and down. The above operation
was divided into three sequences, and then superimposed images were rendered
for each sequence. Misalignment error of superimposition at time t is defined
transforming the position of the transponder in the CT image using equation (3),
and vi,tis a 2D position of the transponder in the laparoscopic image obtained
by manual pointing.
i=3|ui,t− vi,t|, where ui,tis a 2D position of the transponder given by
4.1 Distortion Error Caused by Surgical Instruments
Distortion error caused by the surgical instruments was around 1.0 mm when
the distance between the transponder and the instrument was 0 mm, and 0.4
mm or less when the distance was 20 mm or more as shown in Fig. 3. The error
when the powers were up was almost same as the error when the powers were
364M. Nakamoto et al.
Fig.5. Augmented reality visualization (third sequence). Upper row: original laparo-
scopic images. Lower row: superimposed images.
4.2Accuracy of Superimposition
Superimposed images of the third sequence are shown in Fig. 5. Kidney tis-
sue, pseudo tumor and transponder are depicted as blue, yellow and red part,
respectively. Delay of a few frames were observed.
Average misalignment errors of three sequences were 10.7 ± 4.9, 14.0 ± 2.9,
16.7 ± 9.5 pixels, respectively. Since 1 pixel in the laparoscopic image was ap-
proximately equal to 0.33 mm in this condition, the average misalignment errors
Misalignment error (pixel),
rotation angle (degree)
Fig.6. Misalignment error of augmented reality visualization in third sequence
Realtime Organ Tracking for Endoscopic Augmented Reality Visualization365
corresponded to around 3 to 5 mm. Fig. 6 shows time varying misalignment
error, distance from the position of the initial frame and rotation angle from
the orientation of the initial frame. Change of the distance means the resected
part was lifted up three times in this sequence. Change of the rotation angle
means change of the orientation of the resected part was moderate. When the
resected part was far from the initial position in the last part of the sequence,
the misalignment error increased greatly in proportion to the distance.
5Discussion and Conclusions
We have described endoscopic AR visualization using the Calypso system. We
evaluated distortion error caused by metallic surgical instruments. Since distor-
tion error was 0.4 mm or less when the distance from the transponder to the
instrument was 20 mm or more and around 1 mm even when the instrument
made contact with the transponder, the error would be negligible without dis-
tortion correction in most cases.
In the experiment of AR visualization, we demonstrated that the colored 3D
kidney model was superimposed onto the moving kidney in live laparoscopic
images. The demonstration means potential feasibility and usefulness of the AR
system using the Calypso system. Since the AR system is applicable to internal
organs which can be assumed as a rigid body (i.e. prostate), this study would
contribute to development and progress of the endoscopic surgical navigation
system for the internal organs. Average misalignment error of superimposition
was around 3 to 5 mm. Since the width of the resection margin in laparoscopic
partial nephrectomy is 5 mm, the misalignment error is not small enough to
clinical application. Delay of a few frames means there were some errors in tem-
porary registration. The misalignment error which increased in proportion to the
distance also means there were some errors in integration of the coordinate sys-
tems. Therefore, it is guessed that temporal and spatial misalignments between
the Calypso system and the Polaris are main factors of the error.
Future work includes integration of the Calypso system into the AR system.
Because the Calypso system can not control from the PC currently, complexity
to deal with the Calypso system obstructs accurate registration. If the Calypso
system is integrated, accuracy of temporal and spatial registration would be
1. Ukimura, O., Nakamoto, M., Desai, M., Herts, B., Aron, M., Haber, G.P., Kaouk,
J., Miki, T., Sato, Y., Hashizume, M., Gill, I.: Augmented reality visualization
during laparoscopic urologic surgery: The initial clinical experience. In: The 102nd
American Urological Association 2007 Annual Meeting, Anaheim, CA (2007)
2. Krueger, S., Timinger, H., Grewer, R., Borgert, J.: Modality-integrated magnetic
catheter tracking for x-ray vascular interventions. Physics in Medicine and Biol-
ogy 50, 581–597 (2005)
366M. Nakamoto et al.
3. Mori, K., Deguchi, D., Akiyama, K., Kitasaka, T., Maurer Jr., C.R., Suenaga,
Y., Takabatake, H., Mori, M., Natori, H.: Hybrid bronchoscope tracking using a
magnetic tracking sensor and image registration. In: Duncan, J.S., Gerig, G. (eds.)
MICCAI 2005. LNCS, vol. 3750, pp. 543–550. Springer, Heidelberg (2005)
4. Willoughby, T.R., Kupelian, P.A., Pouliot, J., Shinohara, K., Aubin, M., Roach
III, M., Skrumeda, L.L., Balter, J.M., Litzenberg, D.W., Hadley, S.W., Wei, J.T.,
Sandler, H.M.: Target localization and real-time tracking using the Calypso 4D
localization system in patients with localized prostate cancer. International Journal
of Radiation Oncology Biology Physics 65, 528–534 (2006)
5. Kupelian, P., Willoughby, T., Mahadevan, A., Djemil, T., Weinstein, G., Jani, S.,
Enke, C., Solberg, T., Flores, N., Liu, D., Beyer, D., Levine, L.: Multi-institutional
clinical experience with the Calypso System in localization and continuous, real-
time monitoring of the prostate gland during external radiotherapy. International
Journal of Radiation Oncology Biology Physics 67, 1088–1098 (2007)
6. Kupelian, P.A., Willoughby, T.R., Reddy, C.A., Klein, E.A., Mahadevan, A.: Hy-
pofractionated intensity-modulated radiotherapy (70 gy at 2.5 gy per fraction)
for localized prostate cancer: Cleveland clinic experience. International Journal of
Radiation Oncology Biology Physics 68, 1424–1430 (2007)
7. Rohlfing, T., Maurer Jr., C.R., O’Dell, W.G., Zhong, J.: Modeling liver motion and
deformation during the respiratory cycle using intensity-based nonrigid registration
of gated MR images. Medical Physics 31, 427–432 (2004)
8. Olbrich, B., Traub, J., Wiesner, S., Wiechert, A., Feußner, H., Navab, N.: Respi-
ratory motion analysis: Towards gated augmentation of the liver. In: Proceedings
of Computer Assisted Radiology and Surgery (CARS 2005), Berlin, Germany, pp.
9. von Siebenthal, M., Szekery, G., Gamper, U., Boesiger, P., Lomax, A., Cattin, P.:
4D MR imaging of respiratory organ motion and its variability. Physics in Medicine
and Biology 52, 1547–1564 (2007)
10. Nakamoto, M., Hirayama, H., Sato, Y., Konishi, K., Kakeji, Y., Hashizume, M.,
Tamura, S.: Recovery of respiratory motion and deformation of the liver using
laparoscopic freehand 3D ultrasound system. Medical Image Analysis 11, 429–442
11. Balter, J.M., Wright, J.N., Newell, L.J., Friemel, B., Dimmer, S., Cheng, Y., Wong,
J., Vertatschitsch, E., Mate, T.P.: Accuracy of a wireless localization system for
radiotherapy. International Journal of Radiation Oncology Biology Physics 61, 933–