Realtime Organ Tracking for Endoscopic Augmented Reality Visualization Using Miniature Wireless Magnetic Tracker.
ABSTRACT Organ motion is one of the problems on augmented reality (AR) visualization for endoscopic surgical navigation system. However,
the conventional optical and magnetic trackers are not suitable for tracking of internal organ motion. Recently, a wireless
magnetic tracker, which is called the Calypso 4-D localization system has been developed. Since the sensor of the Calypso
system is miniature and implantable, position of the internal organ can be measured directly. This paper describes AR system
using the Calypso system and preliminary experiments to evaluate the AR system. We evaluated distortion error caused by the
surgical instruments and misalignment error of superimposition. Results of the experiments shows potential feasibility and
usefulness of AR visualization of moving organ using the Calypso system.
- SourceAvailable from: Dale W Litzenberg[show abstract] [hide abstract]
ABSTRACT: The Calypso 4D Localization System is being developed to provide accurate, precise, objective, and continuous target localization during radiotherapy. This study involves the first human use of the system, to evaluate the localization accuracy of this technique compared with radiographic localization and to assess its ability to obtain real-time prostate-motion information. Three transponders were implanted in each of 20 patients. Eleven eligible patients of the 20 patients participated in a study arm that compared radiographic triangulated transponder locations to electromagnetically recorded transponder locations. Transponders were tracked for 8-min periods. The implantations were all successful, with no major complications. Intertransponder distances were largely stable. Comparison of the patient localization on the basis of transponder locations as per the Calypso system with the radiographic transponder localization showed an average (+/-SD) 3D difference of 1.5 +/- 0.9 mm. Upon tracking during 8 min, 2 of the 11 patients showed significant organ motion (>1 cm), with some motion lasting longer that 1 min. Calypso transponders can be used as magnetic intraprostatic fiducials. Clinical evaluation of this novel 4D nonionizing electromagnetic localization system with transponders indicates a comparable localization accuracy to isocenter, (within 2 mm) compared with X-ray localization.International Journal of Radiation OncologyBiologyPhysics 07/2006; 65(2):528-34. · 4.52 Impact Factor
- [show abstract] [hide abstract]
ABSTRACT: To report the clinical experience with an electromagnetic treatment target positioning and continuous monitoring system in patients with localized prostate cancer receiving external beam radiotherapy. The Calypso System is a target positioning device that continuously monitors the location of three implanted electromagnetic transponders at a rate of 10 Hz. The system was used at five centers to position 41 patients over a full course of therapy. Electromagnetic positioning was compared to setup using skin marks and to stereoscopic X-ray localization of the transponders. Continuous monitoring was performed in 35 patients. The difference between skin mark vs. the Calypso System alignment was found to be >5 mm in vector length in more than 75% of fractions. Comparisons between the Calypso System and X-ray localization showed good agreement. Qualitatively, the continuous motion was unpredictable and varied from persistent drift to transient rapid movements. Displacements > or =3 and > or =5 mm for cumulative durations of at least 30 s were observed during 41% and 15% of sessions. In individual patients, the number of fractions with displacements > or =3 mm ranged from 3% to 87%; whereas the number of fractions with displacements > or =5 mm ranged from 0% to 56%. The Calypso System is a clinically efficient and objective localization method for positioning prostate patients undergoing radiotherapy. Initial treatment setup can be performed rapidly, accurately, and objectively before radiation delivery. The extent and frequency of prostate motion during radiotherapy delivery can be easily monitored and used for motion management.International Journal of Radiation OncologyBiologyPhysics 03/2007; 67(4):1088-98. · 4.52 Impact Factor
- [show abstract] [hide abstract]
ABSTRACT: To study the outcomes in patients treated for localized prostate cancer with 70 Gy delivered at 2.5-Gy/fraction within 5 weeks. The study sample included all 770 consecutive patients with localized prostate cancer treated with hypofractionated intensity-modulated radiotherapy at the Cleveland Clinic between 1998 and 2005. The median follow-up was 45 months (maximum, 86). Both the American Society for Therapeutic Radiology and Oncology (ASTRO) biochemical failure definition and the alternate nadir + 2 ng/mL definition were used. The overall 5-year ASTRO biochemical relapse-free survival rate was 82% (95% confidence interval, 79-85%), and the 5-year nadir + 2 ng/mL rate was 83% (95% confidence interval, 79-86%). For patients with low-risk, intermediate-risk, and high-risk disease, the 5-year ASTRO rate was 95%, 85%, and 68%, respectively. The 5-year nadir + 2 ng/mL rate for patients with low-, intermediate-, and high-risk disease was 94%, 83%, and 72%, respectively. The Radiation Therapy Oncology Group acute rectal toxicity scores were 0 in 51%, 1 in 40%, and 2 in 9% of patients. The acute urinary toxicity scores were 0 in 33%, 1 in 48%, 2 in 18%, and 3 in 1% of patients. The late rectal toxicity scores were 0 in 89.6%, 1 in 5.9%, 2 in 3.1%, 3 in 1.3%, and 4 in 0.1% (1 patient). The late urinary toxicity scores were 0 in 90.5%, 1 in 4.3%, 2 in 5.1%, and 3 in 0.1% (1 patient). The outcomes after high-dose hypofractionation were acceptable in the entire cohort of patients treated with the schedule of 70 at 2.5 Gy/fraction.International Journal of Radiation OncologyBiologyPhysics 09/2007; 68(5):1424-30. · 4.52 Impact Factor
Realtime Organ Tracking for Endoscopic
Augmented Reality Visualization Using
Miniature Wireless Magnetic Tracker
Masahiko Nakamoto1, Osamu Ukimura2, Inderbir S. Gill3, Arul Mahadevan3,
Tsuneharu Miki2, Makoto Hashizume4, and Yoshinobu Sato1
1Division of Image Analysis, Graduate School of Medicine, Osaka University, Japan
2Department of Urology, Kyoto Prefectural University of Medicine, Kyoto, Japan
3Cleveland Clinic, Cleveland, OH, USA
4Graduate School of Medical Sciences, Kyushu University, Japan
Abstract. Organ motion is one of the problems on augmented reality
(AR) visualization for endoscopic surgical navigation system. However,
the conventional optical and magnetic trackers are not suitable for track-
ing of internal organ motion. Recently, a wireless magnetic tracker, which
is called the Calypso 4-D localization system has been developed. Since
the sensor of the Calypso system is miniature and implantable, position
of the internal organ can be measured directly. This paper describes AR
system using the Calypso system and preliminary experiments to evalu-
ate the AR system. We evaluated distortion error caused by the surgical
instruments and misalignment error of superimposition. Results of the
experiments shows potential feasibility and usefulness of AR visualiza-
tion of moving organ using the Calypso system.
Laparoscopic partial nephrectomy is a minimally invasive technique to remove
a renal tumor. In order to remove the tumor completely and preserve healthy
tissues as much as possible, augmented reality (AR) visualization of the CT im-
age where the resection line is depicted would be useful because the boundary
between healthy tissues and the tumor is not clear by endoscopic observation .
However, the surgeon holds up the kidney and changes its direction during resec-
tion of the tumor to confirm the resection line from several viewpoints. Therefore,
realtime tracking of the kidney motion is required to superimpose the CT image
onto live endoscopic images accurately.
Magnetic trackers are suitable to track objects inside the body. Conventional
wired magnetic trackers have been used to track flexible tubular instruments like
a catheter or a bronchoscope . However, they have not been employed to
track organ motion because it is necessary to pass the wire through the body.
Recently, a wireless magnetic tracker, which is called the Calypso 4-D localiza-
tion system (Calypso Medical Technologies, Inc., Seattle, WA, USA), has been
developed and applied to tracking of the prostate motion during the radiation
T. Dohi, I. Sakuma, and H. Liao (Eds.): MIAR 2008, LNCS 5128, pp. 359–366, 2008.
c ? Springer-Verlag Berlin Heidelberg 2008
360M. Nakamoto et al.
therapy . Three miniature magnetic sensors, which are called “transpon-
ders”, are implanted into the prostate beforehand, and then their 3D positions
are measured in realtime. Although the measurement and gating methods for
internal organ motion caused by respiration were proposed , these
methods assume periodical and regular motion. Since organ motion caused by
surgical operation is not periodical and regular unlike the organ motion caused
by respiration, the Calypso system is suitable for this purpose.
Our objective is to evaluate feasibility of endoscopic AR system using the
Calypso system. We evaluate the Calypso system from two viewpoints: (1) Dis-
tortion error caused by metallic surgical instruments. (2) Accuracy of superimpo-
sition. The conventional magnetic trackers suffer from magnetic field distortion
caused by metallic objects. Since the kidney is held by metallic surgical instru-
ments during surgical operation, effects of metallic surgical instruments would
not be negligible. Therefore, magnitude of the distortion error is an important
factor for clinical application of the Calypso system. We also demonstrate su-
perimposition of the CT image onto live endoscopic images, and thus accuracy
and feasibility of the endoscopic AR system are evaluated.
We demonstrated AR visualization for laparoscopic partial nephrectomy
(Fig. 1) . In order to assist determination of the resection line, the resec-
tion margin was visualized by colored areas. The kidney is divided to tumor,
resection margin, resection and healthy tissue areas, and these areas are colored
with red, yellow, green and blue (Fig. 1 left). The width of the resection margin
and resection areas is 5 mm. If the surgeon resects the tumor along the green
resection area, the resection margin of 5 mm or more can be kept. In order to
maintain accurate superimposition, the kidney needed to keep staying at the
same position and orientation as when the registration was performed because
the kidney was not tracked. Since the kidney is moved by surgical operation
practically, realtime tracking of the kidney is required.
Fig.1. Augmented reality visualization for laparoscopic partial nephrectomy. Left: 3D
kidney model reconstructed from CT image. Middle: original laparoscopic image. Right:
Realtime Organ Tracking for Endoscopic Augmented Reality Visualization 361
The AR system consists of the Calypso 4D Localization System, Polaris (North-
ern Digital Inc., Waterloo, Ontario, Canada), laparoscope (OTV-S5, Olympus,
Japan) and PC. The Calypso system consists of the miniature implantable elec-
tromagnetic transponder, which is 8.5 mm in length (Fig. 2(a)), 4D electromag-
netic array, which is a source of AC electromagnetic energy (Fig. 2(b)), and
console. The Calypso system measures the position of the transponder in 3.3
frames per second. Measurement volume is 150 × 150 × 270 mm. Bias error
and reproducibility are less than 0.4 mm and 0.6 mm, respectively . Three
transponders are attached to the organ to track position and orientation of a
target organ. The Polaris is also employed to track the laparoscope.
Integration of the coordinate systems of both trackers and registration of
the CT image are required to render superimposed images. Firstly, the matrix
TMT→OT, which transforms the coordinate system from the Calypso system
frame to the Polaris frame, is obtained by point matching algorithm described
as the following:
TMT→OT = argmin
where riand piare positions of the transponder measured by the Calypso sys-
tem and the Polaris, respectively. Secondly, the registration matrix between the
CT and Polaris frames at time t, TIMG→OT,t, is obtained by point matching
algorithm described as the following:
where S is a diagonal matrix of which diagonal factors represent the voxel sizes
of the CT image and qi is a position of the transponder in the CT image.
(a) Transponder is around
8.5 mm in length.
(b) Appearance of Calypso system.
Fig.2. Calypso 4D localization system
362M. Nakamoto et al.
Distance from transponder (mm)
Fig.3. Distortion error caused by surgical instruments
Finally, arbitrary position qt = (x,y,z,1)tin the CT image is projected to
t= (sx?,sy?,s,1)tin the laparoscopic image by the following equation:
where A and Text are intrinsic and extrinsic camera parameters obtained by
camera calibration, and Tcam,tis a rigid transformation matrix measured by the
Because the Calypso system is not connected to the PC in the current system,
position data measured by the Calypso system is imported to the PC offline and
then the superimposed images are rendered after the acquisition of positions and
3.2Experimental Conditions and Methods
We evaluated effects of metallic surgical instruments on the position measured
by the Calypso system. We employed the laparoscope, forceps (CLICKline Bowel
grasper, KARL Storz, Germany) and scissors (CLICKline Sciccors, KARL Storz,
Germany), which were commonly used in laparoscopic surgery, as surgical instru-
ments. The transponder and the surgical instrument were fixed on the operating
table which was made of non-metallic materials. The position of the transpon-
der was measured around 50 times while changing the distance between the
transponder and the instrument from 0 to 150 mm. We performed the mea-
surements for each instrument and performed them for all three instruments.
Powers of the laparoscope and the light source were turned off except when
the distance was 0 mm. The powers of them were turned on and off when the
Realtime Organ Tracking for Endoscopic Augmented Reality Visualization363
(a) Cow kidney with transponders.
(b) Arrangement of instruments.
Fig.4. Experimental setup for augmented reality visualization
distance was 0 mm. Distortion error caused by the surgical instruments is defined
as the difference of positions between with and without the surgical instruments.
We employed a cow kidney to demonstrate AR visualization. In order to
simulate laparoscopic partial nephrectomy, a pseudo tumor was embedded in
the kidney, and three transponders were attached around the pseudo tumor
(Fig. 4(a)). The CT image of the kidney was acquired, and then the kidney and
the pseudo tumor were segmented. The kidney and the laparoscope with optical
markers were fixed on the operating table (Fig. 4(b)). A part of the kidney was
grasped with the forceps and moved up and down, and then the pseudo tumor
was resected with the scissors and lifted up and down. The above operation
was divided into three sequences, and then superimposed images were rendered
for each sequence. Misalignment error of superimposition at time t is defined
transforming the position of the transponder in the CT image using equation (3),
and vi,tis a 2D position of the transponder in the laparoscopic image obtained
by manual pointing.
i=3|ui,t− vi,t|, where ui,tis a 2D position of the transponder given by
4.1 Distortion Error Caused by Surgical Instruments
Distortion error caused by the surgical instruments was around 1.0 mm when
the distance between the transponder and the instrument was 0 mm, and 0.4
mm or less when the distance was 20 mm or more as shown in Fig. 3. The error
when the powers were up was almost same as the error when the powers were
364M. Nakamoto et al.
Fig.5. Augmented reality visualization (third sequence). Upper row: original laparo-
scopic images. Lower row: superimposed images.
4.2Accuracy of Superimposition
Superimposed images of the third sequence are shown in Fig. 5. Kidney tis-
sue, pseudo tumor and transponder are depicted as blue, yellow and red part,
respectively. Delay of a few frames were observed.
Average misalignment errors of three sequences were 10.7 ± 4.9, 14.0 ± 2.9,
16.7 ± 9.5 pixels, respectively. Since 1 pixel in the laparoscopic image was ap-
proximately equal to 0.33 mm in this condition, the average misalignment errors
Misalignment error (pixel),
rotation angle (degree)
Fig.6. Misalignment error of augmented reality visualization in third sequence
Realtime Organ Tracking for Endoscopic Augmented Reality Visualization365
corresponded to around 3 to 5 mm. Fig. 6 shows time varying misalignment
error, distance from the position of the initial frame and rotation angle from
the orientation of the initial frame. Change of the distance means the resected
part was lifted up three times in this sequence. Change of the rotation angle
means change of the orientation of the resected part was moderate. When the
resected part was far from the initial position in the last part of the sequence,
the misalignment error increased greatly in proportion to the distance.
5Discussion and Conclusions
We have described endoscopic AR visualization using the Calypso system. We
evaluated distortion error caused by metallic surgical instruments. Since distor-
tion error was 0.4 mm or less when the distance from the transponder to the
instrument was 20 mm or more and around 1 mm even when the instrument
made contact with the transponder, the error would be negligible without dis-
tortion correction in most cases.
In the experiment of AR visualization, we demonstrated that the colored 3D
kidney model was superimposed onto the moving kidney in live laparoscopic
images. The demonstration means potential feasibility and usefulness of the AR
system using the Calypso system. Since the AR system is applicable to internal
organs which can be assumed as a rigid body (i.e. prostate), this study would
contribute to development and progress of the endoscopic surgical navigation
system for the internal organs. Average misalignment error of superimposition
was around 3 to 5 mm. Since the width of the resection margin in laparoscopic
partial nephrectomy is 5 mm, the misalignment error is not small enough to
clinical application. Delay of a few frames means there were some errors in tem-
porary registration. The misalignment error which increased in proportion to the
distance also means there were some errors in integration of the coordinate sys-
tems. Therefore, it is guessed that temporal and spatial misalignments between
the Calypso system and the Polaris are main factors of the error.
Future work includes integration of the Calypso system into the AR system.
Because the Calypso system can not control from the PC currently, complexity
to deal with the Calypso system obstructs accurate registration. If the Calypso
system is integrated, accuracy of temporal and spatial registration would be
1. Ukimura, O., Nakamoto, M., Desai, M., Herts, B., Aron, M., Haber, G.P., Kaouk,
J., Miki, T., Sato, Y., Hashizume, M., Gill, I.: Augmented reality visualization
during laparoscopic urologic surgery: The initial clinical experience. In: The 102nd
American Urological Association 2007 Annual Meeting, Anaheim, CA (2007)
2. Krueger, S., Timinger, H., Grewer, R., Borgert, J.: Modality-integrated magnetic
catheter tracking for x-ray vascular interventions. Physics in Medicine and Biol-
ogy 50, 581–597 (2005)
366M. Nakamoto et al.
3. Mori, K., Deguchi, D., Akiyama, K., Kitasaka, T., Maurer Jr., C.R., Suenaga,
Y., Takabatake, H., Mori, M., Natori, H.: Hybrid bronchoscope tracking using a
magnetic tracking sensor and image registration. In: Duncan, J.S., Gerig, G. (eds.)
MICCAI 2005. LNCS, vol. 3750, pp. 543–550. Springer, Heidelberg (2005)
4. Willoughby, T.R., Kupelian, P.A., Pouliot, J., Shinohara, K., Aubin, M., Roach
III, M., Skrumeda, L.L., Balter, J.M., Litzenberg, D.W., Hadley, S.W., Wei, J.T.,
Sandler, H.M.: Target localization and real-time tracking using the Calypso 4D
localization system in patients with localized prostate cancer. International Journal
of Radiation Oncology Biology Physics 65, 528–534 (2006)
5. Kupelian, P., Willoughby, T., Mahadevan, A., Djemil, T., Weinstein, G., Jani, S.,
Enke, C., Solberg, T., Flores, N., Liu, D., Beyer, D., Levine, L.: Multi-institutional
clinical experience with the Calypso System in localization and continuous, real-
time monitoring of the prostate gland during external radiotherapy. International
Journal of Radiation Oncology Biology Physics 67, 1088–1098 (2007)
6. Kupelian, P.A., Willoughby, T.R., Reddy, C.A., Klein, E.A., Mahadevan, A.: Hy-
pofractionated intensity-modulated radiotherapy (70 gy at 2.5 gy per fraction)
for localized prostate cancer: Cleveland clinic experience. International Journal of
Radiation Oncology Biology Physics 68, 1424–1430 (2007)
7. Rohlfing, T., Maurer Jr., C.R., O’Dell, W.G., Zhong, J.: Modeling liver motion and
deformation during the respiratory cycle using intensity-based nonrigid registration
of gated MR images. Medical Physics 31, 427–432 (2004)
8. Olbrich, B., Traub, J., Wiesner, S., Wiechert, A., Feußner, H., Navab, N.: Respi-
ratory motion analysis: Towards gated augmentation of the liver. In: Proceedings
of Computer Assisted Radiology and Surgery (CARS 2005), Berlin, Germany, pp.
9. von Siebenthal, M., Szekery, G., Gamper, U., Boesiger, P., Lomax, A., Cattin, P.:
4D MR imaging of respiratory organ motion and its variability. Physics in Medicine
and Biology 52, 1547–1564 (2007)
10. Nakamoto, M., Hirayama, H., Sato, Y., Konishi, K., Kakeji, Y., Hashizume, M.,
Tamura, S.: Recovery of respiratory motion and deformation of the liver using
laparoscopic freehand 3D ultrasound system. Medical Image Analysis 11, 429–442
11. Balter, J.M., Wright, J.N., Newell, L.J., Friemel, B., Dimmer, S., Cheng, Y., Wong,
J., Vertatschitsch, E., Mate, T.P.: Accuracy of a wireless localization system for
radiotherapy. International Journal of Radiation Oncology Biology Physics 61, 933–