Content uploaded by Gerhard Kurz
Author content
All content in this area was uploaded by Gerhard Kurz on Apr 14, 2015
Content may be subject to copyright.
Experimental Evaluation of Kinect and Inertial Sensors
for Beating Heart Tracking
Gerhard Kurz¹, Peter Hegedus², Gabor Szabo², Uwe D. Hanebeck¹
¹ Intelligent Sensor-Actuator-Systems Laboratory (ISAS)
Institute for Anthropomatics, Karlsruhe Institute of Technology (KIT), Germany
² Department of Cardiac Surgery, University of Heidelberg, Germany
Contact: gerhard.kurz@kit.edu
Abstract:
This paper investigates the use of Kinect depth sensors as well as inertial sensors in the context of beating
heart surgery. In the past, various sensors have been used in attempts to track the beating heart, each with its
own distinct set of advantages and disadvantages. With the availability of affordable structured-light depth
sensors such as the Kinect and sufficiently small and low priced inertial sensors, the question of their suitability
for beating heart tracking arises. We performed in-vivo experiments on a porcine heart in order to assess the
feasibility of beating heart tracking based on these sensors.
Keywords: depth camera, RGBD sensor, beating heart surgery, inertial measurement unit, arterial pressure
1 Problem
Motion tracking of the beating heart is of interest for cardiac imaging and radiology. Precise information about the motion
of the heart surface is particularly useful in the context of robotic surgery on the beating heart. In 2001, Nakamura et al.
first introduced the idea of using a teleoperated robot that automatically compensates for the heart motion while presenting
a stabilized view to the surgeon [1]. This allows the surgeon to perform surgery on the beating heart as though it were
still. On-pump coronary bypass operation (OPCAB) on the beating heart has significant medical benefits compared to the
”on-pump” technique of cardioplegia with cardiopulmonary bypass (CPB) through a heart-lung machine. [2] CPB and
the use of a heart-lung machine is always accompanied by heart injury caused by ischemia and the following reperfusion,
but also bears the risk of severe immune reaction, anaemia, and cerebral microembolization by provoking clot formation.
Additionally, beating heart surgery results in shorter hospital stays and is less expensive than heart surgery with CPB [3].
Different types of sensors have been used for beating heart tracking, for example regular cameras [4], [5] or endoscopes
[6], [7], high-speed cameras [1], electrocardiography [8], pressure sensors [9], force sensors [10], and others. The
availability of affordable RGBD (red, green, blue and depth) sensors with an accuracy in the range of 1 mm like the
Microsoft Kinect poses the question of their suitability for beating heart tracking. To our knowledge, this is the first
published work that uses a Kinect sensor for tracking the surface of the beating heart. As increasingly small and low
priced inertial measurement units (IMUs) become available, their applicability to beating heart tracking is of interest as
well. We are interested in capturing both the respiratory motion and the motion due to heartbeat with these sensors.
2 Methods
We performed an in-vivo experiment on a porcine beating heart (see Fig. 1a). From a median thoracotomy the anterior
surface of the heart was approached, then a Microsoft Kinect for Windows sensor was positioned approximately 50 cm
above the beating heart. Unlike the commonly used Xbox 360 Kinect sensor, which has a minimum measuring distance
around 80 cm, a Kinect for Windows sensor can provide depth measurements down to a distance of roughly 40 cm. The
size of the discretization step, i.e., the depth resolution, of the Kinect for Windows depends on the distance to the target
and is 1 mm in the relevant range (see Fig. 2). Kinect data was recorded at a frequency of 30 Hz at a resolution of 640 x
480.
Two InvenSense MPU-9150 IMUs were used to evaluate the applicability of inertial sensors in the context of beating
heart tracking. The size of an MPU-9150 is just 4 mm x 4 mm, which allows placing it on a small PCB with an area of
approximately 1 cm x 1 cm (see Fig. 3). The first IMU was attached to the diaphragm in order to record the movement
due to respiration, whereas the second IMU was attached to the pericardium in order to capture the motion due to
heartbeat. The IMU data consisting of accelerometer and gyroscope measurements was recorded at a frequency of 135
Hz.
Furthermore, in order to compare the results to commonly used medical sensors, arterial blood pressure and air pressure
inside the lung were recorded at a frequency of 1000 Hz. To obtain the actual movement of the heart surface, small
markers were placed on the surface and captured with a PIKE F-210 color camera (see Fig. 4a). The marker positions
were extracted by a color-based segmentation algorithm and their movement was tracked from frame to frame to obtain
their trajectory in image coordinates. The experimental setup is depicted in Fig. 1a and Fig. 1b. Only one of the cameras
visible in Fig. 1b was used in the discussed experiment.
3 Results
We recorded a sample of 80 s with all sensors mentioned above. For readability, we only show an excerpt of 14 s. Time
stamps obtained by the different recordings were converted to the same format to plot data from the different sensors in
a single graph. In the recorded depth images, the heart has a size of about 1000 to 2000 pixels and can thus be modeled
as a point cloud with as many points. This is only a small fraction of the Kinect’s resolution of 307 200 pixels, but still
significantly more than what can be obtained with commonly used approaches based on tracking landmarks using multi-
camera systems.
For our evaluation of the Kinect data, we considered the average depth
∶ 1
49
,
of the depth image d(x, y) in a small area (7 x 7 pixels) around a point (x, y)T. Since the heart movement is not restricted
to the axis perpendicular to the RGBD sensor, this obviously does not represent the motion of a fixed point on the heart
surface.
Our results from the Kinect, the camera-based marker tracking, and the two pressure sensors are depicted in Fig. 5a. The
depth information recorded by the Kinect camera clearly shows both respiratory motion and movement due to the beating
heart. To further analyze the motion recorded by the Kinect sensor, we calculated the power spectral density of the
different sensors (see Fig. 5b), where this is more evident because the frequencies of heartbeat (1.4 Hz) and respiration
(0.2 Hz) clearly stand out. It is interesting to note that the breathing motion is very hard to detect in this experiment by
tracking 2D marker positions since it occurs almost exclusively perpendicular to the image plane of the 2D camera.
The Kinect’s accuracy is obviously affected by the discretization interval of 1 mm, which is quite a significant limitation
for this application. However, this is less apparent in our experiments because we average across a certain area. According
to [7], an accuracy under 0.2 mm is required depending on the size of the vessels. In the future, new structured light
cameras may provide smaller discretization intervals and thus allow more accurate tracking.
We directly compare the IMU data to the pressure recordings (Fig. 6). Not surprisingly, we find that the IMU at the
diaphragm produces a signal with the same periodicity as the air pressure in the lung whereas the IMU at the pericardium
produces a signal with the same periodicity as the arterial blood pressure. Each inhalation and each exhalation respectively
as well as each heartbeat is clearly visible in the appropriate IMUs signals. These motions can be observed in both
accelerometer as well as gyroscope data.
4 Discussion
Our results show the general suitability of depth sensors, such as the Microsoft Kinect, for tracking the surface of the
beating heart. The Kinect is capable of following both heartbeat and breathing motion. While it is clearly possible to gain
information about the movement of the heart in the Kinect data, the accuracy may be too limited to rely solely on the
Kinect sensor for applications in the context of teleoperated beating heart surgery. A combination with more accurate
sensors like a high-resolution multi camera system might be used to remedy this limitation. A fusion algorithm such as
[11] may be employed to combine the advantages of both types of sensors.
The data obtained from the IMUs looks promising as well. Even though it is hard to estimate absolute positions and
orientations based on the IMU data, it is clearly possible to detect each heartbeat and inhalation as well as exhalation.
Since IMUs can typically achieve a fairly high sampling rate, they may be a good addition to slower sensors like color or
depth cameras.
5 Acknowledgments
This work was partially supported by the German Research Foundation (DFG) within the Research Training Group RTG
1126 “Intelligent Surgery - Development of new computer-based methods for the future working environment in visceral
surgery”. We would like to thank Geneviève Foley for her work on the camera-based marker tracking algorithm.
6 References
[1] Y. Nakamura, K. Kishi, and H. Kawakami, “Heartbeat Synchronization for Robotic Cardiac Surgery,” in Proceedings
of the IEEE International Conference on Robotics and Automation, Seoul, Korea, pp. 2014–2019, May 2001.
[2] W. B. Keeling, M. L. Williams, M. S. Slaughter, Y. Zhao, and J. D. Puskas, “Off-pump and on-pump coronary revas-
cularization in patients with low ejection fraction: a report from the society of thoracic surgeons national database.,”
The Annals of Thoracic Surgery, vol. 96, pp. 83–89, July 2013.
[3] T. J. Gilhuly, S. E. Salcudean, and S. V. Lichtenstein, “Evaluating Optical Stabilization of the Beating Heart,” IEEE
Engineering in Medicine and Biology Magazine, pp. 133–140, 2003.
[4] R. Richa, P. Poignet, and C. Liu, “Deformable Motion Tracking of the Heart Surface,” in Proceedings of the 2008
IEEE International Conference on Intelligent Robots and Systems (IROS 2008), 2008.
[5] G. Kurz and U. D. Hanebeck, “Image stabilization with model-based tracking for beating heart surgery,” in Proceed-
ings of CURAC 2012, 2012.
[6] D. Stoyanov and G. Yang, “Stabilization of Image Motion for Robotic Assisted Beating Heart Surgery,” Medical
Image Computing and Computer-Assisted Intervention, vol. 10, pp. 417–424, 2007.
[7] R. Richa, A. P. L. Bo, and P. Poignet, “Towards robust 3d visual tracking for motion compensation in beating heart
surgery,” Medical Image Analysis, vol. 15 (3), pp. 302–315, 2011.
[8] T. Ortmaier, M. Groeger, D. H. Boehm, V. Falk, and G. Hirzinger, “Motion Estimation in Beating Heart Surgery,”
IEEE Transactions on Biomedical Engineering, vol. 52, pp. 1729–1740, Oct. 2005.
[9] E. Bogatyrenko, P. Pompey, and U. D. Hanebeck, “Effcient physics-based tracking of heart surface motion for beating
heart surgery robotic systems,” International Journal of Computer Assisted Radiology and Surgery (IJCARS 2010),
vol. 6, pp. 387–399, Aug. 2010.
[10] Ö. Bebek and M. C. Cavusoglu, “Whisker sensor design for three dimensional position measurement in robotic
assisted beating heart surgery,” in IEEE International Conference on Robotics and Automation (ICRA), 2007.
[11] G. Kurz and U. D. Hanebeck, “Recursive Fusion of Noisy Depth and Position Measurements for Surface Recon-
struction,” in Proceedings of the 16th International Conference on Information Fusion (Fusion 2013), (Istanbul,
Turkey), July 2013.