Conference PaperPDF Available

Experimental Evaluation of Kinect and Inertial Sensors for Beating Heart Tracking

Authors:

Abstract and Figures

This paper investigates the use of Kinect depth sensors as well as inertial sensors in the context of beating heart surgery. In the past, various sensors have been used in attempts to track the beating heart, each with its own distinct set of advantages and disadvantages. With the availability of affordable structured-light depth sensors such as the Kinect and sufficiently small and low priced inertial sensors, the question of their suitability for beating heart tracking arises. We performed in-vivo experiments on a porcine heart in order to assess the feasibility of beating heart tracking based on these sensors.
No caption available
… 
Content may be subject to copyright.
Experimental Evaluation of Kinect and Inertial Sensors
for Beating Heart Tracking
Gerhard Kurz¹, Peter Hegedus², Gabor Szabo², Uwe D. Hanebeck¹
¹ Intelligent Sensor-Actuator-Systems Laboratory (ISAS)
Institute for Anthropomatics, Karlsruhe Institute of Technology (KIT), Germany
² Department of Cardiac Surgery, University of Heidelberg, Germany
Contact: gerhard.kurz@kit.edu
Abstract:
This paper investigates the use of Kinect depth sensors as well as inertial sensors in the context of beating
heart surgery. In the past, various sensors have been used in attempts to track the beating heart, each with its
own distinct set of advantages and disadvantages. With the availability of affordable structured-light depth
sensors such as the Kinect and sufficiently small and low priced inertial sensors, the question of their suitability
for beating heart tracking arises. We performed in-vivo experiments on a porcine heart in order to assess the
feasibility of beating heart tracking based on these sensors.
Keywords: depth camera, RGBD sensor, beating heart surgery, inertial measurement unit, arterial pressure
1 Problem
Motion tracking of the beating heart is of interest for cardiac imaging and radiology. Precise information about the motion
of the heart surface is particularly useful in the context of robotic surgery on the beating heart. In 2001, Nakamura et al.
first introduced the idea of using a teleoperated robot that automatically compensates for the heart motion while presenting
a stabilized view to the surgeon [1]. This allows the surgeon to perform surgery on the beating heart as though it were
still. On-pump coronary bypass operation (OPCAB) on the beating heart has significant medical benefits compared to the
”on-pump” technique of cardioplegia with cardiopulmonary bypass (CPB) through a heart-lung machine. [2] CPB and
the use of a heart-lung machine is always accompanied by heart injury caused by ischemia and the following reperfusion,
but also bears the risk of severe immune reaction, anaemia, and cerebral microembolization by provoking clot formation.
Additionally, beating heart surgery results in shorter hospital stays and is less expensive than heart surgery with CPB [3].
Different types of sensors have been used for beating heart tracking, for example regular cameras [4], [5] or endoscopes
[6], [7], high-speed cameras [1], electrocardiography [8], pressure sensors [9], force sensors [10], and others. The
availability of affordable RGBD (red, green, blue and depth) sensors with an accuracy in the range of 1 mm like the
Microsoft Kinect poses the question of their suitability for beating heart tracking. To our knowledge, this is the first
published work that uses a Kinect sensor for tracking the surface of the beating heart. As increasingly small and low
priced inertial measurement units (IMUs) become available, their applicability to beating heart tracking is of interest as
well. We are interested in capturing both the respiratory motion and the motion due to heartbeat with these sensors.
2 Methods
We performed an in-vivo experiment on a porcine beating heart (see Fig. 1a). From a median thoracotomy the anterior
surface of the heart was approached, then a Microsoft Kinect for Windows sensor was positioned approximately 50 cm
above the beating heart. Unlike the commonly used Xbox 360 Kinect sensor, which has a minimum measuring distance
around 80 cm, a Kinect for Windows sensor can provide depth measurements down to a distance of roughly 40 cm. The
size of the discretization step, i.e., the depth resolution, of the Kinect for Windows depends on the distance to the target
and is 1 mm in the relevant range (see Fig. 2). Kinect data was recorded at a frequency of 30 Hz at a resolution of 640 x
480.
Two InvenSense MPU-9150 IMUs were used to evaluate the applicability of inertial sensors in the context of beating
heart tracking. The size of an MPU-9150 is just 4 mm x 4 mm, which allows placing it on a small PCB with an area of
approximately 1 cm x 1 cm (see Fig. 3). The first IMU was attached to the diaphragm in order to record the movement
due to respiration, whereas the second IMU was attached to the pericardium in order to capture the motion due to
heartbeat. The IMU data consisting of accelerometer and gyroscope measurements was recorded at a frequency of 135
Hz.
Furthermore, in order to compare the results to commonly used medical sensors, arterial blood pressure and air pressure
inside the lung were recorded at a frequency of 1000 Hz. To obtain the actual movement of the heart surface, small
markers were placed on the surface and captured with a PIKE F-210 color camera (see Fig. 4a). The marker positions
were extracted by a color-based segmentation algorithm and their movement was tracked from frame to frame to obtain
their trajectory in image coordinates. The experimental setup is depicted in Fig. 1a and Fig. 1b. Only one of the cameras
visible in Fig. 1b was used in the discussed experiment.
3 Results
We recorded a sample of 80 s with all sensors mentioned above. For readability, we only show an excerpt of 14 s. Time
stamps obtained by the different recordings were converted to the same format to plot data from the different sensors in
a single graph. In the recorded depth images, the heart has a size of about 1000 to 2000 pixels and can thus be modeled
as a point cloud with as many points. This is only a small fraction of the Kinect’s resolution of 307 200 pixels, but still
significantly more than what can be obtained with commonly used approaches based on tracking landmarks using multi-
camera systems.
For our evaluation of the Kinect data, we considered the average depth
∶ 1
49 
󰇛,󰇜


of the depth image d(x, y) in a small area (7 x 7 pixels) around a point (x, y)T. Since the heart movement is not restricted
to the axis perpendicular to the RGBD sensor, this obviously does not represent the motion of a fixed point on the heart
surface.
Our results from the Kinect, the camera-based marker tracking, and the two pressure sensors are depicted in Fig. 5a. The
depth information recorded by the Kinect camera clearly shows both respiratory motion and movement due to the beating
heart. To further analyze the motion recorded by the Kinect sensor, we calculated the power spectral density of the
different sensors (see Fig. 5b), where this is more evident because the frequencies of heartbeat (1.4 Hz) and respiration
(0.2 Hz) clearly stand out. It is interesting to note that the breathing motion is very hard to detect in this experiment by
tracking 2D marker positions since it occurs almost exclusively perpendicular to the image plane of the 2D camera.
The Kinect’s accuracy is obviously affected by the discretization interval of 1 mm, which is quite a significant limitation
for this application. However, this is less apparent in our experiments because we average across a certain area. According
to [7], an accuracy under 0.2 mm is required depending on the size of the vessels. In the future, new structured light
cameras may provide smaller discretization intervals and thus allow more accurate tracking.
We directly compare the IMU data to the pressure recordings (Fig. 6). Not surprisingly, we find that the IMU at the
diaphragm produces a signal with the same periodicity as the air pressure in the lung whereas the IMU at the pericardium
produces a signal with the same periodicity as the arterial blood pressure. Each inhalation and each exhalation respectively
as well as each heartbeat is clearly visible in the appropriate IMUs signals. These motions can be observed in both
accelerometer as well as gyroscope data.
4 Discussion
Our results show the general suitability of depth sensors, such as the Microsoft Kinect, for tracking the surface of the
beating heart. The Kinect is capable of following both heartbeat and breathing motion. While it is clearly possible to gain
information about the movement of the heart in the Kinect data, the accuracy may be too limited to rely solely on the
Kinect sensor for applications in the context of teleoperated beating heart surgery. A combination with more accurate
sensors like a high-resolution multi camera system might be used to remedy this limitation. A fusion algorithm such as
[11] may be employed to combine the advantages of both types of sensors.
The data obtained from the IMUs looks promising as well. Even though it is hard to estimate absolute positions and
orientations based on the IMU data, it is clearly possible to detect each heartbeat and inhalation as well as exhalation.
Since IMUs can typically achieve a fairly high sampling rate, they may be a good addition to slower sensors like color or
depth cameras.
5 Acknowledgments
This work was partially supported by the German Research Foundation (DFG) within the Research Training Group RTG
1126 “Intelligent Surgery - Development of new computer-based methods for the future working environment in visceral
surgery”. We would like to thank Geneviève Foley for her work on the camera-based marker tracking algorithm.
6 References
[1] Y. Nakamura, K. Kishi, and H. Kawakami, “Heartbeat Synchronization for Robotic Cardiac Surgery,” in Proceedings
of the IEEE International Conference on Robotics and Automation, Seoul, Korea, pp. 2014–2019, May 2001.
[2] W. B. Keeling, M. L. Williams, M. S. Slaughter, Y. Zhao, and J. D. Puskas, “Off-pump and on-pump coronary revas-
cularization in patients with low ejection fraction: a report from the society of thoracic surgeons national database.,”
The Annals of Thoracic Surgery, vol. 96, pp. 83–89, July 2013.
[3] T. J. Gilhuly, S. E. Salcudean, and S. V. Lichtenstein, “Evaluating Optical Stabilization of the Beating Heart,” IEEE
Engineering in Medicine and Biology Magazine, pp. 133–140, 2003.
[4] R. Richa, P. Poignet, and C. Liu, “Deformable Motion Tracking of the Heart Surface,” in Proceedings of the 2008
IEEE International Conference on Intelligent Robots and Systems (IROS 2008), 2008.
[5] G. Kurz and U. D. Hanebeck, “Image stabilization with model-based tracking for beating heart surgery,” in Proceed-
ings of CURAC 2012, 2012.
[6] D. Stoyanov and G. Yang, “Stabilization of Image Motion for Robotic Assisted Beating Heart Surgery,” Medical
Image Computing and Computer-Assisted Intervention, vol. 10, pp. 417–424, 2007.
[7] R. Richa, A. P. L. Bo, and P. Poignet, “Towards robust 3d visual tracking for motion compensation in beating heart
surgery,” Medical Image Analysis, vol. 15 (3), pp. 302–315, 2011.
[8] T. Ortmaier, M. Groeger, D. H. Boehm, V. Falk, and G. Hirzinger, “Motion Estimation in Beating Heart Surgery,”
IEEE Transactions on Biomedical Engineering, vol. 52, pp. 1729–1740, Oct. 2005.
[9] E. Bogatyrenko, P. Pompey, and U. D. Hanebeck, “Effcient physics-based tracking of heart surface motion for beating
heart surgery robotic systems,” International Journal of Computer Assisted Radiology and Surgery (IJCARS 2010),
vol. 6, pp. 387–399, Aug. 2010.
[10] Ö. Bebek and M. C. Cavusoglu, “Whisker sensor design for three dimensional position measurement in robotic
assisted beating heart surgery,” in IEEE International Conference on Robotics and Automation (ICRA), 2007.
[11] G. Kurz and U. D. Hanebeck, “Recursive Fusion of Noisy Depth and Position Measurements for Surface Recon-
struction,” in Proceedings of the 16th International Conference on Information Fusion (Fusion 2013), (Istanbul,
Turkey), July 2013.
Conference Paper
Full-text available
We propose an algorithm to combine both depth and position measurements when estimating a continuous surface. Position measurements originate from a fixed point on the surface, whereas depth measurements are determined by the intersection of the surface with a line originating from the depth sensor. Through fusion of both types of measurements, it is possible to benefit from the advantages of different sensors. The surface is obtained through interpolation of control points with splines, which allows a compact representation of the surface. In order to simplify the problem of intersecting the surface with lines originating from the depth sensor, we propose the use of polar or spherical coordinates in surface parameterization. The presented algorithm can be applied in both 2D and 3D settings and is independent of the particular choice of sensors. Our method can recursively include new information as it is obtained by using nonlinear filtering and it considers uncertainties associated with the measurements.
Conference Paper
Full-text available
Performing surgery on the beating heart has significant advantages compared to cardiopulmonary bypass. However, when performed directly, it is very demanding for the surgeon. As an alternative, using a teleoperated robot for compensating the heart motion has been proposed. As an addition, this paper describes how stabilized images are obtained to create the illusion of operating on a stationary heart. For that purpose, the heart motion is tracked with a stochastic physical model. Based on correspondences obtained by motion tracking, image stabilization is considered as a scattered data interpolation problem. The proposed algorithms are evaluated on a heart phantom and in in-vivo experiments on a porcine heart, which show that there is very little residual motion in the stabilized images.
Conference Paper
Full-text available
Minimally invasive direct coronary artery bypass (MIDCAB) requires of surgeons precision of hand skill and mental concentration, since it needs to work on beating hearts. We propose a surgical robot system that compensates motions of organs during operations. The motion canceling robot system consists of three technologies; visual synchronization, motion synchronization and master-slave control. The visual stabilization provides the surgeon with the image of stabilized target point on the video monitor. The surgeon operates the master robot referring to the stabilized image. The motion stabilization, on the other hand, controls the slave robot being synchronized with the heart beat, which is the function of the master-slave control. Master-slave transforms the master motion and controls the slave robot. In this paper, we verify the effectiveness of the prototype system by in-vivo experiment.
Article
Full-text available
Tracking of beating heart motion in a robotic surgery system is required for complex cardiovascular interventions. A heart surface motion tracking method is developed, including a stochastic physics-based heart surface model and an efficient reconstruction algorithm. The algorithm uses the constraints provided by the model that exploits the physical characteristics of the heart. The main advantage of the model is that it is more realistic than most standard heart models. Additionally, no explicit matching between the measurements and the model is required. The application of meshless methods significantly reduces the complexity of physics-based tracking. Based on the stochastic physical model of the heart surface, this approach considers the motion of the intervention area and is robust to occlusions and reflections. The tracking algorithm is evaluated in simulations and experiments on an artificial heart. Providing higher accuracy than the standard model-based methods, it successfully copes with occlusions and provides high performance even when all measurements are not available. Combining the physical and stochastic description of the heart surface motion ensures physically correct and accurate prediction. Automatic initialization of the physics-based cardiac motion tracking enables system evaluation in a clinical environment.
Article
Background: The purpose of this study was to evaluate outcomes of patients with low ejection fraction who underwent surgical coronary revascularization with or without cardiopulmonary bypass (CPB). Methods: The Society of Thoracic Surgeons National Database was queried from January 1, 2008, to June 30, 2011 for patients with an ejection fraction of less than 0.30 who underwent primary, nonemergent coronary artery bypass (CAB) grafting. The entire cohort of 25,667 patients was divided into those who underwent revascularization with (ONCAB, n = 20,509) and without (OPCAB, n = 5,158) CPB. OPCAB patients who were converted to CPB intraoperatively were counted as intended OPCAB and were included in the OPCAB group. Propensity scores were estimated using 32 covariates, and multivariate logistic regression was used to compare risk-adjusted outcomes between groups. Results: Patients undergoing planned OPCAB were older, more frequently female, and had a lower body mass index than those who underwent ONCAB. The OPCAB cohort also had higher rates of prior stroke, peripheral vascular disease, and chronic lung disease. The predicted mortality risk was 2.3% for the OPCAB cohort vs 2.1% for the ONCAB group (p < 0.0001). Of the 5,158 patients who underwent OPCAB, unplanned conversion to CPB occurred in 270 (5.2%). OPCAB was associated with significantly lower adjusted risk of death (odds ratio [OR], 0.82), stroke (OR, 0.67), major adverse cardiac events (OR, 0.75), and prolonged intubation (OR, 0.78). Postoperative transfusion rates were significantly lower in the OPCAB group (44.8% vs 51.6%, p < 0.001). There were no adverse outcomes that occurred more commonly in OPCAB patients. The advantage associated with OPCAB was found in the entire Society of Thoracic Surgeons National Database and among high-volume and low-volume OPCAB centers. Conclusions: In The Society of Thoracic Surgeons National Database, OPCAB is associated with significantly reduced adjusted risk of early morbidity and mortality for patients having coronary bypass grafting with an ejection fraction of less than 0.30.
Conference Paper
In the robotic-assisted off-pump coronary artery bypass graft (CABG) surgery, surgeon performs the operation with intelligent robotic instruments controlled through teleoperation that replace conventional surgical tools. The robotic tools actively cancel the relative motion between the surgical instruments and the point of interest on the beating heart. Measuring the motion of the heart during this operation is an important part of this scheme. In this paper, a novel whisker sensor design to measure the heart motion in three dimensions (3D) is presented. The proposed whisker sensor is a flexible contact sensor. Low stiffness of the sensor prevents damage on the tissue it contacts. This paper explains the design concept, and reports the simulation and measurement results of the prototype whisker position sensor
Conference Paper
With the advent of new applications in cardiac robotic-assisted minimally invasive surgery (MIS), a demand for the design of efficient motion compensation systems was created. In this context, vision-based techniques seem to be a practical way to retrieve the motion of the beating heart since they do not require the introduction of additional sensors in the limited workspace. In this paper, we propose an efficient method for tracking the heart surface which incorporates two novelties. The first is a thin-plate splines (TPS) parametric model for the heart surface deformation that allows us to better track regions of the heart surface with little texture information which undergo large non-rigid deformations. The second novelty is the incorporation of a performing illumination compensation algorithm to cope with arbitrary illumination changes and increase tracking robustness.We also extend this framework for 3D tracking, to enable full compensation of the heart motion. Extensive experiments conducted on in-vivo and ex-vivo heart images attest the notable performance of the algorithm.
Article
In the context of minimally invasive cardiac surgery, active vision-based motion compensation schemes have been proposed for mitigating problems related to physiological motion. However, robust and accurate visual tracking remains a difficult task. The purpose of this paper is to present a robust visual tracking method that estimates the 3D temporal and spatial deformation of the heart surface using stereo endoscopic images. The novelty is the combination of a visual tracking method based on a Thin-Plate Spline (TPS) model for representing the heart surface deformations with a temporal heart motion model based on a time-varying dual Fourier series for overcoming tracking disturbances or failures. The considerable improvements in tracking robustness facing specular reflections and occlusions are demonstrated through experiments using images of in vivo porcine and human beating hearts.
Article
Coronary artery bypass grafting (CABG) is a surgical procedure performed to revascularize the areas of the heart deprived of oxygen by blockages. This procedure is relatively common. Improvement of this form of treatment is the topic of this article. An optical stabilization technique developed with the goal of enabling surgery on the beating heart is proposed and evaluated. The technique proposes that an ECG-triggered strobe light be focused on the surgical site making the heart appear still to the surgeon. The illusion of stillness then allows the surgeon to suture uninhibited by the actual motion. A test apparatus to model the heart was constructed and human-factors testing carried out to evaluate subject performance under conditions resembling those when suturing, with and without this strobe in place. In answer to the specific question of this research, "is strobing of a regular, beating heart beneficial to suturing as compared to suturing under normal lighting conditions?" the answer is "no". This was shown in both the objective and subjective measures. Whereas a strobed heart platform should have been less confusing to the subjects, easier to see, and thereby easier to mark, the opposite was seen. Time to perforrn the test was greater for the test of complicated motion with the strobe than without. Error increased as well. For both regularly lit room tests, subjects had better placement of marks as compared to when they did the same tests under optically stabilized conditions.
Article
Minimally invasive beating-heart surgery offers substantial benefits for the patient, compared to conventional open surgery. Nevertheless, the motion of the heart poses increased requirements to the surgeon. To support the surgeon, algorithms for an advanced robotic surgery system are proposed, which offer motion compensation of the beating heart. This implies the measurement of heart motion, which can be achieved by tracking natural landmarks. In most cases, the investigated affine tracking scheme can be reduced to an efficient block matching algorithm allowing for realtime tracking of multiple landmarks. Fourier analysis of the motion parameters shows two dominant peaks, which correspond to the heart and respiration rates of the patient. The robustness in case of disturbance or occlusion can be improved by specially developed prediction schemes. Local prediction is well suited for the detection of single tracking outliers. A global prediction scheme takes several landmarks into account simultaneously and is able to bridge longer disturbances. As the heart motion is strongly correlated with the patient's electrocardiogram and respiration pressure signal, this information is included in a novel robust multisensor prediction scheme. Prediction results are compared to those of an artificial neural network and of a linear prediction approach, which shows the superior performance of the proposed algorithms.