Content uploaded by Mo Fattahi Sani
Author content
All content in this area was uploaded by Mo Fattahi Sani on Sep 14, 2019
Content may be subject to copyright.
Towards Finger Motion Tracking and Analyses
for Cardiac Surgery
Mohammad Fattahi Sani1, Sajeeva Abeywardena1, Efi Psomopoulou1,
Raimondo Ascione2, and Sanja Dogramadzi1
1Bristol Robotics Laboratory, Coldharbour Lane, Bristol, UK
sanja.dogramadzi@brl.ac.uk
2Bristol Heart Institute, University of Bristol, Bristol, UK
Abstract. Robot Assisted Surgery is attracting increasing amount of
attention as it offers numerous benefits to patients as well as surgeons.
Heart surgery requires a high level of precision and dexterity, in contrast
to other surgical specialties. Robot assisted heart surgery is not as widely
performed due to numerous reasons including a lack of appropriate and
intuitive surgical interfaces to control minimally invasive surgical tools.
In this paper, finger motion of the surgeon is analyzed during cardiac
surgery tasks on an ex-vivo animal model with the purpose of designing
a more intuitive master console. First, a custom finger tracking system
is developed using IMU sensors, which is lightweight and comfortable
enough to allow free movement of the surgeons fingers/hands while using
instruments. The proposed system tracks finger joint angles and finger-
tip positions for three involved fingers (thumb, index, middle). Accuracy
of the IMU sensors has been evaluated using an optical tracking sys-
tem (Polaris, NDI). Finger motion of the cardiac surgeon while using
a Castroviejo instrument is studied in suturing and knotting scenarios.
The results show that PIP and MCP joints have larger Range Of Mo-
tion (ROM), and faster rate of change compared to other finger/thumb
joints, while thumb has the largest Fingertip WorkSpace (FWS) of all
three digits.
Keywords: Robot Assisted Surgery, Finger Tracking, Cardiac surgery
1 Introduction
Minimally invasive surgery brought a remarkable improvement over open surgery
in terms of hospitalization time and patients scarring [1]. On the other hand,
this method has complicated surgery performance due to a range of problems
such as fulcrum effect, inaccurate scaling, and lack of precision, dexterity and
intuitive handling [2] . Robot Assisted Surgery (RAS) has addressed some of
these problems. Among many available robotic systems, Da Vinci is currently
the only widely used in hospitals [3]. In the field of cardiac surgery, however, RAS
has not been widely adopted [3] and its application is mainly limited to mitral
valve repair [2–4] and in smaller extent, coronary artery revascularization [2,3],
2 Mohammad Fattahi Sani et al.
closure of simple septal defects, tricuspid valve repair, and cardiac tumor re-
moval [4] . In the literature, cost and steep learning curve of currently available
systems are counted as reasons for this [1,3,5, 6]. Despite Da Vinci system with
7 DOF master station and a wide range of tools the control of fine movements
required in heart surgeries is still an issue [6–8]. In addition, the lack of haptic
feedback [1, 5, 7, 8] , and the system bulkiness [1, 7] motivate research on more
intuitive control of the robotic surgical tools for this surgical area [8]. The most
important step towards designing a teleoperation system that deals with the
restricted space of heart surgeries and a finer set of motions required is to gain
a thorough understanding of the characteristics of finger/ hand motions of car-
diac surgeons. The characteristics of finger motion required for complex surgical
tasks are necessary for developing better, more customized master-slave robotic
tools for cardiac surgery. Therefore, the aim of our work is to track and analyze
finger/hand motion during cardiac surgery with the purpose of extracting ROM
for each finger and thumb joint as well as FWS during different maneuvers of
cardiac surgery.
2 Related Works
Finger motion tracking can be implemented for many purposes such as tele–
operating a robotic hand [9], or patient motion analysis in order to study dis-
eases [10, 11]. Different methods to implement finger tracking include using: 1)
Inertial Measurement Unit (IMU) based sensors [12,13] 2) optical tracking sys-
tems [9, 11, 14–17] 3) exoskeleton based systems including anthropomorphic ex-
oskeletons attached to fingers, or highly redundant exoskeleton attached on top
of the hand and wrist [18], 4) magnetic sensing [19], 5)flex sensor based [20],
and 6) fusing several methods together [21]. When it comes to surgeon fingers
tracking, optical tracking methods are highly vulnerable to occlusion [22]. Ex-
oskeleton based finger-tracking methods, on the other hand, are usually bulky
for surgery, and in some models suffer from inaccuracy due to misalignments
of the exoskeleton and the finger joints [18]. IMU sensors, are relatively small,
lightweight and occlusion free method of acquiring three rotation angles for each
link of the hand/finger. Nevertheless, their values might drift after a while, and
experience some magnetic interferences.
Hand motion analysis is a useful tool in different applications. Researchers
in [21] analyzed three different methods of data glove, force sensors and EMG
sensors in order to study human hand motion. The hand motion analysis is
typically used to study physical impairments [10, 11], electrical stimulation of
hand [15], analyze joint loads [16], gestures [23], or to find comfort zone of fingers
when interacting with smartphones [17] .
Surgical tool tracking has already been studied in the literature using
either 2D or 3D image processing [24], or mounting optical tracking markers [25].
Therefore, it is apparent that developing a reliable platform for analyzing finger
motions is necessary.
Towards Finger Motion Tracking... 3
Our Contributions: We propose a tracking system customized for hand
tracking of surgeon which is occlusion-free, precise and lightweight to collect
data during specific cardiac surgery tasks. We have identified ROM and FWS of
the three digits (Middle, Index and thumb) in different stages of the surgery.
3 Hand kinematic Model
In order to study hand/finger motions, a good understanding of biomechanics of
hand is essential. A full hand consists of 27 bones including fingers, thumb, palm,
and wrist [22]. Each finger is comprised of three parts of bones called phalanges.
Fingers are attached to Metacarpals through Metacarpophalangeal (MCP) joint,
which is followed by Proximal interphalangeal joint (PIP) and Distal interpha-
langeal joint (DIP). Thumb, however, which plays a crucial role in human ma-
nipulation capability, consists of two phalanges followed by a Metacarpal bone
attached to Carpus [18]. Various approaches have been put forward to model
hand/finger kinematics [26,27]. In this study, we are mainly interested to model
middle finger, index finger and thumb, therefore we utilize simple model pro-
posed in [27] , which can be seen in Fig.1.
Fig. 1. Finger joint models used in our method
As it can be seen in fig.1, DIP and PIP joints in index and middle fingers as
well as IP and MCP joints in thumb all have one revolute joint, whereas MCP
and CMC joints have two DOF revolute joints. We measure absolute orientation
of each phalange by a single IMU. Therefore, the joint angles can be calculated
4 Mohammad Fattahi Sani et al.
by simply subtracting the orientation of two consecutive IMU sensors. Sensors
are aligned with the finger/thumb lengthwise (Yaxis) while the sensors Zaxis
is perpendicular to the the fingers/thumb. Let us assume φ,θ,ψfor rotation
around Z,YandXaxis, respectively. Therefore, for instance, φM CP and θM C P
which show φand θangles for MCP joints, can be calculated as follows:
φMC P =φM C −φMP , θM CP =θM C −θM P (1)
Having all the required joint angles allow calculation of fingertip positions for
the two fingers and the thumb using forward kinematics [26]. DenavitHartenberg
(DH) parameters of the fingers and the thumb and forward kinematics calcula-
tions are used according to SynGrasp toolbox [28].
4 Experimental setup
The tracking system consists of 10 BNO055 IMU sensors (4 on the thumb and 3
on index and middle fingers). IMU sensors are connected to an acquisition board
comprised of multiple Microprocessor Units (MCU) running Arduino firmware.
Each MCU connects to two IMU sensors on a single I2C bus. Orientations from
each sensor are sent to the computer with frequency of 50Hz through a serial port.
Fig.2 shows the experimental implementation of the data acquisition system.
(a) Our custom Hand Tracking system. (b) Reconstructed model of the
hand in computer.
Fig. 2. Experimental setup for hand/finger tracking.
In addition to the sensors internal calibration process, we remove the offset
by measuring the angle values at the beginning of the test.
4.1 Accuracy assessment of IMU finger tracking system
In order to validate accuracy of the hand tracking system,, digit orientations and
joint angles estimated by the IMUs were compared to a ground truth. As it can
be seen in Fig.3a, an angled circle is used as a ground truth to test the accuracy
of IMU sensors for each orientation. Results show that average error is 4.1◦with
Towards Finger Motion Tracking... 5
a standard deviation of 2◦. Researchers in [12] assessed accuracy of the same
sensors in a planar pose and reported average error of 3◦∼6◦with standard
deviation of 1.7◦for different angles. The researchers in [13] reported in Static
angle errors of ≤2◦for their developed IMU-based hand tracking system.
(a) Accuracy assessment setup in planar mode. (b) Accuracy assessment in 3D
space.
Fig. 3. Accuracy assessment setup.
In addition, a dynamic angle verification was carried out using Polaris Spectra
system (NDI) to verify the accuracy of the tracking system. A custom tool with
markers (Fig.3b), which also houses IMU sensors, is designed and attached to
the Index finger, and orientations were measured relative to the marker and the
sensor fixed on the palm. Fig.4 shows simple flexion extension movements of the
index finger and their corresponding error values. According to experimental
tests, the measurement error is less than 8◦for rotating around Z,Yand Xaxis
of the sensor.
(a) Finger joint angles extracted from IMU
sensors and NDI Polaris system.
(b) Joint angle error with respect to NDI
Polaris system.
Fig. 4. Finger motions: Flexion, Extension. IMU sensors compared with NDI Polaris
motion capture system.
6 Mohammad Fattahi Sani et al.
4.2 Cardiac Surgery data collection
An experienced cardiac surgeon performed typical mitral valve surgery and aorta
suturing tasks on an ex-vivo pigs heart specimen with the tracking system fitted
to his hands, as as shown in Fig. 5. The tasks were performed using 7” Castroviejo
Needle Holder Plier Straight and 7” Castroviejo Micro Scissor.
(a) Finger/Hand data collection using our developed system dur-
ing the Mitral Valve surgery.
(b) A set of Castroviejo
instruments used in the
experiment.
Fig. 5. Cardiac surgery data collection setup and instruments.
4.3 Results and discussion
ROM and FWS are two important features of finger motion. They were extracted
in [11] and [15] in order to study human functional abilities and to explore hand
grasp patterns, respectively. In our study, ROM for each joint of the surgeons
fingers has been extracted and is shown in Table.1. Fig. 6,7,8,9 demonstrate joint
angles for a sample suturing operation.
Table 1. Range of motion (ROM) for surgeons fingers. (Degrees)
Action
Type
Index
DIP
Index
PIP
Index
MCP
Middle
DIP
Middle
PIP
Middle
MCP
Thumb
DIP
Thumb
MCP
Thumb
CMC
Wrist
IMU an-
gle
φ φ φ θ φ φ φ θ φ φ φ θ φ θ
Knotting 23 40 30 10 50 85 60 21 35 38 25 15 25 41
Suturing 15 25 19 12 39 60 55 30 22 15 15 20 45 45
*φstands for flexion–extension movement of fingers, whereas θstands for abduction
movements.
Towards Finger Motion Tracking... 7
Fig. 6. Middle finger joint angles
Fig. 7. Index finger joint angles
8 Mohammad Fattahi Sani et al.
Fig. 8. Thumb joint angles
Fig. 9. Wrist joint angles
Towards Finger Motion Tracking... 9
Rate of change is another valuable characteristic of finger joint motion which
demonstrates how fast each joint is moving. Assuming φ[n]as a discrete–time
series of joint angles, the angular velocity (ω) is calculated as follows.
ω[n]= (φ[n]−φ[n−1])/F s (2)
Where Fs =50 is the sampling frequency. Now, the average angular velocity
(ωavg ) of a series with N samples is calculated as follows:
ωavg = 1/N
N
X
n=1
ω[n](3)
Fig. 10 shows rate of change for two different surgical scenarios. In addition,
FWS during the surgery is shown in Fig.11 .
(a) Knotting. (b) suturing.
Fig. 10. Mean rate of change for each joint (joints are numbered from 1 for DIP joint).
A close look into the ROM results show that MCP and PIP joints in middle
finger have a relatively higher range of motion compared to others. Furthermore,
according to Fig.10, PIP and MCP joint are the ones that are changing fastest.
Finally, Fig.11 shows that FWS for the thumb has larger space compared to the
middle and index fingers.
5 Conclusion
In this paper, motion of the fingers and the thumb during typical cardiac surgery
tasks was studied. A custom IMU–based finger tracking system has been devel-
oped which is lightweight, comfortable, and precise enough to track the surgeons
finger motions. The captured data during the heart surgery shows that PIP and
MCP joints are moving faster and more than the other tracked joints. In addi-
tion, the thumb has larger FWS in this study. The outcome of this research will
10 Mohammad Fattahi Sani et al.
Fig. 11. Fingertip position during the suturing process with Castroviejo tool
be further utilized in designing a new intuitive way of controlling tele–operated
surgical robot tools for heart surgeries. In future steps, surgical tools will also
be monitored and their motion analyzed together with finger/thumb motions.
References
1. Peters BS, Armijo PR, Krause C, Choudhury SA, Oleynikov D (2018) Review of
emerging surgical robotic technology. Surg Endosc 32:16361655
2. Rodriguez E, Chitwood WR (2009) Robotics in cardiac surgery. Scand J Surg
98:120124
3. Quint E, Sivakumar G (2019) The role of robotic technology in cardiac surgery.
Univ West Ont Med J 87:4042
4. Dearani JA (2018) Robotic heart surgery: Hype or hope? J Thorac Cardiovasc Surg
155:7778
5. Pettinari M, Navarra E, Noirhomme P, Gutermann H (2017) The state of robotic
cardiac surgery in Europe. Ann Cardiothorac Surg 6:18
6. Scholar MD, Ashford H (2015) Use of Robots on Cardiac Surgery.
7. Simorov A, Stephen Otte R, Kopietz CM, Oleynikov D (2012) Review of surgical
robotics user interface: What is the best way to control robotic surgery? Surg Endosc
26:21172125
8. Cuschieri CFVFFMMFFMA (2010) Technical review of the da Vinci surgical tele-
manipulator. Int J Med Robot 6:468472
9. Cerulo I, Ficuciello F, Lippiello V, Siciliano B (2017) Teleoperation of the SCHUNK
S5FH under-actuated anthropomorphic hand using human hand motion tracking.
Rob Auton Syst 89:7584
10. Medicine P (2016) Kinematic Motion Analysis in Upper Extremity Cerebral Palsy.
Towards Finger Motion Tracking... 11
11. Leitkam ST, Reid Bush T (2014) Comparison Between Healthy and Reduced Hand
Function Using Ranges of Motion and a Weighted Fingertip Space Model. J Biomech
Eng 137:041003
12. Liu H, Xie X, Millar M, Edmonds M, Gao F, Zhu Y, Santos VJ, Rothrock B, Zhu
S-C A Glove-based System for Studying Hand-Object Manipulation via Joint Pose
and Force Sensing.
13. Lin B-S, Lee I-Jung, Chiang P-Y, Huang S-Y, Chih , Peng W A Modular Data
Glove System for Finger and Hand Motion Capture Based on Inertial Sensors. J
Med Biol Eng. doi: 10.1007/s40846-018-0434-6
14. Cerveri P, De Momi E, Lopomo N, Baud-Bovy G, Barros RML, Ferrigno G Finger
Kinematic Modeling and Real-Time Hand Motion Estimation. doi: 10.1007/s10439-
007-9364-0
15. Shin H, Watkins Z, Hu X Exploration of Hand Grasp Patterns Elicitable Through
Non-Invasive Proximal Nerve Stimulation OPEN. doi: 10.1038/s41598-017-16824-1
16. Vignais N, Cocchiarella DM, Kociolek AM, Keir PJ (2012) Dynamic Assessment of
Finger Joint Loads Using Kinetic and Kinematic Measurements. Digit Hum Model
2013 15
17. Le HV, Mayer S, Bader P, Henze N (2018) Fingers Range and Comfortable Area
for One-Handed Smartphone Interaction Beyond the Touchscreen. 112
18. Sarakoglou I, Brygo A, Mazzanti D, Hernandez NG, Caldwell DG, Tsagarakis NG
(2016) Hexotrac: A highly under-actuated hand Exoskeleton for finger tracking and
force feedback. IEEE Int Conf Intell Robot Syst 2016Novem:10331040
19. Chen K-Y, Patel S, Keller S (2016) Finexus: Tracking Precise Motions of Multiple
Fingertips Using Magnetic Sensing. Chi 16 15041514
20. Saggio G, Pallotti A, Sbernini L, Errico V, Paolo F Di (2016) Feasibility of Com-
mercial Resistive Flex Sensors for Hand Tracking Applications. Sensors & Trans-
ducers 201:1726
21. Ju Z, Liu H, Member S (2014) Human Hand Motion AnalysisWith Multisensory
Information. IEEE/ASME Trans Mechatronics 19:456466
22. Wheatland N, Wang Y, Song H, Neff M, Zordan V, Jrg S State of the Art in Hand
and Finger Modeling and Animation.
23. Tits M (2018) Expert Gesture Analysis through Motion Capture using Statistical
Modeling and Machine Learning Mickal Tits. doi: 10.13140/RG.2.2.36839.50084
24. Zhang L, Menglong Ye , Chan P-L, Yang G-Z (2017) Real-time surgical tool track-
ing and pose estimation using a hybrid cylindrical marker. Int J CARS 12:921930
25. Ye M, Zhang L, Giannarou S, Yang G-Z (2016) Real-time 3D Tracking of Articu-
lated Tools for Robotic Surgery.
26. Van Der Hulst FPJ, Schtzle S, Preusche C, Schiele A (2012) A functional anatomy
based kinematic human hand model with simple size adaptation. Proc - IEEE Int
Conf Robot Autom 51235129
27. Li K, Chen I-M, Yeo SH, Lim CK (2011) Development of finger-motion capturing
device based on optical linear encoder. J Rehabil Res Dev 48:69
28. Malvezzi M, Salvietti G, Gioioso G, Prattichizzo D (2015) SynGrasp: A Matlab
Toolbox for Underactuated and Compliant Hands Version 2.2 User Guide.
