Patient Speciﬁc Virtual and Physical Simulation
Platform for Surgical Robot Movability
Evaluation in Single-Access Robot-Assisted
Minimally-Invasive Cardiothoracic Surgery
), Sara Condino2, Sara Sinceri2, Izadyar Tamadon3,
Simona Celi4, Claudio Quaglia3, Michele Murzi4, Giorgio Soldani5,
Arianna Menciassi3, Vincenzo Ferrari2,6, and Mauro Ferrari2
1Computer Science Department, Kettering University, Flint, MI, USA
2Department of Translational Research on New Technologies in Medicine
and Surgery, EndoCAS Center, University of Pisa, Pisa, Italy
3The Biorobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy
4Fondazione Toscana Gabriele Monasterio, Pisa, Italy
5Consiglio Nazionale delle Ricerche, Istituto di Fisiologia Clinica, Pisa, Italy
6Information Engineering Department, University of Pisa, Pisa, Italy
Abstract. Recently, minimally invasive cardiothoracic surgery (MICS)
has grown in popularity thanks to its advantages over conventional
surgery and advancements in surgical robotics.
This paper presents a patient-speciﬁc virtual surgical simulator for the
movability evaluation of single-port MICS robots. This simulator can be
used for both the pre-operative planning to rehearse the case before the
surgery, and to test the robot in the early stage of development before
physical prototypes are built.
A physical simulator is also proposed to test the robot prototype in
a tangible environment. Synthetic replicas of the patient organs are able
to replicate the mechanical behaviors of biological tissues, allowing the
simulation of the physical interactions robot-anatomy.
The preliminary tests of the virtual simulator showed good perfor-
mance for both the visual and physics processes.
After reviewing the physical simulator, a surgeon provided a positive
evaluation of the organ replicas in terms of geometry and mechanical
Keywords: Virtual reality ·Unity game engine ·Computer-assisted
surgery ·Minimally-invasive surgery ·Cardiothoracic surgery ·Robotic
surgery ·Surgical simulation
Springer International Publishing AG 2017
L.T. De Paolis et al. (Eds.): AVR 2017, Part II, LNCS 10325, pp. 211–220, 2017.
DOI: 10.1007/978-3-319-60928-7 18
212 G. Turini et al.
Minimally invasive cardiothoracic surgery (MICS) has grown rapidly over the
past decade, thanks to continuous innovations in surgical techniques, advances
in surgical instruments, and the adoption of robotic technologies. According to
literature evidence, major cardiac operations traditionally performed through a
median sternotomy can be accomplished less invasively through small incisions,
with equivalent safety and durability .
Robotic systems have been utilized successfully to perform complex surgi-
cal procedures such as mitral valve and tricuspid valve repairs [3,9], single and
multiple vessel coronary artery bypass surgeries [9,11], atrial ﬁbrillation abla-
tions , intracardiac tumor resections, atrial septal defect closures, and left
ventricular leads [2,18]. Moreover recent studies demonstrated the feasibility of
performing aortic valve replacement in adults using surgical robots .
Single port access (SPA) surgery, which uses one skin incision for interven-
tions, caught the attention of the surgical community in the past a few years
because of its potential to further reduce the invasiveness of surgical procedures
and post-operative complications. Looking at this potentiality, researchers have
proposed various robotic systems to assist SPA surgery . For example, an
articulated robotic probe has been designed for cardiac surgery applications ,
and a ﬂexible “snakelike” robotic systems have been developed to allow physi-
cians to view, access, and perform complex procedures on the beating heart
through a single-access port .
Despite the reduced invasiveness, SPA robotic surgery has its own limitations.
One of the main issues is that the anatomical region reached from an incision site
is restrained. So the access port placement has to be carefully chosen depending
on: patient anatomy, steps of the surgical procedure, and robot workspace .
Patient-speciﬁc surgical simulators could overcome this limitation by allowing
the surgeon to plan the intervention in order to evaluate the robot workspace
and the optimal access port placement [13,19]. Moreover they can also be used
by robot designers to evaluate the robot dimensioning and distal dexterity in a
realistic scenario, thus improving the quality and shortening the design cycle.
In this paper we present a patient-speciﬁc virtual and physical simulation
platform (Figs. 1and 8b) for surgical robot evaluation in single-access robot-
assisted minimally-invasive cardiothoracic surgery.
2 Modeling of the Patient Anatomy
A computed tomography dataset with contrast medium was used to generate the
3D model. The stack of medical images in DICOM format was processed using a
speciﬁc segmentation pipeline, developed in VMTK, a software for the generation
of 3D virtual models by integrating custom Python scripts. The segmentation
algorithm is based on a hierarchical approach as previously described in .
Basically: once the most simple feature to be identiﬁed is reconstructed, the
associated pixels are excluded from the subsequent segmentation phase. Finally,
Virtual-Physical Simulator for Surgical Robot Movability Evaluation 213
Fig. 1. Overview of the virtual simulator: the surgical robot initially positioned above
the chest, 2 rib parts (in green) can be expanded to facilitate the robot insertion, the
insertion point highlighted by a red dot on the skin, the 3 views from the robot end-
eﬀector cameras (right), and the GUI to control the virtual simulator (bottom). (Color
mesh reconstruction, artefacts removal, and holes ﬁlling stages were performed
to generate the 3D models of the patient anatomy necessary for the surgical
simulation, including: rib cage, aortic arch, ascending aorta, and aortic valve.
3 Development of the Virtual Simulator
The virtual simulator was designed to be a standalone desktop application for
the Microsoft Windows platform (Fig. 1). We used Blender and Unity as the
main tools for the 3D content creation and the software development respectively
(Fig. 2). Both tools were chosen because they are cost-eﬀective and technically
suitable to implement virtual surgical simulators [7,15].
3.1 Modeling of the Virtual Surgical Robot
The virtual surgical robot was modeled accordingly to the current prototype as
designed by the mechanical engineering team. This single-access surgical robot
consists in: a mechanical ﬂexible trunk with a user-controlled torsion, 3 distal
blades allowing its insertion and anchoring into the aorta, and 3 distal cameras
for the inspection of the aorta inner part once the robot is inserted (Fig. 4a).
In Unity, the entire behavior of the virtual robot was controlled by a C#
script component, implementing the following mechanisms:
214 G. Turini et al.
Fig. 2. The virtual simulator project in the Unity game engine editor.
•control the main body torsion using the keyboard arrow keys;
•robot insertion/extraction pressing “page down”/“page up” respectively;
•opening/closing of the robot end-eﬀector blades with keys “e” and “q”;
•robot rotation around its axis using “comma” and “period” keys.
In order to enable the interactions between the virtual surgical robot and
the virtual anatomy, we conﬁgured each robot part with a Collider component,
and a Rigidbody component (Fig. 3). These Unity components enable collision-
detection capabilities, and physics properties respectively.
All the robot Collider components have been conﬁgured using the proper
type (i.e. shape) to approximate the 3D geometry of the respective part, and
setting them to be trigger Colliders. In this way, Unity will be able to detect the
collisions robot-anatomy, but these collisions will not aﬀect the robot positioning.
(a) (b) (c) (d)
Fig. 3. The structure of the Collider components of the virtual robot in Unity: (a),
(b), (c), and (d) show the torsion boundaries of the surgical robot.
Virtual-Physical Simulator for Surgical Robot Movability Evaluation 215
Fig. 4. Detailed view of: (a) the virtual robot end-eﬀector (highlighted in orange),
including 3 blades to open/close the tip and 3 cameras for endovascular inspection;
and (b) the navigation mesh (in blue) baked on the chest of the patient, and used to
implement the interactive navigation of the insertion point on the skin (Color ﬁgure
All the robot Rigidbody components have been conﬁgured to be kinematic.
In this way, we can assign each part its own physics properties, but we disable
any update of its position/orientation performed by the Unity physics engine.
This conﬁguration provides the maximum control to the user, and preserves
the capability to detect all the interactions between the robot and the anatomy.
3.2 Interactive Simulation of the Surgical Robot Movability
The virtual surgical simulator was designed to provide a user-friendly interface
to enable the surgeon to rehearse the robot placement using only: the mouse, its
3 buttons, and a minimal set of keyboard keys (Fig. 1).
The complete interface includes some keyboard controls, a GUI panel (Fig.1),
and some interactions available directly on the 3D virtual anatomy:
•an invisible trackball allows the user to rotate, zoom in, and zoom out the
main view (i.e. the point-of-view of the surgeon) using only mouse buttons
and drag-and-drop (see Figs. 5aand5b);
•the virtual chest is interactive, allowing the user to click on the skin to directly
place the insertion point (see red dot in Fig. 1);
•the insertion point can also be precisely positioned by moving it on the skin
using the “WASD” keys (Fig. 1);
•ﬁve buttons on the GUI panel (Fig. 1) allow the user to rotate the robot in
respect to the access point pivot axis, orthogonal to the skin surface (Fig. 5a);
•an error message is shown on the GUI panel (and an audio signal is played)
whenever a collision between the robot and the rib cage is detected (Fig. 5b).
216 G. Turini et al.
Fig. 5. Two diﬀerent placements of the single-access surgical robot using the virtual
simulator: (a) frontal view of the virtual anatomy and the robot tilted in respect to
the pivot axis (white), and (b) side view of the virtual anatomy with an error message
(red) signaling a collision between the robot and the rib cage.
The trackball was implemented in a C#script component attached to the
main camera, and allows: the rotation of the main view around the virtual
anatomy, with angular limits to avoid uncomfortable points of view; the zooming
in and out implemented modifying the main camera ﬁeld-of-view angle.
The interaction with the virtual chest was implemented through ray-casting,
in a C#script component attached to it. Every time a mouse right click event
was raised, the script converted it into a 3D ray using the Camera class un-
projection capabilities. Then, the script used the Unity physics engine to cast
the 3D ray, detecting its collision with the virtual chest thanks to a Mesh Collider
component added to the skin.
The ﬁne positioning of the insertion point is performed moving it on the
skin using the keyboard. This implementation exploited a Unity NavMesh:a
navigation mesh approximating the “walkable” surface and enabling artiﬁcial
intelligence path-planning capabilities (Fig. 4b). In our project, a NavMesh has
been baked on the chest, and it has been properly conﬁgured to allow movements
only on the almost-ﬂat part of the skin. A NavMeshAgent component attached
to the insertion point enables the movement on the NavMesh. Finally, a C#
script component attached to the chest controls the NavMeshAgent to perform
the proper movement accordingly to the user inputs.
The GUI panel buttons allow the tilting of the virtual robot in respect to
the access point pivot axis (Fig. 5a). This feature has been implemented simply
exploiting the parenting between Unity GameObjects, in this case: the insertion
point (the parent GameObject), and the virtual robot (the child GameObject).
Finally, collisions between the virtual robot and the rib cage are identiﬁed
using Unity collision-detection capabilities. In fact, all the robot parts have their
respective Colliders, and the rib cage has a MeshCollider.
4 Development of the Physical Simulator
The manufacturing of organ physical replicas involves rapid prototyping tech-
niques as described in previous works [6,17]. More particularly a 3D printer
Virtual-Physical Simulator for Surgical Robot Movability Evaluation 217
(a) (b) (c)
Fig. 6. Manufacturing of the ascending aorta distal part and aortic valve: (a) 3D virtual
model of the portion to be reproduced, (b) CAD model of designed mold, and (c) CAD
view of the mold inner core with pins for a correct positioning.
(Dimension Elite 3D Printer) is used to turn the 3D virtual model of the patient
bones into tangible 3D synthetic replicas made of acrylonitrile butadiene styrene
(ABS). This plastic is commonly used for the manufacturing of bone replica for
orthopedic surgery simulations, since it suﬃciently replicates the mechanical
behavior of the natural tissue [1,16,17].
Soft synthetic replica of the whole or a part of an organ can be manufactured
with casting technique, selecting plastic materials with properties tailored to
the speciﬁc application . Injection molds are designed using a computer-aided
design (CAD) software starting from the organ 3D virtual models (Fig. 6).
Figure 7shows the physical simulator developed for cardiac interventions
involving the ascending aorta and the aortic valve, including a replica of:
•the rib cage with bones made of ABS, and a portion of costal cartilage made
of a high hardness silicone rubber to reproduce the elastic behavior of the
natural tissue (highlighted in red in Fig. 8b);
•the aortic arch (with brachiocephalic, left common carotid, and left subclavian
arteries) made of ABS with a pin to anchor it to a base (Fig. 7b);
•the ascending aorta and the aortic valve, which are the anatomical targets of
the intervention, made of soft silicone for a realistic interaction with surgical
instruments (traditional and/or robotic devices);
(a) (b) (c)
Fig. 7. Assembly of the physical simulator: (a) CAD assembly including a portion of
the rib cage, the aortic arch, the ascending aorta, the aortic valve, an aortic valve
support, and a base for a stable positioning of the anatomical parts; (b) the aortic arch
and the aortic valve support with pins; and (c) the CAD assembly of whole simulator.
218 G. Turini et al.
Fig. 8. Side-by-side views of the virtual and physical anatomies: (a) the virtual
anatomy as imported in Unity for the virtual simulator; and (b) the physical anatomy
built using 3D printing technology and silicones to have ﬂexible and rigid parts. (Color
•an aortic valve support made of ABS with a pin for anchoring (Fig. 7b);
•a base with a grid of holes accommodates the pin of the aortic arch and the
aortic valve support (labeled in green in Fig. 8b).
The assembly of the organ replicas can be customized using the grid of holes
in the base of the physical simulator (see green label in Fig. 8b). This grid allows
apin-hole coupling, constraining 5 DOF, that can be used to simulate diﬀerent
anatomical conﬁgurations by repositioning the aortic arch and the aortic valve.
5 Preliminary Results and Future Work
The virtual surgical simulator was tested on a laptop running Microsoft Windows
7 (Intel Core i7 – 2.80 GHz, 16 GHz RAM, GPU nVidia GeForce GT 650 M),
using a virtual 3D environment including: the patient anatomy composed of
approximately 81k vertices and 150k triangles, and the surgical robot made with
roughly 63k vertices and 64k triangles. The update frequency was ranging from
65 to 75 fps, with the physics engine running at 50 fps (default). The memory
required to run the simulator was about 140 MB.
The physical simulator underwent a qualitative evaluation performed by a
surgeon, and both the geometry and mechanical behavior of all the synthetic
organ replicas were positively evaluated. A quantitative evaluation, considering
all the factors aﬀecting the generation of the 3D organ replicas, estimated an
accuracy of less than 2 mm for the physical anatomy.
The virtual simulator described allows the pre-operative planning to rehearse
the surgical case before the actual intervention, and the evaluation of the surgical
robot during the design-development cycle. The physical simulator presented
enables the evaluation of the surgical robot in a synthetic anatomy, testing the
physical interactions between the robot prototype and the organ replicas. Thus,
we can assess the robot payloa d and compli ance performing a wide range of tasks
(not included in the virtual simulation). Furthermore, the virtual and physical
Virtual-Physical Simulator for Surgical Robot Movability Evaluation 219
surgical simulators can also be eﬃciently integrated into the clinical context for
teaching and training purposes.
In the future, we plan to combine the virtual and physical simulators into a
single mixed reality system. Additionally, we will also perform validation studies
to test the face validity of the simulator.
Acknowledgments. The research leading to these results has been supported by the
scientiﬁc project ValveTech (“Realizzazione di una Valvola Aortica Polimerica di Nuova
Concezione ed Impiantabile Tramite Piattaforma Robotica con Tecniche di Chirurgia
Mininvasiva” 2016–2018) funded by the Tuscany Region (Italy) through the call FAS
1. Baccarin, L.S., Casarin, R.C.V., Lopes-da Silva, J.V., Passeri, L.A.: Analysis of
mandibular test specimens used to assess a bone ﬁxation system. Craniomaxillo-
facial Trauma Reconstr. 8(03), 171–178 (2015)
2. Bhatt, A.G., Steinberg, J.S.: Robotic-assisted left ventricular lead placement. Card.
Electrophysiol. Clin. 7(4), 649–659 (2015)
3. Bush, B., Nifong, L.W., Alwair, H., Chitwood Jr., W.R.: Video-atlas on robotically
assisted mitral valve surgery. Ann. Cardiothorac. Surg. 2(6), 846–848 (2013)
4. Carbone, M., Condino, S., Mattei, L., Forte, P., Ferrari, V., Mosca, F.: Anthropo-
morphic ultrasound elastography phantoms: Characterization of silicone materials
to build breast elastography phantoms. In: 2012 Annual International Conference
of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 492–494.
5. Celi, S., Berti, S.: In-vivo segmentation and quantiﬁcation of coronary lesions by
optical coherence tomography images for a lesion type deﬁnition and stenosis grad-
ing. Med. Image Anal. 18(7), 1157–1168 (2014)
6. Condino, S., Carbone, M., Ferrari, V., Faggioni, L., Peri, A., Ferrari, M., Mosca,
F.: How to build patient-speciﬁc synthetic abdominal anatomies. An innovative
approach from physical toward hybrid surgical simulators. Int. J. Med. Robot.
7(2), 202–213 (2011)
7. De Luca, V., Meo, A., Mongelli, A., Vecchio, P., De Paolis, L.T.: Development
of a virtual simulator for microanastomosis: new opportunities and challenges. In:
Paolis, L.T., Mongelli, A. (eds.) AVR 2016. LNCS, vol. 9769, pp. 65–81. Springer,
Cham (2016). doi:10.1007/978-3-319-40651-0 6
8. Degani, A., Choset, H., Wolf, A., Zenati, M.A.: Highly articulated robotic probe
for minimally invasive surgery. In: Proceedings of the 2006 IEEE International
Conference on Robotics and Automation, ICRA 2006, pp. 4167–4172. IEEE (2006)
9. Folliguet, T.A., Dibie, A., Philippe, F., Larrazet, F., Slama, M.S., Laborde, F.:
Robotically-assisted coronary artery bypass grafting. Cardiol. Res. Pract. 2010
10. Folliguet, T.A., Laborde, F.: Robotic aortic valve replacement. In: Chitwood Jr.,
W.R. (ed.) Atlas of Robotic Cardiac Surgery, pp. 265–269. Springer, London (2014)
11. Gong, W., Cai, J., Wang, Z., Chen, A., Ye, X., Li, H., Zhao, Q.: Robot-assisted
coronary artery bypass grafting improves short-term outcomes compared with min-
imally invasive direct coronary artery bypass grafting. J. Thorac. Dis. 8(3), 459–468
220 G. Turini et al.
12. Iribarne, A., Easterwood, R., Chan, E.Y., Yang, J., Soni, L., Russo, M.J., Smith,
C.R., Argenziano, M.: The golden age of minimally invasive cardiothoracic surgery:
current and future perspectives. Future Cardiol. 7(3), 333–346 (2011)
13. Moglia, A., Turini, G., Ferrari, V., Ferrari, M., Mosca, F.: Patient speciﬁc surgical
simulator for the evaluation of the movability of bimanual robotic arms. Stud.
Health Technol. Inf. 163, 379–385 (2011)
14. Neuzil, P., Cerny, S., Kralovec, S., Svanidze, O., Bohuslavek, J., Plasil, P., Jehlicka,
P., Holy, F., Petru, J., Kuenzler, R., et al.: Single-site access robot-assisted
epicardial mapping with a snake robot: preparation and ﬁrst clinical experience.
J. Robot. Surg. 7(2), 103–111 (2013)
15. Paci, A., Marcutti, S., Ricci, S., Casadio, M., Vercelli, G.V., Marchiol`e, P., Cor-
done, M.: eBSim: development of a low-cost obstetric simulator. In: Paolis, L.T.,
Mongelli, A. (eds.) AVR 2016. LNCS, vol. 9769, pp. 101–110. Springer, Cham
(2016). doi:10.1007/978-3-319-40651-0 9
16. Parchi, P., Condino, S., Carbone, M., Gesi, M., Ferrari, V., Ferrari, M., Lisanti,
M.: Total hip replacement simulators with virtual planning and physical replica for
surgical training and rehearsal. In: Proceedings of the 12th IASTED International
Conference on Biomedical Engineering, BioMed 2016, pp. 97–101 (2016)
17. Parchi, P., Ferrari, V., Piolanti, N., Andreani, L., Condino, S., Evangelisti, G.,
Lisanti, M.: Computer tomography prototyping and virtual procedure simulation
in diﬃcult cases of hip replacement surgery. Surg. Technol. Int. 23, 228–234 (2013)
18. Poﬀo, R., Toschi, A.P., Pope, R.B., Celullare, A.L., Ben´ıcio, A., Fischer, C.H.,
Vieira, M.L.C., Teruya, A., Hatanaka, D.M., Rusca, G.F., et al.: Robotic surgery
in cardiology: a safe and eﬀective procedure. Einstein (Sao Paulo) 11(3), 296–302
19. Turini, G., Moglia, A., Ferrari, V., Ferrari, M., Mosca, F.: Patient-speciﬁc surgical
simulator for the pre-operative planning of single-incision laparoscopic surgery with
bimanual robots. Comput. Aided Surg. 17(3), 103–112 (2012)
20. Wutzler, A., Wolber, T., Haverkamp, W., Boldt, L.H.: Robotic ablation of atrial-
ﬁbrillation. J. Vis. Exp. JoVE 99 (2015)
21. Xu, K., Zheng, X.: Conﬁguration comparison for surgical robotic systems using a
single access port and continuum mechanisms. In: 2012 IEEE International Con-
ference on Robotics and Automation (ICRA), pp. 3367–3374. IEEE (2012)