ArticlePDF Available

Dynamic changes of brain functional states during surgical skill acquisition


Abstract and Figures

There is lack of a standardized measure of technical proficiency and skill acquisition for robot-assisted surgery (RAS). Learning surgical skills, in addition to the interaction with the machine and the new surgical environment adds to the complexity of the learning process. Moreover, evaluation of surgeon performance in operating room is required to optimize patient safety. In this study, we investigated the dynamic changes of RAS trainee's brain functional states by practice. We also developed brain functional state measurements to find the relationship between RAS skill acquisition (especially human-machine interaction skills) and reconfiguration of brain functional states. This relationship may help in providing trainees with helpful, structured feedback regarding skills requiring improvement and will help in tailoring training activities.
Content may be subject to copyright.
Dynamic changes of brain functional states
during surgical skill acquisition
Somayeh B. Shafiei
, Ahmed Aly HusseinID
, Khurshid A. Guru
1Department of Mechanical and Aerospace Engineering, University at Buffalo, SUNY, Buffalo, New York,
United States of America, 2Applied Technology Laboratory for Advanced Surgery (ATLAS), Roswell Park
Cancer Institute, Buffalo, New York, United States of America, 3Department of Urology, Roswell Park
Cancer Institute, Buffalo, New York, United States of America, 4Department of Urology, Cairo University,
Cairo, Egypt
There is lack of a standardized measure of technical proficiency and skill acquisition for
robot-assisted surgery (RAS). Learning surgical skills, in addition to the interaction with the
machine and the new surgical environment adds to the complexity of the learning process.
Moreover, evaluation of surgeon performance in operating room is required to optimize
patient safety. In this study, we investigated the dynamic changes of RAS trainee’s brain
functional states by practice. We also developed brain functional state measurements to
find the relationship between RAS skill acquisition (especially human-machine interaction
skills) and reconfiguration of brain functional states. This relationship may help in providing
trainees with helpful, structured feedback regarding skills requiring improvement and will
help in tailoring training activities.
With the widespread adoption of robot-assisted surgery (RAS), it becomes crucial to provide
robots with the capability of collaborating closely with humans in a complex and critical envi-
ronment like the operating room (OR) [1,2]. Understanding how learning and skill acquisi-
tion occur can open new horizons for surgical teaching, skill assessment, and also set a
platform for active interaction with robots. This shared environment encompasses trainees
acquiring the appropriate knowledge, learning the necessary surgical skills and how to interact
with robots on one side, and robots should be equipped appropriately to learn new collabora-
tive tasks on the other [3,4].
The main challenge in skill assessment is the dynamic nature of learning, which results
in brain neuroplasticity changes, where the brain’s neural organization changes during the
development of a motor cognition skill through practice [5]. Learning technical surgical
skills matures through 3 stages. First, the “cognitive” stage as one initially learns the skill and–
thoughtfully- performs it. With practice, trainees become less thoughtful about the steps and
reach “the associative stage” and can operate with fewer disruptions. The final stage is the
PLOS ONE | October 31, 2018 1 / 19
Citation: Shafiei SB, Hussein AA, Guru KA (2018)
Dynamic changes of brain functional states during
surgical skill acquisition. PLoS ONE 13(10):
Editor: Camillo Gualtieri, North Carolina
Neuropsychiatry Clinics, UNITED STATES
Received: November 29, 2017
Accepted: August 19, 2018
Published: October 31, 2018
Copyright: ©2018 Shafiei et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original
author and source are credited.
Data Availability Statement: All relevant data are
within the paper and its Supporting Information
Funding: Roswell Park Alliance Foundation. The
funders had no role in study design, data collection
and analysis, decision to publish, or preparation of
the manuscript.
Competing interests: The authors have declared
that no competing interests exist.
“autonomous” stage, where the trainee can perform automatically without putting much
thought, meanwhile paying more attention towards other aspects of surgery [1,6,7].
Several key brain functional features have been proposed for evaluating surgical skills profi-
ciency, including kinematics and motor control. Hand movement kinematics has been the
main source of information in various motor skill assessment studies [812]. Completion
time, distance by hand, total distance traveled, speed, curvature, and relative phase of two
hands trajectories as well as tool-based metrics of tool out of view, tools collision, tissue dam-
age, camera control, number of critical errors, and clutch usage have been proposed for RAS
skill evaluation [1315]. However, tool-based metrics evaluate surgical skills based on static
measurements and do not represent any dynamic effect and are not able to assess the associ-
ated change in brain functioning and brain neuroplasticity. Still, there is controversy about the
optimal method for cognitive analysis during acquisition of RAS skills [6,1619]. Recording
brain neural oscillations of surgeons while interacting with robots, utilizing electroencephalo-
gram (EEG) neurofeedback, has been proposed to analyze functional dynamics of the brain of
surgeons working in laparoscopy compared to those working in RAS [19]. EEG cognitive met-
rics, such as mental workload and level of engagement, have been used to analyze brain activity
improvement during RAS learning [11,19,20]. However, skill-specific structured feedback to
trainees could not be provided. It has been shown that modular training, defined as decon-
struction of a procedure into smaller modules, can transfer technically challenging surgical
skills in a step-wise fashion under appropriate supervision. Also, a structured feedback would
be more constructive, as it allows tailoring of training activities and focusing on individual
steps [21,22].
High performance level in multifaceted tasks depends upon several factors including the
learner’s ability to develop perceptual, cognitive, and motor skills [5]. The human brain is a
complex system that includes various subsystems which interact with each other and dynami-
cally change over different temporal scales while interacting with changes in the environment.
It has been shown that during motor-cognition skill acquisition, there is a change of functional
connectivity throughout areas of the brain [23,24].
In this study, we hypothesized that the brain functional states reconfiguration during RAS
skill acquisition can be informative toward monitoring surgeon’s performance progress during
training. We analyzed the features extracted from brain functional network during RAS skill
acquisition. Additionally, changes of brain states were quantified using dynamic network anal-
ysis to develop metrics for evaluation of performance in RAS. The main goal was to investigate
dynamic changes of the brain functional states, and to find whether these changes are associ-
ated with performance improvement and practice time.
EEG has been previously used in different applications including mental workload mea-
surement [25] gesture classification in computer aided design areas [26], and stress evaluation
[18]. Here, EEG data were used to extract the brain functional network of RAS surgeon
involved in different surgical tasks.
Architecture of learning
We first sought to address the question: “Are there sets of RAS surgeon’s brain areas that pref-
erentially interact with one another during RAS tasks practice and learning?” To answer this
question, we examined the behavior of a module allegiance matrix (MAM) throughout learn-
ing (sessions). Values of elements of this matrix (MAM
) indicate the probability that two
areas (channels i and j) be assigned to the same community, in the set of functional brain net-
works constructed from all subjects and recordings. MAM matrices during six sessions and
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 2 / 19
different frequency bands were represented in Fig 1 (Data available in S1 File). These architec-
tures display that channels in motor area (Frontal and Central channels [27]) consistently
were assigned to the same community (recruited). The same thing happened to channels in
cognitive (Prefrontal and Parietal channels [27]) and visual areas (Occipital channels [27]).
This may suggest that areas relevant to task execution (motor and visual) and cognitive control
(cognitive) are recruited consistently while other areas might only be transiently recruited.
Dynamic architecture of learning
We then sought to address the question: “Does functional contribution of these consistent
communities or their interactions change with practice during RAS learning process?” To
answer this question, we used Integration and recruitment quantities (see Methods). All sub-
jects started study without any RAS or human-robot interaction experience. During the study,
we considered practice time (seconds) from session to session as practice time or experience of
subject for the next session. It should be noted that practice time may not be the only effective
variable here, as gap between practice sessions may also affect learning.
We used correlation analysis to be able to detect significant effects of practice time on inte-
gration and recruitment quantities and also investigate relationship between these quantities
and performance level. Results of this investigation were represented in Table 1 (Data available
in S2 File).
Significant correlation between performance level and motor-cognitive integration during
the βfrequency band may indicate at higher performance levels (expertise improvement) ten-
dency of channels in motor and cognitive modules to be integrated together decreases. It may
suggest that autonomy of motor and cognitive modules increases to process information inde-
pendently from each other, during βfrequency band, by performance improvement. Beta
waves are characteristics of a strongly engaged mind, able to perform complex mental process-
ing [28]. Hence, improvement of motor and cognitive autonomy at βfrequency band (while
brain is strongly engaged), by performance, may be an informative metric for performance
Significant correlation between practice time and motor-cognitive integration during θfre-
quency band (unconscious band [28]), and motor-visual integration during γfrequency band
(crucial frequency range for self-awareness and insight, cognition, and coordinating simulta-
neous processing throughout brain areas [28]), and visual-cognitive integration during all fre-
quency bands indicates that autonomy of these modules increases by practice. These features
may also be a good measurement for performance evaluation.
Dynamic architecture after long gap in learning
We also sought to address the question: “Are the change in brain functioning permanent or
does it deteriorate again?” Since our recording sessions were not regular and were limited to 6
sessions and also gap between practices was not equal for all sessions, to answer this question,
we found the correlation between practice gap and integration and recruitment, represented
in Table 2 (Data available in S2 File).
This result showed that a higher practice gap was associated with lower recruitment of
motor (αand β), and visual (β) modules. This results in longer learning process. Although
architecture of RAS learning seems to be permanent (Fig 1), dynamic architecture of the brain
during learning was not permanent and it was affected by gap in practice. However, it should
be noted that more recording sessions both before gap and after gap are needed to be able to
answer this question more accurately, and to quantify the rate of deterioration and recovery.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 3 / 19
Change of RAS trainee’s brain functional network and cognitive features by
task outcome (FSRS metrics) improvement throgh individual subjects and
After investigating the dynamic architecture of learning, we also sought to address the ques-
tion: “How does cognitive and functional connectivity features change by performance
Fig 1. Architecture of RAS learning during 6 recording sessions, through frequency bands of θ,α,β, and γchannels 1–20
represent the EEG leads numbers. Channels 1–8 (motor area), channels 9–10 (Visual cortex), channels 11–16 (cognitive area), and
channels 17–20 (other areas such as temporal cortex).
Table 1. Relationship between integration and recruitment through brain areas at different frequency ranges, and average performance level and practice time for
recordings used for extraction of each module allegiance matrix through subjects and sessions. All 524 recordings during 6 sessions were considered to extract integra-
tion and recruitment values. Significant correlations (correlation >0.2 and P-value<0.05) are bolded.
Recruitment (P-value) Integration (P-value) Frequency
Motor Cognitive Visual Motor-Visual Motor-Cognitive Visual-Cognitive
Performance Level 0.56(0.24) 0.32(0.52) -0.48(0.32) 0.61(0.19) -0.58(0.22) -0.66(0.15) θ
-0.17(0.73) -0.36(0.48) -0.31(0.53) 0.28(0.57) -0.47(0.34) -0.66(0.15) α
0.18(0.73) 0.12(0.81) -0.43(0.38) 0.57(0.22) -0.84(0.03) -0.66(0.15) β
0.28(0.59) 0.53(0.27) -0.16(0.76) 0.53(0.27) -0.56(0.24) -0.66(0.15) γ
Practice time (second) -0.17(0.74) -0.20(0.70) -0.72(0.10) -0.79(0.05) -0.90(0.01) -0.89(0.01) θ
-0.72(0.10) -0.22(0.68) -0.41(0.42) 0.79(0.06) -0.77(0.06) -0.89(0.01) α
-0.45 (0.36) -0.55 (0.25) -0.73(0.09) 0.59(0.21) -0.68(0.13) -0.89(0.01) β
-0.30(0.55) 0.34(0.50) -0.46(0.35) 0.85(0.03) -0.81(0.05) -0.89(0.01) γ
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 4 / 19
improvement during RAS skill acquisition?” To answer this question, we calculated the corre-
lations between Fundamental Skills of Robotic Surgery (FSRS)metrics and cognitive and func-
tional connectivity features through all subjects and recordings (Table 3; Data available in S3
Positive correlation between CT and most FSRS metrics is in agreement with results in the
literature [20] because higher FSRS scores are associated with lower performance level and
Table 2. Relationship between integration and recruitment of brain areas and average practice gap for recordings which were used for ‘module allegiance matrix’
extraction. All 524 recordings during 6 sessions were considered to extract integration and recruitment values. Significant correlations (correlation >0.2 and P-
value<0.05) are bolded.
Recruitment (P-value) Integration (P-value) Frequency
Motor Cognitive Visual Motor-Visual Motor-Cognitive Visual-Cognitive
Practice Gap -0.53(0.28) -0.16 (0.78) -0.43(0.39) 0.59(0.20) -0.76(0.07) -0.72(0.10) θ
-0.81(0.04) 0.39 (0.43) -0.07(0.89) 0.75(0.08) -0.65(0.15) -0.72(0.10) α
-0.87 (0.02) -0.65 (0.15) -0.85(0.02) 0.2(0.70) -0.39(0.44) -0.72(0.10) β
-0.78(0.06) 0.03(0.94) -0.06(0.90) 0.76(0.07) -0.44(0.38) -0.72(0.10) γ
Table 3. Relationship between FSRS metrics, CT, and extracted EEG features (cognition and network) at different frequency bands. Twenty-seven subjects per-
formed 5 tasks on robot simulator during six sessions, number of total recording was 524. However, Robotic Surgery Simulator (Ross) scores were not reported for some
recordings. Total number of recordings considered in this correlation analysis was 260. Significant correlations (correlation >0.2 and P-value<0.05) are bolded.
Left Tool
Left-Tool out of
# of-Errors (P-
Right-Tool Grasp (P-
Right-Tool Out-
CT 0.54
Distraction 0.16
LE 0.11
HE -0.10
MW -0.15
APA -0.13
AI -0.01
Strength θ-0.18
Strength α-0.19
Strength β-0.29
Strength γ-0.33
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 5 / 19
higher CT indicates lower expertise level and performance, too. Also, we detected significant
negative correlations between network features (strength and communication) and FSRS met-
rics, which may suggest strength and communication as informative features for performance
evaluation during RAS learning.
Relationship between difficulty level, complexity level, practice time, and
cognitive and brain network features and FSRS metrics
Although all designed tasks were considered for RAS learning, the complexity level (evaluated
by expert surgeons through FSRS curriculum) of tasks were different. Also, difficulty level
(evaluated by subjects not necessarily expert) calculated by using NASA-TLX scores showed
that subjects evaluate tasks very differently. This likely stems from their ability (e.g. intelli-
gence) as subjects are different. We then sought to address the question: “Do difficulty level
and complexity level affect the outcome of subjects while performing tasks?” To answer this
question, we calculated the correlation between these quantities and FSRS metrics (Table 4;
Data available in S3 File). Results showed that performance level decreased (FSRS metrics
increased) by increase of difficulty level. Our previous studies showed that NASA-TLX scores
given by trainee are not reliable for expertise level assessment as we could not find any signifi-
cant correlation between individual NASA-TLX metrics and expertise level (assessed by expert
RAS surgeons) [20,29], maybe because trainee doesn’t have enough proficiency to consider all
necessary factors in evaluation [6,20,29].
The results in Table 4 may suggest that although scores given to individual NASA-TLX are
not reliable for expertise level assessment [6,20,29], difficulty level feature (calculated by using
some of these metrics) can be an informative feature toward performance evaluation during
RAS skill acquisition.
In this study we addressed the hypothesis that continuous practice during human-robot inter-
action (RAS skill acquisition) results in reconfiguration of brain functional states. We recorded
EEG data from 27 subjects when performing four RAS surgical tasks, selected from FSRS
curriculum focusing on human-machine interface skills and one advanced RAS task, on a sim-
ulator over one year. We used network analysis algorithms to extract functional states and
investigate the change of RAS trainees’ brain network dynamics at different frequency ranges.
We found that motor, cognitive, and visual areas formed three separate, functionally cohesive
Table 4. Correlations between task difficulty level, complexity level, and FSRS features at different frequency
bands. Twenty-seven subjects performed 5 tasks on RoSS simulator during six sessions, number of total recording was
524. However, Ross scores are not reported for some recordings. Total number of recordings considered in this corre-
lation analysis was 260. Significant correlations (correlation>0.2 and P-value<0.05) were bolded.
Difficulty level
Complexity level
Clutch usage 0.20 (1x10
)0.003 (0.96)
Left Tool Grasp 0.34 (1.03x10
)-0.005 (0.93)
Left-Tool out of view 0.19 (2x10
) 0.02 (0.74)
# of Errors 0.38 (1.31x10
) 0.43 (4.47x10
Right-Tool Grasp 0.33 (5.74x10
)0.09 (0.14)
Right-Tool Out-of-View 0.27 (6.74x10
)0.05 (0.41)
Tissue Damage 0.22 (3x10
)0.14 (0.02)
Tool-Tool Collision 0.04 (0.49) -0.08 (0.17)
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 6 / 19
modules whose recruitment did not change by practice but whose integration changed at dif-
ferent frequency bands as performance became more automatic.
Since learning RAS skills is very complicated and improvement of performance and rate of
learning depends on several factors including subject intelligence (we didn’t include this factor
in our study), higher practice time may not be necessarily associated with higher performance
level. Therefore, we investigated change of integration between motor, cognitive, and visual
modules by performance improvement and practice time, separately. We observed that
motor-cognitive intergration at βband decreased by performance improvement, while motor-
visual and also cognitive-visual modules did not display significant changes by performance
improvement. This result may introduce the motor-cognitive integration level at βfrequency
band as informative feature to objectify RAS performance evaluation and also to determine
the rate of learning RAS skills for individual subjects (trainees).
However, integration between all three modules showed significant changes, at different
frequency bands, by practice. These results may present autonomy improvement of motor
task executive centers (motor and visual) and cognitive control center (cognitive module) by
practice, to process information more independently and more efficiently. Also, integration
between motor-cognitive (θ), motor-visual (γ), and cognitive-visual (all frequencies) can be
proposed as informative features to estimate practice time each individual trainee needs to
learn RAS skills.
We also investigated whether long gap in practice would affect brain network dynamic
architecture. Results showed that dynamic architecture of brain functioning may not be per-
manent and be affected by factors such as practice gap. Decrease of motor module recruitment
(at αand β) and visual module recruitment (at β) with practice gap may point out that for a
shorter RAS learning curve, trainees need to practice regularly and continuously. Hence, the
deterioration of brain functioning changes as well as individual recovery rate should be consid-
ered in RAS learning curve. Since recordings in this study were limited to 6 sessions, we were
not able to investigate the deterioration and recovery rate for individual subjects and also the
effective factors on these variables. However, it seems change of integration and recruitment of
motor and visual modules can provide useful information in this regard.
In addition to RAS learning architecture and its dynamic, we also explored change of brain
functional network and cognitive features by outcome improvements, and the relationship
between difficulty level, task complexity level, practice time, and cognitive and brain network
features as well as FSRS metrics. Although we didn’t observe any significant relationship
between cognitive features and FSRS metrics, detected significant correlation between strength
and communication features and several FSRS metrics may suggest strength and communica-
tion as informative features to evaluate RAS outcome performance level. Results also showed
that higher task complexity level is associated with higher number of erros, indicating lower
performance level. Also, we found difficulty level was positively correlated with clutch usage,
left tool grasp, number of errors, right tool grasp, right tool out of view, and tissue damage.
This result indicates that higher difficulty level is associated with lower performance level.
Results retrieved from correlation analysis showed that both task complexity level and diffi-
culty level are effective variables on surgical outcome and performance level.
Our results highlighted several important opportunities to study RAS skill acquisition, cog-
nitive neuroscience of learning, rehabilitation, and human-robot interaction challenges. Exist-
ing methods of objective RAS skill evaluation, in brain functioning analysis framework, are
limited to cognitive features, extracted by using power spectral density analyses, which are
unable to accurately depict detailed dynamic changes of brain functioning during learning.
Use of brain functional network provides access to neurophysiological processes that open
new insights into understanding learning and skill acquisition.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 7 / 19
Experimental set up and data recording
We designed an experimental set up, where Electroencephalography (EEG) data from 27 sub-
jects was recorded during six sessions throughout one year of practice, performing five RAS
tasks (Fig 2). The study was conducted in accordance with relevant guidelines and regulations,
and were approved by Roswell Park Cancer Institute Institutional Review Board (IRB: I-
241913). Each subject provided written informed consent before participating.
Subjects. Twenty-seven medical students without any experience in robotic surgery and
human-robot interaction. Participant characteristics were summarized in Table 5.
Surgical tasks. Simulation based robotic curriculum—fundamental skills of robotic sur-
gery (FSRS)—is a virtual reality-based curriculum that contains 4 modules (orientation, motor
skills, basic surgical skills, and intermediate surgical skills) and 16 tasks [30]. Five surgical
Fig 2. Illustration of experimental set up and schematic of RAS tasks included in this study to acquire skills related to human-
machine interface. A 20-channel EEG headset was used to record trainee’s brain activity while performing tasks. Four surgical tasks
were designed using Fundamental Skills of Robotic Surgery (FSRS) curriculum: (A) Instrument control task, (B) Placement Task,
(C) Spatial control II task, (D) Fourth arm tissue dissection. (E) Hands-on surgical task performed by subjects on Robotic Surgery
Simulator (RoSS).
Table 5. Demographics.
Participant characteristics Characteristics options Number of participants
Age, years <30 12
30–45 15
Dominant Hand Right 25
Left 2
Gender Male 17
Female 10
Simulator Experience (and gaming) No experience 27
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 8 / 19
tasks, selected from FSRS curriculum focusing on human-machine interface skills, were
included in this study:
1. Instrument Control Task (Fig 2A). This is task one of module one of FSRS and is designed
for subject knowledge improvement. During this task subjects learn how to move the arms,
which helps orient the user to the feel of the Robotic Surgery Simulator (RoSS) console [31].
2. Ball Placement Task (Fig 2B). This task involves subtasks 5 and 6 of module 2 of FSRS cur-
riculum and is designed to teach psychomotor skills. The trainee is presented with five balls
and five columns. Subjects pick up the balls from the tray using the robot simulator tools
and place them on top of the columns. The task is completed when a ball is placed on top of
every column [31]. Competence of the subjects for this task is evaluated using measures like
the time taken to complete the task (Completion Time—CT), the number of times tool-tool
collision occurs, and the number of times the camera is used and the clutch pedal is pressed
3. Spatial Control II Task (threading string through a series of hoops) (Fig 2C). This task
includes subtasks 7 and 8 of module 2 of FSRS curriculum and is designed to teach psycho-
motor skills. The subject passes a thread through a series of rings to hone his/her spatial
awareness, instrument control and fine motor skills.
4. Fourth Arm Tissue Retraction (Fig 2D). This task is third task of module one of FSRS cur-
riculum and is in the level of intermediate surgical skills [31]. This task combines the train-
ee’s previously acquired skills and requires coordinated control of the 4
arm to retract
tissue [31].
5. Hands-on Surgical Training (HoST) Anastomosis (Fig 2E). This task is designed to teach
advanced RAS skills, and is completely different from the other four tasks. In HoST, a simu-
lator has been developed using the hand movements of a master surgeon, and the trainee’s
hands follow the master’s hand motion. It feels like a master surgeon holds the hands of the
trainee and accompanies him/her throughout the performance. When the trainee’s hand
movement is not correct, the program pushes his/her hands to the correct path.
FSRS scores were not available for cases that trainee could not finish task during maximum
allowed time. In these cases, RoSS system automatically closed the task without reporting the
FSRS scores (260 recordings out of 524). Number of recordings for each task and session were
reported in Table 6.
Twenty-seven subjects started learning robot-assisted surgery from the pre-novice level, and
during one year of practice they learnt required skills to do the basic tasks of RAS on the
Table 6. Number of total recordings and number of recordings with FSRS scores, while 27 RAS trainees performed 5 tasks during 6 sessions with various number
of repeatitions.
Task Session 1 Session 2 Session 3 Session 4 Session 5 Session 6
Total cases With FSRS Total cases With FSRS Total cases With FSRS Total cases With FSRS Total cases With FSRS Total cases With FSRS
1 21 15 22 13 22 14 17 16 13 10 7 7
2 24 17 22 12 22 12 17 14 13 10 7 7
3 32 8 27 11 23 9 17 11 12 7 7 3
4 22 17 20 11 15 13 17 13 13 9 7 5
5 122 0 118 0 94 0 85 0 61 0 35 0
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 9 / 19
simulator. EEG data recording was carried out at the initial session and followed up after one
week, one month, three months, six months, and one year.
EEG data was recorded by placing the channels sensors of a 20-channel EEG headset on the
Frontal (F; cognition and action; F3, Fz, F4, F7, F8 electrode channels), Prefrontal (PF; cogni-
tion; Fp1, Fp2 electrode channels), Central (C; action; C3, Cz, C4 electrode channels), Tempo-
ral (T; perception; T3, T4, T5, T6 electrode channels), Parietal (Pa; cognition; P3, Pz, P4, Poz
electrode channels), and Occipital (O; perception; O1, O2 electrode channels) cortices (Fig 2).
All surgical tasks were done on RoSS system.
EEG data preprocessing. The EEG data from the channels was filtered with a band-pass
filter (0.5–128 Hz). Eye blinks, muscle activity, and environmental effects were considered as
artifacts. Environmental artifacts were removed by applying a 60 Hz notch filter to EEG data
[32,33]. Muscle activity and eye movement were detected using wavelet transform and dis-
criminant function analyses (DFA) as proposed by Berka et. al [33]. Linear discriminant func-
tion analysis was applied to raw data to decontaminate it from eye blinks. Wavelet coefficients
from FzPOz and CzPOz channels were used to classify each signal data points into eye-blink
or non-eye-blink categories [32]. The eye-blink data points were removed from signal.
Brain functional network and community detection
A continuous wavelet transform with the ‘morlet’ wavelet method was used to extract the
spontaneous phase of each channel signal for all recordings. The calculated phases of pair sig-
nals were used to find phase synchronized time points. These points were used to extract
phase synchronization index for pairs of signals [34]. Calculating the phase difference
(t)) between two signals x(t) and y(t) by the following equation, the phase synchronized
point was defined as time point (t) in which dðDφðtÞÞ
dt ¼0[35].
DφxyðtÞ ¼ jφxðtÞ  φyðtÞj
Transferring the range of phase into boundary of φ
2[π,π], the phase difference for all
pairs of channels was normalized by using the range of phase difference (Dφmax
xy ¼2pand
xy ¼0) [35]. The synchronization index Γ
(FB) was calculated through frequency band
(FB) using [35,36]
GxyðFBÞ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
x;yðtÞÞ2þ ½X
where, P
is the number of data-points in the time series. The calculated synchronization index
for all pairs of channels forms the symmetric weighted adjacency matrix (Γ) [35]. The single-
layer community detection algorithm was used to extract functional communities of record-
ings for all subjects through learning sessions. Communities were extracted through (4–60 Hz)
with frequency bands of θ(4-8Hz), α(8-12Hz), β(12-35Hz), and γ(35-60Hz).
Single-layer community detection algorithm was used to detect communities of each adja-
cency matrix, which we defined these communities as functional states.
Single- layer community detection. We partitioned each adjacency matrix into commu-
nities (functional states) using modularity maximization criteria [37]. Modularity index (Q),
as a quality function, was defined as [37].
ij ½Gij gMijdðgi;gjÞ;dðgi;gjÞ ¼ 1gi¼gj
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 10 / 19
where, channel ‘i’ is assigned to cluster g
, channel ‘j’ is assigned to cluster g
, and M
is the
expected weight of the link connecting channels i and j. The coefficient γ>0 is the resolution
parameter in the multiscale modularity-maximization algorithm. Maximization of Q is equiva-
lent to partitioning the matrix into communities such that the total link weight inside the
modules is as large as possible. The generalized Louvain like “greedy” algorithm and ‘Newman-
Girvan (NG)’ null network were used for modularity maximization and community detection
[37,38]. Elements of matrix M are defined as kikj
2mfor vertices i and j, where m is defined as
2Xikiand kikj
2mis the degree of the vertices. Expression XijðGij kikj
2mÞis called the
modularity matrix.
To find the optimum consistent modularity and resolution parameter for each adjacency
matrix Γ, a range of γwas considered and ‘Q’ was calculated at each γ(using generalized Lou-
vain algorithm).
Finally, the resolution limit that resulted in maximum modularity and acceptable number
of communities (6 in this study) was selected as the optimum resolution limit for each
matrix (Fig 3). We selected the number of communities 6 because the network in this study
was small with 20 nodes and we considered 6 brain predefined cognitive systems. Considering
more than 6 communities resulted in several singular communities, which do not convey
meaningful information toward our purpose.
Module-allegiance matrix, brain functional network measurements
Using the functional community each channel was assigned to, the Module Allegiance Matrix
(MAM) was derived. Element (i,j) in the MAM represents the probability that nodes ‘i’ and ‘j’
belong to the same community (Fig 4).
Comparison between module allegiance matrix and predefined brain structural system was
used to extract brain state reconfiguration measurements. Considering the structural connec-
tivity of considered channels (represented in Fig 5), the integration and recruitment coeffi-
cients were calculated.
Features. Different categories of features were measured (NASA-TLX metrics and FSRS
metrics) and calculated (cognitive and functional network features) to find the relationship
between subjective assessment metrics (NASA-TLX), real outcome measures (FSRS metrics
Fig 3. Illustration of selecting optimum modularity value.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 11 / 19
Fig 4. Schematic of module allegiance matrix extraction process. Functional connectivity values were used in adjacency matrix,
which were then used in multilayer community detection algorithm. Extracted functional communities for all recordings were used
to extract MAM.
Fig 5. Comparison between the structural community of the channels and the MAM was used to extract the integration
coefficient, and recruitment coefficient.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 12 / 19
and completion time (CT)) and analytical features retrieved from brain activity (measured by
Subjective metrics. The trainee evaluated each procedure by giving scores to NASA-TLX
indices, with a scale of 1 to 20. This subjective skill evaluation survey has six indices: Mental
Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration [39,40].
Mental Demand (MD): Evaluates the level of mental/perceptual activity demanded to com-
plete the task.
Physical Demand(PD): Level of physical activity required to complete the task.
Temporal Demand(TD): Level of time pressure the subject feels during completing the task.
Performance Score (PS): Quality level of outcome and the level of satisfaction of doing the
task (1 means highest quality and 20 means lowest quality).
Effort (E): Evaluates how hard (mentally and physically) should the subject work to complete
the task.
Frustration: Level of negative (compared to positive) psychological emotions the subject
feels while completing the task.
Task difficulty: We used scores given to these metrics to extract task difficulty (D) level as
We did not use performance scores given by trainee because trainee does not have enough
expertise to be able to evaluate outcome of performed task.
Task complexity: FSRS curriculum modules were used to determine the complexity level
of tasks in Table 7, with 1 being simplest and 5 most complicated task.
FSRS metrics. RoSS scores given by da Vinci simulator technology were used in this
study for outcome evaluation. These metrics include: clutch usage, left tool out of view, num-
ber of errors, right tool out of view, tissue damage, and tool-tool collision. These metrics were
used in correlation analysis, to investigate the relationship between brain state measurements
and RoSS scores. These scores were also used to estimate performance level of doing each task
by subjects. Performance level was calculated as the average of scores given to all 8 FSRS met-
rics as
Performance ¼100 1
Completion time. By synchronizing the recorded EEG and the associated video of the
task, completion time was defined as the total time a trainee was performing a task. The
Table 7. Description of tasks designed in this study, and the reason for the associated skill necessity in RAS.
Task name Instument control Ball placement Spatial control II Fourth arm tissue dissection Hands-on surgical
Task ID (1) (2) (3) (4) (5)
2 4 5 3 1
Skill orientation and control of
Hand-eye-tool coordination &
depth perception & foot control
Hand-eye-tool coordination &
depth perception & foot control
orientation and control of
Motor skills
Reason Lack of tactile feedback in
RAS requires development of
this skill
Remoteness and lack of tactile
feedback requires development
of this skill
Remoteness and lack of tactile
feedback requires development
of this skill
Lack of tactile feedback in
RAS requires development of
this skill
Learn how to have
fine hand
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 13 / 19
completion time can be defined using the number of total data points in signal (N) and the
data recording sampling frequency (f
) as
Cognitive features- power spectral density. Aiming period activity: Brain activity dur-
ing short and long term decision making affects the performance while operating a procedure
[41]. The whole of hand movement is planned and figured out during long-term decision
making, which happens during the Aiming Period (AP). This period has revealed important
effects on performance and expertise level in different tasks [4143]. The brain activity
during this period is Aiming Period Activity (APA), which is the power reduction in spatially
enhanced α(8–12 Hz) and β(13–30 Hz) bands during movement [43]. It has been shown that
the APA depends on the expertise level of the subject [41].
Mental Workload: Mental workload (MW) is related to the engaged memory capacity while
performing a procedure of interest [44]. The main assumption in MW interpretation is that
each person has a relatively fixed cognitive capacity [32]. Commonly, MW refers to the portion
of a person’s total mental capacity which is loaded [45]. Mental workload was estimated using
the PSD of signals at the C3-C4, Cz-POz, F3-Cz, F3-C4, Fz-C3, and Fz-POz channels [32].
To calculate MW, we used the framework developed by the B-Alert EEG series from
Advanced Brain Monitoring (ABM) company, which has been frequently validated in different
studies [32,33,46]. Briefly, this framework calculates a baseline value of the absolute and rela-
tive power spectral variables from the C3-C4, Cz-PO, F3-Cz, Fz-C3, and Fz-PO channels dur-
ing mental arithmetic, grid location, and digit-span baseline tasks. These baselines have been
recorded from 80 healthy subjects, and are available from ABM software. A two-class quadratic
logistic discriminant function analysis (DFA) [32] was used to extract the probability of pre-
senting a high mental workload. The quadratic logistic DFA was established once for one men-
tor based on baseline data collected before surgeries.
Engagement. Engagement reflects the spatial recruitment of the brain regions in process-
ing tasks associated with decision making. These tasks include, but are not limited to, informa-
tion gathering, visual scanning, audio processing, and attention concentration on one aspect
of the environment while ignoring other distractions [32,33]. As with the calculation of MW,
we used the framework developed by the B-Alert EEG series from Advanced Brain Monitoring
(ABM) company. However, in this case, the baselines were drawn from 5 minutes of three dif-
ferent tasks (3-choice vigilance task, eyes open, and eyes closed).These baselines were recorded
from one mentor at the beginning of the whole research study. Here, the absolute and relative
PSD values of the Fz-POz and Cz-POz channels were used in a four-class quadratic logistic dis-
criminant function analysis (DFA) which returned an estimation of the engagement level [32,
33]. The range of this estimation for 1 second epoch is between 0–1 with 0.1 (sleep onset), 0.3
(distraction), 0.6 (low engagement; LE), and 0.9 (high engagement; HE).
Asymmetry index. Negative feelings such as surprise, frustration, fear, and concern have
opposing effects on the activity of the right and left lobes of the frontal cortex [47]. For each
recording, Asymmetry Index (AI) was defined as the difference between the power spectral den-
sity decrease in the left and right frontal lobes in the alpha frequency band, normalized as [47,48]
maxðaÞ  PSDF3
minðaÞ þ PSDF7
maxðaÞ  PSDF7
maxðaÞ  PSDF4
minðaÞ þ PSDF8
maxðaÞ  PSDF8
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 14 / 19
The AI was calculated as the average value over the following pairs of electrodes: (F3 and F4), and
(F7 and F8). During negative feelings, the right frontal lobe shows more intense activity (associ-
ated with lowerαpower [48]) compared to the left lobe [48] (αpower is inversely related to acti-
vation [47]).
Brain functional network features. Strength. The strength of a cognitive system was
defined as the average functional connectivity of electrodes within the system. Adjacency
matrices were used to calculate this feature.
Communication. Communication, C, between two cognitive systems k
and k
can be
defined as the average functional connectivity in electrode pairs where one electrode lies
within the first system and the second electrode lies within the second system [49]. Adjacency
Table 8. Summary of all features used in this study. Four feature categories were considered: Subjective assessment metric, tool-based metrics, cognition, and brain
functional network features.
Feature Description Category Main extraction method
Difficulty level (D) Extracted by using NASA-TLX features, evaluated by the trainee
D = MD + PD + TD + E
Subjective assessment
Complexity level Indicates the complexity of task based on assessment of expert surgeons considered in
FSRS curriculum. This feature is independent from trainee performance and outcome
and is based on the properties of the task.
FSRS curriculum FSRS curriculum
Practice time All subjects started learning without any experience of work with simulator, gaming, and
robotic surgery experience. However, during learning they practice on simulator and
their experience increases from session to session. We measured practice time of each
subject from recording to next recording and considered that (seconds of practice) as
practice time.
Direct measurement Direct measurement
Practice gap Gap between practice sessions Direct measurement Direct measurement
Clutch usage Indicates effectiveness and the level of skill in which the task is performed (economy of
motion) [50]
Tool besed metric
Real measurement by
Left tool out of view Indicates awareness of operative environment in which tasks are Performed [50] Tool besed metric
Real measurement by
Right tool out of view Indicates awareness of operative environment in which tasks are performed [50] Tool besed metric
Real measurement by
Number of errors Indicates effectiveness and the level of skill in which the task is performed (economy of
motion) [50]
Tool besed metric
Real measurement by
Tissue damage Collisions causing damage to the underlying tissue [31].
Indicates awareness of operative environment in which tasks are performed [50]
Tool besed metric
Real measurement by
Tool-Tool collision Indicates awareness of operative environment in which tasks are performed [50] Tool besed metric
Real measurement by
Performance Level Performance level was calculated as:
100 1
Tool besed metric
Real measurement by
Completion time (CT) Indicates total time task is performed Direct measurement
Aiming period activity
Indicates brain activity level during aiming period Cognition Power Spectral Density
(PSD) analysis
Mental Workload
Indicates the level of engaged memory capacity while performing task Cognition Power Spectral Density
(PSD) analysis
Engagement (E) Reflects the spatial cooperation of the brain regions in processing tasks Cognition Power Spectral Density
(PSD) analysis
Strength Indicates the level of total functional connectivity within channels in a specific cognitive
Brain functional
Pairwise phase
Communication Indicates the level of total functional connectivity between channels from different
cognitive systems
Brain functional
Pairwise phase
Integration Average probability that a brain area is in the same network community as areas from
other cognitive systems
architecture feature
Network community
Recruitment Average probability that a brain area is in the same network community as other areas
from its own cognitive system
architecture feature
Network community
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 15 / 19
matrices (Γ) were used to calculate this feature as
where, |S
| is the number of nodes in the cognitive system k, k = 1. . .6, and k
. Note that
the strength of each cognitive system can be calculated by letting k
= k
Integration coefficient. Integration of each area measures the average probability that this
area is in the same network community as areas from other systems. Module Allegiance Matri-
ces were used to calculate integration feature.
Recruitment coefficient. The recruitment coefficient for each area of the network corre-
sponds to the average probability that this area is in the same network community as other areas
from its own system. Module Allegiance Matrices were used to calculate recruitment feature.
All features used in this study were summarized in Table 8.
Current analysis frameworks are unable to describe necessary dynamic changes of brain areas
to make performance of motor cognitive skills autonomous, because of statistical and mathe-
matical limitations. In this study we used dynamic network neuroscience methods to investi-
gate dynamic reconfiguration of brain modules during RAS skill learning, effective factors on
dynamic architecture of learning, and address several questions related to RAS skill acquisi-
tion. Results also suggest integration, recruitment, strength, and communication features to be
used for objective performance evaluation.
Supporting information
S1 File. ‘Module allegiance matrix’ data through different sessions and frequency bands of
θ,α,βγ.Description file explains data.
S2 File. Recruitment, integration, total practice time for subjects up to each session (sec-
ond), gap between practice sessions, and performance level data for six sessions. Descrip-
tion file explains data.
S3 File. Features for 260 recordings used in this study. Data required for extraction of Tables
3and 4, and the description file. Description file explains data.
This work was funded by the Roswell Park Alliance Foundation.
Author Contributions
Conceptualization: Somayeh B. Shafiei.
Data curation: Somayeh B. Shafiei, Ahmed Aly Hussein, Khurshid A. Guru.
Formal analysis: Somayeh B. Shafiei.
Funding acquisition: Khurshid A. Guru.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 16 / 19
Methodology: Somayeh B. Shafiei, Ahmed Aly Hussein.
Validation: Ahmed Aly Hussein.
Writing – original draft: Somayeh B. Shafiei.
Writing – review & editing: Somayeh B. Shafiei, Ahmed Aly Hussein, Khurshid A. Guru.
1. Lalle
´e S, Hamann K, Steinwender J, Warneken F, Martienz U, Barron-Gonzales H, et al., editors. Coop-
erative human robot interaction systems: IV. Communication of shared plans with Naïve humans using
gaze and speech. Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference
on; 2013: IEEE.
2. Shafiei SB, Doyle ST, Guru KA, editors. Mentor’s brain functional connectivity network during robotic
assisted surgery mentorship. Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th
Annual International Conference of the; 2016: IEEE.
3. Lalle
´e S, Lemaignan S, Lenz A, Melhuish C, Natale L, Skachek S, et al., editors. Towards a platform-
independent cooperative human-robot interaction system: I. perception. Intelligent Robots and Systems
(IROS), 2010 IEEE/RSJ International Conference on; 2010: IEEE.
4. Lalle
´e S, Pattacini U, Boucher JD, Lemaignan S, Lenz A, Melhuish C, et al., editors. Towards a plat-
form-independent cooperative human-robot interaction system: Ii. perception, execution and imitation
of goal directed actions. Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Confer-
ence on; 2011: IEEE.
5. Williams AM, Hodges NJ. Skill acquisition in sport: Research, theory and practice: Routledge; 2004.
6. Shafiei SB, Hussein AA, Guru KA. Cognitive learning and its future in urology: surgical skills teaching
and assessment. Current opinion in urology. 2017; 27(4):342–7.
0000000000000408 PMID: 28445190
7. Reznick RK, MacRae H. Teaching surgical skills—changes in the wind. New England Journal of Medi-
cine. 2006; 355(25):2664–9. PMID: 17182991
8. Shafiei SB, Cavuoto L, Guru KA, editors. Motor Skill Evaluation During Robot-Assisted Surgery. ASME
2017 International Design Engineering Technical Conferences and Computers and Information in Engi-
neering Conference; 2017: American Society of Mechanical Engineers.
9. Nisky I, Hsieh MH, Okamura AM, editors. A framework for analysis of surgeon arm posture variability in
robot-assisted surgery. Robotics and Automation (ICRA), 2013 IEEE International Conference on;
2013: IEEE.
10. Teleoperation AVDR. Uncontrolled Manifold Analysis of Arm Joint Angle Variability During Robotic Tel-
eoperation and Freehand Movement of Surgeons and Novices. 2013.
11. Shafiei SB. Investigation of Brain Computer Interface as a New Modality in Human-Surgical Robot Inter-
action: State University of New York at Buffalo; 2018.
12. Shafiei SB, Guru KA, Esfahani ET, editors. Using two-third power law for segmentation of hand move-
ment in robotic assisted surgery. ASME 2015 International Design Engineering Technical Conferences
and Computers and Information in Engineering Conference; 2015: American Society of Mechanical
13. Judkins TN, Oleynikov D, Stergiou N. Objective evaluation of expert and novice performance during
robotic surgical training tasks. Surgical endoscopy. 2009; 23(3):590.
008-9933-9 PMID: 18443870
14. Lin HC, Shafran I, Yuh D, Hager GD. Towards automatic skill evaluation: Detection and segmentation
of robot-assisted surgical motions. Computer Aided Surgery. 2006; 11(5):220–30.
3109/10929080600989189 PMID: 17127647
15. Verner L, Oleynikov D, Holtmann S, Haider H, Zhukov L. Measurements of the level of surgical exper-
tise using flight path analysis from da Vinci robotic surgical system. Stud Health Technol Inform. 2003;
94:373–8. PMID: 15455928
16. Shafiei S, Fiorica T, Hussein A, Ahmed Y, Muldoon S, Guru K. PD41-08 Skill acquisition and its reten-
tion after simulation-based practice during robot-assisted surgery: Can functional brain states help us
forge forward? The Journal of Urology. 2017; 197(4):e810.
17. Shafiei S, Hussein A, Kozlowski J, Ahmed Y, Muldoon S, Guru K. PD46-02 Looking for your own reflec-
tion: Assessing brain functional state of surgical mentor during robot-assisted surgery. The Journal of
Urology. 2017; 197(4):e890.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 17 / 19
18. Shafiei S, Hussein A, Ahmed Y, Kozlowski J, Guru K. MP51-05 DOES TRAINEE PERFORMANCE
2017; 197(4):e695.
19. Bocci T, Moretto C, Tognazzi S, Briscese L, Naraci M, Leocani L, et al. How does a surgeon’s brain
buzz? An EEG coherence study on the interaction between humans and robot. Behavioral and Brain
Functions. 2013; 9(1):14.
20. Guru KA, Shafiei SB, Khan A, Hussein AA, Sharif M, Esfahani ET. Understanding cognitive perfor-
mance during robot-assisted surgery. Urology. 2015; 86(4):751–7.
2015.07.028 PMID: 26255037
21. Volpe A, Ahmed K, Dasgupta P, Ficarra V, Novara G, van der Poel H, et al. Pilot validation study of the
European Association of Urology robotic training curriculum. European urology. 2015; 68(2):292–9. PMID: 25454612
22. Hussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, et al. Development and validation of an
objective scoring tool for robot-assisted radical prostatectomy: prostatectomy assessment and compe-
tency evaluation. The Journal of urology. 2017; 197(5):1237–44.
100 PMID: 27913152
23. Sosnik R, Hauptmann B, Karni A, Flash T. When practice leads to co-articulation: the evolution of geo-
metrically defined movement primitives. Experimental Brain Research. 2004; 156(4):422–38. https:// PMID: 15167977
24. Schaal S, Ijspeert A, Billard A. Computational approaches to motor learning by imitation. Philosophical
Transactions of the Royal Society B: Biological Sciences. 2003; 358(1431):537–47.
25. Aghajani H, Garbey M, Omurtag A. Measuring Mental Workload with EEG+ fNIRS. Frontiers in human
neuroscience. 2017; 11:359. PMID: 28769775
26. Shafiei SB, Esfahani ET, editors. Aligning Brain Activity and Sketch in Multi-Modal CAD Interface.
ASME 2014 International Design Engineering Technical Conferences and Computers and Information
in Engineering Conference; 2014: American Society of Mechanical Engineers.
27. Wu C, Liu Y. Queuing network modeling of driver workload and performance. IEEE Transactions on
Intelligent Transportation Systems. 2007; 8(3):528–37.
28. Jemmer P. Getting in a (brain-wave) state through entrainment, meditation and hypnosis. Hypnotherapy
Journal. 2009; 2:24–9.
29. Hussein AA, Shafiei SB, Sharif M, Esfahani E, Ahmad B, Kozlowski JD, et al. Technical mentorship dur-
ing robot-assisted surgery: a cognitive analysis. BJU international. 2016; 118(3):429–36.
10.1111/bju.13445 PMID: 26864145
30. Stegemann AP, Ahmed K, Syed JR, Rehman S, Ghani K, Autorino R, et al.Fundamental skills of robotic
surgery: a multi-institutional randomized controlled trial for validation of a simulation-based curriculum.
Urology. 2013; 81(4):767–74. PMID: 23484743
31. Raza SJ, Froghi S, Chowriappa A, Ahmed K, Field E, Stegemann AP, et al. Construct validation of the
key components of Fundamental Skills of Robotic Surgery (FSRS) curriculum—a multi-institution pro-
spective study. Journal of surgical education. 2014; 71(3):316–24.
10.006 PMID: 24797846
32. Berka C, Levendowski DJ, Lumicao MN, Yau A, Davis G, Zivkovic VT, et al. EEG correlates of task
engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and envi-
ronmental medicine. 2007; 78(5):B231–B44.
33. Berka C, Levendowski DJ, Cvetinovic MM, Petrovic MM, Davis G, Lumicao MN, et al. Real-time analy-
sis of EEG indexes of alertness, cognition, and memory acquired with a wireless EEG headset. Interna-
tional Journal of Human-Computer Interaction. 2004; 17(2):151–70.
34. Quiroga RQ, Panzeri S. Principles of neural coding: CRC Press; 2013.
35. Jamal W, Das S, Maharatna K, Pan I, Kuyucu D. Brain connectivity analysis from EEG signals using
stable phase-synchronized states during face perception tasks. Physica A: Statistical Mechanics and
its Applications. 2015; 434:273–95.
36. Shafiei SB, Hussein AA, Guru KA. Relationship between Surgeon’s Brain Functional Network Reconfig-
uration and Performance Level During Robot-assisted Surgery. IEEE Access. 2018.
37. Bazzi M, Porter MA, Williams S, McDonald M, Fenn DJ, Howison SD. Community detection in temporal
multilayer networks, with an application to correlation networks. Multiscale Modeling & Simulation.
2016; 14(1):1–41.
38. Jutla IS, Jeub LG, Mucha PJ. A generalized Louvain method for community detection implemented in
39. Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of empirical and theo-
retical research. Advances in psychology. 52: Elsevier; 1988. p. 139–83.
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 18 / 19
40. Rubio S, Dı
´az E, Martı
´n J, Puente JM. Evaluation of subjective mental workload: A comparison of
SWAT, NASA-TLX, and workload profile methods. Applied Psychology. 2004; 53(1):61–86.
41. Haufler AJ, Spalding TW, Santa Maria D, Hatfield BD. Neuro-cognitive activity during a self-paced
visuospatial task: comparative EEG profiles in marksmen and novice shooters. Biological Psychology.
2000; 53(2–3):131–60. PMID: 10967230
42. Salazar W, Landers DM, Petruzzello SJ, Han M, Crews DJ, Kubitz KA. Hemispheric asymmetry, car-
diac response, and performance in elite archers. Research quarterly for exercise and sport. 1990; 61
(4):351–9. PMID: 2132894
43. Babiloni C, Del Percio C, Iacoboni M, Infarinato F, Lizio R, Marzano N, et al. Golf putt outcomes are pre-
dicted by sensorimotor cerebral EEG rhythms. The Journal of Physiology. 2008; 586(1):131–9. https:// PMID: 17947315
44. Carswell CM, Clarke D, Seales WB. Assessing mental workload during laparoscopic surgery. Surgical
innovation. 2005; 12(1):80–90. PMID: 15846451
45. Carswell CM, Lio CH, Grant R, Klein MI, Clarke D, Seales WB, et al. Hands-free administration of sub-
jective workload scales: acceptability in a surgical training environment. Applied ergonomics. 2010; 42
(1):138–45. PMID: 20630495
46. Stikic M, Johnson RR, Tan V, Berka C. EEG-based classification of positive and negative affective
states. Brain-Computer Interfaces. 2014; 1(2):99–112.
47. Tomarken AJ, Davidson RJ, Henriques JB. Resting frontal brain asymmetry predicts affective
responses to films. Journal of personality and social psychology. 1990; 59(4):791. PMID: 2254854
48. Kim M-K, Kim M, Oh E, Kim S-P. A review on the computational methods for emotional state estimation
from the human EEG. Computational and mathematical methods in medicine. 2013; 2013.
49. Shafiei SB, Hussein AA, Muldoon SF, Guru KA. Functional Brain States Measure Mentor-Trainee Trust
during Robot-Assisted Surgery. Scientific reports. 2018; 8(1):3667.
018-22025-1 PMID: 29483564
50. Chowriappa AJ, Shi Y, Raza SJ, Ahmed K, Stegemann A, Wilding G, et al. Development and validation
of a composite scoring system for robot-assisted surgical training—the Robotic Skills Assessment
Score. journal of surgical research. 2013; 185(2):561–9.
PMID: 23910887
Dynamic changes of brain functional states during surgical skill acquisition
PLOS ONE | October 31, 2018 19 / 19
... Non-technical skills are naturally hard to be measured automatically. The possibilities for automated RAMIS non-technical skill assessment are similar to traditional MIS, such as relying on physiological signals measured by additional sensors [34]. ...
... Workload can be defined with self-rating techniques, where a subject fills a questionnaire about his/her personal experience about the task workload. It is naturally a subjective technique, however, there are works in the literature which studied both subjective workload measurements and objective non-technical skill assessment metrics [32,40], or objective physiological parameters [30,34,[41][42][43][44][45][46]. Workload measurements do not only help to assess the personal workload index, but also to define the main stressors and disturbing factors in surgery in general, furthermore, to provide personal training for novices as well. ...
... NASA-TLX is a widely used technique for workload measurement in aviation, military and healthcare. NASA-TLX can be found in traditional MIS mental workload estimation [48][49][50][51][52], and employed in the case of surgical robotics workload assessment as well [8,32,34,40,41,43,45, MRQ estimates workload with 17 items, and it is specifically useful for multitasking workload measurements [77]. SSSQ is based on DSSQ, and both target stress measurement [86], such as CITS [84]. ...
Full-text available
BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome’s point of view is clearly indicated by the available surgical error statistics.
... We partitioned each adjacency matrix, calculated for every second of each recording, into communities (functional states) using the multilayer modularity maximization criteria [33,34]. The expression of a community refers to the phenomenon where brain regions assigned to the same community are more likely to be strongly connected to one another as compared to regions assigned to different communities [35]. ...
... A generalized Louvain-like "greedy" algorithm and Newman-Girvan (NG) null network were used for modularity maximization and community detection [33,34]. A con-Brain Sci. ...
... Recruitment refers to the average probability that a channel is in the same network community as other channels from its own area. The average recruitment for all channels within each area is considered a feature assigned to each area and recording [33]. regions assigned to the same community are more likely to be strongly connected to one another as compared to regions assigned to different communities [35]. ...
Full-text available
Objective: The aim of this work was to examine (electroencephalogram) EEG features that represent dynamic changes in the functional brain network of a surgical trainee and whether these features can be used to evaluate a robot assisted surgeon's (RAS) performance and distraction level in the operating room. Materials and methods: Electroencephalogram (EEG) data were collected from three robotic surgeons in an operating room (OR) via a 128-channel EEG headset with a frequency of 500 samples/second. Signal processing and network neuroscience algorithms were applied to the data to extract EEG features. The SURG-TLX and NASA-TLX metrics were subjectively evaluated by a surgeon and mentor at the end of each task. The scores given to performance and distraction metrics were used in the analyses here. Statistical test data were utilized to select EEG features that have a significant relationship with surgeon performance and distraction while carrying out a RAS surgical task in the OR. Results: RAS surgeon performance and distraction had a relationship with the surgeon's functional brain network metrics as recorded throughout OR surgery. We also found a significant negative Pearson correlation between performance and the distraction level (-0.37, p-value < 0.0001). Conclusions: The method proposed in this study has potential for evaluating RAS surgeon performance and the level of distraction. This has possible applications in improving patient safety, surgical mentorship, and training.
... Our understanding is that this is the first study taking this fact into consideration. Functional brain network reconfigures by practice throughout skill acquisition [62]. Although subjects have different levels of expertise, all of them had required skills to perform considered gestures on the patient, with high performance and under the supervision of a master surgeon in the OR. ...
Full-text available
Surgical gestures detection can provide targeted, automated surgical skill assessment and feedback during surgical training for robot-assisted surgery (RAS). Several sources including surgical videos, robot tool kinematics, and an electromyogram (EMG) have been proposed to reach this goal. We aimed to extract features from electroencephalogram (EEG) data and use them in machine learning algorithms to classify robot-assisted surgical gestures. EEG was collected from five RAS surgeons with varying experience while performing 34 robot-assisted radical prostatectomies over the course of three years. Eight dominant hand and six non-dominant hand gesture types were extracted and synchronized with associated EEG data. Network neuroscience algorithms were utilized to extract functional brain network and power spectral density features. Sixty extracted features were used as input to machine learning algorithms to classify gesture types. The analysis of variance (ANOVA) F-value statistical method was used for feature selection and 10-fold cross-validation was used to validate the proposed method. The proposed feature set used in the extra trees (ET) algorithm classified eight gesture types performed by the dominant hand of five RAS surgeons with an accuracy of 90%, precision: 90%, sensitivity: 88%, and also classified six gesture types performed by the non-dominant hand with an accuracy of 93%, precision: 94%, sensitivity: 94%.
... Indeed, residents and surgeons' performance can be influenced by the interplay of their attributes (e.g., expertise, stress reactivity; Berguer and Smith, 2006;Shafiei et al., 2018), task (e.g., novel, activities that challenge different modalities, codes, and stages; Wickens, 2002), and context (e.g., laparoscopic, robot surgery, open surgery; Smith et al., 2003). Thus, it is crucial that several dimensions of surgeon capabilities and limitations, outside of task performance, be determined to understand the surgeon's performance envelope. ...
Full-text available
Sensory feedback, which can be presented in different modalities-single and combined, aids task performance in human-robotic interaction (HRI). However, combining feedback modalities does not always lead to optimal performance. Indeed, it is not known how feedback modalities affect operator performance under stress. Furthermore, there is limited information on how feedback affects neural processes differently for males and females and under stress. This is a critical gap in the literature, particularly in the domain of surgical robotics, where surgeons are under challenging socio-technical environments that burden them physiologically. In the present study, we posited operator performance as the summation of task performance and neurophysiological cost of maintaining that performance. In a within-subject design, we used functional near-infrared spectroscopy to assess cerebral activations of 12 participants who underwent a 3D manipulation task within a virtual environment with concurrent feedback (visual and visual + haptic) in the presence and absence of a cognitive stressor. Cognitive stress was induced with the serial-7 subtraction test. We found that while task performance was higher with visual than visual + haptic feedback, it degraded under stress. The two feedback modalities were found to be associated with varying neural activities and neural efficiencies, and these were stress-and gender-dependent. Our findings engender further investigation into effectiveness of feedback modalities on males and females under stressful conditions in HRI.
Objective To develop an algorithm for objective evaluation of distraction of surgeons during robot-assisted surgery (RAS). Materials and Methods Electroencephalogram (EEG) of 22 medical students was recorded while performing five key tasks on the robotic surgical simulator: Instrument Control, Ball Placement, Spatial Control II, Fourth Arm Tissue Retraction, and Hands-on Surgical Training Tasks. All students completed the Surgery Task Load Index (SURG-TLX), which includes one domain for subjective assessment of distraction (scale: 1-20). Scores were divided into low (score 1-6, subjective label: 1), intermediate (score 7-12, subjective label: 2), and high distraction (score 13-20, subjective label: 3). These cut-off values were arbitrarily considered based on a verbal assessment of participants and experienced surgeons. A Deep Convolutional Neural Network (CNN) algorithm was trained utilizing EEG recordings from the medical students and used to classify their distraction levels. The accuracy of our method was determined by comparing the subjective distraction scores on SURG-TLX and the results from the proposed classification algorithm. Also, Pearson correlation was utilized to assess the relationship between performance scores (generated by the simulator) and distraction (Subjective assessment scores). Results The proposed end-to-end model classified distraction into low, intermediate, and high with 94%, 89%, and 95% accuracy, respectively. We found a significant negative correlation (r=-0.21; p=0.003) between performance and SURG-TLX distraction scores. Conclusions Herein we report, to our knowledge, the first automated method to assess and quantify distraction while performing robotic surgical tasks on the robotic simulator, which may improve patient safety. Validation in the clinical setting is required.
Full-text available
Objective Investigations into surgical expertise have almost exclusively focused on overt behavioral characteristics with little consideration of the underlying neural processes. Recent advances in neuroimaging technologies, for example, wireless, wearable scalp-recorded electroencephalography (EEG), allow an insight into the neural processes governing performance. We used scalp-recorded EEG to examine whether surgical expertise and task performance could be differentiated according to an oscillatory brain activity signal known as frontal theta—a putative biomarker for cognitive control processes. Design, setting, and participants Behavioral and EEG data were acquired from dental surgery trainees with 1 year (n=25) and 4 years of experience (n=20) while they performed low and high difficulty drilling tasks on a virtual reality surgical simulator. EEG power in the 4–7 Hz range in frontal electrodes (indexing frontal theta) was examined as a function of experience, task difficulty and error rate. Results Frontal theta power was greater for novices relative to experts (p=0.001), but did not vary according to task difficulty (p=0.15) and there was no Experience × Difficulty interaction (p=0.87). Brain–behavior correlations revealed a significant negative relationship between frontal theta and error in the experienced group for the difficult task (r=−0.594, p=0.0058), but no such relationship emerged for novices. Conclusion We find frontal theta power differentiates between surgical experiences but correlates only with error rates for experienced surgeons while performing difficult tasks. These results provide a novel perspective on the relationship between expertise and surgical performance.
Full-text available
The current methods of surgical performance in robot-assisted surgery are subjective. In this study, we propose a cognitive-based objective method to be used for objectifying performance evaluation. Changes in brain functional networks were measured and their relationship with performance level was investigated. We used electroencephalogram data recorded from mentor surgeon’s brain during supervising and performing surgical tasks of varying complexity [Urethrovesical Anastomosis (UVA) and Lymph-node Dissection (LND)]. Multilayer community detection techniques were used to extract functional network communities at frequency bands of θ , α , lower β , upper β , and γ . Results showed different detected communities during supervising and performing LND. However, for UVA some of the functional communities were similar. In simpler tasks, trainees’ performance was close to the mentor’s. Entropy and power distribution through frequency bands confirmed minimum thermodynamic stability occurs at α frequency band. The most thermodynamically stable state occurred during γ frequency band. Time to reach stable state for channels with high entropy level was extracted as brain functional metrics at thermodynamic stability state. These metrics may be used to quantify changes of brain functional network as performance improves.
Full-text available
Mutual trust is important in surgical teams, especially in robot-assisted surgery (RAS) where interaction with robot-assisted interface increases the complexity of relationships within the surgical team. However, evaluation of trust between surgeons is challenging and generally based on subjective measures. Mentor-Trainee trust was defined as assessment of mentor on trainee's performance quality and approving trainee's ability to continue performing the surgery. Here, we proposed a novel method of objectively assessing mentor-trainee trust during RAS based on patterns of brain activity of surgical mentor observing trainees. We monitored the EEG activity of a mentor surgeon while he observed procedures performed by surgical trainees and quantified the mentor's brain activity using functional and cognitive brain state features. We used methods from machine learning classification to identity key features that distinguish trustworthiness from concerning performances. Results showed that during simple surgical task, functional brain features are sufficient to classify trust. While, during more complex tasks, the addition of cognitive features could provide additional accuracy, but functional brain state features drive classification performance. These results indicate that functional brain network interactions hold information that may help objective trainee specific mentorship and aid in laying the foundation of automation in the human-robot shared control environment during RAS.
Conference Paper
Full-text available
Remote manipulation during robot-assisted surgery requires proficiency in perception, cognition, and motor skills. We aim to understand human motor control in remote manipulation of robotic surgical instrument and attempt to measure motor skills. Three features, smoothness, normalized jerk score, and two-thirds power law coefficient, estimating the motor skills of surgeons were analyzed. These features were calculated through segments, extracted from continuous end-effector trajectories during suturing, knot-tying, and needle-passing surgical tasks, performed by 8 right-handed subjects on bench-top models using da vinci surgical kit control system. Each subject repeated each task five times. Totally 1567 segments were extracted, 413, 437, and 717 segments performed by experts, intermediates, and novice subjects, respectively. Dynamic change of kinematic properties was analyzed to evaluate the relationship between considered features and motor skill level. Results show smoothness is significantly correlated with normalized jerk score and both features are significant measures of expertise levels. Also, results show the power law is violated by many end-effector trajectories and there is no relationship between obeying two-thirds power law, smoothness and jerk. We conclude trajectory is improved from non-smooth and jerky in novices to smooth in expert surgeons. This property may be used for motor skill evaluation. It is unlikely that obeying two-thirds power law be a valid property of all end-effector trajectories. However, power law coefficient may be a discriminant feature for levels of expertise. The results are also applicable in a home-based monitoring platform, to monitor motor functioning improvement of stroke patients during rehabilitation process.
Full-text available
We studied the capability of a Hybrid functional neuroimaging technique to quantify human mental workload (MWL). We have used electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) as imaging modalities with 17 healthy subjects performing the letter n-back task, a standard experimental paradigm related to working memory (WM). The level of MWL was parametrically changed by variation of n from 0 to 3. Nineteen EEG channels were covering the whole-head and 19 fNIRS channels were located on the forehead to cover the most dominant brain region involved in WM. Grand block averaging of recorded signals revealed specific behaviors of oxygenated-hemoglobin level during changes in the level of MWL. A machine learning approach has been utilized for detection of the level of MWL. We extracted different features from EEG, fNIRS, and EEG+fNIRS signals as the biomarkers of MWL and fed them to a linear support vector machine (SVM) as train and test sets. These features were selected based on their sensitivity to the changes in the level of MWL according to the literature. We introduced a new category of features within fNIRS and EEG+fNIRS systems. In addition, the performance level of each feature category was systematically assessed. We also assessed the effect of number of features and window size in classification performance. SVM classifier used in order to discriminate between different combinations of cognitive states from binary- and multi-class states. In addition to the cross-validated performance level of the classifier other metrics such as sensitivity, specificity, and predictive values were calculated for a comprehensive assessment of the classification system. The Hybrid (EEG+fNIRS) system had an accuracy that was significantly higher than that of either EEG or fNIRS. Our results suggest that EEG+fNIRS features combined with a classifier are capable of robustly discriminating among various levels of MWL. Results suggest that EEG+fNIRS should be preferred to only EEG or fNIRS, in developing passive BCIs and other applications which need to monitor users' MWL.
Full-text available
Purpose of review: The aim of this study is to provide an overview of the current status of novel cognitive training approaches in surgery and to investigate the potential role of cognitive training in surgical education. Recent findings: Kinematics of end-effector trajectories, as well as cognitive state features of surgeon trainees and mentors have recently been studied as modalities to objectively evaluate the expertise level of trainees and to shorten the learning process. Virtual reality and haptics also have shown promising in research results in improving the surgical learning process by providing feedback to the trainee. Summary: 'Cognitive training' is a novel approach to enhance training and surgical performance. The utility of cognitive training in improving motor skills in other fields, including sports and rehabilitation, is promising enough to justify its utilization to improve surgical performance. However, some surgical procedures, especially ones performed during human-robot interaction in robot-assisted surgery, are much more complicated than sport and rehabilitation. Cognitive training has shown promising results in surgical skills-acquisition in complicated environments such as surgery. However, these methods are mostly developed in research groups using limited individuals. Transferring this research into the clinical applications is a demanding challenge. The aim of this review is to provide an overview of the current status of these novel cognitive training approaches in surgery and to investigate the potential role of cognitive training in surgical education.
Introduction: and Background: Comprehensive training and skill acquisition by urologic surgeons are vital to optimize surgical outcomes and patient safety. We sought to develop and validate an objective and procedure specific tool to assess the quality of robot-assisted radical prostatectomy (RARP)-Prostatectomy Assessment and Competence Evaluation (PACE) METHODS: Development and content validation of PACE was performed by deconstruction of RARP into seven key domains utilizing the Delphi methodology. The reliability and construct validation were then assessed utilizing de-identified videos performed by practicing surgeons and fellows. Consensus for each domain was defined as achieving a content validity index (CVI) ≥0.75. Reliability was assessed using intra-class correlation (ICC) and construct validation using a mixed linear model accounting for multiple ratings on the same video. Consensus was reached after 3 rounds on wording, relevance of skills assessed, and concordance between the score assigned and the skill assessed. ICC ≥ 0.4 was achieved for all domains. The expert group outperformed trainees in all domains but reached statistical significance in bladder drop (4.5 versus 3.4, p=0.002), preparation of the prostate (4.4 versus 3.2, p<0.0001), seminal vesicles and posterior plane dissection (8.3 versus 6.8, p=0.03) and neurovascular bundle preservation (4.1 versus 2.4, p<0.0001). Limitations include the lack of assessment of other key skills as communication and decision making. Conclusions: PACE is a structured, procedure-specific and reliable tool that objectively measures surgical performance during RARP. It can differentiate different levels of expertise, and provide structured feedback to customize training and surgical quality improvement.
Conference Paper
In many complicated cognitive-motor tasks mentoring is inevitable during the learning process. Although mentors are expert in doing the task, trainee's operation might be new for a mentor. This makes mentoring a very difficult task which demands not only the knowledge and experience of a mentor, but also his/her ability to follow trainee's movements and patiently advise him/her during the operation. We hypothesize that information binding throughout the mentor's brain areas, contributed to the task, changes as the expertise level of the trainee improves from novice to intermediate and expert. This can result in the change of mentor's level of satisfaction. The brain functional connectivity network is extracted by using brain activity of a mentor during mentoring novice and intermediate surgeons, watching expert surgeon operation, and doing Urethrovesical Anasthomosis (UVA) procedure by himself. By using the extracted network, we investigate the role of modularity and neural activity efficiency in mentoring. Brain activity is measured by using a 24-channel ABM Neuro-headset with the frequency of 256 Hz. One mentor operates 26 UVA procedures and three trainees with the expertise level of novice, intermediate, and expert perform 26 UVA procedures under the supervision of mentor. Our results indicate that the modularity of functional connectivity network is higher when mentor performs the task or watches the expert operation comparing mentoring the novice and intermediate surgeons. At the end of each operation, mentor subjectively assesses the quality of operation by giving scores to NASA-TLX indexes. Performance score is used to discuss our results. The extracted significant positive correlation between performance level and modularity (r = 0.38, p - value <; 0.005) shows the increase of automaticity and decrease in neural activity cost by improving the performance.