Boosting physical and psychological well-being in
rehabilitation through cognitive technologies
Department of Philosophy
Università degli Studi di Milano
Department of Philosophy
Università degli Studi di Milano
Abstract-Orthopaedic rehabilitation is a hot topic since the
increasing age of the population in western countries implies that
more and more people require invasive and inabilitating
orthopaedic surgery, such as knee substitution, in order to
recover physical functionality, autonomy and quality of life.
Consequently, it’s fundamental to test rehabilitation treatment
able to increase efficacy and reliability of treatment, both with
the aim to improve outcomes and decrease days of hospital stay.
In this sense, Neuro-cognitive driven technology may have a great
impact in this field. Starting by the paradigm of Action
Observation Treatment, we have designed a pilot study using a
3D environment and wearable bio-sensors to boost rehabilitation
in collaboration with an important orthopaedic hospital in Milan,
Italy. We hypothesized that the use of a 3D environment would
decrease the time required for the recovery of motor
functionality with respect to a 2D environment or to a standard
treatment. We sampled 26 patients who accepted to participate in
the study. They were randomly assigned to one of the three arms
of the study. Preliminary data suggest that actually the Action
Observation Treament has the power to boos rehabilitation when
matched with a traditional treatment. Furthermore, the 3D video
stimulation seems to have a higher impact on cognitive and
physical variables, thus suggesting that the use of 3D stimulation
may constitute a cognitive tool to be used in hospital settings,
while 2D videos might be considered an at-home tool to be used
autonomously to maintain and further boost outcomes.
Keywords—Rehabilation; 3d environment; BCI; Orthopedics;
The ICF (International Classification of Functioning)
standard, published in 2001 by WHO (World Health
Organization), has been transposed by 191, including Italy. The
cultural revolution introduced by ICF consists in considering a
person with a disability not as the result of anatomic damage
(menomation) followed by a functional limitation (disability)
transforming him/her in a handicapped individual, but in
focusing on the individual. The ICF evaluation process
considers, for the first time, contextual environment factors
(social roles, cultural – natural or not - environment, political
factors, Insitutions, and so on) and person-related factors
(genre, age, health conditions, adaptability, social background,
education, profession, past experience, behaviours styles),
systematically classifying them through common criteria
comparable in an interdisciplinary way. The model of Health
and of disability presented by ICF is biopsychosocial model,
involving all the contexts of public policies and, particularly,
the welfare, health, education and job ones. Only an
intersectorial cooperation, jointly with an integrate approach
allow to detect solutions which reduce the disability in a
population. The focus is on the terms “health” and “function”,
not on disability. This vision of an individual, even if
introduced in 2001, the ICF actually is in contrast with the
classification systems currently used in rehabilitation. The
Major Diagnostic Category system (MDC), currently in use,
divides rehabilitation areas in orthopaedic, neurologic,
cardiology and pulmonary, strongly in cultural contrast with
the ICF vision.
Moreover, considering the Italian scenario, there is also an
evident epidemiologic datum:
“In 2010, the total number of hospitalizations in
rehabilitation has been about 321,000 patients, the 82% of
which in ordinary regimen and the remaining 18% in daily
regimen, with an incidence of the private offer component of
69%. The rehabilitation in hospitals, in Italy, is a phenomenon
involving mainly elder patients: the over 65 represent the 61%
of the total number (value constant from 2007 to 2010)
(Metodologia per la definizione dei criteri/parametri di
appropriatezza ed efficienza dei ricoveri di riabilitazione
ospedaliera - febbraio 2013 Ministero della Salute). If most of
the patients of rehabilitation are elder people, there is another
datum to consider: when over 65, we register the coexistence in
the same individual of multiple pathologies related to the age,
especially in chronic pathologies. It has been demonstrated, in
fact, that most of the people aged between 65 and 79 show 4,9
pathologies, while in individuals over 80 the number is 5,4.
The association of specific chronic pathologies such as the
organic cardiopathy and the osteoarthritis increases the relative
risk of disability of 13,6 times if compared to an isolated risk of
4,4 for the osteoarthritis and 2,3 for the organic cardiac disease.
With age, the presence of comorbidity increases due to the
increasing of frequency of the chronic pathologies.”
Considering that most of the patients are, so, mainly older
people with a variety comorbidity, the possibilities to apply the
ICF are various and, following the trends, they will be more
and more in future.
We focused specifically on orthopaedic rehabilitation,
starting from the assumption that an older patient needs not
only traditional physical therapies, but also neurological
treatments. The hypothesis, discussed with Doctors of the
Gaetano Pini Orthopaedic Institute in Milan, is that, using
cognitive technologies, we could reduce the risk of other
physical traumas possible in dispensing traditional physical
rehabilitation exercises proposing a double approach:
traditional and virtual, as better described in the following
paragraph. In this way the influence of other pathologies,
making often difficult the physical rehabilitation therapy (such
as, for example, cardiac diseases), is reduced and, at the same
time, also the pain suffered by patients is reduced. It is also
important to underline that the use of cognitive-driven
technologies may also help the patient engagement in the the
therapeutic journey with potential important reflexes on
perceived quality of life and psychological well-being.
II. THE PROJECT: TECHNOLOGICAL SETTING AND PROTOCOL
At the Orthopaedic Rehabilitation Institute Gaetano Pini, in
Milan, we constituted an interdisciplinary group composed of
Doctors specialized in Orthopaedic (but also with the
contribution of other medical specialities), Cognitive
Psychologists, Information Technology specialized in Brain
Computer Interface and Virtual reality. The group designed
and realized a virtual room where a patient can follow a special
rehabilitation program reinforced by the immersive 3D
The reasons why we chose to realize the implemented
systems come from literature and medical experience.
In neurophysiology it is now a well-accepted that the
observation of actions performed by others activates in the
perceiver the same neural circuits of the actual execution of the
same actions . In fact, observing a person performing a
meaningful motor task activates the so-called mirror neuron
system . This means that the brain of a person observing
someone performing everyday actions activates the same areas
involved in the actual execution of those actions, opening
opportunity to develop Neuro-cognitive tools in a wide range
of domains. In particular, a number of studies shown that
action observation is an effective way to enhance (or even to
learn) the performance of specific motor skills . For
instance, in a study, participants put in unusual environment
were required to perform a reaching task after watching a video
depicting a person learning to reach in the same environment or
in a different one. In the first condition, participant performed
better than participant in the second condition, suggesting that
what was learned about the environment through the actions of
others (by the video) was easily transferable to their own action
in a true environment . Moreover, action observation has
been shown to help motor learning in both normal adults and
patients after stroke . It also reported that observing action
may increase the force of performing a giving movement in a
similar way that actually training that movement does .
More recent studies also revealed that action observation is a
powerful cognitive tool for learning a novel complex motor
task that in some condition may be better than motor imagery
[7, 8]. All this evidence established the basis for a
rehabilitative technique known as “Action Observation
Treatment”, already applied in several clinical settings, e.g. the
rehabilitation of motor deficit after stroke [9, 10], Parkinson’s
disease  and motor performance in elderly people .
However, a few studies tested this rehabilitation technique in
orthopaedics. Among these, Bellelli et al.  successfully
tested the possibility to couple a conventional physiotherapy
with the Action Observation Treatment in post-surgical (hip
and knee) orthopaedic patients reporting a faster improvement
in functional measures (Tinetti scales ). Park et al. 
found that treatment based on action observation may have a
positive effect symptoms such as pain and stiffness in patients
undergoing total knee arthroplasty. Finally, Villafañe and
colleagues  found significant improvements in knee
mobility adding an action observation to a conventional
physiotherapy after primary total knee replacement with
respect to a conventional physiotherapy. In particular, in this
study authors used a self-administration protocol in which
patients were trained to watch autonomously some
rehabilitation-related video clips. This protocol in interesting
since it show that is also possible to provide patients tools that
they can use at-home. However, we argue that using a more
immersive and guide environment during hospitalization may
be more effective in helping patients to develop motor skills
that they could be able to boost at-home by the use of not
Moreover, if the motor representation is realized through
immersive 3D technologies, the result is a greater effectiveness
in therapy, thanks to the stronger identification process [15, 16,
We created a virtual room where each patient is provided of
a specific rehabilitation virtual protocol, following a 3D
immersive reproduction of the movement done during the
traditional rehabilitation therapy.
To allow freedom of movement for the patient and the
therapist who follows the rehabilitation session, the immersion
is carried out by screen projection made with double Epson
projectors in passive mode. This kind of choice allows, while
preserving the advantages of the immersivity, not to
completely isolate the patients so that they can interact with the
The first step, currently in progress, requires that a group of
patients carries out the treatment in a traditional mode and then
revise, in passive mode, the movements performed during the
traditional rehabilitation therapy, through 3D videos made with
external, lateral, front and subjective shooting.
Virtual sessions have a duration of about 20 minutes total,
and during this time, the patient wears an EEG-based BCI
(Brain Computer Interface) to detect brain responses to visual
and motor stimuli. We opted for an EEG-based BCI for greater
freedom of movement allowed by the device, compared to a
Thanks to the introduction in research of EEG-based Brain
Computer Interfaces (BCIs) , currently we can measure the
attention level of a participant in an experiment in a
comfortable situation, if compared to medical EEG devices. In
fact, BCI devices, such as the Emotiv Epoch we used, allow
freedom of movement (communicates with the server, using a
bluethoot dongle) and reduce individuals’ anxiety, thanks to
wireless collection of the EEG signal and to the ergonomy of
the device. BCIs process brainwaves into digital signals and it
is used extensively in neuroscience [16, 19], cognitive science
[20, 21, 22, 23] and cognitive psychology [24, 25] research, as
well as in games and education [26, 27] and some other
applications, such as attention training [28, 29] music training
and analysis [30, 31].
We must recall that BCIs collect several cerebral frequency
rhythms: the alpha band (7 Hz–14 Hz), related to relaxed
awareness, contemplation, etc.; the Beta band (14 Hz–30 Hz),
associated with active thinking, active attention, solving
concrete problems; the Delta band (3 Hz–7 Hz), frontally in
adults, posteriorly in children with high amplitude waves,
found during some continuous attention tasks [32, 33]; the
Theta band (4 Hz–7 Hz), usually related to emotional
processing ; the Gamma band (30 Hz–80 Hz), generally
related to cognitive processing of multi-sensorial signals. The
considered commercial BCI, the Emotiv Epoc, collects all the
listed signals and also myographic signals, allowing the use of
facial expressions and head movements (as a gyroscope).
In addition to the group of the ten patients following the
sperimental virtual therapy, we chose also a second group
composed by patients who will watch the same videos but in a
traditional 2D version, and a control group of patients who
only follow the traditional therapy, in order to compare, at the
end of the experiment, the results obtained in terms of
efficiency and recovery of motor functions.
We conducted a pilot study. Informed consent was obtained
from all participants and procedures were conducted according
to the Declaration of Helsinki. The protocol was approved by
the Local Ethical Committee. The study was performed at the
Physical and Rehabilitation Medicine department of “G. Pini”
Hospital in Milan. Patients recovered for rehabilitation after
elective primary total knee replacement were invited to
participate in the present study. Exclusion criteria were 1) age
18 years or younger and 90 years or older, 2) bilateral knee
replacement or previous total knee replacement; 4) pre-existing
motor impairment; 5) Mini-Mental State Examination score
lower than 21; 6) severe vision impairment; 8) delirium on the
admission; 9) unwillingness to participate.
All eligible patients signed written informed consent before
they entered the study. All the eligible patients who agreed to
participate in the present study were randomly assigned either
to the experimental group or to the control group.
From September 2016 and January 2017 we found 31
patients eligible for the study. Three patients refused to
participate stating that they were not interested in experimental
treatments. Other two elegible patients stopped participating
after the first video clip experience, since they found the 3D
stimulation “confusing”. The present study was then conducted
on 26 participants (mean age = 68.5; female = 16), who met the
eligibility criteria and agreed to participate, they were
randomized to the experimental group I (n=9), experimental
group II (9) or control (n=8).
All study variables were measured at baseline and after
intervention. Each patient received 30 minutes of rehabilitation
once a day, 5 days a week for 10 days. The first 30 minutes
were conventional exercises done with a physiotherapist. In the
second part of the program (some hours after the physical
therapy), patients watched a video comfortably seat on an
ergonomic chair in a dedicated room. The experimental group
1 watched a 3D video related to rehabilitation exercised
(similar to the exercised performed before with the
physiotherapist); the second group watched the same video, but
in a traditional 2D version; the control group watched a video-
clip containing relaxing images of landscapes or documentaries
in 2D format.
Data on pain intensity, active and passive range of motion,
functional status, comorbidity burden, psychological distress
and quality of life were recorded at the admission and soon
after the treatment.
The following scales were administered: Visual Analogue
Scale, active range of motion, passive range of motion, Barthel
index , Tinetti scale , Tellegen Absorption Scale ,
Hospital Anxiety and Depression scale (HADs; ). We also
recorded EMG(The mean EMG activity was then computed as
a percentage of the mean EMG activity recorded from the same
muscles during a five-second maximal isometric contraction of
the knee extensor muscles) and frontal EEG (we sampled 30
seconds of the EEG signal at the beginning of the video and
each 5 minutes and we compared the band power with a
We didn’t find any significant difference in psychological
distress between groups, though we find improvements in
anxiety F (1,24) = 3.125; p = 0.34) and depression (F (1,24) =
7.339; p =.022) as measured by the HAD in all conditions. This
trend may be due to a normal adjustment process, potentially
boosted by the engagement in the study and/or the video
stimulation. Though the pain, as measured by the visual
analogical scale, was significantly reduced in all subjects
independent of the group, other subjective variables measured
by analogical scales after the completion of the study (during
the debriefing) showed significant differences. In particular,
patients in group 1 and 2 reported higher values than patients in
group 3 on self-perceived physical and emotional well-being.
They also reported to be more confident in their ability to
completely recovering their functionality after discharge from
From a psychological point of view, it is interesting to
report that the absorption trait as measure by the TAS
positively correlates with motor improvements only in group 1
(r = .398, p = 0.041).
All the groups, as expected, reported significant
improvements in knee mobility (flexion and extension) and
functional state with respect to baseline conditions. However,
experimental groups reported significant better results after 10
days of treatment in active flexion (F (1,24) = 8.819; p =.007)
and extension (F (1,24) = 6.914; p =.012) as well as on Tinetti
scale (gait assessment).
In particular, we found a significant difference between
experimental groups and the control group. The difference
between group I and group II, instead, was not statistical
However, interesting data come from EMG data. In fact,
we found that 3D rehab-related video-clips are able to recruit
involved muscle in a higher way than 2D stimulation. At the
same time, we found a higher activation in gamma band in
frontal area during 3D stimulation. These final data suggest
that 3D videos may have a higher physical and cognitive
impact on patients, potentially boosting their rehabilitation
Our findings suggest that action observation treatment, in
addition to conventional physiotherapy is a safe, usable and
effective tool to improve rehabilitative outcomes after surgery.
In fact, preliminary data show that experimental groups had a
faster recovery as detected by a significant different between
active flexion and extension of the knee. We didn’t find any
other differences between groups about the other variables
considered here. However, we didn’t analyze yet follow-up
data about functional, psychological and physical conditions
after discharge from hospital. Consequently, we cannot
comment long-term consequences of the different treatments
These data suggest that the use of a 3D video stimulation
may be more effective in supporting rehabilitation than 2D
IV. CONCLUSIONS AND FURTHER DEVEOPMENTS
Preliminary data of this pilot study are very promising. In
fact, we showed that Action Observation Treatment, as an
addition to conventional physiotherapy, might have a positive
effect on rehabilitation since it significantly improved clinical
outcomes in a short time.
If the preliminary results may seem encouraging, however,
we need a wider study, in order to consolidate the results of the
investigation and provide a clear picture about the specific
(motor function) and general (pain, quality of life, autonomy
functional and so on) impact of the treatment.
After a more extensive testing, it could be possible, in
future, for experienced staff who assists the patient during
exercises, to customize the level of difficulty and the type of
therapy, to achieve greater progress, reducing the patient's
frustration. Other tools to measure biofeedback signals will be
assessed in order to collect information for personalization of
individual therapeutic program.
In the future, through specific tools to be developed, the
same exercises could be offered by PC or tablet, so that the
intervention can also continue autonomously, at the patient's
 G. Bellelli, G. Buccino, B. Bernardini B. “Action observation treatment
improves recovery of postsurgical orthopedic patients: evidence for a
top-down effect?”, Archives of physical medicine and rehabilitation.
 G. Rizzolatti, L. Fadiga, V. Gallese. “Premotor cortex and the
recognition of motor actions”, Brain research Cognitive brain research.
 S. D. Park, H. S. Song and J. Y. Kim. “The effect of action observation
training on knee joint function and gait ability in total knee replacement
patients”, Journal of exercise rehabilitation. 2014;10:168–171.
 C. Johnson and B. Green. “Submitting manuscripts to biomedical
journals: common errors and helpful solutions”, Journal of manipulative
and physiological therapeutics, 2009;32:1–12.
 E. G. Tangalos, G. E. Smith, R. J. Ivnik. “The Mini-Mental State
Examination in general medical practice: clinical utility and
acceptance”, Mayo Clinic proceedings. 1996;71:829–837
 S. K. Inouye, C. H. van Dyck, C. A. Alessi. “Clarifying confusion: the
confusion assessment method. A new method for detection of delirium”,
Annals of internal medicine, 1990;113:941–948.
 L. A. Wei, M. A. Fearing, E. J. Sternberg. “The Confusion Assessment
Method: a systematic review of current usage”, Journal of the American
Geriatrics Society, 2008;56: 823–830.
 D. l. Judd, D. A. Dennis, A. C. Thomas. “Muscle strength and functional
recovery during the first year after THA”, Clinical orthopaedics and
related research, 2014;472:654–664.
 P. Celnik, B. Webster, D. M. Glasser. “Effects of action observation on
physical training after stroke”, Stroke; a journal of cerebral circulation.
 D. Ertelt, S. Small, A. Solodkin. “Action observation has a positive
impact on rehabilitation of motor deficit safter stroke”, NeuroImage,
2007;36 (Suppl 2):T164–T173.
 E. Pelosin, L. Avanzino, M. Bove. “Action observation improves
freezing of gait in patients with Parkinson’s disease”,
Neurorehabilitation and neural repair. 2010;24: 746–752.
 B. Tia, F. Mourey, Y. Ballay. “Improvement of motor performance by
observational training in elderly people”, Neuroscience letters.
 M. E. Tinetti. “Performance-oriented assessment of mobility problems in
elderly patients”, Journal of the American Geriatrics Society.
 J. H. Villafañe, M. Isgrò, M. Borsatti, P. Berjano, C. Pirali and S.
Negrini, S. “Effects of action observation treatment in recovery after
total knee replacement: A prospective clinical trial”, Clinical
rehabilitation, 2016, 0269215516642605.
 E. Calore, R. Folgieri, D. Gadia, and D. Marini. “Analysis of brain
activity and response during monoscopic and stereoscopic
visualization”. In IS&T/SPIE Electronic Imaging (pp. 82880M-
82880M). International Society for Optics and Photonics, 2012,
 R. Folgieri, L. Lucchiari and D. Marini. “Analysis of brain activity and
response to colour stimuli during learning tasks: an EEG
study”. IS&T/SPIE Electronic Imaging. International Society for Optics
and Photonics, California, US, 2013, 86520I-86520I.
 P. Storz, G. F. Buess, W. Kunert and A. Kirschniak. “3D HD versus 2D
HD: surgical task efficiency in standardised phantom tasks”, Surgical
endoscopy, 2012, 26(5), 1454-1460.
 B. Z. Allison, E. W. Wolpaw and J. R. Wolpaw. “Brain -computer
interface systems: progress and prospects”. Expert Review of Medical
Devices; 2007, 4, 4, 463-74.
 B. Cameli, R. Folgieri and J. P. M. Carrion. “A Study on the Moral
Implications of Human Disgust-Related Emotions Detected Using EEG-
Based BCI Devices”, In Advances in Neural Networks, 2016, pp. 391-
401. Springer International Publishing.
 C. Wang, P. Cesar and E. Geelhoed. “An Invisible Gorilla: Is It a Matter
of Focus of Attention?”, In Advances in Multimedia Information
Processing–PCM 2013 (pp. 318-326). Springer International Publishing.
 R. Folgieri, C. Lucchiari, M. Granato, D. Grechi. “Brain, Technology
and Creativity. BrainArt: A BCI-Based Entertainment Tool to Enact
Creativity and Create Drawing from Cerebral Rhythms”, In Digital Da
Vinci, 2014, pp. 65-97. Springer New York.
 R. Folgieri and R. Zampolini. “BCI promises in emotional involvement
in music and games”, Computers in Entertainment (CIE), 2014, 12(1), 4.
 L. Al-Barrak, E. Kanjo. “NeuroPlace: making sense of a place”, In:
Proceedings of the 4th Augmented Human International Conference
(AH '13) ACM, New York, NY, USA, 2013
 M. Marchesi, B. Riccò. “BRAVO: a brain virtual operator for educa-tion
exploiting brain-computer interfaces”. In: CHI '13 Extended Abstracts
on Human Factors in Computing Systems (CHI EA '13). ACM, New
York, NY, USA, 2013
 A. Banzi, R. Folgieri. “EEG-Based BCI Data Analysis On Visual-
Priming In The Context of a Museum Of Fine Arts”, In Proceedings of
DMS 2012, 18th International Conference on Distributed Multimedia
Systems, 9-11 August 2012, Miami Beach, USA
 S. Dervan, C. McCosker, B. MacDaniel, C O’Nuallain. “Educational
Multimedia. Current Developments in Technology-Assisted Education”,
In: Fourth International Conference on Multimedia and Information and
Communication Technologies in Education. Seville, Spain, 2006
 Z. Merchant, E. T. Goetz, L. Cifuentes, W. Keeney-Kennicutt, and T. J.
Davis. “Effectiveness of virtual reality-based instruction on students'
learning outcomes in K-12 and higher education: A meta-
analysis”, Computers & Education, 2014, 70, 29-40.
 J. D. Cohen, R. F. Lee, K. A. Norman and N. B. Turk-Browne. “Closed-
loop training of attention with real-time brain imaging”, Nature
neuroscience, 2015, 18(3), 470-475.
 L. Jiang, C. Guan, H. Zhang, C. Wang and B. Jiang. “Brain computer
interface based 3D game for attention training and rehabilitation”, In
Industrial Electronics and Applications (ICIEA), 2011 6th IEEE
Conference on(pp. 124-127). IEEE, 2011, June.
 R. Folgieri, M. G. Bergomi and S. Castellani. “EEG-based brain-
computer interface for emotional involvement in games through music”,
In Digital Da Vinci, 2014, pp. 205-236. Springer New York.
 R. Folgieri, M. Zichella. “A BCI-based application in music: Conscious
playing of single notes by brainwaves”. In Computer Entertainment, 10,
3, Article 1, 10 pages, 2012.
 E. Kirmizialsan, Z. Bayraktaroglu, H. Gurvit, Y. Keskin, M. Emre, T.
Demiralp.”Comparative analysis of event-related potentials during
Go/NoGo and CPT: Decomposition of electrophysiological markers of
response inhibition and sustained attention”, Brain Research 1104 (1):
 C. Lucchiari and G. Pravettoni. “Feedback related brain activity in a
gambling task: a temporal analysis of EEG correlates”, Scandinavian
journal of psychology, 51(6), 449-454, 2010.
 M. Balconi and C. Lucchiari. “Consciousness and arousal effects on
emotional face processing as revealed by brain oscillations. A gamma
band analysis”, International Journal of Psychophysiology, 67(1), 41-46,
 C. Collin, D. T. Wade, S. Davies, S. and V. Horne, V. “The Barthel
ADL Index: a reliability study”, International disability studies, 1988,
 G. A. Jamieson. “The modified Tellegen absorption scale: A clearer
window on the structure and meaning of absorption”, Australian Journal
of Clinical and Experimental Hypnosis, 2005, 33(2), 119.
 A. S. Zigmond and R. P. Snaith. “The hospital anxiety and depression
scale”, Acta psychiatrica scandinavica, 1983, 67(6), 361-370.