Sensory Substitution and Spatial Navigation in Early and Late Blind Individuals
Using a New SensoryFusion Application Installed on a Smartphone.
1 1,2 2,3 4 4 4 5 1,5
M.Bleau , S.Paré , I.Djerourou , D.Bernal , C.Knowledge , M.Piszczor , R.Kupers , M.Ptito
1 Chaire de recherche H.Sanders en Sciences de la Vision, École d'Optométrie, Université de Montréal, Québec, Canada ; 2 Département de Pharmacologie et Physiologie, Université de Montréal, Québec, Canada ; 3 Faculté des Sciences de la Vie,
Université de Strasbourg, France and Département des Sciences Biologiques, Université de Montréal, Québec, Canada ; 4 Signal Garden Inc., Silicon Valley, California ; 5 Institute of Neuroscience and Pharmacology, University of Copenhagen, Denmark.
Blind individuals often report diﬃculties while navigating in their environment and detecting
objects beyond the peri-personal space. Sensory substitution devices (SSDs) are therefore
needed to inform them about the environment via other senses like touch and audition (Kupers
& Ptito, 2004; Maidenbaum, Abboud, & Amedi, 2014).
Previous studies on SSDs like the Tongue Display Unit (TDU) (Chebat, Schneider, Kupers, &
Ptito, 2011), the Eyecane (Maidenbaum et al., 2014) and The vOIce (Auvray, Hanneton, &
O'Regan, 2007) have shown the potential of such devices for spatial navigation. Indeed, it has
been shown that the use of SSDs activates brain areas involved in vison guided spatial
navigation in blind and sighted individuals (Kupers, Chebat, Madsen, Paulson, & Ptito, 2010).
However, SSDs are rarely used due to their high cost, the intensive training they require and
the lack of training programs (Elli, Benetti, & Collignon, 2014).
Figure 1. Brain activation patterns during route
recognition using the TDU or a visual control
paradigm. (A) Results of blind participants, showing
activation of occipital and posterior parietal cortices,
precuneus, fusiform gyrus, right parahippocampus. (B)
Blindfolded sighted control subjects activated the
posterior parietal cortex and the precuneus. (C) Sighted
control subjects performing the route recognition task
visually activated the occipital and superior parietal
cortices, the precuneus, fusiform gyrus, and right
In this study, we are testing the currently in development application : SensoryFusion, using
the Signal Garden Electrolyte Engine platform installed on a smartphone. The system uses
multiple sensors to either detect obstacles at a distance directly in front of the user (Detection
mode) or to create a 3D map of the environment (Avoidance mode) and informs the user with
an auditory feedback. In addition, it has been designed to be easy to use and accessible to all
via smartphones, which are increasingly popular in the targeted population (Due, Kupers,
Lange, & Ptito, 2017).
We tested late blind (LB), early blind (EB) individuals and sighted control (SC) for their
navigation ability in an obstacle course using the device. Our goal is to compare their
performance and learning curves to determine if the new application has potential to help
them in daily life activities.
In this study, early blind (EB, n=12), late blind (LB, n=12) and blindfolded-sighted participants (SC, n=24) were tested for their navigation ability in an obstacle course using the SensoryFusion App (Signal Garden Electrolyte Engine platform) installed on an ARCore smartphone.
Subjects characteristics. SC (n=24), LB (n=12) and EB (n=12) were studied.
# of Participant
Cause of Blindness
# of Participant
Retinopathy of Prematurity
The corridor (21 x 2.4m) contained 6
identical obstacles (0.4 x 0.4 x 2m
cardboard boxes) randomly placed across
trials but always distanced by 3m from
1 2 6
ŸDistance estimation performance
ŸTotal crossing time
Ÿ Detect obstacles by pointing at them
Ÿ Estimate the distance between them and
Ÿ Cross the corridor as quickly as possible.
ŸCross the corridor as quickly as possible
ŸAvoid as many obstacles as possibles
Ÿ·Total crossing time
SC LB EB
SC LB EB
SC LB EB
Each group had a similar performance
in the detection task: SC=79,86%,
LB=72,22% and EB=80,34%. ANOVAs
did not ﬁnd any signiﬁcant diﬀerence
For obstacles in center of the corridor,
performance was 82,84% for SC,
87,75% for LB and 83,26% for EB. For
obstacles to the wall, performance was
77,19% for SC, 58,33% for LB and
77,73% for EB.
Only LB had their performance
aﬀected by the position of obstacles
(p<0.05). However, they did not diﬀer
from SC and LB
Eﬀect of obstacles position.
Across trials, the average time to cross
the obstacle course decreased for all 3
gr oup s (C S= 58. 6s , LB =43 .4 s,
Only SC and EB showed a signiﬁcant
improvement in their total crossing
Each group had a similar performance
S C = 8 5 , 9 5 % , L B = 8 8 , 6 1 % a n d
EB=84,03%. ANOVAs did not ﬁnd any
signiﬁcant diﬀerence ( F=0.687
Average of total crossing time for 2 ﬁrst trials, 2 middle trials and 2
end trials for each group in avoidance task.
Across trials, the total crossing time is higher for SC compared to
LB and EB. Furthermore, LB seem to be intermediate between
SC and EB. Total crossing time of SC always diﬀered from LB
(p<.01) and EB individuals (p<.05), but that LB and EB never
diﬀered from each other.
A device has to be assessed according to 3 principles
1) The function for which it was designed
All participants were able to use the application with less than 30 minutes of
training, giving evidence of its ease of use. Furthermore, performance and
learning are similar in SC, LB and EB participant in Detection mode, but in
Avoidance mode, crossings are faster for LB and EB.
2) The beneﬁts that the user gets
According to subjective reports, the application allows to detect obstacles at a
distance and to avoid them, then could even help ﬁnding landmarks. The
Avoidance mode is judged the most relevant in daily life: blind tell that
detecting the totality of obstacles is useless since they search to avoid
obstacles directly in their way. Furthermore, head sweeping can aﬀect
balance and their ability to walk in a straight line.
3) The extra coverage going beyond the capabilities of the white cane
According to participants, the system is interesting mostly for obstacles above
pelvis and thus would be a good complement to the long cane which don't
allow this type of detection (Suterko, 1967). Furthermore, it increases the
detection distance up to 3 meters in front of the person. On the other hand, the
application couldn't be used alone because of the diﬃculty to detect variation
on ground surface. But if this aspect and the comfort is improved, blind
individual could use it alone.
1) New obstacles course which reproduced more real situation like doors,
pole, sidewalk/stair and big obstacles under pelvis.
2) Other studies about Eyecane and TDU in the old and new obstacle course
and a review
3) Cerebral imagery to see the activated brains areas involved while using the
• Auvray, M., Hanneton, S., & O’Regan, J.K. (2007). Learning to perceive with a visuo-auditory substitution system:
Localisation and object recognition with ‘the Voice’. Perception, 36(3),416-430.
• Chebat, D.R., Schneider, F.C., Kupers, R., & Ptito, M. (2011). Navigation with a sensory substitution device in congenitally
blind individuals. Neuroreport, 22(7), 342-347. doi: 10.1097/WNR.0b013e3283462def
• Due, B. L., Kupers, R., Lange, S., & Ptito, M. (2017). Technology enhanced vision in blind and visually impaired individuals.
Working Papers on Interaction and Communication, 1-31.
• Elli, G. V., Benetti, S., & Collignon, O. (2014). Is there a future for sensory substitution outside academic laboratories?.
Multisensory research, 27(5-6), 271-291.
• Kupers, R., & Ptito, M. (2004). “Seeing” through the tongue: cross-modal plasticity in the congenitally blind. Paper
presented at the International Congress Series.
• Maidenbaum, S., Abboud, S., & Amedi, A. (2014). Sensory substitution: closing the gap between basic research and
widespread practical visual rehabilitation. Neuroscience & Biobehavioral Reviews, 41, 3-15.
• Maidenbaum, S., Hanassy, S., Abboud, S., Buchs, G., Chebat, D. R., Levy-Tzedek, S., & Amedi, A. (2014). The "EyeCane", a
new electronic travel aid for the blind: Technology, behavior & swift learning. Restor Neurol Neurosci, 32(6), 813-824.
• Suterko, S. (1967). Long cane training: Its advantages and problems. Paper presented at the Proceedings of the Conference
for Mobility Trainers and Technologists.
• Wiener, W. R., Welsh, R. L., & Blasch, B. B. (2010a). Foundations of orientation and mobility (Vol. 1): American Foundation
for the Blind.
• Wiener, W. R., Welsh, R. L., & Blasch, B. B. (2010b). Foundations of orientation and mobility (Vol. 2): American Foundation