Conference PaperPDF Available

3D sound interactive environments for problem solving


Abstract and Figures

Audio-based virtual environments have been increasingly used to foster cognitive and learning skills. A number of studies have also highlighted that the use of technology can help learners to develop affective skills such as motivation and self-esteem. This study presents the design and usability of 3D interactive environments for children with visual disabilities to help them to solve problems related with the Chilean geography and culture. We introduce AudioChile, a virtual environment that can be navigated through 3D sound to enhance spatiality and immersion throughout the environment. 3D sound is used to orientate, avoid obstacles, and identify the position of diverse personages and objects within the environment. We have found during usability evaluation that sound can be fundamental for attention and motivation purposes during interaction. Learners identified and differentiated clearly environmental sounds to solve everyday problems, spatial orientation, and laterality.
Content may be subject to copyright.
3D Sound Interactive Environments for Problem Solving
Jaime Sánchez & Mauricio Sáenz
Department of Computer Science
University of Chile
Blanco Encalada 2120, Santiago, CHILE
{jsanchez, msaenz}
Audio-based virtual environments have been increasingly used to
foster cognitive and learning skills. A number of studies have also
highlighted that the use of technology can help learners to develop
affective skills such as motivation and self-esteem. This study
presents the design and usability of 3D interactive environments for
children with visual disabilities to help them to solve problems
related with the Chilean geography and culture. We introduce
AudioChile, a virtual environment that can be navigated through 3D
sound to enhance spatiality and immersion throughout the
environment. 3D sound is used to orientate, avoid obstacles, and
identify the position of diverse personages and objects within the
environment. We have found during usability evaluation that sound
can be fundamental for attention and motivation purposes during
interaction. Learners identified and differentiated clearly
environmental sounds to solve everyday problems, spatial
orientation, and laterality.
Categories and Subject Descriptors: J.0 General.
General Terms: Human Factors.
Keywords: Hyperstories, 3D sound, virtual world, problem
solving, role-playing game.
Children with visual disabilities are increasingly having more
contact with virtual environments [2, 4, 7, 8, 9, 10, 18, 19].
Computer games have been used for entertaining purposes through
accessibility interfaces.
Few initiatives have focused on the design of games to develop and
rehearse cognitive skills in children with visual disabilities [3, 11].
Applications conceived for users with visual disabilities have been
developed using auditory information as the main output channel
and haptic devices for input [1, 7, 8, 10, 12]. These systems have
been principally developed to help blind people to overcome their
difficulties with standard interfaces such as the Web page "reader"
Jaws. Other focus is the development of 3D audio interfaces used to
develop the user's skills to recognize spatial environments through
sound. However, we have no record of research work on virtual
environments to solve problems by children with visual disabilities.
Sound has also been used as input/output to develop learning and
cognitive skills in blind children. Most applications use spatial
sound to help children to develop abstract and short-term memory,
haptic perception, collaborative skills, spatial abstraction,
mathematics skills, and algorithmic thinking skills [9, 10, 13, 14, 17,
An experience with audio stimuli to simulate visual cues for blind
learners [8] have found that 3D audio interfaces help blind people to
localize specific points in a 3D space concluding that navigating
virtual environments through sound can be a precise task for blind
people. Other studies describe positive effects of 3D audio-based
virtual environments [2].
Authors have used sensory virtual environments through force
feedback joysticks simulating real world places such as the school
concluding that providing appropriate spatial information through
compensatory channels the performance of blind people can be
improved [4].
A research work replicated a traditional computer game such as
space invader by using 3D sound. Authors used force feedback
joysticks as input interface by letting to play blind to sighted
children to share the same experience [7].
Authors evaluated the skill to hold in mind a specific localization
without concurrent perceptual information or spatial update through
the use of audio stimuli to trace specific places through sound [5].
A study to design and evaluate a spatial audio system that models
the acoustic response of a closed environment with varying sizes
and textures for blind user was implemented concluding that there
was almost no difference in user perception of room sizes between
sounds in real and simulated scenes [3].
Finally, a real-time 3D graphic game for low vision gamers was
developed to propose general methods of making real-time 3D
games accessible for virtual reality applications, e-commerce, and
distance learning [19].
Studies to enhance problem solving skills through 3D sound in blind
children are scarce. This research study presents the design,
development, and usability testing of AudioChile a 3D sound
interactive environment for children with visual disabilities. The
system was designed to assist learners to solve everyday problems
and thus knowing the life, culture, and idiosyncrasy of different
geographical regions of Chile.
Children with visual disabilities perceive the surrounding world
differently to sighted children. Actually, to orient and mobilize in
the real world they have to learn to exploit and use other senses such
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
ASSETS’05, October 9–12, 2005, Baltimore, Maryland, USA.
Copyright 2005 ACM 1-59593-159-7/05/0010...$5.00.
as touch and hearing. To orient spatially within a surrounding
environment is a complex process that requires learning to explore
the smelling and textures as points of reference to search for objects
in the environment.
Children with visual disabilities need to learn how to use their ears
and interpret the surrounding world, localize sounds that serve as
cues to orient and mobilize autonomously. Spatial sound stimuli can
provide a stimulating atmosphere for developing thinking skills to
cope with everyday situations and solve current problems and issues
with independence.
AudioChile is a role playing game. Children have to travel through
main Chilean regions where different hyperstories occur. The
interaction is through sound and music. Within a region the child
performs a personage and navigates through virtual environments
looking for cues and information to do actions and take decisions.
At the end the child can get the whole picture and information about
the adventure.
The main purpose of this study is to design 3D virtual environments
based on hyperstories and assisted by stereo and spatial sound to
develop problem solving skills in children with visual disabilities.
This includes searching, mobilization, localization, designing
strategies, and orientation skills.
Our study also includes a usability evaluation of the virtual
environment. We have studied: 1. 2D and 3D graphic interfaces for
children with residual vision, 2. Stereo and spatial sound interfaces,
and 3. Problem solving strategies developed when using
AudioChile is based on a model for implementing educational
software for children with visual disabilities [15,16]. Basically, the
model is centered on providing facilities for evaluation purposes and
to give prompt feedback to the user. It also clarifies, for
implementation purposes, similarities and differences between
software for people with and without visual disabilities. The
resulting architecture model after applying the developing model is
depicted in Figure 1.
AudioChile has a model (metaphor) that represents three major
Chilean cities. A personage has to navigate through these cities
searching for cues and information to solve a posed problem.
Editors consist of a virtual world and the definition of its properties,
objects, and personages.
The knowledge representation model is based on the mapping of
real world behaviors in a virtual world, state variables (physics,
kinetics, and luminous), and actions.
Figure 1. Architecture of the resulting software [15]
The strategy (AI) is in charge of following up the actions performed
by the user. It provides a compass that perceives the user´s
orientation within the virtual world, helping him/her to navigate and
take actions.
The learner model is such that as the user goes through the
adventure she/he can know and process more information to solve
the problem in order to reach the end of the story.
Evaluation consists of an evaluation of the compass to provide
information to the user to find diverse objects and personages within
the virtual world.
Projection consists of 3D images and sound virtual worlds. It
explores diverse fine representation qualities to get a high standard
of use as well as to solve the problem posed.
AudioChile is composed of diverse hyperstories [6]. Once immersed
in the 3D world the user can adopt a personage that can be a girl or a
boy. Each story consists of an adventure to explore one of three
geographical regions of Chile by navigating, interacting, and solving
tasks and problems. Basically, the child has to find a cooper ingot in
the region of Chiloé and return it to the region of Chuquicamata. To
do this the child has to follow directions and cues given by some of
the personages through the story. They constitute a subset of the
major problem concerning with returning the cooper ingot to
Virtual personages and objects of an adventure have been designed
for a 3D virtual world. All of them are designed according to real
characters of the geographical zone in story.
Interaction occurs through actions such as to take, give, open, push,
pull, look, speak, use, travel, check the backpack, movements, and
turns (90 /180 degrees). These actions can be done by using the
joystick forcefeedback and keyboard.
Diverse interactive elements have associated actions defined by a
matrix that crosses the user’s behavior and different elements to
represent the viability of the operation.
Through the “take” action the personage can load elements during
the adventure that can serve in future actions. To do this there is a
virtual backpack to save these elements. The backpack is an
interactive point that can be accessed and the content saved inside it
can be checked to be used in conjunction with the surrounding
All actions performed in the software such as menu actions and
actions during the story itself have an audio feedback to understand
what is happing in the story. For the menu and some actions there
are stereo sounds. To navigate through the virtual world AudioChile
uses 3D sound to get a better spatiality and immersion. Spatial
sound is provided during obstacles orientation such as the
boundaries of the labyrinth as well as the position of personages and
objects within the virtual environment. When the user performs
illegal actions for a certain element an error feedback is provided.
Interaction through the software can occur by using general and
specific planes.
AudioChile is performed in three Chilean regions: Chiloé,
Valparaíso, and Chuquicamata. Relevant zone information is
provided by hidden cues that allow users to visit and know aspects
of the geography and traditions of Chile. To travel between zones
users must attain certain objectives that can help them in futures
Virtual world navigation is delimited by labyrinths that allow
mobility and freedom to the personage within certain parameters
(can give turns of 90 and 180 degrees, and move forward). In any
moment the user can save the performance in the story or restart it
later from the same leaving point when quitting the game. Each time
the user interacts with users a log file is created to know the results
of the session. This information is used to create a story at the end of
the adventure that summarizes the performed actions through the
game and how they solved the posed problem (see Figure 2).
Figure 2. The Navigation Map
AudioChile has a compass to orient the user when searching for
interaction with different objects and personages in order to solve
the problem posed.
The software presentation allows easy access. Users with residual
vision can identify elements on the screen through the use of
contrasting colors and well defined images.
Mazes & Interfaces
AudioChile has defined mobility environments of three Chilean
cities. They consist of three different labyrinths that the user has
to navigate through the selected personage. Also, there are objects
and personages with their defined positions within the virtual
world (see Figure 3).
Figure 3. Labyrinth of three Chilean cities: (a) Chiloé, (b)
Valparaíso, and (c) Chuquicamata. (Golden liquor is a typical
liquor of Chiloé zone)
Three virtual environments represent the three Chilean cities. Each
virtual world can be navigated through interacting with textures and
special sounds. Each city consists of an introduction through audio
and a typical music of the zone. Within the travel environment there
are some representative places of different cities in order to trace the
travel through Chile (see Figure 4).
(a) (b)
(c) (d)
(e) (f)
Figure 4. Screenshot of the graphic interface of AudioChile: (a)
Personage menu; (b) Chiloé; (c) Travel; (d) Valparaíso; (e)
Chuquicamata, and (f) Option menu (save game)
The sample was consisted of 6 Chilean children with visual
disabilities, 4 boys and 2 girls ages 10 to 15 that attend a blind
school in Santiago, Chile. Three children have low vision and three
children with total blindness. Two of them are blind from birth, one
child acquired blindness during childhood, two with good residual
vision, and one with poor functional residual vision. Two special
education teachers and one usability expert also participated in the
Children participated in four testing sessions each during four
months. Each session was especially dedicated to one child in other
to allow him/her to completely walkthrough the software.
Each usability testing consisted of the following steps:
Introduction to the software. The user receives explanations about
the purpose of testing and how to use the keyboard to interact with
AudioChile. Teachers mediated to help children to orient when
using the keyboard.
Software interaction. Users have to navigate throughout the virtual
environment and according to their needs they can ask teachers for a
better orientation.
Anecdotic record. Key data and observations of the child’s
interaction with the software are registered.
Application of usability questionnaires. The user answers questions
asked by special education teachers.
Session recording through photography. Each session was
photographed to register the child behavior during interaction.
Protocol reports of the session. All the data from the child’s
interaction is registered to get comments and suggestions to improve
the software navigation.
Design and redesign. According to the comments and observations
received the software is redesigned and some new functions are
Three usability questionnaires were used during testing: 1. End-user
questionnaire, 2. Prototype interface evaluation questionnaire, and
3. Problem solving understandability questionnaire. The end-user
questionnaire was applied at the end of the usability sessions. It is
basically a software acceptance test and consists of 18 closed
statements with an answer scale of 1 to 10. It also contains 5 open
questions. The prototype interface evaluation questionnaire was
applied during the usability sessions. It is intended to evaluate
images and audio feedback by including an observation instrument
that has two parts: 1. A set of questions to identify images of
personages and objects in the software, as well as to record
observations during interaction is also included, and 2. A set of
questions to identify input/output sounds and related associations
made by blind children. It also contains observations recorded
during interaction. The problem solving understandability
questionnaire was applied during interaction and consists of a
questionnaire with 10 open questions to evaluate the
understandability of problems and tasks posed and related interface
elements such as instructions, sounds, visual and sound cues, voice,
navigation issues, and strategies to find hidden clues.
Figure 5. A blind user interacting with AudioChile
The usability testing was first implemented in March 2004. Children
pretested early prototypes of AudioChile during interaction (see
Figure 5). The objective was to have an initial feedback about the
sounds and images of the software in order to have early in the
implementation phase information to orient the final design of
interfaces. To get more detailed information we used the interface
evaluation questionnaire.
The second stage of the usability testing was implemented in April
2004 after we processed the data from the first testing and
redesigned and improved the prototype. Therefore at this stage we
had a more advanced prototype. We used the prototype interface
evaluation questionnaire.
The third stage of the usability testing was applied in two parts. We
applied the end-user questionnaire to the same sample in two
different moments: May and July. After each application we
analyzed data from open and closed questions and took decisions
about interface design/redesign. Both applications served to improve
the usability of AudioChile.
The application of the prototype interface evaluation questionnaire
gave us information about the presentation and use of the graphics
and sound interfaces. The interface for the menu to associate actions
of the personage were first planned in 2D format. These initial icons
easily understood by sighted users were not appropriated in their
representation for users with residual vision. For this reason we
changed them with more representative images of associated actions
to have a better effect on the user. This representation was made in a
3D format.
We also realized that there was a need to have sound feedback to
allow certain logic of actions. Then we associated a characteristic
sound to each action. For opposite actions we also provide the same
feedback but inversed. In the same line, we used the keystrokes that
have some relief (F, J) and those that surround them to orient easily
within the keyboard.
The design of AudioChile was specially thought for blind children.
Thus we emphasized the presentation of menus with one function at
the same time based on the concept of circular menu. This idea
came out after realizing that when designing software for sighted
children the pointer of the mouse is crucial in the interface as well as
always maintaining visibility by highlighting relevant controls and
functions. Children with visual disabilities don’t need to have a total
visibility of the menu. Actually they currently select one option and
then go through all the functions of the menu by using the keyboard.
This gives more screen space to improve the design.
The idea of mapping objects through sound according to the four
cardinal points (north, south, east, and west) worked very well for
the user allowing them to identify behaviors such as reaching to a
wall, a border of the forest, and a beach and avoiding to collide with
some elements of the virtual world.
The associated sounds to the steps of the personage in different
types of roads were modified many times to find the adequate sound
for the steps and the type of terrain where the user walked.
From the application of the problem solving understandability
questionnaire we got relevant information about the use of the
software, the understanding of main instructions, the idea of finding
a lost object, and thus understanding the whole problem. Children
used the keyboard adequately and understood that the personage can
take objects from the virtual world and save them in a backpack.
They also identified, received, and understood the information
provided by the software and applied it during the game.
Only one learner needed some mediation to navigate through the
software perhaps due to the fact that the introduction to the software
was too long. He may have lost the main objective and key
directions to use the software. This needs to be improved to get a
better mapping from users.
The application of the end-user questionnaire gave us very
important information. We present this data by contrasting the
software acceptability of children with residual vision and those
totally blind. Later we present data comparing the results of the first
and second testing.
Contrasting users
The motivation to interact with AudioChile was evaluated. To do
this we posed 6 statements in the end-user questionnaire: 1. I like
the software, 2. AudioChile is pleasant, 3. The software is
challenging, 4. AudioChile makes me to be active, 5. I would like to
play the software again, and 12. The software is motivating. The
scores were between 1 (very low) and 10 (very high).
Children with residual vision showed a great motivation to interact
with the software, with average scores of 9-10. Blind children
mentioned lower motivation but it was still high, with scores
between 7 and 9 points (see Figure 6). This is a very good result
because in general children with visual disabilities of our sample
were motivated to interact and walkthrough AudioChile.
Residual Vision
A. I like the software
B. AudioChile is pleasant
C. The software is challenging
D. AudioChile makes m e to be
E. I would like to play the software
F. The software is motivating
Figure 6. Motivation when interacting with AudioChile
A second relevant aspect is the software utilization. The statements
considered were: 9. I felt in control of the software, 11. The
software is easy to use, and 13. AudioChile adapts to my pace.
Users with residual vision did not have difficulties when using the
software (scores 8-10). Blind users had more difficulties to map the
interaction with AudioChile (scores 5-7). Controlling the software
and the easiness of use was more complex to blind children. Most
children with visual disabilities mentioned that AudioChile adapts to
their pace (scores 7-10) (see Figure 7).
Residual Vision
A. I felt in control of the software
B. The software is easy to use
C. AudioChile adapts to m y pace
Figure 7. The use of AudioChile
To evaluate the sounds used in the software we considered the
following statements: 15. I like the sounds of the software, 16. The
sound are clearly identified, and 17. The sounds of AudioChile
convey information.
As depicted in Figure 8 children with residual vision highly
accepted the sounds (score 10). Blind children also accepted the
sound of AudioChile (scores 9-10). This was a very sensitive aspect
of this study because the software relies heavily on diverse sounds
and voices.
Residual Vision
A. I like the s ounds of the
B. The sound are clearlly
C. The sounds of AudioChile
convey information
Figure 8. Perception of sounds of AudioChile
Initial and Final Testing
We also analyzed the user’s final acceptance of the software before
and after we redesigned and improved the software based on the
results and requirements detected with the prototype interface
evaluation questionnaire and the problem solving understandability
The contrasting results are displayed in Figures 9 and 10. We can
observe that motivation increased after redesigning the software
especially in software acceptability aspects such as likeness and
pleasantness of the software as well of the activeness of the user.
The acceptation of sounds increased after software redesign.
These results confirm our beliefs and experience concerning the
importance of testing and retesting the software during
implementation in order to design and redesign as much as it is
possible when designing software for children with visual
First Testing
Last Testing
A. I like the software
B. AudioChile is pl easant
C. The software is challenging
D. AudioChile m akes me to be
E. I would like to play the
software again
F. The software is motivating
Figure 9. Motivation when interacting. First and last testing
First Testing
Last Testing
I like the sounds of the
The sound are clearly
The sounds of AudioChile
conve y informati on
Figure 10. Perception of sounds of AudioChile. First and last
We have presented the design, implementation, and usability testing
of AudioChile, a 3D sound virtual environment to assist the learning
of problem solving in children with visual disabilities.
Children liked, accepted, used, and were very motivated with the
software. After designing and redesigning 3D sound interfaces they
mapped and navigated throughout the virtual environment.
Usability testing was crucial for mapping the end-user and the
understandability of the software. Children with visual disability
played a key role in the design and redesign of AudioChile by
making suggestions, comments, and answering questionnaires to
elicit information about how a 3D sound environment can map their
needs and way of thinking.
Before we planed the usability testing we thought that the main idea
was that the learner could discover events in the game without
including a lot of cues. However after the first testing we had to add
more instructions because learners needed many cues and
instructions to orient themselves in the software to make it similar to
their real environment.
When designing graphic interfaces for children with residual vision
we found a clear issue of mental modeling. For a sighted person it is
natural to see icons and associate them to certain actions. However,
children with residual vision besides having difficulties recognizing
certain icons did not associate them for the designed actions. After
the first usability testing we changed icons to 3D format to obtain
more fidelity in their representation. We also used stereo sounds that
were the opposite in their related actions. For instance, sounds for
opening were the opposite of closing sound.
The use of AudioChile has allowed children to differentiate and
identify surrounding sounds that helped them to orient spatially. It
also has helped them to improve the laterality and spatial concepts
of up and down in relation to the north and south coordinates within
a map. Sounds helped to catch the attention and motivated of
children. The contrasting colors of the interfaces were also
important for users with residual vision.
The visibility of graphic elements in the interface is also relevant.
While sighted users prefer the visibility of possible actions and rapid
access for a user with visual disabilities the main interaction device
was the keyboard and the use of circular menus. This made us to
design menus without caring about visibility by using representative
icons with higher size to have a better feedback for users with
residual vision leaving screen space to improve the representation of
the interface by these users.
Finally, AudioChile is ready to be used with children with visual
disabilities. Software design and redesign have been possible with
the participation of children with visual disabilities in a Chilean
school for blind. We envision a long term application of this
software to evaluate the impact on the development of problem
solving skills and thus helping these children to solving real life
problem through sound.
We are now working on the cognitive testing of AudioChile. We are
interested in knowing what type of problem solving strategies can
be developed by using this virtual environment as well as the paths
followed and how sound help these children to develop cognitive
strategies to solve everyday problems outside the virtual software.
This report was funded by the Chilean National Fund of Science and
Technology, Fondecyt, Project 1030158.
1. Baldis, J. (2001). Effects of spatial audio on memory,
comprehension, and preference during desktop conferences.
Proceeding of the ACM CHI ´01, Vol 3, 1, (2001), pp. 166-173
2. Cooper, M. & Taylor, M. (1998). Ambisonic sound in virtual
environments and applications for the blind people. In
Proceedings of the Second European Conference on Disability,
Virtual Reality, and Associated Technologies, ECDVRAT
1998, Skövde, Sweden, 10-11 September, 1998. pp. 113-118
3. Kurniawan, S., Sporka, A., Nemec, W., Slavik, P. (2004).
Design and users evaluation of a spatial audio system for blind
users. Proceedings of The 5th International Conference on
Disability, Virtual Reality and Associated Technologies,
ICDVRAT 2004 , September 20-22, Oxford, United Kingdom,
pp. 175-182
4. Lahav, O. & Mioduser, D. (2000). Multisensory virtual
environment for supporting blind persons’ acquisition of
spatial cognitive mapping, orientation, and mobility skills. In
Proceedings of the Third International Conference on
Disability, Virtual Reality and Associated Technologies,
ICDVRAT 2000, Sardinia Italia, 23-25 September, 2000. pp.
5. Loomis, J., Lippa, Y., Klatzky, R. & Golledge, R. (2002).
Spatial updating of locations specified by 3-D sound and
spatial language. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 28(2), pp. 335-345. 2002
6. Lumbreras, M & Sánchez, J. (1997). Hyperstories: A model to
specify and design interactive educational stories. En R. Baeza
(Editor). Computer Science, USA: Editorial IEEE Computer
Society, Los alamitos, California. (1997), pp. 135-146
7. McCrindle, R. & Symons, D. (2000). Audio space invaders.
Proceedings of the Third International Conference on
Disability, Virtual Reality and Associated Technologies,
ICDVRAT 2000, Sardinia Italia, 23-25 September, 2000. pp.
8. Mereu, S. & Kazman R. (1996). Audio enhanced 3D interfaces
for visually impaired users. Proceedings of CHI ‘96, ACM
Press. (1996)
9. Sánchez J., Flores, H. (2004). AudioMath: Blind children
learning mathematics through audio. Proceedings of The 5th
International Conference on Disability, Virtual Reality and
Associated Technologies, ICDVRAT 2004 , September 20-22,
Oxford, United Kingdom, pp. 183-189
10. Sánchez, J. (2001). Interactive virtual acoustic environments
for blind children. In Proceedings of ACM CHI ´2001, pp. 23-
25. Seattle, Washington, USA, March 31 – April 5, 2001.
11. Sánchez, J. (1999). Interactive 3D sound hyperstories for blind
children In Proceedings of ACM CHI ´99, (1999). Pittsburg,
PA. pp. 318-325
12. Sánchez, J. (2003). AudioBattleShip: Blind learners
collaboration through sound. In Proceedings of ACM CHI ´03,
Fort Lauderdale Florida. pp. 798-799.
13. Sánchez, J., Aguayo, F. (2005). Blind learners programming
through audio. Published in the Proceedings of ACM CHI
2005. Portland, Oregon, April 2-7, 2005, pp. 1769-1772
14. Sánchez, J., Baloian, N. & Hassler, T. (2004). Blind to sighted
children interaction through collaborative environments.
Lecture Notes in Computer Science 3198, Springer-Verlag, pp.
15. Sánchez, J., Baloian, N. (2005). Modeling audio-based virtual
enviroments for children with visual disabilities. Proceedings
of the World Conference on Educational Multimedia,
Hypermedia & Telecommunications ED-MEDIA 2005, pp.
16. Sánchez, J., Baloian, N., Flores H. (2004). A methodology for
developing audio-based interactive environments for learners
with visual disabilities. Proceedings of the World Conference
on Educational Multimedia, Hypermedia &
Telecommunications ED-MEDIA 2004, Lugano, Switzerland ,
(June 21-26, 2004), pp. 124
17. Sánchez, J., Flores, H. (2004). Memory enhancement through
audio. Published in the Proceedings of The Sixth International
ACM SIGACCESS Conference on Computers and
Accessibility, Assets 2004, Atlanta, Giorgia, USA, October 18-
20, pp. 24-31
18. Sánchez, J., Sáenz, M. (2005). Developing mathematics skills
through audio interfaces. To be published in the Proceedings of
11th International Conference on Human-Computer
Interaction, HCI 2005. Las Vegas, Nevada, USA, July 22-27,
19. Westin, T. (2004). Game accessibility case study: terraformers
– a real – time 3D graphic game. Proceedings of The 5th
International Conference on Disability, Virtual Reality and
Associated Technologies, ICDVRAT 2004, September 20-22,
Oxford, United Kingdom, pp. 95-10
... Le rythme a également été utilisé comme moyen de saisir des commandes via des séquences de gestes de tapotement [63] ou avec des micro-gestes exécutés à un tempo donné [57] ou pour la sélection de cibles en suivant des trajectoires à une vitesse donnée avec gestes en mid-air [36,166]. Dans les interfaces utilisateurs 3D, les guides sonores ont été principalement étudiés pour aider les utilisateurs malvoyants à trouver des cibles dans un environnement 3D [114,146]. ...
Full-text available
Récemment, sont apparus de nouveaux casques grand public de réalité virtuelle (RV) aux capacités comparables à ceux utilisés en laboratoire. L'usage de ces technologies est devenu plus courant et les sessions d'utilisation sont devenues plus longues (jeux vidéo, expression artistique, travail à distance, sport, rééducation...). Durant ces temps d'interaction, les utilisateurs ont leurs bras dans les airs parfois sans repos possible. Aussi, ce type d'interaction dans l'air, qualifié de ``mid-air'', est connu pour provoquer de la fatigue au niveau des bras que l'on nomme ``effet Bras de Gorille'' en interaction humain-machine (IHM). Comprendre ce phénomène pour la conception des applications en RV devient un élément essentiel afin d'assurer un confort d'utilisation et aussi d'éviter des blessures.D'abord, nous discuterons les travaux antérieurs traitant le sujet des interactions mid-air puis de la fatigue musculaire avant de présenter des études portant sur ces deux sujets. Puis, au travers de plusieurs expériences, nous avons cherché à étendre notre compréhension de cette fatigue lors de divers exercices en RV. En particulier, nous nous sommes intéressés à différentes synchronisations des mains, au rythme et au rapport contrôle-affichage (CDR), en nous appuyant sur les contextes d'applications pouvant occasionner de longues sessions d'utilisations comme le jeu vidéo et l'expression artistique.Premièrement, nous avons étudié les différences entre interactions uni et bi-manuelles en terme de fatigue au cours de différentes tâches répétitives de sélection de cibles. Il est apparu que la synchronicité de main devait être choisie au regard de la tâche à effectuer pour optimiser le rapport entre fatigue et efficacité. De plus, il semblerait préférable de laisser à l’utilisateur la possibilité d’utiliser ses mains comme il le souhaite afin qu'il auto-régule sa fatigue. En outre, il se pourrait que les changements de postures des utilisateurs soient des indicateurs de la fatigue. Enfin, nous avons pu vérifier que certaines directions de mouvements étaient plus fatigantes, en particulier celle verticale et certaines diagonales. Secondement, suite notamment à l'analyse des retours des participants, nous avons exploré l'impact d'un rythme imposé aux gestes sur la fatigue des bras et l'expérience de l'utilisateur (UX), lors d'un exercice de suivi de cibles. Ce rythme, en particulier s’il est irrégulier et lent, semblerait pouvoir réduire la sensation de fatigue et améliorer l’expérience de l’utilisateur. De plus, si le rythme est souligné par un son simple, le participant pourrait percevoir la tâche comme étant plus fatigante mais également plus engageante qu'en l'absence de son. Enfin, nous avons voulu observer les effets de variations du CDR sur la fatigue et l'UX. Cette expérience a pris place dans un instrument de musique virtuel immersif (IVMI) afin de motiver les gestes mid-air des participants. Ils ont dû explorer un cube musical alors que leurs gestes étaient visuellement amplifiés ou réduits, plus ou moins fortement. Quand le CDR est modérément modifié, il pourrait avoir un impact bénéfique sur l'UX lors de l’interaction avec un IVMI. Aussi, étonnamment, nous n'avons observé aucun impact significatif sur la fatigue alors que, pour une grande variation du CDR, les utilisateurs parcouraient moins de distance avec leur main et que cette distance était corrélée à la fatigue. Étudier le CDR sur des temps d'interaction plus longs nous permettra peut-être d'observer un impact sur la fatigue.En conclusion, nous avons pu retirer des implications intéressantes sur les choix de conceptions les plus judicieux à effectuer lorsque l'on veut proposer des applications peu fatigantes en RV. Nous avons également proposé des idées de futurs travaux qu'il serait intéressant d'étudier, comme l'utilisation de la manipulation redirigée entre des zones de CDR différenciées ou l'étude des changements de postures comme indicateur de la fatigue.
... Rhythm has been also used as a way to input commands through sequences of tapping gestures [7] or with micro-gestures performed at a given tempo [5] or for target selection by following trajectories at a given speed with mid-air gestures [3,19]. In 3D User Interfaces, sound guides were primarily investigated to help visually impaired users find targets in a 3D environment [13,15]. ...
In this work, we examine the effect of mid-air gesture rhythm on user experience in Virtual Reality. In particular, we investigate gesture regularity, speed and highlighting with a sound guide. We measure the effect of these components on the perceived fatigue, presence, difficulty, success and helpfulness. Our findings indicate that an irregular and slow rhythm leads to a lower arm fatigue. We also find that such an irregular rhythm could increase the user perceived difficulty of the task and the absence of a sound guide could decrease the sense of presence.
... gpsTunes [54] and SWAN [15]) and one-to-one interactions with learners (e.g. AudioChile [50]) or visually-impaired listeners [55]. The user, however, needs to be completely removed from the real space, with interactions (e.g. ...
Conference Paper
Centuries of development in optics have given us passive devices (i.e. lenses, mirrors and filters) to enrich audience immersivity with light effects, but there is nothing similar for sound. Beam-forming in concert halls and outdoor gigs still requires a large number of speakers, while headphones are still the state-of-the-art for personalized audio immersivity in VR. In this work, we show how 3D printed acoustic meta-surfaces, assembled into the equivalent of optical systems, may offer a different solution. We demonstrate how to build them and how to use simple design tools, like the thin-lens equation, also for sound. We present some key acoustic devices, like a "collimator", to transform a standard computer speaker into an acoustic "spotlight"; and a "magnifying glass", to create sound sources coming from distinct locations than the speaker itself. Finally, we demonstrate an acoustic varifocal lens, discussing applications equivalent to auto-focus cameras and VR headsets and the limitations of the technology.
... In the case of the visually impaired, serious games are used for educational purposes and for improving navigational skills. Various experiments demonstrated the positive effects of blind-accessible audio games on the development of learning, navigation, problem-solving and social skills [4] as well as on enhancing motivation and user confidence [5]. ...
Full-text available
The purpose of this paper is to introduce our training strategy, based on gamified 3D virtual environments, for improving sound localization and navigational skills of visually impaired people (VIP). During training, the VIPs will use an advanced sensory substitution device (SSD) which conveys information about the environment. An example of such device is the Sound of Vision device (Sound of Vision website) which performs a visual to audio-haptic encoding of the environment. In this paper we explain how a virtual environment can provide, in such cases, the same scenarios and outputs as real life situations, within full safety and control. Most serious games are specifically designed for training or educating users, rather than entertaining them, and in most cases, users play them out of necessity to gain a practical skill set-which is also the case for our visually impaired users. However, in this paper, we argue that a serious game success is highly dependent upon creating game specific entertainment and even fun, immersion and motivation for the user. Serious games' effectiveness comes from many related factors, such as sense of presence, mental and emotional involvement, motivation, sense and flow achievement, player's feeling of uniqueness and individualization in the environment. In this paper we explain our efforts in designing a "serious-but-entertaining" game with hierarchical levels of difficulty that would provide auditory and haptic stimuli and tune in unique ways general game design and learning principles, such as learning by doing or experimenting, reflection and meta-reflection for transferring the learning outcomes from virtual to live contexts. Particular attention is given to level design and interaction (which, unlike in almost all other cases, is based only on audio-haptic outputs) to maximize immersion and motivation through all available means and taking into account the specificity of our users group.
Audio is one of the essential elements of game design. Even in virtual reality applications, visuals have been thought to have more importance to create impressiveness. However, using its stereo or surround capabilities, it is possible to improve the effectiveness of the game, on the players. Also, creating an audio rich game increases its accessibility to a variety of people. This study involves a game setup with 3D audio used in virtual reality (VR), design of audio games, and Google Cardboard development. Using this game setup, a highly audio dependent mobile virtual reality game is developed and tested by 10 users. The performance of the users and feedback from the qualitative and quantitative comments were analyzed and reported. The main aim of this work is to show that audio can be a powerful game element, based on an explementary work involving a game developed with audio experience on its focus.
Full-text available
Visual disability has a major impact on people’s quality of life. Although there are many technologies to assist people who are blind, most of them do not necessarily guarantee the effectiveness of the intended use. As part of research developed at the University of Chile since 1994, we investigate the interfaces for people who are blind regarding a gap in cognitive impact, which include a broad spectrum of the cognitive process. In this work, first, a systematic literature review concerning the cognitive impact evaluation of multimodal interface for people who are blind was conducted. The study selection criteria include the papers which present technology with a multimodal interfaces for people who are blind that use a method to evaluate the cognitive impact of interfaces. Then, the results of the systematic literature review were reported with the purpose of understanding how the cognitive impact is currently evaluated when using multimodal interfaces for people who are blind. Among forty-seven papers retrieved from the systematic review, a high diversity of experiments was found. Some of them do not present the data results clearly and do not apply a statistical method to guarantee the results. Besides this, other points related to the experiments were analyzed. The conclusion was there is a need to better plan and present data from experiments on technologies for cognition of people who are blind. Moreover, this work also presented a data qualitative analysis based on the Grounded Theory based method to complement and enrich the systematic review results. Finally, a set of guidelines to conduct experiments concerning the cognitive impact evaluation of multimodal interfaces for people who are blind are presented.
3D audio simulations allow for the creation of immersive auditory environments for enhanced and alternative interactive learning. Several supporting teaching and learning philosophies are presented. Experimental research and literature on spatial cognition and sound perception provide further backing. Museums, schools, research and training facilities, as well as online educational websites all significantly can benefit from its use. Design dependence on project purpose, content, and audience is explored. An example installation is discussed.
In their evolution, virtual worlds have become more persistent. Their three-dimensional (3D) objects are more easily ported and interoperable between 3D repositories and may eventually be portable between synthetic world systems. If trend-lines continue, these synthetic spaces will become more integrated into the fabric of virtual learning and research, community-building, socializing, and digital information archival. Their continuity-in-time adds fresh capabilities for learning (human actualization, long-term virtual collaborations), digital resource protection (digital artifact preservation, long-term and evolving simulations, virtual ecologies), human relationship management (customer relationship management and branding, digital governance), and information exchange and management (international exchanges, and immersive long-term 3D libraries and knowledge structures). However, this immersive persistence must be balanced against the needs of temporality, transience, and forgetting.
An increasing number of studies have used technology to help blind people to integrate more fully into a global world. We present software to use mobile devices by blind users. The software considers a system of instant messenger to favor interaction of blind users with any other user connected to the network. Input/Output implementation modules were emphasized creating a 9-button virtual keyboard and associated Text-to-Speech technology (TTS). The virtual keyboard helps to write into the pocketPC without needing external devices, representing a real challenge for novice blind users. The TTS engine was adapted to blind users by adjusting the engine. Usability evaluation of these modules was iteratively applied to end-users. As a result, the integration of the designed modules into a communication system helped us to create a messenger system specially tailored to people with visual disabilities.
Conference Paper
Full-text available
Multimodal serious games are attractive tools for achieving this goal and helping people with visual disabilities to perceive and to interpret the surrounding world. However, it is fundamental to ensure that the games can stimulate cognitive development. The purpose of this study was to investigate the role of multimodal components in the development and evaluation of games and virtual environments targeting the enhancement of cognitive skills in people who are blind. We analyze the state-of-the-art concerning approaches and technologies currently in use for the development of mental maps, cognitive spatial structures, and navigation skills in learners who are blind by using multimodal videogames. Besides, we identify the current approaches used for designing and evaluating multimodal games in this context. In this paper, we discuss the results on these and related topics and draw from them some trends and issues.
Full-text available
Blind children tend to represent spatial environments with cognitive difficulty. The absence of visual stimuli processing impedes the construction of rich representations of the surrounding space. This can be decreased if they are exposed to interactive experiences with acoustic stimuli delivered through spatialized sound software complemented with cognitive tasks tailored to accomplish richer representations. A few studies have approached this issue by using interactive applications that integrate virtual reality and cognitive tasks to enhance spatial orientation skills. The aim of this research has been to design interactive software based on spatialized sound to help blind learners to construct cognitive spatial structures.
Full-text available
We introduce a conceptual model for building interactive hypermedia literary stories. Hyperstories extend and plasticize the idea of "branching games" and the classic "choose your own adventure" stories. The model combines static and dynamic objects embedded in nested contexts to allow flexible navigation through virtual worlds. Hyperstories include the concept of object migration between nodes, underexplored in hypermedia environments. As a result, the model supports objects manipulated by the learner, autonomous objects, characters representing entities that behave independently from the user, and a clear separation between content and interface representation. We discuss different aspects involved in the implementation of hyperstories. Finally, we analyze some further trends and issues in this growing line of research.
Full-text available
Recent literature provides initial evidence that sound can be used for cognitive development purposes in blind children. In this paper we present the design, development, and usability testing of AudioBattleShip, a sound-based interactive environment for blind children. AudioBattleShip is an interactive version of the board Battleship game, providing different interfaces for both sighted and blind people. The interface is based on spatialized sound as a way of navigating and exploring through the environment. The application was developed upon a framework that supports the development of distributed heterogeneous applications by synchronizing only some common objects, thus allowing the easy development of interactive applications with very different interfaces. AudioBattleship was tested for cognitive tasks with blind children, evidencing that it can help to develop and rehearse abstract memory through spatial reference, spatial abstraction through concrete representations, haptic perception through constructing mental images of the virtual space, and cognitive integration of both spatial and haptic references.
Full-text available
Diverse studies using computer applications have been implemented to improve the learning of children with visual disabilities. A growing line of research uses audio-based interactive interfaces to enhance learning and cognition in these children. The development of short-term memory and mathematics learning through virtual environments has not been emphasized in these studies. This work presents the design, development, and usability of AudioMath, an interactive virtual environment based on audio to develop and use short-term memory, and to assist mathematics learning of children with visual disabilities. AudioMath was developed by and for blind children. They participated in the design and usability tested the software during and after implementation. Our results evidenced that sound can be a powerful interface to develop and enhance memory and mathematics learning in blind children.
Full-text available
Whilst advances are underway in various areas to ease and encourage disabled uptake of new technology, very little emphasis to date has been placed on making the games market accessible to all. The aims of the described work have been twofold. Firstly, to prove that the standard features of a traditional space invader game can be replicated using a 3-D audio (ambisonic) environment. Secondly, through combining audio and visual interfaces with force feedback joystick movement that it is possible to produce a multi-modal game that can be played by both sighted and non-sighted users, thereby enabling them to share the same gaming experience. This paper describes the development and features of the resultant Audio Space Invaders game.
Full-text available
The paper reports on the design and evaluation of a spatial audio system that models the acoustic response of a closed environment with varying sizes and textures. To test the fit of the algorithms used, the system was evaluated by nine blind computer users in a controlled experiment using seven distinct sounds in three environments. The statistical analysis reveals that there was insignificant difference in user perception of room sizes between sounds in real and simulated scenes. This system can contribute to the area of VR systems used for training blind people to navigate in real environments.
Full-text available
Mental mapping of spaces, and of the possible paths for navigating through these spaces, is essential for the development of efficient orientation and mobility skills. The work reported here is based on the assumption that the supply of appropriate spatial information through compensatory channels (conceptual and perceptual), may contribute to the blind people's spatial performance. We developed a multi-sensory virtual environment simulating real-life spaces. This virtual environment comprises developer / teacher mode and learning mode.
Terraformers is the result of three years of practical research in developing a real-time 3D graphic game accessible for blind and low vision gamers as well as full sighted gamers. This presentation focus on the sound interface and how it relates to the 3D graphic world, and also include post mortem survey results from gamers and comments to those.