Content uploaded by Ticianne Darin
Author content
All content in this area was uploaded by Ticianne Darin on Sep 28, 2015
Content may be subject to copyright.
Dimensions to Analyze the Design of Multimodal
Videogames for the Cognition of People Who Are Blind
Ticianne Darin
Virtual University Institute
Federal University of Ceara
Humberto Monte, S/N
Fortaleza, Ceará, Brasil
ticianne@virtual.ufc.br
Jaime Sánchez
Department of Computer Science
Universidad de Chile
Blanco Encalada 2120,
Santiago, Chile
jsanchez@dcc.uchile.cl
Rossana Andrade
Department of Computer Science
Federal University of Ceara
Humberto Monte, S/N
Fortaleza, Ceará, Brasil
rossana@great.ufc.br
ABSTRACT
Multimodal serious video games are relevant tools to
enhance the cognitive skills of people who are blind. For this
purpose, it is necessary that designers and developers be able
to create user interfaces and interactions using the
multimodal components properly. Thus, there is a need to
know the key components to be considered for such
applications, as well as to understand their roles and
relationships. In this paper, we propose and discuss a 4-
dimension classification: Interface, Interaction, Cognition,
and Evaluation, to analyze the design of multimodal
videogames for the cognition of people who are blind. Such
classification was assembled from features related to the
design and evaluation of a number of multimodal video
games and virtual environments, identified via literature
review based on systematic review methodology. We also
classify and discuss multimodal applications within the
proposed categorization.
Author Keywords
Multimodal Interfaces, Virtual Environments, Serious
Videogames, People with Visual Disabilities, Design,
Evaluation
ACM Classification Keywords
H.5.m. Information interfaces and presentation:
Miscellaneous.
INTRODUCTION
One of the most significant cognitive issues for people who
are blind is the development of orientation and mobility
(O&M) skills so that the individual can become autonomous
and independent. For an efficient navigation and the
development of orientation skills, a person needs to construct
a mental representation or metal map of the surrounding
environment. However, the absence of vision adds more
complexity to easy tasks that require spatial representation
[11]. It happens because the visual channels handle to
collecting most of the information needed to assemble such
a mental map [13, 23]. Therefore, a person who is blind
needs to use complementary non-visual stimuli to perceive
the environment and construct mental maps.
Receiving space information via complementary sensors
collaborates with the creation of an adequate mental
representation of the environment. There is evidence that
multimodal interfaces, mainly based on audio and haptic
stimuli, can enhance learning and cognition in children who
are blind [14, 22]. There are several experiences with the
design and use of video games for stimulating the
development of various skills in people with visual
dissabilities [52, 53]. In fact, multimodal serious games have
been widely used as tools to improve various spatial and
O&M skills in children and young people [13, 35, 4, 37, 39,
38, 36, 34, 49]. This type of applications has particularities
and must consider the limitations of users as well as ensure
the cognitive goals. Even so, important issues are frequently
neglected in the development and evaluation of these
applications. Using a development cycle that do not consider
the particularities of the target audience, and/or the desired
cognitive skills, or not performing a proper and trustable
evaluation, are recurrent situations in the design of such
video games [27].
There is a need to promote a better understanding and an
adequate, relevant and meaningful use of the multimodal
elements in a serious game. Academics, software engineers,
developers or simply interested end users lack a
comprehensive overview of the development and evaluation
of multimodal videogames aiming to improve cognition. To
the best of our knowledge, there are no studies addressing
these issues. For this reason, the present work seeks to fill in
some of these gaps.
The overall goal of this paper is to give readers a
comprehensive overview of the multimodal video games for
the cognition of people who are blind works to date. Also to
provide necessary insights for the practical understanding of
the issues involved in their design and evaluation. As such,
we performed a bibliographic review based on the systematic
review approach [10] to investigate the role of multimodal
elements in the development and evaluation of games and
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. IHC'15, Brazilian Symposium on Human
Factors in Computing Systems. November 03 - 06, 2015, Salvador, BA,
Brazil. Copyright 2015 SBC. ISSN 2316 - 5138 (pendrive). ISBN 978-
85-7669 (online).
.
virtual environments, aiming to enhance cognitive skills of
people who are blind [27]. In this paper, we propose a
classification based on the significant features identified on
the 21 studied multimodal video games. Derived from these
characteristics, we discuss the challenges and open issues in
the development and evaluation of multimodal video games,
in the context of enhancing cognition of people who are
blind.
METHODOLOGY
We performed a bibliographic review from July to
November in 2014, based on the steps proposed on the
systematic review approach [10, 19]. The systematic review
consists in a secondary study method that reviews existing
primary studies in-depth and describes their results [19]. In
this research approach, a set of search strings correspondent
to the research questions is posed to suitable sources. Then,
the obtained papers are filtered according to a set of
exclusion and inclusion criteria. The resulting papers are
analyzed in order to answer the initial research questions.
There are three main phases of a systematic review:
planning, conducting and reporting the review [10]. To
support the realization of the three stages of the study, we
used the tool StArt [5]. As a reference manager, we relied on
Mendeley [44].
For the planning phase, we defined a protocol that guided the
research objectives and clearly defined the research
questions, the query sources and the selection methods. Two
researchers and two experts performed incremental reviews
to the protocol. The research questions are: Q1: What
strategies have been used for the design of multimodal games
for learners who are blind to enhance cognition
1
? Q2: What
strategies have been used to evaluate usability and quality of
multimodal games for learners who are blind? Q3: What
technologies have been used for the development of
multimodal games for learners who are blind, to enhance
cognition?
For the conducting phase, we posed the search string
addressing the research questions Q1, Q2, and Q3 to eight
sources. The sources were the following digital libraries:
ACM Digital Library, Engineering Village, IEEE Xplore,
Scopus, Science Direct, Springer Link, PubMed, and Web of
Science. The result of submitting the search string to the
eight selected bases was an initial set of 446 papers. Then,
using the snowballing sampling technique [16] we manually
added a set of 52 papers to the original sample. Snowballing
sampling is a technique for gathering research subjects
through the identification of an initial subject which is used
to provide other related subjects. In this research from the
initial set of 446 papers we also gathered 52 relevant related
research works cited in these papers. Thus, the total of papers
obtained was 498. From this amount there were 48 papers
from ACM (9.6%), 136 from IEEE (27.3%), 28 from Scopus
1
Concerning to mental models, cognitive spatial structures, and/or
navigation skills.
(5.6%), 181 from ScienceDirect (36.3%), 50 from Springer
(10%), 4 papers from Web of Science (0.8%), 1 paper from
Pubmed (0.2%) and 52 added manually (10.5%). It is
important to note that, although ScienceDirect had the higher
number of papers, there were not many outcomes related to
the desired area. It happened because this source returned a
vast amount of articles related to cognition and/or people
who are blind, but under the medical point of view.
Figure 1. Search string submitted to the eight selected sources.
To choose the most suitable studies to answer the research
questions, we filtered the papers according to the inclusion
and exclusion criteria (Figure 2). The inclusion criteria
helped us selecting studies describing multimodal serious
video games, some specific entertainment video games and
navigational virtual environments, whose goal was to
enhance cognition. The inclusion criteria also selected
studies describing no application but introducing a model for
the design or the evaluation of multimodal games or
environments for people who are blind. The exclusion
criteria helped us to eliminate papers related to audiences
other than people who are blind and those unrelated to mental
models, navigational and similar cognitive skills.
Figure 2. Filtering process
The first filter (F1) consists of removing the duplicated and
short papers (i.e. less than four pages) and secondary studies
or those published before 1995. F1 excluded 172 papers
(34.5%) so that 326 studies went to the second filter. The
second filter (F2) consists of the application of the specific
exclusion criteria and the inclusion criteria, after reading the
papers’ title and abstract. F2 excluded 216 papers (43.4%)
and included 68 papers (13.7%). These papers went to the
third filter (F3), intending to refine the initially accepted set
of studies. F3 consisted of the examination of the full text of
the 68 articles and the review of the assigned inclusion and
exclusion criteria. F3 excluded 34 articles by criteria and four
duplicated papers (7.6%) and included 30 papers (6%). Most
of the papers eliminated were related to cognition, but not to
multimodal games for people who are blind. From the 30
papers finally selected for data extraction, one paper was
from ACM, two from IEEE, four from Scopus, two from
ScienceDirect, two from Web of Science. Finally, 19 papers
were added manually, through a snowballing sampling.
The relevant papers are from 1999 to 2014, being 80% of the
papers from 2008 on. The selected papers were: [13, 51, 2,
35, 3, 4, 8, 15, 12, 17, 18, 41, 42, 24, 25, 30, 28, 29, 32, 33,
31, 37, 39, 38, 40, 36, 34, 47, 48, 49]. Among these, 25
papers described 21 distinct applications: 17 multimodal
games and four multimodal navigation virtual environment.
Some papers discussed the same application, but from
another point of view.
CLASSIFICATION OF FEATURES
We analyzed the 21 applications and reported the answers to
the aforementioned research questions in a previous work
[27]. In the present work, we analyze the characteristics
related to design and evaluation of the 21 selected
applications. Based on this analysis, we identified that the
studied multimodal applications differentiate mainly in the
interface and interaction characterization, along with the
cognitive aspects and the evaluation performed. Therefore,
we propose a classification of the features existing in the
multimodal video games and environments in four
dimensions related to Interaction, Interface, Cognition, and
Evaluation. It is important to highlight that the features we
describe are those found on the applications’ papers. The
proposed classification shows some trends in interface
characterization and the interaction style, as well as
instruments and activities for evaluation of usability and
cognitive impact. Figure 3 gives an overview of the
identified features, discussed in detail in the next sections.
Several elements are common to the design of multimodal
applications aiming to enhance cognition of individuals who
are blind [27]. The video games and environments combine
those elements according to the target cognitive purpose. The
Interface dimension has three kinds of features: Graphics,
Audio and Adaptation. The main features related to the
Interaction dimension are Feedback and Mode, i.e., the way
a user can interact with the application. Table 1 provides an
overview of the applications studied and their classification,
according to the Interface and Interaction dimensions.
Furthermore, we can distinguish the applications by the type
of skill they aim to improve and by the kind of evaluation
performed. The main feature in the Cognitive dimension is
Skill, with six types present in the studied papers. Although
only part of the applications carried out a proper evaluation,
this process usually focuses on the interaction and interface
features that are meaningful to the cognitive enhancement
proposed.
Figure 3. Classification of the key features in multimodal video
games for the cognition of people who are blind
The evaluation concentrates on the aspects of usability and
cognitive impact verification. Even though there is no model
for usability evaluation of multimodal games and no formal
standardization about the elements to evaluate, it is possible
to identify the most used tools and methods. As so, the
features on Evaluation dimension are Usability and
Cognitive Impact, both related to Typical Activities and
Instruments.
DIMENSION 1: INTERACTION
According to the information available in the analyzed video
games and environments, there are seven modes through
what the user can interact, providing input to the application.
The chosen input device determines the style of interaction
available in the application. It also influences the type of
feedback that the application provides to an interaction. The
Interaction Mode comprehends mouse, keyboard, natural
language, force feedback devices, touchscreen (with or
without a stylus), directional pad, and specific devices,
designed for a particular application. The Interaction
Feedback can be either haptic (kinesthetic or tactile),
sonorous or visual. Some applications combine two or more
interaction modes and feedback types.
The keyboard requires no specific learning and allows direct
interaction since it is a familiar input device, common to
diverse kinds of systems and technologies. In fact, 71% of
Table 1. Classification of the studied applications according to the Interface and Interaction Dimension
the researched applications use the keyboard as a primary or
secondary interaction mode, in the desktop paradigm. The
interaction via the keyboard is associated with a sonorous or
visual feedback. The keyboard is a low cost, accessible and
straightforward device, but it provides no sense of touch or
volume during the interaction. Nevertheless, it may be a
desirable feature in a navigation context to provide non-
visual stimuli that help perceiving the environment physical
characteristics.
Force feedback devices allow the haptic feedback (tactile and
kinesthetic). These devices measure the positions and contact
forces of the user’s hand, displaying contact forces and
positions to the user [44], even without visual clues. The
applications using force feedback devices allow manual
interactions with the multimedia environment through touch.
It allows the user to explore the environment to extract
information, through the tactile feedback and manipulation
for modifying the environment, via the kinesthetic feedback
[9]. The applications use various devices to allow the
function of haptic feedback. Gamepads and joysticks are the
second most used interaction form, present in 33% of the
studied applications. The applications that use gamepads
focus on the tactile feedback, providing sensations of
vibration, pressure, touch, and texture. The tactile feedback
allows the user to perceive contact force, the geometry of an
object and temperature [9]. The applications that utilize
joystick provide kinesthetic sensation, dealing with forces
resulting from position and velocity of the hand motion and
simulating the force and torque [9]. Those applications that
employ specialized joysticks, e.g. 3D touch controllers,
usually combine tactile and kinesthetic, providing haptic
feedback. The force feedback device always has an
alternative interaction mode, usually the keyboard. It assures
the availability of the game, even without the force feedback.
The most common force feedback device is Novint Falcon,
a 3D USB haptic device used in four applications. However,
Novint Falcon has a high cost, about $250 for the standard
version. So, lower cost devices have also been utilized, such
as OWL joystick, Wiimote, and SideWinder joystick.
The use of mouse and touchscreen with stylus configures a
style of interaction that uses a directional pointer to select
items on a display screen. Due to the visual limitation of the
audience, these are the less frequent interaction modes
identified. Two applications (9%) allows the use of mouse
along with the keyboard and one uses the mouse as the
primary interaction mode. However, this last application
main audience is partially sighted users. The use of mouse
relates to a sonorous-only feedback while the stylus also
provides a tactile feedback. The natural language is still an
underutilized feature (found in two applications). The
feedback to this kind of interaction is sonorous. Unlike the
mouse, we expected a wide use of the natural language in this
type of application, as an easier and instinctive way to
interact. However, the accurate and efficient recognition and
processing of the human language is still a challenge in
computer science. A single application uses the directional
pad, a four-way directional control with one button on each
point, found on most video game console gamepads. In this
context, the directional pad interaction occurs in a mobile
application and its feedback is sonorous.
Another option is the development of specific devices to
allow for the interaction of the users who ae blind with the
multimodal video game. For instance, the Digital Clock
Carpet [35] is a device based on a usual cane and a simple
digital carpet that inform the user about directions to a
destination point, based on the hour system. In this particular
case, the feedback is both sonorous and haptic, but adapting
a device to interaction and feedback specific to the context
of a multimodal video game brings a bunch of new
possibilities.
DIMENSION 2: INTERFACE
The interface features that impact most the design of the
studied multimodal video games and environments are
Graphics, Audio and Adaptation (Figure 3). The Graphics
and Adaptation features are directly related since all the
interface adjustments refer to graphical elements. Regarding
Adaptation, the interfaces may support the adjustment of the
size, color or contrast of the graphical elements. The features
include the resize of the elements, the possibility of choosing
a high contrast mode and the presentation of relevant
information in colorblind safe colors. The resizing relates
mainly to text elements, which can be resized with no loss of
content or functionality. The high contrast is to provide
enough contrast between the content and its background so
that people with low vision can read it. The use of colorblind
safe colors is to assure the interface presents the visual
elements in color combinations that are perceivable by
people with any colorblindness.
The Graphical User Interface (GUI) is a vastly utilized
feature in this kind of applications, allowing the user to
interact with graphical elements through direct manipulation.
In their exploratory study [50], the prevalence of visual over
other senses in video games is highlighted and compares the
vastly used option of “turn off sound” with the option to turn
off the graphics. They state that it is uncommon to turn off
the game graphics in a way that it can still be played and
enjoyed. Our results corroborate partially to that conclusion
in the context of multimodal gaming for cognition in players
who are blind since we identified no video game with a
sound-only interface. However, 14% of the applications
studied do allow the users to choose navigating exclusively
through sound, by using a graphic interface for configuration
only.
The reason all the applications studied have a graphic
interface, despite the visual impairment of the main
audience, is to increase the interaction options for users with
visual loss. The interface’s graphics can be either 2D or 3D
combined with icons, images, and text. The association of
such elements helps filling some gaps on the interaction. For
instance, children with residual vision have difficulties
recognizing certain 2D icons and not always associate them
with the designed actions, so the use of 3D icons helps
increasing the fidelity of the representation [37]. Besides,
having a graphical interface a facilitator can support the
interaction with the video game, observing the navigation
and the cognitive aspects [26].
Although the graphics are substantial features to the
interfaces of the studied multimodal video games, the
essential interface feature is the Audio. The papers showed 6
types of audio that can be used solely or combined to provide
sonorous feedback in a multimodal interface: iconic sound,
spatialized sound, spoken audio, stereo sound, speech
synthesis and abstract earcon. All of the analyzed
applications use at least one aural interface element, although
most of the cases combine two or more aural elements. The
prevailing combination is between iconic and spatialized
sound, in 3D environments. Figure 4 illustrates the use of
each audio feature in the considered applications.
Figure 4. Distribution of the audio features found in the 21
studied applications
Iconic sounds are specific sounds associated with each
available object and action in the environment. Every time
the user executes a certain action or interacts with a particular
object, the corresponding representative sound is heard, for
example, distinct steps sound for different kinds of floor. It
is the most common audio feature, occurring in 16
applications (76%). Spatialized or 3D sounds are stereo
sounds that are digitally processed to appear to come from
particular locations in the three-dimensional space, aiming to
simulate the acoustic experienced by a listener within a
specific environment. A 3D sound navigable environment
can serve as an aural representation of the space and
surrounding entities, helping the players who are blind to
assemble a mental image of an environment [17]. It was the
second most used audio feature, present in 11 (52%)
applications. Spoken audio refers to the use of sentences pre-
recorded in a human voice, usually describing the game
status or relevant information about the actions and objects.
The spoken audio occurs in 11 applications (52%) while the
speech synthesis, which is the artificial creation of the human
voice, appears in seven (33%). Five applications combine
these two approaches. Stereo sound consists in the mixing of
two channels of sound recorded in two separate sources, with
the distinction of left and right channels. Stereo sound is
present in five applications (23%) and can provide
information regarding the nature and location of objects,
using intuitive associations (e.g. the sound of flowing water
representing a fountain). Abstract earcons appear in only one
application, using music/tones to represent different objects.
It refers to the use of sounds unrelated to the elements they
represent. The use of abstract earcons requires the user to
learn with what they are associated. However, it is possible
to represent a much wider range of concepts than by using
iconic sounds [3].
DIMENSION 3: COGNITION
The studied multimodal video games and virtual
environments meticulously combine the Interaction and
Interface Dimensions’ features, not only for entertainment
but also for learning and stimulating cognitive processes [35,
3, 4, 8]. The Cognition dimension deals with which cognitive
skills an application aims to improve. Some of the
applications help the development of more than one skill.
Besides, some skills are secondarily stimulated while
playing. The Table 2 summarizes which are the main skills
the studied applications aim to improve. We defined six
types of skills that compose the Cognition dimension: mental
models, mental maps, spatial structures, orientation and
mobility (O&M), problem solving and social collaboration.
Some of the multimodal games for players who are blind did
not specify how they improve cognitive skills. Most of the
applications work O&M skills (44%). It is a set of techniques
that helps visually impaired children and adults to develop
and master the concepts and competencies necessary to be
able to move safely and efficiently within their world. While
navigating the video game environment, people who are
blind can perceive the audio and haptic elements and use
them as references for orientation and mobility [42].
Mental models are constructs that can explain human
behavior and the internal mechanisms that allow people to
understand, explain, and predict the behavior of objects and
systems [20]. The improvement of mental modeling can
involve the user adopting and restructuring a mental model
of spatial dimensions, based on audio and haptic cues after
interacting with a video game [42]. The adequate
orchestration between audio and haptics can also help the
user who is blind to build up a model of a fantasy world and
of how a game works [18, 51]. Specific mental models are
stimulated by 27% of the studied applications.
Mental maps are mental representations of the space being
navigated and its defining features, e.g., overall structure,
spatial components, landmarks, dimensions, and relative
positions [12]. Having a mental map of space is fundamental
to the efficient development of orientation and mobility
techniques [35]. From the applications that improve
cognitive skills, 27% help developing mental maps.
Cognitive spatial structures stimulated by 16% of the
applications studied, relates to spatial and temporal reference
frames, implying the connection between visual and haptic
space [6]. The spatial properties include location, size,
distance, direction, separation and connection, shape,
pattern, and movement. Humans acquire spatial knowledge
and beliefs directly via sensorimotor systems that operate as
they move about the world [45]. While navigating in
multimodal environments, players who are blind can acquire
spatial knowledge indirectly through the maps and images,
3D audio and graphics models, and the language.
Table 2. Classification of the applications according to the
Cognitive Dimension
Problem solving can be understood as the act of consciously
apply rules and procedures to bridge the gap between the
initial problem state and a solution state [7]. Working this
skill while playing multimodal video games involves
navigating and interacting, while solving tasks, challenges
and issues [37, 8]. Besides, it may include other
competencies such as learn and interpret points on a two-
dimensional plane [3], and searching, mobilization,
localization and designing strategies [37]. Skills concerning
to social collaboration refers to help multiple people interact
and share information to achieve a common goal. These
skills can be improved while multimodal gaming when users
who are blind and sighted users solves collaborative tasks
[24]. It creates an inclusive learning context where users can
work together collaboratively and achieve commonly shared
objectives.
DIMENSION 4: EVALUATION
The last dimension we defined to classify the analyzed
games is Evaluation, which concentrates on two main
aspects: usability and cognitive impact verification (Table 3).
It is important to point out that both evaluations are necessary
to assure the quality of multimodal video games with
cognitive enhancement purposes. According to the ISO/IEC
9126 (2001) quality, relates to "[...] all the features of a
product or service that exert their abilities to meet the stated
or involved needs." It makes clear that usability is an
important aspect of the game’s quality. However, without a
proper cognitive impact evaluation one cannot assure that a
particular application can develop or enhance any cognitive
skills for people with visual disabilities [27].
Table 3. Types of evaluation performed on the considered
applications
The usability evaluation is the most frequent type of end
quality (users and expert user’s acceptance) assessment in
this context, performed more often than cognitive impact
evaluation. Among the applications analyzed that aim to
enhance cognition, only 50% performed a cognitive impact
evaluation while 64% carried out at least one type of
usability evaluation. However, in this context both
evaluations are essential. Table 3 shows the evaluations
performed on the set of the video games and environments
analyzed. There is no particular model for the cognitive
impact and usability evaluation of this kind of multimodal
games. However, we identified the most used instruments
2
None of the papers presents a bibliographical reference to these
instruments, except for SUBC and EUQ.
and activities for the usability and cognitive impact
evaluation.
For the usability evaluation we identified three types of
commonly applied instruments: Specialized Questionnaires,
Common Questionnaires, and Likert-based Surveys. The
specialized questionnaires are validated and reusable
instruments, prevailing in the formal evaluations. They
consist of some context-specific sentences, for which the
users can define the degree of fulfillment on a scale. The
specialized questionnaires
2
identified are the Software
Usability for Blind Children Questionnaire (SUBC) [43]; the
End User and Facilitator Questionnaire for Software
Usability (EUQ) [21]; the Software Usability Elements
Questionnaire (SUE), which quantifies the degree to which
the sounds of an application are recognizable; the Open
Question Usability Questionnaire (OQU); and the Initial
Usability Evaluation (IUE). The common questionnaires and
the Likert-based surveys are developed and used
circumstantially, to evaluate a particular application and
consist of a set of factual, opinion, and attitude questions. In
both cases, the authors themselves created the instruments,
and they do not disclosure the validation process or a
particular formalization. The surveys are mainly based on the
context of the application and are applied face to face or via
email [4, 47]. These instruments are prevalent in non formal
evaluations.
The typical activities executed in the usability evaluation of
the multimodal games and environments analyzed are
Observation, Interview, and Heuristics Evaluation. The
observation is direct and involves an investigator viewing
users as they use the application, and taking notes on
usability aspects [17]. The interviews are semi-structured
and occur after the user interacts with the application. The
questions intend to establish the subject’s previous
experience, identify aspects of the interface that were helpful
or problematic, and prompt for suggestions on improving the
experience [3]. In the heuristic evaluation, usability experts
inspect the application’s interface and compare it with
usability principles in a checklist. For example, the Heuristic
Evaluation of the Videogame (HEV), based on
Schneiderman’s golden rules and Nielsen’s usability
heuristics, and the Heuristic Evaluation Questionnaire
(HEQ) [35]. The evaluations often combine such activities
and instruments.
For the cognitive impact evaluation, the instruments most
commonly used are Logs, Checklists, Questionnaires and
Modeling Kits. The four sorts of instruments can be used
throughout the assessment’s activities, combined with
observation and interviews. The goal is to collect data that
will allow an investigator to observe, compare, analyze and
measure the skills and the development of the subjects. The
typical activities identified relates to the research structure of
the quasi-experimental design of non-equivalent groups [1],
considering experimental and control groups and a two-
sample test analysis (pretest-posttest).
The four typical activities identified are Pretest, Training
Tasks, Cognitive Tasks and Postest, performed in this order.
The pretest involves the users executing an activity under
observation so that an investigator can establish the subjects’
initial skills [42, 38]. The training tasks refers to the skills
that the users need to have before using the video game [33].
Through the training tasks, the user can be familiar with the
gameplay [36]. The cognitive tasks focus on developing the
specific desired skills based on the software interface [33].
Through the utilization of the video game, each skill in
Cognition dimension can be worked on, to strengthen the
development of these skills by using the video game. Once
the cognitive tasks are completed the posttest takes place, to
determine whether or not there was any impact on the
previously evaluated skills [33, 17, 42]. In the posttest, each
subject is asked to represent in an adequate way the learned
skills. The representation can be either a graphic or physical
model, an oral description, a test or another suitable
approach. Besides, the representation data can be gathered
from, for example, logs and in-depth or structure interviews.
All the generated data is analyzed and compared with the
pretest and cognitive tasks data, determining the cognitive
impact of the application.
CONCLUSION
Multimodal interfaces through video gaming can help to
improve cognitive skills and thus to make that the playful
aspects of the game and their associated technologies
positively influence the motivation of end-users [26]. While
developing these applications, one must carefully consider
several factors, such as the context of use, the desired skills
to be developed and the severity of the audience’s visual
impairment [27]. The analyzed papers show trends in
interface characterization and the interaction style, as well as
instruments and activities for evaluation of usability and
cognitive impact. However, there are some gaps related to
when and how to employ the interface and interaction
elements to fulfill the application’s cognitive requirements.
Significant issues remain neglected in the evaluation of
multimodal video games for blind learner’s cognition
enhancement. This paper aimed to help reducing the
problems aforementioned by presenting necessary insights
for the practical understanding of the issues involved in their
design and evaluation of such applications. We proposed a
classification of the features for design and evaluation of
multimodal games for cognition of people who are blind,
considering four dimensions: Interaction, Interface,
Cognition, and Evaluation. Besides, we provide readers with
a comprehensive overview of the multimodal video games
for the cognition of people who are blind works to date. As
future work we intend to propose a methodology for
evaluating the usability and the cognitive impact of interface
elements and interaction components in this kind of games.
The results will allow us to analyze and discuss the impact
of interface elements on cognition and to propose guidelines
for the better practical use of such elements.
ACKNOWLEDGMENTS
This paper is funded by: the Program STIC-AmSud-
CAPES/CONYCIT/MAEE, project KIGB-Knowing and
Interacting while Gaming for the Blind, 2014; the Chilean
National Fund of Science and Technology, Fondecyt
#1150898; and by Basal Funds for Centers of Excellence,
Project FB0003, from the Associative Research Program of
CONICYT
REFERENCES
1. Campbell, D. T., Stanley, J. C. and Gage, N.L.
Experimental and Quasi-Experimental Designs for
Research. Boston: Houghton Mifflin, 1963.
2. Connors, E.C., Chrastil, E.R., Sánchez, J., Merabet, L.B.:
Action video game play and transfer of navigation and
spatial cognition skills in adolescents who are blind.
Frontiers in Human Neuroscience (FNHUM) 8: article 133,
2014. Frontiers (2014)
3. Dulyan, A., Edmonds, E.: AUXie: initial evaluation of a
blind-accessible virtual museum tour. In Proceedings of the
22nd Australasian Computer-Human Interaction
Conference. Pages 272-275. Brisbane, Australia (2010)
4. Espinoza, M., Sánchez, J., Campos, M. B.: Videogaming
Interaction for Mental Model Construction in Learners
Who Are Blind. In Proceedings of 8th International
Conference, UAHCI 2014, Held as Part of HCI
International 2014, pages 525-536. Crete, Greece (2014)
5. Fabbri, S., Hernandes, E., Di Thommazo, A., Belgamo, A.,
Zamboni, A., Silva, C.: Managing literature reviews
information through visualization. In International
Conference on Enterprise Information Systems.14th.
ICEIS, Wroclaw, Poland, Jun, 2012. Lisbon: SCITEPRESS
(2012)
6. Freksa, C. Spatial and temporal structures in cognitive
processes. In: Foundations of computer science. Springer
Berlin Heidelberg, 1997. p. 379-387.
7. Glatzeder, B.M. (eds.), Towards a Theory of Thinking, On
Thinking, 3 DOI 10.1007/978-3-642-03129-8_1, ©
Springer-Verlag Berlin Heidelberg 2010
8. Guerrero, J., Lincon, J.: AINIDIU, CANDI, HELPMI:
ICTs of a personal experience. Engineering Applications
(WEA), 2012 Workshop on. Pages 1-7. Bogotá, Colombia
(2012)
9. Hamam, A., Eid, M. and Saddik. A. 2013. Effect of
kinesthetic and tactile haptic feedback on the quality of
experience of edutainment applications. Multimedia Tools
Appl. 67, 2 (2013)
10. Kitchenham, B. and Charters, S.: Guidelines for
performing systematic literature reviews in software
engineering. Technical Report EBSE 2007-001, Keele
University and Durham University Joint Report (2007)
11. Kolb, B and Whishaw, I.: Neuropsicología Humana 5a
edición. Editorial Médica Panamericana, 2006, pp. 555
(2006)
12. Lahav, O. and Mioduser, D.: Construction of cognitive
maps of unknown spaces using a multysensory virtual
environment for people who are blind. Computers in
Human Behavior 24(3), pp. 1139-1155 (2008)
13. Lahav, O. and Mioduser, D.: Haptic-feedback support for
cognitive mapping of unknown spaces by people who are
blind. International Journal on Human-Computer Studies,
66, pp. 23–35 (2008)
14. Lahav, O. Schloerb, D.W. Kumar, S. and Srinivasan, M.A.:
BlindAid: A learning environment for enabling people who
are blind to explore and navigate through unknown real
spaces. Proceedings Virtual Rehabilitation 2008
Conference, Vancouver, Canada, pp.193-197 (2008)
15. Lahav, O., Schloerb, D. W., Srinivasan, M. A.: Virtual
Environment System in Support of a Traditional
Orientation and Mobility Rehabilitation Program for
People Who Are Blind. Presence, vol. 22, pages 235-254.
MIT Press (2013)
16. Lewis-Beck, M.S., Bryman, A., and Liao, T.F.: The SAGE
Encyclopedia of Social Science Research Methods (2004)
17. Lumbreras, M., Sánchez, J.: Interactive 3D Sound
Hyperstories for Blind Children. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems, pages 318-325, May 1999. Pittsburgh, USA
(1999)
18. Mccrindle, R. J., Symons, D.: Audio Space Invaders. In
Proceedings of the Third International Conference on
Disability, Virtual Reality and Associated Technologies.
Pages 59-65 (2000)
19. Petersen, K., Feldt, R., Mujtaba, S. Mattsson, M.:
Systematic mapping studies in software engineering, in:
EASE ’08: Proceedings of the 12th International
Conference on Evaluation and Assessment in Software
Engineering, University of Bari, Italy. (2008)
20. Rouse, W. B., & Morris, N. M. (1986). On looking into the
black box: Prospects and limits in the search for mental
models. Psychological bulletin, 100(3), 349.
21. Sánchez, J. End-user and facilitator questionnaire for
Software Usability (EUQ). Usability evaluation test.
University of Chile (2003)
22. Sánchez, J. and Tadres, A.: Audio and haptic based virtual
environments for orientation and mobility in people who
are blind. In Proceedings of the 12th international ACM
SIGACCESS conference on Computers and accessibility
(ASSETS '10). ACM, New York, NY, USA, 237-238
(2010)
23. Sánchez, J. and Zúñiga, M.: Evaluating the Interaction of
Blind Learners with Audio-Based Virtual Environments.
Cybersychology & Behavior, Volume 9, Number 6, 2006,
pp. 717 (2006)
24. Sánchez, J., Aguayo, F.: AudioGene: Mobile learning
genetics through audio by blind learners. Learning to live
in the knowledge society. IFIP 20th World Computer
Congress, IFIP TC 3 ED-L2L Conference September,
2008. Pages 79-86. Milano, Italy (2008)
25. Sánchez, J., Campos, M. B., Espinoza, M., Merabet, L. B.:
Audio Haptic Videogaming for Developing Wayfinding
Skills in Learners Who are Blind. In Proc. ACM
International Conference on Intelligent User Interfaces
(ACM IUI), pp. 199-208, Feb 2014. Haifa, Israel (2014)
26. Sánchez, J., Campos, M.B. and Espinoza, M. 2014.
Multimodal gaming for navigation skills in players who are
blind. In Proceedings of the 13th Brazilian Symposium on
Human Factors in Computing Systems (IHC '14).
Sociedade Brasileira de Computação, Porto Alegre, Brazil,
Brazil, 169-179.
27. Sánchez, J., Darin, T. and Andrade, R. Multimodal
Videogames for the Cognition of People Who Are Blind:
Trends and Issues. Universal Access in Human-Computer
Interaction. Access to Learning, Health and Well-Being.
Springer International Publishing, 2015. 535-546.
28. Sánchez, J., Espinoza, M., Campos, M. B., Leite, L. L.
(2014) Modeling Videogames for Mental Mapping in
People Who Are Blind. In Proceedings of 8th
International Conference, UAHCI 2014, Held as Part of
HCI International 2014, pages 605-616. Crete, Greece.
29. Sánchez, J., Espinoza, M., Campos, M. B., Merabet, L. B.:
Enhancing Orientation and Mobility Skills in Learners who
are Blind through Videogaming. In Proceedings ACM
Creativity and Cognition (ACM C&C), pp. 353-356, Jun
2013. Sydney, Australia (2013)
30. Sánchez, J., Espinoza, M.: Designing Serious Videogames
through Concept Maps. In M. Kurosu (ed.), Proc. 17th
Human-Computer Interaction International (HCII), pp.
299-308, volume 2, Jul 2013. Las Vegas, Nevada, USA.
Springer-Verlag, Berlin/Heidelberg, Germany. Lecture
Notes in Computer Science (2013)
31. Sánchez, J., Flores, H.: Virtual Mobile Science Learning
for Blind People. Cyberpsychology & behavior: the impact
of the Internet, multimedia and virtual reality on behavior
and society, vol. 11, number 3 (2008)
32. Sánchez, J., Guerrero, L., Saenz, M., Flores, H.: A Model
to Develop Videogames for Orientation and Mobility. In
Klaus Miesenberger, Joachim Klaus, Wolfgang Zagler,
Arthur Karshmer (ed.), Proc. 12th International Conference
on Computers Helping People with Special Needs
(ICCHP), pp. 296-303, Jul 2010. Vienna, Austria.
Springer-Verlag, Berlin/Heidelberg, Germany. Lecture
Notes in Computer Science vol. 6180 (part II) (2010)
33. Sánchez, J., Mascaró, J.: Audiopolis, Navigation through a
Virtual City Using Audio and Haptic Interfaces for People
Who Are Blind. In Constantine Stephanidis (ed.), Proc.
15th Human-Computer Interaction International (HCII),
pp. 362-371, Jul 2011. Orlando, Florida, USA. Springer-
Verlag, Berlin/Heidelberg, Germany. Lecture Notes in
Computer Science (2011)
34. Sánchez, J., Sáenz, M. (2006). Three-Dimensional Virtual
Environments for Blind Children. CyberPsychology &
Behavior. Vol. 9, issue 2, page 200.
35. Sánchez, J., Saenz, M., Garrido, J.M. Usability of a
multimodal video game to improve navigation skills for
blind children. ACM Transactions on Accessible
Computing. Proceedings of the 11th international ACM
SIGACCESS conference on Computers end Accessibility.
Pages 35-42. (2010)
36. Sánchez, J., Sáenz, M., Pascual-Leone, A., Merabet, L. B..
Enhancing Navigation Skills through Audio Gaming. In
Proc. ACM Conference on Human Factors in Computing
Systems (CHI’10), pp. 3991-3996, Apr 2010. Atlanta, GA,
USA. (2010)
37. Sánchez, J., Sáenz, M.: 3D sound interactive environments
for blind children problem solving skills. Behaviour & IT.
Vol. 25, issue 4, pages 367-378. From the 7h International
ACM SIGACCESS Conference on Computers and
Accessibility. Baltimore, USA (2006)
38. Sánchez, J., Sáenz, M.: Metro Navigation for the Blind.
Computers and Education (CAE) 55(3):970-981, Jul 2010.
Elsevier Science, Amsterdam, The Netherlands (2010)
39. Sánchez, J., Sáenz, M.: Video Gaming for Blind Learners
School Integration in Science Classes. In Tom Gros, Jan
Gulliksen, Paula Kotzé, Lars Oestreicher, Philippe
Palanque, Raquel Oliveira Prates, Marco Winckler (ed.),
Proc. 12th IFIP TC13 Conference on Human-Computer
Interaction (Interact), pp. 36-49, Aug 2009. Uppsala,
Sweden. Springer-Verlag, Berlin/Heidelberg, Germany.
Lecture Notes in Computer Science vol. 5726 (Part I)
(2009)
40. Sánchez, J., Tadres, A., Pascual-Leone, A., Merabet, L.:
Blind children navigation through gaming and associated
brain plasticity. In Virtual Rehabilitation International
Conference 2009, June 29-July 2, 2004, Haifa, Israel, 29-
36 (2009)
41. Sánchez, J.: A model to design interactive learning
environments for children with visual disabilities.
Education and Information Technologies. Vol. 12, issue 3,
pages 149-163. Kluwer Academic Publishers-Plenum
Publishers (2007)
42. Sánchez, J.: Development of navigation skills through
audio haptic videogaming in learners who are blind. In
Proc. Software Development for Enhancing Accessibility
and Fighting Info-exclusion (DSAI’12), pp. 102-110, Jul
2012. Douro, Portugal (2012)
43. Sanchez, J.: Software Usability for Blind Children
Questionnaire (SUBC). Usability evaluation test,
University of Chile (2003)
44. Singh, J. Mendeley: A free research management tool for
desktop and web. Journal of pharmacology &
pharmacotherapeutics 1, 62-63 (2010).
45. Smelser, N. J. and Baltes, P.B. (Eds.), International
Encyclopedia of the Social & Behavioral Sciences (pp
14771-14775). Oxford: Pergamon Press. 2001
46. Tan, H. Z., Srinivasan, M. A., Eberman, B., & Cheng, B.
(1994). Human factors for the design of force-reflecting
haptic interfaces. Dynamic Systems and Control, 55(1),
353-359.
47. Torrente, J., Blanco, Á. d., Serrano-Laguna, Á., Vallejo-
Pinto, J. Á., Moreno-Ger, P., Fernández-Manjón, B.:
Towards a Low Cost Adaptation of Educational Games for
People with Disabilities. Computer Science and
Information Systems, Vol. 11, No. 1, 369–391 (2014)
48. Torrente, J., Blanco, A., Moreno-Ger, P., Martinez-Ortiz,
I., Fernandez-Manjon, B.: Implementing Accessibility in
Educational Videogames with e-Adventure. In Proceedings
of the 1st ACM International Workshop on Multimedia
Technologies for Distance Learning. Pages 57-66. Beijing,
China (2009)
49. Trewin, S., Hanson, V. L., Laff, M. R., Cavender, A.:
PowerUp: An Accessible Virtual World. In Proceedings of
the 10th international ACM SIGACCESS conference on
Computers and accessibility, pages 177-184. Halifax,
Canada (2008)
50. Valente, L., Sieckenius, C.S. and Feijó, B. 2008. An
exploratory study on non-visual mobile phone interfaces
for games. In Proceedings of the VIII Brazilian Symposium
on Human Factors in Computing Systems (IHC '08).
Sociedade Brasileira de Computação, Porto Alegre, Brazil,
Brazil, 31-39.
51. Westin, T.: Game accessibility case study: Terraformers –
a real-time 3D graphic game. In: Proceedings of the 5th
International Conference on Disability, Virtual Reality and
Associated Technologies, ICDVRAT 2004, Oxford, UK,
pp. 95–100 (2004)
52. Yuan, B. and Folmer, E.: Blind hero: enabling guitar hero
for the visually impaired. In Proceedings of the 10th
International ACM SIGACCESS Conference on
Computers and acces-sibility, pp 69-176 (2008)
53. Yuan, B.: Towards Generalized Accessibility of Video
Games for the Visually Impaired. Ph.D. Dissertation. Univ.
of Nevada, Reno, Reno, NV, USA. Advisor (S Frederick C.
Harris and Eelke Folmer. AAI3355610. (2009)
SAME SIZEmmmmmmmm
