Content uploaded by Ticianne Darin
Author content
All content in this area was uploaded by Ticianne Darin on Jul 18, 2019
Content may be subject to copyright.
'LPHQVLRQVIRUWKH'HVLJQDQG(YDOXDWLRQRI0XOWLPRGDO
9LGHRJDPHVIRUWKH&RJQLWLRQRI3HRSOH:KR$UH%OLQG
Ticianne Darin
Virtual University Institute
Federal University of Ceara
Humberto Monte, S/N
Fortaleza, Ceará, Brasil
ticianne@virtual.ufc.br
Jaime Sánchez
Department of Computer Science
Universidad de Chile
Blanco Encalada 2120,
Santiago, Chile
jsanchez@dcc.uchile.cl
Rossana Andrade
Department of Computer Science
Federal University of Ceara
Humberto Monte, S/N
Fortaleza, Ceará, Brasil
rossana@great.ufc.br
ABSTRACT
Multimodal serious video games are relevant tools to
enhance the cognitive skills of people who are blind. For this
purpose, it is necessary that designers and developers be able
to create interfaces and interactions using the multimodal
elements properly. Thus, there is a need to know the relevant
elements for such applications, as well as to understand their
roles and relationships. In this paper, we propose and discuss
a 4-dimension classification: Interface, Interaction,
Cognition, and Evaluation. Such classification was
assembled from the features related to the design and
evaluation of 21 multimodal video games and environments,
identified via a bibliographic review based on the systematic
review approach. Besides, we classify and discuss the 21
multimodal applications into the proposed classification.
Author Keywords
Multimodal Interfaces, Virtual Environments, Serious
Videogames, Visual Impairment, Design, Evaluation
ACM Classification Keywords
H.5.m. Information interfaces and presentation:
Miscellaneous.
INTRODUCTION
One of the most significant cognitive issues for people who
are blind is the development of orientation and mobility
skills so that the individual can become autonomous. For an
efficient navigation and the development of orientation
skills, a person needs to construct a mental representation of
the surrounding environment. However, the absence of
vision adds unnecessary complexity to easy tasks that require
spatial representation [11]. It happens because the visual
channels handle collecting most of the information needed to
assemble such a mental map [13, 23]. Therefore, a blind
person needs to use complementary non-visual stimuli to
perceive the environment and construct mental maps.
Receiving space information via complementary sensors
collaborates with the creation of an adequate mental
representation of the environment. There are evidence that
multimodal interfaces, mainly based on audio and haptic
stimuli, can foment learning and cognition in blind children
[14, 22]. There are several experiences with the design and
use of video games for stimulating the development of
various abilities in people with visual impairment [52, 53].
In fact, multimodal serious games have been widely used as
tools to improve various spatial and O&M skills in children
and young people [13, 35, 4, 37, 39, 38, 36, 34, 49]. This
type of application has particularities and must consider the
limitations of the users, as well as ensure the cognitive goals.
Even so, important issues are frequently neglected in the
development and evaluation of these applications. Using a
development cycle that do not consider the audience or the
desired cognitive skills, or not performing a proper and
trustable evaluation are recurrent situations in the design of
such video games [27].
There is a need to promote a better understanding and an
adequate, relevant and meaningful use of the multimodal
elements in a serious game. Academics, software engineers,
developers or simply interested end users lack a
comprehensive overview of the development and evaluation
of multimodal videogames aiming to improve cognition. To
the best of our knowledge, there are no studies addressing
these issues. For this reason, the present work seeks to fill in
some of these gaps.
The overall goal of this paper is to give readers a
comprehensive overview of the multimodal video games for
the cognition of people who are blind works to date. Also to
provide necessary insights for the practical understanding of
the issues involved in their design and evaluation. As such,
we performed a bibliographic review based on the systematic
review approach [10] to investigate the role of multimodal
elements in the development and evaluation of games and
virtual environments, aiming to enhance cognitive skills of
people who are blind [27]. In this paper, we propose a
classification based on the significant features identified on
the 21 studied multimodal video games. Derived from these
characteristics, we discuss the challenges and open issues in
the development and evaluation of multimodal video games,
in the context of enhancing cognition of people who are
blind.
METHODOLOGY
We performed a bibliographic review from July to
November in 2014, based on the steps proposed on the
systematic review approach [10, 19]. The systematic review
consists in a secondary study method that reviews existing
primary studies in-depth and describes their results [19]. In
this research approach, a set of search strings correspondent
to the research questions is posed to suitable sources. Then,
the obtained papers are filtered according to a set of
exclusion and inclusion criteria. The resulting papers are
analyzed in order to answer the initial research questions.
There are three main phases of a systematic review:
planning, conducting and reporting the review [10]. To
support the realization of the three stages of the study, we
used the tool StArt [5]. As a reference manager, we relied on
Mendeley [44].
For the planning phase, we defined a protocol that guided the
research objectives and clearly defined the research
questions, the query sources and the selection methods. Two
researchers and two experts performed incremental reviews
to the protocol. The research questions are: Q1: What
strategies have been used for the design of multimodal games
for blind learners to enhance cognition1? Q2: What strategies
have been used to evaluate usability and quality of
multimodal games for blind learners? Q3: What technologies
have been used for the development of multimodal games for
blind learners, to enhance cognition?
For the conducting phase, we posed the search string
addressing the research questions Q1, Q2, and Q3 to eight
sources. The sources were the following digital libraries:
ACM Digital Library, Engineering Village, IEEE Xplore,
Scopus, Science Direct, Springer Link, PubMed, and Web of
Science.
Figure 1. Search string submitted to the eight selected sources.
The result of submitting the search string to the eight
selected bases was an initial set of 446 papers. Then, using
1 Concerning to mental models, cognitive spatial structures, and/or
navigation skills.
the snowballing sampling technique [16] we manually added
a set of 52 papers to the original sample. Snowballing
sampling is a technique for gathering research subjects
through the identification of an initial subject which is used
to provide other related subjects. In this research, from the
initial set of 446 papers we gathered 52 relevant related
works, cited in these papers. Thus, the total of papers
obtained was 498. From this amount, there were 48 papers
from ACM (9.6%), 136 from IEEE (27.3%), 28 from Scopus
(5.6%), 181 from ScienceDirect (36.3%), 50 from Springer
(10%), 4 papers from Web of Science (0.8%), 1 paper from
Pubmed (0.2%) and 52 added manually (10.5%). It is
important to note that, although ScienceDirect has had most
of the results, there were not many outcomes related to the
desired area. It happened because this source returned a vast
amount of articles related to cognition and/or blind people,
but under the medical point of view.
To choose the most suitable studies to answer the research
questions, we filtered the papers, according to the inclusion
and exclusion criteria (Figure 2). The including criteria
helped us selecting studies describing multimodal serious
video games, some specific entertainment video games and
navigational virtual environments, whose goal was to
increase cognition. The including criteria also selected
studies describing no application but introducing a model for
the design or the evaluation of multimodal game or
environment for blind people. The exclusion criteria helped
us to eliminate papers related to audiences other than blind
and unrelated to mental models, navigational and similar
cognitive skills.
Figure 2. Filtering process
The first filter (F1) consists of removing the duplicated and
short papers (i.e. less than four pages) and secondary studies
or those published before 1995. F1 excluded 172 papers
(34.5%) so that 326 studies went to the second filter. The
second filter (F2) consists of the application of the specific
exclusion criteria and the inclusion criteria, after reading
papers¶ title and abstract. F2 excluded 216 papers (43.4%)
and included 68 papers (13.7%). These papers went to the
third filter (F3), intending to refine the initially accepted set
of studies. F3 consisted of the examination of the full text of
the 68 articles and the review of the assigned inclusion and
exclusion criteria. F3 excluded 34 articles by criteria and four
duplicated papers (7.6%) and included 30 papers (6%). Most
of the eliminated papers related to cognition, but not to
multimodal games for blind people. From the 30 papers
finally selected for data extraction, one paper was from
ACM, two from IEEE, four from Scopus, two from
ScienceDirect, two from Web of Science. Finally, 19 papers
were added manually, through a snowballing sampling.
The relevant papers are from 1999 to 2014, being 80% of the
papers from 2008 on. The selected papers were: [13, 51, 2,
35, 3, 4, 8, 15, 12, 17, 18, 41, 42, 24, 25, 30, 28, 29, 32, 33,
31, 37, 39, 38, 40, 36, 34, 47, 48, 49]. Among these, 25
papers described 21 distinct applications: 17 multimodal
games and four multimodal navigation virtual environment.
Some papers discussed the same application, but from
another point of view.
CLASSIFICATION OF FEATURES
We analyzed the 21 applications and reported the answers to
the aforementioned research questions in a previous work
[27]. In the present work, we analyze the characteristics
related to design and evaluation of the 21 selected
applications. Based on this analysis, we identified that the
studied multimodal applications differentiate mainly in the
interface and interaction characterization, along with the
cognitive aspects and the evaluation performed. Therefore,
we propose a classification of the features existing in the
multimodal video games and environments in four
dimensions related to Interaction, Interface, Cognition, and
Evaluation. It is important to highlight that the features we
GHVFULEHDUHWKRVHIRXQGRQWKHDSSOLFDWLRQV¶SDSHUV
The proposed classification shows some trends in interface
characterization and the interaction style, as well as
instruments and activities for evaluation of usability and
cognitive impact. Figure 3 gives an overview of the
identified features, discussed in detail in the next sections.
Several elements are common to the design of multimodal
applications aiming to enhance cognition of individuals who
are blind [27]. The video games and environments combine
those elements according to the target cognitive purpose. The
Interface dimension has three kinds of features: Graphics,
Audio and Adaptation. The main features related to the
Interaction dimension are Feedback and Mode, i.e., the way
a user can interact with the application. Table 1 provides an
overview of the applications studied and their classification,
according to the Interface and Interaction dimensions.
Furthermore, we can distinguish the applications by the type
of skill they aim to improve and by the kind of evaluation
performed. The main feature in the Cognitive dimension is
Skill, with six types present in the studied papers. Although
only part of the applications carried out a proper evaluation,
this process usually focuses on the interaction and interface
features that are meaningful to the cognitive enhancement
proposed.
Figure 3. Classification of the key features in multimodal video
games for the cognition of people who are blind
The evaluation concentrates on the aspects of usability and
cognitive impact verification. Even though there is no model
for usability evaluation of multimodal games and no formal
standardization about the elements to evaluate, it is possible
to identify the most used tools and methods. As so, the
features on Evaluation dimension are Usability and
Cognitive Impact, both related to Typical Activities and
Instruments.
DIMENSION 1: INTERACTION
According to the information available in the analyzed video
games and environments, there are seven modes through
what the user can interact, providing input to the application.
The chosen input device determines the style of interaction
available in the application. It also influences the type of
feedback that the application provides to an interaction. The
Interaction Mode comprehends mouse, keyboard, natural
language, force feedback devices, touchscreen (with or
without a stylus), directional pad, and specific devices,
designed for a particular application. The Interaction
Feedback can be either haptic (kinesthetic or tactile),
sonorous or visual. Some applications combine two or more
interaction modes and feedback types.
The keyboard requires no specific learning and allows direct
interaction since it is a familiar input device, common to
diverse kinds of systems and technologies. In fact, 71% of
the researched applications use the keyboard as a primary or
secondary interaction mode, in the desktop paradigm. The
interaction via the keyboard is associated with a sonorous or
visual feedback. The keyboard is a low cost, accessible and
straightforward device, but it provides no sense of touch or
volume during the interaction. Nevertheless, it may be a
Table 1. Classification of the studied applications according to the Interface and Interaction Dimensions
desirable feature in a navigation context to provide non-
visual stimuli that help perceiving the environment physical
characteristics.
Force feedback devices allow the haptic feedback (tactile and
kinesthetic). These devices measure the positions and contact
IRUFHV RI WKH XVHU¶V KDQG GLVSOD\LQJ FRQWDFW IRUFHV DQG
positions to the user [44], even without visual clues. The
applications using force feedback devices allow manual
interactions with the multimedia environment through touch.
It allows the user to explore the environment to extract
information, through the tactile feedback and manipulation
for modifying the environment, via the kinesthetic feedback
[9]. The applications use various devices to allow the
function of haptic feedback. Gamepads and joysticks are the
second most used interaction form, present in 33% of the
studied applications. The applications that use gamepads
focus on the tactile feedback, providing sensations of
vibration, pressure, touch, and texture. The tactile feedback
allows the user to perceive contact force, the geometry of an
object and temperature [9]. The applications that utilize
joystick provide kinesthetic sensation, dealing with forces
resulting from position and velocity of the hand motion and
simulating the force and torque [9]. Those applications that
employ specialized joysticks, e.g. 3D touch controllers,
usually combine tactile and kinesthetic, providing haptic
feedback. The force feedback device always has an
alternative interaction mode, usually the keyboard. It assures
the availability of the game, even without the force feedback.
The most common force feedback device is Novint Falcon,
a 3D USB haptic device used in four applications. However,
Novint Falcon has a high cost, about $250 for the standard
version. So, lower cost devices have also been utilized, such
as OWL joystick, Wiimote, and SideWinder joystick.
The use of mouse and touchscreen with stylus configures a
style of interaction that uses a directional pointer to select
items on a display screen. Due to the visual limitation of the
audience, these are the less frequent interaction modes
identified. Two applications (9%) allows the use of mouse
along with the keyboard and one uses the mouse as the
primary interaction mode. However, this last application
main audience is partially sighted users. The use of mouse
relates to a sonorous-only feedback while the stylus also
provides a tactile feedback. The natural language is still an
underutilized feature (found in two applications). The
feedback to this kind of interaction is sonorous. Unlike the
mouse, we expected a wide use of the natural language in this
type of application, as an easier and instinctive way to
interact. However, the accurate and efficient recognition and
processing of the human language is still a challenge in
computer science. A single application uses the directional
pad, a four-way directional control with one button on each
point, found on most video game console gamepads. In this
context, the directional pad interaction occurs in a mobile
application and its feedback is sonorous.
Another possibility is the development of specific devices to
allow the interaction of the blind users with the multimodal
video game. For instance, the Digital Clock Carpet [35] is a
device based on a usual cane and a simple carpet that inform
the user about directions to a destination point, based on the
hour system. In this particular case, the feedback is both
sonorous and haptic, but adapting a device to an interaction
and feedback specific to the context of a multimodal video
game brings a bunch of new possibilities.
DIMENSION 2: INTERFACE
The interface features that most affect the design of the
studied multimodal video games and environments are
Graphics, Audio and Adaptation (Figure 3). The Graphics
and Adaptation features are directly related, since all the
interface adjustments refer to graphical elements. Regarding
Adaptation, the interfaces support adjustments to provide
accessibility covering the size, color or contrast of the
graphical elements. The features include the resize of the
elements, the possibility of choosing a high contrast mode
and the presentation of relevant information in colorblind
safe colors. The resizing relates mainly to text elements,
which can be resized with no loss of content or functionality.
The high contrast is to provide enough contrast between the
content and its background so that people with low vision
can read it. The use of colorblind safe colors is to assure the
interface presents the visual elements in color combinations
that are perceivable by people with any colorblindness.
The Graphical User Interface (GUI) is a vastly utilized
feature in this kind of application, allowing the user to
interact with graphical elements, through direct
manipulation. In their exploratory study, [50] highlights the
prevalence of visual over other senses in video games and
FRPSDUHVWKHYDVWO\XVHGRSWLRQRI³WXUQRIIVRXQG´ZLWKWKH
option to turn off the graphics. They state that it is
uncommon to turn off the game graphics in a way that it can
still be played and enjoyed. Our results corroborate partially
to that conclusion in the context of multimodal gaming for
cognition in players who are blind since we identified no
video game with a sound-only interface. However, 14% of
the considered applications do allow the users to choose
navigating exclusively by sound, using a graphic interface
for configuration only.
The reason all the considered applications have a graphic
interface, despite the visual impairment of the main
audience, is to increase the interaction possibilities for
partially sighted users. The interface¶V graphics can be either
2D or 3D combined with icons, images, and text. The
association of such elements helps filling some gaps on the
interaction. For instance, children with residual vision have
difficulties recognizing certain 2D icons and not always
associate them with the designed actions, so the use of 3D
icons helps increasing the fidelity of the representation [37].
Besides, with a graphical interface a facilitator can support
the interaction with the video game, observing the navigation
and the cognitive aspects [26].
Although the graphics are substantial features to the
interfaces of the studied multimodal video games, the
essential interface feature is the Audio. The papers showed 6
types of audio that can be used solely or combined to provide
sonorous feedback in a multimodal interface: iconic sound,
spatialized sound, spoken audio, stereo sound, speech
synthesis and abstract earcon. All of the analyzed
applications use at least one aural interface element, although
most of the cases combine two or more aural elements. The
prevailing combination is between iconic and spatialized
sound, in 3D environments. Figure 4 illustrates the use of
each audio feature in the considered applications.
Figure 4. Distribution of the audio features found in the 21
studied applications
Iconic sounds are specific sounds associated with each
available object and action in the environment. Every time
the user executes a certain action or interacts with a particular
object, the corresponding representative sound is heard, for
example, distinct steps sound for different kinds of floor. It
is the most common audio feature, occurring in 16
applications (76%). Spatialized or 3D sounds are stereo
sounds that are digitally processed to appear to come from
particular locations in the three-dimensional space, aiming to
simulate the acoustic experienced by a listener within a
specific environment. A 3D sound navigable environment
can serve as an aural representation of the space and
surrounding entities, helping the players who are blind to
assemble a mental image of an environment [17]. It was the
second most used audio feature, present in 11 (52%)
applications. Spoken audio refers to the use of sentences pre-
recorded in a human voice, usually describing the game
status or relevant information about the actions and objects.
The spoken audio occurs in 11 applications (52%) while the
speech synthesis, which is the artificial creation of the human
voice, appears in seven (33%). Five applications combine
these two approaches. Stereo sound consists in the mixing of
two channels of sound recorded in two separate sources, with
the distinction of left and right channels. Stereo sound is
present in five applications (23%) and can provide
information regarding the nature and location of objects,
using intuitive associations (e.g. the sound of flowing water
representing a fountain). Abstract earcons appear in only one
application, using music/tones to represent different objects.
It refers to the use of sounds unrelated to the elements they
represent. The use of abstract earcons requires the user to
learn with what they are associated. However, it is possible
to represent a much wider range of concepts than by using
iconic sounds [3].
DIMENSION 3: COGNITION
The studied multimodal video games and virtual
environments meticulously combine the Interaction and
Interface Dimensions¶ IHDWXUHV, not only for entertainment
but also for learning and stimulating cognitive processes [35,
3, 4, 8]. The Cognition dimension deals with which cognitive
skills an application aims to improve. Some of the
applications help the development of more than one skill.
Besides, some skills are secondarily stimulated while
playing. The Table 2 summarizes which are the main skills
the studied applications aim to improve. We defined six
types of skills that compose the Cognition dimension: mental
models, mental maps, spatial structures, orientation and
mobility (O&M), problem solving and social collaboration.
Some of the multimodal games for players who are blind did
not specify how they improve cognitive skills. Most of the
applications work O&M skills (44%). It is a set of techniques
that helps visually impaired children and adults to develop
and master the concepts and competencies necessary to be
able to move safely and efficiently within their world. While
navigating the video game environment, people who are
blind can perceive the audio and haptic elements and use
them as references for orientation and mobility [42].
Mental models are constructs that can explain human
behavior and the internal mechanisms that allow people to
understand, explain, and predict the behavior of objects and
systems [20]. The improvement of this skill involves the user
adopting and restructuring a mental model of spatial
dimensions, based on audio and haptic cues after interacting
with a video game [42]. The adequate orchestration between
audio and haptics can also help the user who is blind to build
up a model of the fantasy world and of how a game works
[18, 51]. Specific mental models are stimulated by 27% of
the studied applications.
Mental maps are mental representations of the space being
navigated and its defining features, e.g., overall structure,
spatial components, landmarks, dimensions, and relative
positions [12]. Having a mental map of space is fundamental
to the efficient development of orientation and mobility
techniques [35]. From the applications that improve
cognitive skills, 27% help developing mental maps.
Cognitive spatial structures, stimulated by 16% of the
applications, relates to spatial and temporal reference frames,
implying the connection between visual and haptic space [6].
The spatial properties include location, size, distance,
direction, separation and connection, shape, pattern, and
movement. Humans acquire spatial knowledge and beliefs
directly via sensorimotor systems that operate as they move
about the world [45]. While navigating in multimodal
environments, players who are blind can acquire spatial
knowledge indirectly through the maps and images, 3D
audio and graphics models, and the language.
Table 2. Classification of the applications according to the
Cognitive Dimension
Problem solving can be understood as the act of consciously
apply rules and procedures to bridge the gap between the
initial problem state and a solution state [7]. Working this
skill while playing multimodal video games involves
navigating and interacting, while solving tasks, challenges
and issues [37, 8]. Besides, it may include other
competencies such as learn and interpret points on a two-
dimensional plane [3], and searching, mobilization,
localization and designing strategies [37].
Skills concerning to social collaboration refers to help
multiple people interact and share information to achieve a
common goal. These skills can be improved while
multimodal gaming when blind and sight users solves
collaborative tasks [24]. It creates an inclusive learning
context, where users can work together collaboratively and
achieve shared objectives.
DIMENSION 4: EVALUATION
The last dimension we defined to classify the analyzed
games is Evaluation, which concentrates on two main
aspects: usability and cognitive impact verification (Table 3).
It is important to point out that both evaluations are necessary
to assure the quality of multimodal video games with
cognitive enhancement purposes. According to the ISO/IEC
9126 (2001) quality, relates to "[...] all the features of a
product or service that exert their abilities to meet the stated
or involved needs." It makes clear that usability is an
LPSRUWDQWDVSHFWRIWKHJDPH¶VTXDOLW\+RZHYHUZLWKRXWD
proper cognitive impact evaluation one cannot assure that a
particular application can develop or enhance any cognitive
skills for people with visual disabilities [27].
Table 3. Types of evaluation performed on the considered
applications
The usability evaluation is the most frequent type of quality
assessment in this context, performed more often than
cognitive impact evaluation. Among the analyzed
applications that aim to enhance cognition, only 50%
performed a cognitive impact evaluation while 64% carried
out at least one type of usability evaluation. However, in this
context both evaluations are essential. Table 3 shows the
evaluations performed on the set of analyzed video games
and environments. There is no particular model for the
cognitive impact and usability evaluation of this kind of
multimodal games. However, we identified the most used
instruments and activities for the usability and cognitive
impact evaluation.
For the usability evaluation, we identified three types of
commonly applied instruments: Specialized Questionnaires,
Common Questionnaires, and Likert-based Surveys. The
specialized questionnaires are validated and reusable
instruments, prevailing in the formal evaluations. They
consist of some context-specific sentences, for which the
2 None of the papers presents a bibliographical reference to these
instruments, except for SUBC and EUQ.
users can define the degree of fulfillment on a scale. The
specialized questionnaires2 identified are the Software
Usability for Blind Children Questionnaire (SUBC) [43]; the
End User and Facilitator Questionnaire for Software
Usability (EUQ) [21]; the Software Usability Elements
Questionnaire (SUE), which quantifies the degree to which
the sounds are recognizable; the Open Question Usability
Questionnaire (OQU); and the Initial Usability Evaluation
(IUE). The common questionnaires and the Likert-based
surveys are developed and used circumstantially, to evaluate
a particular application and consist of a set of factual, opinion
and attitude questions. In both cases, the authors themselves
created the instruments, and they do not claim to base it on
any validated instrument or particular formalization. The
surveys are mainly based on the context of the application
and are applied personally or via email [4, 47]. These
instruments are prevalent in informal evaluations.
The typical activities executed in the usability evaluation of
the multimodal games and environments analyzed are
Observation, Interview, and Heuristics Evaluation. The
observation is direct and involves an investigator viewing
users as they use the application, and taking notes on
usability aspects [17]. The interviews are semi-structured
and occur after the user interacts with the application. The
TXHVWLRQV LQWHQG WR HVWDEOLVK WKH VXEMHFW¶V SUHYLRXV
experience, identify aspects of the interface that were helpful
or problematic, and prompt for suggestions on improving the
experience [3]. In the heuristic evaluation, usability experts
LQVSHFW WKH DSSOLFDWLRQ¶V LQWHUIDFH DQG FRPSDUH LW with
usability principles in a checklist. For example, the Heuristic
Evaluation of the Videogame (HEV), based on
6FKQHLGHUPDQ¶V JROGHQ UXOHV DQG 1LHOVHQ¶V XVDELOLW\
heuristics, and the Heuristic Evaluation Questionnaire
(HEQ) [35]. The evaluations often combine such activities
and instruments.
For the cognitive impact evaluation, the instruments most
commonly used are Logs, Checklists, Questionnaires and
Modeling Kits. The four sorts of instruments can be used
throughout the assessment¶V DFWLYLWLHV, combined with
observation and interviews. The goal is to collect data that
will allow an investigator to observe, compare, analyze and
measure the skills and the development of the subjects. The
typical activities identified relates to the research structure of
the quasi-experimental design of non-equivalent groups [1],
considering experimental and control groups and a two-
sample test analysis (pretest-posttest).
The four typical activities identified are Pretest, Training
Tasks, Cognitive Tasks and Postest, performed in this order.
The pretest involves the users executing an activity under
observation so that an investigator can establish WKHVXEMHFWV¶
initial skills [42, 38]. The training tasks refers to the skills
that the users need to have before using the video game [33].
Through the training tasks, the user can be familiar with the
gameplay [36]. The cognitive tasks focus on developing the
specific desired skills based on the software interface [33].
Through the utilization of the video game, each of the skills
in Cognition dimension can be worked on, to strengthen the
development of these skills by using the video game. Once
the cognitive tasks are completed, the posttest takes place, to
determine whether or not there was any impact on the
previously evaluated skills [33, 17, 42]. In the posttest, each
subject is asked to represent in an adequate way the learned
skills. The representation can be either a graphic or physical
model, an oral description, a test or another suitable
approach. Besides, the representation data can be gathered
from, for example, logs and in-depth or structure interviews.
All the generated data is analyzed and compared with the
pretest and cognitive tasks data, determining the cognitive
impact of the application.
CONCLUSION
Multimodal interfaces through video gaming can help to
improve cognitive skills and thus to make that the playful
aspects of the game and their associated technologies
positively influence the motivation of end-users [26]. While
developing these applications, one must carefully consider
several factors, such as the context of use, the desired skills
to be developed and the severity of WKH DXGLHQFH¶V visual
impairment [27]. The analyzed papers show trends in
interface characterization and the interaction style, as well as
instruments and activities for evaluation of usability and
cognitive impact. However, there are some gaps related to
when and how to employ the interface and interaction
elements to IXOILOOWKHDSSOLFDWLRQ¶VFRJQLWLYH UHTXLUHPHQWV
Significant issues remain neglected in the evaluation of
multimodal video games IRU EOLQG OHDUQHU¶V FRJQLWLRQ
enhancement. This paper aimed to help reducing the
problems aforementioned by presenting necessary insights
for the practical understanding of the issues involved in their
design and evaluation of such applications. We proposed a
classification of the features for design and evaluation of
multimodal games for cognition of people who are blind,
considering four dimensions: Interaction, Interface,
Cognition, and Evaluation. Besides, we provide readers with
a comprehensive overview of the multimodal video games
for the cognition of people who are blind works to date. As
future work we intend to propose a methodology for
evaluating the usability and the cognitive impact of interface
elements and interaction components in this kind of games.
The results will allow us to analyze and discuss the impact
of interface elements on cognition and to propose guidelines
for the better practical use of such elements.
ACKNOWLEDGMENTS
This paper is funded by the Program STIC-AmSud-
CAPES/CONYCIT/MAEE, project KIGB-Knowing and
Interacting while Gaming for the Blind, 2014; by the Chilean
National Fund of Science and Technology, Fondecyt
#1150898 and by Basal Funds for Centers of Excellence,
Project FB0003, from the Associative Research Program of
CONICYT
REFERENCES
1. Campbell, D. T., Stanley, J. C. and Gage, N.L.
Experimental and Quasi-Experimental Designs for
Research. Boston: Houghton Mifflin, 1963.
2. Connors, E.C., Chrastil, E.R., Sánchez, J., Merabet,
L.B.: Action video game play and transfer of navigation
and spatial cognition skills in adolescents who are blind.
Frontiers in Human Neuroscience (FNHUM) 8: article
133, 2014. Frontiers (2014)
3. Dulyan, A., Edmonds, E.: AUXie: initial evaluation of a
blind-accessible virtual museum tour. In Proceedings of
the 22nd Australasian Computer-Human Interaction
Conference. Pages 272-275. Brisbane, Australia (2010)
4. Espinoza, M., Sánchez, J., Campos, M. B.:
Videogaming Interaction for Mental Model
Construction in Learners Who Are Blind. In
Proceedings of 8th International Conference, UAHCI
2014, Held as Part of HCI International 2014, pages
525-536. Crete, Greece (2014)
5. Fabbri, S., Hernandes, E., Di Thommazo, A., Belgamo,
A., Zamboni, A., Silva, C.: Managing literature reviews
information through visualization. In International
Conference on Enterprise Information Systems.14th.
ICEIS, Wroclaw, Poland, Jun, 2012. Lisbon:
SCITEPRESS (2012)
6. Freksa, C. Spatial and temporal structures in cognitive
processes. In: Foundations of computer science.
Springer Berlin Heidelberg, 1997. p. 379-387.
7. Glatzeder, B.M. (eds.), Towards a Theory of Thinking,
On Thinking, 3 DOI 10.1007/978-3-642-03129-8_1, ©
Springer-Verlag Berlin Heidelberg 2010
8. Guerrero, J., Lincon, J.: AINIDIU, CANDI, HELPMI:
ICTs of a personal experience. Engineering
Applications (WEA), 2012 Workshop on. Pages 1-7.
Bogotá, Colombia (2012)
9. Hamam, A., Eid, M. and Saddik. A. 2013. Effect of
kinesthetic and tactile haptic feedback on the quality of
experience of edutainment applications. Multimedia
Tools Appl. 67, 2 (2013)
10. Kitchenham, B. and Charters, S.: Guidelines for
performing systematic literature reviews in software
engineering. Technical Report EBSE 2007-001, Keele
University and Durham University Joint Report (2007)
11. Kolb, B and Whishaw, I.: Neuropsicología Humana 5a
edición. Editorial Médica Panamericana, 2006, pp. 555
(2006)
12. Lahav, O. and Mioduser, D.: Construction of cognitive
maps of unknown spaces using a multysensory virtual
environment for people who are blind. Computers in
Human Behavior 24(3), pp. 1139-1155 (2008)
13. Lahav, O. and Mioduser, D.: Haptic-feedback support
for cognitive mapping of unknown spaces by people
who are blind. International Journal on Human-
Computer Studies, 66, pp. 23±35 (2008)
14. Lahav, O. Schloerb, D.W. Kumar, S. and Srinivasan,
M.A.: BlindAid: A learning environment for enabling
people who are blind to explore and navigate through
unknown real spaces. Proceedings Virtual
Rehabilitation 2008 Conference, Vancouver, Canada,
pp.193-197 (2008)
15. Lahav, O., Schloerb, D. W., Srinivasan, M. A.: Virtual
Environment System in Support of a Traditional
Orientation and Mobility Rehabilitation Program for
People Who Are Blind. Presence, vol. 22, pages 235-
254. MIT Press (2013)
16. Lewis-Beck, M.S., Bryman, A., and Liao, T.F.: The
SAGE Encyclopedia of Social Science Research
Methods (2004)
17. Lumbreras, M., Sánchez, J.: Interactive 3D Sound
Hyperstories for Blind Children. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems, pages 318-325, May 1999. Pittsburgh, USA
(1999)
18. Mccrindle, R. J., Symons, D.: Audio Space Invaders. In
Proceedings of the Third International Conference on
Disability, Virtual Reality and Associated
Technologies. Pages 59-65 (2000)
19. Petersen, K., Feldt, R., Mujtaba, S. Mattsson, M.:
Systematic mapping studies in software engineering, in:
($6( ¶ 3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO
Conference on Evaluation and Assessment in Software
Engineering, University of Bari, Italy. (2008)
20. Rouse, W. B., & Morris, N. M. (1986). On looking into
the black box: Prospects and limits in the search for
mental models. Psychological bulletin, 100(3), 349.
21. Sánchez, J. End-user and facilitator questionnaire for
Software Usability (EUQ). Usability evaluation test.
University of Chile (2003)
22. Sánchez, J. and Tadres, A.: Audio and haptic based
virtual environments for orientation and mobility in
people who are blind. In Proceedings of the 12th
international ACM SIGACCESS conference on
Computers and accessibility (ASSETS '10). ACM, New
York, NY, USA, 237-238 (2010)
23. Sánchez, J. and Zúñiga, M.: Evaluating the Interaction
of Blind Learners with Audio-Based Virtual
Environments. Cybersychology & Behavior, Volume 9,
Number 6, 2006, pp. 717 (2006)
24. Sánchez, J., Aguayo, F.: AudioGene: Mobile learning
genetics through audio by blind learners. Learning to
live in the knowledge society. IFIP 20th World
Computer Congress, IFIP TC 3 ED-L2L Conference
September, 2008. Pages 79-86. Milano, Italy (2008)
25. Sánchez, J., Campos, M. B., Espinoza, M., Merabet, L.
B.: Audio Haptic Videogaming for Developing
Wayfinding Skills in Learners Who are Blind. In Proc.
ACM International Conference on Intelligent User
Interfaces (ACM IUI), pp. 199-208, Feb 2014. Haifa,
Israel (2014)
26. Sánchez, J., Campos, M.B. and Espinoza, M. 2014.
Multimodal gaming for navigation skills in players who
are blind. In Proceedings of the 13th Brazilian
Symposium on Human Factors in Computing Systems
(IHC '14). Sociedade Brasileira de Computação, Porto
Alegre, Brazil, Brazil, 169-179.
27. Sánchez, J., Darin, T. and Andrade, R. Multimodal
Videogames for the Cognition of People Who Are
Blind: Trends and Issues. Universal Access in Human-
Computer Interaction. Access to Learning, Health and
Well-Being. Springer International Publishing, 2015.
535-546.
28. Sánchez, J., Espinoza, M., Campos, M. B., Leite, L. L.
(2014) Modeling Videogames for Mental Mapping in
People Who Are Blind. In Proceedings of 8th
International Conference, UAHCI 2014, Held as Part
of HCI International 2014, pages 605-616. Crete,
Greece.
29. Sánchez, J., Espinoza, M., Campos, M. B., Merabet, L.
B.: Enhancing Orientation and Mobility Skills in
Learners who are Blind through Videogaming. In
Proceedings ACM Creativity and Cognition (ACM
C&C), pp. 353-356, Jun 2013. Sydney, Australia (2013)
30. Sánchez, J., Espinoza, M.: Designing Serious
Videogames through Concept Maps. In M. Kurosu (ed.),
Proc. 17th Human-Computer Interaction International
(HCII), pp. 299-308, volume 2, Jul 2013. Las Vegas,
Nevada, USA. Springer-Verlag, Berlin/Heidelberg,
Germany. Lecture Notes in Computer Science (2013)
31. Sánchez, J., Flores, H.: Virtual Mobile Science Learning
for Blind People. Cyberpsychology & behavior: the
impact of the Internet, multimedia and virtual reality on
behavior and society, vol. 11, number 3 (2008)
32. Sánchez, J., Guerrero, L., Saenz, M., Flores, H.: A
Model to Develop Videogames for Orientation and
Mobility. In Klaus Miesenberger, Joachim Klaus,
Wolfgang Zagler, Arthur Karshmer (ed.), Proc. 12th
International Conference on Computers Helping People
with Special Needs (ICCHP), pp. 296-303, Jul 2010.
Vienna, Austria. Springer-Verlag, Berlin/Heidelberg,
Germany. Lecture Notes in Computer Science vol. 6180
(part II) (2010)
33. Sánchez, J., Mascaró, J.: Audiopolis, Navigation
through a Virtual City Using Audio and Haptic
Interfaces for People Who Are Blind. In Constantine
Stephanidis (ed.), Proc. 15th Human-Computer
Interaction International (HCII), pp. 362-371, Jul 2011.
Orlando, Florida, USA. Springer-Verlag,
Berlin/Heidelberg, Germany. Lecture Notes in
Computer Science (2011)
34. Sánchez, J., Sáenz, M. (2006). Three-Dimensional
Virtual Environments for Blind Children.
CyberPsychology & Behavior. Vol. 9, issue 2, page 200.
35. Sánchez, J., Saenz, M., Garrido, J.M. Usability of a
multimodal video game to improve navigation skills for
blind children. ACM Transactions on Accessible
Computing. Proceedings of the 11th international ACM
SIGACCESS conference on Computers end
Accessibility. Pages 35-42. (2010)
36. Sánchez, J., Sáenz, M., Pascual-Leone, A., Merabet, L.
B.. Enhancing Navigation Skills through Audio
Gaming. In Proc. ACM Conference on Human Factors
LQ &RPSXWLQJ 6\VWHPV &+,¶ SS -3996, Apr
2010. Atlanta, GA, USA. (2010)
37. Sánchez, J., Sáenz, M.: 3D sound interactive
environments for blind children problem solving skills.
Behaviour & IT. Vol. 25, issue 4, pages 367-378. From
the 7h International ACM SIGACCESS Conference on
Computers and Accessibility. Baltimore, USA (2006)
38. Sánchez, J., Sáenz, M.: Metro Navigation for the Blind.
Computers and Education (CAE) 55(3):970-981, Jul
2010. Elsevier Science, Amsterdam, The Netherlands
(2010)
39. Sánchez, J., Sáenz, M.: Video Gaming for Blind
Learners School Integration in Science Classes. In Tom
Gros, Jan Gulliksen, Paula Kotzé, Lars Oestreicher,
Philippe Palanque, Raquel Oliveira Prates, Marco
Winckler (ed.), Proc. 12th IFIP TC13 Conference on
Human-Computer Interaction (Interact), pp. 36-49, Aug
2009. Uppsala, Sweden. Springer-Verlag,
Berlin/Heidelberg, Germany. Lecture Notes in
Computer Science vol. 5726 (Part I) (2009)
40. Sánchez, J., Tadres, A., Pascual-Leone, A., Merabet, L.:
Blind children navigation through gaming and
associated brain plasticity. In Virtual Rehabilitation
International Conference 2009, June 29-July 2, 2004,
Haifa, Israel, 29-36 (2009)
41. Sánchez, J.: A model to design interactive learning
environments for children with visual disabilities.
Education and Information Technologies. Vol. 12, issue
3, pages 149-163. Kluwer Academic Publishers-Plenum
Publishers (2007)
42. Sánchez, J.: Development of navigation skills through
audio haptic videogaming in learners who are blind. In
Proc. Software Development for Enhancing
Accessibility and Fighting Info-H[FOXVLRQ '6$,¶
pp. 102-110, Jul 2012. Douro, Portugal (2012)
43. Sanchez, J.: Software Usability for Blind Children
Questionnaire (SUBC). Usability evaluation test,
University of Chile (2003)
44. Singh, J. Mendeley: A free research management tool
for desktop and web. Journal of pharmacology &
pharmacotherapeutics 1, 62-63 (2010).
45. Smelser, N. J. and Baltes, P.B. (Eds.), International
Encyclopedia of the Social & Behavioral Sciences (pp
14771-14775). Oxford: Pergamon Press. 2001
46. Tan, H. Z., Srinivasan, M. A., Eberman, B., & Cheng,
B. (1994). Human factors for the design of force-
reflecting haptic interfaces. Dynamic Systems and
Control, 55(1), 353-359.
47. Torrente, J., Blanco, Á. d., Serrano-Laguna, Á., Vallejo-
Pinto, J. Á., Moreno-Ger, P., Fernández-Manjón, B.:
Towards a Low Cost Adaptation of Educational Games
for People with Disabilities. Computer Science and
Information Systems, Vol. 11, No. 1, 369±391 (2014)
48. Torrente, J., Blanco, A., Moreno-Ger, P., Martinez-
Ortiz, I., Fernandez-Manjon, B.: Implementing
Accessibility in Educational Videogames with e-
Adventure. In Proceedings of the 1st ACM International
Workshop on Multimedia Technologies for Distance
Learning. Pages 57-66. Beijing, China (2009)
49. Trewin, S., Hanson, V. L., Laff, M. R., Cavender, A.:
PowerUp: An Accessible Virtual World. In Proceedings
of the 10th international ACM SIGACCESS conference
on Computers and accessibility, pages 177-184. Halifax,
Canada (2008)
50. Valente, L., Sieckenius, C.S. and Feijó, B. 2008. An
exploratory study on non-visual mobile phone interfaces
for games. In Proceedings of the VIII Brazilian
Symposium on Human Factors in Computing Systems
(IHC '08). Sociedade Brasileira de Computação, Porto
Alegre, Brazil, Brazil, 31-39.
51. Westin, T.: Game accessibility case study: Terraformers
± a real-time 3D graphic game. In: Proceedings of the
5th International Conference on Disability, Virtual
Reality and Associated Technologies, ICDVRAT 2004,
Oxford, UK, pp. 95±100 (2004)
52. Yuan, B. and Folmer, E.: Blind hero: enabling guitar
hero for the visually impaired. In Proceedings of the
10th International ACM SIGACCESS Conference on
Computers and acces-sibility, pp 69-176 (2008)
53. Yuan, B.: Towards Generalized Accessibility of Video
Games for the Visually Impaired. Ph.D. Dissertation.
Univ. of Nevada, Reno, Reno, NV, USA. Advisor (S
Frederick C. Harris and Eelke Folmer. AAI3355610.
(2009)
SAME SIZEmmmmmmmm
