ArticlePDF Available
Universal Accessibility as a Multimodal Design Issue
Zeljko Obrenovic, Julio Abascal, Dusan Starcevic
In recent years, many research activities have focused on design that aims to produce
universally accessible systems, taking into account special needs of various user groups.
These special needs are associated with various factors, including speech, motor, hearing,
and vision impairments, cognitive limitations, emotional and learning disabilities, as well as
aging, but also with various environmental factors [3].
Fields that address this problem, such as Usability, Universal Accessibility, Universal
Design, or Inclusive Design [8] have been developed as relatively independent domains, but
they share many things with other HCI disciplines. However, researchers and practitioners
are often not aware of interconnections among concepts of universal accessibility and
"ordinary" HCI. In view of that, in this paper we wanted to show that there is a fundamental
connection between multimodal interface design and universal accessibility, and that
awareness of these links can help both disciplines. Researchers from these areas may use
different terminology, but they can mean pretty much the same. Based on these ideas, we
propose a unified generic framework where these areas can be joined.
Accessibility and Multimodal Interaction
Universal accessibility and related approaches such as "Inclusive Design" or "Design for All",
aim to produce systems that can be used by everyone, regardless of their physical or
cognitive skills. As this design philosophy tend to enhance the usability of the product, it can
also be extremely valuable for non-disabled users trying to use the system under suboptimal
conditions [3]. The growing interest in accessibility and universal usability for information
and communications technologies have resulted in various solutions that developers can use.
For example, many guidelines about accessibility, especially for Web design, are already
available [10]. In addition, conferences such as ACM Conference on Universal Usability
(CUU), and ACM Conference on Assistive Technologies (ASSETS), as well as journals such
as International Journal on Universal Access in the Information Society, offer good sources of
various practical and theoretical work in this area. Developers can also use various practical
solutions and tools, such as Web site compliance checkers, semi-automatic Web site repair
tools, or Web adaptation facilities that transform existing Web content "on the fly". There are
also activities in developing tools that use of guidelines to automatically verify Web
accessibility [1].
Multimodal interaction is a characteristic of everyday human discourse, in which we speak,
shift eye gaze, gesture, and move in an effective flow of communication. Enriching human-
computer interaction with these elements of natural human behavior is the primarily task of
multimodal user interfaces. Many studies have explored multimodal interaction from
different viewpoints [2]. Sharon Oviatt gave a practical definition of multimodal systems,
saying that they combine human natural input modalities—such as speech, pen, touch, hand
gestures, eye gaze, and head and body movements—in a coordinated manner with
multimedia system output [6]. Matthew Turk and George Robertson further refined the
difference between multimedia and multimodal systems, saying that multimedia research
focuses on the media, while multimodal research focuses on the human perceptual channels
[9]. Multimodal interfaces can improve accessibility for diverse users and usage contexts,
advance performance stability, robustness, expressive power and efficiency of
communication [6].
While multimodal interaction research focuses on adding more natural human
communication channels into human-computer interaction (HCI), accessibility research is
looking for substitute ways of communication when some of these channels, due to various
restrictions, are of limited bandwidth. What makes a difference between these two areas is a
focus of their research. Therefore, many things from both areas can be generalized so that we
can get unified and more abstract view on them. In this way, some existing solutions from
one of the domains could find their usage in another domain.
The Unified Framework
Treating user interfaces as multimodal systems, can clearly help design for universal
accessibility, as multimodal interfaces describe human-computer interaction in terms of
modalities, e.g. in terms of communication channels established between the computer and
the user. Limiting environment characteristics or limited abilities of a user can be viewed as a
break or decrease of throughput in these channels (Figure 1).
Figure 1. Modalities, constraints, and effects. Computers and humans establish various
communication channels over which they exchange messages or effects. Modalities process or
produce these effects, while various interaction constraints reduce or completely eliminate some of
these effects.
If we describe user interfaces as a set of communication channels, and connect these
descriptions with user, environment, and device profiles, we can easily see if the multimodal
interface will be appropriate for the user in a specific situation. However, to create a unified
view on multimodal system design and accessibility, we need a semantic framework where
we can explicitly and formally establish relations among concepts from both domains.
Therefore, our first step was to formally define a unified modeling framework for
description of multimodal human-computer interaction and various user and environment
characteristics using the same terms. Proposed framework does not define any specific
interaction modality - such as speech, gesture, graphics, and so on - nor a constraint, such as
low vision, immobility, or various environment conditions, but defines a generic unified
approach for describing such concepts. Proposed framework, therefore, focuses on the
notions of an abstract modality and abstract constraint, defining their common
characteristics regardless of their specific manifestations. This work is the extension of our
previous work in modeling multimodal human-computer interaction [4].
The Model of Multimodal HCI and Constraints
Our approach is based on the idea that user interfaces can be viewed as one-shot, higher-
order messages sent from designers to users [7]. While designing a user interface the
designer defines an interactive language that determines which effects and levels will be
included in the interaction. Therefore, we model user interfaces with modalities they use,
where we define a modality as a form of interaction designed to engage some of human
capabilities, e.g. to produce effects on users, or to process effects produced by the user
(Figure 2a). In our model, modalities can be simple or complex: a complex modality integrates
other modalities to create simultaneous use of them, e.g. to provide modality fusion of
fission mechanisms, while a simple modality represents a primitive form of interaction. In the
paper we do not focus on the detailed description of multimodal integration, but on high-
level effects that a modality system or some of his parts use. We defined input and output
types of a simple modality, using the computer as a reference point. An input modality
requires some user devices to transfer human output into a form suitable for computer
processing, and we classified them into event-based and streaming-based classes. Event-based
input modalities produce discrete events in reaction to user actions, such in the case of user
input via a keyboard or mouse. Streaming-based modalities sample input signals with some
resolution and frequency, producing the time-stamped array of sampled values. We
introduced a special class of streaming modality, a recognition-based modality, which adds
additional processing over streaming data, searching for patterns. An output modality
presents data to the user, and this presentation can be static or dynamic. More elaborate
description of this model can be found in [4].
While we describe human-computer interaction in terms of modalities, we describe various
accessibility issues in terms of interaction constraints (Figure 2b). Interaction constrictions can
be viewed as filters on usage of some effects. Constraints are organized as basic and complex.
We have identified two types of basic constraints: user constraints and external constraints.
User constraints are classified into user features, user states and user preferences. User features
describe the longterm ability of user to exploit some of the effects, and this description can
include some of the user disabilities, such as low vision or immobility. A user state constraint,
further classified in emotional and cognitive context, describes user's temporary ability to use
some effects. User preferences describe how much is the user eager to make use of some
effects, e.g. it is a user's subjective mark of the effects they prefer or dislike.
Figure 2. Simplified model of computing modalities (a) and constraints (b).
External constraints are categorized as device constraints, environment constraints, and social
context. Device constraints describe restrictions on usage of some effects which are a
consequence of device characteristics. For example, a mouse is limited to capture movement
in two-dimensional space with some resolution, while output devices, such us screens on
PDAs and other mobile devices, have limited resolution and a limited number of colors.
Environmental constraints describe how the interaction environment influences the effects. For
example, when driving a car, in most of the situations, users are not able to watch the screen
and, therefore, this situation greatly reduces usage of visual effects. In addition, various
other environmental factors, such as lightning or noise, greatly affect the usage of other
effects. Social context describes social situation in which the interaction occurs. Proposed
model allows flexible definition of various simple and complex constraints of different types.
The resulting constraint in a particular situation will be a combination of user's state, abilities
and preferences, as well as various external factors relevant to that situation.
Common ground: The effects
Entities that connect modalities and constraints in our model are effects. We have classified
effects used by modalities and affected by constraints in five main categories [4]:
Sensual effects,
Perceptual effects,
Motor effects,
Linguistic effect, and
Cognitive effects.
These effects are based on various sources, such as World Health Organization International
Classification of Functioning, Disability and Health. In our model, these concepts are
subclasses of the Effect class presented in Figure 2. Sensory effects describe processing of
stimuli performed by human sensory apparatus. Perceptual effects are more complex effects
that human perceptual system gets by analyzing data received from sensors, such us, shape
recognition, grouping, highlighting, or 3D cues. Motor effects describe human mechanical
action, such as hand movement or pressure. Linguistics effects are associated with human
speech, listening, reading and writing. Cognitive effects take place at higher level of human
information processing, such as memory processes, or attention.
Effects are often interconnected. For example, all perceptual effects are a consequence of
sensory effects. These relations among effects are important because in this way a designer
can see what side-effects will be caused by his intention to use some effects.
Using the Framework
Proposed unified framework can be used to describe various interaction modalities and
interaction constraints. By combining these descriptions, and by using effects as a common
ground, it is possible to see if the designed interface will be appropriate for a concrete
situation, and it can enable adaptation of user interfaces according to user profiles and
situational parameters.
Describing user interfaces, users and environment
We model user interfaces with their modalities, where we describe each modality with effect
it uses in order to be operative. For example, Table 1 shows effects produced by some
common simple or complex modalities, such as simple text presentation, aimed hand
movement, visual menu interaction, and speech user interfaces.
Table 1. A list of some of the modalities and associated effects.
Modality Composed of: Effect which the modality uses. Effect type
Pixel Visual sensory processing sensory
Correct central field vision
Normal vision sharpness
Shape recognition of grouped pixels perceptual
Grouping of letters by proximity Word
Shape recognition of words
Text line Grouping of words by good continuation perceptual
Simple textual
Paragraph Grouping of lines by proximity
Highlighting by the shape of the first line
Hand movement Hand movement
(input modality) Pressure
Highlighting by shape of the cursor
Highlighting by motion
Highlighting by depth (cursor shadow)
Aimed hand
Visual feedback
Attention cognitive
Selection See aimed hand movement. -
Grouping by surrounding of menu borders perceptual
Grouping of items by proximity perceptual
Highlighting by shape and color (selected
Visual reading of item text linguistic
Visual menu
Understanding of the menu language linguistic
Speech input Speaking linguistic
Listening linguistic
Simple speech
interaction Speech output
Attention cognitive
Concrete user interface can then be described using these high-level descriptions of
modalities, while we can get detailed description of used effects automatically through
mappings, such us those shown in Table 1. It is also possible to have several alternative
mappings among modalities and effects according to different theories. For example, simple
textual presentation in Table 1 is described according to Gestalt psychology, but it is also
possible to provide description of these modalities according to the other theories.
Accessibility issues, such as user abilities and environmental constrains, are described with
the constraints. User abilities can be described in several ways. For example, one approach is
to create individual profiles of each user, associating all the effects with values describing the
user's capability to exploit them. For simplicity, the profile could include only the effects
which are different from a typical user. Alternatively, it is possible to define a repository or
user ability categories, where each category is described with a set of effects that it reduces.
These categories can describe some disability, or other factors, such as average abilities of
different age groups. For example, Table 2 shows effects that are reduced by some
disabilities. In modeling and analyzing user interfaces, very important role play the relations
among effects. If, for example, we describe that the user is not capable of processing a sound, it
means not only sensory, but also all the audio perceptual effects will not be appropriate for
that user.
Table 2. A list of some of the disabilities and associated effects.
Disability as a user ability
Effects reduced by a disability
Blindness Absence of all visual stimulus processing, and therefore
associated visual perceptual effects
Poor acuity (poor sharpness) Reduced visual sharpness
Clouded vision Reduced visual sharpness
Tunnel vision Reduced peripheral vision
Central field loss Reduced central vision
Color blindness Reduced color sensation and contrast processing
Deafness Absence of all audio stimulus processing, and associated
audio perceptual effects
Hard of Hearing Reduced audio sensory processing, and associated audio
perceptual effects
Weakness Reduced movement and pressure
Limitations of muscular control Reduced movement and pressure
Limitations of sensation Reduced pressure
Joint problems Reduced movement
Pain associated with movement Reduced movement
Dyslexia Reduced linguistic effects
Attention Deficit Disorder Reduced attention
Memory Impairments Reduced memory processes
In a similar way we can describe constraints introduced by the environment conditions. For
example, driving a car is a complex constraint that integrates different user end
environmental parameters. Table 3 show simplified description of this constraint where we
identified that it depends on traffic situation, weather conditions, noise level, lighting, user
current state, as well as a number of people in the car. Constraints can also be
interconnected, for example, lightning and weather conditions can affect the user current
state, while a number of people in the car can influence the noise level. This description can
be useful in determining which modalities to use in particular situation. When the car is
stopped, it is possible to use central field vision of the user, as well as the other effects in
higher amount. On the other hand, traffic jam further limits possible usage of these effects,
allowing their use in a very low level.
Table 3. Driving a car as a complex interaction constraint, composed of various simpler
constraints, with associated effects.
Constraints Situations
(constraint subclasses)
Influence on the usage of effect.
Car stopped No specific reductions.
Normal traffic situation It is not convenient to require of the user to
use hands
. Also, user's
central field
is directed toward the road.
Traffic situation
Traffic jam In addition to the normal traffic situation,
additional limitation is usage of user
as the user is more focused and stressed.
Insignificant noise level No specific reductions.
Normal noise level A user's
audio perception
audio 3D cues
, and
can be used provided that they are of
significant intensity.
Noise level
High noise level
All audio effects
are significantly reduced.
Day No specific reductions.
constraint) Fog
Driving conditions are tougher; user is more
focused and stressed.
Dry No specific reductions.
Weather condition
constraint) Snowy
Driving conditions are tougher; user is more
focused and stressed.
The driver is relaxed No specific reductions. User current state
(emotional context) The driver is stressed Limited usage of attention requests and
complex interaction modalities.
The driver is alone No specific reductions. Number of people in
the car
(social context)
The driver is not alone
Other user
can use the application. Can affect
the noise level.
Analysis and Transformations
Presented descriptions of multimodal interfaces and interaction constraints can be used for
various purposes. In a simpler form, they can serve as metadata about some user interface, or
as a part of a user profile. However, with formal descriptions of user interfaces and
constraints, it is possible to develop tools that analyse and transform the content in order to
see if it is suitable for some situation or for some user.
We have developed a Web service, which have formalized the proposed framework, creating
a database of effects, and standard description of modalities, and constraints. This service
can receive the description of user interface, expressed in terms of modalities, and then
evaluate it, for example, giving the list of effects, or giving a list of potential problems in
some environments, as well as the list of user groups that could have a problem to use this
interface. To increase the awareness about importance of accessibility aspects, we also
enabled that these reports contain data about percentage of people who suffer from some
interaction limitations (for example, about 8% of men and 0.4% of women have some form of
color blindness).
Various other applications, such as dynamic adaptation and content repurposing are also
possible. By connecting descriptions of user interfaces, user profiles, and other constraints,
we can analyze and transform content in various ways. Proposed framework, therefore, can
be a good basis for adaptation and content repurposing, that attack the problem of
developing content for various users and devices. More detailed description of our previous
work in this area can be found in [5].
Discussion and Conclusions
Proposed approach can bring developers and researchers several advantages. From the
developer's point of view, one advantage is that it is possible to design more flexible and
more reusable solutions, aimed for a broader set of situations. Most of the previous work in
designing solutions for people with disabilities concentrated on a specific set of disabilities,
or on specific situations. Having in mind great diversity of disabilities and situations, it is
clear that development and maintenance of such systems is rather complex. With our
approach, developers can concentrate on more generic effects, providing solutions for
different levels of availability of specific effects. In this way it is possible to create adaptable
solutions which adjust to user features, states, preferences, and environmental
Another important advantage is that our framework enables treating different situations in a
same way. As user features and preferences are described in the same way as environmental
characteristics, it is possible to use solutions aimed for user with some disability, for non-
disabled user in situations that limit the interaction in the same way as some disability limits
the other user. Besides providing more universal solutions, this could also solve some of the
ethical problems, as design is not concerned with disabilities, and usage of the term
'disability' often introduces negative reactions, but with various effects and their constraints.
Some of the constraints are not a consequence of user physical limitations. For example,
when using a secondary language, a foreign user may experiences similar problems as the
user who has cognitive disabilities that affect linguistic effects. Therefore, these situations do
not have to be treated differently, and some of the solutions from one of the domains can be
reused in another.
1. J. Abascal et al, "The use of guidelines to automatically verify Web accessibility", Universal Access in
the Information Society, Vol. 3, No. 1, 71-79 (2004).
2. M.M. Blattner and E.P. Glinter, "Multimodal Integration", IEEE Multimedia, Winter, 14-24 (1996).
3. C. Nicolle, J. Abascal (Eds.) Inclusive Design Guidelines for HCI. Taylor & Francis. London (2001).
4. Z. Obrenovic and D. Starcevic, "Modeling Multimodal Human-Computer Interaction", IEEE
Computer, Vol. 37, No. 9, 62-69 (2004).
5. Z. Obrenovic, D. Starcevic, and B. Selic, "A Model Driven Approach to Content Repurposing", IEEE
Multimedia, Vol. 11, No. 1, 62-71 (2004).
6. S. Oviatt, T. Darrell, and M. Flickner, "Multimodal interfaces that flex, adapt, and persist", Comm. of
the ACM, Vol. 47 , No. 1, 30-33 (2004).
7. R. Prates, C. De souza, and C. Barbosa, “A Method for Evaluating the Communicability of User
Interfaces”, Interactions, Jan– Feb 2000, pp. 31-38.
8. A. Savidis and C. Stephanidis, "Unified User Interface Design: Designing Universally Accessible
Interactions", International Journal of Interacting with Computers, 16 (2), pp. 243-270, (2004).
9. M. Turk, G. Robertson, "Perceptual user interfaces (introduction)", Comm. of the ACM, Vol. 43, No.
3, March 2000, pp. 33-35.
10. Web Accessibility Initiative (WAI),, Last visited September 2004.
... It is strongly related to Shneiderman (2000)'s idea of 'universal usability': break technology barriers, bridge knowledge gaps, and accommodate user diversity. Example approaches to broaden the accessibility of cartographic products include: the use of multi-layered interfaces to bridge the knowledge gap between novice and experts (see Roth, 2013b); and the use of multi-modal interaction to engage a multiplicity of user capabilities in the computer-human communication process (see Obrenovic, Abascal, and Starcevic, 2007). Challenges of accessible maps were recently discussed in (Froehlich et al., 2019), and recommendations to make web maps more accessible were suggested in (Hennig, Zobl, and Wasserburger, 2017). ...
Full-text available
The availability of open data and of tools to create visualizations on top of these open datasets have led to an ever-growing amount of geovisualizations on the Web. There is thus an increasing need for techniques to make geovisualizations FAIR - Findable, Accessible, Interoperable, and Reusable. This article explores what it would mean for a geovisualization to be FAIR, presents relevant approaches to FAIR geovisualizations and lists open research questions on the road towards FAIR geovisualizations. The discussion is done using three complementary perspectives: the computer, which stores geovisualizations digitally; the analyst, who uses them for sensemaking; and the developer, who creates them. The framework for FAIR geovisualizations proposed, and the open questions identified are relevant to researchers working on findable, accessible, interoperable, and reusable online visualizations of geographic information.
... The growing possibilities with auditory display allows for the expansion beyond visual display more efficiently than ever before, and creates opportunities to compliment, augment, or replace visual displays, inviting designers and educators to expand their modal palette and reimagine traditional graphics-centric learning tools, incorporate and express musical and linguistic skills, and increase the capacity of digital learning tools to support inclusive learning opportunities. Inclusive learning tools include multimodal displays that can be adapted [3] to meet the needs of the educator and learner(s) in the moment-decreasing needs for alternative material creation for learners with diverse needs, and increasing opportunities for learning and collaboration amongst learners with diverse needs. Here we share findings from an investigation of teacher and student perceptions related to multimodality and inclusion after using interactive physics simulations with non-speech auditory display (sonifications and sound effects). ...
... Multimodal interfaces have been considered to improve accessibility for a number of users and usage contexts (Obrenovic, Abascal, and Starcevic 2007), including the diverse needs of older users (Himmelsbach et al. 2015;Munteanu and Salah 2017). ...
Full-text available
This paper presents key aspects and trade-offs that designers and Human-Computer Interaction practitioners might encounter when designing multimodal interaction for older adults. The paper gathers literature on multimodal interaction and assistive technology, and describes a set of design challenges specific for older users. Building on these main design challenges, four trade-offs in the design of multimodal technology for this target group are presented and discussed. To highlight the relevance of the trade-offs in the design process of multimodal technology for older adults, two of the four reported trade-offs are illustrated with two user studies that explored mid-air and speech-based interaction with a tablet device. The first study investigates the design trade-offs related to redundant multimodal commands in older, middle-aged and younger adults, whereas the second one investigates the design choices related to the definition of a set of mid-air one-hand gestures and voice input commands. Further reflections highlight the design trade-offs that such considerations bring in the process, presenting an overview of the design choices involved and of their potential consequences.
... This operator takes an action and an obtrusiveness level and returns the set of concrete interaction mechanisms required for that action. Some examples of the relationship between the obtrusiveness levels and the actions for the Takeover task are the following: The selection of the concrete interaction mechanisms that best suit each obtrusiveness level can be based on existing multimodal design taxonomies and frameworks that depend on the cognitive characteristics of interaction modalities, i.e., how information carried by different modalities is perceived by the human perceptual-sensory system [39][40][41]42]. For example, an image uses less mental workload than a text, and visual-auditory combinations impose less cognitive load than visual combinations [39]. ...
Full-text available
Autonomous capabilities are required in AmI environments in order to adapt systems to new environmental conditions and situations. However, keeping the human in the loop and in control of such systems is still necessary because of the diversity of systems, domains, environments, context situations, and social and legal constraints, which makes full autonomy a utopia within the short or medium term. Human-system integration introduces an important number of challenges and problems that have to be solved. On the one hand, humans should interact with systems even in those situations where their attentional, cognitive, and physical resources are limited in order to perform the interaction. On the other hand, systems must avoid overwhelming the user with unnecessary actions. Therefore, appropriate user-centered methods for AmI development should be used to help designers analyze and design human-in-the-loop interactions in AmI environments. This paper presents a user-centered design method that defines a process with a set of tools and techniques that supports the process steps in order to systematically design, prototype, and validate human-in-the-loop (HiL) solutions. The process starts with the definition of the HiL design, which defines how the system cooperates with the human. This HiL design is built using a conceptual framework that focuses on achieving human-system interactions that get human attention and avoid obtrusiveness. Then, we provide a software infrastructure to generate a prototype based on the HiL design and validate it by having end-users use a web simulator. The feedback data generated during the prototype user validation is gathered and used by a machine learning tool that infers the user’s needs and preferences. Finally, these inferences are used to automatically enhance the human-in-the-loop designs and prototypes. We have validated the proposed method through a twofold perspective: an experiment to analyze the perception of interaction designers regarding their acceptance of the design method and another experiment to evaluate the usefulness of the “smart” prototyping technique. The results obtained point out the acceptability of the proposed method by designers and the useful adaptations provided by the “smart” prototyping technique to achieve a HiL design that adapts well to users’ preferences and needs.
... This paper adopts the definition of inclusion from the The World Wide Web Consortium [185] because it is broad, as it can encompass social, economic and technical factors as well as accessibility and usability. 7 In the definition, accessibility is assumed to enable the greatest extent of people to interact with systems, potentially by means of approaches such as Inclusive Design [31], Universal Design (also known as Design For All) [122,123,126,159,169,170,177], Ability-Based Design [183], and Design for User Empowerment [91,92]. 8 In particular, Design for User Empowerment [92] proposes that people with disabilities should contribute to all aspects of development (therefore, including design and implementation). ...
Full-text available
One strategy toward universalizing play is enabling more people to develop their own games. In this paper, our efforts toward a framework for inclusive creation of inclusive games are discussed. The hypothesis is that if end-users used creation tools suitable to their interaction needs and followed a collaborative work model to iteratively improve accessibility features to be inserted into a software architecture able to modify human-computer interaction at use-time, then they would be able to create games satisfying heterogeneous interaction needs of possible players. To verify the hypothesis, the architecture, the collaborative work model, and a game creation platform (Lepi) were designed to support game creation and play activities. Abilities were focused to provide opportunities for contributions based on skills, interests, and knowledge of people. The framework was evaluated over ten meetings spanning four months by people with alcohol and drug addiction from a public healthcare service. With the framework, participants were able to create their own games despite their different interaction needs (including low literacy, no previous contact with computers, emotional disabilities). By following the collaborative work model, they enabled people with different interaction needs than their own to play their games. Hence, with the framework, opportunities were provided to enable people with different interaction needs to contribute, create, and play. Game creation became a jigsaw puzzle, on which each piece (contribution) allowed people to create and play according to their abilities and skills.
Multimodal human–computer interaction has been sought to provide not only more compelling interactive experiences, but also more accessible interfaces to mobile devices. With the advance in mobile technology and in affordable sensors, multimodal research that leverages and combines multiple interaction modalities (such as speech, touch, vision, and gesture) has become more and more prominent. This article provides a framework for the key aspects in mid-air gesture and speech-based interaction for older adults. It explores the literature on multimodal interaction and older adults as technology users and summarises the main findings for this type of users. Building on these findings, a number of crucial factors to take into consideration when designing multimodal mobile technology for older adults are described. The aim of this work is to promote the usefulness and potential of multimodal technologies based on mid-air gestures and voice input for making older adults' interaction with mobile devices more accessible and inclusive.
Full-text available
Universal accessibility (or design) is a trend that promotes accessibility for everyone in various ways. One of its attributes is to ensure that everyone has equal learning opportunities, especially with the ‘access to information’ format. This applies to arranging a conference that includes conference organizers, plenary speakers, performers, conference presenters, and audio describers preparing to provide information and sensorial accessibility to the conference participants. Unfortunately for contemporary conferences, individuals with different needs are likely to experience language barriers due to their linguistic differences, hearing loss, and/or challenges in understanding and/or accessing visual information. A performing arts conference, Partition/Ensemble 2020, hosted by the Canadian Association of Theatre Research, serves as a case study for examining the process of arranging and providing language interpreters and text transcriptions, including audiovisual descriptions. During the COVID pandemic in the summer of 2020, the conference organizers decided to have a relaxed virtual conference. This designation had an impact on the preparation with four languages in different modalities: English (spoken and written), French (spoken and written), American Sign Language (signed), and Langue des signes québécoise (signed). From this linguistic learning experience, individuals who participated in this conference (e.g. conference organizers, plenary speakers, and audio describers) share their thoughts and insights for the implementation of an accessible conference (whether hosted in-person or online) with the goal of reducing language barriers. The authors of this article consider what it means to incorporate a diversity of languages simultaneously with different modalities and the challenges of accessibility with this endeavour.
Facebook Reactions are a collection of animated icons that enable users to share and express their emotions when interacting with Facebook content. The current design of Facebook Reactions utilizes visual stimuli (animated graphics and text) to convey affective information, which presents usability and accessibility barriers for visually-impaired Facebook users. In this paper, we investigate the use of sonification as a universally-accessible modality to aid in the conveyance of affect for blind and sighted social media users. We discuss the design and evaluation of 48 sonifications, leveraging Facebook Reactions as a conceptual framework. We conducted an online sound-matching study with 75 participants (11 blind, 64 sighted) to evaluate the performance of these sonifications. We found that sonification is an effective tool for conveying emotion for blind and sighted participants, and we highlight sonification design strategies that contribute to improved efficacy. Finally, we contextualize these findings and discuss the implications of this research with respect to HCI and the accessibility of online communities and platforms.
Full-text available
Services to Empower YOU” (SEU) responde a um desafio societal que visa fornecer a pessoas com deficiência ou incapacidade uma ferramenta agregadora e inclusiva através da qual possam requisitar com confiança serviços. Este artigo apresenta os resultados iniciais obtidos no projeto SEU que compreende o estudo da conceção e implementação de interfaces digitais adaptadas às necessidades específicas de cada utilizador, assim como o estudo de um modelo de negócio que permita aos prestadores de serviços fornecerem serviços para um público com características especiais. Partindo do levantamento das necessidades de pessoas com incapacidades e do estudo das normas de acessibilidade digital, especificou-se um Modelo de Adaptação para interfaces em função das características de cada tipo de utilizador e um Modelo de Serviços, que estende o modelo base de plataformas facilitadoras de serviços offline. Estes dois modelos foram aplicados no desenvolvimento do protótipo web apresentado. Palavras-chave: Plataforma de serviços web. Acessibilidade digital. Interfaces multimodais. Inclusão. Pessoas com deficiência ou incapacidade. "Services to Empower YOU" - Inclusive Multimodal Interfaces in a Service Platform for People with Disabilities Abstract: “Services to Empower YOU” (SEU) responds to a societal challenge that aims to provide people with disabilities with an aggregating and inclusive tool through which they can confidently request services that are appropriate to them. This article presents the initial results achieved in project SEU, which comprises the study of the design and implementation of digital interfaces adapted to the specific needs of each user, as well as the study of a business model that allows service providers to offer services for an audience with special characteristics. Starting from the survey of the needs of people with disabilities and the study of digital accessibility standards, two models were proposed: an Adaptation Model for designing interfaces according to the characteristics of each type of user; and a Services Model, which extends the base model of platforms facilitating offline services. These two models were instantiated in the initial web prototype presented. Keywords: Web services platform. Digital accessibility. Multimodal user-interfaces. Inclusion. People with disabilities.
This paper presents key aspects that designers and Human–Computer Interaction practitioners might encounter when designing multimodal interaction for older adults, focusing on the trade-offs that might occur as part of the design process. The paper gathers literature on multimodal interaction and assistive technology, and describes a set of design challenges specific for older users. Building on these main design challenges, four trade-offs in the design of multimodal technology for this target group are presented and discussed. To highlight the relevance of the trade-offs in the design process of multimodal technology for older adults, two of the four reported trade-offs are illustrated with two user studies that investigate mid-air and speech-based interaction with a tablet device. The first study explores the design trade-offs related to redundant multimodal commands in older, middle-aged and younger adults, whereas the second one investigates the design choices related to the definition of a set of mid-air one-hand gestures and voice input commands for older adults. Further reflections highlight the design trade-offs that such considerations bring in the process, providing an overview of the design choices involved and of their potential consequences.
Full-text available
In the information society, the notion of “computing-platform” encompasses, apart from traditional desktop computers, a wide range of devices, such as public-use terminals, phones, TVs, car consoles, and a variety of home appliances. Today, such computing platforms are mainly delivered with embedded operating systems (such as Windows CE, Embedded/ Personal Java, and Psion Symbian), while their operational capabilities and supplied services are controlled through software. The broad use of such computing platforms in everyday life puts virtually anyone in the position of using interactive software applications in order to carry out a variety of tasks in a variety of contexts of use. Therefore, traditional development processes, targeted towards the elusive “average case”, become clearly inappropriate for the purposes of addressing the new demands for user- and usage-context diversity and for ensuring accessible and high-quality interactions. This paper will introduce the concept of unified user interfaces, which constitutes our theoretical platform for universally accessible interactions, characterized by the capability to self-adapt at run-time, according to the requirements of the individual user and the particular context of use. Then, the unified user interface development process for constructing unified user interfaces will be described, elaborating on the interactive-software engineering strategy to accomplish the run-time self-adaptation behaviour.
Full-text available
Accessibility is one of the key challenges that the Internet must currently face to guarantee universal inclusion. Accessible Web design requires knowledge and experience from the designer, who can be assisted by the use of broadly accepted guidelines. Nevertheless, guideline application may not be obvious, and many designers may lack experience to use them. The difficulty increases because, as the research on accessibility is progressing, existing sets of guidelines are updated and new sets are proposed by diverse institutions. Therefore, the availability of tools to evaluate accessibility, and eventually repair the detected bugs, is crucial. This paper presents a tool, EvalIris, developed to automatically check the accessibility of Websites using sets of guidelines that, by means of a well-defined XML structure, can be easily replaced or updated.
Full-text available
The multimedia metamodel defines platform-independent multimedia concepts, opening the way for novel approaches to designing content repurposing solutions. Designers can use the metamodel to create content and add metadata to existing content, simplifying content analysis and repurposing. This model-driven approach and proposed design solutions are useful not only for many multimedia designers and researchers, where the model-driven tools can help them create better multimedia interfaces, but also for lecturers and students of multimedia courses. In the latter case, the unified multimedia metamodel offers context for sometimes subtle relationships between multimedia concepts. The metamodel can also facilitate the collaborative creation of broader knowledge about multimedia phenomena. In our future work, we plan to extend the proposed metamodel and include domains from related fields, such as user modeling and intelligent tutoring systems that deal with high-level user models. We are also designing multimodal test environments, reusable multimedia components, and data mining tools for evaluating various aspects of multimedia and multimodal communication.
The article reports that state-of-the-art multimodal interfaces can process two or more combined input modes using recognition-based technologies in order to accurately identify users or to interpret their communicative intent. There are several classes of multimodal systems that are relatively mature, including ones that process speech and manual input, audiovisual speech input, and multibiometric input. In many important ways, these new interface designs emulate basic biological principles and systems, including the coordinated multimodal communication patterns that have evolved so impressively in humans. During multimodal communication, people's multisensory perception can achieve remarkable accuracy through fusion of different information sources.
Designing universally accessible user interfaces means designing for diversity in end-users and contexts of use, and implies making alternative design decisions, at various levels of the interaction design, inherently leading to diversity in the final design outcomes. Towards this end, a design method leading to the construction of a single interface design instance is inappropriate, as it cannot accommodate for diversity of the resulting dialogue artifacts. Therefore, there is a need for a systematic process in which alternative design decisions for different design parameters may be supported. The outcome of such a design process realizes a design space populated with appropriate designed dialogue patterns, along with their associated design parameters (e.g. user- and usage-context-attribute values). This paper discusses the Unified Interface Design Method, a process-oriented design method enabling the organization of diversity-based design decisions around a single hierarchical structure, and encompassing a variety of techniques such as task analysis, abstract design, design polymorphism and design rationale.
This is a conference paper. Let us say you wish to design and develop a new product or technology. You want to ensure that as many people as possible are able to use it—not only is it politically and in some countries legally correct but with our ageing population it makes good economic sense. You want to follow existing design advice, of which you have been told there is much around. But where is the best place to look, and who do you go to for more specific information? You may decide to look for published materials, such as design guidelines. However, when you find guidelines, you cannot always be sure that they will be applicable to your specific product or technology. If you follow these guidelines, can you be sure that your product will be usable by more people?
Advances in multimedia, virtual reality, and immersive environments have expanded human computer interaction beyond text and vision to include touch, gestures, voice, and 3D sound. Although well developed single modalities for communication already exist, we do not really understand the general problem of designing integrated multimodal systems. We explore this issue and the diverse approaches to it, with emphasis on a generic platform to support multimodal interaction