Content uploaded by Matteo Matteucci
Author content
All content in this area was uploaded by Matteo Matteucci
Content may be subject to copyright.
Content uploaded by Matteo Matteucci
Author content
All content in this area was uploaded by Matteo Matteucci
Content may be subject to copyright.
An Adaptive and Predictive Environment to
Support Augmentative and Alternative
Communication
Nicola Gatti, Matteo Matteucci, and Licia Sbattella
Dipartimento di Elettronica e Informazione
Politecnico di Milano
Piazza Leonardo da Vinci 32, I-20133 Milano, Italy
{ngatti,matteucci,sbattella}@elet.polimi.it
Abstract. In this paper we describe Bliss2003, an Information and
Communication Technology (ICT) aid for verbal impaired people sup-
porting the use of Augmentative and Alternative Communication (AAC)
languages. Bliss2003 allows to compose messages in Bliss and other
AAC languages (i.e. PCS, PIC, etc.), to translate them in natural lan-
guage, to send and receive them via email, or to vocally synthetize them.
Bliss2003 is characterized by a predictive module that allows for a more
efficient selection of graphical symbols and more natural sessions of com-
munication by adapting a model of the user language behavior.
1 Introduction
Communication represents the main way in which man can live his sociality;
in fact, only by communication, man can express thought, emotions, and ideas.
Communication plays an important role in everyday life; by communicating man
can express necessities, feelings, request for information and aid. Nowadays, mil-
lions of verbal impaired people live currently in the world [1]; their communica-
tion capabilities are permanently or temporarily corrupted and, for this reason,
most of them suffer a condition of social exclusion.
To help verbal impaired people, ad-hoc alternative languages have been de-
veloped by the International Society for Augmentative and Alternative Com-
munication (ISAAC) established in 1983 in USA [2]. Among the currently most
adopted AAC languages [3] we can cite: Bliss, PCS, PIC, PICSYMB, CORE,
Rebus. Each of them is based on a peculiar dictionary of words represented by
pictures or symbols, and specific composition rules simple enough to be learnt
and used by disabled people [4].
ICT, especially in cases of verbal impaired people, has been widely used to de-
velop effective tools for rehabilitation. Bliss2003, developed from APBliss [5],
is an adaptive and predictive environment for AAC developed to provide people
during the communicative process with a personal table of Bliss symbols and a
set of intelligent tools. Bliss2003 can translate the composed messages to nat-
ural language making use of a syntactic/semantic analyzer, vocally synthetize
2 Gatti, Matteucci, and Sbattella
them, and exchange them via email. With respect to traditional AAC software
aids, Bliss2003 addresses a set of novel features: an innovative predictive com-
position assistant based on a discrete implementation of auto-regressive hidden
Markov model [6] called DAR-HMM, the simultaneous support of several AAC
languages, and the adoption of a graphical interface designed on purpose for
impaired users. Bliss2003 has been registered by the Blissymbolics Communi-
cation International (BCI) center of Toronto and has been adopted for hundreds
hours of experimental validation by several Italian clinics for verbal impaired
people. In the following section we introduce the Bliss2003 environment and
in Section 3 we describe its the main novelty: the predictive composition assis-
tant. A case study, taken from the validation activity of the tool, is presented in
Section 4.
2 The Bliss2003 Graphical Environment
The collaboration of a psychologist expert on graphic trace helped us to de-
sign a specific graphical interface that improves usability in order to reduce the
difficulties a verbal impaired person can undertake interacting with graphical
applications [7]. The main issues we have taken into account regard the icono-
graphic style, the graphical structure of the windows, their place, the number
of buttons and their function as well. The whole iconographic collection has
been drawn by a web-designer with the requirements of a simple and clear inter-
face, intuitive and unambiguous, with well-blended colors and uniform strokes
to assist visual difficulties while supporting content decoding (see Figure 1).
Bliss2003 is organized in seven environments to facilitate the user con-
centration by reducing buttons, functions and information that the user can
simultaneously find on the screen. This environment-based structure has been
designed to facilitate users since they express an higher concentration in specific
environments than in a general environment that requires to discriminate among
many choices. Environments are also related to the capabilities of the user and
the adoption of a specific one should be accomplished in collaboration with a
therapist.
Since the universality of Bliss language, Bliss messages can be synthesized
in any natural language. The user can set more speech features such as male or
female voice, tonality, volume, emphasis and so on. The vocal synthesis makes
easier the interaction with other people. In addition the verbal impaired user can
operate a syntactic concordance, using a syntactic/semantic analyzer, of a Bliss
message into natural language according to grammar rules for verb, personal
ending using gender concordance and prepositions in order to achieve a sentence
syntactically correct. The transposition of messages makes more comprehensible
a sentence for people that do not know Bliss language.
Similarly to other AAC communication software application, Bliss2003 is
focused on the symbols table. In particular, Bliss2003 provides a master table
composed of all Bliss symbols – about 2000 – and a personal table composed
of just the symbols used by the user. Through the symbol editor environment
An Adaptive and Predictive Environment to Support AAC 3
Fig. 1. Bliss2003 graphical user interface. The underlined spots refer to the Main
Command Bar (1), the Environment Option Bar (2), the Environment Command Bar
(3), the Data Zone (4), and the Temporary Processing Zone (5).
therapists can load PCS, PIC, or Rebus collection, so that Bliss2003 could be
employed as a generic software for alternative communication and could be used
as a multiuser system (in clinic, with people who communicate using different
alternative codes). The provision of multi-language support evidenced extreme
utility in rehabilitative processes. Furthermore Bliss2003 interface supports
devices other then classical mouse and keyboard, in order to overcome motor
disabilities in the user (e.g., joystick, touch sensor, switch buttons, graphic tables,
etc.).
3 Predictive composition assistant
The Bliss2003 environment provides a predictive composition assistant, named
caba2l[8], which supports disables in composing sentences, by speeding up
the symbols selection process, reducing the strain and the composition time,
while increasing their self esteem. Literature about AAC languages application
is rich of works concerning alphabetical prediction systems [9], but it is almost
completely lack of systems performing symbols prediction [10]. To overcome
this issue we have designed an innovative prediction system for Bliss language.
Caba2lis a prediction system able to suggest the user a set of Bliss symbols as
next probable choice for his/her sentence according to the last symbol selected
and the history of sentences previously composed. In other word, when a symbol
is selected from the table, caba2lshows a list of symbols that it considers as
most probable ones – with respect to its model of the user language behavior
– to continue his/her text. For instance, in the sentence “I want to eat” the
4 Gatti, Matteucci, and Sbattella
next symbol will probably be some kind of food, so the assistant should suggest
“pizza”, “pasta”, “bread”, “cake”, on the basis of last selected symbol and user
characteristics. Caba2lproposes a limited number of symbols defined by the
therapist depending disable capabilities and operates a scansion of them by
adopting an adaptive rate similarly to [11].
The composition assistant is based on a novel auto-regressive Hidden Markov
Model implementation, called Discrete Auto-Regressive Hidden Markov Model
(DAR-HMM) that we have developed specifically for symbolic prediction. Bliss
symbols have been divided in six grammatic categories (adverbs, verbs, adjec-
tives, substantives, persons, punctuation), and each one of them has been di-
vided in sub-categories using semantic network formalism [12] (about 30 sub-
categories). In the following, we give a brief formal description of DAR-HMM
according to the notation adopted by Rabiner in [6] to specify hidden Markov
models:
–S,{si}, (sub)categories set with N=|S|;
–V,{vj}, predictable symbols set with M=|V|;
–V(i)={v(i)
k}, set of symbols predictable in (sub)category i
–O(t)∈V, observed symbol at time t;
–Q(t)∈S, (sub)category at time t;
–πi(t) = P(Q(t) = si), probability of sibeing (sub)category at time t;
–aii0=P(Q(t+ 1) = si|Q(t) = si0), transition probability si0→si;
–bi
k=P(O(0) = v(i)
k|Q(0) = si), probability of observing v(i)
kfrom (sub)category
siat t= 0;
–bii0
kk0=P(O(t) = v(i)
k|Q(t) = si, O(t−1) = v(i0)
k0), probability of observing
v(i)
kfrom the subcategory sihaving just observed v(i0)
k0.
Given λ=< Π0={πi(0)}, A ={aii0}, B ={bii0
kk0}>the vector of parameters
describing a specific language behavior model, we can predict the first observed
symbol as the most probable one at time t= 0:
ˆ
O(0) = arg max
v(i)
kP(O(0) = v(i)
k|λ)
= arg max
v(i)
k
(P(O(0)|Q(0), λ)P(Q(0)))
= arg max
v(i)
kbi
k·πi(0).
Then, using the DAR-HMM generative model described in Figure 2, to predict
the tth symbol of a sentence we want to maximize the symbol probability in the
present (hidden) state given the last observed symbol:
P(O(t) = v(i)
k, Q(t) = si|O(t−1) = v(i0)
k0, λ).
An Adaptive and Predictive Environment to Support AAC 5
Fig. 2. Symbols emission in DAR-HMM; Siis the ith state (Bliss subcategory), Ojis
the jth observed symbol
Recalling that we can compute the probability of the current (hidden) state as
P(Q(t)) =
N
XP(Q(t)|Q(t−1))P(Q(t−1)) =
=
N
X
i0=1
πi0(t−1)aii0=πi(t),
we obtain the a recursive form for symbol prediction at time t:
ˆ
O(t) = arg max
v(i)
k bii0
kk0·
N
X
i0=1
πi0(t−1)aii0!.
In caba2l, the probability tables on λvector are computer using a data
set of Bliss sentences from the user and can adapt their values according to
the evolution of the composition capabilities of the disable. In doing this, we
have adopted a variation of the Baum-Welch algorithm, an iterative algorithm
based on the Expectation-Maximization method [6], adapting this technique
to the specific case of DAR-HMM. This feature is particularly relevant in the
rehabilitation cases where, during the rehabilitation, the dictionary of the disable
increases and his/her linguistic capabilities improve (an interested reader can
retrieved a deeper analysis of caba2lin [8]).
4 Tool Validation
According to recent studies [13], 37% of impaired people dismiss to use the
rehabilitation tool mainly because of lack of a real match to their needs. For this
reason, in the development of Bliss2003, we have collaborated with two Italian
clinical centers operating on verbal impairment – PoloH and SNPI of Crema
(Italy) – to focus on an extensive experimentation of the new tool. Among the
users who collaborated with us we have chosen to report the case study we
consider as the most significant: “Elisa” an eighteen-year-old girl, who has been
communicating with Bliss language for ten years.
6 Gatti, Matteucci, and Sbattella
Problem Imp D1 D2
speeding up communication 5 4 2
aiding the symbol selection process 5 4 2
compose Bliss message with much
autonomy
5 4 1
performing the social integration
making easier communication with
non-Bliss interlocutor
5 5 3
exploiting communicative found 5 4 2
increasing user’s attention degree 4 4 2
making powerful learning 4 4 3
Total score 137 70
Table 1. IPPA interviews comparison
From a disable perspective the most significant reason for assessing effective-
ness is to assure that his/her problem has been solved. The effectiveness of an
assistive technology provision, in its most basic form, can thus be defined as the
degree to which the problem is actually solved in relation to its intended aim.
We dealt with this by using two international ad-hoc protocols: IPPA [14] and
MPT [15]. In the following we report the results we have obtained using IPPA
and MPT protocols introducing an overall evaluation of Bliss2003 (we do not
describe the evaluation of each single components, such as composition assistant
or multi-language adoption).
4.1 IPPA (Individually Prioritized Problem Assessment)
IPPA is a protocol centered on the verbal impaired user, in particular on the
evaluation of the difficulties the disable undertakes interacting with the aid. The
user is asked to identify a set of problems that he/she experiences in daily life and
that he/she hopes to eliminate or decrease. This is done at the very beginning
during the service delivery process so that the user is not influenced by service
provider. The evaluation consists in the comparison of interviews regarding the
use of different aids, and reporting the difficulties that the user undertakes in the
accomplishment of the problems previously identified. The identification process
is an interactive process and we take care to designate problems on the basis of
user’s concrete activities. Table 1 reports the problems identified by Elisa.
A first interview has been accomplished before the use of Bliss2003 and
refers to an AAC communication software previously adopted by Elisa at SNPI;
Elisa with her parents and clinical staff assigned scores (on a five-grades scale)
to the tool both with respect to the importance (Imp) of the problem and the
level of difficulty of performing the specific activity (D1). During the follow-
up interview, a few months after Elisa has started using Bliss2003, she had
assigned a new difficulty score for the same activities (D2). The total score for
each interview is calculated summing each difficulty score weighted by relative
importance factor. The difference between the total IPPA score before and after
An Adaptive and Predictive Environment to Support AAC 7
Interview Positive Indifferent Negative
1a10 12 9
2a22 6 3
Table 2. SOTU questionnaire comparison
Area Scores 1st int. Scores 2nd int.
Disability 1.214 3.28
Aid 4.3 4.6
Environment 3.571 3.71
Character 4.027 4.25
Table 3. ATD PA questionnaire comparison
the provision of Bliss2003 represents an index of the aid effectiveness. The data
reported in Table 1 evidence that the difficulty score is decreased in all identified
problems while the total score is almost cut by half. Similarly to the Elisa case
of study the interviews obtained from other disables evidence a lower difficulty
score in comparison with the previously adopted AAC communication software.
4.2 MPT (Matching Person and Technology)
MPT is a validation protocol founded on active dialogue between the disable
and the assistive technology expert. It allows to identify disables needs and their
point of view on assistive technology aids so that we can develop a software
to prevent assistive technology abandonment. In particular we have used three
MPT instruments:
–MPT working sheet: to define targets and guidelines with possible informa-
tion technology solutions;
–SOTU questionnaire: to analyze user’s personal and social characteristics;
–ATD PA questionnaire: to analyze the features of assistive technology aid
and its applicative domain.
These instruments are structured as questions with closed answer which have
been compiled by Elisa parents, her clinical staff and us as information technol-
ogy experts, according to two session: before using Bliss2003 and a few months
after using it.
Comparing the first interview score with the second one about SOTU ques-
tionnaire (shown in Table 2) the negative answers decrease in the light of the
progress on the social and communicative integration. Taking into account the
ATD PA questionnaire results (shown in Table 3) the score is higher thus indi-
cating that Bliss2003 has improved Elisa’s communication capabilities.
8 Gatti, Matteucci, and Sbattella
5 Conclusions
In this paper we have introduced Bliss2003 an adaptive and predictive environ-
ment to support augmentative and alternative communication. In particular we
described its overall architecture, its innovative predictive composition assistant
and the experimental activity we did to validate the tool.
References
1. Bloomberg, K., Johnson, H.: A statewide demographic survey of people with severe
communication impairments. Augmentative and Alternative Communication 6
(1990) 50–60
2. ISAAC: International society for augmentative and alternative communication.
http://www.isaac-online.org (1983) Last accessed October 1st, 2003.
3. Gava, M.: ...e se manca la parola, quale comunicazione e quale linguaggio ? AAC:
una risposta nell’ambito delle disabilit`a verbali. Riabilitazione Oggi 16 (1999) 8–42
4. Shane, B.: Augmentative Communication: an introduction. Blackstone, Toronto,
Canada (1981)
5. Somalvico, M., Gava, M., Squillace, M., Stefanato, M.: APBliss per Windows. In:
Convegno Nazionale di Informatica Didattica e Disabilit`a, Napoli (1995)
6. Rabiner, L.: A tutorial on hidden markov models and selected applications in
speech recognition. In: Proc. of the IEEE. 77, IEEE Computer Society Press
(1989) 257–286
7. Taylor, A., Arnott, J., Alm, N.: Visualisation to assist non-speaking users of aug-
mentative communication systems. In: Proc. of the Int. Conf. of the IEEE on Inf.
Vis., London, UK (2001) 251–256
8. Gatti, N., Matteucci, M.: Caba2la Bliss predictive composition assistant for
verbal impaired people. In: Proceedings of ICEIS 2004. Volume 5. (2004) 89–96
9. Simpson, R.C., Koester, H.H.: Adaptive one-switch row-column scanning. IEEE
Trans. on Rehabilitation Engineering 7(1999) 464–473
10. Higginbotham, D.J., Lesher, G.W., Moulton, B.J.: Techniques for aumenting scan-
ning communication. Augmentative and Alternative Communication 14 (1998)
81–101
11. Cronk, S., Schubert, R.: Development of a real time expert system for automatic
adaptation of scanning rates. In: Proc. of the Conf. on Rehabilitation Technology,
RESNA. Volume 7., Washington, DC, USA (1987) 109–111
12. Quillian, M.: Semantic memory. In: Minsky ed. Semantic Information Processing.
MIT Press, Cambridge (1968)
13. S.I.V.A.: Assistive technology research and information center. http://www.siva.it
(1980) Last accessed October 1st, 2003.
14. Andrich, R., deWitte, L., Ferrario, M., Lorentsen, O., Oberg, B., Oortwijn, W.,
Persson, J., VanBeekum, T., Wessels, R.: IPPA, a user-centred approach to assess
effectiveness of assistive technology provision. Technology and Disability 13 (2000)
105–116
15. Scherer, M.J.: MPT: Matching person & technology. Technical report, Fond. Don
Gnocchi IRCCS-ONLUS, SIVA, Milan, Italy (1999)
