Content uploaded by Markel Vigo
Author content
All content in this area was uploaded by Markel Vigo on Jun 19, 2020
Content may be subject to copyright.
Technology for supporting web information search and learning in Sign Language
Inmaculada Fajardo
a,*
, Markel Vigo
b
, Ladislao Salmerón
a
a
Department of Developmental and Educational Psychology, University of Valencia, Blasco Ibañez, 26010 Valencia, Spain
b
Department of Computer Architecture and Technology, University of the Basque Country, Manuel Lardizabal 1, 20018 San Sebastian, Spain
article info
Article history:
Received 1 October 2008
Received in revised form 9 March 2009
Accepted 12 May 2009
Available online 21 May 2009
Keywords:
Web accessibility
Deafness
Sign Language
Information search
e-Learning
Video Technology
abstract
Sign Languages (SL) are underrepresented in the digital world, which contributes to the digital divide for
the Deaf Community. In this paper, our goal is twofold: (1) to review the implications of current SL gen-
eration technologies for two key user web tasks, information search and learning and (2) to propose a
taxonomy of the technical and functional dimensions for categorizing those technologies. The review
reveals that although contents can currently be portrayed in SL by means of videos of human signers
or avatars, the debate about how bilingual (text and SL) versus SL-only websites affect signers’ compre-
hension of hypertext content emerges as an unresolved issue in need of further empirical research. The
taxonomy highlights that videos of human signers are ecological but require a high-cost group of experts
to perform text to SL translations, video editing and web uploading. Avatar technology, generally associ-
ated with automatic text-SL translators, reduces bandwidth requirements and human resources but it
lacks reliability. The insights gained through this review may enable designers, educators or users to
select the technology that best suits their goals.
Ó2009 Elsevier B.V. All rights reserved.
1. Introduction
According to the World Health Organization, 278 million people
in the world are deaf or hard of hearing (World Health Organiza-
tion, 2006) and many of them have Sign Language (SL) as their
mother language. The diversity of regional variations of SL consti-
tutes a set of minority languages relatively underrepresented in
the digital world. Thus, members of the Deaf Community usually
face non-native language web sites where accessibility barriers
may emerge (e.g. Fajardo et al., 2006;Smith, 2006).
In order to ensure the Deaf Community’s social inclusion, SL has
to be properly incorporated into Information Technologies. Fur-
thermore, as Kralisch and Berendt (2005) highlight, the web inclu-
sion of minority languages might not be only an ethical but a
commercial issue, related, for instance, to financial investment to
create bilingual websites. In Spain, there are around 400,000 users
of Spanish Sign Language (INE, Instituto Nacional de Estadística,
2006) who use oral and written Spanish as a second language
and could benefit from inclusive policies. Spanish and Catalan Sign
Languages have held the status of State Official Languages in Spain
since 2007, as is also the case in other Member States of the Euro-
pean Union.
The World Wide Web Consortium (W3C) addresses, to some ex-
tent, deaf users’ issues in the Web Content Accessibility Guidelines,
WCAG 1.0 (Chisholm et al., 1999). In particular, Guideline 1.4 rec-
ommends that audio and its textual transcription should be syn-
chronized, while guideline 14 states that deaf users would
benefit from simple and clear written language. Besides the vague-
ness of these guidelines, it is important to note that they just deal
with the captioning or text transcription of auditory content, leav-
ing out the visual–spatial characteristics of SL, thus causing infor-
mation loss. In addition, as clear implementation techniques are
not provided, non-experts find it difficult to create WCAG-compli-
ant web pages for the Deaf. WCAG 2.0 (Caldwell et al., 2008) be-
came a W3C candidate recommendation in December 2008. As
opposed to WCAG 1.0, SL issues are more extensively addressed
in this set of guidelines. Guideline 1.2, entitled ‘‘Time-based Media:
provide alternatives for time-based media”, is one of those guide-
lines whose fulfilment enhances web content perceivability by
users, including the deaf. Specifically, success criterion 1.2.6 states
‘‘Sign Language interpretation is provided for all prerecorded audio
content in synchronized media”. Satisfying success criterion 1.2.6
is necessary in order to meet the most demanding conformance le-
vel, the AAA success criteria. Specifically, it claims to provide SL
interpretation for synchronized media by means of the following
techniques:
– Embedding a SL interpreter in the video stream in order to pro-
vide SL transcription of audio content.
0953-5438/$ - see front matter Ó2009 Elsevier B.V. All rights reserved.
doi:10.1016/j.intcom.2009.05.005
*Corresponding author. Address: Avda. Blasco Ibáñez, 21, 46010 Valencia, Spain.
Tel.: +34 963983847.
E-mail addresses: infabra@uv.es (I. Fajardo), markel@si.ehu.es (M. Vigo),
Ladislao.Salmeron@uv.es (L. Salmerón).
Interacting with Computers 21 (2009) 243–256
Contents lists available at ScienceDirect
Interacting with Computers
journal homepage: www.elsevier.com/locate/intcom
– Techniques and examples of synchronizing video of the SL inter-
preter in order to display it in a different overlay on the image
using SMIL (Michel, 2008) technology.
However, this success criterion just focuses on the transcription
of auditory content. Nothing is mentioned about link content or
content transcription, which, most of the time, is not auditory
but textual.
In order to ensure accessibility of digital resources such as
the World Wide Web (WWW), many countries have intro-
duced laws in this regard.
1
Identifying levels of conformance
with guidelines is of paramount importance, as most policies
rely on standards such as the above-mentioned WCAG or simi-
lar. However, the most demanding levels of compliance, includ-
ing those referring to deafness, are seldom required in order to
meet these policies.
In addition, government initiatives and standardista contribu-
tions cannot be effectively applied if they are not accompanied
by research conducted in different areas such as Human–Computer
Interaction, Computer Science, Psychology, or Sociology. These sci-
entific disciplines could help to answer questions such as the fol-
lowing, quoted from Cunliffe and Herring (2005, pp. 135–136):
1. How should the impact (whether positive or negative) of tech-
nology in minority language use be measured and quantified?
2. How can the linguistic dimensions of the digital divide be mea-
sured and how can its significance be assessed?
3. How does interface design influence language behaviour, e.g.
how can design be used to promote minority language use in
bilingual contexts, or to better support users accessing content
in their non-native language?
Or, added by us:
4. What kind of technology is available and how appropriate is it
to include SL on the web?
5. How do SL technologies affect the information scent (assess-
ment of the semantic similarity between searching goals and
hyperlink choices) and knowledge acquisition on the web?
Although it is vital to find the answers to all of these ques-
tions, the objective of this article is principally to answer the last
two. Therefore, the aim of the following sections is to shed some
light on how current SL technologies can provide deaf users with
a satisfactory user experience while interacting with the WWW.
Note that some of the prototypes presented were not intended
for the WWW but they are mentioned in this paper as their
foundations are sound and can be deployed in web
environments.
Section 2describes the most relevant SL generation systems
for the web, emphasizing their strengths and weaknesses with
regard to their usefulness for web information search. Similarly,
the same procedure has been followed in Section 3, focusing on
web learning. In Section 4, based on the previous systems re-
view, we propose a taxonomy that classifies SL generation sys-
tems according to a set of relevant dimensions which are not
only functional but also technical, such as location rendering,
underlying technology (programming language, mark-up, etc.)
or dimensionality. The purpose of the taxonomy is to help in
the selection of the system or set of functionalities that best
suits the goals of designers, educators or users while considering
the availability of resources.
2. Information search on the WWW by means of Sign Language
A web document or hypertext system is composed of a set of
information nodes connected by links or hyperlinks. Information
search refers to the user’s behaviour when looking for pieces of
information within or between web pages or hypertexts by means
of queries in search engines and/or by following hyperlinks. What-
ever the mechanism used, one of the most influential theories of
information search, the Information Foraging Theory (Pirolli and
Card, 1999), predicts that a particular hyperlink will be followed
when the trade-off between information gained and cost of access
is low. Therefore, in order to calculate such trade-offs, individuals
have to assess the semantic similarity between the search goals
and hyperlink choices (called information scents) presented on a
web page or in a hypertext node. As Pirolli (2004) found, users
seem to use semantic scents for judging not only when visiting a
website but also when leaving it.
The obvious problem for SL users is that semantic cues are com-
monly only available in a non-native language (link words) in
which they present low levels of reading proficiency (e.g. Leybaert
et al., 1982; Alegría, 1999; Asensio, 1989; Goldin-Meadow and
Mayberry, 2001). Consequently, deaf signers find it difficult to
use a scent following strategy with textual cues as some empirical
studies seem to indicate (Fajardo et al., 2009). Apparently, an easy
solution to increase information search for deaf signer users is the
use of graphical hyperlinks or icons, since they facilitate the pro-
cess of semantic decision-making according to the classical Picture
Superiority Effect (Nelson et al., 1976;Paivio, 1991). In addition,
Namatame et al. (2007) observed that deaf participants were even
more accurate than hearing participants in a simple visual search
and match task where participants were asked to pair directory
names, typically used in representative web sites, with pictograms
(Experiment 2). In contrast, Fajardo et al. (2006) did not observe
accuracy differences between deaf and hearing users with using
graphical material. Across two experiments, users were asked to
find targets in a hypertext system with several layers of nodes (a
more complex search task than in Namatame et al.). Although deaf
users were faster in a graphical than in a textual hypertext, deaf
and hearing participants were equally accurate when very familiar
and frequent pictures were used as hyperlinks (Fajardo et al.,
2008a). When unfamiliar pictures were used as hyperlinks, both
types of users found less targets, were slower and became more
disoriented in the graphic hypertext than in the textual hypertext
(Fajardo et al., 2006). Therefore, it seems reasonable to assume that
SL scent cues could be a more appropriate solution for improving
deaf signers’ information searching than text or icons, but here
we find a technological issue: is there any technology available
to provide SL scent cues? As described in the following sections,
we distinguish between mechanisms for making it possible to fol-
low hyperlinks in SL and mechanisms for supporting queries in SL
(for an extended description of this functional dimension for SL
generation techniques, see the dimension Task in the taxonomy
proposed in Section 4).
2.1. Hyperlinking by means of Sign Language
To enable not only content access but link-following in SL, a
number of research projects have developed a set of more or less
sophisticated techniques. Starting with an apparently simple idea,
the Cogniweb project (Fajardo et al., 2008b,c) developed two alter-
natives consisting of videos embedded in small frames which con-
tain SL translation (performed by human signers) of each textual
hyperlink in the menu. This mechanism is called Sign Language
Scent (SLS) here. In the first approach, when the cursor hovers over
a link, the embedded video located at the left-bottom of the page
1
Available at http://www.w3.org/WAI/Policy/.
244 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
starts signing so that the user grasps the meaning of the link (see
Fig. 1, left). In the second approach, every link is related to its
own video that signs whenever the user clicks on it (see Fig. 1,
right). In two experiments (Fajardo et al., 2008b,c), it was observed
that deaf signer users were more efficient with the second version.
Furthermore, deaf signer users became less disorientated (mea-
sured by the number of page visited per search) and were less
dependent on their verbal categorical reasoning abilities using
SLS than just using textual hyperlinks. These results would repre-
sent empirical support for the usage of SL videos linked to textual
hyperlinks as an efficient web navigation mechanism for deaf sign-
er users.
These two approaches were aimed at providing both SL and text
information scent cues; that is, to support bilingual navigation. It is
a desirable goal from the point of view of the Access for All princi-
ple,but quite challenging from a technical viewpoint. One of the
main limitations of the Cogniweb Project is the lack of an authoring
tool to support video processing and web page edition, two tasks
that require high expertise and are highly time-consuming.
Other more sophisticated approaches are aimed at creating web
sites exclusively in SL, which would allow both content access and
navigation in this language. These approaches are based on the
Hypervideo technology which consists of links inserted in videos
so that the user can retrieve further information about concepts
conveyed in certain sequences. Links are represented by means
of text or a static image, which may not convey SL concepts in a
proper manner. Based on this technology, Fels et al. (2006) devel-
oped the Signlinking system. Each Signlink is a period of time within
the video clip, defined by the author. When the video reaches a
Signlink, a link indicator is displayed to notify the user. Then, the
user can click and follow the link or continue playing the content.
A similar approach is followed in a system designed by Kaibel
et al. (2006). In this system, the user can get an overview of the
hyperlink videos by hovering the mouse over each snapshot. In
addition, there is a hand or a text-symbol at the bottom-left of each
hyperlink video indicating which type of document the hypervideo
leads to. A video containing a hand-symbol leads to a SL page
whilst a video that contains a text-symbol will lead to a traditional
hypertext document.
It is not clear which of these two approaches to hypervideo for
SL navigation is more effective. Regarding Signlinking, throughout
successive re-designs of the system, in the sequence where the
signer was referring to the concept requiring a hyperlink, the link
indicator evolved from a red rectangle surrounding the upper torso
of the signer to a small red icon located at the top-left corner of the
video window. Two usability studies showed that deaf signer par-
ticipants were confused about the meaning of the red rectangle
while navigating through signed web pages (study 1). However,
the understanding of a different group of users improved and num-
ber of navigation errors decreased after replacing the rectangles
with the arrow-into-document icon (study 2). As the authors ad-
mit, these two studies helped to improve the design of the video
link indicators and other elements such as video link density indi-
cators in a video but some important questions remain unan-
swered. For instance, with regard to the advantage that this kind
of technology could bring to deaf signers compared to a text-only
version of a website: do video-link web pages improve the link-fol-
lowing strategy of signers compared to traditional text-link web
pages? This question, as is the case with most of the questions
highlighted throughout this article, can only be resolved by means
of experimentation.
2.2. Queries in Sign Language
In addition to the hyperlink-following strategy, a complemen-
tary mechanism for searching for information on the web is the
use of queries in search engines. Taking into account the tridimen-
sionality of SL and the fact that it cannot be alphabetically se-
quenced, most of the multimedia dictionaries for signers use
query systems that just allow users word-to-SL search.
In order to overcome this limitation, Vettori and Felice (2008)
developed e-Lis (see Fig. 2 for a snapshot of the system), which al-
lows signers to make queries in SL, in a SL-Italian/Italian-SL dictio-
nary. A notable feature of e-Lis is that its dialogue allows users to
make queries using signs by selecting their formational units.
However, instead of 3D avatars, pre-stored 2D pictures are used.
Buttussi et al. (2007) aim at providing holistic search tools using
3D virtual avatars for word-to-SL, SL-to-word and SL-to-SL
searches with 3DictSL. The purpose of the SL-to-SL search is to find
equivalent signs in different SLs. The SL-to-word or SL-to-SL are the
more challenging tasks and signs are introduced by successively
selecting 3D formational units of a sign or cheremes; i.e. the hand
shape, orientation, location and movement of the palm. They also
provide online authoring tools for creating new dictionaries where
Fig. 1. Examples of the two versions of web page with textual hyperlinks supported by SL videos used in the Cogniweb project. In the version on the left, videos with
corresponding translation of the textual hyperlinks were presented in a unique frame of the screen placed in a distant location. In the version on the right, videos were placed
close to each hyperlink. The videos were activated by clicking or passing the mouse cursor over the hyperlink.
I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256 245
new signs can be stored by composing existing cheremes. How-
ever, the creation of a new dictionary from scratch turns out to
be a cumbersome task for a small group of experts.
Among other factors, it would be necessary to test the under-
standability of the 3D signs transmitted by avatars in 3DictSL ver-
sus 2D pictures in e-LIS versus written text, and the usability of the
search-by-chereme interfaces. To date, none of these two ap-
proaches that aim at making queries in SL possible have been val-
idated empirically which seems to be the common limitation of
most of the techniques we describe in this article.
3. Web learning in Sign Language
Web and hypertext learning research is mostly, although not
exclusively, focused on reading comprehension. Reading compre-
hension in hypertext refers to the process of knowledge acquisition
from the texts provided on websites and in hypertext systems. The
construction–integration model of text comprehension (Kintsch,
1988, 1998; Van Dijk and Kintsch, 1983) frames most of the re-
search on hypertext comprehension (see a review by Salmerón
et al., 2005). According to this model, the reader generates two
mental representations from the text: the textbase (a propositional
representation of the information within the text) and the situa-
tion model (a representation of what the text is about which inte-
grates the information with the readers’ prior knowledge). Unlike
traditional printed materials, hypertext systems are characterized
by a non-linear structure of information which allows an active
learning process, making users select the pieces of text to read
next. The idea that the user controls the reading process may
sound appealing, but the fact is that hypertext may be beneficial
only when the readers can rely on their reading skills to deal with
the multi-linearity of hypertext (Foltz, 1996). This fact becomes
especially relevant for the majority of prelingually deaf people,
who usually show low levels of reading proficiency (e.g. Leybaert
et al., 1982; Asensio, 1989; Alegría, 1999; Goldin-Meadow and
Mayberry, 2001). Quoting Foltz, ‘‘readers with poor reading skills
are using a lot more controlled processing for their reading processing
and thus will likely have a greater amount of interference from the
additional task of navigating the text”. This will not let them gener-
ate as many inferences about the text as they read it, making it
harder to integrate the information.
One alternative to overcome the poor reading skills of deaf
users may consist of translating written information from hyper-
text into SL, as in ViSiCAST by Elliott et al. (2000) and eSign by
Kennaway et al. (2007). The idea of providing content in SL seems
to be a viable solution but again generates many questions: how
can hypertext content be provided in SL? How does SL affect the
comprehension of text? Should SL content entirely replace the
texts or just support them? Do bilingual web sites enhance or
interfere with deaf people’s content comprehension? Is it feasible
to translate all hypertext content to SL? As a young and relatively
illiterate language, is the SL vocabulary suited to conveying the
technical or scientific concepts of some academic subjects and for-
mal documents? In the next two subsections (3.1 and 3.2), we de-
scribe the advantages and weaknesses of technologies based on
bilingual versus SL-only approaches which has been a long-stand-
ing debate in deaf education.
3.1. Bilingual (sign plus text) web sites
Digital video technology has allowed the incorporation of SL
into the web by providing SL translations of texts performed by hu-
man signers or avatars. The two most popular ways of displaying
the SL video that translates the text are: (1) in a different browser
window (see example in Fig. 3); and (2) embedded in the same
web page as that in which the text appears (see example in Fig. 4).
Far from being a trivial issue, the choice between these two op-
tions might have important repercussions on the learning process.
Research conducted with hearing users in the field of multimedia
learning may serve as a guide for evaluation of the usability of the-
ses video technologies. In particular, two principles emerge as rel-
evant to our goal: the spatial contiguity (Mayer, 2005) and
redundancy (Sweller, 2005) principles.
According to the spatial contiguity principle, the user’s under-
standing of the message transmitted through words and corre-
sponding pictures will increase when they are presented near to
each other, rather than far apart from each other on the page
(Mayer, 2005). If we apply this principle to the understanding of
the text and corresponding SL translations, we would expect that
the embedded video approach would be superior to the separate
pop-up windows approach, because the text and the SL videos
are more closely linked.
Fig. 2. e-Lis Dialogue for making sign enquiries using formational units (Vettori and Felice, 2008).
246 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
On the other hand, the redundancy principle states that hearing
users’ comprehension is hampered if the multimedia system in-
cludes different sources of the same information, e.g.graphics
and redundant texts. A possible explanation is that in these situa-
tions readers are forced to divide their attention between the
sources of information (both presented in visual format in the
aforementioned example), thus increasing the cognitive load of
the task. In the case of SL hypertext, including both the written
information and the exact SL translation at the same time may
hamper deaf users’ comprehension. In addition, as Richards et al.
(2005) highlight, having to pay attention to different sources would
create literacy barriers due to the continuous switching between
their language of choice and a second language. The redundancy is-
sue could be even more relevant for deaf learners because they
commonly exhibit difficulties integrating information coming from
two differences sources (Marschark et al., 2006). This redundancy
problem would affect both embedded and separate videos, but it
could be mitigated by enhancing the text–sign synchronisation.
Fig. 3. Example of SL videos displayed in a separated pop-up window from http://www.diariosigno.com.
Fig. 4. Example of SL videos displayed in an embedded frame from http://www.faas.es.
I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256 247
In that sense, Videotext.web (Austrian Sign Language Service
Centre
2
and the Indeed!
3
company) enables designers to highlight
the sentence of the text that is being signed in the video (see
Fig. 5). However, even if a priori it is a feasible approach, it has not
been empirically tested and could generate more problems than
solutions.
In general, one important question which emerges from the
technological solutions mentioned up to now is how bilingual
websites may affect signers’ comprehension of hypertext content.
This issue has actually unleashed a great debate and much research
in the context of education for deaf children. In particular, in order
to test whether multimedia technology can improve the text com-
prehension of deaf students, Gentry et al. (2004/2005) conducted
an experiment in which deaf participants were presented stories
in 4 different formats: printed, printed plus pictures, printed plus
SL, and printed plus pictures plus SL. The lowest comprehension
scores, measured by means of a story retelling activity, appeared
for the printed-only condition, while the printed plus pictures con-
dition obtained the highest scores. The use of signs improved per-
formance compared to the text-only condition but not to the
printed format plus picture condition. The authors suggest that
an explanation for that is that deaf children develop meta-cogni-
tive strategies to decode text by using pictures, because pictures
have been traditionally used in deaf education. However, they
would not have this type of strategy available to decode text using
SL, due to a lack of practice.
Thus, in order to facilitate development of meta-cognitive strat-
egies for decoding text using signs, literacy teaching for deaf chil-
dren should be supported by this kind of multilingual approach in
the early stages of education. Similarly, we could argue that
bilingual (text and SL) access to hypertext content would be im-
proved through practice with the kind of new emerging technolo-
gies presented here. Longitudinal studies would be needed to
answer this question.
This explanation of the phenomenon based on the peculiarities
of deaf students’ instruction is sound but, as we noted before, if the
design of text plus pictures material fulfils the contiguity and
redundancy principles, hearing learners would also be able to take
advantage of the picture and texts combination (Mautone and
Mayer, 2007; Moreno and Mayer, 2002). Actually, adjunct pictures
to text have been frequently used in instructional material (both
for printed books and computer-based material) for hearing stu-
dent education. That means that a more general factor could be
explaining the superiority of text plus pictures in deaf people.
However, to our knowledge, there are no studies which directly
compare deaf and hearing students regarding the facilitative effect
of picture aids on comprehension, which, once again, moves this
idea into the category of interesting hypothesis for future testing.
A third alternative explanation for the superiority of text plus
pictures over SL-only material in deaf learners is based on the fact,
highlighted above, that SL is a young and relatively illiterate lan-
guage and consequently may lack the terminology (vocabulary)
to translate or express the technical or scientific terms common
in some academic disciplines, such as biology or mathematics.
We could argue that SL users would prefer precise terms in their
second language to ambiguous and inappropriate terms in their
mother tongue.
Dowaliby and Lang (1999) performed a comprehension study
with 144 deaf college students using a scientific text on the topic
of ‘‘The Human eye, its function”. They compared five instructional
conditions: (1) text only, (2) text followed by content videos (a vi-
deo showing a visual example of the information) (3) text followed
by sign videos (a video of the text signed) (4) text followed by ad-
junct questions (a written practice question after each text and
Fig. 5. Example of sign-text synchronization using Videotext.web technology, retrieved from http://www.oegsbarrierefrei.at/.
2
http://www.oegsbarrierefrei.at/ (last accessed on February 2009).
3
Indeed! available at http://www.indeed.at/web/products/ (last accessed on
February 2009).
248 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
feedback on the response) and (5) all of these together (full
condition). Although they did not report any difficulty related to
the translation of the scientific text to English-like signing with
American Sign Language, the performance of students provided
with a text-only version of the information was similar to those
who studied the written text plus a signed video of the informa-
tion. Although these results may be partially interpreted in terms
of a ‘‘lack of technical terminology” hypothesis, future usability
studies should include a version with a signed video only to clearly
compare the effectiveness of both written and signed languages in
supporting the comprehension of scientific texts.
Nevertheless, the study by Dowaliby and Lang (1999) demon-
strates that technical supports that facilitate deaf students’ com-
prehension of narrative stories, such as adjunct pictures (Gentry
et al., 2004/2005), may not be so effective for supporting the read-
ing of a technical text. The authors constructed a third version of
the scientific text on the ‘Human eye’ which included the written
text plus a video showing a visual example of the information. Con-
trary to what has been found for narratives, this version did not
improve comprehension. Indeed, the best condition to foster scien-
tific comprehension in their study was the written text plus ad-
junct questions (a written practice question after each text and
feedback on the response).
In summary, there are technologies and procedures for generat-
ing bilingual (sings plus text) web sites but, according to the
empirical insights gained until now, to foster deaf readers’ compre-
hension of the narrative content of web sites, adjunct pictures are
more effective than adjunct SL. Future research will help to test
new procedures to effectively combine both types of information.
In the meantime, existing technology may support the creation
of SL-only web sites instead of bilingual approaches, so the next
question is could the exclusive use of SL foster content comprehension,
instead of the use of SL as adjunct or complementary information?
This could be a way of reducing the impact of redundant informa-
tion coming from different sources and in need of integration. Let
us answer this question in the next section.
3.2. SL-only web sites
Hypervideo technology, introduced in Section 2, makes it possi-
ble to embed clickable anchors in video streams, allowing the
development of sign-only web sites. As described above, Fels
et al. (2006) developed an authoring tool for creating Signlinked
web pages in which both the content and navigation mechanisms
could be provided exclusively in SL The educational potential of
this approach could be enormous but, as we argued in the previous
section, there are still no empirical studies proving its validity to
improve, for instance, deaf signers’ comprehension of academic
material.
From a human resources perspective, one of the main disadvan-
tages of the use of human signers is that it is necessary to have a
group of experts available for the translations from text to SL,
recording, editing and uploading videos to the web. A solution to
this problem would be the use of SL transcriptions but due to,
among other factors, the visual–spatial and temporal nature of
the SL, its transcription turns out to be quite complex. However,
there are some widely accepted notation systems. For example,
Stokoe (Stokoe, 1960) for American Sign Language, Hamburg
Notation System, HamNoSys (Prillwitz et al., 1989) or SignWriting
(Sutton, 1977) can be used for such a task. These notation systems
represent the basic parameters of signs (movements, hand shape
and location) by means of pictograms. They are often used by ad-
vanced automatic SL generators as intermediate representation of
real signs (Kennaway et al., 2007). They are also useful for small
screens and for devices with limited computing resources. On the
other hand, these notations have to be learned, which can be as
difficult as learning a written language. In order to find a more
economical and less time-consuming method of producing SL
material compared to the expensive human signers’ videos or the
difficult to learn notation systems, several groups are developing
different solutions to automatically translate text to SL and synthe-
size it using avatars.
ViSiCAST by Elliott et al. (2000) was one of the preliminary pro-
jects that aimed at providing signing avatars on the World Wide
Web (WWW). Aimed at achieving that developers adopt SL gener-
ation technologies, its creation could not entail a large amount of
financial investment or development effort. However, ViSiCAST re-
quired motion capture techniques to record the body, hand and
face gestures of a human signer. These captures were then con-
verted into highly realistic animations which were stored in a data-
base of reusable signs. Therefore, in order to find a solution to this
costly process of motion capture, ViSiCAST evolved into the eSign
project. Using an XML-based language called Signing Gesture
Markup Language (SiGML) it was decided that once a browser
plug-in was developed for such language, an avatar would tran-
scribe a web page into SL. The SiGML-to-avatar philosophy pro-
vides the framework with a set of tools for content creation. The
avatar is rendered on the user’s computer; saving the bandwidth
that video broadcasting would otherwise require. Kennaway
et al. (2007) report the results of a set of studies, conducted in
three different countries, to test the quality of the synthetic signs
developed in the e-Sign project. The recognition rate by native Sign
Language users was around 95% for single signs and ranged be-
tween 35% and 58% for signed sentences and chunks of text. These
results support the use synthetic signs for single words mainly.
However, those studies were conducted with small sample sizes
and important methodological details such as the number of trials
per test, length of sentences and order of presentation are not re-
ported, which reduces the generalization and validity of the re-
sults. In addition, the tools for content creation are domain
dependent (weather forecast or online employment office) and if
the plug-in is not supported by the user’s browser, the video con-
tent will be downloaded instead. Therefore, the user cannot take
advantage of one of the key features of the system, which is low
bandwidth SL generation.
As far as on-the-fly SL creation is concerned, Francik and Fabian
(2002) developed a real-time humanoid avatar that signed Polish SL
in real time. It consists of a humanoid that lacks facial expression.
Huenerfauth (2008) goes one step further, applying the latest
Machine Translation technologies for an efficient automatic SL
generation that considers the Classifier Predicate phenomenon
(CP). CPs are those SL phenomena that ‘‘make use the space around
the signer in a topologically meaningful way” (Huenerfauth, 2008).
Huenerfauth et al. (2008) designed an experimental study to com-
pare the effectiveness between their CP-aware system and Signed
English animations (words in an English sentence were replaced
with their corresponding signs without taking into account the
SL grammar). They found that American Sign Language (ASL) users
judged the CPs transmitted by their system significantly more nat-
ural, grammatically correct and understandable than Signed Eng-
lish. In addition, the CP-capable system proved to be more
effective than the Signed English animations as ASL users were
more accurate in selecting the corresponding animated visualiza-
tion (out of three animations) that both systems aimed at
transmitting.
In conclusion, both hypervideos with human signers and avatar
technology can offer web content exclusively in SL and could solve
the language switching problems which could emerge in bilingual
contexts. Paradoxically, SL-only approaches might not be as effec-
tive in facilitating the comprehension of academic information by
deaf students (e.g. Marschark et al., 2006; Rodriguez-Ortiz, 2007)
as oral languages are for hearing students. That is, the level of
I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256 249
comprehension obtained by hearing students exposed to spoken
lectures is higher than the level of comprehension of deaf students
exposed to signed or interpreted lessons.
For instance, Marschark et al. (2006) compared the effect on the
comprehension of two academic lectures (one on soil mechanics
and one on visual perception) by deaf signer students using real-
time text transcriptions, SL interpreting and both (Experiment 1).
They found that real-time text alone significantly improved the
performance of deaf students compared to the other two condi-
tions. The rest of experiments reported failed to replicate the
real-time text advantage but they did find that the deaf students’
performance in all conditions was below that of the hearing stu-
dents who received oral lectures. Rodriguez-Ortiz recently con-
ducted a study to evaluate to what extent deaf individuals
understand Spanish Sign Language interpretation. She found that
signing deaf participants extracted less information than hearing
signer participants from academic oral lectures (domestic violence
and apartment searching for university students). These results
replicate Marschark et al.’s findings (Experiment 1), but, using
more complete measures of comprehension (text based, infer-
ences, etc.). Rodriguez-Ortiz suggests differences in discourse pro-
cessing strategies as a possible explanation for the difference
between hearing and deaf people (Banks et al., 1990), as well as
less language exposure during childhood and the later acquisition
of spoken language, which could determine the acquisition of
meta-comprehension abilities.
Alternatively, these results could again be explained by the
‘‘lack of technical terminology in SL” hypothesis, referred to in
the previous section. Actually, although Marschark et al. (2006)
used introductory-level lectures, the topics were highly specific
(soil mechanics and visual perception). The topics used by
Rodriguez-Ortiz (2007) possibly involve a more common vocabu-
lary (domestic violence and apartment searching for university
students) but with no information about the word frequency or
the validity of the SL terminology used in these experiments, this
hypothesis is not falsifiable.
We are not aware of any empirical research comparing deaf
learners’ comprehension of academic content using SL-only
material (not interpreted but recorded) versus regular texts (not
real-time text transcriptions as in Marschark et al., 2006, nor
transliterations). As mentioned earlier, a study of these character-
istics would help to refute or support the hypothesis that SL is not a
good vehicle to transmit academic content.
3.3. Towards a rational combination of SL and text
The hitherto discussed literature indicates that the use of com-
plete SL webs, with either a bilingual or monolingual approach, could
not directly improve deaf signers’ access to web content. Therefore,
we propose that future research efforts should move away from the
dichotomy of SL webs versus written-text webs in an attempt to clar-
ify which kind of information in textual webs should be translated
into SL in order to foster deaf students’ comprehension.
Within this framework, empirical studies should explore how
key comprehension components can be supported by the inclusion
of SL generation technologies. Oakhill and Cain (2007) have identi-
fied four components that explain comprehension of hearing stu-
dents: vocabulary, inference making skill, comprehension
monitoring, and understanding of text structure. We later discuss
these issues with regard to deaf students’ comprehension in SL
hypertexts and propose a testable hypothesis for future studies.
The vocabulary and background knowledge has also proven to
be an important predictor of deaf students’ reading comprehension
ability (Garrison et al., 1992). Thus, in order to activate background
knowledge of deaf readers a testable alternative is the use of SL
summaries presented before reading. This strategy improves the
comprehension of easy narrative texts (fables) of deaf students
(Andrews et al., 1994). Further studies should test the effectiveness
of providing SL summaries in textual webs and others types of text
(e.g. expositive, more complex narrative texts), for example, com-
paring deaf users’ reading comprehension of web textual content
with and without previous presentation of videos with SL summa-
ries. This solution reduces the switching between languages and
the redundancy of information sources (SL and textual), which
could be counterproductive for deaf people, as discussed above.
Regarding the second component of reading comprehension,
inference making can be improved by enhancing text cohesion.
Along these lines, meta-textual elements (e.g. paragraphs, titles,
headlines, etc.) are fundamental for textual cohesion in hearing
readers (Rouet and Le Bigot, 2007). Therefore, bilingual webs
should facilitate deaf students’ identification of these elements
by translating important meta-textual elements into SL. A poten-
tial usability experiment with deaf students might compare two
types of informational webs: one including titles of news in a tex-
tual version, and a second using SL titles.
A third comprehension component that should be considered
in bilingual educational webs for deaf students is the support for
comprehension monitoring (i.e. assessment of one’s own under-
standing during reading). We have already discussed the effec-
tiveness of using textual adjunct questions during reading in
fostering deaf students’ monitoring and learning (Dowaliby and
Lang, 1999). The extent to which this support can be enhanced
by providing SL adjunct questions is still an open question for
future research.
The last comprehension component considered, understanding
of text structure, can be enhanced in SL hypertext systems by using
graphical overviews or content maps, which display hypertext nodes
and their relations. Graphical overviews have been used to promote
the comprehension of hearing users (Salmerón et al., in press). In-
deed, textual content maps improve the reading comprehension of
deaf students as well (O’Donell and Adenwalla, 1991), so they could
be used with the same purpose in SL hypertext. The digital newspa-
per of the Andalusian Deaf Community (Spain) includes a bilingual
content map (text and SL banners) showing the main sections of
the newspaper (Fig. 3). The SL banners are activated when the mouse
is placed over them, being the static graphic the most representative
part of each sign. For more complex hypertext or text content (i.e.
scientific hypertext), the content map might not only include the
main contents or sections which compose the hypertext system
but also a representation of the relationships between sections
(e.g. bi-directional or unidirectional links).
As we have reiterated throughout this article, the solutions dis-
cussed, extracted from models and data from previous studies con-
ducted with hearing and deaf individuals with textual information,
should not be implemented directly without prior empirical test-
ing with deaf signers and SL webs.
In the last two sections, we have mentioned a set of techniques
for SL generation which can facilitate information search and
learning on the web. However, apart from this functional dimen-
sion, there are other aspects that web designers, educators or deaf
users need to take into account before making a choice among the
systems available. Our precedent review has allowed us to abstract
a series of dimensions which configures a taxonomy of techniques
for SL generation on the web that is useful in guiding selection. We
present it in the following section.
4. Taxonomy of techniques for Sign Language generation on the
WWW
After reviewing the literature regarding SL generation, the
following dimensions have been revealed as being sufficiently
250 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
relevant to differentiate systems: SL communication modality,
signing-space, recording, task, authoring tool support, user-testing
and applicability in other devices. Next, we define each dimension
of the taxonomy (see a summary of technologies according to this
taxonomy in Appendix A).
SL communication modality refers to the way SL is conveyed. We
have identified at least two modalities: written notation (tran-
scription) and signed communication (Human Signer or Avatar
Signer).
Written notation of gestures: as we mentioned earlier, there are
some widely accepted notation systems, such as the Stokoe
Notation (Stokoe, 1960) for American Sign Language, the
Hamburg Notation System, HamNoSys (Prillwitz et al., 1989)
or SignWriting (Sutton, 1977). Advanced SL generators often
use them as intermediate representation of real signs; for exam-
ple, the SiGML notation system used in the eSign project is based
on HamNoSys (Kennaway et al., 2007). One of the main draw-
backs of notation systems is that their use is not generalized.
Signed communication by human signers: consist of human
signers who are pre-recorded and later displayed on web pages
by downloading or streaming videos. Expert and native signers
are generally the agents in the videos, so the SL should be effec-
tively articulated. The reliability of SL transmission is high. How-
ever, it requires broadband Internet access. In addition, video
creation can be highly expensive considering that web pages
are regularly updated. As far as implementation and deployment
are concerned, on-the-fly video broadcasting requires non-triv-
ial architectures in order to provide, for instance, video
streaming.
Signed communication by avatar signers: the above-mentioned
drawbacks caused by the use of human signers (Kennaway et al.,
2007) make it more efficient to provide mark-up that will subse-
quently be translated into a signing avatar. In other words, this
modality has the potential to generate SL on demand and for as
long as animations are rendered on the users’ computer. How-
ever, realism, expressiveness and readability are diminished. In
addition, note that the plug-ins that transform mark-up into ani-
mations are dependent on browsers and their respective release
versions. Thus, if the goal is to reach a broad audience it will also
require more financial investment. When it comes to virtual
avatars, they have some intrinsic characteristics:
Avatar type: if the objective of the avatar is to imitate a human
signer it is called a realistic avatar. Old developments that resem-
ble robots are called humanoids. For example, the above-men-
tioned ViSiCAST and the system developed by Huenerfauth
(2008) use humanoids. However, for certain specific purposes, such
as SL learning for children, cartoons can be used (Yanagimoto,
2004).
Dimensionality: this refers to the spatial dimensions of avatars
as virtual avatars can be 3D or 2D. Hand-shape recognition is a
key task in order to convey information accurately and only 3D
avatars can provide a complete picture as 2D avatars tend to hide
key information such as hand shape, palm orientation, location and
movement.
Location rendering: this characterizes the location (server-side
or client-side) where the processing of the avatar takes place. Gen-
erally, human signing videos embedded in web pages are broadcast
on the Internet in the same way as some virtual avatars’ anima-
tions are; that is, the pre-recorded videos (of human or avatar sign-
ers) are placed on the server-side. However, some novel
technologies allow the downloading of scripts or mark-up docu-
ments that browser plug-ins transform into avatars. This dimen-
sion has implications on the resources used as mark-up requires
less bandwidth than video content, and browser plug-ins interpret
this content dynamically on the client-side.
Technology: avatars for the web can be developed using several
technologies such as programming languages (VRML), graphics li-
braries (DirectX or OpenGL), mark-up (X3D) or by following ab-
stract representations (H-ANIM). However, as mentioned above,
some of these solutions, such as X3D-based ones, require brow-
ser-dependent plug-ins development, which may cause audience
segmentation, and due to financial considerations they only work
with one browser.
Some other dimensions to be considered for both human and
avatar signers are the following:
Task: this refers to the functionalities offered or the interaction
task that the SL generation system provides signer users with.
Cunliffe and Herring (2005) distinguish between the use of lan-
guage for consumption (e.g. reading web sites) and for production
(e.g. writing messaging) in multimedia systems and on the web.
Regarding consumption, SL generation systems should allow
signer users to efficiently and accurately perform at least two of
the most relevant and investigated web tasks such as Information
Search (Brumby and Howes, 2008; Kitajima et al., 2000; Pirolli and
Fu, 2003; Pirolli, 2004), and Learning (Foltz, 1996; Rouet and
Levonen, 1996;Salmerón et al., 2005;Spiro and Jenhg, 1990) which
have been the focus of our previous sections. Although some
authors distinguish between web searching (looking for a known
piece of information) and web browsing (looking around to see
what is available), in this paper, we have used the term ‘‘search”
in a broader sense, which encompasses both task subtypes. Accord-
ing to Jul and Furnas (1997), two search tactics can be followed:
queries and navigation, each one having different implications
for SL generation systems. Making queries implies SL generation
from text to SL or SL to text while navigation implies SL translation
of textual hyperlinks or generation of new mechanisms of naviga-
tion in SL.
Signer users’ learning or knowledge acquisition on the web re-
quires that SL generation systems generate bilingual or SL-only
hypertext and hypermedia.
Finally, production happens when users produce SL to commu-
nicate with a computer or other remote users (SignWriting mes-
saging by means of notation systems or signed communication,
recognized and interpreted by automatic SL recognition systems).
The production task is outside the scope of our paper, so we refer
to it just to make readers aware of its existence.
Summing up, a SL generation system must allow users to per-
form four tasks which imply the following challenges:
Web search by using queries: SL generation from written words or
SL.
Web search by navigation: SL generation from hyperlinks and
hypertext.
Learning: SL generation from hypertext and hypermedia.
Communication: SignWriting messaging
Recording: this dimension characterizes whether SL generation
is created on-the fly or needs to be recorded in advance. Pre-re-
corded content does not only apply to video but also to the devel-
opment of mark-up that will be later transformed into a SL
interpreter. Regarding the dynamic creation of SL, Kennaway
et al. (2007) state that automatic translation from written text to
SL cannot be achieved in the near future and it should be consid-
ered as challenging as translating web pages from one written lan-
guage to another. As we have seen, in order to address this issue
some authors such as Huenerfauth (2008) provide novel solutions
which consider Classifier Predicates, one of the more demanding SL
elements to be conveyed.
I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256 251
Applicability in other devices: with the advent of the mobile and
ubiquitous web, mobile devices, interactive TV and home appli-
ances can also embed web browsers as the increase in computing
performance allows these devices to handle more demanding tech-
nologies such as scripting or video support. PDAs or mobile phones
have several physical limitations due to their small size. Note that
the limited screen size of these devices constrains the readability
of SL as information conveyed by face expressions and hands is
more difficult to perceive. Some authors have partially addressed
mobile communication, proposing a system for mobile messaging
making use of SignWritting notation (Ahmed and Kuen Seong,
2006).
User-testing: computer systems involving user interaction
should have comprehensive user-testing. This is a Boolean dimen-
sion: existent versus non-existent. In the table, user-testing com-
ments are described more accurately.
The following dimensions just impact on signed commu-
nication:
Signing-space: most approaches show a whole body or waist-up
signer (whether be avatar or human), but in the case of virtual ava-
tars some approaches are incomplete. It is necessary to generate
facial expression, hand position, orientation, shape and movement,
and body posture for a holistic sign generation, in order to
adequately convey SL. Some avatars lack the facial expression
(Buttussi et al., 2007), while others lack body motion (Adamo-
Villani and Beni, 2005). Only a few approaches consider that the
surrounding space can be used for communicative purposes
(Huenerfauth, 2008).
SL broadcasting: SL generation can be divided into two modali-
ties depending on the visibility of the information sources. Simul-
taneous translation from hypermedia (text, video or sound) to SL
entails that video is embedded into the web document and both
information sources are shown at the same time. Alternatively,
translation can be broadcast separately, for example, via pop-ups.
Authoring tool support: one of the key factors for a SL generating
system to be successful and widely adopted is the authoring tool
support, without which developers cannot develop SL content.
The purpose of authoring tools can range from creating a mark-
up language for avatars to generating hyperlinks for videos of hu-
man signers or including subtitles.
Other: the specific SL (e.g. Italian, Spanish, American, etc.)
should also be considered as SL generation systems just implement
one or at most three SLs while others allow SL corpuses to be cre-
ated. Finally, providing mechanisms for the user to manipulate SL
video reproduction is also extremely helpful. Features such as con-
tent preview when the mouse is over the content or content for-
warding allows the user to master the SL generation application.
Huenerfauth (2009) demonstrated that providing these mecha-
nisms increases user performance in a SL comprehension task
and SL animations are better understood. Considering this variable,
we have introduced the category ‘‘Reviewing Supported’’ in Table
1. When the user only has limited access to such features (for in-
stance, he is only allowed to pause but not to forward or rewind
content), the system is labeled as ‘‘limited’’ in this column. When
the systems provide mechanisms for previewing content, under-
stood as an advanced technique for content reviewing, the system
is assigned the label ‘‘preview’’.
The Fig. 6 represents how these SL generation characteristics re-
late to each other in the taxonomy.
Considering the described dimensions, a holistic approach for
SL generation on the web should be efficient in terms of imple-
mentation, cost and bandwidth. Ideally, SL should be automati-
cally generated and SL representations should be capable of
handling hyperlinks. Providing tools for search tasks and other
additional features, such as mobile or ubiquity support, will also
be extremely significant in the following years. Thus, the chal-
lenge will be to provide SL generation with the same features
as traditional hypertext at the lowest implementation cost. In
addition to such technical requirements, our taxonomy also con-
siders functional dimensions such as the type of web task that
technologies might make available: search and learning. The
worth of current technologies is assessed according to their
functional validity and, when applicable, to the rest of the
dimensions in the taxonomy.
Fig. 6. Taxonomy for SL generation on the WWW.
252 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
5. Conclusions
The digital divide for the Deaf Community partially derives
from the low representation of SL in the digital world, which forces
signers to navigate in a non-native language. SL generation sys-
tems could solve this problem but their validity and usability is
not very clear.
This review has shown that in order to enable web information
search for signer users, SL cues could be provided in hypertext by
means of SL menus or embedding hyperlinks in video clips. More-
over, several notations for SL could be used to implement query
systems for SL. However, empirical evidence of the validity of most
of the technological approaches does not exist so they should be
applied with precaution or pre-tested with the target users
beforehand.
With regard to web knowledge acquisition and learning, we
have discussed some previous findings indicating that SL may
not be as efficient as it can be thought for learning information.
This type of finding, together with technical implementation
concerns, leads to the conclusion that it might not be either fea-
sible or desirable to use SL-only approaches or to translate all
textual content of a website to SL. The brief set of notes pro-
posed in this paper should be taken into consideration for fu-
ture research about the rational combination of text and signs
in hypertext.
On the other hand, the proposed taxonomy helps to categorize
SL generation techniques in another set of relevant and interde-
pendent dimensions such as location rendering, SL modality com-
munication, etc. For instance, regarding the modality dimension,
we have seen the two main approaches: human signers and ava-
tars. The main advantage of human signers’ videos over avatars
is that they convey SL more realistically and thus readability is en-
hanced. However, the weak points are twofold: from the end-user
point of view a high-speed Internet connection is required. This en-
tails that the user should have broadband connection coverage and
can afford the network connection. From the content providers’
point of view, using human signers can be an expensive option
and involves using non-conventional client–server architectures.
As a final remark we can state that web designers, web develop-
ers, educators or users could take advantage of the proposed taxon-
omy to critically evaluate and select the technological approach
that best fits their needs. We also hope that our review may bring
to light for researchers the need to investigate those issues that re-
main unresolved.
Acknowledgement
Markel Vigo is the recipient of a pre-doctoral fellowship from
the Department of Education, Universities and Research of the Bas-
que Government.
I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256 253
Appendix A
Summary of technologies for SL generation on the web according to the dimension of the taxonomy proposed.
System Communication
modality
Main
applicable
task
Object of
transcription
Signing-space Recording Authoring
tool
support
Applicability
in other
devices
Reviewing
supported
Specific Sign
Language
Results of user-testing
ViSiCAST Avatar Web searching:
navigation
Text-to-SL Face, hands,
arms, torso
Pre-
recorded
Yes Mobile
devices, TV
Yes British, German
and Dutch SL
Improvement in reading
comprehension3D
Elliott et al. (2000) Realist
eSign by Kennaway
et al. (2007)
Client-side
rendering
Francik and Fabian
(2002)
Avatar Web searching:
navigation
Text-to-SL Hands, arms,
torso
On-the fly N/A N/A N/A Polish None
3D
Humanoid Speech-to-
SL
Huenerfauth (2008) Avatar Web searching:
navigation
Text-to-SL Face, hands,
arms, torso
and
surroundings
On-the-
fly
N/A N/A N/A American Sign
Language (ASL)
Over performed other
systems in grammatically,
understandability and
naturalness
3D
Realist
VRML,
DirectX,
OpenGL
SigningLink Human Web searching:
navigation
Hyperlink face, hands,
arms, torso
pre-
recorded
SignEd depending
on device
support
limited ASL Satisfactory results but needs
revisionFels et al. (2006)
Kaibel et al., 2006 Human Web searching:
navigation
Hyperlink Face, hands,
arms, torso
Pre-
recorded
Yes Depending
on device
support
Yes German SL None
3DictSL Avatar Web searching:
querying
Word-to-SL Hands, body User-
driven
Yes Depending
on device
support
Preview Italian SL but it is
theoretically
language
independent
None
3D
Buttussi et al.
(2007)
Realist SL-to-word
X3D and H-
Anim
SL-to-SL
e-Lis Avatar Web searching:
querying
word-to-SL Face, hands,
arms, torso
User-
driven
Yes Depending
on device
support
Preview Italian SL None
2D
Vettori and Felice
(2008)
Humanoid SL-to-word
SignWriting in
Mobile Phones
Written
notation
Communication text, image
and sound
N/A User-
driven
Yes Mobile
devices
Yes N/A Satisfactory
Ahmed and Kuen
Seong (2006)
Sign Language
Subtitling
Adamo-Villani
and Beni, (2005)
Avatar Web searching:
navigation
video Semandroid:
head and
hands
Pre-
recorded
Yes Mobile
devices
Yes ASL Satisfactory
3D
Realist
254 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
References
Adamo-Villani, N., Beni, G., 2005. Sign Language Subtitling. ACM SIGGRAPH 2005
Educators program. ACM Press.
Ahmed, A.S., Kuen Seong D.S., 2006. SignWriting on mobile phones for the deaf. In:
Proceedings of the 3rd International Conference on Mobile Technology,
Applications and Systems, Mobility. ACM Press.
Alegría, J., 1999. La lectura en el niño sordo: elementos para una discusión (Reading
in the deaf child: elements for discussion). In: VV. AA. Actes du colloque:
Lenguaje Escrito y Sordera. Enfoques teóricos y derivaciones practicas,
Publicaciones Universidad Pontificia de Salamanca, Salamanque, pp. 59-76.
Andrews, J.F., Winograd, P., DeVille, G., 1994. Deaf children reading fables: using ASL
summaries to improve reading comprehension. American Annals of the Deaf
139, 378–386.
Asensio, M., 1989. Los procesos de lectura en los deficientes auditivos. PhD
dissertation. Universidad Autónoma de Madrid.
Banks, J., Gray, C., Fyfe, R., 1990. The written recall of printed stories by severely deaf
children. British Journal of Educational Psychology 60, 192–206.
Brumby, D.P., Howes, A., 2008. Strategies for guiding interactive search: an
empirical investigation into the consequences of label relevance for
assessment and selection. Human–Computer Interaction 23, 1–46.
Buttussi, F., Chittaro, L., Coppo, M., 2007. Using Web3D technologies for
visualization and search of signs in an international sign language dictionary.
In: Proceedings of the 12th International Conference on 3D Web Technology,
Web3D, ACM Press, pp. 61–70.
Caldwell, B., Cooper, M., Guarino Reid, L., Vanderheiden, G., (Eds.), 2008. Web
Content Accessibility Guidelines 2.0. W3C Recommendation. <http://
www.w3.org/TR/WCAG20/> (accessed 11.11.2008).
Chisholm, W., Vanderheiden, G., Jacobs, I., (Eds.), 1999. Web Content Accessibility
Guidelines 1.0. W3C Recommendation. http://www.w3.org/TR/WAI-
WEBCONTENT/ (accessed 05.05.99).
Cunliffe, D., Herring, S.C., 2005. Introduction to minority languages, multimedia and
the Web. New Review of Multimedia and Hypermedia 11 (2), 131–137.
Dowaliby, F., Lang, H.G., 1999. Adjunct aids in instructional prose: a multimedia
study with deaf college students. Journal of Deaf Studies and Deaf Education 4,
270–282.
Elliott, R., Glauert, J.R.W., Kennaway, J.R., Marshall, I., 2000. The development of
language processing support for the ViSiCAST project. In: Proceedings of the 4th
International ACM Conference on Assistive Technologies, ASSETS 2000, ACM
Press, pp. 101–108.
Fajardo, I., Cañas, J.J., Salmerón, L., Abascal, J., 2006. Improving deaf users’
accessibility in hypertext information retrieval: are graphical interfaces useful
for them? Behaviour & Information Technology 25 (6), 455–467.
Fajardo, I., Arfé, B., Benedetti, P., Altoé, G.M., 2008a. Hyperlink format,
categorization abilities and memory span as contributors to deaf users
hypertext access. Journal of Deaf Studies and Deaf Education 13 (2), 241–256.
Fajardo, I., Parra, E., Cañas, J.J., Abascal, J., López J.M., 2008b. Web information search
in sign language. In: symposium Informal Learning on the Web: Individual
Differences and Evaluation Processes, XXIX International Congress of
Psychology, pp. 20-25, Berlin, July 2008.
Fajardo, I., Parra, E., Cañas, J.J., Abascal, J., López J.M., Gea, M., 2008c. Web textual
hyperlinks supported with sign language videos. In: Workshop on Cognition
and the Web 2008, Granada, Spain. 26, April 2008.
Fajardo, I., Cañas, J.J., Salmerón, L., Abascal, J., 2009. Information structure and
practice as facilitators of deaf users’ navigation in textual websites. Behaviour
and Information Technology. Tailor & Francis.
Fels, J., Hardman, J., Lee, D.G., 2006. Sign language Web pages. American Annals of
the Deaf 151, 423–433.
Foltz, P.W., 1996. Comprehension, coherence, and strategies in hypertext and linear
text. In: Rouet, J.F., Levonen, J.J., Dillon, A., Spiro, R. (Eds.), Hypertext and
Cognition. LEA, Mahwah, New Jersey, pp. 109–136.
Francik, J., Fabian, P., 2002. Animating Sign Language in real time. In: 20th IASTED
International Multi-Conference, Applied Informatics AI 2002, pp. 276–281.
Garrison, W., Dowaliby, F., Long, G., 1992. Reading comprehension test item
difficulty as a function of cognitive processing variables. American Annals of the
Deaf 137, 22–30.
Gentry, M.M., Chinn, K.M., Moulton, R.D., 2004–2005. Effectiveness of multimedia
reading materials when used with children who are deaf. American Annals of
the Deaf 149, 394–402.
Goldin-Meadow, S., Mayberry, R., 2001. How do profoundly deaf children learn to
read? Learning Disabilities Research and Practice 16 (4), 222–229.
Huenerfauth, M., 2008. Generating American Sign Language animation: overcoming
misconceptions and technical challenges. Universal Access in the Information
Society 6 (4), 419–434.
Huenerfauth, M., 2009. A linguistically motivated model for speed and pausing in
animations of american sign language. ACM Transactions on Access Computing
2 (2). no. 9.
Huenerfauth, M., Zhao, L., Gu, E., Allbeck, J., 2008. Evaluation of American Sign
Language generation by native ASL signers. ACM Transactions on Access
Computing 1 (1) (no. 3).
Instituto Estatal de Estadística (Spanish Statistical Office), 2006. <http://
www.ine.es/>.
Jul, S., Furnas, G.W., 1997. Navigation in electronic worlds: a CHI 97 Workshop. ACM
SIGCHI Bulletin, ACM Press, pp. 44–49.
Kaibel, A. Grote, K., Knoerzer, K, Sieprath, H., Kramer, F., 2006. Hypertext in Sign
Language. In: 9th ERCIM ‘‘User Interfaces for All” Workshop Adjunct
Proceedings. <http://www.ui4all.gr/workshop2006/publications/adjunct-
proceedings.html>.
Kintsch, W., 1988. The role of knowledge in discourse comprehension: a
construction–integration model. Psychological Review 2 (95), 163–182.
Kintsch, W., 1998. Comprehension: A Paradigm for Cognition. Cambridge University
Press, New York.
Kitajima, M., Blackmon, M.H., Polson, P.G.A., 2000. Comprehension-based model of
Web navigation and its application to Web usability analysis. In: People and
Computers XIV. Springer, pp. 357–373.
Kennaway, J.R., Glauert, J. R.W., Zwitserlood, I., 2007. Providing signed content on
the internet by synthesized animation. ACM Transactions on Computer–Human
Interaction 14 (3), 1–29.
Kralisch, A., Berendt, B., 2005. Language-sensitive search behaviour and the role of
domain knowledge. The New Review of Hypermedia and Multimedia 11 (2),
221–246.
Leybaert, J., Alegria, J., Morais, J., 1982. On automatic reading processes in the deaf.
Cahiers de Psychologie Cognitive 2, 185–192.
Marschark, M., Leigh, G., Sapere, P., Burnham, D., Convertino, C., Stinson, M., Knoors,
H., Vervloed, M.P.J., Noble, W., 2006. Benefits of sign language interpreting and
text alternatives to classroom learning by deaf students. Journal of Deaf Studies
and Deaf Education 11, 421–437.
Mautone, P.D., Mayer, R.E., 2007. Cognitive aids for guiding graph comprehension.
Journal of Educational Psychology 99, 640–652.
Mayer, R.E., 2005. Principles for reducing extraneous processing in multimedia
learning: coherence, signalling, redundancy, spatial contiguity, and temporal
contiguity principles. In: Mayer, R.E. (Ed.), The Cambridge Handbook of
Multimedia Learning. Cambridge University Press, Cambridge, pp. 183–200.
Michel, T., 2008. W3C Synchronized Multimedia Home Page. <http://www.w3.org/
AudioVideo/>.
Moreno, R., Mayer, R.E., 2002. Verbal redundancy in multimedia learning: When
reading helps listening. Journal of Educational Psychology 94, 156–163.
Namatame, M., Nishizaki, Y., Kitajima, M., 2007. Improving usability of web pages
for hard-of-hearing persons: an investigation of the utility of pictograms. HCI
International Posters, 176–180.
Nelson, D.L., Reed, U.S., Walling, J.R., 1976. Picture superiority effect. Journal of
Experimental Psychology: Human Learning and Memory 2, 523–528.
O’Donell, A.M., Adenwalla, D., 1991. Using cooperative learning and concepts maps
with deaf college students. In: Martin, D.S. (Ed.), Advances in Cognition,
Learning and Deafness. Gallaudet University Press, DC, pp. 348–355.
Oakhill, J., Cain, K., 2007. Issues of causality in children’s reading comprehension. In:
McNamara, D.S. (Ed.), Reading Comprehension Strategies: Theory,
Interventions, and Technologies. Lawrence Erlbaum, Mahwah, NJ, pp. 47–72.
Paivio, A., 1991. Dual-coding theory: retrospect and current status. Canadian
Journal of Psychology 45, 255–287.
Pirolli, P., Card, S.K., 1999. Information foraging. Psychological Review 106 (4), 643–
675.
Pirolli, P., Fu, W.-T.F., 2003. SNIF-ACT: a model of information foraging on the World
Wide Web. In: Proceedings of the Ninth International Conference on User
Modelling.
Pirolli, P.L., 2004. The use of proximal information scent to forage for distal content
on the World Wide Web. In: Kirlik, A. (Ed.), Adaptive Perspectives on Human–
Technology Interaction. Oxford University Press, Cambridge UK, pp. 247–266.
Prillwitz, S., Leven, R., Zienert, H., Hanke, T., Henning, J., HamNoSys, 1989. Version
2.0; Hamburg Notation System for Sign Languages. An introductory guide.
International Studies on Sign Language and Communication of the Deaf 5,
Signum.
Richards, J., Fels, D., Hardman, J., 2005. The Educational Potential of the Signing
Web, Instructional Technology and Education of the Deaf.
Rodriguez-Ortiz, I.R., 2007. Sign Language Comprehension: the Case of Spanish Sign
Language. Journal of Deaf Studies and Deaf Education Advance Access published
December 13, 2007.
Rouet, J.F., Le Bigot, L., 2007. Effects of academic training on metatextual knowledge
and hypertext navigation. Metacognition and Learning 2 (2–3), 157–168.
Rouet, J.F., Levonen, J.L., 1996. Studying and learning with hypertext: empirical
studies and their implications. In: Rouet, J.F., Levonen, J.L., Dillon, A., Spiro, R.J.
(Eds.), Hypertext and Cognition. Erlbaum, Hillsdale, NJ.
Salmerón, L., Baccino, T., Cañas, J.J., Madrid, R.I., Fajardo, I., in press. Do graphical
overviews facilitate or hinder comprehension in hypertext? Computers &
Education.
Salmerón, L., Cañas, J.J., Kintsch, W., Fajardo, I., 2005. Reading strategies and
hypertext comprehension. Discourse Processes 40, 171–191.
Smith, C.E., 2006. Where is it? How deaf adolescents complete fact-based internet
search tasks. American Annals of the Deaf 5, 519–529.
Spiro, R., Jenhg, C., 1990. Cognitive flexibility and hypertext: theory and technology
for the nonlinear multidimensional traversal of complex subject matter. In: Nix,
D., Spiro, R. (Eds.), Cognition, Education and Multimedia: Exploring ideas in
High-Technology. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 163–205.
Stokoe, W., 1960. Sign Language structure: an outline of the visual communication
systems of the American deaf. Studies in Linguistics, Occasional Papers, 8.
Sutton, V., 1977. Sutton movement shorthand; writing tool for research. In: Stokoe
(Ed.), Proceedings of the 1st National Symposium on Sign Language Research
and Teaching. National Association of the deaf, Silver Spring, MD, pp. 267–
296.
I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256 255
Sweller, J., 2005. The redundancy principle in multimedia learning. In: Mayer, R.E.
(Ed.), The Cambridge Handbook of Multimedia Learning. Cambridge University
Press, Cambridge, pp. 183–200 (159–167).
Van Dijk, T.A., Kintsch, W., 1983. Strategies of Discourse Comprehension. Academic
Press, Nueva York.
Vettori, C., Felice, M., 2008. e-LIS: ELECTRONIC Bilingual Dictionary Italian Sign
Language-Italian. In: Proceedings of the XIII EURALEX International Congress.
Videotext.web. Austrian Sign Language Service Centre and indeed! <http://
www.oegsbarrierefrei.at/<<http://www.indeed.at/web/products/>.
World Health Organization (WHO), 2006. Deafness and hearing impairment.
<http://www.who.int/mediacentre/factsheets/fs300/en/index.html>.
Yanagimoto, M., 2004. Sign Language Animation Site ‘‘Hello! Astroboy”. ACM
SIGGRAPH 2004. ACM Press.
256 I. Fajardo et al. / Interacting with Computers 21 (2009) 243–256
