Content uploaded by Markel Vigo
Author content
All content in this area was uploaded by Markel Vigo on Jun 19, 2020
Content may be subject to copyright.
Notice
This is the author copy of the paper:
Amaia Aizpurua, Myriam Arrue, Markel Vigo (2015) Prejudices, mem-
ories, expectations and confidence influence experienced accessibility on
the Web. Computers in Human Behavior 51, 152–160. Elsevier
The published paper is available at http://dx.doi.org/10.1016/j.
chb.2015.04.035
Note that there might be some inconsistencies between this and the above
publication so use this copy at your own risk.
1
Prejudices, memories, expectations and confidence
influence experienced accessibility on the Web
Amaia Aizpuruaa,b, Myriam Arruea, Markel Vigob
aSchool of Computer Science, University of the Basque Country, UPV/EHU,
Donostia-San Sebasti´an, Spain
bSchool of Computer Science, University of Manchester,
Manchester, United Kingdom
Abstract
Evidence suggests that compliance with accessibility standards does not always
guarantee a satisfying user experience on the Web. The literature indicates that
addressing the expectations users have about online content and functionalities
is crucial to bridge this gap. We examine the role played by subjectiveness, ex-
perience and, particularly, expectations on how users experience the accessibility
on the Web. To do so, 11 blind participants were enquired through interviews
and questionnaires about 12 tasks they completed in four websites. Thematic
analysis on the transcriptions reveals that expectations are often built up on
previous experiences and preconceived ideas. Particularly, the content which
is explicitly labelled as accessible arises the curiosity and creates high expec-
tations about the accessibility of the website. We also find that, in addition
to unmet expectations, prejudices on branding issues and the memories evoked
by past experiences or emotional bonds does not only affect the way in which
users perceive and experience accessibility, but also the overall user experience.
Identifying the nature of expectations is key (i) to formalise more exhaustive
user testing protocols and (ii) to complement and complete existing accessibility
guidelines.
Keywords: Behavioral sciences, Blind users, Screen readers, Web, Web
accessibility, User Experience
1. Introduction
The Web Content Accessibility Guidelines (WCAG) are the internationally
recognised standard for Web accessibility (Caldwell et al., 2008; Chisholm et al.,
1999). However, evidence suggests that compliance to these accessibility stan-
dards does not necessarily guarantee a satisfying user experience on the Web.
Studies that corroborate such evidence state that guidelines compliant websites
can be inaccessible for specific users in specific situations. The other way around
also applies: non-compliant websites do not necessarily have to pose a challenge
Preprint submitted to Computers in Human Behavior May 28, 2015
to users. For instance, Petrie et al. (2004) conducted a user study with 51 par-
ticipants with disabilities, where the authors observed, identified and classified
the difficulties that users encountered. They found that 45% of the observed
problems were not related to any violation of WCAG 1.0 checkpoints (Chisholm
et al., 1999). The second version of guidelines, namely WCAG 2.0 (Caldwell
et al., 2008) was released to address the weaknesses exhibited by the previ-
ous versions and to cater for the technological updates that occurred hitherto.
Power et al. (2012) conducted an empirical study about the problems identified
by 32 screen reader users on the Web. Results revealed that only 50.4% of
the problems encountered by participants were covered by WCAG 2.0 success
criteria (henceforth SC). Consistent coverage figures —measured in terms of
the percentage of actual problems addressed by guidelines— were reported by
Rømen and Svanæs (2012). Among those problems not covered by these SC,
the 13.5% of all user problems are related to unmet expectations in terms of
unexpected content (Power et al., 2012).
Even if guidelines are an invaluable starting point for building accessible
sites, the above-mentioned findings indicate that there is a need to explore com-
plementary ways of building accessible websites beyond conformance to guide-
lines. In this regard, we claim that understanding how users experience and
perceive Web accessibility is vital to bridge this gap (see Section 2). We expand
on this by exploring how subjective dimensions and especially how user expecta-
tions influence on the perception and experience of accessibility. Inspired by the
work of McCarthy and Wright (2004), who identified that previous experiences,
prejudices and branding are the key dimensions that shape user expectations, we
analyse how these dimensions influence the perception of blind users about Web
accessibility. To do so, we conducted an exploratory study in which 11 blind
participants were enquired about the tasks they had to accomplish in four web-
sites (see Section 3). These websites contained different accessibility problems,
aesthetic properties and experiential values. The study included semi-structured
interviews, user observation and questionnaires that participants filled out.
Our analysis (see Section 4) reveals that expectations are built up on previ-
ous experiences (either in the physical world or in the Web) and on preconceived
ideas. We found that unmet expectations are often related to participants’ un-
certainty. We also learned that if a Web page is explicitly labelled as accessible
and the participant notices it, this does not only spark interest, but also creates
expectations that are not always satisfied. As a consequence, if high expecta-
tions are not met deception and frustration can be more severe. We also find
that memories, past experiences and emotional bonds do also influence the per-
ception of website accessibility. Prejudices towards a brand can influence this
perception too. We conclude that all the above does not only affect user per-
ception about the accessibility of a given website, but also the user experience.
The main outcome of this work suggests that users’ perceived accessibility, is
not only determined by conformance to guidelines, but also by other experiential
and more subjective dimensions, as discussed in Section 5. Specifically, we reach
this conclusion by examining the following axes:
– We explore how the dimensions that build user expectations including past
3
experiences and preconceptions reflect on the navigation of blind users on
the Web.
– We uncover how these dimensions and their consequences determine how
blind users perceive the accessibility of a website.
– We explore the association between real life experiences and Web experi-
ences.
– We discuss how a website is actually perceived when it contains an ‘acces-
sible’ version.
2. Background
In recent years efforts on Web accessibility have been devoted to defining
guidelines, comparing evaluation methods and metrics or providing tool sup-
port for intelligent browsing. The publication of the WCAG guidelines has not
only led to the implementation of worldwide policies1, but also to a wide range
of automated evaluation tools2, supporting developers building accessible web-
sites. Most efforts have been directed to evaluate and measure Web accessibility
according to standards with the aim of improving websites (Lazar et al., 2004).
Even if those efforts are necessary in order to advance towards having a better
understanding of accessibility problems on the Web, they do not seem to be
enough. A website may have an adequate level of accessibility, but still not
be sufficient for users. Even if a developer or evaluator considers a website to
be accessible, users may have a different perception about its accessibility. A
website designed to be compliant to accessibility standards fails if users cannot
experience that accessibility. Since compliance to accessibility standards does
not necessarily guarantee a satisfying experience, we pose that the actual per-
ception and experience of accessibility barriers have to be examined, beyond
guidelines.
Users’ perceived accessibility does not always match to that represented in
terms of compliance to guidelines. There are several factors that play a key role
in this regard: for instance, some users have a repertoire of skills which allow
them to overcome barriers by employing workarounds, namely coping tactics or
strategies (Vigo and Harper, 2013). In other cases, users do not notice barriers
if these do not prevent them from accomplishing their tasks.
Nevertheless, individuals find the Web a mean, not only for working or
achieving informational goals, but also for different purposes like communica-
tion, entertainment, social networking or contributing to building the Web, to
name a few. In order to establish the scope of this study, and suggest its
generalisability to similar website types we use the classification proposed by
De Marsico and Levialdi (2004). They proposed a taxonomy which establishes
different categories of websites based on a three-dimensional space, derived from
1Web Accessibility Policy Resources: http://www.w3.org/WAI/policy- res
2A complete list of Web accessibility evaluation tools: http://www.w3.org/WAI/ER/tools/
complete
4
Aristotle’s rhetorical triangle (ethos, logos, pathos). The three axes of the taxon-
omy represent different communication features of websites: 1. personal/social,
which refers to the different target audiences; 2. site/info is the type of informa-
tion provided by the website; 3. communication style refers to the effort made
by the designer to address the users’ affective sphere.
Considering the wide range of websites that exist and thus, the variety of
possibilities offered by the Web, non-instrumental information merits attention
too. The ISO 9241-11 (1998) standard supports this claim by establishing that
subjective qualities such as learnability can also contribute to the usability of a
product. Beyond traditional evaluation techniques in Human-Computer Inter-
action, which have emphasised more on the instrumental aspects that charac-
terise user interaction (such as error rates and task completion times), the User
Experience (UX) is intended to provide a more holistic approach to understand
how users experience the interaction. With the aim of bridging the HCI and
UX research, Bargas-Avila and Hornbæk (2011) conducted a systematic survey
on how previous UX studies address issues like situation, context of use, dimen-
sions, when is UX assessed and how UX data is collected. Since UX takes into
account the user interaction from a broader perspective, it brings a wider range
of possibilities to our research context by considering subjective aspects related
to users’ perceptions and expectations.
Little attention has been paid to how blind users experience the accessibility
barriers on websites. Generally, as blind users are not specially trained on
Web accessibility, their perception of Web accessibility can be very subjective
and may not match to that of experts. The initial findings of Aizpurua et al.
(2013) indicate that expectations play an important role when it comes to how
blind users perceive accessibility barriers. This perception is not only about the
website and its accessibility, but about what they expect from their engagement
with a website. Also, users may assess the accessibility of website based on UX
qualities, which at first sight might have little to do with the website’s actual
accessibility. Our research builds upon this work and based on a more in-depth
analysis, adds new evidence to the relatively unexplored research corpus about
the intersection of user experience and Web accessibility.
3. Method
The unexplored and subjective nature of the experienced accessibility calls
for a preliminary qualitative approach (Adams et al., 2008) that will inform
subsequent stages of our research. An exploratory study was conducted to
collect the data that was analysed using qualitative research methods. The
following sections describe our methodological approach in order to uncover
the interplay between experienced Web accessibility and other experiential and
subjective aspects.
3.1. Participants
Eleven legally blind participants —four female participants—, who were rep-
resentative of the user group being studied (Sears and Hanson, 2011), were re-
5
Table 1: Demographic data and characterisation of expertise on the Web
Id Gender Age Familiarity Frequency of Self-reported Observed
with the Web Web use expertise expertise
P01 F 29 >7 years daily advanced expert
P02 F 29 >7 years daily advanced expert
P03 M 39 >7 years daily advanced advanced
P04 M 54 4-6 years daily intermediate intermediate
P05 M 43 1-3 years weekly beginner intermediate
P06 M 21 1-3 years weekly beginner beginner
P07 M 64 >7 years daily intermediate advanced
P08 M 58 >7 years daily intermediate intermediate
P09 F 54 4-6 years daily intermediate advanced
P10 M 64 4-6 years daily beginner intermediate
P11 F 42 >7 years weekly intermediate beginner
cruited in partnership with the National Organization of Spanish Blind People
(ONCE)3. The median age of participants was 43 years, with a range of 21–64
years. Table 1 shows demographic data as well as the characterisation of user
expertise in two ways: self-reported and observed skills (columns 6 and 7 re-
spectively). The former was collected by asking participants to rate their Web
expertise on a four-item scale: expert, advanced, intermediate and beginner. In
order to avoid the bias of self-judgements we also assessed Web expertise from
the perspective of an external observer. This last indicator was computed using
observations of the first author and the reports of the instructor, who trained
the participants for improving their computer skills at the local delegation of
ONCE. Specifically, the first author paid attention to the strategies employed
and the confidence shown when carrying out the tasks. As shown in Table 1,
self-rated and observed skills did not always match. This is in line with the
conclusions of a study (van der Geest et al., 2014), which shows that self-rated
Internet competence of visually impaired users is not always related to their
actual performance on common Internet tasks.
3.2. Apparatus
All participants were legally blind and utilised screen readers to navigate
on the Web: ten participants were Jaws users (version 10, except P01 and P03
who used version 12) on Internet Explorer/Windows (XP and and Win 7), while
just one participant (P02) was a VoiceOver user on Safari over MacOS. The first
three participants were observed in the research facility of the HCI laboratory
at the School of Computer Science, and participants brought their own laptop.
Remaining sessions took place in a room at the ONCE delegation in Donostia-
San Sebasti´an, where the remaining eight participants used the same laptop and
keyboard, which were provided by the ONCE.
3http://www.once.es/new/otras-webs/english
6
3.3. Stimuli
In order to let the subjective dimensions emerge, we recruited local partic-
ipants and selected websites of restaurants that are popular and well-known in
the area where they lived. We focused on only one type of website, so that we
could establish analogous tasks across different stimuli, and also to minimise
confounding factors related to the type of stimuli. According to De Marsico
and Levialdi (2004)’s taxonomy, the restaurant websites we used fall onto the
categories of commercial sites, are targeted to a general audience and exhibit
an informative-seductive communication style.
When we sampled the websites, the main selection criteria was the accessi-
bility level of the websites, as we wanted to study how users experienced web-
sites with different accessibility issues. Unfortunately we did not find any local
restaurant website that was fully compliant with WCAG 2.0 level AA. Also, it
must be noted that some websites were discarded due to the severe accessibility
problems they contained in the homepage. Another selection criteria was about
branding issues, W1 and W3 represent internationally well-known restaurants
with an innovative character (see Figure 1). Their culinary style is based on
creativity, investigation and experimentation. Their ‘author’s cuisine’ has been
recognised worldwide by the award of Michelin stars. In contrast, the other
two restaurants (W2 and W4), even if they are quite popular in Donostia-San
Sebasti´an, their style is based on traditional Basque cuisine. As it can be ob-
served in Figure 1 the visual design of the websites is aligned with the style of
the restaurant: the internationally renowned restaurants (those having an inno-
vative and avant-garde style: W1 and W3) use a reduced colour palette, have
had their typographies designed and contain high-quality close-up pictures. On
the other hand, the design of the traditional restaurants is less ambitious.
These websites had been chosen by following this selection process: we first
listed 25 websites of local restaurants; then we evaluated their homepages, us-
ing four automated Web accessibility evaluation tools: AChecker (Gay and
Li, 2010), EvalAccess (Abascal et al., 2004), TAW 4and WAVE5. Based on
the automated tests of the aforementioned tools we classified websites in four
groups: very accessible, accessible, not so accessible, poorly accessible. Within
each group, we narrowed down our selection by evaluating again the homepage
and two more Web pages using the WAQM metric Vigo et al. (2007). Then
we merged the four groups into two (highly accessible and poorly accessible).
Within each group, we classified websites based on branding issues, considering
the style of the restaurant: traditional and innovative. After that, we carried
out an expert evaluation where the Barrier Walkthrough (BW) method (Bra-
jnik, 2006) was applied. Based on the results, we finally selected the two most
and least accessible websites for each type of restaurant (traditional and inno-
vative). Once we had the four websites, we evaluated them according to the
WCAG 2.0 AA conformance level. Both, VoiceOver and Jaws screen readers
4http://www.tawdis.net
5http://wave.webaim.org/
7
were used during the manual Web accessibility evaluations performed.
In general, W1 and W2 present higher accessibility levels than W3 and W4.
Their homepages satisfy 73%, 69%, 52% and 36% of SC for the AA level of
WCAG 2.0 respectively. It must be noted that W1 is the accessible version
of the restaurant’s website (although we did not tell participants about that
detail). The most severe accessibility issue of W3 is that the navigation menu
consists of seven images, all with the same ‘alt’ text, which was the word image.
In W4, the main accessibility problem is related to Flash based dynamic content
that cannot be accessed by the screen reader.
3.4. Procedure
Each session was conducted with one participant at a time. Once the partic-
ipants were informed about the objectives of the study and the procedure of the
session, they signed the consent form. In order to obtain non-biased answers,
and make participants feel free to respond as honestly as possible, we high-
lighted that we had no relationship nor conflict of interest with the websites.
We also made it clear that there were no right or wrong answers by emphasising
that we were interested in their insights and personal opinions. We explained
participants that they were playing the role of testers rather than being the
tested subjects. Then, we collected demographic data and enquired participants
about their expertise and familiarity with information technology. After that,
participants were interviewed about their previous experiences and expectations
regarding restaurant websites. Once the interview had finished, they could start
to navigate the first website. After the navigation with each website, by means
of questionnaires and semi-structured interviews, we collected information about
their browsing experience. This information was diverse and encompassed their
perceptions about website aspects, the problems they encountered, the acces-
sibility barriers found and the affective considerations evoked by the websites.
Each session, was video and audio recorded to enable subsequent analysis. At
the end of the session each participant was rewarded with a USB memory stick
for taking part.
3.5. Tasks
Each participant was asked to try to complete three tasks on each website.
These tasks were the same for each website: 1. freely navigate on the website,
in order to become familiar with it; 2. find information about the gastronomic
offer; 3. find the means offered by the website to make a booking. We prevented
a learning effect from happening because each website structured its content in a
different manner. However, in order to minimise order effects, the task order was
counterbalanced. The idea was to let the users explore the website through tasks
which were related to the typical expectations regarding restaurant websites.
Hence, the tasks were deliberately not very specific and tried to trigger real world
situations, where users have an informational goal in mind but explicit directions
to accomplish are absent. Participants were told that the time estimated for
each task was between 5-10 minutes but we highlighted that they could spend
8
(a) W1
(b) W2
(c) W3
(d) W4
Figure 1: Screenshots of websites.
9
longer time if they wanted, or even shorter time if they finished or they had lost
their interest.
3.6. Data Analysis
We followed the six phases of the thematic analysis (Braun and Clarke,
2006): 1. Familiarising with the data; 2. Generating initial codes; 3. Searching
for themes; 4. Reviewing themes; 5. Defining and naming themes; 6. Produc-
ing the report. We performed an open coding phase in order to identify the
emerging themes in the data. Transcripts were printed on paper and by using
highlighters of different colours we identified potential themes. In order to re-
view and validate the identified themes, we analysed all transcripts again using
NVivo 10, a qualitative data analysis software. To ensure the reliability of the
coding scheme, we asked an external coder to review the transcripts and use the
initial coding scheme to code the data. We computed Cohen’s Kappa statistic to
measure the average level of agreement between both coders and a coefficient of
0.86 was obtained, which indicates a substantial level of agreement. Even if the
agreement was quite high, the results were shared and the discrepancies were
discussed and resolved. This exercise led to some changes in terms of adding
and deleting some instances on the final coding scheme.
4. Results
This section presents the themes and sub-themes found with regard to the
experiential aspects that emerged from the interaction of participants with the
websites6.
4.1. General Expectations (10)
We define a expectation as anything expected by the participant, no matter
if it is met or not. That is, expectations are any assumption that may or may not
be realistic. We learned that the general expectations participants had with re-
gard to the restaurant websites, were built up on to previous experiences: either
real (physically at restaurants) or on the Web (with similar websites). Not sur-
prisingly, most participants expected to find information about the gastronomic
offer, prices, contact, location, regular timetable or bookings.
Experiences at restaurants. Participants relied on past experiences at restau-
rants to express what they expected from these websites. Beyond specific con-
tent, three participants wanted restaurant websites to provide them with enough
and useful information to control the situation when they were physically at the
restaurant. As a consequence of having experienced uncomfortable situations
in the past, they reported that they usually try to avoid these situations by
visiting the website before going to the restaurant. This provides them with
6The number between parentheses indicates the number of users who reported each sub-
theme.
10
useful information to figure out what they can find in the restaurant, and also
increases their sense of autonomy and security. This strategy is a way to take
precaution to avoid potential uncomfortable situations, and thus, it can facili-
tate the real experience at the restaurant: “There are many times that you have
such a long menu, which is a pain, because the waiters don’t even ask you later.
They say ‘we have this, this and this’, when the menu is about 30 dishes! .. . but
they decide for you what you don’t want . . . Of course, if someone has to read
the menu you can spend 15 minutes there reading the menu. I understand that.
Well, in order to avoid this I check the menu on the Web” (P02).
Experiences on the Web. In addition to real experiences related to restaurants,
past Web experiences with other restaurant websites may also contribute to
the creation of expectations. For example, the two participants who habitually
visited restaurant websites did not expect them to be accessible as they often
came up with many inaccessible websites: “They are very inaccessible, I tell
you! Horrible, they are horrible, besides many of them have Flash, and now
with HTML5 Flash is not needed . . . Online reservations are oof! . . . a nightmare
. . . a nightmare and you end up calling them [on the phone]” (P02).
4.2. Website Specific Expectations
General expectations refer to any restaurant website, while specific expec-
tations are related to what participants expected from a particular website.
In comparison to general expectations, these are more related to the aspects
of a particular website such as content and functionality. While participants
navigated a website, the expectations they had with regard to its content or
functionality were not always met.
4.2.1. Content-Related Expectations
Content found where not expected (4). Encountering content in unexpected lo-
cations was not a major problem provided that the content was a relevant piece
of information to complete the given task. Some even seem to be surprised of
having found it unexpectedly: e.g. the telephone number was very handy at
the beginning of the homepage in W2 “We already have a task done! [Laughs]”
(P09-W2). However, if they found something they did not want it did not match
their mental model participants got confused: “Yes, before the first link, or the
. . . well, there was ‘History’, the link was ‘Restaurant’, well I don’t know what
it does there ” (P03-W4).
Content not found where expected (10). Sometimes participants did not find
the content they wanted on a website. Either because of difficulties with the
website, the assistive technology, because their search strategy was not effective
or because the content they were looking for was not in the website. The
majority of the comments from participants were related to information about
the given tasks, especially about the gastronomic offer: “The menu was what I
wanted to see . . . the menu and the prices” (P04-W1) and “It does not have no
menu nor anything. Because if you go to a website I could not care less about
the history, what I want to know is what to eat” (P06-W3).
11
Content not found as expected (11). This happened when the content was in a
different form than expected by the participant. For instance, P09 thought that
the menu would come in a document such as a PDF, or a list of dishes on the
same page. However, in the W2 website, information about the gastronomic
offer was conveyed through links. In fact, participants had to navigate through
different pages that were located at different depths in order to get the infor-
mation. “The menu, I expected to see a document all together . . . and there was
no menu as such, it was like a bunch of links” (P09-W2).
Not accessed content (4). Expectations sometimes go beyond the content that
participants visited. Sometimes only by the link text, they made assumptions
about the content beyond the link, even without exploring the Web page. For
instance, one participant imagined that the video on the W3 website was a
welcoming video: “There was also a welcoming written text, with an attachment
or a link to Youtube, well I understand it’s for welcoming, because it was next
to the written text” (P03-W3).
Content labelled as ‘accessible’ (5). It was surprising to observe how the ‘ac-
cessible’ keyword aroused the interest of the participants. It can also generate
expectations regarding the content to which it refers. There was a video on
the W2 homepage, which had a link next to it that said ‘accessible version of
the video’. None of the participants played the video neither visited the link of
the accessible version of the video but later on the interview four participants
mentioned the detail of the accessible video: “I wanted to look at it because
it said ‘accessible video’ . . . I forgot to check it out but I wanted to look at it,
huh? For me when they put accessible makes me want to try it, right? Then,
sometimes you say ‘to hell with this accessibility!’ Isn’t that so? But anyhow, it
denotes a level of interest; the designer of the Web may have done it thinking
about something or someone . . . Well, that is appreciated” (P10-W2).
In addition, as previously mentioned, W1 website was in this case the ac-
cessible version of the website. So, it contained a link with the ‘normal version’
text. As soon as P09 realised that she was navigating on the accessible version
of the website she said: “Wow! . . . finding an adapted version, even if we say it
does not influence our mood, we are already applauding! ” (P09-W1). Neverthe-
less, she did not experienced W1 to be accessible, and after the navigation she
said: “As I was browsing I was feeling as if I lacked resources, because I figured
that if the version was accessible, it’d be for some reason, and if I wasn’t find-
ing information, I was thinking: others [referring to participants] would have
found it, because it’s accessible, and I look silly because I’m not able to find the
information” (P09-W1). In addition, she reported that it is frustrating when
something is claimed to be accessible apparently, but finally it turns out that is
not: “that bothers me very much .. . it’s bothering when a company claims to be
as such . . . then you get disappointed . . . it feels like being cheated ” (P09-W1).
Beyond the specific case of W1, we observed different positions and opinions
with regard to the accessible or the normal version of a website. Some expert
participants said that they usually access the normal version of the websites,
12
even when accessing it for the first time. “I always try the other [referring to the
“normal” version], but because I want to and I am weird” (P02). In contrast,
most of the other participants (non-experts) stated that they usually visit the
accessible one if it was available, mainly because they expected to be the most
convenient or easiest way. “of course I will always go to the accessible one, I
always hit the button to make it more accessible for me . . . I don’t know in what
way ‘accessibilizes’ it for me, because I do not know much about computers”
(P05).
P07 reported that with one particular online newspaper he used to access
the text only version, until he realised that it was less complete than the orig-
inal one. This stopped him from visiting alternative versions of websites and
hitherto he sticks to original versions. Another participant said that he usually
visits the accessible version of the website, especially if he is browsing in an
operative mode, rather than in a more exploratory mode: “yes I go there [to
the alternative version], especially if you’re in a working attitude, I mean that
if you’re going to do something in particular ” (P10).
We observed two different positions on alternative website versions. Some
participants thought that it is not a good approach, and would prefer the website
to be accessible without the need of having an alternate version: “I always try
the other, and I do not find that page [the accessible version]accessible . . . I
don’t want to be separated .. . aren’t we going towards the integration? ” (P02).
In contrast, others feel good about having an accessible version, as long as it is
conveniently handy. “I don’t go looking for it . . . if I knew that all the websites
have an accessible feature, I assure you that is the first place I would go .. . ”
(P08).
4.2.2. Functionality-Related Expectations
In addition to the content, expectations are also related to the functionality
of the website. In this context functionality refers to processes and achievable
outcomes.
Expected functionality not present (10). This happened when a website did not
provide participants with the expected functionality. For example, in W4 a
participant expected that by activating a button he could make a booking: “I
wanted to press the button and the reserve by email would show up” (P05-W4).
Functionality does not work as expected (9). This refer to situations where web-
site elements do not work as expected by participants. For instance, links that
unexpectedly opened a new window or a new application misled the partici-
pants: “I don’t know where it has taken me to, to a very strange place, and then
you restored it. Well, that could be also as a sign of something that .. . I cannot
tell because I don’t know where it took me . . . but I was thinking ‘well, let’s see if
it takes me to a section for bookings’, that’s why I entered, but it didn’t happen”
(P10-W3).
13
4.3. Preconceptions (9)
Participants’ preconceived ideas regarding the restaurant represented by the
website (e.g. branding issues) also affected expectations on the website which
can later impact on their perception of the accessibility of the website. Some
participants were bothered —and even swore— by the experienced lack of ac-
cessibility of the W3 website. The anger was fuelled by the fact that W3 is an
internationally well-known restaurant: “If it’s so luxurious, they should bother
to put it right but they don’t care, because as they have lots of money, they would
say ‘people who know me they know where I am’ . . . Restaurants will say: ‘ahh
I’m already rich, I don’t care if they [referring to blind customers] don’t come”
(P06-W3).
Surprisingly, two of the restaurants of the websites (W1 and W3) used in
our study, had previously carried out initiatives to promote accessibility and
get closer to people with visual impairments: the accessible pintxo7initiative,
and offering the menu in Braille, respectively. In the case of the W1 restaurant,
the pintxo was specially created for blind people with the aim that they could
enjoy this type of ‘social snack’, which is very traditional in the area where
participants lived. The objective of the initiative was to raise awareness on the
difficulties that blind people have to face in their daily routine. Our findings
suggest that being aware of these initiatives carried out by restaurants in order
to promote accessibility, can influence expectations regarding the accessibility
of the website. “Surprised, pleasantly surprised, I did not expect .. . I knew that
the restaurant had done some pro-accessibility things, not on the Web but in
general. And that they have been doing some things with the ONCE and such,
but I did not think . . . yes the accessible pintxo, silly things in my opinion but
. . .well, maybe that served them to realise that an accessible site is needed” (P02-
W1). P09 knew that two restaurants had their menus in Braille. Consequently
she assumed that their respective websites would be accessible: “maybe you’re
not going to eat at the Arzak or Akelarre restaurant [two well-known upmarket
local restaurants, the latter is the restaurant of W3], but I guess their website is
accessible because they have the menu in Braille . . . even if it’s never updated! ”
(P09).
4.4. Evoked Memories (6)
The participants’ memories from their past experiences emerged while carry-
ing out their tasks. These memories were triggered by the recall of the experience
of having been at that restaurant, or by content of the website that reminded
participants of an emotional bond to someone or something: “I was happy, be-
cause apart from having been at Mugaritz [the restaurant corresponding to W1],
Mugaritz is located in an area to which I have much affection” (P05-W1). P09
remembered that time when she went to the W4 restaurant with her father.
She remembered fondly as an endearing and satisfying experience and expected
7Apintxo is the Basque equivalent of a Spanish tapa: a miniature dish you can eat while
standing up by the bar.
14
the Web experience corresponded to the real one: “I felt happy because I like
playing with the pages, and because I was remembering when I had been in [the
restaurant of W4] with my father . . . I’ve been once or twice. In both cases I had
dinner with very few people in the dinning room .. . A little disappointed for that
reason, because for such a restaurant of quality, and especially for the intimate
and satisfying eating experience I’ve had, nevertheless it’s like the website did
not match up to my experience ” (P09).
P10 commented that he recalled that his parents used to go to have dinner at
the restaurant of W4 on the festivities of Donostia-San Sebasti´an: “I also have
remembered that my father and my mother used to go there on the festivities.
They used to come to Donostia and then go to this restaurant to have dinner”
(P10-W4). Checking the wine list on W2, P10 noticed that there was a selection
of wines from the Canary Islands, which caught his attention. He said that he
wanted to see if there was a special wine called Malvasia: “I have seen the types
of wines it reminded my of the Canary Islands, because there is a typical wine
that is the Malvasia . . . I entered Canary Islands link ” (P10-W2).
4.5. Uncertainty
4.5.1. Confusion Generated by the Website (11)
We observed that participants’ non-met expectations are often related to
certain level of uncertainty and confusion. It is not only about expectations
that have not been met, but the associated feeling of uncertainty about why
they were not fulfilled. For instance, most participants expected to find a menu
on the websites. Even if W1 contained information about gastronomic offer, it
did not contain a menu as such, however participants reacted differently: one
of the expert participants was aware that the page did not contain it. “It has
one fault, which is the menu, I want a menu on the page. Yes, it tells me a
few starters, a few seconds and such, and that’s fine, but the extended menu”
(P02-W1). In contrast, some non-expert participants did not find the expected
menu and were not sure if it was because of ‘their fault’ or due to issues with
the website. “the thing is that I have not been able to find, the main goal I
have not achieved, eh? then, probably other blind colleagues would have found
it, because I’m sure that the average of the technical resources is higher than
mine” (P10-W1).
When participants were confused about whether it was because of something
related to the website, the screen reader, or due to they had done something
wrong. This uncertainty about not knowing what happened and why can affect
the perceived Web accessibility and UX. In the W3, P05 accessed the Youtube
link thinking that he would probably find information on the menu: “The thing
is that I don’t know if there was something else . . . clearly I’m not sure that
I could reach everywhere .. . that’s why I got into Youtube, because there they
might tell you ” (P05-W3). “I don’t know, I got lost with the buttons . . . the thing
is that I don’t know why buttons exist ... I don’t have the concepts clear about
the difference between a link and a button” (P10-W4). The number of com-
ments about how confusing Web content was completely unbalanced between
15
the most accessible websites (W1, W2 get 13 comments) and the least acces-
sible ones (W3, W4), which get 39 comments. This supports the relationship
between accessibility problems and confusing situations found elsewhere (Vigo
and Harper, 2013).
4.5.2. Confusion Related to Confidence and Expertise (7)
The uncertainty of participants was closely related to their perception of
their own skills and their confidence. Intermediate and advanced participants
exhibited a lack of confidence and blamed on themselves when encountering
problems : “I see that I am very clumsy. If I had a better opinion about myself,
I’ll probably be more critical with the Web accessibility” (P10-W1). Those par-
ticipants at both ends of the spectrum (i.e. beginners and experts) attributed the
cause of problems to the website or the screen reader rather than to themselves.
4.6. Experienced Web Accessibility
After completing the tasks, we collected accessibility ratings as the websites
and enquired participants about the problems they encountered. Table 2 shows
the accessibility ratings given by participants on a 7-item Likert-scale where 1
indicated ‘very inaccessible’ and 7 ‘very accessible’. The median values suggest
that participants perceived W1, W2 and W4 to be similarly accessible. However,
a smaller standard deviation in W2 and W3 indicates a broader consensus among
participants than in W1 and W4. In the case of the latter two, although opinions
varied, most participants considered them to be accessible.
Table 2: Participants’ experienced Web accessibility ratings (1: very inaccessible–7: very
accessible)
website AA compliance median mode SD
W1 73% 6 7 1.95
W2 69% 6 6 1.42
W3 52% 2 1 1.62
W4 36% 6 6 1.95
Website 1. P06 withdrew from the website as JAWS was getting blocked: “some-
times JAWS gets stuck and doesn’t respond ” (P06). P09 and P10 were not very
confident talking about accessibility barriers: they both agreed on the large
amount of visual information that W1 contained. Even if images had an alter-
native text, the fact that the text was not meaningful for them was perceived as
a barrier: “I may have preferred a description of the person, some biographical
data, or a phrase, or how the person entitles his dishes .. . [but] when I hear
‘the portrait of our people’ [as alternative text] .. . this leaves me overwhelmed”
(P10).
Participants also complained about not being able to find the menu even
if this website did not contain a menu as such: “I haven’t found any practical
information, not even the menu . . . you cannot access the content. Even if the
sections are marked up, you click on those links and once you are in a section
there is nothing there . . . the most frustrating thing is going to a restaurant and
not knowing what to eat” (P09).
16
Website 2. Only P01 rated it as non-accessible, which was mainly due to net-
work connection issues. She said the page seemed to be accessible, but because
of problem when loading Web pages she could not give it a better score: “I
think it’s accessible, but I couldn’t go beyond the links” (P01).
Website 3. All participants except P11, who said the Web page was too simple
and lacked major difficulties, perceived W3 to be non-accessible. The fact that
P11 did not navigate beyond the homepage may explain this outlying percep-
tion. Most participants mentioned that the navigation menu of the website was
completely inaccessible because each menu item was a graphic link with the
same alternative text (i.e. image). Consequently many of them did not even try
to click on those links and their navigation was exclusive to the homepage.
The mental map of some participants was mismatched to the information
architecture of W3. P03, who was able to distinguish between the structure
from the content of the website, was ambivalent about the accessibility/non-
accessibility of the website and did not lean towards any end of the scale. Ac-
cording to P03, W3 had a very simple structure, which made navigation easy.
However, the scarcity of content was considered a problem: “when it comes to
move, you can move without any problem, the problem is related to the infor-
mation in those links” (P03).
Website 4. Most participants thought W4 was quite accessible while two par-
ticipants (P01 and P07) were ambivalent about it. We observed the majority
of participants were not aware about the presence of non-accessible dynamic
content and therefore did not perceive the accessibility barriers. Nevertheless
they were suspicious about something going on: “unless I missed something to
browse .. . there wasn’t anything or I was not able to seek . . . I wonder if there
was a longer menu” (P09). In line with this P10 was not sure whether they
were missing some content. Both P09 and P10 were disappointed and regretted
their ratings when they were told that there was actually more content: “this
devalues the accessibility score I gave .. . it’s like there is a walk adapted for
people with disabilities (i.e. like a safe path) and there is a hole in the middle,
and I walk with my guide dog and step near the edge of the hole. I do not fall
over but it could have happened” (P10).
In some cases that participants experienced a website as accessible we found
comments about possibly visiting again the website, and in some cases partici-
pants commented their willingness to go to the restaurant: “I would love to go
to the restaurant ” (P05-W1). In contrast, in other cases when a website was not
perceived to be accessible, some participants expressed their disappointment in
terms of not having the intention to return to the website, neither to visit the
restaurant: “if their page is not accessible, I’m not interested in the restaurant,
I’d rather go to the neighbouring restaurant where I’m treated better” (P06-W3).
17
5. Discussion
Results corroborate that blind users’ experienced Web accessibility does not
always correspond to compliance with accessibility guidelines. In order to better
understand the experienced accessibility, we identify how experiential aspects
may influence on the users’ judgement about the accessibility on websites.
5.1. Prejudices, Past Experiences and Memories Determine Expectations
Before visiting a website most participants already had preconceived ideas
and expectations regarding the website, which were mainly about on previous
experiences either because of having navigated on similar websites, or due to
past experiences at restaurants. For instance, three participants who had ex-
perienced uncomfortable situations when ordering at restaurants expected the
website to provide them with practical information to avoid such problematic
situations. Taking measures to avoid awkward situations is an strategy doc-
umented elsewhere (Ray and Ryder, 2003), which indicates that the Web is
instrumental for people with disabilities to plan their journeys well in advance.
We also found that having had an endearing and satisfying experience at
a particular restaurant in the past could also influence the expectations about
the website of the restaurant. Nevertheless, the actual Web experience can also
determine real life experiences: some participants showed either willingness to
revisit the website and go to the restaurant because their good experience on the
restaurant website. On the other hand, they showed no interest on restaurants
and their websites if the experience was poor. This provides additional evidence
to support a business case for accessibility and user experience.
Preconceptions did not only create expectations, but affected how accessi-
bility was experienced. Participants were more demanding with those restau-
rants that were better positioned to have an accessible website due to their
international prestige, and alleged resources and affluence. In contrast, evoked
memories have an impact on the experienced Web accessibility as participants
rated the accessibility of those websites that evoked fond memories higher. We
suspect that traumatic memories will negatively influence too.
5.2. Unmet Expectations
While trying to accomplish their tasks participants’ expectations are not
always met. In general, this happens when there is discrepancy between what
was expected and what is encountered. The underlying reasons of the mismatch
are due to (i) the obstacles within the website including accessibility barriers,
usability flaws and confusing information architecture; (ii) problems with the
screen reader; (iii) their preferences, previous knowledge, skills and expertise.
Unmet expectations are not always explicitly reported as such: when par-
ticipants come across with unexpected content or functionalities this is often
reported as a problem of the website when actually there is a mismatch be-
tween the mental model of users and the model of the website. The existence
of accessibility barriers made some content non reachable and therefore non-
accessible in our study. Not being able to find such content is not blamed on
18
accessibility barriers which are not directly perceived, but users think that the
website does not contain such content or that they were not able to find it.
Specifically, intermediate and advanced users take on the responsibility of the
failure, while those on the extremes (i.e. experts and beginners) blame external
factors such as the website and the screen reader. In the case of beginners they
may know fewer things (e.g. basic screen reader commands for Web navigation)
but well learned, while experts may well know the causes of problems. When
it comes to intermediate and advanced participants, they may be aware of the
wide range of possibilities offered by the Web but at the same time they know
the limits of their own internet skills, which lessens their confidence on their
judgements.
5.3. The Role of Uncertainty and Confidence
Non-accessible sites create more uncertain situations than the accessible
ones, which suggests there is a relationship between the the lack of accessi-
bility and uncertainty. This finding reinforces the fact that uncertain situations
trigger coping strategies (Vigo and Harper, 2013) and the fact that uncertainty
is caused by unmet expectations and accessibility barriers.
While uncertain situations can be triggered when websites are visited for the
first time, we observed that this uncertainty remained after finishing the task.
This lasting effect has an impact beyond the perceived accessibility and affects
the user experience. Therefore reducing the uncertainty would probably lead
to a better user experience. Designing with familiarity and learnability princi-
ples helps in removing this uncertainty while may help boosting the experience
of users (Vigo and Harper, 2014). Understandably, supporting and training
users in order to acquire skills will increase their confidence to cope better with
uncertainty.
5.4. ‘Accessible’ Versions of Websites
Most non-expert participants stated that they usually visit the accessible
version of a website if this is available, whereas expert participants reported
that they usually access the main version, even on the first time. Some par-
ticipants do not access accessible versions of websites they visit periodically
because alternate versions are usually inconsistent, lack information and are
not updated as regularly as the main version. Some rejected accessible versions
on the grounds of discrimination. Anecdotally, one participant reported that the
choice depends on the navigation modality, whether navigation is exploratory or
directed. On directed navigation —i.e. the user has a specific goal in mind— the
accessible version is more useful. Conversely, if the navigation is of exploratory
nature the main version is preferred.
‘Accessible’ versions do not only arise the curiosity of participants, but cre-
ates high expectations about their accessibility. Those webmasters managing
different website versions must be careful because if these high expectations
are not met, it generates sheer frustration. The above implies that accessible
19
versions of websites should be carefully provided: they should be not only acces-
sible, but easily findable and noticeable. For example they should be included in
the results of search engines somehow along with the main website. In addition,
if an accessible version exists, the link to access to it should always be easy to
find.
These findings have implications on user testing practices in that the isolated
aspects influence the assessment and ratings of users in a considerable way. This
phenomena calls for identifying these situations and calibrating assessments and
ratings on users tests.
We claim that all the above-mentioned aspects —i.e. past experiences, prej-
udices, evoked memories, unmet expectations and confidence— strongly affect
how users perceive and experience the accessibility of websites. Because of this,
we state that experienced accessibility is a highly subjective quality, which does
not necessarily correspond to compliance with accessibility standards. Because
guidelines are defined for standardisation purposes, their technical specification
has, understandably, a unified and integral view of accessibility that can diffi-
culty cope with the subjective aspects we have discovered.
5.5. Methodological Considerations
Restaurant websites were the object of this study. Participants were familiar
with the product (i.e. the restaurant) but not with their website —they all were
first time visitors. Following the previously mentioned classification for web-
sites (De Marsico and Levialdi, 2004), the findings are generalisable to those
websites that fall into the axes of commercial sites which target a general au-
dience and exhibit an informative-seductive communication style. For instance,
the websites of products/services such as local shops or charities that appeal to
emotions in order to engage the customers.
While the outcomes are applied to blind users, we hypothesise that the
results are generalisable to broader audiences. The role that prejudices, expec-
tations and memories play on the user experience may well be independent of
the users’ abilities. Thus, the outcomes related to these aspects are the ones
that are more directly transferable. The uncertain situations and the design
barriers may also impact on able-bodied users, especially on those who are less
skilled and knowledgeable about Web conventions. However, more evidence is
required in order to suggest the generalisability of these outcomes to broader
populations.
6. Conclusions
We identify the experiential aspects that influence blind users’ perception
and experience of accessibility barriers on the Web including
•prejudices;
•past experiences;
20
•memories;
•expectations;
•and confidence;
Experienced accessibility is not only an intrinsic quality of websites, but it is also
constructed from what individuals preconceive and expect from them. Conse-
quently, these factors introduce subjectivity to the results obtained in controlled
user testing protocols. The identification of the factors and calibration of results
in order to correct theses deviations is critical if we want to achieve reproducible,
comparable and reliable results.
7. Acknowledgements
We are grateful to Encarnacion Bri˜nas for her crucial help with recruitment,
and for providing facilities at the ONCE delegation; and to all participants,
for their willingness to take part and contribute to the study. A copy of the
websites and the results of their corresponding Web accessibility evaluations can
be found in http://sipt07.si.ehu.es/amaia/stimuli. A. Aizpurua held a
PhD scholarship from the Department of Education, Universities and Research
of the Basque Government when the study was conducted. The work has been
partly funded by the Department of Education, Universities and Research of
the Basque Government (grant IT395-10) and the Basque Advanced Informatics
Laboratory (BAILab), which belongs to the University of the Basque Country
UPV/EHU (grant UFI11/45).
References
Abascal, J., Arrue, M., Fajardo, I., Garay, N., and Tom´as, J. (2004). The use of
guidelines to automatically verify web accessibility. Universal Access in the
Information Society, 3(1):71–79.
Adams, A., Lunt, P., and Cairns, P. (2008). A qualititative approach to HCI
research. pages 138–157.
Aizpurua, A., Arrue, M., and Vigo, M. (2013). Uncovering the role of expecta-
tions on perceived web accessibility. In Proc. of, ASSETS, pages 74:1–74:2,
New York, NY, USA. ACM.
Bargas-Avila, J. A. and Hornbæk, K. (2011). Old wine in new bottles or novel
challenges: A critical analysis of empirical studies of user experience. In Proc.
of, CHI, pages 2689–2698, New York, NY, USA. ACM.
Brajnik, G. (2006). Web accessibility testing: When the method is the culprit.
In ICCHP, LNCS 4061, pages 156–163. Springer Verlag.
Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology.
Qualitative Research in Psychology, 3(2):77–101.
21
Caldwell, B., Cooper, M., Reid, L. G., and Vanderheiden, G.
(2008). Web content accessibility guidelines (WCAG) 2.0.
http://www.w3.org/TR/WCAG20/.
Chisholm, W., Vanderheiden, G., and Jacobs, I. (1999). Web con-
tent accessibility guidelines (WCAG) 1.0. http://www.w3.org/TR/WAI-
WEBCONTENT/.
De Marsico, M. and Levialdi, S. (2004). Evaluating web sites: Exploiting user’s
expectations. International Journal of Human-Computer Studies, 60(3):381–
416.
Gay, G. and Li, C. Q. (2010). Achecker: Open, interactive, customizable, web
accessibility checking. In Proc. of, W4A, pages 23:1–23:2, New York, NY,
USA. ACM.
ISO 9241-11 (1998). ISO 9241-11: Ergonomic requirements for office work with
visual display terminals (vdts) – part 11 guidance on usability.
Lazar, J., Dudley-Sponaugle, A., and Greenidge, K.-D. (2004). Improving
web accessibility: a study of webmaster perceptions. Computers in Human
Behavior, 20(2):269 – 288.
McCarthy, J. and Wright, P. (2004). Technology as Experience. The MIT Press.
Petrie, H., Hamilton, F., and King, N. (2004). Tension, what tension?: Website
accessibility and visual design. In Proc. of, W4A, pages 13–18, New York,
NY, USA. ACM.
Power, C., Freire, A., Petrie, H., and Swallow, D. (2012). Guidelines are only
half of the story: Accessibility problems encountered by blind users on the
web. In Proc. of, CHI, pages 433–442, New York, NY, USA. ACM.
Ray, N. M. and Ryder, M. E. (2003). “ebilities” tourism: an exploratory dis-
cussion of the travel needs and motivations of the mobility-disabled. Tourism
Management, 24(1):57 – 72.
Rømen, D. and Svanæs, D. (2012). Validating wcag versions 1.0 and 2.0 through
usability testing with disabled users. Universal Access in the Information
Society, 11(4):375–385.
Sears, A. and Hanson, V. (2011). Representing users in accessibility research.
In Proc. of, CHI, pages 2235–2238, New York, NY, USA. ACM.
van der Geest, T., van der Meij, H., and van Puffelen, C. (2014). Self-assessed
and actual internet skills of people with visual impairments. Universal Access
in the Information Society, 13(2):161–174.
Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., and Abascal, J. (2007). Quan-
titative metrics for measuring web accessibility. In Proc. of, W4A ’07, pages
99–107, New York, NY, USA. ACM.
22
Vigo, M. and Harper, S. (2013). Coping tactics employed by visually dis-
abled users on the web. International Journal of Human-Computer Studies,
71(11):1013 – 1025.
Vigo, M. and Harper, S. (2014). A snapshot of the first encounters of visually
disabled users with the web. Computers in Human Behavior, 34(0):203 – 212.
23
