Exploring the relationship between web accessibility and user experience

Abstract

Understanding the interplay between the user experience (UX) and Web accessibility is key to design Web sites that, beyond access, could provide a better UX for people with disabilities. In this paper we examine the relationship between UX attributes and Web accessibility. We measured accessibility in two ways: the perceived accessibility as reported by participants and accessibility in terms of conformance to guidelines. Findings uncover that perceived Web accessibility is significantly correlated with 27 of the 35 UX attributes analysed, suggesting these two qualities are closely related. The relationship between UX and conformance to WCAG 2.0 is more elusive: we only found significant correlations between the hedonic attributes original, innovative and exciting.

Full text

Exploring the Relationship between Web Accessibility
and User Experience
Amaia Aizpuruaa,b,∗
, Simon Harperb, Markel Vigob
aFaculty of Informatics, University of the Basque Country, UPV/EHU,
Donostia-San Sebasti´an, Spain
bSchool of Computer Science, University of Manchester,
Manchester, United Kingdom
Abstract
Understanding the interplay between the user experience (UX) and Web acces-
sibility is key to design Web sites that, beyond access, could provide a better UX
for people with disabilities. In this paper we examine the relationship between
UX attributes and Web accessibility. We measured accessibility in two ways:
the perceived accessibility as reported by participants and accessibility in terms
of conformance to guidelines. Findings uncover that perceived Web accessibility
is significantly correlated with 27 of the 35 UX attributes analysed, suggesting
these two qualities are closely related. The relationship between UX and con-
formance to WCAG 2.0 is more elusive: we only found significant correlations
between the hedonic attributes original,innovative and exciting.
Keywords: Blind users, Screen readers, Web, Web accessibility, User
Experience
1. Introduction
The World Wide Web has an incredible potential to make our lives better
due to the wide range of services offered through it. The Web can be specially
helpful for people with disabilities, as barriers to communication and interaction
that many people face in the physical world are removed. While the Web was5
∗Corresponding author
Email address: amaia.aizpurua@manchester.ac.uk (Amaia Aizpurua )
Preprint submitted to International Journal of Human Computer Studies April 12, 2016

designed to be universally accessible, in practice this does not always happen
(Lopes et al., 2010) mainly because Web sites are often designed without con-
sidering human diversity. This leads to poorly designed Web sites which can
potentially exclude significant segments of the population. Since the Web is a
mainly visual environment, navigating the Web is particularly challenging for10
blind users. Although assistive technology such as screen readers have been
an incredible breakthrough, blind users still face a wide range of difficulties
on the Web. In fact, blind users face not only more challenges than sighted
users (Bigham et al., 2007), but are also disadvantaged when compared to other
groups of users with disabilities (Petrie et al., 2004).15
The World Wide Web Consortium (W3C) through the Web Accessibility
Initiative (WAI) published the Web Content Accessibility Guidelines (WCAG)
(Chisholm et al., 1999; Caldwell et al., 2008) to promote the design of accessible
Web content. While guidelines are an invaluable starting point, prior empirical
research (Power et al., 2012) indicates that WCAG 2.0 only cover around half20
of the problems that blind users encounter on the Web. This implies that a
Web site may have an adequate level of accessibility in terms of conformance to
guidelines, but still not be sufficient for users. Our experience, which is informed
by a series of studies with blind participants, corroborates that a Web site with
a significant number of WCAG 2.0 success criteria violations can be perceived25
to be accessible; on the contrary, a Web site which is compliant to guidelines
may not be always perceived to be accessible (Aizpurua et al., 2013).
Research on the behavioural aspects of blind users on the Web has been
mainly focused on the analysis of performance in terms of efficiency, errors com-
mitted and satisfaction (Leuthold et al., 2008), and on examining the strategies30
employed to overcome the barriers they encounter and the situations that trig-
ger the use of coping tactics (Vigo & Harper, 2013). These works have provided
a valuable knowledge about how blind users behave and navigate on the Web
although they say little about the ‘intangible’ aspects of the experience. In other
words, since behaviour is accompanied by subjective experience, in addition to35
assessing objective qualities such as performance-related aspects of the interac-
2

tion, the interaction with Web sites should be explored in a more holistic way. In
this respect user experience (UX) provides a framework to understand how users
may perceive an interactive artefact from diverse facets including aesthetics, af-
fect or trust (Law et al., 2009). We claim that having a better understanding40
of blind users’ subjective experience on the Web cannot be disassociated from
an analysis of how this experience affects the perceived accessibility.
To shed some light on this association we examined the relationship be-
tween the UX and Web accessibility by comparing the subjective experience
reported by users, with the perceived Web accessibility and with conformance45
to accessibility guidelines. User experience scores were collected by means of
questionnaires and semi-structured interviews based on the UX model proposed
by Hassenzahl (2005). In Section 4.1 we examined the relationship between
UX and perceived Web accessibility after-use and found that perceived Web
accessibility (PWA) is significantly correlated with most of the UX attributes,50
suggesting the close relationship between these qualities. Secondly, we inves-
tigated how UX attributes are related to different Web Accessibility Indicators
(AIs), which were measured using different accessibility evaluation methods.
The outcomes of this analysis reveal in Section 4.2 that compliance to WCAG
2.0 guidelines is significantly correlated to three UX attributes that belong to55
the hedonic quality: original,innovative and exciting. In Section 4.3 we anal-
ysed the interviews with participants in order to provide possible explanations
for these relationships. Finally, results and implications are discussed in Section
5. This work is not only novel due to the number of covered UX attributes, but
also because we study the experience of blind users with regard to a more holis-60
tic view of Web accessibility, as the subjective perception of participants and a
more normative assessment of accessibility have been considered.
3

2. Background
2.1. The Importance of the Experience
Usability is a clearly defined concept: the ISO 9241-11 (ISO, 1998) defines65
usability as “the extent to which a product can be used by specified users to
achieve specified goals with effectiveness, efficiency and satisfaction in a specified
context of use”. However, Web sites are much more than interactive artefacts
for accomplishing specific tasks. Individuals find the Web as a mean, not only
to achieve informational goals, but also for activities related to communication,70
leisure, social networking or contributing to building the Web (Lindley et al.,
2012). Users value interactive artefacts on the basis of how well they satisfy
their needs in a particular situation, beyond the objective features that derive
from design choices including content, style or functionalities. Therefore, the
success of a Web site may not only depend on its actual characteristics, but also75
on how well these are perceived by users (Hassenzahl, 2005). This highlights
the importance of considering the subjective aspects of the interaction in order
to understand the actual experience of users. Because the HCI community has
acknowledged the importance of these non-instrumental aspects, several works
have focused on defining and setting the scope of UX: according to a survey,80
UX is considered dynamic, context-dependent and subjective and therefore,
difficult to agree upon (Law et al., 2009). Similarly, Bargas-Avila & Hornbæk
(2011) found a lack of consistency in the methods employed to evaluate different
UX attributes. This is a symptom of not having a common framework that
allows researchers and practitioners to understand the UX attributes and the85
relationships among them. So far, only a few models have been proposed, for
instance, the most comprehensive ones are the UX model (Hassenzahl, 2005)
and the CUE model (Th¨uring & Mahlke, 2007). Both models share the same
foundations, as both include instrumental and non-instrumental qualities, the
emotional reactions of users and their appraisal of interactive artefacts.90
4

2.2. Web Accessibility, Usability and UX
There is little agreement when it comes to defining web accessibility, which
causes some tensions between the community of users, researchers and acces-
sibility advocates (Yesilada et al., 2012). Moreover, while it cannot be denied
that accessibility and usability are two qualities that interact with each other95
it has always been difficult to define the scope and extent of this relationship.
In fact, if accessibility and usability are not properly integrated, Web sites can
turn out to be either accessible but barely usable, or usable but barely acces-
sible (Leporini & Patern`o, 2008). In a study run by Petrie & Kheir (2007) it
was found that sighted and blind users have in common 14% of the problems100
they encounter, suggesting this figure as the overlap between accessibility and
usability.
There are few reliable research works about the relationship of UX and
Web accessibility. One exception is a study about the relationship between
aesthetic features and accessibility (Mbipom & Harper, 2011). The authors105
found that there was a relationship between the visually clean dimension of
aesthetics and conformance to Web accessibility. It must be noted that even if
Web accessibility evaluations were carried out following the barrier walkthrough
method (Brajnik, 2006), the aesthetic judgements were made by sighted users.
This suggests we require a more comprehensive understanding of the relationship110
between Web accessibility and UX as experienced and reported by blind users.
In addition, the experiential and subjective aspects of Web accessibility remain
largely unexplored. In previous work (Aizpurua et al., 2015) it was uncovered
that experiential aspects such as prejudices, evoked memories, expectations can
influence on how blind users experience the accessibility of a Web site.115
2.3. The Experience of Blind Users on the Web
In order to understand the experience of blind users on the Web, previous
work has focused on identifying the problems they encounter (Theofanos & Re-
dish, 2003; Murphy et al., 2008; Power et al., 2012), analysing their performance
(Leuthold et al., 2008), examining their navigation behaviour and the coping120
5

strategies they use (Vigo & Harper, 2013). Works about the subjective aspects
of the interaction of blind users with the Web are scarce: Lazar et al. (2006)
conducted a study that examined the frustrating experiences and mood changes
of 100 participants with visual disabilities when browsing the Web. One of the
main findings showed that frustration causes individuals’ mood to deteriorate.125
More specifically, the factors that had the strongest negative impact on mood
were those related to the ability to complete the work.
3. Method
3.1. Participants
Eleven participants (4 females and 7 males) took part in the study, as shown130
in Table 1. The median age of participants was 43 years, with a range of 21–64
years. Expertise varied among participants: 2 were experts, 2 beginners, 4 inter-
mediates and 3 advanced. In an interview which took place before the navigation
tasks we asked participants to rate their expertise on the Web on a four-item
scale: expert, advanced, intermediate and beginner. Due to the observed lack135
of reliability of self-reported values, Web expertise was also assessed based on
external observation of the navigation skills shown by participants. The first au-
thor looked at the strategies employed and the confidence of participants when
carrying out the proposed tasks. These assessments were corroborated by the
facilitator in charge of the computer training facility at the National Organi-140
sation of Spanish Blind People (ONCE), who had trained the participants. In
line with the findings of van der Geest et al. (2014), who found that self-rated
competence of visually impaired users is not always related to their actual per-
formance, Table 1 shows that self-rated and observed skills do not necessarily
match.145
3.2. Apparatus
All participants were legally blind and utilised screen readers to navigate on
the Web: ten participants were Jaws users (version 10, except for P01 and P03
6

Table 1: Demographic data and characterisation of Web expertise
part. id gender age Web familiarity frequency of Web use expertise
self-reported observed
P01 f 29 >7 years daily advanced expert
P02 f 29 >7 years daily advanced expert
P03 m 39 >7 years daily advanced advanced
P04 m 54 4–6 years daily intermediate intermediate
P05 m 43 1–3 years weekly beginner intermediate
P06 m 21 1–3 years weekly beginner beginner
P07 m 64 >7 years daily intermediate advanced
P08 m 58 >7 years daily intermediate intermediate
P09 f 54 4–6 years daily intermediate advanced
P10 m 64 4–6 years daily beginner intermediate
P11 f 42 >7 years weekly intermediate beginner
who used version 12) on Internet Explorer and Windows XP and Windows 7
respectively, while just one participant (P02) was a VoiceOver user on Safari150
over MacOS. The first three participants were observed in the research facility
of the HCI laboratory at the School of Computer Science of the University of the
Basque Country, where they brought their own laptop. Remaining sessions took
place in a room at the ONCE delegation in Donostia-San Sebasti´an, where these
eight participants used the same laptop and keyboard, which were provided by155
the ONCE.
3.3. Stimuli Selection
We recruited local participants and selected Web sites of restaurants that
were popular in the area where participants lived in order to let the subjective
dimensions emerge. We focused on one type of Web site for two main reasons: 1)160
to establish analogous tasks across different stimuli; and 2) to minimise potential
confounding factors resulting from different types of Web sites.
In order to select the final set of Web sites we first listed 25 Web sites of
local restaurants of different styles. We then analysed their homepages, using
four automated Web accessibility evaluation tools: AChecker (Gay & Li, 2010),165
7

EvalAccess (Abascal et al., 2004), TAW 1and WAVE2. Using the WAQM metric
(Vigo et al., 2007) we evaluated again the homepage and two more Web pages.
Based on the results we classified the Web sites into 2 groups: highly accessible
and poorly accessible sites. Within each group we separated Web sites into two
other groups considering the style of the restaurant: traditional and innovative.170
Then we performed manual accessibility evaluations: we applied the Barrier
Walkthrough (BW) inspection method (Brajnik, 2006) in three Web pages of
each Web site. Based on the results, we finally selected the two most and least
accessible Web sites for each type of restaurant.
We evaluated the final four Web sites against the AA conformance level of175
WCAG 2.03. The selected Web sites contained different features and problems:
in general, W1 and W2 satisfied more AA level success criteria (SC) than W3
and W4. The homepages of W1, W2, W3 and W4 satisfied respectively 73%,
69%, 52% and 36% of the SC for the AA level of WCAG 2.0. The most severe
accessibility problem of W3 was that the seven links that conform the navigation180
menu, all of them have the same text which is ‘image’. In the case of W4, the
main accessibility problem is about Flash content which is not accessible using
a screen reader.
Regarding the above-mentioned selection criteria with respect to branding,
W1 and W3 represent internationally well-known restaurants with an innovative185
character and a culinary style based on research and creativity. In contrast, W2
and W4 Web sites correspond to restaurants that are locally popular and their
style is based on traditional Basque cuisine. The visual design of the Web sites
is in line with the style of the restaurant. The Web sites of the internationally
well-known restaurants (W1 and W3) have more stylised visual layouts, as they190
have had their typographies designed and make use of high-quality close-up
pictures. In contrast, the Web sites of the traditional-style restaurants (W2
1http://www.tawdis.net
2http://wave.webaim.org/
3Both VoiceOver and Jaws screen readers were used on BW and WCAG 2.0 evaluation.
8

(a) W1: high accessibility and innovative (b) W2: high accessibility and traditional
(c) W3: low accessibility and innovative (d) W4: low accessibility and traditional
Figure 1: Screenshots of Web sites.
and W4) have a more basic and less elaborated visual aesthetic designs (see a
snapshot of their homepages in Figure 1).
3.4. Procedure195
Each session was conducted with one participant at a time. Once the par-
ticipants were informed about the objectives of the study and the procedure of
the session they signed a consent form. In order to reduce bias in the obtained
answers we told participants that we had no conflict of interests with the Web
sites, and that we had only selected those Web sites for the purpose of the study.200
We let the participants know that we were interested in their personal opinions
in order to let them play the role of testers rather than tested subjects. We also
made it clear that there were no right or wrong answers. Thus, they would feel
free to respond as honestly as possible.
9

Then, we asked each participant questions about demographics, and their205
Web expertise, including their familiarity with the Web and the frequency of
access to the Web. After that, each participant was interviewed about his pre-
vious experiences and expectations regarding restaurant Web sites. Once the
interview had finished, the participant could start to navigate the first Web
site. Following a within-subject design, each participant was asked to complete210
the same three consecutive tasks within each Web site (more in Section 3.5).
Repeating tasks on all Web sites did not introduced a potential learning effect
bias as each Web site structured its content in a different manner. In order to
minimise order effects, Web site navigation order was counterbalanced. Once
they finished the tasks or withdrew from them, participants were asked to rate215
the items of the questionnaires (more in Section 3.6). Then they were inter-
viewed about their browsing experience. Each session, including the navigation
on Web sites and interviews, was video and audio recorded to enable subsequent
analysis.
3.5. Tasks220
The three tasks were: 1) freely navigate on the Web site in order to become
familiar with it; 2) find information about the gastronomic offer; and 3) find the
means offered by the Web site to make a booking. The idea was to let the users
explore the Web site through real tasks, which would induce a more naturalistic
behaviour. Even if tasks were set in advance, they were not very specific and225
were kept open, as participants were not given explicit clues or directions to
follow in order to find the information. This allowed the participants to explore
and browse each Web site with ample opportunities. Participants were told
that the time estimated for each task was between 5-10 minutes. However we
insisted on their freedom to spend more time or withdraw from navigating if230
they wanted. For this reason completion times were irrelevant and not useful
for the scope of our study.
10

3.6. Data Collection
We used existing instruments to capture the UX of participants in order
to focus on their perception of the Web sites and the emotional reactions they235
had. The instruments we selected after reviewing the literature were: Attracdiff
2 (Hassenzahl, 2008) and the emotion word prompt list (EWPL) by Petrie &
Precious (2010) both in Table 2. Attracdiff 2 was used for collecting the in-
sights from participants about the Web sites. This questionnaire consists of a
set of 23 word pairs reflecting opposite adjectives that can be rated on a 7-point240
scale to assess perceptions of users about an interactive artefact on pragmatic
quality (PQ) and hedonic quality (HQ) attributes also including judgements on
beauty and goodness. PQ refers to the usability of the artefact, and focuses
on task-related aspects. In contrast, HQ refers to more subjective quality at-
tributes in terms of stimulation, identity communication (identification) and245
valued memories. The emotions that emerged during the interaction with the
Web sites were obtained by means of the emotion word prompt list (EWPL).
The EWPL consists of 11 emotional words: annoyed,bored,confident,confused,
disappointed,frustrated,happy,interested,hopeful,pleased and unsure that can
be rated through 7-point Likert items where 1 means low intensity and 7 high250
intensity. We translated both questionnaires into Spanish.
Information about the Web accessibility perceptions of participants was ob-
tained by asking participants to rate the accessibility of each Web site in a
7-point Likert-type question, from 1 (very inaccessible) to 7 (very accessible).
We also used semi-structured interviews to gather deeper insights on the aspects255
collected by the questionnaires. Some of the prompts we gave them aimed at
knowing more about the moment and the reasons for their emotional reactions,
the problems they encountered while navigating, the positive and negative as-
pects of the Web sites, etc.
The accessibility indicators (AIs) were obtained applying the different evalu-260
ation methods described in Section 3.3. As a result, we computed 37 accessibil-
ity indicators that derived from four main sources: the TAW online automated
11

evaluation tool4(tool), the metrics from the WAQM software (m) and those
obtained through inspection methods: conformance to WCAG 2.0 (sc) and the
Barrier Walkthrough method (bw ). These four main AIs were broken down into265
more specific indicators resulting in a list of 37 AIs: the automatically detected
problems (e), warnings that require manual verification (w); the four accessibil-
ity principles of the WCAG 2.0 guidelines: perceivable (p), operable (o), under-
standable (u) and robust (r); the conformance level (a,aa) and the number of
satisfied (sat) and not-satisfied (nsat) success criteria (sc). For instance, mrep-270
resents the average accessibility score computed by WAQM, tool w p refers to
the number of warnings reported by the tool for the pprinciple, tool e u corre-
sponds to the number of errors reported by the tool for the uprinciple, sc sat a
represents the number of satisfied sc for the A conformance level, sc nsat aa p
indicates the number of not satisfied sc for the AA conformance level for the p275
principle, and bw corresponds to the number of barriers found using the Barrier
Walkthrough method.
3.7. Data Analysis
We run analyses of statistical correlation to observe the relationships be-
tween: 1) UX attributes and perceived Web accessibility (PWA) as reported by280
participants; and 2) UX attributes and accessibility indicators (AIs) as gener-
ated by tools and inspections by experts. The statistical software used was R5.
We also analysed data from the interviews in order to better understand the
importance of the identified relationships and the reasons why they emerged.
4We chose the TAW online tool because it provides a straightforward way to discriminate
automatically reported violations and warnings that require human verification.
5http://www.r-project.org/
12

4. Results285
4.1. UX and Perceived Web Accessibility
Table 2 shows that most correlations between PWA and UX attributes (27
out of 35) are statistically significant when we compute Kendall’s Tau test6.
We found strong significant (τ > 0.5 and p < 0.001) correlations between PWA
and six attributes of the hedonic quality-identification (integrating, professional,290
valuable, inclusive, brings me closer to people, presentable), one attribute of the
hedonic quality-stimulation (creative), five attributes of the pragmatic quality
(simple, practical, direct, clear, manageable), goodness, appeal, annoyed, dis-
appointed, frustrated, happy, interested and pleased. We also found significant
and moderate correlations (0.4>τ>0.5 and p < 0.001) between PWA and one295
attribute of hedonic quality-identification (classy), three attributes of hedonic
quality-stimulation (original, exciting, new), beauty, bored and confused.
4.2. UX and Accessibility Indicators
We examined the relationship between the UX attributes rated by partici-
pants and the accessibility indicators corresponding to Web sites. Since Likert300
scales can actually be considered somewhere between an ordinal and a true
interval scale (Maxwell, 2006), it has been a subject of debate for years how
these scales should be appropriately analysed (Carifio & Perla, 2007; Norman,
2010). Due to the exploratory nature of our research, we decided to apply both
parametric and non-parametric tests including Pearson, Spearman and Kendall305
correlation tests. If we focus on large effect sizes (above 0.5) the Pearson and
Spearman test yielded some significant correlations, while the Kendall test did
not. Table 3 shows that, predominantly, conservative–innovative (HQS 4) and
lame–exciting (HQS 5) attributes are the ones with more strong and significant
correlations. We observe that these correlations correspond to those criteria in310
which conformance to WCAG 2.0 guidelines was evaluated by human testers,
6Kendall’s Tau is preferred to Spearman test because it performs better with small sample
sizes (Clark-Carter, 2004).
13

Table 2: Correlations between PWA and UX attributes for the Kendall test [N=44, 11 par-
ticipants x 4 Web sites]. Significance of tests are reported at p < 0.05 (*), p < 0.005 (**),
p < 0.001 (***).
UX attributes Kendall’s Tau (τ)
Attracdiff – Hedonic quality-identification
HQI 1 isolating–integrating 0.63∗∗∗
HQI 2 amateurish–professional 0.64∗∗∗
HQI 3 gaudy–classy 0.44∗∗∗
HQI 4 cheap–valuable 0.62∗∗∗
HQI 5 noninclusive–inclusive 0.68∗∗∗
HQI 6 takes me distant from people–brings me closer to people 0.66∗∗∗
HQI 7 unpresentable–presentable 0.67∗∗∗
Attracdiff – Hedonic quality-stimulation
HQS 1 typical–original 0.43∗∗∗
HQS 2 standard–creative 0.55∗∗∗
HQS 3 cautious–corageous 0.18
HQS 4 conservative–innovative 0.29∗
HQS 5 lame–exciting 0.47∗∗∗
HQS 6 easy–challenging -0.19
HQS 7 commonplace–new 0.45∗∗∗
Attracdiff – Pragmatic Quality
PQ 1 technical–human 0.37∗∗
PQ 2 complicated–simple 0.52∗∗∗
PQ 3 impractical–practical 0.64∗∗∗
PQ 4 cumbersome–direct 0.53∗∗∗
PQ 5 unpredictable–predictable 0.39∗∗
PQ 6 confusing–clear 0.55∗∗∗
PQ 7 unruly–manageable 0.60∗∗∗
Attracdiff – Evaluational Constructs
beauty 0.48∗∗∗
goodness 0.68∗∗∗
appeal 0.61∗∗∗
EWPL
annoyed -0.56∗∗∗
bored -0.46∗∗∗
confident 0.10
confused -0.47∗∗∗
disappointed -0.58∗∗∗
frustrated -0.58∗∗∗
happy 0.59∗∗∗
interested 0.54∗∗∗
hopeful 0.41∗∗
pleased 0.54∗∗∗
unsure -0.26∗
14

and specially the ones related to the number of satisfied and non-satisfied success
criteria for the Perceivable principle.
Table 3: Correlations between Accessibility Indicators and UX attributes for the Pearson test
[N=88, 11 participants x 4 Web sites x 2 Web pages)] for p <0.001. Values between brackets
correspond to the Kendall test.
HQI 1 HQS 4 HQS 5 PQ 2 PQ 7
isolating conservative lame complicated unruly
AI integrating innovative exciting simple manageable
m u 0.54
tool w u -0.50 -0.50 -0.50
sc sat aa 0.52 (0.54)
sc sat aa p 0.50
sc sat aa o 0.50
sc sat aa u (0.50)
sc sat a p 0.51 (0.53)
sc sat a u (0.50)
sc nsat aa -0.50 (-0.51)
sc nsat aa p -0.52 (-0.52)
sc nsat aa o -0.50
sc nsat a p -0.52 (-0.53) -0.51 (-0.52)
Figures 2, 3 and 4 show the correlation matrices corresponding to the coef-
ficients obtained from Pearson, Spearman and Kendall tests respectively, pro-315
viding a general overview of all the correlations in addition to those with large
effect sizes in Table 3. Matrices show that the more and stronger correlations are
between typical–original (HQS 1), conservative–innovative (HQS 4) and lame–
exciting (HQS 5) attributes and those AIs generated through the expert evalu-
ation of Web site compliance to WCAG 2.0, particularly again to those success320
criteria belonging to the Perceivable principle.
In order to check the consistency and robustness of the results we performed
Pearson’s correlation test using sampling with replacement. We applied the
bootstrapping technique for different numbers of bootstrap replicates (R=100,
500, 1000, 1500, 5000, 10000) for the same confidence level (0.95). We obtained325
15

Figure 2: Correlation matrix for the AIs and UX attributes, using the Pearson test [N=88,
11 participants x 4 Web sites x 2 Web pages, p < 0.001].
the bias (the difference between the mean of the R bootstrap samples and the
original estimate), the standard error (the standard deviation of the R bootstrap
samples) and the confidence intervals for R samples. We did not observe big
differences on the confidence intervals depending on the size of R. On the other
hand the bias and the standard error are very low, which suggests the similarity330
with the original estimate. For instance, if we take the bootstrap results for
the correlation between the HQS 5 attribute and the sc sat aa AI, the obtained
minimum and maximum values for the bias and the standard error were [-0.0094,
0.001], [0.066, 0.071] respectively. We also applied the bootstrapping technique
by relaxing the confidence level (0.90, 0.92). In this case the confidence intervals335
kept stable even when increasing alpha. As the bias and the standard error
obtained as a result of applying the bootstrapping technique is very low, none
of the confidence intervals include the zero value and the range of the intervals
16

Figure 3: Correlation matrix for the AIs and UX attributes, using the Spearman test [N=88,
11 participants x 4 Web sites x 2 Web pages, p < 0.001].
is not very wide we conclude that the correlations are robust.
4.3. Analysis of the Interviews340
We looked into the transcriptions of the interviews in order to better un-
derstand the practical importance of the identified correlations and the rea-
sons why they emerged. For this analysis we only focused on those three UX
attributes typical-original (HQS 1), conservative-innovative (HQS 4) and lame-
exciting (HQS 5), which showed stronger and more consistent correlations across345
the three statistical tests we performed.
We queried the transcripts using the synonyms and antonyms of the word-
pairs of the identified UX attributes. Then we annotated and coded the excerpts
17

Figure 4: Correlation matrix for the AIs and UX attributes, using the Kendall test [N=88, 11
participants x 4 Web sites x 2 Web pages, p < 0.001].
we retrieved. We used the Merriam-Webster online dictionary7, the Collins
online dictionary8and the qualitative data analysis software NVivo 10 for this350
analysis.
4.3.1. Hedonic Quality-Stimulation: Typical–Original (HQS 1)
Typical can be defined as being or serving as a representative example of a
particular type, characteristic, having the qualities associated with the members
of a particular group or kind. We looked up synonyms (such as archetypal,355
standard, model, normal, classic) and opposites (including unique, unusual,
unexpected, exceptional) in the transcripts. The majority of comments were
observations about situations that occur regularly such as coming across non-
7http://www.merriam-webster.com/thesaurus/
8http://www.collinsdictionary.com/dictionary/english-thesaurus
18

accessible Web sites. Two expert users (P01, P02) said that finding accessible
sites is still uncommon, particularly when it comes to restaurant Web sites:360
“Restaurant Web sites are those that I find especially non-accessible (P01)”.
P05 said that he would not spend much time on W3 if he was not participating
in the study: “This is the typical page I say – out! [snap of fingers] ”.
If a Web site provides an accessible version, the link pointing to it, is often
located at the bottom of the Web page and thus, not easily reachable: “Normally365
the buttons for the accessible version are at the bottom (.. . ) normally I cannot
get to the accessible version (. . . ) it always happens (P02)”. Since not all Web
sites have an alternative version, it is not very usual to find one: “It’s assumed
that the most suitable for us is the accessible one, because the ‘normal’ one is
the one that everyone uses (P07)”.370
Original is related to something unusual, novel, not known or experienced be-
fore. We looked up synonyms (new, novel, different, unusual, unknown) and
opposites (old, standard, traditional, normal, usual, ordinary). Most excerpts
about originality and accessibility have to do with content (i.e. textual infor-
mation) and a few refer to the layout of the Web site (i.e. how the information375
is arranged). Regarding textual content the uncommon name of the dishes on
W1 drew the attention of participants. Unlike ordinary restaurants, this one
organises cultural events like live music or plays to enhance the gastronomic
experience: “I found the Web site funny and very interesting. It called my at-
tention that the guy is in the artistic wave. The mix with culture, theatre, music380
(. . . ) (P05) ”.
Regarding unconventional layouts, the gastronomic information on W2 was
conveyed through links in a hierarchical multi-level layout, instead of providing
the information on the same page, which was not expected by P09: “The menu,
I was expecting a document, right? and then of course, it was not a menu as385
such, it was like a bunch of links”. P02 said that it was unusual to find the
link to the accessible version of the Web site quite at the beginning of the Web
page on W1: “No, it’s not usually the case (.. . ) it’s quite at the top”. The fact
that the content was placed before the navigation menu on W1 attracted P03
19

participant’s attention.390
Apart from the aspects related to Web content and layout, some participants
valued the novel experience of visiting a restaurant Web site for the first time:
“First time I visit a restaurant Web site, I didn’t know what it could contain
and I really liked it because it was non-accessible but if you try, you get to
know new things (P06-W1)”. Many participants praised and linked originality395
and the accessibility of W1 and W2 Web sites although this was not always
the case: “It’s not original, it gets to the point and is very professional, it is
not arty (. . . ) the page lacks an artistic touch (P05-W2)”. This suggests that
not only the accessibility of a Web site, but the quality of the textual content
provided on the Web site influences the perception of originality: “The text400
meets the objective of informing, and the more interesting the better, we ignore
all the visual part of about the venue and the dishes so textual explanations are
interesting for us (P10)”.
4.3.2. Hedonic Quality-Stimulation: Conservative–Innovative (HQS 4)
This attribute is closely related to the previous one (typical-original, HQS 1)405
as many comments coded as original were also coded into the innovative cate-
gory. Nevertheless, there is a subtle but important difference about the meaning:
unlike HQS 1, HQS 4 deals more with conservatism and innovation, which are
somehow related to progress and evolution.
Conservative represents a tendency to favour the preservation of established410
ideas, conditions, values or institutions, opposing innovation. Some synonyms
are traditional, conventional, moderate, cautious and reactionary. While liberal,
radical, progressive, innovative, and imaginative are some examples of opposites.
We found few examples of comments which relate both conservative and
inaccessibility: “The feeling is that the page hasn’t served me for anything (.. . )415
in addition to annoying me I feeling the site is totally conservative, it has no
innovation, it is not accessible at all (P04-W3)”. P05 goes further and suggests
that the lack of accessibility is an indicator of non-evolved society or country:
“I think they haven’t thought much about people who are not able to see, right?
20

Regarding accessibility, nowadays it should already be there (.. . ) I’m sure that420
in the Netherlands these non-accessible Web sites aren’t developed any more
. . . (P05-W3)”.
Innovative relates to showing a noteworthy use of the imagination and creativ-
ity especially in creating new things and inventing. Synonyms include novel,
new, original, different, fresh, unusual and unfamiliar. Most comments who425
link innovation and accessibility refer to the W1 Web site: “Seeing a different
design has aroused my curiosity (. . . ) what I have liked most is its innovative
character. . . (P10-W1)”.
4.3.3. Hedonic Quality-Stimulation: Lame–Exciting (HQS 5)
Lame means falling short of a standard, painful or weak, unconvincing,430
not effective or enthusiastic, conventional or uninspiring. Synonyms include de-
fective, unconvincing, poor, inadequate, weak, insufficient and unsatisfactory.
Most comments were about the W3 Web site, where we found at least one com-
ment for each participant about W3: “It is not easy to handle. The information
is quite hard to find with a screen reader, unspecific (.. . ) It’s too much effort,435
you waste a lot of time, the information you get is not very reliable. The messy
links were not very clarifying, they do not give an idea of the content in each
link (. . . ) it was like a labyrinth, too complicated (. . . ) to get something specific
you would have to invest much time, effort and I’m not sure if one would get
to anything concrete (. . . ) it forces you to have to do all the tour of the entire440
page. It’s like to get a room in a hotel you would have to go through the 360
rooms it has (. . . ) is the most difficult one of the four Web sites, for a screen
reader user is unsatisfactory (P07)”.
We only found comments from one participant who would stress the lameness
of W4: “Disappointed, confused in many moments, completely bored and very445
annoyed (P02)”. We also found a few comments about the most accessible Web
sites, W1 and W2: “It’s not a very clear content as to the presented links. It’s
promising, it seems it will provide information, but the information that exposes
is very literary. Very literary and very repetitive for a screen reader (P07-W1)”.
21

Exciting is related to causing great emotional or mental stimulation. Synonyms450
for exciting would be stimulating, inspiring, thrilling or sensational while oppo-
sites include boring, dull, dreary, monotonous, uninspiring. Most comments are
about the accessible Web sites, W1 and W2: “I found the Web page attractive
(. . . ) a desirable place to go (.. . ) a Web site with very specific and clear infor-
mation, and very attractive content (P03-W2)”. Some participants suggested455
the possibility to revisit the Web site and other even showed their willingness
to go to the restaurant: “I’ll check it at home, maybe I’ll write them an email
telling them that the Web site is perfect (P06-W2)”.
5. Discussion
5.1. On the Relationship between UX Attributes and Perceived Web Accessibility460
Results suggest that perceived Web accessibility is associated to most UX
attributes. The strong and moderate significant statistical correlations found
between PWA and the attributes belonging to the hedonic quality-identification
(i.e. inclusive, presentable, brings me closer to people, professional, integrating,
valuable and classy) indicate that participants may feel closer or more identified465
with Web sites they experience to be accessible. And the other way around:
they may feel more distant from Web sites perceived as non-accessible, as if
these Web sites were foreign artefacts that are not designed for them.
We found a relationship between PWA and pragmatic quality, which rep-
resents the usability perceived by participants. A Web site that participants470
considered accessible is related to aspects such as practical, manageable, direct,
clear and simple and predictable; whereas a non-accessible perception of a Web
site is related to aspects like impractical, unruly, cumbersome, confusing com-
plicated and unpredictable. Results for the relationship between PWA and the
hedonic quality-stimulation attribute indicate that Web sites experienced as ac-475
cessible are related to perceptions such as creative, original, exciting and new.
In contrast, Web sites considered to be non-accessible are related to perceptions
like standard, typical, lame and commonplace. Findings also support that a
22

positive accessibility perception is related to appraisals of goodness, appeal and
beauty. This suggests that a Web site which is experienced to be accessible480
is perceived as good, appealing and beautiful, while a non-accessible Web site
is considered as bad, repelling and ugly. In summary, participants perceived
positive qualities on Web sites experienced as accessible, and the opposite ef-
fect happened, in Web sites perceived as non-accessible negative qualities were
predominant.485
We also found strong and moderate correlations of PWA with emotion-
bearing words. PWA is positively related to emotional words with positive
valence (i.e. happy, pleased, interested and hopeful) and negatively related to
emotional words with negative valence (i.e. disappointed, frustrated, annoyed,
confused and bored). Accessible Web sites are related to positive emotional490
reactions, while non-accessible ones are correlated to the negative ones. This
indicates that participants may feel better on a Web site they experience as
accessible than when navigating on a Web site perceived as non-accessible.
These results show that the experienced accessibility of participants is not
only associated to perceptions on task-oriented aspects, but also to even more495
subjective and experiential ones like hedonic aspects, emotional reactions or
appraisals on beauty, goodness and appeal. While these outcomes are not sur-
prising, we provide empirical evidence indicating that perceived accessibility
and user experience could be understood as interchangeable qualities for blind
users. A practical implication of these findings is about informing the design500
of instruments and protocols to be used in studies involving users. Because we
know the UX attributes that are strongly correlated to PWA, UX terminology
could be used as an indirect way to elicit information about how users perceive
or experience the accessibility of a Web site on questionnaires, questions on
focus groups and interviews. Participants will probably be more familiarised505
with terms representing UX attributes (e.g. emotional reactions such as disap-
pointment and frustration) than with technical terms about the Web, assistive
technologies and accessibility. The identified attributes do not only serve as
proxies for perceived Web accessibility, but they can also facilitate the commu-
23

nication during user studies, leading to a better understanding of the experience510
of blind users with a Web site.
5.2. On the Relationship between UX Attributes and Web Accessibility Indica-
tors
We found evidence to support the relationship between the UX attributes
corresponding to the hedonic quality-stimulation typical-original (HQS 1), conservative-515
innovative (HQS 4), lame-exciting (HQS 5) and AIs representing the confor-
mance to WCAG 2.0 guidelines. We observed a slight predominance of AIs
corresponding to the Perceivable principle of WCAG 2.0. Accessible Web sites
(in terms of a higher number of satisfied SC or fewer number of non-satisfied
SC) are perceived to be original, innovative and exciting, whereas non-accessible520
ones (in terms of a lower number of satisfied SC or higher number of non-satisfied
SC) are perceived as typical, conservative and lame.
Comments of participants during the interviews provided additional evidence
to support the correlations found between compliance to guidelines and three
hedonic quality-stimulation attributes: original, innovative and exciting. Par-525
ticipants may consider accessible Web sites original because they still find many
non-accessible Web sites on the Web. Expert users claimed there are still many
non-accessible Web sites (especially restaurant Web sites) and coming across
an accessible Web site is considered a novelty. In line with this, some users
appreciated the uncommon event of encountering alternative and theoretically530
more accessible versions of Web sites. We also learned that the perception of
originality is not only influenced by the Web site’s accessibility: the quality of
textual content and its arrangement boosted the perception of originality. Nev-
ertheless, we are cautious about this statement as user expertise and familiarity
with the domain seem to play the role of moderator variables.535
Accessible Web sites were considered to be innovative and related to progress
and evolution, while non-accessible ones were regarded as conservative. A clear
relationship between lame and lack of accessibility as well as between exciting
and accessibility was observed. With regard to lame, participants gave some
24

illustrative metaphors which reflect how it is like to navigate on a Web site with540
serious accessibility problems: “a labyrinth, a loop going nowhere, trying each
room of a hotel to select just one, a fortress for the accessibility, a bunker.. . ”.
Conversely, participants were strongly motivated on accessible Web sites, which
led to Web site revisitation or even to physically go to the restaurant featured
on the Web site. The experience of accessing a different type of Web site for545
the first time may have contributed to some extent to the motivation of par-
ticipants. This suggests that the hedonic quality-stimulation is not only driven
by the characteristics of the stimuli, but by other experiential aspects, such as
expectations and previous experiences on the Web.
Nevertheless, we found it surprising the unbalanced number of comments550
about the lame attribute on the Web sites with low accessibility (W3 and W4).
One possible explanation for this is that the severity of accessible barriers may
have more impact than their number. For instance, W3 did not have proper
text alternatives for the navigational image links, which had devastating con-
sequences on the hedonic quality-stimulation attributes. Even if the content555
about the gastronomic offer was accessible users were totally demotivated when
exploring the homepage. On the other hand, the texts of the navigation menu
link in W4 were clear and concise although the content about gastronomic offer
was not completely accessible. This can be explained in light of previous works
that highlighted the importance of the information architecture and the under-560
standability of the texts in navigation menus. Blind users use them to get an
overview of Web sites, which helps them form a mental model of the Web site
(Leuthold et al., 2008).
Few works in the literature relate UX attributes and Web accessibility. One
exception is the study by Mbipom & Harper (2011) where accessibility indi-565
cators were computed using the Barrier Walkthrough method and aesthetic
judgements were made by sighted users. They found that Web pages judged
on the classical aesthetics attribute as being visually clean showed significant
correlations with accessibility. No correlation was found between the expressive
aesthetic attributes and accessibility indicating that an expressive design is not570
25

necessarily in conflict with accessibility. In fact, expressive aesthetics (Lavie &
Tractinsky, 2004; Hassenzahl, 2008) match with the hedonic quality-stimulation
attribute from Hassenzahl’s model. Specifically the original,innovative and ex-
citing attributes map to original,creative and fascinating expressive aesthetics
attributes respectively. Hence, our findings do not only corroborate that an575
expressive design is not necessarily in conflict with accessibility, but we provide
additional evidence on the interplay between Web aesthetics and accessibility.
In this context, we emphasise that web aesthetics should be conceived beyond
the visual representation and content of Web sites. In order to increase the
aesthetic perception of Web sites the information architecture and the quality580
of texts should be paid attention.
Whether compliance to accessibility guidelines implies a satisfying user ex-
perience is a controversial topic. Our findings suggest that compliance to guide-
lines benefits the original,innovative and exciting attributes of the hedonic
quality-stimulation attribute. It seems reasonable to assume that an accessible585
Web site is more likely to offer users new impressions and opportunities than
a non-accessible Web site. If the content of a Web site is accessible, users will
have more chances to be stimulated and motivated to navigate on that Web site
than on a poorly accessible Web site.
5.3. Implications for Design590
The accessibility problems participants encountered are those covered by pre-
vious works and guidelines (Theofanos & Redish, 2003; Leuthold et al., 2008;
Leporini & Patern`o, 2008). These works provide already a substantial body of
knowledge on the design recommendations to build more accessible Web sites.
Our findings corroborate how critical information architecture and navigation595
menus are, how beneficial it is to provide ‘skip navigation’ links and the effect
of text quality of the aesthetic perception of Web sites. As far as design recom-
mendations are concerned, our findings stress the criticality of the mentioned
features in that they do not only improve accessibility, but they also boost a
positive perception of Web sites.600
26

6. Conclusions
In order to acquire a better understanding of the interplay between UX and
Web accessibility, we analysed the relationships between UX attributes and per-
ceived Web accessibility (PWA) and accessibility indicators derived from confor-
mance of Web sites to guidelines (AIs). Results revealed that most UX attributes605
are significantly correlated with PWA indicating that perceived accessibility is
related to hedonic and pragmatic qualities. As concepts that belong to these UX
attributes (e.g. interested, disappointed, frustrated and annoyed) are probably
more familiar to users than technical terms about the Web and assistive technol-
ogy, they can be employed to facilitate the communication between researchers610
and in ethnography, contextual enquiry, focus groups or interview studies. We
also uncover significant relationships between three hedonic quality-stimulation
attribute pairs (typical-original,conservative-innovative and lame-exciting ) and
accessibility indicators that represent the number of satisfied WCAG 2.0 suc-
cess criteria. These attributes can be understood as proxy measures for Web615
accessibility conformance as far as blind users are concerned. Including these
attributes in questionnaires or other sort of enquiry method would be an indirect
way to obtain estimates of conformance to accessibility guidelines.
Our findings may be generalisable to the remit of the type, domain and
qualities of the Web sites used in the study. The Web sites under test have620
more hedonic than pragmatic qualities since their main objective is to attract
potential customers rather than to provide a service or functionality. Thus,
at most, and using the Web site classification framework by De Marsico &
Levialdi (2004) we could generalise these results to commercial sites, which
target a general audience and with an informative-seductive communication625
style. Future studies should address the possibility of generalising results to
other type of stimuli.
27

7. Acknowledgements
A copy of the Web sites, the results of their Web accessibility evaluations
and the generated datasets can be found in http://sipt07.si.ehu.es/amaia/630
stimuli. A. Aizpurua held a Ph.D. scholarship from The Department of Educa-
tion, Universities and Research of the Basque Government when the study was
conducted. The work has been partly funded by the same department under the
Grant IT395-10 and the Basque Advanced Informatics Laboratory (BAILab) at
the University of the Basque Country UPV/EHU (Grant UFI11/45).635
References
Abascal, J., Arrue, M., Fajardo, I., Garay, N., & Tom´as, J. (2004). The
use of guidelines to automatically verify web accessibility. Universal Access
in the Information Society,3, 71–79. URL: http://dx.doi.org/10.1007/
s10209-003-0069-3.640
Aizpurua, A., Arrue, M., & Vigo, M. (2013). Uncovering the role of expectations
on perceived web accessibility. In Proceedings of the 15th International ACM
SIGACCESS Conference on Computers and Accessibility ASSETS ’13 (pp.
74:1–74:2). New York, NY, USA: ACM. URL: http://doi.acm.org/10.
1145/2513383.2513411.645
Aizpurua, A., Arrue, M., & Vigo, M. (2015). Prejudices, memories, expectations
and confidence influence experienced accessibility on the web. Computers in
Human Behavior,51, Part A, 152 – 160. URL: http://www.sciencedirect.
com/science/article/pii/S0747563215003222.
Bargas-Avila, J. A., & Hornbæk, K. (2011). Old wine in new bottles or650
novel challenges: A critical analysis of empirical studies of user experience.
In Proceedings of the SIGCHI Conference on Human Factors in Comput-
ing Systems CHI ’11 (pp. 2689–2698). New York, NY, USA: ACM. URL:
http://doi.acm.org/10.1145/1978942.1979336.
28

Bigham, J. P., Cavender, A. C., Brudvik, J. T., Wobbrock, J. O., & Lander,655
R. E. (2007). Webinsitu: A comparative analysis of blind and sighted browsing
behavior. In Proceedings of the 9th International ACM SIGACCESS Confer-
ence on Computers and Accessibility Assets ’07 (pp. 51–58). New York, NY,
USA: ACM. URL: http://doi.acm.org/10.1145/1296843.1296854.
Brajnik, G. (2006). Web accessibility testing: When the method is the culprit.660
In ICCHP, LNCS 4061 (pp. 156–163). Springer Verlag.
Caldwell, B., Cooper, M., Reid, L. G., & Vanderheiden, G. (2008). Web con-
tent accessibility guidelines (WCAG) 2.0. URL: http://www.w3.org/TR/
WCAG20/.
Carifio, J., & Perla, R. J. (2007). Ten Common Misunderstandings, Misconcep-665
tions, Persistent Myths and Urban Legends about Likert Scales and Likert
Response Formats and their Antidotes. Journal of Social Sciences ,3, 106–
116.
Chisholm, W., Vanderheiden, G., & Jacobs, I. (1999). Web content accessibility
guidelines (WCAG) 1.0. URL: http://www.w3.org/TR/WAI-WEBCONTENT/.670
Clark-Carter, D. (2004). Quantitative Psychological Research: A Student’s
Handbook . Psychology Press.
De Marsico, M., & Levialdi, S. (2004). Evaluating web sites: Exploiting user’s
expectations. The International Journal of Human-Computer Studies,60 ,
381–416. URL: http://dx.doi.org/10.1016/j.ijhcs.2003.10.008.675
Gay, G., & Li, C. Q. (2010). Achecker: Open, interactive, customizable,
web accessibility checking. In Proceedings of the 2004 International Cross-
disciplinary Workshop on Web Accessibility (W4A) W4A (pp. 23:1–23:2).
New York, NY, USA: ACM. URL: http://doi.acm.org/10.1145/1805986.
1806019.680
van der Geest, T., van der Meij, H., & van Puffelen, C. (2014). Self-assessed and
actual internet skills of people with visual impairments. Universal Access in
29

the Information Society,13 , 161–174. URL: http://dx.doi.org/10.1007/
s10209-013-0304-5.
Hassenzahl, M. (2005). The thing and i: Understanding the relationship between685
user and product. In Funology (pp. 31–42). Springer Netherlands volume 3 of
Human-Computer Interaction Series. URL: http://dx.doi.org/10.1007/
1-4020-2967-5_4.
Hassenzahl, M. (2008). The interplay of beauty, goodness, and usability in
interactive products. Human-Computer Interaction,19 , 319–349. URL:690
http://dx.doi.org/10.1207/s15327051hci1904_2.
ISO (1998). Iso 9241-11: Ergonomic requirements for office work with visual
display terminals (vdts). part 11: Guidance on usability.
Lavie, T., & Tractinsky, N. (2004). Assessing dimensions of perceived visual aes-
thetics of web sites. The International Journal of Human-Computer Studies ,695
60 , 269–298. URL: http://dx.doi.org/10.1016/j.ijhcs.2003.09.002.
Law, E. L.-C., Roto, V., Hassenzahl, M., Vermeeren, A. P., & Kort, J. (2009).
Understanding, scoping and defining user experience: A survey approach.
In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (pp. 719–728). New York, NY, USA: ACM. URL: http://doi.acm.700
org/10.1145/1518701.1518813.
Lazar, J., Feng, J., & Allen, A. (2006). Determining the impact of computer
frustration on the mood of blind users browsing the web. In Proceedings
of the 8th International ACM SIGACCESS Conference on Computers and
Accessibility Assets ’06 (pp. 149–156). New York, NY, USA: ACM. URL:705
http://doi.acm.org/10.1145/1168987.1169013.
Leporini, B., & Patern`o, F. (2008). Applying web usability cri-
teria for vision-impaired users: Does it really improve task per-
formance? International Journal of Human-Computer Interac-
30

tion,24 , 17–47. URL: http://dx.doi.org/10.1080/10447310701771472.710
arXiv:http://dx.doi.org/10.1080/10447310701771472.
Leuthold, S., Bargas-Avila, J. A., & Opwis, K. (2008). Beyond web con-
tent accessibility guidelines: Design of enhanced text user interfaces for
blind internet users. International Journal of Human-Computer Studies,66 ,
257 – 270. URL: http://www.sciencedirect.com/science/article/pii/715
S1071581907001413.
Lindley, S. E., Meek, S., Sellen, A., & Harper, R. (2012). ”it’s simply integral
to what i do”: Enquiries into how the web is weaved into everyday life. In
Proceedings of the 21st International Conference on World Wide Web WWW
’12 (pp. 1067–1076). New York, NY, USA: ACM. URL: http://doi.acm.720
org/10.1145/2187836.2187979.
Lopes, R., Gomes, D., & Carri¸co, L. (2010). Web not for all: A large scale
study of web accessibility. In Proceedings of the 2010 International Cross
Disciplinary Conference on Web Accessibility (W4A) W4A ’10 (pp. 10:1–
10:4). New York, NY, USA: ACM. URL: http://doi.acm.org/10.1145/725
1805986.1806001.
Maxwell, K. (2006). What you need to know about statistics. In Web Engineer-
ing (pp. 365–408). Springer Berlin Heidelberg. URL: http://dx.doi.org/
10.1007/3-540-28218-1_12.
Mbipom, G., & Harper, S. (2011). The interplay between web aesthetics and730
accessibility. In The Proceedings of the 13th International ACM SIGAC-
CESS Conference on Computers and Accessibility ASSETS ’11 (pp. 147–154).
New York, NY, USA: ACM. URL: http://doi.acm.org/10.1145/2049536.
2049564.
Murphy, E., Kuber, R., McAllister, G., Strain, P., & Yu, W. (2008). An em-735
pirical investigation into the difficulties experienced by visually impaired in-
ternet users. Universal Access in the Information Society,7, 79–91. URL:
http://dx.doi.org/10.1007/s10209-007-0098-4.
31

Norman, G. (2010). Likert scales, levels of measurement and the laws of statis-
tics. Advances in Health Sciences Education ,15 , 625–632. URL: http:740
//dx.doi.org/10.1007/s10459-010-9222-y.
Petrie, H., Hamilton, F., & King, N. (2004). Tension, what tension?: Website
accessibility and visual design. In Proceedings of the 2004 International Cross-
disciplinary Workshop on Web Accessibility (W4A) W4A ’04 (pp. 13–18).
New York, NY, USA: ACM. URL: http://doi.acm.org/10.1145/990657.745
990660.
Petrie, H., & Kheir, O. (2007). The relationship between accessibility and
usability of websites. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems CHI ’07 (pp. 397–406). New York, NY, USA:
ACM. URL: http://doi.acm.org/10.1145/1240624.1240688.750
Petrie, H., & Precious, J. (2010). Measuring user experience of websites: Think
aloud protocols and an emotion word prompt list. In CHI ’10 Extended
Abstracts on Human Factors in Computing Systems CHI EA ’10 (pp. 3673–
3678). New York, NY, USA: ACM. URL: http://doi.acm.org/10.1145/
1753846.1754037.755
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012). Guidelines are only half
of the story: Accessibility problems encountered by blind users on the web.
In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems CHI ’12 (pp. 433–442). New York, NY, USA: ACM. URL: http:
//doi.acm.org/10.1145/2207676.2207736.760
Theofanos, M. F., & Redish, J. G. (2003). Bridging the gap: Between acces-
sibility and usability. interactions,10 , 36–51. URL: http://doi.acm.org/
10.1145/947226.947227.
Th¨uring, M., & Mahlke, S. (2007). Usability, aesthetics and emo-
tions in humantechnology interaction. International Journal of Psychol-765
ogy,42 , 253–264. URL: http://dx.doi.org/10.1080/00207590701396674.
arXiv:http://dx.doi.org/10.1080/00207590701396674.
32

Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., & Abascal, J. (2007). Quan-
titative metrics for measuring web accessibility. In Proceedings of the 2004
International Cross-disciplinary Workshop on Web Accessibility (W4A) W4A770
’07 (pp. 99–107). New York, NY, USA: ACM. URL: http://doi.acm.org/
10.1145/1243441.1243465.
Vigo, M., & Harper, S. (2013). Coping tactics employed by visually disabled
users on the web. The International Journal of Human-Computer Studies ,
71 , 1013–1025. URL: http://dx.doi.org/10.1016/j.ijhcs.2013.08.002.775
Yesilada, Y., Brajnik, G., Vigo, M., & Harper, S. (2012). Understanding
web accessibility and its drivers. In Proceedings of the International Cross-
Disciplinary Conference on Web Accessibility W4A ’12 (pp. 19:1–19:9). New
York, NY, USA: ACM. URL: http://doi.acm.org/10.1145/2207016.
2207027.780
33