Web Accessibility Metrics For A Post Digital World
ABSTRACT This paper argues that, as we move towards a 'post-digital' world where use of the Web becomes normalised, there is a need to address Web accessibility measurement challenges within a wider real-world context. Strategy and policy that defines Web accessibility purely by the conformance of digital resources with technical guidelines can lead to a danger that 'good enough' solutions may fail to be deployed; they also fail to consider a wider measure of user experience in accessibility measurement. We propose that metrics should draw on aspects of user experience to provide a more meaningful, real-world measure of the impact (or not) of accessibility barriers and therefore priority in addressing them. Metrics should also consider context in terms of the quality of effort taken by organisations to provide an inclusive experience; one option for doing so is the framework provided by British Standard 8878 Code of Practice for Web Accessibility. In both cases, challenges exist in the complexity of defining and implementing such metrics.
- SourceAvailable from: Helen L. Petrie[show abstract] [hide abstract]
ABSTRACT: This chapter introduces a range of evaluation methods that assist developers in the creation of interactive electronic products, services and environments (eSystems) that are both easy and pleasant to use for the target audience. The target audience might be the broadest range of people, including people with disabilities and older people or it might be a highly specific audience, such as university students studying biology. The chapter will introduce the concepts of accessibility, usability and user experience as the criteria against which developers should be evaluating their eSystems, and the iterative user-centred design lifecycle as the framework within which the development and evaluation of these eSystems can take place. Then a range of methods for evaluating accessibility, usability and user experience will be outlined, with information about their appropriate use and strengths and weaknesses.C Stepanidis. 01/2009;
Web Accessibility Metrics For A Post Digital World
School of Computing
University of Dundee
+44 1382 385598
University of Bath
+44 1225 383943
1. Problem Addressed
This paper argues that, as we move towards a 'post-digital'
world where use of the Web becomes normalised, there is a
need to address Web accessibility measurement challenges
within a wider real-world context. Strategy and policy that
defines Web accessibility purely by the conformance of
digital resources with technical guidelines can lead to a
danger that 'good enough' solutions may fail to be deployed;
they also fail to consider a wider measure of user
experience in accessibility measurement.
We propose that metrics should draw on aspects of user
experience to provide a more meaningful, real-world
measure of the impact (or not) of accessibility barriers and
therefore priority in addressing them. Metrics should also
consider context in terms of the quality of effort taken by
organisations to provide an inclusive experience; one option
for doing so is the framework provided by British Standard
8878 Code of Practice for Web Accessibility. In both cases,
challenges exist in the complexity of defining and
implementing such metrics.
Accessibility metrics provide a valuable role in establishing
the extent to which a web resource can be effectively used
by people with specific access needs. W3C WCAG exists
as the de facto standard for measuring technical web
content accessibility; it can be incorporated into other
standards and policy, and act as a focus for automated
However there are limitations to WCAG as a means of
measuring accessibility of wider real-world online
experiences (Kelly et al 2005; Sloan et al 2006). These
limitations include the extent and nature of accessibility
barriers covered by the guidelines, the relationship of
guideline conformance to observed user experience (as
found in e.g. Petrie and Kheir, 2007), and the focus on
product rather than the process undertaken to create that
product. This is potentially problematic for countries such
as the UK, where legislation protecting the rights of
disabled people focuses on access to information and
services and the responsibilities of organisations to take
action to promote equality, rather than technical
More recently, implications are now emerging of in a post-
digital environment in which it becomes increasingly
difficult to differentiate between Web resources and other
examples of digital resources as use of digital means to
deliver information, services and experiences becomes
normalised (White, 2009). The emergence of mobile and
tablet devices as platforms for Web content and experience
delivery blurs boundaries further, while economic
constraints place new pressures on the need to rapidly
publish, appropriate and reuse digital content for different
purposes. This leads to a pragmatic desire not to reject
potentially useful content even if it contains known
accessibility problems for specific groups of people, and
where resource constraints limit the potential for addressing
these problems quickly, directly, and fully.
For organisations who wish – or who are required – to
measure progress in delivering
experiences, there is a need for more sophisticated models
of understanding and addressing accessibility issues, in a
way that is sympathetic to current digital authoring
practices and online experiences. This would allow
accessibility policy to incorporate a conformance-driven
model of measurement within the context of a more holistic
measure of the level of inclusivity.
Measuring accessibility should not be restricted to web
pages; but rather the extent to which goals can be
completed by disabled people relative to other members of
the target audience; which may mean alternative equivalent
routes, using alternative online resources if necessary.
Further, this definition could incorporate more subjective,
experiential issues, such as those presented by Hassenzahl
(2010), that may modify the apparent impact of a specific
accessibility barrier – either by lessening its impact due to a
very positive use experience, or increasing its impact
because of a negative user experience. In each case, the
default WCAG priority of a technical barrier – and the
urgency required to deal with it may change.
Techniques for measuring accessibility, usability and user
experience of e-systems are discussed in Petrie and Bevan
(2009), although there is no detailed discussion of areas of
overlap between accessibility and user experience.
document accessibility of e-learning resources, provides a
promising example of how the steps taken to optimise the
accessibility of an online experience – beyond guideline
conformance to include user engagement and reaction - can
be documented, publicised and updated over time.
BS 8878 (BSI 2010), published in the UK, presents a
framework for organisations to follow in the process of
commissioning accessible web sites, from definition of
accessibility, to procurement – whether internal or
outsourcing – to evaluating the accessibility of the solution
delivered. As a process standard focused primarily on
managers and decision-makers, it is complementary to
existing technical standards for accessibility.
the Techdis Accessibility Passport
4. Major Difficulties
The obvious difficulties in defining and implementing an
accessibility metric that incorporates quality of user
experience and the quality of the process undertaken to
provide that experience are the complexity of the
environment to be measured – i.e. not just a collection of
resources that enable an experience, but also evidence of
organisational activity taken to enhance inclusion. These
require precise and logical definition; so measurement may
take substantial time and effort, and will inevitably have a
degree of subjectivity, which needs to be carefully
controlled if measures are to be reliable and meaningful.
Additionally, there is an obvious need for such a metric to
take into account evidence of decision-making, in particular
justification for failing to address accessibility barriers –
while still encouraging Web resource providers to continue
to focus on removing barriers wherever possible.
BS 8878 provides a framework that allows definition – and
measurement – of the process undertaken by organisations
to procure an optimally accessible web site, but is at present
a copyrighted work and not freely available. In comparison
to a purely technical WCAG conformance report, the nature
of the data being gathered for measurement means that
inevitably the measurement process is longer; but it also
provides a richer set of data giving context – and therefore
justification – to current levels of accessibility.
This paper presents a theoretical proposal, and as such no
outcomes from our work exist. However, many of the ideas
expressed in earlier work encouraging adoption of a more
holistic approach to accessibility are reflected in the text of
BS8878; while the Techdis Accessibility Passport provides
an example of how rich documentation of accessibility of a
resource could be achieved.
The logical outcome should be the formalisation of an
accessibility metric that can be adopted at an organisational
level or higher, as a means of defining effectiveness of an
online accessibility strategy. This would incorporate
technical guideline conformance and measures of user
experience to disabled people alongside measures of the
process adopted of procuring and providing accessible web
6. Open Research Avenues
Following the iterative development of an inclusive user
experience metric, the next step would be to evaluate the
effectiveness of such a metric. The authors are currently
exploring (Kelly 2011) the potential of using such a metric
to evaluate the online accessibility strategy of Higher
Education institutions – specifically collections of academic
web content they provide, such as institutional repositories
of research output.
BSI (2010) BS 8878:2010 Web accessibility – Code of
Hassenzahl M. (2010) Experience Design – Technology for
All the Right Reasons. Morgan & Claypool.
Kelly, B. (2011) Metrics, This Time For Web Accessibility.
Retrieved 31 October 2011 from
Kelly B., Sloan D., Phipps L., Petrie H. and Hamilton F.
(2005) Forcing standardization or accommodating
diversity? A framework for applying the WCAG in the real
world. Proceedings of the 2005 International Cross-
Disciplinary Workshop on Web Accessibility (W4A)
(Chiba, Japan, 10 May 2005). New York:ACM Press, 46-
Petrie, H and Bevan N. (2009) The Evaluation of
Accessibility, Usability and User Experience. The
Universal Access Handbook, C Stepanidis (ed), CRC Press,
Petrie, H. and Kheir, O. (2007) The relationship between
accessibility and usability of websites. In the Proceedings of
CHI '07: ACM Annual Conference on Human Factors in
Computing Systems. New York: ACM Press, 397-406.
Sloan D., Kelly B., Heath A., Petrie H., Hamilton F. and
Phipps L. (2006) Contextual Accessibility: Maximizing the
Benefit of Accessibility Guidelines. Proceedings of the
2006 International Cross-Disciplinary Workshop on Web
Accessibility (W4A) (Edinburgh, Scotland, 23 May 2006).
New York: ACM Press, 121-131.
White, D. (2009) PostDigital: Escaping the Kingdom of the
New? Retrieved 31 October 2011 from