ArticlePDF Available

Tool independence for the Web Accessibility Quantitative Metric

Taylor & Francis
Disability and Rehabilitation: Assistive Technology
Authors:
Markel Vigo at The University of Manchester
Markel Vigo
Giorgio Brajnik at University of Udine
Giorgio Brajnik
Myriam Arrue at University of the Basque Country
Myriam Arrue
Julio Abascal at University of the Basque Country
Julio Abascal
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

The Web Accessibility Quantitative Metric (WAQM) aims at accurately measuring the accessibility of web pages. One of the main features of WAQM among others is that it is evaluation tool independent for ranking and accessibility monitoring scenarios. This article proposes a method to attain evaluation tool independence for all foreseeable scenarios. After demonstrating that homepages have a more similar error profile than any other web page in a given web site, 15 homepages were measured with 10,000 different values of WAQM parameters using EvalAccess and LIFT, two automatic evaluation tools for accessibility. A similar procedure was followed with random pages and with several test files obtaining several tuples that minimise the difference between both tools. One thousand four hundred forty-nine web pages from 15 web sites were measured with these tuples and those values that minimised the difference between the tools were selected. Once the WAQM was tuned, the accessibility of 15 web sites was measured with two metrics for web sites, concluding that even if similar values can be produced, obtaining the same scores is undesirable since evaluation tools behave in a different way.
Figures - uploaded by Markel Vigo
Author content
All figure content in this area was uploaded by Markel Vigo
Content may be subject to copyright.
Content uploaded by Markel Vigo
Author content
All content in this area was uploaded by Markel Vigo
Content may be subject to copyright.
!
Tool!Independence!for!the!Web!Accessibility!Quantitative!Metric!!
"#$%&'!()*+,-!.)+$*)+!/$#01)%2-!"3$)#4!5$$6&,!#17!86')+!59#:;#',!
,!<1)=&$:)>3!+?!>@&!/#:A6&!B+61>$3-!C1?+$4#>)%#!D#%6'>#>&#-!2EE,F!G+1+:>)#-!HI#)1J!
4#$%&'-!43$)#4-!06')+K:)J&@6J&:!
!
2!G)I#$>)4&1>+!7)!"#>&4#>);#!&!C1?+$4#>);#-!<1)=&$:)>L!7&!<7)1&-!MM,EE!<7)1&-!C>#'3!
*)+$*)+K7)4)J61)67J)>!
!
AbstractJ! N@&! O&9! 5;;&::)9)')>3! P6#1>)>#>)=&! "&>$);! QWAQMR! #)4:! #>! #;;6$#>&'3!
4&#:6$)1*!>@&!#;;&::)9)')>3!+?!S&9!I#*&:J!T1&!+?!>@&!4#)1!?&#>6$&:!+?!WAQM!#4+1*!
+>@&$:!):!>@#>!)>!):!&=#'6#>)+1!>++'!)17&I&17&1>!?+$!$#1%)1*!#17!#;;&::)9)')>3!4+1)>+$)1*!
:;&1#$)+:J!N@):!I#I&$!I$+I+:&:!#!4&>@+7!>+!#>>#)1!&=#'6#>)+1!>++'!)17&I&17&1;&!?+$!#''!
?+$&:&&#9'&! :;&1#$)+:J! 5?>&$! 7&4+1:>$#>)1*! >@#>! @+4&I#*&:! @#=&! #! :)4)'#$! &$$+$!
I$+?)'&! >@#1! #13! +>@&$! S&9! I#*&! )1! #! *)=&1! S&9! :)>&-! ,U! @+4&I#*&:! S&$&! 4&#:6$&7!
S)>@!,EEEE!7)??&$&1>!=#'6&:!+?!WAQM!I#$#4&>&$:!6:)1*!>@&!#6>+4#>);!&=#'6#>)+1!>++':!
?+$! #;;&::)9)')>3! V=#'5;;&::! #17! WCDNJ! H)4)'#$! I$+;&76$&! S#:! ?+''+S&7! S)>@! $#17+4!
I#*&:!#17!S)>@! :&=&$#'! >&:>!?)'&:!+9>#)1)1*!:&=&$#'!>6I'&:!>@#>!4)1)4)X&!>@&!7)??&$&1;&!
9&>S&&1! 9+>@! >++':J! ,YYZ! S&9! I#*&:! ?$+4! ,U! S&9! :)>&:! S&$&! 4&#:6$&7! S)>@! >@&:&!
>6I'&:! #17! >@+:&! =#'6&:! >@#>! 4)1)4)X&7! >@&! 7)??&$&1;&! 9&>S&&1! >@&! >++':! S&$&!
:&'&;>&7J!T1;&!WAQM!S#:!>61&7-!>@&!#;;&::)9)')>3!+?!,U!S&9!:)>&:!S#:!4&#:6$&7!S)>@!
>S+! 4&>$);:! ?+$! S&9! :)>&:! ;+1;'67)1*! >@#>! &=&1! )?! :)4)'#$! =#'6&:! ;#1! 9&! I$+76;&7-!
+9>#)1)1*!>@&! :#4&!:;+$&:! ):! 617&:)$#9'&! :)1;&! &=#'6#>)+1! >++':! 9&@#=&! )1! #! 7)??&$&1>!
S#3J!
!
1!Introduction!
C1!$&;&1>!3&#$:!#!*$&#>!7&#'!+?!$&:&#$;@!@#:!9&&1!;#$$)&7!+6>!)1!>@&!?)&'7!+?!4&>$);:!?+$!
S&9!#;;&::)9)')>3!#::&::4&1>-! >@#>! ):-! @+S!>+!4&#:6$&! >@&! #;;&::)9)')>3! '&=&'!+?!#! S&9!
:)>&J!"&#:6$)1*!>@&!#;;&::)9)')>3!'&=&'!):!1&;&::#$3!:)1;&!4+$&!#17!4+$&!:;&1#$)+:!#$&!
9#:&7! +1! :6;@! '&=&':! Q:&&! &[#4I'&:! 9&'+SR-! #17! )1! 4#13! ;#:&:! >@&3! $&A6)$&! @)*@!
#;;6$#;3!)1!+$7&$!>+!$#>&!#17!#::&::!>@&!#;;&::)9)')>3!+?!S&9!:)>&:J!
N@&!4+:>!9$+#7'3!6:&7!4&>$);!?+$!#;;&::)9)')>3!):!>@&!A6#')>#>)=&!4&>$);!I$+I+:&7!93!>@&!
O&9! 5;;&::)9)')>3! C1)>)#>)=&,! QO5CR! )1! >@&! ;+1>&[>! +?! O&9! 5;;&::)9)')>3! B+1>&1>!
.6)7&')1&:!,JE!QB@):@+'4!&>!#'J-!,ZZZR!S@);@!#$&!6:&7!?+$!4&#:6$)1*!>@&!;+1?+$4#1;&!
+?! #! S&9I#*&! S)>@! >@&! #?+$&4&1>)+1&7! *6)7&')1&! :&>J! 5! S&9! I#*&! :#>):?3)1*! #''!
;@&;%I+)1>:!+9>#)1:!#!\555]!$#>)1*-!)?!)>!:#>):?)&:!#''!I$)+$)>3!,!#17!I$)+$)>3!2!;@&;%I+)1>:!
)>!*&>:! \55]-!>@&! \5]!$#>)1*! ):! +9>#)1&7! )?! +1'3! #''! I$)+$)>3! ,! ;@&;%I+)1>:! #$&! :#>):?)&7-!
,!N@&!O&9!5;;&::)9)')>3!C1)>)#>)=&-!O5CJ!5=#)'#9'&!#>!@>>I^__SSSJSMJ+$*_O5C_!
This!is!the!author!copy!of!Tool!Independence!for!the!Web!Accessibility!Quantitative!Metric.!Disability!
&!Rehabilitation:!Assistive!Technology!4(4),!248"263.!Informa!Healthcare.!Available!at!
http://informahealthcare.com/doi/abs/10.1080/17483100902903291!
Note!that!there!might!be!some!inconsistencies!between!this!and!the!above!publication!so!use!this!copy!at!
your!own!risk.
#17!?)1#''3!>@&!S&9!I#*&!):!\1+1!;+1?+$4#1>]!+>@&$S):&J!`+>);&!@+S!93!06:>!=)+'#>)1*!#!
:)1*'&!I$)+$)>3!,!;@&;%I+)1>!>@&!I#*&!9&;+4&:!1+1!;+1?+$4#1>J!
N@):! 4&>$);! Q>@#>! '&#7:! >+! +$7&$&7! :349+');! =#'6&:! a555-! 55-! 5-! `BbR! ):! 1+>! I$&;):&!
&1+6*@! )1! +$7&$! >+! $#>&! #17! ;'#::)?3! S&9! #II');#>)+1:! #;;+$7)1*! >+! >@&)$! #;;&::)9)')>3!
'&=&'J! D+$! &[#4I'&-! #! S&9! I#*&! ?6'?)'')1*! #''! I$)+$)>3! ,! ;@&;%I+)1>:! S+6'7! +9>#)1! >@&!
:#4&! #;;&::)9)')>3! =#'6&! #:! #1+>@&$! S&9! I#*&! ?6'?)'')1*! #''! I$)+$)>3! ,! ;@&;%I+)1>:! #17!
#'4+:>!#''!I$)+$)>3!2!;@&;%I+)1>:^!9+>@!+?!>@&4!S+6'7!*&>!>@&!5!'&=&'!;+1?+$4#1;&J!N@):!
;$)>&$)+1!:&&4:!>+!9&!9#:&7!+1!>@&!#::64I>)+1!>@#>!)?!#!S&9!I#*&!?#)':!>+!#;;+4I'):@!
+1&!+?! >@&!;@&;%I+)1>:!)1!#!'&=&'-!)>!):!:+!)1#;;&::)9'&!#:!)?!)>!?#)'&7!>+!?6'?)'!#''! +?!>@&4J!
5'>@+6*@!>@):!):!I+::)9'&-!:&=&$#'!:;&1#$)+:!&[):>!>@#>!$&A6)$&!#!@)*@&$!$&:+'6>)+1!?+$!>@&!
4&>$);!>@#>!*+!9&3+17!#!:4#''!+$7&$&7!:;#'&!+?!#;;&::)9)')>3!:;+$&:J%
Quality%Assurance%(QA)%within%Web%Engineering%
O&9! V1*)1&&$)1*! 7&?)1&:! :I&;)?);! 4&>@+7+'+*)&:-! 4+7&':! #17! >&;@1)A6&:! ?+$! S&9!
#II');#>)+1:!7&=&'+I4&1>J!H)1;&!)>:!?)1#'!*+#'!):!>+!+9>#)1!@)*@!A6#')>3!S&9!#II');#>)+1:-!
#!P6#')>3!5::6$#1;&!I$+;&::!):!+?!#!I#$#4+61>!)4I+$>#1;&J!N@):!&1>#)':!>@&!#II');#>)+1!
+?! 4&>$);:-! 4&>@+7:! #17! A6#')>3! 4+7&':! >@$+6*@+6>! >@&! 7&=&'+I4&1>! I$+;&::J! 5:! #!
;+1:&A6&1;&-! 4&#:6$&4&1>! +?! S&9! 6:#9)')>3! #17! #;;&::)9)')>3! :@+6'7! 9&! I&$?+$4&7!
76$)1*!>@&!7)??&$&1>!:>#*&:!+?!>@&!')?&;3;'&!#17!:@+6'7!9&!;#I#9'&!>+!3)&'7!I$&;):&!#17!
:>#17#$7! $&:6'>:c! I$&;):)+1! ):! 1&&7&7! )1! +$7&$! >+! 9&! #9'&! >+! 7):>)1*6):@! '&=&':! +?!
#;;&::)9)')>3!>@#>!4#3!9&!;'+:&!>+!&#;@!+>@&$-!#17!:>#17#$7)X#>)+1!):!1&&7&7!)1!+$7&$!>+!
9&!#9'&!>+!;+4I#$&!7)??&$&1>!=&$:)+1:!+?!#!S&9!:)>&J!
C1!>@):!:&1:&-!A6#')>3!4+7&':!:6;@!#:!2PB(MP!Q");@!&>!#'J-!2EEMR!#17!O&9PV"!QT':)1#!
#17! d+::)-! 2EE2R! @#=&! 9&&1! 7&?)1&7J! N@&! ;@#$#;>&$):>);:! +?! S&9! #II');#>)+1:! #17! >@&!
1&;&::#$3! 4&>$);:! ?+$! >@&)$! &=#'6#>)+1! #$&! )1;'67&7! )1! >@&:&! 4+7&':J! C1! 9+>@! A6#')>3!
4+7&':-! S&9! #;;&::)9)')>3! ):! #1! #>>$)96>&! S@);@! :@+6'7! 9&! 4&#:6$&7! )1! +$7&$! >+!
*6#$#1>&&!>@&!A6#')>3!+?!>@&!I$+76;>J!D+$!>@):!$&#:+1-!A6#1>)>#>)=&!4&>$);:!#$&!&::&1>)#'!
)1! +$7&$! >+! #;;6$#>&'3! 4&#:6$&! 6:#9)')>3! #17! #;;&::)9)')>3! I$+I&$>)&:! :)1;&! 9+>@! #$&!
1&&7&7! ?+$! P5J! C1! #77)>)+1-! $#1%)1*! I$+>+>3I&:! +?! #! I$+76;>! #;;+$7)1*! >+! >@&)$!
#;;&::)9)')>3! '&=&'! ):! 6:&?6'! )1! +$7&$! >+! #::&::! >@&! )4I#;>! +?! ;@#1*&:-! 6I7#>&:! )1!
?61;>)+1#')>)&:-! &>;J! )1! #13! )>&$#>)=&! 7&=&'+I4&1>! I$+;&::J! N@&$&?+$&-! A6#1>)>#>)=&!
4&>$);:! ?+$! 4&#:6$)1*! #17! $#1%)1*! I$+>+>3I&:! #;;+$7)1*! >+! >@&)$! #;;&::)9)')>3! '&=&'!
S+6'7!9&!6:&?6'J%
Considering%Web%Accessibility%in%Information%Retrieval%
C=+$3!&>!#'J!Q2EEYR!;+176;>&7!#! :>673!S)>@!=):6#'!)4I#)$&7!6:&$:!)1!+$7&$!>+!7&>&$4)1&!
>@&!?#;>+$:!S@);@!S+6'7!)4I$+=&!:&#$;@!&1*)1&!$&:6'>:!?+$!>@+:&!6:&$:J!C>!):!;+1;'67&7!
>@#>! :+4&! 6:&$:! S+6'7! ')%&! >+! %1+S! #77)>)+1#'! 7&>#)':! #9+6>! :&#$;@! $&:6'>:-! :6;@! #:!
S@&>@&$! $&>$)&=&7! I#*&:! #$&! #;;&::)9'&! >+! >@&4! +$! 1+>-! #17! >@&! I#I&$! $&;+44&17:!
:+$>)1*! $&:6'>:! #;;+$7)1*! >+! #;;&::)9)')>3! +$! 6:#9)')>3! ;$)>&$)#J! d&e$#1%)1*! $&:6'>:!
#;;+$7)1*!>+!6:&$:f!=):6#'!#9)')>)&:!S+6'7!)4I$+=&!>@&)$!:&#$;@!&[I&$)&1;&J!
C1! >@):! :&1:&-! .++*'&! @#:! '#61;@&7! \.++*'&! 5;;&::)9'&! H&#$;@]2!S@&$&! $&:6'>:! #$&!
+$7&$&7!#;;+$7)1*!>@&!;$)>&$)#!:>#>&7!)1!>@&)$!D5P^!\I#*&:!S)>@!?&S!=):6#'!7):>$#;>)+1:!
#17!I#*&:!>@#>!#$&!')%&'3!>+!$&17&$!S&''!S)>@!)4#*&:!>6$1&7!+??]J!g+S&=&$-!>@):!):!1+>!#!
2!5=#)'#9'&!#>!@>>I^__'#9:J*++*'&J;+4_#;;&::)9'&_!
;+4I$&@&1:)=&! #II$+#;@! :)1;&! +1'3! :+4&! *6)7&')1&:! ?+$! =):6#''3! )4I#)$&7! 6:&$:! #$&!
9&)1*!;+1:)7&$&7!#17!6:&$:!S)>@!+>@&$!>3I&!+?!7):#9)')>)&:!#$&!1+>!>#%&1!)1>+!#;;+61>!#>!
#''J! 5$$6&! &>! #'J! Q2EEFR! I$+I+:&! #! 4+7&'! >+! #7&A6#>&'3! ;+49)1&! $&:6'>:! I$+=)7&7! 93!
>$#7)>)+1#'!Cd!:3:>&4:-!:6;@!#:!:&#$;@!&1*)1&:!?+$!>@&!OOO-!S)>@!#;;6$#>&!#;;&::)9)')>3!
4&#:6$&4&1>J! d&:6'>:! #$&! $&e$#1%&7! #;;+$7)1*! >+! >@&)$! #;;&::)9)')>3! '&=&'! +$! >@&3! #$&!
:@+S1!)1!>@&!+$7&$!I$+=)7&7! 93! >@&! :&#$;@! &1*)1&! 96>! '#9&''&7! S)>@!>@&)$!#;;&::)9)')>3!
:;+$&J!N@&!?+$4&$!4+7#')>3!#>>#;@&:!)4I+$>#1;&!>+!>@&!#;;&::)9)')>3!+?!$&:6'>:-!S@)'&!)1!
>@&!'#>>&$!I$&?&$&1;&!):!*)=&1!>+!>@&!$&'&=#1;&!+?!>@&!$&:6'>:! ?+$! #! *)=&1! A6&$3J! C1! #13!
;#:&-!>@&:&!4+7#')>)&:!;#1!+1'3!9&!#;@)&=&7!)?!#;;&::)9)')>3!;#1!9&!4&#:6$&7J%
Accessibility%Monitoring%
T1;&!#!S&9!:)>&!@#:!9&&1!7&=&'+I&7-!%&&I)1*!>$#;%!+?!>@&!&=+'6>)+1!+?!)>:!#;;&::)9)')>3!
'&=&'!@#:!#!I#$#4+61>!)4I+$>#1;&!:)1;&!)>!4#3!9&!?$#4&7!93!'&*#'!$&:>$);>)+1:J!G6&!>+!
>@&!1#>6$&!+?!>@&!OOO-!6I7#>&:!)1!S&9!:)>&:!#$&!A6)>&!?$&A6&1>!#17!')>>'&!;+1>$+'!;#1!
9&!&[&$>&7!+1!)>:!;+1>&1>!Q?+$!&[#4I'&-!;+1:)7&$!;+''#9+$#>)=&!S&9!:)>&:!')%&!D');%$!+$!#!
1&S:!S&9!:)>&!')%&!B``J;+4RJ!H)1;&!6I7#>&:!;#1!7&;$&#:&!>@&!#;;&::)9)')>3!'&=&'-!:+4&!
S&9! :)>&:! ?)17! >@&4:&'=&:! )1! ')49+! :)>6#>)+1! S@);@! ;#1! $&:6'>! )1! '&*#'! )::6&:! #17!
I+::)9'3! #74)1):>$#>)=&! ?)1&:J! N@&$&?+$&-! 4+1)>+$)1*! +?! >@&! &=+'6>)+1! +?! S&9!
#;;&::)9)')>3! $&A6)$&:! 4&>$);:! >@#>! #$&! I$&;):&! &1+6*@! )1! +$7&$! >+! #=+)7! >@+:&!
;)$;64:>#1;&:J! N@&:&! 4&>$);:! :@+6'7! 9&! 6:&7! ?+$! $#1%)1*! I6$I+:&:-! :+! +$7)1#'! =#'6&:!
4#3!9&!&1+6*@J!!
5;;&::)9)')>3!4+1)>+$)1*! I$+;&::&:! 4#3!9&! #':+! @&'I?6'! ?+$! I69');!)1:>)>6>)+1:! )1! +$7&$!
>+!%&&I!>$#;%!+?!>@&)$!&e*+=&$14&1>!S&9!:)>&:f!#;;&::)9)')>3!'&=&'J!C1!#77)>)+1-!)>!4#3!9&!
6:&?6'!?+$!;+4I#$)1*! >@&! #;;&::)9)')>3! '&=&'!+?!7)??&$&1>!S&9!:)>&:!#17!;$&#>)1*!$#1%)1*!
'):>:!#:!#1!#;;&::)9)')>3!+9:&$=#>+$3J!
()*+!&>! #'J!Q2EEhR!$&I+$>!#!:>673!+1!>@&!9&@#=)+6$!+?!>@&!Web%Accessibility%Qualitative%
Metric% (WAQM)! S@&1! &=#'6#>)1*! >@&! #;;&::)9)')>3! +?! S&9! :)>&:! S)>@! 7)??&$&1>! >++':-!
V=#'5;;&::!#17!WCDNJ!<:)1*!#6>+4#>&7!>++':!?+$!>&:>)1*!#;;&::)9)')>3!):!:&&1!93!4#13!#:!
+1&! +?! >@&! 4+:>! &??&;>)=&! S#3! >+! ;+I&! S)>@! #;;&::)9)')>3-! :)1;&! +1'3! 93! 6:)1*! >++':!
QI+::)9'3!)1!#77)>)+1!>+!@64#1!067*4&1>R!>@&!:+'6>)+1:!+6>')1&7!)1!I$&=)+6:!:;&1#$)+:!
;#1!9&!4#7&!=)#9'&J!C1! >@&! :>673-! ,MiM!S&9!I#*&:!?$+4! ,U! S&9! :)>&:! S&$&!&=#'6#>&7!
#*#)1:>! OB5.! ,JE! 6:)1*! 9+>@! >++':! #17! 4&#:6$&7! 6:)1*! WAQMJ! N@&! ;+1;'6:)+1! S#:!
>@#>! =#'6&:! I$+76;&7! 93! WAQM! +1! >@&! 9#:):! +?! 7#>#! +9>#)1&7! 93! >@&! >S+! 7)??&$&1>!
>++':! #$&! ;+4I'&>&'3! 7)??&$&1>-! 96>! >@&$&! &[):>:! #! :>$+1*! ;+$$&'#>)+1c! HI&#$4#1f:!
;+$$&'#>)+1!>&:>!+1! >@&!#;;&::)9)')>3!)17&[!I$+76;&7!93!WAQM!'&#7:! >+! jQ,MiMRkEJh,Z!
S)>@!#! @)*@! :)*1)?);#1;&! '&=&'! QIlEJ,,URJ! N@&! ;+1;'6:)=&! )1>&$I$&>#>)+1! +?! >@&! $&:6'>:! ):!
>@#>!>@&!4&>$);!;#1!9&!6:&7!?+$!$#1%)1*!S&9!I#*&:!S@&1!6:)1*!V=#'5;;&::!#17!WCDN!#17!
S)>@!$&:I&;>! >+!S&9!:)>&:!:)4)'#$!>+!>@&!+1&:!S&!>&:>&7J!<:&!+?!&)>@&$!>++'!S+6'7!'&#7!
>+! :)4)'#$! $#1%)1*! +?! S&9! I#*&:! #17! S&9! :)>&:J! N@):! ):! >@&! ;#:&! ?+$! >@&! C1?+$4#>)+1!
d&>$)&=#'!#17!>@&!5;;&::)9)')>3!"+1)>+$)1*!:;&1#$)+:-!S@&$&!+1'3!#1!+$7)1#'!:;#'&!+?!>@&!
#;;&::)9)')>3! '&=&'! ):! 1&&7&7J! C1! ?#;>-! *)=&1! >@&! &[):>)1*! ;+$$&'#>)+1-! 7&:I)>&! >@&! '#$*&!
A6#1>)>#>)=&! 7)??&$&1;&! 9&>S&&1! >@&! =#$)#9'&:-! S&! S+6'7! &[I&;>! >@#>! :)4)'#$! $&:6'>:!
S+6'7!@+'7!#':+!?+$!>++':!+>@&$!>@#1!V=#'5;;&::!#17!WCDNJ!
H&;+17'3-! ?+$! >#:%:! S@&$&! #;;&::)9)')>3! '&=&':! 1&&7! >+! 9&! 4&#:6$&7! +1! #! A6#1>)>#>)=&!
:;#'&-!')%&!)1!>@&!P5!I$+;&::&:!S)>@)1!O&9!V1*)1&&$)1*!:;&1#$)+-!>@&1!>@&!;@+);&!+?!>@&!
>++'!;+1:>$#)1:!>@&!&=#'6#>+$J!T1;&!#!>++'!):!;@+:&1-!>@&!P5!I$+;&::!1&&7:!>+!;+1>)16&!
6:)1*! >@&! :#4&! >++'J! `+! ;+4I#$):+1:! #$&! #''+S&7! S)>@! 7#>#! I$+76;&7! 93! 7)??&$&1>!
>++':J!g+S&=&$-!S@&1!+1'3!#!:)1*'&!>++'!):!6:&7-!>@&!4&>$);!):!;#I#9'&!+?!I$+76;)1*!7#>#!
>@#>! ;#1! 9&! 6:&7! >+! ?)1&'3! ;+4I#$&! 7)??&$&1>! S&9! :)>&:-! #17! I$+76;)1*! >@&$&?+$&! #!
$#1%&7!'):>J!
N@&! :>673! &[I'#)1&7! >@&:&! $&:6'>:! )1! >@&! ?+''+S)1*! >&$4:J! N++':! @#=&! #! 7)??&$&1>!
;+=&$#*&! +?! *6)7&')1&:! )1! >&$4:! +?! A6#1>)>3! #17! #;;6$#;3J! V=#'5;;&::! ;+=&$:! ?&S&$!
;@&;%I+)1>:! >@#1! WCDN! 7+&:J! C1! #77)>)+1-! WCDN! ;@&;%:! 4+$&! >&:>:! )1! #! ;@&;%I+)1>! >@#1!
V=#'5;;&::!7+&:J!N@&$&?+$&-!WCDN! 7):;+=&$:! 4+$&!#;;&::)9)')>3!9#$$)&$:! >@#1! V=#'5;;&::!
7+&:J!N@):! ):! >@&! 4#)1! $&#:+1!S@3! WCDN!3)&'7:! '+S&$! =#'6&:J! 51+>@&$! $&#:+1!4)*@>! 9&!
>@&!?#;>!>@#>!:+4&!I#$#4&>&$:!)1!WAQM!#$&!>++'!7&I&17&1>J!!
51! )4I+$>#1>! $&;+44&17#>)+1! I$+=)7&7! 93! >@&! O5C! >+! #;;&::)9)')>3! &=#'6#>+$:! ):! >+!
6:&! #>! '&#:>! >S+! >++':! >+! #6>+4#>);#''3! &=#'6#>&! >@&! ;+1?+$4#1;&! S)>@! $&:I&;>! >+!
OB5.! ,JE! *6)7&')1&! :&>! Q59+6em#@$#-! 2EEiRJ! C1! >@):! S#3-! &=#'6#>+$:! ;#1! &[#4)1&!
$&:6'>:!)17&I&17&1>'3!I$+76;&7! 93! >@&! >++':-!S@);@!)1!*&1&$#'!;+=&$!)1!7)??&$&1>!S#3:!
>@&! +??);)#'! ;@&;%I+)1>:J! N@&$&?+$&! )>! ):! '+*);#'! >+! *&1&$#')X&! :6;@! #! $&;+44&17#>)+1!
#':+! >+! I&+I'&! S#1>)1*! >+! 4&#:6$&! >@&! #;;&::)9)')>3! '&=&':J! g&1;&! >@&! )4I+$>#1;&! +?!
#::&::)1*! @+S! 46;@! #! 4&>$);! 7&I&17:! +1! >++':-! #17! >+! ?)17! )?-! #17! 617&$! S@);@!
#::64I>)+1:-!#!4&>$);!'&#7:!>+!:)4)'#$!$&:6'>:!S@&1!6:)1*!>S+!7)??&$&1>!>++':J!
N@):! I#I&$! #77$&::&:! >@&:&! )::6&:! ')4)>&7! >+! WAQM! #II')&7! >+! $&:6'>:! I$+76;&7! 93!
V=#'5;;&::!#17!WCDNJ!N@&!I#I&$!:@+S:!@+S!93!>61)1*!I#$#4&>&$:!)1!WAQM!>@&!4&>$);!
;#1!9&!4#7&!>++'!)17&I&17&1>!?+$!>@&!:;&1#$)+:!>@#>!@#=&!9&&1!?+$&:&&1J!C1!#77)>)+1-!
I#$#4&>&$!>61)1*!):!;+1>&[>6#')X&7!S)>@)1!#!4&>@+7!:+!>@#>!>@&!>61)1*!I$+;&::!;#1!9&!
$&I');#>&7!#':+!S)>@!+>@&$!&=#'6#>)+1!>++':!9&:)7&:!V=#'5;;&::!#17!WCDNJ!
2!Related!Work!on!Web!Metrics!
C1!$&;&1>!3&#$:-! #! *$+S)1*!)1>&$&:>!+1!S&9!#;;&::)9)')>3!4&>$);:!@#:!#$):&1!4)$$+$&7! )1!
:69:>#1>)#'!:;)&1>)?);!')>&$#>6$&-!S@);@!S&!9$)&?'3!$&=)&SJ!
H6'')=#1!#17!"#>:+1!Q2EEER!I$+I+:&7!>+!4&#:6$&!#;;&::)9)')>3!=)#!>@&!\?#)'6$&!$#>&]!QfrR!
9&>S&&1!#;>6#'!#17!I+>&1>)#'!I+)1>:!+?!?#)'6$&-!93!&=#'6#>)1*!#!:69:&>!+?!F!;@&;%I+)1>:!
?$+4! OB5.! ,JEJ! D+$! )1:>#1;&-! ,E! I);>6$&:! 4)::)1*! #1! #II$+I$)#>&! >&[>6#'! 7&:;$)I>)+1!
+6>!+?! ,EE! S+6'7!'&#7! >+! frkEJ,!S@)'&! U! )4#*&:!+6>! +?!2U!'&#7! >+! frkEJ2J! N@&! #6>@+$:!
;'#)4! >@#>! >@&! I+>&1>)#'! I+)1>:! +?! ?#)'6$&! Q+$! accessibility% opportunities)!:@+6'7! 9&!
I&1#')X&7!:)1;&!>@&3! 4)*@>! ;+1>#)1!#;;&::)9)')>3!9#$$)&$:J!N@6:-!#!1+$4#')X&7!)1;)7&1;&!
?)*6$&!):!;#';6'#>&7!?+$!>@&:&!accessibility%opportunitiesMJ!
.+1Xn'&X!&>! #'J!Q2EEMR!7&=&'+I&7!o5C-!S@);@!:>#17:! ?+$!\o)>!?+$!>@&!5;;&::)9)')>3!>+! >@&!
C1>&$1&>]-!#!:&>!+?!#II');#>)+1:!#)4)1*! #>! &1@#1;)1*! >@&! #;;&::)9)')>3! +?!S&9!I#*&:!?+$!
=):6#''3! )4I#)$&7! 6:&$:J! O)>@)1! >@):! ;+1>&[>-! )>! S#:! 7&=&'+I&7! #1! #II');#>)+1! >+!
4&#:6$&!>@&!#;;&::)9)')>3!'&=&'!+?!S&9!I#*&:!:+!>@#>!>@&!6:&$!;#1!%1+S!)>:!#;;&::)9)')>3!
'&=&'! 9&?+$&@#17J! 5! *$&#>! 1649&$! +?! 4&>$);:! #$&! 7&?)1&7! )1! $&*#$7! S)>@! OB5.! ,JE!
;@&;%I+)1>:J! D+$! )1:>#1;&-! >@&$&! #$&! >S+! 4&>$);:! ?+$! ;@&;%I+)1>! UJM^! percentage% of%
tables%with%summaries!#17!percentage%of% tables% with% descriptive% summariesJ! `+>! +1'3!
I&$;&1>#*&!>&$4:!#$&!6:&7!>+!7&?)1&!#!4&>$);!96>!#':+!#9:+'6>&!1649&$:!+?!)>&4:-!:6;@!
#:!number%of%colours%used%as%background!?+$!;@&;%I+)1>!2J2J!C1!#77)>)+1-!#!1+$4#')X&7!
+=&$#''!#;;&::)9)')>3! =#'6&! ):! ;#';6'#>&7! 6:)1*!>@&! O&9PV"! 4&>@+7! QT':)1#! #17!d+::)-!
3 Further explanations are not given in this regard because the way that the normalize incidence figure is
calculated is quite confusing.
2EE2RJ!N@&! ?#;>!>@#>!>@&!4&>$);:!#$&!#6>+4#>);#''3! +9>#)1&7!#17!>@#>!=):6#''3!)4I#)$&7!
6:&$:!I$+=)7&7!?&&79#;%!76$)1*!>@&!7&=&'+I4&1>!+?!>@&! I$+0&;>! #$&! >@&! :>$+1*! I+)1>:!
+?!>@):!#II$+#;@J!
D6%67#! &>! #'J! Q2EEUR! I$+I+:&7! >S+! #;;&::)9)')>3! 4&>$);:! ?+$! 9')17! 6:&$:^! navigability-!
S@);@!4&#:6$&:!@+S!S&''!>@&!S&9!;+1>&1>!):!:>$6;>6$&7!#17!listenability-!S@);@!$&'#>&:!
>+! >@&! #II$+I$)#>&1&::! +?! #'>&$1#>)=&! >&[>J! /+>@! I#$#4&>&$:! #$&! #6>+4#>);#''3!
;#';6'#>&7!)1!>@&!Accessibility%Designer!>++'!93!N#%#*)!&>!#'J!Q2EEYRJ!p&>-!>@&$&!):!1+!6:&$!
>&:>)1*! >@#>! 7&4+1:>$#>&:! >@&! =#')7)>3! +?! >@&! 4&>$);:! #17! >@&! S#3! >@&:&! 4&>$);:! #$&!
;#';6'#>&7!):!1+>!$&=&#'&7J!
/#)'&3! #17! /6$7! Q2EEUR! 6:&7! >$&&e4#I:! >+! $&I$&:&1>! >@&! #;;&::)9)')>3! '&=&'! +?! #! S&9!
:)>&J! N@&3! ;'#)4! >@#>! >@):! )1?+$4#>)+1! =):6#')X#>)+1! >&;@1)A6&! ):! 4+$&! )1>&$#;>)=&! #17!
&#:)&$! >+! ;+4I$&@&17! ?+$! >@&! S&9! :)>&! 4#)1>&1#1;&! >#:%J! V#;@! 1+7&! S)>@)1! #! >$&&!
$&I$&:&1>:! #! S&9! I#*&! #17! >@&! :)X&! #17! ;+'+6$! +?! >@&! 1+7&! =#$3! 7&I&17)1*! +1! )>:!
#;;&::)9)')>3! '&=&'-! S@);@! ):! 4&#:6$&7! S)>@! >@&! Overall% Accessibility% Metric,%
∑+
=
c
cc
ElementsAttributes
WB
OAM
!S@&$&! Bc! ):! >@&! 1649&$! +?! 9#$$)&$:! ?+617! S)>@)1!
;+1?)7&1;&! '&=&'! c% #17! Wc! ;+$$&:I+17:! >+! >@&! S&)*@>! +?! >@#>! ;+1?)7&1;&! '&=&'J! N@&$&!
#$&! ?+6$! ;+1?)7&1;&! '&=&':! 7&I&17)1*! +1! @+S! ;&$>#)1! ):! #1! &=#'6#>)+1! >++'! S@&1!
&=#'6#>)1*! #! OB5.! ,JE! ;@&;%I+)1>^! ;@&;%I+)1>:! '#9&''&7! #:! certain!S&)*@! ,E-! high!
;&$>#)1>3! ;@&;%I+)1>:! S&)*@! F-! S@)'&! low! +1&:! *&>! Y! #17! >@&! 4+:>! 61;&$>#)1! +1&:!
+9>#)1! ,J! N@&$&?+$&-! >@&! @)*@&$! >@&! ;&$>#)1>3! '&=&'! ):! >@&! 4+$&! >@&! ;+$$&:I+17)1*!
9#$$)&$!):!I&1#')X&7J!N@):!):!7)=)7&7!93!>@&!:64!+?!>@&!>+>#'!1649&$!+?!gN"W!#>>$)96>&:!
#17!&'&4&1>:!)1!#!S&9I#*&J!N@&! 4#0+$! 7$#S9#;%! +?! >@):!4&>$);!):!>@#>!>@&! =#'6&:! #$&!
619+617&7J!
N@&!Web% Accessibility%Quantitative% Metric%(WAQM)% 93!5$$6&! &>!#'J! Q2EEUR!+=&$;+4&:!
>@&!')4)>#>)+1:!+?!>@&!#9+=&4&1>)+1&7!4&>$);:!93!#6>+4#>);#''3!I$+=)7)1*!1+$4#')X&7!
$&:6'>:!>@#>!;+1:)7&$!>@&!S&)*@>:!+?!>@&!OB5.!,JE!I$)+$)>)&:-!&[I'+)>)1*!>@&!)1?+$4#>)+1!
)1! >@&! $&I+$>:! I$+76;&7! 93! #1! &=#'6#>)+1! >++'-! V=#'5;;&::! Q59#:;#'! &>! #'J-! 2EEYRJ!
V=#'6#>)+1! $&I+$>:! #$&! 9#:&7! +1! OB5.! ,JE! 96>! WAQM! #':+! I$+=)7&:! #1! #;;&::)9)')>3!
=#'6&! ?+$! &#;@! OB5.! 2JE! QB#'7S&''! &>! #'J-! 2EEFR! *6)7&')1&! QPerceivable-! Operable-!
Understandable-!RobustR! :)1;&! $&:6'>:! #$&! )1>&$?#;&7!6:)1*! #! 4#II)1*! ?61;>)+1J! T1;&!
OB5.!,JE!;@&;%I+)1>:!#$&!*$+6I&7!93!>@&)$!OB5.!2JE!4&49&$:@)I!#17!>@&)$!I$)+$)>)&:!
)1! >@&! OB5.! ,JE-! ?#)'6$&! $#>&:! #$&! ;+4I6>&7! ?+$! &#;@! :69*$+6IJ! H)1;&! ?#)'6$&! $#>&:!
>&17!>+!I)'&!6I!;'+:&!>+!E-!7):;$)4)1#>)+1!#4+1*!?#)'6$&!$#>&:!):!1+>!=&$3!&??&;>)=&J!N@6:-!
#!?61;>)+1!>+!:I$&#7!+6>!>@&:&!=#'6&:!):!#II')&7!>+!>@&!?#)'6$&!$#>&:J!5:!WAQM!$&')&:!+1!
$&I+$>:! 3)&'7&7! 93! #6>+4#>);! >++':-! ;@&;%I+)1>:! >@#>! ;#1! 9&! #6>+4#>);#''3! &=#'6#>&7!
@#=&! #! :>$+1*! )1?'6&1;&! +1! >@&! ?)1#'! :;+$&:! &=&1! )?! >@&! :&4)e#6>+4#>);! I$+9'&4:! #$&!
#':+!;+1:)7&$&7J!"+$&!7&>#)':!#9+6>!>@&!WAQM!;#1!9&!?+617!)1!>@&!1&[>!:&;>)+1J!
Metrics%for%a%Web%Site%and%Sampling%Issues%
N@&!I$&=)+6:'3!4&1>)+1&7!4&>$);:!4#3!;+1:)7&$!>@&!#;;&::)9)')>3!=#'6&!?+$!#!S@+'&!S&9!
:)>&! 96>! #$&! ?+;6:&7! +1! :)1*'&! I#*&:J! g#;%&>>! &>! #'! Q2EEYR! #17! q#$4#1>+! #17! m&1*!
Q2EEUR! I$+I+:&7! >@&! Web% Accessibility% Barrier% (WAB)% score! #)4)1*! #>! 4&#:6$)1*!
A6#1>)>#>)=&'3! >@&! #;;&::)9)')>3! +?! #! S&9! :)>&! 9#:&7! +1! 2U! OB5.! ,JE! ;@&;%I+)1>:J! C1!
&#;@! I#*&! p-! #! ?#)'6$&! $#>&! QfrR! 9&>S&&1! #;>6#'! #17! I+>&1>)#'! &$$+$:! ):! ;#';6'#>&7! ?+$!
&#;@!;@&;%I+)1>!c! #17!7)=)7&7!93!>@&!$&;)I$+;#'!+?!>@&!I$)+$)>3! +?!>@&!;@&;%I+)1>!Q,-!2!
+$! MR-!
( )
∑∑
=
p c cP priority
cpfr
N
WAB ,1
J! N@&! ?)1#'! $&:6'>! ):! 7)=)7&7! 93! >@&! >+>#'! 1649&$! +?!
S&9!I#*&:J!g)*@&$!=#'6&:!)4I'3!'+S&$!#;;&::)9)')>3!'&=&':J!N@&!4+:>!)4I+$>#1>!?&#>6$&:!
+?!>@&!4&>$);!#$&!>@#>!)>!):!#6>+4#>);#''3!;#';6'#>&7!6:)1*!#1!#6>+4#>);!&=#'6#>)+1!>++'!
#17!>@&!?#;>!>@#>!>@&!:;+I&!)1;'67&:!>@&!S@+'&!S&9!:)>&J!T1!>@&!+>@&$!@#17-!)>:!=#'6&:!
$#1*&! ):! 619+617&7! Q1+>! 1+$4#')X&7R! #17! S&)*@>)1*! @#:! 1+>! :+')7! &4I)$);#'!
?+617#>)+1:J!
C1!>@&! ;+1>&[>! +?! >@&! <1)?)&7! O&9! V=#'6#>)+1! "&>@+7+'+*3Y! Q<OV"R! :&=&$#'! 4&>$);:!
@#=&! 9&&1! I$+I+:&7! 76$)1*! )>:! 7&=&'+I4&1>! I$+;&::J! N@&! ?)$:>! I69');! 4)'&:>+1&! S#:!
<OV"! EJU! #17! >@&! '#:>! =&$:)+1! >+! 7#>&! Q"#3! 2EEFR! ):! <OV"! ,J2! 93! (&''&4#1! &>! #'J!
2EEhJ!51!&[>&1:)+1! +?! >@&!<OV"! EJU! 4&>$);!r,Es! #II')&:! >+!#! :#4I'&! +?! I#*&:!I! )1! #!
O&9!:)>&!>+!#!*)=&1!6:&$!*$+6I!6!#17!):!7&?)1&7!93!
( )
∏−−=
i
ubibSRupA 11),(
-!S@&$&!9!
$&I$&:&1>:!>@&!9#$$)&$!>3I&!#17!)!):!>@&!)7&1>)?)&$!?+$!#!9#$$)&$J!d)9!$&I+$>:!S@&>@&$!>@&!
9#$$)&$!9!#:!9&&1!7&>&;>&7!#>!'+;#>)+1!)-!>@6:!>@&!=#'6&!+?!d)9!;#1!9&!&)>@&$!E!+$!,J!N@&!
)1;)7&1;&!>@#>!#!*)=&1!9#$$)&$!@#:!+1!#!6:&$!*$+6I!):!$&I$&:&1>&7!93!>@&!:&=&$)>3!=#'6&!
H69-!$#1*)1*!?$+4!E!>+! ,! 7&I&17)1*! +1! >@&!)4I#;>J!/3!6:)1*!>@):! 4&>$);-! '+S&$! =#'6&:!
)17);#>&! @)*@&$! #;;&::)9)')>3! '&=&':J! W#>&$-! )1! <OV"! ,J2! >@&! 4&>$);! ):! :)*1)?);#1>'3!
:)4I')?)&7!#17!>@&!?#)'6$&!$#>&!):!#7+I>&7!?+$!#!:)1*'&!O&9!I#*&J!N@&!4&>$);!#II')&7!>+!
>@&! O&9! :)>&! ;+1:):>:! +?! >@&! ;#';6'#>)+1! +?! >@&! 4&#1! =#'6&! +?! &=&$3! :)1*'&! I#*&!
)1;'67&7!)1!#!:#4I'&!+?!I#*&:J!
!
/$#01)%! #17! W+46:;)+! Q2EEhR! I$+I+:&! SAMBA-! #! 4&>@+7+'+*3! >@#>! )1=+'=&:! 1+>! +1'3!
&=#'6#>)+1!>++':! 96>! #':+! &[I&$>! $&=)&S&$:! )1! >@&! ;+1>&[>! +?!>@&! Barrier% Walkthrough!
4&>@+7!7):;6::&7!)1!Q/$#01)%-!2EEic!2EEhRJ!5!:#4I'&!+?!$&:6'>:!I$+=)7&7!93!#;;&::)9)')>3!
&=#'6#>)+1!>++':!#$&!6:&7!93!#!I#1&'!+?!&[I&$>:!)1!+$7&$!>+!?)17!#;;&::)9)')>3!9#$$)&$:!?+$!
7)??&$&1>!&17!6:&$! *$+6I:c!&[I&$>:!#$&!#:%&7!>+!#::)*1!:&=&$)>)&:!>+! >@&:&! 9#$$)&$:!#17!
>@&1!#II$+I$)#>&!*&1&$#')X#>)+1:! ;#1! 9&!)1?&$$&7! ?+$! >@&! &1>)$&!S&9! :)>&J! O)>@!:6;@!#!
I$+;&::-!&[I&$>:! #':+! ;+1:)7&$! >@&! &$$+$! $#>&! +?! >@&! &=#'6#>)+1! >++'! S@);@! #??&;>:! >@&!
=#'6&:!;+4I6>&7!93!>@&!4&>$);J!5''!>@&:&!)::6&:!#$&!I6>!>+*&>@&$!)1!>@&!'#:>!:>&I!+?!>@&!
4&>@+7-!>@&!;+4I6>#>)+1!+?!#;;&::)9)')>3!)17&[&:J!d#>@&$!>@#1!4&#:6$)1*!;+1?+$4#1;&!
>+!;&$>#)1!*6)7&')1&:-!SAMBA!#)4:!#>!4&#:6$)1*!>@&!#;;&::)9)')>3!'&=&'!+?!#!S&9!:)>&!?+$!
7)??&$&1>!6:&$! *$+6I:J! 5:! '+1*! #:! #! >&:>! >+! 9#$$)&$! 4#II)1*! ?61;>)+1! ):! I$+=)7&7-! >@&!
SAMBA! 4&>@+7! ):! )17&I&17&1>! ?$+4! >@&! &=#'6#>)+1! >++'J! g+S&=&$! )>! $&4#)1:!
61&[I'+$&7!S@&>@&$!>@&!=#'6&:!I$+76;&7!93!SAMBA!#$&!)17&I&17&1>!?$+4!>@&!>++'J!
N@&!)::6&!+?!:&'&;>)1*!#!$&I$&:&1>#>)=&!:#4I'&!#$):&:!S@&1!4&#:6$)1*!>@&!#;;&::)9)')>3!
+?!'#$*&!S&9! :)>&:!>@#>!;#1!@+:>!@617$&7:!+?!>@+6:#17:!+?!S&9! I#*&:J! N@&!<OV"!,J2!
$&;+44&17:! >@#>! #''! I#*&:! :@+6'7! 9&! &=#'6#>&7! )?! #! S&9! :)>&! ;+1>#)1:! '&::! >@#1! UE!
I#*&:! #17! :#4I')1*! 4&>@+7:! :@+6'7! 9&! #II')&7! S@&1! 4+$&! S&9! I#*&:! #$&! @+:>&7J!
<1?+$>61#>&'3-!<OV"!+??&$:!1+!&[I'#1#>)+1!+1!:6;@!#!>@$&:@+'7-!1+$!#77)>)+1#'!7&>#)':!
#9+6>! >@&! :#4I')1*! 4&>@+7:! >@#>! ;#1! 9&! 7&I'+3&7J! H+4&! $&4#$%#9'&! ?)17)1*:! #$&!
7):;6::&7! )1! >@):! $&*#$7! 93! /$#01)%! &>! #'J! Q2EEhR-! S@+-! +1! >@&! 9#:):! +?! #1! &[@#6:>)=&!
&[I&$)4&1>#'! :>673! +?! #! 7+X&1! +?! :#4I')1*! 4&>@+7:! #17! >@$&&! 4&>$);:-! '&#7! >+! >@&!
;+1;'6:)+1!>@#>!>@&! :)X&! +?!>@&!:#4I'&-!>@&!4&>$);!#17!>@&!:#4I'&! 4&>@+7! @#=&!>+!9&!
4 Available at http://www.wabcluster.org/
;#$&?6''3!;@+:&1!9&;#6:&! >@&3! #$&!>)*@>'3!;+6I'&7J! C1! +>@&$!S+$7:-!>@&! 4&>$);:! =#'6&:!
#$&!:>$+1*'3!7&I&17&1>!+1!:#4I'&!:)X&!#17!+1!>@&!:#4I')1*!#'*+$)>@4J!
3!The!Web!Accessibility!Quantitative!Metric!
N@&!WAQM!93!5$$6&!&>!#'J!Q2EEUR!#)4:!#>!;#';6'#>)1*!#6>+4#>);#''3!#;;&::)9)')>3!:;+$&:!
+?! S&9! I#*&:J! 51! #II');#>)+1! >@#>! I&$?+$4:! >@):! >#:%! @#:! 9&&1! )1>&*$#>&7! )1>+! >@&!
#;;&::)9)')>3!&=#'6#>)+1!>++'! V=#'5;;&::-!>#%)1*!#7=#1>#*&!+?!)>:!4+76'#$)>3J!V=#'6#>)+1!
$&I+$>:!I$+76;&7! 93! V=#'5;;&::! #$&! >@&! )1I6>! ?+$! >@&! 4&>$);:! ;#';6'#>)+1! #II');#>)+1J!
N@6:-!>+!>@&!*$&#>&$!&[>&1>!)>!;+1:)7&$:!I$+9'&4:!>@#>!;#1!9&!#6>+4#>);#''3!?+617!#17!
>+! >@&! '&::&$! &[>&1>! >@&! :&4)e#6>+4#>);! &$$+$:! QS#$1)1*:R! >@#>! $&A6)$&! @64#1!
067*&4&1>J! T>@&$! )::6&:! >@#>! ;#11+>! 9&! 7&>&;>&7! 93! &=#'6#>)+1! >++':! :6;@! #:! OB5.!
,YJ,!;@&;%I+)1>!\6:&!>@&! ;'&#$&:>! #17!:)4I'&:>!'#1*6#*&]!#$&! 7):;#$7&7J! C>!):!I+::)9'&!
>+!6:&! >@&! WAQM! 93! 6:)1*!#! ;+44+1!&=#'6#>)+1! $&I+$>! ?+$4#>! #:! )>!S#:! I&$?+$4&7!
S)>@!WCDN!&=#'6#>)+1!>++'!)1!()*+!&>!#'J!2EEhJ!
N@&!WAQM%#::64&:!>@#>!>@&!:;+$&:!;#';6'#>&7!S)>@!>@&!4&>$);!:@+6'7!9&!1+$4#')X&7!
#17!$#>@&$!>@#1!>#%)1*!)1>+!#;;+61>!#9:+'6>&!1649&$!+?!#;;&::)9)')>3!I$+9'&4:!#!?#)'6$&!
$#>&!Q#;>6#'!&$$+$:!7)=)7&7! 93! I+>&1>)#'! &$$+$:R!):!;#';6'#>&7J!N@&! I$)+$)>3! +?! #!=)+'#>&7!
;@&;%I+)1>! )1! OB5.! ,JE! ):! #':+! ;+1:)7&$&7! #17! #?>&$! &4I)$);#'! >&:>)1*-! =#'6&:! S&$&!
#::)*1&7! >+! >@&:&! I$)+$)>)&:^! EJFE! ?+$! I$)+$)>3! ,! ;@&;%I+)1>:-! EJ,i! ?+$! I$)+$)>3! 2!
;@&;%I+)1>:!#17!EJEY!?+$!I$)+$)>3!M!;@&;%I+)1>:J!C>!S#:!+9:&$=&7!>@#>!?#)'6$&!$#>&:!>&17!
>+! I)'&! 6I! ;'+:&! >+! E! 4#%)1*! 7)??);6'>! >@&! 7):;$)4)1#>)+1! 9&>S&&1! >@+:&! I#*&:! >@#>!
+9>#)1&7!@)*@! #;;&::)9)')>3! :;+$&:J!N@&$&?+$&-! S&! >$#1:?+$4! >@&! ?#)'6$&!$#>&! #;;+$7)1*!
>+!#! @3I&$9+'&!?61;>)+1!Q:&&!D)*6$&!,R!>@#>!:I$&#7:! +6>!=#'6&:!S@);@!#$&!;'+:&!>+! E!93!
6:)1*! #! ?61;>)+1! >@#>! 9&@#=&:! #:! >@&! @3I&$9+'&! 7+&:Q:&&! D)*6$&! 2RJ! N@):! ?61;>)+1!
;@#1*&:! 7&I&17)1*! +1! >@&! =#'6&:! +?! =#$)#9'&:! a! #17! b! S@);@! @#=&! >+! 9&! >61&7!
7&I&17)1*!+1!>@&!&=#'6#>)+1!>++'J!
!
!
Figure!1.!Ideal!hyperbole!that!discriminates!
failure!rates!close!to!0!
!
!
Figure!2.!An!approach!to!the!hyperbole!depicted!
in!Figure!1!
!
/&:)7&:! #1! #=&$#*&! #;;&::)9)')>3! :;+$&-! #! =#'6&! ?+$! &#;@! OB5.! 2JE! *6)7&')1&!
QPerceivable-! Understandable-! Operable-! RobustR! ):! #':+! I$+=)7&7! &=&1! )?! &=#'6#>)+1:!
#$&! ;#$$)&7! +6>! #*#)1:>! OB5.! ,JE! *6)7&')1&:! :)1;&! #! 4#II)1*! ?61;>)+1U! #''+S:!
)1>&$?#;)1*!>@&!$&:6'>:J!
<:)1*! &=#'6#>)+1! $&I+$>:! $&>6$1&7! 93! V=#'5;;&::! 4#%&:! )>! &#:3! >+! *#>@&$! #''! >@&!
1&;&::#$3!7#>#!?+$!4&>$);!;#';6'#>)+1-!:6;@!#:!;@&;%I+)1>!1#>6$&!Q#6>+4#>);#''3!?+617!
+$!#!S#$1)1*!>@#>!:@+6'7! 9&! ;@&;%&7! 93!#1!&[I&$>R-!>@&!I+>&1>)#'! &$$+$:! Qt! =#'6&R-!>@&!
>)4&:!&#;@!>&:>!?#)':!>+!;+4I'3!S)>@!>@&!*6)7&')1&:!7&?)1)>)+1!Qe!=#'6&R-!#17!)>:!I$)+$)>3J!
5''! >@&:&! I#$#4&>&$:! #$&! *$+6I&7! )1! 2! *$+6I:! Q#6>+4#>);! #17! S#$1)1*! )::6&:RJ! V#;@!
*$+6I!;+1>#)1:!,2! :69*$+6I:! ;'#::)?)&7! 93!>@&)$!I$)+$)>3!)1!OB5.!,JE!QM!I$)+$)>)&:R!#17!
>@&)$! 4&49&$:@)I! )1! &#;@! OB5.! 2JE! *6)7&')1&! QqT<dR! #;;+$7)1*! >+! >@&! I$&=)+6:'3!
4&1>)+1&7!4#II)1*J!N@&!A6#1>)>#>)=&!#;;&::)9)')>3!4&>$);!):!;#';6'#>&7!93!>@&!?+''+S)1*!
#'*+$)>@4^!
!
%
%for i n each checkpoint in a guideline {P,O,U,R} loop
for j in each type of checkpoint {automatic, warning} loop
for k in each priority{1,2,3} loop
x'=calculate_x'_point(a,b)
if (failure_rate(e,t)<x’) then
Aijk=calculate_S_line(b, e ,t)
else
Aijk=calculate_V_line(a, e, t)
end loop
(1)
ijk
k
kij AwA ∑
=
×=
3
1
end loop
(2)
i
j
ijij
iN
AN
A
∑×
=
end loop
(3)
N
AN
Ai
ii
∑×
=
S@&$&![f!I+)1>-!H!')1&!#17!(!')1&!#$&!;#';6'#>&7!)1!>@&!?+''+S)1*!S#3^!
!
b
a
a
x100
100
'
−
−
=
!
100
100 +
!
"
#
$
%
&−
×=bt
e
A
!
a
t
e
aA +
!
"
#
$
%
&×−=
!
!
[f!;#';6'#>)+1!
H!')1&!
(!')1&!
!
Aijk!):!>@&!#;;&::)9)')>3!:;+$&!*)=&1!93!&)>@&$!H!')1&!+$!(!')1&!S@)'&!)1!step!(1)!Aij!;+1:)7&$:!
>@&! I$)+$)>)&:! +?! =)+'#>&7! ;@&;%I+)1>:! 46'>)I'3)1*! 5)0%! :;+$&:! 93! >@&)$! ;+$$&:I+17)1*!
S&)*@>)1*:!wk!S@&$&! k1kEJF-! k2kEJ,i! #17! k3kEJEY!+9>#)1)1*! =#'6&:! :6;@! #:! AP,automaticJ!
N@):! I#$>);6'#$! ;#:&! 4&#1:! >@#>! =#'6&:! #$&! ;#';6'#>&7! ?+$! #6>+4#>);#''3! &=#'6#>&7!
;@&;%I+)1>:!)1!>@&!q&$;&)=#9'&!*6)7&')1&J!C1! step! (2)-! #1! #=&$#*&! =#'6&!?+$!&#;@!qT<d!
*6)7&')1&!):!;#';6'#>&7!93!S&)*@>)1*!Aij!=#'6&:!S)>@!>@&!1649&$!+?!#6>+4#>#9'&!&$$+$:!
Ni,automatic! #17! S#$1)1*:! Ni,warning! )1! &#;@! i! *6)7&')1&J! D)1#''3-! )1! step! (3)! #1! +=&$#''!
#;;&::)9)')>3! =#'6&! ):! +9>#)1&7! S&)*@>)1*! &#;@! qT<d! *6)7&')1&! S)>@! >@&! 1649&$! +?!
;@&;%I+)1>:!>@&3!;+1>#)1-!NiJ!
4!Tuning!the!Parameters!
U!5=#)'#9'&!#>!@>>I^__SSSJSMJ+$*_Nd_2EEi_OGeOB5.2Ee2EEiEY2h_#II&17)[GJ@>4'!
N@&!*+#'! +?!I#$#4&>&$! >61)1*!):! >+! ?)17! Q#! :&>! +?R! #7&A6#>&! =#'6&:! ?+$! a! #17! b! )1! >@&!
I$&=)+6:'3! 4&1>)+1&7! @3I&$9+'&! #II$+#;@! >@#>! 4)1)4)X&! >@&! #;;&::)9)')>3! :;+$&!
4&#:6$&7! S)>@! WAQM! 6:)1*! &=#'6#>)+1! $&I+$>:! I$+76;&7! 93! WCDN! #17! V=#'5;;&::J! C1!
+>@&$! S+$7:-! ?)17)1*! :4#''! =#'6&:! +?!
EvalAccessLI FT WAQMWAQMdiff −=
J! C7&#''3! >@&!
9&:>!a!#17!b!=#'6&:!S+6'7!9&!>@+:&!S@+!4#%&!diffkEJ!N@&$&?+$&-!S&!;@+:&!#!:&>!+?!S&9!
I#*&:! #17! ;#';6'#>&7! WAQM! S)>@! 7)??&$&1>! =#'6&:! +?! a! #17! b! #17! +9:&$=&! S@);@!
;+49)1#>)+1!+?!>@&:&!=#'6&:!?+$!WCDN!#17!V=#'5;;&::!4#%&:!4)1)464!>@&!=#'6&!+?!diffJ!
N@):! )4I')&:! >@#>! ;+4I6>)1*! #''! >@&! I+::)9)')>)&:! ):! A6)>&! >)4&! ;+1:64)1*i-!
)__( 4
nsamplesofnumberO×
J! H)1;&! a! #17! b! I#$#4&>&$:! #$&! ;+1>)16+6:! >@&3! @#=&!
9&&1!7):;$&>)X&7! )1!+$7&$!>+!')4)>!>@&!;+4I6>#>)+1:!+?!>@&!#II$+#;@!>+!>@&! @3I&$9+'&-!
)1!+>@&$!S+$7:!>@&!=#'6&!+?!nJ!N@&!4&>@+7!):!I&$?+$4&7!#:!?+''+S:^!
,J G):;$&>)X&!
[ ]
100..0∈a
! #17!
[ ]
1..0∈b
! I#$#4&>&$:! )1! >@&! ?+''+S)1*! S#3^!
[ ]
95..5∈a
!S)>@!#! :>&I! +?!,E! 61)>:! :+!>@&$&! #$&! ,E! =#'6&:!QU-! ,U-! 2U-JJ-!ZUR! #17!
[ ]
95.0..05.0∈b
! S)>@! #! :>&I! +?! EJ,! 61)>:! +9>#)1)1*! #1+>@&$! ,E! =#'6&:! QEJEU-!
EJ,U-JJ-EJZUR!
2J H&'&;>!#! :#4I'&!+?! S&9! I#*&:! ?$+4! 7)??&$&1>! S&9! :)>&:J! q$&?&$#9'3! S&9! I#*&:!
>@#>!S+6'7!9&!7)??&$&1>!+1&!?$+4!#1+>@&$!:)1;&!@)*@!=#$)#9)')>3!):!I$&?&$$&7J!
! !
!for i from 5 to 95 (step 10) loop
for j from 0.5 to 0.95 (step 0.1) loop
for k from 5 to 95 (step 10) loop
for l from 0.5 to 0.95 (step 0.1) loop
for m in all_samples loop
results_ea(m)=WAQM_with_EvalAccess(m,i,j)
results_lift(m)=WAQM_with_LIFT(m,k,l)
end loop
calculate_mean(results_ea)
calculate_mean(results_lift)
end loop
end loop
end loop
end loop!
Proving%the%validity%of%the%sample%
5:! )>! S#:! &4I)$);#''3! =#')7#>&7! 93! /$#01)%! &>! #'J! Q2EEhR-! #;;&::)9)')>3! =#'6&:! 4&#:6$&7!
S)>@!WAQM! +1! #!:)1*'&! S&9!I#*&! ):! A6)>&! I$&7);>)=&! +?!>@&! #;;&::)9)')>3! =#'6&! +?! >@&!
S@+'&! S&9! :)>&J! D6$>@&$4+$&-! :)1;&! `)&':&1! #17! N#@)$! Q2EEER! :>#>&! >@#>! >@&! 6:#9)')>3!
'&=&'!+?!>@&!@+4&I#*&!):!A6)>&!I$&7);>)=&!+?!>@&!*'+9#'!#;;&::)9)')>3!+$!6:#9)')>3!'&=&'!+?!
#! S&9! :)>&-! S&! ;@+:&! >@&! @+4&! I#*&:! +?! ,U! S&9! :)>&:! S@);@! @#7! 9&&1! I$&=)+6:'3!
7+S1'+#7&7! #17! :>+$&7J! g+S&=&$! S&! #)4&7! #>! 7&4+1:>$#>)1*! &4I)$);#''3! >@#>! >@&!
:#4I'&!S#:!1+>!9)#:&7J!N@6:-!S&!;#>&*+$)X&7!>@$&&!S&9!:)>&:!?$+4!>@&!+1&:!&=#'6#>&7!
)1! ()*+! &>! #'J! Q2EEhR! )1! >@$&&! #;;&::)9)')>3! '&=&':^! @)*@! #;;&::)9)')>3! QB)>3! <1)=&$:)>3R-!
4&7)64! #;;&::)9)')>3! QG617&&! <1)=&$:)>3R! #17! '+S! #;;&::)9)')>3! Qq$&>+$)#! <1)=&$:)>3R!
#17! :&'&;>&7! >@&! @+4&! I#*&! #17! #! $#17+4! I#*&! Q6:)1*! #! 61)?+$4! $#17+4! :#4I')1*!
#'*+$)>@4R! S)>@)1! >@&! :#4&! S&9! :)>&! )1! +$7&$! >+! ;+4I#$&! >@&! =#'6&:! I$+76;&7! 93!
WAQM!#:!a!#17!b!I#$#4&>&$:!;@#1*&7J!
i!<:)1*!#!qB!S)>@!q&1>)64!Y!M.gX!I$+;&::+$!#17!,./!+?!d5"!)>!>++%!#'4+:>!22!@+6$:!>+!I$+;&::!,U!
I#*&:!,EEEE!>)4&:J!
N6I'&:!*)=&1!93!Qa-! b-! WAQMEvalAccess-! WAQMLIFT-!diffR!S@&$&!:&'&;>&7! 93! $&:>$);>)1*! >+!
>@&!,t!I&$;&1>)'&!S)>@!$&:I&;>!>+!diff! Q)J&J! ,t! +?! >@&! >6I'&:! S&$&! ;@+:&1! >@#>! @#7! >@&!
:4#''&:>!diff!S@);@!;+1:):>&7!+?!,EM!#17!,FY!>6I'&:!+6>!+?!iEE!?+$!@+4&I#*&:!#17!>@&!
$#17+4! I#*&! $&:I&;>)=&'3RJ! N@&! @):>+*$#4:! ?+$! I#$#4&>&$! a! )1! D)*6$&! M! #17! ?+$!
I#$#4&>&$!b!)1!D)*6$&!Y! I$+=)7&! #! *$#I@);#'! 7&:;$)I>)+1! +?! >@&! 7):>$)96>)+1! +?!#!#17!9J!
<II&$!@):>+*$#4:!;+$$&:I+17!>+!$&:6'>:!?+$!@+4&I#*&:!6:)1*!WCDN!Q'&?>R!#17!V=#'5;;&::!
Q$)*@>R! S@)'&! >@&! @):>+*$#4:! #>! >@&! 9+>>+4! ;+$$&:I+17! >+! #! $#17+4! I#*&! 4&#:6$&7!
S)>@!WCDN!Q'&?>R!#17!V=#'5;;&::!Q$)*@>RJ!
!
!
!
Figure!3.!Histograms!for!a!parameter!
Figure!4.!Histograms!for!b!parameter!
!
d&:6'>:! :@+S! >@#>! ?+$! V=#'5;;&::! >@&$&! ):! 1+! :)*1)?);#1>! 7)??&$&1;&! 9&>S&&1! >@&! 7#>#!
9#:&7!+1! >@&!@+4&I#*&! #17! >@#>! 9#:&7! +1! >@&! $#17+4'3! :&'&;>&7! +1&-! #:! ?#$! #:! >@&!
7):>$)96>)+1!+?!a!+$! b! ):!;+1;&$1&7!?+$!>@&!9+>>+4!,t!+?!>@&!7#>#! S)>@! $&:I&;>!>+!diffJ!
O@&1!)>!;+4&:!>+!WCDN-!@+S&=&$-!>@&$&!):!#!:)*1)?);#1>!7)??&$&1;&!9&>S&&1!7#>#!7&$)=&7!
?$+4! >@&! >S+! >3I&:! +?! I#*&:J! C1! ?#;>! O&';@f:! NS+! H#4I'&! >&:>! :@+S:! >@#>! >@&$&! ):! #!
7)??&$&1;&!?+$!a!Q"kYE!?+$!@+4&!I#*&:!=:!"kYZ!?+$!$#17+4!I#*&:R!#17!b!Q"kEJ2Y!?+$!
@+4&!I#*&:!=:J!"kEJM!?+$!$#17+4!I#*&:R!?+$!WCDN!96>!>@&!#;>6#'!7)??&$&1;&!):!$&'#>)=&'3!
:4#''!QZt!#17!it!$&:I&;>)=&'3RJ!5:! #! $&:6'>-! S@&1!'++%)1*!#>!>@&!>6I'&:! >@#>! 4)1)4)X&!
>@&!7)??&$&1;&:-!>@&!?#;>+$! @+4&! I#*&_$#17+4! 7+&:!1+>!I'#3! #13! $+'&! ?+$!V=#'5;;&::J!
517!&=&1!)?!>@&$&! #$&! 7)??&$&1;&:!?+$!WCDN-!>@&3!#$&!$&'#>)=&'3!:4#''J!H)4)'#$!$&:6'>:!#$&!
+9>#)1&7!)?!$&:6'>:!#$&!$&:>$);>&7!>+!>@&!9+>>+4!Ut!I&$;&1>)'&!+?!>@&!7#>#!S)>@!$&:I&;>!
>+!diffJ!
/&:)7&:-!)>!;#1!9&!;+1;'67&7! ?$+4! >@&:&! 7#>#! >@#>!9&:>!;+49)1#>)+1:!>+!4)1)4)X&! diff!
#17!7)??&$&1;&:!9&>S&&1!>++':!#$&^!
• aLIFTkYU!-%bLIFTkEJ,U,!aEvalAccesskYU!#17%bEvalAccesskEJ,U!
• aLIFTkiU!-%bLIFTkEJEU,!aEvalAccesskiU!#17%bEvalAccesskEJEU!
Observing%the%Behaviour%of%a%and%b%
T1;&!>@#>!>@&!,U!@+4&I#*&:!:#4I'&!S#:!=#')7#>&7-!>@&!#'*+$)>@4!>+!;+4I6>&!WAQM!
S)>@! V=#'5;;&::! #17! WCDN! ;#1! 9&! $61J! C1! #77)>)+1-! S&! ;#1! #':+! #1#'3:&! $&:6'>:! #17!
+9:&$=&!@+S! I#$#4&>&$:! a! #17! b!9&@#=&! S)>@!9+>@! >++':! )1! +$7&$! >+!617&$:>#17! >@&!
$+'&!>@&:&!I#$#4&>&$:!I'#3!>@$+6*@+6>!>@&!S@+'&!$#1*&!+?!=#'6&:J!H)1;&!>@&!I$+;&::!):!
A6)>&! >)4&! ;+1:64)1*-! $&4+=)1*! $#1*&:! +?! =#'6&:! >@#>! 7+! 1+>! )1?'6&1;&! +1! >@&! ?)1#'!
$&:6'>! ):! 6:&?6'! )1! +$7&$! >+! :#=&! >)4&! >@&! 1&[>! >)4&! I#$#4&>&$:! #$&! >61&7! ?+$! #! 1&S!
>++'J!T1;&!#*#)1-!:&'&;>)1*!>@&!>6I'&:!93!$&:>$);>)1*!>+!>@&!,t!I&$;&1>)'&!S)>@!$&:I&;>!>+!
7)??-!S&!+9>#)1!>@&!$&:6'>:!:@+S1!)1!D)*6$&:!U!#17!iJ!!
!
Figure!5.!Boxplots!corresponding!to!aLIFT,!bLIFT,!aEvalAccess,!bEvalAccess,!diff!and!mean!of!the!WAQM!values!
for!the!15!homepages!calculated!with!EvalAccess!and!LIFT!for!the!1%!of!tuples!containing!the!
smallest!values!of!diff4
N@&! $)*@>4+:>! 9+[! I'+>:! )1! D)*6$&! U! :@+S! >@#>! >@&! $#1*&! +?! 4&#1:! +?! WAQMEvalAccess!
#17! +?! WAQMLIFT!):! rUY-hEs-! #17! >@#>! UEt! +?! >@&! =#'6&:! #$&! ;+1;&1>$#>&7! )1! >@&!
$&'#>)=&'3! :4#''! )1>&$=#'! riU-hEs! Q*)=&1! 93! >@&! ?)$:>! #17! >@)$7! A6#$>)'&:R! ?+$! 9+>@! >++':J!
q#$#4&>&$!a! 7+&:! 1+>!I'#3! #!$&'&=#1>! $+'&! )1! +9>#)1)1*! diff!=#'6&:! >@#>! #$&! ;'+:&! >+!E!
S@&$&#:! b! 7+&:J! C1! ?#;>-! aLIFT! #17! aEvalAccess%:I#1! #! $&'#>)=&'3! '#$*&! $#1*&J! bLIFT! #17!
bEvalAccess%:I#1! >@&! &1>)$&! $#1*&-! 96>! S@)'&! >@&! ?+$4&$! ):! ;+1;&1>$#>&7! )1! >@&! $#1*&!
rEJUU-EJZUs! S)>@! #! 4&#1! +?! EJhE-! >@&! '#>>&$! ;+1;&1>$#>&:! )1! rEJMU-! EJFUs! S)>@! #! 4&#1!
&A6#'!>+!EJUZJ!
5II$+I$)#>&! =#'6&:! ?+$! a% #17! b% #$&! >@&! 4&#1! =#'6&:! +?! &#;@! =#$)#9'&^! aLIFTkUF-%
bLIFTkEJhE,!aEvalAccesskYF!#17%bEvalAccesskEJUZJ!5!4+$&!7&>#)'&7!#1#'3:):!+?!>@&!9&@#=)+6$!+?!
bLIFT!#17!bEvalAccess!S@&1!diff!*&>:!;'+:&!>+!E!;#1!9&!+9:&$=&7!)1!D)*6$&!iJ!
!
!
Figure!6.!Scatter!plot!for!bEvalAccess!(x!axis)!and!bLIFT4(y!axis)!
N@&!:;#>>&$!I'+>!)1!D)*6$&!i! :@+S:! >@&! :@#I&! +?! >@&! $&'#>)+1! 9&>S&&1! bEvalAccess!Q[!#[):R!
#17! bLIFT! Q3! #[):R! S@&1! $&:>$);>&7! >+! >@&! :69:&>:! S@&$&! diff!):! :4#''&$! >@#1! >@&! ,t!
I&$;&1>)'&J! N@&! ;+$$&'#>)+1! >@#>! ;#1! 9&! )7&1>)?)&7! 9&>S&&1! >@&! >S+! =#$)#9'&:! ):!
;+1?)$4&7!93!>@&!HI&#$4#1f:!$#1%!;+$$&'#>)+1!>&:>!$@+!@#:!#!=#'6&!+?!EJhi!S)>@!QIl2J2e
,iRJ! N@6:-! &=&1! )?! >@&! >S+! =#$)#9'&:! #$&! 1+>! 7)$&;>'3! ;+$$&'#>&7! >@&$&! ):! #! :>$+1*!
;+$$&'#>)+1!+1!>@&!$#1%:!+?!>@&)$!=#'6&:-!S@);@!&1>#)':!>@#>!>@&3!4+=&!:31;@$+1+6:'3J!
%
Using%Test%Files%
C1!+$7&$!>+!@#=&!4+$&!)1?+$4#>)+1!#9+6>!>@&!&??&;>!+?!>@&:&!I#$#4&>&$:!#17!>+!9&!#9'&!
>+!:&'&;>! >@&! 9&:>! =#'6&:!>@&! :#4&! >61)1*!I$+;&76$&! S#:! ?+''+S&7!6:)1*! #! :&>!+?! >&:>!
?)'&:J!N@&:&!>&:>!?)'&:!@#=&!#!I$&e7&>&$4)1&7!?#)'6$&!$#>&^!!
• Low%Accessibility%level%web%page% (LA)! ;+1>#)1:! )4#*&:!S)>@+6>!>&[>!&A6)=#'&1>-!
>#9'&:! S)>@+6>! :644#$3-! :+4&! ')1%:! S@);@! +I&1! I+Ie6I! S)17+S:-! #6>+e
$&?$&:@)1*!#17!)1;+$$&;>!7+;64&1>!'#1*6#*&!7&?)1)>)+1J!
• Accessible%web%page%(A)!;+1>#)1:!>@&!:#4&!&'&4&1>:!#17!#>>$)96>&:!96>!>@&3!7+!
1+>! ;#6:&! #13! #;;&::)9)')>3! &$$+$^! )4#*&:! @#=&! >&[>! &A6)=#'&1>-! >#9'&:! @#=&!
:644#$3-! ')1%:! 7+! 1+>! +I&1! 1&S! S)17+S:-! >@&$&! ):! 1+! #6>+e$&?$&:@! #17!
'#1*6#*&!):!S&''!7&?)1&7J!
• V'&4&1>:!)1! Medium% Accessibility%level% web% page% (MA)! #$&!>@&! :#4&! #:!)1! >@&!
Low%Accessibility!?)'&!96>!@#'?!+?!>@&!I+>&1>)#'!&$$+$:!#$&!#;>6#'!&$$+$:J!
• C1!Worse%than%MA-!M_Y!+?!>@&!I$&=)+6:'3!4&1>)+1&7!I+>&1>)#'!&$$+$:!#$&!#;>6#'!
&$$+$:J!
• Better% than% MA!):! ;+4I+:&7! +?! >@&! :#4&! &'&4&1>:! 96>! ,_Y! +?! >@&4! @#=&! #1!
#;>6#'!&$$+$J!
• Empty% web% page% E% +1'3! ;+1>#)1:! >@&! 1&;&::#$3! :>$6;>6$#'! gN"W! >#*:! S)>@+6>!
#13!;+1>&1>!&'&4&1>J!
5?>&$! #II'3)1*! 9+>@! >++':! >+! >@&! >&:>! ;+''&;>)+1-! #17! ;+4I6>)1*! >@&! ;+$$&:I+17)1*!
WAQM! =#'6&:! ?+$! #''! ;+49)1#>)+1:! +?! a! #17! +?! b,% S&! ?+617! >@#>! 1+1&! +?! >@&! >6I'&:!
3)&'7&7!#!=#'6&!?+$!diff!&A6#':! >+! EJ! g+S&=&$!>@&!9&:>!;+49)1#>)+1:!#$&!>@&!?+''+S)1*!
+1&:^!
• aLIFTkU!-%bLIFTkEJEU,!aEvalAccesskFU!#17%bEvalAccesskEJFU!
• aLIFTkU-%bLIFTkEJEU,!aEvalAccesskFU%#17%bEvalAccesskEJZU!
H644#$)X)1*-!6:)1*!7)??&$&1>!4&>@+7:!7)??&$&1>!=#'6&:!?+$!a!#17!b!@#=&!9&&1!+9>#)1&7J!
N#9'&!,!:@+S:!>@&:&!=#'6&:!#:!S&''!#:!:+4&!$&4#$%:!#9+6>!>@&)$!?&#>6$&:J!
!
!
!
!
!
!
!
Table!1.!Best!values!for!a!and!b!produced!by!different!methods!
"&>@+7!
aLIFT%
bLIFT%
aEvalAccess%
bEvalAccess%
$&4#$%:!
g+4&I#*&:!
UF!
EJhE!
YF!
EJUZ!
e!
H#4I'&!
YU!
EJ,U!
YU!
EJ,U!
=#'6&:!>@#>!4)1)4)X&!diff!9&>S&&1!>@&!I#*&:!
#17!
EvalAccessLI FT aa −
!#17!
EvalAccessLI FT bb −
!
iU!
EJEU!
iU!
EJEU!
N&:>!?)'&:!
U!
EJEU!
FU!
EJFU!
1+1&!+?!>@&!>6I'&:!'&#7:!>+!diff!kE!
U!
EJEU!
FU!
EJZU!
H)1;&!$&:6'>:!#$&!A6)>&!;+1>$#7);>+$3-!>@&!,U!S&9!I#*&:!@#=&!9&&1!4&#:6$&7!)1!+$7&$!
>+!+9:&$=&!S@);@!=#'6&:!+9>#)1!>@&!$&:6'>:!>@#>!'&#7!WAQM!>+!9&!>++'!)17&I&17&1>!?+$!
#''!>@&!:;&1#$)+:J!
5!Recalculation!with!new!values!
N@&!#)4!+?!>@&!$&;#';6'#>)+1!):!>+!;+4I#$&!1&S!$&:6'>:!S)>@!>@&!+1&:!+9>#)1&7!)1!()*+!
&>! #'J! 2EEh! S@&$&! HI&#$4#1f:! $#1%)1*! $@+! QjQ,MiMRkEJh,ZR! I$+=&7! >@#>! >@&$&! S#:! #!
:>$+1*!;+$$&'#>)+1!9&>S&&1!=#'6&:!$#1%&7!93!WAQMEvalAccess!#17!WAQMLIFT!6:)1*!ak2E!
#17!bkEJM!?+$!9+>@!>++':J!g+S&=&$!$&:6'>:!S&$&!:+!7)??&$&1>!>@#>!>@&!6:&!+?!>@&!4&>$);!
)1!>@&! P6#')>3! 5::6$#1;&! :;&1#$)+! S#:! ;+1:>$#)1&7! 93! >@&! >++'J! C>! ;#1!9&! +9:&$=&7!)1!
D)*6$&!h!#!4#$%&7! 7)??&$&1;&! 9&>S&&1! >@&4J! 5! 9+[! I'+>! :@+S:! >@&! 7):>$)96>)+1! +?! >@&!
=#'6&:! 93! @)*@')*@>)1*! >@&! 4&7)#1! Q>@&! >@);%! @+$)X+1>#'! ')1&R-! >@&! ,:>! #17! M$7! A6#$>)'&!
Q>@&!9+>>+4!#17!>+I!+?!>@&!9+[R-!#17!>@&!+6>')&$:!Q=#'6&:!9&3+17!>@&!S@):%&$:RJ!C>!;#1!
9&! $&#7)'3! :&&1! >@#>! WAQMEvalAccess! Q>@&! '&?>4+:>! 9+[! I'+>R! ):! 46;@! @)*@&$! >@#1!
WAQMLIFT!Q9+[!I'+>!)1!>@&!;&1>$&R^!4+$&!>@#1!hUt!+?!>@&!I#*&:!&=#'6#>&7!93!WCDN!@#=&!
#1!:;+$&! >@#>! ):! :4#''&$! >@#1!Y2-! S@&$&#:! ?&S&$! >@#1! 2Ut! +?! >@&!I#*&:! &=#'6#>&7! 93!
V=#'5;;&::! @#=&! #! :;+$&! '+S&$! >@#1! UYJ! "&7)#1! =#'6&! ?+$! WAQMEvalAccess!):! iZ-! ?+$!
WAQMLIFT! )>! ):! 2FJ! C1! #77)>)+1! S&! ;#1! :&&! >@#>! >@&! :I$&#7! +?! WAQMEvalAccess! ):! '#$*&$!
>@#1!WAQMLIFT!Q>@&!7):>#1;&!9&>S&&1!>@&!A6#$>)'&:!?+$!WAQMEvalAccess!):!Mh-!S@&$&#:!)>!
):!,i!e!'&::! >@#1! @#'?! e! ?+$! WAQMLIFT!RJ! WAQMEvalAccess!$&#;@&:!,EE-!S@&$&#:!WAQMLIFT!
$&#;@&:!F2c! 9+>@! :>#$>! ?$+4! ,,J! C>! ;#1! 9&! :&&1! )1! $)*@>4+:>!9+[! I'+>!>@#>! UEt!+?! >@&!
I#*&:!@#=&!#! 7)??&$&1;&!
EvalAccessLI FT WAQMWAQMdiff −=
!>@#>!')&:! 9&>S&&1! 2h! #17!
Yh!Q4&7)#1kYERJ!
0
10
20
30
40
50
60
70
80
90
100
A_eval A_lift diff
!
Figure!7.!Box!plots!show!the!clear!difference!between!WAQMEvalAccess4(left)4and4WAQMLIFT!(center)4
C1! #77)>)+1-! @):>+*$#4:! )1! D)*6$&! F! #17! D)*6$&! Z! :@+S! >@#>! S@)'&! WAQMEvalAccess!):!
1&*#>)=&'3!:%&S&7!WAQMLIFT!)>!):!I+:)>)=&'3J!
0
50
10 0
150
200
250
300
350
400
10 20 30 40 50 60 70 80 90 10 0
!
Figure!8.!Histogram!of!absolute!frequency!of!
WAQMEvalAccess!
0
50
100
150
200
250
300
350
400
450
500
10 20 30 40 50 60 70 80 90 100
!
Figure!9.!Histogram!of!absolute!frequency!of!
WAQMLIFT!
%
Recalculation%of%Results%
N@&!,U!S&9! :)>&:! ;+1>#)1)1*!#! >+>#'! 1649&$! +?!,YYZ! S&9! I#*&:!@#=&!9&&1! 4&#:6$&7!
#*#)1! S)>@! 9+>@! >++':! 96>! $&:>$);>&7! >+! >@&! a! #17! b! =#'6&:! +9>#)1&7! )1! >@&! I$&=)+6:!
:&;>)+1J!`+>! +1'3!:)4)'#$!$&:6'>:!S#1>!>+!9&!I$+76;&7! 96>!S&!#':+!#)4!#>!;@&;%)1*! >@&!
9&@#=)+6$! +?! >@&! 4&>$);! )1! >@+:&! :;&1#$)+:! >@#>! $&A6)$&! #1! +$7)1#'! :;#'&! +?! >@&!
#;;&::)9)')>3!'&=&'J!N@6:-!HI&#$4#1f:!;+$$&'#>)+1:!#$&!#':+!;#';6'#>&7!)1!&#;@!;#:&J!N#9'&!
2!:@+S:! >@#>!>@&!:>$+1*&$!;+$$&'#>)+1!;#1!9&! ?+617!)1!>6I'&!uM!S@&$&!akiU!#17!bkiU-!
jkEJih!QIl2J2e,iR!S@);@!):!#!4+7&$#>&!;+$$&'#>)+1J!N6I'&:!u,!#17!u2!#':+!:@+S!#!:)4)'#$!
9&@#=)+6$! &=&1! )?! >@&! ;+$$&'#>)+1! ):! :')*@>'3! '+S&$J! N6I'&:! ;+$$&:I+17)1*! >+! =#'6&:!
+9>#)1&7! 93! 6:)1*! >&:>! ?)'&:! :@+S! >@&! '+S&$! ;+$$&'#>)+1! S@);@! ):! #$+617! EJUE! Q'+Se
4+7&$#>&R!96>!:>)''!:)*1)?);#1>J!
Table!2.!Tuples!with!a!and!b4values!and!Spearman’s!rank!correlation!as!well!as!data!to!interpret!the!
data!distribution.!Note!that!the!upper!row!in!each!tuple!corresponds!to!EvalAccess!while!the!lower!
one!correspond!to!values!for!LIFT%
>6I'&!
aLIFT%
bLIFT%
aEA%
bEA%
HI&#$4#1!j!Q,YYZR!
4)1!
,:>!P!
"&7)#1!
M$7!P!
4#[!
"&#1!
u,!
UF!
EJhE!
YF!
EJUZ!
EJiY-!Il2J2e,i!
2!
UY!
65!
Z,!
,EE!
iF!
,Y!
UZ!
64!
h2!
ZZ!
iY!
u2!
YU!
EJ,U!
YU!
EJ,U!
EJi,-!Il2J2e,i!
2!
Y2!
i,!
FZ!
,EE!
i,!
,M!
YE!
YF!
iE!
ZF!
YZ!
uM!
iU!
EJEU!
iU!
EJEU!
EJih-!Il2J2e,i!
M!
MY!
UU!
FU!
,EE!
Uh!
,Y!
YM!
YF!
UF!
ZU!
UE!
uY!
U!
EJEU!
FU!
EJFU!
EJU,-!Il2J2e,i!
Z!
Ui!
iZ!
Z,!
,EE!
h,!
,!
,E!
2E!
YY!
ZU!
2i!
uU!
U!
EJEU!
FU!
EJZU!
EJUE-!Il2J2e,i!
,2!
UF!
ih!
Z,!
,EE!
h,!
,!
,E!
,F!
YY!
ZU!
2U!
N@&!>6I'&!>@#>!:@+S:!>@&!4+$&!:)4)'#$!7):>$)96>)+1!9&>S&&1!:>#>):>);#'!=#'6&:!I$+76;&7!
93!WCDN!#17!V=#'5;;&::!):!>@&!;$)>&$)+1!>+!:&'&;>!>@&!4+:>!#7&A6#>&!a!#17!b!I#$#4&>&$:J!
N@&!,:>!A6#$>)'&-!M$7!A6#$>)'&!#17!>@&!4&7)#1!#$&!&:I&;)#''3!;+1:)7&$&7J!H)1;&!>6I'&:!uY!
#17! uU! :@+S! >@#>! >@&:&! =#'6&:! #$&! ;+4I'&>&'3! 7)??&$&1>! >@&3! #$&! 7):;#$7&7J! T1! >@&!
+>@&$!@#17-!)1!>@&!$&:>!+?!>@&!>6I'&:!=#'6&:!*&>!;'+:&$J!!
0
20
40
60
80
100
WAQM_EA WAQM_LIFT diff
!
Figure!10.!Box!plots!for!WAQMEvalAccess4(left)!and!WAQMLIFT4(right)4when!aEA=48,!bEA=0.59,!aLIFT=58!and!
bLIFT=0.70!
N@&$&?+$&-!&)>@&$!+?!>@&4!S+6'7!9&!#II$+I$)#>&!96>!?+$!?6$>@&$!;#';6'#>)+1:!>6I'&!u,!):!
:&'&;>&7!#:!>@&!4&7)#1!):!#'4+:>!>@&!:#4&!#:!;#1!9&!:&&1!)1!D)*6$&!,EJ!C>!;#1!9&!#':+!
+9:&$=&7!>@#>! 1+S!WCDN!+9>#)1:!@)*@&$!=#'6&:!e!4#[kZ,!e!;+4I#$&7!>+!D)*6$&!h!S@&$&!
4#[kF2!#17!diff!4)1)4)X&:!'3)1*!9&>S&&1!i!#17!2YJ!C>!;#1!#':+!9&!7&76;&7!?$+4!D)*6$&!
,E! >@#>! WCDN! =#'6&:! #$&! 4+$&! ;+1;&1>$#>&7! S@);@! ):! ;+1?)$4&7! 93! >@&! @):>+*$#4! )1!
D)*6$&!,2J!
0
50
100
150
200
250
300
350
400
10 20 30 40 50 60 70 80 90 100
!
Figure!11.!Histogram!of!absolute!frequency!of!
WAQMEvalAccess4when!a=48!and!b=0.59!
0
100
200
300
400
500
600
10 20 30 40 50 60 70 80 90 100
!
Figure!12.!Histogram!of!absolute!frequency!of!
WAQMLIFT4when!a=58!and!b=0.70!
!
B+4I#$&7!>+!>@&!@):>+*$#4:!)1!D)*6$&:!F!#17!Z!>@&!@):>+*$#4:!#9+=&!@#=&!=#'6&:!4+$&!
;+1;&1>$#>&7! )1! >@&! $#1*&! rUEeFEs! S@&$&! WAQMEvalAccess! @#:! iEt! +?! =#'6&:! S@&$&#:!
WAQMLIFT!@#:!>@&!ZFt!+?!>@&4J%
%
Accessibility%Formulas%for%a%Web%Site%
NS+!?+$46'#:!#$&!I$+I+:&7!>+!4&#:6$&!#!S&9!:)>&!S)>@!WAQMJ!N@&!?)$:>!+1&!#::64&:!
>@#>!#''!I#*&:!@#=&!>@&!:#4&!)4I+$>#1;&!)1!#!S&9!:)>&!1+!4#>>&$!>@&)$!7&I>@!)1!>@&!:)>&!
+$! >@&! ?61;>)+1#')>3! >@&3! )4I'&4&1>J! N@&$&?+$&! #! :>#>):>);#'! 4&#1! ):! ;#';6'#>&7! 6:)1*!
&=&$3!:)1*'&!I#*&!)1!#!S&9!:)>&J!
N@&! :&;+17! ?+$46'#! #::64&:! >@#>! #''! >@&! I#*&:! )1! #! :)>&! :@+6'7! 1+>! 9&! &A6#''3!
S&)*@>&7J!N@&!7&&I&$!#!I#*&!):!)1!>@&!@)&$#$;@3!+?!>@&!:)>&!>@&!'&::!)4I#;>!:@+6'7!@#=&!
+1!>@&!?)1#'!=#'6&!#:!>@&!?+''+S)1*!?+$46'#!:@+S:^!
∑
∑
−
−
×
=
i
i
i
i
Site e
eA
A
!
O@&$&!i!):!>@&!7&I>@!)17&[-!:>#$>)1*!?$+4!E!?+$!>@&!@+4&!I#*&J!N@6:-!>@&!#;;&::)9)')>3!>+!
>@&!@+4&!I#*&!):!S&)*@&7!93!,JEE-!>@&!?)$:>!7&I>@! '&=&'! 93! EJMh-! >@&! :&;+17! 7&I>@! 93!
EJ,Y-! >@&! >@)$7! +1&! 93! EJEU-! &>;J! `+>&! >@#>! 1+1&! +?! >@&! ?+$46'#:! ;+1:)7&$! :#4I')1*!
)::6&:J!!
N@&!?+''+S)1*!>#9'&!:@+S:!>@&!WAQM!=#'6&!?+$!&#;@!S&9!:)>&!S)>@!>@&!:&'&;>&7!=#'6&:!
?+$!+?! a% #17!bJ! N#9'&! M! :@+S:! >@&!#;;&::)9)')>3! :;+$&:! +9>#)1&7!#II'3)1*! >@&!?+$46'#:!
?+$!#!S&9!:)>&!*$+6I)1*!>@&!#?+$&4&1>)+1&7!,YYZ!S&9!I#*&:J!
Table!3.!Accessibility!scores!obtained!using!different!formulas!for!a!web!site!(mean,!depth)!with!
EvalAccess!and!LIFT!as!well!as!the!difference!between!both!approaches!when!different!tools!are!used!
(diffmean!and!diffdepth).%
!
WAQMEvalAccess%
WAQMLIFT!
diffmean%
diffdepth%
S&9!:)>&!
4&#1!
7&I>@!
4&#1!
7&I>@!
<1)=&$:)>3!+?!B#49$)7*&!
ZY!
ZF!
hY!
h,!
2E!
2h!
B)>3!<1)=&$:)>3!W+17+1!
Z2!
ZU!
iU!
h,!
2h!
2Y!
N@&!<1)=&$:)>3!+?!o#1:#:!
FE!
Fh!
h,!
hE!
Z!
,h!
W#1;#:>&$!<1)=&$:)>3!
hZ!
FF!
ii!
h,!
,M!
,h!
<1)=&$:)>3!+?!`)*&$)#!
hh!
iF!
ih!
h,!
,E!
M!
<1)=&$:)>3!+?!B#'*#$3!
hY!
ih!
iZ!
iZ!
U!
2!
<1)=&$:)>3!+?!/&$%&'&3!
hY!
hE!
ih!
iZ!
h!
,!
N@&!C$):@!N)4&:!
iF!
i,!
iY!
iU!
Y!
Y!
<1)=&$:)>3!+?!/+'>+1!
iU!
iU!
iY!
i2!
M!
M!
<1)=&$:)>3!+?!q$&>+$)#!
iM!
UZ!
UU!
Uh!
i!
2!
<1)=&$:)>3!+?!G617&&!
UZ!
UU!
iE!
i2!
M!
h!
N@&!/&'?#:>!N&'&*$#I@!
UY!
Ui!
Uh!
UM!
M!
M!
G#)'3!V[I$&::!
UY!
Yh!
UY!
Yh!
E!
h!
N@&!H371&3!"+$1)1*!g&$#'7!
U2!
U,!
iE!
iM!
F!
,2!
T6>'++%!C17)#!
ME!
M,!
UF!
UF!
2F!
2h!
!
O@&1!+1&!>++'!):!:&'&;>&7!>@&!7)??&$&1;&!9&>S&&1!>@&!>S+!4&>$);:!Q4&#1e7&I>@R!):!1+>!
=&$3!:)*1)?);#1>J! C1!>@&!;#:&!+?!4&#:6$&4&1>!S&)*@>)1*!7&I>@:-!:)1;&!>@&!@+4&I#*&!):!
@)*@&$!S&)*@>&7!>@#1!7&&I&$!I#*&:-!>@):!1+1e$&'&=#1>!7)??&$&1;&!4#3!9&!;#6:&7!76&!>+!
>@&! ?#;>! >@#>! #''! I#*&:! )1! #! :)>&! #$&! =&$3! :)4)'#$! >+! >@&! @+4&I#*&! )1! >&$4:! +?!
#;;&::)9)')>3! )::6&:J! d&*#$7)1*! >@&! ;+4I#$):+1! 9&>S&&1! >@&! :;+$&:! +9>#)1&7! S)>@!
7)??&$&1>!>++':-!N#9'&!M!:@+S:!>@#>!&=&1!:)4)'#$!$&:6'>:!#$&!+9>#)1&7!Q)1!4#13!;#:&:!diff!
):! ;'+:&! >+! ER! :+4&! +>@&$! ;#:&:! :@+S! 4+$&! =#$)#9)')>3! )1! >@&)$! $&:6'>:! Q2F! #:! #!
4#[)464RJ!D)*6$&!,M!:@&7:!4+$&!')*@>!+1!>@&!9&@#=)+6$!+?!diffmean!#17!diffdepth.!
0
5
10
15
20
25
30
mean depth
!
Figure!13.!Box!plots!for!diffmean!and!diffdepth!
(#'6&:!#$&!4+$&!;+1;&1>$#>&7!?+$!diffmean!Q,:>!PkMJU-!M$7!Pk,,JUR!S@);@!$#1*&:!?$+4!E!
>+!2F!S@)'&!diffdepth!$#1*&:!?$+4!,!>+!2h!#17!,:>!Pk!#17!M$7!Pk,hJ!N@&!4&7)#1!)1!9+>@!
;#:&:!):!&A6#':!>+!hJ!N@6:-!)>!:&&4:!>@#>!>@&!?+$46'#!?+$!#!S&9!:)>&!;#';6'#>&7!S)>@!>@&!
4&#1!+?!#''!WAQM!$&76;&:!9&>>&$!>@&!7)??&$&1;&!9&>S&&1!9+>@!>++':-!#>!'&#:>!?+$!>@&:&!
S&9!:)>&:J!C1!+$7&$!>+!4#%&!$&:6'>:!;'&#$&$-!D)*6$&!,Y!:@+S:!>@&!#;;&::)9)')>3!:;+$&:!?+$!
&#;@! >++'! )1! >@&! ;#:&! +?! >@&! ?+$46'#! >@#>! 6:&:! >@&! 4&#1! #II$+#;@! >+! ;#';6'#>&! >@&!
#;;&::)9)')>3!+?!#!S@+'&!:)>&J!
0
10
20
30
40
50
60
70
80
90
100
U of Cambr idge
City U London
U of Kansas
Lanca ster U
U of Niger ia
U of Calga ry
U of Berke ley
The Irish Times
U of Bolton
U of Preto ria
U of Dundee
Belfast Telegraph
Daily Expre ss
Sydney M Herald
Outlook India
EvalAccess
LIFT
!
Figure!14.!Accessibility!scores!obtained!by!different!websites!by!EvalAccess!and!LIFT!using!the!mean!
approach!
(#'6&:! )1! D)*6$&! ,Y! $&?'&;>! >@&! ?#)'6$&! $#>&! +?! OB5.! ,JE! ;@&;%I+)1>:! ;+1:)7&$&7! 93!
&)>@&$! >++'! #17! :+4&@+S! >@&3! #$&! $&'#>&7! >+! >@&! &[):>)1*! ?#)'6$&! I+)1>:! QS@#>! S&9!
7&=&'+I&$:!;+6'7!@#=&!7+1&!S$+1*R!#17!?#)'6$&:!7&>&;>&7!93!&#;@!>++'!)1!#!7&>&$4)1&7!
S&9! :)>&J! W)>>'&! ;#1! 9&! :#)7! #9+6>! S@#>! &#;@! :I&;)?);! 4&#:6$&! 4&#1:! S)>@! $&*#$7! >+!
:>$6;>6$&-!)1>&$#;>)+1!:>3'&-!'++%! #17! ?&&'-! &>;J!N@&!S&9!:)>&:! 6:&7! ?+$!>@&!&[I&$)4&1>!
#$&! )1?+$4#>)+1e;&1>$);! :)>&:! S@&$&! )1?+$4#>)+1! :&#$;@)1*! I$&=#)':! +=&$! I&$?+$4)1*!
>#:%:!:6;@!#:!?)'')1*!+6>!#!?+$4J!p&>-!WAQM!):!;#';6'#>&7!6:)1*!OB5.!2JE!q&$;&)=#9)')>3-!
<17&$:>#9)')>3-! d+96:>1&::! #17! TI&$#9)')>3! #>>$)96>&:J! N@&$&?+$&! @)*@! #;;&::)9)')>3!
:;+$&:!)4I'3!>@#>!>@&:&!?+6$!I$+I&$>)&:!#$&!:#>):?)&7!>+!:+4&!&[>&1>J!
6!Conclusions!
N@&!+90&;>)=&!+?!>@):!I#I&$!):!>+!?)17!#7&A6#>&!=#'6&:!?+$!>@&!WAQM!)1!+$7&$!>+!+9>#)1!
>++'! )17&I&17&1;&J! C1! +>@&$! S+$7:-! 1+! 4#>>&$! S@);@! &=#'6#>)+1! >++'! ):! 6:&7! ?+$!
4&#:6$&4&1>-! $&:6'>:! S)''! 9&! :)4)'#$! )1! >@&! >@$&&! ?+$&:&&1! #II');#>)+1! :;&1#$)+:! #17!
>++'!)1>&$;@#1*&#9)')>3!S+6'7!9&!+9>#)1&7J!
D)$:>! +?! #''-! )>! @#:! 9&&1! &4I)$);#''3! 7&4+1:>$#>&7! >@#>! >@&! @+4&I#*&! #17! #13! +>@&$!
S&9! I#*&! )1! #! S&9! :)>&! @#=&! #! :)4)'#$! #;;&::)9)')>3! &$$+$! I$+?)'&-! #>! '&#:>! #>! >@&! S&9!
:)>&:! 6:&7! )1! >@):! &[I&$)4&1>J! N@&1-! >@$&&! 4&>@+7:! @#=&! 9&&1! 6:&7! >+! +9>#)1! >@&!
=#'6&:! >@#>! 4)1)4)X&! >@&! 7)??&$&1;&! +?! WAQM! 9&>S&&1! WCDN! #17! V=#'5;;&::J! D)$:>'3-!
@+4&I#*&:! ?$+4! ,U! S&9! :)>&:! @#=&! 9&&1! 4&#:6$&7! S)>@! 9+>@! >++':! #17! ,EEEE!
7)??&$&1>! ;+49)1#>)+1:! +?! a! #17! b! I#$#4&>&$:! @#=&! 9&&1! 6:&7J! 5! *$&#>! 1649&$!+?!
>@&:&!;+49)1#>)+1:! 3)&'7&7! diffkEJ!N@6:-! >@&! 4&#1!+?! >@+:&! =#'6&:!>@#>! 4)1)4)X&! >@&!
7)??&$&1;&! S&$&! :&'&;>&7J! H&;+17'3-! >#%)1*! #7=#1>#*&! +?! >@&! 7&4+1:>$#>)+1! +?! >@&!
=#')7)>3!+?! @+4&I#*&:!#:! #1! #7&A6#>&! :#4I'&! ?+$! +9>#)1)1*! a! #17! b! =#'6&:! >@&! :#4&!
I$+;&::! S#:! ?+''+S&7! #17! 1&S! =#'6&:! S&$&! +9>#)1&7J! D)1#''3-! )>! S#:! I$&;&7&7! )1! >@&!
:#4&!S#3!96>!S)>@!:&=&$#'!>&:>!?)'&:J!
,YYZ!S&9!I#*&:!S&$&!#*#)1!4&#:6$&7!S)>@!>@&!>6I'&:!:&'&;>&7!S)>@!&#;@!4&>@+7!#17!
:>#>):>);#'! #1#'3:):! S#:! ;#$$)&7! +6>! )1! +$7&$! >+! ;@++:&! >@&! 4+:>! #II$+I$)#>&! =#'6&:J!
HI&#$4#1f:!$#1%)1*!;+$$&'#>)+1!S#:! '+S&$! >@#1! )1!>@&!I$&=)+6:! &[I&$)4&1>! )1! ()*+!&>!
#'J! Q2EEhR! e! S#:! jQ,MiMRkEJh,Z! QIlEJ,,UR! e! 96>! )>! ):! :>)''! 4+7&$#>&-! $#1*)1*! 9&>S&&1!
jQ,YYZRkEJi,!#17! jQ,YYZRkEJih! QIl2J2e,iRJ! g+S&=&$-! >@&!7)??&$&1;&! 9&>S&&1! WAQM!
:;+$&:!):!$&#''3!'+S! )1! :+4&! ;#:&:!:I&;)#''3!S@&1!aLIFTkUF!-%bLIFTkEJhE,!aEvalAccesskYF!#17%
bEvalAccesskEJUZJ! 5:! #! $&:6'>-! 7)??&$&1>! #II');#>)+1! :;&1#$)+:! +?! #;;&::)9)')>3! 4&>$);:!
$&A6)$&!>@&!?+''+S)1*!=#'6&:!+?!a%#17!b^!
• N@&! :;&1#$)+:! >@#>! $&A6)$&! +$7)1#'! :;#'&! +?! =#'6&:! QC1?+$4#>)+1! d&>$)&=#'! #17!
#;;&::)9)')>3! 4+1)>+$)1*R! =#'6&:! :&'&;>&7! )1! ()*+! &>! #'J! Q2EEhR-! ak2E! #17! 9kEJM!
#$&!4+$&!#II$+I$)#>&J!
• D+$!>@&!P6#')>3!5::6$#1;&! :;&1#$)+! +$! #13! :;&1#$)+!>@#>!4#%&:!6:&! +?! ;#$7)1#'!
=#'6&:! aLIFTkUF! -% bLIFTkEJhE,!aEvalAccesskYF! #17% bEvalAccesskEJUZ! #$&! >@&! 4+:>!
:6)>#9'&J!N@&:&!=#'6&:! ;#1!#':+!9&!6:&7!)1!#''!:;&1#$)+:!:)1;&!>@&3! @#=&!I$+=&7!
>+!S+$%!S&''!S)>@!+$7)1#'!=#'6&:J!
C>!S+6'7!9&!)1>&$&:>)1*!>+!+9:&$=&!>@&!9&@#=)+6$!+?!#1+>@&$!&=#'6#>)+1!>++'!S)>@!>@&:&!
=#'6&:!+?!a%#17!b!)1!+$7&$!>+!#:;&$>#)1!S@&>@&$!)>!9&@#=&:!')%&!WCDN!#17!V=#'5;;&::!+$!)>!
):!1&;&::#$3!>+!?+''+S!>@&!>61)1*!4&>@+7!#*#)1J!
C>!:@+6'7!9&!;+1:)7&$&7!>@#>!>@&!:&'&;>&7!=#'6&:!7+! 1+>!3)&'7!+I>)4#'!96>!A6)>&!:)4)'#$!
$&:6'>:J!p&>-!)>!;#11+>!9&!#)4&7!#>!+9>#)1)1*!>@&!:#4&!$&:6'>:!?+$!9+>@!>++':!:)1;&!&#;@!
&=#'6#>)+1! >++'! ;+=&$:! #;;&::)9)')>3! )::6&:! )1! #! 7)??&$&1>! S#3J! N@&$&?+$&-! 7)??&$&1>!
?#)'6$&!$#>&:!#$&!+9>#)1&7!7&I&17)1*!+1!>@&!>&:>!;+=&$#*&!#!>++'!)4I'&4&1>:J!
D)1#''3-!>S+! ?+$46'#:! @#=&! 9&&1! I$+I+:&7! )1! +$7&$! >+! 4&#:6$&!#! S@+'&! S&9:)>&^! >@&!
?)$:>! +1&! #::64&:! >@#>! #''! I#*&:! @#=&! >@&! :#4&! )4I+$>#1;&! S@)'&! >@&! :&;+17! +1&!
#::)*1:!7)??&$&1>!S&)*@>:!7&I&17)1*!+1!@+S!7&&I!#!S&9!I#*&!):!'+;#>&7!)1!#!S&9!:)>&J!
d&:6'>:!:@+S!>@#>!>@&!?+$4&$!4)1)4)X&:!9&>>&$!>@&!7)??&$&1;&!9&>S&&1!>++':-!#>!'&#:>!)1!
>@&!S&9!:)>&:!S&!6:&7!)1!>@):!&[I&$)4&1>J!
!
References!
!
r59#:;#'!&>!#'J-!2EEYs!59#:;#'-!8J-!5$$6&-!"J-!D#0#$7+-!CJ-!.#$#3-!`J!#17!N+4n:-!8J!Q2EEYR!<:&!+?!.6)7&')1&:!
>+!56>+4#>);#''3!(&$)?3!O&9!5;;&::)9)')>3J!C1>&$1#>)+1#'!8+6$1#'!+1!<1)=&$:#'!5;;&::!)1!>@&!C1?+$4#>)+1!
H+;)&>3!Q<5CHR!MQ,R-!I#*&:!h,ehZ-!HI$)1*&$J!
r59+6em#@$#-!HJ-!2EEis!59+6em#@$#-!HJ!Q&7JR!Q2EEiR!B+1?+$4#1;&!V=#'6#>)+1!+?!O&9!H)>&:!?+$!
5;;&::)9)')>3J!O&9!5;;&::)9)')>3!C1)>)#>)=&J!5=#)'#9'&!#>!@>>I^__SSSJSMJ+$*_O5C_&=#'_;+1?+$4#1;&J@>4'!
r5$$6&!&>!#'J-!2EEUs!5$$6&-!"J-!()*+-!"J!#17!59#:;#'-!8J!Q2EEUR!P6#1>)>#>)=&!"&>$);:!?+$!O&9!5;;&::)9)')>3!
V=#'6#>)+1J!q$+;&&7)1*:!+?!>@&!O+$%:@+I!+1!O&9!"&>$);:!#17!"&#:6$&4&1>!-!;+e'+;#>&7!#>!>@&!U>@!
C1>&$1#>)+1#'!B+1?&$&1;&!+1!O&9!V1*)1&&$)1*-!CBOV!2EEUJ!<1)=&$:)>3!+?!O+''+1*+1*!H;@++'!+?!CN!#17!
B+4I6>&$!H;)&1;&J!
r5$$6&!&>!#'-!2EEFs!5$$6&-!"J-!()*+-!"J!#17!59#:;#'-!8J!Q2EEFR!O&9!5;;&::)9)')>3!5S#$&1&::!)1!H&#$;@!
V1*)1&:!d&:6'>:J!C1>&$1#>)+1#'!8+6$1#'!+1!<1)=&$:#'!5;;&::!)1!>@&!C1?+$4#>)+1!H+;)&>3!Q<5CHR!hQ,e2R-!I#*&:!
,EMe,,i-!HI$)1*&$J!
r/#)'&3!#17!/6$7-!2EEUs!/#)'&3-!8J!#17!/6$7-!VJ!Q2EEUR!N$&&e"#I!():6#'):#>)+1!?+$!O&9!5;;&::)9)')>3J!2Z>@!
B+4I6>&$!H+?>S#$&!#17!5II');#>)+1:!B+1?&$&1;&-!BT"qH5BfEU-!I#*&:!2hUe2FE-!CVVV!B+4I6>&$!H+;)&>3!
q$&::J!
r/$#01)%-!2EEis!/$#01)%-!.J!Q2EEiR!O&9!5;;&::)9)')>3!N&:>)1*!S)>@!/#$$)&$:!O#'%>@$+6*@J!5=#)'#9'&!#>!
SSSJ7)4)J61)67J)>_*)+$*)+_I$+0&;>:_9S!
r/$#01)%-!2EEhs!/$#01)%-!.J!Q2EEhR!d#1%)1*!O&9:)>&:!>@$+6*@!q$)+$)>)X&7!O&9!5;;&::)9)')>3!/#$$)&$:J!
N&;@1+'+*3!#17!q&$:+1:!S)>@!G):#9)')>)&:!B+1?&$&1;&J!BH<`-!B#')?+$1)#!H>#>&!<1)=&$:)>3!`+$>@$)7*&J!
r/$#01)%!#17!W+46:;)+-!2EEhs!/$#01)%-!.J!#17!W+46:;)+-!dJ!Q2EEhR!H5"/5^!#!H&4)e56>+4#>);!"&>@+7!?+$!
"&#:6$)1*!/#$$)&$:!+?!5;;&::)9)')>3J!q$+;&&7)1*:!+?!>@&!Z>@!5B"!HC.5BBVHH!B+1?&$&1;&!+1!B+4I6>&$:!
#17!5;;&::)9)')>3-!5HHVNHfEh-!I#*&:!YMeYZ-!5B"!q$&::J!
r/$#01)%!&>!#'J-!2EEhs!/$#01)%-!.J-!"6'#:-!5J!#17!q)>>+1-!BJ!Q2EEhR!V??&;>:!+?!H#4I')1*!"&>@+7:!+1!O&9!
5;;&::)9)')>3!&=#'6#>)+1:J!q$+;&&7)1*:!+?!>@&!Z>@!5B"!HC.5BBVHH!B+1?&$&1;&!+1!B+4I6>&$:!#17!
5;;&::)9)')>3-!5HHVNHfEh-!I#*&:!UZeiU-!5B"!q$&::J!
r/v@'&$!&>!#'J-!2EEis!/v@'&$-!BJ-!g&;%-!gJ-!q&$');%-!TJ-!`)&>X)+-!5J-!#17!<''=&)>e"+&-!`J!Q2EEiR!C1>&$I$&>)1*!
d&:6'>:!?$+4!W#$*&!H;#'&!56>+4#>);!V=#'6#>)+1!+?!O&9!5;;&::)9)')>3J!C1!oJ!")&:&19&$*&$!&>!#'J!Q&7:JR-!
B+4I6>&$:!g&'I)1*!q&+I'&!S)>@!HI&;)#'!`&&7:!2EEiJ!W&;>6$&!`+>&:!)1!B+4I6>&$!H;)&1;&!YEi,-!I#*&:!
,FYe,Z,-!HI$)1*&$J!
rB#'7S&''!&>!#'J-!2EEFs!B#'7S&''-!/J-!B++I&$-!"J-!.6#$)1+!d&)7-!WJ!#17!(#17&$@&)7&1-!.J!Q&7:RJ!O&9!
B+1>&1>!5;;&::)9)')>3!.6)7&')1&:!2JEJ!QOMB!B#17)7#>&!d&;+44&17#>)+1RJ!5=#)'#9'&!#>!
@>>I^__SSSJSMJ+$*_Nd_OB5.2E_!Q2EEF-!5I$)'!MER!
rB@):@+'4!&>!#'J-!,ZZZ!s!B@):@+'4-!OJ-!(#17&$@&)7&1-!.J!#17!8#;+9:-!CJ!QV7:JRJ!Q,ZZZRJ!O&9!B+1>&1>!
5;;&::)9)')>3!.6)7&')1&:!,JEJ!5=#)'#9'&!#>!@>>I^__SSSJSMJ+$*_Nd_O5CeOV/BT`NV`N_!
rD6%67#!&>!#'J-!2EEUs!D6%67#-!oJ-!H#)>+-!HJ-!N#%#*)!gJ!#17!5:#%#S#!BJ!Q2EEUR!q$+I+:)1*!1&S!4&>$);:!>+!
&=#'6#>&!S&9!6:#9)')>3!?+$!>@&!9')17J!V[>&17&7!59:>$#;>:!+1!g64#1!D#;>+$:!)1!B+4I6>)1*!H3:>&4:-!
BgCfEU-!I#*&:!,MFhe,MZE-!5B"!q$&::J!
r.+1Xn'&X!&>!#'J-!2EEMs!.+1Xn'&X-!8J-!"#;w#:-!"J-!d+7$w*6&X-!dJ!#17!Hn1;@&X-!DJ!Q2EEMR!5;;&::)9)')>3!
"&>$);:!+?!O&9!q#*&:!?+$!/')17!V17e<:&$:J!C1!8J"J!B6&=#!W+=&''&!&>!#'J!Q&7:JR-!O&9!V1*)1&&$)1*J!W&;>6$&!
`+>&:!)1!B+4I6>&$!H;)&1;&!2h22-!I#*&:!ii,ehEh-!HI$)1*&$J!
rC=+$3!&>!#'J-!2EEYs!C=+$3-!"J-!p6-!HJ-!#17!.$+1&43&$-!oJ!Q2EEYR!H&#$;@!$&:6'>!&[I'+$#>)+1^!#!I$&')4)1#$3!
:>673!+?!9')17!#17!:)*@>&7!6:&$:f!7&;):)+1!4#%)1*!#17!I&$?+$4#1;&J!BgC!V[>&17&7!59:>$#;>:!2EEY-!IIJ!
,YUMe,YUi-!5B"!q$&::J!
rg#;%&>>!&>!#'J-!2EEYs!g#;%&>>-!HJ-!q#$4#1>+-!/J!#17!m&1*-!xJ!Q2EEYR!5;;&::)9)')>3!+?!C1>&$1&>!S&9:)>&:!
>@$+6*@!>)4&J!q$+;&&7)1*:!+?!i>@!C1>&$1#>)+1#'!5B"!HC.5BBVHH!B+1?&$&1;&!+1!B+4I6>&$:!#17!
5;;&::)9)')>3-!5HHVNH!2EEY-!I#*&:!M2eMZ-!5B"!q$&::J!
r");@!&>!#'J-!2EEMs!");@-!WJ-!D$#1;@-!"J!#17!.#)+-!WJ!Q2EEMR!V=#'6#>)1*!#17!G&:)*1)1*!O&9!H)>&!P6#')>3J!
CVVV!"6'>)4&7)#!,EQ,R-!I#*&:!MYeYM-!CVVV!B+4I6>&$!H+;)&>3!q$&::J!
r`)&':&1!#17!N#@)$-!2EEEs!`)&':&1-!8J!#17!N#@)$-!"J!Q2EEER!g+4&I#*&!<:#9)')>3^!UE!O&9:)>&:!
G&;+1:>$6;>&7-!`&S!d)7&$:!q69'):@)1*J!
rT':)1#!#17!d+::)-!2EE2s!T':)1#-!WJ!#17!d+::)-!.J!Q2EE2R!"&#:6$)1*!O&9!5II');#>)+1!P6#')>3!S)>@!
O&9PV"J!CVVV!"6'>)4&7)#!ZQYR-!I#*&:!2Ee2Z-!CVVV!B+4I6>&$!H+;)&>3!q$&::J!
rq#$4#1>+!#17!m&1*-!2EEUs!q#$4#1>+-!/J!#17!m&1*-!xJ!Q2EEUR!"&>$);!?+$!O&9!5;;&::)9)')>3!V=#'6#>)+1J!
8+6$1#'!+?!>@&!54&$);#1!H+;)&>3!?+$!C1?+$4#>)+1!H;)&1;&!#17!N&;@1+'+*3!UiQ,MR-!I#*&:!,MZYe,YEY-!
O)'&3J!
rH6'')=#1!#17!"#>:+1-!2EEEs!H6'')=#1-!NJ!#17!"#>:+1-!dJ!Q2EEER!/#$$)&$:!>+!6:&^!6:#9)')>3!#17!;+1>&1>!
#;;&::)9)')>3!+1!>@&!O&9f:!4+:>!I+I6'#$!:)>&:J!q$+;&&7)1*:!+?!>@&!5B"!B+1?&$&1;&!+1!<1)=&$:#'!
<:#9)')>3-!B<<fEE-!I#*&:!,MZe,YY-!5B"!q$&::J!
rN#%#*)!&>!#'J-!2EEYs!N#%#*)-!gJ-!5:#%#S#-!BJ-!D6%67#-!oJ!#17!"#&7#-!8J!Q2EEYR!5;;&::)9)')>3!G&:)*1&$^!
():6#')X)1*!<:#9)')>3!?+$!>@&!/')17J!q$+;&&7)1*:!+?!>@&!i>@!5B"!HC.5BBVHH!B+1?&$&1;&!+1!B+4I6>&$:!#17!
G):#9)')>3-!5HHVNH!2EEY-!I#*&:!,hhe,FY-!5B"!q$&::J!
r(&''&4#1!&>!#'J-!2EEhs!(&''&4#1-!VJ-!"&&$9&'7-!BJ-!H>$+99&-!BJ-!o+;@-!8J!(&'#:;+-!BJ5J-!H1#I$67-!"J!#17!
`)&>X)+-! 5J! GeO5/Y-! <1)?)&7! O&9! V=#'6#>)+1! "&>@+7+'+*3! Q<OV"! ,J2! B+$&RJ! 2EEhJ! 5=#)'#9'&! #>!
@>>I^__SSSJS#9;'6:>&$J+$*_6S&4,y2_!
r()*+!&>!#'J-!2EEhs!()*+-!"J-!5$$6&-!"J-!/$#01)%-!.J-!W+46:;)+-!dJ!#17!59#:;#'-!8J!Q2EEhR!P6#1>)>#>)=&!
"&>$);:!?+$!"&#:6$)1*!O&9!5;;&::)9)')>3J!q$+;&&7)1*:!+?!>@&!2EEh!C1>&$1#>)+1#'!B$+::eG):;)I')1#$3!
B+1?&$&1;&!+1!O&9!5;;&::)9)')>3!QOY5R-!I#*&:!ZZe,Eh-!5B"!q$&::J!
... The Failure Rate (FR) was developed by Sullivan and Matson in 2000 [16]. According to Vigo, et al., [17], this metric relates the actual points of failure with the potential points of failure. For instance, if a web page has ten images, all these images are potential barriers if they are not properly defined. ...
... 17). Equation 17 is not, however, adequate once the user experience is a subjective aspect. For this reason, the authors developed Eq. 18, where they make use of machine learning. ...
... UWEM already calculates a score for each website, by calculating its web pages' average score. Besides applying the UWEM metric to websites as Vigo, et al. define [17], we decided to additionally use another procedure to convert this metric into a website metric, as will be described further on. WIE, Conservative, Optimistic and Strict are four simple metrics that can be easily applied with our data, as they only require the number of applied success criteria, the number of elements, the number of warnings, the number of fails and the number of passes. ...
Large-scale study of web accessibility metrics
Article
Full-text available
Evaluating the accessibility of web resources is usually done by checking the conformance of the resource against a standard or set of guidelines (e.g., the WCAG 2.1). The result of the evaluation will indicate what guidelines are respected (or not) by the resource. While it might hint at the accessibility level of web resources, often it will be complicated to compare the level of accessibility of different resources or of different versions of the same resource from evaluation reports. Web accessibility metrics synthesize the accessibility level of a web resource into a quantifiable value. The fact that there is a wide number of accessibility metrics, makes it challenging to choose which ones to use. In this paper, we explore the relationship between web accessibility metrics. For that purpose, we investigated eleven web accessibility metrics. The metrics were computed from automated accessibility evaluations obtained using QualWeb. A set of around three million web pages were evaluated. By computing the metrics over this sample of nearly three million web pages, it was possible to identify groups of metrics that offer similar results. Our analysis shows that there are metrics that behave similarly, which, when deciding what metrics to use, assists in picking the metric that is less resource intensive or for which it might be easier to collect the inputs.
View
... Automated web accessibility tools can generate different results and do not capture all the problems when analysing the homepages. Vigo et al. [55] analysed the use of web accessibility evaluation tools, specifically, the coverage, completeness and correctness with regard to WCAG 2.0 conformance, finding a large number of problems in the three variables analysed. They concluded that the use of automated tests entails that "1 of 2 success Fig. 1 Map of international tourist arrivals and level of conformance AA and AAA by country criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives". ...
... Considering the limitations of the use of web accessibility evaluation tools (the focus on homepages produces a similar error profile than any other webpage in a given site) [55], and that TAW only analyzes the home pages, it may be argued that Hong Kong, Japan and South Korea homepages are clear examples of good practices in their websites. They have a very low number of issues in the analysis, compared to other websites. ...
Accessibility of tourism websites: the level of countries’ commitment
Article
Full-text available
Various initiatives have been undertaken in recent years to improve the accessibility of tourism in a large number of countries. One of the areas in which work has been done has been the inclusion and accessibility of information for all, especially for people with disabilities and other users with access needs. A focus of this work has been to the digital world, given the increasing relevance of the internet in daily life. This paper aims to determine whether the different nation states are applying appropriate standards, both under their own regulations and under those from United Nations agreements regarding accessibility to online tourism information to foster more sustainable tourism. In the study of official tourism organization website home pages of the countries contained in the World Tourism Organization reports, cluster analysis was used to identify common behaviour patterns. The main result was the need to improve the compatibility and ease of navigation of websites, as well as identifying Japan, Hong Kong and South Korea as countries that can serve as examples of good practice.
View
... More recently, Vigo et al. proposed a quantitative measurement for Web accessibility called Web Accessibility Quantitative Metric (WAQM) [96,97], to accurately measure the accessibility of Web content. This approach can provide a means for quantifying quality assurance, information retrieval, and monitoring Web accessibility. ...
... It assumes that all Web pages in a Website has equal importance, but this is not true in practice. Thus, in [97], Vigo et al. extended their approach, so that Web pages that are deeper in the hierarchy will have lesser impact on the final value. ...
View
... Currently, the version of the Web Content Accessibility Guidelines (WCAG) has arrived in version 2.0, and 2.1 whose main goal is to determine how to make Web content more accessible to people with disabilities. Accessibility involves a variety of disabilities, including vision, hearing, physical, speech, cognitive, language, learning, and neurological disabilities [9][10][11]. There are many tools that can be used as a guide or guideline in developing website 3 accessibility, such as EvalAccess 2.0, Achecker, TAW, and Watchfire Bobby [1] and also Wave [12], however it is recommended to use WCAG because it is considered to have a more complete and widely used standard in the world. ...
Comparison of accessibility of OER repositories of developed countries and developing countries based on WCAG 2.0 guidelines
Conference Paper
Full-text available
  • Dec 2019
Abstract. Access to quality education in the era of ICT technology is growing rapidly, by utilizing this technology ideally it is not a problem, thus the presence of technology can be a problem solver that occurs such as inequality of quality educational material and access to the intended materials, this is certainly a dream all parties because, in reality, all this is not easy. The biggest challenge in equalizing the quality of education is experienced by almost all countries in the world, especially countries with large territories and populations. Inequalities and gaps in differences and equal access to quality education material occur in almost every country in the world. Distance learning can be carried out easily by utilizing ICT infrastructure. Almost every country in the world strives to provide open wide access through the use of web-based applications to facilitate and facilitate the population of the needs of teaching materials according to the population. This article will discuss, analyse and explore the comparison of the accessibility of the Open Educational Resources (OER) repository in developed countries with developing countries.
View
... Moreover, it is reasonable to presume that accessibility issues emerging from the homepage assessment are likely to propagate to other web pages. For example, Vigo et al (2009) demonstrated that homepages have a more similar error profile than any other web page in a given website. Further, the homepage plays the role of the index to other hyperlinked documents within the website and is the façade that shapes the end user's first impression upon visiting a given e-Government portal (Abdul Latif and Masrek, 2010). ...
View
... Further, it is reasonable to presume that accessibility issues emerging from the homepage assessment are likely to propagate to other web pages. For example, Vigo et al (2009) demonstrated that homepages have a more similar error profile than any other web page in a given web site. Further, the homepage plays the role of the index to other hyperlinked documents within the web site and is the fac¸ade that shapes the end user's first impression upon visiting a given e-government portal (Abdul Latif and Masrek, 2010). ...
Accessibility as an integral factor in e-government web site evaluation: The case of Dubai e-government
Article
Full-text available
  • May 2014
  • Inform Tech People
Purpose – The purpose of this paper is to examine the extent to which accessibility is taken into account in the assessment and ranking of e-government web sites through the lens of a specific study related to Dubai e-government. Design/methodology/approach – The paper considers a case study related to Dubai e-government and it evaluates the accessibility of each of the 21 Dubai e-government web sites, based on the Web Content Accessibility Guidelines (WCAG) 2.0 and using an automated accessibility testing tool. A bivariate correlation analysis is performed to assess the correlation between web site ranking and accessibility score. Findings – The research reveals that contrary to common intuition and some earlier studies, there is a weak correlation between e-government web site ranking score and web site accessibility. Research limitations/implications – The paper uses an accessibility metric that is a proxy indicator of web accessibility and is not a real assessment of accessibility as experienced by a person with disability. Practical implications – When re-examined through the lens of Rawls's moral theory, this research suggests that accessibility should be given a higher priority in the general evaluation and ranking of e-government web sites. Social implications – The paper promotes universal accessibility to e-government information and services. Originality/value – The paper uses ethical arguments to highlight the need to comprehensively consider accessibility as a major criterion in the assessment and ranking of e-government web sites.
View
... Even if it would be desirable to define metrics for websites, a study by Nielsen and Tahir [26] states that the usability of the homepage predicts the usability of the whole site. Supporting this statement, Vigo et al. [42] provide empirical data to corroborate such a statement, at least in the context of automatic accessibility measurement. ...
Quantitative assessment of mobile web guidelines conformance
Article
Full-text available
Conformance metrics for the mobile web can play a crucial role as far as engineering mobile websites are concerned, especially if they are automatically obtained. In this way, developers can have an idea in numeric terms of how suitable their developments are for mobile devices. However, there are a plethora of devices with their own particular features (screen size, formats support, etc.) that restrict a unified automatic assessment process. This paper proposes a tool-supported method for device-tailored assessment in terms of conformance with Mobile Web Best Practices 1.0, including the definition of five quantitative metrics for automatically measuring mobile web conformance: Navigability, Page layout, Page definition, User input and Overall score. The behaviour of these metrics was analysed for different devices and different web paradigms, both mobile web pages and their equivalent desktop pages. As expected, the results show that mobile web pages on more capable devices score higher. In addition, 20 users took part in an experiment aimed at discovering how conformance-based scores relate to usability dimensions. The results demonstrate that automatic scoring approaches strongly correlate with usability scores obtained by direct observation, such as task completion time and user satisfaction. This correlation is even stronger for the device-tailored assessment than the one that assumes a general profile for all devices. For instance, results show a strong negative correlation between Overall score and task completion time: ρ (9) = −0.81, (p < 0.05) for the generalist approach and ρ (9) = −0.88 for the device-tailored one, entailing that mobile web guidelines and the metrics based on their conformance capture usability aspects. This result challenges the widely accepted belief that conformance to guidelines does not imply more usable web pages, at least for web accessibility conformance.
View
Determining the Accessibility of e-Government Websites in Sub-Saharan Africa Against WCAG 2.0 Standard
Chapter
  • Jan 2021
The purpose of this study was to determine the conformance levels of government websites in Sub-Saharan Africa (SSA) to Web Content Accessibility Guidelines (WCAG) 2.0 and examine which macro factors influenced the accessibility these websites. The findings indicated that the majority of government websites in SSA still had a long way to go to become accessible based on the WCAG 2.0 standards. None of the 217 government websites examined adhered to all the WCAG 2.0 guidelines. Cross country analysis showed that there are three macro factors influencing e-government accessibility in SSA, namely Human Development Index (HDI), Corruption Perception Index (CPI), and percentage of the active population (15-64 years). Countries with high HDI levels and low CPI levels tend to have websites with fewer accessibility errors, while those for countries with high percentage of the active population have more accessibility errors.
View
Accessibility of local government websites: influence of financial resources, county characteristics and local demographics
Article
  • Aug 2020
Although many studies have investigated how well government websites have implemented website accessibility standards, such as Section 508 and WCAG 2.0, very little research has been conducted exploring what factors influence the level of implementation. Based on the organizational innovation model proposed by Frances Berry, this study examines 342 county government websites in the USA and analyzes how several factors, such as budget resources, local demographics and forms of county government, influence the accessibility of the sites. The result of the OLS estimate indicates that the complexity of the websites and the county population density are the most important predictors of web accessibility. County budget is also a marginal predictor, while the percentage of the population with disabilities is a negative predictor. Given the slow progress in the implementation of higher accessibility standards in the USA, the findings of this study provide timely implications for policymakers and governments to improve the quality of their websites.
View
Determining the Accessibility of e-Government Websites in Sub-Saharan Africa Against WCAG 2.0 Standard
Article
  • Jan 2017
The purpose of this study was to determine the conformance levels of government websites in Sub- Saharan Africa (SSA) to Web Content Accessibility Guidelines (WCAG) 2.0 and examine which macro factors influenced the accessibility these websites. The findings indicated that the majority of government websites in SSA still had a long way to go to become accessible based on the WCAG 2.0 standards. None of the 217 government websites examined adhered to all the WCAG 2.0 guidelines. Cross country analysis showed that there are three macro factors influencing e-government accessibility in SSA, namely Human Development Index (HDI), Corruption Perception Index (CPI), and percentage of the active population (15-64 years). Countries with high HDI levels and low CPI levels tend to have websites with fewer accessibility errors, while those for countries with high percentage of the active population have more accessibility errors.
View
Barriers to use: usability and content accessibility on the Web's most popular sites
Article
Full-text available
  • Jan 2000
Content accessibility is a key feature in highly usable Web sites, but reports in the popular press typically report that 95% or more of all Web sites are inaccessible to users with disabilities. The present study is a content accessibility compliance audit of 50 of the Web's most popular sites, undertaken to determine if content accessibility can be conceived and reported in continuous, rather than dichotomous, terms. Preliminary results suggest that a meaningful ordinal ranking of content accessibility is not only possible, but also correlates significantly with the results of independent automated usability assessment procedures.
View
View
Effects of sampling methods on web accessibility evaluations
Conference Paper
Full-text available
  • Oct 2007
Except for trivial cases, any accessibility evaluation has to be based on some method for selecting pages to be analyzed. But this selection process may bias the evaluation. Up to know, not much is known about available selection methods, and about their eectiveness and eciency. The paper addresses the following open issues: how to define the quality of the selection process, which processes are better than others, how to measure their dierence in quality, which factors may aect
View
Accessibility designer: visualizing usability for the blind.
Conference Paper
Full-text available
  • Jan 2004
These days, accessibility-related regulations and guidelines have been accelerating the improvement of Web accessibility. One of the accelerating factors is the development and deployment of accessibility evaluation tools for authoring time and repair time. They mainly focus on creating compliant Web sites by analyzing the HTML syntax of pages, and report that pages are compliant when there are no syntactical errors. However, such compliant pages are often not truly usable by blind users. This is because current evaluation tools merely check if the HTML tags are appropriately used to be compliant with regulations and guidelines. It would be better if such tools paid more attention to real usability, especially on time-oriented usability factors, such as the speed to reach target content, the ease of understanding the page structure, and the navigability, in order to help Web designers to create not simply compliant pages but also usable pages for the blind. Therefore, we decided to develop Accessibility Designer (aDesigner), which has capabilities to visualize blind users' usability by using colors and gradations. The visualization function allows Web designers to grasp the weak points in their pages, and to recognize how accessible or inaccessible their pages are at a glance. In this paper, after reviewing the related work, we describe our approach to visualize blind users' usability followed by an overview of Accessibility Designer. We then report on our evaluations of real Web sites using Accessibility Designer. After discussing the results, we conclude the paper.
View
SAMBA: a semi-automatic method for measuring barriers of accessibility
Conference Paper
Full-text available
  • Oct 2007
Although they play an important role in any assessment pro- cedure, web accessibility metrics are not yet well developed and studied. In addition, most metrics are geared towards conformance, and therefore are not well suited to answer questions whether the web site has critical barriers with re- spect to some user group. The paper addresses some open issues: how can acces- sibility be measured other than by conformance to certain guidelines? How can a metric merge results produced by ac- cessibility evaluation tools and by expert reviewers? Does it consider error rates of the tool? How can a metric consider also severity of accessibility barriers? Can a metric tell us if a web site is more accessible for certain user groups rather than others? The paper presents a new methodology and associated metric for measuring accessibility that eciently
View
Accessibility designer
Article
  • Sep 2003
These days, accessibility-related regulations and guidelines have been accelerating the improvement of Web accessibility. One of the accelerating factors is the development and deployment of accessibility evaluation tools for authoring time and repair time. They mainly focus on creating compliant Web sites by analyzing the HTML syntax of pages, and report that pages are compliant when there are no syntactical errors. However, such compliant pages are often not truly usable by blind users. This is because current evaluation tools merely check if the HTML tags are appropriately used to be compliant with regulations and guidelines. It would be better if such tools paid more attention to real usability, especially on time-oriented usability factors, such as the speed to reach target content, the ease of understanding the page structure, and the navigability, in order to help Web designers to create not simply compliant pages but also usable pages for the blind. Therefore, we decided to develop Accessibility Designer (aDesigner), which has capabilities to visualize blind users' usability by using colors and gradations. The visualization function allows Web designers to grasp the weak points in their pages, and to recognize how accessible or inaccessible their pages are at a glance. In this paper, after reviewing the related work, we describe our approach to visualize blind users' usability followed by an overview of Accessibility Designer. We then report on our evaluations of real Web sites using Accessibility Designer. After discussing the results, we conclude the paper.
View
View
View
Proposing new metrics to evaluate web usability for the blind
Conference Paper
  • Apr 2005
Accessibility-related regulations and guidelines are contributing to the steady improvement of Web accessibility. There are various accessibility evaluation tools, and they also help Web authors make their pages compliant with guidelines. As a result, an increasing number of Web pages are compliant with the evaluation tools. These days, however, blind people face the serious problem that reading Web pages is quite difficult. Improvements in information density by using visual effects such as two-dimensional layouts are making it difficult for blind people to understand the page structure. Also, inappropriate alternative texts mislead or confuse blind users.In this paper, to evaluate these kinds of usability problems, we introduce two metrics: navigability and listenability. Navigability evaluates how well structured the Web content is by using headings, intra-page links, labels, etc. Listenability denotes how appropriate the alternative texts are. By using these metrics, we summarize the historical transition of Web usability for blind people.
View
Search result exploration: a preliminary study of blind and sighted users' decision making and performance
Conference Paper
  • Apr 2004
We conducted a preliminary study to examine sighted and blind users' decision-making behavior and performance during the search process. We manipulated the search result's relevance to a task, the search result presentation, and the effort required to process the corresponding web page. We found that users leveraged page features to gauge the amount of effort that is required to explore search pages and made exploration decisions accordingly. Users' desire to know additional page details varied based on their visual ability and the results' relevance. We quantified the cost/benefit tradeoff of additional page features and suggest ways to better support diverse Web searchers.
View