No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Comparative Analysis of Accessibility Testing Tools and Their Limitations in RIAs
Abstract
Accessibility is a required quality for websites today, and several tools exist to test for this quality. These tools are highly advantageous, but sadly they also have some limitations. A particular set of challenges they face is in the evaluation of Rich Internet Applications (RIAs). In this paper, we carry out an experiment to compare and analyze different accessibility testing tools as they evaluate 10 educational websites. We judged these tools based on their error detection, guideline coverage, speed, similarity to one another, and their relative performance when evaluating RIAs. The experiment findings revealed the strength and limitations of each tool. The results of this experiment also exposed that there are many guidelines and success criteria that accessibility testing tools are not able to cover, and that some evaluation tools are similar to each other in terms of the results they produce. Lastly, this experiment highlights a discrepancy in the behavior of the tools when evaluating RIAs compared to when evaluating static websites, although some more than others. This experiment has some limitations which we presented. As a future work, we intend to work with an expert to determine the accuracy of the results produced from the experiment. We also intend to delve deeper into the limitations of these tools and come up with possible solutions.KeywordsAccessibilityAccessibility evaluation toolsOnline educationWeb Content Accessibility Guideline
... Such rules apply to native apps, mobile web apps, and web content. WCAG defines accessibility in terms of four principles which include ease of operation (operable), understandable app content (understandable), robustness (robust), and coherent app content (perceivable) [43,49]. ...
... Accessibility in online learning involves the addition of tools and features into applications to meet the needs of all users [4][5][6]. It is important for users to offer reviews regarding the accessibility of education applications so that such feedback can be used to improve the apps [7,9,43]. In this section, we divide the section into three categories as follows: user reviews, accessibility in user reviews, and LMS in the mobile application. ...
... Accessibility in digital environments refers to designing and developing products, services, and content that can be perceived, understood, and operated by a wide range of users, regardless of their physical, intellectual, or cognitive abilities [7,23,33]. Despite the widespread accessibility adoption in web applications over the years, studies have shown that most web applications still present many accessibility violations. 1 Accordingly, some researchers surveyed people involved in software development and identified many barriers to the adoption of accessibility during the development process: lack of awareness or knowledge, lack of time and support from high-level management, lack of accessibility requirements, and lack of understanding of accessibility standards and guidelines [9,14,18,25,28]. Most barriers are related to the development process or individual perspectives, leaving a gap in the technical challenges developers find when they try to incorporate accessibility into their design. ...
Many standards and guidelines are available to promote the accessibility of digital products and services, yet developers may still struggle to comprehend how they can be applied. In this study, we investigate Stack Overflow posts in order to comprehend the challenges developers confront in the context of web accessibility. Our analysis involved manually labeling 5,092 posts associated with web accessibility to ensure the credibility and consistency of our results. In addition, a formal open-coding approach was employed to generate a taxonomy that delineates the various aspects developers discuss concerning accessibility in web applications. Our analysis revealed that accessibility posts had increased significantly through the years and that approximately 18.36% of the accessibility issues were associated with screen reader behavior. Our study also indicates that most discussions are a call to assistance to improve their application accessibility for visual impairments. The analysis of developers’ queries on Stack Overflow reveals several distinct trends. These findings provide a more comprehensive understanding of the challenges that developers face when it comes to web accessibility.
Web accessibility means that people with some type of disability can make use of theWeb in the same conditions as the rest of the people. When we talk about web accessibility, we refer to a web design and development that allows these people to perceive, understand, navigate and interact with the Web. Web accessibility also benefits other people, including elderly people whose abilities have declined as a result of age. The Web is an essential resource in human activity: education, employment, government, commerce, health, entertainment and many others benefit of the power of the Web. The aim of this systematic literature review is to analyze the empirical methods of evaluating accessibility to educational websites, disabilities and their errors described in a total of 25 selected studies. The results show that in 20 of the 25 papers, web accessibility was evaluated with automatic tools, in 2 papers it was evaluated with real users and in the other 3 papers with automatic tools, real users and experts. There is also evidence that all the educational websites analyzed in the papers need to correct errors. In conclusion, educational websites do not meet any version of the Web Content Accessibility Guidelines (WCAG) and their conformance levels. According to the results, the empirical evaluation methods used for web accessibility could be improved by adopting automatic evaluation tools for website construction and manual mechanisms with web accessibility experts. The challenge for educational institutions is to carry out web accessibility projects to comply with WCAG and other web accessibility standards and current laws of educational inclusion.
With the growth of e-services in the past two decades, the concept of web accessibility has been given attention to ensure that every individual can benefit from these services without any barriers. Web accessibility is considered one of the main factors that should be taken into consideration while developing webpages. Web Content Accessibility Guidelines 2.0 (WCAG 2.0) have been developed to guide web developers to ensure that web contents are accessible for all users, especially disabled users. Many automatic tools have been developed to check the compliance of websites with accessibility guidelines such as WCAG 2.0 and to help web developers and content creators with designing webpages without barriers for disabled people. Despite the popularity of accessibility evaluation tools in practice, there is no systematic way to compare the performance of web accessibility evaluators. This paper first presents two novel frameworks. The first one is proposed to compare the performance of web accessibility evaluation tools in detecting web accessibility issues based on WCAG 2.0. The second framework is utilized to evaluate webpages in meeting these guidelines. Six homepages of Saudi universities were chosen as case studies to substantiate the concept of the proposed frameworks. Furthermore, two popular web accessibility evaluators, Wave and SiteImprove, are selected to compare their performance. The outcomes of studies conducted using the first proposed framework showed that SiteImprove outperformed WAVE. According to the outcomes of the studies conducted, we can conclude that web administrators would benefit from the first framework in selecting an appropriate tool based on its performance to evaluate their websites based on accessibility criteria and guidelines. Moreover, the findings of the studies conducted using the second proposed framework showed that the homepage of Taibah University is more accessible than the homepages of other Saudi universities. Based on the findings of this study, the second framework can be used by web administrators and developers to measure the accessibility of their websites. This paper also discusses the most common accessibility issues reported by WAVE and SiteImprove.
Achieving web accessibility has become a global aim nowadays, especially with the significant inclusion of the Internet in our daily lives. There exist different accessibility evaluation methods that support web developers and designers during websites' development life cycle, and each has different benefits and drawbacks. However, many studies show that there is a continued lack of equality between disabled and non-disabled people in benefitting from the web, especially with the prolonged growth of web development standards and techniques, which raises a demand for continuous improvement in web accessibility evaluation methods in order to gain reliable results. Given this demand, our paper investigates the advancements in web accessibility evaluation methods for the past five years (2011-2015). The results reveal a lack of significant evolution of these evaluation methods. Based on our findings, more efforts are required for enhancing accessibility evaluation techniques in order to achieve better web accessibility.
In this research we will compare between five tools used for web accessibility evaluation. This comparison will use list of faults to be injected in web pages that later will be tested. The research show that tools does not cover all accessibility faults and provide results that miss lead the designer.
The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives.
Accessibility stands as a quality requirement for Web applications. However, current accessibility automatic evaluation tools are not capable of evaluating DOM dynamic generated content that characterizes Ajax applications and RIAs - Rich Internet Applications. In this context, this paper describes an approach for testing accessibility requirements in RIA, by using acceptance tests. The authors had implemented a set of assistive technology user scenarios in the acceptance tests, in order to guarantee keyboard accessibility in web applications. As the scenarios were implemented as acceptance tests scenarios, they provide accessibility analysis over all layers of the software, from server-side to client-side implementations (JavaScript and dynamically generated DOM elements) in RIA. The test scenarios are automatically executed, and by doing so, fit the Continuous Integration process of constant delivery of new functionalities in Web projects.
Although they play an important role in any assessment pro- cedure, web accessibility metrics are not yet well developed and studied. In addition, most metrics are geared towards conformance, and therefore are not well suited to answer questions whether the web site has critical barriers with re- spect to some user group. The paper addresses some open issues: how can acces- sibility be measured other than by conformance to certain guidelines? How can a metric merge results produced by ac- cessibility evaluation tools and by expert reviewers? Does it consider error rates of the tool? How can a metric consider also severity of accessibility barriers? Can a metric tell us if a web site is more accessible for certain user groups rather than others? The paper presents a new methodology and associated metric for measuring accessibility that eciently
Web access for users with disabilities is an important goal and challenging problem for web content developers and designers. This paper presents a comparison of different methods for finding accessibility problems affecting users who are blind. Our comparison focuses on techniques that might be of use to Web developers without accessibility experience, a large and important group that represents a major source of inaccessible pages. We compare a laboratory study with blind users to an automated tool, expert review by web designers with and without a screen reader, and remote testing by blind users. Multiple developers, using a screen reader, were most consistently successful at finding most classes of problems, and tended to find about 50% of known problems. Surprisingly, a remote study with blind users was one of the least effective methods. All of the techniques, however, had different, complementary strengths and weaknesses.
The Web accessibility discipline strives for the study and improvement of front-end Web design towards people with disabilities. Best practices such as WCAG dictate how Web pages should be created accordingly. On top of WCAG, several evaluation procedures enable the measurement of the quality level of a Web page. We leverage these procedures in an automated evaluation of a nearly 30 million Web page collection provided by the Portuguese Web Archive. Our study shows that there is high variability regarding the accessibility level of Web pages, and that few pages reach high accessibility levels. The obtained results show that there is a correlation between accessibility and complexity (i.e., number of HTML elements) of a Web page. We have also verified the effect of the interpretation of evaluation warnings towards the perception of accessibility.
The fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.
The World Wide Web Consortium (W3C) has provided the most important set of guidelines for web accessibility which is popularly known as Web Content Accessibility Guidelines (WCAG). The accessibility analysis of higher education websites becomes paramount important to make them inclusive considering the growing number of enrollments of persons with disabilities (PwDs) in higher education, in countries such as India. This paper presents the accessibility analysis of higher education websites with the case study of college websites (N=44) affiliated with the University of Kashmir and Cluster University Srinagar. The study has been carried out with two major accessibility evaluation tools called web accessibility test, denoted as TAW and the accessibility engine powering browser extensions called the accessibility engine, denoted as aXe. This paper lists the major accessibility barriers exposed by these sites in terms of metrics such as a number of problems, warnings and a status of success criteria violations. With respect to TAW tool, a number of problems observed were 2646, a large number of warnings to the scale of 15995 and the not reviewed items were 1356. With aXe tool, the total violations observed were 1951 and items needing review were 1733. Findings of the statistical analysis are also presented in this paper. This paper presents a roadmap of steps for making these websites inclusive and barrier-free for PwDs.
Popularity of Internet applications has reached significant scales. In consequence, a wide diversity of solutions has been created based on Web features. Rich Internet Application (RIA) is a relevant term adopted for technological advances in software developed for the Web, which refers to Web applications aimed to provide users with a desktop similar experience. RIAs usually have wider capabilities in comparison with traditional hypertext applications, specially regarding to the interactive elements of their interfaces. New possibilities that have emerged from RIA are essential to support relevant aspects of Web 2.0, such as participation and collaboration. As among other applications, developing accessible and usable RIAs is a valuable and fundamental aspect for development teams, since these new interaction features available on the Web are not always accessible for people with disabilities or reduced mobility. For this reason, this mini-course aims to present the main concepts usually used to evaluate accessibility and usability of RIA; it is an overview of perspectives about practices and theoretical references, from Standards for Quality up to the implementation resources of Web applications. The approach of this mini-course covered practices on main RIA coding techniques, and methods of usability and accessibility evaluation as Guidelines Review and Heuristic Evaluation. Moreover, this mini-course was developed aiming newcomers and professionals that want to specialize their skills on the development and evaluation of usable and accessible RIAs.
Rich Internet Applications (RIAs) technologies are challenging the way in which the Web is being developed. However, from the UI accessibility point of view, these technologies pose new challenges that the Web Accessibility Initiative of the W3C is trying to solve through the use of a standard specification for Accessible Rich Internet Applications (WAI-ARIA). Currently, the introduction of properties defined in WAI-ARIA is being done in an ad-hoc manner due to the lack of models, methodologies and tools to support the design of accessible RIA UIs. In this paper we propose a semantic approach to deal with this modeling issue by extending the RUX-Method, a model-based method to build RIA UIs. The approach includes the validation process of the accessibility issues at two different levels: the UI structure and the interactions behavior.
Website accessibility standards should be higher to help disabled people -vox
- S Smith
Smith, S.: Website accessibility standards should be higher to help disabled people
-vox, May 2019. https://www.vox.com/the-goods/2019/2/5/18210912/websitesada-compliance-lawsuits. Accessed 04 Mar 2022
Accessibility testing -manual or automated?
- Investis Digital
Investis Digital: Accessibility testing -manual or automated? January 2020.
https://www.investisdigital.com/blog/web-design-and-development/accessibilitytesting-manual-or-automated. Accessed 23 Oct 2021
8 free web-based website accessibility evaluation tools -usability geek
- J Misfud
Misfud, J.: 8 free web-based website accessibility evaluation tools -usability geek.
https://usabilitygeek.com/10-free-web-based-web-site-accessibility-evaluationtools/
Combination of automatic and manual testing for web accessibility
- J M Mucha
Mucha, J.M.: Combination of automatic and manual testing for web accessibility.
Master's thesis, Grimstad, Norway (2018). Accessed 05 Aug 2019
Comparative analysis of online web accessibility evaluation tools
- C Timbi-Sisalima
- C I M Amor
- S O Tortosa
- J R Hilera
- J Aguado-Delgado
Timbi-Sisalima, C., Amor, C.I.M., Tortosa, S.O., Hilera, J.R., Aguado-Delgado, J.:
Comparative analysis of online web accessibility evaluation tools. In: Proceedings
of the 25th International Conference on Information Systems Development, ISD
2016, pp. 562-573. University of Economics, ACM Press, New York (2016)
5 free must have web accessibility testing tools -leader in offshore accessibility testing -section 508 compliance -wcag conformance -barrierbreak
- A Kumar
Kumar, A.: 5 free must have web accessibility testing tools -leader in offshore accessibility testing -section 508 compliance -wcag conformance -barrierbreak,
October 2021. https://www.barrierbreak.com/5-free-must-have-web-accessibilitytesting-tools/