The State of Accessibility in Blackboard: Survey and User Reviews Case Study

Full text

The State of Accessibility in Blackboard: Survey and User Reviews
Case Study
Wajdi Aljedaani
University of North Texas
Mohammed Alkahtani
University of North Texas
Stephanie Ludi
University of North Texas
Mohamed Wiem Mkaouer
Rochester Institute of Technology
Marcelo M. Eler
University of São Paulo
Marouane Kessentini
Oakland University
Ali Ouni
ETS Montreal, University of Quebec
ABSTRACT
Context: Nowadays, mobile applications (or apps) have become vi-
tal in our daily life, particularly within education. Many institutions
increasingly rely on mobile apps to provide access to all their stu-
dents. However, many education mobile apps remain inaccessible
to users with disabilities who need to utilize accessibility features
like talkback or screen reader features. Accessibility features have
to be considered in mobile apps to foster equity and inclusion in
the educational environment allowing to use of such apps without
limitations. Gaps in the accessibility to educational systems persist.
Objective: In this paper, we focus on the accessibility of the Black-
board mobile app, which is one of the most common Learning
Management Systems (LMS) used by many universities, especially
during the current COVID-19 pandemic.
Method: This study is divided into two-fold. First, we conduct a
survey using questionnaires and interviews to explore the extent
to which students consider the Blackboard mobile app usability. A
Total of 1,308 hearing students and 65 deaf and hard-of-hearing
students participated in the study. Second, we collected 15,478 user
reviews from the Google Play Store and analyzed the reviews to
extract accessibility issues.
Result: We observed that most deaf and hard-of-hearing students
found diculty in the Blackboard mobile app, compared to hearing
students. Also, our app store analysis showed that only 31% of the
reviews reported violations of accessibility principles that apps
like Blackboard must comply with. This study highlights these
violations and their corresponding implications to support LMS
frameworks in becoming more inclusive for all users.
CCS CONCEPTS
•Human-centered computing →Empirical studies in acces-
sibility;Ubiquitous and mobile devices.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specic permission
and/or a fee. Request permissions from permissions@acm.org.
W4A ’23, April 30–May 01, 2023, Austin, TX, USA
©2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 979-8-4007-0748-3/23/04. . . $15.00
https://doi.org/10.1145/3587281.3587291
KEYWORDS
Mobile Applications, user reviews, accessibility, Blackboard, learn-
ing management system.
ACM Reference Format:
Wajdi Aljedaani, Mohammed Alkahtani, Stephanie Ludi, Mohamed Wiem
Mkaouer, Marcelo M. Eler, Marouane Kessentini, and Ali Ouni. 2023. The
State of Accessibility in Blackboard: Survey and User Reviews Case Study.
In 20th International Web for All Conference (W4A ’23), April 30–May 01,
2023, Austin, TX, USA. ACM, New York, NY, USA, 12 pages. https://doi.org/
10.1145/3587281.3587291
1 INTRODUCTION
The use of mobile devices, particularly smartphones, has signi-
cantly risen in the last few years [
21
,
51
]. Consequently, the number
of mobile apps, applications designed to run on mobile devices, has
also increased. Accordingly, mobile accessibility has gained much
attention in the last few years [
16
,
24
,
37
] given that there is an
estimated 650 million people with disability globally, representing
nearly 10-15% of the world’s total population [
37
,
39
]. Even though
accessibility impacts the overall quality of a product for any user,
it is focused on users with disabilities as it is “the extent to which
products, systems, services, environments, and facilities can be used
by people from a population with the widest range of characteristics
and capabilities to achieve a specied goal in a specied context of
use” [
19
,
37
]. Accessibility for mobile computing is an important
topic, especially for persons with disabilities, given that there are
approximately 650 million disabled people globally, representing
nearly 10-15% of the world’s total population [
37
,
39
], including
students.
In addition to the massive adoption of mobile apps in education
to manage day-by-day activities, most students had their educa-
tional activities conducted through Learning Management Systems
(LMS) as most universities worldwide were constrained to oer
online courses to keep their educational programs running due
to the COVID-19 pandemic. In that sense, investigating the acces-
sibility of LMS systems is of utmost importance once their non-
compliance with accessibility guidelines can potentially hinder or
prevent students from learning any content, thus excluding people
with disabilities from the educational process.
In particular, Blackboard LMS is one of the most adopted plat-
forms for online education worldwide. In previous studies, Li
[32]
assessed students’ acceptance of the Blackboard LMS platform in
84

W4A ’23, April 30–May 01, 2023, Austin, TX, USA Aljedaani, et al.
the United States. They found a need for more compatible con-
tent and activities to be introduced for mobile learning. Another
study by Alkhaldi and Abualkishik
[10]
reports on the challenges of
dealing with poor mobile network signals in Saudi Arabia to make
LMS platforms such as Blackboard more accessible to the general
population. Furthermore, a study by Kinash et al
. [31]
found that
students were positive towards Blackboard mobile learning. Yet,
little is known about the extent to which Blackboard successfully
meets the expectations of students with disabilities because most
studies focused on general usability aspects.
Therefore, this paper aims to present an investigation we con-
ducted to learn how accessible Blackboard LMS is from the student’s
perspective. We thus employed two strategies. First, we surveyed
students to gather their perceptions of LMS compliance with general
usability and accessibility guidelines by considering Blackboard as
a case study. In this investigation, in addition to surveying students
without impairment, we surveyed students with hearing impair-
ment once there are many barriers they might face, especially when
online content is made available through multimedia artifacts (e.g.,
audio and video). Our study ndings will highlight the unanswered
accessibility issues that users are currently facing. Second, we lever-
age recent Blackboard public user reviews, the ocial medium for
mobile users to share their feedback with the app maintainers. In
this study, we identied comments regarding any type of disabil-
ity or accessibility barrier. User reviews represent the wisdom of
the crowd [
11
], and various successful apps have been known to
interactively respond to their user’s feedback by addressing their
concerns in the app’s newer releases [
14
,
38
,
53
]. To the best of our
knowledge, none of the previous studies assessed the accessibility
of the Blackboard mobile app platform using user reviews.
We framed our investigation around the following research ques-
tions:
RQ
1
: To what extent do students nd the Blackboard
mobile application easy-to-use?
This research question discovers the extent to which students
are able to use the Blackboard application. To do so, we performed
a large-scale survey with 1,373 students and 65 deaf and Harding
of hearing students. This 5-questions survey targets the general
usability of blackboard, especially when being the main learning
medium, given that most universities are currently oering online
courses due to the COVID-19 pandemic. We also conducted follow-
up interviews with 8 students to reect on the ndings of the
survey.
RQ
2
: What accessibility issues are reported by the users
of Blackboard app?
Since the ndings of our previous research question cannot be
generalized, we also decided to explore user reviews for further
analysis. To address this research question, we crawled and an-
alyzed 15,478 user reviews publicly posted by Blackboard users
on the Google Play Store. We used quantitative and qualitative
procedures to lter out these reviews and extract only accessibility-
related ones. Our ndings will inform app developers of the most
common accessibility issues so that they can be resolved in current
and future applications. Also, our curated set of reviews is available,
as part of our replication package, for reproducibility and extension
purposes1.
2 BACKGROUND
2.1 Mobile Accessibility Standards/Guidelines
Apps accessibility in mobiles is controlled by benchmarks and stan-
dards stipulated by the World Wide Web Consortium (W3C)
2
. The
W3C, through the Web Accessibility Initiative (WAI), provides a
range of guidelines and standards that are frequently updated to
address any emerging issues in accessibility. The guidelines mainly
include Web Content Accessibility Guidelines (WCAG), User Agent
Accessibility Guidelines (UAAG), Accessible Rich Internet Appli-
cations (WAI-ARIA), and Authoring Tool Accessibility Guidelines
(ATAG). Although there are many standards on accessibility, those
that relate to mobile accessibility are yet to be developed, meaning
that the current guidelines in use are the W3C through the WCAG
2.0 principles. Such rules apply to native apps, mobile web apps,
and web content. WCAG denes accessibility in terms of four prin-
ciples which include ease of operation (operable), understandable
app content (understandable), robustness (robust), and coherent
app content (perceivable) [43, 49].
Other than the standards provided by W3C, other independent
organizations, such as the British Broadcasting Corporation (BBC),
have drafted their own accessibility standards. A document called
BBC Mobile Accessibility Guidelines by the BBC contains these
guidelines [
18
]. The rules contained in the BBC guidelines are
similar to those provided by the W3C [
54
]. These standards mainly
guide principles, audio and video, designs of the apps, focus, forms,
images, links, notications, scripts, and dynamic content, structure,
and text equivalence, as described in detail in Table 1.
Although BBC accessibility guidelines provide detailed instruc-
tions, there are additional standards given by WCAG 2.0 in relation
to mobile accessibility. Some of the additional guidelines include
rules on how to reduce content in the mobile version, eliminating
form elds that are beside their labels, accessibility of interactive
controls, noticeability of apps content, text minimization, provi-
sion of clear guidelines, and adjustment of the app to the dierent
orientation of the device. The accessibility guidelines provided by
BBC and WCAG are crucial. However, other organizations may also
develop accessibility guidelines to complement the existing ones.
2.2 Mobile Learning and LMS
With the increasing need to oer online education by universi-
ties and institutions worldwide, the adoption of learning manage-
ment systems (LMS) has also increased [
50
]. Indeed, mobile learn-
ing oers several benets, including location-based services, cost-
eectiveness, and education aid, among others. It has also been con-
sidered that LMS systems help improve students’ problem-solving
skills, performance, and knowledge and create an individualized
learning system [
30
,
50
]. Five authoring tools are considered part
of a learning management system: content collaboration, content
delivery, content development, content distribution, and content
management [
30
]. During the current Coronavirus (COVID-19) pan-
demic, the need to reduce physical interaction in higher education
1https://wajdialjedaani.github.io/MobileBB/
2https://www.w3.org/
85

The State of Accessibility in Blackboard: Survey and User Reviews Case Study W4A ’23, April 30–May 01, 2023, Austin, TX, USA
TVTC College
Survey Participants
(65)
Participants
(1,308)
Resul ts Analysi s
Hearin g Students
Deaf Students RQ1
15,478
Google Play
Store Blac kboard User Reviews Data
Preproces sing
Step 2
Disc ard
Emoj is & Lan g
Step 3
Disc ard Nois e
in Rev iews
Remov e Duplic ated
Reviews
Step 1
7,534
ML Tool
Final
Reviews
Non-Accessibility
Accessibility
Manual
labellin g
Resul ts
Analysi s
RQ2
Interview
Participants
(4)
Participants
(4)
Hearin g Students
Deaf Student s
Figure 1: Approach Overview.
institutions has increased the adoption of LMS to facilitate mobile
learning [
23
]. Hence, LMS systems have helped many institutions
deliver instruction exclusively online, where content is managed,
distributed, and delivered to students.
2.3 Blackboard
The Blackboard LMS provides a personalized intuition that helps
learners to engage with their tutors, provides data handling ca-
pabilities, and is exible to any teaching approach [
20
]. Matthew
Pittinsky and Michael Chasen developed Blackboard in 1997. It is
considered an excellent LMS system because it is easily available
on many devices, provides quick feedback, makes tracking easier,
and has better communication [
2
]. Learning institutions can choose
between two Blackboard systems: the Networked Transaction Envi-
ronment (NTE), which helps in supporting commercial transactions,
and the Networked Learning Environment (NLE), which oers aca-
demic capabilities that support online learning. Blackboard is one
of the most popular LMS systems today and is used by many insti-
tutions globally.
2.4 Technical and Vocational Training
Corporation (TVTC)
The Technical and Vocational Training Corporation (TVTC) was
established in 1980 and mandated to oer higher education in Saudi
Arabia. The institution oers vocational training and development,
imparting practical skills to students. Under the TVTC umbrella,
there are technical colleges, vocational training centers, and sec-
ondary institutions. Students can enroll in any of the dierent
TVTC branches distributed across Saudi Arabia. It is also important
to note that the TVTC is a regulatory body for nearly 1,000 private
training institutions in Saudi Arabia. Oering courses in various
technical majors, the institution is important for skills development
in Saudi Arabia [
52
]. We chose to conduct our case study at this
university because it has a signicant body of deaf and Harding of
hearing students.
3 RELATED WORK
Accessibility in online learning involves the addition of tools and
features into applications to meet the needs of all users [
4
–
6
]. It is
important for users to oer reviews regarding the accessibility of
education applications so that such feedback can be used to improve
the apps [7, 9, 43]. In this section, we divide the section into three
categories as follows: user reviews, accessibility in user reviews,
and LMS in the mobile application.
3.1 User Reviews
Scholars have found that analyzing user feedback helps gather
customers’ opinions and feedback on a given app, which helps in
improving future applications [
33
,
46
]. Getting the perspectives of
app users is a good step towards understanding what users desire in
the app and fullling such needs [
35
]. In most cases, developers get
user feedback through mobile devices [
48
], which they use in the
software lifecycle of future apps [
41
]. There are various techniques
for getting user reviews, including online surveys, opinion mining
from various online review sites, and sentiment analysis [
13
]. For
instance, Balachandran and Kirupananda used a sentiment analysis
tool to get online reviews for an online evaluation system in a
Sri Lankan university [
15
]. Most people do not understand the
importance of giving app feedback, and studies have shown that
only an estimated 1% of users share their reviews [
15
]. Commonly
used apps can be improved if users provide feedback [
25
,
56
]. For
example, if learners give feedback on their LMS apps, developers can
identify the services they desire and ensure that they are suciently
accessible by [
11
]. Furthermore, dierent apps present dierent
accessibility challenges depending on whether they are on the
iOS or Android platforms, and feedback on either platform greatly
86

W4A ’23, April 30–May 01, 2023, Austin, TX, USA Aljedaani, et al.
Table 1: List of the keywords used to identify user reviews refer to accessibility. We followed the BBC standards and guidelines
for mobile accessibility [18].
Guideline Description Relevant Keywords
These guidelines require a focus on three principles of developing usable and inclusive applications. First, developers
Principles should utilize all web standards as required. Secondly, there should be the utilization of interactive controls. Thirdly, content Accessibility, disability, screen reader, blind
and functionality in the app should support native features of the app. talkback, operable, impaired, impairment
Applications should provide alternative formats such as transcripts, sign language, or subtitles. Autoplay should be
Audio/video disabled, and the user should be provided with play/pause/stop or mute buttons to control audio. There should be no Subtitle, sign language, audio description,
conict between audio in application media of native assistive technology. transcript, autoplay, mute, volume, can’t hear
The color in the app background should have appropriate contrast, and touch targets must be large enough to be Contrast, background color, icker, visual cue,
Design touched eectively. Visible state change should be experienced in every item in the app that has been focused on. touch size, overlap, font size, dark/light mode,
Unnecessary or frequent ickering of content must be avoided. eyestrain, seizure, can’t see
There should be a logical organization of items, and users should be oered alternative input methods. Interactive
Focus and inactive elements should be focusable and non-focusable, respectively. Keyboard traps should be eliminated, and Focusable, control focus, keyboard trap, focus
focus should not change suddenly when the app is utilized. order, navigable, input/type
Forms Every form of control must have a label. All labels must have a logical grouping, and a default input format must be Unique label, missing label, visible label
given. Labels should be close to their form controls. layout, voice-over
Images Text images should not be included. Any background images that have content should have another accessible Image of text, hidden text, text alternative,
alternative. background image
Any navigation links must indicate the function of the link. If a link to an alternative format is clicked, the user
Links should be notied of the redirection to the alternative. Several links that redirect to the same source should be put Link description, unique desc., duplicate link,
together in one link. alternative format
Notications Error messages should be clear. Any notications given must be easily seen or heard. There should be standard Operating inclusive, haptic, vibration, feedback,
system notications where necessary. alert dialog, understandable, unfamiliar
Dyn. content Applications should be made in a progressive manner that enables every user to benet from them. Appropriate Animated content, page refresh, automatic
notications should be given for automatic page refreshes. Flexible interaction input control must be given. refresh, timeout, adaptable, input sign
Every page on the application should be uniquely identied. Content should be arranged in a hierarchical and logical.
Structure manner with appropriate headings. One accessible component should be used to group interface objects, controls or Page title, screen title, heading, header
elements. unique descriptive
Applications should give the objective of a specic image or its editorial aim. In addition, visual formatting must be
Text equivalent complemented by other ways to give meaning. There should be no conict between decorative images with assistive Alternative text, non-visual, content description
technology. Every element must have well-placed and eective accessibility properties. decorative content, no-text-content
facilitates app development [
26
]. Our study looks at user reviews
within the context of learning applications.
3.2 Accessibility in User Reviews
Various scholars have investigated accessibility in user reviews in
the past. For example, Eler et al. [
25
] evaluated user comments
about the accessibility of mobile applications and found that people
rarely provide accessibility-related reviews even when faced with
such issues. In another study, Yan and Ramachandran investigated
the accessibility of mobile apps by looking at their Graphical User
Interface (GUI) features and adherence to accessibility guidelines.
They found that a majority of the apps had violated accessibility
guidelines and had multiple accessibility issues [
56
]. Another study
by AlOmar et al. designed a model to help automatically identify
accessibility user reviews, which was found more accurate than
random classiers or keyword-based detectors [
11
]. We have noted
that all the previous studies did not focus on accessibility user
reviews in education applications, which is the focus of our current
study.
3.3 LMS in Mobile Application
Several studies have focused on learning management systems
in mobile applications. For example, Papadakis et al. found that
students use mobile phones as an electronic document repository
when accessing content from the Moodle platform. However, the
phones were limited in terms of their reliability and usability [
36
].
Another study by Liu et al. focused on the use of mobile apps in
inquiry-based learning (IBL), which found that updated functional
features promoted the use of IBL [
44
]. Albidinova et al. investi-
gated the development of a mobile application for learning in a
university and pointed out the need for such innovations in higher
education institutions after a piloting experiment [
34
]. In Saudi Ara-
bia, Arturki et al. [
1
] and Sahrir et al. [
12
] investigated the use of
Blackboard and IIUM I-Taleem, respectively, and documented their
usefulness in providing learning support for students. In Egypt,
applying the Easy-Edu LMS platform has proven helpful in uni-
versities because of its use of agile-based systems that enable the
detection and prevention of issues faster way [47].
Several scholars have investigated the accessibility of LMS edu-
cational applications. For example, a study by Batanero-Ochaita et
al. [
27
] evaluated the accessibility of learning management systems
by blind, deaf, and deaf-blind students and found that students
had positive perceptions towards the use of learning management
systems that were adapted to their needs because they improved
content accessibility. Furthermore, conducting a study at King Saud
University, Alturki et al. [
1
] found that Blackboard was usable and
accessible. However, there was a need to customize the LMS system
to cater to teachers’ needs. Furthermore, at the University of Dar es
Salaam in Tanzania, Mtebe et al. [
17
] found that the use of Mobile
Moodle made it easier to use the Moodle platform and enabled them
to do their learning activities more eectively.
Collecting user reviews of LMS systems has been considered one
of the strategies for increasing user satisfaction and improving the
experience of video education app users [
40
]. Some of the common
problems that could aect app accessibility include user interfaces,
structure, communicative features, and mobile features [
55
], as
well as inaccessible content, slow downloading of Moodle’s pages,
screen size, diculties in submitting assignments, among others
[
3
]. In the current study, we wish to explore the accessibility of
four android educational applications, which, to the best of our
knowledge, have not been addressed by previous studies.
87

The State of Accessibility in Blackboard: Survey and User Reviews Case Study W4A ’23, April 30–May 01, 2023, Austin, TX, USA
Table 2: Participants demographics information. Each partic-
ipant (P#) answered the interview questions.
Participant Age Major Year Student Type
P1 22 Electronic Engineering 2 Hearing
P2 24 Mechanical Engineering 4 Hearing
P3 22 Computer Networking 3 Hearing
P4 20 Electronic Engineering 1 Hearing
P5 23 Computer Technology 3 Deaf/Hard-of-hearing
P6 22 Business 2 Deaf/Hard-of-hearing
P7 21 Business 3 Deaf/Hard-of-hearing
P8 23 Computer Technology 4 Deaf/Hard-of-hearing
Table 3: Set of interviews questions.
First- Background and Demographics
Years of age, and study major
Do you use the Blackboard mobile application on your phone?
Second- Generic Views
How would you describe your experience while using the Blackboard mobile application?
Were you able to access the class materials via the Blackboard mobile application?
How often would you use the Blackboard mobile application?
Third- Accessibility Challenges
How easy was the application to use?
How is the navigation of the Blackboard mobile application?
Fourth- Students Recommendations
Are there any features that you think you need but are missing in the mobile application?
What do you think the Blackboard mobile application should improve on?
4 STUDY DESIGN
This section presents the details of our approach used in this study,
as provided in Figure 1. The information covered in this section
contains the survey details, interview procedures, and user reviews.
This section provides the details of our survey with 1,373 stu-
dents. Then we elaborate on the follow-up interviews with 8 stu-
dents. To get more insight into the users’ reports about accessibility-
related, we collected all the user reviews related to the Blackboard
app. Next, we detail our ltration process to identify whether user
reviews were accessibility-related or non-accessibility-related. Fi-
nally, we explain our manual analysis to label the user reviews
based on the accessibility guidelines.
4.1 Survey
To get an overview of the issues surrounding the accessibility of the
Blackboard LMS platform, we conducted the survey at Technical
and Vocational Training Corporation (TVTC) college
3
, which was
the study’s location and focus. Our participants were divided into
hearing students (1,308 participants) and deaf students (65 partici-
pants). The questionnaire was in the Arabic language, which was
the native language of the respondents. We asked ve questions in
the survey, which are given in Table 4. We sent the questionnaire
using Google forms
4
, which made it easier and more convenient to
reach the respondents by sending them a link to the form. The anal-
ysis of the results was crucial in elaborating on students’ perception
of the accessibility of the Blackboard platform.
4.2 Interview
To complement the survey data, we conducted interviews so that
respondents would give us their views and opinions. Given the
3https://www.tvtc.gov.sa/index-en.html
4https://www.google.com/forms/about/
Table 4: Set of survey questions.
Q1- What is your Gender?
Male
Female
Other
Q2- What is your major?
Computer technology and related elds
Business and related elds
Mechanical and related elds
Electronic and related elds
Electrical and related elds
Telecommunication and related elds
Food Processing Technology and Related to It (Food Processing)
Chemical and related elds
Tourism and Hospitality and related elds
Civil, Architectural and related elds
Other
Q3- How satised are you with using the Blackboard platform on
your mobile phone?
Extremely satised
Satised
Neutral
Dissatised
Extremely dissatised
Q4- Based on your experience using the Blackboard application on
your mobile phone, how easy and user-friendly is the app for you?
Extremely easy
Easy
Neutral
Dicult
Extremely dicult
Q5- What would you like to see improved in the Blackboard mobile app?
Open-end question
importance of validity in interviews, we utilized investigator trian-
gulation. We used a voluntary sample of 8 students, four deaf and
four hearing students. We created an interview schedule with open-
ended and closed-ended questions. The interview’s semi-structured
nature allows respondents to reect, in more in-depth, on their
answers. The nine questions used in the interview are given in
Table 3.
We conducted the interviews using the Zoom platform
5
and
used the Arabic language, which was the native language of the
respondents. We oered a $25 prepaid gift card to motivate the in-
terviewees to participate. It is important to mention that, during the
interviews with deaf students, we hired a sign language interpreter
as an accommodation for the students because none of us was
competent in sign language. During the interviews, students were
encouraged to comment on any usability or accessibility issues they
experienced. Thus, deaf students also mentioned interface issues
unrelated to hearing impairment. The demographic information of
the interviewees is given in Table 2.
Following the interviews, we transcribed the data and translated
it from Arabic to English. The translation accuracy was ensured by
all the authors separately as follows: when the rst author trans-
lated the work from English to Arabic, it was passed on to the second
author and later to the third author, who separately compared the
translated scripts to the original Arabic ones. We utilized thematic
analysis to analyze the qualitative data. We perused through the
interview transcripts, created codes, revised them, and deduced the
themes from the pattern of answers given.
5https://zoom.us/
88

W4A ’23, April 30–May 01, 2023, Austin, TX, USA Aljedaani, et al.
Table 5: Present an example of the eliminated reviews.
Step Example
Emojis 
language Algunas veces las videoconferencias no se conectan
Noise Love it!!
4.3 User Reviews Collection and Preprocessing
The rst stage in this section was to get user reviews on the acces-
sibility of the Blackboard platform, where we collected the reviews
from the Google Play Store [
29
]. We collected all the reviews relat-
ing to Blackboard, and a total of 15,478 reviews were received. The
next step was data preprocessing which involved three steps:
•
Step (1)- Discard Emojis & Languages: We removed
any reviews that only contained emojis or images, such as
thumbs up and others. These reviews that include only emo-
jis are not helpful and assist us in understanding the acces-
sibility issues in the reviews. We also removed any reviews
written in languages other than English, such as Chinese
or Arabic, since our study was in English. After mining the
collected data, we eliminated 292 reviews containing only
emojis or written in a dierent language than English.
•
Step (2)- Remove Duplicated Reviews: We removed any
reviews that were posted twice or severally by the same
user. We eliminated such duplicate reviews because they
were only repeating themselves and adding no value to the
dataset. In this Step, we eliminated 1,670 reviews, and we
only kept the unique reviews in the dataset.
•
Step (3)- Discard Noise in Reviews: We discarded noise
by removing all reviews that were in less than ve words
format, such as ’super’ or ’awesome’ or ’great’ or ’good app’
etc., which were not useful in getting user feedback. This
step involved removing 5,982 reviews. Table 5 presents an
example of three steps of data preprocessing.
4.3.1 User Reviews Filtering.After the data preprocessing, we
remained with 7,534 reviews subjected to machine learning to know
whether the user reviews were related to either accessibility or non-
accessibility. To do so, we used our previous model [
11
] to help us
automatically identify the type of user reviews. This means we put
all the Blackboard user reviews as an input of the model, and the
model determines the two subsets of the dataset, accessibility and
non-accessibility, as an output. We used this model to reduce the
human eort needed to lter the user reviews manually. After we
utilized the ML model, the model’s output was distinguished from
3,813 reviews (50.61%) out of the 7,534 reviews as accessibility.
4.3.2 User Reviews Labeling.Since we are using machine learn-
ing to identify accessibility user reviews, there could be user reviews
that are not related to accessibility, which can be a false positive of
automated detection. Therefore, to address this issue, we performed
a manual analysis to lter the false positive reviews and classify
the reviews based on the accessibility guideline in Table 1. More
precisely, we employed three-step iterations, described in content
analysis method [
42
,
45
] involving two of the authors of this paper,
who have complementary expertise in line with the goal of our
analysis. The rst author is a software engineer with four years
of working on mining software repositories for mobile and pub-
lishing more than two papers in the accessibility eld. The second
researcher is a bachelor’s student in computer engineering and
has two years of experience in Natural Language Processing (NLP)
techniques for mobile computing.
From this point forward: we introduce both of them as inspectors;
they label a total of 3,813 user reviews, each using the approach
outlined below:
Iteration (1): In the rst stage: the inspectors analyze the
3,813 user reviews individually. The inspectors read all the user
reviews during the analysis process and strive to identify any non-
accessibility user reviews labeled as accessibility (false positive).
After each inspector completed the labeling, the inspectors dis-
cussed the false positive user reviews. In the discussion, the two
inspectors intend to stimulate a consensus. Afterward, the inspec-
tors decided to eliminate 1,498 reviews. Thus, the inspectors ended
up with 2,315 user reviews.
Iteration (2): In the second stage: the inspectors categorize the
2,315 accessibility user review proceeded from the rst iteration.
Both of the inspectors aimed to categorize based on the guidelines of
the BBC standards and guidelines for mobile accessibility [
18
] and
described in Table 1. During the categorizing process, the inspectors
allow categorizing the user review and labeling them with one or
more guidelines. After each inspector completed the labeling, the
inspectors opened a discussion about the process of categorizing
reviews. While discussing, the inspectors identied an issue during
this iteration.
Review 1. “I can’t review some content. That’s kind of
important in school.”
Review 2. “It doesn’t load anything anymore on my
phone ever so disappointed”
Review 3. “The app is really easy and accessible, but it
doesn’t always load the pages that I need”
The above examples demonstrate one thing in common: they do
not elaborate enough to make them seem like accessibility reviews.
Afterward, the inspectors decided these false positives were still
non-accessible, even though they were worded in an accessible way.
Hence, the inspectors eliminated 13 reviews. So, the inspectors
ended up with its iteration with 2,302 accessibility user reviews.
Iteration (3): After the second iteration, where the inspectors
categorized 2,302 accessibility user reviews based on the guidelines,
they concentrated on dealing with the multi-guideline reviews.
There were 164 accessibility user reviews belonging to two guide-
lines. The inspectors have to decide for each review what is the
deciding factor in labeling each guideline. The inspectors opened
a discussion about the multi-guideline reviews. The following is
an example of one of the user reviews that was labeled as multi-
guidelines.
“This app is awful, the last one wasn’t great, but it
was still better than this one. It sends me notications
for the same grade like 10 times, and half of the links
don’t work, and it’s always glitching.”
89

The State of Accessibility in Blackboard: Survey and User Reviews Case Study W4A ’23, April 30–May 01, 2023, Austin, TX, USA
The review listed above has a couple of problems, yet some do
not take precedence over others, as this example was labeled noti-
cations and links guidelines. After the discussion, the inspectors
decided to label the review to the primary concern of the review,
which is link guidelines.
Additional validation: To validate the procedures executed
by the inspectors, who individually examined and labeled all the
accessibility user reviews related to Blackboard reviews, we fol-
lowed the directions of Aljedaani et al. [
8
] by picking a 9% sample
of the entire data set (239 out of 2,302 reviews). The selected sample
satised the 95% condence level, while the condence interval was
6. Then, we randomly selected 239 reviews out of the 2,302 reviews.
Afterward, the selected sample was given to two of the authors for
labeling them. The selected data were not previously disclosed to
the authors. The review procedure lasted for seven days to avoid
fatigue. During the labeling process, the authors had the ability to
look for terms/keywords online that they could not understand.
The labeling of the data was followed by a comparison with the
labeled reviews from the original dataset. Finally, we investigated
the inter-rater agreement level between the two datasets using
Cohen’s Kappa Coecient [
22
], which gave us an agreement level
of 0.87. As noted by Fleiss et al. [
28
], an agreement level between
(i.e., 0.81–1.00) implies almost perfect agreement.
5 STUDY RESULTS
RQ
1
: To what extent do students nd the Blackboard
mobile application easy-to-use?
In this question, we wanted to gauge students’ experience with
the Blackboard mobile app, from a usability perspective. According
to the responses outlined in Figure 2, the majority (85%) of the deaf
students found Blackboard extremely dicult to use. We found the
high dissatisfaction rate with Blackboard among deaf students to
be a signicant source of concern and sought more explanation
from the students. This is particularly interesting since only 12.1%
of hearing students have found it extremely dicult (cf. Figure 3).
One of the essential features of an LMS system is having a suit-
able interface for the needs of deaf students. Compared to hearing
students, deaf students have e-learning challenges requiring LMS
systems to have the complicity, consistency, navigation, and proper
typography.
This nding has driven our interview to seek more insights about
why deaf students experience diculties when using Blackboard.
As mentioned in Section 4, deaf students could freely refer to any
issues that make it challenging to use Blackboard, not only barriers
related to hearing impairment. For instance, participant P6 had a
problem with the Blackboard interface and explained that:
“Blackboard was not a friendly interface. I had an issue
locating the exam component since there are a lot of
headers and sub-headers in the navigation bar, and the
font was tiny and hard to read.” (P6)
In our study, some students could not access the materials. P8
who faced such a challenge said that:
“I usually like education apps so that it can be more
convenient to learn from my phone. However, I Can’t
watch videos or download les in the blackboard app.
Then, I stopped using the app and switched to the web
version.” (P8)
P5 noted that:
“Because I am deaf and rely on visualization, I cannot
have the video caption in the app for videos.” (P5)
Deaf students also had problems when doing exams due to a lack
of pictures, and (P7) said:
“I liked the app, and it was easy to use. My issue always
was in the exam where the picture is not shown in the
exams, and I missed a few questions for that reason.”
(P7)
Although previous studies indicated that deaf students had posi-
tive perceptions towards learning management systems [
27
], this
study reveals another dimension of accessibility problems that neg-
atively impact a subset of students. Therefore, we compared the
survey results of deaf students with hearing students, and the re-
sults are given in Figure 2 and Figure 3.
Extremely dicult
85%
Dicult
1%
Natural
9% Easy
2% Extremely easy
3%
Figure 2: Percentage of the deaf students (no. 65) participated
in the survey.
Extremely dicult
12.1%
Dicult
9.9%
Natural
20%
Easy
19%
Extremely easy
39%
Figure 3: Percentage of the hearing students (no. 1308) par-
ticipated in the survey.
Compared to deaf students, most hearing students (39%) found it
extremely easy to use the Blackboard LMS application, 20% found
it natural, and only 10% and 12% found the platform complex and
extremely dicult respectively, as shown in Figure 3.
We wanted to identify the reviews of those students who found it
dicult to use the platform. In our study, some students complained
that materials were inaccessible. P2 said that:
90

W4A ’23, April 30–May 01, 2023, Austin, TX, USA Aljedaani, et al.
“When we were locked out in the COVID-19 pandemic,
I used the application because I had no laptop to access.
I tried to access the class material via the Blackboard
app, but it is not opening documents. So, I have to ask
my colleagues to share the class materials.” (P2)
The user-friendliness of an education app is crucial because it
indicates the ease of use of the application by students. According
to the respondents, the site layout of the app, navigation labels, and
overall app design should make it appealing to users, which was
not the case for Blackboard. P1 noted that:
“I used several education apps to teach my younger
brothers, and they were easy and simple to use. For
my studies, I used the Blackboard. I found it a highly
complex app to navigate if you don’t know what you’re
doing. It needs to be more user-friendly since it is an
educational app.” (P1)
P3 also highlighted that some links were not working:
“I love the app, and I used to study my class materials
on it. Sometimes links in the app are not working, and
the show me page was not found.” (P3)
It would be plausible to suggest that the app was not user-friendly
and easy to use. In a bilingual or multilingual learning environment,
app users should be able to switch between languages. For instance,
Arabic and English are often used in Saudi Arabia, even though
some people only understand one of the languages. For the case
of the students we interviewed, their native language was Arabic,
and they indicated that changing the language in Blackboard was a
problem. According to (P4):
“When I downloaded the app, it was in the English
language. I tried to change the language to Arabic, but
I could not. I asked my father to change the language
for me, but I realized that with the Arabic language,
the application layout was weird and hard to read or
navigate.” (P4)
These results are consistent with previous studies [
1
] that have
found Blackboard LMS usable and accessible, although improve-
ments to the platform could make it easier to use.
RQ
2
: What accessibility issues are reported by the users
of Blackboard app?
In this second research question, we were looking at the user
reviews given about Blackboard on Google Play Store. Table 8
shows the number of reviews and their percentages in relation to
the various guidelines.
From Table 8, the majority of the reviews (1062 representing
46.13%) related to Principle, followed by Focus (24.15%), and No-
tications (11.60%). Table 7 presents an example of accessibility
reviews relating to each guideline. The principle guideline requires
that apps be easy to operate, accessible to all users, robust in use,
and understandable. From the reviews we analyzed, the principle
guideline was not met, and one of the reviews indicated that:
“For a student who is Blind that uses accessibility soft-
ware, JAWS, a screen reader that the Blind uses to oper-
ate on a computer, the Blackboard application does not
Table 6: Present the results of the accessibility reviews after
labeling.
Guideline # of Reviews % Percentage
Principle 1062 46.13%
Focus 556 24.15%
Notications 267 11.60%
Design 145 6.30%
Forms 93 4.04%
Audio/video 69 3.00%
Links 59 2.56%
Dyn.content 38 1.65%
Images 10 0.43%
Editorial 3 0.14%
Structure 0 0.00%
Text equivalent 0 0.00%
Total 2,302
function correctly when jaws require that some activa-
tion requires the user to do a double tap to activate some
functions of radio dialogues, especially when taking an
exam to open the test and when navigating to the next
test question the application did not move to the next
question but went and submitted the exam test.no be.”
The focus guideline requires that the page be navigable, focusable,
and the input type be compatible with the users. In relation to the
Focus principle, one of the reviews suggested that:
“Not user friendly. Especially disappointed that the word
"organization" is spelled incorrectly throughout the app.
How can one take an app like this seriously if the de-
signer can’t spell, and nobody else corrects them???”
On the Notications principle, students should be able to get
notications on updates in the courses, and upload materials, among
other things, even without their laptops. If the app does not provide
notications, students may miss important communication from the
university or lecturers, which means missed learning opportunities.
For an application such as Blackboard, which many universities
use, it was unfortunate that some users said it lacked notications.
One of the reviews said:
“Fix the goddamn notications I missed requirements
because of this!”
For the design guideline, a learning app should have proper in-
teractive elements, consistent navigation options, an appropriate
background color, provide light/dark models, reduce eye strain on
the users, and have a good font size. However, some users did not
nd Blackboard to have a good design, and one of the reviews
indicated that:
“It works, but what’s with the zany button click anima-
tions, etc? They’re really distracting, I’d like to see them
toned down a good bit.”
91

The State of Accessibility in Blackboard: Survey and User Reviews Case Study W4A ’23, April 30–May 01, 2023, Austin, TX, USA
Table 7: An example of the accessibility reviews related to each guideline.
Guideline Example
for a student who is Blind that uses accessibility software, JAWS, a screen reader that the Blind uses to operate on a
computer, the Blackboard application does not function correctly when jaws require that some activation requires the user
Principle to do a double tap to activate some function actions of radio dialogues, especially when taking an exam to open the test and
when navigating to the next test question, the application did not move to the next question but went and submitted the
exam test.no be.
Design Please add dark mode, plus the feature to customize, organize and categorize modules, really need it. And I need to
re-login almost every day, please make the login stick.
Notications What’s the point if it doesn’t even have notications to remind you of anything? You might as well just use the browser
version and auto sign-in every time.
Focus Hard to navigate and often causes glitches in test submission. Dicult to nd what you need.
Forms After so many updates and this app still has the same problem. It does it open all the les (PDF, PowerPoint) they do
not load.
I downloaded this app in the hopes that the inset videos would be easier to watch on my phone. The web browser doesn’t
Audio/video allow you to see the whole video, just a cropped amount. Unfortunately, the app does not allow for videos, and I just see
code instead. But I would really like to see video capability added to this app. I do like that it has due dates listed for
assignments, and for that alone, I am happy to keep the app for now. Overall though, it is somewhat unsatisfying.
Links Absolutely awful. The app is entirely unusable. Every link opens in an embedded browser and results in an error.
Dyn.Content Some of the worst UI designs I’ve ever seen in an app. There are so many unnecessary animations tied to small actions;
you can’t click on a link without something wobbling or showing a folder opening up and papers falling out. The worst
part is, the animations clearly take up a huge amount of resources because the app will actually lag before they show,
which makes the whole app feel clunky and slow. If they got rid of all of these things, the app would at least feel usable.
Images Unable to view any images sent by the instructor (sent individually or inside a quiz).
Editorial Activity steam updates after a very long time. Also, when opening a new announcement, it opens old announcements, not
the recent one. The push notications come on after more than 24 hours. Denitely not pleased about the app and how
slow it responds to everything. Also, I can’t hear my collaborative sessions, although all my microphone settings have been
activated to be on and can be accessed by blackboard.
It is important to note that the issues we found aecting accessi-
bility from the user reviews have been reported in previous studies
[
3
,
55
]. Therefore, grouping them in terms of the guidelines helped
us to classify them for easier identication.
Additionally, we attempted to determine the nature of the issues
that app users reported. We outlined 16 pertinent issues faced by
the users in Table 8, along with the frequency of reviews that re-
port those issues. We also provide a sample review related to each
category of the listed issues. We observed that the majority of the
reviews are related to the anomalies associated with notications,
followed by grade visibility. Users also encountered problems such
as submitting assignments, homework, and PDF les, among others.
We established that even though Blackboard is a widespread appli-
cation, previous studies have shown that user feedback can benet
such applications by enabling developers to make improvements
[
25
,
56
]. Therefore, it would be important for developers to address
the accessibility issues indicated for a better user experience.
6 STUDY DISCUSSION
The results of our case study give a broad understanding of the
accessibility challenges currently experienced by users in perceiv-
ing, understanding, and operating the Blackboard LMS. Our results
show that most hearing students nd Blackboard easy to use, yet
many have issues related to navigation, accessibility of materials,
and changing language, among others. On the other hand, the ma-
jority of non-hearing students nd Blackboard extremely dicult
to use, not only due to accessibility barriers but also due to other
usability problems.
In that sense, our study gives some evidence of the accessibility
issues of Blackboard and their associated consequences, in addition
to pinpointing areas where Blackboard mobile application needs to
be improved for a better user experience. We believe our ndings
will show the importance of delivering accessible educational sys-
tems and benet the developer community by showing necessary
improvements not only to Blackboard but to any LMS that shares
similar features. Accordingly, we present some takeaways from our
investigation.
Takeaway 1: Both deaf and hearing students frequently rely on
videos that have been uploaded on Blackboard to learn the contents
of a course. Therefore, they should be able to easily access and
download videos and material for later and further studies. This
requirement may seem obvious, but it is important to show evidence
that many students have problems nding and downloading study
materials.
Takeaway 2: Lack of captions in videos causes problems for
deaf students. Deaf students cannot hear what the teacher is saying
and solely rely on captions. Unfortunately, some of the videos on
Blackboard did not have captions according to students, making
it challenging for deaf students to perceive and understand the
content of any class. This problem happens both with live-streaming
content and pre-recorded videos. One of the limitations of making
available videos with proper captions is that it heavily depends
on the content generator (e.g., teachers and instructors). In that
92

W4A ’23, April 30–May 01, 2023, Austin, TX, USA Aljedaani, et al.
Table 8: Categories of issues reported by the users during their app usage.
No. Issue Type # of Reviews Example
1 Quiz 19 Quizzes with images don’t show the image (so I can’t tell what label B is pointing at, for instance). Notications show up repeatedly for the
same things even after you’ve dismissed them, and there is no way to adjust settings. This is the fastest I’ve ever uninstalled an app.
2 Homework 40 This app crashes a lot. Sometimes the app works, sometimes it doesn’t, and given the fact that students lile myself have to upload homework
assignments it is very fustrating knowing that this app could crash when the assignment you must summit is due.
This app is the bane of my existence. It glitched while saving a draft of an exam I was writing, and crashed whenever I attempted to open the
3 Exam 17 draft, causing me to lose time and have to email the professor my exam submission. Beyond that, it has frequent glitches and often refuses to
work for no apparent reason.
4 Material 43 This doesn’t not work. The previous app allowed access to my school and the material. All of the sudden my school is no longer available and
the app doesn’t work or recognize my University. Please bring back the old Mobil app.
5 Lecture Recording 16 The most recent update has broken echo360 lecture recordings can’t watch them anymore.
6 Course Content 39 I can never see full course content and it doesn’t allow me to submit my assignments.
7 Grade Visibility 220 Use to work great, until this year. Can’t view my grades and some content. When I click on grades it says "something went wrong". Tried
reaching out to them and put in a help ticket, still nothing 2 weeks later.
8 Assignment 179 I can never see full course content and it doesn’t allow me to submit my assignments.
9 Announcement 125 I hate that you have color coded grades. This app doesnt update regularly causing me to miss announcements and content postings for classes.
10 Video 67 Worst app Couldn’t load Videos,recorded lecture and lecture slides. Durning class I couldn’t see the lecture Worst experience.
11 PDF 49 The app has become pretty useless as it doesn’t recognise my pdf or doxc readers and won’t open any les. The only thing I can check are
updated grades, but often I get a notication and then the grade won’t show within the app.
12 lecture 41 Can no longer stream a lecture while doing something else. Lecture stops if not in foreground. Old app was better/easier to use.
13 Image/Figure 20 Unable to view any images sent by the instructor (sent individually or inside a quiz).
14 Audio 9 It’s good but i can’t open the content if it’s an Audio.
15 Link 83 Can not access to any links that provided by proesors.
16 Notication 436 Crashes constantly and sends the same notication repeatedly after I look at the app and clear the notication.
sense, we believe any LMS should have a policy to enforce uploaded
videos to be properly captioned. That could be done automatically
or by manual inspection.
Takeaway 3: The use of a Graphical User Interface (GUI) in LMS
systems is essential for deaf students because it enables them to
visualize what they are learning. It is crucial to have high-quality im-
ages that are informative to students. Our ndings further highlight
the signicance of user-friendly apps, which enable users of bilin-
gual or multilingual environments to switch between languages
quickly.
Takeaway 4: Students should be able to eciently and eec-
tively utilize the app, attend lessons, take assignments, and even
download materials. It is worth mentioning that students should be
able to see all les that have been uploaded and be able to download
them, provide alternative access to learning materials, and use the
correct format that can be opened using common programs. Again,
some of the responsibility lies in the content creator. In that sense,
the LMS should have a policy to enforce alternative formats to
study materials made available to students.
Takeaway 5: The ndings from RQ
2
indicated that the Black-
board application lacked certain application guidelines that are
required for a better user experience for students with disabilities
and those without disabilities. We classied those guidelines to
identify the corresponding issue more precisely in terms of prin-
ciples, focus, notications, and design. We further outline the key
implications obtained from RQ2:
•
Principle Guideline: For students with disabilities and those
without, it requires that every user be able to navigate
through the platform, know the information that is pre-
sented, understand it, and frequent upgrades are made to
improve app accessibility.
•
Focus Guideline: All content should be suciently described
with unique labels, and there should be logical and intuitive
navigation order of focusable elements.
•
Notications Principle: To prevent students from missing
training and learning opportunities, students should be noti-
ed of any updates about coursework.
•
Design Guideline: There should be visual cues, as well as form
elements that have clearly associated labels. For students
with visual impairments, accommodations should be made
for them so they can access the materials.
7 THREATS TO VALIDITY
In this section, we present potential threats to the validity of our
study.
Detecting accessibility user reviews. We used a previously
implemented model that used supervised learning to formulate
the identication of accessibility reviews as a binary classication
[
11
]. However, there might be a false positive in the process of
automatically tagging the reviews as accessibility. To migrate this
issue, we manually read the 2,302 reviews while we labeled them
in the guidelines, and we made sure that there was no such case
where data was labeled falsely by automatic processing.
Inclusion of all possible keywords in the labeling process.
There are dierent keywords that users use to express their con-
cerns related to accessibility. One potential concern is whether the
set of keywords belonging to each of the guidelines used in this
research covers all possible keywords. For mitigating this threat,
keywords dened by [
25
] have been employed in our study. More-
over, variants of these keywords have been adopted to ensure that
the authors do not miss any relevant review during the manual
validation and labeling process.
Sample size. The number of interviewed students does not be
representative. Therefore, we had to mitigate this risk by perform-
ing a survey of 1,308 hearing students and 65 deaf and hard-of-
hearing students. We have also mined the issues in the app store to
target a wider population of users as well.
93

The State of Accessibility in Blackboard: Survey and User Reviews Case Study W4A ’23, April 30–May 01, 2023, Austin, TX, USA
8 CONCLUSION
This study explored the student perception and user reviews of the
Blackboard app from the accessibility standpoint. We believe that
our research would contribute to the existing literature on the ease
of use of educational applications and be the rst study to utilize
Blackboard user reviews from the Google Play Store to analyze the
accessibility of the application. We established that most students,
especially the deaf ones (85%), found it extremely dicult to use the
Blackboard application. Similar results were conrmed when we
analyzed user reviews from the Google Play Store, where (31%) of
the user reviews were related to accessibility. Some of the reasons
that made the Blackboard application inaccessible were the lack
of notications, unavailability of captions, distracting animations,
diculty changing language, and lack of video captions. Our results
provide valuable insights for educational application developers to
improve the accessibility and usability of the applications.
REFERENCES
[1]
Gulmira M Abildinova, Aitugan K Alzhanov, Nazira N Ospanova, Zhymatay
Taybaldieva, Dametken S Baigojanova, and Nikita O Pashovkin. 2016. Developing
a Mobile Application" Educational Process Remote Management System" on the
Android Operating System. International Journal of Environmental and Science
Education 11, 12 (2016), 5128–5145.
[2]
Abdulaziz Aldiab, Harun Chowdhury, Alex Kootsookos, Firoz Alam, and Hamed
Allhibi. 2019. Utilization of Learning Management Systems (LMSs) in higher
education system: A case review for Saudi Arabia. Energy Procedia 160 (2019),
731–737.
[3]
Abdulsalam K Alhazmi, Athar Imtiaz, Fatima Al-Hammadi, and Ezzadeen Kaed.
2021. Success and Failure Aspects of LMS in E-Learning Systems. International
Journal of Interactive Mobile Technologies 15, 11 (2021).
[4]
Wajdi Aljedaani, Mona Aljedaani, Eman Abdullah AlOmar, Mohamed Wiem
Mkaouer, Stephanie Ludi, and Yousef Bani Khalaf. 2021. I cannot see you—the
perspectives of deaf students to online learning during covid-19 pandemic: Saudi
arabia case study. Education Sciences 11, 11 (2021), 712.
[5]
Wajdi Aljedaani, Mona Aljedaani, Mohamed Wiem Mkaouer, and Stephanie
Ludi. 2023. Teachers Perspectives on Transition to Online Teaching Deaf and
Hard-of-Hearing Students during the COVID-19 Pandemic: A Case Study. In 16th
Innovations in Software Engineering Conference. 1–10.
[6]
Wajdi Aljedaani, Rrezarta Krasniqi, Sanaa Aljedaani, Mohamed Wiem Mkaouer,
Stephanie Ludi, and Khaled Al-Raddah. 2022. If online learning works for you,
what about deaf students? Emerging challenges of online learning for deaf and
hearing-impaired students during COVID-19: a literature review. Universal access
in the information society (2022), 1–20.
[7]
Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, and Yasir Javed.
2022. Automatic classication of accessibility user reviews in android apps. In
2022 7th international conference on data science and machine learning applications
(CDMA). IEEE, 133–138.
[8]
Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, Ali Ouni, and Ilyes
Jenhani. 2022. On the identication of accessibility bug reports in open source
systems. In Proceedings of the 19th international web for all conference. 1–11.
[9]
Wajdi Aljedaani, Furqan Rustam, Stephanie Ludi, Ali Ouni, and Mohamed Wiem
Mkaouer. 2021. Learning sentiment analysis for accessibility user reviews. In
2021 36th IEEE/ACM International Conference on Automated Software Engineering
Workshops (ASEW). IEEE, 239–246.
[10]
Ayman N Alkhaldi and Abdallah M Abualkishik. 2019. The mobile blackboard
system in higher education: Discovering benets and challenges facing students.
International Journal of Advanced and Applied Sciences 6, 6 (2019), 6–14.
[11]
Eman Abdullah AlOmar, Wajdi Aljedaani, Murtaza Tamjeed, Mohamed Wiem
Mkaouer, and Yasmine N El-Glaly. 2021. Finding the Needle in a Haystack: On
the Automatic Identication of Accessibility User Reviews. In Proceedings of the
2021 CHI Conference on Human Factors in Computing Systems. 1–15.
[12]
Uthman T Alturki, Ahmed Aldraiweesh, et al
.
2016. Evaluating the usability and
accessibility of LMS “Blackboard” at King Saud University. Contemporary Issues
in Education Research (CIER) 9, 1 (2016), 33–44.
[13]
Asm Iftekhar Anam and Mohammed Yeasin. 2013. Accessibility in smartphone ap-
plications: what do we learn from reviews?. In Proceedings of the 15th International
ACM SIGACCESS Conference on Computers and Accessibility. 1–2.
[14]
Elsa Bakiu and Emitza Guzman. 2017. Which feature is unusable? Detecting
usability and user experience issues from user reviews. In 2017 IEEE 25th Interna-
tional Requirements Engineering Conference Workshops (REW). IEEE, 182–187.
[15]
Luxchippiriya Balachandran and Abarnah Kirupananda. 2017. Online reviews
evaluation system for higher education institution: An aspect based sentiment
analysis tool. In 2017 11th international conference on software, knowledge, infor-
mation management and applications (SKIMA). IEEE, 1–7.
[16]
Mars Ballantyne, Archit Jha, Anna Jacobsen, J Scott Hawker, and Yasmine N El-
Glaly. 2018. Study of accessibility guidelines of mobile applications. In Proceedings
of the 17th international conference on mobile and ubiquitous multimedia. 305–315.
[17]
Concepción Batanero-Ochaíta, Luis De-Marcos, Luis Felipe Rivera, Jaana
Holvikivi, José Ramón Hilera, and Salvador Otón Tortosa. 2021. Improving
Accessibility in Online Education: Comparative Analysis of Attitudes of Blind
and Deaf Students Toward an Adapted Learning Platform. IEEE Access 9 (2021),
99968–99982.
[18]
BBC. 2020. The BBC Standards and Guidelines for Mobile Accessibility. https:
//www.bbc.co.uk/guidelines/futuremedia/accessibility/mobile.
[19]
Nigel Bevan, James Carter, and Susan Harker. 2015. ISO 9241-11 revised: What
have we learnt about usability since 1998?. In International Conference on Human-
Computer Interaction. Springer, 143–151.
[20]
Blackboard. 2021. Blackboard Learn: An Advanced LMS. https://www.blackboard.
com/en-eu/teaching- learning/learning-management/blackboard- learn.
[21]
Witold Chmielarz. 2020. The usage of smartphone and mobile applications from
the point of view of customers in Poland. Information 11, 4 (2020), 220.
[22]
Jacob Cohen. 1960. A coecient of agreement for nominal scales. Educational
and psychological measurement 20, 1 (1960), 37–46.
[23]
Claudiu Coman, Lauren
t
,
iu Gabriel
T
,
îru, Luiza Mese
s
,
an-Schmitz, Carmen Stan-
ciu, and Maria Cristina Bularca. 2020. Online teaching and learning in higher
education during the coronavirus pandemic: students’ perspective. Sustainability
12, 24 (2020), 10367.
[24]
José-Manuel Díaz-Bossini and Lourdes Moreno. 2014. Accessibility to mobile
interfaces for older people. Procedia Computer Science 27 (2014), 57–66.
[25]
Marcelo Medeiros Eler, Leandro Orlandin, and Alberto Dumont Alves Oliveira.
2019. Do Android app users care about accessibility? an analysis of user reviews
on the Google play store. In Proceedings of the 18th Brazilian Symposium on
Human Factors in Computing Systems. 1–11.
[26]
Marcelo Medeiros Eler, José Miguel Rojas, Yan Ge, and Gordon Fraser. 2018.
Automated accessibility testing of mobile apps. In 2018 IEEE 11th International
Conference on Software Testing, Verication and Validation (ICST). IEEE, 116–126.
[27]
Menna Elkhateeb, Abdulaziz Shehab, and Hazem El-Bakry. 2019. Mobile learning
system for egyptian higher education using agile-based approach. Education
Research International 2019 (2019).
[28]
Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, et al
.
1981. The measurement
of interrater agreement. Statistical methods for rates and proportions 2, 212-236
(1981), 22–23.
[29]
Google. 2021. Google Play Store. https://play.google.com/store?hl=en_US&gl=
US.
[30]
J Hemabala and ESM Suresh. 2012. The frame work design of mobile learn-
ing management system. International Journal of Computer and Information
Technology 1, 2 (2012), 179–184.
[31]
Shelley Kinash, Jerey Brand, and Trishita Mathew. 2012. Challenging mobile
learning discourse through research: Student perceptions of Blackboard Mobile
Learn and iPads. Australasian journal of educational technology 28, 4 (2012).
[32]
Xiaoqing Li. 2020. Students’ Acceptance of Mobile Learning: An Empirical Study
Based on Blackboard Mobile Learn. In Mobile Devices in Education: Breakthroughs
in Research and Practice. IGI Global, 354–373.
[33]
Xiaozhou Li, Zheying Zhang, and Kostas Stefanidis. 2018. Mobile app evolution
analysis based on user reviews. In New Trends in Intelligent Software Methodologies,
Tools and Techniques. IOS Press, 773–786.
[34]
Caihua Liu, Muneera Bano, Didar Zowghi, and Matthew Kearney. 2021. Analysing
user reviews of inquiry-based learning apps in science education. Computers &
Education 164 (2021), 104119.
[35]
Walid Maalej, Hans-Jörg Happel, and Asarnusch Rashid. 2009. When users
become collaborators: towards continuous and context-aware user input. In
Proceedings of the 24th ACM SIGPLAN conference companion on Object oriented
programming systems languages and applications. 981–990.
[36]
Yichuan Man, Cuiyun Gao, Michael R Lyu, and Jiuchun Jiang. 2016. Experience
report: Understanding cross-platform app issues from user reviews. In 2016 IEEE
27th International Symposium on Software Reliability Engineering (ISSRE). IEEE,
138–149.
[37]
Diego Mayordomo-Martínez, Juan M Carrillo-de Gea, Ginés García-Mateos,
José A García-Berná, José Luis Fernández-Alemán, Saúl Rosero-López, Salvador
Parada-Sarabia, and Manuel García-Hernández. 2019. Sustainable accessibil-
ity: a mobile app for helping people with disabilities to search accessible shops.
International journal of environmental research and public health 16, 4 (2019), 620.
[38]
Montassar Ben Messaoud, Ilyes Jenhani, Nermine Ben Jemaa, and Mo-
hamed Wiem Mkaouer. 2019. A multi-label active learning approach for mobile
app user review classication. In International Conference on Knowledge Science,
Engineering and Management. Springer, 805–816.
[39]
Daniel Mont. 2021. Combatting the Costs of Exclusion for Children with Disabil-
ities and Their Families. UNICEF (2021).
94

W4A ’23, April 30–May 01, 2023, Austin, TX, USA Aljedaani, et al.
[40]
Joel S Mtebe and Aron W Kondoro. 2016. Using Mobile Moodle to enhance
Moodle LMS accessibility and usage at the University of Dar es Salaam. In 2016
IST-Africa Week Conference. IEEE, 1–11.
[41]
Meiyappan Nagappan and Emad Shihab. 2016. Future trends in software engi-
neering research for mobile apps. In 2016 IEEE 23rd International Conference on
Software Analysis, Evolution, and Reengineering (SANER), Vol. 5. IEEE, 21–32.
[42]
Mariaclaudia Nicolai, Luca Pascarella, Fabio Palomba, and Alberto Bacchelli. 2019.
Healthcare Android apps: a tale of the customers’ perspective. In Proceedings of
the 3rd ACM SIGSOFT International Workshop on App Market Analytics. 33–39.
[43]
Obianuju Okafor, Wajdi Aljedaani, and Stephanie Ludi. 2022. Comparative
Analysis of Accessibility Testing Tools and Their Limitations in RIAs. In HCI
International 2022–Late Breaking Papers: HCI for Health, Well-being, Universal
Access and Healthy Aging: 24th International Conference on Human-Computer
Interaction, HCII 2022, Virtual Event, June 26–July 1, 2022, Proceedings. Springer,
479–500.
[44]
Stamatios Papadakis, Michail Kalogiannakis, Eirini Sifaki, and Nikolas Vidakis.
2017. Access moodle using smart mobile phones. A case study in a Greek Uni-
versity. In Interactivity, game creation, design, learning, and innovation. Springer,
376–385.
[45]
Luca Pascarella, Davide Spadini, Fabio Palomba, Magiel Bruntink, and Alberto
Bacchelli. 2018. Information needs in contemporary code review. Proceedings of
the ACM on Human-Computer Interaction 2, CSCW (2018), 1–27.
[46]
Lucas Pelloni, Giovanni Grano, Adelina Ciurumelea, Sebastiano Panichella, Fabio
Palomba, and Harald C Gall. 2018. Becloma: Augmenting stack traces with
user review information. In 2018 IEEE 25th International Conference on Software
Analysis, Evolution and Reengineering (SANER). IEEE, 522–526.
[47]
Muhammad Sabri Sahrir, Nurkhamimi Zainuddin, and Mohd Shahrizal Nasir.
2016. Learning preference among Arabic language learners via mobile learning
management system platform (mobile LMS) using i-Taleem. International Journal
of Current Research in Life Sciences 5, 01 (2016), 509–514.
[48]
Norbert Sey, Florian Graf, and Neil Maiden. 2010. Using mobile re tools to
give end-users their own voice. In 2010 18th IEEE International Requirements
Engineering Conference. IEEE, 37–46.
[49]
Colin Shanley. 2016. Cracking Accessibility on Mobile Devices: The denitive
eld guide to accessibility and digital inclusion for business managers and project
teams.
[50]
Won Sug Shin and Minseok Kang. 2015. The use of a mobile learning manage-
ment system at an online university and its eect on learning satisfaction and
achievement. International Review of Research in Open and Distributed Learning
16, 3 (2015), 110–130.
[51]
Statista. 2021. Number of mobile (cellular) subscriptions worldwide from 1993 to
2021. https://www.statista.com/statistics/262950/global-mobile-subscriptions-
since-1993/.
[52]
UNESCO. 2020. Including learners with disabilities in COVID-19 education
responses. https://en.unesco.org/news/including-learners- disabilities-covid- 19-
education-responses.
[53]
Rajesh Vasa, Leonard Hoon, Kon Mouzakis, and Akihiro Noguchi. 2012. A prelim-
inary analysis of mobile app user reviews. In Proceedings of the 24th Australian
computer-human interaction conference. 241–244.
[54]
W3C. 2020. Web Content Accessibility Guidelines (WCAG) 2.1. https://www.w3.
org/TR/WCAG21/.
[55]
Jiahui Wang and Chen Li. 2020. Research on User Satisfaction of Video Education
Application Based on Reviews. In 2020 IEEE 10th International Conference on
Electronics Information and Emergency Communication (ICEIEC). IEEE, 340–343.
[56]
Shunguo Yan and PG Ramachandran. 2019. The current status of accessibility in
mobile apps. ACM Transactions on Accessible Computing (TACCESS) 12, 1 (2019),
1–31.
95