Available via license: CC BY 4.0
Content may be subject to copyright.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
VOLUME XX, 2017 1
(Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.Doi Number
Training future ICT engineers in the field of
accessibility and usability: a methodological
experience
María Luisa Pertegal-Felices1, Diego Marcos-Jorquera2*, Antonio Jimeno-Morenilla2,
Raquel Gilar-Corbi1, Higinio Mora2
1Department of Developmental Psychology and Didactics, University of Alicante, Carret. de San Vicente s/n, 03690 San Vicente del Raspeig, Spain
2Department of Computer Technology, University of Alicante (Spain), Carretera de San Vicente s/n, 03690 San Vicente del Raspeig, Spain
Corresponding author: D. Marcos-Jorquera (e-mail: dmarcos@dtic.ua.es).
ABSTRACT Nowadays, digital culture affects all levels of society. However, differences exist between
individuals, commonly named as the “digital divide,” which impedes the equal access to the benefits of new
technologies. The Usability and Accessibility (UA) module is a core, first-semester module during the first
year of the Multimedia Engineering degree at the University of Alicante. The UA module’s main objective
is to provide students with the necessary concepts and tools to design and develop products with usability
and accessibility features, thus achieving end products that are more usable and accessible, regardless of the
end users’ status, ability or situation. This paper presents a new learning methodology aimed at making
students become everyday users of their own digital products. Daily use of these products improves the UA
learning process, since students can appreciate their accessibility and usability in everyday life conditions for
a better understanding of how their own design decisions affect potential users. A non-equivalent control
group design with pre- and post-test control groups was used to test the research hypothesis. The results of
this study showed a significant improvement in their academic performance compared to the control group.
INDEX TERMS ICT engineers’ training, Accessibility, Digital divide, Usability divide.
I. INTRODUCTION
The World Summit on the Information Society (WSIS)
created one of the broadest and highest platforms in the area
of communications in the history of the United Nations [1].
This multi-year event included two intergovernmental
summits: Geneva 2003 and Tunis 2005. The concluding report
confirmed that information and communications technologies
(ICTs) have immense impact on practically every aspect of our
lives [2]. Technology provides millions of people throughout
the world with unprecedented opportunities to reach higher
development levels. It can reduce many obstacles, notably
time and distance, and promote dialogue between people,
nations and civilisations. Furthermore, it is an effective tool
for increasing productivity, generating economic growth,
creating employment and boosting employability, as well as
improving the quality of life for all. However, ICTs are not
equally distributed among countries (developed and
developing), or within societies, creating a digital divide
between citizens.
A. DIGITAL DIVIDE
“Digital divide” is a concept that came from President
Clinton’s mandate. This concept intended to express the
differences between people in the United States who had
access to new technologies and those who did not [3], [4]. The
Digital Divide was related to the efforts that the government
had to make to ensure the necessary investments to help
people access new technologies [5].
Although the concept originated in individuals being able to
access technology, it does have several dimensions. The
“global divide” determines the access differences between
industrialised and developing countries. The “social divide”
refers to accessibility related to socio-economic differences
between people that have access to the Internet and those who
do not. Lastly, the “democratic divide” exists among those
who are online and marks the differences in how they use ICTs
for engaging, mobilising and participating in public life [3].
This last dimension has a much wider scope and in addition to
Internet access, refers to all tools related to ICTs (mobile
telephones, network technologies, telecommunications,
personal digital assistants [PDAs] and other devices). The
“democratic divide” therefore helps us evaluate and
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
2 VOLUME XX, 2017
understand the differences in technology access among
groups, people and geographical areas, their normalised use
and ability to enjoy the advantages associated with them [6].
The dimension “digital literacy” appears in the area of
education, which refers to the basic skills needed to face digital
life on a critical level (as an information
discriminator/selector) or on a security level when using ICTs
[7]. This article focuses on the problems associated with the
“online” group who are affected by the dimensions
“democratic divide” and “digital literacy.” Hereinafter, the
concept “Digital Divide” shall be understood to be referring to
these areas.
According to the Organisation for Economic Cooperation
and Development (OECD), the term “digital divide” is defined
in terms of access to ICTs, the Internet and the skills needed
to use these technologies. The OECD defines technology as a
social process and considers that ICT skills are as important as
access to technology itself in bridging the digital divide [8].
To this end, although people have access to technology, the
difference lies in how they use it and their user skills, which
leads to the definition of the “techno-rich” and “techno-poor”
in accordance with the quality of technology use [9]. This
same concept is introduced by Van Seters, de Gaay Fortman,
and de Ruijter [10] when they confirm that poverty is not only
measured by economic or social terms. The world is now also
divided by those who have mastered new ICT skills and those
who have not.
Although governments have made several efforts to bridge
this gap, the subject is still a concern for many. The OECD
[11] developed a project that evaluated the work-related skills
of people aged between 16 and 65 years of age, examining the
ICT skills of 200,000 people in 33 countries. The results
showed that an important percentage of adults were lacking
basic ICT skills, demonstrating that they were not able to solve
problems in technology-rich environments. However, using
ICTs in the workplace is not the only limitation that the
“digitally illiterate” face. Digital illiteracy also affects other
dimensions such as identity and security.
A study by Best, Manktelow and Taylor [12] reports that
the use of online technologies provides benefits related to self-
esteem, perceived social support, increased social capital, safe
identity experimentation and increased opportunity for self-
disclosure. However, the same report states that little
knowledge of how to use these environments is also conducive
to damaging aspects such as increased exposure to harm,
social isolation, depression and cyber-bullying.
Information security is another of the subjects that most
concerns public and private entities. The divide is often not
associated with a lack of knowledge but with a disparity
between the objectives of those who design ICTs and those
who use them. Albrechtsen and Hovden [13] proved that
Security system managers and users were not in tune with one
another, in the sense that both had different points of view
concerning the same problem. The study revealed that the
differing interpretations were due to the Security managers
basing their practical method on unrealistic assumptions about
users, which were therefore poorly aligned with the users’
daily work dynamics.
B. HOW TO BRIDGE THE DIVIDE
Since 2005, the United Nations (UN) has established
mechanisms and recommendations that prompt governments
to set specific policies aimed at reducing technological
inequality [14]. This institution agreed to create “an
information society for all” as one of its fundamental
principles to guarantee that the opportunities that ICTs offer
are beneficial to everyone. To ensure that this principle is
achieved, all interested parties must collaborate in different
aspects, including developing and broadening ICT
applications and promoting ability, confidence and security in
ICT usage. The UN’s plan of action was centred on developing
and promoting ICTs in areas such as infrastructure, content
and applications. In line with the UN’s recommendations,
governments are adopting measures. Technological
infrastructure can be one of the strengths in a country, while
another could be changes to teaching practices to make
citizens digitally literate [15]. To this end, policies are being
developed, which aim to promote access to technology and its
use, seeking digital literacy, the rational use of technology and
its impact on daily life, security, etc. New technologies are
being taken into account in curricula at all levels of education,
which include “digital competence” as a key skill [16]–[18].
The academic world has also sought to make these
governmental policies more effective. Several studies provide
recommendations on how to bridge the digital divide. For van
Deursen and van Dijk [19], one of the main issues lies in the
lack of skills, but this problem increases when governments
assume that citizens are more skilful than they actually are.
The Dutch government’s expectations that citizens are capable
of completing the digital tasks requested —examined in the
paper— were not justified. The authors recommend two types
of policy to change this situation: recommendations to
improve government websites and to improve citizens’
qualification levels. Van Dijk and Hacker [20] confirm that the
digital divide is related to different categories of income,
employment, education, age and ethnicity. They showed that
the differential access of skills and usage is likely to increase,
projecting that the usage gap will grow. Furthermore, their
studies reveal the surprising effect that age and gender have in
comparison to education and relate the usage gap to evolution
of the information and network society.
Once governments have adopted measures to provide the
public with the physical tools needed to bridge the digital
divide (computers, Internet access, etc.), then the public need
to make effective use of them. The objective is to achieve a
digital citizenship, which, according to Ribble [21] is reached
when certain characteristics are acquired, such as
understanding cultural, social and human issues and practising
legal and ethical behaviour; advocating safe, legal and
responsible use of information and technology; exhibiting
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
3 VOLUME XX, 2017
positive attitude towards using technology that supports
collaboration, learning and creativity; demonstrating personal
responsibility for lifelong learning; and exhibiting leadership
for digital citizenship [22]. The best way to obtain digital
citizenship is through education. To ensure that students
actively participate in this digital citizenship, their training
must be directed toward digital access, digital commerce,
digital communication, digital literacy, digital etiquette,
digital law, digital rights and responsibilities, digital health
and wellness and digital security [23], [24].
C. USABILITY DIVIDE
Although citizens eventually obtain good digital skills, a
new divide can arise that prevents them from effectively
accessing the benefits of ICTs. This new obstacle, called the
“usability divide” [25] refers to the difficulties that citizens
face to make use of technology and the degree of simplicity
and effectiveness that technology creators provide their
creations. Citizens must have minimum skills to be able to use
the technology, but engineers must also be in tune with their
potential users’ needs to provide solutions that facilitate and
maximise product usage. This paper will focus on the ICT
engineer side.
Several studies demonstrate the difficulties that engineers
can create with regard to ICT access. In addition to the
aforementioned study by Albrechtsen and Hovden [13] on the
lack of understanding between Security system managers and
users, Shneiderman’s paper [26] describes serious difficulties
involved in creating a successful ecommerce business. For
Shneiderman, it is not only about using low-cost hardware or
broadband networks, emarket services are often too complex,
unusable and even irrelevant for too many users. In this
situation, design and usability are the key factors for success.
The cause of these problems is often linked to designers
making incorrect assumptions about users’ knowledge and
requirements. These false assumptions are related to the
difficulty in understanding technical vocabulary and/or
advanced concepts that are not well explained. Unfortunately,
most designers are not aware of how destructive this could be
for novice or even expert users [26].
What is usability for an ICT engineer? It is a broad concept
that has several definitions. According to international
standards [27], “usability” is the degree to which the software
product is understood, learned, used and attractive to the user
when used under specified conditions. For Nielsen [28], it
refers to the user's experience when interacting with a website.
Therefore, a “usable” product is one that can be clearly and
simply understood by users.
One of the main barriers that engineers must overcome is
being responsible for designing products that they may not
necessarily use, i.e., they need to be able to predict what would
be a good user interface. Engineers who design products
without knowing the users’ needs have to use their intuition to
detect them [29], [30]. Designers are extremely different to
most of their products’ users, not only concerning irrelevant
aspects such as what they like, but with other essential aspects,
such as what they believe is easy to use. Trusting said personal
preferences and intuitions is often misleading and the reason
why product designs can be disastrous [31].
The terms usability and accessibility often cause confusion.
The term accessibility refers to the universal access to
information of all people, regardless of the circumstances and
the devices used. For Hassan and Martin [32], accessibility is
the highest possible number of users being able to access and
use a web service or product, regardless of individuals’ own
limitations (abilities, knowledge, languages, experiences, etc.)
or those derived from the usage context. However, for these
authors, accessibility does not only involve the need to
facilitate access, but the need to facilitate use. A design is
accessible when it is usable for more people in more situations
or usage contexts [33], making efficiently and satisfactorily
carrying out and achieving tasks possible for all users [28]. For
Henry [33], accessibility is a subset of usability; it should be
understood as “part of” and a “requirement for” usability.
Therefore, the distinction between usability and accessibility
indicated by Henry may be considered difficult, and
unnecessary in many cases.
Future ICT engineers learn about Usability as a Software
Engineering concept at universities. To this end, it is mainly in
these institutions where several experiments have been
conducted to bridge the “usability divide.” In the academic
environment, Nielsen’s books [28], [31] are one of the basic
references used by usability students. At Toronto University,
Baecker, Booth, Jovicic, McGrenere and Moore [34] prepared
three projects to make complex designs more accessible to
novice users, including a text processor that gave users control
over the interface’s complexity. From the University of Graz
(Austria), Holzinger [35] reviewed inspection methods
designed to make detecting usability problems easier and other
end-user-oriented test methods that provide feedback on the
real use of applications. In other works, procedures based on
the evaluation of Usability Engineering and User Experience
are proposed, as in [36] where it is applied to the evaluation of
scientific special interest internet services.
Although many contributions have been made on the
concept of usability, there are very few studies aimed at
changing the way that future ICT engineers perceive and
practice usability. Øvad and Larsen [37] presented a study in
which students had to sit a “focused workshop” where medical
professionals informed the future developers of their daily
work needs in order to make designing medical applications
easier. However, this type of specific course limits the time
students have for the rest of their studies.
D. OBJECTIVES AND HYPOTHESIS
The objective of this research is to check if a new practical
methodology to teach usability and accessibility concepts can
improve future ICT engineers’ skills related to these concepts.
This experiment was conducted with Usability and
Accessibility students enrolling Multimedia Engineering at
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
4 VOLUME XX, 2017
the University of Alicante (Spain). The research hypothesis is
that making students everyday users of the products designed
in class will improve their usability and accessibility skills.
Thanks to this methodology, students will gain a better
understanding of how their design decisions could affect the
potential users of their applications.
The paper is organised as follows: The methodology section
shows how the research was conducted and what the learning
experience consists of. The results section evaluates the effect
that the experience has had on the experimental student group
compared to the control group. Lastly, the conclusion section
discusses the most relevant contributions from this study.
II. METHOD
A. PARTICIPANTS
A new degree emerged in response to the growing demand
for ICT specialists who are able to design new multimedia
projects in the digital leisure and entertainment sector and
content management sector for dissemination on information
networks: Multimedia Engineering. This degree is placed
between traditional engineering and computer engineering and
its scope of work is specially aimed at the design and
development of websites, web apps, mobile apps and video
games.
This study was conducted at the University of Alicante,
which has offered this degree since the 2010/2011 academic
year. The degree has been widely accepted since it was
introduced at the university, having filled all places each year.
Furthermore, each year group has graduated with a high
employment rate, which justifies the role played by these new
professionals.
The “Usability and Accessibility” module is part of the
Multimedia Engineering degree. Its objective is to analyse and
create user interfaces with usability characteristics, which are
easy-to-use, understandable and concise, as well as accessible,
meaning that they can be used by a maximum number of users,
regardless of their characteristics, access devices or context,
especially focusing on users with any type of disability.
The following skills are developed during the Multimedia
Engineering degree’s Usability and Accessibility module:
• Developing, maintaining, administering and evaluating
multimedia services and systems that satisfy all user
requirements and act in a reliable and efficient way,
complying with quality standards.
• Creating, designing and evaluating human-computer
interfaces that guarantee accessibility and usability.
• Designing, producing and managing multilingual and
multimodal systems of multimedia content with the
objective of guaranteeing their internationalisation,
localisation, accessibility and usability.
The UA instructional model is based on the Task-centered
Instructional strategy following the principles proposed by
Merrill [38]. This type of strategy, particularly suitable for
engineering education, involves the student in four distinct
phases of learning: (1) activation of prior experience, (2)
demonstration of skills, (3) application of skills, and (4)
integration or these skills into real world activities.
The study was conducted during the first term of the
2017/2018 academic year. The 70 students enrolled on the
Usability and Accessibility module participated in the study.
The course lasted 15 weeks with four hours of face-to-face
classes per week. Figure 1 shows the milestones associated
with the course programme.
Three projects are carried out during the module. For the
first two projects (P1 and P2), students are asked to design
human-computer interfaces with which users strengthen the
module’s basic concepts, learning to correctly use different
interactive elements and controls, as well as the basic rules and
good practices associated with developing usable interfaces.
The third project (P3) proposes a more complex activity and it
was during this project that the intervention was conducted. In
addition to the aforementioned, this project includes
accessibility concepts that universalises the interfaces
proposed, creating designs that maximise accessibility and
remove barriers for potential users with physical, cognitive
and technological issues.
The main objective that this project sets out is to create a
user interface with certain requirements and restrictions that
allow students to explore the usability and accessibility
problems that users often experience. More specifically,
students were asked to create a touchscreen interface for an
oven. An everyday electrical appliance was chosen so that
students could explore this module’s fundamental aspects,
promoting user-oriented design as an essential work method.
To this end, the application’s potential user is very
heterogeneous (age, sex and technical capacities) and the
product is widely known by users, which have already
assimilated a way to work with it. This creates a simulation,
encouraging them to produce a more natural and approachable
interface. Furthermore, the new interface would be
incorporated into the oven on a small, 5-inch touchscreen. This
restriction would condition the design decisions that the
students make and enable them to explore usability and
accessibility actions on specific interfaces covered in the
module, such as smartphone interfaces.
The third project (P3) comprises two stages; each has a
different objective, delivery date and separate evaluation. The
first stage (P3/1 milestone in Figure 1) consists of designing
the interface. Students must submit their interface designs and
a functional description for the interface. The objective of this
stage is to evaluate the design decisions that the students make.
During the second stage (P3/2 milestone in Figure 1), students
must develop the interface, which allows them to evaluate the
interactive processes and the experience of the end user that
they have proposed.
.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
5 VOLUME XX, 2017
FIGURE 1. Academic calendar for the Usability and Accessibility module.
Students must only use standard web technology to develop
the project during this stage (HTML, CSS and JavaScript) so
that the oven can be used from the integrated touchscreen
interface and monitored and managed from any Internet-
connected device. The objective of this is to strengthen
usability, such as the interfaces' flexibility and adaptability and
accessibility, such as device-independent access.
Students were given a JavaScript library, which simulates
access to the oven’s sensors and parameters (see Figure 2).
The sensors monitor data related to the oven (e.g.,
temperature) while the parameters act on the oven’s
electronics (e.g., switching a given element on or off). The
library has an engine that fully simulates the oven’s functions,
i.e., if the door is closed and an element is switched on, the
temperature will gradually rise. If all elements are switched
off, the temperature will drop, a process that speeds up if the
door is open. This activity aims to make students more familiar
with the reality of engineering project development.
FIGURE 2. General diagram of the oven’s management system
Table I shows the oven’s sensors and parameters. The
library has an Application Programming Interface (API) so
students can create an interface that can check the oven’s
sensors and modify the parameters.
Students have to submit a technical report on how to use the
oven’s API and a document detailing the functional
requirements that the interface must include: cooking
temperature selection, oven function selection, cooking timer,
warning alarms (open door, long cooking periods, etc.), pre-
set cooking selection, etc.
TABLE I
OVEN SENSORS AND PARAMETERS
Cat*
Name
Type
Description
S
External temp.
Number
External air temperature (ºC)
S
Internal temp.
Number
Internal air temperature (ºC)
S
Door
Boolean
Indicates the door’s state:
open (true) or closed (false)
S
Time
Date/Time
Data and time setting for the
oven
S
Power
consumption
Number
Oven’s power consumption
in watts
P
Upper
resistance
Boolean
Switch the upper resistance
on (true) or off (false)
P
Lower
resistance
Boolean
Switch the lower resistance
on (true) or off (false)
P
Grater
Boolean
Switch the grater on (true)
or off (false)
P
Fan
Boolean
Switch the fan on (true) or
off (false)
P
Beep
Boolean
Switch the alarm sound on
(true) or off (false)
*Category: S, sensor; P, parameter.
B. INTERVENTION
The objective of the intervention was to prove the research
hypothesis and to check that immersing students in more
realistic situations during project development could lead to
improved results, i.e., more usable and accessible products. To
do so, 70 students were divided into two groups: a control
group of 43 students who followed a traditional methodology
and an experimental group of 27 students who followed the
immersion method.
Both groups carried out the same activity for Project 3’s
design stage (P3/1); however, differences were set out during
the development stage (P3/2). The control group students were
only provided with an oven simulator library to develop the
application (A in Figure 3). This library exposes the API to
access the oven's functionality and uses keyboard commands
so that the user can interact with the oven (for example, to open
or close the door). The degree of immersion in this
environment is very low, since the user does not interact with
an oven and does not have a feedback on what his interface is
doing on it. On the other hand, the experimental group
students were provided with a complete simulation
environment, where the developed interface would be
integrated in a virtual oven to which students could connect (B
on Figure 3). The simulation environment uses web
technology and is portable, meaning that students were able to
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
P3/2
P1 P2 P3/1
Control
Experimental
Week
70 70
43
27
USER AGENTS OVEN SIMULATOR LIBRARY
Engine
Sensors
User
Interface
Parameter
Parameter
Parameter
Sensors
Sensor
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
VOLUME XX, 2017 6
work with it from home. Furthermore, experimental group
students were encouraged to use the system away from the
academic environment, using everyday devices, such as
mobile phones, tablets and laptops and even involve friends
and family members in the project development, ensuring
better self-assessment of the product they developed.
FIGURE 3. Simulation environment for the experimental group (B) and
the control group (A).
The oven simulator was implemented in JavaScript via the
NodeJS platform, given that it is a cross-platform environment
that supports web standards. The oven manager was
developed on this platform, which simulates an oven in
operation, allowing access to the oven’s sensors and
parameters. Express (version 4.15.2) offers a website where
the oven simulator can be viewed. The simulator always
shows all of the oven’s sensors and parameters and simulates
a real oven’s appearance and operations. For example, by
clicking and swiping over the door, it can be opened or closed.
The upper part shows the interface created by the student via
an HTML iframe. The result is a realistic simulation of how
the end product would work once it is manufactured. A two-
way channel based on the WebSocket protocol was used to
coordinate changes performed through the oven manager
(temperature changes, door opening, element activation, etc.).
In addition to the simulator, the channel can also integrate
other external screens, such as mobile phones or tablets, where
the oven’s interface can be viewed. The WebSocket node
module (version 1.0.24) was used to implement it.
Both groups (control and experimental) programmed the
interface using the same tools, accessing the oven with the
same API (Table I). The only difference was the realistic
immersion that the experimental group experienced through
the virtualised environment, which placed those students in
situations that are much more similar to the end users’
experience. Through the simulator, the actions performed by
the student in the user interface produce (through the library)
a visual response in the oven simulator (for example,
activating the alarm produces a beep or turning on a resistance,
makes it light up with a warm color), and the actions
performed by the user in the oven simulator cause (also
through the library) to trigger events in the user interface (for
example if the oven door is opened by pressing on it, the user
interface is notified).
In order to ensure that the products developed by the
students are successful, in terms of usability and accessibility,
and taking into account that many of them have never been
end users of a furnace, they are instructed in the basic
operation of a traditional oven. In addition, they are
encouraged to make real use of their prototype, with all the
programming sequence of resistances and times for cooking
various recipes. With these same realistic procedures, the
evaluation of the products by the teachers is carried out later.
This allows students to become end users of their own product,
and to evaluate the user experience using the designed
interface.
III. RESULTS
A quasi-experimental “with a non-equivalent control
group” design was adopted [39] with pre-test-post-test control
groups. Consequently, the statistical procedure used the
general linear model with repeated measures. The time of
assessment (pre-test and post-test) was used as the intra-
subject factor, and participation in the project (belonging to the
experimental or control group) was the inter-subject factor. All
statistical analyses were conducted using SPSS (version 23.0).
In the case of the pre-test assessments, the usability of the
proposed design has been evaluated. This evaluation was prior
to the intervention, and was carried out stablishing objective
criteria on the fulfillment of the design made with the usability
directives studied in the subject: Ease of learning, Flexibility,
Consistency, Robustness, Recoverability, Response time,
Adaptation of homework, and decreased cognitive load. The
evaluation in the post-test was focused on two fundamental
aspects, the fulfillment of the proposed design and, therefore,
the correct use of the usability principles, and the
incorporation of accessibility rules. This evaluation is very
objective, since it can be done automatically using software
tools, or it can be systematized with a revision of the delivered
source code. The correction of the projects was carried out
Tablet
WEB BROWSER
Smartphone
WEB BROWSER
Oven Simulator
NODE
Oven Manager
WEB BROWSER
expresswebsocket
HTTP
HTTP (WS)
HTTP (WS)
HTTP
(WS)
Oven
Library
API
Oven
Library
API
PC
WEB BROWSER
Oven
Library
API Key
commands
Interacts Interacts
ENVIRONMENT WITH OVEN SIMULATOR
(EXPERIMENTALGROUP)
ENVIRONMENT WITHOUT OVEN SIMULATOR
(CONTROL GROUP)
Interacts Interacts
A
B
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
VOLUME XX, 2017 7
blindly, and regardless of which group each student belonged
to.
The sample’s normality test indicates a normal distribution.
Box’s M test shows homogeneity of the variance-covariance
matrices (p = .785). To assess the programme’s effect on
students’ performance, the students’ grades were compared
before (pre-test) and after the experiment (post-test). The
independent variable or factor is belonging to one group or the
other and the criteria or dependent variables are the grades
obtained by the subjects at each stage of the project (P3/1 and
P3/2). These students are graded in the interval [0-10].
The inter-subject test values (see Table II) show that the
average of all grades differed from zero since the tests were
significant (p < .000) for intercept, but not for belonging to one
group or the other (p = .201), which confirms that there are no
significant differences between the student groups.
TABLE II
TEST FOR INTER-SUBJECT EFFECTS
Source
Type III error
gl
F
Sig.
η2 partial
Obs.
power
a
Intersection
6655.864
1
642.900
.000
.904
1.000
Group
35.526
1
3.432
.068
.048
.447
Error
703.995
68
a computed using alpha = .05
Table III shows proof of intra-subject effects with regard to
programme application. Said test values show that the
interaction effect between the time of the test (pre-test and
post-test) and applying the programme with the new
methodology is significant (p = .008). The power observed
was .768, meaning that it is correct to reject the null hypothesis
that the variances are equal. The effect size (η2), proportion of
total variation attributable to a factor or, the magnitude of
difference between one time or another [40], which produces
the interaction between the test time and the programme
application is .099.
TABLE III
TEST OF INTRA-SUBJECTS EFFECTS
Source
Type III
error
gl
F
Sig.
η2 partial
Obs.
power a
Usability
3.028
1
3.716
.058
.052
.476
Usability * Group
6.079
1
6.079
7.458
.008
.099
Error (Usability)
55.421
68
a computed using alpha = .05
Finally, a t-test was conducted on the mean differences to
check if there were any differences between the experimental
group and the control group pre-test and post-test (Table IV).
Table IV shows that there are no significant differences at pre-
test, meaning that both groups started under similar
circumstances, as the inter-subject test already suggested.
TABLE IV
STUDENT’S T-TEST ON THE DIFFERENCE IN MEANS
Moment
t
gl
Sig.
Difference
Std dev.
Pretest (P3/1)
-,981
68
,330
-,606
,618
Postest (P3/2)
-2,741
68
,008
-1,462
,539
a. The parameter has been assigned the value zero because it is redundant.
b. computed using alpha = .05
With regard to the post-test, the test produced a significant
difference between the two groups (p = .008). This difference
represents 1.462 points more than the experimental group.
FIGURE 4. Average performance of the groups for the pre-test (P3/1)
and post-test (P3/2) under scoring range [0-10].
IV. DISCUSSION
Information and Communication Technologies are
changing the way we relate to the world. However, not all
citizens are able to enjoy the benefits of using ICTs equally.
The internationally recognised term Digital Divide highlights
the inequalities that citizens face with regard to technology
usage. This could be due to not having access to technology or
not being able to make the most of the advantages that they
offer us despite being connected. This research focused on the
latter group, i.e., those who are affected by the “usability
divide”.
Usability divide can be observed from two approaches and
mechanisms can be implemented to bridge the gap in both
cases. From the users’ point of view, digital literacy provides
citizens with digital skills so that technology use does not
represent an obstacle. From the ICT engineers’ point of view,
they must make sure that their creations are easy to understand
and aligned with users’ needs.
During this research, a new teaching methodology was
developed for students on the Multimedia Engineering degree
aimed at improving their usability and accessibility skills.
Improving these skills will help improve the applications that
these future engineers develop, making them more useful for
users. This methodology places the student in the real world,
in which students solve real problems that are significant and
important to them and that are similar to the problems they will
face in their professional future. This type of learning
facilitates the transfer of the knowledge acquired and which
7.53
7.66
6.93
6.20
6
6,5
7
7,5
8
P3/1 P3/2
Exp
Control
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
VOLUME XX, 2017 8
will also last better over time [41]. This methodology is
aligned with project-based learning in which satisfactory
results have also been obtained for engineering students [42].
The methodological experience is based on making students
become everyday users of their own products. This change has
two objectives: to make the engineers become end users of
their products and to encourage engineers to design a product
that is very familiar to them. Improvements of this type have
been reported previously, e.g., Marcos-Jorquera, Pertegal-
Felices, Jimeno-Morenilla, & Gilar-Corbí [43] showed that
end users’ accessibility and usability skills improved when
they participated in the product design alongside ICT
engineers. Pan, Miao, Yu, Leung, & Chin [44] also showed
that product familiarity improved the user experience, thus
making it more usable.
The research reported significant differences between
students that followed the methodology and those that
continued to use the module’s usual method. The experimental
group achieved improved grades in comparison to their pre-
test results, while the control group’s grades were lower. The
control group’s lower grades are common for this module at
the University of Alicante, given that the post-test is evaluated
as being more complicated than the pre-test. Although there
was no significant difference between the groups for the pre-
test results, there was for the post-test, with the experimental
group being 14% higher.
In the future, there are plans to improve the teaching of
engineers by having them develop everyday products, not only
for their personal use, but also for their friends and family
outside the academic environment. This method aims to
improve products’ accessibility and usability characteristics.
Other future research could be aimed at evaluating the impact
Multimedia Engineering graduate students have on this
methodology. Some of the variables that could be evaluated
are motivation and teaching style. Reliability and validity
psychometrics tests would be used for such (MAPLE, MSLQ,
etc.).
In addition, it would be interesting to incorporate other
approaches such as participatory design that could greatly
enrich the learning process in the subject. However, this
approach would imply the realization of group projects, and
the number of samples obtained when performing the analysis
of results would be drastically reduced. That is why, for this
case, it would be essential to increase the number of
participants in the study.
ACKNOWLEDGEMENTS
This work was funded by the I3CE Network Program of
research in university teaching at the Institute of Educational
Sciences of the University of Alicante (ICE call 2018-19). Ref.
4331.
REFERENCES
[1] S. Zehle, ‘World Summit on the Information Society’,
in The Wiley-Blackwell Encyclopedia of Globalization,
American Cancer Society, 2012.
[2] WSIS Civil Society Plenary, ‘Shaping Information
Societies for Human Needs’, 2003. [Online].
Available: http://www.itu.int/wsis/docs/geneva/civil-
society-declaration.pdf. [Accessed: 30-May-2018].
[3] P. Norris, Digital divide: Civic engagement,
information poverty, and the Internet worldwide.
Cambridge University Press, 2001.
[4] T. N. Friemel, ‘The digital divide has grown old:
Determinants of a digital divide among seniors’, New
Media Soc., vol. 18, no. 2, pp. 313–331, 2016.
[5] A. S. Santoyo and E. M. Martínez, La brecha digital:
mitos y realidades. Uabc, 2003.
[6] P. Maya Álvarez, ‘La brecha digital, brecha social. Los
recursos humanos en el desarrollo y la capacitación a
través del aprendizaje digital ('elearning’)’, 2008.
[7] D. Buckingham, ‘Defining digital literacy’, in
Medienbildung in neuen Kulturräumen, Springer,
2010, pp. 59–71.
[8] J. Edwyn, ‘Learning to bridge the digital divide’,
Organ. Econ. Coop. Dev. OECD Obs. Paris, no. 224,
pp. 43–45, 2001.
[9] J. Castaño, J. M. Duart, and T. Sancho, ‘Una segunda
brecha digital entre el alumnado universitario’, Cult.
Educ., vol. 24, no. 3, pp. 363–377, 2012.
[10] van Seters, Paul, De Gaay Fortman, B., de Ruijter,
Arie, and Department of Politics and Public
Administration, Globalization and its new divides.
Dutch University Press, 2003.
[11] M. Kankaraš, G. Montt, M. Paccagnella, G. Quintini,
and W. Thorn, ‘Skills Matter: Further Results from the
Survey of Adult Skills. OECD Skills Studies.’, OECD
Publ., 2016.
[12] P. Best, R. Manktelow, and B. Taylor, ‘Online
communication, social media and adolescent
wellbeing: A systematic narrative review’, Child.
Youth Serv. Rev., vol. 41, pp. 27–36, 2014.
[13] E. Albrechtsen and J. Hovden, ‘The information
security digital divide between information security
managers and users’, Comput. Secur., vol. 28, no. 6,
pp. 476–490, 2009.
[14] M. Beckedahl, ‘Much more could have been
achieved’, in World Summit on the Information
Society, 2005.
[15] ‘UNESCO ICT Competency Framework for Teachers
- UNESCO Biblioteca Digital’. [Online]. Available:
https://unesdoc.unesco.org/ark:/48223/pf0000213475.
[Accessed: 24-May-2019].
[16] ‘La Integración de las tecnologías de la información y
la comunicación en los sistemas educativos -
UNESCO Biblioteca Digital’. [Online]. Available:
https://unesdoc.unesco.org/ark:/48223/pf0000150785.
[Accessed: 24-May-2019].
[17] U. Europea, ‘Digital agenda for Europe’, Consult. En
Httpec Eur. Eudigital-Agenda, 2010.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2985077, IEEE Access
Author Name: Preparation of Papers for IEEE Access (February 2017)
VOLUME XX, 2017 9
[18] I. Peña-López, ‘Students, computers and learning.
Making the connection’, 2015.
[19] A. J. A. M. van Deursen and J. A. G. M. van Dijk,
‘Improving digital skills for the use of online public
information and services’, Gov. Inf. Q., vol. 26, no. 2,
pp. 333–340, Apr. 2009, doi:
10.1016/j.giq.2008.11.002.
[20] J. van Dijk and K. Hacker, ‘The Digital Divide as a
Complex and Dynamic Phenomenon’, Inf. Soc., vol.
19, no. 4, pp. 315–326, Sep. 2003, doi:
10.1080/01972240309487.
[21] M. Ribble, ‘Passport to digital citizenship’, Learn.
Lead. Technol., vol. 36, no. 4, pp. 14–17, 2008.
[22] A. Isman and O. Canan Gungoren, ‘Digital
citizenship.’, Turk. Online J. Educ. Technol.-TOJET,
vol. 13, no. 1, pp. 73–77, 2014.
[23] M. Ribble, Digital citizenship in schools: Nine
elements all students should know. International
Society for Technology in Education, 2015.
[24] F. Cruz-Jesus, M. R. Vicente, F. Bacao, and T.
Oliveira, ‘The education-related digital divide: An
analysis for the EU-28’, Comput. Hum. Behav., vol.
56, pp. 72–82, 2016.
[25] A. Lane, ‘The impact of openness on bridging
educational digital divides’, Int. Rev. Res. Open
Distrib. Learn., vol. 10, no. 5, 2009.
[26] B. Shneiderman, ‘Universal usability’, Commun. ACM,
vol. 43, no. 5, pp. 85–85, 2000.
[27] D. D. J. Suwawi, E. Darwiyanto, and M. Rochmani,
‘Evaluation of academic website using ISO/IEC 9126’,
in 2015 3rd International Conference on Information
and Communication Technology (ICoICT), 2015, pp.
222–227.
[28] J. Nielsen, Designing web usability: The practice of
simplicity. New Riders Publishing, 1999.
[29] A. Sonderegger, S. Schmutz, and J. Sauer, ‘The
influence of age in usability testing’, Appl. Ergon., vol.
52, pp. 291–300, 2016.
[30] A. Queirós, A. Silva, J. Alvarelhão, N. P. Rocha, and
A. Teixeira, ‘Usability, accessibility and ambient-
assisted living: a systematic literature review’,
Univers. Access Inf. Soc., vol. 14, no. 1, pp. 57–66,
2015.
[31] J. Nielsen, Usability engineering. Elsevier, 1994.
[32] Y. Hassan Montero and F. J. Martín Fernández, ‘Qué
es la accesibilidad web’, No Solo Usabilidad, no. 2,
2003.
[33] S. L. Henry, ‘Understanding web accessibility’, in Web
Accessibility, Springer, 2006, pp. 1–51.
[34] R. Baecker, K. Booth, S. Jovicic, J. McGrenere, and G.
Moore, ‘Reducing the gap between what users know
and what they need to know’, in Proceedings on the
2000 conference on Universal Usability, 2000, pp. 17–
23.
[35] A. Holzinger, ‘Usability engineering methods for
software developers’, Commun. ACM, vol. 48, no. 1,
pp. 71–74, 2005.
[36] E.-M. Schön, J. Hellmers, and J. Thomaschewski,
‘Usability Evaluation Methods for Special Interest
Internet Information Services’, Int. J. Interact.
Multimed. Artif. Intell., vol. 2, pp. 26–32, Jun. 2014,
doi: 10.9781/ijimai.2014.263.
[37] T. Øvad and L. B. Larsen, ‘Experiences from training
agile software developers in focused workshops’, Proc
IHCI, pp. 355–359, 2014.
[38] M. D. Merrill, First principles of Instruction in CM
Reigeluth & Carr (Eds.) Instructional design Theories
and Models: Building a Common knowledge Base
(Vol. III). New York: Routledge Publishers, 2009.
[39] D. T. Campbell and J. C. Stanley, Experimental and
quasi-experimental designs for research. Ravenio
Books, 2015.
[40] R. Ledesma, G. Macbeth, and N. Cortada De Kohan,
‘Effect size: A conceptual review and applications
with the ViSta statistical system’, Rev. Latinoam.
Psicol., vol. 40, no. 3, pp. 425–439, 2008.
[41] S. Vosniadou, C. Ioannides, A. Dimitrakopoulou, and
E. Papademetriou, ‘Designing learning environments
to promote conceptual change in science’, Learn.
Instr., vol. 11, no. 4, pp. 381–419, Aug. 2001, doi:
10.1016/S0959-4752(00)00038-4.
[42] J. Mills and D. Treagust, ‘Engineering Education, Is
Problem-Based or Project-Based Learning the
Answer’, Aust J Eng Educ, vol. 3, Jan. 2003.
[43] D. Marcos-Jorquera, M. L. Pertegal-Felices, A.
Jimeno-Morenilla, and R. Gilar-Corbi, ‘An
interdisciplinary practical for multimedia engineering
students’, IEEE Trans. Educ., vol. 60, no. 1, pp. 8–15,
2016.
[44] Z. Pan, C. Miao, H. Yu, C. Leung, and J. J. Chin, ‘The
effects of familiarity design on the adoption of
wellness games by the elderly’, in 2015
IEEE/WIC/ACM International Conference on Web
Intelligence and Intelligent Agent Technology (WI-
IAT), 2015, vol. 2, pp. 387–390.
