ChapterPDF Available

UX Evaluation Design of UTAssistant: A New Usability Testing Support Tool for Italian Public Administrations

Chapter

UX Evaluation Design of UTAssistant: A New Usability Testing Support Tool for Italian Public Administrations

Abstract and Figures

Since 2012, usability testing in Italian public administration (PA) has been guided by the eGLU 2.1 technical protocols, which provide a set of principles and procedures to support specialized usability assessments in a controlled and predictable way. This paper describes a new support tool for usability testing that aims to facilitate the application of eGLU 2.1 and the design of its User eXperience (UX) evaluation methodology. The usability evaluation tool described in this paper is called UTAssistant (Usability Tool Assistant). UTAssistant has been entirely developed as a Web platform, supporting evaluators in designing usability tests, analyzing the data gathered during the test and aiding Web users step-by-step to complete the tasks required by an evaluator. It also provides a library of questionnaires to be administered to Web users at the end of the usability test. The UX evaluation methodology adopted to assess the UTAssistant platform uses both standard and new bio-behavioral evaluation methods. From a technological point of view, UTAssistant is an important step forward in the assessment of Web services in PA, fostering a standardized procedure for usability testing without requiring dedicated devices, unlike existing software and platforms for usability testing.
Content may be subject to copyright.
UX Evaluation Design of UTAssistant: A New Usability
Testing Support Tool for Italian Public Administrations
Stefano Federici1[0000-0001-5681-0633], Maria Laura Mele1[0000-0002-2714-4683], Rosa Lanzi-
lotti2[0000-0002-2039-8162], Giuseppe Desolda2[0000-0001-9894-2116], Marco Bracalenti1[0000-0001-
5681-0633], Fabio Meloni1[0000-0002-4161-9956], Giancarlo Gaudino3, Antonello Cocco3
1 Department of Philosophy, Social and Human Sciences and Education University of Perugia,
Perugia, Italy
stefano.federici@unipg.it (marialaura.mele,
marco.bracalenti90,fa.meloni)@gmail.com
2 Department of Computer Science, University of Bari Aldo Moro, Bari, Italy
(rosa.lanzilotti,giuseppe.desolda)@uniba.it
3 ISCOM Superior Institute of Communication and Information Technologies, Ministry of
Economic Development, Rome, Italy
(giancarlo.gaudino,antonello.cocco)@mise.gov.it
Abstract. Since 2012, usability testing in Italian public administration (PA) has
been guided by the eGLU 2.1 technical protocols, which provide a set of princi-
ples and procedures to support specialized usability assessments in a controlled
and predictable way. This paper describes a new support tool for usability testing
that aims to facilitate the application of eGLU 2.1 and the design of its User eX-
perience (UX) evaluation methodology. The usability evaluation tool described
in this paper is called UTAssistant (Usability Tool Assistant). UTAssistant has
been entirely developed as a Web platform, supporting evaluators in designing
usability tests, analyzing the data gathered during the test and aiding Web users
step-by-step to complete the tasks required by an evaluator. It also provides a
library of questionnaires to be administered to Web users at the end of the usa-
bility test. The UX evaluation methodology adopted to assess the UTAssistant
platform uses both standard and new bio-behavioral evaluation methods. From a
technological point of view, UTAssistant is an important step forward in the as-
sessment of Web services in PA, fostering a standardized procedure for usability
testing without requiring dedicated devices, unlike existing software and plat-
forms for usability testing.
Keywords: Experimental UX Evaluation Methodology, Usability Evaluation
Tool, Public Administration, UX Semi-Automatic Assessment, International
Usability Standards.
1 Introduction
This paper describes the design of an experimental methodology that aims to evaluate
the User eXperience (UX) of a new Web platform, called UTAssistant (Usability Tool
Assistant) [1]. This is a semi-automatic usability evaluation tool that supports practi-
tioners in usability evaluations of Web systems and services provided by a public ad-
ministration (PA), according to the eGLU 2.1 technical protocol [2]. This protocol pro-
vides a set of principles and procedures to support specialized usability assessments in
a controlled and predictable way.
The UX evaluation design of UTAssistant described in this work is an experimental
methodology for assessing the UTAssistant platform with end-users and Web managers
of PA Web sites, both in a laboratory setting and using a Web-based recruitment plat-
form. The methodology proposed here includes several types of end-users, with the aim
of assessing (i) the UTAssistant method through bio-behavioral measurements; (ii) the
usability evaluation process of UTAssistant with Web managers in Italian PA; (iii) a
heuristic evaluation of UTAssistant conducted by experts in UX; and (iv) a usability
evaluation of UTAssistant with a highly representative number of end-users using a
Web-based recruitment platform.
2 Usability Testing of Italian Public Administration Web
Services
In October 2012, the Department of Public Function of the Italian Ministry for Simpli-
fication and Public Administration formed a working group called GLU (Working
Group on Usability). The GLU team was composed of Italian universities, central and
local Italian PAs, and other independent information and communication companies.
The purpose of GLU is to support PA practitioners involved in Web content manage-
ment, website development, or e-government systems development in performing usa-
bility evaluations, and particularly those who are not usability experts. The primary
goal of GLU is to collect and identify golden rules for developing and evaluating sys-
tems that are easy to use and appropriate for this purpose. To this end, GLU developed
a set of guiding protocols that are able to operatively support both the analysis and
evaluation of graphical user interfaces for the Web. GLU can guide Web masters, and
its protocols are explorative tools that can investigate how good or satisfactory the ex-
perience is for a user when using a PA Web service, e.g. searching for certain infor-
mation, consulting or downloading a digital document, or completing an online form.
GLU protocols guide PA practitioners in exploratory analyses to better understand the
problems (or strengths) of their Web service, in order to collect use cases for future
development. Since 2013, GLU has developed four different usability evaluation pro-
tocols called eGLU 1.0, eGLU 2.0, eGLU 2.1 [2], and eGLU-M [3]. Three protocols
(1.0 and 2.0) are designed for desktop solutions (2.1), while the other (eGLU-M) is
designed for mobile platforms [3].
The eGLU 1.0 protocol was developed in May 2013 [4]. The protocol involves two
levels of analysis, basic and advanced, which can be used independently of each other
according to the testing period and the practitioner’s skill. The basic level is specifically
recommended for performing quick analyses to check the main problems affecting the
usability of a short number of Web pages. It is a macroscopic analysis that asks users
to freely navigate the content of the main pages of a given Web service, and then to
complete a questionnaire to investigate the quality of the interaction. In a basic level
analysis, practitioners primarily collect information on how many navigation tasks us-
ers achieved or failed, how difficult it was for users was to perceive or understand Web
interface elements, and user satisfaction. The advanced level analysis is recommended
for practitioners who need a more detailed analysis of interaction problems. At this
level, participants are required to report their actions and thoughts during their interac-
tion with the system. Compared to the basic level, an advanced analysis allows practi-
tioners a greater level of detail and information on user interactions, both in terms of
the users’ navigation paths and the difficulties they encountered in perceiving or under-
standing information during the tasks. Both basic and advanced levels describe how to
create and describe tasks for users, how to set parameters, the apparatus involved, and
the selection of participants. The eGLU 1.0 protocol provides practitioners with practi-
cal advice on how to properly conduct the test, including how to verbally describe both
the goals of the test and the instructions to participants. Both the basic and advanced
levels follow five phases, which describe: (i) how to prepare testing documents; (ii)
how to prepare tools and materials; (ii) how to conduct the test; (iv) how to handle the
collected data; and (v) how to draw up the evaluation report. eGLU 1.0 recommends
the use of at least one of two usability assessment questionnaires: (i) the System Usa-
bility Scale (SUS) [5, 6] or the Usability Evaluation (Us.E. 2.0) questionnaire [7].
The eGLU 2.0 protocol was released in 2014. Compared to eGLU 1.0, eGLU 2.0
provides practitioners with an easier and simpler methodology for conducting evalua-
tion tests, together with a wide range of design and evaluation approaches and methods
from which practitioners can freely choose according to their needs. eGLU 2.0 consists
of two parts: the first gives recommendations and instructions to practitioners on how
to design and conduct tests, while the second focuses on advanced design methods and
evaluation techniques, and describes which alternative and/or complementary usability
methods can be used.
In the same way as eGLU 1.0, eGLU 2.0 offers a first-level usability test methodol-
ogy that is suitable for both expert and non-expert usability evaluation practitioners.
eGLU 2.0 involves three phases, which describe how to (i) prepare, (ii) execute, and
(iii) analyze the results. The protocol recommends using at least one of three usability
assessment questionnaires: (i) the SUS [5, 6], the Us.E. 2.0 questionnaire [7], and the
Usability Metric for User Experience, lite version (UMUX-LITE) [8-10]. The second
part of eGLU 2.0 involves several in-depth analyses of and extensions to the basic pro-
cedure. These schedules can be useful in planning, conducting or analyzing the inter-
action, and increase the possibility of intervention via Web site redesign by providing
elements from a broader and more complex range of methodological approaches com-
pared to the basic protocol procedure. The advanced techniques described in eGLU 2.0
are the kanban board, scenarios and personas, evaluation strategies using the think-
aloud verbal protocol, the methodology of the ASPHI non-profit organization founda-
tion (http://www.asphi.it/), and the usability cards method
(http://www.usabilitycards.com/).
An updated version of the methods and techniques proposed in eGLU 2.0 was de-
veloped in 2015 [2] with the eGLU 2.1 protocol. eGLU 2.1 is distributed together with
the eGLU-M (eGLU-mobile) protocol [3], which is specifically designed for usability
evaluations using mobile devices. Although the evaluation of mobile websites and Web
services has some aspects that are operationally different from evaluations using desk-
top devices, the approach, methodology and phases of the exploratory analysis proce-
dure remain substantially unchanged. The development of a new version of the protocol
is currently in progress, and its release is expected in 2018.
3 UTAssistant: A New Usability Testing Support Web Platform
for Italian Public Administration
UTAssistant is a Web platform, designed and developed within the PA++ Project. The
goal of this platform is to provide Italian PA with a lightweight and simple tool for
conducting user studies based the eGLU 2.1 protocol, without requiring installation on
user devices.
One of the most important requirements driving the development of this platform
was the need to perform remote usability tests with the aim of stimulating users to par-
ticipate in a simpler and more comfortable way. To accomplish this, UTAssistant was
developed as a Web platform so that the stakeholders involved, namely the evaluator
(Web manager of a PA site) and users (typically of PA Web sites), can interact using
their PCs, wherever and whenever they prefer. This is possible due to the recent evolu-
tions of the HTML5 and JavaScript standards, which allow Web browsers to gather
data from PC devices such as the webcam, microphone, mouse, and keyboard. This
represents an important contribution to state-of-the-art of usability test tools, since re-
mote participation fosters wider adoption of these tools and consequently of the usabil-
ity testing technique. Indeed, the existing tools for usability testing require software
installation on a PC with specific requirements (e.g. Morae®
https://www.techsmith.com/morae.html [11]).
The following sub-sections describe how UTAssistant supports evaluators in design-
ing a user study and analyzing the results, and how users are supported by UTAssistant
in completing the evaluation tasks.
3.1 Usability Test Design
A usability test starts from the test design, which mainly consists of: (i) creating a script
to introduce the users to the test; (ii) defining a set of tasks; (iii) identifying the data to
be gathered (e.g. the number of clicks and the time required by the user to accomplish
a task, audio/video/desktop recording, logs, etc.); and (iv) deciding which question-
naire(s) to administer to users.
UTAssistant facilitates evaluators in performing these activities by means of three
wizard procedures. The first guides evaluators in specifying: (a) general information
(e.g. a title, the script); (b) the data to gather during execution of the user task (e.g.
mouse/keyboard data logs, webcam/microphone/desktop recordings); and (c) the post-
test questionnaire(s) to administer. The second procedure assists evaluators in creating
the task lists; for each task, start/end URLs, the goal and the duration have to be speci-
fied. Finally, the third procedure requires evaluators to select the users, either from a
list of users already registered to the platform, or by typing their email addresses. The
invited users receive an email including the instructions for participating in the usability
test. The following sub-section illustrates how UTAssistant aids users in performing
the test.
3.2 Usability Test Execution
Following the creation of the usability test design, users receive an email with infor-
mation about the evaluation they are asked to complete, and a link to access UTAssis-
tant. After clicking on this link, users can carry out the evaluation test, which starts by
giving general information about the platform use (e.g. a short description of the toolbar
with useful commands), the script for the evaluation and, finally, privacy policies indi-
cating which data will be captured, such as mouse/keyboard logs and webcam/micro-
phone/desktop recordings.
Fig. 1. An example of execution of a task. The UTAssistant toolbar is shown at the top of the
evaluated website page.
Following this, UTAssistant administers each task, one at a time. The execution of
each task is closely guided by the platform, which shows the task description in a pop-
up window, and then opens the Web page at which users are asked to start the task
(Figure 1). To keep the platform as minimally invasive as possible during execution of
the evaluation test, we grouped all the functions and indications in a toolbar placed at
the top of the Web page. This toolbar indicates the title of the current task, its goal, the
duration of the task, the task number, and a button to move to the next task, which
shows the message “Complete Questionnaire” when the user finishes the last task and
is asked to complete the questionnaire(s). During execution of the task, the platform
collects all data identified by the evaluator at the design stage, in a transparent and non-
invasive way.
3.3 Evaluation of Test Data Analysis
One of the most time-consuming phases of a usability test is the data analysis, since
evaluators are required to manually collect, store, merge and analyze a huge amount of
data such as mouse logs, video/audio recordings and questionnaire results. Due to the
effort required, this phase becomes a deterrent towards the adoption of usability testing
techniques. UTAssistant automates all of these activities, thus removing the barriers to
the gathering of usability test data. The evaluators have access to the data analysis re-
sults via the control panel and can exploit several functionalities that provide useful
support in discovering usability issues. The next sub-sections present an overview of
some of these tools.
3.4 Task Success Rate (Effectiveness)
Analysis of the results of the usability test often starts by investigating the task success
rate, an essential indicator of the effectiveness of the website in supporting the execu-
tion of a set of tasks. This metric is calculated as the percentage of tasks correctly com-
pleted by users. It can be also calculated for each task, thereby estimating the percentage
of users who completed that task. UTAssistant calculates these frequencies and displays
them in a table (Figure 2).
Fig. 1. Example of a table reporting the success rates of a study. The columns display the tasks,
while the rows show a list of users. The last row reports the success rate for each task, while the
last column depicts the success rate for each user. The overall success rate is reported below the
table.
3.5 Questionnaire Results
Another phase requiring a great deal of effort by evaluators is the analysis of the results
of the questionnaire. Using UTAssistant, evaluators can administer one or more ques-
tionnaires at the end of each usability evaluation. The platform automatically stores the
user’s answers and produces results in the form of statistics and graphs. For example,
if the SUS [5, 6] questionnaire is used, UTAssistant calculates the global SUS score (a
unidimensional measure of perceived usability [12]), the usability score and the learna-
bility score. In addition, different visualizations can display these results from different
perspectives, e.g. a histogram of each user’s SUS scores, a box-plot of SUS
score/learnability/usability (Figure 3), and a score compared with the SUS evaluation
scales (Figure 3).
Fig. 2. Example of SUS score plotted on the three SUS scales.
3.6 Audio/Video Analysis
While users are executing the tasks, UTAssistant can record the user’s voice using the
microphone, their facial expressions using the webcam, and the desktop display using
a browser plugin. This recorded content can be analyzed by evaluators in order to un-
derstand, for example, low performance in executing a particular task or the reasons for
a low success rate. To support a more effective audio/video analysis, UTAssistant pro-
vides annotation tools, so that when evaluators detect the existence of difficulties, as
indicated by means of verbal comments or facial expressions, they can annotate the
recorded audio/video tracks. If the evaluators decide to record both camera and desktop
videos, the video tracks are merged and displayed together.
3.7 Mouse/Keyboard Logs Analysis (Efficiency)
Important information about the efficiency of performing tasks is given by metrics such
as the time and number of clicks required to complete each task. UTAssistant tracks the
user’s behavior by collecting mouse and keyboard logs. Based on the collected data,
the platform shows performance statistics for each task, such as the number of pages
visited, the average number of clicks and the time that each user needed to complete
the task.
Fig. 3. Summary of metrics measuring performance related to three tasks.
4 UX Evaluation Methodology for Assessing UTAssistant
4.1 Methodology
The UX evaluation design proposed here is an experimental methodology, consisting
of four phases:
Phase 1. Heuristic evaluation of the UTAssistant platform;
Phase 2. Usability evaluation with PA practitioners, under workplace condi-
tions;
Phase 3. Usability evaluation with Web end-users, under experimental labor-
atory conditions;
Phase 4. Usability evaluation with Web end-users, under remote online con-
ditions.
4.2 Objective
This experimental methodology aims to provide a new approach to assessment of the
UTAssistant semi-automatic usability evaluation tool. This methodology combines ex-
pert assessment methods with usability evaluation models, under workplace, labora-
tory, and remote online conditions. The implementation of the UX evaluation method-
ology for the UTAssistant platform is planned for future work.
4.3 Methods and Techniques
The experimental methodology proposed here involves both usability assessment and
psychophysiological measurement methods. Different methods and techniques are used
in each phase, as described below.
Phase 1. Heuristic evaluation. This is an inspection method [13-16] which consists
of experts assessing the usability of a product. In general, the experts involved in heu-
ristic evaluation use a list of principles, also called heuristics, to compare the product
with a baseline representing how the product should meet the main usability require-
ments. Heuristics are based on sets of features for the ideal matching of a user model.
At the end of each evaluation, the expert carrying out a heuristic evaluation provides a
list of problems and related suggestions.
Phase 2. Usability evaluation under workplace conditions. During Phase 2, users
follow the evaluation methodology provided in the eGLU 2.1 protocol, as explained
above in Section 2. A tailored procedure that applies the eGLU 2.1 protocol for usability
evaluation tests is provided to the PAs involved in the UTAssistant experimental pro-
ject. This protocol differs from eGLU 2.1 in that it uses the think-aloud (TA) technique
rather than the partial concurrent think-aloud (PCTA) technique (see Section 4.3, Phase
3). The TA technique is used in traditional usability testing methods [17-24] and is
especially useful in indoor conditions such as laboratories or workplaces. The TA tech-
nique asks users to verbalize (“think aloud”) each action and the problems they encoun-
ter during their interaction with the system. Evaluators are asked to transcribe and ana-
lyze each user action in order to identify interaction problems.
Phase 3. Usability evaluation under laboratory conditions. Usability testing of the
UTAssistant platform is also conducted under laboratory conditions. In this phase, eval-
uators use the PCTA technique, created by the current authors [25-28] to provide a
technique for easily comparing collected data with blind, cognitively disabled, and non-
disabled users. The PCTA technique asks users to interact silently with the interface
and to ring a bell on the desk whenever they identify a problem. In the PCTA technique,
all user interactions are registered. As soon as the test is complete, the user is invited to
identify and verbalize any problems experienced during the interaction [29].
Any psychophysiological reactions of the users that may occur during this interac-
tion are measured using two bio-behavioral measurement techniques: (i) facial expres-
sion recognition; and (ii) electroencephalography (EEG). The EEG method allows
practitioners to record the electrical activity generated by the brain using electrodes
placed on the users scalp. Due to its high temporal resolution, the EEG is able to ana-
lyze which areas of the brain are active at any given moment. The scientific community
also recognizes a limited number of facial expressions (about 45) as universally able to
express hundreds of emotions resulting from the combination of seven basic emotions
[30]: joy, anger, surprise, fear, contempt, sadness, and disgust. In human beings, the
user is mostly unaware of the ways in which facial muscles express basic emotions
[31]. An analysis of involuntary facial expressions returns information about the emo-
tional impact on the users of an interaction with a given interface.
Phase 4. Usability evaluation under remote online conditions. In this phase, users
are recruited through a Web recruitment platform and redirected to the UTAssistant
Web platform. This methodology has previously been validated for psychological stud-
ies [32, 33].
4.4 Material and Equipment
Phases 1, 3, and 4 use the UTAssistant platform to evaluate the Ministry of Economic
Development (MiSE) website (http://www.sviluppoeconomico.gov.it), while Phase 2
uses the platform to evaluate the websites of each PA involved under workplace con-
ditions. All phases are conducted using either desktop or laptop computer with a screen
size of between 13and 15, and a minimum resolution of 1024x640. Computers
should be equipped with a Google Chrome browser
(http://www.google.com/intl/en/chrome). Computers should be plugged into a power
source, and the brightness of the display should be set to the maximum level. In each
phase, different materials and equipment are used, as described below.
Phase 1. Heuristic evaluation. Many heuristic lists are proposed in the literature [29].
In this work, we use 10 heuristics for Web interface analysis created by Nielsen and
Molich [16]; these take into account many aspects of the user interaction such as safety,
flexibility, and efficiency of use. The Nielsen heuristics are based on 10 principles de-
rived from a factorial analysis carried out on a list of 249 problems detected by many
usability evaluations.
Phase 2. Usability evaluation under workplace conditions. This phase uses a tailored
protocol asking managers of PA websites to evaluate them in conjunction with users.
This evaluation should be done using the UTAssistant platform with a desktop or laptop
computer.
Phase 3. Usability evaluation under laboratory conditions. In this phase, UX experts
are asked to measure user interaction by means of two bio-behavioral measurement
devices: a facial expression recognition system, and an EEG. Both devices return data
that can be synchronized using a biometric synchronization platform called iMotions
(http://imotions.com).
Phase 4. Usability evaluation under remote online conditions. Tests are adminis-
tered using an online recruitment procedure involving a crowdsourcing platform for
psychological research called Prolific Academic (http://www.prolific.ac).
4.5 Subjects
Phase 1. Heuristic evaluation. A heuristic evaluation requires a small set of between
three and five expert evaluators.
Phase 2. Usability evaluation under workplace conditions. PA Web managers are
asked to conduct their tests with a minimum of five participants.
Phase 3. Usability evaluation under laboratory conditions. Ten participants are in-
volved, equally divided by gender.
Phase 4. Usability evaluation under remote online conditions. One hundred users
should be recruited. Participants should be equally divided by gender and language (50
native English speakers, and 50 native Italian speakers).
4.6 Procedure
Phase 1. Heuristic evaluation. Experts are asked to evaluate the main actions required
by the UTAssistant platform to assess a website. In particular, experts are asked to
evaluate the user experience of an evaluator using UTAssistant during the following
actions:
Create a new usability test with UTAssistant in order to evaluate the MiSE
website.
Define four user tasks.
Determine which user questionnaires will be administered to users at the end
of the test.
Define which export data the system should record during the interaction.
Export navigation, questionnaire and log data.
Use the help function.
Phase 2. Usability evaluation under workplace conditions. The Web managers in-
volved in this phase are asked to evaluate the usability of their PA website. Web man-
agers are asked to perform the same actions as required in Phase 1, and then to evaluate
their websites with users recruited from within their workplace. Users should be asked
to navigate the administration website to carry out four tasks, presented in the form of
usage scenarios. A help service embedded into the platform is provided to users, which
activates an error message and automatically sends a request to a remote help service.
Phase 3. Usability evaluation under laboratory conditions. In this phase, users are
required to perform the test in a quiet and sufficiently bright environment, using a com-
fortable chair placed at least 50 centimeters from the screen of a desktop or laptop com-
puter. Users are asked to navigate the MiSE website to carry out the four tasks previ-
ously created by the UX expert conducting the sessions. Tasks should be presented to
the users in form of usage scenarios.
Phase 4. Usability evaluation under remote online conditions. Online participants
should be redirected to the UTAssistant platform to evaluate the MiSE website. In this
phase, participants are asked to set up their devices as required in Phases 1, 2, and 3,
and to perform the same tasks as defined in Phase 3, presented in the form of scenarios.
4.7 Data Collection
At the end of each phase, evaluators are asked to store their collected data in a database
hosted by the Superior Institute of Communication and Information Technologies
(ISCOM), Italy. Stored data will be analyzed, phase by phase, in aggregate form. Sta-
tistical analyses and comparisons will be carried out using the IBM SPSS platform, and
then discussed and disseminated through reports and conference papers.
5 Conclusion
The paper describes UTAssistant, a semi-automatic usability assessment web platform
for Italian PA, and proposes a new experimental evaluation methodology for assessing
the UX of the proposed platform. Both UTAssistant and the experimental assessment
methodology were developed as part of a multidisciplinary project involving design
engineers, UX experts and PA Web managers. UTAssistant is a new tool aimed at the
international scientific community; its goal is to provide a standardized model to guide
non-experts in the usability of PA websites, in a quick and straightforward way, to meet
international usability protocols and standards. Unlike the most commonly usability
evaluation methods, the assessment methodology proposed for evaluating the UTAs-
sistant platform uses bio-behavioral measures in addition to the standard validated us-
ability assessment methodologies. The methodology proposed here provides an evalu-
ation strategy that avoids the involvement of social desirability factors (often related to
explicit satisfaction questionnaires), since bio-behavioral measures are hidden from us-
ers. This work is part of a two-year project (20172018) involving Italian PAs, the
University of Bari and the University of Perugia. In future work, the proposed experi-
mental methodology will be implemented to assess the UX of the UTAssistant plat-
form.
References
1. Desolda, G., Gaudino, G., Lanzilotti, R., Federici, S., Cocco, A.: Utassistant: A Web
Platform Supporting Usability Testing in Italian Public Administrations. In: 12th Edition of
CHItaly: CHItaly 2017, pp. 138142 (2017)
2. Dipartimento della Funzione Pubblica: Il Protocollo Eglu 2.1: Come Realizzare Test Di
Usabilità Semplificati Per I Siti Web E I Servizi Online Delle Pa. Formez PA, Rome, IT
(2015)
3. Dipartimento della Funzione Pubblica: Il Protocollo Eglu-M: Come Realizzare Test Di
Usabilità Semplificati Per I Siti Web E I Servizi Online Delle Pa. Formez PA, Rome, IT
(2015)
4. Borsci, S., Federici, S., Mele, M.L.: Eglu 1.0: Un Protocollo Per Valutare La
Comunicazione Web Delle Pa. Diritto e Pratica Amministrativa, vol. 1, pp. 910. Il Sole 24
Ore, Milan, IT (2014)
5. Brooke, J.: Sus: A “Quick and Dirty” Usability Scale. In: Jordan, P.W., Thomas, B.,
Weerdmeester, B.A., McClelland, I.L. (eds.) Usability Evaluation in Industry, pp. 189194.
Taylor & Francis, London, UK (1996)
6. Borsci, S., Federici, S., Lauriola, M.: On the Dimensionality of the System Usability Scale
(Sus): A Test of Alternative Measurement Models. Cogn. Process. 10, 193197 (2009)
7. Di Nocera, F.: Usability Evaluation 2.0: Una Descrizione (S)Oggettiva Dell’usabilità. In:
Ergoproject (ed.), Rome, IT (2013)
8. Borsci, S., Federici, S., Mele, M.L., Conti, M.: Short Scales of Satisfaction Assessment: A
Proxy to Involve Disabled Users in the Usability Testing of Websites. In: Kurosu, M. (ed.)
Human-Computer Interaction: Users and Contexts, Part Iii, Hcii 2015, vol. LNCS 9171, pp.
3542. Springer, Los Angeles, CA (2015)
9. Borsci, S., Federici, S., Bacci, S., Gnaldi, M., Bartolucci, F.: Assessing User Satisfaction
in the Era of User Experience: Comparison of the Sus, Umux and Umux-Lite as a Function
of Product Experience. International Journal of Human-Computer Interaction 31, 484495
(2015)
10. Lewis, J.R., Utesch, B.S., Maher, D.E.: Umux-Lite: When There’s No Time for the Sus.
In: Conference on Human Factors in Computing Systems: CHI ’13, pp. 20992102 (2013)
11. Tomlin, C.: 14 Usability Testing Tools Matrix and Comprehensive Reviews. Useful
Usability, vol. 2018, USA (2014)
12. Lewis, J.R., Sauro, J.: Revisiting the Factor Structure of the System Usability Scale.
Journal of Usability Studies 12, 183192 (2017)
13. Molich, R., Nielsen, J.: Improving a Human-Computer Dialogue. Commun. ACM 33,
338–348 (1990)
14. Nielsen, J.: Heuristic Evaluation. In: Nielsen, J., Mack, R.L. (eds.) Usability Inspection
Methods. John Wiley & Sons, New York, NY (1994)
15. Nielsen, J.: Enhancing the Explanatory Power of Usability Heuristics. In: SIGCHI
Conference on Human Factors in Computing Systems: CHI ’94, pp. 152158 (1994)
16. Nielsen, J., Molich, R.: Heuristic Evaluation of User Interfaces. In: SIGCHI Conference
on Human Factors in Computing Systems: CHI ’90, pp. 249256 (1990)
17. Bettman, J.R.: An Information Processing Theory of Consumer Choice. Addison-
Wesley, Cambridge, MA (1979)
18. Bettman, J.R., Park, C.W.: Implications of a Constructive View of Choice for Analysis
of Protocol Data: A Coding Scheme for Elements of Choice Processes. Advances in
Consumer Research 7, 148153 (1980)
19. Biehal, G., Chakravarti, D.: Experiences with the Bettman-Park Verbal-Protocol
Coding Scheme. Journal of Consumer Research 8, 442448 (1982)
20. Biehal, G., Chakravarti, D.: Information-Presentation Format and Learning Goals as
Determinants of Consumers’ Memory Retrieval and Choice Processes. Journal of Consumer
Research 8, 431441 (1982)
21. Biehal, G., Chakravarti, D.: Consumers’ Use of Memory and External Information in
Choice: Macro and Micro Perspectives. Journal of Consumer Research 12, 382405 (1986)
22. Biehal, G., Chakravarti, D.: The Effects of Concurrent Verbalization on Choice
Processing. Journal of Marketing Research (JMR) 26, 84–96 (1989)
23. Green, A.: Verbal Protocol Analysis. The Psychologist 8, 126129 (1995)
24. Kuusela, H., Spence, M.T., Kanto, A.J.: Expertise Effects on Prechoice Decision
Processes and Final Outcomes: A Protocol Analysis. European Journal of Marketing 32, 559
(1998)
25. Federici, S., Borsci, S., Stamerra, G.: Web Usability Evaluation with Screen Reader
Users: Implementation of the Partial Concurrent Thinking Aloud Technique. Cogn. Process.
11, 263272 (2010)
26. Federici, S., Borsci, S., Mele, M.L.: Usability Evaluation with Screen Reader Users: A
Video Presentation of the Pcta’s Experimental Setting and Rules. Cogn. Process. 11, 285288
(2010)
27. Federici, S., Borsci, S.: Usability Evaluation: Models, Methods, and Applications. In:
Stone, J., Blouin, M. (eds.) International Encyclopedia of Rehabilitation, pp. 1–17. Center for
International Rehabilitation Research Information and Exchange (CIRRIE), Buffalo, NY
(2010)
28. Borsci, S., Federici, S.: The Partial Concurrent Thinking Aloud: A New Usability
Evaluation Technique for Blind Users. In: Emiliani, P.L., Burzagli, L., Como, A., Gabbanini,
F., Salminen, A.-L. (eds.) Assistive Technology from Adapted Equipment to Inclusive
Environments: AAATE 2009, vol. 25, pp. 421425. IOS Press, Amsterdam, NL (2009)
29. Borsci, S., Kurosu, M., Federici, S., Mele, M.L.: Computer Systems Experiences of
Users with and without Disabilities: An Evaluation Guide for Professionals. CRC Press, Boca
Raton, FL (2013)
30. Ekman, P.: Darwin and Facial Expression: A Century of Research in Review. Academic
Press, New York, NY (1973)
31. Qu, F., Yan, W.-J., Chen, Y.-H., Li, K., Zhang, H., Fu, X.: “You Should Have Seen the
Look on Your Face…”: Self-Awareness of Facial Expressions. Frontiers in Psychology 8,
832 (2017)
32. Mele, M.L., Millar, D., Rijnders, C.E.: Validating a Quality Perception Model for
Image Compression: The Subjective Evaluation of the Cogisen’s Image Compression Plug-
In. In: Human-Computer Interaction. Theory, Design, Development and Practice: 18th
International Conference, HCI International 2016, pp. 350359 (2016)
33. Mele, M.L., Millar, D., Rijnders, C.E.: Using Spatio-Temporal Saliency to Predict
Subjective Video Quality: A New High-Speed Objective Assessment Metric. In: Human-
Computer Interaction. User Interface Design, Development and Multimodality: 19th
International Conference, HCI International 2017, pp. 353368 (2017)
... However, this expert evaluator will always provide a certain level of subjectivity in their analysis [4,5]. Although these disadvantages can be discouraging, despite the benefits they provide considering the finished software product, they can be mitigated by implementing usability evaluation tools [5][6][7][8]. ...
... Many tools directly benefit usability evaluation activities in an automated way, allowing, for example, during a usability test to store user registration data such as (i) keystrokes, (ii) clicks made with the mouse, and (iii) the distances traveled by the mouse pointer, among others. These tools allow, in some cases, to analyze the data collected to provide feedback to developers and usability experts, providing information on usability errors and, depending on the tool, automatically correcting them [5][6][7][8][9]. ...
... To form the CG, a traditional search for studies related to the research context and, according to the previous explanation, that responds to the research questions was carried out. As a result of this search, six studies were identified [5,7,8,[17][18][19]. Before building the search string, it is verified if the CG studies are found in the Scopus database since it is the one that hosts the most studies. ...
Chapter
Usability is one of the most critical indicators in determining the quality of a software product. It corresponds to how users can use a software system to achieve specific objectives with effectiveness, efficiency, and satisfaction. A usability evaluation is necessary to ensure that the software system is usable, but this has certain disadvantages (e.g., a high cost of time and budget for the evaluation to be implemented). While these disadvantages can be a bit daunting despite the benefits they provide, some tools can automatically generate and support usability testing. We conducted a systematic mapping study to identify the tools that support automatic usability evaluation. We identified a total of 15 primary studies. In addition, we classify the tools into four categories: measure usability, support usability evaluation, detect usability problems, and correct usability problems. We identified that the automatic evaluation of the usability of web platforms and mobile devices is the most interesting.
... This work describes the heuristic evaluation of eGLU-box, a new remote semiautomatic usability assessment tool that overcomes each of the aforementioned limits. eGLU-box is a re-engineered version of a previous platform called UTAssistant [5][6][7][8], a web-based usability assessment tool developed to provide the Italian Public Administration with an online tool to conduct remote user studies. Both UTAssistant and its renewed version eGLU-box are designed according to usability guidelines provided by GLU, a group working on usability founded by the Department of Public Function, Ministry for Simplification and Public Administration in 2010. ...
... The re-engineering process of UTAssistant was made possible by previous studies by Federici and colleagues, who evaluated user experience (UX) expert users of public administration (PA) websites [6]. In laboratory conditions, they used psychophysiological techniques [5] to measure the underlying reactions of participants through the recognition of facial expressions and electroencephalography (EEG). ...
... This is why in 2017 a web platform called UTAssistant was developed in line with the last Italian PA usability protocol, eGLU 2.1 [9]. Thanks to a UX evaluation of UTAssistant with expert users [6] and in laboratory conditions with two biobehavioral implicit measures [5], a re-engineering process of UTAssistant led to the development of the current version of the platform, eGLU-box. It is divided into two modules, one (the "tester module") for the practitioner who has to create, administer, and analyze the test, and another (the "end-user module") for end-users for whom the test is intended. ...
Chapter
This paper illustrates the heuristic evaluation of a web-based tool for usability testing for Public Administrations called eGLU-box. eGLU-box is an online platform aiming at supporting practitioners in the process of designing usability tests, analyzing data, and helping step-by-step participants to complete assessment tasks. Web users of Public Administrations can report their perceived quality of experience by completing a library of questionnaires shown to them by eGLU-box at the end of the test. This work is part of a multi-step user experience (UX) evaluation methodology to assess the platform. The UX evaluation methodology of eGLU-box uses standard and bio-behavioural evaluation methods. This work shows the results of the heuristic evaluation of eGLU-box involving five human factors experts and 20 practitioners working in Italian Public Administrations. Findings show that most of the problems are rated as minor problems and related to Nielsen’s heuristic, “visibility of the system.” Only 9% of problems are rated as major problems. These major problems are related to the “problematic match between system and the real world” heuristic. Evaluators provided indications for improvements that will be applied for the next version of the platform.
... The potential to reach a larger sample of test subjects was evident. While there were certain methodological reservations, for example, it would not be possible to detect other, verbal and nonverbal, signals such as in laboratory testing, these reservations were balanced by the significant advantages of remote, unmoderated usability [61], user experience and usage load measurements. ...
Article
Full-text available
This is the latest article in a series of research on the family-centered design concept. The theoretical context was revisited and expounded to support its usefulness in conjunction with a user-centered design approach within distinct application domains. A very important contribution is applied through the development of the instruments—website capture, a public testing platform, results processing and the Web Content Accessibility Guide 2.1. recommendation tool—to conduct unmoderated remote testing of user interfaces that corresponds to the requirements of general digitalization efforts as well as the response to current and future health risks. With this set of instruments, an experiment was conducted to address the differences in usage, and performance-wise and user-based evaluation methods, of the eDavki public tax portal, among two generations, adults and elderly citizens, and between an original and an adapted user interface that respects accessibility and other recommendations. The differences found are further discussed and are congruent to particularities that have been modified within interfaces.
... The study was organized according to a within-subject design, with visualization as an independent variable and as within-subject factors the two visualizations. Each visualization showed data gathered in a study conducted with 15 employees of an Italian Administration, who used a Web platform for onsite and remote usability tests [4,5]. With each system, the participants performed the following tasks: 1. Identify the page that caused most problems; 2. Identify the path(s) that led to the task's failure; 3. Identify the path that led to the task success following the ideal path, 4. Identify the paths that led to the task success following alternative paths; 5. Identify backward paths, if possible. ...
Chapter
Full-text available
This paper reports on an on-going work that investigates the use of visualization techniques to help evaluators discovering usability problems by visualizing data collected during usability tests of web sites. Two visualization techniques are described and some results of the evaluation study that compared the two techniques are provided.
Conference Paper
Full-text available
Even if the benefits of the usability testing are remarkable, it is scarcely adopted in the software development process. To foster its adoption, this paper presents a Web platform, UTAssistant, that supports people, also without skills in Human-Computer Interaction (HCI), in evaluating Web site usability.
Article
Full-text available
In 2009, we published a paper in which we showed how three independent sources of data indicated that, rather than being a unidimensional measure of perceived usability, the System Usability Scale apparently had two factors: Usability (all items except 4 and 10) and Learnability (Items 4 and 10). In that paper, we called for other researchers to report attempts to replicate that finding. The published research since 2009 has consistently failed to replicate that factor structure. In this paper, we report an analysis of over 9,000 completed SUS questionnaires that shows that the SUS is indeed bidimensional, but not in any interesting or useful way. A comparison of the fit of three confirmatory factor analyses showed that a model in which the SUS's positive-tone (odd-numbered) and negative-tone (even-numbered) were aligned with two factors had a better fit than a unidimensional model (all items on one factor) or the Usability/Learnability model we published in 2009. Because a distinction based on item tone is of little practical or theoretical interest, we recommend that user experience practitioners and researchers treat the SUS as a unidimensional measure of perceived usability, and no longer routinely compute Usability and Learnability subscales.
Article
Full-text available
The awareness of facial expressions allows one to better understand, predict, and regulate his/her states to adapt to different social situations. The present research investigated individuals’ awareness of their own facial expressions and the influence of the duration and intensity of expressions in two self-reference modalities, a real-time condition and a video-review condition. The participants were instructed to respond as soon as they became aware of any facial movements. The results revealed that awareness rates were 57.79% in the real-time condition and 75.92% in the video-review condition. The awareness rate was influenced by the intensity and (or) the duration. The intensity thresholds for individuals to become aware of their own facial expressions were calculated using logistic regression models. The results of Generalized Estimating Equations (GEE) revealed that video-review awareness was a significant predictor of real-time awareness. These findings extend understandings of human facial
Article
Subjects made an initial choice using external product information. Some concurrently verbalized this choice, whereas others did not. Next, they received more information on new brands and a new attribute for all brands. Both verbalizers and nonverbalizers then made a second choice using some of the first choice information incidentally acquired in memory. All subjects verbalized this second choice. The effects of the first choice verbalization manipulation were examined by analyzing the second choice protocols along with the choice outcome and task perception measures. In comparison with verbalizers, nonverbalizers did more problem framing and brand processing during earlier phases of the second choice. However, choice outcomes did not differ. The findings suggest that the verbalization manipulation may have altered the first choice, creating memory differences that affected some subsequent tasks. Retrieval measures corroborate this conclusion. The concurrent verbal protocol method is evaluated on the basis of these findings.
Conference Paper
We describe a new Objective Video Quality Assessment (VQA) metric, consisting of a method based on spatio-temporal saliency to model human visual perception of quality. Accurate measurement of video quality is an important step in many video-based applications. Algorithms that are able to significantly predict human perception of video quality are still needed to evaluate video processing models, in order to overcome the high cost and time requirement for large-scale subjective evaluations. Objective quality assessment methods are used for several applications, such as monitoring video quality in quality control systems, benchmarking video compression algorithms, and optimizing video processing and transmission systems. Objective Video Quality Assessment (VQA) methods attempt to predict an average of human perception of video quality. Therefore subjective tests are used as a benchmark for evaluating the performance of objective models. This paper presents a new VQA metric, called Sencogi Spatio-Temporal Saliency Metric (Sencogi-STSM). This metric generates subjective quality scores of video compression in terms of prediction efficacy and accuracy than the most used objective VQA models. The paper describes the spatio-temporal model behind the proposed metric, the evaluation of its performance at predicting subjective scores, and the comparison with the most used objective VQA metrics.
Conference Paper
User experience has a fundamental role in determining the effectiveness of image compression methods. This work presents the subjective evaluation of a new compression plug-in for current compression formats developed by Cogisen. The quality of image compression methods is often evaluated by objective metrics based on subjective quality datasets, rather than by using subjective quality evaluation tests. Cogisen’s compression method follows an adaptive compression process that evaluates the saliency of any image and calculates the level of compression beyond which viewers shall be aware of image quality degradation. The Single Stimulus Continuous Quality Scale method was used to conduct the subjective quality evaluation of image compression. Pictures compressed by the Facebook Mobile lossy JPEG compression and by the Cogisen plug-in integrated in the Facebook Mobile compression settings were used. The results of the user quality evaluation of pictures show about a 45 % compression improvement, with no loss in perceived image quality, for pictures compressed by the Cogisen plug-in compared to jpeg pictures as compressed by Facebook Mobile.