Conference PaperPDF Available

UTAssistant: a Web Platform Supporting Usability Testing in Italian Public Administrations



Even if the benefits of the usability testing are remarkable, it is scarcely adopted in the software development process. To foster its adoption, this paper presents a Web platform, UTAssistant, that supports people, also without skills in Human-Computer Interaction (HCI), in evaluating Web site usability.
UTAssistant: a Web Platform
Supporting Usability Testing in Italian
Public Administrations
Even if the benefits of the usability testing are
remarkable, it is scarcely adopted in the software
development process. To foster its adoption, this paper
presents a Web platform, UTAssistant, that supports
people, also without skills in Human-Computer
Interaction (HCI), in evaluating Web site usability.
Author Keywords
Usability testing; public administration web sites;
automatic and semi-automatic tools.
ACM Classification Keywords
H.5.2 Human-centered computing~Usability testing.
HCI literature proposes different methods for evaluating
usability of interactive systems. Among them, usability
testing is generally considered the most complete form
of evaluation, because it assesses usability through
samples of real users and permits to gather a wide
range of qualitative and quantitative data, whose
analysis allow to detect possible issues about the
system and the interface elements which cause
them [7].
Copyright is held by the author/owner(s).
CHItaly ’17, September 18-20, 2017, Cagliari, Italy.
Giuseppe Desolda
Università di Bari
via Orabona, 4
70125 Bari, Italy
Rosa Lanzilotti
Università di Bari
via Orabona, 4
70125 Bari, Italy
Antonello Cocco
Viale America 201
00144 Roma, Italy
Giancarlo Gaudino
Viale America 201
00144 Roma, Italy
Stefano Federici
Università di Perugia
Piazza G. Ermini, 1
06123 Perugia, Italy
Although the benefits of this technique are
unquestionable, it is hardly adopted due to the required
effort and time needed to set up reliable user testing. A
frequently used technique is the thinking aloud in which
users are asked to verbalize their actions during the
interaction with the system to complete a set of critical
tasks [7]. During the task execution, the evaluators can
collect quantitative data, e.g. task time/clicks, and
qualitative data, e.g. facial expressions and
externalized comments. In addition, questionnaires are
typically administered to evaluate the users’ perceived
usability. Then, all the collected data are analyzed to
evaluate the software usability with respect to the
administered tasks, thus producing statistics like the
task completion time or number of clicks to accomplish
tasks, task success rate, heat map of the user
behavior, questionnaire results and graphs, and so on.
To foster the adoption of usability testing technique, in
the past few years, there has been massive growth of
usability testing tools. For example, professional
platforms like Morae, Ovo Logger and User Testing
support HCI experts in most of the evaluation activities
[2][9, 11]. Despite their powerful features and
automatization of data gathering and analysis, their use
is limited by their price and/or the need to install them
on dedicate machines (e.g., Morae and Ovo Logger).
Simpler and cheaper tools are also available but they
typically cover part of the evaluation process, for
example the user behavior analysis obtained gathering
and elaborating the mouse movements/clicks (e.g.,
UsabilityTool) or the facial expressions and externalized
comments recording (e.g. Camtasia).
This paper proposes a Web platform, called
UTAssistant, which assists evaluators, even without
skills in HCI, in performing usability testing.
UTAssistant automatizes and simplifies the usability
testing process in all its phases, i.e. design, execution
and data analysis.
UTAssistant is a Web platform designed and developed
within the PA++ Project, an initiative coordinated by
the High Institute of Communication and Information
Technology (ISCOM) scientific and technical body of the
Italian Ministry of Economic Development (MISE),
whose main goal is to promote the usability practices in
the Italian Public Administration (PA). In particular, the
PA++ (A Public Administration + mobile and + usable:
design and evaluation of Public Administration Web
sites) project involves a grouo of people that are
members of the GLU (Gruppo di Lavoro per l’Usabilità,
in English, Working Group on Usability). The GLU is
related to the Italian Ministry of Public Administration
and is working to support people of the PA website staff
in performing usability evaluation activities of their
websites and other e-government systems. Among the
different activities of the GLU, in May 2015, a new
version of the eGLU and eGLU-M Protocol 2.1 has been
published [1]; its aim is to guide web masters and web
editors, who do not have experience on usability and
UX evaluation, in performing usability testing of the
websites they work on, by committing limited resources
in terms of time and people.
UTAssistant has been conceived to support remote
usability testing executed by using desktop and mobile
devices, from the test design to the analysis of the data
collected following the guidelines provided by the eGLU
2.1 and eGLU-M evaluation protocol. UTAssistant has
been entirely developed as Web platform. The idea is to
provide Italian PA with a lightweight and simple service
that does not require any installation on user devices or
special requirements. In fact, thanks to the use of the
recent evolutions of the HTML5 standard, the browser
gathers data from desktop and mobile peripherals like
webcam, microphone, mouse, keyboard, and so on.
From a technological point of view this is an important
step forward with respect to the state-of-the-art, which
can foster a wide adoption of this tool and,
consequently, of the usability testing technique.
Indeed, the existing software for usability testing,
require dedicated devices (e.g. Morae) or cover part of
the evaluation process and often do not support the
evaluation of web site used on mobile devices that
today represent the most common interaction device.
In the following sections, it is described how
UTAssistant supports evaluators in designing a usability
test and in analyzing the results, as well as how
participants are driven by UTAssistant in completing
usability tests.
Test Design
The first phase of a usability testing is its design. The
most important issue of this phase is the study creation
that mainly consists in defining a set of tasks users
have to complete during the test execution. After the
logging into the system, evaluators create a new study:
a wizard procedure guides them in designing the test
following three main steps. At the first step of the test
design, the evaluators have to define: a) general
information about the study (e.g., a title, a short
introduction), b) data UTAssistant has to gather while
participants carry out the tasks (e.g., mouse/keyboard
data log, audio/video from participant
webcam/microphone), and c) questionnaires
participants have to complete during/at the end of the
study. According to the eGLU protocol, UTAssistant
actually implements post-questionnaires like SUS [5],
NPS [8], UMUX-Lite [6], USE [10]. Another post-
questionnaire, not mentioned by the eGLU protocols
but also implemented in UTAssinstant are: AttrakDiff,
[4] and the NASA-TLX questionnaire [3]. After defining
the study general information, at the second step
evaluators define the task list. For each task, they have
to provide the title, the starting/ending URL, the goal,
the duration. Finally, the third step requires evaluators
to specify the study participants, by selecting them
from a list of users already registered in the platform or
by typing their email. Then the study has to be saved.
The invited participants receive an email describing all
the instructions to participate in the usability testing.
Test Execution
A usability testing execution is typically managed by
one or more HCI experts, which assist participants in
carrying out the study tasks, gather data (e.g.,
externalized comments, task times/click) and
administer the questionnaires. UTAssistant aims to
automatize all the execution steps guiding participants
even without the physical presence of evaluators.
Once the participant receives an email, by clicking the
URL reported inside it, UTAssistant is open in a browser
on the desktop or mobile device. First, UTAssistant
shows general indications about the platform (e.g., a
short description of the toolbar with all the useful
commands) as well as the privacy policies indicating
that data like mouse/keyboard log data and
webcam/microphone stream could be captured. Then,
UTAssistant administers the tasks one at a time. For
each task, the platform shows the goal in a pop-up
window. A minimal toolbar is positioned on top of the
task web page reporting the current task goal and
description, the duration time, the task number, and
the buttons to go to the next task, or stop the study. If
the user is executing the last task, the “Next task
button becomes “Fill in the questionnaire”. All the
steps, are completely assisted by UTAssistant that
simulates the presence of the HCI expert.
Test Data Analysis
One of the most time-consuming phase of a usability
testing is the data analysis. HCI experts have to store,
merge and analyze the mouse logs, the video/audio
recordings, the questionnaire results, and so on.
UTAssistant assists evaluators in performing these
activities, thus making easier and lighter the entire
analysis. UTAssistant provides different types of tools
for the automatic data analysis and to produce
information that facilitate the evaluators’ work in
detecting usability issues.
One of the most important usability metric is the task
success rate, i.e. the percentage of tasks that users
complete correctly during the study [7]. UTAssistant
creates a report showing the general success rate
(“Tasso di successo medio complessivo” in Italian as
shown at the bottom of Figure 1) and a table reporting
the details for each task (column)/user (row).
In addition, for each task UTAssistant shows: its title,
the number of pages the user visited to carry out it, the
number of notes evaluator made during the analysis,
the date and time in which the task starts and finishes,
the average time and the average number of clicks all
the users needed to complete the task.
A different analysis of the mouse/keyboard logs is
provided by the heatmaps, i.e., graphical visualizations
overlapped to a web page that show where people
click/touch, move the mouse, scroll. It uses hotter
colors, e.g. orange or bright red, on areas that get the
most clicks/movements and colder colors, e.g. blues
and purples, on areas that get the less
clicks/movements. For each web page, UTAssistant
builds a heatmap that evaluators can analyze to
understand if some area of the web page wrongly
attracted the users’ attention due to some user
interface elements. Figure 2 shows an example of
mouse heatmap.
UTAssistant automatically analyzes questionnaire data
and provides proper statistics and graphs to summarize
the results and to help evaluators in understanding the
perceived usability, cognitive load and so on.
This paper has presented a web-based platform, called
UTAssistant, developed in the project PA++, to support
people working in PA website staff in performing
usability tests. A preliminary evaluation of the
UTAssistant, involving 2 people acting as evaluators
and 6 people as participants, showed that evaluators
appreciated a lot the support to the evaluation
activities, while participants enjoyed the possibility to
participate in a user test, without being observed.
This work is supported by ISCOM under grant PA++ (A
Public Administration + mobile and + usable: design
and evaluation of Public Administration Web sites).
Figure 1. Summary of the success rate of a
performed study.
Figure 2. Example of heatmap overlapped to
a page involved in a task execution.
1. eGLU and eGLU-M Protocol 2.1. Retrieved from Last
Access June 4th, 2017
2. Morae. Retrieved from Last
Access June 10th, 2017
3. Hart S. G. and Staveland L. E. (1988).
Development of NASA-TLX (Task Load Index):
Results of empirical and theoretical research.
Advances in psychology 52(1), 139-83.
4. Hassenzahl M., Burmester M., and Koller F. (2003).
AttrakDiff: A questionnaire to measure perceived
hedonic and pragmatic quality. In Proc. of the
Mensch & Computer. 187-96.
5. Lewis J. and Sauro J. (2009). The Factor Structure
of the System Usability Scale. In Human Centered
Design - HCD 2009, Kurosu M. Ed. Lecture Notes in
Computer Science, Vol. 5619, Springer Berlin
Heidelberg, 94-103
6. Lewis J. R., Utesch B. S., and Maher D. E. (2013).
UMUX-LITE: when there's no time for the SUS. In
Proc. of the Conference on Human Factors in
Computing Systems (CHI '13). Paris, France, ACM,
New York, NY, USA, 2099-102.
7. Preece J., Sharp H., and Rogers Y. 2015.
Interaction Design-beyond human-computer
interaction. John Wiley & Sons.
8. Reichheld F. F. (2003). The one number you need
to grow. Harvard business review 81(12), 46-55.
9. Ovo Logger. Retrieved from Last Access June
10th, 2017
10. Terenzi M., Di Nocera F., and Ferlazzo F. (2006).
Firmitas, Utilitas, Venustas: Assessing the Validity
of the Usability Evaluation (Us. E.) Questionnaire.
Developments in Human Factors in Transportation,
Design, and Evaluation. Maastricht, the
Netherlands: Shaker Publishing, 249-53.
11. UserTesting. Retrieved from Last Access June
10th, 2017
... This work describes the heuristic evaluation of eGLU-box, a new remote semiautomatic usability assessment tool that overcomes each of the aforementioned limits. eGLU-box is a re-engineered version of a previous platform called UTAssistant [5][6][7][8], a web-based usability assessment tool developed to provide the Italian Public Administration with an online tool to conduct remote user studies. Both UTAssistant and its renewed version eGLU-box are designed according to usability guidelines provided by GLU, a group working on usability founded by the Department of Public Function, Ministry for Simplification and Public Administration in 2010. ...
This paper illustrates the heuristic evaluation of a web-based tool for usability testing for Public Administrations called eGLU-box. eGLU-box is an online platform aiming at supporting practitioners in the process of designing usability tests, analyzing data, and helping step-by-step participants to complete assessment tasks. Web users of Public Administrations can report their perceived quality of experience by completing a library of questionnaires shown to them by eGLU-box at the end of the test. This work is part of a multi-step user experience (UX) evaluation methodology to assess the platform. The UX evaluation methodology of eGLU-box uses standard and bio-behavioural evaluation methods. This work shows the results of the heuristic evaluation of eGLU-box involving five human factors experts and 20 practitioners working in Italian Public Administrations. Findings show that most of the problems are rated as minor problems and related to Nielsen’s heuristic, “visibility of the system.” Only 9% of problems are rated as major problems. These major problems are related to the “problematic match between system and the real world” heuristic. Evaluators provided indications for improvements that will be applied for the next version of the platform.
... Based on the eGLU 2.1 protocol, a new web platform called UTAssistant (Usability Tool Assistant) was developed. It is a lightweight and simple, semiautomatic usability evaluation tool to support practitioners in the usability evaluation of web systems and services provided by the PA Desolda et al., 2017;Catarci et al., 2018). Federici and colleagues (2018) described the design of an experimental methodology aimed at evaluating the User eXperience (UX) of UTAssistant. ...
Conference Paper
Full-text available
This work shows the user experience (UX) assessment of a web-based platform for the semi-automatic usability evaluation of websites, UTAssistant, which is primarily addressed to workers in public administration (PA). The study is part (Phase 1) of a multiple assessment methodology which consists of four phases in total: (1) UX in laboratory conditions; (2) Usability evaluation in remote online conditions; (3) Usability evaluation in workplace conditions; and (4) Heuristic evaluation. In Phase 1, a UX study in laboratory conditions was carried out. Participants' UX of a PA website navigation through UTAssistant was evaluated by both traditional self-report usability assessment tools (SUS and UMUX) and bio-behavioral measurement techniques (facial expression recognition and electroencephalography). Results showed that using the UTAssistant usability assessment tool for webpages did not affect users' perceived usability in terms of self-reports and affective states, which were mostly neutral for all the assessment session. However, frontal alpha asymmetry EEG's scores showed a higher sensitivity of UTAssistant users to the duration of the trial, with a decrease in motivation displayed as the trial ensued. However, this result did not seem to affect emotional experience.
... This paper describes the design of an experimental methodology that aims to evaluate the User eXperience (UX) of a new Web platform, called UTAssistant (Usability Tool Assistant) [1]. This is a semi-automatic usability evaluation tool that supports practitioners in usability evaluations of Web systems and services provided by a public administration (PA), according to the eGLU 2.1 technical protocol [2]. ...
Full-text available
Since 2012, usability testing in Italian public administration (PA) has been guided by the eGLU 2.1 technical protocols, which provide a set of principles and procedures to support specialized usability assessments in a controlled and predictable way. This paper describes a new support tool for usability testing that aims to facilitate the application of eGLU 2.1 and the design of its User eXperience (UX) evaluation methodology. The usability evaluation tool described in this paper is called UTAssistant (Usability Tool Assistant). UTAssistant has been entirely developed as a Web platform, supporting evaluators in designing usability tests, analyzing the data gathered during the test and aiding Web users step-by-step to complete the tasks required by an evaluator. It also provides a library of questionnaires to be administered to Web users at the end of the usability test. The UX evaluation methodology adopted to assess the UTAssistant platform uses both standard and new bio-behavioral evaluation methods. From a technological point of view, UTAssistant is an important step forward in the assessment of Web services in PA, fostering a standardized procedure for usability testing without requiring dedicated devices, unlike existing software and platforms for usability testing.
... The study of shared visual elements between artificial systems and humans leads to new interesting perspectives of artificial intelligence which is increasingly closer to the human ones. The current work presents a usability evaluation Web platform called UTAssistant ("Usability Tool Assistant") [10], a semiautomatic digital tool developed to support PA web service practitioners in designing usability assessment tests from the outset through the data analysis process, by following the principles and recommendations of both eGLU 2.1 and eGLU-M. UTAssistant aims to provide Italian PA with an easy-to-use tool for assessing the usability of PA websites and services; there is no need to install any software on evaluators' devices, as required by the existing tools for usability testing (e.g., Morae, ...
Conference Paper
Full-text available
In the framework of the AVI 2018 Conference, the interuniversity center ECONA has organized a thematic workshop on "Digital Interaction: where are we going?". Six contributions from the ECONA members investigate different perspectives around this thematic.
Full-text available
In 2009, we published a paper in which we showed how three independent sources of data indicated that, rather than being a unidimensional measure of perceived usability, the System Usability Scale apparently had two factors: Usability (all items except 4 and 10) and Learnability (Items 4 and 10). In that paper, we called for other researchers to report attempts to replicate that finding. The published research since 2009 has consistently failed to replicate that factor structure. In this paper, we report an analysis of over 9,000 completed SUS questionnaires that shows that the SUS is indeed bidimensional, but not in any interesting or useful way. A comparison of the fit of three confirmatory factor analyses showed that a model in which the SUS's positive-tone (odd-numbered) and negative-tone (even-numbered) were aligned with two factors had a better fit than a unidimensional model (all items on one factor) or the Usability/Learnability model we published in 2009. Because a distinction based on item tone is of little practical or theoretical interest, we recommend that user experience practitioners and researchers treat the SUS as a unidimensional measure of perceived usability, and no longer routinely compute Usability and Learnability subscales.
Conference Paper
Full-text available
In this paper we present the UMUX-LITE, a two-item questionnaire based on the Usability Metric for User Experience (UMUX) [6]. The UMUX-LITE items are This system's capabilities meet my requirements and This system is easy to use." Data from two independent surveys demonstrated adequate psychometric quality of the questionnaire. Estimates of reliability were .82 and .83 -- excellent for a two-item instrument. Concurrent validity was also high, with significant correlation with the SUS (.81, .81) and with likelihood-to-recommend (LTR) scores (.74, .73). The scores were sensitive to respondents' frequency-of-use. UMUX-LITE score means were slightly lower than those for the SUS, but easily adjusted using linear regression to match the SUS scores. Due to its parsimony (two items), reliability, validity, structural basis (usefulness and usability) and, after applying the corrective regression formula, its correspondence to SUS scores, the UMUX-LITE appears to be a promising alternative to the SUS when it is not desirable to use a 10-item instrument.
Companies spend lots of time and money on complex tools to assess customer satisfaction. But they're measuring the wrong thing. The best predictor of top-line growth can usually be captured in a single survey question: Would you recommend this company to a friend? This finding is based on two years of research in which a variety of survey questions were tested by linking the responses with actual customer behavior--purchasing patterns and referrals--and ultimately with company growth. Surprisingly, the most effective question wasn't about customer satisfaction or even loyalty per se. In most of the industries studied, the percentage of customers enthusiastic enough about a company to refer it to a friend or colleague directly correlated with growth rates among competitors. Willingness to talk up a company or product to friends, family, and colleagues is one of the best indicators of loyalty because of the customer's sacrifice in making the recommendation. When customers act as references, they do more than indicate they've received good economic value from a company; they put their own reputations on the line. And they will risk their reputations only if they feel intense loyalty. The findings point to a new, simpler approach to customer research, one directly linked to a company's results. By substituting a single question--blunt tool though it may appear to be--for the complex black box of the customer satisfaction survey, companies can actually put consumer survey results to use and focus employees on the task of stimulating growth.
Protocol 2.1. Retrieved from http
  • Eglu-M Eglu
eGLU and eGLU-M Protocol 2.1. Retrieved from Last Access June 4th, 2017
Retrieved from Last Access
  • Morae
Morae. Retrieved from Last Access June 10th, 2017
Retrieved from Last Access
  • Ovo Logger
Ovo Logger. Retrieved from Last Access June 10th, 2017
Venustas: Assessing the Validity of the Usability Evaluation (Us. E.) Questionnaire. Developments in Human Factors in Transportation, Design, and Evaluation
  • Utilitas Firmitas
Firmitas, Utilitas, Venustas: Assessing the Validity of the Usability Evaluation (Us. E.) Questionnaire. Developments in Human Factors in Transportation, Design, and Evaluation. Maastricht, the Netherlands: Shaker Publishing, 249-53.
Retrieved from Last Access
  • Usertesting
UserTesting. Retrieved from Last Access June 10th, 2017