Conference PaperPDF Available

Bio-behavioral and Self-Report User Experience Evaluation of a Usability Assessment Platform (UTAssistant)

Authors:

Abstract

This work shows the user experience (UX) assessment of a web-based platform for the semi-automatic usability evaluation of websites, UTAssistant, which is primarily addressed to workers in public administration (PA). The study is part (Phase 1) of a multiple assessment methodology which consists of four phases in total: (1) UX in laboratory conditions; (2) Usability evaluation in remote online conditions; (3) Usability evaluation in workplace conditions; and (4) Heuristic evaluation. In Phase 1, a UX study in laboratory conditions was carried out. Participants' UX of a PA website navigation through UTAssistant was evaluated by both traditional self-report usability assessment tools (SUS and UMUX) and bio-behavioral measurement techniques (facial expression recognition and electroencephalography). Results showed that using the UTAssistant usability assessment tool for webpages did not affect users' perceived usability in terms of self-reports and affective states, which were mostly neutral for all the assessment session. However, frontal alpha asymmetry EEG's scores showed a higher sensitivity of UTAssistant users to the duration of the trial, with a decrease in motivation displayed as the trial ensued. However, this result did not seem to affect emotional experience.
Bio-behavioral and Self-Report User Experience Evaluation of a
Usability Assessment Platform (UTAssistant)
Stefano Federici1, Maria Laura Mele1, Marco Bracalenti1, Arianna Buttafuoco1, Rosa Lanzilotti2,
Giuseppe Desolda2
1Department of Philosophy, Social and Human Sciences and Education, University of Perugia, Piazza Ermini 1, Perugia,
Italy
stefano.federici@unipg.it, {marialaura.mele, marco.bracalenti, arianna.buttafioco}@gmail.com}
2Department of Computer Science, University of Bari Aldo Moro, Bari, Italy
{rosa.lanzilotti,giuseppe.desolda}@uniba.it}
Keywords: User Experience, UX Bio-behavioral Methods, Semi-automatic Usability Assessment, Usabi lit y Assessment
Tools.
Abstract: This work shows the user experience (UX) assessment of a web-based platform for the semi-automatic
usability evaluation of websites, UTAssistant, which is primarily addressed to workers in public
administration (PA). The study is part (Phase 1) of a multiple assessment methodology which consists of
four phases in total: (1) UX in laboratory conditions; (2) Usability evaluation in remote online conditions;
(3) Usability evaluation in workplace conditions; and (4) Heuristic evaluation. In Phase 1, a UX study in
laboratory conditions was carried out. Participants’ UX of a PA website navigation through UTAssistant
was evaluated by both traditional self-report usability assessment tools (SUS and UMUX) and bio-
behavioral measurement techniques (facial expression recognition and electroencephalography). Results
showed that using the UTAssistant usability assessment tool for webpages did not affect users’ perceived
usability in terms of self-reports and affective states, which were mostly neutral for all the assessment
session. However, frontal alpha asymmetry EEG’s scores showed a higher sensitivity of UTAssistant users
to the duration of the trial, with a decrease in motivation displayed as the trial ensued. However, this result
did not seem to affect emotional experience.
1 INTRODUCTION
In October 2012, the Department of Public Function,
Italian Ministry for Simplification and Public
Administration, founded the “GLU” (Working Group
on Usability), that is working to support the public
administration (PA) website staff in performing
usability evaluation activities on their websites and
other Italian e-government systems. Its primary goals
were a collection of the best usability experiences of
PA websites, the development of a practical tool that
could operatively support the analysis and evaluation
of the user interfaces, and the testing of this tool. In
2013, the GLU developed the first guide protocol
(eGLU; Borsci et al., 2014) for Italian web editors of
the PA, to investigate any difficulties a site user could
have in finding information, consulting documents, or
filling in online forms. The actual version of the
eGLU protocol, released in 2015, is the 2.1
(Dipartimento della Funzione Pubblica, 2015). Based
on the eGLU 2.1 protocol, a new web platform called
UTAssistant (Usability Tool Assistant) was
developed. It is a lightweight and simple,
semiautomatic usability evaluation tool to support
practitioners in the usability evaluation of web
systems and services provided by the PA (Federici et
al., 2018; Desolda et al., 2017; Catarci et al., 2018).
Federici and colleagues (2018) described the
design of an experimental methodology aimed at
evaluating the User eXperience (UX) of UTAssistant.
This methodology was applied toward assessing the
platform with end-users and web managers of PA
websites, both in a laboratory setting and through a
web-based recruitment platform. The methodology
adopted used both usability assessment and
psychophysiological measurements applied in four
different conditions: (1) UX in laboratory conditions;
(2) usability evaluation in remote online conditions,
(3) usability evaluation in workplace conditions; and
(4) heuristic evaluation.
This paper describes Phase 1, in which evaluators
used the Partial Concurrent Thinking Aloud (PCTA)
technique (Federici et al., 2010; Borsci and Federici,
2009) to apply traditional usability methods based on
self-report questionnaires. Users
psychophysiological reactions as occurred during the
interaction were also measured using two bio-
behavioral detection techniques: facial expression
recognition and electroencephalography (EEG), in
order to evaluate the emotional impact, the interface
with UTAssistant had on users.
2 USING BIO-BEHAVIORAL
MEASURES IN UX
EVALUATION
The term UX” refers to a new and broad concept that
can be defined according to different perspective. In
this paper, it was considered the definition of UX as
“person’s perceptions and responses resulting from
the use and/or anticipated use of a product(Brooke,
1996; Borsci et al., 2015). In particular, the
interactive experience of a user is affected by the
amount of user interaction with a product, the
perceived usability and aesthetics of interface
(Brooke, 1996; Borsci et al., 2015). Therefore, in a
UX evaluation test it should be included a set of
subjective and objective measures of interactive
experience.
Traditional methods to assess practical and
psychological aspects of UX are based on self-
reporting methodologies, which include
questionnaires, interviews and contextual inquiry and
represent the subjective perception of product use
(Ganglbauer et al., 2009; Isbister et al., 2006). All the
abovementioned techniques ask users to express their
opinion about or evaluate the usability of a product or
its features (Mele and Federici, 2012). Self-reporting
methodologies can be used before and/or after the
product use according to the dimensions that they are
intended to evaluate. Even though evaluating short-
term practical experience is relevant to understanding
the dynamic changes related to situational and
emotional factors, it is also necessary to employ
reliable methods that are able to evaluate user
experience during the interaction with a product as
well (Ganglbauer et al., 2009; Vermeeren et al., 2010;
Jokela, 2010).
In the last decades, implicit methods are
increasingly employed in the evaluation of UX.
Implicit methods are based on the analysis of the
users’ behavior independent of their awareness of
perceptions and their conscious control (Mele and
Federici, 2012; Jimenez-Molina et al., 2018).
Specifically, psychophysiological measurements
have been determined to be reliable approaches for
measuring the many aspects of interaction user
experience (Ganglbauer et al., 2009).
Usually, the psychophysiological techniques
most used in the field of UX evaluation are:
electromyography, galvanic skin response (GSR),
respiration rate, electroencephalography (EEG),
electrocardiography (ECG), eye tracking and
electrodermal activity (EDA).
The state of the art regarding psychophysiological
methods to evaluate user experience shows three
main advantages of employing these techniques: (a)
they are able to investigate cognitive processes such
as, for example, the changes in mental workload that
may be caused by a difficult interaction; (b) they
provide insights in the affective aspects of interaction
such as users’ engagement and satisfaction during
product use; (c) they provide constant feedback
during the interaction, allowing designers to
customize product features for the users during the
interface itself. Furthermore, using implicit methods
may involve several areas of interest such as web
browsing, gaming, software and website design, and
research on packaging and organoleptic properties of
food. In order to investigate more about the
advantages of using psychophysiological methods,
we conducted a literature review to address the
following questions: first, which type of implicit
methods are mainly used in the evaluation of user
experience; second, whether implicit techniques have
never been combined with traditional methods; and
last, what is the additional value of
psychophysiological techniques.
The literature shows that the psychophysiological
method most employed for UX purposes is ECG,
followed by EEG and GSR equally with EDA. The
primacy of ECG can be explained by its temporal
precision of detection and recording, which is simple,
effective, low cost, noninvasive and continuous
(Goshvarpour et al., 2017). Moreover, the literature
shows that implicit methods are frequently combined
with traditional ones because using self-report
together with psychophysiological techniques allows
practitioners to overcome shortcomings of traditional
methods (Poore et al., 2017) and to enhance the data
concerning cognitive and affective aspects of UX
(Muñoz Cardona et al., 2016). A particularly useful
example of such a combined instrument is the Game
Engagement Questionnaire (Brockmyer et al., 2009).
Finally, it is widely accepted that an additional
value of psychophysiological methods in evaluating
UX is provided by their objectivity and their capacity
to bypass language and memory restrictions (Muñoz
Cardona et al., 2016). Further, with their use it is
possible to assess continuously the UX with high
temporal precision and without interference
stemming from product exploration (Zhang et al.,
2018; Jimenez-Molina et al., 2018; Rodriguez-
Guerrero et al., 2017; Muñoz Cardona et al., 2016;
Poore et al., 2017; Vourvopoulos and Bermúdez i
Badia, 2016; Yan et al., 2016; Pallavicini et al., 2013;
Nacke et al., 2011; Jun et al., 2010). Furthermore, the
great amount of information collected permits the
obtaining of a holistic understanding of UX regarding
both practical and psychological aspects. Finally, the
psychophysiological techniques are helpful in
guiding adaptation of product features to user needs
(Mendoza-Denton et al., 2017; Christensen and
Estepp, 2013).
3 UTASSISTANT: A NEW
USABILITY TESTING TOOL
FOR ITALIAN PUBLIC
ADMINISTRATION
UTAssistant is a web platform to provide the Italian
Public Administration with a lightweight and simple
tool to conduct user studies, according to the eGLU
2.1 protocol, without requiring any installation on
user devices (Federici et al., 2018).
One of the most important objectives driving the
platform development was the need to perform
remote usability studies, with the aim of stimulating
users to participate in usability tests in a simpler and
more comfortable way (Jokela, 2010). To accomplish
such requirements, UTAssistant has been developed
as a web platform so that the involved stakeholders,
namely evaluators and web users, act from their PCs
wherever and whenever they desire. With respect to
the state-of-the-art of usability test tools, this aspect
represents an important contribution, since remote
participation fosters a wider adoption of this tool and,
consequently, of the usability test technique. Indeed,
the existing tools for usability tests require software
installation on PCs with specific requirements (e.g.
Morae® https://www.techsmith.com/morae.html).
3.1 Usability test design
Every usability test starts from the evaluation design,
which mainly consists of: (i) creating a script to
introduce the users to the test; (ii) defining a set of
tasks; (iii) identifying data to be gathered (e.g.
number of clicks and time required by user to
accomplish a task, audio/video/desktop recording,
logs, etc.); and (iv) deciding which questionnaire(s)
to administer to users.
UTAssistant facilitates evaluators in performing
these activities by means of three wizard procedures.
The first procedure guides evaluators in specifying:
(a) general information (e.g. a title, the script); (b)
data to gather during user task execution (e.g.
mouse/keyboard data logs,
webcam/microphone/desktop recordings); (c) post-
test questionnaire(s) to administer. After that, the
second procedure assists evaluators in creating the
task lists. For each task, starting/ending URLs, the
goal and the duration must be specified. Finally, the
third procedure requires evaluators to determine the
users they evaluate, by selecting them from a list of
users already registered to the platform or by typing
their email address.
3.2 Usability test execution
After creation of the usability test design, the users
receive an email with information about the
evaluation they must complete and a link to access the
UTAssistant. By clicking on the link, the users can
carry out the evaluation test, which starts by
providing general instructions about the platform use
(e.g. a short description of the toolbar with all the
useful commands), the script of the evaluation and,
finally, privacy policies pertaining to how data such
as mouse or keyboard logs and webcam, microphone
and desktop recordings will be captured.
Afterwards, UTAssistant administers all the tasks
one at a time. Each task execution is strongly guided
by the platform that, after displaying the task
description in a pop-up window, opens the webpage
from which users must start the task execution. To
minimize the invasiveness of the platform during the
evaluation test execution, we grouped all the
functions and indications—such as the current task
goal and instructions, duration time, task number, and
buttons to go to the next task or stop the evaluation—
in a toolbar placed at the top of the webpage. The
toolbar displays the button to go to the next task:
“Complete Questionnaire” when the users finish the
last task and then move to complete the
questionnaire(s). During task execution, the platform
collects all the data the evaluator set during the study
design in a transparent and noninvasive way.
3.3 Evaluation test data analysis
UTAssistant automates all activities (such as collect,
store, merge and analyze) related to data analysis,
removing barriers in gathering usability test data. The
evaluators access the data analysis results in their
control panel, by exploiting different tools that
provide a useful support in finding usability issues. In
the next subsections, an overview of some of such
tools is reported.
3.4 Task success rate
UTAssistant calculates the task success rate (the
percentage of tasks that users correctly complete
during the test, that can be also calculated for each
task, estimating the percentage of users who complete
that task), and visualizes them in a table in which the
columns represent the tasks, while the rows show the
users. The last row reports the success rate for each
task while the last column depicts the success rate for
each user. The global success rate is reported below
the table.
3.5 Questionnaire results
Thanks to UTAssistant, the evaluators can administer
one or more questionnaires at the end of the usability
evaluation. The platform automatically stores all the
user’s answers and produces results by means of
statistics and graphs. For example, if the System'
Usability' Scale (SUS) (Brooke, 1996; Borsci et al.,
2015; Jokela, 2010) has been chosen, UTAssistant
calculates statistics such as global SUS score,
usability score and learnability score. In addition,
different visualizations depict the results from
different perspectives, e.g. a histogram of users’ SUS
scores, a box-plot of SUS score/learnability/usability,
and a score that is compared to SUS evaluation scales.
3.6 Audio/video analysis
While users execute the study tasks, UTAssistant
collects and stores the user’s voice (through the
microphone), facial expressions (through the
webcam), and desktop (through a browser plug-in).
The implemented player also provides an annotation
tool so that, when evaluators detect some difficulties
externalized by means of verbal comments or facial
expressions, they can document the recorded
audio/video tracks. If the evaluators decide to record
both camera and desktop videos, their tracks are
merged in a picture-in-picture fashion.
3.7 Mouse/keyboard logs analysis
Finally, UTAssistant tracks user’s behavior by
collecting mouse and keyboard logs and, starting
from the collected data, the platform shows
performance statistics for each task.
4 UX EVALUATION OF
UTASSISTANT
This paper aims at evaluating the UX of UTAssistant
under laboratory conditions through two bio-
behavioral implicit measures, i.e. facial expression
recognition and electroencephalography, and two
explicit measures, the SUS (Lewis and Sauro, 2017;
Borsci et al., 2015) and the Usability Metric for User
Experience (UMUX) (Finstad, 2010; Borsci et al.,
2015). The methodology adopted in this usability
study was the PCTA (Borsci et al., 2013).
4.1 Methods
The quality of the interaction with UTAssistant is
measured through bio-behavioral measurement
techniques which are able to measure the underlying
psychophysiological reactions of participants through
(i) the recognition of facial expressions and (ii)
electroencephalography (EEG).
(i) The scientific community recognizes a limited
number of facial expressions (45 action units) which
are considered universally connected to hundreds of
emotions resulting from the combination of seven
basic emotions (Ekman et al., 2002): joy, anger,
surprise, fear, contempt, sadness and disgust. In
humans, facial movements related to basic emotions
are unaware and automatic. The analysis of
involuntary facial expressions provides information
on the emotional impact that an interface can elicit
during the interaction.
(ii) The EEG method allows recording the
electrical activity generated by the neurons through
electrodes positioned on the participant’s scalp.
Thanks to a high temporal resolution, the EEG allows
analysis of which brain areas are active at a given
moment. In this study, the frontal alpha (8–12 Hz)
asymmetry index was derived for each participant.
Frontal alpha asymmetry (FAA) reflects the levels of
approach/withdraw cognitive processes calculated by
the difference between right-hemispheric electrodes,
in which an increase in alpha power is an index of
withdrawal motivation or related negative emotion
(Gruzelier, 2014), and their left-hemispheric
electrodes counterparts (1):
ln(R)-ln(L)
(1)
in which an increase in alpha power reflects positive
emotions and approach motivation. Since positive
FAA values indicate larger relative right-hemispheric
power, an increase in FAA may reflect withdrawal
motivation and negative emotions. In alpha frontal
asymmetry, the power between left and right
hemisphere is normalized between 0 (perfect
symmetry) and 1 (maximal asymmetry).
The experiment follows the PCTA method
(Borsci et al., 2013). The PCTA requires that user and
evaluator do not verbally interact during the whole
duration of a task, but whenever users find a difficulty
or want to express an opinion about the quality of
their navigation, they are instructed to indicate this
with a signal (generally the sound of a desk bell). In
order to remember the problem at the end of the task,
the evaluator takes note of the actions the user was
performing before ringing the bell, while the
interaction is video recorded. The signal is designed
to serve as a memorandum for discussing problems
with the users at the end of the trial (Federici et al.,
2010; Borsci and Federici, 2009).
4.2 Materials and Apparatuses
The UTAssistant web-based platform was used to
evaluate the Italian Ministry of Economic
Development (MiSE) website
(http://www.sviluppoeconomico.gov.it) on a 15”
Lenovo ThinkPad T540p laptop computer, with a
1920 x 1200 screen resolution. The browser used to
access the UTAssistant platform is Google Chrome
(http://www.google.com/intl/en/chrome). During the
test, the computer was set to maximum brightness and
was constantly plugged in to power.
We use the iMotions platform
(https://imotions.com) to synchronize the data
collected by the Affectiva facial expression
recognition software (https://www.affectiva.com/)
through an integrated webcam and (EEG data
recorded by the EPOC+ 16-electrode headset
(https://www.emotiv.com/epoc/). A table bell was
placed next to the mouse. Affectiva is able to
calculate the emotional valence (positive, negative
and neutral) and the seven basic human emotions
(both ranging from 1 to 100). In this work, basic
emotions are computed with a threshold = 20 (only
facial expressions valued by algorithm as equal to or
higher than the 20% probability of a human emotion
is accepted). An external Logitech Webcam 250
camera was positioned on a tripod stand behind the
participant.
Two questionnaires are used to assess the
perceived usability of the system, i.e. the SUS and the
UMUX. The SUS is an easy and short tool for
measuring the usability of a system (Lewis and Sauro,
2017; Borsci et al., 2015). The SUS is a reliable 10-
item questionnaire based on a five-point Likert scale
ranging from 1 (strong disagreement) to 5 (strong
agreement). Compared to the SUS, the UMUX is
shorter and is based on the effectiveness, efficiency
and satisfaction dimensions of usability as defined by
the ISO 9241 (Finstad, 2010; Borsci et al., 2015;
Finstad, 2013). The UMUX is a reliable four-item
questionnaire based on a seven-point Likert scale.
4.3 Procedure
Participants are invited to perform the test in a
sufficiently bright and silent laboratory environment,
sitting in a comfortable chair placed at least 50
centimeters away from the computer screen. After
being informed about the general aim and methods of
the test, the subjects are asked to sign a consent form.
After the application of the EEG headset, the test
instructions are presented either by the UTAssistant
platform (experimental group) or directly by the
experimenter (control group).
Participants were divided into two groups: an (i)
experimental group and a (ii) control group. Subjects
were asked to browse the MiSE website following
four consecutive tasks, each one presented to users in
form of scenario. (i) The experimental group received
the description of the four scenarios automatically
presented in a written form one by one by the
UTAssistant platform. The participants assigned to
(ii) the control group were instructed about the
content of each task directly by the experimenter. The
maximum duration allowed for completion of each
task was five minutes, beyond which participants
were invited to proceed with the next task until the
test was completed. In order to follow the PCTA
method described in the section 4.1, participants are
asked not to verbalize any problems that may arise
during the test, but to report them at the end of the
trial. At the end of the session, participants are asked
to complete the two usability assessment
questionnaires in digital form, i.e. the SUS and the
UMUX scales.
4.4 Subjects
Thirty participants took part to the experiment, mean
age = 27.35 years old, equally divided by gender. Ten
subjects were assigned to the control group (mean age
= 20.67 years old), 20 subjects were assigned to the
two experimental groups (mean age = 26.76 years
old).
4.5 Results
The results of SUS questionnaire and UX bio-
behavioral data are described as follows.
4.5.1 Questionnaire results
The mean and standard deviations of the scores
obtained by both the experimental group (SUS mean
= 59.875; SUS S.D. = 23.500; UMUX mean = 54.85;
UMUX S.D. = 15.806) and the control group (SUS
mean = 63.75; SUS S.D. = 17.209; UMUX mean =
59.191; UMUX S.D. = 13.997) were evaluated.
The one-way ANOVA between groups analysis
found no significant difference between experimental
group and control group analysis for the SUS score
(F(1,28) = 0.213, p > 0.05) and for the UMUX scores
(F(1,28) = 0.213, p > 0.05).
4.5.2 UX bio-behavioral data
EEG. The mean frontal alpha asymmetry (FAA)
index of the whole session was calculated for both
experimental (mean FAA = 0.122) and control group
(mean FAA = 0.005). Figure 1 shows FAA index
mean values for each task. Correlations between time
and FAA (N = 204.987) show a highly significant
positive correlation (Pearson’s r = 0.035, p = 0.000).
Figure 1: Frontal alpha asymmetry of both experimental
group and control group.
The one-way ANOVA between groups shows a
significant difference in FAA values between control
group and experimental group (F(1,119) = 5.351, p =
0.022). No significant difference in frontal alpha
asymmetry values among the four tasks was found
(one way ANOVA F(3,117) = 0.643, p > 0.05)).
Figure 2: Overall mean affective valence time percentage
of both experimental group and control group.
Facial expression recognition. The overall affective
valence for both experimental and control groups was
calculated (Figure 2). No significant difference
between the experimental and control groups was
found for affective valence (one way ANOVA
between groups F(1,85) = 0.271, p > 0.05), whereas a
significant difference was found for valence type in
both groups (one way ANOVA F(2,84) = 1397.407,
p = 0.000).
The mean time (milliseconds) percent of each
basic emotion has been calculated for experimental
and control groups in the four tasks (Figures 3 and 4).
Figure 3: Mean time (ms) percent of emotions calculated
during tasks for the experimental group.
Figure 4: Mean time (ms) percent of emotions calculated
during tasks for the control group.
One-way ANOVA between groups shows no
difference between experimental and control group
related to basic emotions (F(1,54) = 0.042, p > 0.05)
and among tasks (F(3,52) = 0.342, p > 0.05) for both
groups. A significant difference was found among
emotional values (F(6,49) = 8.901, p = 0.000).
Bonferroni post hoc tests were used to determine
which basic emotions are significantly different from
each other. Bonferroni post hoc tests showed that the
only emotion significantly different at p < 0.01 was
surprise (mean = 1.355; SD = 0.186) compared to
anger, sadness, disgust and joy.
4.5.3 Comparisons between questionnaires
and bio-behavioral measures
Correlations between both SUS and UMUX scores
and frontal alpha asymmetry values were computed,
showing no significant correlation between both SUS
(Pearson’s r = 0.135, p > 0.05) and UMUX (Pearson’s
r = -.240, p > 0.05). No correlation between SUS and
UMUX scores was found as well for all participants
(Pearson’s r = 0.018, p > 0.05). Moreover,
correlations between frontal alpha asymmetry and
affective values showed no significant correlation
between both basic emotion (Pearson’s r = -0.124, p
> 0.05) and affective valence (Pearson’s r = 0.017, p
> 0.05).
4.6 Discussion
The study provides four main findings on
comparisons between the experimental group
(browsing the studied website through UTAssistant)
and the control group (conducting the test directly on
the studied website):
(1). There is a significant difference between the
frontal alpha asymmetry (FAA) measured on the
control group during the four usability tasks
compared to the experimental group. FAA showed no
increase during the tasks in the control group, but it
increased in the experimental group over the trial
time. The positive asymmetry values for the
experimental group are therefore significantly higher
than the control, meaning that there is a greater
positive activity relative to the frontal area of the right
hemisphere. As described in Section 4.1, increasing
right frontal activity is related to a withdrawal from
stimuli. In this work, the increase of FAA suggests a
decrease in motivation and a more negative approach
to the interaction with the system.
(2). There is no significant difference in affective
valence between the groups. Despite a possible
decrease in motivation, a difference in emotion and
emotional value (positive, negative, neutral) is not
confirmed on an emotional level during the entire test
for both groups. The significantly greater and
constant affective valence during the test was neutral
for both groups. From the post hoc analyses within
basic emotions, it emerges that the most significant
emotion for all subjects is surprise, which can most
likely be attributed to the content of the browsed
website or the nature of the task.
(3). What happens for the FAA among the groups
does not happen for the results of the self-reports.
There is no significant difference between the control
group and the experimental group for both the SUS
and the UMUX. This means that both groups are
homogeneous in their attribution of quality of
usability, although there is a decrease in motivation
for those who use UTAssistant. We should also note
that neither measure investigates the level of
motivation, unlike the FAA.
(4). The results of the self-reports do not correlate
with both the FAA values or with the affective values.
In this study, it seems that self-reports are not enough
to predict bio-behavioral measures that indicate
motivation levels during the interaction.
5 CONCLUSIONS
This work is about the user experience (UX)
assessment of a web-based usability assessment
platform called UTAssistant, primarily applied to the
field of public administration. The study has been
carried out under laboratory conditions using
traditional usability assessment methodologies (such
as the Partial Concurrent Thinking Aloud method and
usability questionnaires) and bio-behavioral
measures: electroencephalography and facial
expression recognition. Results showed that the
perception of usability of the system as measured by
self-reports is similar for both experimental and
control group. Also, the emotions behind the
interaction are mostly neutral in affective values for
both groups. However, electroencephalography
measurements of alpha activity seem to be more
sensitive to the duration effect for participants using
UTAssistant to complete tasks, with a decrease in
motivation that increases with the increase of test
duration. However, this is not followed by a feeling
of negative emotions that is observable through or
manifest in facial expressions. Future works will
focus on extending the investigation to other bio-
behavioral information provided by technologies
such as eye-tracking tools or electrodermal activity
sensors.
ACKNOWLEDGEMENTS
This work is supported and funded by Department of
Public Function, Italian Ministry for Simplification
and Public Administration (PA). The software
UTAssistant is designed and developed by G.D. and
R.L.
REFERENCES
Borsci, S. & Federici, S. 2009. The Partial Concurrent
Thinking Aloud: A New Usability Evaluation
Technique for Blind Users. In: Emiliani, P. L.,
Burzagli, L., Como, A., Gabbanini, F. & Salminen, A.-
L. (eds.) Assistive Technology from Adapted
Equipment to Inclusive Environments: AAATE 2009.
Amsterdam, NL: IOS Press.
Borsci, S., Federici, S., Bacci, S., Gnaldi, M. & Bartolucci,
F. 2015. Assessing User Satisfaction in the Era of User
Experience: Comparison of the SUS, UMUX and
UMUX-LITE as a Function of Product Experience.
International Journal of Human-Computer Interaction,
31, 484–495.
Borsci, S., Federici, S. & Mele, M. L. 2014. eGLU 1.0: un
protocollo per valutare la comunicazione web delle PA.
Diritto e Pratica Amministrativa. Milan, IT: Il Sole 24
Ore.
Borsci, S., Kurosu, M., Federici, S. & Mele, M. L. 2013.
Computer Systems Experiences of Users with and
without Disabilities: An Evaluation Guide for
Professionals, Boca Raton, FL, CRC Press.
Brockmyer, J. H., Fox, C. M., Curtiss, K. A., McBroom, E.,
Burkhart, K. M. & Pidruzny, J. N. 2009. The
development of the Game Engagement Questionnaire:
A measure of engagement in video game-playing.
Journal of Experimental Social Psychology, 45, 624–
634.
Brooke, J. 1996. SUS: A “quick and dirty” usability scale.
In: Jordan, P. W., Thomas, B., Weerdmeester, B. A. &
McClelland, I. L. (eds.) Usability Evaluation in
Industry. London, UK: Taylor & Francis.
Catarci, T., Amendola, M., Bertacchini, F., Bilotta, E.,
Bracalenti, M., Buono, P., Cocco, A., Costabile, M. F.,
Desolda, G., Di Nocera, F., Federici, S., Gaudino, G.,
Lanzilotti, R., Marrella, A., Laura Mele, M., Pantano,
P. S., Poggi, I. & Tarantino, L. Digital interaction:
where are we going? Proceedings of the 2018
International Conference on Advanced Visual
Interfaces: AVI 2018, May 29–Jun 01 2018 Castiglione
della Pescaia (GR), IT. ACM Digital Library, 1–5.
Christensen, J. C. & Estepp, J. R. 2013. Coadaptive Aiding
and Automation Enhance Operator Performance.
Human Factors, 55, 965–975.
Desolda, G., Gaudino, G., Lanzilotti, R., Federici, S. &
Cocco, A. UTAssistant: A Web Platform Supporting
Usability Testing in Italian Public Administrations. In:
Bottoni, P., Gena, C., Giachetti, A., Iacolina, S. A.,
Sorrentino, F. & Spano, L. D., eds. 12th Edition of
CHItaly: CHItaly 2017, Sep 18–20 2017 Cagliari, IT.
CEUR-WS.org, 138–142.
Dipartimento della Funzione Pubblica 2015. Il Protocollo
eGLU 2.1: Come realizzare test di usabilità semplificati
per i siti web e i servizi online delle PA. Rome, IT:
Formez PA.
Ekman, P., Friesen, W. V. & Hager, J. C. 2002. Facial
Action Coding System. The Manual, Salt Lake City,
UT, A Human Face.
Federici, S., Borsci, S. & Stamerra, G. 2010. Web usability
evaluation with screen reader users: Implementation of
the Partial Concurrent Thinking Aloud technique.
Cognitive Processing, 11, 263–272.
Federici, S., Mele, M. L., Lanzilotti, R., Desolda, G.,
Bracalenti, M., Meloni, F., Gaudino, G., Cocco, A. &
Amendola, M. UX Evaluation Design of UTAssistant:
A New Usability Testing Support Tool for Italian
Public Administrations. 20th International Conference
on Human-Computer Interaction, Jul 15–20 2018 Las
Vegas, NV. Springer International Publishing, 55–67.
Finstad, K. 2010. The Usability Metric for User Experience.
Interacting with Computers, 22, 323–327.
Finstad, K. 2013. Response to commentaries on ‘The
Usability Metric for User Experience’. Interacting with
Computers, 25, 327–330.
Ganglbauer, E., Schrammel, J., Deutsch, S. & Tscheligi, M.
Applying psychophysiological methods for measuring
user experience: possibilities, challenges and
feasibility. Workshop on user experience evaluation
methods in product development, 2009. Citeseer.
Goshvarpour, A., Abbasi, A. & Goshvarpour, A. 2017.
Fusion of heart rate variability and pulse rate variability
for emotion recognition using lagged poincare plots.
Australasian Physical & Engineering Sciences in
Medicine, 40, 617–629.
Gruzelier, J. H. 2014. EEG-neurofeedback for optimising
performance. I: A review of cognitive and affective
outcome in healthy participants. Neuroscience &
Biobehavioral Reviews, 44, 124–141.
Isbister, K., Höök, K., Sharp, M. & Laaksolahti, J. The
sensual evaluation instrument: developing an affective
evaluation tool. Conference on Human Factors in
Computing Systems: SIGCHI ’06, Apr 22–27, 2006
2006 Montréal, CA. ACM, 1163–1172.
Jimenez-Molina, A., Retamal, C. & Lira, H. 2018. Using
Psychophysiological Sensors to Assess Mental
Workload During Web Browsing. Sensors, 18, 458.
Jokela, T. Determining usability requirements into a call-
for-tenders: a case study on the development of a
healthcare system. 6th Nordic Conference on Human-
Computer Interaction: Extending Boundaries, Oct 16-
20 2010 Reykjavik, IS. ACM, 256–265.
Jun, J., Ou, L.-C., Oicherman, B., Wei, S.-T., Luo, M. R.,
Nachilieli, H. & Staelin, C. 2010. Psychophysical and
Psychophysiological Measurement of Image Emotion.
Color and Imaging Conference, 2010, 121–127.
Lewis, J. R. & Sauro, J. 2017. Revisiting the Factor
Structure of the System Usability Scale. Journal of
Usability Studies, 12, 183–192.
Mele, M. L. & Federici, S. 2012. A psychotechnological
review on eye-tracking systems: Towards user
experience. Disability and Rehabilitation: Assistive
Technology, 7, 261–281.
Mendoza-Denton, N., Eisenhauer, S., Wilson, W. & Flores,
C. 2017. Gender, electrodermal activity, and
videogames: Adding a psychophysiological dimension
to sociolinguistic methods. Journal of Sociolinguistics,
21, 547–575.
Muñoz Cardona, J. E., Cameirão, M. S., Paulino, T.,
Bermudez i Badia, S. & Rubio, E. Modulation of
Physiological Responses and Activity Levels during
Exergame Experiences. 8th International Conference
on Games and Virtual Worlds for Serious Applications:
VS-GAMES ’16, Sep 7–9, 2016 2016 Barcelona, ES.
1–8.
Nacke, L. E., Stellmach, S. & Lindley, C. A. 2011.
Electroencephalographic Assessment of Player
Experience:A Pilot Study in Affective Ludology.
Simulation & Gaming, 42, 632–655.
Pallavicini, F., Cipresso, P., Raspelli, S., Grassi, A., Serino,
S., Vigna, C., Triberti, S., Villamira, M., Gaggioli, A.
& Riva, G. 2013. Is virtual reality always an effective
stressors for exposure treatments? some insights from a
controlled trial. BMC Psychiatry, 13, 52.
Poore, J. C., Webb, A. K., Cunha, M. G., Mariano, L. J.,
Chappell, D. T., Coskren, M. R. & Schwartz, J. L.
2017. Operationalizing Engagement with Multimedia
as User Coherence with Context. IEEE Transactions on
Affective Computing, 8, 95–107.
Rodriguez-Guerrero, C., Knaepen, K., Fraile-Marinero, J.
C., Perez-Turiel, J., Gonzalez-de-Garibay, V. &
Lefeber, D. 2017. Improving Challenge/Skill Ratio in a
Multimodal Interface by Simultaneously Adapting
Game Difficulty and Haptic Assistance through
Psychophysiological and Performance Feedback.
Frontiers in Neuroscience, 11.
Vermeeren, A. P. O. S., Law, E. L.-C., Roto, V., Obrist, M.,
Hoonhout, J. & Väänänen-Vainio-Mattila, K. User
experience evaluation methods: current state and
development needs. 6th Nordic Conference on Human-
Computer Interaction Extending Boundaries:
NordiCHI ’10, Oct 16–20, 2010 2010 Reykjavik, IC.
ACM, 521–530.
Vourvopoulos, A. & Bermúdez i Badia, S. 2016. Motor
priming in virtual reality can augment motor-imagery
training efficacy in restorative brain-computer
interaction: a within-subject analysis. Journal of
NeuroEngineering and Rehabilitation, 13, 69.
Yan, S., Ding, G., Li, H., Sun, N., Wu, Y., Guan, Z., Zhang,
L. & Huang, T. 2016. Enhancing Audience
Engagement in Performing Arts Through an Adaptive
Virtual Environment with a Brain-Computer Interface.
21st International Conference on Intelligent User
Interfaces: IUI '16. Sonoma, CA: ACM.
Zhang, L., Sun, S., Zhang, K., Xing, B. & Xu, W. 2018.
Using psychophysiological measures to evaluate the
multisensory and emotional dynamics of the tea
experience. Journal of Food Measurement and
Characterization, 12, 1399–1407.
... A large part of our research work deals with e-government websites, specifically websites of small municipalities in Italy, which are designed and managed by web editorial staff by using Content Management Systems [6]. These websites offer general information (e.g., addresses of public offices, their work hours and contacts), and provide online services to citizens, with the potential of drastically reducing costs and waiting times. ...
... The ten items have opposite polarity. For odd items (1,3,5,7,9), the most positive response is 5; for even items (2,4,6,8,10), the most positive response is 1. For each returned questionnaire, SUS provides a single score, which ranges from 0 to 100, calculated as it follows: ...
... For even items (2,4,6,8,10): subtract the user response from 5. In this way, all values scale from 0 to 4 (with 4 being the most positive response). ...
Article
Full-text available
Daily experiences in working with various types of computer systems show that, despite the offered functionalities, users have many difficulties, which affect their overall User eXperience (UX). The UX focus is on aesthetics, emotions and social involvement, but usability has a great influence on UX. Usability evaluation is acknowledged as a fundamental activity of the entire development process in software practices. Research in Human-Computer Interaction has proposed methods and tools to support usability evaluation. However, when performing an evaluation study, novice evaluators still have difficulties to identify usability problems and to understand their causes: they would need easier to use and possibly automated tools. This article describes four visualization techniques whose aim is to support the work of evaluators when performing usability tests to evaluate websites. Specifically, they help detect “usability smells”, i.e. hints on web pages that might present usability problems, by visualizing the paths followed by the test participants when navigating in a website to perform a test task. A user study with 15 participants compared the four techniques and revealed that the proposed visualizations have the potential to be valuable tools for novice usability evaluators. These first results should push researchers towards the development of further tools that are capable to support the detection of other types of UX smells in the evaluation of computer systems and that can be translated into common industry practices.
... This work describes the heuristic evaluation of eGLU-box, a new remote semiautomatic usability assessment tool that overcomes each of the aforementioned limits. eGLU-box is a re-engineered version of a previous platform called UTAssistant [5][6][7][8], a web-based usability assessment tool developed to provide the Italian Public Administration with an online tool to conduct remote user studies. Both UTAssistant and its renewed version eGLU-box are designed according to usability guidelines provided by GLU, a group working on usability founded by the Department of Public Function, Ministry for Simplification and Public Administration in 2010. ...
... The re-engineering process of UTAssistant was made possible by previous studies by Federici and colleagues, who evaluated user experience (UX) expert users of public administration (PA) websites [6]. In laboratory conditions, they used psychophysiological techniques [5] to measure the underlying reactions of participants through the recognition of facial expressions and electroencephalography (EEG). This work describes the usability evaluation of the renewed platform by a heuristic evaluation with both UX experts and PA practitioners. ...
... This is why in 2017 a web platform called UTAssistant was developed in line with the last Italian PA usability protocol, eGLU 2.1 [9]. Thanks to a UX evaluation of UTAssistant with expert users [6] and in laboratory conditions with two biobehavioral implicit measures [5], a re-engineering process of UTAssistant led to the development of the current version of the platform, eGLU-box. It is divided into two modules, one (the "tester module") for the practitioner who has to create, administer, and analyze the test, and another (the "end-user module") for end-users for whom the test is intended. ...
Chapter
This paper illustrates the heuristic evaluation of a web-based tool for usability testing for Public Administrations called eGLU-box. eGLU-box is an online platform aiming at supporting practitioners in the process of designing usability tests, analyzing data, and helping step-by-step participants to complete assessment tasks. Web users of Public Administrations can report their perceived quality of experience by completing a library of questionnaires shown to them by eGLU-box at the end of the test. This work is part of a multi-step user experience (UX) evaluation methodology to assess the platform. The UX evaluation methodology of eGLU-box uses standard and bio-behavioural evaluation methods. This work shows the results of the heuristic evaluation of eGLU-box involving five human factors experts and 20 practitioners working in Italian Public Administrations. Findings show that most of the problems are rated as minor problems and related to Nielsen’s heuristic, “visibility of the system.” Only 9% of problems are rated as major problems. These major problems are related to the “problematic match between system and the real world” heuristic. Evaluators provided indications for improvements that will be applied for the next version of the platform.
... It is an online tool that allows PA webmasters to create usability tests for a particular website and to invite participants to carry them out. eGLU-box is a re-engineered version of a previous platform called UTAssistant [4][5][6], a web-based usability assessment tool that was developed to provide the PA with an online tool to conduct remote user studies. Both UTAssistant and its newer version, eGLU-box, were designed according to usability guidelines provided by GLU, a group working on usability that was founded by the Department of Public Function, Ministry for Simplification and Public Administration in 2010. ...
... eGLU-box was designed for the conduction of usability tests both remotely and in a laboratory. In this case, eGLU-box data can be combined with software that captures bio-behavioral data (such as electroencephalographic, skin conductance, heart rate, eye movement, and facial expression data) [5]. ...
Chapter
Smartphones and tablets now offer consumers unique advantages such as portability and accessibility. Developers are also working with a mobile-first approach, and are prioritizing mobile applications over desktop versions. This study introduces eGLU-box Mobile, an application for performing a drive usability test directly from a smartphone. An experimental study was conducted in which the participants were divided into two groups: an experimental group, which used the new mobile application from a smartphone, and a control group, which used the desktop application from a computer. The participants’ behavior was assessed using explicit (self-report questionnaires) and implicit measures (eye movement data). The results were encouraging, and showed that both the mobile and desktop versions of eGLU-box enabled participants to test the usability with a similar level of UX, despite some minimal (although significant) differences in terms of satisfaction of use.
... Technology advances in recent years, in particular the advent of HTML5 (introduced in 2014) and JavaScript ES6 (introduced in 2015), have enabled previously unexplored scenarios in remote usability testing. Currently, it is possible to develop more powerful tools for remote usability testing of websites that, for example, work across browsers, are OSindependent, do not require access to the website source code, and permit screen recording and user-interaction tracking as well as the use of peripheral devices such as webcams and microphones to capture the face and speech of the participants during tests [25][26][27][28][29]). The functionality provided by these recent tools permits collecting much more quantitative (e.g., questionnaire answers, task success/time) and qualitative data (e.g., audio/video recording), which may reveal further usability issues. ...
Article
Full-text available
Remote usability testing is performed by evaluators who are in different physical locations from the participants (synchronous remote testing) and possibly operating at different times (asynchronous remote testing). The tools developed in recent years to support remote tests exploit web technology based on HTML5 and JavaScript ES6 and thus enable previously unexplored scenarios. However, studies providing evidence on the benefits or drawbacks of utilizing recent web-based tools have not yet been reported in the literature. This article sheds some light on the impact of such tools on asynchronous remote usability testing of websites by reporting an experimental study with 100 participants and 15 evaluators to compare real-time laboratory tests with asynchronous remote tests. The study investigates: 1) how the metrics results of asynchronous remote usability tests performed through a web-based tool differ from those of usability tests conducted in real-time laboratory settings; and 2) how the experience of participants differs in the two types of tests. The lessons learned in the study are instrumental in informing the design of future tools. Some results of particular interest indicate that the web technology used by the tool for asynchronous remote testing affects task execution times and participants’ satisfaction. Another indication is that slow internet connections must be managed in asynchronous remote testing; slow connections introduce delays when transferring large amounts of collected data, which, together with the lack of human support, make participants of asynchronous remote tests more prone to feel negative emotions.
... UX includes the dimensions of usability [2] and concurrently attempts to enlarge the assessment's factors with a focus on cognitive, aesthetics, and qualitative aspects of interaction measured throughout time [3]. Accordingly, to fully model the perceived experience of a user, practitioners should include a set of repeated objective and subjective measures in their evaluation protocols to enable satisfaction and benefit analysis as a "subjective sum of the interactive experience" [4]. Several standardized tools have been developed to measure satisfaction, realization of benefit and perceived usability of user with and without disabilities [5][6][7][8][9][10][11]. ...
Chapter
Full-text available
To fully model the perceived experience of a user, practitioners should include a set of repeated objective and subjective measures in their evaluation protocols to enable satisfaction and benefit analysis as a “subjective sum of the interactive experience.” It is also well known that if the UX of a product is assessed at the end of the design process, product changes are much more expensive than if the same evaluation were conducted throughout the development process. In this study, we aim to present how these concepts of UX and UCD inform the process of selecting and assigning assistive technologies (ATs) for people with disabilities (PWD) according to the Matching Person and Technology (MPT) model and assessments. To make technology the solution to the PWD’s needs, the MPT was developed as an international measure evidence-based tool to assess the best match between person and technology, where the user remains the main actor in all the selection, adaptation, and assignment process (user-driven model). The MPT model and tools assume that the characteristics of the person, environment, and technology should be considered as interacting when selecting the most appropriate AT for a particular person’s use. It has demonstrated good qualitative and quantitative psychometric properties for measuring UX, realization of benefit and satisfaction and, therefore, it is a useful resource to help prevent the needs and preferences of the users from being met and can reduce early technology abandonment and the consequent waste of money and energy.
... In another article [48], the relationship of GSR characteristics with performance metrics was analyzed, identifying that the tasks with a lower rate of completion have a non-significant tendency to cause higher GSR values and a significant correlation between attractiveness, efficiency, dependability, and novelty with GSR data. On the other hand, the usability of a web application was evaluated looking for correlations between subjective questionnaires, EEG, and emotions through facial expressions [49], concluding that EEG measurements are necessary since it was observed that the decrease in motivation was not reflected in the self-reports, but in the increase of brain activity. Arndt et al. [50] analyzed the perception of quality in video fragments using EEG and eye-tracking data, including pupillometry. ...
Article
Full-text available
Traditional evaluation of user experience is subjective by nature, for what is sought is to use data from physiological and behavioral sensors to interpret the relationship that the user’s cognitive states have with the elements of a graphical interface and interaction mechanisms. This study presents the systematic review that was developed to determine the cognitive states that are being investigated in the context of Quality of Experience (QoE)/User Experience (UX) evaluation, as well as the signals and characteristics obtained, machine learning models used, evaluation architectures proposed, and the results achieved. Twenty-nine papers published in 2014–2019 were selected from eight online sources of information, of which 24% were related to the classification of cognitive states, 17% described evaluation architectures, and 41% presented correlations between different signals, cognitive states, and QoE/UX metrics, among others. The amount of identified studies was low in comparison with cognitive state research in other contexts, such as driving or other critical activities; however, this provides a starting point to analyze and interpret states such as mental workload, confusion, and mental stress from various human signals and propose more robust QoE/UX evaluation architectures.
... Se ha utilizado un enfoque multimétodo y holístico, considerando datos de diversa naturaleza, tales como; valores pragmáticos, valores de percepción y valores biométricos. Existen trabajos relacionados como el de Federici, et al., [18] donde emplean métodos implícitos para la evaluación de la UX basados en el análisis del comportamiento de los usuarios independientemente de su conciencia de percepciones y su control consciente. A diferencia, este trabajo de investigación, integra valores pragmáticos como el tiempo de consecución de tareas y tasa de errores. ...
Article
Full-text available
La industria 4.0 y una vida laboral en continua transmutación, plantean preguntas acerca de cuáles serán las competencias que requerirán las personas para la correcta apropiación tecnológica y la mejora del desempeño. El objetivo de esta investigación es analizar la implicación que tienen en la apropiación de una interfaz de inteligencia competitiva diversas características de diseño de interfaces que refuerzan la consecución de tareas y la UX mediante una nueva aproximación multimétodo. En esta línea, existen diversos marcos metodológicos con enfoque multimétodo, tales como el marco de Usabilidad del Sistema (SU) [1], el modelo Componentes de la Experiencia de Usuario (CUE) [2], el marco “Factory2Fit work well being” [3] o la herramienta Experience Capturer (XC) [4]. A su vez, este trabajo de investigación analiza el impacto de la familiarización por el orden de uso en la apropiación del software analizado. Los resultados recogidos indican que las interfaces diseñadas teniendo en cuenta la consecución de objetivos, orientadas a cubrir las necesidades de los usuarios y que siguen unos patrones establecidos de diseño ofreciendo una solución precisa y consistente, consiguen mejores resultados durante la ejecución de tareas. En esta línea, en términos de aceptación tecnológica, se ha observado que un diseño que transmite confianza, facilita la navegación, ofrece control y feedback sobre las acciones, provee de una estética clara, precisa y coherente, y que además es tolerante a errores, ayuda a que la tecnología sea apropiada de forma más rápida. Por otro lado, se ha observado que el factor de orden tiene consecuencias significativas durante la ejecución de las tareas y la valoración de los usuarios, por lo que se considera un factor clave en el campo de conocimiento de la UX en relación al análisis del proceso de aprendizaje y apropiación de soluciones digitales.
... In turn, (Mandryk et al. 2006), (Dirican andGöktürk 2011), and(Certický et al. 2019) listed the following psychophysiological measures useful for the enhancement of user experience study: electroencephalography (EEG); electro-dermal activity (EDA)/galvanic skin response (GSR); cardiovascular measures: heart rate (HR), heart rate variability (HRV), blood pressure (BP), blood volume pulse (BVP); electromyogram (EMG); eye movements: fixations, saccades, gaze and blinks; pupil diameter and pupil dilation; respiration measures: respiration amplitude (RespAmp) and respiration rate (RespRate). (Federici et al. 2019) presented a study on the user experience (UX) assessment of a web-based platform designed for the semi-automatic usability evaluation of other websites. They employed two bio-behavioral measurement techniques, namely facial expression recognition and electroencephalography. (Maia and Furtado 2019) in their study, investigated and proposed during HCI experiments the employment of psychophysiological measures, taking into consideration emotions of the user and its connection with possible usability issues. ...
Article
Full-text available
This paper addresses usability problems of data entry techniques and patterns in web systems and web sites. In order to carry out a comparative analysis of the solutions a dedicated web application was developed as an experimental tool for performing usability testing. The application was designed in the form of a test divided into scenarios and exercises on filling up selected models. Furthermore, the application recorded usability metrics, questionnaires, open and closed questions, as well as users’ personal data. 40 participants, who took part remotely in the experiment, were split into two distinct groups and completed two separate tests that differed in input data and tested solutions. Some measures referring to the effectiveness, efficiency, and user satisfaction were collected. The results were thoroughly analyzed and the recommendations for the usage of particular models were formulated.
Chapter
In the competitive realm of consumer-driven markets, companies strive to meet rising expectations by creating digital products that not only meet functional needs but also evoke authentic emotional responses. In this article, we integrate cognitive neuroscience methods with user experience research, citing them as powerful tools for uncovering users’ subconscious responses and unconscious preferences. Techniques such as eye tracking, GSR, EEG and facial coding combined with traditional UX research methods provide a comprehensive understanding of the user experience. Despite implementation challenges, a review of 64 experiments conducted between 2017 and 2022 highlights the growing use of these methods. A generalized procedure based on these experiments is proposed for designing and conducting UX research using cognitive neuroscience methods. By sharing best practices, this article serves as a valuable guide for researchers exploring the evolving intersection of cognitive neuroscience and UX design, facilitating the creation of impactful experiments in this emerging field.
Chapter
This paper shows a chatbot solution for eGLU-box Pro, a usability testing platform for Italian Public Administration (PA). eGLU-box Pro is a web-based tool designed to help PA practitioners in creating remote usability tests and analyzing participants’ answers and interaction data after they complete the usability tasks. The impact of the chatbot solution on users’ experience was assessed by bio-behavioral evaluation methods such as eye tracking, electroencephalography, and facial expression recognition. This work describes the platform and its integrated chatbot solution and shows the results of a preliminary laboratory study involving 20 end-users. The study is part of an ongoing design and development project based on a user-centered approach.
Article
Full-text available
In recent years, the user experience has gradually become the main focus in the design of consumer products, rather than the products’ features and functions. Therefore, designers should consider the user experience to be of paramount importance in the development of a successful product. In addition, user experience is not static, but rather it changes over the different stages of usage due to the variations in consumers’ sensory modalities and emotional responses. Therefore, this study aims to use psychophysiological measures to explore how the interaction between the user and the product at each different stage influences sensory modalities and emotional responses to products. The experiment was conducted in five stages according to the different levels of interaction for two tea products. Subjective data about sensory modalities and emotions related to the products were collected. In addition, the research gathered objective data to examine the feasibility of relating physiological measures to subjective experiences. The results showed that the tea experience tends to be dynamic over the different stages of usage, demonstrating important variations in the different sensory modalities and emotional feedback. In addition, physiological signals such as galvanic skin response, have been shown to be of potential value for evaluation of the user’s emotional reaction. Our findings could be important not only for a theoretical understanding of sensory and emotional feedback in a product experience but also as a measure of user experience using psychophysiological methods.
Conference Paper
Full-text available
Even if the benefits of the usability testing are remarkable, it is scarcely adopted in the software development process. To foster its adoption, this paper presents a Web platform, UTAssistant, that supports people, also without skills in Human-Computer Interaction (HCI), in evaluating Web site usability.
Article
Full-text available
In 2009, we published a paper in which we showed how three independent sources of data indicated that, rather than being a unidimensional measure of perceived usability, the System Usability Scale apparently had two factors: Usability (all items except 4 and 10) and Learnability (Items 4 and 10). In that paper, we called for other researchers to report attempts to replicate that finding. The published research since 2009 has consistently failed to replicate that factor structure. In this paper, we report an analysis of over 9,000 completed SUS questionnaires that shows that the SUS is indeed bidimensional, but not in any interesting or useful way. A comparison of the fit of three confirmatory factor analyses showed that a model in which the SUS's positive-tone (odd-numbered) and negative-tone (even-numbered) were aligned with two factors had a better fit than a unidimensional model (all items on one factor) or the Usability/Learnability model we published in 2009. Because a distinction based on item tone is of little practical or theoretical interest, we recommend that user experience practitioners and researchers treat the SUS as a unidimensional measure of perceived usability, and no longer routinely compute Usability and Learnability subscales.
Article
Full-text available
Knowledge of the mental workload induced by a Web page is essential for improving users' browsing experience. However, continuously assessing the mental workload during a browsing task is challenging. To address this issue, this paper leverages the correlation between stimuli and physiological responses, which are measured with high-frequency, non-invasive psychophysiological sensors during very short span windows. An experiment was conducted to identify levels of mental workload through the analysis of pupil dilation measured by an eye-tracking sensor. In addition, a method was developed to classify mental workload by appropriately combining different signals (electrodermal activity (EDA), electrocardiogram, photoplethysmo-graphy (PPG), electroencephalogram (EEG), temperature and pupil dilation) obtained with non-invasive psychophysiological sensors. The results show that the Web browsing task involves four levels of mental workload. Also, by combining all the sensors, the efficiency of the classification reaches 93.7%.
Article
Full-text available
In order to harmonize robotic devices with human beings, the robots should be able to perceive important psychosomatic impact triggered by emotional states such as frustration or boredom. This paper presents a new type of biocooperative control architecture, which acts toward improving the challenge/skill relation perceived by the user when interacting with a robotic multimodal interface in a cooperative scenario. In the first part of the paper, open-loop experiments revealed which physiological signals were optimal for inclusion in the feedback loop. These were heart rate, skin conductance level, and skin conductance response frequency. In the second part of the paper, the proposed controller, consisting of a biocooperative architecture with two degrees of freedom, simultaneously modulating game difficulty and haptic assistance through performance and psychophysiological feedback, is presented. With this setup, the perceived challenge can be modulated by means of the game difficulty and the perceived skill by means of the haptic assistance. A new metric (FlowIndex) is proposed to numerically quantify and visualize the challenge/skill relation. The results are contrasted with comparable previously published work and show that the new method afforded a higher FlowIndex (i.e., a superior challenge/skill relation) and an improved balance between augmented performance and user satisfaction (higher level of valence, i.e., a more enjoyable and satisfactory experience).
Article
Full-text available
Background The use of Brain–Computer Interface (BCI) technology in neurorehabilitation provides new strategies to overcome stroke-related motor limitations. Recent studies demonstrated the brain's capacity for functional and structural plasticity through BCI. However, it is not fully clear how we can take full advantage of the neurobiological mechanisms underlying recovery and how to maximize restoration through BCI. In this study we investigate the role of multimodal virtual reality (VR) simulations and motor priming (MP) in an upper limb motor-imagery BCI task in order to maximize the engagement of sensory-motor networks in a broad range of patients who can benefit from virtual rehabilitation training. Methods In order to investigate how different BCI paradigms impact brain activation, we designed 3 experimental conditions in a within-subject design, including an immersive Multimodal Virtual Reality with Motor Priming (VRMP) condition where users had to perform motor-execution before BCI training, an immersive Multimodal VR condition, and a control condition with standard 2D feedback. Further, these were also compared to overt motor-execution. Finally, a set of questionnaires were used to gather subjective data on Workload, Kinesthetic Imagery and Presence. Results Our findings show increased capacity to modulate and enhance brain activity patterns in all extracted EEG rhythms matching more closely those present during motor-execution and also a strong relationship between electrophysiological data and subjective experience. Conclusions Our data suggest that both VR and particularly MP can enhance the activation of brain patterns present during overt motor-execution. Further, we show changes in the interhemispheric EEG balance, which might play an important role in the promotion of neural activation and neuroplastic changes in stroke patients in a motor-imagery neurofeedback paradigm. In addition, electrophysiological correlates of psychophysiological responses provide us with valuable information about the motor and affective state of the user that has the potential to be used to predict MI-BCI training outcome based on user’s profile. Finally, we propose a BCI paradigm in VR, which gives the possibility of motor priming for patients with low level of motor control.
Article
Full-text available
Traditional approaches for assessing user engagement within multimedia environments rely on methods that are removed from the human computer interaction itself, such as surveys, interviews and baselined physiology. We propose a context coherence approach that operationalizes engagement as the amount of independent user variation that covaries in time with multimedia contextual events during unscripted interactions. This can address questions about the features of multimedia users are most engaged and how engaged users are without the need for prescribed interactions or baselining. We assessed the validity of this approach in a psychophysiological study. 40 participants played interactive video games. Intake and post-stimulus questionnaires collected subjective engagement reports that provided convergent and divergent validity criteria to evaluate our approach. Estimates of coherence between physiological variation and in-game contextual events predicted subjective engagement and added information beyond physiological metrics computed from baselines taken outside of the multimedia context. Our coherence metric accounted for task-dependent engagement, independent of predispositions; this was not true of a baselined physiological approach that was used for comparison. Our findings show compelling evidence that a context-sensitive approach to measuring engagement overcomes shortcomings of traditional methods by making best use of contextual information sampled from multimedia in time-series analyses.
Article
User experience research on the affective dimensions of videogame players’ interactions has looked to psychophysiological measures as proxies for emotional states. We add to the growing body of literature on this topic and offer our research as a novel application to the methodological toolbox of sociolinguistics. We present a case study of an interaction occurring in a mixed‐gender group (3m/1f) playing a video game that has a potentially sexualized component in gameplay, using a Nintendo Wii‐mote joystick as a mimetic penis in the game Mario Party 8 . Through a detailed analysis of (1) gameplay interaction, including talk, laughter and gesture, (2) participants’ electrodermal activity ( EDA ), and (3) a post‐game debriefing, we track the varied engagements of the players in‐game and with each other, showing how the design of the game itself and the entanglement of hardware and software design with gendered gameplay serve to marginalize and exclude the female participant.
Conference Paper
Audience engagement is an important indicator of the quality of the performing arts but hard to measure. Psychophysiological measurements are promising research methods for perceiving and understanding audience's responses in real-time. Currently, such research are conducted by collecting biometric data from audience when they are watching a performance. In this paper, we draw on techniques from brain-computer interfaces (BCI) and knowledge from quality of performing arts to develop a system that monitor audience engagement in real time using electroencephalography (EEG) measurement and seek to improve it by triggering the adaptive performing cues when the engagement level decreased. We simulated the immersive theatre performances to provide audience a high-fidelity visual-audio experience. An experimental evaluation is conducted with 48 participants during two different performance studies. The results showed that our system could successfully detect the decreases in audience engagement and the performing cues had positive effects on regain audience engagement. Our research offers the guidelines for designing theatre performances from the audience's perception.