Chapter

Heuristic Evaluation of eGLU-Box: A Semi-automatic Usability Evaluation Tool for Public Administrations

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

This paper illustrates the heuristic evaluation of a web-based tool for usability testing for Public Administrations called eGLU-box. eGLU-box is an online platform aiming at supporting practitioners in the process of designing usability tests, analyzing data, and helping step-by-step participants to complete assessment tasks. Web users of Public Administrations can report their perceived quality of experience by completing a library of questionnaires shown to them by eGLU-box at the end of the test. This work is part of a multi-step user experience (UX) evaluation methodology to assess the platform. The UX evaluation methodology of eGLU-box uses standard and bio-behavioural evaluation methods. This work shows the results of the heuristic evaluation of eGLU-box involving five human factors experts and 20 practitioners working in Italian Public Administrations. Findings show that most of the problems are rated as minor problems and related to Nielsen’s heuristic, “visibility of the system.” Only 9% of problems are rated as major problems. These major problems are related to the “problematic match between system and the real world” heuristic. Evaluators provided indications for improvements that will be applied for the next version of the platform.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Due to COVID-19 restrictions, the study had to be performed remotely. The tool eGLU-Box PA [11] has been used to perform the remote study. The participants have been recruited among the students at the University of Bari and at the University of Trento on a voluntary basis. ...
Chapter
In this paper, we present the design and the evaluation of an authoring tool for End-User Development, which supports the definition of Trigger-Actions rules that combines events and states in the triggers. The possibility of using either states or events in triggers has already been discussed in the literature. However, it is recognized that the state/event distinction is difficult to manage for users. In this paper, we propose an authoring tool that provides explicit support for managing this distinction. We compare it with a state-of-the-art authoring tool that implements the classical event-event paradigm.
... Given the COVID-19 pandemic, the study was performed remotely. To facilitate the remote execution, a tool for remote user testing, eGLU-Box PA, was used [17]. Three evaluators (HCI experts) were involved. ...
Chapter
Full-text available
Given the spread of the Internet of Things (IoT) technology, in several contexts there is a growing need for laypeople to configure their smart devices. Task Automation Systems (TASs) have emerged as tools to simplify the definition of rules for personalizing the behavior of such devices. However, one aspect often neglected by current TASs, which is instead typical of IoT technologies, relates to the security and privacy threats exposed by the “connected” devices. This paper tries to address this problem and illustrates a user-centered design that eventually led to identify a visual paradigm that facilitates the end users in understanding and controlling security and privacy threats.
... The study was organized according to a within-subject design, with visualization as an independent variable and as within-subject factors the two visualizations. Each visualization showed data gathered in a study conducted with 15 employees of an Italian Administration, who used a Web platform for onsite and remote usability tests [4,5]. With each system, the participants performed the following tasks: 1. Identify the page that caused most problems; 2. Identify the path(s) that led to the task's failure; 3. Identify the path that led to the task success following the ideal path, 4. Identify the paths that led to the task success following alternative paths; 5. Identify backward paths, if possible. ...
Chapter
Full-text available
This paper reports on an on-going work that investigates the use of visualization techniques to help evaluators discovering usability problems by visualizing data collected during usability tests of web sites. Two visualization techniques are described and some results of the evaluation study that compared the two techniques are provided.
Chapter
Due to the importance of usability, multiple usability evaluation methods have been proposed that help Human-Computer Interaction (HCI) specialists determine whether the interfaces of a software product are usable, easy to use, understandable, and attractive. Among these methods, the heuristic evaluation proposed by Jakob Nielsen is the one that stands out. Although Nielsen offers general guidelines for the execution of heuristic evaluations, very few authors of different studies present a formal agreement or process on how the evaluations should be carried out, which leads us to the problem of the absence of a comparative analysis that allows determining the most appropriate formal evaluation process to carry out heuristic inspections. To complement, some proposals found in the literature compare the results of the execution of heuristic evaluations, the definition of new heuristics, qualification forms, and the formalization of the complete process in 5 phases. Although these proposals contribute to the formalization of the heuristic evaluation process, the literature review has not provided a comparative analysis that allows determining which is the most appropriate, which could cause usability evaluators to interpret and carry out their own procedure, which would lead to inaccuracies in the results and increase the probability of improperly executing the inspection. The purpose of this study was to elaborate a comparative table of the formal processes; this allowed the grouping of the various studies found in the literature, to select a process, and thus, and to model by a BPMN tool the whole process. Finally, this process modeled was validated by expert judgement of HCI specialists.
Chapter
This paper shows a chatbot solution for eGLU-box Pro, a usability testing platform for Italian Public Administration (PA). eGLU-box Pro is a web-based tool designed to help PA practitioners in creating remote usability tests and analyzing participants’ answers and interaction data after they complete the usability tasks. The impact of the chatbot solution on users’ experience was assessed by bio-behavioral evaluation methods such as eye tracking, electroencephalography, and facial expression recognition. This work describes the platform and its integrated chatbot solution and shows the results of a preliminary laboratory study involving 20 end-users. The study is part of an ongoing design and development project based on a user-centered approach.
Chapter
Heuristic evaluation belongs to the usability inspection methods and is considered one of the most popular methods since it allows to discover over 75% of the total usability problems involving only 3 to 5 usability experts, in comparison with user tests. However, certain problems and challenges have been identified at the time of their execution. In this study we present the results of conducting a Systematic Literature Review (SLR) to identify case studies, challenges, problems, and opportunities on the execution of heuristic evaluations in the context of a research for the automation and formalization of the process. For this SLR, we have employed the protocol proposed by Kitchenham and Charters. The research was carried out on September 7 of 2020 and retrieved a total of 167 studies of which 37 were selected for this review. The results show that the main challenges are related to the low suitability of the chosen set of heuristics and the low expertise of usability evaluators. Additionally, we have identified that exist very few software solutions that support and automate the process. Finally, we have found that there were many protocols to follow when applying a heuristic evaluation, like the definition of new usability heuristics for a given case of study. According to the results obtained, we can conclude that it is necessary to develop and validate a tool based on a formal protocol that supports and automate the heuristic evaluation process, that gives solutions to the challenges and opportunities identified in this research.
Conference Paper
Full-text available
This work shows the user experience (UX) assessment of a web-based platform for the semi-automatic usability evaluation of websites, UTAssistant, which is primarily addressed to workers in public administration (PA). The study is part (Phase 1) of a multiple assessment methodology which consists of four phases in total: (1) UX in laboratory conditions; (2) Usability evaluation in remote online conditions; (3) Usability evaluation in workplace conditions; and (4) Heuristic evaluation. In Phase 1, a UX study in laboratory conditions was carried out. Participants' UX of a PA website navigation through UTAssistant was evaluated by both traditional self-report usability assessment tools (SUS and UMUX) and bio-behavioral measurement techniques (facial expression recognition and electroencephalography). Results showed that using the UTAssistant usability assessment tool for webpages did not affect users' perceived usability in terms of self-reports and affective states, which were mostly neutral for all the assessment session. However, frontal alpha asymmetry EEG's scores showed a higher sensitivity of UTAssistant users to the duration of the trial, with a decrease in motivation displayed as the trial ensued. However, this result did not seem to affect emotional experience.
Article
Full-text available
The present article examined the effects of using different extra-laboratorial testing procedures in usability testing. Three experiments were conducted using different artefacts (website, computer-simulated mobile phone, fully operational smartphone) to compare different methodological approaches in field testing (synchronous and asynchronous remote testing, classical field testing) to lab-based testing under different operational conditions (dual task demands, poor product usability). Typical outcome variables of usability testing were measured, including task completion time, click rate, perceived usability and workload. Overall, the results showed no differences between field and lab-based testing under favourable operational conditions. However, under difficult operational conditions (i.e. dual task demands, poor product usability) differences between field and lab-based testing emerged (corresponding to small and medium effect sizes). The findings showed a complex pattern of effects, suggesting that there was no general advantage of one testing procedure over another.
Chapter
Full-text available
Since 2012, usability testing in Italian public administration (PA) has been guided by the eGLU 2.1 technical protocols, which provide a set of principles and procedures to support specialized usability assessments in a controlled and predictable way. This paper describes a new support tool for usability testing that aims to facilitate the application of eGLU 2.1 and the design of its User eXperience (UX) evaluation methodology. The usability evaluation tool described in this paper is called UTAssistant (Usability Tool Assistant). UTAssistant has been entirely developed as a Web platform, supporting evaluators in designing usability tests, analyzing the data gathered during the test and aiding Web users step-by-step to complete the tasks required by an evaluator. It also provides a library of questionnaires to be administered to Web users at the end of the usability test. The UX evaluation methodology adopted to assess the UTAssistant platform uses both standard and new bio-behavioral evaluation methods. From a technological point of view, UTAssistant is an important step forward in the assessment of Web services in PA, fostering a standardized procedure for usability testing without requiring dedicated devices, unlike existing software and platforms for usability testing.
Conference Paper
Full-text available
In the framework of the AVI 2018 Conference, the interuniversity center ECONA has organized a thematic workshop on "Digital Interaction: where are we going?". Six contributions from the ECONA members investigate different perspectives around this thematic.
Conference Paper
Full-text available
Even if the benefits of the usability testing are remarkable, it is scarcely adopted in the software development process. To foster its adoption, this paper presents a Web platform, UTAssistant, that supports people, also without skills in Human-Computer Interaction (HCI), in evaluating Web site usability.
Article
Full-text available
This paper describes the concepts of TEA, a flexible tool that supports user tests by automating repetitive tasks and collecting data of user inputs and actions. TEA was specifically designed for user studies in the World Wide Web and is able to interact with a web browser. Building on a web intermediary (WBI) and a framework for web enhancement tools (SCONE), TEA can be applied in a range of test settings - providing either a controlled laboratory environment or a quick tool for collecting informal data.
Article
Il protocollo eGLU 1.0 (scaricabile dal sito http://www.funzionepubblica.gov.it/glu) è stato creato con lo scopo di individuare le aree di un sito web percepite come problematiche dagli utenti, al fine di determinare, in maniera efficace ed efficiente, in quali sezioni del sito occorra investire e coinvolgere esperti in valutazione per un’analisi approfondita dell’interazione.
Article
This paper shows how to use the UsaProxy HTTP proxy to perform logging of user activity for AJAX web applications. UsaProxy is a special-purpose HTTP proxy which modifies HTML pages before forwarding them to the client browser. It adds JavaScript code which collects data about mouse movement, clicks, key presses and other types of interaction without affecting the user's browsing experience in any way. Using Gmail as an example for an AJAX application, the paper explains in detail how to prepare for a UsaProxy-based user test and how to interpret the log files generated by the system.
Conference Paper
The idea of conducting usability tests remotely emerged ten years ago. Since then, it has been studied empirically, and some software organizations employ remote methods. Yet there are still few comparisons involving more than one remote method. This paper presents results from a systematic empirical comparison of three methods for remote usability testing and a conventional laboratory-based think-aloud method. The three remote methods are a remote synchronous condition, where testing is conducted in real time but the test monitor is separated spatially from the test subjects, and two remote asynchronous conditions, where the test monitor and the test subjects are separated both spatially and temporally. The results show that the remote synchronous method is virtually equivalent to the conventional method. Thereby, it has the potential to conveniently involve broader user groups in usability testing and support new development approaches. The asynchronous methods are considerably more time-consuming for the test subjects and identify fewer usability problems, yet they may still be worthwhile.
Conference Paper
Heuristic evaluation is an informal method of usability analysis where a number of evaluators are presented with an interface design and asked to comment on it. Four experiments showed that individual evaluators were mostly quite bad at doing such heuristic evaluations and that they only found between 20 and 51% of the usability problems in the interfaces they evaluated. On the other hand, we could aggregate the evaluations from several evaluators to a single evaluation and such aggregates do rather well, even when they consist of only three to five people.
Il Protocollo Eglu 2.1: Come Realizzare Test Di Usabilità Semplificati Per I Siti Web E I Servizi Online Delle Pa
  • Dipartimento Della
  • Funzione Pubblica
Dipartimento della Funzione Pubblica: Il Protocollo Eglu 2.1: Come Realizzare Test Di Usabilità Semplificati Per I Siti Web E I Servizi Online Delle Pa. Formez PA, Rome, IT (2015)