No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Heuristic Evaluation of eGLU-Box: A Semi-automatic Usability Evaluation Tool for Public Administrations
Abstract
This paper illustrates the heuristic evaluation of a web-based tool for usability testing for Public Administrations called eGLU-box. eGLU-box is an online platform aiming at supporting practitioners in the process of designing usability tests, analyzing data, and helping step-by-step participants to complete assessment tasks. Web users of Public Administrations can report their perceived quality of experience by completing a library of questionnaires shown to them by eGLU-box at the end of the test. This work is part of a multi-step user experience (UX) evaluation methodology to assess the platform. The UX evaluation methodology of eGLU-box uses standard and bio-behavioural evaluation methods. This work shows the results of the heuristic evaluation of eGLU-box involving five human factors experts and 20 practitioners working in Italian Public Administrations. Findings show that most of the problems are rated as minor problems and related to Nielsen’s heuristic, “visibility of the system.” Only 9% of problems are rated as major problems. These major problems are related to the “problematic match between system and the real world” heuristic. Evaluators provided indications for improvements that will be applied for the next version of the platform.
... The advent of HTML5 in 2014 and JavaScript EM6 in 2015 led to a new generation of tools for remote usability testing. Examples are Loop11 [25], Lookback [26], Userlytics [27], UserTesting [28] and eGLU-Box PA [36,37]. These tools are characterized by similar novel and useful features that might contribute to overcoming some limitations of the previous generation of tools. ...
... In the analysis of the studies in the literature on remote usability tests, we did not find any studies addressing remote tests performed by using recent web-based tools (after 2014). As discussed in Section II.A, these modern tools (e.g., Loop11 [25], Lookback [26], Userlytics [27], UserTesting [28], and eGLU-Box PA [36,37]) are characterized by integrated webbased environments that allow participants to perform a usability test by simply using a website; in general, users do not need to install new software on their personal computers, as required in the past [17-20, 32, 33]. Thanks to advances in web technologies, such tools automatically collect quantitative (e.g., participant logs, task time, success rate) and qualitative data (e.g., the recording of a webcam, microphone, and screen). ...
... Other web managers were asked to use other versions of eGLU-Box PA to perform usability tests of their websites. Some of the studies carried out to evaluate eGLU-Box PA at various stages are reported in [37,40,41], in which further details on the design, development, and use of this tool can also be found. Please note that a previous version of eGLU-Box PA was called UTAssistant. ...
Remote usability testing is performed by evaluators who are in different physical locations from the participants (synchronous remote testing) and possibly operating at different times (asynchronous remote testing). The tools developed in recent years to support remote tests exploit web technology based on HTML5 and JavaScript ES6 and thus enable previously unexplored scenarios. However, studies providing evidence on the benefits or drawbacks of utilizing recent web-based tools have not yet been reported in the literature. This article sheds some light on the impact of such tools on asynchronous remote usability testing of websites by reporting an experimental study with 100 participants and 15 evaluators to compare real-time laboratory tests with asynchronous remote tests. The study investigates: 1) how the metrics results of asynchronous remote usability tests performed through a web-based tool differ from those of usability tests conducted in real-time laboratory settings; and 2) how the experience of participants differs in the two types of tests. The lessons learned in the study are instrumental in informing the design of future tools. Some results of particular interest indicate that the web technology used by the tool for asynchronous remote testing affects task execution times and participants’ satisfaction. Another indication is that slow internet connections must be managed in asynchronous remote testing; slow connections introduce delays when transferring large amounts of collected data, which, together with the lack of human support, make participants of asynchronous remote tests more prone to feel negative emotions.
... Evaluation methods can be conducted in person or remotely [33]. When it is conducted remotely, the evaluator can follow the study synchronously or asynchronously and is mediated by ad-hoc tools [32,43,52,102]. The remote usability testing is most relevant for HCAI systems designed for large-scale or cross-cultural use where users might be placed worldwide. Finally, it is worth mentioning that there are tools that partially automate usability testing [18,47,48,102]. ...
The rapid growth in interest in Artificial Intelligence (AI) has been a significant driver of research and business activities in recent years. This raises new critical issues, particularly concerning interaction with AI systems. This article first presents a survey that identifies the primary issues addressed in Human-Centered AI (HCAI), focusing on the interaction with AI systems. The survey outcomes permit to clarify disciplines, concepts, and terms around HCAI, solutions to design and evaluate HCAI systems, and the emerging challenges; these are all discussed with the aim of supporting researchers in identifying more pertinent approaches to create HCAI systems. Another main finding emerging from the survey is the need to create Symbiotic AI (SAI) systems. Definitions of both HCAI systems and SAI systems are provided. To illustrate and frame SAI more clearly, we focus on medical applications, discussing two case studies of SAI systems.
... However, the data analysis and management lack flexibility, resulting in researchers still relying on advanced knowledge of data analysis and additional third-party tools like IBM SPSS [29] or writing custom scripts in either R [49] or Python. Additionally, platforms are available to support usability evaluations at large [2,15,31,36], for example, Zeng [52] developed a web application with a specific focus on improved user experience (UX), targeted to perform quantitative analysis on measurements obtained from usability tests. However, the platform lacks guidance in selecting appropriate statistical methodologies or settings for GUR. ...
Utilizing a validated scale to assess player experience (PX) in video games can yield important findings. Nevertheless, insufficient quantitative data analysis expertise among games user research (GUR) self-taught practitioners, poses obstacles to effectively analyzing and interpreting the data obtained from various measurement scales. Hence, in this paper, we present a guided interface based web platform to support data analysis of the Player Experience Inventory (PXI), following a user-centered design (UCD) process, incorporating insights from Games User Research (GUR) specialists. The interface, extending a current platform for PXI data evaluation, aspires to enable data analysis with a high degree of flexibility, offering explanatory support for both, inexperienced and experienced analysts. The findings from user testing sessions indicated that the platform was perceived as user-friendly. However, findings also suggest that more accessible explanations of the statistical approaches are needed to aid in understanding data analysis and interpretation of outcomes. CCS CONCEPTS • Human-centered computing → HCI design and evaluation methods; • Information systems → Web interfaces. KEYWORDS player experience evaluation, quantitative data analysis, guided interface ACM Reference Format:
... The value of informal user testing with the thinking-aloud protocol has been well known since the seminal book by Nielsen (1993); this method is highly appreciated and used widely, especially in formative evaluation or in situations in which resources are limited or evaluators are less experienced. Such informal user testing is currently widely used in the website evaluation protocols adopted by several Italian public administration institutions (see, e.g., Federici et al., 2019;Federici et al., 2021). ...
... In view of this, the present study compares a mobile version of eGLU-box developed for the Italian public administration (PA), a national government, with the original version for desktop use. eGLU-box PA (in the following eGLU-box) is a web platform that allows PA webmasters in Italy to evaluate the usability of their websites and digital services [2]. It was developed based on the eGLU LG 2018.1 protocol, and is designed to run on a personal computer or Mac [3]. ...
Smartphones and tablets now offer consumers unique advantages such as portability and accessibility. Developers are also working with a mobile-first approach, and are prioritizing mobile applications over desktop versions. This study introduces eGLU-box Mobile, an application for performing a drive usability test directly from a smartphone. An experimental study was conducted in which the participants were divided into two groups: an experimental group, which used the new mobile application from a smartphone, and a control group, which used the desktop application from a computer. The participants’ behavior was assessed using explicit (self-report questionnaires) and implicit measures (eye movement data). The results were encouraging, and showed that both the mobile and desktop versions of eGLU-box enabled participants to test the usability with a similar level of UX, despite some minimal (although significant) differences in terms of satisfaction of use.
... Compared with other aspects of the digital artefact's development process, the evaluation stage still suffers from inconsistency in paradigms, methodologies, and procedures, as well as a lack of objectiveness, partially due to the monetary and time costs involved [2]. In order to reduce costs, efforts have been made to develop solutions such as remote user testing [3] and semi-automatic usability evaluations [4]. However, these methods focus on streamlining data collection for UX evaluation, and the need for tools to assist UX professionals in analyzing the data gathered from usability tests remains. ...
As the digital era is witnessing huge commercial success driven by design innovations focusing on usability as well as utility, businesses have also been positioning user experience (UX) research as the cornerstone of the competitiveness of their products and services. However, UX data analysis can be costly and demanding in terms of monetary and human resources, making it imperative to develop a cost-effective solution to assist UX evaluation. In this article, we present the design, development, and deployment process of the UX Calculator (uxcalc.web.app), a lightweight web-based tool that enables practitioners to run statistical tests tailored to the main analyses involved in a UX assessment. This tool allows designers to perform two-group comparisons and get robust conclusions about the UX performance without extra training in statistical programming. The interaction logic of this tool, together with the methodological considerations, practical implications, and future work for the tool’s improvement are discussed in detail.KeywordsEvaluation Methods and TechniquesUser ExperienceUsability TestUX PerformanceUX Calculator
... The questionnaires were administered to two groups of participants conducting the usability test in two conditions, namely, home-space and offce-space. All the participants performed the experimental tasks through a usability assessment web-based platform called eGLU-box PA, a tool developed by the PA to evaluate the usability of its digital platforms and services [10]. The results showed no differences in the overall experience between the two experimental conditions. ...
The COVID-19 pandemic has brought several changes in everyday life, one of them being the application of Remote Working (RW). RW is the new way of working, thanks to this new modality all workers, with certain work requirements, were able to carry out their work from home without having to go to the office. Given the strict rules relating to lockdown, if this method had not been applied many people would not have been able to work and today many companies would probably be closed. But which advantages and disadvantages can RW have compared to classical work? Can it bring more inclusiveness and accessibility for every one or only for workers with specific requirements (for example, for workers that need to take care of family members with disabilities)? This paper attempts to answer these questions. The University of Perugia in collaboration with the Ministry of Economic Development has created the “Job-satisfying” project. In this project 24 participants were divided into two groups (home-space group and office-space group) and each of these had to complete some tasks and complete questionnaires. Generally, no significant difference emerged but some interesting results were encountered: those who took the experimentation from home, that have children, obtained higher scores relating to the sense of working autonomy, support from superiors and satisfaction of relationships at work. This data seems to argue that working from home can improve inclusiveness.
... To promote the adoption of UX evaluation techniques, remote usability and UX techniques have been proposed. For example, software platforms for remote testing allow participants to test systems from their locations, at times when it is convenient for them.ExamplesofmorerecenttoolsareUserlytics [14],Loop11 [10], eGLU-Box PA [7]. eGLU-Box PA also provides advanced visualization techniques to support the UX experts in detecting issues concerning UX, called "UX smells" [3]. ...
This demo presents SERENE, a Web platform for the UX semi-automatic evaluation of websites. It exploits Artificial Intelligence to predict visitors’ emotions starting from their interaction logs. The predicted emotions are shown by interactive heatmaps overlapped to the webpage to be analyzed. The concentration of negative emotions in a specific area of the webpage can help the UX experts identify UX problems.
... Due to COVID-19 restrictions, the study had to be performed remotely. The tool eGLU-Box PA [11] has been used to perform the remote study. The participants have been recruited among the students at the University of Bari and at the University of Trento on a voluntary basis. ...
In this paper, we present the design and the evaluation of an authoring tool for End-User Development, which supports the definition of Trigger-Actions rules that combines events and states in the triggers. The possibility of using either states or events in triggers has already been discussed in the literature. However, it is recognized that the state/event distinction is difficult to manage for users. In this paper, we propose an authoring tool that provides explicit support for managing this distinction. We compare it with a state-of-the-art authoring tool that implements the classical event-event paradigm.
... Given the COVID-19 pandemic, the study was performed remotely. To facilitate the remote execution, a tool for remote user testing, eGLU-Box PA, was used [17]. Three evaluators (HCI experts) were involved. ...
Given the spread of the Internet of Things (IoT) technology, in several contexts there is a growing need for laypeople to configure their smart devices. Task Automation Systems (TASs) have emerged as tools to simplify the definition of rules for personalizing the behavior of such devices. However, one aspect often neglected by current TASs, which is instead typical of IoT technologies, relates to the security and privacy threats exposed by the “connected” devices. This paper tries to address this problem and illustrates a user-centered design that eventually led to identify a visual paradigm that facilitates the end users in understanding and controlling security and privacy threats.
... This platform, called eGLU Box PA, provides Italian PAs with a lightweight and simple service that does not require any installation on user devices and has no other special requirements (e.g., specific operating system and/or processors). Experiments conducted with eGLU Box PA have shown that it provides great support to PA organizations in designing and running usability testing [36]. eGLU Box PA is in two versions, Italian and English, so that it can be used worldwide. ...
Daily experiences in working with various types of computer systems show that, despite the offered functionalities, users have many difficulties, which affect their overall User eXperience (UX). The UX focus is on aesthetics, emotions and social involvement, but usability has a great influence on UX. Usability evaluation is acknowledged as a fundamental activity of the entire development process in software practices. Research in Human-Computer Interaction has proposed methods and tools to support usability evaluation. However, when performing an evaluation study, novice evaluators still have difficulties to identify usability problems and to understand their causes: they would need easier to use and possibly automated tools. This article describes four visualization techniques whose aim is to support the work of evaluators when performing usability tests to evaluate websites. Specifically, they help detect “usability smells”, i.e. hints on web pages that might present usability problems, by visualizing the paths followed by the test participants when navigating in a website to perform a test task. A user study with 15 participants compared the four techniques and revealed that the proposed visualizations have the potential to be valuable tools for novice usability evaluators. These first results should push researchers towards the development of further tools that are capable to support the detection of other types of UX smells in the evaluation of computer systems and that can be translated into common industry practices.
... The study was organized according to a within-subject design, with visualization as an independent variable and as within-subject factors the two visualizations. Each visualization showed data gathered in a study conducted with 15 employees of an Italian Administration, who used a Web platform for onsite and remote usability tests [4,5]. With each system, the participants performed the following tasks: 1. Identify the page that caused most problems; 2. Identify the path(s) that led to the task's failure; 3. Identify the path that led to the task success following the ideal path, 4. Identify the paths that led to the task success following alternative paths; 5. Identify backward paths, if possible. ...
This paper reports on an on-going work that investigates the use of visualization techniques to help evaluators discovering usability problems by visualizing data collected during usability tests of web sites. Two visualization techniques are described and some results of the evaluation study that compared the two techniques are provided.
Due to the importance of usability, multiple usability evaluation methods have been proposed that help Human-Computer Interaction (HCI) specialists determine whether the interfaces of a software product are usable, easy to use, understandable, and attractive. Among these methods, the heuristic evaluation proposed by Jakob Nielsen is the one that stands out. Although Nielsen offers general guidelines for the execution of heuristic evaluations, very few authors of different studies present a formal agreement or process on how the evaluations should be carried out, which leads us to the problem of the absence of a comparative analysis that allows determining the most appropriate formal evaluation process to carry out heuristic inspections. To complement, some proposals found in the literature compare the results of the execution of heuristic evaluations, the definition of new heuristics, qualification forms, and the formalization of the complete process in 5 phases. Although these proposals contribute to the formalization of the heuristic evaluation process, the literature review has not provided a comparative analysis that allows determining which is the most appropriate, which could cause usability evaluators to interpret and carry out their own procedure, which would lead to inaccuracies in the results and increase the probability of improperly executing the inspection. The purpose of this study was to elaborate a comparative table of the formal processes; this allowed the grouping of the various studies found in the literature, to select a process, and thus, and to model by a BPMN tool the whole process. Finally, this process modeled was validated by expert judgement of HCI specialists.
This paper shows a chatbot solution for eGLU-box Pro, a usability testing platform for Italian Public Administration (PA). eGLU-box Pro is a web-based tool designed to help PA practitioners in creating remote usability tests and analyzing participants’ answers and interaction data after they complete the usability tasks. The impact of the chatbot solution on users’ experience was assessed by bio-behavioral evaluation methods such as eye tracking, electroencephalography, and facial expression recognition. This work describes the platform and its integrated chatbot solution and shows the results of a preliminary laboratory study involving 20 end-users. The study is part of an ongoing design and development project based on a user-centered approach.
Heuristic evaluation belongs to the usability inspection methods and is considered one of the most popular methods since it allows to discover over 75% of the total usability problems involving only 3 to 5 usability experts, in comparison with user tests. However, certain problems and challenges have been identified at the time of their execution. In this study we present the results of conducting a Systematic Literature Review (SLR) to identify case studies, challenges, problems, and opportunities on the execution of heuristic evaluations in the context of a research for the automation and formalization of the process. For this SLR, we have employed the protocol proposed by Kitchenham and Charters. The research was carried out on September 7 of 2020 and retrieved a total of 167 studies of which 37 were selected for this review. The results show that the main challenges are related to the low suitability of the chosen set of heuristics and the low expertise of usability evaluators. Additionally, we have identified that exist very few software solutions that support and automate the process. Finally, we have found that there were many protocols to follow when applying a heuristic evaluation, like the definition of new usability heuristics for a given case of study. According to the results obtained, we can conclude that it is necessary to develop and validate a tool based on a formal protocol that supports and automate the heuristic evaluation process, that gives solutions to the challenges and opportunities identified in this research.
This work shows the user experience (UX) assessment of a web-based platform for the semi-automatic usability evaluation of websites, UTAssistant, which is primarily addressed to workers in public administration (PA). The study is part (Phase 1) of a multiple assessment methodology which consists of four phases in total: (1) UX in laboratory conditions; (2) Usability evaluation in remote online conditions; (3) Usability evaluation in workplace conditions; and (4) Heuristic evaluation. In Phase 1, a UX study in laboratory conditions was carried out. Participants' UX of a PA website navigation through UTAssistant was evaluated by both traditional self-report usability assessment tools (SUS and UMUX) and bio-behavioral measurement techniques (facial expression recognition and electroencephalography). Results showed that using the UTAssistant usability assessment tool for webpages did not affect users' perceived usability in terms of self-reports and affective states, which were mostly neutral for all the assessment session. However, frontal alpha asymmetry EEG's scores showed a higher sensitivity of UTAssistant users to the duration of the trial, with a decrease in motivation displayed as the trial ensued. However, this result did not seem to affect emotional experience.
The present article examined the effects of using different extra-laboratorial testing procedures in usability testing. Three experiments were conducted using different artefacts (website, computer-simulated mobile phone, fully operational smartphone) to compare different methodological approaches in field testing (synchronous and asynchronous remote testing, classical field testing) to lab-based testing under different operational conditions (dual task demands, poor product usability). Typical outcome variables of usability testing were measured, including task completion time, click rate, perceived usability and workload. Overall, the results showed no differences between field and lab-based testing under favourable operational conditions. However, under difficult operational conditions (i.e. dual task demands, poor product usability) differences between field and lab-based testing emerged (corresponding to small and medium effect sizes). The findings showed a complex pattern of effects, suggesting that there was no general advantage of one testing procedure over another.
In the framework of the AVI 2018 Conference, the interuniversity center ECONA has organized a thematic workshop on "Digital Interaction: where are we going?". Six contributions from the ECONA members investigate different perspectives around this thematic.
Even if the benefits of the usability testing are remarkable, it is scarcely adopted in the software development process. To foster its adoption, this paper presents a Web platform, UTAssistant, that supports people, also without skills in Human-Computer Interaction (HCI), in evaluating Web site usability.
This paper describes the concepts of TEA, a flexible tool that supports user tests by automating repetitive tasks and collecting data of user inputs and actions. TEA was specifically designed for user studies in the World Wide Web and is able to interact with a web browser. Building on a web intermediary (WBI) and a framework for web enhancement tools (SCONE), TEA can be applied in a range of test settings - providing either a controlled laboratory environment or a quick tool for collecting informal data.
Since 2012, usability testing in Italian public administration (PA) has been guided by the eGLU 2.1 technical protocols, which provide a set of principles and procedures to support specialized usability assessments in a controlled and predictable way. This paper describes a new support tool for usability testing that aims to facilitate the application of eGLU 2.1 and the design of its User eXperience (UX) evaluation methodology. The usability evaluation tool described in this paper is called UTAssistant (Usability Tool Assistant). UTAssistant has been entirely developed as a Web platform, supporting evaluators in designing usability tests, analyzing the data gathered during the test and aiding Web users step-by-step to complete the tasks required by an evaluator. It also provides a library of questionnaires to be administered to Web users at the end of the usability test. The UX evaluation methodology adopted to assess the UTAssistant platform uses both standard and new bio-behavioral evaluation methods. From a technological point of view, UTAssistant is an important step forward in the assessment of Web services in PA, fostering a standardized procedure for usability testing without requiring dedicated devices, unlike existing software and platforms for usability testing.
Il protocollo eGLU 1.0 (scaricabile dal sito http://www.funzionepubblica.gov.it/glu) è stato creato con lo scopo di individuare le aree di un sito web percepite come problematiche dagli utenti, al fine di determinare, in maniera efficace ed efficiente, in quali sezioni del sito occorra investire e coinvolgere esperti in valutazione per un’analisi approfondita dell’interazione.
This paper shows how to use the UsaProxy HTTP proxy to perform logging of user activity for AJAX web applications. UsaProxy is a special-purpose HTTP proxy which modifies HTML pages before forwarding them to the client browser. It adds JavaScript code which collects data about mouse movement, clicks, key presses and other types of interaction without affecting the user's browsing experience in any way. Using Gmail as an example for an AJAX application, the paper explains in detail how to prepare for a UsaProxy-based user test and how to interpret the log files generated by the system.
The idea of conducting usability tests remotely emerged ten years ago. Since then, it has been studied empirically, and some software organizations employ remote methods. Yet there are still few comparisons involving more than one remote method. This paper presents results from a systematic empirical comparison of three methods for remote usability testing and a conventional laboratory-based think-aloud method. The three remote methods are a remote synchronous condition, where testing is conducted in real time but the test monitor is separated spatially from the test subjects, and two remote asynchronous conditions, where the test monitor and the test subjects are separated both spatially and temporally. The results show that the remote synchronous method is virtually equivalent to the conventional method. Thereby, it has the potential to conveniently involve broader user groups in usability testing and support new development approaches. The asynchronous methods are considerably more time-consuming for the test subjects and identify fewer usability problems, yet they may still be worthwhile.
Heuristic evaluation is an informal method of usability analysis where a number of evaluators are presented with an interface design and asked to comment on it. Four experiments showed that individual evaluators were mostly quite bad at doing such heuristic evaluations and that they only found between 20 and 51% of the usability problems in the interfaces they evaluated. On the other hand, we could aggregate the evaluations from several evaluators to a single evaluation and such aggregates do rather well, even when they consist of only three to five people.
Il Protocollo Eglu 2.1: Come Realizzare Test Di Usabilità Semplificati Per I Siti Web E I Servizi Online Delle Pa
- Dipartimento Della
- Funzione Pubblica
Dipartimento della Funzione Pubblica: Il Protocollo Eglu 2.1: Come Realizzare Test Di
Usabilità Semplificati Per I Siti Web E I Servizi Online Delle Pa. Formez PA, Rome, IT
(2015)