Simone Borsci

University of Twente | UT · Dept. Learning, Data analysis, and Technology - Cognition, Data and Education group - CODE

Intelligent systems, such as chatbots, are likely to strike new qualities of UX that are not covered by instruments validated for legacy human–computer interaction systems. A new validated tool to evaluate the interaction quality of chatbots is the chatBot Usability Scale (BUS) composed of 11 items in five subscales. The BUS-11 was developed mainly...

Borsci, Simone and Prati, Elisa and Willemse, Cesco and Stefánsdóttir, Ásthildur and Peruzzini, Margherita, Development and Face Validity of The User Experience in Human Robot Collaboration Interaction Scale (UX-HRCI Scale) (August 05, 2024). Available at SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4922500

Background Effective teamwork is crucial to providing safe and high-quality patient care, especially in acute care. Crew Resource Management (CRM) principles are often used for training teamwork in these situations, with escape rooms forming a promising new tool. However, little is known about escape room design characteristics and their effect on...

Purpose The objective of this study is to adapt and evaluate the Turkish version of the Chatbot Usability Scale (BUS-11) through a confirmatory factorial analysis method. The BUS-11 scale has been established in various languages except for Turkish; thus, its validation and dissemination could serve as a means to improve chatbot interaction satisfa...

Invited talk - Guest Speech

In an increasingly complex everyday life, algorithms – often learnt from data, i.e. machine learning (ML) – are used to make or assist operational decisions. However, developers and designers usually are not entirely aware of how to reflect on social justice while designing ML algorithms and applications. Algorithmic social justice – i.e., designin...

This study describes an evidence-based Clinical Pathway Mapping (CPM) visualisation method that can be used to enhance healthcare quality in the face of complex systems and constrained resources. The CPM visualisation method we are proposing highlights the connection among the key components of the healthcare work system – individuals, tasks, tools...

This paper presents a human factors qualitative study on an AI application for managing sepsis in Intensive Care Units (ICUs). The study involved semi-structured interviews with nine ICU clinicians and nurses across three London hospitals. It consisted of two parts: the first applied methods to understand sepsis resuscitation processes and establis...

Introduction The International Federation of Medical and Biological Engineering created a multidisciplinary working group to discuss assessments of artificial intelligence and machine learning (AI/ML) applications in health care. Engineers, clinicians, and economists identified evidence generation as a critical topic. Heart failure (HF) was selecte...

Purpose This research aims to explore digital feedback needs/preferences in online education during lockdown and the implications for post-pandemic education. Design/methodology/approach An empirical study approach was used to explore feedback needs and experiences from educational institutions in the Netherlands and Germany (N = 247) using a surv...

Artificial intelligence and machine learning (AI/ML) are playing increasingly important roles, permeating the field of medical devices (MDs). This rapid progress has not yet been matched by the Health Technology Assessment (HTA) process, which still needs to define a common methodology for assessing AI/ML-based MDs. To collect existing evidence fro...

Smartphones and tablets now offer consumers unique advantages such as portability and accessibility. Developers are also working with a mobile-first approach, and are prioritizing mobile applications over desktop versions. This study introduces eGLU-box Mobile, an application for performing a drive usability test directly from a smartphone. An expe...

Chatbot-based tools are becoming pervasive in multiple domains from commercial websites to rehabilitation applications. Only recently, an eleven-item satisfaction inventory was developed (the ChatBot Usability Scale, BUS-11) to help designers in the assessment process of their systems. The BUS-11 has been validated in multiple contexts and language...

In the last decade, the field of Human-Robot Collaboration (HRC) has received much attention from both research institutions and industries. Robot technologies are in fact deployed in many different areas (e.g., industrial processes, people assistance) to support an effective collaboration between humans and robots. In this transdisciplinary contex...

Chatbot-based tools becoming pervasive in multiple domains from commercial websites to rehabilitation applications. Only recently an eleven-item satisfaction inventory was developed (the chatBot Usbaliy Scale, BUS-11) to help designers in the assessment process of their systems. The BUS-11 has been validated in multiple contexts and languages i.e.,...

The Bot Usability Scale (BUS) is a standardised tool to assess and compare the satisfaction of users after interacting with chatbots to support the development of usable conversational systems. The English version of the 15-item BUS scale (BUS-15) was the result of an exploratory factorial analysis; a confirmatory factorial analysis tests the repli...

Working from home (WFH) remotely is a modality of working that requires the careful design of systems of rules and tools to enable people to exchange information and perform actions. WFH is expected to expand after the COVID-19 pandemic. How to assess and compare in a reliable way the experience of workers with different (sociotechnical) systems of...

The COVID-19 pandemic has brought several changes in everyday life, one of them being the application of Remote Working (RW). RW is the new way of working, thanks to this new modality all workers, with certain work requirements, were able to carry out their work from home without having to go to the office. Given the strict rules relating to lockdo...

Digital twins (DTs) have been found useful in manufacturing, construction, and maintenance. Adapting DTs to serve cities, the question arises of what an urban digital twin should contain and how it should be orchestrated to serve a city’s dynamical ecosystem, along with how to enhance the efficiency of the city. We are aligning with the commonplace...

Standardised tools to assess a user’s satisfaction with the experience of using chatbots and conversational agents are currently unavailable. This work describes four studies, including a systematic literature review, with an overall sample of 141 participants in the survey (experts and novices), focus group sessions and testing of chatbots to (i)...

The European Union (EU) Commission’s whitepaper on Artificial Intelligence (AI) proposes shaping the emerging AI market so that it better reflects common European values. It is a master plan that builds upon the EU AI High-Level Expert Group guidelines. This article reviews the masterplan, from a culture cycle perspective, to reflect on its potenti...

Background NG (nasogastric) tubes are used worldwide as a means to provide enteral nutrition. Testing the pH of tube aspirates prior to feeding is commonly used to verify tube location before feeding or medication. A pH at or lower than 5.5 was taken as evidence for stomach intubation. However, the existing standard pH strips lack sensitivity, espe...

Introduction: Working from home (WFH) remotely is a modality of working that requires the careful design of systems of rules and tools to enable people to exchange information and perform actions. WFH is expected to expand after the COVID-19 pandemic, and how best to reliably assess and compare the experience of workers with different (sociotechnic...

Artificial Intelligence (AI) systems could improve system efficiency by supporting clinicians in making appropriate referrals. However, they are imperfect by nature and misdiagnoses, if not correctly identified, can have consequences for patient care. In this paper, findings from an online survey are presented to understand the aptitude of GPs (n =...

This paper shows a chatbot solution for eGLU-box Pro, a usability testing platform for Italian Public Administration (PA). eGLU-box Pro is a web-based tool designed to help PA practitioners in creating remote usability tests and analyzing participants’ answers and interaction data after they complete the usability tasks. The impact of the chatbot s...

A health technology assessment (HTA) is commonly defined as a multidisciplinary approach used to evaluate medical, social, economic, and ethical issues related to the use of a health technology in a systematic, transparent, unbiased, robust manner. To help inform HTA recommendations, the surveillance of social media platforms can provide important...

This book constitutes the proceedings of the 8th International Conference on End-User Development, IS-EUD 2021, held in July 2021. Due to COVID-19 pandemic the conference was held virtually. The paper track received 26 submissions, of which 11 full and 4 short papers were selected after a rigorous double-blind review process. The papers focus on “D...

People with disabilities or special needs can benefit from AI-based conversational agents, which are used in competence training and well-being management. Assessment of the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understand their potential long-term benefits. This will in turn hel...

Introduction: People with disabilities or special needs can benefit from AI-based conversational agents (i.e., chatbots) that are used for competence training and well-being management. Assessing the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understanding their potential long-term be...

Introduction Individuals with chronic lung disease (eg, cystic fibrosis (CF)) often receive antimicrobial therapy including aminoglycosides resulting in ototoxicity. Extended high-frequency audiometry has increased sensitivity for ototoxicity detection, but diagnostic audiometry in a sound-booth is costly, time-consuming and requires a trained audi...

Introduction: Health technology assessments (HTA) are tools for policy making and resource allocation. Early HTAs are increasingly used in design and development of new technologies. Conducting early HTAs is challenging, due to a lack of evidence and significant uncertainties in the technology and the market. A multi-disciplinary approach is consid...

The value of technology to health care is not dependent on clinical and economic considerations alone. In this chapter we describe health related as well as economic, social, legal, ethical, and organizational criteria to include in early health technology assessment. Moreover, we provide recommendations on methods to collect data on these criteria...

The main role of human factors and ergonomics professionals within health care is to generate evidence to support the development and the redesign of services and products. To achieve safe and resilient health system environments, both technology and people should collaborate in a resonant way by ensuring an optimal experience to patients. Human fa...

Objective: To ascertain the reliability of a standardised, short-scale measure of satisfaction in the use of new healthcare technology i.e., the LITE version of the usability metric for user experience (UMUX-LITE). Whilst previous studies have demonstrated the reliability of UMUX-LITE, and its relationship with measures of likelihood to recommend...

A recent contribution to the ongoing debate concerning the concept of usability and its measures proposed that usability reached a dead end – i.e. a construct unable to provide stable results and to unify scientific knowledge. Extensive commentaries rejected the conclusion that researchers need to look for alternative constructs to measure the qual...

Every day people use multiple technologies to perform complex tasks, such as buying products online, informing their decision making, or supporting their work activities. In many cases, as in healthcare, high risk processes are dependent on the technology to deliver the appropriate service - e.g. diagnostics outcomes, deliver treatments to patients...

Objectives Current health technology assessment (HTA) methods guidelines for medical devices may benefit from contributions by biomedical and clinical engineers. Our study aims to: (i) review and identify gaps in the current HTA guidelines on medical devices, (ii) propose recommendations to optimize the impact of HTA for medical devices, and (iii)...

The reliability and clinical utility of a medical device are required to demonstrate its impact and to enhance its success in the market. To fully establish these attributes it is necessary to investigate the context of use and the needs of the users. In healthcare, there is a growing use of human factors and health economics methods to generate ev...

Aims To test a hypothesis that cataract operating room (OR) productivity can be improved with a femtosecond laser (FL) using a hub-and-spoke model and whether any increase in productivity can offset additional costs relating to the FL. Methods 400 eyes of 400 patients were enrolled in a randomised-controlled trial comparing FL-assisted cataract su...

This chapter discusses the relationship between the accessibility and usability constructs and how they relate to the user experience (UX) theoretical approach. We present an integrated model of interaction evaluation, a new evaluation perspective based on UX that is intended to be used as a framework for evaluating users’ interactions with assisti...

Introduction. The slow adoption of innovation into healthcare calls into question the manner of evidence generation for medical technology. This paper identifies potential reasons for this including a lack of attention to human factors, poor evaluation of economic benefits, lack of understanding of the existing healthcare system and a failure to re...

[This corrects the article DOI: 10.1186/s41512-016-0001-y.].

[This corrects the article DOI: 10.1186/s41512-016-0001-y.].

Aims (1) To model the process of use and usability of pH strips (2) to identify, through simulation studies, the likelihood of misreading pH strips, and to assess professional’s acceptance, trust and perceived usability of pH strips. Methods This study was undertaken in four phases and used a mixed method approach (an audit, a semi-structured inte...

Semi-structured interview—Results of standardised questionnaire. (DOCX)

Online survey—Results of standardised questionnaire. (DOCX)

Questionnaires and evaluation scales. (DOCX)

Preliminary graphic representation of NG-tube and pH strips use. (TIF)

Revised graphic representation of NG-tube and pH strips use. (TIF)

Aim To provide a quantitative assessment of cataract theatre lists focusing on productivity and staffing levels/tasks using time and motion studies. Methods National Health Service (NHS) cataract theatre lists were prospectively observed in five different institutions (four NHS hospitals and one private hospital). Individual tasks and their timing...

Script of the slides 1. This paint of Giorgio De Chirico expresses the solitude of objects, and the solitude of artists who design a product. I think that this represents quite well the challenges faced by manufacturers when they aim to design something which could potentially change the world. Especially when they try to design products which have...

The classical approach of ROC (receiver operating characteristic) analysis considers event (disease) status and biomarker of an individual as fixed over time; however in practice both the disease status and biomarker change over time. Individuals who are disease-free earlier may develop the disease later due to longer study follow-up, and also have...

Joris de Groot, Christiana Naaktgeboren, Hans Reitsma, Carl Moons Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, 3584 CG Utrecht, Netherlands Correspondence: Joris de Groot (j.degroot-17@umcutrecht.nl) A major contributor to the rising problem of overdiagnosis, with the subsequent risk of overtreatment, is t...

Invited Speech at The 3rd Diagnostics and the Developing World Conference (DDW 2017), London

NIHR DEC London: Approach to support design of diagnostics innovation (Workshop)

An empirical analysis was performed to compare the effectiveness of different approaches to training a set of procedural skills to a sample of novice trainees. Sixty-five participants were randomly assigned to one of the following three training groups: (1) learning-by-doing in a 3D desktop virtual environment, (2) learning-by-observing a video (sh...

http://atlasofscience.org/how-much-does-abandonment-of-assistive-technology-cost-italys-national-health-service/

Workshop presentation

This paper reports a study which demonstrates the advantages of using virtual-reality-based systems for training automotive assembly tasks. Sixty participants were randomly assigned to one of the following three training experiences to learn a car service procedure: (1) observational training through video instruction; (2) an experiential virtual t...

Use of in-vitro point of care devices - intended as tests performed out of laboratories and near patient - is increasing in clinical environments. International standards indicate that interaction assessment should not end after the product release, yet human factors methods are frequently not included in clinical and empirical studies of these dev...

Background: This study was an extension of research which began in the Umbria region in 2009. Aim: To investigate the extent to which assistive technology (AT) has been abandoned by users of the Italian National Health Service (ULHS) and the reasons for this. Design: Observational study. Setting: Users who received a hearing device (HD) or m...

Purpose: The study brings together three aspects rarely observed at once in assistive technology (AT) surveys: (i) the assessment of user interaction/satisfaction with AT and service delivery, (ii) the motivational analysis of AT abandonment, and (iii) the management/design evaluation of AT delivery services. Methods: 15 health professionals and...

In a telephone interview users were asked to respond to a three-section questionnaire consisting of both open and closed questions. The questionnaire included the Italian versions of the Assistive Technology Use Follow-up Survey (ATUFS) 1.1, adapted to collect data on use and non-use of AT, and the Quebec User Evaluation of Satisfaction with Assist...

Usability and trust have been observed to be related in several domains including web retail, information systems, and e-health. Trust in technology reflects beliefs about the attributes of a technology. Research has shown that trust is a key factor for the success of different systems – e.g., e-market, e-commerce, and social networks. Trust in tec...

The consortium of the Innovate UK funded Live Augmented Reality Training Environments (LARTE) project, composed of Jaguar Land Rover (JLR), Holovis International Ltd and The University of Nottingham, developed a new concept of a 3D multiplatform training system to train the procedural skills of service maintenance operators. The LARTE tool was desi...

Short scales of user satisfaction analysis are largely applied in usability studies as part of the measures to assess the interaction experience of users. Among the traditional tools, System Usability Scale (SUS), composed of 10 items, is the most applied quick evaluation scale. Recently, researchers have proposed two new and shorter scales: the Us...

Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience (UX) analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human-computer interaction (HCI) literature the relationship between the outcomes of standardized satisfactio...

Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience (UX) analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human-computer interaction (HCI) literature the relationship between the outcomes of standardized satisfactio...

While simulation methods have proved to be very effective in identifying efficiency gains, low stakeholder engagement creates a significant limitation on the achievement of simulation modeling projects in practice. This study reports causal factors—at two hierarchical levels (i.e., primary and secondary)—that could significantly affect low stakehol...

The debate on effectiveness of virtual and mixed reality (VR/MR) tools for training professionals and operators is long-running with prominent contributions arguing that there are several shortfalls of experimental approaches and assessment criteria reported within the literature. In the automotive context, although car-makers were pioneers in the...

Il protocollo eGLU 1.0 (scaricabile dal sito http://www.funzionepubblica.gov.it/glu) è stato creato con lo scopo di individuare le aree di un sito web percepite come problematiche dagli utenti, al fine di determinare, in maniera efficace ed efficiente, in quali sezioni del sito occorra investire e coinvolgere esperti in valutazione per un’analisi a...

Background. The assistive technology assessment (ATA) process developed by Federici and Scherer [1] provides professionals with guidelines for an effective outcome in the AT selection and assignment process with the aim to match person and assistive technology (AT). Purpose. A case study open out as the ATA process is put into practice, leading the...