PresentationPDF Available

Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten

Authors:

Abstract

Präsentation zum World Usability Day in Osnabrück am 12.11.2020
Messung der Qualität des Benutzererlebnisses
bei der Verwendung von Sprachassistenten
Andreas M. Klein Prof. Dr. Maria Rauschenberger
Abteilung für Computersprachen und -systeme Fachbereich Technik
Universität Sevilla, Spanien Hochschule Emden/Leer, Deutschland
andreas.klein@iwt2.org maria.rauschenberger@hs-emden-leer.de
Last updated: 20-11-12
WUDOS 2020 12. November 2020
Department of Computer
Languages and Systems
University of Seville
Inhalt
2
1. Vorwort (Preface)
2. Einleitung (Introduction)
3. Methodik (Methodology)
4. Studie (Study for scale construction)
5. Ergebnisse und Diskussion (Results and discussion)
6. Fazit und Ausblick (Conclusion and future work)
7. Literaturhinweise (References)
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Preface
3
Measuring User Experience Quality of Voice Assistants
2020 15th Iberian Conference on Information Systems and Technologies (CISTI)
Andreas M. Klein Martin Schrepp
University of Applied Sciences Emden/Leer, Germany SAP SE, Walldorf, Germany
Andreas Hinderks Jörg Thomaschewski
University of Seville, Spain University of Applied Sciences Emden/Leer, Germany
DOI: 10.23919/CISTI49556.2020.9140966 Publisher: IEEE
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Preface
4
Construction of UEQ+ Scales for Voice Quality
2020 Mensch und Computer (MuC ’20)
Andreas M. Klein Martin Schrepp
University of Applied Sciences Emden/Leer, Germany SAP SE, Walldorf, Germany
Andreas Hinderks Jörg Thomaschewski
University of Seville, Spain University of Applied Sciences Emden/Leer, Germany
DOI: https://dl.acm.org/doi/10.1145/3404983.3410003 Publisher: ACM
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Preface
5
Exploring Voice Assistant Risks and Potential with
Technology-based Users
2020 16th International Conference on Web Information Systems and
Technologies (WEBIST)
Andreas M. Klein Maria Rauschenberger
University of Seville, Spain Social Computing Systems, Max Planck Institute for
Software Systems, Saarbrücken, Germany
Andreas Hinderks Jörg Thomaschewski
University of Seville, Spain University of Applied Sciences Emden/Leer, Germany
Publisher: SCITEPRESS
https://www.researchgate.net/publication/345253241_Exploring_Voice_Assistant_Risks_and_Potential_with_Technology-based_Users
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Introduction
6
Voice assistants (VAs):
oGreat future prognoses [1]
oWide range of application [2]
oGeneral-purpose assistants
§‘Adaptive Voice (Vision) Assistants’ [3]
§Siri (Apple), Alexa (Amazon), Google
Assistant (Google), Cortana (Microsoft),
Bixby (Samsung),…
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Introduction
Voice User Interfaces (VUI)
“VUI is what a person interacts with when communicating with spoken
language application.” [4]
User Experience (UX)
oISO 9241-210 [5]: Holistic concept includes all types of reactions
(emotional, cognitive, physical) before, during and after use of the
product.
oSet of distinct quality criteria [6] includes the classical usability
criteria (e.g. Efficiency) and non-goal directed criteria (e.g.
Stimulation).
7
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Introduction
Why measuring UX quality of VAs?
oGreat future prognoses, wide range of
application [1], [2]
oUseful tool for evaluating [7]
oExisting tools measure usability [8]
oConsideration of complete UX
oExploring improvements [9]
8
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Methodolody
UEQ+ Framework [10] (Modular questionnaire concept)
oContains 16 scales to measure different UX aspects.
oScales can be combined to create product-related questionnaire.
oScales type 1: User interaction with graphical user interfaces (GUI) [7]
Research gap
oUEQ+ lacks scales for VUI.
UX aspects of VUI differ significantly from those of GUI.
oHearing and voice function differently than the eyes.
Construction of scales to measure UX aspects of VAs.
9
http://ueqplus.ueq-research.org
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Methodolody
10
UX aspects of VAs
Component
User (Expectations)
- Naturalness
-Confidentiality
System (Properties)
- Functionality
- Purpose
Context
- Environment
- Purpose of activity
Scale
Response behaviour
- According to social norms
- Human conversationalist
Response quality
-Current information
- Intention of the user fulfilled
Comprehensibility
- No special formulations (syntax)
- Intention of the user recognized
Components that influence UX in HCI [11] compared to derived scales for VUIs.
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Study for scale construction
Scale construction
oPool of candidate items for the three UX aspects for VAs.
oIntegration into the UEQ+ framework scale format.
Study
oOnline questionnaire in German language.
oStudents and members of the University of Applied Sciences in
Emden/Leer (Germany).
o96 persons participated voluntarily.
oParticipant average age: 35 years (59 male, 35 female, 2 no answer)
11
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Results and discussion
Voice assistants rated by the participants:
12
0 5 10 15 20 25 30 35 40
Alexa
Siri
Google Assistant
Others
Number of participants
35
27
26
8
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Results and discussion
Factorial analysis of item sets (a total of 30 candidate items)
Principal component analysis [13] (Varimax rotation)
Assumption of three factors confirmed:
oResponse behavior loaded high on factor 2 (low on the others)
oResponse quality loaded high on factor 3 (low on the others)
oComprehensibility loaded high on factor 1 (low on the others)
Details analysis is described in the research protocol [11].
13
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Results and discussion
Response behaviour:
oSet of candidate items and loadings on factor 2 (varimax rotation)
14
No. Items
(German – Original Version)
Items
(English Translation)
Loading
1technisch menschlich technical human 0.66
2künstlich natürlich artificial natural 0.80
3fremd vertraut unfamiliar familiar 0.66
4ungewöhnlich gewöhnlich unusual usual 0.25
5langsam schnell slow fast 0.48
6unangenehm angenehm
unpleasant
pleasant 0.75
7
unsympathisch
sympathisch
unlikeable likable 0.81
8unfreundlich freundlich unfriendly friendly 0.66
9langweilig
unterhaltsam
boring
entertaining
0.68
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Results and discussion
Response behaviour:
oItems with highest loadings (factor 2) and introducing sentence:
15
In my opinion the response behaviour of the voice assistant is
artificial
!
!
!
!
!
!
!
natural
unpleasant
!
!
!
!
!
!
!
pleasant
unlikable
!
!
!
!
!
!
!
likable
boring
!
!
!
!
!
!
!
entertaining
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Results and discussion
Response quality:
oItems with highest loadings (factor 3) and introducing sentence:
Comprehensibility:
oItems with highest loadings (factor 1) and introducing sentence:
16
inappropriate
!
!
!
!
!
!
!
suitable
useless
!
!
!
!
!
!
!
useful
not helpful
!
!
!
!
!
!
!
helpful
unintelligent
!
!
!
!
!
!
!
intelligent
In my opinion the voice assistant has understood my voice commands
complicated
!
!
!
!
!
!
!
simple
inaccurate
!
!
!
!
!
!
!
accurate
unambiguous
!
!
!
!
!
!
!
ambiguous
enigmatic
!
!
!
!
!
!
!
explainable
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Results and discussion
Product-related questionnaire
oSelect relevant scales
oProduct-specific UX aspects first [14]
oFurther criteria e.g. for marketing [14]
oMore information: UEQ+ Handbook (http://ueqplus.ueq-research.org)
17
Questionnaire example 1:
Users applying smart home
Questionnaire example 2
Voice assistant customer service
§Perspicuity
§Trust
§Trustworthiness of content
§Quality of content
§Efficiency
§Perspicuity
§Dependability
§Trust
§Response behaviour
§Response quality
§Comprehensibility
Voice quality scales
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
Conclusion and future work
Conclusion:
oApproach based on UEQ+
oConstruction of VA scales
oStudy for scale construction
oThe factorial analysis confirmed:
§Response behaviour
§Response quality
§Comprehensibility
oTwo example using new VA scales.
Future work:
oValidation of scales specific to VAs.
18
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
Department of Computer
Languages and Systems
University of Seville
References
[1] Tuzovic, S. and Paluch, S. (2018). Conversational Commerce – A New Era for Service Business Development?, pages 81–100.
Springer Fachmedien Wiesbaden, Wiesbaden.
[2| Tractica (2020). Tractica. https://tractica.omdia.com/newsroom/press-releases/voice-and-speechrecognition-software-market-to-
reach-6-9-billionby-2025/.
[3] Knote, R., Janson, A., S¨ollner, M., and Leimeister, J. M. (2019). Classifying smart personal assistants: An empirical cluster
analysis. In Proceedings of the 52nd Hawaii International Conference on System Sciences.
[4] Cohen, M. H., Giangola, J. P., and Balogh, J. (2004). Voice User Interface Design. Addison Wesley Longman Publishing Co.,
Inc., USA.
[5] ISO9241-210, Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems (2010).
[6] Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction Design: Beyond HCI, 4th edn. Wiley, Chichester, 2015.
[7] Klein, A. M., Hinderks, A., Schrepp, M., and Thomaschewski, J. (2020b). Construction of UEQ+ Scales for Voice Quality. In
Proceedings of the Conference on Mensch und Computer, MuC ’20, page 1–5, New York, NY, USA. ACM.
[8] Hone, K. S., & Graham, R. (2000). Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural
Language Engineering, 6(3-4), 287303.
[9] Klein, A. M., Hinderks, A., Rauschenberger, M., & Thomaschewski, J., (2020a). Exploring Voice Assistant Risks and Potential
with Technology-based Users. In Proceedings of 16th International Conference on Web Information Systems and technology
(WEBIST), pages 1–8, SCITEPRESS.
[10] Schrepp, M. and Thomaschewski, J. (2019). Design and Validation of a Framework for the Creation of User Experience
Questionnaires. International Journal of Interactive Multimedia and Artificial Intelligence, 5(7):88–95.
[11] Hassenzahl, Tractinsky, User experience—a research agenda, Behaviour & Information Technology, Vol. 25, No. 2, March-April
2006, 91 97.
[12] Klein, A. M., Hinderks, A., Schrepp, M., & Thomaschewski, J., (2020b). Protocol for Measuring User Experience Quality of Voice
Assistants. DOI: 10.13140/RG.2.2.12816.35848
[13] Revelle, W. (2018). Psych: Procedures for personality and psychological research, Northwestern University, Evanston, Illinois,
USA, https://CRAN.R-project.org/package=psych Version = 1.8.12.
[14] Winter, D., Hinderks, A., Schrepp, M. & Thomaschewski, J., (2017). Welche UX Faktoren sind für mein Produkt wichtig? In: S.
Hess & H. Fischer (Eds.), Mensch und Computer 2017—Usability Professionals. Gesellschaft für Informatik e.V. (pp. 191–200).
19
WUDOS 2020 Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten Klein et al.
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
Voice user interfaces (VUIs) or voice assistants (VAs) such as Google Home or Google Assistant (Google), Cortana (Mircosoft), Siri (Apple) or Alexa (Amazon) are highly available in the consumer sector and present a smart home trend. Still, the acceptance seems to be culture-dependent, while the syntax of communication poses a challenge. So, there are some basic questions: 'Why do people buy VAs?' 'What do they use them for?' 'What could be improved in the future?'. We explore the opinion of a German technology-based user group to identify the challenges and opportunities of VAs. We focus on the interaction behaviour, frequency of use, concerns, and opinions of this target group as they show a higher variety of interaction as well as privacy concerns in representative population studies. Our preliminary findings confirm previous results (missing accuracy of commands and serious concerns about privacy issues) and show that technology-based users from Germany are intensive users, although with particular concerns about data collection. Probably, there is a correlation between privacy concerns and speech intelligibility as queries relating to VAs are problematic due to repetitions and refinement.
Conference Paper
Full-text available
The UEQ+ is a modular framework for the construction of UX questionnaires. The researcher can pick those scales that fit his or her research question from a list of 16 available UX scales. Currently, no UEQ+ scales are available to allow measuring the quality of voice interactions. Given that this type of interaction is increasingly essential for the usage of digital products, this is a severe limitation of the possible products and usage scenarios that can be evaluated using the UEQ+. We describe in this paper the construction of three specific scales to measure the UX of voice interactions. Besides, we discuss how these new scales can be combined with existing UEQ+ scales in evaluation projects. CCS CONCEPTS • Human-centred computing • Human computer interaction • HCI design and evaluation methods
Data
Full-text available
Klein, A. M., Hinderks, A. Schrepp, M. and Thomaschewski, J., (2020) describe the construction of three scales to measure UX aspects specific to voice systems and how these can be used in the UEQ+ framework (Schrepp & Thomaschewski, 2019) to measure UX of such systems. This research report gives detailed information about data analysis done for VUI scale construction and the first validations of the extension scales.
Article
Full-text available
Existing user experience questionnaires have a fixed number of scales. Each of these scales measures a distinct aspect of user experience. These questionnaires can be used with little effort and provide a number of useful support materials that make the application of such a questionnaire quite easy. However, in practical evaluation scenarios it can happen that none of the existing questionnaires contains all scales necessary to answer the research question. It is of course possible to combine several UX questionnaires in such cases, but due to the variations of item formats this is also not an optimal solution. In this paper, we describe the development and first validation studies of a modular framework that allows the creation of user experience questionnaires that fit perfectly to a given research question. The framework contains several scales that measure different UX aspects. These scales can be combined to cover the relevant research questions.
Article
Full-text available
Over the last decade, 'user experience' (UX) became a buzzword in the field of human – computer interaction (HCI) and interaction design. As technology matured, interactive products became not only more useful and usable, but also fashionable, fascinating things to desire. Driven by the impression that a narrow focus on interactive products as tools does not capture the variety and emerging aspects of technology use, practitioners and researchers alike, seem to readily embrace the notion of UX as a viable alternative to traditional HCI. And, indeed, the term promises change and a fresh look, without being too specific about its definite meaning. The present introduction to the special issue on 'Empirical studies of the user experience' attempts to give a provisional answer to the question of what is meant by 'the user experience'. It provides a cursory sketch of UX and how we think UX research will look like in the future. It is not so much meant as a forecast of the future, but as a proposal – a stimulus for further UX research.
Article
Full-text available
Applications of speech recognition are now widespread, but user-centred evaluation methods are necessary to ensure their success. Objective evaluation techniques are fairly well established, but previous subjective techniques have been unstructured and unproven. This paper reports on the first stage in the development of a questionnaire measure for the Subjective Assessment of Speech System Interfaces (SASSI). The aim of the research programme is to produce a valid, reliable and sensitive measure of users' subjective experiences with speech recognition systems. Such a technique could make an important contribution to theory and practice in the design and evaluation of speech recognition systems according to best human factors practice. A prototype questionnaire was designed, based on established measures for evaluating the usability of other kinds of user interface, and on a review of the research literature into speech system design. This consisted of 50 statements with which respond...
Chapter
Digitization, the rise of the Internet and mobile devices have changed the way people interact with each other and with companies. In recent years, the voice interface has become a growing feature in mobile devices. Industry reports indicate that in mid-2016, 20 percent of Android searches were voice-based and Siri received two bn. requests per week. ComScore predicts that by 2020, 50 percent of all searches will be voice searches. Furthermore, it is anticipated that voice will become the default method to control a variety of interfaces including mobile devices, Internet of Things (IoT) appliances, and automobiles.
Psych: Procedures for personality and psychological research
  • W Revelle
Revelle, W. (2018). Psych: Procedures for personality and psychological research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psych Version = 1.8.12.