Conference Paper

User Experience (UX) Evaluation Based on Interaction-Related Mental Models

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

In recent years, user experience (UX) has gained importance in product development because of the increased product complexity, the availability of innovative technologies, etc. UX evaluation methods and tools developed up to now keep users’ emotions in the right consideration; nevertheless, they do not exploit mental models at best. This research aims at developing a UX evaluation method based on the so-called interaction-related mental models, a specific type of mental models focused on interaction matters. The description of the method proposed here considers also its adoption in the field and the results are compared with those obtained by a classic usability evaluation method. Although the scope of the proposed method is quite limited now (the UX evaluation focuses on CAD software packages only), the research results seem very promising. Nevertheless, this limitation will be overcome in the near future.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... They range from simple tools like the User Experience Questionnaire-UEQ (Rauschenberger, Pérez Cota, & Thomaschewski, 2013) and the IPTV-UX Questionnaire (Bernhaupt & Pirker, 2011) to more sophisticated platforms and systems to evaluate web applications like the weSPOT inquiry-based learning platform (Bedek, Firssova, Stefanova, Prinsen, & Chaimala, 2014) and the Personalized Mobile Assessment Framework (Harchay, Cheniti-Belcadhi, & Braham, 2014). There are even more articulated and complete approaches, like the Valence method (Burmester, Mast, Kilian, & Homans, 2010), the quantitative evaluation of the meCUE questionnaire (Minge & Thuring, 2018) and the irMMs-based method (Filippi & Barattin, 2017). The irMMs-based method, first starting point of this research, evaluates the quality of UX considering emotions and mental models together. ...
... The irMMs-based method evaluates the experiences of users interacting with products by exploiting interaction-related mental models (irMMs). The irMMs are mental models consisting of lists of users' meanings and emotions, including the users and products' behaviors determined by these meanings and emotions (Filippi & Barattin, 2017). The generation of an irMM comes during the user's attempt to satisfy a specific need in a specific situation of interaction. ...
Article
The literature offers methods and tools for user experience (UX) evaluation. Among them, the irMMs-based method exploits mental models related to specific situations of interaction. The literature also proposes frameworks and models to describe product development activities. One of these, the X for Design framework (XfD), allows modelling different activities and suggests modifications in order to improve them. This research aims at verifying the capabilities of the XfD in improving the irMMs-based method. Once improved thanks to the XfD, the irMMs-based method is adopted together with the original release of it and with the well-known Think Aloud usability evaluation method in evaluating the UX of a CAD software package. The comparison of the results starts demonstrating the capabilities of the XfD in improving UX evaluation activities. The research outcomes can be of interest for researchers who can exploit the XfD suggestions to deep their knowledge about human cognitive processes and for industrial practitioners who can apply the suggestions proposed by the XfD to their own evaluation activities to make them more effective.
... Product development processes need methods and tools for evaluating the UX. The literature already offers several UX evaluation possibilities, from simple questionnaires like the User Experience Questionnaire -UEQ [6] and the IPTV-UX Questionnaire [7] up to more articulated and complete approaches like the Valence method [8] and the irMMs-based method [9], the starting point of this research. The irMMs-based method considers emotions and mental models to evaluate the quality of the UX. ...
... The irMMs-based method evaluates the UX by exploiting the interaction-related mental models (irMMs). The irMMs consist of lists of users' meanings and emotions along with users' actions and products' reactions/feedback determined by these meanings and emotions [9]. The generation of an irMM comes during the user's attempt to satisfy a specific need in a specific situation of interaction. ...
Chapter
User experience (UX) evaluation and design have become important components of the product development process. The UX describes the users’ subjective perceptions, responses and emotional reactions while interacting with products or services, considering users’ emotions and cognitive activities as the basic elements of the experience. The emotions encompass physiological, affective, behavioral, and cognitive components; the cognitive activities generate and exploit the mental models that govern the human behavior. The literature offers several methods to evaluate the UX; one of them, the irMMs-based method, considers both users’ emotions and mental models to evaluate the quality of the UX. Nevertheless, its current release misses the contribution of users who already know the product under evaluation. This research aims at improving the irMMs-based method by considering also users familiar with those products. The expected benefits of this improvement refer to the completeness of the evaluation results and to the definition of relationships between these results and the evaluation activities that allow them to be discovered. All of this can be useful for both researchers and designers who are willing to increase their knowledge about the generation and exploitation of mental models and to select the most suitable evaluation activities to perform time by time depending on the characteristics of the results they are interested in and on the resources available.
... The irMMs-based method allows qualifying and quantifying the experiences of users interacting with products [2]. This method exploits interaction-related mental models (irMMs), cognitive processes that users generate before to act in order to satisfy a specific need in a specific situation of interaction [22]. Recently, the irMMs-based method reached the release 2.0 thanks to the meCUE questionnaire [3] involvement. ...
Article
Full-text available
Nowadays, User eXperience (UX) is one of the main concerns in developing innovative products. The literature already offers several UX design methods and tools. Among them, the interaction related Mental Models—based UX evaluation method appeared some years ago. Recently, the modular evaluation of key Components of User Experience questionnaire, a tool for quantifying products’ UX, integrated this method. The knowledge acquired thanks to this synergy suggested the way to define and exploit guidelines for interactive redesign of the UX of products. Because of their origin, these guidelines encompass all UX characteristics reputed as meaningful like usefulness, subjective feelings, motivational aspects, emotions, etc. Moreover, these guidelines are generic enough to be used in different situations and on different products; their structure and functioning make them independent from the methods and tools used to collect data and to implement them, widening in this way their applicability in almost any UX interactive redesign activity by almost any researcher/practitioner. The redesign suggestions offered by these guidelines, as well as the procedures to collect and analyze the data to define other guidelines and suggest the best ways to implement them, push different competencies (engineers, UX experts, psychologists, etc.) to work in synergy, just like an interactive design context requires. To date, this research defined ten guidelines; a first adoption in the field starts witnessing their goodness and effectiveness. The paper describes in detail how to define and exploit the guidelines, as well as the first adoption of them in a real redesign context.
... "The literature already offers several UX evaluation aids, from simple questionnaires, like the User Experience Questionnaire -UEQ and the IPTV-UX (User Experience for Interactive Television) Questionnaire, up to more articulated and complete methods, like the Layered Emotions Measurement Tool and the Valence method… The research described in this paper suggests a UX evaluation method based on the exploitation of both emotions and mental models" [16]. ...
Article
Full-text available
Improving IS (Information System) end-user experience is one of the most important tasks in the analysis of end-users behaviour, evaluation and identification of its improvement potential. However, the application of Machine Learning methods for the UX (User Experience) usability and effic iency improvement is not widely researched. In the context of the usability analysis, the information about behaviour of end-users could be used as an input, while in the output data the focus should be made on non-trivial or difficult attention-grabbing events and scenarios. The goal of this paper is to identify which data potentially can serve as an input for Machine Learning methods (and accordingly graph theory, transformation methods, etc.), to define dependency between these data and desired output, which can help to apply Machine Learning / graph algorithms to user activity records.
... These documents, named user guides, report the activities to do and these activities develop orderly through the twelve UX items described before. The irMMs method reached release 2.0 thanks to the adoption of the meCUE (modular evaluation of key Components of User Experience) questionnaire, specifically devoted to the quantification of the UX [23,24]. This was the final step of the irMMs method development that enabled it to deal with the whole set of UX items. ...
Article
Full-text available
Some literature has already demonstrated the widespread influence of human personality on product design. Nevertheless, most of the existing user experience (UX) design methods and tools do not fully exploit knowledge about user personality in selecting the best participants to maximize the effectiveness of the design efforts. This research tries to fill the gap by introducing PERSEL, the ready-to-use PERsonality-based SELector. PERSEL is a Microsoft Excel workbook, free to download, which allows expression of the objectives (needs) and assessment of the user personality; in turn, PERSEL suggests the best users to be involved in UX redesign activities and in what way, in order to get solutions answering to the needs in the best possible way. A comparison of the solutions generated by the first adoption of PERSEL in the field with those coming from the involvement of users selected without obeying any specific criterion, begins validating the research results, mainly in terms of PERSEL functioning and effectiveness.
Conference Paper
Full-text available
This paper outlines the early stages of development of a visual measurement instrument specifically designed to measure emotions in digital media, with the focal point being websites. For this reason, eight emotion terms most relevant to digital media were selected and visualized by an expressive cartoon figure using theories of facial caricaturing. An early prototype of the instrument will be presented in conclusion.
Article
Full-text available
Designing interactive computer systems to be efficient and easy to use is important so that people in our society may realize the potential benefits of computer-based tools .... Although modern cognitive psychology contains a wealth of knowledge of human ...
Article
Full-text available
Developer, manager and user feedback is needed to optimize products. Besides the basic Software qualities textendash usability and user experience are important properties for improving your product. Usability is well known and can be tested with e.g. a usability test or an expert review. In contrast user experience describes the whole impact a product has on the end-user. The timeline goes from before, while and after the use of a product. We present a tool that allows you to evaluate the user experience of a product with little effort. Furthermore the tool is available in different languages and we are using the new Spanish Version. We show how this tool can be used for a continuous user experience assessment.
Article
Full-text available
In this paper, we introduce a general framework for product experience that applies to all affective responses that can be experienced in human-product interaction. Three distinct components or levels of product experiences are discussed: aesthetic experience, experience of meaning, and emotional experience. All three components are distinguished in having their own lawful underlying process. The aesthetic level involves a product’s capacity to delight one or more of our sensory modalities. The meaning level involves our ability to assign personality or other expressive characteristics and to assess the personal or symbolic significance of products. The emotional level involves those experiences that are typically considered in emotion psychology and in everyday language about emotions, such as love and anger, which are elicited by the appraised relational meaning of products. The framework indicates patterns for the processes that underlie the different types of affective product experiences, which are used to explain the personal and layered nature of product experience
Conference Paper
Full-text available
This paper describes a method for formative evaluation of the user experience based on the user experience model of Hassenzahl [11]. It captures positive and negative feelings during the exploration of an interactive product. In a subsequent retrospective interview phase users indicate for each instance of a positive or negative feeling the product design aspects inducing it. This phase further employs the laddering interview technique [24] to reveal the meaning of product design aspects to the user and the underlying fulfilled or frustrated needs. The generated information helps designers to understand and optimize the user experience potential of a product.
Article
Full-text available
Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.
Article
The term user experience (UX) encompasses the concepts of usability and affective engineering. However, UX has not been defined clearly. In this study, a literature survey, user interview and indirect observation were conducted to develop definitions of UX and its elements. A literature survey investigated 127 articles that were considered to be helpful to define the concept of UX. An in-depth interview targeted 14 hands-on workers in the Korean mobile phone industry. An indirect observation captured daily experiences of eight end-users with mobile phones. This study collected various views on UX from academia, industry, and end-users using these three approaches. As a result, this article proposes definitions of UX and its elements: usability, affect, and user value. These results are expected to help design products or services with greater levels of UX. © 2011 Wiley Periodicals, Inc. This article was published online on 20 October 2011. An error was subsequently identified. This notice is included in the online and print versions to indicate that both have been corrected 13 June 2013.
Article
The study of mental representations constructed by students in their interactions with the world, its phenomena and artefacts, constitutes an important line of research in science education, according to what has been published in recent issues of the most important journals in the area. In this article we review what is being understood by mental model, conceptual model, and modelling, emphasizing the contributions this line of theory can bring to science education and research.
The Design of Everyday Things: Revised and Expanded Edition
  • D Norman
Norman, D.: The Design of Everyday Things: Revised and Expanded Edition. Basic Book, New York (2013)
Definition and exploitation of interaction-related mental models in design
  • S Filippi
  • D Barattin
Filippi, S., Barattin, D.: Definition and exploitation of interaction-related mental models in design. In: Poster contribution in the 7th International Conference on Design Computing and Cognition, (DCC 2016) (2016)