PreprintPDF Available

Emotional Design

Authors:
Preprints and early-stage research may not have been peer reviewed yet.

Abstract and Figures

Emotional design has been well recognized in the domain of human factors and ergonomics. In this chapter, we reviewed related models and methods of emotional design. We are motivated to encourage emotional designers to take multiple perspectives when examining these models and methods. Then we proposed a systematic process for emotional design, including affective-cognitive needs elicitation, affective-cognitive needs analysis, and affective-cognitive needs fulfillment to support emotional design. Within each step, we provided an updated review of the representative methods to support and offer further guidance on emotional design. We hope researchers and industrial practitioners can take a systematic approach to consider each step in the framework with care. Finally, the speculations on the challenges and future directions can potentially help researchers across different fields to further advance emotional design.
Content may be subject to copyright.
CHAPTER 9
EMOTIONAL DESIGN
Feng Zhou, Assistant Professor
Department of Industrial and Manufacturing Systems Engineering
University of Michigan-Dearborn
Yangjian Ji, Professor
School of Mechanical Engineering
Zhejiang University
Roger Jianxin Jiao, Associate Professor
The George W. Woodruff School of Mechanical Engineering
Georgia Institute of Technology
To Appear in the 5th edition of the Handbook of Human Factors and Ergonomics
1
arXiv:2010.03046v1 [cs.HC] 6 Oct 2020
Contents
1 INTRODUCTION 3
1.1 WhatisEmotion ............................... 3
1.2 Emotion in Human Factors and Ergonomics . . . . . . . . . . . . . . . . 4
2 CONNECTING EMOTION TO DESIGN 5
2.1 EmotionalAssociations............................ 5
2.2 Factors Influencing Emotional Experience . . . . . . . . . . . . . . . . . 6
2.3 Models and Methods Related to Emotional Design . . . . . . . . . . . . . 8
2.3.1 Norman’s Emotional Design . . . . . . . . . . . . . . . . . . . . . 8
2.3.2 Jordan’s Four Pleasures . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.3 Kansei Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.4 Affective Computing . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.5 Emotional and Cognitive Design for Mass Personalization . . . . . 12
2.3.6 Summary ............................... 13
3 A SYSTEMATIC PROCESS FOR EMOTIONAL DESIGN 14
3.1 Affective-cognitive Needs Elicitation and Measurement . . . . . . . . . . 15
3.1.1 User Research for Needs Elicitation . . . . . . . . . . . . . . . . . 15
3.1.2 Affect and Cognition Measurement . . . . . . . . . . . . . . . . . 16
3.2 Affective-cognitive Needs Analysis . . . . . . . . . . . . . . . . . . . . . . 18
3.2.1 Qualitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.2 Quantitative Methods . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3 Affective-cognitive Needs Fulfilment . . . . . . . . . . . . . . . . . . . . . 21
3.3.1 Quality Function Deployment . . . . . . . . . . . . . . . . . . . . 21
3.3.2 Machine Learning Methods . . . . . . . . . . . . . . . . . . . . . 22
4 CHALLENGES AND FUTURE DIRECTIONS 23
4.1 Measuring Emotion and Cognition in Naturalistic Setting . . . . . . . . . 23
4.2 Integration of Affect and Cognition . . . . . . . . . . . . . . . . . . . . . 24
4.3 Emotional Design for Product Ecosystems . . . . . . . . . . . . . . . . . 25
5 CONCLUSIONS 26
References 26
2
1 INTRODUCTION
1.1 What is Emotion
The concept of emotion is closely related to affect, which is an encompassing term
and it consists of emotions, feelings, moods, and evaluations (Simon, Clark, & Fiske,
1982). The most important concept is probably emotion. Nevertheless, in psychology, the
theories about emotion consider it a ‘very confused and confusing field of study’ (Ortony,
Clore, & Collins, 1988) and thus there is no consensus on a definition. Various factors
are associated with emotions, including subjective factors, environmental factors, and
neural and hormonal processes. In this chapter, we make use of the summary of emotion
provided by Kleinginna and Kleinginna (1981), which incorporates the key elements of
definitions in psychology as follows:
(1) Emotions give rise to affective experience, such as pleasure or displeasure.
(2) Emotions stimulate us to generate cognitive explanations – to attribute the cause to
ourselves or to the environment, for example.
(3) Emotions trigger a variety of internal adjustments in the autonomic nervous system,
such as an increased heart rate and a decreased skin conductance response.
(4) Emotions elicit behaviors that are often, but not always, expressive (laughing or
crying), goal-directed (approaching or avoiding), and adaptive (removal of a potential
threat).
Feelings can be used to describe physical sensation of touch through either ex-
perience or perception, and are subjective representations of emotions, which can be
consciously felt (Davidson, Sherer, & Goldsmith, 2009). Thus, they are often used as
self-reported measures for emotions in the literature (e.g., Zhou, Qu, Helander, & Jiao,
2011).
Moods are associated with affective states with a longer duration (Picard, 1997).
They can last for hours, days, or even longer without an attributed object. Emotions
are often short-lived, but when an emotion, thought, or action, is repeatedly activated, it
can result in a mood (Russell, 2003). For instance, a negative mood can be produced by
repeated negative emotions, thoughts, actions or induced by drugs or medication (Picard,
1997).
Subjective evaluation is often defined as a valenced affective response that can assess
an object or a situation with positive or negative opinions, views, or reactions (Simon et
al., 1982).
Russell (2003) used core affect to describe all the emotionally charged events, in-
cluding emotion, mood, and evaluation. It has two important dimensions, i.e., valence
(pleasure-displeasure) and arousal (sleepy-activated). Compared to the discrete emotion
models, such as basic emotions proposed by Paul Ekman (1992), who argued that there
were six basic emotions (i.e., anger, disgust, fear, happiness, sadness, surprise) and that
they could be recognized by facial expressions across different cultures, Russell (1989) ar-
3
gued that valence and arousal were two important dimensions in the continuous emotion
model. Individual emotions can be specified with these two dimensions. For example,
excitement is characterized by positive valence and high arousal while sad is characterized
by negative valence and low arousal.
1.2 Emotion in Human Factors and Ergonomics
Traditional human factors and ergonomics (HFE) researchers mainly addressed the
physical and cognitive aspects of the human to prevent frustration, pain, stress, fatigue,
overload, injury, and death in the design, development, and deployment of products and
systems (Wickens, Gordon, & Liu, 1998; Hancock, Pepe, & Murphy, 2005). Since the
1990s, researchers in HFE have started to advocate positive experience of the human,
including flow experience (Csikszentmihalyi, 1990) and hedonomics (Helander & Tham,
2003; Hancock et al., 2005). Contrary to traditional HFE which prevented negative
aspects of human experience, positive psychology advocated positive aspects of human
experience, such as happiness, well-being, and positivity (Csikszentmihalyi & Seligman,
2000). This notion started to influence HFE and one good example is the concept of
the flow experience, in which one is so intensely absorbed and immersed in the task that
it results in positive emotions, exploratory behavior, and behavioral perceived control
(Csikszentmihalyi, 1990) during the human-product interaction. This can only happen
when the task difficulty level matches the user’s skill level with a clear set of goals and im-
mediate feedback. The view of positive psychology further influenced pioneer researchers
in HFE for pursuing hedonomics (Helander & Tham, 2003; Hancock et al., 2005; He-
lander & Khalid, 2005). It promotes pleasurable experience and individuation in the
process of human-product interaction. Pleasurable experience goes beyond safety, re-
liability, and usability to include joy, fun, and positive experience resulted from users’
appraisal, perception, and interaction with the product while individuation emphasizes
customization and personalization of the tools for individuals to optimize efficiency and
pleasure (Hancock et al., 2005). Recently, hedonomics has been proposed to reach its
fullest potential to collective goals in organizational and social contexts, such as the
workplace (Oron-Gilad & Hancock, 2017).
In addition, organizations, conferences, and special issues related to emotion and
design in HFE have also been burgeoning. In 1999, the Design and Emotion Society was
built (Desmet, 1999) with the First International Conference on Design and Emotion held
in Delft, The Netherlands. Since then, it has been held bi-annually, where researchers
and industry practitioners and leaders interact with each other in the domain of design
and emotion. At the 10-th anniversary of the International Conference on Design and
Emotion, a special issue was created in the International Journal of Design to synthesize
different design and emotion related studies (Desmet & Hekkert, 2009). In addition, the
International Conference on Kansei Engineering and Emotion Research was created in
2007 and held bi-annually to invite related researchers and industrial practitioners and
4
leaders exchange knowledge in emotion and Kansei research (Nagamachi, 1995) in product
design and development. Both the International Conference on Kansei Engineering and
the International Conference on Affective and Pleasurable Design are affiliated with the
International Conference on Applied Human Factors and Ergonomics series. Emotion
related topics on design also frequently appear in the ACM CHI Conference on Human
Factors in Computing Systems, which is the premier international conference of Human-
Computer Interaction, and recent examples include (Altarriba Bertran, M´arquez Segura,
& Isbister, 2020; Dmitrenko et al., 2020).
2 CONNECTING EMOTION TO DESIGN
2.1 Emotional Associations
Core affect is object-free without directing anything, i.e., no emotional associations,
whereas affective quality related to or belonged to the product has the ability to cause
a change in core affect during the human-product interaction process so that it can be
attributed to the product to create emotional associations (Russell, 2003; Zhou, Xu, &
Jiao, 2011). Note core affect is within the user, but affective quality lies in the product.
Similar to core affect, affective quality can also be described with valence and arousal as a
dimensional construct. Valence, as the intrinsic pleasure or displeasure, of a product fea-
ture often governs the fundamental user responses or reactions in the interaction process,
i.e., likes and attraction, which encourage approach, versus dislikes or aversion, which
lead to withdrawal and avoidance (Bradley, Codispoti, Cuthbert, & Lang, 2001; Zhou,
Xu, & Jiao, 2011). Despite the distinct personalities, emotional baggage, and unique dis-
positions, there are common psychology principles that are common to all humans that
we can use to build emotional associations (Walter, 2011), such as the baby face bias,
the golden ratio rule, and the Gestalt principles. For example, designers can make use
of the baby face bias to motivate users and high baby schema infants were considered as
more cute and elicited stronger motivation for caretaking than low baby schema infants
(Glocker et al., 2009), the golden ratio rule is widely applied in website design, such as
Twitter (Walter, 2011), and Gestalt principles of perceptual organization can make a
design coherent and orderly and, therefore, pleasant to look at (Desmet & Hekkert, 2007;
Schifferstein & Hekkert, 2011).
Arousal also influences the resulted emotional responses to human-product inter-
action. It can be defined as a psychological and physiological level of awakeness and it
can influence a person’s sensory alertness, mobility, and readiness to respond (Kubovy,
1999). Studies showed that there was an optimal level of arousal for individual task per-
formance, i.e., the inverted-U shape Yerkes-Dodson law (Yerkes, Dodson, et al., 1908).
For example, a state of high vigilance is still required in human-automation interaction in
conditional automated driving for the driver to be ready for takeover transitions (Ayoub,
Zhou, Bao, & Yang, 2019; Zhou, Alsaid, et al., 2020; Du et al., 2020). In other interac-
5
tive applications and areas, including training, learning, and gaming, an optimal level of
arousal is also important in order to maintain or prevent particular alertness for optimal
performance and positive emotions (Zhou, Qu, et al., 2011; Zhou, Qu, Jiao, & Helander,
2014; Zhou, Lei, Liu, & Jiao, 2017), which can be similar to the flow experience in the
human-product interaction process.
2.2 Factors Influencing Emotional Experience
We exam the factors that influence emotional experience using the appraisal theory
(Ortony, Clore, & Collins, 1990; Ellsworth & Scherer, 2003; Clore & Ortony, 2013), in
which human users assess stimuli with regard to their perceived significance consider-
ing their goals and needs comparing with their coping capabilities with corresponding
consequences and the compatibility of the actions with perceived social norms and self-
ideals. Under such a framework, we categorize the factors into human needs, product
quality, and ambient factors as well as their dynamic relationships, which was termed as
human-product-ambience interaction (Zhou, Xu, & Jiao, 2011; Zhou, Ji, & Jiao, 2013).
According to the motivational theory in psychology (Maslow & Lewis, 1987), human
needs comprise a five-tier hierarchy, and from the bottom upwards, they are physiological,
safety, love and belonging, esteem, and self-actualization. Correspondingly, for product
design, human needs can be divided into a similar hierarchy of needs, including functional,
reliable, usable, pleasurable, and individuation (Hancock et al., 2005; Walter, 2011). With
regard to emotional design, the higher level of user needs that go beyond the instrumental
ones (Hassenzahl & Tractinsky, 2006), i.e., affective needs, including pleasurable and
individuation, are defined in a broader perspective to focus on emotional responses and
aspirations (Jiao et al., 2007), and are deeply implanted in the lower levels of basic
needs to minimize pain and maximize pleasure, both psychologically and physically. The
strength of such pain or pleasure is built on the user’s appraisal process and ensuing
results. During the interaction process between the human user and the product, the
user evaluates whether the tasks involved are facilitating (affective) needs fulfillment. If
so, positive emotional responses can be elicited.
As mentioned earlier, good affective quality related to or within the product can
greatly satisfy affective needs by attributing positive emotional responses to the product
(features). For example, if an automated school bus is able to assure safety in transporting
children, the parents will have no anxiety or worry, but rather trust and ease (Ayoub et
al., 2020). Moreover, from affective computing’s point of view, smart products equipped
with emotion sensing capabilities may help frustrated users and prevent other negative
emotions (e.g., anger in driving) by designed interventions (Picard & Klein, 2002). For
example, in education and learning, many researchers made use of the emotional lens
to prevent negative emotional responses, optimize learning performance, and advocate
positive emotional outcomes (Yadegaridehkordi, Noor, Ayub, Affal, & Hussin, 2019).
Consistent with the flow experience (Csikszentmihalyi, 1990; Csikszentmihalyi &
6
Seligman, 2000), the appraisal theory also considers users’ ability to deal with the tasks
in the human-product interaction process by reaching, modifying, postponing, or giving
up goals or needs to modulate their emotional responses (Ellsworth & Scherer, 2003).
When one’s coping capabilities match the task challenge levels (dynamically), one is able
to sustain the flow experience continuously. Examples in HFE often compare novice users
with experienced users, old users with young users, and male users with female users, etc.
in evaluating product performance, usability, and affective quality. For instance, ordinary
use cases were compared with extraordinary use cases in order to elicit latent customer
needs that could delight customers unexpectedly (Zhou, Jiao, & Linsey, 2015). For
another example, trust in automated vehicles consisted of multiple interacting variables,
including the age of the drivers, risks, and reliability of the vehicle and younger drivers
reduced their trust significantly more than older drivers when there was automation
failures (Rovira, McLaughlin, Pak, & High, 2019).
Other particular ambient factors that can potentially influence one’s emotional re-
sponses include environmental settings and cultural differences (Zhou, Xu, & Jiao, 2011;
Zhou et al., 2013). These factors can be considered as moderator variables that can either
improve or weaken the relationships between the user factors and product factors. The
environmental settings are factors that influence where the product will be used and how
the product will be used in combination with other products. These factors affect users’
perception of product value and assessment. For example, the interior setting of a plane,
including the humidity level, the noise level, the lighting, the interior color and pattern,
can significantly influence a passenger’s flying experience (Zhou, Ji, & Jiao, 2014b). For
another example, a Kindle device is supposed to be used in various environments and
places, and the designer need to consider whether it is sensitive to various environmental
settings (e.g., light conditions, parental control for kids) (Zhou et al., 2015). In addition,
the sequence effect states that a product is positively evaluated in isolation, but can be
eventually not used or possessed due to its unfitness with other products that are pre-
viously purchased, including furniture, computer hardware and software, and appliances
(Bloch, 1995). For instance, when Microsoft rolled out the Windows Vista operating sys-
tem, its compatibility issues (e.g., the Aero interface) caused negative emotional responses
(Livingston & Thurott, 2007).
Cultural factors can also influence users’ perception on products due to the fact
that humans are socially living species. Typical examples include aesthetic stereotypes,
national shapes and colors, social rules and norms, historical beliefs, customs, practices,
and so on (Qin, Song, & Tian, 2019). For example, participants from countries with indi-
vidualistic cultures (e.g., United States, Canada, Germany, and United Kingdom) liked
angular patterns while those from countries with collective cultures (e.g., Japan, South
Korea, and Hong Kong) preferred round patterns. Not only across cultures, designers
should take cultural differences into the design process, but also across inter-generations
within one culture. For example, how young people perceive traditional cultural design
7
can significantly influence their emotional responses to and attitudes towards cultural
product design (Chai, Bao, Sun, & Cao, 2015).
2.3 Models and Methods Related to Emotional Design
2.3.1 Norman’s Emotional Design
The book Emotional Design: Why We Love (or Hate) Everyday Things by Donald
Norman (2004) described three levels of cognitive processes that give rise to emotional
associations between the human user and the product, i.e., visceral, behavioral, and
reflective (see Figure 1). The visceral level focuses on the immediate sensory reactions to
the product’s physical features (e.g., look, feel, and sound), which are directly related to
valenced reactions to the product (i.e., approach or avoidance). Users’ visceral reactions
to a product are wired in and the design principles tend to be universal. This is consistent
with the baby face bias, the golden ratio rule, and the Gestalt principles mentioned above
and good visceral design needs skilled visual and industrial designers. For example, Park,
Lee, and Kim (2011) explored a new interactive touch system on a mobile touch screen
by making use of the weight factor in the Laban’s Effort system, and they found that it
significantly improved the physical feel of the interface emotionally at the visceral level.
Visceral design is prevalent in industries like automotive (e.g., Mini Cooper and Tesla
vehicles), electronics (e.g., iWatch and MacBook Pro), packaging design, and so on.
Visceral level:
Subconscious
Biologically determined
Attractiveness
Rapid judgement
First impression
inhibit or
enhance
inhibit or
enhance
Behavioral level:
Subconscious
Influenced by training
Usability
Understandability
Performance
Function
Reflective level:
Conscious
Influenced by experience
Extending much longer
Value
Meaning
Culture
Figure 1: The key features involved in Norman’s emotional design
The visceral level informs the behavioral level and the user subconsciously evaluates
the design in terms of whether it helps complete goals with effectiveness, efficiency, and
satisfaction. Behavioral design aims to improve human-product interaction, focusing on
usability, performance, and function. Traditional HFE emphasizes heavily on usability
and performance and in this sense, behavioral design tends to be consistent with human-
centered design in that it puts user needs foremost (Norman, 2013). Many user research
methods (Baxter, Courage, & Caine, 2015) in human-centered design are useful to dis-
8
cover user needs for good behavioral design, such as observation, ethnography, contextual
inquiry, and scenario-based design. Thus, good behavioral design begins with understand-
ing user needs, generating ideas, testing concepts, and obtaining feedback, and iteratively
refines the product. For example, an autonomous system was designed for school buses
using human-centered design in order to meet the needs of the parents (e.g., trust) and
kids (e.g., fun, safety) at the same time (Ayoub et al., 2020). From affective computing
point of view, systems that use behavioral (e.g., facial expressions) (Zhou, Kong, Fowlkes,
Chen, & Lei, 2020) and physiological measures (e.g., heart rate, galvanic skin response
(GSR)) (Zhou, Qu, et al., 2011) to continuously monitor the human-product interaction
process can potentially respond to interaction issues to improve behavioral design. For
example, multiple physiological measures were used to monitoring driver states in order
to improve in-vehicle system usability (Zhou, Ji, & Jiao, 2014a).
With accumulation of the interaction between the user and the product, at the
reflective level, the user consciously assesses the benefits, values, culture, and meaning
brought by the product, which often forms emotional bonds between the user and the
product. At this reflective level, the real value of the product can be way beyond the value
at the visceral and behavioral levels by meeting people’s affective needs, and establishing
their self-image and identity in the society. A good example was described in (Helander,
Khalid, Lim, Peng, & Yang, 2013) about the user’s emotional intent or desire for vehicles
and for a long time, consumption of vehicles is always more than just rational economic
choices and it connects the users by aesthetic, emotional, and sensory responses to driving
and symbolic relationships at both the social and cultural levels (Sheller, 2004). Unlike
the previous two levels, users consciously evaluate the product at the reflective level, the
real values are influenced by knowledge, experience, and culture to a great extent. For
example, many special objects are associated with personal experience and memories of
their own, which are often not the objects themselves, but rather the relationships and
attachment to them as described in the book The Meaning of Things by (Csikszentmihalyi
& Halton, 1981). In addition, at the reflective level, users can sometimes forgive the
negative experience involved at the visceral or behavioral levels. For example, long term
customer experience and loyalty can often be sustained if good customer services are
provided along the customer journeys by fixing defects in the initial interactions with
the product, integrating multiple business functions, and creating and delivering positive
customer experience (Lemon & Verhoef, 2016).
2.3.2 Jordan’s Four Pleasures
Jordan (2000) considered products as living objects that could elicit both positive
and negative emotional responses and products should be designed to be useful, usable,
and pleasurable. He proposed four pleasures, i.e., physiological, social, psychological, and
ideological, to support pleasurable design. Physiological pleasure refers to pleasures gener-
ated from sensory responses, including visual, auditory, tactile, olfactory, and so on, which
9
seem to be consistent with visceral design in Norman’s emotional design. One example in
vehicle related research is to make use of the odors inside the vehicle (e.g., rose compared
to leather) to reduce the effect of visually induced motion sickness to improve physio-
logical pleasure (Keshavarz, Stelzmann, Paillard, & Hecht, 2015). Social pleasure is the
enjoyment as a result of social interaction with others using the product as the medium.
Direct examples are popular social media apps nowadays. Others can be the talking
points involved in the social interactions, such as smart speakers, and those indicate
users’ specific social groups, such as Porsches for ’yuppies’ (Jordan, 1997). Psychological
pleasure is derived from conducting and accomplishing a task through human-product
interaction, which tends to be similar to the behavioral level in Norman’s emotional de-
sign. It focuses on enjoyment as a result of achieving tasks with usable products. For
example, an assistance system was developed for the elderly to aid their activities in daily
living and due to its proactive and case-driven characteristics, it was usable and pleasur-
able at the same time (Zhou, Jiao, Chen, & Zhang, 2010). Ideological pleasure is related
to personal aspirations and values and is derived from artistic products, such as books,
music, movies, and products that embody their values. For example, some consumers
were willing to buy sustainable and organic foods due to social identity and attitudes
towards environmental responsibility (Bartels & Onwezen, 2014). Thus, products that
embed such values can be popular among these consumers.
2.3.3 Kansei Engineering
Kansei engineering was originated in Japan as early as in the 1970s and it maps
users’ Kansei into product attributes in the design process using engineering methods,
where Kansei is defined as the state of mind where knowledge, emotion, and passion are
harmonized (Nagamachi, 1995). The key questions in Kansei engineering are 1) how
to understand Kansei accurately, 2) how to reflect and translate Kansei understanding
into design elements, and 3) how to create a system and organization for Kansei-oriented
design (Nagamachi & Lokman, 2016). Although Kansei can be represented with different
forms, adjectives are most frequently used (Zhou, Jiao, Schaefer, & Chen, 2010) with
semantic differential scales (e.g., simple - complex, spacious - narrow, boring - interesting)
(Osgood, May, Miron, & Miron, 1975).
There are three major types of Kansei engineering methods. The type I method uses
a tree structure to decomposes the 0-Order Kansei concept into nth order sub-concepts
until these sub-concepts can be mapped to physical design elements without difficulty.
The success of this method not only depends on the understanding of user Kansei, but also
the decomposition of design elements that form the product. For example, a speedometer
was decomposed into meter layouts, meter types and numbers, panel colors, materials,
and so forth, to match user Kansei, and the contribution of each design element to specific
Kansei was identified by the partial correlation coefficients based on subjective evaluation
with semantic differential scales (Jindo & Hirasago, 1997). The type II method uses
10
expert systems to automatically map Kansei sub-concepts to physical design elements by
constructing a Kansei database, which allows the designers to understand user Kansei
better. The type III method uses hybrid mapping, i.e., forward mapping from Kansei to
design elements and backward mapping from design elements to Kansei. The backward
mapping starts from the designers and the mapping relationships can then be revised
and validated by user evaluation. For example, Zhou et al. (2010) used both K-optimal
rule discovery for forward Kansei mapping (from Kansei to design elements) and ordinal
logistic regression for backward Kansei mapping (from design elements to Kansei) to
support truck cab interior design. Other methods were also proposed in order to deal
with the issues involved in the previous three types, such as uncertainty of user Kansei,
product element presentation (e.g., virtual reality), and effectiveness of expert systems
(Marghani, da Silva, Knapik, & Verri, 2013). For example, a deep learning method based
on long short-term memory was used to extract user Kansei from online product reviews,
which improved the efficiency and effectiveness of understanding user Kansei and reduced
uncertainty involved in user Kansei (W. Wang, Wang, Li, Tian, & Tsui, 2019).
2.3.4 Affective Computing
Picard (1997) coined the term affective computing in 1997, which aims to design
and develop systems that can recognize, interpret, respond to, and simulate human emo-
tions. This is consistent with the view that emotional intelligence is one of the basic
components of intelligence (Goleman, 1995). There are two major areas of research in
affective computing, including 1) recognizing and responding to user emotions, i.e., affect
sensing, and 2) simulating emotions in machines, affect generation, in order to enrich and
facilitate interactivity between humans and machines.
First, affect sensing refers to a system that can recognize emotion by collecting
data through sensors and building algorithms to recognize emotion patterns (Picard,
1997) based on Ekman’s discrete emotion model (Ekman, 1992) or Russell’s dimensional
emotion model (Russell, 2003). According to the component process model of emotion
(Scherer, 2005), many researchers used psychophysiological signals (e.g., GSR, electroen-
cephalogram (EEG), heart rate), facial and vocal expressions, and/or gestures to recog-
nize emotions. For example, GSR, facial electromyography (EMG), and EEG were used
to predict emotions using a machine learning technique named rough set to recognize
seven discrete emotions (Zhou, Qu, et al., 2011, 2014). Recently, deep learning models
were also used to recognize emotions, such as bi-level convolutional neural networks for
fine-grained emotion recognition using Russell’s dimensional emotion model (Zhou, Kong,
et al., 2020). By recognizing and monitoring users’ emotions, the system can respond to
the users to improve learning in education (Wu, Huang, & Hwang, 2016), communica-
tions for autistic children (Messinger et al., 2015), and video gaming (Guthier, D¨orner,
& Martinez, 2016), to name but a few.
Second, many researchers simulate human emotions in social robots and virtual
11
agents to optimize the interaction between human-robot/agent interaction. The capa-
bilities of recognizing and expressing emotions assign characteristics to social robots and
virtual agents, which can form impressions during social interactions, especially when the
non-human entities are human-like, i.e., anthropomorphism (Eyssel & Kuchenbrandt,
2012). These social robots and agents can be widely applied in offices, hotels, educa-
tion, personal assistants, avatars, entertainment, nursing care, therapy, and rehabilitation
(Dautenhahn, 2002; Breazeal, 2011; Thalmann, Yumak, & Beck, 2014). For example, a
previous study showed that social robots were used as tutors or peer learners, which
achieved similar cognitive and affective outcomes compared to human tutors (Belpaeme,
Kennedy, Ramachandran, Scassellati, & Tanaka, 2018).
2.3.5 Emotional and Cognitive Design for Mass Personalization
Mass personalization is a strategy of producing goods and services to meet individual
customers’ latent needs and the surplus is positive both for customers and producers con-
sidering both the values and costs associated (Kumar, 2007; Zhou et al., 2013). Note this
is different from mass customization, which aims to customize products and services for
individual customers at a mass production price (Tseng & Jiao, 2001). The major differ-
ences are 1) mass personalization is fulfilled at the personal level, i.e., market-of-one with
customer co-creation (e.g., Netflix movie recommendation), while mass customization is
fulfilled for a certain market segment, i.e., market-of-few, with customer configuration
(e.g., Apple computer configuration), 2) mass personalization emphasizes on high-level
non-instrumental needs, including cognitive needs and emotional needs with values out-
performing costs, while mass customization focuses on functional needs with near mass
production efficiency, and 3) mass personalization is usually producer-initiated to delight
customers with a surprise while mass customization is mostly user-initiated within the
configuration defined by the producer. Furthermore, mass personalization is not person-
alization per se, but personalization with affordable fulfilment costs for both customers
and producers (Kumar, 2007). Many of the personalization techniques are now based
on big data analytics and artificial intelligence and once the algorithms are developed,
the costs associated with them tend to be minimal to provide personalized, satisfactory
services for the majority of users (Alkurd, Abualhaol, & Yanikomeroglu, 2020). What
mass personalization emphasizes is latent customer needs that users might not be aware
of (Zhou et al., 2015), mainly including affective and cognitive needs according to their
profiles, behavioral patterns, affective and cognitive states, aesthetics preferences, and
so on (Zhou et al., 2013). We have explained affective needs above. Cognitive needs
are those non-functional requirements of how products and systems are designed to ac-
commodate human cognitive limitations (Zhou, Xu, & Jiao, 2011), which are similar to
what behavioral design addresses in Norman’s emotional design (Norman, 2004). Under
the framework of mass personalization, we aim to integrate both affective and cognitive
needs to create positive user experience throughout the product life cycle.
12
2.3.6 Summary
Table 1: Summary of different models related to emotional design
Model Focus Major Methods Advantages Limitations
Emotional
Design
Visceral,
behavioral,
reflective
Human-centered
design methods,
industrial design
Solid theory
support
in psychology
No straightforward
mapping from
three designs
to specific product
parameters
Four
Pleasures
Physiological,
social,
psychological,
ideological
pleasures
Human-centered
design methods
(user research),
Pleasure analysis
A framework of
four pleasures
and applicable
qualitative
research
methods
Similar to the
limitations
of emotional
design
Kansei
Engineering
Translate
Kansei
into product
elements
Subjective
evaluation,
expert systems
Widely applied
in Japan
with successes
Uncertainty of Kansei,
Averaged Kansei for
sampled participants,
Cultural barriers to be
applicable in other
countries
Affective
Computing
Emotion
recognition,
Emotion
generation
Machine
learning,
artificial
intelligence
Development of
deep learning
models
Heavily dependent on
models trained on
a specific dataset,
privacy and moral issues
Mass
Personalization
Personalized
products for
individuals
affective and
cognitive needs
Engineering design
methods, machine
learning models,
human-centered
design method
Solid support in
engineering design
and human factors
engineering
Only applicable for
certain products
and services,
May need big data
for personalization
By reviewing multiple models and methods related to emotional design, we summa-
rize them in Table 1. Emotional design and the four-pleasure framework are deeply rooted
in human-centered design and go beyond it to include fun and pleasure. Other meth-
ods include sustainable design, participatory design, and even universal design. Thus,
both can make full use of many qualitative methods in human-centered design, which
are useful to incorporate affective needs in the design process. However, they do suffer
some limitations, including 1) research quality can heavily depend on researchers’ skills
with subjectivity, 2) data analysis can be time-consuming, 3) results can be difficult to
verify, and 4) there is no straightforward mapping between design elements and affective
needs (Zhou, Jiao, Schaefer, & Chen, 2010; Anderson, 2010). Kansei engineering has
been widely applied in Japan with successes in different areas, such as automotive indus-
try, cosmetics, and clothing (Nagamachi & Lokman, 2016). However, the subjectivity,
uncertainty, and cultural barriers associated with Kansei have restricted its applications
to other countries. In addition, the designed product is often the result of the averaged
Kansei of the sampled participants with perceptual preferences although other presenta-
tion methods have been proposed, such as virtual reality (Marghani et al., 2013). The
data collection process is often time-consuming with active participation of customers and
13
researchers (W. Wang et al., 2019). Affective computing mainly uses machine learning
and artificial intelligence techniques for the machine to recognize and simulate emotions.
With the development of deep learning techniques, more sophisticated and successful
models have been built (e.g., Zhou, Kong, et al., 2020). However, the models trained on
a specific dataset can be unfair for those who are less representative (e.g., black people)
in the dataset (Lohr, 2018). The privacy and moral issues associated with giving the
machine the capabilities to monitor and intervene users’ emotional states are still under
debate (Daily et al., 2017). Mass personalization can reuse many methods in engineer-
ing design for all the steps involved and many user research methods in human-centered
design can also be used, especially for affective-cognitive needs elicitation. It is built on
top of mass customization and incorporates affective and cognitive needs from HFE and
thus is complementary to mass customization. However, mass personalization tends to be
applicable to the ’soft’ characteristics of the product that are changeable and adaptable
to personalize individual customers, such as those that create the experience of drinking
coffee in a certain store though it is built on the ’hard’ components of the product that
can be configurable (i.e., mass customization), such as the coffee cups, beans, and other
ingredients (Zhou et al., 2013).
3 A SYSTEMATIC PROCESS FOR EMOTIONAL DESIGN
Affective-cognitive
needs fulfillment
Affective-cognitive
needs elicitation
Affective-cognitive
needs analysis
User research methods
Affect and cognition measurement
Stakeholders, goals, constraints
identification
Formal representation and
synthesis of needs
Idea generation and selection
Expert systems and
machine learning models
Engineering design methods
Figure 2: The proposed three-step process for emotional design
By examining the advantages and disadvantages of different models and methods
related to emotional design, we propose a three-step systematic process based on mass
14
personalization and human-centered design to transform customers’ affective and cogni-
tive needs from the customer domain into design elements in the designer domain, includ-
ing affective-cognitive needs elicitation, affective-cognitive needs analysis, and affective-
cognitive needs fulfillment (Zhou et al., 2013) as shown in Figure 2. The first step aims to
elicit affective and cognitive needs of customers systematically, and many user research
methods can be applied in this step. One of the key issues is how to measure affect and
cognition constructs involved in affective and cognitive needs. At the same time, this step
also identifies the involved stakeholders (e.g., customers and manufacturers), goals, use
cases, and constraints. The second step aims to understand affective and cognitive needs
and transform them into explicit requirements for engineers and marketers. Formal rep-
resentations should be used to synthesize the needs from the first step, concepts should
be generated and selected based on the priorities of customer needs. The last step aims to
identify the mapping relationships between the requirements and product specifications
with an iterative process of prototype testing. This three-step process itself should also
be iterative to refine the product. At the same time, many machine learning models and
expert systems involved in affective computing and Kansei engineering can also be used
to support emotional design. The related work is reviewed below.
3.1 Affective-cognitive Needs Elicitation and Measurement
3.1.1 User Research for Needs Elicitation
Many user research methods in human-centered design have been proposed for af-
fective and cognitive needs elicitation (Baxter et al., 2015). For example, in order to
understand the cognitive needs of healthcare workers when designing medical software,
Johnson and Turley (2006) used a think-aloud protocol. A diary study (with introductory,
mid-study, and final interview with each participant, spaced 7 days apart) was conducted
over two weeks to understand the informational needs of mobile phones (Sohn, Li, Gris-
wold, & Hollan, 2008). Observation was used in public transportation, such as trains,
in order to understand user needs to support none-driving related tasks in automated
vehicles (Pfleging, Rang, & Broy, 2016). Contextual inquiry was used to gain a deeper
understanding of how drivers interact with vehicles’ infotainment systems to create pos-
itive driver-vehicle interactive experience (Gellatly, Hansen, Highstrom, & Weiss, 2010).
For more examples, please refer to (Baxter et al., 2015).
For affective needs elicitation, Ng and Khong (2014) reviewed various methods for
affective human-centered design for video games and proposed two types of methods,
including user-feedback methods (e.g., focus group, survey, interviews, usability testing
methods) and non-intrusive methods (e.g., observation on facial and vocal expressions,
physiological sensors). Then, Ng, Khong, and Nathan (2018) applied interviews as a
user-feedback method and observation as a non-intrusive method to affective video game
design, where the interviews were used to measure subjective feelings while the observa-
15
tion was used to understand participants’ emotional responses during their game playing.
Many researchers in Kansei engineering applied surveys, questionnaires, and focus groups
to collect Kansei from users (W. Wang et al., 2019). For example, Quan, Li, and Hu (2018)
used questionnaires to elicit Kansei by reviewing clothes images from both designers and
consumers. Kwong, Jiang, and Luo (2016) utilized conjoint and lead user surveys to
understand customer Kansei of electric irons. Akay and Kurt (2009) interviewed users
and surveyed magazines to understand customer Kansei of mobile phones. The sample
sizes in these Kansei studies were relatively small (<20) and in order to reduce the
possible subjective biases (Pryzant et al., 2020), online product reviews can be readily
collected from websites (e.g., Amazon.com) in large quantities. For example, a large
amount of review data for Kindle tablets and Amazon product ecosystems were crawled
from Amazon to understand reviewers’ emotional responses and satisfaction (Zhou et al.,
2015; Ayoub, Zhou, Xu, & Yang, 2019; Zhou, Ayoub, Xu, & Jessie Yang, 2020). Human
agents or avatars are also used to elicit emotional responses for interactive interfaces. For
example, an affective avatar was designed based on a human–avatar taxonomy to address
social communication disorders (E. Johnson et al., 2018). Wizard-of-oz methods were
used in automated driving by hiding the drivers to elicit emotional responses and natural
behaviors of the passengers (Ayoub et al., 2020) and pedestrians (Currano et al., 2018).
Methods that can elicit both affective and cognitive needs at the same time are reported,
too. For example, Coursaris and van Osch (2016) proposed a cognitive-affective model of
perceived user satisfaction and used experiment design by manipulating colors in website
design to understand participants’ perceived cognitive (effectiveness and efficiency) and
emotional (aesthetics and playfulness) responses.
3.1.2 Affect and Cognition Measurement
Subjective Measures: These methods are probably the most frequently used ones
to measure affect (e.g., emotional responses and feelings) and cognition (e.g., cognitive
workload) with efficiency. For example, in affective computing, in order to train a model
that can recognize emotions, the ground truth used as labels in training is often produced
by subjective self-reports. For example, Zhou et al. (2011, 2014) used participants’ self-
reported emotional responses to static images and sound clips as labels to train machine
learning models. One of the possible issues is the forced-choice method among a list
of discrete emotions leading to a relative judgement, which may not necessarily reflect
the participant’s real emotional responses (Russell, 1993). In order to obtain a reliable
set of labels, crowd-sourcing labeling with multiple workers can be useful. For example,
Barsoum et al. (2016) used multiple crowd workers to label each facial expression image
and found the agreement increased from less than 40% with 3 workers to over 90% with
9 workers. For dimensional emotion recognition, the reliability is often measured among
individual raters, such as AffectNet (Mollahosseini, Hasani, & Mahoor, 2017).
In Kansei engineering, researchers often applied semantic differential scales with
16
participants’ self-reported measures (Nagamachi & Lokman, 2016). For example, Lu and
Petiot (2014) applied semantic differential scales in designing eyeglasses. Other useful
methods include self-assessment manikin on valence, arousal,and dominance (Bradley &
Lang, 1994), the product emotion measurement instrument (PrEmo) using animations of
cartoon characters with a small number of basic emotions (Desmet, Hekkert, & Hillen,
2003), and the experience sampling method to collect daily experience over a longer
period of time (Larson & Csikszentmihalyi, 2014). For instance, the self-assessment
manikin instrument was applied to measure valence and arousal and their influence on the
interaction between drivers and automated vehicles (Du et al., 2020) and the experience
sampling method was used to identify the antecedents of daily positive emotions (Goetz,
Frenzel, Stoeger, & Hall, 2010).
In order to measure cognitive constructs, many survey-based tools have been de-
veloped. The most frequently used survey tools for measuring cognitive workload are
probably the NASA Task Load Index (Hart, 2006), the Workload Profile (Tsang & Ve-
lazquez, 1996), and the Subjective Workload Assessment Technique (Reid & Nygren,
1988). For example, Ayoub and Zhou (2020) used the NASA Task Load Index to mea-
sure cognitive workload of automated vehicle interfaces in the context of lane changing
events. Rubio et al. (2004) compared these three tools and found all of them had good
validity, but the Workload Profile had better sensitivity and diagnostic powers.
Objective Measures: Subjective measures are easy to implement. However, they
tend to be susceptible to subjective biases (Pryzant et al., 2020). Objective measures,
such as behavioral and physiological measures, are less susceptible to voluntary control.
The typical behavioral measures include facial and vocal expressions, poses, and gestures
while physiological measures include eye tracking data, GSR, heart and respiration activ-
ity, EMG, EEG, and so on. For example, eye tracking data was used to measure attention,
GSR was used to measure arousal, and heart rate and heart rate variability were used to
measure cognitive workload in the interaction between the driver and automated vehicles
(Du et al., 2020). Facial expressions were used to understand participants’ dimensional
emotional states (Zhou, Kong, et al., 2020) and trust in human-automation interaction
(Neubauer et al., 2020). Due to the fact that one measure is not able to reliably mea-
sure emotion or cognition, many researchers often use multiple measures together. For
example, both facial expressions and head poses were used to measure emotions and
understand human interaction by visualizing depth information (Kalliatakis, Stergiou, &
Vidakis, 2017). GSR, respiration rate, facial EMG, and EEG were used to measure partic-
ipants’ emotional responses to visual and auditory stimuli (Zhou, Qu, et al., 2011, 2014).
Koelstra et al. (2011) used 32 channels of EEG data, electrooculography, zygomaticus
major EMG, trapezius EMG, GSR, respiration, plethysmograph, and peripheral skin
temperature to measure participants’ emotional responses. Capitalizing on this dataset,
many researchers applied machine learning models to recognize emotions, (e.g., Piho &
Tjahjadi, 2018; Cui et al., 2020).
17
From the behavioral point of view, another important tool in understanding the cog-
nitive demands, thought processes, knowledge, and goals is cognitive task analysis, which
combines the features of the work domain and the cognitive demands imposed on the user
(Schraagen, Chipman, & Shute, 2000). There are many techniques (including subjective
methods) used in cognitive task analysis, such as critical incident/decision analysis, cog-
nitive field observation, hierarchical task analysis, sequence analysis, knowledge audit,
and so on (Crandall, Klein, Klein, Hoffman, et al., 2006). For example, main display
patterns and themes were identified using cognitive task analysis to support software
design (Pfautz & Roth, 2006). Zanesco (2020) applied sequence analysis to understand
the dynamic thought process using time series data across different cognitive tasks.
3.2 Affective-cognitive Needs Analysis
Some of the elicitation methods mentioned above have their analysis components,
such as cognitive task analysis (Crandall et al., 2006). Since affect and cognition tend to
be qualitative in nature, many of the methods for affective-cognitive needs analysis are
qualitative methods and were used in human-centered design, such as grounded theory
(Strauss & Corbin, 1994) and affinity diagram (Spool, 2004). However, qualitative meth-
ods can be time-consuming when the data amount is large. With the development of
big data and machine learning techniques, many efficient and quantitative methods are
proposed.
3.2.1 Qualitative Methods
Qualitative methods can potentially produce different types of representation of
needs, such as profiles, patterns and themes, importance and priorities, concepts and
classifications, and so on. Among many, personas are often created as profiles of archetype
users in order to analyze users’ needs by scrutinizing their goals, needs, wants, and pains,
based on which different scenarios can be created for activities of empathetic role-play
(Pruitt & Adlin, 2010). For example, personas were used to analyze emotional needs for
automated public transportation services in a multi-stakeholder context (Kong, Cornet,
& Frenkler, 2018). Affinity diagram, developed by Jiro Kawakita and also named the
KJ method, is widely used for customer needs analysis in terms of identifying themes
and patterns and assigning importance and priority (Spool, 2004). For example, over 800
Kansei words were grouped into 43 clusters using an affinity diagram for web design in four
different steps, including initial study, exploratory study, KJ method, and confirmatory
study (Lokman & Kamaruddin, 2010). Ayoub et al. (2020) also used an affinity diagram
to prioritize the emotional needs of parents and kids based on personas for designing
an automated school bus. Another important method is grounded theory (Strauss &
Corbin, 1994) and it aims construct theories and identify patterns and themes using
specific coding schemes of the data systematically, especially text data generated in the
18
elicitation process. For instance, Brown and Cairns (2004) used grounded theory to
categorize immersion into engagement, engrossment, and total immersion for game design,
and such grouping was able to be applied to software design. Zhou, Yang, and Zhang
(2020) applied grounded theory to code comments on YouTube videos of automated
driving and identified major human factors issues of automated vehicles. In addition,
both task analysis and cognitive task analysis can be used to identify themes and concepts
to support decision making. For example, a task-based needs analysis was proposed
to obtain insights from task selection, task discourse analysis, task difficulty, and task
sequencing for designing foreign language instructions (Malicka, Gilabert Guerrero, &
Norris, 2019). The critical decision method as an approach to cognitive task analysis
was used to identify a list of critical cues and judgements, such as action, knowledge,
appraisal, and anticipation in training (Hoffman, Crandall, & Shadbolt, 1998).
Graphic methods are also useful in affective-cognitive analysis, such as mind maps,
concept maps, and cognitive maps. A mind map can be used to visually represent hi-
erarchical relationships among pieces of information, usually with one focus (Hopper,
2012). For example, individual mind maps from each participant were used to represent
Kansei words and then an overall mind map was developed by aggregating individual
ones (Huang, Chen, Wang, & Khoo, 2014). Unlike mind maps, a concept map is defined
as a relationship diagram using labeled arrows (e.g., ”consist of”, ”give rise to”) to con-
nect different concepts in a hierarchical structure (Novak & Ca˜nas, 2006). For example,
concept maps were used to represent knowledge of retired NASA engineers to help train
novices (Coffey & Carnot, 2003). Cognitive maps use causal links to represent concepts
and it was used to represent the decision-making process of different team members in
new product design (Carbonara & Scozzi, 2006). Fuzzy cognitive maps incorporate fuzzi-
ness involved in the relationships between concepts and were used to capture the causal
reasoning process in geographic information system design (Liu & Satur, 1999).
When affective and cognitive needs are synthesized, potential solutions can be gen-
erated to satisfy these needs. In human-centered design, idea generation or brainstorming
aims to generate as many ideas as possible in the first place and then systematic analysis
can be done to select the optimal candidates. Sketching is widely used in idea generation
because it is quick, inexpensive, disposable, plentiful, with distinct gestures and minimal
details (Buxton, 2010). Scribble sketching can rapidly sketch the idea anytime, anywhere
and is used not only to generate ideas but also collect existing ideas while 10puls10 aims
to generate 10 or more ideas and then select the most promising one to generate 10
detailed variations (Greenberg, Carpendale, Marquardt, & Buxton, 2011). While these
techniques are widely used in idea generation, many researchers tend to develop new
sketching tools, especially digital ones. For example, Spatial Sketch was developed as a
3D sketch application to combine physical movement and object fabrication in the real
world using cut planar materials (Willis, Lin, Mitani, & Igarashi, 2010). DataToon in-
cluded elements of comics to create data-driven storyboards that blended analysis and
19
presentation with pen and touch interactions (N. W. Kim et al., 2019). In order to gen-
erate a large number of ideas, crowdsourcing (Majchrzak & Malhotra, 2013) has been
widely used. For example, Schuurman et al. (2012) explored crowdsourcing for idea gen-
eration and selection for smart city innovation. In order to deal with a large volume of
the ideas produced by crowd sourcing, Hoornaert et al. (2017) identified three sources to
select ideas, including the content, the contributor, and the crowd’s feedback on the idea.
Faste et al. (2013) proposed new ideas using digital collaborative ideation, including
chainstorming (i.e., passing ideas along the communication chain), cheatstorming (i.e.,
brainstorming without generating ideas), and tweetstorming (i.e., a digital chainstorm-
ing that used cheatstorming) and found that brainstorming was not only pooling existing
ideas but also involving the sharing and interpretation of concepts in unintended and
unanticipated ways.
3.2.2 Quantitative Methods
One of the limitations of the qualitative methods mentioned above is that the data
analysis process tends to be laborious and subjective. For example, qualitative persona
methods tend to be time-consuming if the designers have to examine a large number of
participants. In order to overcome this issue, Hence, Zhang, Brown, and Shankar (2016)
proposed a quantitative method by making use of the click streams from 2400 users
and a hierarchical clustering model to identify typical personas in order to improve user
experience. The grounded theory and affinity diagram also suffer from similar issues for
analyzing a large amount of text data. In order to automate this process, recently, natural
language processing techniques have achieved great successes in various tasks, especially
for those with deep learning models (Devlin, Chang, Lee, & Toutanova, 2018). Zhou et
al. (2020) used latent Dirichlet allocation to identify the topics from over 90,000 online
reviews on Amazon product ecosystems and sentiment analysis was used to automatically
predict their sentiment polarity and intensity. Wang et al. (2019) proposed a heuristic
deep learning model to automatically generate multiple Kansei pairs by mining a large
number of online product reviews.
Other quantitative methods aim to group, quantify, or prioritize affective and cog-
nitive needs with statistical criteria, which can potentially overcome subjective biases
(Pryzant et al., 2020) involved in the qualitative methods. For example, principal com-
ponent analysis was used to identify the major Kansei concepts among the collected data
(Barnes & Lillford, 2009) and a Kansei clustering method was proposed with design struc-
ture matrices, where partial correlation coefficients were used as the distances between
Kansei adjectives (Huang, Chen, & Khoo, 2012). Similarly, a combination of design
structure matrices and genetic algorithms was used to identify the connections between
Kansei for optimal clusters (Yang, Chen, Gu, Gu, & Yu, 2016). Conjoint analysis was
used to measure utilities of product profiles linked to the affective needs of truck cabs
based on ratings on Likert scales (Jiao et al., 2007).
20
Affective and cognitive needs are also associated with different levels of uncertainties
and ambiguities due to their qualitative nature. Researchers also proposed quantitative
methods to deal with this issue. For example, due to the individual differences among par-
ticipants, Kansei individuality was modeled using fuzzy set theory (Nakamori & Ryoke,
2004). Grey relationship degree analysis was used to identify the priority of Kansei adjec-
tives (Kang, 2020b). Fuzzy product rules based on rough set were used in order to deal
with the uncertainty, complexity, and dynamics associated with user experience model-
ing and quantification, which included both affective needs and cognitive needs (Zhou,
Jiao, Xu, & Takahashi, 2011). Zhai et al. (2009a, 2009b) made use of the rough num-
bers in rough set to model the uncertainties involved in affective and cognitive needs to
produce reliable priorities. Li et al. (2017) applied evidence theory’s reliability indices
(e.g., support and confidence) to the rules generated by rough set using neural networks
to improve precision in Kansei knowledge. Su et al. (2020) used convolutional neural
networks to identify the importance of different Kansei attributes to overcome subjective
biases (Pryzant et al., 2020).
As a quantitative technique, deep learning has been applied to automatically gen-
erate design concepts. For example, Raina, McComb and Cagan (2019) used a deep
convolutional autoencoder to imitate human designers to generate high-level semantic
information from image designs without any objectives. Recently, a more powerful gen-
erative deep learning model, i.e., generative adversarial nets (Goodfellow et al., 2014),
has been proposed and it has better capabilities to extract key information contained in
the design space to generate new designs and requires minimal input from the designer.
For example, Shu et al. (2020) trained a generative adversarial network to generate 3D
aircraft models and after three iterations of the training-evaluation process, the produced
design had statistically significant improvement based on evaluation in a simulated envi-
ronment. Chen et al. (2019) proposed a semantic ideation network and a visual concept
combined model based on generative adversarial network and their model was able to gen-
erate cross-domain concepts efficiently with quantity and novelty. The selection process
was conducted by domain experts.
3.3 Affective-cognitive Needs Fulfilment
The fulfillment step aims to create mapping relationships between affective and
cognitive needs of customers and product specification in the form of design features or
elements. Both traditional engineering methods and machine learning models are widely
used though many engineering design methods apply machine learning models, too.
3.3.1 Quality Function Deployment
One of the most frequently used engineering design methods to translate customer
needs, especially functional needs, to product specifications is probably quality function
21
deployment (Prasad, 1998). Quality function deployment is a method that first trans-
forms qualitative customer needs into quantitative parameters, then deploys the functions
to form product quality, and then translates product quality into design elements, and
finally to specific manufacturing processes (Akao, 1994). For example, Jin et al. (2009)
used quality function deployment to develop evaluation models for overall emotional fac-
tors, detailed emotional factors, usability factors, and physical design specifications in
three sequential processes, based on which the emotional factors affecting physical de-
sign specifications were generated. In order to better deal with the uncertainty involved
in affective and cognitive needs, fuzzy and rough quality function deployment was also
used. For example, Kang et al. (2018) integrated the evaluation grid method with
fuzzy or rough quality function deployment to build relationships between affective needs
and design elements, where a fuzzy analytic hierarchy process was integrated with qual-
ity function deployment to prioritize the affective needs involved. Later, Kang (2020a)
applied both fuzzy quality function deployment and rough set theory to develop relation-
ships between aesthetic product elements and customer satisfaction. Similarly, Zhai et
al. (2008) combined quality function deployment with rough set, where rough numbers
were used to help deal with subjective variables involved in affective needs and design
parameters.
3.3.2 Machine Learning Methods
Regression models are among the first to construct the relationships between the
customer needs and product design elements. For instance, a general linear regression
model was used to connect design specifications and participants’ emotional responses
to website design (J. Kim, Lee, & Choi, 2003). Likewise, a multiple regression model
was used to link usability factors to design elements (Han, Yun, Kim, & Kwahk, 2000).
However, linear relationships might not well represent the relationships between customer
needs and design specifications (Zhou, Jiao, Schaefer, & Chen, 2010).
One of the frequently used nonlinear methods is association rule mining that can di-
rectly link needs and design elements using if-then rules (Jiao, Zhang, & Helander, 2006).
Using goodness evaluation of the rules, helpful rules can be identified (Zhou, Jiao, Schae-
fer, & Chen, 2010). The uncertainty involved in customer needs can be mitigated using
fuzzy set theory as mentioned above. Kwong, Jiang, and Luo (2016) used chaos-based
fuzzy regression to understand both the concerns and satisfactions of design, engineer-
ing, and marketing issues in Kansei engineering. By combing if-then rules and fuzzy
set theory, Akay and Kurt (2009) proposed a neuro-fuzzy if-then rules to identify the
relationships between physical form design elements and customers’ affective responses
for mobile phone design. However, the association rules identified can still be spurious
if they are found just by co-occurrences. Another method often used is neural networks
and the powerful nonlinear modeling capability can help identify the relationships be-
tween affective-cognitive needs and design attributes. For example, Hsiao and Huang
22
(2002) used neural networks to build the connection between product form parameters
and Kansei adjectives. Kang (2020b) applied neural networks as a mapping function
to identify the important Kansei factors and product design elements for vehicle booth
design. Many researchers made use of the advantages of multiple machine learning mod-
els for the fulfillment task. Quan, Li, and Hu (2018) proposed a deep transfer learning
model to generate new product models by reconstructing and merging color and pattern
features for clothes, and then used a neural network model to identify the relationships
between product elements and Kansei. Wang (2011) proposed a combined approach of
grey system theory and support vector regression to capture the bidirectional relation-
ships between customers’ affective needs and design elements. Recently, deep learning
models have also been applied in this fulfillment task and compared to traditional ma-
chine learning models, deep learning models are more successful. For example, Wang et
al. (2018) applied two deep learning techniques, named Deep Belief Network and Re-
stricted Boltzmann machines, to classify multiple affective attributes of customer reviews
and compared to traditional learning models they used, i.e., support vector machine and
softmax regression, the accuracy of the deep learning models was 50% higher.
4 CHALLENGES AND FUTURE DIRECTIONS
Upon reviewing work published in the past related to emotional design, we speculate
recent trends and possible future directions below.
4.1 Measuring Emotion and Cognition in Naturalistic Setting
Emotion is dynamic and short-lived and many studies (e.g., Du et al., 2020) pre-
sented above measured emotion in controlled laboratories and some were limited to a
small number of basic emotions (e.g., Zhou, Qu, et al., 2011, 2014). However, in many
scenarios, emotion recognition should be conducted in naturalistic settings and the num-
ber of emotions in different applications need to go beyond the six basic emotions (Zhou,
Kong, et al., 2020). Regardless of the good performance of emotion recognition from
the reported studies in laboratory conditions, applications using emotion recognition in
naturalistic settings still remain an open challenge, such as in the context of learning,
driving, entertainment, and robotics (Avots, Sapi´nski, Bachmann, & Kami´nska, 2019).
For example, in the Affective Behavior Analysis in-the-wild 2020 Competition, it was
reported that the best average concordance correlation coefficient for valence and arousal
recognition was only 0.447, and the best weighted performance between accuracy and
f1-score for seven basic emotion and eight action unit detection was only 0.509 and 0.607,
respectively (Kollias, Schulc, Hajiyev, & Zafeiriou, 2020).
Another challenge is the ambiguities and uncertainties involved in subjective human
emotion and cognition, as evidenced in Kansei engineering. Furthermore, many studies
only recruited a small number of participants and the sampled Kansei might be biased
23
for the targeted user groups. For example, only four participants were reported in (Jiang,
Kwong, Siu, & Liu, 2015) and five participants were reported in (Kwong et al., 2016).
Due to the time-consuming data collection process while customer needs might change
from time to time rapidly with an increasing number of new products in the market,
it is necessary to develop an efficient method to collect data from a large number of
participants continuously (W. Wang et al., 2019).
One of the possible solutions to deal with such challenges is to make use of the
technologies of the internet of things with the convergence of deep learning techniques,
commodity sensors, and embedded systems. Such technologies have contributed a surge
in data traffic, making real time measuring of user states, including emotions and cogni-
tive states a possibility with advanced deep learning models and computational resources
(Gubbi, Buyya, Marusic, & Palaniswami, 2013). First, deep learning techniques have
achieved great successes in many applications, such as computer vision (Hassaballah &
Awad, 2020), natural language processing (Devlin et al., 2018), and emotional and cogni-
tive states recognition (Zhou, Kong, et al., 2020). Second, the development of the internet
of things offers enough training data to improve the performance of deep learning mod-
els (Chan et al., 2020). For example, wearable sensors (e.g., smart watches) and other
commodity sensors can be readily used to collect various data about the user (physio-
logical and behavioral data) on the fly over wireless networks (Alkurd et al., 2020). In
addition, human-computer integration is emerging, in which computational and human
systems can be interwoven closely in a wider social-technical system (Mueller et al., 2020).
Furthermore, user-generated data on websites, such as online product reviews on Ama-
zon.com, can be utilized to understand and update customer needs more efficiently and
effectively with a large number of users on a daily basis (Li, Tian, Wang, Wang, & Huang,
2018; Zhou, Ayoub, et al., 2020). Third, high-performance computing resources, such as
graphic processing units and tensor processing units, allow the training of large-scale deep
learning models for big data possible (Q. Zhang, Yang, Chen, & Li, 2018). Under such
circumstances, smart and personalized services based on artificial intelligence that can
respond to user feedback immediately might be the cardinal competitive advantage for all
service providers (Alkurd et al., 2020). At the same time, the contextual data should also
be utilized in order to optimize and personalize the interaction process between humans
and the system (Zhou, Xu, & Jiao, 2011).
4.2 Integration of Affect and Cognition
Affect and cognition have been treated as independent entities (Zajonc, 1980). How-
ever, studies have shown that affect and cognition are highly interrelated and should be
integrated (Jiao, Zhou, & Chu, 2017). For example, Kahneman (2003) pointed out that
there were two systems for decision making, i.e., the analytic system (i.e., the cognitive
system) and the experiential system (i.e., the affective system). The analytic system
deliberately uses cognitive processes to make reasonable decisions while the experiential
24
system operates outside of conscious thoughts and utilizes emotion-related associations,
past experience and intuitions for reactive decision making. The behavioral level in Nor-
man’s emotional design directly points out that understandability and usability are the
two key factors in contributing to positive emotional responses (Norman, 2004). The
appraisal theory emphasizes the cognitive component of emotions, which can be used to
distinguish different emotions (Ortony et al., 1990). Ahn and Picard (2005) also proposed
an integrated framework of affect and cognition for decision and learning. Such integrated
perspectives were also emphasized in our previous work for user experience modeling and
design (Zhou, Xu, & Jiao, 2011; Zhou et al., 2013; Zhou, Ji, & Jiao, 2014b).
Based on the view of cognitive and affective systems when we process information,
Norman’s emotional design gives us good guidelines and implications at three different
levels as well as the interactions between affect and cognition. For example, positive affect
allows us to think more broadly and creatively, supporting idea generation (Isen, Daub-
man, & Nowicki, 1987) while negative affect narrows one’s cognitive scope (Rathunde,
2000). However, a recent study suggested that motivational intensity (i.e., how strongly
one was compelled to approach or avoid something) that influenced one’s cognitive scope
rather than emotional valence (Harmon-Jones, Gable, & Price, 2013). In their study, they
found that 1) amusement (a low motivational emotion) elicited by watching a cat video
broadened the participants’ attentional focus while desire (a high motivational emotion)
elicited by watching a delicious-looking dessert video narrowed their attentional focus and
2) sadness (a low motivational emotion) broadened their attentional focus while disgust
(a high motivational emotion) narrowed their attentional focus. Therefore, despite the
common consensus on integrated affect and cognition, the interaction between affect and
cognition tends to be less clear (Storbeck & Clore, 2007) and it is challenging to develop
analytic models to integrate affective and cognitive needs, to measure subjective experi-
ence, and to extract the mapping relationships between integrated affective and cognitive
needs and design elements (Jiao et al., 2017). More research in this aspect is called for
to support emotional design.
4.3 Emotional Design for Product Ecosystems
Many companies now focus more on the overall experience of their product ecosys-
tems rather than one single individual product in order to improve their competitiveness.
A product ecosystem has a focal product at the center during the human-product interac-
tion process, with numerous other peripheral supporting products and services to deliver
an entire experience so that other more disjointed offerings cannot compete (Zhou, Xu,
& Jiao, 2011). Good examples include the Apple product ecosystem and the Amazon
product ecosystem. From the experiential point of view, the roles of affect and cognition
as an antecedent, a consequence, and a mediator of human-product-ambience interaction
within a product ecosystem over time form one’s total user experience (Hassenzahl &
Tractinsky, 2006; Zhou, Xu, & Jiao, 2011). First, within a product ecosystem, when
25
one is interacting with a product, that product becomes the focal product and others
become supporting products, named as ambience. Without the supporting role of other
ambient products, the interaction with the focal product might be isolated in a narrower
scope. For example, Levin (2014) emphasized the interaction between multiple devices,
such as smart phones, computers, tablets, and TVs to create user experience with an
ecosystem approach. Gawer and Cusumano (2014) pointed out that within a company’s
platform, numerous derivative products and services should be designed within a common
ecosystem perspective to support innovation and user experience and within an industry
platform, complementary products, services, and technologies should be designed and
developed within a business ecosystem to promote innovation. Second, the role of affect
and cognition should be expanded from just the consequence of design. Traditional af-
fective computing emphasizes to respond retrospectively to users’ emotions during the
interaction process to aid frustrated users (Picard & Klein, 2002). Within a product
ecosystem, the system should proactively predict users’ emotions and examine the roles
of affect and cognition as the antecedent and mediator factors as well. Third, we empha-
size not only the temporary interaction between the user and the ecosystem but also over
a longer period of time to form a totality of user experience. This echoes Norman’s reflec-
tive design to some extent. With the complexity and dynamics of user experience over
a longer period of time, emotional designers need to consider all relevant factors in the
product ecosystem. This is consistent with the perspective of measuring emotion in the
naturalistic setting and advanced technologies of the internet of things with convergences
of deep learning, big data, and wireless networks could potentially help.
5 CONCLUSIONS
Emotional design has been well recognized in the domain of human factors and
ergonomics. In this chapter, we reviewed related models and methods of emotional de-
sign. We are motivated to encourage emotional designers to take multiple perspectives
when examining these models and methods. Then we proposed a systematic process for
emotional design, including affective-cognitive needs elicitation, affective-cognitive needs
analysis, and affective-cognitive needs fulfillment to support emotional design. Within
each step, we provided an updated review of the representative methods to support and
offer further guidance on emotional design. We hope researchers and industrial prac-
titioners can take a systematic approach to consider each step in the framework with
care. Finally, the speculations on the challenges and future directions can potentially
help researchers across different fields to further advance emotional design.
References
Ahn, H., & Picard, R. W. (2005). Affective-cognitive learning and decision making: A motivational
26
reward framework for affective agents. In International conference on affective computing and
intelligent interaction (pp. 866–873).
Akao, Y. (1994). Development history of quality function deployment. The Customer Driven Approach
to Quality Planning and Deployment,339 , 90.
Akay, D., & Kurt, M. (2009). A neuro-fuzzy based approach to affective design. The International
Journal of Advanced Manufacturing Technology,40 (5-6), 425.
Alkurd, R., Abualhaol, I., & Yanikomeroglu, H. (2020). Big-data-driven and ai-based framework to
enable personalization in wireless networks. IEEE Communications Magazine,58 (3), 18–24.
Altarriba Bertran, F., M´arquez Segura, E., & Isbister, K. (2020). Technology for situated and emergent
play: A bridging concept and design agenda. In Proceedings of the 2020 chi conference on human
factors in computing systems (pp. 1–14).
Anderson, C. (2010). Presenting and evaluating qualitative research. American journal of pharmaceutical
education,74 (8).
Avots, E., Sapi´nski, T., Bachmann, M., & Kami´nska, D. (2019). Audiovisual emotion recognition in
wild. Machine Vision and Applications,30 (5), 975–985.
Ayoub, J., Mason, B., Morse, K., Kirchner, A., Tumanyan, N., & Zhou, F. (2020). Otto: An autonomous
school bus system for parents and children. In Extended abstracts of the 2020 chi conference on
human factors in computing systems (pp. 1–7).
Ayoub, J., & Zhou, F. (2020). Investigating drivers’ trust in autonomous vehicles’ decisions of lane
changing events. In Proceedings of the human factors and ergonomics society annual meeting
(Vol. 64).
Ayoub, J., Zhou, F., Bao, S., & Yang, X. J. (2019). From manual driving to automated driving: A
review of 10 years of autoui. In Proceedings of the 11th international conference on automotive
user interfaces and interactive vehicular applications (pp. 70–90).
Ayoub, J., Zhou, F., Xu, Q., & Yang, J. (2019). Analyzing customer needs of product ecosystems using
online product reviews. In International design engineering technical conferences and computers
and information in engineering conference (Vol. 59186, p. V02AT03A002).
Barnes, C., & Lillford, S. P. (2009). Decision support for the design of affective products. Journal of
Engineering Design,20 (5), 477–492.
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016). Training deep networks for facial expression
recognition with crowd-sourced label distribution. In Proceedings of the 18th acm international
conference on multimodal interaction (pp. 279–283).
Bartels, J., & Onwezen, M. C. (2014). Consumers’ willingness to buy products with environmental
and ethical claims: the roles of social representations and social identity. International Journal of
Consumer Studies,38 (1), 82–89.
Baxter, K., Courage, C., & Caine, K. (2015). Understanding your users: a practical guide to user
research methods. Morgan Kaufmann.
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., & Tanaka, F. (2018). Social robots for
education: A review. Science robotics,3(21).
Bloch, P. H. (1995). Seeking the ideal form: Product design and consumer response. Journal of
marketing,59 (3), 16–29.
Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. J. (2001). Emotion and motivation i:
defensive and appetitive reactions in picture processing. Emotion ,1(3), 276.
Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic
differential. Journal of behavior therapy and experimental psychiatry,25 (1), 49–59.
Breazeal, C. (2011). Social robots for health applications. In 2011 annual international conference of
the ieee engineering in medicine and biology society (pp. 5368–5371).
27
Brown, E., & Cairns, P. (2004). A grounded investigation of game immersion. In Chi’04 extended
abstracts on human factors in computing systems (pp. 1297–1300).
Buxton, B. (2010). Sketching user experiences: getting the design right and the right design. Morgan
kaufmann.
Carbonara, N., & Scozzi, B. (2006). Cognitive maps to analyze new product development processes: A
case study. Technovation,26 (11), 1233–1243.
Chai, C., Bao, D., Sun, L., & Cao, Y. (2015). The relative effects of different dimensions of traditional
cultural elements on customer product satisfaction. International Journal of Industrial Ergonomics,
48 , 77–88.
Chan, K. Y., Kwong, C., Wongthongtham, P., Jiang, H., Fung, C. K., Abu-Salih, B., . . . Jain, P.
(2020). Affective design using machine learning: a survey and its prospect of conjoining big data.
International Journal of Computer Integrated Manufacturing,33 (7), 645–669.
Chen, L., Wang, P., Dong, H., Shi, F., Han, J., Guo, Y., .. . Wu, C. (2019). An artificial intelligence
based data-driven approach for design ideation. Journal of Visual Communication and Image
Representation,61 , 10–22.
Clore, G. L., & Ortony, A. (2013). Psychological construction in the occ model of emotion. Emotion
Review,5(4), 335–343.
Coffey, J. W., & Carnot, M. J. (2003). Graphical depictions for knowledge generation and sharing. In
International conference on information and knowledge sharing.
Coursaris, C. K., & Van Osch, W. (2016). A cognitive-affective model of perceived user satisfaction
(campus): The complementary effects and interdependence of usability and aesthetics in is design.
Information & Management,53 (2), 252–264.
Crandall, B., Klein, G., Klein, G. A., Hoffman, R. R., et al. (2006). Working minds: A practitioner’s
guide to cognitive task analysis. Mit Press.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper & Row.
Csikszentmihalyi, M., & Halton, E. (1981). The meaning of things: Domestic symbols and the self.
Cambridge university press.
Csikszentmihalyi, M., & Seligman, M. E. (2000). Positive psychology: An introduction. American
Psychologist,55 (1), 5–14.
Cui, H., Liu, A., Zhang, X., Chen, X., Wang, K., & Chen, X. (2020). Eeg-based emotion recognition
using an end-to-end regional-asymmetric convolutional neural network. Knowledge-Based Systems,
106243.
Currano, R., Park, S. Y., Domingo, L., Garcia-Mancilla, J., Santana-Mancilla, P. C., Gonzalez, V. M.,
& Ju, W. (2018). ¡ vamos! observations of pedestrian interactions with driverless cars in mexico.
In Proceedings of the 10th international conference on automotive user interfaces and interactive
vehicular applications (pp. 210–220).
Daily, S. B., James, M. T., Cherry, D., Porter III, J. J., Darnell, S. S., Isaac, J., & Roy, T. (2017).
Affective computing: historical foundations, current applications, and future trends. In Emotions
and affect in human factors and human-computer interaction (pp. 213–231). Elsevier.
Dautenhahn, K. (2002). Design spaces and niche spaces of believable social robots. In Proceedings. 11th
ieee international workshop on robot and human interactive communication (pp. 192–197).
Davidson, R. J., Sherer, K. R., & Goldsmith, H. H. (2009). Handbook of affective sciences. Oxford
University Press.
Desmet, P. (1999). To love and not to love: Why do products elicit mixed emotions. In Proceedings of
the 1st international conference on design and emotion (pp. 67–74).
Desmet, P., & Hekkert, P. (2007). Framework of product experience. International journal of design,
1(1), 57–66.
28
Desmet, P., & Hekkert, P. (2009). Special issue editorial: Design & emotion. International Journal of
Design,3(2).
Desmet, P., Hekkert, P., & Hillen, M. (2003). Values and emotions; an empirical investigation in the
relationship between emotional responses to products and human values. In Proceedings of the fifth
european academy of design conference, barcelona, spain.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint arXiv:1810.04805 .
Dmitrenko, D., Maggioni, E., Brianza, G., Holthausen, B. E., Walker, B. N., & Obrist, M. (2020).
Caroma therapy: pleasant scents promote safer driving, better mood, and improved well-being in
angry drivers. In Proceedings of the 2020 chi conference on human factors in computing systems
(pp. 1–13).
Du, N., Zhou, F., Pulver, E. M., Tilbury, D. M., Robert, L. P., Pradhan, A. K., & Yang, X. J. (2020).
Examining the effects of emotional valence and arousal on takeover performance in conditionally
automated driving. Transportation research part C: emerging technologies,112 , 78–87.
Ekman, P. (1992). An argument for basic emotions. Cognition & emotion,6(3-4), 169–200.
Ellsworth, P. C., & Scherer, K. R. (2003). Appraisal processes in emotion. Oxford University Press.
Eyssel, F., & Kuchenbrandt, D. (2012). Social categorization of social robots: Anthropomorphism as a
function of robot group membership. British Journal of Social Psychology,51 (4), 724–731.
Faste, H., Rachmel, N., Essary, R., & Sheehan, E. (2013). Brainstorm, chainstorm, cheatstorm, tweet-
storm: new ideation strategies for distributed hci design. In Proceedings of the sigchi conference
on human factors in computing systems (pp. 1343–1352).
Gawer, A., & Cusumano, M. A. (2014). Industry platforms and ecosystem innovation. Journal of product
innovation management,31 (3), 417–433.
Gellatly, A. W., Hansen, C., Highstrom, M., & Weiss, J. P. (2010). Journey: General motors’ move
to incorporate contextual design into its next generation of automotive hmi designs. In Proceed-
ings of the 2nd international conference on automotive user interfaces and interactive vehicular
applications (pp. 156–161).
Glocker, M. L., Langleben, D. D., Ruparel, K., Loughead, J. W., Gur, R. C., & Sachser, N. (2009).
Baby schema in infant faces induces cuteness perception and motivation for caretaking in adults.
Ethology,115 (3), 257–263.
Goetz, T., Frenzel, A. C., Stoeger, H., & Hall, N. C. (2010). Antecedents of everyday positive emotions:
An experience sampling analysis. Motivation and Emotion,34 (1), 49–62.
Goleman, D. (1995). Emotional intelligence. new york, ny, england. Bantam Books, Inc.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., . .. Bengio, Y. (2014).
Generative adversarial nets. In Advances in neural information processing systems (pp. 2672–2680).
Greenberg, S., Carpendale, S., Marquardt, N., & Buxton, B. (2011). Sketching user experiences: The
workbook. Elsevier.
Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of things (iot): A vision,
architectural elements, and future directions. Future generation computer systems,29 (7), 1645–
1660.
Guthier, B., D¨orner, R., & Martinez, H. P. (2016). Affective computing in games. In Entertainment
computing and serious games (pp. 402–441). Springer.
Han, S. H., Yun, M. H., Kim, K.-J., & Kwahk, J. (2000). Evaluation of product usability: development
and validation of usability dimensions and design elements based on empirical models. International
Journal of Industrial Ergonomics,26 (4), 477–488.
Hancock, P. A., Pepe, A. A., & Murphy, L. L. (2005). Hedonomics: The power of positive and pleasurable
ergonomics. Ergonomics in design,13 (1), 8–14.
29
Harmon-Jones, E., Gable, P. A., & Price, T. F. (2013). Does negative affect always narrow and positive
affect always broaden the mind? considering the influence of motivational intensity on cognitive
scope. Current Directions in Psychological Science,22 (4), 301–307.
Hart, S. G. (2006). Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the human factors
and ergonomics society annual meeting (Vol. 50, pp. 904–908).
Hassaballah, M., & Awad, A. I. (2020). Deep learning in computer vision: Principles and applications.
CRC Press.
Hassenzahl, M., & Tractinsky, N. (2006). User experience-a research agenda. Behaviour & information
technology,25 (2), 91–97.
Helander, M. G., & Khalid, H. M. (2005). Underlying theories of hedonomics for affective and pleasurable
design. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 49, pp.
1691–1695).
Helander, M. G., Khalid, H. M., Lim, T. Y., Peng, H., & Yang, X. (2013). Emotional needs of car buyers
and emotional intent of car designers. Theoretical Issues in Ergonomics Science,14 (5), 455–474.
Helander, M. G., & Tham, M. P. (2003). Hedonomics—affective human factors design. Ergonomics,
46 (13-14), 1269–1272.
Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). Use of the critical decision method to elicit expert
knowledge: A case study in the methodology of cognitive task analysis. Human factors,40 (2),
254–276.
Hoornaert, S., Ballings, M., Malthouse, E. C., & Van den Poel, D. (2017). Identifying new product ideas:
waiting for the wisdom of the crowd or screening ideas in real time. Journal of Product Innovation
Management,34 (5), 580–597.
Hopper, C. H. (2012). Practicing college learning strategies. Cengage Learning.
Hsiao, S.-W., & Huang, H.-C. (2002). A neural network based approach for product form design. Design
studies,23 (1), 67–84.
Huang, Y., Chen, C.-H., & Khoo, L. P. (2012). Kansei clustering for emotional design using a combined
design structure matrix. International Journal of Industrial Ergonomics,42 (5), 416–427.
Huang, Y., Chen, C.-H., Wang, I.-H. C., & Khoo, L. P. (2014). A product configuration analysis
method for emotional design using a personal construct theory. International Journal of Industrial
Ergonomics,44 (1), 120–130.
Isen, A. M., Daubman, K. A., & Nowicki, G. P. (1987). Positive affect facilitates creative problem
solving. Journal of personality and social psychology,52 (6), 1122.
Jiang, H., Kwong, C., Siu, K. W. M., & Liu, Y. (2015). Rough set and pso-based anfis approaches to
modeling customer satisfaction for affective product design. Advanced Engineering Informatics,
29 (3), 727–738.
Jiao, J. R., Xu, Q., Du, J., Zhang, Y., Helander, M., Khalid, H. M., . . . Ni, C. (2007). Analytical affective
design with ambient intelligence for mass customization and personalization. International Journal
of Flexible Manufacturing Systems,19 (4), 570–595.
Jiao, J. R., Zhang, Y., & Helander, M. (2006). A kansei mining system for affective design. Expert
Systems with Applications,30 (4), 658–673.
Jiao, J. R., Zhou, F., & Chu, C.-H. (2017). Decision theoretic modeling of affective and cognitive needs
for product experience engineering: key issues and a conceptual framework. Journal of Intelligent
Manufacturing,28 (7), 1755–1767.
Jin, B. S., Ji, Y. G., Choi, K., & Cho, G. (2009). Development of a usability evaluation framework
with quality function deployment: from customer sensibility to product design. Human Factors
and Ergonomics in Manufacturing & Service Industries,19 (2), 177–194.
30
Jindo, T., & Hirasago, K. (1997). Application studies to car interior of kansei engineering. International
journal of industrial ergonomics,19 (2), 105–114.
Johnson, C. M., & Turley, J. P. (2006). The significance of cognitive modeling in building healthcare
interfaces. International journal of medical informatics,75 (2), 163–172.
Johnson, E., Herv´as, R., Guti´errez L´opez de la Franca, C., Mond´ejar, T., Ochoa, S. F., & Favela, J.
(2018). Assessing empathy and managing emotions through interactions with an affective avatar.
Health Informatics Journal,24 (2), 182–193.
Jordan, P. (1997). Putting the pleasure into products. IEE review ,43 (6), 249–252.
Jordan, P. (2000). The four pleasures-a framework for pleasures in design. In Proceedings of conference
on pleasure based human factors design, groningen. the netherlands: Philips design.
Kahneman, D. (2003). A perspective on judgment and choice: mapping bounded rationality. American
psychologist,58 (9), 697.
Kalliatakis, G., Stergiou, A., & Vidakis, N. (2017). Conceiving human interaction by visualising depth
data of head pose changes and emotion recognition via facial expressions. Computers ,6(3), 25.
Kang, X. (2020a). Aesthetic product design combining with rough set theory and fuzzy quality function
deployment. Journal of Intelligent & Fuzzy Systems(Preprint), 1–16.
Kang, X. (2020b). Combining grey relationship analysis and neural network to develop attractive
automobile booth design. Computational Intelligence and Neuroscience,2020 .
Kang, X., Yang, M., Wu, Y., & Ni, B. (2018). Integrating evaluation grid method and fuzzy quality
function deployment to new product development. Mathematical Problems in Engineering,2018 .
Keshavarz, B., Stelzmann, D., Paillard, A., & Hecht, H. (2015). Visually induced motion sickness can
be alleviated by pleasant odors. Experimental brain research,233 (5), 1353–1364.
Kim, J., Lee, J., & Choi, D. (2003). Designing emotionally evocative homepages: an empirical study of
the quantitative relations between design factors and emotional dimensions. International Journal
of Human-Computer Studies,59 (6), 899–940.
Kim, N. W., Riche, N. H., Bach, B., Xu, G., Brehmer, M., Hinckley, K., .. . others (2019). Datatoon:
drawing data comics about dynamic networks with pen+ touch interaction. In Proceedings of the
conference on human factors in computing systems (chi). acm (Vol. 12).
Kleinginna, P. R., & Kleinginna, A. M. (1981). A categorized list of emotion definitions, with suggestions
for a consensual definition. Motivation and emotion ,5(4), 345–379.
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., .. . Patras, I. (2011).
Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective
computing,3(1), 18–31.
Kollias, D., Schulc, A., Hajiyev, E., & Zafeiriou, S. (2020). Analysing affective behavior in the first abaw
2020 competition. arXiv preprint arXiv:2001.11409 .
Kong, P., Cornet, H., & Frenkler, F. (2018). Personas and emotional design for public service robots:
A case study with autonomous vehicles in public transportation. In 2018 international conference
on cyberworlds (cw) (pp. 284–287).
Kubovy, M. (1999). On the pleasures of the mind. Well-being: The foundations of hedonic psychology,
1999 , 134–154.
Kumar, A. (2007). From mass customization to mass personalization: a strategic transformation.
International Journal of Flexible Manufacturing Systems,19 (4), 533.
Kwong, C., Jiang, H., & Luo, X. (2016). Ai-based methodology of integrating affective design, engineer-
ing, and marketing for defining design specifications of new products. Engineering Applications of
Artificial Intelligence,47 , 49–60.
Larson, R., & Csikszentmihalyi, M. (2014). The experience sampling method. In Flow and the foundations
of positive psychology (pp. 21–34). Springer.
31
Lemon, K. N., & Verhoef, P. C. (2016). Understanding customer experience throughout the customer
journey. Journal of marketing,80 (6), 69–96.
Levin, M. (2014). Designing multi-device experiences: An ecosystem approach to user experiences across
devices. ” O’Reilly Media, Inc.”.
Li, Z., Shi, K., Dey, N., Ashour, A. S., Wang, D., Balas, V. E., . . . Shi, F. (2017). Rule-based back
propagation neural networks for various precision rough set presented kansei knowledge prediction:
a case study on shoe product form features extraction. Neural Computing and Applications,28 (3),
613–630.
Li, Z., Tian, Z., Wang, J., Wang, W., & Huang, G. (2018). Dynamic mapping of design elements and
affective responses: a machine learning based method for affective design. Journal of Engineering
Design,29 (7), 358–380.
Liu, Z.-Q., & Satur, R. (1999). Contextual fuzzy cognitive map for decision support in geographic
information systems. IEEE Transactions on Fuzzy Systems,7(5), 495–507.
Livingston, B., & Thurott, P. (2007). Windows vista secrets: do what you never thought possible with
windows vista. John Wiley & Sons.
Lohr, S. (2018). Facial recognition is accurate, if you’re a white guy. New York Times,9.
Lokman, A. M., & Kamaruddin, K. A. (2010). Kansei affinity cluster for affective product design. In
2010 international conference on user science and engineering (i-user) (pp. 38–43).
Lu, W., & Petiot, J.-F. (2014). Affective design of products using an audio-based protocol: application
to eyeglass frame. International Journal of Industrial Ergonomics,44 (3), 383–394.
Majchrzak, A., & Malhotra, A. (2013). Towards an information systems perspective and research agenda
on crowdsourcing for innovation. The Journal of Strategic Information Systems,22 (4), 257–268.
Malicka, A., Gilabert Guerrero, R., & Norris, J. M. (2019). From needs analysis to task design: Insights
from an english for specific purposes context. Language Teaching Research,23 (1), 78–106.
Marghani, V. G. R. E., da Silva, F. C., Knapik, L., & Verri, M. A. (2013). Kansei engineering: Types
of this methodology. In Emotional engineering vol. 2 (pp. 127–147). Springer.
Maslow, A., & Lewis, K. (1987). Maslow’s hierarchy of needs. Salenger Incorporated,14 , 987.
Messinger, D. S., Duvivier, L. L., Warren, Z. E., Mahoor, M., Baker, J., Warlaumont, A., & Ruvolo, P.
(2015). Affective computing, emotional development, and autism.
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2017). Affectnet: A database for facial expression,
valence, and arousal computing in the wild. IEEE Transactions on Affective Computing,10 (1),
18–31.
Mueller, F. F., Lopes, P., Strohmeier, P., Ju, W., Seim, C., Weigel, M., . . . others (2020). Next steps
for human-computer integration. In Proceedings of the 2020 chi conference on human factors in
computing systems (pp. 1–15).
Nagamachi, M. (1995). Kansei engineering: a new ergonomic consumer-oriented technology for product
development. International Journal of industrial ergonomics,15 (1), 3–11.
Nagamachi, M., & Lokman, A. M. (2016). Innovations of kansei engineering. CRC Press.
Nakamori, Y., & Ryoke, M. (2004). Modeling of fuzziness in multivariate analysis. In 2004 ieee interna-
tional conference on systems, man and cybernetics (ieee cat. no. 04ch37583) (Vol. 1, pp. 294–298).
Neubauer, C., Gremillion, G., Schaefer, K. E., Perelman, B. S., La Fleur, C., & Metcalfe, J. S. (2020).
Analysis of facial expressions: Explaining affective state and trust-based decisions during interac-
tion with automation (Tech. Rep.). CCDC Army Research Laboratory Aberdeen Proving Ground
United States.
Ng, Y., & Khong, C. W. (2014). A review of affective user-centered design for video games. In 2014 3rd
international conference on user science and engineering (i-user) (pp. 79–84).
32
Ng, Y., Khong, C. W., & Nathan, R. J. (2018). Evaluating affective user-centered design of video games
using qualitative methods. International Journal of Computer Games Technology,2018 .
Norman, D. (2004). Emotional design: Why we love (or hate) everyday things. Basic Civitas Books.
Norman, D. (2013). The design of everyday things: Revised and expanded edition. Basic books.
Novak, J. D., & Ca˜nas, A. J. (2006). The theory underlying concept maps and how to construct them.
Florida Institute for Human and Machine Cognition,1(1), 1–31.
Oron-Gilad, T., & Hancock, P. A. (2017). From ergonomics to hedonomics: Trends in human factors
and technology—the role of hedonomics revisited. In Emotions and affect in human factors and
human-computer interaction (pp. 185–194). Elsevier.
Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. Cambridge UK:
Cambridge University.
Ortony, A., Clore, G. L., & Collins, A. (1990). The cognitive structure of emotions. Cambridge university
press.
Osgood, C. E., May, W. H., Miron, M. S., & Miron, M. S. (1975). Cross-cultural universals of affective
meaning (Vol. 1). University of Illinois Press.
Park, D., Lee, J.-H., & Kim, S. (2011). Investigating the affective quality of interactivity by motion
feedback in mobile touchscreen user interfaces. International Journal of Human-Computer Studies,
69 (12), 839–853.
Pfautz, J., & Roth, E. (2006). Using cognitive engineering for system design and evaluation: A visual-
ization aid for stability and support operations. International Journal of Industrial Ergonomics,
36 (5), 389–407.
Pfleging, B., Rang, M., & Broy, N. (2016). Investigating user needs for non-driving-related activities
during automated driving. In Proceedings of the 15th international conference on mobile and
ubiquitous multimedia (pp. 91–99).
Picard, R. (1997). Affective computing. cambridge, massachustes institure of technology. The MIT Press.
Picard, R., & Klein, J. (2002). Computers that recognise and respond to user emotion: theoretical and
practical implications. Interacting with computers,14 (2), 141–169.
Piho, L., & Tjahjadi, T. (2018). A mutual information based adaptive windowing of informative eeg for
emotion recognition. IEEE Transactions on Affective Computing.
Prasad, B. (1998). Review of qfd and related deployment techniques. Journal of manufacturing Systems,
17 (3), 221–234.
Pruitt, J., & Adlin, T. (2010). The persona lifecycle: keeping people in mind throughout product design.
Elsevier.
Pryzant, R., Martinez, R. D., Dass, N., Kurohashi, S., Jurafsky, D., & Yang, D. (2020). Automatically
neutralizing subjective bias in text. In Proceedings of the aaai conference on artificial intelligence
(Vol. 34, pp. 480–489).
Qin, Z., Song, Y., & Tian, Y. (2019). The impact of product design with traditional cultural proper-
ties (tcps) on consumer behavior through cultural perceptions: Evidence from the young chinese
generation. Sustainability,11 (2), 426.
Quan, H., Li, S., & Hu, J. (2018). Product innovation design based on deep learning and kansei
engineering. Applied Sciences,8(12), 2397.
Raina, A., McComb, C., & Cagan, J. (2019). Learning to design from humans: Imitating human
designers through deep learning. Journal of Mechanical Design,141 (11).
Rathunde, K. (2000). Broadening and narrowing in the creative process: A commentary on fredrickson’s”
broaden-and-build” model.
Reid, G. B., & Nygren, T. E. (1988). The subjective workload assessment technique: A scaling procedure
for measuring mental workload. In Advances in psychology (Vol. 52, pp. 185–218). Elsevier.
33
Rovira, E., McLaughlin, A. C., Pak, R., & High, L. (2019). Looking for age differences in self-driving
vehicles: examining the effects of automation reliability, driving risk, and physical impairment on
trust. Frontiers in Psychology,10 , 800.
Rubio, S., D´ıaz, E., Mart´ın, J., & Puente, J. M. (2004). Evaluation of subjective mental workload: A
comparison of swat, nasa-tlx, and workload profile methods. Applied Psychology,53 (1), 61–86.
Russell, J. A. (1993). Forced-choice response format in the study of facial expression. Motivation and
Emotion,17 (1), 41–51.
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological review,
110 (1), 145.
Russell, J. A., Weiss, A., & Mendelsohn, G. A. (1989). Affect grid: a single-item scale of pleasure and
arousal. Journal of personality and social psychology,57 (3), 493.
Scherer, K. R. (2005). What are emotions? and how can they be measured? Social science information,
44 (4), 695–729.
Schifferstein, H. N., & Hekkert, P. (2011). Product experience. Elsevier.
Schraagen, J. M., Chipman, S. F., & Shute, V. J. (2000). State-of-the-art review of cognitive task
analysis techniques. Cognitive task analysis, 467–487.
Schuurman, D., Baccarne, B., De Marez, L., & Mechant, P. (2012). Smart ideas for smart cities:
Investigating crowdsourcing for generating and selecting ideas for ict innovation in a city context.
Journal of theoretical and applied electronic commerce research,7(3), 49–62.
Sheller, M. (2004). Automotive emotions: Feeling the car. Theory, culture & society,21 (4-5), 221–242.
Shu, D., Cunningham, J., Stump, G., Miller, S. W., Yukish, M. A., Simpson, T. W., & Tucker, C. S.
(2020). 3d design using generative adversarial networks and physics-based validation. Journal of
Mechanical Design,142 (7).
Simon, H. A., Clark, M., & Fiske, S. (1982). Affect and cognition. Hillsdale: Lawrence Erlbaum.
Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. (2008). A diary study of mobile information needs.
In Proceedings of the sigchi conference on human factors in computing systems (pp. 433–442).
Spool, J. M. (2004). The kj-technique: A group process for establishing priorities. User interface
engineering.
Storbeck, J., & Clore, G. L. (2007). On the interdependence of cognition and emotion. Cognition and
Emotion,21 (6), 1212–1237.
Strauss, A., & Corbin, J. (1994). Grounded theory methodology. Handbook of qualitative research,
17 (1), 273–285.
Su, Z., Yu, S., Chu, J., Zhai, Q., Gong, J., & Fan, H. (2020). A novel architecture: Using convolutional
neural networks for kansei attributes automatic evaluation and labeling. Advanced Engineering
Informatics,44 , 101055.
Thalmann, N. M., Yumak, Z., & Beck, A. (2014). Autonomous virtual humans and social robots in
telepresence. In 2014 ieee 16th international workshop on multimedia signal processing (mmsp)
(pp. 1–6).
Tsang, P. S., & Velazquez, V. L. (1996). Diagnosticity and multidimensional subjective workload ratings.
Ergonomics,39 (3), 358–381.
Tseng, M. M., & Jiao, J. R. (2001). Mass customization. Handbook of industrial engineering,3, 684–709.
Walter, A. (2011). Designing for emotion. A book apart New York.
Wang, J., Wang, W., Tian, Z., & Li, Z. (2018). Classification of multiple affective attributes of customer
reviews: Using classical machine learning and deep learning. In Proceedings of the 2nd international
conference on computer science and application engineering (pp. 1–5).
Wang, K. C. (2011). A hybrid kansei engineering design expert system based on grey system theory and
support vector regression. Expert Systems with Applications,38 (7), 8738–8750.
34
Wang, W., Wang, J., Li, Z., Tian, Z., & Tsui, E. (2019). Multiple affective attribute classification of online
customer product reviews: A heuristic deep learning method for supporting kansei engineering.
Engineering Applications of Artificial Intelligence,85 , 33–45.
Wickens, C. D., Gordon, S. E., & Liu, Y. (1998). An introduction to human factors engineering. Longman
New York.
Willis, K. D., Lin, J., Mitani, J., & Igarashi, T. (2010). Spatial sketch: bridging between movement
& fabrication. In Proceedings of the fourth international conference on tangible, embedded, and
embodied interaction (pp. 5–12).
Wu, C.-H., Huang, Y.-M., & Hwang, J.-P. (2016). Review of affective computing in education/learning:
Trends and challenges. British Journal of Educational Technology,47 (6), 1304–1323.
Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, M. N. B., Affal, H. B., & Hussin, N. B. (2019). Affective
computing in education: A systematic review and future research. Computers & Education,142 ,
103649.
Yang, Y.-p., Chen, D.-k., Gu, R., Gu, Y.-f., & Yu, S.-h. (2016). Consumers’ kansei needs clustering
method for product emotional design based on numerical design structure matrix and genetic
algorithms. Computational intelligence and neuroscience,2016 .
Yerkes, R. M., Dodson, J. D., et al. (1908). The relation of strength of stimulus to rapidity of habit-
formation. Punishment: Issues and experiments, 27–41.
Zajonc, R. B. (1980). Feeling and thinking: Preferences need no inferences. American psychologist,
35 (2), 151.
Zanesco, A. P. (2020). Quantifying streams of thought during cognitive task performance using sequence
analysis. Behavior Research Methods, 1–21.
Zhai, L.-Y., Khoo, L.-P., & Zhong, Z.-W. (2008). A rough set enhanced fuzzy approach to quality
function deployment. The International Journal of Advanced Manufacturing Technology,37 (5-6),
613–624.
Zhai, L.-Y., Khoo, L.-P., & Zhong, Z.-W. (2009a). A dominance-based rough set approach to kansei
engineering in product development. Expert Systems with Applications,36 (1), 393–402.
Zhai, L.-Y., Khoo, L.-P., & Zhong, Z.-W. (2009b). A rough set based qfd approach to the management of
imprecise design information in product development. Advanced Engineering Informatics,23 (2),
222–228.
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018). A survey on deep learning for big data. Information
Fusion,42 , 146–157.
Zhang, X., Brown, H.-F., & Shankar, A. (2016). Data-driven personas: Constructing archetypal users
with clickstreams and user telemetry. In Proceedings of the 2016 chi conference on human factors
in computing systems (pp. 5350–5359).
Zhou, F., Alsaid, A., Blommer, M., Curry, R., Swaminathan, R., Kochhar, D., . . . Lei, B. (2020).
Driver fatigue transition prediction in highly automated driving using physiological features. Expert
Systems with Applications,147 , 113204.
Zhou, F., Ayoub, J., Xu, Q., & Jessie Yang, X. (2020). A machine learning approach to customer needs
analysis for product ecosystems. Journal of Mechanical Design,142 (1).
Zhou, F., Ji, Y., & Jiao, J. R. (2013). Affective and cognitive design for mass personalization: status
and prospect. Journal of Intelligent Manufacturing,24 (5), 1047–1069.
Zhou, F., Ji, Y., & Jiao, J. R. (2014a). Augmented affective-cognition for usability study of in-vehicle
system user interface. Journal of computing and information science in engineering,14 (2).
Zhou, F., Ji, Y., & Jiao, J. R. (2014b). Prospect-theoretic modeling of customer affective-cognitive
decisions under uncertainty for user experience design. IEEE Transactions on Human-Machine
Systems,44 (4), 468–483.
35
Zhou, F., Jiao, J. R., Chen, S., & Zhang, D. (2010). A case-driven ambient intelligence system for elderly
in-home assistance applications. IEEE Transactions on Systems, Man, and Cybernetics, Part C
(Applications and Reviews),41 (2), 179–189.
Zhou, F., Jiao, J. R., & Linsey, J. S. (2015). Latent customer needs elicitation by use case analogical
reasoning from sentiment analysis of online product reviews. Journal of Mechanical Design,137 (7).
Zhou, F., Jiao, J. R., Schaefer, D., & Chen, S. (2010). Hybrid association mining and refinement
for affective mapping in emotional design. Journal of Computing and Information Science in
Engineering,10 (3).
Zhou, F., Jiao, J. R., Xu, Q., & Takahashi, K. (2011). User experience modeling and simulation for
product ecosystem design based on fuzzy reasoning petri nets. IEEE transactions on systems, man,
and cybernetics-part A: Systems and humans,42 (1), 201–212.
Zhou, F., Kong, S., Fowlkes, C. C., Chen, T., & Lei, B. (2020). Fine-grained facial expression analysis
using dimensional emotion model. Neurocomputing.
Zhou, F., Lei, B., Liu, Y., & Jiao, J. R. (2017). Affective parameter shaping in user experience prospect
evaluation based on hierarchical bayesian estimation. Expert Systems with Applications,78 , 1–15.
Zhou, F., Qu, X., Helander, M. G., & Jiao, J. R. (2011). Affect prediction from physiological measures
via visual stimuli. International Journal of Human-Computer Studies,69 (12), 801–819.
Zhou, F., Qu, X., Jiao, J., & Helander, M. G. (2014). Emotion prediction from physiological signals:
A comparison study between visual and auditory elicitors. Interacting with computers,26 (3),
285–302.
Zhou, F., Xu, Q., & Jiao, J. R. (2011). Fundamentals of product ecosystem design for user experience.
Research in Engineering Design,22 (1), 43–61.
Zhou, F., Yang, X. J., & Zhang, X. (2020). Takeover transition in autonomous vehicles: a youtube
study. International Journal of Human–Computer Interaction,36 (3), 295–306.
36
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Miryoku engineering is a design concept based on customer preferences, with the goal of creating attractive products or spaces. However, traditional Miryoku engineering faces two main issues: (1) the upper Kansei factor ranks the weights by the number of mentions, but it does not represent the importance of customers; (2) the mapping connection between the upper Kansei factor and the lower specific conditions adopts a statistical analysis method, which easily leads to the omission of key information. With the development of computer-based artificial intelligence, it repeatedly simulates human thinking with simple calculation rules, which has the advantages of fewer errors and faster speed. Therefore, on the three-level evaluation grid diagram platform established by Miryoku engineering, this paper first uses grey relationship analysis to comprehensively evaluate the priority order of Kansei words. Secondly, for the key Kansei factors, a morphological deconstruction table that connects the original reasons and specific conditions is established. Orthogonal design is used to screen representative combinations of design elements and create sample models by using the 3D software. Finally, the neural network was used to establish a mapping function between the key Kansei factors and the representative product design elements, and based on this, the most perceptually attractive product design was discovered. As a case study, the automobile booth was used to validate the effectiveness of the proposed method and significantly improve exhibitor design decisions and attendees’ satisfaction.
Conference Paper
Full-text available
Human-Computer Integration (HInt) is an emerging paradigm in which computational and human systems are closely interwoven. Integrating computers with the human body is not new. however, we believe that with rapid technological advancements, increasing real-world deployments, and growing ethical and societal implications, it is critical to identify an agenda for future research. We present a set of challenges for HInt research, formulated over the course of a five-day workshop consisting of 29 experts who have designed, deployed and studied HInt systems. This agenda aims to guide researchers in a structured way towards a more coordinated and conscientious future of human-computer integration.
Book
Full-text available
Deep learning algorithms have brought a revolution to the computer vision community by introducing non-traditional and efficient solutions to several image-related problems that had long remained unsolved or partially addressed. This book presents a collection of eleven chapters, where each chapter explains deep learning principles for a specific topic, introduces reviews of up-to-date techniques, and presents research findings to the computer vision community. The book covers a broad scope of topics in deep learning concepts and applications such as accelerating the convolutional neural network inference on field-programmable gate arrays, fire detection in surveillance applications, face recognition, action and activity recognition, semantic segmentation for autonomous driving, aerial imagery registration, robot vision, tumor detection, and skin lesion segmentation as well as skin melanoma classification. The content of this book has been organized such that each chapter can be read independently from the others. The book is a valuable companion for researchers, for postgraduate and possibly senior undergraduate students who are taking an advanced course in related topics, and for those who are interested in deep learning with applications in computer vision, image processing, and pattern recognition.
Article
Streams of thought vary in content from one moment to the next, and these temporal patterns have been argued to be critical to understanding the wandering mind. But few analytic methods have been proposed that can account for both the content and temporal ordering of categorical experience sampling thought probes over time. In the present study, I apply sequence analytic methods to quantify the dynamics of thought from time series sequences of categorical experience sampling thought probes delivered across five different cognitive tasks in the same individuals (N = 545). Analyses revealed some patterns of consistency in streams of thought within individuals, but also demonstrated considerable variability within and between task sessions. Hierarchical clustering of sequence dissimilarities further revealed common typologies of mind wandering across individuals. These findings demonstrate the application of sequence analytic methods for quantifying the dynamics of thought over the course of task performance and show that contextual task constraints are associated with how streams of thought unfold over time. More broadly, sequence analysis provides a valuable framework for investigation of time ordered cognitive and behavioral processes across psychological domains.
Article
Kansei evaluation is crucial to the process of Kansei engineering. However, traditional methods are subjective and random. In order to eliminate the differences of individual evaluation criteria in product Kansei attributes evaluation, and further improve the evaluation efficiency, a novel automatic evaluation and labeling architecture for product Kansei attributes was proposed in this paper based on Convolutional Neural Networks (CNNs). The architecture consists of two modules: (1) Target detection module (Faster R-CNN was taken as an example), (2) Fine-Grained classification module (DFL-CNN was taken as an example). A case study was provided to validate the proposed architecture. The proposed architecture transformed design evaluation tasks into the recognition and classification tasks. The experiments achieved 98.837%, 96.899%, 86.047%, and 81.008% accuracy in the binary, triple, and two five-classification tasks, respectively. Our results proved the feasibility of using computer vision to mimic human vision for the automatic evaluation of Kansei attributes.
Article
Automated facial expression analysis has a variety of applications in human-computer interaction. Traditional methods mainly analyze prototypical facial expressions of no more than eight discrete emotions as a classification task. However, in practice, spontaneous facial expressions in a naturalistic environment can represent not only a wide range of emotions, but also different intensities within an emotion family. In such situations, these methods are not reliable or adequate. In this paper, we propose to train deep convolutional neural networks (CNNs) to analyze facial expressions explainable in a dimensional emotion model. The proposed method accommodates not only a set of basic emotion expressions, but also a full range of other emotions and subtle emotion intensities that we both feel in ourselves and perceive in others in our daily life. Specifically, we first mapped facial expressions into dimensional measures so that we transformed facial expression analysis from a classification problem to a regression one. We then tested our CNN-based methods for facial expression regression and these methods demonstrated promising performance. Moreover, we improved our method by a bilinear pooling which encodes second-order statistics of features. We showed that such bilinear-CNN models significantly outperformed their respective baseline models.
Article
The authors present a Generative Adversarial Network (GAN) model that learns how to generate 3D models in their native format so that they can either be evaluated using complex simulation environments, or realized using methods such as additive manufacturing. Once initially trained, the GAN can create additional training data itself by generating new designs, evaluating them in a physics-based virtual environment, and adding the high performing ones to the training set. A case study involving a GAN model that is initially trained on 4045 3D aircraft models is used for demonstration, where the training data set updated with GAN-generated and evaluated designs, results in enhanced model generation, both in the geometric feasibility and performance of the designs. Z-tests on the performance scores of the generated aircraft models indicate a statistically significant improvement in the functionality of the generated models after three iterations of the training-evaluation process. In the case study, a number of techniques are explored to structure the generate-evaluate process in order to balance the need to generate feasible designs with the need for innovative designs.