Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
SPECIAL ISSUE ARTICLE
Optimizing the digital customer journey—Improving user
experience by exploiting emotions, personas and situations
for individualized user interface adaptations
Christian Märtin | Bärbel Christine Bissinger | Pietro Asta
Faculty of Computer Science, Augsburg
University of Applied Sciences, Augsburg,
Germany
Correspondence
Christian Märtin, Faculty of Computer Science,
Augsburg University of Applied Sciences,
Augsburg, Germany.
Email: christian.maertin@hs-augsburg.de
Abstract
This paper discusses a novel approach for exploiting emotions and situation-aware
software adaptation methods for individualizing some of the touch points of the digital
customer journey and thereby optimizing customer experience and effectiveness of
e-commerce applications. Our approach uses emotion recognition, eye-tracking, and other
individual tracking methods as well as customer personas for adapting interactive web
applications by accessing a flexible adaptation framework at runtime. The framework
allows for individualization at runtime by applying situation-aware adaptations. Two
experimental customer studies were carried out in the e-commerce domain in order to
provide a basis for exploitable emotion- and persona-related situational changes. The
results of the studies were used to demonstrate the potential of our situation analytic
adaptation approach with examples from a commercial beauty-products e-business portal.
1|INTRODUCTION
One aspect of digitalization in marketing is the design of IT-based solu-
tions for the steps or cycles of the customer journey (Følstad &
Kvale, 2018). The customer journey is the customer's interaction at sev-
eral touch points with a service or several services of one or more service
providers in order to achieve a specific goal (Halvorsrud et al., 2016).
A generic customer journey is divided into the following five
phases: awareness, where the customer is made aware of the product
or service, favorability, where the interest of the customer is
increased, so that the customer begins to take a closer look at the
product and to inform herself about it, consideration, which increas-
ingly triggers the customer's desire to own the product, intent to
purchase, where the customer's intention to buy the product is being
initiated, and, finally conversion, where the product will ultimately be
bought by the customer. It also makes sense to add a post purchase
phase to the customer journey. A distinction is made between direct
and indirect touch points. The website, advertising spots and
advertisement in general are understood as direct touch points. Indi-
rect touch points for example include rating portals, user forums and
blogs and can only be influenced to a limited extent by the provider.
Optimizing the customer journey is very important and necessary
for the success of e-commerce providers. By using suitable tracking
technologies, the behavior of consumers can be analyzed in real-time.
Analysis can also reveal all contact points created through advertising.
Thanks to this knowledge, it is possible to identify optimization poten-
tial. It must be the provider's goal at every touch point, to create a sit-
uation that leads to optimum user experience (UX) for the potential
customer (Stein & Ramaseshan, 2016). UX in the customer journey is
often described as customer experience (CX). CX aims at “a cus-
tomer's cognitive, emotional, behavioral, sensorial, and social”reac-
tions to the offerings of a provider or a business “during the
customer's entire purchase journey”(Lemon & Verhoef, 2016).
This paper is a revised and extended version of (Märtin et al., 2020).
In our previous research we have designed and engineered the SitAdapt
system that is able to observe activities and recognize emotions of users
Received: 1 November 2020 Revised: 10 March 2021 Accepted: 18 May 2021
DOI: 10.1002/cb.1964
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any
medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
© 2021 The Authors. Journal of Consumer Behaviour published by John Wiley & Sons Ltd.
J Consumer Behav. 2021;1–12. wileyonlinelibrary.com/journal/cb 1
during their work in any interactive application environment. By applying
situation rules the system exploits the gathered data in order to individu-
ally adapt the user interface and the behavior of the observed interactive
application (Märtin et al., 2019).
In this paper we lay the foundation for using SitAdapt for accom-
panying individual users during the different phases of the customer
journey in the e-business domain. We create a suitable experimental
setting for analyzing the following research questions:
•Q1: When are emotions triggered in e-business environments and
how can they be exploited for optimizing the digital customer
journey?
•Q2: Can predefined personas lead to better customer experience
in e-business environments?
•Q3: Can direct observation of users lead to more individualized
adaptations of the interactive e-business applications?
•Q4: Can situation-aware adaptations help to achieve better task
accomplishment of the users?
•Q5: Can situation-aware adaptations help to achieve the providers'
business goals more effectively?
The data we collected during our experimental studies give prelimi-
nary answers to these research questions and provide valuable input for
triggering and targeting larger and more specific subsequent studies.
The remainder of this paper is organized as follows:
•Section 2 gives an overview of related and previous work relevant
to the scope of this paper.
•Section 3 presents two experimental studies that evaluate the role
of emotions and customer personas in order to raise customer
experience for the b2c part of an existing e-commerce portal for
beauty products.
•Section 4 discusses, how SitAdapt can exploit the findings and results
of the studies by specifying and triggering appropriate situation rules.
•Section 5 concludes the paper and gives an outlook on our planned
future work.
2|MATERIALS AND METHODS
This section discusses some of the methods from the disciplines that
have influenced and inspired our work. We also discuss the function-
ality of the SitAdapt system.
2.1 |Emotions, decision-making and user
experience
2.1.1 | Emotions and decision-making
In our society, emotions and emotional reactions are often perceived
with a negative connotation even though emotions play an important
role in all human actions. Emotions directly influence interaction
between humans as well as human–computer-interaction (HCI). This
means that emotions are not only linked to social environments or to
the entertainment industry, but that they are also essential for human
communication, human perception and decision-making (Picard, 2000).
In the past, scientists were assuming that human beings primarily
behave in a rational way. Today's expertise and research prove the
opposite. It is now known that emotions trigger actions and are
involved in human processes like
•Evaluation of situations
•Motivation
•Preparation
•Actions
•Regulations
•Social Tasks (Broschart & Monschein, 2017)
These processes cover a broad field of behavior and activities. It
can be seen that the power of emotions influences the decision mak-
ing processes in many situations and areas of life.
In 1994, a study by the neuroscientist Ant
onio Dam
asio showed,
how emotions affect decision making. Using a strategic card game,
Dam
asio could prove that emotions are essential for long-term plan-
ning. Some of the participants of the study had a disorder of the cere-
bral cortex and were not able to evaluate actions emotionally. These
players were at a disadvantage compared to the other participants,
because the average persons could use their emotions to help them
make long-term decisions which was necessary in this strategic card
game (Dam
asio, 2006; Müsseler & Rieger, 2017).
We human beings are often not aware of how much influence
emotions have on our choices. Even if we assume that we make a
decision based on rational reasons, this is usually not the case - espe-
cially for long-term considerations. In some situations, it is also possi-
ble that emotions are the only reason for our judgment. In these
cases, no logical element is necessary to pass a resolution. However,
since we often consider a logical component to be important in our
judgment, the logical element is added afterwards in order to rein-
force and rationalize the emotionally made decision (Broschart &
Monschein, 2017). This is how people avoid negative emotions, which
would arise when opinions, attitudes, perceptions or thoughts do not
match. This phenomenon is called cognitive dissonance, which is per-
ceived as an uncomfortable tension for people and which needs to be
reduced in order to build a coherent overall picture and a harmonious
balance (Moser, 2015). The opinion that decisions need to be made
rational leads to the phenomenon that humans subconsciously add a
logical component after the decision has already been made. This
shows that supposedly rational decisions can often be traced back to
emotions without people being aware of it.
Even though emotions have such a high significance, there is no
explicit definition for it and many different statements can be found.
Kleinginna and Kleinginna have collected the similarities of around
100 assertions and definitions and have created a working definition
that combines cognitive, mental, physiological and physical aspects
(Kleinginna Jr. & Kleinginna, 1981). One possibility for categorization
2MÄRTIN ET AL.
are fundamental emotion models which are based on Darwin's find-
ings of basic emotions. A famous example are the six basic emotions
discussed by Ekman, which refer to facial expressions and should be
valid regardless of a person's language and culture (Ekman, 1992a;
Ekman, 1992b; Stürmer & Schmidt, 2014). Another option to catego-
rize emotions are dimensional models. Two prominent dimensions are
valence and arousal which are for example used in the model of
Russel. The dimension of valence indicates the extent to which the
emotion is felt positively or negatively. The arousal dimension shows
how strong the emotion is and how relevant the environment is (Val-
enza et al., 2014).
2.1.2 | Emotions and user experience
Since emotions play a central role in human behavior and decision-
making, they are meaningful for marketing and advertising measures,
in classical and digital marketing. The choices of the customers deter-
mine whether a product or a company will be accepted or declined in
the market. Consumer excitement, great anticipation and loyalty to a
brand or products are achievements which can only be reached by
creating and maintaining emotional value.
This is not only valid for an individual product, but for the whole
experience users gather with the products, brands or companies.
While usability describes the extent to which an interactive system is
effective, efficient and satisfying to use in a specified context of use, user
experience (UX) considers satisfaction before, during and after use
(UXQB, 2020).
For online presence or digital touch points a smooth interaction
due to compliance with usability rules, that is, an expected and fault-
less operation, is nowadays a prerequisite. A friendly user interface
still leads to a competitive advantage, but does not mean that users
are addressed emotionally or that they have a great experience by
using the interface. This is only achieved when users feel excitement
and a joy of use (Dorau, 2011).
When emotions are taken into account while creating and
adapting user interfaces, more intelligent and human like interfaces
can be designed. The attention of the user can thus be controlled
more easily, which results in a more natural perception for the user.
Computer systems and user interfaces can be adapted to people and
not the other way around (Picard, 2000).
2.2 |Business intelligence in e-commerce
Whenever consumers enter the individual phases of the customer
journey and get into contact with a provider through direct and indi-
rect touch points, huge amounts of data are produced and transmitted
to the respective provider. In order to generate valuable knowledge
from such data, business intelligence (BI) can be applied. BI combines
the use of various methods and technologies that serve the collection,
administration, evaluation and presentation of mainly external but also
internal data in digital form (Gluchwoski & Chamoni, 2016). By using
adequate tools data patterns, cross-connections and correlations can
be discovered and trends can be predicted. In the focus of the applied
analytical methods are data mining that uses statistical algorithms as
well as the hypothesis-based online analytics processing (OLAP),
which allows a multi-dimensional analysis of data. In (Chen
et al., 2012) the authors predicted for the now current generation of
BI and analysis tools to focus on mobile and sensor-based content,
location-awareness, person-centered and context-relevant analysis as
well as mobile visualization techniques and HCI aspects.
Results from BI analysis can have an impact on the design of e-
commerce sites and provide insights on how to organize content
parts, advertising, navigation and presentational aspects. In our
approach the large quantities of user and situational data recorded by
the SitAdapt 2.1 recording component (see Section 2.4) can be
accessed by BI tools. Analysis results can therefore evolutionarily
influence customer experience in all phases and for all direct touch
points in future sessions.
2.3 |Persona-based design
Another technique with positive impact on the customer journey and
e-commerce in general can be the use of personas. In the context of
this paper a “persona is an archetype of a class of users synthesizing
goals and behavior patterns as well as skills, attitudes and environ-
ment. The user's characteristics […] must be effective for the design
problem at hand.”(De Marsico & Levialdi, 2004). For each persona, an
avatar is defined, that is equipped with authentic features such as
name, photo, curriculum vitae, age, income, marital status, hobbies,
education, etc. (Klünen, 2019). In our approach to optimizing the cus-
tomer journey, a persona represents a certain customer group of a
provider. Most information required to design personas, as well as all
knowledge about their preferences, results from expert analysis of
existing customer data. The use of personas helps us to create a cus-
tomized approach for specific customer groups, that is, to work with
specific visual, verbal and audio attributes. Also, persona-specific
product preferences are taken into account. This allows the persona-
specific tailoring of the marketing measures, in particular the choice of
advertising material to be used, as well as of the web design of the e-
commerce site.
Personas are also often used in HCI and software engineering
(Jahavery et al., 2009). In (McGinn & Kotamraju, 2008) an alternative
approach to finding personas is discussed: In order to reach specific
HCI design goals, a survey was conducted with members of the differ-
ent target groups, resulting in useful personas with attributes specific
to tasks of each user group, but without organizational overhead.
2.4 |Context- and situation-awareness
The concept of context-aware computing was first proposed for dis-
tributed mobile computing by (Schilit & Theimer, 1994). In addition to
technical aspects the definition of context also included
MÄRTIN ET AL.3
environmental and social attributes. Later the term situation-
awareness appeared in psychology and the cognitive sciences with
the aim to support correct task accomplishment of human operators
in complex situations by defining situation-dependent guidance (Flach
et al., 2004). In recent years interactive software has made huge steps
towards understanding of and reacting to varying situations. To cap-
ture the individual requirements of a situation, (Chang, 2016) pro-
poses that a situation consists of an environmental context Ethat
covers the user's operational environment, a behavioral context Bthat
covers the user's social behavior by interpreting his or her actions,
and a hidden context Mthat includes the users' mental states and
emotions.
2.4.1 | SitAdapt
Our own work, the SitAdapt system (Märtin et al., 2019), (Herdin &
Märtin, 2020) was inspired by Chang. The system (Figure 1) uses a
broad set of visual and other observation and monitoring tools and
has been tested and evaluated for a number of applications from
different e-business domains. All data recorded in a user session
are stored in very fine-grained situation profiles (minimum time res-
olution: 1/60 s). Possible adaptations of the target web application
are planned and premodeled at development time. At runtime they
are triggered by situation rules and generated by activating and
exploiting domain-dependent and independent actions and/or HCI-
patterns. For applying SitAdapt inthee-commercedomainwehave
exploited the results of two experimental user studies in order to
optimize our approach (see Section 3). The current software ver-
sion, the SitAdapt 2.1 system, is now able to cover broad parts of
the customer journey. Section 4 demonstrates the capabilities of
the SitAdapt system with application examples from e-commerce.
2.4.2 | SitAdapt architecture
SitAdapt 2.1 consists of the following parts:
•The data interfaces use the different APIs of the devices (eye-
tracker, wristband, facial expression recognition software interface,
metadata from the application) to collect data about the user.
SitAdapt 2.1 uses two different data types for generation and
adaptation of the user interface received from the different input
devices.
FIGURE 1 Structure and
components of the SitAdapt system
(Märtin et al., 2019) [Colour figure can
be viewed at wileyonlinelibrary.com]
TABLE 1 Data input from the eye tracking system (Herdin &
Märtin, 2020)
Attribute Possible values Description
LeftPupilDiameter Between circa
2.0 and 8.0
Describes the dilation of
the subject's left pupil in
mm
RightPupilDiameter Between circa
2.0 and 8.0
Describes the dilation of
the subject's right pupil
in mm
LeftEyeX Between 0.0
and 1.0
Indicates the normalized x-
coordinate of the
subject's left eye
LeftEyeY Between 0.0
and 1.0
Indicates the normalized y-
coordinate of the
subject's left eye
RightEyeX Between 0.0
and 1.0
Indicates the normalized x-
coordinate of the
subject's right eye
RightEyeY Between 0.0
and 1.0
Indicates the normalized y-
coordinate of the
subject's right eye
4MÄRTIN ET AL.
•The recording component synchronizes the different input records
with a timestamp. In Table 1, for instance, the attribute value ranges are
listed that can be received from the eye tracking system API. Table 2
shows the possible attributes and values from emotion tracking.
•The database writer stores the data from the recording component
and from the browser in the database, where the raw situations
and situation profiles are managed. It also controls the communica-
tion with the rule editor.
•The rule editor (Figure 2) allows the definition and modification of situ-
ation rules, for example, for specifying the different user states and
the resulting actions. The rule editor can use all input data types and
attribute values as well as their temporal changes for formulating rule
conditions. At runtime rules are triggered by the situation analytics
component for adapting the user interface, if the conditions of one or
more rules apply. However, situation rules can also activate HCI-
patterns in a pattern repository (Märtin et al., 2019).
•The situation analytics component analyzes and assesses situations
by exploiting the observed data. Situation rules are triggered when
the rule conditions are satisfied. Situation rules interact with the
situation profiles stored in the SitAdapt 2.1 database.
•The evaluation/decision component uses the data that are provided
by the situation analytics component to decide whether an adapta-
tion of the user interface is currently meaningful and necessary.
The component evaluates one or more applicable situation rules
and has to solve possible conflicts between the rules.
•The adaptation component generates the necessary modifications
of the interactive target user application.
3|RESULTS
3.1 |Experimental user study 1: Can emotions be
exploited for UX optimization in e-commerce?
To discover what triggers emotions of users, and how this information
can be used to optimize the user experience, we did a test in our lab
with customers and potential customers of Dr. Grandel GmbH, a Ger-
man manufacturer in the cosmetics sector. The experimental study
was focused on emotional responses at digital touch points. In the
study the real world b2c website of the company with an interactive
tool, an online shop and an online magazine was used. Other elements
were three different online advertising measures.
We wanted to find out whether the usage of these different ele-
ments triggers emotional responses of the users. In case they do, we
wanted to know which emotions were shown and what the concrete
triggers for the measured responses were.
3.1.1 | Research subject
The first part of the study was centered around an interactive tool on
the website, which helps users to find the most suitable products for
them. The users were asked to describe themselves with the support
of predefined answers and pictures. This is a very individual procedure
and lots of decisions need to be made that involve emotions. Another
part of the research with the website was focused on the online shop
and the process from product search to product selection. The last
element of the test that was related to the website included an
online-magazine with editorial articles for entertainment and news
about trends and products in the cosmetics industry. Additionally,
selected advertisements on external websites, advertising banners on
the own online channels as well as newsletters and Instagram posts
were analyzed. All these elements were tested in order to find out,
whether they trigger emotions that can be measured.
According to Jakob Nielsen, the best results for the usability eval-
uation of an interactive application will appear when only five users
are testing the target object with as many small tests as possible
(Williams, 2004). As we differentiated between two user groups, we
used two groups, each of them comprising of four people, which is
recommended when testing with different user groups (Tullis &
Albert, 2013).
One group consisted of female customers of the company who
already knew the company and its products. The second group con-
sisted of female users, who did not know the products yet, but who
did fit into the target group of the company's products.
3.1.2 | Test setting and tools
The experimental study included a pretest to gather information
about the earlier experience of the participants and to assign the par-
ticipants to the different groups, a lab test, which was the main part
of the study, and a post-test to enquire the individual personal opin-
ion of the participants.
The lab test was divided into three test scenarios with eight tasks
which were building on one another. During the test, the participant
was seated in an ordinary office room in front of a personal computer.
The researchers could observe the participants through a one-way
mirrored window and with a webcam that recorded the sessions and
logged interaction details.
To measure emotional responses of the participants during differ-
ent stimuli, Tobii Studio eye-tracking software, FaceReader 7 facial
expression recognition software and an Empatica E4 wristband to
measure the heart rate and skin conductance as indicators for some
emotional states were used.
3.1.3 | Selected results of the study
Interactive tool
The interactive tool with the individual result could trigger emotions
which could be measured. Figure 3 shows a section of the results of
the facial expression recognition software during the interaction of a
participant with the tool. Since emotions do not always occur in
a pure form, it is possible that the facial expression shows several
MÄRTIN ET AL.5
emotional states. The FaceReader software therefore shows the ana-
lyzed expressions proportionally. For classifying basic emotions
FaceReader uses a deep neural network trained with 10,000
annotated facial images for detecting the basic emotions in the algo-
rithmically runtime-generated 3D model which shows the exact posi-
tion of 500 relevant key points in the currently observed face. The
TABLE 2 Input data from emotion tracking (Märtin et al., 2019)
Attribute Possible values Description
Angry between 0.0 and 1.0 Indicates to what degree the user appears
angry
Disgusted between 0.0 and 1.0 Indicates to what degree the user appears
disgusted
Happy between 0.0 and 1.0 Indicates to what degree the user appears
happy
Neutral between 0.0 and 1.0 Indicates to what degree the user appears
neutral
Sad between 0.0 and 1.0 Indicates to what degree the user appears
sad
Scared between 0.0 and 1.0 Indicates to what degree the user appears
scared.
Surprised between 0.0 and 1.0 Indicates to what degree the user appears
surprised
Contempt between 0.0 and 1.0 Indicates to what degree the user appears
full of contempt
Contempt is defined as the feeling that a
person or a thing
is beneath consideration, worthless, or
deserving scorn
Valence between 1.0 and 1.0 Describes how positive/negative an
emotion is. 1 means very negative.
Arousal between 0.0 and 1.0 Describes how strong the emotion is. A
person that is yelling in anger, for
example, has a higher arousal than a
person who is only pulling their eyebrows
together.
Quality between 0.0 and 1.0 Roughly describes the quality of the
observation
Age Agerange (“from-to”) Describes a numeric interval for the
estimated age of the user
Beard ”None,”“Some,”“Full”Describes the extent of the subject's lower
facial hair
Moustache “None,”“Some,”“Full”Describes the extent of the subject's upper
facial hair.
Glasses ”Yes,”“No”Describes whether or not the user is
wearing glasses
Ethnicity “Caucasian,”“Eastern Asian,”“South
Asian,”“African,”“Other”
Describes the ethnicity of the subject based
on visual appearance.
Gaze Direction Identity “Left,”“Right,”“Forward”Describes in which direction the subject is
looking.
Identity “unknown person,”“no identification”Identifies the user based on the current
Noldus Face-Reader session's profiles.
Left Eye “Open,”“Closed”Describes the state of the user's left eye.
Right Eye “Open,”“Closed”Describes the state of the user's right eye.
Mouth “Open,”“Closed”Describes the state of the user's mouth.
Left Eyebrow “Raised,”“Lowered,”“Neutral”Describes the state of the user's left
eyebrow.
Right Eyebrow “Raised,”“Lowered,”“Neutral”Describes the state of the user's right
eyebrow.
6MÄRTIN ET AL.
FaceReader software measures the intensity of each basic emotion in
a range between 0 and 1. Note, however, that the sum of the intensi-
ties of all currently involved emotions is not typically equal to 1, as
the intensity of each emotion is handled separately (Loijens &
Krips, 2019). In this example, the dominant expression of the partici-
pant during the usage of the tool is happy which proves a positive user
experience during the interaction for this user.
The results also show (see Figure 4a,b) that the arousal and the
valence, which are indicators for emotional reactions and are also
measured by FaceReader, change with the beginning of the interac-
tion with the tool. The valence indicates, whether an emotional state
is positive or negative. The arousal shows the activeness of an emo-
tional state. This example illustrates the correlation of valence and
arousal for physical reactions. In the second part of the graph, at
09:37:00, the interaction with the tool starts. For the observed user
the graph proves that the activeness of emotional states is higher and
more positive during an interactive touchpoint.
Magazine and newsletter
Figure 5 shows the emotional reaction of one of the participants,
when an unexpected visual appeared, after a button leading to some
promotional action was pressed. The values of the basic emotions
happy and surprised are reaching quite significant levels.
In the next example (Figure 6) a participant is looking at a specific
winter skin cream. Upon reading the detailed description of the prod-
uct Winter Silk Cream, the user's emotional state significantly changes
to happy. A situation rule could now exploit this knowledge to give
additional information about other winter products.
Advertisements
In the next example (Figure 7a,b), the system has gathered a priori
knowledge about the varying gaze behavior of participants, who are
known customers of the business or participants who do not know
the products by distinguishing between the lab-created heat maps.
These heat maps show that customers focus on the images, while
noncustomers concentrate on the textual description of the products.
The gaze behavior with respect to this image can be used to catego-
rize anonymous users. The customer experience during the
FIGURE 2 SitAdapt rule editor (Märtin et al., 2019) [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 3 Emotions measured by FaceReader 7 software [Colour
figure can be viewed at wileyonlinelibrary.com]
MÄRTIN ET AL.7
prepurchase phase can be improved. When the system assumes a
returning customer, the focus of her further customer journey will
be put on showing esthetic images, while in the other case more
descriptive information will be given during the rest of the customer
journey.
These examples of our experimental findings show that the potential
for UX optimization in e-commerce by exploiting emotions is very high
and promising. Thus research question Q1 can be answered positively.
The findings provide valuable input for the analysis of research questions
Q3, Q4, and Q5 in Section 4. In Section 4 we discuss, how the observation
FIGURE 4 (a) Valence of measured emotions during the test for a specific participant; (b) Arousal of measured emotions during the test for a
specific participant [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 5 Visual for a special promotion and measured emotional values [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 6 A significant value of the emotion “happy”is observed for a candidate, when she is finding the right product [Colour figure can be
viewed at wileyonlinelibrary.com]
8MÄRTIN ET AL.
of changes of the emotional and other attribute values can be exploited
for triggering situation rules and runtime adaptations.
3.2 |Experimental user study 2: How personas can
be exploited for optimizing customer experience
Our co-operation partner has been using personas in digital marketing
for a long time. The three recent personas were derived from cus-
tomer data and refined in a workshop of the marketing and
e-commerce teams. The personas reflect the company's main cus-
tomer groups. Each persona profile contains a short biography as well
as information on personality, age, buying behavior, e-commerce and
web affinity, motivation to buy, and the preferred reference channels.
A psychography is also included.
In order to provide a justification for the preconfigured persona-
specific web shop adaptations and to receive important information
about product preferences, a usability test was carried out in our lab
with 10 female test subjects, who represented the three persona-
groups, but so far had not been customers of the company. Test sce-
narios were developed for gaining insights into product preferences
as well as persona-specific visual and verbal design. For example, the
test subjects had to simulate the shopping process as part of a test
task and put products into their shopping cart. It was considered
whether they would also buy the specific products intended for their
persona-group. Which products are preferably bought by which per-
sona had been determined in advance by analyzing real
customer data.
In a different task the test candidates had to search the magazine
on the company's website and spontaneously click on three articles
that particularly appealed to them. It was checked, which articles were
clicked on and whether subjects from the same persona-group chose
the same articles. In addition, it was tested on which words and verbal
approaches the said persons paid particular attention. The participants
were also shown pictures from the company's Instagram account.
Each participant could select any number of pictures. As with the
other tasks before, the existence of persona-specific relationships,
especially with regard to visual incentives, was checked.
The knowledge gained through the laboratory tests showed
that some, but not all preconfigured adaptations and product
offerings that were based on data analysis results could be asso-
ciated with the relevant personas. Therefore, with respect to the
FIGURE 7 (a) Gaze heat-map of regular customers of the business; (b) Gaze heat-map of new visitors of the website [Colour figure can be
viewed at wileyonlinelibrary.com]
MÄRTIN ET AL.9
tested web-shop environment and the predefined personas, there
was no clear positive answer to research question Q2. However,
the evaluation results from the visual and textual selection tests
were used to re-design the persona-specific configurations of the
website. This will lead to significant optimizations of the aware-
ness, consideration and conversion phases of the customer jour-
ney. The results have also given us insight into which types of
visual and verbal information found on e-business sites are prom-
ising candidates for being exploited in situation-rules for individ-
ual runtime-adaptation.
4|DISCUSSION
This section discusses, how the findings of the experimental stud-
ies can be exploited for generating individualized situation-aware
adaptations at runtime with SitAdapt. In Figure 5 the emotional
reaction of a test subject is shown, when an unexpected visual
appeared, after a button leading to some promotional action was
pressed. The values of the basic emotions happy and surprised are
reaching quite significant levels. These data are available in the
SitAdapt 2.1 user profile, whereas the current situation (applica-
tion meta-data, visuals, etc.) is recorded during the entire user
session and stored in the situation profile in the SitAdapt data-
base.Foralltouchpointsthatare visited during this session the
informationinthesituationprofilecanbeaccessedatruntime.
This combined information can be exploited by a SitAdapt 2.1 sit-
uation rule that triggers adaptations at runtime, for example, for
promoting a special offer:
<SituationRule> SpecialPromotion
FOR N <Situation
i
>IN3s
<Gaze_Tracking> Contains Field Promotion
Image(Id) (>0)
AND <Emotion> happy (>0.30)
AND <Emotion> surprised (>0.20)
<Action> SHOW AT 10s Promotion1TextFUI
<Action> WAIT Promotion1TextFUIInput
<Action> LINK Promotion1TextFUIInput
TaskModel PromotionProcessingTask
(Promotion1)
As explained in paragraph 3.1.1 the intensities of emotions mea-
sured by FaceReader 7 can vary between 0 and 1. In our tests we
found that above a certain threshold each emotion can also clearly be
noticed by a human observer. The levels chosen in our situation rules
represent such clearly perceptible levels.
Thus, if the user focuses on the image and after 3 s the emotions
have changed to perceptible levels, a promotion dialog box is dis-
played in the user interface. The web application will wait for a user
input and activate the promotion processing task specified in the task
model.
In the next example we specify a situation rule that could be used
in the situation in the web shop environment that is shown in
Figure 6 where SitAdapt recognizes the user's interest in a certain
product, that is, the winter silk cream, because of the repeated gazing
to the field and the raised level of emotion “happy.”After 2 min a text
is displayed that notifies the user that in case of the purchase of prod-
uct (Id) within a certain time span, a voucher is granted for the user's
next purchase. A link to the voucher-processing task in the task model
is activated:
<SituationRule> OfferingVoucher
FOR N <Situation
i
> IN 120s
<Eye_Tracking> Field Product Product(Id)
<Gaze_Tracking> Contains Field Product
Product(Id) (>5)
AND <Pulse> (85-100)
AND <PulseRate> rising
AND <Emotion> happy (>0.25)
AND <StressLevel> orange
<Action> SHOW AT 120s VoucherText1FUI
<Action> WAIT VoucherText1FUIInput
<Action> LINK VoucherText1FUIInput
TaskModel VoucherProcessingTask
The activation of the voucher processing task in the task model at
runtime will trigger the generation of a dialog box with the voucher
text and the possible user options. A special reusable pattern for this
task could be available in the pattern repository. At runtime the indi-
vidual product would be mapped to the pattern and the final user
interface with the voucher text and the possible user options would
be displayed.
Another interesting topic would be the automatic recognition
of the persona of an unknown customer. Figure 7a,b from the
first experimental study demonstrate the different gazing behav-
ior of two different groups of customers. Such knowledge in the
form of heat-maps could also be assembled for a group of regular
customers, whose persona is known. In addition, the emotional
and gazing behavior ranges of the members of a specific persona-
group could be measured during some typical standard tasks
when working with the website. The results of the persona-
related experimental study described in paragraph 3.2 have pro-
vided knowledge about which visual and textual characteristics of
web user interfaces and business content can be correlated to a
specific persona. Our findings suggest that atmospheric settings
showing real persons with different styling and hair or fashion
colors rather allow the selection of a specific persona than the
visual confrontation of a candidate with the product basket asso-
ciated with the personas by a-priori analysis of real customer
data. We are currently defining the most promising relationships
between visual content characteristics and specific personas. The
best matches will then be coded as the left-hand-sides of a group
of persona-related situation-rules.
A decision about the probable persona of a new customer could
be made with such rules and the persona-specific content, style and
layout adjustments of the website could be triggered by the SitAdapt
system.
10 MÄRTIN ET AL.
By assessing the potential of our experimental results and
their exploitation by the situation-aware adaptation process pos-
sible with the SitAdapt system we can clearly state that our lab-
based approach paves the way for giving positive answers to our
research questions Q3, Q4, and Q5. We are now able to orches-
trate larger studies with fine-tuned scenarios in order to statisti-
cally evaluate our experimental findings.
5|CONCLUSION AND FUTURE WORK
Inthispaperwehavediscussedanewsoftwareapproachfor
accompanying and optimizing the customer journey that combines
analysis of existing customers' data, persona-based design and
situation-aware runtime adaptations in order to allow for better
task accomplishment of both, the e-commerce provider, and the cli-
ent. Two experimental studies directed on measuring usability and
user experience for a real-world b2c e-commerce portal for beauty
products have demonstrated that a high-level of customer experi-
ence can be achieved, if the capabilities of the available tools are
exploited consequently.
We are currently evaluating the observation data of a larger study
conducted between November 2020 and January 2021 targeted on
emotion recognition in visual and motion picture settings where, in addi-
tion to the SitAdapt version described in this paper, the participants also
used an EEG-based brain-computer interface (BCI). We will carefully ana-
lyze the results of this new study and consider the consequences for the
creation of new e-business scenarios.
The next step will be to get the SitAdapt 2.1 system from the lab
into real life by using advanced camera-based visual tracking technol-
ogy. Also, a set of situation rules that fulfill the standards and require-
ments of both service providers and customers has to be specified
and evaluated. For optimizing the quality of the situation rules we are
planning to add a reinforcement-learning component to the SitAdapt
system.
In contrast to using SitAdapt within a pure lab environment, the
real-world application of such an individualized emotion tracking and
adaptation system could lead to major privacy and ethical issues. In
order to manage these aspects, we will apply accepted principles of
value sensitive design as, e.g., discussed in (Friedman & Hendry, 2019).
ACKNOWLEDGMENTS
Part of this work was carried out in cooperation with Dr. Grandel
GmbH, Augsburg, Germany. We greatly acknowledge the opportunity
to run the SitAdapt tools and user tests on their enterprise e-business
platform. We also greatly acknowledge the valuable hints of the
reviewers.
DATA AVAILABILITY STATEMENT
Data available on request from the authors
ORCID
Christian Märtin https://orcid.org/0000-0002-3646-1920
REFERENCES
Broschart, S., & Monschein, R. (2017). Der Content Faktor. Schreiben Sie
Texte, die gefunden und gelesen werden (Professional Series, 1. Auflage).
Franzis.
Chang, C. K. (2016). Situation analytics: A foundation for a new software
engineering paradigm. IEEE Computer,2016,24–33.
Chen, H., Chiang, R. H. L., & Storey, V. C. (2012). Business intelligence and
analytics: From big data to big impact. MIS Quarterly,36(4), 1165–1188.
Dam
asio, A. (2006). Descartes' error. Vintage Books.
De Marsico, M., & Levialdi, S. (2004). Evaluating web sites: Exploiting
user's expectations. International Journal of Human-Computer Studies,
60(3), 381–416.
Dorau, R. (2011). Emotionales Interaktionsdesign. Gesten und Mimik inter-
aktiver Systeme. Springer.
Ekman, P. (1992a). Are there basic emotions? Psychological Review 1992,
99(3), 550–553.
Ekman, P. (1992b). An argument for basic emotions. Cognition and Emotion,
1992,6(3/4), 169–200.
Flach, J. M., Mulder, M., & Van Paassen, M. M. (2004). The concept of situ-
ation in psychology. In S. Banbury & S. Tremblay (Eds.), A cognitive
approach to situation awareness: Theory and applications (pp. 42–60).
Ashgate Publishing.
Følstad A., & Kvale K. (2018). Customer journeys: a systematic literature
review. Journal of Service Theory and Practice,28(2), 196–227. http://
doi.org/10.1108/jstp-11-2014-0261.
Friedman, B., & Hendry, D. G. (2019). Value sensitive design: Shaping tech-
nology with moral imagination. MIT Press.
Gluchwoski, P., & Chamoni, P. (2016). Analytische Informationssysteme.
Springer-Verlag.
Halvorsrud, R., Kvale, K., Følstad, A. (2016). Improving Service Quality
through Customer Journey Analysis, J. of service theory and practice,
vol. 26(6), 840–867.
Herdin, C., & Märtin, C. (2020). Modeling and runtime generation of
situation-aware adaptations. In M. Kurosu (Ed.), HCII 2020, LNCS
12181 (pp. 71–81, Springer Nature Switzerland). Cham, Switzerland:
Springer Nature. https://doi.org/10.1007/978-3-030-49059-1_5
Jahavery, H., Deichman, A., Seffah, A., & Taleb, M. (2009). A user-centered
framework for deriving a conceptual design from user experiences:
Leveraging personas and patterns to create usable designs. In A.
Seffah, J. Vanderdonckt, & M. C. Desmarais (Eds.), Human-centered
software engineering (pp. 53–81). Springer.
Kleinginna, P. R., Jr., & Kleinginna, A. M. (1981). A categorized list of emo-
tion definitions, with suggestions for a consensual definition. Motiva-
tion and Emotion,5(4), 345–379.
Klünen, K. (2019). Persona, Gablers Wirtschaftslexikon, Retrieved from
https://wirtschaftslexikon.gabler.de/definition/persona-119156
Lemon, K. N., & Verhoef, P. C. (2016). Understanding customer experience
throughout the customer journey. Journal of Marketing: AMA/MSI Spe-
cial Issue,80,69–97.
Loijens, L., & Krips, O. (2019). FaceReader methodology note. Noldus Infor-
mation Technology.
Märtin, C., Asta, P., & Bissinger, B. C. (2020). Optimizing the digital customer
journey—Improving user experience by persona-based and situation-
aware adaptations. In F. J. Martinez-Lopez & S. D'Alessandro (Eds.),
Advances in Digital Marketing and eCommerce, SPBE (pp. 141–148). Cham,
Switzerland: Springer Nature. https://doi.org/10.1007/978-3-030-
47595-6_18
Märtin, C., Kampfer, F., Herdin, C., & Yameni, L. B. (2019). Situation analyt-
ics and model-based user Interface development: A synergetic
approach for building runtime-adaptive business applications. Complex
Systems Informatics and Modeling Quarterly, CSIMQ,20,1–19. https://
doi.org/10.7250/csimq.2019-20.01
Moser, K. (Ed.). (2015). Wirtschaftspsychologie (2. Auflage). Springer.
McGinn, J., Kotamraju, N. (2008). Data-Driven Persona Development, Proc.
CHI 2008, ACM. New York, NY, U.S.A.: ACM.
MÄRTIN ET AL.11
Müsseler, J., & Rieger, M. (2017). Allgemeine Psychologie (3. Aufl.
2017). Berlin, Hei-delberg: Springer Berlin Heidelberg; Imprint.
Springer.
Picard, R. W. (2000). Affective computing (1st ed.). MIT Press.
Schilit, B. N., & Theimer, M. M. (1994). Disseminating active map informa-
tion to Mobile hosts. IEEE Network,8(5), 22–32.
Stein, A., & Ramaseshan, B. (2016). Towards the identification of customer
experience touch point elements. Journal of Retailing and Customer Ser-
vices,30,8–19.
Stürmer R., & Schmidt, J. (2014). Erfolgreiches Marketing durch
Emotionsforschung, Haufe Gruppe, Freiburg, Germany.
Tullis, T., & Albert, W. (2013). Measuring the user experience. Collecting,
analyzing, and presenting usability metrics (2nd ed., [third printing]).
Mor- gan Kaufmann.
UXQB: The international usability and user experience qualification board,
Retrieved from http://www.uxqb.org (2020)
Valenza G., Citi, L., Lanata, A., Scilingo, E. P., & Barbieri, R. (2014). Reveal-
ing Real-Time Emotional Responses: a Personalized Assessment based
on Heartbeat Dynamics, nature, Scientific Reports 4, 4998 https://doi.
org/10.1038/srep04998
Williams, J. R. (2004). Developing performance support for computer sys-
tems: A strategy for maximizing usability and learnability (p. 297). CRC
Press.
AUTHOR BIOGRAPHIES
Christian Märtin is a professor of computer architecture and intel-
ligent systems at Augsburg University of Applied Sciences. He
holds a diploma degree (M.Sc.) in computer science from the Frie-
drich Alexander University of Erlangen and a Dr.-Ing. (Ph.D.)
degree in computer science from the University of Rostock. His
main research interests are in human-computer interaction (HCI),
software engineering, and computer performance evaluation. He
is author of more than 130 research papers and of three text-
books. Before joining the university, he held several R&D posi-
tions in the computer manufacturing industry, where he finally
headed the HCI research group at the Olivetti Research Lab in
Nuremberg, Germany.
Bärbel Christine Bissinger studied computer science and business
information systems at Augsburg University of Applied Sciences,
Furtwangen University of Applied Sciences and the Eastern Insti-
tute of Technology, New Zealand. In 2018 she graduated in Augs-
burg with a Master of Science. Her master's thesis focused on the
measurement of emotions to optimize user experience. She
received an award from VDI for her work. She holds keen interest
in human-centered software development and is now working at
Capgemini, a global leader in consulting, digital transformation,
technology and engineering services.
Pietro Asta started his professional career in 2008 as an insurance
and finance trainee. After successfully completing his traineeship,
he worked for several years in the financial services sector before
he decided to study business information systems at Augsburg
University of Applied Sciences. He graduated with a bachelor's
degree in 2017, before completing his master's degree two years
later. In his master's thesis, Pietro dealt with the topics of user
experience and emotions in digital marketing as well as persona-
related advertising. He is now working as a business analyst for a
local consulting company.
How to cite this article: Märtin, C., Bissinger, B. C., & Asta, P.
(2021). Optimizing the digital customer journey—Improving
user experience by exploiting emotions, personas and
situations for individualized user interface adaptations. Journal
of Consumer Behaviour,1–12. https://doi.org/10.1002/
cb.1964
12 MÄRTIN ET AL.
