Technical ReportPDF Available

AFFINT process. Design of Affect-aware Applications. Introduction

Authors:

Abstract and Figures

Let me introduce you to an affective intervention design process (AFFINT) that combines analysis of the designed application with knowledge of affective phenomena and affective tools in order to formulate an affective intervention model for an application. The proposed AFFINT process facilitates the designing of an affective intervention model for every intelligent system.
No caption available
… 
Content may be subject to copyright.
Shall systems react to human emotions?
Affective states influence the way people work or act and although computers are considered as purely logical,
humancomputer interaction (HCI) is significantly influenced by emotions.
Modern intelligent systems are designed to support human activities. The systems often adapt control flow and
functionality to the user’s cognitive receptiveness. New methods and techniques of affective computing has
allowed intelligent systems to adapt also to users’ emotional states in order to support user effectiveness.
Is there a challenge in designing an affect-aware application?
Let us consider an example of an intelligent tutoring system (ITS) that supports the effective learning process.
Some emotions, e.g. boredom, frustration and more, are known to disturb learning and an ITS may recognize
and address these undesired emotional states in order to keep a user on the desired educational path.
In other words, affect-aware systems could make an intervention when certain objectives (e.g. task
effectiveness) are not met; however, the system should refrain from making interventions if it could disturb
the user in his/her execution of appropriate tasks.
How to design appropriate and effective affective interventions?
Let me introduce you to an affective intervention design process (AFFINT) that combines analysis of the
designed application with knowledge of affective phenomena and affective tools in order to formulate an
affective intervention model for an application. The proposed AFFINT process facilitates the designing of an
affective intervention model for every intelligent system.
AFFINT process
Design of Affect-aware Applications
Introduction
Definitions:
Affective application is an application, that aims at emotional interaction with the user eg. games,
conversational bots etc.
Affect-aware application is an application of any system goal, that additionally recognizes user affective
state and uses that information in the flow control.
Affective applications might be affect-aware as well as affect-aware applications might be affective or
not. Both types constitute a family of affect-related applications.
Definitions:
An affective intervention is a modification of a standard control path or system behavior that is a
response to user’s affective state and aims at providing effective execution of a task.
An affective intervention model defines when and how affective interventions should be performed.
AFFINT process enables design of valuable and natural affective interventions.
AFFINT process is used is game design and e-learning systems design.
AFFINT process is supported with scientific research.*
You are free to use AFFINT process in your commercial and non-commercial projects.
CC-BY licence applies. While using the process or this introduction please refer to:
*Landowska A, Szwoch M, Szwoch W, Methodology of Affective Intervention Design for Intelligent
Systems, Interacting with Computers, Oxford Journals, 2016, doi: 10.1093/iwc/iwv047
How to use AFFINT process?
The criteria for a good intervention model for an affect-aware application might be defined as follows:
a system should make an affective intervention only when required (i.e. it makes no interventions when to
do so would disturb the user);
a system should refrain from affective intervention when emotional state is uncertain, or should check
whether intervention is required;
interventions should be a natural element of an interaction with an application;
interventions should be tailored to the user and current interaction state;
interventions should be random in order to provide a realistic impression.
AFFINT design process intends to find a balance between what is technically available in emotion perception
and intervention, and what is effective for the application goals. The AFFINT approach proposes a 10-step
process that allows for the formulation of an affective intervention model compromising the three important
aspects: system characteristics (including goals, functionality and interface metaphor); knowledge of affective
phenomena that support effectiveness in the particular application domain and available affective computing
solutions that are applicable in the system context.
The structure of the AFFINT process is presented in Figure below (arrows indicate compulsory precedence,
and activity numbers the suggested sequence of execution). The AFFINT process contains 10 activities
(steps) and is divided into four layers: system definition, affective intervention solution set, affective
intervention model and evaluation of intervention model. Each layer contains activities which, performed
together, make it possible to design an affective intervention model. Detailed description of the activities is
provided in the consecutive pages in the suggested order of execution, but you might define your own
execution sequence as well.
AFFINT process
Design of Affect-aware Applications
How-to
Activity and results
Activity name: Application goals and tasks
Activity layer: System definition layer
Suggested order: 1
Preceding activities: none
What to do?
Design of an affective intervention requires identifying the main goals of the application first, as the
intervention purpose is to support the process that a human performs within the system.
Different applications can support human in different ways some are designed to entertain (games), some to
analyze (information systems), browse (web pages), learn (tutoring systems), store (databases), to name just a
few. Systems that adapt to human affect can be divided into affective and affect-aware applications. The
affective system aims at human-system interaction and evoking emotions on purpose, while affect-aware
software goal is to support certain process, eg. learning, controlling, driving, calculating and so on. Affective
systems aim at entertainment and user satisfaction rather than effectiveness of the supported process (as it is
for affect-aware applications). Affect-aware applications should foster cognitive and emotional states that are
efficient for the process they support (e.g. concentration, flow, etc.).
AFFINT process
Design of Affect-aware Applications
Activity 1. Application goals and tasks
Checklist:
What are the application goals?
What are the main application tasks/functionality?
Which human process is supported with an application?
In what way the application should supports users?
Expected outcomes:
Definition of application goals, list of tasks and expected features
Activity and results
Activity name: Effective emotional activations
Activity layer: Affective intervention solution set layer
Suggested order: 2
Preceding activities: 1. Application goals and tasks
What to do?
Effective emotional activations (EEA) comprise a definition of a subset of emotional states of a human that
are optimal for performing the process that is supported with an application. In other words, it defines which
emotional states are effective for the application goals, irrespective of it is entertainment application or
supporting tool. The EEA can be expressed in a number of ways:
open label set (advisable ones and/or discouraged ones),
a subset of discrete representation model or
sub-space in a multidimensional representation model of affect.
Definition of effective affective activations for the process can be also delivered in negation form (e.g. for
application goals it would be efficient to eliminate frustration and boredom).
The choice of Effective emotional activations could be based on:
psychological knowledge (based on literature review) about efficient and unproductive emotional
activations in the specific area of application,
interviews and surveys among domain experts (psychologists or domain professionals e.g. teachers for
affective tutoring systems)
experimental and observational studies performed as a part of the development process,
user-centred design methods,
common sense, e.g. driver monitoring system tries to prevent him not to fall asleep, so low-arousal
states should be alarming.
If there are no existing studies on emotions in the specific area, and no domain experts to retrieve knowledge
by interviews or surveys, user opinions might be considered, however perhaps the user assumptions should be
confirmed with some evidence in the development process.
It's important to emphasis, that the definition of effective and undesired emotional states should precede the
choice of the affect representation model for the system. The latter could be performed with AFFINT 5th
activity of choosing emotion recognition granularity and methods, as available input channels and available
emotion recognition methods can also impose constraints on the chosen emotion representation model.
AFFINT process
Design of Affect-aware Applications
Introduction
AFFINT process
Design of Affect-aware Applications
Activity 2. Effective emotional activations
Checklist:
Which emotional reactions foster effectiveness of the process supported by the application?
Which emotions suppress effectiveness of the process supported by the application?
Expected outcomes:
Effective Emotional Activations (either as a label set, subset of emotional states in discrete mode
or a subspace in dimensional model)
Activity and results
Activity name: Available input channels in application environment
Activity layer: System definition layer
Suggested order: 3
Preceding activities: None
What to do?
Recognition of an emotional state of an application user can be based on diverse characteristics, including
facial expressions, body posture, textual inputs, behavioural patterns, physiological parameters, voice
modulation and so on. However, in most cases only a small subset of them is available in the system context
of use and some of the available channels might be too noisy to reliably recognize user emotional state.
Therefore, the purpose of 3rd activity within AFFINT process is to describe, which of the input channels used
in emotion recognition are available and to estimate the level of the noise attached to them in daily use of
application. For example, in tutoring systems that are designed for distant e-learning we can usually rely only
on textual input and behavioural patterns of application usage, and rarely on video and sound channels.
Availability of the channels can be also user-dependent or time-dependent. Thus, it is important to distinguish,
which input channels will be available for all, and which for some of the users as well as to identify possible
timing limitations. Some input channels might be a subject to some kind of disturbance, or noise that could
affect emotion recognition process. For example for mobile devices sound channel can be significantly
disturbed by surrounding environment noise when used outdoor. Video channel, which is frequently used in
emotion recognition, can be significantly disturbed by insufficient or uneven light or by camera location.
Realistic definition of the availability of input channels is an essential step in defining, which affective
computing solutions are applicable in the application environment.
It is worth noting, that the input channels should be also permitted to use, not only available in the system
context.
AFFINT process
Design of Affect-aware Applications
Activity 3. Available input channels
Expected outcomes:
A list of available input channels followed by evaluation of user-dependence, time-dependence,
expected noise/disturbances and permission necessity (for each channel separately)
Checklist:
Which input channels are available in the system generic context (video, textual input, peripherals
usage, biosignals, sound, etc.)?
Are available channels user-dependent?
Are available channels time-dependent?
Are available channels susceptible to noise?
Do available channels require specific permission from the user?
Activity and results
Activity name: Available emotion recognition solutions and representation models
Activity layer: Affective intervention solution set layer
Suggested order: 4
Preceding activities: 3. Available input channels in application environment
What to do?
This activity is based on the step 3, as available emotion recognition solutions depend mostly on the available input
channels. Specialized equipment, like electroencephalography devices or physiological sensors are rarely available
outside laboratory environments, therefore their use is usually limited to research experiments.
If video input channel is available, one can consider recognition of user emotional state from facial expressions analysis,
which is one of the most popular and well explored solutions, but also from body posture analysis. Some of the newest
solutions recognize selected physiological characteristics, such as heart rate from video image, which is a new and
promising direction to consider. It is also important, that depending only on video channel in emotion recognition has
several drawbacks, such as:
the video channel can be of low quality due to insufficient light or wrong camera location,
facial expressions could be misleading (e.g. some people smile, when embarrassed) or falsified,
recognition from facial expression almost immediately imposes use of Ekman six basic emotions representation model,
as Facial Action Coding System (FACS) is the mostly known solution (Ekman, 1999), and the representation model is
not sufficient for some applications (e.g. it is hard to express boredom).
Using of sound channel is one of the promising approaches to emotion recognition, as often our voice modulation reflects
the emotional state more than the words we say.
Behavioral patterns of mouse and keyboard usage, as well as the user’s interaction with a system, can be used for
emotion recognition, however accuracy of emotion recognition based only on this information channel is limited and can
be improved by using multimodal approach, e.g. fusing recognition results from multiple channels. For textual inputs
sentiment analysis would also be an option.
AFFINT process
Design of Affect-aware Applications
Activity 4. Available emotion recognition solutions and
representation models
Expected outcomes:
A list of available effective computing solutions for emotion recognition and representation.
Checklist:
Which emotion recognition methods are possible considering the available input channels?
If the channel is temporarily not available (user constraints, time constraints, permission not
granted, noise) what is the alternative solution?
Is the chosen emotion recognition method reliable enough for the purpose of the application?
Which model of representation is the most frequently used for that emotion recognition method?
Are there other algorithms that provide results in a different representation model?
Activity and results
Activity name: Emotion recognition granularity and methods
Activity layer: Affective intervention model layer
Suggested order: 5
Preceding activities: 2. Effective emotional activations
4. Available emotion recognition solutions and representation models
What to do?
This activity decides on emotion recognition granularity and methods as a synthesis of the step 2 (that defines,
what is required) and step 4 (that outlines, what is technically available in the system context). As a result of
this activity a decisions should be undertaken on: which characteristics will be used in emotion recognition, as
well as, which emotion representation model will be used. The Effective emotional activations (EEA)
description, if provided in words only, should be expressed in the chosen emotion recognition representation
model. For example, if an application is reacting to stress only and the chosen input channel is facial
expression, in this step there is a need to map the outputs of the emotion recognition process (including six
basic emotions: joy, anger, fear, surprise, disgust, sadness) into the notion of stress. It is worth noting, that
some mappings require lots of knowledge on affective phenomena and might be inaccurate.
There are some available solutions (libraries) for emotion recognition from text, facial expressions or sound,
but there are no of-the-shelf solutions for other input channels or for multimodal emotion recognition to apply.
In those cases, data gathering, labeling and learning must be performed to obtain a specialized classifier that
would suite the application most. One may also choose a self-adapting approach and implement a learning
component into the target application. That approach is quite common, especially when application is
designed for a personal use and personalized classifier could be more accurate than general one.
One may also choose to predict an emotional state, for example with OCC model (Steunebrink, et al. 2009)
instead of recognizing it or choose a combined approach (prediction confirmed by recognition process).
AFFINT process
Design of Affect-aware Applications
Activity 5. Emotion recognition granularity and methods
Expected outcomes:
Decision on: emotion recognition methods, emotion representation model and EEA definitions
within that model.
Activity and results
Activity name: Available output channels in application interface metaphor
Activity layer: System definition layer
Suggested order: 6
Preceding activities: None
What to do?
Within this activity another consideration on the designed application should be made, i.e. output channels
that are available in the system environment, e.g. voice messages, music and audio effects, colours, labels,
animations, embodied characters etc. Usually there are many available output channels and this activity
requires some more reflection on the interface metaphor used in the application.
Interface metaphor is a concept of interaction around which an interface of an application is designed. A role
of the metaphor in user interface is to facilitate orientation, understanding and learning in the application
improving its intuitiveness and usability. Discrepancy between the metaphor used and what occurs in an
application is called mismatch and may decrease user satisfaction with the system. Affective interventions
should follow the interface metaphor in the application and should use similar output channels. There are
some cases when a mismatch could be done on purpose, however they must be carefully considered and
designed.
AFFINT process
Design of Affect-aware Applications
Activity 6. Available output channels in application interface metaphor
Expected outcomes:
Description of system responses, that are: available in the system context and natural within the
chosen interface metaphor.
Checklist:
Which output channels are available in the system generic context (textual, visual, sound)?
Which systems reactions are natural in the chosen interaction metaphor?
Are available channels user-dependent? Might be temporarily switched off? Can a user be asked to
turn them on (eg. speakers)?
Activity and results
Activity name: Possible affective interventions of an application
Activity layer: Affective intervention solution set layer
Suggested order: 7
Preceding activities: 6. Available output channels in application interface metaphor
What to do?
The goal of this activity is to list all possible system reactions to any change of user’s affective state.
Reactions (interventions) might be as subtle as a change of music tones or color of an icon, but also as
spectacular as a full screen animation, a reaction of an embodied character or a message from the off-voice.
As designer's imagination can be boundless, the considerations on the interface metaphor should limit it.
Within this activity one can choose to define a range of possible system reactions in a form of scenarios, e.g.
sequences of changes or actions undertaken in all states, including co-occurrence of events.
AFFINT process
Design of Affect-aware Applications
Activity 7. Possible affective interventions of an application
Expected outcomes:
List of possible system reactions in form of scenarios.
Checklist:
What are the possible system’s responses to user actions or states (provide a wide range of system
reactions)?
What scenarios can define each of the reactions?
Can you define any gradation of specific system response?
Activity and results
Activity name: Affective intervention triggering rules
Activity layer: Affective intervention model layer
Suggested order: 8
Preceding activities: 2. Effective emotional activations
7. Possible affective interventions of an application
What to do?
The next activity aims at pairing emotional states that a designed system should react to with affective
interventions that the system would perform. This step requires precedence of activities 2 (Effective emotional
activations), 5 (emotion recognition granularity) and 7 (possible affective intervention set). When defining
affective interventions triggering rules, one should consider:
gradation of an emotional state (perhaps different reaction should be performed for high and low
arousal of the emotion),
uncertainty of the recognition process (in case of uncertainty of different affective intervention
scenario might be proposed),
randomization of some affective interventions for more realistic impression.
AFFINT process
Design of Affect-aware Applications
Activity 8. Affective intervention triggering rules
Expected outcomes:
List of affective intervention triggering rules in the form: emotional state possible scenarios.
Checklist:
Which emotional states could be paired with which scenarios?
Is emotion gradation changing the performed scenario?
Does uncertainty of the detected emotional state change the scenario?
Can you list more than one scenario for each emotional state you want the system to react to?
Activity and results
Activity name: Affective intervention constraint rules
Activity layer: Affective intervention model layer
Suggested order: 9
Preceding activities: 6. Available output channels in application interface metaphor
8. Affective intervention triggering rules
What to do?
The activity in AFFINT process was added to consider limitations of triggering affective interventions for the
purpose of providing natural impression of interaction. Let us imagine a system, that asks its user "Are you
angry?" every 15 seconds or an animation that is triggered every time the user stops working for a moment.
The system should not perform interventions that are annoying or seem unnatural considering interface
metaphor used in system design. For every affective intervention scenario three constraints should be
considered: compatibility with the interface metaphor, conformance with a current state of a user-system
interaction and possible repeatability interference with a natural user experience.
AFFINT process
Design of Affect-aware Applications
Activity 9. Affective intervention constraint rules
Expected outcomes:
List of timing and repetitiveness conditions associated with each of the scenarios, that fit into
interface metaphor.
Checklist:
Is the scenario of affective intervention compatible with interface metaphor (would it be
considered natural for the user)?
Would the scenario be in conformance with a state of a user-system interaction?
How often the scenario could be repeated, still providing natural perception?
Activity and results
Activity name: Validation with end users
Activity layer: Evaluation of intervention model
Suggested order: 10
Preceding activities: 5. Emotion recognition granularity and methods
8. Affective intervention triggering rules
9. Affective intervention constraint rules
What to do?
The last activity in AFFINT process was added to evaluate the designed intervention model and probably loop
back to some step in the AFFINT process. The suggested method for evaluation is validation with end users,
as a ultimate reference for correctness of the chosen approach. The most reliable approach is to evaluate
affective interventions against a sham condition.
.
AFFINT process
Design of Affect-aware Applications
Activity 10. Validation with end users
Expected outcomes:
Two lists of emotional interventions: (1) acceptable by the users and supporting the desired
emotions and (2) the ones to reconsider.
Checklist:
Which system interventions surprised user? Was it a pleasant or unpleasant surprise?
Which emotional states were detected and which did not occurred?
Did affective interventions supported the effective execution of the tasks?
To sum up...
Affective computing is one of the research areas that is currently intensively explored. It provides many tools
that focus on, in particular, emotion recognition from various characteristics.
However, affective interventions of an intelligent application should take into consideration the system
characteristics, knowledge of affective phenomena that support effectiveness in the particular application
domain and available affective computing solutions that are applicable in the system environment.
The proposed affective intervention (AFFINT) approach provides a 10-step process that enables the design of
affective intervention models for intelligent systems. The process supports designers in defining affective
intervention models that are adequate and natural in the system context.
Main reference
All of the textual descriptions in this document are derived from the following publication. While using the
AFFINT process or this introduction please refer to:
Landowska A, Szwoch M, Szwoch W (2016) Methodology of Affective Intervention Design for Intelligent
Systems, Interacting with Computers, Oxford Journals, doi: 10.1093/iwc/iwv047
Cited references
(Ekman, 1999) Ekman, P. and Davidson, R. (1999) The Nature of Emotion: Fundamental Questions. Oxford
University Press.
(Steunebrink, et al. 2009) Steunebrink, B.R., Dastani, M.M. and Meyer, J.-J.Ch. (2009) The OCC Model
Revisited. In Reichardt, D. (ed.), Proc. 4th Workshop on Emotion and ComputingCurrent Research
and Future Impact. Paderborn, Germany.
Acknowledgements
This study was partially supported by Polish-Norwegian Research Programme operated by the National
Centre for Research and Development under the Norwegian Financial Mechanism 20092016 in the frame of
Project Contract no. Pol-Nor/210629/51/2013, and under frame of the Project Contract no. Pol-
Nor/209260/108/2015 as well as by DS Programs of the ETI Faculty, Gdañsk University of Technology.
AFFINT process
Design of Affect-aware Applications
Summary and Acknowledgments
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.