Epistemic Affordances and Active Inference in Socio-Cognitive Design

Full text

Art Style | Art & Culture International Magazine
______ ______
53
Epistemic Affordances and Active Inference
in Socio-Cognitive Design
Klaus Schwarzfischer
Abstract
Methodological problems arise when a special case is confused with the general
principle. You will find affordances only for ‘artifacts’ if analysis is restricted to
‘artifacts.’ The general principle, however, is an ‘invitation character,’ which
triggers an action. Consequently, an action-theoretical approach is
recommended. Accordingly, humans are not passive-receptive beings but
actively produce action effects that open up the world to us (through ‘active
inferences’). This ‘ideomotor approach’ focuses on the so-called ‘epistemic
actions,’ which guide our perception as conscious and unconscious cognitions.
Accordingly, the seemingly passive perception is dissolved into a multitude of
epistemic actions (e.g. eye movements, tactile operations, etc.). The action
theoretical approach of ›enactive cognition’ takes into account that every form is
consistently processualized. Thus, each ‘Gestalt’ is understood as the process
result of interlocking cognitions of ‘forward modelling’ (which produces
anticipations and enables prognoses) and ‘inverse modelling’ (which makes
hypotheses about genesis and causality). These cognitions are fed by previous
experiences of real interactions, which later change into mental trial treatments,
which are highly automated and often unconsciously. Every object can have such
affordances that call for instrumental or epistemic action. In the simplest case, it
is the body and the facial expressions of our counterpart that can be understood
as a question and provoke an answer/reaction. In the same way, our own body
and facial expressions act as affordances to our counterpart. Thus, emotion is not
only to be understood as expression (output) according to the scheme ‘input-
processing-output,’ but acts itself as a provocative act (input). The reaction to this
clarifies what kind of situation we are in. Any unclear situation thus shows
affordances to epistemic actions. Consequently, artifacts are neither necessary
nor sufficient conditions for affordances. Rather, they exist in all areas of
cognition—from enactive cognition to embodied cognition and social cognition.
Klaus Schwarzfischer, “Epistemic Affordances and Active Inference in Socio-Cognitive Design,”
Art Style, Art & Culture International Magazine, no. 7 (March 11, 2021): 53-72. DOI: http://doi.org/10.5281/zenodo.4589345

Art Style | Art & Culture International Magazine
______ ______
54
The Efficiency of a Systematic Approach
This paper argues from a design theorist view why artifacts are neither specific
for design nor for affordances. Some readers might be perplexed by that, but
their irritation will vanish—as this text progresses from prejudgments to solid
knowledge. A short preliminary round should help to show to which extent this
optimism is reasonable.
We often gain a methodical surplus, if we look for the simplest case of a
phenomenon. Ideally, we generate a precise definition of an object, of which we
have just had unrelated examples before. Mostly, the object needs to be
resolved in its generative processes by an operationalizing—as Siegfried J.
Schmidt proposes with regard to scientific theory (Schmidt, 2010). The
microgenesis of those objects (in German Aktualgenese) consists of specific
processes which are resulting in that object. In the context of design and
aesthetics this may feel unusual, because of the fact that mostly artifacts are
thematized there. In his dialogue ‘Hippias Major,’ however, Plato already made
the point that it does not suffice to list beautiful things in order to understand the
essence of beauty. In fact, we have to explicate the definite process which is the
base of each and every aesthetic experience (Schwarzfischer, 2019). A similar
problem can be found in design theory, also, where designed artifacts were
focussed for a long time without a strict definition of the term ‘design.’ I showed
that operationalizing is fruitful in this case also (Schwarzfischer, 2010 a and 2010
b
1
). The focus will change from artifacts (as possible results of processes) to
interventions themselves (as processes which are definable by necessary and
sufficient conditions).
What exactly will be the methodic benefit of that operationalizing? First of all,
this is the crucial step from description to explanation of a phenomenon—and
therefore, from story telling to science. The explanation may result in a process
flow diagram, but formalization is optional. Even when no such model like a block
diagram is generated, the understanding of the phenomenon now is
fundamentally different. This progress means the step from a special case to the
general principle. Before that, we just had single examples of something which
we met in random situations. Now, we are able to adress the possibility space of
the phenomenon. Especially on ‘design’ an enormous difference shows up: We
will no longer analyze exclusively artifacts (which are just a special case). The
process of formation itself will be focussed (as the general principle). Then, any
volitional action is a design problem because to define ‘design’ as an

Art Style | Art & Culture International Magazine
______ ______
55
intervention seems to be the only option (according to Schwarzfischer, 2010 a
and 2010 b). That action-theoretical approach is astonishingly productive as this
paper develops. Similarly, this holds good for ‘aesthetics,’ although there is a
traditional fixation towards ‘artworks’ which prevented those theories from
unfolding actual explanatory force and/or noteworthy use of application.
Essential for this would be the capability for prognosis and the concomitant
falsifiability (which necessarily exeeds a description of the given). Both require a
theoretical access to cases that did not occur yet, in terms of adressing the
possibility space (from which prognoses can be derived). For aesthetics this was
accomplished systematically by modelling the aesthetic experience itself as a
microcognitive process instead of ‘measuring’ static artifacts (Schwarzfischer,
2019). As a result, it becomes apparent that the basic process of any aesthetic
experience can be identified as an evolutionary learning reinforcement. Thus, any
acting can produce an aesthetic experience (e.g. playing, dancing or thinking as
well as alone as with others). Amazingly, in this field it was again an action-
theoretical approach which based the operationalization—although the verdict
of ‘disinterested appreciation’ dominated (and paralyzed) aestheticians for
generations
2
. To escape that, we chose a finer-grained analysis. Thus, each
perception dissolves in several perceptual acts as a feature of any active observer
(like men and animals on the contrary to trees). Concrete perceptual acts are eye
movements or tactile scanning, for example. We can interpret those perceptual
acts as ‘epistemic actions’ (which do not aim to change the world but just derive
information in order to optimize the cognitive model of the world)
3
.
This paper aims to outline the general principle of affordances instead of
attributing a ‘stimulative nature’ (German Aufforderungscharakter) to artifacts
only—as widely spread in literature. A more systematical and historical view
refers to active observers, which is why an action-theoretical approach may be
productive again. As a result, the possibility space of affordances is much wider
than one might think when reading the conservative analyzes of artifacts
4
. On
closer consideration, the spectrum of affordances ranges from intra-psychic
microcognitions to processes of social cognition. This is less surprising when
reflecting that affordances is orignially a concept of cognitive sciences
5
.

Art Style | Art & Culture International Magazine
______ ______
56
Roots of the Affordance Concept in Cognitive Biology
The term ›affordances‹ was coined by the perceptual psychologist James J.
Gibson as widely known
6
. The idea of affordances, though, is a bit older than
Gibson’s references to the Gestalt psychologists Kurt Koffka and Kurt Lewin
suggest. Especially the concept Aufforderungscharakter (as an ‘invitation
character’ or a ‘call to action’) by Lewin is generally identical with ‘affordances’
and may therefore be rated as a parent
7
. Andrea Borsato (in press) goes into the
differences:
8
[…] both authors [Lewin und Gibson] emphasize that such
a connection does not exist per se, but just for a specific
subject in a specific situtation, because the postbox does
not point per se to post a letter by its ›invitation
character‹ or by its ‘affordance’ but just for a subject who
wrote a letter and wants to get it delivered. Already J. J.
Gibson clarifies that this parallelism isn’t more far-
reaching. After done action the object loses its ›invitation
character‹ but not its ‘affordance:‘ For Lewin, the postbox
does not call any longer for action when the letter is
posted, whereas for Gibson the postbox keeps its
‘affordance’ also when the posting is done.
This quotation explains why in design the affordance is closely coupled with
artifacts. In design theory affordance is seen as an object feature (acc. to Gibson)
and not as a subject disposition (acc. to Lewin). The postbox example highlights
the difference because the ‘invitation character’ as a disposition has a dynamical
history itself— the ‘invitation character’ arises by writing a letter and leaves by
posting the letter. Whereas Gibson claims that an affordance exists in a stable
manner over time. Hence, it is a feature of the postbox. This assumption is
problematic as postboxes (like every object) initially obtain their functional role
by the users socialization. And, indeed, Gibson indicates, that affordances
presuppose specific processes of learning.
9
Isn’t this inconsistent, when
affordances require learning processes of the subject and are features of the
object? This is not the place to clarify these contradictions in detail or to
reconstruct the full history of the concept ‘affordance’ (particularly because other
contributions in this volume address aspects of it). For our purpose it seems more
productive to introduce a less known author who was a contemporary of Koffka
and Lewin.

Art Style | Art & Culture International Magazine
______ ______
57
Jakob von Uexküll published his much-noticed book ‘Theoretical Biology’
10
in
1920 and already in 1909 he introduced the term ‘Umwelt’
11
. This phenomenal
lifeworld of animals is species-specific and therefore different for each species, as
each species reacts distinctively to different objects in its environment.
12
Refering
to Immanuel Kant,
13
he argues that in the Umwelt of an animal exclusively those
objects exist which can be perceived with the particular sensorium of that species
and that for this species only these objects can have meaning at all. Although
Uexküll rejected Darwin’ s theory of evolution, his phenomenological inspired
approach to the inner world (German Innenwelt) of animals gained wide
influence to growing disciplines like animal psychology and ethology. For
example, Konrad Lorenz underlines the relevance of learning processes for
behaviour—but now, the animal acquires the ‘invitation character’ by exploratory
action: “Jakob von Uexküll once said that all things in the ‘world’ (Umwelt) of
animals are ‘action things‹ (Aktionsdinge) (1909). This is particularly true of the
objects with which an animal has made itself familiar through exploration, and
has then ‘laid ad acta’ for later reference.”
14
In this sentence, not just the
behavioral scientist is speaking because Konrad Lorenz is a founder of
evolutionary epistemology as well.
15
Also in the context of affordances it is of special interest which knowledge is
innate and which is only acquired through the experiences of the individual. That
learning is mainly favored by two factors: First, the disposition of a biological
observer system that every learning is subjectively experienced positively.
Second, the disposition that this biological observer system is to be understood
as a ‘curiosity being,’ as Lorenz calls it. From this wording, it becomes clear that
these curiosity beings—be it humans or other animals—actively deal with their
environments. By no means do they passively wait for an external impulse before
responding to it (as the behavioristic stimulus-response scheme would provide).
It is precisely this phenomenological perspective which places value on
subjective experience that Konrad Lorenz already found in Uexküll. The latter
refers to Kant, who explains the principle of the unrecognizability of the ‘thing-in-
itself.’ Not only is the sensory equipment of a living being the reason why each
species lives in its own Umwelt, which is fundamentally different from that of
another species. In addition, there are the very different possibilities to act on
this environment. The perceptual actions mentioned above also belong to these
possibilities of action. Each of these perceptual actions can be understood as an

Art Style | Art & Culture International Magazine
______ ______
58
epistemic action, which resembles a question to the environment: If you do not
ask, you will not receive an answer and thus far less information about the
environment. The effects of action are an indispensable source of information
that has been and is systematically underestimated by positivists. Many, if not
most, aspects of things cannot be seen as long as the observer remains passive
(think, for example, of the haptic qualities, the weight or the hardness of objects).
Therefore, interaction with the environment is an essential method without which
it is impossible to construct a comprehensive cognitive reality model. Gibson
explicitely formulates the principle:
16
“We must perceive in order to move, but we
must move in order to perceive.” Uexküll recognized this and formalized it in his
‘functional circle,’ thus laying an important foundation for biocybernetics.
Figure 1: The structural elements in the ‘function-circle’ due to Jakob von Uexküll, 1926, 157
(Source: own depiction from Schwarzfischer, 2019, 306)

Art Style | Art & Culture International Magazine
______ ______
59
The function-circle expresses the theoretical design of Uexküll, on which
the biocybernetics of the 20th century is based. Essential is the
differentiation into four distinct areas:
[1.] The ‘Umwelt’ (as the subjective environment) is everything
that does not belong to the observer system itself (but e.g.
the newborn human does not yet know what belongs to him
or her and what does not).
[2.] At the center of the ‘world of action’ (as the effector world) is
what is known as efferent nerves (these are the descending
nerve pathways that lead from the brain to the muscles and
that are necessary for action, i.e. for interacting with the
environment). Because it is part of the activity, the physical
self-perception of the actor (the ‘interoception’) belongs to
the world of action. The world of effects pursues implicit or
explicit goals, the realization of which is to be understood as
‘top-down processes.’
[3.] The bottom-up processes in the ›world as sensed‹ (as the
receptor world) work up the sensory input—based on the
afferent nerve pathways (which run from the sensory
periphery to the brain and which are the medium for
noticing the environment). This is the perception of the
outside world (the ‘exteroception’).
[4.] In evolutionary and developmental terms, the ‘inner world’ is
the most recent area because it encompasses the higher
cognitive processes of mental probing and conscious
reflection. However, the elementary basic structures are
significantly older. Basal is, among other things, the principle
of ‘reafference,’ with which it is possible to distinguish, for
example, whether the world has moved or whether I have
only performed an eye movement. (Thereby the efferent
muscle commands are set off against the afferent sensor
signals, which the word ‘reafference’ expresses). Even
complex memory structures are further developments of
what is called the ‘new circle’ in figure 1. All effectors are
grouped there as (E) and the (S) denotes the sensors.
Schwarzfischer further developed the model (2019, 307–308).
Thus, an ‘inner world’ can also be attributed to a counterpart
in the environment, if this counterpart behaves non-trivially
(so that this counterpart can be attributed with the control of
behavior by intentions).
17

Art Style | Art & Culture International Magazine
______ ______
60
Two Fundamental Perspectives on Affordances:
The Sensomotoric and the Ideomotoric Approach
In a way, Uexküll anticipates the ‘pragmatic turn’ in the cognitive sciences, as
propagated by Andreas Engel, Karl Friston and Danica Kragic, 2015. This
‘pragmatic turn’ describes the paradigm shift from the ‘sensorimotor approach’
to the ‘ideomotor approach.’
[s.] The sensorimotor approach is based on traditional
cognitivistic thinking, which assumes a (quasi-)linear input-
processing-output scheme. Accordingly, the sensory
organs are the input that is processed centrally in the brain,
resulting in an output (which either consists of a motor
action or in a cognitive perceptual judgement, e.g. in an
aesthetic or ethical evaluation). Primary is always the
perception, which then leads to a secondary action—hence
the name ‘sensorimotor approach.’
18
[i.] The ideomotor approach assumes an active observer,
whose actions generate the observable phenomena as
effects of action. For example, the infant recognizes only
through its own movements (as ‘ideomotor’ means self-
moving) what belongs to its body and what belongs to the
environment. Similarly, observers only recognize the
structured depth of the spatial arrangement when they act,
as Gibson (1979, 123–125), shows with the help of flying
birds. In general, according to the ideomotor approach,
activity is primary and the perception of resulting action
effects is secondary. Even the still undirected movements
of an infant induce an input without which the development
of higher cognitive structures would not be possible. This is
called embodied cognition (because without physical
action, cognition would not be possible) or enactive
cognition (this is generating cognition because the activity
itself generates action knowledge, relying on processes of
self-organization rather than explicit knowledge or
reflection).

Art Style | Art & Culture International Magazine
______ ______
61
This distinction is relevant for a comprehensive understanding of affordances,
since two very different types of calls to action are evident: First, there are
affordances to react physically to a given situation. This corresponds to the
affordance concept in design, where real actions are triggered to change the
situation (e.g. when a door handle has the affordance to open the door and enter
the building).
This corresponds to an affordance to ›instrumental actions‹ as described above.
Second, affordances to ‘epistemic actions’ (which do not aim to change the
world) can be distinguished from those. Rather, they help to improve knowledge
of the world. Information is not passively registered but actively provoked. Real
interventions can be addressed as well as mental trial treatments or
communicative inquiries in the social space.
19
What exactly provokes an affordance that calls for action? It calls for interaction.
But a concrete action (be it physical action, mental test treatment, or
communicative simulation) presupposes a goal, which can be explicit or implicit.
In the case of ›instrumental action‹ this goal represents a desired state of the
world. In the case of the ›epistemic action‹, a question to the world is necessary,
which is to be answered. In relation to the cognitive observer system three things
can happen:
[1.] A stimulus in the perceptual field suggests bottom-up a goal that
is being considered (e.g. a deviation from the original intention, as
children with Attention Deficit Hyperactivity Disorder experience
everywhere). Here something is suggested as a goal.
[2.] A perceived or imagined object may prove to be suitable as a
means to achieve the goal that dominates the situation top-down
(e.g. in an escalating marital dispute a cup is suddenly seen as a
‘projectile’). Now the suitability as a means is determined.
[3.] In the case of ‘epistemic affordance,’ the adressing of a role in a
means-to-purpose relation is not that easy. The form of a
statement (as a propositional sentence), which corresponds to the
two varieties of the ‘instrumental affordance’ in [1.] and [2.], does
not do justice to it. Rather, something approaches the observer in
the form of a question, which can be answered in the next step by
an ‘epistemic action.’ Accordingly such an action cannot be
falsified as it would be possible in a statement. As a question it can
only be accepted or rejected.

Art Style | Art & Culture International Magazine
______ ______
62
Relevant in evolutionary and developmental psychological terms is that such
questions can be implicit or explicit. This is significant because in early stages
only an implicit question is possible as a motivation for exploration. A biologically
founded curiosity thus becomes the basis for any further development. Gerald
Hüther shows why ideomotor activity has priority, even though the movements at
first seem more random than arbitrary:
From about the 7th week of pregnancy onwards, it can be
observed how the embryo swimming in the amniotic sac
performs its first, still very uncoordinated movements. Initially,
these are rather twitches, which are triggered by the
contraction of certain muscles of the trunk and extremities. [...]
From the very beginning, learning takes place in the brain by
using and exercising the corresponding bodily functions. In the
course of this long and complicated learning process, the
embryo is enabled to move its trunk, legs and arms in an
increasingly coordinated manner [...]. What is true for the
central nervous control of the body muscles is equally true—
although less clearly visible or measurable—for the formation of
all those neural circuit patterns that are involved in the control
and coordination of all other body functions.
20
Even the initially random movement of the embryo induces action effects that
gradually make it possible to derive a cognitive body schema (which is still
unconscious or implicit or embodied). Without this self-representation, it would
be impossible to consider oneself later as an acting subject during mental trial
treatments. First, one’s own body must be appropriated to the extent that
reliable prognoses of movements (as anticipations of action effects) are possible.
Because they are embodied, these actions are always perceptible to others. Every
action (whether it is sensorimotor-reactive or ideomotor-proactive) is thus public.
Consequently, the production of affordances and the reactions to them are always
both: they are of epistemological-embodied-cognitive interest (because ideomotor)
and of action-theoretical-cultural relevance (because public).

Art Style | Art & Culture International Magazine
______ ______
63
Epistemic Affordances in Every Gestalt Perception
The action-theoretical paradigm represented here can also be applied to those
areas that traditionally tend towards a passive observer. How this increases the
explanatory value is to be shown with the simplest possible example.
Furthermore, all complex scenarios are composed of its basic processes.
Figure 2: Simple example of a shape construction by the observer
(Source: own depiction from Schwarzfischer 2019, 83)
Why should we speak of a ‘Gestalt-Construction’ when we look at figure 2? The
adult observer is usually not aware of any processes of ›construction‹. Rather, the
›content‹ of the image is immediately evident. But this impression is deceptive,
because the supposedly given is an interpreted (and thus a product, something
made) that differs considerably from what is actually seen (as sensor data). These
deviations can be recorded quite precisely as processes of construction:

Art Style | Art & Culture International Magazine
______ ______
64
[1.] The ‘circle’ is no circle, because only an arrangement of single
points can be seen in reality. Already the recognition of a
regularity (a rotational symmetry) is put in by the observer as a
cognition (based on unconscious conclusions).
[2.] The ‘square’ seems to lie on the circle and therefore hides a
part of it. This also cannot be seen directly but is concluded.
For the pixels of figure 2 are lying on one and the same plane
(the paper), so there is no graduation of depth within the figure.
This is added by the cognitive observer system by using
probabilities that have developed and proven themselves in the
empirical lifeworld (i.e. outside of this illustration).
[3.] This interpretation—that the ‘square’ lies on the ‘circle’—
creates another fact (done), which is not a date (given). The
assumption that the ‘circle’ under the ‘square’ has the same
regularity as the visible ‘parts’ is obviously a hypothesis and not
a direct perception.
All three constructions, which were given here as examples, can be interpreted as
unconscious cognitive processes on a very small scale. Such microcognitions
underpin our entire life. All objects (not only the visual ones) are therefore to be
understood as ‘process results’ of such microcognitions.
21
In adults, these
processes run very fast, highly automated and unconsciously. They usually only
become conscious when the results are unexpected, for example in the case of
optical illusions, etc. The situation is different in infants and small children, where
the learning processes are so slow that the external observer can, so to speak,
‘watch them think in real time.’ Here, ideomotor activity is slowly transformed
into expectations of action effects, as described by Arvid Herwig, 2014. Enactive
cognition plays the central role here because the action is primary and the ability
to anticipate the action effects only gradually develops from it. This is exactly
what happens unconsciously when we look at Figure 2. The ›spatial structure‹ of
the elements is not seen but unconsciously calculated. These cognitive processes
can be explained, as the contributions in Engel, Friston & Kragic, 2015, show.
Furthermore, it can be shown that the enactive prior experiences are condensed
to an aniticipative probability (which is called ‘prior’ in Bayesian reasoning). Each
individual action changes this assessment depending on whether the expectation
is confirmed or disappointed (which leads to the Bayesian ‘posterior’—which is
immediately used again as ›prior‹ for the next action). Without these interactions
with the environment, learning that recognizes even the most elementary
connections is not possible. In the famous study by Richard Held and Alan Hein

Art Style | Art & Culture International Magazine
______ ______
65
from 1963, young cats were raised in complete darkness. They were only able to
see under controlled conditions (when light was on), forming two groups: One
cat moved actively and the other cat was only passively moved by a device. The
visual impressions in these bright learning phases were identical for both animals
because they moved in the same way through the same scene. But those cats
that were only passively moved did not learn to see. They moved in the test
phase as if they were blind. In relation to our example in Figure 2, this means that
even the microcognitive processes have to be learned under specific conditions.
We can speak of active inference (i.e. active inference based on action effects),
even though there is no reflected consciousness in the narrower sense yet.
The reality construction of every human being is full of interpretation based more
or less on real sensor data. The process model of Schwarzfischer from 2019,
refines the ›function-circle‹ by Jakob von Uexküll and extends it into a multi-level
model. The process results of the active inferences have a very different range in
spatial, temporal and factual terms. Therefore, at least three such levels have to
be differentiated.
22
The ‘conclusions’ of the first-level active inferences are again
used as ›premises‹ for the second-level active inferences, etc. In our example
from figure 2, this means that here at the lowest level a perceptual hypothesis is
formed, which would be tested in early childhood by a real action. Later on, a
mental trial treatment is sufficient, which, if routine is sufficient, is automated and
unconscious (so that we think we can ‘see’ it immediately without any cognitive
effort, but it is actually an immediate knowledge or remembrance).
In this context, affordances to epistemic actions can be defined as all
uncertainties that cannot be deductively derived from the cognitive reality
model. Wherever only probabilities are involved (i.e. when the hypothesis
character indicates that it could be different), a cognitive process is set up to
check or complete the uncertain information (as in the example in Figure 2). Two
types of such cognitive processes are important here: Forward modelling takes
the current perception and forms hypotheses about possible future
developments (e.g. the expected action effects of epistemic or instrumental
actions). Inverse modelling forms hypotheses about possible causes (e.g. the
history of the current situation). Put simply, two types of affordances to epistemic
actions can be distinguished: Either the question ‘What for?’ Or the question
‘Where from?’ is provoked. Figure 1 can illustrate this because it combines
temporal and structural aspects. Thus the arrows in the ‘function-circle’ represent
a direction and thus a time axis, because it is a succession of events. At the same
time, the areas represent structural units (e.g. the efferent nerves of the ‘action
world,’ the afferent nerves of the ‘perception world’ and the brain-based working

Art Style | Art & Culture International Magazine
______ ______
66
memory of the ‘inner world’). All events in the ›action world‹ are spun forward to
their possible/probable effects in the ‘perception world.’
23
Every phenomenon in
the ‘perception world’ can be investigated by inverse modelling for its causes in
the ‘action world.’
Affordances in design often refer to the direct request for an instrumental action,
which can be understood as forward modelling (because the means has an
affordance towards the end). However, the design of affordances to epistemic
actions is also a design problem (but is traditionally more associated with
criminology, pedagogy or science). Since these are less ‘world problems’ than
’knowledge problems,’ forward modelling and inverse modelling are equally
common. Forward modelling is intended to make it easier for the ›process
customer‹ to operate in the desired direction. Inverse modeling is intended to
strengthen the understanding of causes and interrelationships—and in the next
step to support desirable actions. ‘Desirable’ does not mean that a extrinsical
target is internalized without reflecting this. On the contrary, especially if the
autonomy of the subject is to be promoted, its ability to act and reflect must be
increased. This is done by improving cognitive processes, which we can formalize
as forward modelling and inverse modelling. Autonomy is improved if the subject
is enabled to understand even complex process chains across several links and to
assess their probability. This makes it more difficult to manipulate this subject by
the proverbial ‘dangling a carrot in front of somebody’ or by too clumsy
falsifications of history—because the stages of reflection range from enactive
anticipation of immediate effects of action to theoretical reflections of possible
worlds. There, logical investigations are carried out as epistemic actions. Even if
in these possible worlds no instrumental action can be taken, instrumental
actions are conceivable in our life-world, which are supposed to turn a possible
world into reality. Every design fulfills this definition. Consequently, non-existent
objects can have an affordance to epistemic and instrumental actions.

Art Style | Art & Culture International Magazine
______ ______
67
Emotional Expression as Affordance and Epistemic Action
At first glance, all actions in social space are public and thus observable by
others. This is true for instrumental actions that are actually performed. For
epistemic actions, though, this applies to a limited extent (think, for example, of
mental trial treatment) as well as for instrumental acts of omission (such as the
selfish concealment of information). A complete catalog of affordances that
encourage such actions is not feasible here due to a lack of space. However, in
introductions one aspect is regularly neglected when the artifacts are in the focus
of media sociology.
24
Especially the subpersonal processes of enactive cognition
deserve our attention because they typically occur automatically and
unconsciously. They usually only enter consciousness when the course of action is
disturbed.
25
An example can illustrate this: We go for a walk—either in the forest
or in a lively old town. What is the difference between dodging trees and
walkers? In a busy shopping mall, our bodies are in dialogue; in the forest, only I
react to the static trees. The mutual affordances when avoiding passersby cannot
be traced back to a linear scheme of ‘input-processing-output.’ Here we are
dealing with circular processes because both are obstacles for each other and
react to each other. Subliminally, we perceive finest approaches of evasive
movements and tend to the other side.
Most of the time, this is done without conscious reflection. From time to time,
however, it happens that both of us try to sidestep the same side, which can lead
to exhilarating choreographies. This happens when the affordances of one actor
are not felt by the other to be clear or unambiguous enough. In a busy
pedestrian zone, passersby can by no means retreat to fixed rules, as is usually
the case in automobile traffic. Each regulation must be ›negotiated‹ as an
individual case (in the double sense of the word). This is not a rare special case
that only occurs when strolling. Rather, the reciprocal negotiation of who
occupies which semantic role in a concrete situation represents the normal case.
Epistemic actions are used for this purpose, which in turn act as affordances on
the counterpart.
Thus, Wendy Wilutzky, 2015, argues for a changed understanding of emotions
that are not only the expression of a state of mind (which would correspond to
the linear scheme of ‘input-processing-output’). At least as important is the
function of emotions in shaping the current situation. In this context, the emotion
or facial/bodily expression is not seen as the output/result/end of a social
interaction but its input/tool/start in the sense of an epistemic action. The social
counterpart is forced to react to this emotion—whereby ignoring it also

Art Style | Art & Culture International Magazine
______ ______
68
represents a reaction that does not break off the sequence of events but only
changes it. If emotions are interpreted in this way, they represent affordances for
communicative action from which it is almost impossible to escape. In addition,
emotional expressivity itself is to be understood as an epistemic action because a
situation that was previously unclear is clarified by forcing a response/reaction.
Here, also, evolutionary and developmental psychological roots can be found
that go back a long way. For example, teenagers provoke a reaction from
others—both in their peer group and at home—by varying degrees of
expressivity (from feigned coolness to slamming doors). Only from this reaction,
teenagers become clear about which roles they can sensibly grab in this
structure. The ‘handbag dog’ of our neighbors (biologically it is a ›German Spitz‹)
does it in a similar way, when it yelps at everyone and reads from the reaction its
position in the social ranking, which it did not know before.
Of course, these examples are not only epistemic actions because there is always
an attempt to not only get to know the social structure but also to actively shape
it. This becomes clear when the will to shape is as lacking as curiosity about the
world. The absence of affordances in depression not only paralyzes the affected
individual because instrumental and epistemic actions are greatly reduced. In
addition, the depressive sends out much fewer signals through facial
expressions, which leads to systemic distortions in his or her environment, as
Keith Dixon and Hans-Ueli Fisch (1993, 30) quantitatively demonstrate. As a
result, the depressive in turn stimulates little reactions because of offering few
affordances. This lack of responsiveness of depressives is problematic from an
evolutionary and development-psychological point of view since a central basis
of social cognition thus fails. All primates use ‘social referencing’ to pass on
semantic ratings (the so-called ‘valence’) to infants and toddlers through facial
expressions.
26
The children always cast a casual glance at their mother, who
reacts with an encouraging smile or a serious facial expression.This non-verbal
response signals that a ‘good’ or ‘bad’ situation exists and regulates the child’s
behavior. In one study, the children moved the least far away when the mother
made a ‘fear face’ and the furthest when the mother radiated ‘joy.’
27
Social referencing can therefore be regarded as epistemic action. Both parts of
this dialogue of non-verbal questioning (as an epistemic action to obtain
information) and mimic response (as an instrumental action to control the child)
fulfill the criteria of affordance. This is because the behaviors are used to provoke
a specific reaction by the counterpart. Thus, social referencing can be
understood as an evolutionary and development-psychologically very old form of
affordance. At the same time, it is an example of how social situations are shaped
by affordances that do not require artifacts as carriers. Thus, artifacts are neither
a specific feature of design nor of affordances—as claimed in the introduction.

Art Style | Art & Culture International Magazine
______ ______
69
Author Biography
Klaus Schwarzfischer, Dr. phil., began his career as a multimedia artist and
communication designer. He later studied information design at Donau-Universität
Krems and was awarded the Young Academics Achievement Award of the Society for
Gestalt Theory and its Applications (GTA) for his master thesis in 2015. In 2019, he
received his doctorate in media sciences from the University of Tübingen under
supervision of Klaus Sachs-Hombach. Klaus Schwarzfischer’s main research interests are
cognitive semiotics, Gestalt psychology, design theory, and empirical aesthetics. In
addition, he was a member of the advisory board of the German Society for Semiotics
and head of the design section from 2005 to 2019. Relevant book publications (in
German): Reality as a Design Problem (2008), Transdisciplinary Design (2010), Integrative
Aesthetics (2014), Empirical Aesthetics (2016) and Aesthetics of Reality Construction: How
Are Competing Aesthetic (Design) Preferences Possible? A cognitive-semiotic approach
(2019). Email: ks@indukt.de
Bibliography
Borsato, Andrea. „Zur Phänomenologie der Aufforderungscharaktere.” Grazer philosophische Studien
– International Journal for Analytic Philosophy, 2021, in press. Accepted draft online
https://www.academia.edu/43488594/Zur_Ph%C3%A4nomenologie_der_Aufforderungschara
ktere (21.12.2020)
Dewey, John. Logic. The Theory of Inquiry. New York: Henry Holt & Co., 1938.
Dixon, Keith A. & Fisch, Hans-Ueli. “Ethologische Aspekte der depressiven Erkrankung.” In: Hell, Daniel
(Ed.): Ethologie der Depression. Stuttgart, Jena & New York: Gustav Fischer, 1993, 19–32.
Dror, Itiel E. & Harnad, Stevan. “Offloading Cognition onto Cognitive Technology.” In: Dror, Itiel E. &
Harnad, Stevan (Eds.). Cognition Distributed: How Cognitive Technology Extends Our Minds.
Amsterdam: John Benjamins Publishing, 2008, 1–23.
Engel, Andreas K.; Friston, Karl J. & Kragic, Danica (Eds.). The Pragmatic Turn. Toward Action-Oriented
Views in Cognitive Science. Cambridge, MA: MIT Press, 2015.
von Foerster, Heinz & Pörksen, Bernhard. Understanding Systems, Conversations on Epistemology and
Ethics. Heidelberg: Carl-Auer-Systeme-Verlag, 2002.
Fox, Richard; Panagiotopoulos, Diamantis & Tsouparopoulou, Christina. “Affordanz.” In: Ott, Michael
R.; Sauer, Rebecca & Meier, Thomas (Eds.). Materiale Textkulturen. Konzepte – Materialien –
Praktiken (= Materiale Textkulturen 1). Berlin, Boston & München: Walter de Gruyter, 2015,
63–71:
https://www.degruyter.com/downloadpdf/books/9783110371291/9783110371291.63/97831
10371291.63.pdf (14.1.2021)
Gibson, James J. The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin, 1966.
–––––. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979.
Grinschgl, Sandra; Meyerhoff, Hauke S. & Papenmeier, Frank. “Interface and interaction design: How
mobile touch devices foster cognitive offloading.” Computers in Human Behavior, Vol. 108 (July
2020), 106317. URL: https://doi.org/10.1016/j.chb.2020.106317 (14.01.2021)
Harrington, Anne. Reenchanted Science. Holism in German Culture from Wilhelm II. to Hitler.
Princeton: Princeton University Press, 1996.
Held, Richard & Hein, Alan. “Movement-produced stimulation in the development of visually guided
behavior.”: Journal of Comparative and Physiological Psychology, Vol. 56 (1963), No. 5, 872–876.

Art Style | Art & Culture International Magazine
______ ______
70
Herwig, Arvid. “Experimentelle Handlungsforschung: Die individuelle Perspektive.” In: Prinz, Wolfgang
(Ed.). Experimentelle Handlungsforschung. Kognitive Grundlagen der Wahrnehmung und
Steuerung von Handlungen. Stuttgart: Kohlhammer, 2014, 56–105.
Höhl, Stefanie & Pauen, Sabina. “Preparedness to learn about the world: Evidence from infant
research.” In: Breyer, Thiemo (Ed.) (2015): Epistemological dimensions of evolutionary
psychology. New York, Heidelberg & Dordrecht: Springer, 2015, 159–173
Hubig, Christoph. Die Kunst des Möglichen I. Technikphilosophie als Reflexion der Medialität. Bielefeld:
Transcript, 2006.
Humphrey, Jonah. “Fantastic affordances: Architectures of extreme perceptions.” In: Weber, Ralf &
Technische Universität Dresden (Eds.). Proceedings of the 21st Biennial Congress of
International Association of Empirical Aesthetics IAEA, August 25–28th, 2010, Dresden
Germany. Dresden: TU Dresden, 2010. (CD-ROM Edition ISBN 978-3-86780-183-6)
Hüther, Gerald. “Wie Embodiment neurobiologisch erklärt werden kann.” [2006] In: Storch, Maja;
Cantieni, Benita; Hüther, Gerald & Tschacher, Wolfgang. Embodiment. Die Wechselwirkung von
Körper und Psyche verstehen und nutzen (3rd ed.). Bern: Hogrefe, 2017, 73–97.
Jacob, Frank & Wisch, Kristina. “Affordances – Produkteigenschaften die zur Nutzung motivieren und
wie man sie gestaltet.” In: Bosenick, Tim; Hassenzahl, Marc; Müller-Prove, Matthias & Peissner,
Matthias (Eds.). Usability Professionals 2006 (Berichtband des vierten Workshops des German
Chapters der Usability Professionals Association e.V.). Stuttgart: Fraunhofer IRB Verlag, 2006,
134–137.
James, William. The principles of psychology, Vols. I, II. Cambridge, MA:
Harvard University Press, 1890.
Kant, Immanuel. Kritik der reinen Vernunft. Riga: Johann Friedrich Hartknoch, 1781.
Kirsh, David & Maglio, Paul. “On Distinguishing Epistemic from Pragmatic Action.”
Cognitive Science, Vol. 18 (1994), 513–549.
Klinnert, Mary D.; Campos, Joseph; Sorce, James F.; Emde, Robert N. & Svejda, Marilyn. “Emotions as
Behavior Regulators: Social Referencing in Infancy.” In: Plutchik, Robert & Kellerman, Henry
(Eds.). Emotion. Theory, Research, and Experience. Volume 2: Emotions in Early Development.
New York & London: Academic Press, 1983, 57–86.
Koffka, Kurt. Principles of Gestalt Psychology. New York: Harcourt Brace & Co, 1935.
Lewin, Kurt. Vorsatz, Wille und Bedürfnis. Berlin: Springer, 1926.
Lorenz, Konrad. Behind The Mirror: A Search for a Natural History of Human Knowledge. New York:
Harcourt Brace Jovanovich, 1978.
–––––. Vergleichende Verhaltensforschung. Grundlagen der Ethologie. Wien u.a.: Springer, 1978.
–––––. The Foundations of Ethology: The Principal Ideas and Discoveries in Animal Behavior. New York:
Simon & Schuster, 1982.
Norman, Donald A. The Psychology of Everyday Things. New York: Basic Books, 1988.
–––––. Things That Make Us Smart: Defending Human Attributes in the Age of the Machine. Reading
u.a.: Addison-Wesley, 1993.
–––––. “Affordances and Design.” jnd.org: just noticeable difference, 2018.
https://jnd.org/affordances_and_design/ [retrieval 21.10.2020]
Piaget, Jean. Principles of Genetic Epistemology. (Jean Piaget: Selected Works, Vol. VII).
Oxford/Abingdon (UK): Routledge, 1972.
Sauer, Martina: „Ästhetik und Pragmatismus. Zur funktionalen Relevanz einer nicht-diskursiven
Formauffassung bei Cassirer, Langer und Krois.“ In: Image – Zeitschrift für interdisziplinäre
Bildwissenschaft, Vol. 20 (August 2014), 49–69
Schmidt, Siegfried J. Die Endgültigkeit der Vorläufigkeit. Prozessualität als Argumentationsstrategie.
Weilerswist: Velbrück, 2010.
Schwarzfischer, Klaus. “Was ist transdisziplinäres Design? Beobachtende Systeme und die
Möglichkeiten der Intervention.” In: Romero-Tejedor, Felicidad & Jonas, Wolfgang (Eds.).
Positionen zur Designwissenschaft. Kassel: Kassel University Press, 2010, 119–122.

Art Style | Art & Culture International Magazine
______ ______
71
–––––. Transdisziplinäres Design: Design als Intervention und System-Therapie. Was verbindet jede
Gestaltung in den Bereichen Wirtschaft, Kultur, Politik sowie Gesundheit? Regensburg: InCodes
Verlag, 2010.
–––––. Ästhetik der Wirklichkeits-Konstruktion. Wie sind konkurrierende ästhetische
(Design-)Präferenzen möglich? Ein kognitiv-semiotischer Ansatz. Würzburg:
Königshausen & Neumann Verlag, 2019.
Stock, Armin & Stock, Claudia. “A short history of ideo-motor action.” Psychological Research,
Vol. 68 (2004), 176–188.
Strübing, Jörg. Grounded Theory. Zur sozialtheoretischen und epistemologischen Fundierung eines
pragmatischen Forschungsstils (3rd ed.). Wiesbaden: Springer VS, 2014.
von Uexküll, Jakob. Umwelt und Innenwelt der Tiere. Berlin: Springer, 1909.
–––––. Theoretical Biology. New York: Harcourt, Brace & Co., 1926.
–––––. A Stroll Through the Worlds of Animals and Men: A Picture Book of Invisible Worlds [1934]. In:
Claire H. Schiller (Ed.). Instinctive Behavior: The Development of a Modern Concept. New York:
International Universities Press, 1957, 5–80.
Wilutzky, Wendy. “Emotions as pragmatic and epistemic actions.” Frontiers in Psychology, Vol.6
(2015), Article 1593, doi: 10.3389/fpsyg.2015.01593
Zillien, Nicole. “Die (Wieder-)Entdeckung der Medien – Das Affordanzkonzept in der
Mediensoziologie.” Sociologia Internationalis, 46 (2008), No. 2, 161–181.
Notes
1
This connection was first presented by Schwarzfischer at a talk about ›A Systemsemiotic
Approach to Design‹ at EAD06, 6th European Academy of Design in Bremen, March 29th 2005.
2
Sauer, 2014, extends the arts to the wider scope of pragmatism. Whereas, Schwarzfischer, 2019,
completely leaves behind the arts, when combining the act-theoretical approaches of Piaget,
1972, and Lorenz, 1982. The resulting account is plausible from phylogenetic as well as
ontogenetic perspectives.
3
Kirsh & Maglio, 1994, distinguish ›epistemic action‹ (which aim at a gain of knowledge about the
world) and ›pragmatic action‹ (which target to produce a specific state of the world). From a
semiotic point of view we better do not speak of ›pragmatic action‹ here because any action has
pragmatic aspects. Therefore, I will call them ›instrumental actions‹ instead of ›pragmatic actions‹.
This differentiation is relevant for ›design‹ as we noted above: Hence, any intervention is design.
In technical terms, shaping the world means that we try to realize a target state—this means, we
apply an ›instrumental action‹. To get the actual value (and to decide whether it is different from
the target value) we need an observation (which is formally a measurement of any kind). This is an
›epistemic action‹ which can be very simple or rather complicated (e.g. as an experimental
design). It shows that the distinction by Kirsh & Paul, 1994, is an analytical one, because in
everyday life both types occur combined. For example, any experimental design (as an ›epistemic
action‹) needs several ›instrumental actions‹ to be realized and to work.
4
For example: Norman, 1988, 1993 and 2018; Jacob & Wisch, 2006; Humphrey, 2010; Fox,
Panagiotopoulos & Tsouparopoulou, 2015.
5
The cognitive scientist Norman, 1993: 139 ff., analyzes affordances with respect to ›distributed
cognition‹. He demonstrates how affordances are used to minimize cognitive load and user errors.
The targeted anchoring of affordances in a situation is unburdening the user, why it is also called
›cognitive offloading‹—see Dror & Harnad, 2008, or Grinschgl et al., 2020.
6
Gibson, 1966, 285.
7
Lewin, 1926, 28 and 59–62.

Art Style | Art & Culture International Magazine
______ ______
72
8
Borsato (2021, in press); translation by Klaus Schwarzfischer.
9
Gibson, 1966, 285
10
Uexküll, Theoretische Biologie, [German Edition], 1920; Uexküll, Theoretical Biology, [English
Edition], 1926
11
Uexküll, 1909, Umwelt und Innenwelt der Tiere.
12
Uexküll, 1934, 47–48, tries to reconstruct the ›first-person-perspective‹ of animals to be able to
grasp their subjective reality: He speaks of the ›sitting tone‹ (which a basket has for a dog), the
›climbing tone‹ (which a ladder emits), and the ›beating tone‹ (which an object emits for someone
looking for a weapon). This is clearly reminiscent of Koffka, 1935, 7: »To primitive man each thing
says what it is and what he ought to do with it: a fruit says, ›Eat me‹; water says, ›Drink me‹;
thunder says, ›Fear me‹, and woman says, ›Love me‹.« Who has taken over this affordance
concept from whom can remain open here. Because of his phenomenological methodology and
his closeness to Gestalt psychology, Harrington, 1996, has dedicated an entire chapter to Uexküll.
13
Kant, 1781, Kritik der reinen Vernunft.
14
Lorenz, 1982, 328. In the German version of this sentence, Konrad Lorenz, 1978, 259–260,
speaks explicitly of ›Neugierwesen‹ (curiosity beings).
15
Lorenz, 1978, Behind The Mirror.
16
Gibson, 1966, 223
17
This attribution means that these are non-trivial systems according to Heinz von Foerster, which
provide different outputs despite the same input—see Foerster & Pörksen, 2002, 54–57.
18
In the present context, the term ›sensorimotor‹ should not be confused with the term Piaget,
1972, 20–31, uses in his genetic epistemology for the first phase of life, which he also calls ›sensori-
motor‹ In fact, however, Piaget is referring to what is called ›ideomotor‹ in more recent literature.
The conceptual differentiation between ›sensorimotor‹ and ›ideomotor‹ was not yet established in
Piaget’s time, although the principle can be traced back to James, 1890, see Stock & Stock, 2004.
19
Consequently, both ›instrumental actions‹ and ›epistemic actions‹ can occur in all three
spheres, which the technology philosopher Hubig, 2006, 141–142 and 241, distinguishes: The
cognitive intellectual technique, the physical real technique and the institutional social technique.
Hence, all three areas are fields that can be designed-—and must.
20
Hüther, 2017, 83–84 [translation by KS]
21
See Schmidt, 2010, 104–105
22
In the lifeworld, however, even finer differentiations are necessary, so that Schwarzfischer 2019,
317, proposes a ›normal version‹ of the process model with eight levels each in the top-down
pathway (the world of action) and the bottom-up pathway (the world as sensed).
23
This is the basis of ›reafference‹, as described by Lorenz 1982, 63, by reference to rats that are
frustrated when the expected action effects do not occur although the action is performed.
24
For example, see Zillen, 2008
25
For more detailed information, please refer to the ›inquiry cycle‹ by Dewey, 1938. This can shed
light on scientific problems as well as on everyday irritations in the course of the action. A
visualization with detailed explanation can be found in Strübing, 2014, 43, and Schwarzfischer,
2019, 129–130.
26
Höhl & Pauen, 2015, show that the social dimension already plays a major role in learning in
infants (e.g. joint attention and social referencing, in which the observed facial expressions of the
other person assign an object its valence). A ›fear face‹ of the mother assigns a negative valence
to the object, whereupon the infant does not approach the object further. A mother’s smile
assigns a positive valence to the object and the infant continues to approach the object with
encouragement. This observation disproves the common saying »Burned child shuns fire« at least
in important parts. This is because the assumed direct interaction with the object does not have to
have taken place at all since the assignment of the valence is already done at sight through social
referencing.
27
For details of this study see Klinnert et al., 1983, 67