Content uploaded by Christoph von der Malsburg
Author content
All content in this area was uploaded by Christoph von der Malsburg
Content may be subject to copyright.
The Correlation Theory of Brain
Function
Christoph von der Malsburg
Max-Planck-Institute for Biophysical Chemistry
P.O.Box 2841, D-3400 Gottingen, W.-Germany
Abstract
A summary of brain theory is given so far as it is contained within the
framework of Localization Theory. Diculties of this \conventional the-
ory" are traced back to a specic deciency: there is no way to express
relations between active cells (as for instance their representing parts of
the same object). A new theory is proposed to cure this deciency. It
introduces a new kind of dynamical control, termed synaptic mo dulation,
according to which synapses switch between a conducting and a non-
conducting state. The dynamics of this variable is controlled on a fast
time scale by correlations in the temporal ne structure of cellular signals.
Furthermore, conventional synaptic plasticity is replaced by a rened ver-
sion. Synaptic modulation and plasticity form the basis for short-term
and long-term memory, respectively. Signal correlations, shaped by the
variable network, express structure and relationships within ob jects. In
particular, the gure-ground problem may be solved in this way. Synaptic
modulation introduces exibilityinto cerebral networks which is necessary
to solve the invariance problem. Since momentarily useless connections
are deactivated, interference between dierent memory traces can be re-
duced, and memory capacity increased, in comparison with conventional
associative memory.
Originally published July 1981 as Internal Report 81-2, Dept. of Neurobiology, Max-
Planck-Institute for Biophysical Chemistry, 3400 Gottingen, W.-Germany
1
Contents
1 Introduction 4
2 Conventional Brain Theory 4
2.1 Localization Theory
: : : : : : : : : : : : : : : : : : : : : : : : :
4
2.1.1 The Macroscopic Level
: : : : : : : : : : : : : : : : : : : :
4
2.1.2 The Microscopic Level
: : : : : : : : : : : : : : : : : : : :
5
2.1.3 The Brain as a Pro jection Screen
: : : : : : : : : : : : : :
5
2.2 The Problem of Nervous Integration
: : : : : : : : : : : : : : : :
6
2.2.1 The General Question
: : : : : : : : : : : : : : : : : : : :
6
2.2.2 Representation of Structured Objects
: : : : : : : : : : :
6
2.2.3 Invariance
: : : : : : : : : : : : : : : : : : : : : : : : : : :
7
2.2.4 Memory
: : : : : : : : : : : : : : : : : : : : : : : : : : : :
7
2.2.5 Self-Organization
: : : : : : : : : : : : : : : : : : : : : : :
7
2.2.6 Control of Action
: : : : : : : : : : : : : : : : : : : : : : :
7
2.3 Proposed Solutions
: : : : : : : : : : : : : : : : : : : : : : : : : :
8
2.3.1 Synaptic Plasticity
: : : : : : : : : : : : : : : : : : : : : :
8
2.3.2 Feature Detectors
: : : : : : : : : : : : : : : : : : : : : :
9
2.3.3 Cell Assemblies
: : : : : : : : : : : : : : : : : : : : : : : :
9
2.3.4 Associative Memory
: : : : : : : : : : : : : : : : : : : : :
10
2.3.5 Visual Perception, Perceptrons
: : : : : : : : : : : : : : :
11
3 The Correlation Theory of Brain Function 12
3.1 Modications to Conventional Theory
: : : : : : : : : : : : : : :
12
3.1.1 Correlations between Cellular Signals
: : : : : : : : : : :
12
3.1.2 Synaptic Mo dulation
: : : : : : : : : : : : : : : : : : : : :
13
3.1.3 Rened Plasticity
: : : : : : : : : : : : : : : : : : : : : : :
13
3.2 Elementary Discussion
: : : : : : : : : : : : : : : : : : : : : : : :
14
3.2.1 Sources of Correlations
: : : : : : : : : : : : : : : : : : : :
14
3.2.2 Eects of Correlations
: : : : : : : : : : : : : : : : : : : :
14
3.2.3 Correlation Dynamics
: : : : : : : : : : : : : : : : : : : :
14
3.3 Network Structures
: : : : : : : : : : : : : : : : : : : : : : : : : :
15
3.3.1 The Topological Network
: : : : : : : : : : : : : : : : : :
15
3.3.2 The Correlate of a Topological Network
: : : : : : : : : :
15
3.3.3 Projection between Topological Networks
: : : : : : : : :
16
3.3.4 Composite Elements
: : : : : : : : : : : : : : : : : : : : :
17
3.3.5 The Synergetic Control of Action
: : : : : : : : : : : : : :
18
3.4 Applications of Correlation Theory
: : : : : : : : : : : : : : : : :
19
3.4.1 Visual Elements
: : : : : : : : : : : : : : : : : : : : : : :
19
3.4.2 Figure-Ground Discrimination
: : : : : : : : : : : : : : :
20
3.4.3 Invariant Image Representation
: : : : : : : : : : : : : : :
20
3.4.4 Interpretation of an Image
: : : : : : : : : : : : : : : : : :
21
2
4 Discussion 22
4.1 The Text Analogy
: : : : : : : : : : : : : : : : : : : : : : : : : :
22
4.2 The Bandwidth Problem
: : : : : : : : : : : : : : : : : : : : : : :
23
4.3 Facing Experiment
: : : : : : : : : : : : : : : : : : : : : : : : : :
24
4.4 Conclusion
: : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
24
A Literature 25
3
1 Introduction
The purpose of this paper is to p oint out a specic deciency in existing brain
theory and to propose a way in which this gap could be lled. Although it
leaves open a number of technical questions and presents more a program than
an accomplished theory, at least a language is developed to describe processes
from the cellular to the cognitive level.
Searching for the function of the brain in all generalitymay be regarded as a
venture bound to fail in view of the diversity of function of even a single brain.
It is clear that any answer to the question can only be of a very general kind,
much as a \general theory of painting" can only be a description of the process
by which pigments are prepared and applied to canvas with a brush, and could
say nothing about art, subject, style and interpretation.
There is every reason to believe in the existence of general principles govern-
ing the function of the brain. Cerebral anatomy is surprisingly homogeneous in
spite of the diversity of functional modalities represented in its dierent parts.
The rapid cultural development of man has created elds of mental activity
for which the brain cannot have been prepared by phylogeny in any detailed
way. Both arguments seem to force the conclusion that the brain is governed
by general principles of organization.
2 Conventional Brain Theory
The literature on brain theory is vast and cannot be summarized here. This
chapter concentrates on a set of ideas which is fairly consistent in itself and with
experiments. My account passes over most of the rich and sometimes ingeneous
detail to which these ideas have been worked out in the literature. However, I
try to bring out points where the ideas fail.
2.1 Lo calization Theory
2.1.1 The Macroscopic Level
Observation of behavioural defects caused by localized lesions of the brain has
rmly established that dierent parts of the brain are preoccupied with dierent
modalities,
topics of mental activity [Luria, 1973]. Examples are vision, audi-
tion, motor control, basic emotions and drives (e.g. aggression, pleasure and
hunger), various aspects of speech, and long-term planning of action. The abil-
itytolaydown long-term memory can be destroyed by a specic local lesion;
however, already existing long-term memory is not aected thereby. Memory
traces and the ability to recall seem to be localized together with the modalities
to which they refer.
Several kinds of hierarchy can be construed on the basis of the modalities.
For instance, sleep-waking regulation, drives, emotions and planning all exert
4
global control over the rest of the mind. However, I will treat all localized
topics as on the same level. The term hierarchy will be reserved to forms of
cooperation of lo calized objects.
There are aspects of the mind's function which can not be localized in parts
of the brain. Among these are consciousness and reason.
2.1.2 The Microscopic Level
In recent decades localization theory has been rened down to the microscopic
level. The information carrying units are thought to be nerve cells. These
produce signals in the form of short (1 msec) electro-chemical pulses, which can
be recorded with the help of ne electrodes.
How are these signals to be interpreted? For the more central stages of the
brain neurophysiology has answered this question in an operational way. In a
vast majority of experiments signals are evaluated in terms of a peri-event time
histogram. An eventisconstituted by a stimulus presented to the brain or a
response evoked from the brain. The time shortly before and after the eventis
divided into a raster of small intervals (typically 1 to 10 msec). The event is
repeated and the mean number of spikes falling into eachinterval is recorded.
The mean frequency reacts to the event, if at all, by a short increase or decrease.
The art of the experimenter consists in nding an event which inuences the
activity of the cell he is recording from. In this way a topic can be assigned to
the cell. Atypical topic is \an edge of light with a particular spectral component
moving in a particular direction over a point of the retina".
The success of neurophysiology with this type of experiment has been tremen-
dous. It is true, not all of brain has been mapped in this way, and in fact it may
not be practical to do so, because some of the events may be dicult to nd.
Nevertheless, many scientists are ready to extrapolate the microscopic version
of localization theory to all of the central nervous system. In this ultimate form
localization theory can be summarized as \one cell - one topic": cells are the
atoms of meaning.
2.1.3 The Brain as a Projection Screen
The topology of the periphery is preserved in the central representations, e.g.
neighboring points of a sensory surface project to neighboring points on a central
sheet of cells. To each point of a sense organ (retina, cochlea, skin) there cor-
responds a small region centrally, often called hypercolumn. Single cells in that
region are specialized to particular combinations of qualityvalues describing the
point of the sensory surface (e.g. spectral distribution, direction of movement,
stereo-depth).
A single peripheral surface is represented by several central elds, whichmay
vary in their emphasis on dierent qualities, and which usually are connected
by topologically ordered bre pro jections.
5
Again, this picture has to be extrapolated to cover all of brain. It presents
an experimental challenge to nd the precise terms of this extrapolation. Let
us suppose it will turn out to be possible. Then the physiological-anatomical
picture is that of a screen on which patterns appear corresp onding to input,
output and internal processing (e.g. emotional, planning), similar to the moving
pictures on a colour television screen.
2.2 The Problem of Nervous Integration
2.2.1 The General Question
The picture of the brain as a pro jection screen is very suggestive, and in its prin-
cipal traits it is well founded in experimental observation. However, the picture
poses a problem, that of nervous integration: in what way do the dierent pat-
terns of activityinteract? To be sure, the machinery for cellular interactions is
conspicuously there: each cell is contacted on its dendritic and somatic mem-
brane by many synapses through which its membrane potential can be raised
or lowered upon arrival of nerve impulses. The axonal bres and branches for
the transport of impulses from cells to synapses ll the larger part of the brain's
volume. The nervous integration question more precisely asks how this machin-
ery is organized. The problem calls for new concepts, and at present it cannot
be attacked experimentally.
One can train or ask a subject to react to the presentation of an apple by
pressing a button: on command the subject can organize its brain so that upon
the appearance of a certain class of patterns in the visual modality another
pattern of a certain structure is created in the motor modality. This simple
everyday example alone combines in it several complex organization processes
which will now be named.
2.2.2 Representation of Structured Objects
In our cultural world we form symbols of a higher order by the juxtaposition
(in time or space) of symbols of a lower order, e.g. words out of letters or
phonemes. According to lo calization theory neurons are the basic symbols in
the brain. Their position is xed and cannot be used to form groupings. Another
code is required to represent associtation of cells into patterns forming symbols
on a higher level.
When we analyse a complex visual scene it is important to break it down
into patterns which are simple enough so that we can hope to recognize them,
e.g. identify them with ob jects wesaw before. A single pattern in turn has to
be broken down into subpatterns, possibly through several stages, e.g. man -
arm - hand - nger - joint (cf. [Sutherland, 1968]). It should be p ossible to
group neurons into such a hierarchy of patterns in a exible way, without the
introduction of new hardware for new patterns.
6
2.2.3 Invariance
It is an everyday experience that there are objects of a relatively xed structure,
which aect our senses in an enormously variable way. E.g. the picture of an
apple can vary in persp ective and in position and size on the retina, depending
on the relative coordinates of eye and apple. It is important to reduce this
variabilitytoaninvariant representation of the intrinsic structure of the ob ject,
in order to be able to generalize, i.e. draw the same conclusions from the per-
ception of an object, irrespectiveofvariations in its appearance [cf. Sutherland,
1968]. An analogous discussion applies to motor patterns.
2.2.4 Memory
There must be a physical basis for the gradual acquisition of information. We
usually discuss it under two aspects. According to one the brain changes to
acquire new abilities. This will be the subject of the subsequent paragraph.
The other is the ability of the brain to recall complex patterns which were
active at an earlier time. With this memory in the narrow sense, recall should
be evoked by an activity pattern or an input which are suciently close to the
original pattern.
2.2.5 Self-Organization
Self-organization refers to the ability of the brain to organize structures and
activity patterns. The term \organize" implies that the process is directed
toward some useful goal, which still has to be dened. A goal we already
mentioned is the retention of earlier states of activity. In this way the brain can
become independent of external sources of information and can build models
for phenomena. Other goals will be dened in later sections.
The ability to organize itself sets the brain in sharp contrast to the computer,
which relies entirely on a human programmer. It also is the basis of the reliability
of the brain, being able to \repair" deviations from an optimal conguration.
Self-organization puts heavy constraints on possible functional schemes for the
brain.
2.2.6 Control of Action
The metaphor of the brain as a projection screen assigns a passive role to it. In
realitywe know that the brain is spontaneously active: The \pro jector" is an
integral part of it, to stay with the metaphor. Accordingly, a solution to the
nervous integration problem has to include a scheme for the control of processes
and the global integration of action.
7
2.3 Prop osed Solutions
Localization theory, section 2.1, proposes a basic frame into whichany functional
scheme for the brain has to be tted. It p oses the nervous integration problem,
some aspects of whichhave been presented in 2.2. This section discusses some
potential solutions whichhave been prop osed in the literature, and points out
some problems which they do not solve.
2.3.1 Synaptic Plasticity
A synaptic connection can be characterized by the size of the postsynaptic
conductance transient (PCT) which is produced in the postsynaptic cell upon
arrival of a nerve impulse in the presynaptic bre. PCT size may slowly change
under the control of the neural signals on the presynaptic and the postsynaptic
side. This leads to a feedback situation: PCT size (together with the presy-
naptic signal) has inuence on the postsynaptic signal, which in turn controls
the change of the PCT. If this feedback is positive and if the changes impressed
on the PCT are permanent (non-decaying) one speaks of synaptic plasticity.
The formation of new synapses may be included in the denition of synaptic
plasticity [Ariens Kappers et al., 1936]. In the case of an excitatory synapse
the EPCT (excitatory PCT) is increased (or a synapse established) after co-
incidence of neural activity on the presynaptic and postsynaptic side. In the
framework of localization theory this is a straight-forward implementation of the
idea of an association and of Pavlows conditioned reex. It is usually assumed
that plastic synaptic changes need seconds to become established and hours to
consolidate (show full eect and stabilize). Synaptic plasticity has been shown
experimentally to exist [Bliss and Gardner-Medwin, 1973, Baranyi and Feher,
1981], although it is, in the presence of controlling signals, intrinsically dicult
to demonstrate its non-decaying nature.
The instability which is caused by positive feedback has to be controlled
somehow. Several schemes have been proposed: an upper limit to the synaptic
weight (PCT size for a single pulse); limitation of the sum of all synaptic weights
converging on a cell or diverging from a cell; and stabilization of the mean level
of activity in the postsynaptic cell. The latter says that if the time average (over,
say, several hours) of cell activity exceeds a certain value either all excitatory
synapses converging onto the cell are reduced in weightby a certain factor (and
if this average is too low the synapses are increased in weight), or the inhibitory
synapses are increased in weight.
Synaptic plasticity is thought to be the basis of memory. The p ositive feed-
backinvolved in it leads to the kind of instability that is required for pattern
generation and self-organization. In this sense synaptic plasticity is analogous
to self-reproduction in biological evolution.
8
2.3.2 Feature Detectors
In the context of theoretical discussions within the frame of localization theory
a cell in the sensory part of the brain is termed a
feature detector, feature
being
the term for the eventby which the cell is triggered [see e.g. Sutherland, 1968].
Feature detectors may dier in level. On the lowest level they respond to the
signal of a peripheral sensory cell. On the highest conceivable level feature
detectors respond to the appearance of entire ob jects [Barlow, 1972]. They are
then referred to as cardinal cells. Feature detectors of intermediate level are
found experimentally (a typical feature has been described in 2.1.2).
Fairly specic feature detectors are proposed in many models of perception
as a basis for the discrimination between complex patterns. The p ostulated
level of feature detectors is regulated by a trade-o. The higher the level (the
more specic the cells) the smaller the overlap of the sets of cells responding
to dierent patterns and the easier the task of discriminating between them.
High-level features, on the other hand, mean large numbers of cells, less exi-
bility (because specic trigger features must be adapted to particular pattern
universes) and low duty cycle for each particular cell.
Many models employ cardinal cells because they seem to solve the problem
indicated in 2.2.2, representation of complex objects. In reality that problem is
not solved by cardinal cells. Either a cardinal cell is able to represent a whole
class of objects. Then the individual object cannot be represented in detail,
because the signal of a single cardinal cell is too crude. Or there has to be a
cardinal cell for each pattern (a person with a new facial expression constituting
a new pattern!). The number of cardinal cells required would then be forbidding
(even if the invariance problem 2.2.3 had been solved somehow), and it would be
impossible to recognize new patterns which diered from familiar ones merely in
detail. In addition, a cardinal cell would have to be silent (possibly for decades)
until its pattern appeared again, but there is every reason to believe that a cell
which is forced to stay silentforaday (e.g. by deaerentation) will change its
synaptic make-up to become active again.
From this discussion it follows that high-level feature detectors do not solve
any of the nervous integration problems. Low-level feature detectors, on the
other hand, are an experimental fact and have to be the building blocks of any
theory under the roof of localization theory.
2.3.3 Cell Assemblies
Stimulation of some part of the brain will switch on many cells simultaneously.
It therefore appears natural in the context of localization theory to regard sets
of simultaneously activated nerve cells as the basic internal objects. The ner-
vous integration problem requires that such sets should not just be passively
activated by stimuli, that they should rather be dynamical units, integrated by
interactions. The
cell assembly
[Hebb, 1949] is a model idea describing a certain
9
system of suchinteractions.
A cell assembly is a set of neurons cross-connected such that the whole
set is brought to become simultaneously active upon activation of appropriate
subsets, which have to be suciently similar to the assembly to single it out
from overlapping others. In view of the uctuating nature of cellular signals
activation of cells in an assembly is simultaneous only on a coarse time scale,
longer than, say, 50 msec.
Assembly reconstitution, or its completion from parts, has been proposed as
the fundamental process of brain function. Important special cases would be the
attachment of abstract symbolic representations to sensory patterns (recogni-
tion), the reactivation of supplementary information stored by past experience,
and the generation of a response pattern which has previously b een associated
with a stimulus. According to this view, the action of the brain is controlled
by a succession of such completion processes, alternating with the (partial) de-
cay of assemblies (due to some delayed disfacilitation) leaving residues which,
together with new stimuli, form the germs for other assemblies.
Analysis of long perio ds of brain activitywould reveal a hierarchyofsub-
patterns which appear as part of many assemblies. The dynamics of assembly
completion possibly could be interpreted as interaction among subassemblies,
analogous to excitation and inhibition exchanged between single cells. Sub-
assembly interactions would havetob e realized with the help of the synaptic
interactions of the constituent cells.
It is an unsolved question whether assembly interactions with these speci-
cations are possible [see Legendy, 1967]. However, the assembly concept has a
more fundamental aw. When a particular assembly is active, there is no basis
on which it could be analysed into subassemblies: it just consists of a mass of
simultaneously active cells. (The above analysis into subassemblies was only
possible in a Gedankenexperiment.) This must lead to serious ambiguities. For
instance, when we see a visual pattern, it is not only necessary to know which
collection of features apply to it, but also in whichway they are grouped. Even
if the feature set is so complete that it can only be combined in one wayinto
an image it is important to know this combination. (When we see two people
in the street we usually don't confuse which jacket is worn together with which
trouser by one of them.) In particular, it must be possible to represent the
result of a successful gure-ground discrimination.
2.3.4 Associative Memory
Assemblies are supposed to be formed by synaptic plasticity. A pair of simul-
taneously stimulated cells establishes or strengthens its synaptic connection (in
case there is a ber bridging their distance). If this happens to many pairs
of cells in a repeatedly activated pattern an assembly can be formed. Several
detailed schemes for this process have been proposed and analyzed under the
name of associative memory. Analysis has been made possible by simplifying
10
assumptions (e.g. linearity, only one exchange of interactions, small overlap). It
has been shown that manyoverlapping assemblies can be stored and retrieved
in the same network without to o muchinterference between them.
The lack of internal structure in assemblies leads to a serious diculty of
associative memories: each memory trace recalls a xed subset of cells without
possible variation apart from noise. However, cognitive psychology makes it ob-
vious that realistic memory traces often correspond to a network of subpatterns
connected in a exible way to t a particular situation.
2.3.5 Visual Perception, Perceptrons
Visual perception presents two outstanding problems, gure-ground discrimina-
tion and invariants extraction. The perceptron approach [Rosenblatt, 1961] to
perception, which makes use of most of the ideas reviewed so far, demonstrates
quite explicitely the inadequacies of those ideas to solve the two problems men-
tioned.
Perceptrons are meant to be models for sensory subsystems of the brain.
A typical perceptron consists of threshold units (neurons) of three kinds,
S
,
A
and
R
, i.e. sensory, association and response units. These are arranged in
layers which are sequentially connected:
S
!
A
!
R
. Cross-connections within
a layer, or even backward connections may also exist. The
A
-units play the
role of feature detectors. The
A
!
R
connections are modiable by synaptic
plasticity.
The prominent feature of a perceptron is its ability to reorganize itself in
response to the repeated activation of a certain subset
s
of
S
-units such that
subsequently a specic
R
-unit res precisely when
s
is presented.
The invariance problem calls for a single
R
-unit to respond to the presen-
tation of a pattern
p
in any position in
S
. Rosenblatt proposed to solve the
problem by the introduction of a second layer
A
0
of feature detectors, sand-
wiched between
A
and
R
:
S
!
A
!
A
0
!
R
. A unit
a
0
in
A
0
responds to
the presentation of a certain feature in arbitrary position within
S
. Unit
a
0
is enabled to do so by the presence of units
a
i
in
A
,
i
= 1
;:::;N
a
0
, each of
which responds to the particular feature in a position
x
i
in
S
. All units
a
i
have
a connection to
a
0
, which res if at least one cell of the set
f
a
i
g
is activated.
Many dierent feature detectors analogous to
a
0
are presentin
A
0
. The pattern
p
will activate the same subset of
A
0
, independent of its position in
S
.Aspe-
cic
R
-unit can now be trained to react to this subset. Activity in a unit of
A
0
makes a statement about the presence of a particular subpattern (or feature)
of
p
. In order to generalize over position in
S
information about the position
of the subpattern is discarded. If the features are suciently complex it may
be possible in principle to recover the relationships of overlap and reconstruct
the full pattern
p
,inaway analogous to solving a jigsaw puzzle. This recon-
struction, however, is nowhere done in a perceptron, and the recognition of
p
has to be done on the basis of the uncorrelated feature set represented by the
11
active units in
A
0
. This is only p ossible if the features represented in
A
0
are of
suciently high level, which means that they are very numerous, or specialized
to a particular universe of patterns in
S
, or both. The machinery needed, par-
ticularly in
A
, is gigantic (as is demonstrated by a recent simulated version of
a perceptron [Fukushima, 1980]). It is evident that an enormous improvement
over the perceptron could be made with the help of a scheme by which the
overlap conditions of subpatterns would be expressed.
An
R
-unit can supress all
a
0
units not belonging to its own trigger set if
appropriate inhibitory back-couplings
R
!
A
0
are present. Rosenblatt proposed
to solve the selective-attention problem in this way. He recognized, however,
that this is no solution to the general gure-ground problem, since learning
and recognition of a gure have to precede the suppression of its background.
He admitted that new concepts were needed for the problem of gural unity
[Rosenblatt, 1961, p.555]. Again, this calls for a scheme by which cells in
A
0
could express their relatedness in terms of separate gures.
3 The Correlation Theory of Brain Function
3.1 Mo dications to Conventional Theory
This section introduces i) a scheme for the interpretation of cellular signals
which is a renement of the one given in 2.1.2, and ii) a short-term analogue of
synaptic plasticity.
3.1.1 Correlations between Cellular Signals
In paragraph 2.1.2 I discussed the experimental procedure by which the corre-
lation of a cellular signal to an event is detected. The averaging in the peri-
event-histogram method is important to get rid of an apparently random time
structure within the cellular response. This time structure will now become
important.
Consider the spike trains emitted bytwo cells in the central nervous system.
These signals maybe evaluated in terms of a
correlation
1
. It is supposed to
measure the similaritybetween the two signals and should at least discriminate
between synchrony and asynchrony in their temporal ne structure (with a
resolution of 2 or 5 msec). It has to be assumed that the trivial state in which
all cells are globally correlated with each other is suppressed by a system of
inhibitory connections which permits only a small fraction of all cells to be
activeatany one time.
1
The term 'correlation' is not meant to imply a specic mathematical formulation.
12
3.1.2 Synaptic Modulation
The synaptic connection between brain cells
i
and
j
is characterized by a
strength
w
ij
. It is a measure for the size of the PCT evoked in cell
i
upon
arrival of a spike from cell
j
. I here postulate that the weight
w
ij
of an exci-
tatory synapse depends on twovariables with dierent time-scale of behaviour,
a
ij
and
s
ij
. The set
f
s
ij
g
constitutes the permanent network structure. Its
modication (synaptic plasticity) is slow and is the basis for long-term memory.
The new dynamic variable
a
ij
, termed
state of activation
of synapse
ij
,changes
on a fast time-scale (fractions of a second) in response to the correlation be-
tween the signals of cells
i
and
j
. With no signals in
i
and
j
,
a
ij
decays towards
a resting state
a
0
, within times typical for short-term memory. With strong
correlation between the signals the value
a
ij
changes such that
w
ij
increases
(activation).
With uncorrelated signals
a
ij
changes such that
w
ij
decreases to
zero
(inactivation).
This behaviour of the variable
a
ij
will be referred to here
as
synaptic modulation.
It can change the value of
a
ij
signicantly within a
fraction of a second. Not all synapses from a given cell to other cells can grow
at the same time, since the inhibitory system referred to in 3.1.1 prevents those
target cells from all ring simultaneously; also the synapses received by a cell
compete with each other, for the same reason. The physical basis for synaptic
modulation is not clear; it might correspond to the accumulation or depletion
of some chemical substance at a strategic location in or near the synapse. The
relevant postsynaptic signal is here taken to be the cell's output spike train, but
it may also be a more local dendritic signal. As a simple example one could
assume
w
ij
=
a
ij
s
ij
with 0
a
ij
1 and a resting state
a
0
within the interval
(0
;
1).
3.1.3 Rened Plasticity
The variables
f
s
ij
g
are controlled by what I shall call
rened synaptic plasticity:
strong correlation between the temporal ne structure in the signals of cells
i
and
j
causes
s
ij
to grow; this growth may b e thought to be limited in the usual
way (e.g. by sum-rules). Absence of correlation does not directly reduce
s
ij
.
The analogy between synaptic modulation and rened plasticity is apparent.
Both are controlled by correlations in the signals of cell pairs in a positive
feedback fashion. They dier in time-scale of decay (seconds for
a
ij
, decades
to permanent for
s
ij
), and of build-up; and they dier in the way they are
controlled. The
a
ij
react only to the two locally available signals and are both
increased and decreased by correlations and their absence. The
s
ij
are only
increased by local signals and are decreased in resp onse to the growth of other
synapses.
13
3.2 Elementary Discussion
3.2.1 Sources of Correlations
Correlations between the signals of cells can be caused by time structure in
sensory signals exciting the cells. However, there is a more important source of
correlations. Time structure in cellular signals can be created spontaneously,
e.g. by a tendency of cells to form bursts of spikes. Correlations arise if this
time structure is transmitted to other cells by excitation or inhibition.
3.2.2 Eects of Correlations
One eect of the correlation between signals in cells
i
and
j
was already men-
tioned: activation of the synaptic weight
w
ij
. Specic connection patterns
(e.g.reex arcs) can be created in this way, and a plurip otent network can be
turned temp orarily into a specialized machine.
Secondly, a correlation between the signals of cells
i
and
j
enables them to
positively interact to excite a third cell
k
(if
w
ki
,
w
kj
6
= 0): the individual signals
may not be able to transcend the threshold of cell
k
, whereas simultaneously
arriving signals may. Two subnetworks with uncorrelated activity patterns may
coexist without interfering.
Thirdly, correlations control (rened) synaptic plasticity. The absence of
correlations between two activity objects, even if they sometimes coexist within
the same span of a second, keeps them from developing mutual synapses.
3.2.3 Correlation Dynamics
The dynamical system introduced so far for the cellular signals and temporary
synaptic strengths forms a basis for organization. The correlation between the
signals in cells
i
and
j
and the states of activation
a
ij
and
a
ji
of their common
synapses form a positive feedback loop, driving
a
ij
and
a
ji
away from the resting
state
a
0
, and the signal pair away from a structureless uncorrelated state. In
this way correlations can stabilize their own existence and cease to be transitory
and shaky statistical epiphenomena. Dierent synapses on one cell compete with
each other, as was pointed out above, and certain sets of synapses co operate with
each other. For instance, the direct pathway from cell
i
to cell
j
cooperates with
the indirect pathway from
i
to a cell
k
to cell
j
. These dynamical interactions
between synapses and corresp onding signals tend to stabilize certain optimal
connectivity patterns (together with their corresponding signal patterns). These
can be characterized locally as having sparse connections (to avoid competition),
which are arranged so as to have optimal local cooperation b etween them.
The slow comp onent
s
ij
of synaptic strength is plastically modied only by
strong correlations, i.e. mainly when connectivity forms an optimal pattern.
Therefore the structure of the permanent network tends to be a superposition
of optimal connectivity patterns. When input to the network activates certain
14
cells (possibly in a correlated fashion), a dynamical process of organization sets
in, as a result of which synapses forming an optimal network are fully activated
and all other synapses are deactivated.
3.3 Network Structures
It is not clear how optimal connectivity patterns can be characterized globally.
This chapter proceeds on the basis of the conjecture that they have a topological
structure to them, the neighbourhoods of whichmay correspond to overlapping
sets of co-active cells. (Mathematical rigour is not attempted in the notation.)
3.3.1 The Topological Network
Let
S
be a set of
n
cells,
E
an appropriate space of low dimensionality,
m
a map
assigning to each cell of
S
a pointin
E
, and
p
a natural number,
p
n
. The
topologically structured network
~
S
=(
S; m; E; p
) is constructed by connecting
each cell of
S
with its
p
nearest neighbors in
E
by excitatory synapses. I will
refer to
~
S
simply as a
topological network.
A topological network embedded in ordinary space is a very common idea (cf.
e.g. [Beurle, 1955]). The point made here is that there is no need for a network
to stay with this natural embedding. This has the important consequence that
there is a huge number of topological networks on the same set
S
,even if
E
,
p
and the set of assigned points in
E
are kept xed. Namely, instead of
m
one
can consider
Pm
, with any permutation
P
of the assigned points in
E
. See g.
1 for an example.
3.3.2 The Correlate of a Top ological Network
Before one can deduce the dynamical behaviour of the network activitymuch
more detail has to be specied. However, we are here only interested in certain
aspects of the dynamics and a few assumptions suce.
Let synaptic weights be constant for the moment. Since an activated excita-
tory synapse between two cells creates correlations in their signals, \neighbors"
in a topological network are correlated. The topological aspect is important
since on the one hand the topological network is cooperative: each synapse is
helped by others in establishing a correlation (e.g. 1-3 and 1-2-3 in g. 1a
cooperate); on the other hand, the network can still be decomposed into many
weakly coupled subnetworks, in contrast to globally cross-connected networks.
Two kinds of signal pattern can exist in a topological network. In one there
are waves running through the network (see e.g. [Beurle, 1955], for an example
see g. 1). The diuse network of inhibition keeps the cells from ring in global
simultaneity. The other kind of pattern stresses the analogy to the spin-lattice
of a ferromagnet [Little, 1974]: a cell randomly switches between an active and
a silent state. In doing so it is inuenced by its \neighbors". If a ma jority
15
of neighbors is active, the cell will also be more likely to re, if a majorityof
neighbors is silent, the cell will more likely be silent. The strength of coupling
between the b ehaviour of the cell and its environment can be characterized by
a parameter
T
, analogous to the temperature of the spin lattice. For
T
= 0 the
coupling is rigid: all cells in the network switch up and down simultaneously. For
innite
T
there is no coupling and cells are independent. We are interested here
in intermediate values of
T
, for which a cell is correlated with its neighbors and
this correlation decreases over a characteristic \distance" which is a fraction of
the \diameter" of the network. With either kind of cellular activity the structure
of the network is expressed in the signals by correlations. Such a signal structure
will be called a
correlate.
Now allow the synaptic activities
f
a
ij
g
to vary. Consider a set
C
of cells
which are excited by input activity. Supp ose
C
is part of several sets of cells
S
0
;S
00
;:::
which are internally connected by topological networks
~
S
0
;
~
S
00
;:::
. If
these topologies are independent and all synapses are in the resting state,
C
is globally coupled in a non-topological fashion. The connectivityin
C
is then
probably unstable. A stable state can be reached after one of the topological
networks, say
~
S
0
, has been activated and the others have been inactivated. In
order for this to happen, the complementof
C
in
S
0
has to be invaded, to ll
the holes left by
C
in the topology of
~
S
0
. After the network with the topology
of
~
S
0
has been activated, activity can no longer invade the rest of the other sets
S
00
;:::
, because the
p
-environments of their cells, even if they are activein
S
0
,
never re synchronously.
Correlate reconstruction is the fundamental process of correlation theory. It
must take place on the fast time scale of thought processes. Its synergetics is a
complicated matter and needs further detailed work. An important special case
is discussed in the next paragraph.
3.3.3 Projection between Topological Networks
Consider two structurally identical networks
~
S
1
and
~
S
2
on disjoint sets
S
1
and
S
2
of
n
cells each. The two sets are connected to each other by a one-to-one
projection
R
of activated synapses connecting cells in corresponding position,
so that
R
corresponds to an isomorphism. This denes on
S
1
[
S
2
again a
topological structure which can carry a correlate, with correlations at short
distance in
~
S
1
and
~
S
2
, and between cells in
S
1
and
S
2
which correspond to
each other according to
R
. This special kind of top ological correlate can be
approached from dierent starting congurations, as will be discussed now.
3.3.3.1
Consider rst the case with
R
in the resting state and correlates corre-
sponding to
~
S
1
and
~
S
2
activein
S
1
and
S
2
but not mutually correlated.
R
will
haveaweak synchronizing inuence on pairs of corresponding cells in
~
S
1
and
~
S
2
. The so induced correlations will activate the synapses of
R
, and strengthen
the
S
1
{
S
2
correlations, until the stationary state is reached with fully acti-
vated
R
and the activity strongly correlated between
S
1
and
S
2
. On the other
16
hand, if on
S
2
a network with a considerably dierent topological structure were
activated,
R
would be deactivated.
The case is very reminiscent of the basic two-cells-one-synapse situation:
correlation in (internal structure of) the correlates on
S
1 and
S
2
leads to
R
-
activation, lack of correlation to deactivation. In this sense
S
1
and
S
2
can be
regarded as composite analoga to single cells.
3.3.3.2
Let
R
be a system of synapses connecting each cell of
S
1
with each
cell of
S
2
. Let
~
S
1
and
~
S
2
be isomorphic topological networks on
S
1
and
S
2
.
The synapses of
R
initially are in their resting state. A very similar system,
referring to an ontogenetic problem, was simulated in [Willshaw and von der
Malsburg, 1976] with two-dimensional
E
, and was treated analytically for one-
dimensional
E
in [Haussler and v.d. Malsburg, 1983]. There, it was shown that
R
can dynamically reduce to a one-to-one projection b etween the isomorphically
corresponding cells in
~
S
1
and
~
S
2
. The system is able to sp ontaneously break
the symmetry between several possible projections.
3.3.3.3
Several top ological networks
~
S
2
,
~
S
0
2
,
~
S
00
2
;:::
may exist in
S
2
(in addition
to
~
S
1
and
R
). Before a topological correlate can be established on
S
1
[
S
2
, several
decisions have to be made: between
~
S
2
,
~
S
0
2
,
~
S
00
2
;:::
and between possible one-to-
one mappings corresponding to one of the structures on
S
2
. These decisions have
to be made simultaneously. This is likely to cause chaos instead of a specic
correlate. However, if symmetries between the various structures are slightly
broken already in the inital state, an ordered stationary state may be reached
securely, as is made likely by extrapolation from experience with a case similar
to 3.3.3.2.
3.3.4 Composite Elements
In section 3.1 I have introduced the basic machinery of correlation theory in
terms of cells, correlations of their signals, synapses and their modulation. The
discussion of 3.3.3 has prepared the way to the use of a very similar language on
a higher level. The idea consists in considering sets of topologically connected
cells instead of single cells as network elements. The sets may then be termed
composite elements.
Likewise the ensemble of cellular signals of a set maybe
regarded as a
composite signal
, and the ensemble of bres connecting two com-
posite elements as a
composite connection
. The correlation between two cellular
signals was dened in terms of synchrony and asynchronybetween spike trains.
Correlation between the signals of two composite elements has to be dened
as a structural corresp ondence between the composite signals in terms of the
composite connection between the elements. Each single synapse between two
composite elements should be modulated by a globally evaluated correlation be-
tween the composite signals. This is made p ossible by the fact that a temporal
correlation in the signals locally available to the synapse can only be established
in the context of a global correlation between the elements, as was discussed in
3.3.3.1.
17
Composite elements can again form networks:
S
i
,
S
j
;:::
, with composite
connections
R
ij
. For a correlate b etween the composite elements to form it is
necessary that the dierent composite connections be locally consistent with
each other. Introduce an arbitrary but xed numbering of cells in each element.
A one-to-one projection
R
ij
is then equivalenttoapermutation matrix
P
ij
in
which each non-zero element corresponds to a synapse. In a triplett of elements
S
i
,
S
j
,
S
k
the permutation
P
ik
must be the same as
P
ij
P
jk
in order to be
consistent. Stated dierently, the composite permutation matrix corresponding
to a closed chain of connections must be unity:
P
ij
P
jk
P
ki
=1. (The condition
can be relaxed for chains of elements which are longer than the correlation
length. This opens the door to the whole complexity and richness of non-trivial
ber bundle or gauge eld structures.) Also on this new level the dynamical
interactions between signals and synapses stabilize certain preferred connectivity
patterns and correlations, and again it may be conjectured that they have a
topological structure.
In applications it may be necessary to introduce super-comp osite elements.
Paragraph 3.6.4 will give an example. The elaboration of particular structures
is, however, a complex dynamical and mathematical problem.
3.3.5 The Synergetic Control of Action
How can the dynamical behaviour of the brain's network structure be charac-
terized globally? Suppose the state of the brain at time
t
could b e understood
as a superposition of structures, termed modes, with the following properties:
A mode is a subnetwork of active cells and synapses which, if left to itself,
would reproduce its form, possibly change its amplitude. (Decomposition into
modes has been rigorously carried out in a neuronal system in [Haussler and v.d.
Malsburg, 1983].) To predict the state of the brain at time
t
+
t
, decompose
its state at
t
into modes, let each of them growordecay for the interval
t
,
and superpose the results again. With the help of a global control parameter it
often can be achieved that only one or a few modes grow and all others decay.
It is conceivable that such global control exists in the brain. If only one mode
grows it soon dominates the state of the system. If several modes are related by
a symmetry they grow or decay with the same speed. This is the reason why
symmetry breaking, i.e. the selection of one of the related modes, is dicult to
achieve.
The distinguishing feature which allows a mode to grow fast is maximal
local self-amplication and optimal cooperation of locally converging dynamical
inuences, e.g. correlation between signals converging on one cell.
If growth of a mode is suciently slow there is time for the exchange of
signals between all parts of the network. All locally available information is
then integrated into the one global decision - growth or decay. After a mode
has grown and established itself for some time, conditions may cease to be
favourable for it, either because the mo de has prepared the way for a successor
18
mode which starts to compete successfully, or because the environment has
changed, or simply because of some kind of fatigue. Thus the brain is ruled by
a succession of modes. This view emphasizes the analogy to many other self-
organizing systems [Haken, 1978], and would put the brain into sharp contrast
to the computer and other man-made systems with detailed central control.
Memory may be thought of as the ability of the brain to change its network so
as to improve the success of modes whichwere once active. In the extreme case
an entire global mode which once dominated the brain's state for a short moment
can be reestablished. Aphysical basis for this ability is synaptic plasticity, which
reinforces those networks which are strongly activated.
3.4 Applications of Correlation Theory
3.4.1 Visual Elements
Light stimulation of one retinal point can directly aect several thousand neu-
rons in visual cortex. Together they form a composite elementoflowest level,
a
primary visual element
. Each neuron is specically sensitive to a particular
combination of qualityvalues characterizing the stimulus: level of brightness or
darkness, spectral distribution, spatial distribution of light within a receptive
eld, stereo depth, direction and sp eed of movement. Visual cortex contains
multiple representations of the retina. These are interconnected by diusely
retinotopic ber pro jections. Primary visual elements may be composed of cells
in several visual areae and even in thalamus. The part of the brain formed by
primary visual elements will here be termed
V
.
Consider a particular visual element while the eyes are slowly scanning over
a scene. When a light edge crosses the receptive eld of the element, a subset of
cells is activated simultaneously. The subset describes the quality of the edge of
light. This simultaneous excitation triggers activation of synapses and forma-
tion of a correlate within the active subset of the element under consideration.
A subnetwork results which now represents a composite feature detector. Its
signal expresses a composite quality which can be recognized even from mix-
tures of signals from dierent visual elements. Confusion is excluded by signal
correlations within a set of bres coming from one primary visual element.
Visual elements have been introduced here as those collections of cells which
are aected from one retinal point. One could possibly also consider somewhat
larger patches of cortex (and thalamus) as elements. Those larger elements
would then be capable of forming correlates corresponding to patches of visual
texture. There is no need for the brain's \hardware" to contain complex feature
detector cells. Only cells responding to rather simple stimuli are required, from
which complex composite feature detectors can be \manufactured on the spot"
by activation of synaptic networks.
19
3.4.2 Figure-Ground Discrimination
Suppose all visual elements in the primary region
V
are integrated byaber
system which connects feature sensitive cells in one element with cells specic
for the same local quality in many other elements. Suppose two elements so
interconnected are stimulated by a similar comp osite quality and correlates have
formed in both of them, so that the situation described in 3.3.3.1 is given. In due
course the connection between the elements will b e activated and the composite
signals of the elements will correlate with each other. On the other hand, if the
two elements were stimulated by radically dierent composite qualities, mutual
synapses would be deactivated and the signals would decouple.
Suppose a visual scene contains a region
F
characterized by local qualities
whichchange continuously from pointtopoint inside
F
and whichchange dis-
continuously across the boundary of
F
. (A prominent role among these qualities
will be played by dierential velocity caused by p ersp ectivemovement or object
displacement.) The mechanism just described will set up a network of activated
synapses and a correlate which lls the region of primary visual cortex excited
by the image of
F
. All elements taking part in it will signal this fact bymu-
tual local correlations. There will be no correlations across the boundary of the
network.
In this way the scene is decomposed into a patchwork of gures. Moreover,
a gure is decomposed into a hierarchy of parts, the strongest correlations sig-
nalling aliation to one part of the gure, weaker ones aliation to adjacent
parts, and so on. This decomp osition of the visual scene into a hierarchy of
correlates starts already prior to recognition of patterns, a stage of the process
whichwas termed \preattentive vision" by B. Julesz [1981].
3.4.3 Invariant Image Representation
The correlation structure described in the last paragraph has to be built up
anew for each image xation. Another part of the visual system, to which I
will refer as
I
, can accumulate visual information over longer perio ds of time
and independently of retinal image location. A physical prerequisite for this
is a ber system which connects each element of
V
with each element of
I
.
(This strong assumption can later be relaxed considerably.) If all these b ers
were activated at the same time a great jumble of signals would converge on
the elements in
I
. It is, however, possible to deactivate most connections and
activate only topologically ordered one-to-one projections b etween
V
and
I
.
I assume that the elements in
V
and in
I
are tied together by topological net-
works
N
V
and
N
I
, respectively. (This is a statement about p ermanentweights.)
The topology is the natural one of two-dimensional visual space. Consider for
simplicity a situation in
V
with just two active correlates
F
and
G
.
F
refers to
gure and
G
to ground. Correlations in both
F
and
G
are top ologically struc-
tured by activated subnetworks of
N
V
. The components of
N
V
connecting
F
20
with
G
are deactivated. Initially there may only be spontaneous activityin
I
,
the correlations of which are topologically structured by
N
I
. Connections from
F
and
G
which converge on one element of
I
carry noncorrelated signals and
cannot cooperate to cause excitation, correlation or synaptic activation. If one
considers
I
and just the
F
-part of
V
as two super-elements, the situation is that
of 3.3.3.2. As was pointed out there, a stationary state will be reached in which
a one-to-one projection is activated which connects neighboring elements in
F
to neighboring elements in
I
. If symmetries are not broken by other inuences,
the scale and orientation of the
F
!
I
projection will be such that the image
of
F
ts best into
I
. At the same time the correlate structure of the intra-
and inter-element networks in
F
is transferred to the corresp onding elements in
I
. (This is analogous to the transfer of retinal markers to the tectum in [v.d.
Malsburg and Willshaw, 1977].)
The simulations of [Willshaw and von der Malsburg, 1979] have shown that
the simultaneous presence of two independent correlates, like
F
and
G
, can lead
to a nal state with two independent mappings of the kind described, one for
F
and one for
G
. The network
I
can then tune its correlate to
F
or
G
.
New mappings between
V
and
I
have to be set up for each new image
xation. This is enormously facilitated by relevant structure in
I
built up during
previous xations. Relative image distortions between xations are digested by
distortions in the pro jections which are established. Over the course of many
xations more and more information about a gure, although arriving through
dierent parts of the retinae, can be gradually accumulated in
I
.
After a mapping between
V
and
I
has been activated, information can be
transferred from
I
backto
V
. The aerent information can thus be scrutinized
by the retrograde activation of composite feature correlates.
In distinction to the perceptron approach to the invariance problem, the
geometrical structure of the gure is explicitly represented in
I
. There is no
need to recover it from the distribution of active feature detectors (cf. 2.3.5).
3.4.4 Interpretation of an Image
Before an image can be recognized it must be broughtinto contact with ideas of
shapes and ob jects stored in memory. Let us invoke a part
M
of the brain. To
a neurophysiologist
M
would appear similar to
I
. However, it would be domi-
nated by specic connection patterns whichhave been layed down previously by
synaptic plasticity and which correspond to abstract schemata of ob jects. These
can be revived by resonance with structures in
I
to carry correlates. Recognition
between structures in
I
and in
M
is possible on the basis of a correspondence of
detailed network structure, which in turn is expressed in terms of correlations in
signals. The situation was discussed in 3.3.3.3. Several relevant memory traces
may be activated simultaneously or consecutively.
The representation of an object in
I
has to be fairly insensitive to image
size, position, orientation and (slight) distortion. It therefore lacks information
21
about these parameters and it is necessary for structures such as
M
to have
access to the primary image in
V
. This is possible with the help of full direct
ber systems connecting
M
with
V
. These can be easily structured during
a xation because all elements of an image in
V
are functionally labelled by
correlations with the corresp onding elements in
I
. The original image can be
scrutinized by the selective set-up of part-networks referring to parts of it. A
full interpretation of an image is constituted by a correlate in a sup er-network
composed of many super-elements such as
V
,
I
and
M
, partly b elonging to
other modalities of the brain.
Memory in its direct form precisely reestablishes correlates whichwere previ-
ously active. The observed great exibility of memory traces could be explained
if memory in this extreme form were restricted to certain substructures of the
brain, like the
M
mentioned above. For instance, we know that the memory
trace corresponding to a human face leaves unspecied all accidental aspects,
e.g. persp ective, illumination and expression. The trace has to be comple-
mented by particular correlates in other areae, like
V
and
I
, before it can be
compared with a real image. This exibility cannot b e accounted for with cell
assemblies which cannot b e analyzed into parts.
4 Discussion
4.1 The Text Analogy
The relationship between correlation theory and conventional brain theory may
be claried with the analogy to the way our cultural world makes use of sym-
bols. When we write text we employ a set of basic symb ols, letters or ideograms.
Out of these we form higher symbols, words, sentences, paragraphs. Wedoso
by<