Content uploaded by Paulin Melatagia
Author content
All content in this area was uploaded by Paulin Melatagia on Oct 16, 2014
Content may be subject to copyright.
Available at: http://www.ictp.it/~pub−off IC/2005/111
United Nations Educational Scientific and Cultural Organization
and
International Atomic Energy Agency
THE ABDUS SALAM INTERNATIONAL CENTRE FOR THEORETICAL PHYSICS
DYNAMICAL BEHAVIOUR OF NEURAL NETWORKS
ITERATED WITH MEMORY
Paulin Yonta Melatagia1, Ren´e Ndoundam2
Department of Computer Science, University of Yaounde I,
P.O. Box 812, Yaounde, Cameroon
and
The Abdus Salam International Centre for Theoretical Physics, Trieste, Italy
and
Maurice Tchuent´e3
Department of Computer Science, University of Yaounde I,
P.O. Box 812, Yaounde, Cameroon.
Abstract
We study memory iteration where the updating consider a longer history of each site and the
set of interaction matrices is palindromic. We analyze two different ways of updating the net-
works: parallel iteration with memory and sequential iteration with memory that we introduce
in this paper. For parallel iteration, we define Lyapunov functional which permits us to char-
acterize the periods behaviour and explicitely bounds the transient lengths of neural networks
iterated with memory. For sequential iteration, we use an algebraic invariant to characterize the
periods behaviour of the studied model of neural computation.
MIRAMARE – TRIESTE
November 2005
1pmelatag@uycdc.uninet.cm
2Regular Associate of ICTP. ndoundam@uycdc.uninet.cm
3tchuente@uycdc.uninet.cm
Introduction
Caianiello [1] has suggested that the dynamic behaviour of a neuron in a neural network with
k-memory can be modeled by the following recurrence equation:
xi(t) = 1
n
X
j=1
k
X
s=1
aij (s)xj(t−s)−bi
, t ≥k(1)
where
•iis the index of a neuron, i= 1, ..., n.
•xi(t)∈ {0,1}is a variable representing the state of the neuron iat time t.
•kis the memory length, i.e., the state of a neuron iat time tdepends on the states
xj(t−1) , ..., xj(t−k) assumed by all the neurons (j= 1, ..., n) at the previous steps
t−1, ..., t −k(k≥1).
•aij (s) (1 ≤i, j ≤nand 1 ≤s≤k) are real numbers called the weighting coefficients.
More precisely, aij (s) represents the influence of the state of the neuron jat time t−son
the state assumed by the neuron iat time t.
•biis a real number called the threshold.
•1is the Heaviside function: 1(u) = 0 if u < 0, and 1(u) = 1 if u≥0
Neural networks were introduced by Mc Culloch and Pitts [15], and are being investigated in
many fields of artificial intelligence as a computational paradigm alternative to the conventional
Von Neumann model. Neural networks are able to simulate any sequential machine or Turing
machine if an infinite number of cells is provided. Recently, neural networks have been studied
extensively as tools for solving various problems such as classification, speech recognition, and
image processing [6]. The field of appplication of threshold functions is large [10, 13, 6].
Neural network is usually implemented by using electronic components or is simulated in
software on a digital computer. One way in which the collective properties of a neural network
may be used to implement a computational task is by way of the concept of energy minimization.
The Hopfield network has attracted a great deal of attention in the literature as a content-
addressable memory [12].
Since neural network models have also been inspired by neurophysiological knowledge, the
theoretical results may help to broaden understanding of the computational principles of mental
processes.
The dynamics generated by Eq. (1) have been studied for some particular one-dimensional
systems:
•when k= 1, one obtains a Mc Culloch and Pitts neural network [15]. Some results about
the dynamical behaviour of these networks can be found in [15, 13, 14].
•when n= 1, one obtains a single neuron (proposed by Caianiello and De Luca [2]) with
memory that does not interact with other neurons. See [2, 3, 4, 17] for some results.
There are few results in the study of evolution induced by Eq. (1). In [8] Goles established the
following result:
Theorem 1 [8]. If the class of interaction matrices (A(s) : s= 1, ..., k)is palindromic the pe-
riods Tof parallel iteration with memory satisfies T|k+ 1.
2
In [18] Tchuent´e generalized the preceding result by showing that the parallel iteration of a
network of automata Ncan be sequentially simulated by another network N′whose local tran-
sition functions are the same as those of N. By implementing a binary Borrow-Save counter,
Ndoundam and Tchuent´e show that:
Theorem 2 [16].There exist a Caianiello automata network of size 2n+ 2 and memory length
kwhich describes a cycle of length k2nk.
In this work, we show some dynamical results for parallel iteration with memory of neural
network (Eq. (1)) when non-trivial regularities on coupling coefficients are satisfied. We also
define the sequential iteration with memory of neural network and characterize its periodic
behaviour. Our approach consists in defining appropriate Lyapunov functional [13] or algebraic
invariant [11] .
The remainder of the paper is organized as follows: in Section 1, some basic definitions and
preliminary results are presented. In Section 2 we define two Lyapunov functional for parallel
iteration with memory of neural network, characterize its periodic behaviour and bound its
transient length. We also compare this bound with another obtained using sequential simulation
of parallel iteration. In Section 3, we introduce sequential iteration with memory of neural
network and study its periodic behaviour using an algebraic invariant.
1 Definitions and preliminary results
A neural network Niterated with a memory of length kis defined by N= (I , A (1) , ..., A (k), b),
where I={1, ..., n}is the set of neurons indexes, A(1) , ..., A (k) are matrices of interactions
and b= (bi:i∈ {1, ..., n}) is the threshold vector. Let {x(t)∈ {0,1}n:t≥0}be the trajectory
starting from x(0),..., x(k−1); since {0,1}nis finite, this trajectory must sooner or later
encounter a state that occurred previously - it has entered an attractor cycle. The trajectory
leading to the attractor is a transient. The period (T) of the attractor is the number of states in
its cycle, which may be just one - a fixed point. The transient length of the trajectory is noted
τ(x(0) , ..., x (k−1)). The transient length of the neural network is defines as the greatest of
transient lengths of trajectories, that is:
τ(A(1) , ..., A (k), b) = max {τ(x(0) , ..., x (k−1)) : x(t)∈ {0,1}n,0≤t≤k−1}
The period and the transient length of sequences generated are good measures of the complexity
of the neural network.
The updates of the state values of each neuron depends on the type of iteration associated
to the model. The sequential iteration consists of one by one updating the neurons in a pre-
established periodic order (i1, i2, ..., in), where I={i1, i2, ..., in}. The parallel iteration consists
of updating the value of all the neurons at the same time.
Let us now recall some tools introduced by Goles et al.. Let X= (x(0) , ..., x (T−1)) a
T-cycle of the network, the T-cycle at site iis denoted Xi= (xi(0) , ..., xi(T−1)). The period
of Xiis denoted τ(Xi). Let k, l ∈Z; the quantity ∆Vk,l (Xi, Xj) for i, j ∈Iwhich is a difference
of covariances among cycles Xi, Xj, is defined as:
∆Vk,l (Xi, Xj) = 1
TX
t∈ZT
(xi(t)xj(t+k)−xj(t)xi(t+l)) (2)
which can also be written:
∆Vk,l (Xi, Xj) = 1
TX
t∈ZT
xi(t) (xj(t+k)−xj(t−l)) (3)
where ZT=Z/mod T (the cyclic group of integers mod T ).
3
Remark 1 ∆Vk,l (Xi, Xi) = ∆Vk,−l(Xi, Xi) for i∈I.
Remark 2 ∆Vk,l (Xi, Xj) = −∆Vl,k (Xj, Xi) for i, j ∈I.
It was shown that:
Lemma 1 [11]. If τ(Xi)|k+lthen ∆Vk,l (Xi, Xj) = 0 for any j∈I.
Let Γ0(Xi) = {t∈ZT:xi(t) = 0}and Γ1(Xi) = {t∈ZT:xi(t) = 1}a partition of ZT. The
period of the sequence Xiis equal to the period of Γ1(Xi). A subset C⊂Γ1(Xi) is called
ak-chain (for k≥1) iff it is on the form C={t+kl ∈ZT: 0 ≤l < s}for some s≥1.
Let C+l={t+l(mod T ) : t∈C}, l ∈Z. A k-chain is said to be maximal if it is not strictly
contained in another k-chain. ζ(Xi) is the class of maximal k-chains of the sequence Xi.ζk(Xi)
is the class of k-periodic maximal k-chains, i.e. ζk(Xi) = {C∈ζ(Xi) : C=C+k}. It was
established that:
Lemma 2 [11]. τ(Xi)|kiff ζ(Xi) = ζk(Xi).
Call scthe cardinality of C∈ζ(Xi). If C∈ζk(Xi) choose anyone of its elements and call it tc.
Otherwise, take tcsuch that tc−k(mod T )/∈C. Then
C={tc+lk (mod T ) : 0 ≤l < sc}.
For any C∈ζ(Xi) write ¯
tc=tc+ (sc−1) k(mod T ) which is an element of C. Using maximal
(k+l)-chains, we have [11]:
∆Vk,l (Xi, Xj) = 1
TX
C∈ζ(Xi)\ζk+l(Xi) X
t∈C
(xj(t+k)−xj(t−l))!(4)
and by simplification:
∆Vk,l (Xi, Xj) = 1
TX
C∈ζ(Xi)\ζk+l(Xi)
(xj(¯
tc+k)−xj(tc−l)) (5)
For more details about these mathematical tools, the reader must refer to [10].
2 Parallel iteration with memory
Let us consider the parallel iteration with memory of a finite neural network N= (I, A (1) , ..., A (s), b)
give by Eq. (1) ; we can assume that:
n
X
j=1
k
X
s=1
aij (s)uj6=bi,∀i∈I, ∀u= (u1, ..., un)∈ {0,1}n(6)
We also assume that the set of interaction matrices (A(s) : s= 1, ..., k) satisfiy the palindromic
condition:
A(k+ 1 −s) = A(s)tfor s = 1, ..., k (7)
i.e.:
aij (k+ 1 −s) = aji (s)∀i, j ∈ {1, ..., n},∀s∈ {1, ..., k}(8)
Let {x(t) : t≥0}be a trajectory of the parallel iteration, we define the following functional
for t≥k:
E(x(t)) = −
n
X
i=1
k−1
X
s=0
xi(t−s)
n
X
j=1
k−s
X
s′=1
aij (s′)xj(t−s−s′)
−bi
k
X
s=0
xi(t−s)
(9)
4
Proposition 1 If the class of interaction matrices (A(s) : s= 1, ..., k)is palindromic, then the
functional E(x(t)) is a strictly decreasing Lyapunov functional for the parallel iteration with
memory of neural network.
Proof . Let ∆tE=E(x(t))−E(x(t−1)). As the set of interaction matrices (A(s) : s= 1, ..., k)
satisfiy Eq. (7), we have:
∆tE=
n
P
i=1 n
P
j=1 k−1
P
s=0
xi(t−1−s)
k−s
P
s′=1
aij (s′)xj(t−1−s−s′)−bi
k
P
s=0
xi(t−1−s)!
−
n
P
i=1 n
P
j=1 k−1
P
s=0
xi(t−s)
k−s
P
s′=1
aij (s′)xj(t−s−s′)−bi
k
P
s=0
xi(t−s)!
=
n
P
i=1 n
P
j=1 k
P
s=1
xi(t−s)
k−s+1
P
s′=1
aij (s′)xj(t−s−s′)−bi
k+1
P
s=1
xi(t−s)!
−
n
P
i=1 n
P
j=1 k−1
P
s=0
xi(t−s)
k−s
P
s′=1
aij (s′)xj(t−s−s′)−bi
k
P
s=0
xi(t−s)!
=
n
P
i=1 n
P
j=1 k−1
P
s=1
xi(t−s)k−s+1
P
s′=1
aij (s′)xj(t−s−s′)−
k−s
P
s′=1
aij (s′)xj(t−s−s′)
+xi(t−k)aij (1) xj(t−k−1) −xi(t)
k
P
s′=1
aij (s′)xj(t−s′)+bi(xi(t)−xi(t−k−1))
=
n
P
i=1 n
P
j=1 k−1
P
s=1
xi(t−s)aij (k−s+ 1) xj(t−k−1) + xi(t−k)aij (1) xj(t−k−1)
−xi(t)
k
P
s=1
aij (s)xj(t−s)+bi(xi(t)−xi(t−k−1))
=
n
P
i=1 n
P
j=1 k
P
s=1
xi(t−s)aij (k−s+ 1) xj(t−k−1) −xi(t)
k
P
s=1
aij (s)xj(t−s)!
+
n
P
i=1
(bi(xi(t)−xi(t−k−1)))
=
n
P
i=1 n
P
j=1 k
P
s=1
xj(t−s)aji (k−s+ 1) xi(t−k−1) −xi(t)
k
P
s=1
aij (s)xj(t−s)!
+
n
P
i=1
(bi(xi(t)−xi(t−k−1)))
=
n
P
i=1 n
P
j=1 xi(t−k−1)
k
P
s=1
xj(t−s)aij (s)−xi(t)
k
P
s=1
aij (s)xj(t−s)!
+
n
P
i=1
(bi(xi(t)−xi(t−k−1))) (by using Eq. (8))
=−
n
P
i=1 (xi(t)−xi(t−k−1)) n
P
j=1
k
P
s=1
aij (s)xj(t−s)−bi!!
From the definition of parallel iteration with memory, for any i∈Iwe find:
−(xi(t)−xi(t−k−1))
n
X
j=1
k
X
s=1
aij (s)xj(t−s)−bi
≤0.
5
So we conclude ∆tE≤0 and ∆tE < 0 iff xi(t)6=xi(t−k−1).
We now give another proof of the Theorem 1 using the preceding Lyapunov functional.
Proof of Theorem 1. Let X= (x(0) , ..., x (T−1)) a cycle of period T. From the proof
of Proposition 1 we found that, E(x(0)) = ... =E(x(T−1)) iff ∀t= 0, ..., T −1, xi(t) =
xi(t+k+ 1) for all i= 1, ..., n, which implies that τ(Xi)|k+ 1 ∀i∈I. Then T|k+ 1.
To study the transient phase, we will work with another Lyapunov functional derived from
E(x(t)). Define:
E∗(x(t)) = −
n
P
i=1 k−1
P
s=0
(2xi(t−s)−1) n
P
j=1
k−s
P
s′=1
aij (s′) (2xj(t−s−s′)−1)!!
+
n
P
i=1 2bi−
n
P
j=1
k
P
s=1
aij (s)!k
P
s=0
(2xi(t−s)−1)!(10)
Proposition 2 If the class of interaction matrices (A(s) : s= 1, ..., k)is palindromic, then the
functional E∗(x(t)) is a strictly decreasing Lyapunov functional for the parallel iteration with
memory of neural network .
Proof . As the set of interaction matrices (A(s) : s= 1, ..., k) satisfiy Eq. (7), we have:
∆tE∗=E∗(x(t)) −E∗(x(t−1))
=−
n
P
i=1 (2xi(t)−2xi(t−k−1)) n
P
j=1
k
P
s=1
aij (s) (2xj(t−s)−1) − 2bi−
n
P
j=1
k
P
s=1
aij (s)!!!
=−
n
P
i=1 2 (xi(t)−xi(t−k−1)) 2
n
P
j=1
k
P
s=1
aij (s)xj(t−s)−
n
P
j=1
k
P
s=1
aij (s)−2bi+
n
P
j=1
k
P
s=1
aij (s)!!
=−4
n
P
i=1 (xi(t)−xi(t−k−1)) n
P
j=1
k
P
s=1
aij (s)xj(t−s)−bi!!
Then ∆tE∗= 4∆tE. Proposition 1 allows us to conclude the result.
Denotes by ¯
Xthe set of all initial conditions which do not belong to a period of length k+ 1:
¯
X={x(0) ∈ {0,1}nsuch that x(0) 6=x(k+ 1)}
Recall that ¯
Xis empty iff the transient length of the neural network is null. If ¯
X6=∅define:
e=min −(E(x(k+ 1)) −E(x(k))) : x(0) ∈¯
X(11)
We note e= 0 if ¯
X= 0.
Proposition 3 Let {x(t) : t≥0}be a trajectory; E∗(x(t)) is bounded by:
E∗(x(t)) ≥ − (k+ 1 )
2b−
k
X
s=1
A(s).¯
1
1
−
k
X
s=1
(k−s+ 1) kA(s)k(12)
and
E∗(x(t)) ≤
2b−
k
X
s=1
A(s).¯
1
1
−2k
n
X
i=1
ei+
k
X
s=2
(s−1) kA(s)k(13)
6
where:
ei=min
n
X
j=1
k
X
s=1
aij (s)uj(s)−bi
:u(s)∈ {0,1}n, s = 1, ..., k
(14)
and kA(s)k=
n
P
i=1
n
P
j=1
|aij (s)|,kuk1=
n
P
i=1
|ui|for any vector u∈Rn,¯
1 = (1, ..., 1)tis the
1-constant vector.
Proof . Consider the i-th term in Eq. (10), i.e.:
(E∗(x(t)))i=−
k−1
P
s=0
(2xi(t−s)−1) n
P
j=1
k−s
P
s′=1
aij (s′) (2xj(t−s−s′)−1)!
+ 2bi−
n
P
j=1
k
P
s=1
aij (s)!k
P
s=0
(2xi(t−s)−1)
The following lower bound is obtained:
(E∗(x(t)))i≥ −
n
X
j=1
k−1
X
s=0
k−s
X
s′=1
|aij (s′)| − (k+ 1)
2bi−
n
X
j=1
k
X
s=1
aij (s)
Since
k−1
P
s=0
k−s
P
s′=1
|aij (s′)|=
k
P
s=1
(k−s+ 1) |aij (s)|, we find:
E∗(x(t)) ≥ −
k
X
s=1
(k−s+ 1) kA(s)k − (k+ 1)
2b−
k
X
s=1
A(s).¯
1
1
One can write (E∗(x(t)))ias follows:
(E∗(x(t)))i=−
k−1
P
s=0
(2xi(t−s)−1) 2
n
P
j=1
k−s
P
s′=1
aij (s′)xj(t−s−s′)−
n
P
j=1
k−s
P
s′=1
aij (s′)
−2
n
P
j=1
k
P
s′=k−s+1
aij (s′)xj(t−s−s′) + 2
n
P
j=1
k
P
s′=k−s+1
aij (s′)xj(t−s−s′)
−2bi+
n
P
j=1
k
P
s′=1
aij (s′)!+ 2bi−
n
P
j=1
k
P
s=1
aij (s)!(2xi(t−k)−1)
=−
k−1
P
s=0
(2xi(t−s)−1) 2
n
P
j=1
k
P
s′=1
aij (s′)xj(t−s−s′) +
n
P
j=1
k
P
s′=k−s+1
aij (s′)−2bi
−2
n
P
j=1
k
P
s′=k−s+1
aij (s′)xj(t−s−s′)!+ 2bi−
n
P
j=1
k
P
s=1
aij (s)!(2xi(t−k)−1)
=−2
k−1
P
s=0
(2xi(t−s)−1) n
P
j=1
k
P
s′=1
aij (s′)xj(t−s−s′)−bi!
+
k−1
P
s=0
(2xi(t−s)−1) n
P
j=1
k
P
s′=k−s+1
aij (s′) (2xj(t−s−s′)−1)!
+ 2bi−
n
P
j=1
k
P
s=1
aij (s)!(2xi(t−k)−1)
The following upper bound is obtained:
(E∗(x(t)))i≤ −2kei+
n
X
j=1
k−1
X
s=0
k
X
s′=k−s+1
|aij (s′)|+
2bi−
n
X
j=1
k
X
s=1
aij (s)
7
Since
n
P
j=1
k−1
P
s=0
k
P
s′=k−s+1
|aij (s)|=
n
P
j=1
k
P
s=1
(s−1) |aij (s)|, we find:
E∗(x(t)) ≤ −2k
n
X
i=1
ei+
k
X
s=2
(s−1) kA(s)k+
2b−
k
X
s=1
A(s)¯
1
1
Theorem 3 If the class of interaction matrices (A(s) : s= 1, ..., k)is palindromic, then the
transient length τ(A(1) , ..., A (k), b)of parallel iteration with memory of neural network is
bounded by:
τ(A(1) , ..., A (k), b)≤1
4e(k+ 2)
2b−
k
P
s=1
A(s)¯
1
1
+k
k
P
s=1
kA(s)k − 2k
n
P
i=1
eiif e > 0
τ(A(1) , ..., A (k), b) = 0 if e = 0
(15)
Proof . Let {x(t) : t≥0}a trajectory with τ(x(0) , ..., x (k−1)) = τ(A(1) , ..., A (k), b) and
denote t0=τ(A(1) , ..., A (k), b) + k. Since x(0) ∈¯
X,x(0) 6=x(k+ 1); then from Proposition
1, −(E∗(x(k+ 1)) −E∗(x(k))) >0. We deduce that, τ(A(1) , ..., A (k), b) = 0 iff e= 0.
Assume e > 0. For any 0 ≤t≤t0−k−1 we have x(t)∈¯
X. By definition of ewe deduce:
E∗(x(t)) ≤E∗(x(t−1)) −4e for any k + 1 ≤t≤τ(A(1) , ..., A (k), b) + k
Then E∗(x(t0)) ≤E∗(x(k)) −4e(t0−k). From Proposition 3 we find:
−
k
X
s=1
(k−s+ 1) kA(s)k − (k+ 1)
2b−
k
X
s=1
A(s).¯
1
1
≤E∗(x(t0)) ≤E∗(x(k)) −4e(t0−k)
≤ −2k
n
X
i=1
ei+
k
X
s=2
(s−1) kA(s)k+
2b−
s
X
k=1
A(s)¯
1
1
−4e∗τ(A(1) , ..., A (k), b)
Then: τ(A(1) , ..., A (k), b)≤1
4e(k+ 2)
2b−
k
P
s=1
A(s)¯
1
1
+k
k
P
s=1
kA(s)k − 2k
n
P
i=1
ei.
For k= 1, this bound is obtained in [9] and a family of neural network which attains it is
given.
Remark 3 By using construction suggested in [18] to simulated sequentially a network of au-
tomata, we obtain for N= (I, A(1), ..., A(k), b) the sequential iterating network N′= (I′, A′, b′)
where:
•I′=
k+1
S
p=1
Ipwith Ip={(p−1)n+ 1, ..., (p−1)n+n}
•A′=
0A(k)A(k−1) ··· A(1)
A(1) 0 A(k)··· A(2)
.
.
..
.
..
.
..
.
.
A(k−1) A(k−2) A(k−3) ··· A(k)
A(k)A(k−1) A(k−2) ··· 0
0 is the 0-constant matrix
•b′= (b1, b2, ..., bk+1), bp=b∀p∈ {1, ..., k + 1}
8
This simulation can be used to bound the transient length of parallel iteration with memory.
Indeed let t≥k+ 1 such that t= (p−1) mod (k+ 1) (1 ≤p≤k+ 1). The updating of N′can
be written:
xi(t) = xi(t−1) if i ∈Iqand q 6=p
xi(t) = 1 k+1
P
q=1 P
j∈Iq
ai′j′((p−q)mod (k+ 1))xj(t−1) −bi!if i ∈Ip
(16)
where i′=i−(p−1)n,j′=j−(q−1)nand a(0) = 0.
It is a block sequential iteration on neural network (sequential with respect to the order of
blocks Ipand parallel within each block).
2.1 Transient length of block sequential iteration on neural network
Let N= (I, A, b) a neural network and I1, ..., Ilan ordered partition (with respect the order of
Z) of I={1, ..., m}; i.e i < i′if i∈Ir,i′∈Ir′with r < r′. The block sequential updating of N
is : at time t > 0, t= (r−1) mod l (1 ≤r≤l):
xi(t) = xi(t−1) if i ∈Ir′and r′6=r
xi(t) = 1 l
P
r′=1 P
j∈Ir′
aij xj(t−1) −bi!if i ∈Ir
(17)
We now show the following proposition:
Proposition 4 If Ais a real symmetric matrix with aij = 0 for i, j ∈Ir(1 ≤r≤l), then the
functional [5]:
Eb(x(t)) = −1
2
m
X
i=1
xi(t)
m
X
j=1
aij xj(t) +
m
X
i=1
bixi(t) (18)
is a strictly decreasing Lyapunov functional for the block sequential iteration of the neural net-
work.
Proof . Suppose t= (r−1) mod l (1 ≤r≤l). Let ∆tEb=Eb(x(t)) −Eb(x(t−1)). We have:
∆tEb=−1
2 m
P
i=1
xi(t)
m
P
j=1
aij xj(t)−
m
P
i=1
xi(t−1)
m
P
j=1
aij xj(t−1)!+
m
P
i=1
bi(xi(t)−xi(t−1))
=−1
2 P
i∈Ir
(xi(t)−xi(t−1))
l
P
r′=1,r′6=rP
j∈Ir′
aij xj(t−1)
+
l
P
r′=1,r′6=rP
j∈Ir′
xj(t−1) P
i∈Ir
aij (xi(t)−xi(t−1))!+P
i∈Ir
bi(xi(t)−xi(t−1))
=−P
i∈Ir
(xi(t)−xi(t−1)) l
P
r′=1,r′6=rP
j∈Ir′
aij xj(t−1) −bi!
From the definition of block sequential iteration, for any i∈Irwe find:
−(xi(t)−xi(t−1))
l
X
r′=1,r′6=rX
j∈Ir′
aij xj(t−1) −bi
≤0
So we concluded ∆tEb≤0. Futhermore if xi(t)6=xi(t−1) then ∆tEb<0.
9
Corollary 1 If Ais a real symmetric matrix with aij = 0 for i, j ∈Ir(1 ≤r≤l), then the
periods Tof the block sequential iteration on the neural network satisfies T=l.
Proof . If X= (x(0), ..., x(T−1)) is a cycle of period T, from the proof of Proposition 4 we
found that, Eb(x(0)) = ... =Eb(x(T−1)) iff ∀t= 0, ..., T −1, xi(t+ 1) = xi(t) for all i∈I.
Since at time t=lall neurons activated their local transition function, we conclude that T=l.
Let E∗
b(x(t)) be the functional [5] defined by:
E∗
b(x(t)) = −1
2
m
X
i=1
(2xi(t)−1)
m
X
j=1
aij (2xj(t)−1) +
m
X
i=1
2bi−
m
X
j=1
aij
(2xi(t)−1) (19)
Proposition 5 Let Abe a real symmetric matrix with aij = 0 for i, j ∈Ir(1≤r≤l). The
difference ∆tE∗
b=E∗
b(x(t)) −E∗
b(x(t−1)) = 4∆tEband then E∗
b(x(t)) is a strictly decreasing
Lyapunov functional for the block sequential iteration on neural network.
Proof . By symmetry of Awe have:
∆tE∗
b=−P
i∈Ir
(2xi(t)−2xi(t−1)) l
P
r′=1,r′6=rP
j∈Ir′
aij (2xj(t−1) −1) − 2bi−
m
P
j=1
aij !!
=−4P
i∈Ir
(xi(t)−xi(t−1)) l
P
r′=1,r′6=rP
j∈Ir′
aij xj(t−1) −bi!
= 4∆tEb
The functional E∗
b(x(t)) is a more appropriate Lyapunov functionnal to study the transient
length of N. Indeed, it is easy to show that |E∗
b(x(t))| ≤ 1
2kAk+
2b−A1
1. Denotes by ¯
X∗
the set of all initial conditions which do not belong to a period of length l:
¯
X∗={x(0) ∈ {0,1}nsuch that x(0) 6=x(l)}
If ¯
X∗6=∅define:
e∗=min −(Eb(x(l)) −Eb(x(l−1))) : x(0) ∈¯
X∗(20)
We note e∗= 0 if ¯
X∗= 0.
Proposition 6 Let Abe a real symmetric matrix with aij = 0 for i, j ∈Ir(1≤r≤l). The
transient length of block sequential iteration τb(A, b)is bounded by:
τb(A, b)≤1
4e∗kAk+ 2
2b−A1
(21)
Proof . Let {x(t) : t≥0}a trajectory with the transient length equals to τb(A, b). Assume
e∗>0. For any 0 ≤t≤τb(A, b)−1, x(t)∈¯
X∗. We deduce
E∗
b(x(t)) ≤E∗
b(x(t−1)) −4e∗for any l ≤t≤τb(A, b) + l−1
Then E∗
b(x(τb(A, b) + l−1)) ≤E∗
b(x(l−1)) −4e∗∗τb(A, b); which implies:
−1
2kAk −
2b−A1
1≤E∗
b(x(τb(A, b) + l−1)) ≤E∗
b(x(l−1)) −4e∗∗τb(A, b)≤
1
2kAk+
2b−A1
1−4e∗∗τb(A, b)
So τb(A, b)≤1
4e∗kAk+ 2
2b−A1
Remark 4 Since N′is a block sequential iterating neural network with a′
ij = 0 when i, j ∈Ip
(1 ≤p≤k+ 1), its transient length τb(A′, b′) is bounded by:
τb(A′, b′)≤1
4e
A′
+ 2
2b′−A′1
(22)
10
To compare this bound with the ones obtained in Theorem 3, Eq. (22) must be rewritten as
follows:
τb(A′, b′)≤1
4e 2(k+ 1)
2b−
k
X
s=1
A(s)1
+ (k+ 1)
k
X
s=1
kA(s)k!(23)
Let
τ=1
4e (k+ 2)
2b−
k
X
s=1
A(s)¯
1
1
+k
k
X
s=1
kA(s)k − 2k
n
X
i=1
ei!(24)
and
τ′=1
4e 2(k+ 1)
2b−
k
X
s=1
A(s)1
+ (k+ 1)
k
X
s=1
kA(s)k!(25)
we find
τ′−τ=1
4e k
2b−
k
X
s=1
A(s)¯
1
1
+
k
X
s=1
kA(s)k+ 2k
n
X
i=1
ei!>0 (26)
Hence, the first bound (τ) is better than the ones obtained by sequential simulation of the
parallel iteration of a neural network with memory.
3 Sequential iteration with memory
We define the sequential iteration with memory as follows: the update of the neurons when the
network evolves from t−1 to toccurs hierachically according to a pre-established periodic order
on I(we shall assume, without loss of generality, that the order on Iis the same order as I
posseses as a subset of Z). Thus, when the neuron ichanges from xi(t−1) to xi(t), all the
vertices j < i have already evolved. The states considered for the iteration are xj(t+ 1 −s) for
j < i and xj(t−s) for j≥i;s= 1, ..., k.
Thus the configuration of the system are x(t)∈ {0,1}n, the set of interaction matrices is
{A(s) = (aij (s) : i, j ∈ {1, ..., n}) : s= 1, ..., k}and the threshold vector is b= (bi:i∈ {1, ..., n}).
Since a sum over an empty set of indexes is null (
i−1
P
j=1
= 0 if i= 1), the sequential updating with
memory of the neural network is written:
xi(t) = 1
i−1
X
j=1
k
X
s=1
aij (s)xj(t+ 1 −s) +
n
X
j=i
k
X
s=1
aij (s)xj(t−s)−bi
(27)
When k= 1, we obtain a Mc Culloch and Pitts neural network iterating sequentially [15, 7].
Let Tbe the period of the neural network. Let X= (x(0) , ..., x (T−1)) be a T-cycle. For
any couple of local cycles (Xi, Xj) we define the sequential functional (algebraic invariant) by:
L(Xi, Xj) =
k
P
s=1
aij (s) ∆Vk−s+1,s−1(Xi, Xj)if j < i
k−1
P
s=1
aij (s) ∆Vk−s,s (Xi, Xj)if j =i
k
P
s=1
aij (s) ∆Vk−s,s (Xi, Xj)if j > i
(28)
which can be written as:
11
L(Xi, Xj) =
k
P
s=1
aij (s)
TP
t∈ZT
xi(t) (xj(t+k−s+ 1) −xj(t−s+ 1)) if j < i
k−1
P
s=1
aij (s)
TP
t∈ZT
xi(t) (xj(t+k−s)−xj(t−s)) if j =i
k
P
s=1
aij (s)
TP
t∈ZT
xi(t) (xj(t+k−s)−xj(t−s)) if j > i
(29)
From Lemma 1 we find:
if τ(Xi)|kthen L(Xi, Xj) = 0 for any j∈I(30)
Now for evolution Eq. (27) we establish the following lemma:
Lemma 3 For any family of interaction matrices (A(s) : s= 1, ..., k)such that aii (k)≥0for
any i∈Iwe have: X
j∈I
L(Xi, Xj)≤0for any i∈I(31)
L(Xi, Xj) = 0 for any j∈Iiff X
j∈I
L(Xi, Xj) = 0 iff τ(Xi)|k(32)
X
j∈I
L(Xi, Xj)<0iff τ(Xi)does not divide k(33)
X
i∈IX
j∈I
L(Xi, Xj) = 0 iff τ(Xi)|kfor any i∈I(34)
Proof . Consider Γ0(Xi) = {t∈ZT:xi(t) = 0}, Γ1(Xi) = {t∈ZT:xi(t) = 1},ζ(Xi) the
class of maximal k-chains included in Γ1(Xi), and ζk(Xi) the subclass of maximal k-chains
which are k-periodic. We have:
P
j∈I
L(Xi, Xj) = P
j<i
L(Xi, Xj) + L(Xi, Xi) + P
j>i
L(Xi, Xj)
=1
TP
C∈ζ(Xi)\ζk(Xi) P
j<i
k
P
s=1
aij (s) (xj(¯
tc+k−s+ 1) −xj(tc−s+ 1))
+
k−1
P
s=1
aii (s) (xi(¯
tc+k−s)−xi(tc−s)) + P
j>i
k
P
s=1
aij (s) (xj(¯
tc+k−s)−xj(tc−s))!
=1
TP
C∈ζ(Xi)\ζk(Xi) P
j<i
k
P
s=1
aij (s)xj(¯
tc+k−s+ 1) + P
j≥i
k
P
s=1
aij (s)xj(¯
tc+k−s)
− P
j<i
k
P
s=1
aij (s)xj(tc−s+ 1) + P
j≥i
k
P
s=1
aij (s)xj(tc−s) + aii (k) (xi(¯
tc)−xi(tc−k))!!
For any C∈ζ(Xi)\ζk(Xi), we have:
xi(¯
tc+k) = 0,then X
j<i
k
X
s=1
aij (s)xj(¯
tc+k−s+ 1) + X
j≥i
k
X
s=1
aij (s)xj(¯
tc+k−s)< bi
On the other hand,
xi(tc) = 1,then X
j<i
k
X
s=1
aij (s)xj(tc−s+ 1) + X
j≥i
k
X
s=1
aij (s)xj(tc−s)≥bi
12
Then as τ(Xi) does not divide k, the set ζ(Xi)\ζk(Xi) is not empty so:
X
j∈I
L(Xi, Xj)<−aii(k)
TX
C∈ζ(Xi)\ζk(Xi)
(xi(¯
tc)−xi(tc−k))
But xi(¯
tc) = 1 and xi(tc−k) = 0 so P
i∈I
L(Xi, Xj)<0. Hence Eqs. (32) and (33) are verified.
Eqs. (32) and (33) imply (31) and finally Eqs. (31), (32) and (33) imply Eq. (34).
Now assume that the set of interaction matrices (A(s) : s= 1, ..., k) satisfiy:
diag (A(s)) = diag (A(s+ 1)) ∀s= 1, ..., k −1 (35)
where diag (A(s)) = (aii (s) : i∈I)
Theorem 4 If the class of interaction matrices (A(s) : s= 1, ..., k)is palindromic (i.e. satisfy
(7)) and satisfy (35) then the period Tof the neural network iterated sequentially with k-memory
satisfies T|k.
Proof . If aij (s) = aji (k−s+ 1) for s= 1, ..., k then
L(Xi, Xj) + L(Xj, Xi) = 0 ∀i, j ∈I . (36)
Indeed, for i6=jwe have:
L(Xi, Xj) =
k
P
s=1
aij (s) ∆Vk−s,s (Xi, Xj)
=
k
P
s=1
aji (k−s+ 1) ∆Vk−s,s (Xi, Xj)
=
k
P
s′=1
aji (s′) ∆Vs′−1,k−s′+1 (Xi, Xj)
Since ∆Vk,l (Xi, Xj) = −∆Vl,k (Xj, Xi) for k, l ∈Z, i, j ∈I, we have:
L(Xi, Xj) = −
k
P
s′=1
aji (s′) ∆Vk−s′+1,s′−1(Xj, Xi)
=−L(Xj, Xi).
and:
L(Xi, Xi) =
k−1
P
s=1
aii (s) ∆Vk−s,s (Xi, Xi)
=
k−1
P
s=1
aii (s) ∆Vk−s,−s(Xi, Xi)
=aii(k)
T
k−1
P
s=1 P
C∈ζ(Xi)\ζk(Xi)P
t∈C
(xi(t+k−s)−xi(t+s))!
=aii(k)
TP
C∈ζ(Xi)\ζk(Xi)P
t∈Ck−1
P
s=1
(xi(t+k−s)−xi(t+s))
Since
k−1
P
s=1
(xi(t+k−s)−xi(t+s)) = 0, we have L(Xi, Xi) = 0 for i∈I.
From Eqs. (34) and (36) we deduce that if the sequence of interaction matrices (A(s) : s= 1, ..., k)
verify conditions (7) and (35) then P
i∈IP
j∈I
L(Xi, Xj) = 0 for any cycle X.
Theorem 4 is a generalization of the following theorem established in [7] for Mc Culloch and
Pitts neural network iterating sequentially.
13
Theorem 5 [7]. Assume the matrix of interactions Ato be symmetric with non-negative di-
agonal entries aii ≥0for any i∈I. Then the period Tof the sequential iteration of Neural
Network N= (I, A, b)is T= 1, so any initial condition converges to a fixed point.
Remark 5 When the set of interaction matrices (A(s) : s= 1, ..., k) is not palindromic, we can
get limit cycles of periods > k. Take n≥6, k=n−3, b= (1, ..., 1), the interaction matrices
A(2) = ... =A(k−1) = 0 and:
A(1) =
0 1 0 ··· 0 0 0
0 0 1 ··· 0 0 0
.
.
..
.
..
.
..
.
..
.
..
.
.
0 0 0 ··· 0 0 1
1 0 0 ··· 0 0 0
, A (k) =
0 0 0 ··· 0 0 1
−1 0 0 ··· 0 0 0
.
.
..
.
..
.
..
.
..
.
..
.
.
0 0 0 · · · −100
0 0 0 ··· 0−1 0
Then the initial condition:
x(i) = (0, ..., 0
| {z }
n−i−2
,1,0, ..., 0)
| {z }
i+1
for i = 0, ..., k −2, x (k−1) = (1,0,1,0, ..., 0)
belongs to the following n−1 cycle:
x(0) = (0,0,0,0, ..., 0,0,1,0)
x(1) = (0,0,0,0, ..., 0,1,0,0)
x(2) = (0,0,0,0, ..., 1,0,0,0)
.
.
.
x(k−2) = (0,0,0,1, ..., 0,0,0,0)
x(k−1) = (1,0,1,0, ..., 0,0,0,0)
x(k) = (0,1,0,0, ..., 0,0,0,0)
x(k+ 1) = (1,0,0,0, ..., 0,0,0,1)
x(k+ 2) = (0,0,0,0, ..., 0,0,1,0)
x(k+ 3) = (0,0,0,0, ..., 0,0,0,0)
.
.
.
x(k+n−3) = (0,0,0,1, ..., 0,0,0,0)
x(k+n−2) = (1,0,1,0, ..., 0,0,0,0)
Remark 6 If the diagonals of interaction matrices A(s), s = 1, ..., k are not ≥0, we can get
limit cycles of long periods. Take the following matrices of interactions (n≥2 and k= 3):
A(1) = A(3)t=
−1 1 0 ··· 0 0 0
0−1 1 ··· 0 0 0
.
.
..
.
..
.
..
.
..
.
.
0 0 0 ··· 0−1 1
1 0 0 ··· 0 0 −1
A(2) =
−1 0 0 ··· 0 0 0
0−1 0 ··· 0 0 0
.
.
..
.
..
.
..
.
..
.
..
.
.
0 0 0 ··· 0−1 0
0 0 0 ··· 0 0 −1
14
and the constant threshold vector b= (1, ..., 1). The initial condition:
x(0) = (0, ..., 0) , x (1) = (0, ..., 0) , x (2) = (1,0, ..., 0)
belongs to a cycle of length 2n+ 1 as follows:
x(0) = (0,0,0,0, ..., 0,0,0)
x(1) = (0,0,0,0, ..., 0,0,0)
x(2) = (1,0,0,0, ..., 0,0,0)
x(3) = (0,0,0,0, ..., 0,0,0)
x(4) = (0,1,0,0, ..., 0,0,0)
x(5) = (0,0,0,0, ..., 0,0,0)
x(6) = (0,0,1,0, ..., 0,0,0)
.
.
.
x(2n−1) = (0,0,0,0, ..., 0,0,0)
x(2n) = (0,0,0,0, ..., 0,0,1)
x(2n+ 1) = (0,0,0,0, ..., 0,0,0)
x(2n+ 2) = (0,0,0,0, ..., 0,0,0)
x(2n+ 3) = (1,0,0,0, ..., 0,0,0)
Remark 7 If the diagonals of interaction matrices A(s), s = 1, ..., k are not equals, we can get
limit cycles of periods > k. Take n≥5, k= 3, b= (1, ..., 1),
A(1) = A(3)t=
0−1 0 ··· 0 0 0
0 0 −1··· 0 0 0
.
.
..
.
..
.
..
.
..
.
..
.
.
0 0 0 ··· 0 0 −1
1 0 0 ··· 0 0 0
A(2) =
0 0 0 ··· 0 0 0
0 1 0 ··· 0 0 0
.
.
..
.
..
.
..
.
..
.
..
.
.
0 0 0 ··· 0 1 0
0 0 0 ··· 0 0 0
It is easy to see that for any 2 ≤p≤n−3, the initial condition:
x(0) = (0, ..., 0) , x (1) = (0, ..., 0
| {z }
n−1−p
,1,0, ..., 0)
| {z }
p
, x (2) = (1,0, ..., 0,1)
15
belongs to a 6-cycle as follows:
x(0) = (0,0,0,0, ..., 0,0,0)
x(1) = (0, ..., 0
| {z }
n−1−p
,1,0, ..., 0)
| {z }
p
x(2) = (1,0,0,0, ..., 0,0,1)
x(3) = (0, ..., 0
| {z }
n−1−p
,1,0, ..., 0)
| {z }
p
x(4) = (0,0,0,0, ..., 0,0,0)
x(5) = (1,0, ..., 0
| {z }
n−1−p
,1,0, ..., 0,1)
| {z }
p
x(6) = (0,0,0,0, ..., 0,0,0)
x(7) = (0, ..., 0
| {z }
n−1−p
,1,0, ..., 0)
| {z }
p
x(8) = (1,0,0,0, ..., 0,0,1) .
4 Conclusion
We study neural networks of Caianiello under some assumptions on interaction matrices. For
parallel iteration, using Lyapunov functional, we characterize the periods and bounds explicitely
the transient lengths of neural networks. The bound is compared with the ones obtained by
sequential simulation of the parallel iteration of a neural network with memory and proves
more better. We introduce sequential iteration with memory of neural networks and, using an
algebraic invariant, characterize its period behaviour.
Acknowledgments. This work was done within the framework of the Associateship Scheme
of the Abdus Salam International Centre of Theoretical Physics (ICTP), Trieste, Italy. Financial
support from the Swedish International Development Cooperation Agency (SIDA) is acknowl-
edged.
References
[1] E.R. Caianiello, Decision Equations and Reverberations, Kybernetik, 3(2), 1966.
[2] E.R. Caianiello, A. De Luca Decisions equation for binary systems: applications to neuronal
behavior, Kybernetik, 3(1966), 33-40.
[3] M. Cosnard, D. Moumida, E. Goles, and T. de. St. Pierre, Dynamical Behavior of a Neural
Automaton with Memory, Complex systems, 2 (1988), 161-176.
[4] M. Cosnard, M. Tchuent´e, and G. Tindo, Sequences Generated by Neuronal Automata with
Memory, Complex systems, 6 (1992), 13-20.
[5] F. Fogelman Souli´e, E. Goles, G. Weisbuch, Transient length in sequential iteration of thresh-
old functions, Disc. App. Maths, 6, 1983, 95-98.
[6] F. Fogelman Souli´e, et al, Automata networks and artificial intelligence, in: F. Fogelman,
Y. Robert, M. Tchuente (Eds), Automata Networks in Computer Science: Theory and
Applications, Manchester, 1987, 133-186.
16
[7] E. Goles, Fixed Point Behaviour of Threshold Functions on a Finite set, SIAM J. on Alg.
and Disc. Alg. Meths., 3(4), 1982, 529-531.
[8] E. Goles Dynamical Behaviour of Neural Networks, SIAM J. Disc. Alg. Math., 6, 1985,
749-754.
[9] E. Goles, Lyapunov Functions associated to Automata Networks, in Automata Networks in
Computer Science, F. Fogelman, Y. Robert, M. Tchuent´e eds., Manchester University Press,
1987, 58-81.
[10] E. Goles, S. Martinez, Neural and automata Networks, Dynamical Behaviour and Applica-
tions, Kluwer Academic Publishers, Norwell, MA 1990.
[11] E. Goles, J. Olivos Comportement P´eriodique des Fonctions `a Seuil Binaires et Applications,
Disc. App. Math., 3, 1981, 95-105.
[12] S. Haykin, Neural Networks: A Comprehensive Foundations, Prentice-Hall, Upper Saddle
River, NJ, 1999.
[13] J.J. Hopfield, Neural Networks and Physical Systems with Emergent Collective Computa-
tional Abilities, Proc. Natl. Acad. Sci. USA, 79, 1982, 2554-2558.
[14] M. Matamala, Recursive Construction of Periodic Steady State for Neural Networks, The-
oretical Computer Science, 143 (2) (1995), 251-267.
[15] W.S. McCulloch, W. Pitts, A logical calculus of the ideas immanent in nervous activity,
Bull. Math. Biophys., 5 (1943), 115-133.
[16] R. Ndoundam, M. Tchuent´e, Cycles exponentiels des r´eseaux de Caianiello et compteurs
en arithm´etique redondante, Technique et science informatiques, Volume 19-n07/2000, 985-
1008.
[17] R. Ndoundam, and M. Tchuent´e, Exponential Period of Neural Recurrence Automata with
Excitatory Memory, Complex systems, 15 (2004), 79-88.
[18] M. Tchuent´e, Sequential simutation of parallel iterations and applications, Theoretical Com-
puter Science, 48 (1986), 135-144.
17
