Design of Evolvable Biologically Inspired Classifiers.

Full text

Page 1

Design of Evolvable Biologically Inspired
Classifiers
Rinde R.S. van Lon, Pascal Wiggers, Léon J.M. Rothkrantz, and Tom Holvoet
Abstract Complex systems are emergent, self-organizing and adaptive systems.
They are pervasive in nature and usually hard to analyze or understand. Often they
appear intelligent and show favorable properties such as resilience and anticipation.
In this paper we describe a classifier model inspired by complex systems theory.
Our model is a generalization of neural networks, boolean networks and genetic
programming trees called computational networks. Designing computational net-
works by hand is infeasible when dealing with complex data. For designing our
classifiers we developed an evolutionary design algorithm. Four extensions of this
algorithm are presented. Each extension is inspired by natural evolution and theories
from the evolutionary computing literature. The experiments show that our model
can be evolutionary designed to act as a classifier. We show that our evolved clas-
sifiers are competitive compared to the classifiers in the Weka classifier collection.
These experiments lead to the conclusion that using our evolutionary algorithm to
design computational networks is a promising approach for the creation of classi-
fiers. The benefits of the evolutionary extensions are inconclusive, for some datasets
there is a significant performance increase while for other datasets the increase is
very minimal.
Rinde R.S. van Lon
DistriNet Labs, Department of Computer Science, Katholieke Universiteit Leuven
e-mail: rinde.vanlon@cs.kuleuven.be
Pascal Wiggers
Man-Machine Interaction Group, Department of Mediamatics, Delft University of Technology
Léon J.M. Rothkrantz
Man-Machine Interaction Group, Department of Mediamatics, Delft University of Technology
Tom Holvoet
DistriNet Labs, Department of Computer Science, Katholieke Universiteit Leuven
1

Page 2

2Rinde R.S. van Lon, Pascal Wiggers, Léon J.M. Rothkrantz, and Tom Holvoet
1 Introduction
Many problems in the world can be formulated as classification tasks. In fact, hu-
mans are classifying things, sounds, objects, scents and feelings all the time. Most
of these ‘problems’ are solved naturally by humans, while in fact some of them are
quite hard. That is, computers have not been able to solve some of these problems.
Classifiers are typically implemented using parameterized models, e.g. Bayesian
networks or neural networks, in which the model structure is specified by hand and
the parameters are learned by collecting statistics over the data. Such models capture
average behavior, but fail to model large variations and exceptions to the rule, they
lack adaptivity. Looking at software systems in general it can be noted that they do
not possess the adaptive properties which are so distinctive for natural systems. In
the world around us, adaptation is everywhere. Every organism is constantly adapt-
ing to its environment. According to Salehie and Tahvildari [30] there is a need for
a new approach for developing adaptive software.
In nature different adaptation strategies exist, the focus of this paper lies on so
called complex systems. Examples of complex systems are as diverse as ant colonies
[9], immune systems [14], the human mind [25] or a market economy. A common
feature of these systems is that they consist of autonomous interacting entities, it is
hard to pinpoint the location of the adaptivity in such a system. Yet their tremendous
power is apparent when looking at them from an engineering point of view.
In this paper we use complex systems as an inspiration for the design of a classi-
fier model. Our model is a variant of neural networks which we call computational
networks (CN). A computational network is a graph with mathematical functions
as nodes. These function nodes are similar to the functions in genetic program-
ming trees [22]. The model is a generalization of random boolean networks, neural
networks and genetic programming trees. A computational network allows more
expressive functions, it poses no restrictions on the graph topology (recurrent loops
are allowed) and it allows real values.
Designing computational networks by hand is nearly impossible due to some of
their complex system properties (which we will discuss in section 2). As an alter-
native we turn to artificial evolution for the design task. An advantage of evolving
classifiers based on complex systems compared to the traditional approach is that
the classifier can be structurally adapted to a specific problem. If the problem re-
quires a more adaptive model, evolution should be able to design the model to be
more adaptive. Another motive for using evolution is that natural complex systems
have emerged through natural evolution, as such it is plausible that complex sys-
tems can be designed using artificial evolution. An evolutionary algorithm (EA) is
constructed that designs computational networks. Several extensions for this EA are
proposed and investigated to verify their usefulness.
Our long term goal is to develop adaptive classifiers which are capable of han-
dling dynamism such as is encountered in the real world. In this paper, however, we
focus on creating static classifiers inspired by the theory of complex systems. We
hope that in future work our model can be a starting point towards more adaptive
classifiers.

Page 3

Design of Evolvable Biologically Inspired Classifiers3
First, in section 2 we discuss some properties of complex systems that serve as
inspiration for our model and the design choices we made. In section 3 we present
our model, so-called computational networks and in section 4 we present our evolu-
tionary design algorithm. Section 5 describes the extension comparison experiment
and the performance comparison experiment. Section 6 discusses the obtained ex-
perimental results and we conclude this paper in section 7.
2 Complex Systems
Research of complex systems is done in different disciplines. The field of Complex
Adaptive Systems (CAS) [14] is concerned with an analysis of complex systems as
there is currently not a complete mathematical model of it. Collective intelligence
(CI), is more concerned with modeling the behavior of complex systems. For exam-
ple,in[31]Schutidentifiesseveralcharacteristicpropertiesofcollectiveintelligence
systems. Swarm intelligence (SI) is inspired by natural complex systems, and is a
paradigm for implementing adaptive systems [2, 20]. Further, the field of artificial
life (a-life) aims to construct artifacts that approach the complexity of biological
systems [1]. Typical examples of a-life are the work of Sims [32] and the research
field of cellular automata, where from simple rules complex patterns can emerge.
The remainder of this section presents a definition for complex systems and a
set of properties based on an extensive study of the related literature. Additionally,
severalfundamentaldesignchoicesarepresentedforthedesignofcomplexsystems.
2.1 Definition
Our working definition of complex systems is:
Complex systems are adaptive systems composed of emergent and self-organizing
structures.
This definition says that in order to obtain an adaptive system, emergence and self-
organization are required. In the next two sections a total of six properties are iden-
tified for emergence and self-organization. In our view, these properties are required
to obtain an emergent and self-organizing system. Consequently, these six proper-
ties need to be included in the model in order to obtain an adaptive system. Using
these properties the design choices in section 2.2 are made.
2.1.1 Emergence
Emergence is a concept lacking a concise definition [16]. However, Corning [3]
defines emergence as the arising of novel structures during the process of self-

Page 4

4Rinde R.S. van Lon, Pascal Wiggers, Léon J.M. Rothkrantz, and Tom Holvoet
organization in complex systems. This shows the close relation between the topics
of self-organization and emergence, just as in the proposed definition. A more sim-
ple definition of emergence is posed by Schut [31] as “the whole is more than the
sum of its parts”. This lack of a proper definition results in a lot of controversy how
emergent properties are to be distinguished from non-emergent properties [4]. Nev-
ertheless, three properties are presented which, in our opinion, belong to emergence.
The distinction between a local and a global perspective or level is often asso-
ciated with emergence [6]. Local is the level of the individual which describes the
behavior of the individual. Global is the system wide perspective, which describes
characteristics on a system level. Emergence occurs when global behavior of a sys-
tem is not evident from the local behavior of its elements [20].
Interaction between individuals is an important property for emergence. When
parts interact in less simple ways, reduction does not work, the interaction as well
as the parts have to be studied [16]. The stability of an economy can be seen as an
emergent property caused by the interactions of the participants (e.g. buyers, sellers)
in this economy [20].
This interaction between individuals gives rise to possible non-linear relations in
a system. This non-linearity is often stated as: small causes can have large effects,
and large causes can have small effects.
2.1.2 Self-organization
De Wolf and Holvoet [6] state that self-organizing systems appear to organize them-
selves without external direction, manipulation, or control. Di Marzo Serugendo [7]
statesthatself-organizingsystemsgiverisetoemergentproperties.Weidentifythree
properties which are in our opinion crucial to obtain self-organization.
Decentralization is an obvious property of self-organization, there is no central
controller that supervises all the parts. All individuals are autonomous.
The interaction discussed in section 2.1.1 gives rise to feedback. Feedback is a
circular cause-and-effect relation [13]. There is positive, self-reinforcing feedback
which allows a deviation to enlarge in an explosive way. And there is negative, self-
counteracting feedback which stabilizes the system.
Self-organizing systems sometimes appear to behave randomly. The occurrence
of randomness in complex systems is arguably more a limitation of the observer
than an intrinsic property of a system. If the observer would be able to perceive
every part of the system and completely understand the relations between the parts,
the randomness would disappear.

Page 5

Design of Evolvable Biologically Inspired Classifiers5
2.2 Design of complex systems
The design of a complex system is a delicate task, that is, the design of a useful
complex system is complicated. As was stated in section 2.1.1, reduction does not
work,it isvery hardtoassemble partswhichon aglobal levelyielddesired behavior.
Several attempts have been made to create complex systems, for example,
Mitchell and Hofstadter [26] have created an emergent architecture capable of mak-
ing analogies. In subsequent research their idea was applied to music prediction,
geometric puzzles [10], and auditory perception [8].
An alternative approach is automating the design of complex systems. Artifi-
cial evolution is widely used for this purpose. One of the most prominent examples
is genetic programming (GP) pioneered by Koza [22]. GP evolves a population of
computer programs (in the form of syntax trees) to perform a task. Another example
is the earlier mentioned work of Sims [32], where robot behavior and morphology
were evolutionary designed. Evolution has also successfully been applied for the
design of hardware [35], the design of robots [29], the development of an acous-
tic communication system between robots [36] and for designing cellular automata
[34]. Another automated approach is that of learning classifier systems pioneered
by John Holland [15].
We made several design decisions for modeling complex systems. In the follow-
ing sections these decisions are presented, the choices were listed in [37].
2.2.1 Centralization or decentralization
The choice between a centralized or decentralized control structure of a complex
system is clear. As discussed in section 2.1.2, self-organization and decentralization
are important properties of complex systems. As such, a decentralized approach is
recommended.
2.2.2 Differentiation
This choice concerns the homogeneity of the individuals in the complex systems.
Natural complex systems of both types exist, ants in an ant colony are (largely) the
same, while there are several types of antibodies in the immune system. Groß et al.
[11] show that homogenous robots are capable of a division of labor, meaning that
robots perform different kind of tasks. An advantage of the homogenous option over
the heterogenous option is that it reduces the design task significantly. As compro-
mise, we prefer a semi-homogenous option which reduces the design task but still
allows a limited amount of variety among individuals.