Conference PaperPDF Available

On the Locality of Neural Meta-representations


Abstract and Figures

We consider the usage of artificial neural networks for representing genotype-phenotype maps, from and into continuous decision variable domains. Through such an approach, genetic representations become explicitly controllable entities, amenable to adaptation. With a view towards understanding the kinds of space transformations neural networks are able to express, we investigate here the typical representation locality given by arbitrary neuro-encoded genotype-phenotype maps. We consistently find high locality space transformations being carried out, across all tested feedforward neural network architectures, in 5, 10 and 30 dimensional spaces.
Content may be subject to copyright.
On the Locality of Neural Meta-Representations
Luís F. Simões
VU University Amsterdam
A. E. Eiben
VU University Amsterdam
We consider the usage of artificial neural networks for repre-
senting genotype-phenotype maps, from and into continuous
decision variable domains. Through such an approach, ge-
netic representations become explicitly controllable entities,
amenable to adaptation. With a view towards understand-
ing the kinds of space transformations neural networks are
able to express, we investigate here the typical represen-
tation locality given by arbitrary neuro-encoded genotype-
phenotype maps. We consistently find high locality space
transformations being carried out, across all tested feedfor-
ward neural network architectures, in 5, 10 and 30 dimen-
sional spaces.
Categories and Subject Descriptors
I.2.8 [Artificial Intelligence]: Problem Solving, Control
Methods, and Search—Heuristic methods
Representations; Genotype-Phenotype map; Neural networks
Adjusting the parameters and/or components of evolu-
tionary algorithms (EA) during a run offers several advan-
tages [1]. Over the last two decades, we have witnessed
a plethora of methods being introduced for automatically
adapting all sorts of EA parameters and components, with
one notable exception: the genotype-phenotype (G-P) map-
ping, a.k.a. the representation. Few papers ever addressed
its adaptation.
We propose neural networks (NN) as a generic frame-
work for representing representations (i.e. NNs as a meta-
representation). Following such an approach, changes to
the architecture and/or weights of a NN providing a prob-
lem’s genotype-phenotype mapping will effectively change
the representation space explored by evolution. Feasible ap-
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage, and that copies bear this notice and the full ci-
tation on the first page. Copyrights for third-party components of this work must be
honored. For all other uses, contact the owner/author(s). Copyright is held by the
GECCO’14, July 12–16, 2014, Vancouver, BC, Canada.
ACM 978-1-4503-2881-4/14/07. .
proaches for implementing such representation learning in-
clude co-evolution, self-adaptation, and meta-evolution.
In this paper, we formally introduce the approach of us-
ing NNs as genotype-phenotype maps, and then follow with
an investigation into the locality of representations defined
though these means.
Neural networks form the basis of our approach. Being
proven global function approximators (given a sufficiently
large number of hidden layer neurons), feedforward neural
networks are in principle capable of expressing any possible
G-P map. Also, the community has ample experience with
neuroevolution methods, which can be deployed for the ef-
fective learning of representations.
In general, evolutionary search is guided by a fitness func-
tion fthat can be decomposed into a genotype-phenotype
mapping fg: ΦgΦpand a phenotype-fitness mapping
fp: ΦpIR, cf. [2, Sec. 2.1.2]. When considering an EA
that searches in a continuous genotypic space, for solutions
that decode into continuous phenotypes, we then have that
ΦgIRm, and ΦpIRn, where mand nstand, respec-
tively, for the dimensionalities of the considered genotypic
and phenotypic spaces. A genotype-phenotype map is then
a transformation fg: IRmIRn.
Let Nbe a fully connected, feedforward neural network
with llayers and dkneurons on its k-th layer (k= 1..l). If
Lkis the vector representing the states of the dkneurons in
its k-th layer, then the network’s output can be determined
ij Lk1
j), i =,
where bkrepresents the biases for neurons in the k-th
layer, and wk
ithe weights given to signals neuron Lk
from neurons in the preceding layer. A sigmoidal activation
function σ(y)=1/(1 + ey) is used throughout this pa-
per. The network’s output, Ll, is then uniquely determined
through b,w, and L1, the input vector fed to its input layer.
Without loss of generality, in the sequel we assume that
the given phenotype space is an ndimensional hypercube.
(If needed, the interval [0,1] can be mapped with a trivial
linear transformation to the actual user specified lower and
upper bounds for each variable under optimization.) Using a
neural network as a G-P map, we then obtain a setup where
the number of output neurons dl=nand the mapping itself
is fg: [0,1]d1[0,1]dl. To specify a given G-P mapping
network we will use the notation N(mp, ma), where mpand
maare the map parameters1and map arguments, defined
as follows. The vector mp[1,1]dcontains the definition
of all weights and biases in the network, while the vector ma
designates the input vector fed into the network. With this
notation, we obtain a formal framework where genotypes
are map arguments to the neural net and the representa-
tion is fg=N(mp, .). Given a genotype xg[0,1]d1, the
corresponding phenotype is fg(xg) = N(mp, xg)[0,1]dl.
As shorthand for a considered NN architecture, we will
use notation such as 30-5-10, to indicate a fully connected
feeedforward neural network with l= 3 layers, having d1=
30 neurons in its input layer, d2= 5 neurons in the hid-
den layer, and d3= 10 neurons in the output layer (Φg=
In the sizing of the neural network employed to represent
a G-P map, we are given a trade-off between its capacity
to define arbitrary space transformations, and our capacity
to effectively train it. To better understand which kinds
of mappings might be defined in practice through this ap-
proach, we focus on one often-studied representation prop-
erty, locality, and address the following research question:
what is the locality of representations given by small
to medium sized neural networks?
A representation’s locality [2, Sec. 3.3] describes how
well neighboring genotypes correspond to neighboring phe-
notypes. In a representation with perfect (high) locality,
all neighboring genotypes correspond to neighboring pheno-
types. Theoretical and experimental evidence [2] support
the view that high locality representations are important
for efficient evolutionary search, as they do not modify the
complexity of the problems they are used for.
Map characterization.
The G-P map design space is explored by randomly sam-
pling (with uniform probability) NN weights and biases, in
the range [1,1], thus providing the definition of map pa-
rameters, mp. We follow by generating a large number of
map arguments, ma(10000, to be precise), in the range
[0,1], according to a quasi-random distribution. Sobol se-
quences are used to sample the genotype space (maΦg),
so as to obtain a more evenly spread coverage. The mascat-
tered in the genotype space are subsequently mapped into
the phenotype space. The Euclidean metric is used to mea-
sure distances between points within the genotype space, as
well as within the phenotype space.
We characterize the locality of representations definable
by a given neural network architecture, by randomly sam-
pling the space of possible network configurations (mp) in
that architecture. A sample of 1000 points is taken, out
of the 10000 magiven by the Sobol sequence (mentioned
above), and a mutated version generated. A mutation is al-
ways a random point along the surface of the hypersphere
centered on the original genotype, and having a radius equal
to 1% the maximum possible distance in the Φghypercube.
The mutated mais mapped into the phenotype space, and
“Map parameters” named by analogy with the strategy pa-
rameters (e.g., standard deviations of a Gaussian mutation)
traditionally used in Evolution Strategies.
Figure 1: Locality of representations expressible by
different sized neural networks. Shown: empiri-
cal cumulative distribution functions of distances in
phenotype space between pairs of neighboring geno-
its distance there to the original point’s phenotype mea-
sured. Given we wish to consider phenotype spaces having
distinct numbers of dimensions, and importantly, given the
fact that each different G-P map encodes a different subset
of the phenotype space, it becomes important to normalize
phenotypic distances, in a way that makes then comparable.
To that end, we identify the hyperrectangle that encloses
all the phenotypes identified in the initial scatter of 10000
points, and use the maximum possible distance value there
to normalize phenotype distances.
Figure 1 characterizes the locality of representations defin-
able by different NN architectures. Each of the shown distri-
butions was obtained by analyzing 1000 randomly generated
G-P maps having that architecture, and thus represents a
total of 106measured phenotype distances. We consistently
observe high locality representations resulting from all stud-
ied NN architectures: a mutation step of 1% the maximum
possible distance in genotype space is in all cases expected
to take us across a distance in phenotype space of at most
1% the maximum possible distance among phenotypes
representable by the considered G-P map.
We investigated the usage of neural networks as a meta-
representation, suited to the encoding of genotype-phenotype
maps for arbitrary pairings of fitness landscapes and meta-
heuristics that are to search on them.
Small to moderately sized feedforward neural networks
were found to define, on average, high locality representa-
tions (where structure of the phenotypic fitness landscape is
locally preserved in the genotype space).
Lu´ıs F. Sim˜oes was supported by FCT (Minist´erio da
Ciˆencia e Tecnologia) fellowship SFRH/BD/84381/2012.
[1] A. E. Eiben, R. Hinterding, and Z. Michalewicz.
Parameter Control in Evolutionary Algorithms. IEEE
Transactions on Evolutionary Computation,
3(2):124–141, 1999.
[2] F. Rothlauf. Representations for Genetic and
Evolutionary Algorithms. Springer, Berlin, Heidelberg,
2nd edition, Nov. 2006.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
One of the major challenges for researchers in the field of management science, information systems, business informatics, and computer science is to develop methods and tools that help organizations, such as companies or public institutions, to fulfill their tasks efficiently. However, during the last decade, the dynamics and size of tasks organizations are faced with has changed. Firstly, production and service processes must be reorganized in shorter time intervals and adapted dynamically to the varying demands of markets and customers. Although there is continuous change, organizations must ensure that the efficiency of their processes remains high. Therefore, optimization techniques are necessary that help organizations to reorganize themselves, to increase the performance of their processes, and to stay efficient. Secondly, with increasing organization size the complexity of problems in the context of production or service processes also increases. As a result, standard, traditional, optimization techniques are often not able to solve these problems of increased complexity with justifiable effort in an acceptable time period. Therefore, to overcome these problems, and to develop systems that solve these complex problems, researchers proposed using genetic and evolutionary algorithms (GEAs). Using these nature-inspired search methods it is possible to overcome some limitations of traditional optimization methods, and to increase the number of solvable problems. The application of GEAs to many optimization problems in organizations often results in good performance and high quality solutions.
Full-text available
The issue of controlling values of various parameters of an evolutionary algorithm is one of the most important and promising areas of research in evolutionary computation: it has a potential of adjusting the algorithm to the problem while solving the problem. In the paper we: 1) revise the terminology, which is unclear and confusing, thereby providing a classification of such control mechanisms, and 2) survey various forms of control which have been studied by the evolutionary computation community in recent years. Our classification covers the major forms of parameter control in evolutionary computation and suggests some directions for further research