Statistical Analysis of Parameter Setting in
Real-Coded Evolutionary Algorithms
Maria I. Garc´ ıa Arenas, Pedro´Angel Castillo Valdivieso,
Antonio M. Mora Garc´ ıa, Juan J. Merelo Guerv´ os,
Juan L. Jim´ enez Laredo, and Pablo Garc´ ıa-S´ anchez
Department of Architecture and Computer Technology
University of Granada, Spain
Abstract. When evolutionary algorithm (EA) applications are being
developed it is very important to know which parameters have the great-
est influence on the behavior and performance of the algorithm. This pa-
per proposes using the ANOVA (ANalysis Of the VAriance) method to
carry out an exhaustive analysis of an EA method and the different pa-
rameters it requires, such as those related to the number of generations,
population size, operators application and selection type. When under-
taking a detailed statistical analysis of the influence of each parameter,
the designer should pay attention mostly to the parameter presenting
values that are statistically most significant. Following this idea, the sig-
nificance and relative importance of the parameters with respect to the
obtained results, as well as suitable values for each of these, were obtained
using ANOVA on four well known function optimization problems.
When using search heuristics such as evolutionary algorithms (EAs), simulated
annealing and local search algorithms, components such as genetic operators,
selection and replacement mechanisms, and the initial population, must first be
chosen. The parameters used to apply some of these elements determine the way
they operate and influence the results obtained. Obtaining suitable values for
them is an expensive, time-consuming and laborious task.
One of the most common ways of setting these parameters is by hand, af-
ter intensive experimentation with different values . As Eiben states, the
straightforward approach is to generate and test [8,28]. An alternative is to use
a meta-algorithm to optimise the parameters , that is, to run a higher level
algorithm that searches for an optimal and general set of parameters to solve a
wide range of optimisation problems.
However, as some authors remark, solving specific problems requires specific
parameter value sets [3,18,10] and, as Harik  claims, nobody knows the “op-
timal” parameter settings for an arbitrary real-world problem. Therefore, estab-
lishing the optimal set of parameters for a sufficiently general case is a difficult
R. Schaefer et al. (Eds.): PPSN XI, Part II, LNCS 6239, pp. 452–461, 2010.
c ? Springer-Verlag Berlin Heidelberg 2010
Statistical Analysis of Parameter Setting in Real-Coded EAs453
Current best practices are based on intensive test, ad-hoc choices and con-
ventions [29,8,28,9], that is why new practices, based on solid tuning methods
(i.e. robust mathematical methods), are needed. Such a methodology is what we
intend to present in this paper.
Genetic algorithm users adjust their main design parameters (crossover prob-
ability, mutation probability, population size, number of generations, selection
rate) by hand [6,19]. The decision as to which values are best is usually made
in terms of the most common values or experimental formulae given in the bib-
liography, or by trial and error [15,22].
However, other researchers have proposed determining a good set of evolu-
tionary algorithm parameters by analogy, undertaking a theoretical analysis
[2,13,14,16,25,31]. Establishing parameters by analogy means using suitable sets
of parameters to solve similar problems. However they do not explain how to
measure the similarity between problems. Also, a clear protocol has not been
proposed for situations when the similarity between problems implies that the
most suitable sets of parameters are also similar [18,10]. Weyland has described
a theoretical analysis of both an evolutionary algorithm  and simulated an-
nealing algorithm  to search for the optimal parameter setting to solve the
longest common subsequence problem. However, Weyland does not carry out
this approach to practice.
Some authors have proposed practical approaches that eliminate the need for
a parameter search in genetic algorithms . In these works, a set of parameters
is found, but instead of finding them by means of intense experimentation, the
parameter settings are backed up with theoretical work - meaning that these
settings are robust.
New approaches to the problem of establishing parameter values  have
been proposed. Several proposals are based on setting parameter values on-line
(during the run) instead of testing and comparing different values before the
run (parameter tuning). In this sense, some authors propose self-adaptation of
parameters (coding those parameters in the individual’s genome); others pro-
pose non-static parameter settings techniques (controlled by feedback from the
search and optimization process) [9,28,30]. However, control strategies also have
parameters and there are indicators that good tuning works better than control
Finally, authors proposed in  using the ANOVA (ANalysis Of the VAriance)
 statistical method to analyze the main parameters involved in the design of
a neuro-genetic algorithm.
It is very important to know which parameter values involved in the design
of an optimization method have the greatest influence on its behaviour and per-
formance and to obtain accurate values for those parameters. In any case, after
performing a detailed statistical analysis of the influence of each parameter, the
designer should pay attention mostly to the parameter providing the values that
are statistically most significant. In this paper, we propose using the ANOVA
statistical method as a powerful tool to analyze a real-coded EA to solve function
454M.I.G. Arenas et al.
The ANOVA method allows us to determine whether a change in the results
(responses) is due to a change in a parameter (variable or factor) or due to
a random effect. Thus it is possible to determine the variables that have the
greatest effect on the method that is being evaluated.
The theory and methodology of ANOVA was mainly developed by R.A. Fisher
during the 1920s . ANOVA examines the effects of one, two or more quan-
titative or qualitative variables (called factors) on one quantitative response.
ANOVA is useful in a range of disciplines when it is suspected that one or more
factors might affect a response. ANOVA is essentially a method used to anal-
yse the variance to which a response is subjected, dividing it into the various
components corresponding to the sources of variation, which can be identified.
With ANOVA, we test a null hypothesis that all the population means are
equal against the alternative hypothesis that there is at least one mean that is
not equal to the others. We find the sample mean and variance for each level
(value) of the main factor. Using these values, we obtain a significance value
(Sig. Level). If this level is lower than 0.05, then the influence of the factor is
statistically significant at the confidence level of 95%.
After applying ANOVA (to determine if means are different), another tests
must be used to determine which are different; that will give information about
which parameter values are more accurate. In this sense, either TukeyHDS
(Tukey’s Honestly Significant Difference)  or Bonferroni  tests can be used.
In this paper, ANOVA will be used to determine the most important param-
eters of an EA (in terms of their influence on the results), and to establish the
most suitable values for such parameters (thus obtaining an optimal operation).
The rest of this paper is structured as follows: Section 2 describes the EA and
the parameters we propose to be evaluated. This section contains an exhaustive
analysis of the method, showing how the parameters are determined. Section 3
details the experimental setup and the statistical study using ANOVA. Obtained
results are analysed in order to establish the most suitable values. Finally, a brief
conclusion and future work is presented in section 4.
2The EA Method and the Experimental Setup
The purpose of this study is to analyze the dynamics of a typical EA [3,11],
to determine which parameters influence the obtained fitness and to find an
adequate value for the following parameters:
– Generations (G): number of generations.
– Population Size (P): number of individuals in the population.
– Selector (S): Selection operator to generate the offspring. In this paper a
roulette wheel selector, a random selector and a selector based on always
taking the best individual in the popultation are proposed.
– Operators Combination (O): This parameter refers to the percentage of
offspring generated using either an uniform mutator or a BLX-α crossover
Statistical Analysis of Parameter Setting in Real-Coded EAs455
The response variable used to perform the statistical analysis is the fitness in the
last generation. The changes in the response variable are produced when a new
combination of parameters is considered.
Table 1. Parameters (factors) and the abbreviation used as reference later that deter-
mine the EA behaviour and values used to apply ANOVA
Generations (G) Population Operators
- Crossover only (C)
- Mutation only (M)
- Roulette Wheel
- Crossover (80%) and
Mut. (20%) (C8M2)- Always the best
- Crossover (90%) and
Mut. (10%) (C9M1)
Thus, 12960 runs were carried out for each problem (30 times * 6 levels for
G * 6 levels for P * 4 levels for O * 3 levels for S, that represent the possible
combinations) to obtain the fitness for each combination.
The application of ANOVA consisted in running an EA using these parameter
combinations to obtain the best fitness. Then R1was used to obtain the ANOVA
table. In this paper a simplified table is shown, including for each factor, the sum
of squares (Sum Sq), the value of the statistical F (F value) and its significance
level (Sig. Level). As previously stated, if the latter is smaller than 0.05, then
the factor effect is statistically significant at a 95% confidence level (what means
that some means are different for these parameter values).
In order to evaluate the EA and its parameters, four function approximation
problems are used:
– The Griewangk function  is a continuous multimodal function with a high
number of local optima. Its global optimum is located at (0,...,0) . This
problem has been used with vectors of 100 real numbers in the interval
– The Rastrigin function  is a multimodal real function optimisation prob-
lem, whose global optimum is located at point 0 and whose minimum value
is 0. This problem has been addressed with vectors of 100 real numbers in
the interval [−512,512].
– The NormalizedSchwefel function  is a multimodal separablerealfunction
optimisation problem, whose global optimum is located at point (x0,...,xd)
withxi= 420.96,andwhoseminimumvalueis(y0,...,yd)withyi= −418.9828.
Vectors of 100 real numbers in the interval [−512,512] were used.
– The Shekel function  is a multimodal real function optimisation problem,
whose optimum for dimension 5 is located at point x1 = 8.02, x2 = 9.15,
456M.I.G. Arenas et al.
x3= 5.11, x4= 7.62, x5= 4.56 and whose minimum value is −10.4056. This
problem has been addressed with vectors of 5 real numbers in the interval
In all cases, the fitness of an individual is calculated as the distance to the
optimum for that function (the optimum is known).
3 Statistical Study and Results Obtained
In this section, the ANOVA statistical tool is applied to determine whether the
influence on parameter values (factors) is significant in the obtained fitness, (to
obtain an optimal operation). The set of tests carried out to apply the ANOVA
method and thus to determine the most suitable parameter values, is described
in detail. In all cases, the goal is to obtain the smallest fitness for the optimised
Table 2. ANOVA tables for the fitness (response) with the EA parameters as factors.
Those parameters with a significance level over 95% are in bold. Although the full
ANOVA table includes combinations of 2, 3 and 4 parameters, only results related to
single parameters are shown.
Param. Sum Sq F value Pr(> F)
O4540135 154.4671 <2e−16
S 2781525 94.6346 <2e−16
Param. Sum Sq F value Pr(> F)
Param. Sum Sq F value Pr(> F)
O 52393 135.3265 <2.2e−16
S 36652 94.6682 <2.2e−16
Param. Sum Sq F value Pr(> F)
P 79.09 18.7111 1.9e−05
S118.79 28.1017 1.9e−07
Normalized Schwefel Shekel
We will determine if the influence on a parameter value is significant in the
value of the approximation function (fitness).
Table 2 shows the result of applying ANOVA on proposed approximation
function problems. Parameters with a signification level over 95% are highlighted
in boldface. The ANOVA analysis shows that O and S parameters influence the
obtained fitness, which indicates that changes in these parameters influence the
results significantly. However, this influence is not as important for the rest of
the parameters in all the cases (problems).
Statistical Analysis of Parameter Setting in Real-Coded EAs457
Table 3. Griewangk function: obtained error for the different parameter levels. This
table shows the effect each level has on the fitness.
160 320 640 1280 2560 5120
129.5 95.3 81.6 74.3 74.9 70.2
289.0 57.8 1.7
100 200 400 800 1600 3200
104.5 93.1 91.1 83.1 79.1 75.0
Table 4. Rastrigin function: obtained error for the different parameter levels. This
table shows the effect each level has on the fitness.
525797 387416 321216 296155 288133 295020
419618 382480 359795 337218 316751 297876
In Normalized Schwefel and Shekel, the P parameter is significant too. This
fact shows how for each problem different set of parameters can influence results
in a different manner.
Once the parameters with greater influence on the results are determined,
accurate parameter values should be established in order to obtain an optimal
operation. To do so, tables of means are calculated to show the effect each level
has on the approximation error.
The obtained error for the Griewangk, Rastrigin, Normalized Schwefel and
Shekel functions and the different parameter levels are shown in Tables 3, 4, 5
Lets examine each one of the parameters in turn:
– Focusing attention to the operator combinations (O), using either only mu-
tation or only crossover leads to worse fitness results, which was only to be
expected. According to the tables, using a low crossover and high mutation
probabilities to generate offspring in each generation is the more accurate.
However, as the mutation role is to generate diversity, reducing too much
the mutation probability leads to premature convergence of the population.
458M.I.G. Arenas et al.
Table 5. Normalized Schwefel function: obtained error for the different parameter
levels. This table shows the effect each level has on the fitness.
8.146208 7.848331 8.151095 9.062385 8.641955 7.645236
Table 6. Shekel function: obtained error for the different parameter levels. This table
shows the effect each level has on the fitness.
160 320 640 1280 2560 5120
7.32 6.90 6.78 6.70 6.64 6.63
8.10 4.14 7.49
100 200 400 800 1600 3200
7.57 7.30 6.94 6.65 6.34 6.19
On the other hand, applying too much mutation affects exploration, leading
to random search.
– Both P and G have not been reported as significant according to the ANOVA
table in all problems. However, as it can be observed, using the higher values
yields better fitness values, although those values lead to a higher number of
evaluations and time needed to run the algorithm. Logically, the greater the
number of generations or population size, the more possibilities there are of
achieving a good individual from the current population, as there exists a
greater variety of individuals.
– Taking into account the S parameter, using the roulette wheel selector yields
much better results than using a random selector or always taking the best
individual. The reason for this is that the roulette selector produces the
greatest diversification in the EA solutions. The third selector is the most
elitist; obtained results using this one selector (taking the best individual)
are much worse, which indicates that too much selective presion is not appro-
priate (the population must be diverse, otherwise a premature convergence
of the algorithm might occur).
Statistical Analysis of Parameter Setting in Real-Coded EAs459
These conclusions have been confirmed by applying the Bonferroni statistical
test . As can be seen, these results are in agreement with those available in
bibliography. In any case, in this paper we have verified the adequacy of these
parameter values through a rigorous statistical study.
4Conclusions and Work in Progress
A statistical study of the different parameters involved in the design of an EA
has been carried out by using ANOVA, which consists of a set of statistical
techniques that analyze and compare experiments by describing the interactions
and interrelations between the variables (factors) of the system. The motivation
of the present statistical study lies in the great variety of alternatives that a
designer has to take into account when designing an EA.
Proposed methodology has been applied to four well known function approxi-
mation problems, widely used by practitioners, having determined which param-
eters have a higher influence on obtained results (a change on those parameters
will affect the fitness). Likewise, the more accurate value has been determined
for those parameters in order to obtain an optimal operation.
Accurate results are obtained using the higher value tested for population size
and number of generations, although this increases the number of evaluations
and time needed to run the algorithm. Obviously, as the number of generations
or individuals in the population increases, there is a greater probability that the
fitness of the best individual will be higher.
Paying attention to the selector operator, a roulette wheel selector yields much
better results than using a random selector or always taking the best individual,
due to the fact that the roulette selector produces the greatest diversification in
the EA solutions.
As far as the genetic operator application rates are concerned, reducing too
much the mutation probability leads to premature convergenceof the population,
while applying too much mutation is like a random search. According to the
tables, using a low mutation and high crossover probabilities to generate offspring
in each generation is the more accurate.
This methodology based on ANOVA and Bonferroni statistical tests could be
helpful for practitioners in analyzing and adjusting parameters of any optimisa-
Our work in progress includes the analysis of modified EA considering other
selection schemes, new genetic operators and other meta-heuristics. As future
work, the implementation of a parameter control method would be of interest,
as proposed by Eiben et al. . In this case, ANOVA could be used to anal-
yse not only the optimisation method parameters but also the control strategy
This work has been supported in part by the Spanish MICYT project NoHNES
(Spanish Ministerio de Educacion y Ciencia - TIN2007-68083), the Junta de
460 M.I.G. Arenas et al.
Andaluc´ ıa P06-TIC-02025,P07-TIC-03044,P08-TIC03928projects and the Ja´ en
University UJA-08-16-30 project.
1. Griewank, A.O.: Generalized descent for global optimization. Journal of Optimiza-
tion Theory and Applications 34(1), 11–39 (1981)
2. B¨ ack, T.: Optimal mutation rates in genetic search. In: Forrest, S. (ed.) Proceed-
ings of the 5th International Conference on Genetic Algorithms, pp. 2–8. Morgan
Kaufmann, San Francisco (1993)
3. B¨ ack, T., Fogel, D., Michalewicz, Z.: Handbook of Evolutionary Computation.
Institute of Physics Publishing Ltd. Bristol and Oxford University Press, NY (1997)
4. Castillo, P.A., Merelo, J.J., Romero, G., Prieto, A., Rojas, I.: Statistical Analysis
of the Parameters of a Neuro-Genetic Algorithm. IEEE Trans. on Neural Net-
works 13(6), 1374–1394 (2002) ISSN:1045-9227
5. Cho, H., Olivera, F., Guikema, S.D.: A derivation of the number of minima of the
griewank function. Applied Mathematics and Computation 204(2), 694–701 (2008)
6. Davis, L.: Handbook of genetic algorithms. Van Nostrpand Reinhold, NY (1991)
7. Dickinson, P., Chow, B.: Some properties of the Tukey test to Duckworth’s spec-
ification by Peter Dickinson, Bryant Chow. Office of Institutional Research, Uni-
versity of Southwestern Louisiana, Lafayette, Louisiana, USA (1971)
8. Eiben, A.E.: Principled Approaches to tuning EA parameters. In: Tutorials - IEEE
Congress on Evolutionary Computation (CEC 2009) (2009), http://www.few.vu.
9. Eiben, A.E., Michalewicz, Z., Schoenauer, M., Smith, J.E.: Parameter control in
evolutionary algorithms. In: Parameter Setting in Evolutionary Algorithms. Stud-
ies in Computational Intelligence, vol. 54, pp. 19–46. Springer, Heidelberg (2007)
ISBN 978-3-540-69431-1, ISSN 1860-949X
10. Eiben, A.E., Hinterding, R., Michalewicz, Z.: Parameter Control in Evolution-
ary Algorithms. IEEE Transactions on Evolutionary Computation 3(2), 124–141
(1999), doi:10.1109/4235.771166, ISSN: 1089-778X
11. Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Springer, Hei-
delberg (2003) ISBN 3-540-40184-9
12. Fisher, R.A.: Theory of Statistical Estimation. In: Proceedings of the Cambridge
Philosophical Society, vol. 22, pp. 700–725 (1925)
13. Goldberg, D.E.: Genetic Algorithms in Search, Optimization and Machine Learn-
ing. Addison-Wesley Longman Publishing Co., Inc., Boston (1989)
14. Goldberg, D.E., Deb, K., Theirens, D.: Toward a better understanding of mixing in
genetic algorithms. In: Belew, R.K., Booker, L.B. (eds.) Proc. of the 4th Int. Conf.
on Genetic Algorithms, pp. 190–195. Morgan Kaufmann, San Francisco (1991)
15. Grefenstette, J.J.: Optimization of control parameters for genetic algorithms. IEEE
Trans. Systems, Man, and Cybernetics, SMC- 16(1), 122–128 (1986)
16. Harik, G., Cant´ u-Paz, E., Goldberg, D.E., Miller, B.L.: The gambler’s ruin prob-
lem, genetic algorithms, and the sizing of populations. In: Proceedings of the 4th
IEEE Conference on Evolutionary Computation, pp. 7–12. IEEE Press, Los Alami-
17. Harik, G.R., Lobo, F.G.: A parameter-less genetic algorithm. In: Banzhaf, W.,
Daida, J., Eiben, A.E., Garzon, M.H., Honavar, V., Jakiela, M., Smith, R.E. (eds.)
Proceedings of the Genetic and Evolutionary Computation Conference, Orlando,
Florida, USA, 13-17, vol. 1, pp. 258–265. Morgan Kaufmann, San Francisco (1999)
Statistical Analysis of Parameter Setting in Real-Coded EAs 461
18. Hinterding, R., Michalewicz, Z., Eiben, A.E.: Adaptation in Evolutionary Com-
putation: A Survey. In: Proceedings of the 4th IEEE Conference on Evolutionary
Computation, pp. 65–69. IEEE Press, Los Alamitos (1997)
19. Jagielska, I., Matthews, C., Whitfort, T.: An investigation into the application
of neural networks, fuzzy logic, genetic algorithms, and rough sets to automated
knowledge acquisition problems. Neurocomputing 24(1-3), 37–54 (1999)
20. Jansen, T., Weyland, D.: Analysis of evolutionary algorithms for the longest com-
mon subsequence problem. In: GECCO 2007: Proceedings of the 9th Annual Con-
ference on Genetic and Evolutionary Computation, pp. 939–946. ACM, New York
21. De Jong, K.A.: An analysis of the behavior of a class of genetic adaptive systems.
Ph.D. thesis, University of Michigan, Ann Arbor (1975)
22. Kim, D., Kim, C.: Forecasting time series with genetic fuzzy predictor ensemble.
IEEE Transactions on Fuzzy Systems 5(4), 523–535 (1997)
23. Lobo, F.G., Lima, C.F., Michalewicz, Z.: Parameter Setting in Evolutionary Algo-
rithms. Springer Publishing Company, Incorporated, Heidelberg (2007)
24. Savin, N.E.: The bonferroni and the scheff multiple comparison procedures. Review
of Economic Studies (XLVII), pp. 255–273 (1980)
25. Schaffer, J.D., Morishima, A.: An adaptive crossover distribution mechanism for
genetic algorithms. In: Grefenstette, J.J. (ed.) Proceedings of the 2nd International
Conference on Genetic Algorithms and Their Applications, pp. 36–40. Lawrence
Erlbaum Associates, Mahwah (1987)
26. Schwefel, H.-P.: Numerical Optimization of Computer Models. John Wiley & Sons,
Inc., New York (1981)
27. Shekel, J.: Test functions for multimodal search techniques. In: Fifth Annual
Princeton Conference on Information Science and Systems, pp. 354–359 (1971)
28. Smit, S.K., Eiben, A.E.: Comparing parameter tuning methods for evolutionary
algorithms. In: Haddow, P., et al. (eds.) CEC 2009: Proc. of the Eleventh conference
on Congress on Evolutionary Computation, Piscataway, NJ, USA, pp. 399–406.
IEEE Press, Los Alamitos (2009)
29. Smit, S.K., Eiben, A.E.: Using Entropy for Parameter Analysis of Evolutionary
Algorithms. In: Beielstein, B., et al. (eds.) Empirical Methods for the Analysis of
Optimization Algorithms. Natural Computing Series. Springer, Heidelberg (2009)
30. Smit, S.K., Eiben, A.E.: Parameter tuning of evolutionary algorithms: Generalist
vs. specialist. In: Di Chio, C., et al. (eds.) EvoApplicatons 2010. LNCS, vol. 6024,
pp. 542–551. Springer, Heidelberg (2010)
31. Thierens, D.: Dimensional analysis of allele-wise mixing revisited. In: Ebeling, W.,
Rechenberg, I., Voigt, H.-M., Schwefel, H.-P. (eds.) PPSN 1996. LNCS, vol. 1141,
pp. 255–265. Springer, Heidelberg (1996)
32. Torn, A., Zilinskas, A.: Global Optimization. LNCS, vol. 350, p. 124. Springer,
33. Weyland, D.: Simulated annealing, its parameter settings and the longest common
subsequence problem. In: GECCO 2008: Proceedings of the 10th Annual Confer-
ence on Genetic and Evolutionary Computation, pp. 803–810. ACM, New York