![Computational effort for even-parity problems found by Koza [19]](https://www.researchgate.net/profile/Lee-Spector-2/publication/225173354/figure/tbl1/AS:393805959319555@1470902065153/Computational-effort-for-even-parity-problems-found-by-Koza-19_Q320.jpg)
Content uploaded by Lee Spector
Author content
All content in this area was uploaded by Lee Spector on Mar 26, 2016
Content may be subject to copyright.
A preview of the PDF is not available
Genetic Programming and Autoconstructive Evolution with the Push Programming Language
Abstract and Figures
Push is a programming language designed for the expression of evolving programs within an evolutionary computation system. This article describes Push and illustrates some of the opportunities that it presents for evolutionary computation. Two evolutionary computation systems, PushGP and Pushpop, are described in detail. PushGP is a genetic programming system that evolves Push programs to solve computational problems. Pushpop, an autoconstructive evolution system, also evolves Push programs but does so while simultaneously evolving its own evolutionary mechanisms.
Figures - uploaded by Lee Spector
Author content
All figure content in this area was uploaded by Lee Spector
Content may be subject to copyright.
... This implies that the potential performance improvement that can be gained from optimizing parameters is always limited by the algorithm that is chosen. To limit the bias imposed by the choice of a specific (evolutionary) algorithm, researchers have proposed to instead evolve the structure of an EA itself (e.g., [17], [16], [27]). ...
... In those studies, varying numbers of metalevels are used, each responsible for evolving the program used in the level below and potentially itself. The Push language [27] was later dedicated to enable autoconstructive evolution whereby a population of programs is expected to produce other programs. ...
Various variants of the well known Covariance Matrix Adaptation Evolution Strategy (CMA-ES) have been proposed recently, which improve the empirical performance of the original algorithm by structural modifications. However, in practice it is often unclear which variation is best suited to the specific optimization problem at hand. As one approach to tackle this issue, algorithmic mechanisms attached to CMA-ES variants are considered and extracted as functional \emph{modules}, allowing for combinations of them. This leads to a configuration space over ES structures, which enables the exploration of algorithm structures and paves the way toward novel algorithm generation. Specifically, eleven modules are incorporated in this framework with two or three alternative configurations for each module, resulting in algorithms. A self-adaptive Genetic Algorithm (GA) is used to efficiently evolve effective ES-structures for given classes of optimization problems, outperforming any classical CMA-ES variants from literature. The proposed approach is evaluated on noiseless functions from BBOB suite. Furthermore, such an observation is again confirmed on different function groups and dimensionality, indicating the feasibility of ES configuration on real-world problem classes.
... In addition, GPU solutions have been created for grammar-guided GP [20] and Tangled Program Graphs [132]. For stack-based GP, at least one GPU study has been conducted [101], although this was for simple stack-based GP [88], and not for a more complex system like PushGP [109]. Finally, we note that the application area of evolvable hardware [129,130] has leveraged FPGA devices, although this has been with the primary intention of evolving circuits, rather than accelerating the GP procedure via a single circuit, which is the purpose of our architecture presented in Sect. 4. ...
This paper establishes the potential of accelerating the evaluation phase of tree-based genetic programming through contemporary field-programmable gate array (FPGA) technology. This exploration stems from the fact that FPGAs can sometimes leverage increased levels of both data and function parallelism, as well as superior power/energy efficiency, when compared to general-purpose CPU/GPU systems. In this investigation, we introduce a fixed-depth, tree-based architecture that can fully parallelize tree evaluation for type-consistent primitives that are unrolled and pipelined. We show that our accelerator on a 14nm FPGA achieves an average speedup of 43 when compared to a recent open-source GPU solution, TensorGP, implemented on 8nm process-node technology, and an average speedup of 4,902 when compared to a popular baseline GP software tool, DEAP, running parallelized across all cores of a 2-socket, 28-core (56-thread), 14nm CPU server. Despite our single-FPGA accelerator being 2.4 slower on average when compared to the recent state-of-the-art Operon tool executing on the same 2-processor, 28-core CPU system, we show that this single-FPGA system is 1.4 better than Operon in terms of performance-per-watt. Importantly, we also describe six future extensions that could provide at least a 64–192 speedup over our current design. Therefore, our initial results provide considerable motivation for the continued exploration of FPGA-based GP systems. Overall, any success in significantly improving runtime and energy efficiency could potentially enable novel research efforts through faster and/or less costly GP runs, similar to how GPUs unlocked the power of deep learning during the past fifteen years.
... Compared to standard GP, which requires the evaluation of a specialized interpreter for each candidate program, GSGP mutation is performed by a linear combination of semantic vectors, making the evolutionary loop easy to parallelize on a GPU. A notable aspect of GSGP-CUDA is that it does not use trees to represent models, it uses a linear representation and a stack-based interpreter that is similar to PushGP [40]. While [5,7] use a tree representation, GSM and GSC do not require or depend on it. ...
Geometric Semantic Genetic Programming (GSGP) reimagined how to search for symbolic models using an evolutionary process. It addressed one of the main weaknesses of standard Genetic Programming (GP) by performing the search directly within the semantic space of a problem, which can define a convex and unimodal fitness landscape. Since the method does not require the syntactic evaluation of offspring, GSGP allowed more efficient and effective learning systems compared to standard GP. However, recent benchmarking results have suggested that GSGP is no longer a competitive approach, particularly in the symbolic regression domain, despite its previous success in several real-world tasks. Therefore, the research question of this work is an empirical one, stated as: Is GSGP still a competitive symbolic regression method 10 years after it was proposed? A comprehensive benchmark of black-box problems and comparisons with state-of-the-art methods were used to answer this question. In particular, a recently developed parallel version of GSGP is used, extending the implementation to also include the previously proposed optimal mutation step computation, as well as using the analytical quotient operator instead of a protected division. Results show that with simple, but important, extensions to the original GSGP algorithm, the answer to the research question is yes.
... We use the PushGP system to evolve solutions to the problems described above. PushGP is a GP system that evolves programs written in the Push programming language [25,28]. The Push programming language is a stack-based language that facilitates the use of multiple data types and complex programming paradigms such as iteration and recursion [27]. ...
Lexicase selection is a semantic-aware parent selection method, which assesses individual test cases in a randomly-shuffled data stream. It has demonstrated success in multiple research areas including genetic programming, genetic algorithms, and more recently symbolic regression and deep learning. One potential drawback of lexicase selection and its variants is that the selection procedure requires evaluating training cases in a single data stream, making it difficult to handle tasks where the evaluation is computationally heavy or the dataset is large-scale, e.g., deep learning. In this work, we investigate how the weighted shuffle methods can be employed to improve the efficiency of lexicase selection. We propose a novel method, fast lexicase selection, which incorporates lexicase selection and weighted shuffle with partial evaluation. Experiments on both classic genetic programming and deep learning tasks indicate that the proposed method can significantly reduce the number of evaluation steps needed for lexicase selection to select an individual, improving its efficiency while maintaining the performance.
Context: Program synthesis tools reduce software development costs by generating programs that perform tasks depicted by some specifications. Various methodologies have emerged for program synthesis, among which search-based algorithms have shown promising results. However, the proliferation of search-based program synthesis tools utilising diverse search algorithms and input types and targeting various programming tasks can overwhelm users seeking the most suitable tool. Objective: This paper contributes to the ongoing discourse by presenting a comprehensive review of search-based approaches employed for program synthesis. We aim to offer an understanding of the guiding principles of current methodologies by mapping them to the required type of user intent, the type of search algorithm, and the representation of the search space. Furthermore, we aim to map the diverse search algorithms to the type of code generation tasks in which they have shown success, which would serve as a guideline for applying search-based approaches for program synthesis. Method: We conducted a literature review of 67 academic papers on search-based program synthesis. Results: Through analysis, we identified and categorised the main techniques with their trends. We have also mapped and shed light on patterns connecting the problem, the representation and the search algorithm type. Conclusions: Our study summarises the field of search-based program synthesis and provides an entry point to the acumen and expertise of the search-based community on program synthesis.
In the past, field-programmable gate arrays (FPGAs) have had some notable successes when employed for Boolean and fixed-point genetic programming (GP) systems, but the more common floating-point representations were largely off limits, due to a general lack of efficient device support. However, recent work suggests that for both the training and inference phases of floating-point-based GP, contemporary FPGA technologies may enable significant performance and energy improvements—potentially multiple orders of magnitude—when compared to general-purpose CPU/GPU devices. In this chapter, we highlight the potential advantages and challenges of using FPGAs for GP systems, and we showcase how novel algorithmic considerations likely need to be made in order to extract the most benefits from specialized hardware. Primarily, we consider tree-based GP, although we include suggestions for other program representations. Overall, we conclude that the GP community should earnestly revisit the use of FPGA devices, especially the tailoring of state-of-the-art algorithms to FPGAs, since valuable enhancements may be realized. Most notably, FPGAs may allow for faster and/or less costly GP runs, in which case it may also be possible for better solutions to be found when allowing an FPGA to consume the same amount of runtime/energy as another platform.
This article compares basic genetic programming, genetic programming with automatically defined functions (ADFs), and genetic programming with ADFs using a restricted form of recursion on a planning problem involving tree search. The results show that evolution of a recursive program is possible and further that, of the three techniques explored, genetic programming with recursive ADFs performs the best for the tree search problem. Additionally, genetic programming using ADFs (recursive and non-recursive) outperforms genetic programming without ADFs. The scalability of these techniques is also investigated. The computational effort required to reach a solution using ADFs with recursion is shown to remain essentially constant with world size, while genetic programming with non-recursive ADFs scales linearly at best, and basic genetic programming scales exponentially. Finally, many solutions were found using genetic programming with recursive ADFs which generalized to any world size. Iterative structures are an essential component in programming. Iteration and recursion lead to more compact programs and, in many cases, facilitate generalization. Since genetic programming is a genetic algorithm which evolves programs, iterative and recursive capability is desirable. Additionally, the ability to create simpler programs means that an evolved solution may take less computational effort to find. The evolution of iteration and recursion using genetic programming was first explored by Koza(1992). To allow the evolution of iterative solutions to the block stacking problem, Koza introduced the Do Until (DU) operator. The DU operator takes two arguments: the first specifies the work to be iteratively executed, and the second specifies a predicate which when satisfied will end the iteration. To prevent infinite loops, an upper limit on the number of iterations is externally imposed. Variations of this form of restricted iteration have proven useful in several problem domains including protein sequencing and optical character recognition (Koza 1994), (Andre 1994a), (chapter 8). Koza(1992) also investigates a very limited form of recursion for sequence induction. Programs capable of producing the Fibonacci sequence are achieved by allowing the S- expression to reference previously computed values in the sequence. Although much work has been done in the synthesis of recursive functional programs using other machine learning methods (see (Banerjee 1987), (Hutchinson 1994)), little has been done to explore the evolution of recursion using genetic programming (see (Nordin 1995)). This article investigates a method for evolving recursive programs for tree search. Many techniques used in AI systems, including those for game-playing and planning, depend on an agent's ability to search tree structures (Nilsson 1971), but are not optimal. These systems are thus prime candidates for genetic programming. Similar to restricted
Genetic programming is a powerful method for automatically generating computer programs via the process of natural selection (Koza, 1992). However, in its standard form, there is no way to restrict the programs it generates to those where the functions operate on appropriate data types. In the case when the programs manipulate multiple data types and contain functions designed to operate on particular data types, this can lead to unnecessarily large search times and/or unnecessarily poor generalization performance. Strongly typed genetic programming (STGP) is an enhanced version of genetic programming that enforces data-type constraints and whose use of generic functions and generic data types makes it more powerful than other approaches to type-constraint enforcement. After describing its operation, we illustrate its use on problems in two domains, matrix/vector manipulation and list manipulation, which require its generality. The examples are (1) the multidimensional least-squares regression problem, (2) the multidimensional Kalman filter, (3) the list manipulation function NTH, and (4) the list manipulation function MAPCAR.
Genetic Programming (GP) automatically generates computer programs to solve specified
problems. It develops programs through the process of a “create-test-modify” cycle which is
similar to the way a human writes programs. There are various functional programming techniques
that human programmers can use to accelerate the program development process. This
research investigated the applicability of some of the functional techniques to GP and analyzed
their impact on GP performance.
Among many important functional techniques, three were chosen to be included in this
research, due to their relevance to GP. They are polymorphism, implicit recursion and higher-order
functions. To demonstrate their applicability, a GP system was developed with those
techniques incorporated. Furthermore, a number of experiments were conducted using the
system. The results were then compared to those generated by other GP systems which do not
support these functional features. Finally, the program search space of the general even-parity
problem was analyzed to explain how these techniques impact GP performance.
The experimental results showed that the investigated functional techniques have made
GP more powerful in the following ways: 1) polymorphism has enabled GP to solve problems
that are very difficult for standard GP to solve, i.e. nth and map programs; 2) higher-order
functions and implicit recursion have enhanced GP’s ability in solving the general even-parity
problem to a greater degree than with any other known methods. Moreover, the
analysis showed that these techniques directed GP to generate program solutions in a way that
has never been previously reported. Finally, we provide the guidelines for the application of
these techniques to other problems.
Introduction: Adaptive and Self-Adaptive Evolutionary ComputationsEvolutionary computations have proven to be powerful yet general methods for searchand optimization. As with most search and optimization techniques, evolutionarycomputations include a number of operational parameters whose values significantly alterthe behavior of the algorithm on a given problem, and usually in unpredictable ways.Typically, the optimal parameter values are not only problem dependent but population...