Kenneth O. Stanley

Kenneth O. Stanley
University of Central Florida | UCF · Department of Electrical Engineering & Computer Science

About

205
Publications
57,231
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
14,692
Citations

Publications

Publications (205)
Preprint
This paper pursues the insight that large language models (LLMs) trained to generate code can vastly improve the effectiveness of mutation operators applied to programs in genetic programming (GP). Because such LLMs benefit from training data that includes sequential changes and modifications, they can approximate likely changes that humans would m...
Article
Full-text available
Reinforcement learning promises to solve complex sequential-decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse1 and deceptive2 feedback. Avoiding these pitfalls requires a thorough exploration of t...
Preprint
Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired by how biological motifs such as cells a...
Preprint
The promise of reinforcement learning is to solve complex sequential decision problems by specifying a high-level reward function only. However, RL algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse and deceptive feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but despite sub...
Preprint
Recent advances in machine learning are consistently enabled by increasing amounts of computation. Reinforcement learning (RL) and population-based methods in particular pose unique challenges for efficiency and flexibility to the underlying distributed computing frameworks. These challenges include frequent interaction with simulations, the need f...
Preprint
Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and al...
Preprint
The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and ada...
Preprint
Continual lifelong learning requires an agent or model to learn many sequentially ordered tasks, building on previous knowledge without catastrophically forgetting it. Much work has gone towards preventing the default tendency of machine learning models to catastrophically forget, yet virtually all such work involves manually-designed solutions to...
Article
Evolution provides a creative fount of complex and subtle adaptations that often surprise the scientists who discover them. However, the creativity of evolution is not limited to the natural world: Artificial organisms evolving in computational environments have also elicited surprise and wonder from the researchers studying them. The process of ev...
Preprint
Evolutionary-based optimization approaches have recently shown promising results in domains such as Atari and robot locomotion but less so in solving 3D tasks directly from pixels. This paper presents a method called Deep Innovation Protection (DIP) that allows training complex world models end-to-end for such 3D environments. The main idea behind...
Preprint
This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to...
Preprint
[***** N.B. The full paper is available open access at https://arxiv.org/abs/1909.04430 *****] Nature's spectacular inventiveness, reflected in the enormous diversity of form and function displayed by the biosphere, is a feature of life that distinguishes living most strongly from nonliving. It is, therefore, not surprising that this aspect of life...
Conference Paper
Designing evolutionary algorithms capable of uncovering highly evolvable representations is an open challenge in evolutionary computation; such evolvability is important in practice, because it accelerates evolution and enables fast adaptation to changing circumstances. This paper introduces evolvability ES, an evolutionary algorithm designed to ex...
Conference Paper
Minimal criterion coevolution (MCC) was recently introduced to show that a very simple criterion can lead to an open-ended expansion of two coevolving populations. Inspired by the simplicity of striving to survive and reproduce in nature, in MCC there are few of the usual mechanisms of quality diversity algorithms: no explicit novelty, no fitness f...
Conference Paper
Neural architectures inspired by our own human cognitive system, such as the recently introduced world models, have been shown to outperform traditional deep reinforcement learning (RL) methods in a variety of different domains. Instead of the relatively simple architectures employed in most RL experiments, world models rely on multiple different n...
Conference Paper
How can progress in machine learning and reinforcement learning be automated to generate its own never-ending curriculum of challenges without human intervention? The recent emergence of quality diversity (QD) algorithms offers a glimpse of the potential for such continual open-ended invention. For example, novelty search showcases the benefits of...
Preprint
Designing evolutionary algorithms capable of uncovering highly evolvable representations is an open challenge; such evolvability is important because it accelerates evolution and enables fast adaptation to changing circumstances. This paper introduces evolvability ES, an evolutionary algorithm designed to explicitly and efficiently optimize for evo...
Article
[***** The full paper is available open access at https://arxiv.org/abs/1909.04430 *****] Nature's spectacular inventiveness, reflected in the enormous diversity of form and function displayed by the biosphere, is a feature of life that distinguishes living most strongly from nonliving. It is, therefore, not surprising that this aspect of life shou...
Preprint
Neural architectures inspired by our own human cognitive system, such as the recently introduced world models, have been shown to outperform traditional deep reinforcement learning (RL) methods in a variety of different domains. Instead of the relatively simple architectures employed in most RL experiments, world models rely on multiple different n...
Article
Full-text available
Nature's spectacular inventiveness, reflected in the enormous diversity of form and function displayed by the biosphere, is a feature of life that distinguishes living most strongly from nonliving. It is, therefore, not surprising that this aspect of life should become a central focus of artificial life. We have known since Darwin that the diversit...
Preprint
A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to im...
Article
Full-text available
Much of recent machine learning has focused on deep learning, in which neural network weights are trained through variants of stochastic gradient descent. An alternative approach comes from the field of neuroevolution, which harnesses evolutionary algorithms to optimize neural networks, inspired by the fact that natural brains themselves are the pr...
Preprint
While the history of machine learning so far encompasses a series of problems posed by researchers and algorithms that learn their solutions, an important question is whether the problems themselves can be generated by the algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricu...
Conference Paper
Recent advances in deep neuroevolution have demonstrated that evolutionary algorithms, such as evolution strategies (ES) and genetic algorithms (GA), can scale to train deep neural networks to solve difficult reinforcement learning (RL) problems. However, it remains a challenge to analyze and interpret the underlying process of neuroevolution in su...
Conference Paper
While neuroevolution (evolving neural networks) has been successful across a variety of domains from reinforcement learning, to artificial life, to evolutionary robotics, it is rarely applied to large, deep neural networks. A central reason is that while random mutation generally works in low dimensions, a random perturbation of thousands or millio...
Conference Paper
An evolution strategy (ES) variant based on a simplification of a natural evolution strategy recently attracted attention because it performs surprisingly well in challenging deep reinforcement learning domains. It searches for neural network parameters by generating perturbations to the current set of parameters, checking their performance, and mo...
Preprint
Recent advances in deep neuroevolution have demonstrated that evolutionary algorithms, such as evolution strategies (ES) and genetic algorithms (GA), can scale to train deep neural networks to solve difficult reinforcement learning (RL) problems. However, it remains a challenge to analyze and interpret the underlying process of neuroevolution in su...
Article
How can we build agents that keep learning from experience, quickly and efficiently, after their initial training? Here we take inspiration from the main mechanism of learning in biological brains: synaptic plasticity, carefully tuned by evolution to produce efficient lifelong learning. We show that plasticity, just like connection weights, can be...
Article
Full-text available
Biological evolution provides a creative fount of complex and subtle adaptations, often surprising the scientists who discover them. However, because evolution is an algorithmic process that transcends the substrate in which it occurs, evolution's creativity is not limited to nature. Indeed, many researchers in the field of digital evolution have o...
Article
Full-text available
An evolution strategy (ES) variant recently attracted significant attention due to its surprisingly good performance at optimizing neural networks in challenging deep reinforcement learning domains. It searches directly in the parameter space of neural networks by generating perturbations to the current set of parameters, checking their performance...
Article
Full-text available
While neuroevolution (evolving neural networks) has a successful track record across a variety of domains from reinforcement learning to artificial life, it is rarely applied to large, deep neural networks. A central reason is that while random mutation generally works in low dimensions, a random perturbation of thousands or millions of weights is...
Article
Full-text available
Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because...
Article
Full-text available
Because stochastic gradient descent (SGD) has shown promise optimizing neural networks with millions of parameters and few if any alternatives are known to exist, it has moved to the heart of leading approaches to reinforcement learning (RL). For that reason, the recent result from OpenAI showing that a particular kind of evolution strategy (ES) ca...
Article
Full-text available
Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed explo...
Conference Paper
The fields of artificial life and evolutionary robotics have seen growing interest in evolution as a source of creativity, as opposed to a tool for optimization. New intentionally divergent algorithms such as novelty search with local competition (NSLC) and MAP-Elites accordingly attempt to harness evolution's aptitude for divergence in a new searc...
Article
Full-text available
Natural evolution has produced a tremendous diversity of functional organisms. Many believe an essential component of this process was the evolution of evolvability, whereby evolution speeds up its ability to innovate by generating a more adaptive pool of offspring. One hypothesized mechanism for evolvability is developmental canalization, wherein...
Article
Full-text available
Biological neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifetime learning. The interplay of these elements leads to the emergence of adaptive behavior and intelligence, but the complexity of the whole system of interactions is an obstacle to the understanding of the key factors at pla...
Conference Paper
Inspired by natural evolution’s affinity for discovering a wide variety of successful organisms, a new evolutionary search paradigm has emerged wherein the goal is not to find the single best solution but rather to collect a diversity of unique phenotypes where each variant is as good as it can be. These quality diversity (QD) algorithms therefore...
Article
Full-text available
An ambitious goal in evolutionary robotics (ER) is to evolve increasingly complex robotic behaviors with minimal human design effort. Reaching this goal requires evolutionary algorithms that can unlock from genetic encodings their latent potential for evolvability. One issue clouding this goal is conceptual confusion about evolvability that often o...
Article
Full-text available
We present a survey of the first 21 years of web-based artificial life (WebAL) research and applications, broadly construed to include the many different ways in which artificial life and web technologies might intersect. Our survey covers the period from 1994-when the first WebAL work appeared-up to the present day, together with a brief discussio...
Article
Full-text available
We describe the content and outcomes of the First Workshop on Open-Ended Evolution: Recent Progress and Future Milestones (OEE1), held during the ECAL 2015 conference at the University of York, UK, in July 2015. We briefly summarize the content of the workshop's talks, and identify the main themes that emerged from the open discussions. Two importa...
Conference Paper
While evolutionary algorithms (EAs) have long offered an alternative approach to optimization, in recent years backpropagation through stochastic gradient descent (SGD) has come to dominate the fields of neural network optimization and deep learning. One hypothesis for the absence of EAs in deep learning is that modern neural networks have become s...
Conference Paper
In a departure from conventional optimization where the goal is to find the best possible solution, a new class of evolutionary algorithms instead search for quality diversity (QD) -- a maximally diverse collection of individuals in which each member is as high-performing as possible. In QD, diversity of behaviors or phenotypes is defined by a beha...
Article
Full-text available
While evolutionary computation and evolutionary robotics take inspiration from nature, they have long focused mainly on problems of performance optimization. Yet evolution in nature can be interpreted as more nuanced than a process of simple optimization. In particular, natural evolution is a divergent search that optimizes locally within each nich...
Article
This article presents a lightweight platform for evolving two-dimensional artificial creatures. The aim of providing such a platform is to reduce the barrier to entry for researchers interested in evolving creatures for artificial life experiments. In effect the novel platform, which is inspired by the Sodarace construction set, makes it easy to se...
Conference Paper
In contrast to the conventional role of evolution in evolutionary computation (EC) as an optimization algorithm, a new class of evolutionary algorithms has emerged in recent years that instead aim to accumulate as diverse a collection of discoveries as possible, yet where each variant in the collection is as fit as it can be. Often applied in both...
Patent
Full-text available
Various embodiments are disclosed for generating an image from a Compositional Pattern Producing Network (CPPN). One such method includes receiving, in the CPPN, a series of polar coordinates {r,θ}; outputting, by the CPPN, a series of pixel values, each of the pixel values corresponding to one of the polar coordinates; and displaying the pixel val...
Article
The impact of game content on the player experience is potentially more critical in casual games than in competitive games because of the diminished role of strategic or tactical diversions. Interestingly, until now procedural content generation (PCG) has nevertheless been investigated almost exclusively in the context of competitive, skills-based...
Chapter
Building on the previous chapter’s analysis of how objectives can derail educational systems, this chapter expands the argument against objectives to innovation as a whole. In particular, the focus is on examining how obsession with objectives can harm science funding, collaborative innovation, business investment, and artistic discovery. The probl...
Chapter
Considering the hidden societal toll of pursuing objectives, this chapter and the next illustrate how too many objectives can stifle everything from individual achievement to educational systems and scientific innovation. The main focus of this chapter is the harmful effects of outcomes and objectives upon education. The insights so far bring into...
Chapter
This chapter, the first case study, reconsiders the drive behind natural evolution in light of the myth of the objective. Much discussion of evolution in popular culture emphasizes survival and reproduction as a central objective of the process. This chapter turns this perspective on its head, painting natural evolution as an open-ended exploration...
Chapter
Have you ever noticed that almost every task in life is framed by an objective? Whether you’re completing a job, studying for a test, or even looking for love, you almost always have an objective. Our culture is increasingly obsessed with them. If somehow you don’t have an objective, you’re less likely to be taken seriously, and someone is sure to...
Chapter
The question answered in this chapter is, why does the world work in such an unexpected way? Why is it often easier to succeed without objectives? To expose the way that objectives hamper discovery, this chapter surveys scientific experiments, natural evolution, and technological innovation. The problem is that the precursors to a great achievement...
Chapter
Strangely enough, the insight that inspired this book originated in a then unique (and exotic) experiment in online picture breeding. This chapter tells the story of that experiment and how it led to the unexpected insight that objectives can thwart discovery. Because it appeared it might lead to something interesting, we created a online system wh...
Chapter
There’s a common assumption that you can achieve anything if you put your mind to it. So it’s no surprise that our culture stresses driving full speed towards our goals, whether in starting a business, pursuing a career, or even in choosing hobbies or mates. The accepted wisdom is that it’s wasteful to meander through life without a consistent dire...
Chapter
This chapter provides a case study in how objectives can hold back progress in a particular field of scientific research. The field of artificial intelligence (AI) focuses on the ambitious aim of creating a human-level computer intelligence, an objective which is fascinating for the profound implications it entails if is ever achieved. Interestingl...
Chapter
Because it conflicts with deep-seated beliefs, it can be difficult to accept that deep innovations and discoveries are more likely when the search is aimless. It’s natural to be skeptical. To provide a concrete example of how abandoning objectives can be practical, this chapter explains an alternative to complete aimlessness that still avoids objec...
Chapter
Because they violate common intuitions, there are many ways to interpret the results of novelty search described in the previous chapter. This chapter continues to build the argument that there is no reliable way to consistently achieve particular ambitious goals. The results from novelty search don’t point to a method for solving any particular pr...
Chapter
Letting go of objectives provides a psychological challenge because without them it might seem that we are left drifting without guideposts. This culminating chapter argues against the false security of objectives by celebrating the alternative pathways to achievement that are more effective than objectives. In particular, it offers lessons in how...
Article
Full-text available
Abstract An important goal in both artificial life and biology is uncovering the most general principles underlying life, which might catalyze both our understanding of life and engineering lifelike machines. While many such general principles have been hypothesized, conclusively testing them is difficult because life on Earth provides only a singu...
Article
Many tools for computer-assisted composition contain built-in music-theoretical assumptions that may constrain the output to particular styles. In contrast, this article presents a new musical representation that contains almost no built-in knowledge, but that allows even musically untrained users to generate polyphonic textures that are derived fr...
Article
The question of how to best design a communication architecture is becoming increasingly important for evolving autonomous multiagent systems. Directional reception of signals, a design feature of communication that appears in most animals, is present in only some existing artificial communication systems. This paper hypothesizes that such directio...
Article
An important phenomenon seen in many areas of biological brains and recently in deep learning architectures is a process known as self-organization. For example, in the primary visual cortex, color and orientation maps develop based on lateral inhibitory connectivity patterns and Hebbian learning dynamics. These topographic maps, which are found in...
Conference Paper
Recent work on novelty and behavioral diversity in evolutionary computation has highlighted the potential disadvantage of driving search purely through objective means. This paper suggests that leveraging human insight during search can complement such novelty-driven approaches. In particular, a new approach called novelty-assisted interactive evol...
Article
To address the difficulty of creating online collaborative evolutionary systems, this paper presents a new prototype library called Worldwide Infrastructure for Neuroevolution (WIN) and its accompanying site WIN Online (http://winark.org/). The WIN library is a collection of software packages built on top of Node.js that reduce the complexity of cr...
Article
Unlike unsupervised approaches such as autoencoders that learn to reconstruct their inputs, this paper introduces an alternative approach to unsupervised feature learning called divergent discriminative feature accumulation (DDFA) that instead continually accumulates features that make novel discriminations among the training set. Thus DDFA feature...
Patent
Full-text available
A system for inducing an effect in a raw audio signal comprises a computing device for receiving a first audio signal and a second audio signal from a signal source, and the second audio signal comprises the first audio signal induced with an effect. The system further comprises logic that parameterizes the effect in the second audio signal into an...
Article
The robustness of animal behavior is unmatched by current machines, which often falter when exposed to unforeseen conditions. While animals are notably reactive to changes in their environment, machines often follow finely tuned yet inflexible plans. Thus, instead of the traditional approach of training such machines over many different unpredictab...
Conference Paper
Full-text available
This paper introduces a meta-optimization algorithm called NeuroEvolutionary Meta-Optimization (NEMO) that evolves an algorithm targeted at optimizing only within a specific problem class. More specifically, a form of neural network is evolved that acts as the controller of a kind of optimization algorithm that can potentially exploit problem class...
Conference Paper
This paper introduces a meta-optimization algorithm called NeuroEvolutionary Meta-Optimization (NEMO) that evolves an algorithm targeted at optimizing only within a specific problem class. More specifically, a form of neural network is evolved that acts as the controller of a kind of optimization algorithm that can potentially exploit problem class...
Conference Paper
Natural brains effectively integrate multiple sensory modalities and act upon the world through multiple effector types. As researchers strive to evolve more sophisticated neural controllers, confronting the challenge of multimodality is becoming increasingly important. As a solution, this paper presents a principled new approach to exploiting indi...
Conference Paper
Legged robots can potentially venture beyond the limits of wheeled vehicles. While creating controllers for such robots by hand is possible, evolutionary algorithms are an alternative that can reduce the burden of hand-crafting robotic controllers. Although major evolutionary approaches to legged locomotion can generate oscillations through popular...
Conference Paper
The ambulatory capabilities of legged robots offer the potential for access to dangerous and uneven terrain without a risk to human life. However, while machine learning has proven effective at training such robots to walk, a significant limitation of such approaches is that controllers trained for a specific robot are likely to fail when transferr...
Article
Diversity maintenance techniques in evolutionary computation are designed to mitigate the problem of deceptive local optima by encouraging exploration. However, as problems become more difficult, the heuristic of fitness may become increasingly uninformative. Thus, simply encouraging genotypic diversity may fail to much increase the likelihood of e...
Article
Multiagent systems present many challenging, real-world problems to artificial intelligence. Because it is difficult to engineer the behaviors of multiple cooperating agents by hand, multiagent learning has become a popular approach to their design. While there are a variety of traditional approaches to multiagent learning, many suffer from increas...