No file available
To read the file of this research,
you can request a copy directly from the authors.
Implicit Generative Modeling for Efficient Exploration
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.
ResearchGate has not been able to resolve any citations for this publication.
We propose a simple algorithm to train stochastic neural networks to draw samples from given target distributions for probabilistic inference. Our method is based on iteratively adjusting the neural network parameters so that the output changes along a Stein variational gradient direction (Liu & Wang, 2016) that maximally decreases the KL divergence with the target distribution. Our method works for any target distribution specified by their unnormalized density function, and can train any black-box architectures that are differentiable in terms of the parameters we want to adapt. We demonstrate our method with a number of applications, including variational autoencoder (VAE) with expressive encoders to model complex latent space structures, and hyper-parameter learning of MCMC samplers that allows Bayesian inference to adaptively improve itself when seeing more data.
We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use sequential density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary sequential density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.
Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.
The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of state-of-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.
Bayesian methods are appealing in their flexibility in modeling complex data
and their ability to capture uncertainty in parameters. However, when Bayes'
rule does not result in closed-form, most approximate Bayesian inference
algorithms lacks either scalability or rigorous guarantees. To tackle this
challenge, we propose a scalable yet simple algorithm, Particle Mirror Descent
(PMD), to iteratively approximate the posterior density. PMD is inspired by
stochastic functional mirror descent where one descends in the density space
using a small batch of data points at each iteration, and by particle filtering
where one uses samples to approximate a function. We prove result of the first
kind that, after T iterations, PMD provides a posterior density estimator
that converges in terms of KL-divergence to the true posterior in rate
. We show that PMD is competitive to several scalable Bayesian
algorithms in mixture models, Bayesian logistic regression, sparse Gaussian
processes and latent Dirichlet allocation.
To maximize its success, an AGI typically needs to explore its initially
unknown world. Is there an optimal way of doing so? Here we derive an
affirmative answer for a broad class of environments.
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
We propose a general purpose variational inference algorithm that forms a natural counterpart of gradient descent for optimization. Our method iteratively transports a set of particles to match the target distribution, by applying a form of functional gradient descent that minimizes the KL divergence. Empirical studies are performed on various real world models and datasets, on which our method is competitive with existing state-of-the-art methods. The derivation of our method is based on a new theoretical result that connects the derivative of KL divergence under smooth transforms with Stein's identity and a recently proposed kernelized Stein discrepancy, which is of independent interest.
Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.
The mutual information is a core statistical quantity that has applications
in all areas of machine learning, whether this is in training of density models
over multiple data modalities, in maximising the efficiency of noisy
transmission channels, or when learning behaviour policies for exploration by
artificial agents. Most learning algorithms that involve optimisation of the
mutual information rely on the Blahut-Arimoto algorithm --- an enumerative
algorithm with exponential complexity that is not suitable for modern machine
learning applications. This paper provides a new approach for scalable
optimisation of the mutual information by merging techniques from variational
inference and deep learning. We develop our approach by focusing on the problem
of intrinsically-motivated learning, where the mutual information forms the
definition of a well-known internal drive known as empowerment. Using a
variational lower bound on the mutual information, combined with convolutional
networks for handling visual input streams, we develop a stochastic
optimisation algorithm that allows for scalable information maximisation and
empowerment-based reasoning directly from pixels to actions.
We introduce Adam, an algorithm for first-order gradient-based optimization
of stochastic objective functions. The method is straightforward to implement
and is based an adaptive estimates of lower-order moments of the gradients. The
method is computationally efficient, has little memory requirements and is well
suited for problems that are large in terms of data and/or parameters. The
method is also ap- propriate for non-stationary objectives and problems with
very noisy and/or sparse gradients. The method exhibits invariance to diagonal
rescaling of the gradients by adapting to the geometry of the objective
function. The hyper-parameters have intuitive interpretations and typically
require little tuning. Some connections to related algorithms, on which Adam
was inspired, are discussed. We also analyze the theoretical convergence
properties of the algorithm and provide a regret bound on the convergence rate
that is comparable to the best known results under the online convex
optimization framework. We demonstrate that Adam works well in practice when
experimentally compared to other stochastic optimization methods.
This book chapter is an introduction to and an overview of the
information-theoretic, task independent utility function "Empowerment", which
is defined as the channel capacity between an agent's actions and an agent's
sensors. It quantifies how much influence and control an agent has over the
world it can perceive. This book chapter discusses the general idea behind
empowerment as an intrinsic motivation and showcases several previous
applications of empowerment to demonstrate how empowerment can be applied to
different sensor-motor configuration, and how the same formalism can lead to
different observed behaviors. Furthermore, we also present a fast approximation
for empowerment in the continuous domain.
A novel curious model-building control system is described which
actively tries to provoke situations for which it learned to expect to
learn something about the environment. Such a system has been
implemented as a four-network system based on Watkins' Q-learning
algorithm which can be used to maximize the expectation of the temporal
derivative of the adaptive assumed reliability of future predictions. An
experiment with an artificial nondeterministic environment demonstrates
that the system can be superior to previous model-building control
systems, which do not address the problem of modeling the reliability of
the world model's predictions in uncertain environments and use ad-hoc
methods (like random search) to train the world model
In the classical stochastic k-armed bandit problem, ineachofasequenceofT rounds, adecisionmaker chooses one of k arms and incurs a cost chosen from an unknown distribution associated with that arm. The goal is to minimize regret, defined as the difference between the cost incurred by the algo- rithm and the optimal cost. In the linear optimization version of this problem (firstconsideredbyAuer(2002)), weviewthearms as vectors in Rn, and require that the costs be lin- ear functions of the chosen vector. As before, it is assumed that the cost functions are sampled in- dependently from an unknown distribution. In this setting, the goal is to find algorithms whose run- ning time and regret behave well as functions of the number of rounds T and the dimensionality n (rather than the number of arms, k, which may be exponential in n or even infinite). We give a nearly complete characterization of this problem in terms of both upper and lower bounds for the regret. In certain special cases (such as when the decision region is a polytope), the regret is polylog(T). In general though, the optimal re- gret is ( p T) — our lower bounds rule out the possibility of obtaining polylog(T) rates in gen- eral. We present two variants of an algorithm based on the idea of "upper confidence bounds." The first, due to Auer (2002), but not fully analyzed, obtains regret whose dependence on n and T are both es- sentially optimal, but which may be computation- ally intractable when the decision set is a polytope. The second version can be efficiently implemented when the decision set is a polytope (given as an in- tersection of half-spaces), but gives up a factor of p n in the regret bound. Our results also extend to the setting where the set of allowed decisions may change over time.
We show how a standard tool from statistics | namely condence bounds | can be used to elegantly deal with situations which exhibit an exploitation-exploration trade-o. Our technique for designing and analyzing algorithms for such situations is general and can be applied when an algorithm has to make exploitation-versus-exploration decisions based on uncertain information provided by a random process. We apply our technique to two models with such an exploitation-exploration trade-o. For the adversarial bandit problem with shifting our new algorithm suers only ~ O (ST)1=2 regret with high probability over T trials with S shifts. Such a regret bound was previously known only in expectation. The second model we consider is associative reinforcement learning with linear value functions. For this model our technique improves the regret from ~ O T 3=4 to ~ O T 1=2 .
This paper reviews the literature on Bayesian experimental design. A unified view of this topic is presented, based on a decision-theoretic approach. This framework casts criteria from the Bayesian literature of design as part of a single coherent approach. The decision-theoretic structure incorporates both linear and nonlinear design problems and it suggests possible new directions to the experimental design problem, motivated by the use of new utility functions. We show that, in some special cases of linear design problems, Bayesian solutions change in a sensible way when the prior distribution and the utility function are modified to allow for the specific structure of the experiment. The decision-theoretic approach also gives a mathematical justification for selecting the appropriate optimality criterion.
Wasserstein variational gradient descent: From semi-discrete optimal transport to ensemble variational inference
- Luca Ambrogioni
- Umut Guclu
- Yagmur Gucluturk
- Marcel Van Gerven
Luca Ambrogioni, Umut Guclu, Yagmur Gucluturk, and Marcel van Gerven. Wasserstein variational
gradient descent: From semi-discrete optimal transport to ensemble variational inference. arXiv
preprint arXiv:1811.02827, 2018.
Large-scale study of curiosity-driven learning
- Yuri Burda
- Harri Edwards
- Deepak Pathak
- Amos Storkey
- Trevor Darrell
- Alexei A Efros
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros.
Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018a.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
- Kurtland Chua
- Roberto Calandra
- Rowan Mcallister
- Sergey Levine
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information
Processing Systems, pp. 4754-4765, 2018.
Learning latent dynamics for planning from pixels
- Danijar Hafner
- Timothy Lillicrap
- Ian Fischer
- Ruben Villegas
- David Ha
- Honglak Lee
- James Davidson
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James
Davidson. Learning latent dynamics for planning from pixels. In International Conference on
Machine Learning, pp. 2555-2565, 2019.
Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning
- Volodymyr Mnih
- Koray Kavukcuoglu
- David Silver
- Alex Graves
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602, 2013.
Randomized prior functions for deep reinforcement learning
- Ian Osband
- John Aslanides
- Albin Cassirer
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement
learning. In Advances in Neural Information Processing Systems, pp. 8617-8629, 2018.
Count-based exploration with neural density models
- Georg Ostrovski
- G Marc
- Aäron Bellemare
- Rémi Van Den Oord
- Munos
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on
Machine Learning-Volume 70, pp. 2721-2730. JMLR. org, 2017.
Self-supervised exploration via disagreement
- Deepak Pathak
- Dhiraj Gandhi
- Abhinav Gupta
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement.
In International Conference on Machine Learning, pp. 5062-5071, 2019.
Searching for activation functions
- Prajit Ramachandran
- Barret Zoph
- Quoc V Le
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv
preprint arXiv:1710.05941, 2017.
Hypergan: A generative model for diverse
- Neale Ratzlaff
- Li Fuxin
Neale Ratzlaff and Li Fuxin. Hypergan: A generative model for diverse, performant neural networks.
arXiv preprint arXiv:1901.11058, 2019.
- Yuri Burda
- Harrison Edwards
- Amos Storkey
- Oleg Klimov
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network
distillation. arXiv preprint arXiv:1810.12894, 2018b.
- Benjamin Eysenbach
- Abhishek Gupta
- Julian Ibarz
- Sergey Levine
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need:
Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
- Meire Fortunato
- Mohammad Gheshlaghi Azar
- Bilal Piot
- Jacob Menick
- Ian Osband
- Alex Graves
- Vlad Mnih
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves,
Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration.
arXiv preprint arXiv:1706.10295, 2017.
- Karol Gregor
- Danilo Jimenez Rezende
- Daan Wierstra
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv
preprint arXiv:1611.07507, 2016.
- P Timothy
- Jonathan J Lillicrap
- Alexander Hunt
- Nicolas Pritzel
- Tom Heess
- Yuval Erez
- David Tassa
- Daan Silver
- Wierstra
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,
David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv
preprint arXiv:1509.02971, 2015.
- Ian Osband
- Daniel Benjamin Van Roy
- Zheng Russo
- Wen
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized
value functions. arXiv preprint arXiv:1703.07608, 2017.
Model-based active exploration
- Pranav Shyam
- Wojciech Jaśkowski
- Faustino Gomez
Pranav Shyam, Wojciech Jaśkowski, and Faustino Gomez. Model-based active exploration. In
International Conference on Machine Learning, pp. 5779-5788, 2019.