A Predictive Reinforcement Model of Dopamine Neurons for Learning Approach Behavior

Motor Control Laboratory, Arizona State University, Tempe 85287-0404, USA.
Journal of Computational Neuroscience (Impact Factor: 1.74). 01/1999; 6(3):191-214. DOI: 10.1023/A:1008862904946
Source: PubMed


A neural network model of how dopamine and prefrontal cortex activity guides short- and long-term information processing within the cortico-striatal circuits during reward-related learning of approach behavior is proposed. The model predicts two types of reward-related neuronal responses generated during learning: (1) cell activity signaling errors in the prediction of the expected time of reward delivery and (2) neural activations coding for errors in the prediction of the amount and type of reward or stimulus expectancies. The former type of signal is consistent with the responses of dopaminergic neurons, while the latter signal is consistent with reward expectancy responses reported in the prefrontal cortex. It is shown that a neural network architecture that satisfies the design principles of the adaptive resonance theory of Carpenter and Grossberg (1987) can account for the dopamine responses to novelty, generalization, and discrimination of appetitive and aversive stimuli. These hypotheses are scrutinized via simulations of the model in relation to the delivery of free food outside a task, the timed contingent delivery of appetitive and aversive stimuli, and an asymmetric, instructed delay response task.

Download full-text


Available from: José L Contreras-Vidal, Oct 05, 2015
50 Reads
  • Source
    • "Computational models [151–155] of dopaminergic neuronal firing have noted similarities between the response patterns of dopaminergic neurons and the well-known learning algorithms, especially temporal difference reinforcement learning algorithms. However, there has been considerable debate regarding the circuit mechanisms underlying reward prediction error computation [154]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Dopamine, acetylcholine, and serotonin, the main modulators of the central nervous system, have been proposed to play important roles in the execution of movement, control of several forms of attentional behavior, and reinforcement learning. While the response pattern of midbrain dopaminergic neurons and its specific role in reinforcement learning have been revealed, the role of the other neuromodulators remains rather elusive. Here, we review our recent studies using extracellular recording from neurons in the pedunculopontine tegmental nucleus, where many cholinergic neurons exist, and the dorsal raphe nucleus, where many serotonergic neurons exist, while monkeys performed eye movement tasks to obtain different reward values. The firing patterns of these neurons are often tonic throughout the task period, while dopaminergic neurons exhibited a phasic activity pattern to the task event. The different modulation patterns, together with the activity of dopaminergic neurons, reveal dynamic information processing between these different neuromodulator systems.
    Neural Plasticity 10/2011; 2011(2090-5904):579840. DOI:10.1155/2011/579840 · 3.58 Impact Factor
  • Source
    • "GABA A R-mediated inhibition can effectively counteract glutamatergic excitation of DA neurons (Tepper et al., 1998, Lobb et al., 2010). Recent experimental evidence, together with modeling studies in the past, suggests that GABAergic inhibition might mediate the transient pause in DA neuron firing when the expected reward is omitted after cue presentation (Fig. 1) (Brown et al., 1999, Contreras-Vidal and Schultz, 1999, Jhou et al., 2009, Lobb et al., 2010, Lobb et al., 2011a). In this scenario, GABAergic inhibition will cancel reward-induced glutamatergic excitation when the expected reward is presented after the cue, hence no change in DA neuron firing. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Although the roles of dopaminergic signaling in learning and behavior are well established, it is not fully understood how the activity of dopaminergic neurons is dynamically regulated under different conditions in a constantly changing environment. Dopamine neurons must integrate sensory, motor, and cognitive information online to inform the organism to pursue outcomes with the highest reward probability. In this article, we provide an overview of recent advances on the intrinsic, extrinsic (i.e., synaptic), and plasticity mechanisms controlling dopamine neuron activity, mostly focusing on mechanistic studies conducted using ex vivo brain slice preparations. We also hope to highlight some unresolved questions regarding information processing that takes place at dopamine neurons, thereby stimulating further investigations at different levels of analysis.
    Neuroscience 08/2011; 198:95-111. DOI:10.1016/j.neuroscience.2011.08.023 · 3.36 Impact Factor
  • Source
    • "In this model, an animal experiments with its motor repertoire and receives sensory-driven evaluative feedback that reinforces action sequences that improve output. The combination of randomness and selection can result in extremely sophisticated behaviors and has been successfully used to engineer motor control in robots and to model BG-dependent sequence learning in primates (Beiser and Houk 1998; Contreras-Vidal and Schultz 1999; Gutnisky and Zanutto 2004; Suri and Schultz 1998, 1999). The fact that songbirds require auditory feedback during all stages of song learning to properly master their songs by imitation is well established (Konishi 1965) and is consistent with the RL view of vocal learning. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Over the past four decades songbirds have become a widely used model organism for neuroscientists studying complex sequential behaviors and sensory-guided motor learning. Like human babies, young songbirds learn many of the sounds they use for communication by imitating adults. This remarkable behavior emerges as a product of genetic predispositions and specific individual experiences. Research on different aspects of this behavior has elucidated key principles that may underlie vertebrate motor learning and motor performance in general, including (1) the mechanisms by which neural circuits generate sequential behaviors, (2) the existence of specialized neuronal circuits for the generation of exploratory variability, (3) the importance of basal ganglia-forebrain circuits for learning sequentially patterned behaviors, including speech and language, and (4) the existence of genetic toolkits that may have been coopted multiple times during evolution to play a role in learned vocal communication, such as the transcription factor FoxP2 and its molecular targets. This review presents new techniques, experiments, and findings in areas where songbirds have made significant contributions toward understanding of some of the most fundamental questions in neuroscience.
    ILAR journal / National Research Council, Institute of Laboratory Animal Resources 01/2010; 51(4):362-77. DOI:10.1093/ilar.51.4.362 · 2.39 Impact Factor
Show more