Predictive reinforcement model of dopamine neurons for learning approach behavior

Motor Control Laboratory, Arizona State University, Tempe 85287-0404, USA.
Journal of Computational Neuroscience (Impact Factor: 1.74). 01/1999; 6(3):191-214. DOI: 10.1023/A:1008862904946
Source: PubMed


A neural network model of how dopamine and prefrontal cortex activity guides short- and long-term information processing within the cortico-striatal circuits during reward-related learning of approach behavior is proposed. The model predicts two types of reward-related neuronal responses generated during learning: (1) cell activity signaling errors in the prediction of the expected time of reward delivery and (2) neural activations coding for errors in the prediction of the amount and type of reward or stimulus expectancies. The former type of signal is consistent with the responses of dopaminergic neurons, while the latter signal is consistent with reward expectancy responses reported in the prefrontal cortex. It is shown that a neural network architecture that satisfies the design principles of the adaptive resonance theory of Carpenter and Grossberg (1987) can account for the dopamine responses to novelty, generalization, and discrimination of appetitive and aversive stimuli. These hypotheses are scrutinized via simulations of the model in relation to the delivery of free food outside a task, the timed contingent delivery of appetitive and aversive stimuli, and an asymmetric, instructed delay response task.

Download full-text


Available from: José L Contreras-Vidal,
  • Source
    • "Perhaps this and not better timing ab initio by the dopamine-replete is involved. The authors point to the congruency of their findings with those of Schultz's team: ''how dopamine and prefrontal cortex activity guides short-and long-term information processing within the cortico-striatal circuits during rewardrelated learning and approach behavior " [25]. If, and only if, further research on soccer performance shows that good time interval estimation, i.e. having a boot in the right place at just the right instant, is more important than those other executive skills where A1+ can be beneficial then it will become predictable that few A1+ carriers will reach the top in soccer. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Elite Spanish professional soccer players surprisingly showed a preponderance of an allele coding for nitric oxide synthase (NOS) that resulted in lower nitric oxide (NO) compared with Spanish endurance and power athletes and sedentary men. The present paper attempts a speculative explanation. Soccer is an ‘‘externally-paced” (EP) sport and team work dependent, requiring ‘‘executive function skills”. We accept that time interval estimation skill is, in part, also an executive skill. Dopamine (DA) is prominent among the neurotransmitters with a role in such skills. Polymorphisms affecting dopamine (especially DRD2/ANKK1-Taq1a which leads to lower density of dopamine D2 receptors in the striatum, leading to increased striatal dopamine synthesis) and COMT val 158 met (which prolongs the action of dopamine in the cortex) feature both in the timeinterval estimation and the executive skills literatures. Our paper may be a pioneering attempt to stimulate empirical efforts to show how genotypes among soccer players may be connected via neurotransmitters to certain cognitive abilities that predict sporting success, perhaps also in some other externally-paced team sports. Graphing DA levels against time interval estimation accuracy and also against certain executive skills reveals an inverted-U relationship. A pathway from DA, via endogenous morphine and mu3 receptors on endothelia, to the generation of NO in tiny quantities has been demonstrated. Exercise up-regulates DA and this pathway. With somewhat excessive exercise, negative feedback from NO down-regulates DA, hypothetically keeping it near the peak of the inverted-U. Other research, not yet done on higher animals or humans, shows NO ‘‘fine-tuning” movement. We speculate that Caucasian men, playing soccer recreationally, would exemplify the above pattern and their nitric oxide synthase (NOS) would reflect the norm of their community, whereas professional players of soccer and perhaps other EP sports, with DA boosted by very frequent and intense practice and extra stress from public scrutiny, would potentially have their negative feedback system overwhelmed, were it not that many of them carry the C allele of the NOS3-786T/C polymorphism. Then, even very high DA would not result in so much NO as to shut the system down. We add some evolutionary speculations.
    Medical Hypotheses 09/2015; 85:905-909. · 1.07 Impact Factor
  • Source
    • "Computational models [151–155] of dopaminergic neuronal firing have noted similarities between the response patterns of dopaminergic neurons and the well-known learning algorithms, especially temporal difference reinforcement learning algorithms. However, there has been considerable debate regarding the circuit mechanisms underlying reward prediction error computation [154]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Dopamine, acetylcholine, and serotonin, the main modulators of the central nervous system, have been proposed to play important roles in the execution of movement, control of several forms of attentional behavior, and reinforcement learning. While the response pattern of midbrain dopaminergic neurons and its specific role in reinforcement learning have been revealed, the role of the other neuromodulators remains rather elusive. Here, we review our recent studies using extracellular recording from neurons in the pedunculopontine tegmental nucleus, where many cholinergic neurons exist, and the dorsal raphe nucleus, where many serotonergic neurons exist, while monkeys performed eye movement tasks to obtain different reward values. The firing patterns of these neurons are often tonic throughout the task period, while dopaminergic neurons exhibited a phasic activity pattern to the task event. The different modulation patterns, together with the activity of dopaminergic neurons, reveal dynamic information processing between these different neuromodulator systems.
    Neural Plasticity 10/2011; 2011(2090-5904):579840. DOI:10.1155/2011/579840 · 3.58 Impact Factor
  • Source
    • "GABA A R-mediated inhibition can effectively counteract glutamatergic excitation of DA neurons (Tepper et al., 1998, Lobb et al., 2010). Recent experimental evidence, together with modeling studies in the past, suggests that GABAergic inhibition might mediate the transient pause in DA neuron firing when the expected reward is omitted after cue presentation (Fig. 1) (Brown et al., 1999, Contreras-Vidal and Schultz, 1999, Jhou et al., 2009, Lobb et al., 2010, Lobb et al., 2011a). In this scenario, GABAergic inhibition will cancel reward-induced glutamatergic excitation when the expected reward is presented after the cue, hence no change in DA neuron firing. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Although the roles of dopaminergic signaling in learning and behavior are well established, it is not fully understood how the activity of dopaminergic neurons is dynamically regulated under different conditions in a constantly changing environment. Dopamine neurons must integrate sensory, motor, and cognitive information online to inform the organism to pursue outcomes with the highest reward probability. In this article, we provide an overview of recent advances on the intrinsic, extrinsic (i.e., synaptic), and plasticity mechanisms controlling dopamine neuron activity, mostly focusing on mechanistic studies conducted using ex vivo brain slice preparations. We also hope to highlight some unresolved questions regarding information processing that takes place at dopamine neurons, thereby stimulating further investigations at different levels of analysis.
    Neuroscience 08/2011; 198:95-111. DOI:10.1016/j.neuroscience.2011.08.023 · 3.36 Impact Factor
Show more