The short-latency dopamine signal: a role in discovering novel actions?

Neuroscience Research Unit, Department of Psychology, University of Sheffield, Sheffield, S10 2TP, UK.
Nature reviews Neuroscience (Impact Factor: 31.38). 01/2007; 7(12):967-75. DOI: 10.1038/nrn2022
Source: PubMed

ABSTRACT An influential concept in contemporary computational neuroscience is the reward prediction error hypothesis of phasic dopaminergic function. It maintains that midbrain dopaminergic neurons signal the occurrence of unpredicted reward, which is used in appetitive learning to reinforce existing actions that most often lead to reward. However, the availability of limited afferent sensory processing and the precise timing of dopaminergic signals suggest that they might instead have a central role in identifying which aspects of context and behavioural output are crucial in causing unpredicted events.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Intrinsic motivations drive the acquisition of knowledge and skills on the basis of novel or surprising stimuli or the pleasure to learn new skills. In so doing, they are different from extrinsic motivations that are mainly linked to drives that promote survival and reproduction. Intrinsic motivations have been implicitly exploited in several psychological experiments but, due to the lack of proper paradigms, they are rarely a direct subject of investigation. This article investigates how different intrinsic motivation mechanisms can support the learning of visual skills, such as "foveate a particular object in space", using a gaze contingency paradigm. In the experiment participants could freely foveate objects shown in a computer screen. Foveating each of two "button" pictures caused different effects: one caused the appearance of a simple image (blue rectangle) in unexpected positions, while the other evoked the appearance of an always-novel picture (objects or animals). The experiment studied how two possible intrinsic motivation mechanisms might guide learning to foveate one or the other button picture. One mechanism is based on the sudden, surprising appearance of a familiar image at unpredicted locations, and a second one is based on the content novelty of the images. The results show the comparative effectiveness of the mechanism based on image novelty, whereas they do not support the operation of the mechanism based on the surprising location of the image appearance. Interestingly, these results were also obtained with participants that, according to a post experiment questionnaire, had not understood the functions of the different buttons suggesting that novelty-based intrinsic motivation mechanisms might operate even at an unconscious level.
    PLoS ONE 03/2015; 10(3):e0118705. DOI:10.1371/journal.pone.0118705 · 3.53 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Orienting attention to a new target requires prior disengagement of attention from the current focus. Previous studies indicate that the superior colliculus (SC) plays an important role in attention. However, recordings of responses of SC neurons during attentional disengagement have not yet been reported. Here, we analyzed rat SC neuronal activity during performance of an attention-shift task with and without disengagement. In this task, conditioned stimuli (CSs; right and/or left light-flash or sound) were sequentially presented. To obtain an intracranial self-stimulation reward, rats were required to lick a spout when an infrequent conditioned stimulus appeared (reward trials). In the disengagement reward trials, configural stimuli consisting of an infrequent stimulus and frequent stimulus in the former trials were presented; in the non-disengagement reward trials, only an infrequent stimulus was presented. Of the 186 SC neurons responding to the CSs, 41 showed stronger responses to the CSs in the disengagement reward trials than in the non-disengagement reward trials (disengagement-related neurons). Furthermore, lick latencies in the disengagement reward trials were negatively correlated with response magnitudes to the CSs in half of the disengagement-related neurons. These disengagement-related neurons were located mainly in the deep layers of the SC. Another 70 SC neurons responded to the CSs in both disengagement and non-disengagement reward trials, suggesting that these neurons were involved in attention engagement. Our results suggest complementary mechanisms of attentional shift based on two subpopulations of neurons in the SC.
    Frontiers in Integrative Neuroscience 02/2015; 9:9. DOI:10.3389/fnint.2015.00009
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: We offer a formal treatment of choice behaviour based on the premise that agents minimise the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (or intrinsic) value. Minimising expected free energy is therefore equivalent to maximising extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximising information gain or intrinsic value (reducing uncertainty about the causes of valuable outcomes). The resulting scheme resolves the exploration-exploitation dilemma: epistemic value is maximised until there is no further information gain, after which exploitation is assured through maximisation of extrinsic value. This is formally consistent with the Infomax principle, generalising formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk-sensitive (Kullback-Leibler) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems, ad hoc softmax parameters become the expected (Bayes-optimal) precision of beliefs about, or confidence in, policies. This article focuses on the basic theory, illustrating the ideas with simulations. A key aspect of these simulations is the similarity between precision updates and dopaminergic discharges observed in conditioning paradigms.
    Cognitive neuroscience 02/2015; DOI:10.1080/17588928.2015.1020053 · 2.38 Impact Factor