Review. Explicit neural signals reflecting reward uncertainty

Department of Physiology, Development and Neuroscience, University of Cambridge, Downing Street, Cambridge CB2 3DY, UK.
Philosophical Transactions of The Royal Society B Biological Sciences (Impact Factor: 7.06). 11/2008; 363(1511):3801-11. DOI: 10.1098/rstb.2008.0152
Source: PubMed


The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.

Download full-text


Available from: Colin Farrell Camerer,
  • Source
    • "side , the activity of the rAI has also been associated with behavioral contingencies explained in terms of attention ( Eckert et al . , 2009 ; Menon and Uddin , 2010 ; Nelson et al . , 2010 ) , response inhibition ( Aron and Poldrack , 2006 ; Aron et al . , 2014 ; Cai et al . , 2014 ) , and other forms of uncertainty ( Preuschoff et al . , 2008 ; Schultz et al . , 2008 ; Bossaerts , 2010 ; Jones et al . , 2010 , 2011 ; Sarinopoulos et al . , 2010 ; Payzan - LeNestour and Bossaerts , 2011 , 2012 ; Payzan - LeNestour et al . , 2013 ; Symmonds et al . , 2013 ; Nursimulu and Bossaerts , 2014 ) . In other words , the sole fact that the activity in the rAI is negatively modulated by temporal PEs is not suff"
    [Show abstract] [Hide abstract]
    ABSTRACT: We have previously shown that temporal prediction errors (PEs, the differences between the expected and the actual stimulus’ onset times) modulate the effective connectivity between the anterior cingulate cortex and the right anterior insular cortex (rAI), causing the activity of the rAI to decrease. The activity of the rAI is associated with efficient performance under uncertainty (e.g., changing a prepared behavior when a change demand is not expected), which leads to hypothesize that temporal PEs might disrupt behavior-change performance under uncertainty. This hypothesis has not been tested at a behavioral level. In this work, we evaluated this hypothesis within the context of task switching and concurrent temporal predictions. Our participants performed temporal predictions while observing one moving ball striking a stationary ball which bounced off with a variable temporal gap. Simultaneously, they performed a simple color comparison task. In some trials, a change signal made the participants change their behaviors. Performance accuracy decreased as a function of both the temporal PE and the delay. Explaining these results without appealing to ad hoc concepts such as “executive control” is a challenge for cognitive neuroscience. We provide a predictive coding explanation. We hypothesize that exteroceptive and proprioceptive minimization of PEs would converge in a fronto-basal ganglia network which would include the rAI. Both temporal gaps (or uncertainty) and temporal PEs would drive and modulate this network respectively. Whereas the temporal gaps would drive the activity of the rAI, the temporal PEs would modulate the endogenous excitatory connections of the fronto-striatal network. We conclude that in the context of perceptual uncertainty, the system is not able to minimize perceptual PE, causing the ongoing behavior to finalize and, in consequence, disrupting task switching.
    Frontiers in Psychology 08/2015; 6:1-10. DOI:10.3389/fpsyg.2015.01185 · 2.80 Impact Factor
  • Source
    • "We have studied reward learning in a passive learning situation. It is known that existing RPE schemes do not fully account for learning in this setting (Dayan and Niv, 2008; Schultz and Dickinson, 2000): For example, they have limited capacity for subjective uncertainty (Preuschoff and Bossaerts, 2007; Schultz et al., 2008) and simply associate each cue or " state " with a single value. Experimental evidence points to simple learning in the absence of RPEs, e.g. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Surprise drives learning. Various neural “prediction error” signals are believed to underpin surprise-based reinforcement learning. Here, we report a surprise signal that reflects reinforcement learning but is neither un/signed reward prediction error (RPE) nor un/signed state prediction error (SPE). To exclude these alternatives, we measured surprise responses in the absence of RPE and accounted for a host of potential SPE confounds. This new surprise signal was evident in ventral striatum, primary sensory cortex, frontal poles, and amygdala. We interpret these findings via a normative model of surprise. Hum Brain Mapp, 2014. © 2014 Wiley Periodicals, Inc.
    Human Brain Mapping 09/2014; 35(9). DOI:10.1002/hbm.22513 · 5.97 Impact Factor
  • Source
    • "Recent research demonstrating the role of this dopaminergic system in formulating “reward prediction errors” is consonant with this understanding. Unpredicted reward is followed by increase in phasic dopaminergic activity whereas unpredicted non-reward is followed by a decrease and unchanged when reward is entirely predicted (Schultz, 2000, 2002; Schultz and Dickinson, 2000; Schultz et al., 2008). Unpredicted reward instantiates the activity of the BAS, therefore, and predicted reward maintains its operation. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Interpretation of managerial activity in terms of neuroscience is typically concerned with extreme behaviors such as corporate fraud or reckless investment (Peterson, 2007; Wargo et al., 2010a). This paper is concerned to map out the neurophysiological and cognitive mechanisms at work across the spectrum of managerial behaviors encountered in more day-to-day contexts. It proposes that the competing neuro-behavioral decisions systems (CNBDS) hypothesis (Bickel et al., 2012b) captures well the range of managerial behaviors that can be characterized as hyper- or hypo-activity in either the limbically-based impulsive system or the frontal-cortically based executive system with the corresponding level of activity encountered in the alternative brain region. This pattern of neurophysiological responding also features in the Somatic Marker Hypothesis (Damasio, 1994) and in Reinforcement Sensitivity Theory (RST; Gray and McNaughton, 2000; McNaughton and Corr, 2004), which usefully extend the thesis, for example in the direction of personality. In discussing these theories, the paper has three purposes: to clarify the role of cognitive explanation in neuro-behavioral decision theory, to propose picoeconomics (Ainslie, 1992) as the cognitive component of competing neuro-behavioral decision systems theory and to suggest solutions to the problems of imbalanced neurophysiological activity in managerial behavior. The first is accomplished through discussion of the role of picoeconomics in neuro-behavioral decision theory; the second, by consideration of adaptive-innovative cognitive styles (Kirton, 2003) in the construction of managerial teams, a theme that can now be investigated by a dedicated research program that incorporates psychometric analysis of personality types and cognitive styles involved in managerial decision-making and the underlying neurophysiological bases of such decision-making.
    Frontiers in Human Neuroscience 04/2014; 8(1):184. DOI:10.3389/fnhum.2014.00184 · 3.63 Impact Factor
Show more