-
[show abstract]
[hide abstract]
ABSTRACT: In this chapter, we address the question of delay discounting from the perspective of computational neuroscience. We first review why agents must discount future rewards in order to make reasoned decisions and then discuss the role of delay discounting in the context of the temporal-difference reinforcement learning family of decision-making algorithms. These algorithms require exponential discounting functions to achieve mathematical stability, but as noted in the other chapters in this volume, humans and other animals show hyperbolic discounting functions. In the second half of the chapter, we review four theories for this discrepancy: (a) competition between two decision-making systems, (b) interactions between multiple exponential discounting functions, (c) normalization by estimates of average reward, and (d) effects of errors in temporal perception. All four theories are likely to contribute to the effect of hyperbolic discounting. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
10/2012;
-
[show abstract]
[hide abstract]
ABSTRACT: The encoding and storage of experience by the hippocampus is essential for the formation of episodic memories and the transformation of individual experiences into semantic structures such as maps and schemas. The rodent hippocampus compresses ongoing experience into repeating theta sequences, but the factors determining the content of theta sequences are not understood. Here we first show that the spatial paths represented by theta sequences in rats extend farther in front of the rat during acceleration and higher running speeds and begin farther behind the rat during deceleration. Second, the length of the path is directly related to the length of the theta cycle and the number of gamma cycles in it. Finally, theta sequences represent the environment in segments or 'chunks'. These results imply that information encoded in theta sequences is subject to powerful modulation by behavior and task variables. Furthermore, these findings suggest a potential mechanism for the cognitive 'chunking' of experience.
Nature Neuroscience 06/2012; 15(7):1032-9. · 15.53 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Decisions result from an interaction between multiple functional systems acting in parallel to process information in very different ways, each with strengths and weaknesses. In this review, the authors address three action-selection components of decision-making: The Pavlovian system releases an action from a limited repertoire of potential actions, such as approaching learned stimuli. Like the Pavlovian system, the habit system is computationally fast but, unlike the Pavlovian system permits arbitrary stimulus-action pairings. These associations are a "forward'' mechanism; when a situation is recognized, the action is released. In contrast, the deliberative system is flexible but takes time to process. The deliberative system uses knowledge of the causal structure of the world to search into the future, planning actions to maximize expected rewards. Deliberation depends on the ability to imagine future possibilities, including novel situations, and it allows decisions to be taken without having previously experienced the options. Various anatomical structures have been identified that carry out the information processing of each of these systems: hippocampus constitutes a map of the world that can be used for searching/imagining the future; dorsal striatal neurons represent situation-action associations; and ventral striatum maintains value representations for all three systems. Each system presents vulnerabilities to pathologies that can manifest as psychiatric disorders. Understanding these systems and their relation to neuroanatomy opens up a deeper way to treat the structural problems underlying various disorders.
The Neuroscientist 04/2012; 18(4):342-59. · 4.57 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Nanoelectrodes are an emerging biomedical technology that can be used to record intracellular membrane potentials from neurons while causing minimal damage during membrane penetration. Current nanoelectrode designs, however, have low aspect ratios or large substrates and thus are not suitable for recording from neurons deep within complex natural structures, such as brain slices.
We describe a novel nanoelectrode design that uses nanowires grown on the ends of microwire recording electrodes similar to those frequently used in vivo.
We demonstrate that these nanowires can record intracellular action potentials in a rat brain slice preparation and in isolated leech ganglia.
Nanoelectrodes have the potential to revolutionize intracellular recording methods in complex neural tissues, to enable new multielectrode array technologies and, ultimately, to be used to record intracellular signals in vivo.
Nanomedicine 04/2012; 7(6):847-53. · 5.05 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Although delay discounting, the attenuation of the value of future rewards, is a robust finding, the mechanism of discounting is not known. We propose a potential mechanism for delay discounting such that discounting emerges from a search process that is trying to determine what rewards will be available in the future. In this theory, the delay dependence of the discounting of future expected rewards arises from three assumptions. First, that the evaluation of outcomes involves a search process. Second, that the value is assigned to an outcome proportionally to how easy it is to find. Third, that outcomes that are less delayed are typically easier for the search process to find. By relaxing this third assumption (e.g. by assuming that episodically-cued outcomes are easier to find), our model suggests that it is possible to dissociate discounting from delay. Our theory thereby explains the empirical result that discounting is slower to episodically-imagined outcomes, because these outcomes are easier for the search process to find. Additionally, the theory explains why improving cognitive resources such as working memory slows discounting, by improving searches and thereby making rewards easier to find. The three assumptions outlined here are likely to be instantiated during deliberative decision-making, but are unlikely in habitual decision-making. We model two simple implementations of this theory and show that they unify empirical results about the role of cognitive function in delay discounting, and make new neural, behavioral, and pharmacological predictions.
European Journal of Neuroscience 04/2012; 35(7):1052-64. · 3.63 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Precommitment, or taking away a future choice from oneself, is a mechanism for overcoming impulsivity. Here we review recent work suggesting that precommitment can be best explained through a distributed decision-making system with multiple discounting rates. This model makes specific predictions about precommitment behavior and is especially interesting in light of the emerging multiple-systems view of decision-making, in which functional systems with distinct neural substrates use different computational strategies to optimize decisions. Given the growing consensus that impulsivity constitutes a common point of breakdown in decision-making processes, with common neural and computational mechanisms across multiple psychiatric disorders, it is useful to translate precommitment into the common language of temporal difference reinforcement learning that unites many of these behavioral and neural data.
Frontiers in Neuroscience 01/2012; 6:138.
-
[show abstract]
[hide abstract]
ABSTRACT: This chapter describes addiction as a failure of decision-making systems. Existing computational theories of addiction have
been based on temporal difference (TD) learning as a quantitative model for decision-making. In these theories, drugs of abuse
create a non-compensable TD reward prediction error signal that causes pathological overvaluation of drug-seeking choices.
However, the TD model is too simple to account for all aspects of decision-making. For example, TD requires a state-space
over which to learn. The process of acquiring a state-space, which involves both situation classification and learning causal
relationships between states, presents another set of vulnerabilities to addiction. For example, problem gambling may be partly
caused by a misclassification of the situations that lead to wins and losses. Extending TD to include state-space learning
also permits quantitative descriptions of how changing representations impacts patterns of intertemporal choice behavior,
potentially reducing impulsive choices just by changing cause-effect beliefs. This approach suggests that addicts can learn
healthy representations to recover from addiction. All the computational models of addiction published so far are based on
learning models that do not attempt to look ahead into the future to calculate optimal decisions. Adeeper understanding of
how decision-making breaks down in addiction will certainly require addressing the interaction of drugs with model-based look-ahead
decision mechanisms, atopic that remains unexplored.
12/2011: pages 163-187;
-
[show abstract]
[hide abstract]
ABSTRACT: Many medical devices that are implanted in the body use wires or wireless radiofrequency telemetry to communicate with circuitry outside the body. However, the wires are a common source of surgical complications, including breakage, infection and electrical noise. In addition, radiofrequency telemetry requires large amounts of power and results in low-efficiency transmission through biological tissue. As an alternative, the conductive properties of the body can be used to enable wireless communication with implanted devices. In this article, several methods of intrabody communication are described and compared. In addition to reducing the complications that occur with current implantable medical devices, intrabody communication can enable novel types of miniature devices for research and clinical applications.
Expert Review of Medical Devices 07/2011; 8(4):427-33. · 2.63 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Hippocampal pyramidal cells can be divided into place cells, which fire action potentials when an animal is in specific locations, and silent cells, which are not spatially selective. In this issue of Neuron, Epsztein et al. find intracellular differences between place and silent cells by using whole-cell recordings in freely moving rats.
Neuron 04/2011; 70(1):3-5. · 14.74 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Intertemporal choice has provided important insights into understanding addiction, predicted drug-dependence status, and outcomes of treatment interventions. However, such analyses have largely been based on the choice of a single commodity available either immediately or later (e.g., money now vs. money later). In real life, important choices for those with addiction depend on making decisions across commodities, such as between drug and non-drug reinforcers. To date, no published study has systematically evaluated intertemporal choice using all combinations of a drug and a non-drug commodity.
In this study, we examine the interaction between intertemporal choice and commodity type in the decision-making process of cocaine-dependent individuals.
This study of 47 treatment-seeking cocaine addicts analyzes intertemporal choices of two commodities (equated amounts of cocaine and money), specifically between cocaine now vs. cocaine later (C-C), money now vs. money later (M-M), cocaine now vs. money later (C-M), and money now vs. cocaine later (M-C).
Cocaine addicts discounted significantly more in the C-C condition than in M-M (P = 0.032), consistent with previous reports. Importantly, the two cross-commodity discounting conditions produced different results. Discounting in C-M was intermediate to the C-C and M-M rates, while the greatest degree of discounting occurred in M-C.
These data indicate that the menu of commodities offered alter discounting rates in intertemporal choice and that the greatest rate is obtained when the drug is the later available commodity. Implications for understanding intertemporal choices and addiction are addressed.
Psychopharmacologia 04/2011; 217(2):177-87. · 4.08 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Extensive evidence implicates the ventral striatum in multiple distinct facets of action selection. Early work established a role in modulating ongoing behavior, as engaged by the energizing and directing influences of motivationally relevant cues and the willingness to expend effort in order to obtain reward. More recently, reinforcement learning models have suggested the notion of ventral striatum primarily as an evaluation step during learning, which serves as a critic to update a separate actor. Recent computational and experimental work may provide a resolution to the differences between these two theories through a careful parsing of behavior and the instrinsic heterogeneity that characterizes this complex structure.
Current opinion in neurobiology 03/2011; 21(3):387-92. · 7.21 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Both humans and non-human animals have the ability to navigate and make decisions within complex environments. This ability is largely dependent upon learning and memory processes, many of which are known to depend on NMDA-sensitive receptors. When humans come to difficult decisions they often pause to deliberate over their choices. Similarly, rats pause at difficult choice points. This behavior, known as vicarious trial and error (VTE), is hippocampally dependent and entails neurophysiological representations of expectations of future outcomes in hippocampus and downstream structures. In order to determine the dependence of VTE behaviors on NMDA-sensitive receptors, we tested rats on a Multiple-T choice task with a reward-delivery reversal known to elicit VTE. Rats under the influence of NMDA-receptor antagonists (CPP) showed a significant reduction in VTE, particularly at the reward reversal, implying a role for NMDA-sensitive receptors in the generation of vicarious trial and error behaviors.
Neurobiology of Learning and Memory 02/2011; 95(3):376-84. · 3.42 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: A functional interaction between the hippocampal formation and the ventral striatum is thought to contribute to the learning and expression of associations between places and rewards. However, the mechanism of how such associations may be learned and used is currently unknown. We recorded neural ensembles and local field potentials from the ventral striatum and CA1 simultaneously as rats ran a modified T-maze. Theta-modulated cells in ventral striatum almost invariably showed firing phase precession relative to the hippocampal theta rhythm. Across the population of ventral striatal cells, phase precession was preferentially associated with an anticipatory ramping of activity up to the reward sites. In contrast, CA1 population activity and phase precession were distributed more uniformly. Ventral striatal phase precession was stronger to hippocampal than ventral striatal theta and was accompanied by increased theta coherence with hippocampus, suggesting that this effect is hippocampally derived. These results suggest that the firing phase of ventral striatal neurons contains motivationally relevant information and that phase precession serves to bind hippocampal place representations to ventral striatal representations of reward.
Journal of Neuroscience 02/2011; 31(8):2843-54. · 7.11 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Decision-making studies across different domains suggest that decisions can arise from multiple, parallel systems in the brain: a flexible system utilizing action-outcome expectancies and a more rigid system based on situation-action associations. The hippocampus, ventral striatum, and dorsal striatum make unique contributions to each system, but how information processing in each of these structures supports these systems is unknown. Recent work has shown covert representations of future paths in hippocampus and of future rewards in ventral striatum. We developed analyses in order to use a comparative methodology and apply the same analyses to all three structures. Covert representations of future paths and reward were both absent from the dorsal striatum. In contrast, dorsal striatum slowly developed situation representations that selectively represented action-rich parts of the task. This triple dissociation suggests that the different roles these structures play are due to differences in information-processing mechanisms.
Neuron 07/2010; 67(1):25-32. · 14.74 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Replay of behavioral sequences in the hippocampus during sharp wave ripple complexes (SWRs) provides a potential mechanism for memory consolidation and the learning of knowledge structures. Current hypotheses imply that replay should straightforwardly reflect recent experience. However, we find these hypotheses to be incompatible with the content of replay on a task with two distinct behavioral sequences (A and B). We observed forward and backward replay of B even when rats had been performing A for >10 min. Furthermore, replay of nonlocal sequence B occurred more often when B was infrequently experienced. Neither forward nor backward sequences preferentially represented highly experienced trajectories within a session. Additionally, we observed the construction of never-experienced novel-path sequences. These observations challenge the idea that sequence activation during SWRs is a simple replay of recent experience. Instead, replay reflected all physically available trajectories within the environment, suggesting a potential role in active learning and maintenance of the cognitive map.
Neuron 03/2010; 65(5):695-705. · 14.74 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: A vast literature implicates the ventral striatum in the processing of reward-related information and in mediating the impact of such information on behavior. It is characterized by heterogeneity at the local circuit, connectivity, and functional levels. A tool for dissecting this complex structure that has received relatively little attention until recently is the analysis of ventral striatal local field potential oscillations, which are more prominent in the gamma band compared to the dorsal striatum. Here we review recent results on gamma oscillations recorded from freely moving rats. Ventral striatal gamma separates into distinct frequency bands (gamma-50 and gamma-80) with distinct behavioral correlates, relationships to different inputs, and separate populations of phase-locked putative fast-spiking interneurons. Fast switching between gamma-50 and gamma-80 occurs spontaneously but is influenced by reward delivery as well as the application of dopaminergic drugs. These results provide novel insights into ventral striatal processing and highlight the importance of considering fast-timescale dynamics of ventral striatal activity.
Frontiers in Neuroscience 01/2010; 4:28.
-
[show abstract]
[hide abstract]
ABSTRACT: Decisions can arise in different ways, such as from a gut feeling, doing what worked last time, or planful deliberation. Different decision-making systems are dissociable behaviorally, map onto distinct brain systems, and have different computational demands. For instance, "model-free" decision strategies use prediction errors to estimate scalar action values from previous experience, while "model-based" strategies leverage internal forward models to generate and evaluate potentially rich outcome expectancies. Animal learning studies indicate that expectancies may arise from different sources, including not only forward models but also Pavlovian associations, and the flexibility with which such representations impact behavior may depend on how they are generated. In the light of these considerations, we review the results of van der Meer and Redish (2009a), who found that ventral striatal neurons that respond to reward delivery can also be activated at other points, notably at a decision point where hippocampal forward representations were also observed. These data suggest the possibility that ventral striatal reward representations contribute to model-based expectancies used in deliberative decision making.
Frontiers in Neuroscience 01/2010; 4:6.
-
[show abstract]
[hide abstract]
ABSTRACT: Addiction and many other disorders are linked to impulsivity, where a suboptimal choice is preferred when it is immediately available. One solution to impulsivity is precommitment: constraining one's future to avoid being offered a suboptimal choice. A form of impulsivity can be measured experimentally by offering a choice between a smaller reward delivered sooner and a larger reward delivered later. Impulsive subjects are more likely to select the smaller-sooner choice; however, when offered an option to precommit, even impulsive subjects can precommit to the larger-later choice. To precommit or not is a decision between two conditions: (A) the original choice (smaller-sooner vs. larger-later), and (B) a new condition with only larger-later available. It has been observed that precommitment appears as a consequence of the preference reversal inherent in non-exponential delay-discounting. Here we show that most models of hyperbolic discounting cannot precommit, but a distributed model of hyperbolic discounting does precommit. Using this model, we find (1) faster discounters may be more or less likely than slow discounters to precommit, depending on the precommitment delay, (2) for a constant smaller-sooner vs. larger-later preference, a higher ratio of larger reward to smaller reward increases the probability of precommitment, and (3) precommitment is highly sensitive to the shape of the discount curve. These predictions imply that manipulations that alter the discount curve, such as diet or context, may qualitatively affect precommitment.
Frontiers in Behavioral Neuroscience 01/2010; 4:184.
-
[show abstract]
[hide abstract]
ABSTRACT: A tetrode is a bundle of four microwires that can record from multiple neurons simultaneously in the brain of a freely moving animal. Tetrodes are usually electroplated to reduce impedances from 2-3 MΩ to 200-500 kΩ (measured at 1 kHz), which increases the signal-to-noise ratio and allows for the recording of small amplitude signals. Tetrodes with even lower impedances could improve neural recordings but cannot be made using standard electroplating methods without shorting. We were able to electroplate tetrodes to 30-70 kΩ by adding polyethylene glycol (PEG) or multi-walled carbon nanotube (MWCNT) solutions to a commercial gold-plating solution. The MWCNTs and PEG acted as inhibitors in the electroplating process and created large-surface-area, low-impedance coatings on the tetrode tips.
Sensors and Actuators A Physical 12/2009; 156(2):388-393. · 1.80 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: This mini-symposium aims to integrate recent insights from anatomy, behavior, and neurophysiology, highlighting the anatomical organization, behavioral significance, and information-processing mechanisms of corticostriatal interactions. In this summary of topics, which is not meant to provide a comprehensive survey, we will first review the anatomy of corticostriatal circuits, comparing different ways by which "loops" of cortical-basal ganglia circuits communicate. Next, we will address the causal importance and systems-neurophysiological mechanisms of corticostriatal interactions for memory, emphasizing the communication between hippocampus and ventral striatum during contextual conditioning. Furthermore, ensemble recording techniques have been applied to compare information processing in the dorsal and ventral striatum to predictions from reinforcement learning theory. We will next discuss how neural activity develops in corticostriatal areas when habits are learned. Finally, we will evaluate the role of GABAergic interneurons in dynamically transforming cortical inputs into striatal output during learning and decision making.
Journal of Neuroscience 10/2009; 29(41):12831-8. · 7.11 Impact Factor
