[Show abstract][Hide abstract] ABSTRACT: High impulsivity is an important risk factor for addiction with evidence from endophenotype studies. In addiction, behavioral control is shifted toward the habitual end. Habitual control can be described by retrospective updating of reward expectations in ‘model-free’ temporal-difference algorithms. Goal-directed control relies on the prospective consideration of actions and their outcomes, which can be captured by forward-planning ‘model-based’ algorithms. So far, no studies have examined behavioral and neural signatures of model-free and model-based control in healthy high-impulsive individuals. Fifty healthy participants were drawn from the upper and lower ends of 452 individuals, completing the Barratt Impulsiveness Scale. All participants performed a sequential decision-making task during functional magnetic resonance imaging (fMRI) and underwent structural MRI. Behavioral and fMRI data were analyzed by means of computational algorithms reflecting model-free and model-based control. Both groups did not differ regarding the balance of model-free and model-based control, but high-impulsive individuals showed a subtle but significant accentuation of model-free control alone. Right lateral prefrontal model-based signatures were reduced in high-impulsive individuals. Effects of smoking, drinking, general cognition or gray matter density did not account for the findings. Irrespectively of impulsivity, gray matter density in the left dorsolateral prefrontal cortex was positively associated with model-based control. The present study supports the idea that high levels of impulsivity are accompanied by behavioral and neural signatures in favor of model-free behavioral control. Behavioral results in healthy high-impulsive individuals were qualitatively different to findings in patients with the same task. The predictive relevance of these results remains an important target for future longitudinal studies.
Full-text · Article · Oct 2015 · Translational Psychiatry
[Show abstract][Hide abstract] ABSTRACT: We constantly face decisions to adapt our behavior to changing internal states and external environments. The decision-making process is proposed to involve two basic systems: a flexible, deliberative goal-directed system, and an automatic but inflexible habitual system . These two systems are highly functional when operating in balance, with considerable inter-individual variability . Maladaptive behavioral adaptation, however, occurs when the two decision-making systems are out of balance, and computational psychiatry  accounts suggest that this imbalance underlies psychiatric disorders including addiction, obsessive-compulsive disorder and impulse control disorders. We studied the balance of these two systems influenced by inter-individual factors including impulsivity and stress, and in multiple psychiatric populations, by the use of computational modeling accounts of reinforcement learning. These accounts describe goal-directed behavior with ‘model-based’ algorithms, assuming an agents’ mental model of the environment and flexible behavioral adaptation due to computationally complex forward-planning. Habitual behavior is associated with ‘model-free’ accounts of learning represented by retrospective updating from past rewards by neglecting the environmental structure, resulting in more inflexible and slower behavioral adaptation for the advantage of computational efficiency. We employed a probabilistic serial reversal task designed to examine behavioral adaptation in a dynamic environment, and combined computational modeling of choice behavior with functional and structural MRI. We show impaired behavioral adaptation under influence of stress , impulsivity and in psychiatric patients. We find evidence for reduced reliance on model-based strategies and at the neural level show reduced local brain structure and representation of model-based signatures in the ventral striatum and medial prefrontal cortex.
 Balleine, B.W. & Dickinson, A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–19 (1998).
 Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
 Stephan, K.E. & Mathys, C. Computational approaches to psychiatry. Curr. Opin. Neurobiol. 25, 85–92 (2014).
 Radenbach, C. et al. The interaction of acute and chronic stress impairs model-based behavioral control. Psychoneuroendocrinology 53, 268–280 (2015).
[Show abstract][Hide abstract] ABSTRACT: Background.
A dysfunctional differentiation between self-relevant and irrelevant information may affect the perception of environmental
stimuli as abnormally salient. The aberrant salience hypothesis assumes that positive symptoms arise from an attribution of
salience to irrelevant stimuli accompanied by the feeling of self-relevance. Self-referential processing relies on the activation
of cortical midline structures which was demonstrated to be impaired in psychosis. We investigated the neural correlates of
self-referential processing, aberrant salience attribution, and the relationship between these 2 measures across the psychosis
Twenty-nine schizophrenia patients, 24 healthy individuals with subclinical delusional ideation, and 50 healthy individuals
participated in this study. Aberrant salience was assessed behaviorally in terms of reaction times to task irrelevant cues.
Participants performed a self-reference task during fMRI in which they had to apply neutral trait words to them or to a public
figure. The correlation between self-referential processing and aberrant salience attribution was tested.
Schizophrenia patients displayed increased aberrant salience attribution compared with healthy controls and individuals with
subclinical delusional ideation, while the latter exhibited intermediate aberrant salience scores. In the self-reference task,
schizophrenia patients showed reduced activation in the ventromedial prefrontal cortex (vmPFC), but individuals with subclinical
delusional ideation did not differ from healthy controls. In schizophrenia patients, vmPFC activation correlated negatively
with implicit aberrant salience attribution.
Higher aberrant salience attribution in schizophrenia patients is related to reduced vmPFC activation during self-referential
judgments suggesting that aberrant relevance coding is reflected in decreased neural self-referential processing as well as
in aberrant salience attribution.
[Show abstract][Hide abstract] ABSTRACT: Theoretical and animal work has proposed that prefrontal cortex (PFC) glutamate inhibits dopaminergic inputs to the ventral striatum (VS) indirectly, whereas direct VS glutamatergic afferents have been suggested to enhance dopaminergic inputs to the VS. In the present study, we aimed to investigate relationships of glutamate and dopamine measures in prefrontostriatal circuitries of healthy humans. We hypothesized that PFC and VS glutamate, as well as their balance, are differently associated with VS dopamine. Glutamate concentrations in the left lateral PFC and left striatum were assessed using 3-Tesla proton magnetic resonance spectroscopy. Striatal presynaptic dopamine synthesis capacity was measured by fluorine-18-L-dihydroxyphenylalanine (F-18-FDOPA) positron emission tomography. First, a negative relationship was observed between glutamate concentrations in lateral PFC and VS dopamine synthesis capacity (n = 28). Second, a positive relationship was revealed between striatal glutamate and VS dopamine synthesis capacity (n = 26). Additionally, the intraindividual difference between PFC and striatal glutamate concentrations correlated negatively with VS dopamine synthesis capacity (n = 24). The present results indicate an involvement of a balance in PFC and striatal glutamate in the regulation of VS dopamine synthesis capacity. This notion points toward a potential mechanism how VS presynaptic dopamine levels are kept in a fine-tuned range. A disruption of this mechanism may account for alterations in striatal dopamine turnover as observed in mental diseases (e.g., in schizophrenia).
Full-text · Article · Jul 2015 · The Journal of Neuroscience : The Official Journal of the Society for Neuroscience
[Show abstract][Hide abstract] ABSTRACT: Background: Current models of obsessive-compulsive disorder (OCD) link symptomatology to alterations in cortico-striatal circuits. OCD patients show difficulties in tasks probing a balance of habitual/model-free versus goal-directed/model-based control. Gray matter volume (GMV) in cortico-striatal regions is associated with model-based control. Here we investigated structural alterations in OCD versus controls and its relation to model-based control as measured using a sequential decision-making task.
Methods: GMV of 27 OCD patients and 27 matched controls was compared and associated with OCD severity (Y-BOCS) within the patient group using VBM-DARTEL. Sequential decision-making was analyzed using computational modeling to explore the degree of model-based vs. model-free control which was associated with GMV in both groups.
Results: OCD patients showed reduced GMV in left medial OFC (t=3.85,p(FWE-SVC)<0.05) compared with controls. Within patients GMV was reduced with increasing OCD severity in the right insula (t=7.27,p(FWE_whole-brain)<0.05). We observed no behavioral group differences in omega, the relative weight of model-free vs. model-based control, and no association with disease severity. GMV in ventral striatum correlated with omega in controls (t=3.60,p(uncorr.)<0.001), but not in OCD patients resulting in a significant group by covariate interaction (t=4.50,p(FWE-SVC)<0.05).
Conclusions: Structural orbitofrontal alterations in OCD patients confirm previous meta-analyses. No difference in the balance between model-free vs. model-based control was observed in OCD patients compared with controls, although we confirm a positive association between ventral striatal GMV and model-based actions only in controls. This absent association in OCD patients suggests that behavioral control in OCD is differently related to fronto-striatal integrity.
Keywords: Obsessive-Compulsive Disorder, Gray Matter Volume, Decision Making, Computational Modeling
Supported by: DFG SCHL1969/2-1
[Show abstract][Hide abstract] ABSTRACT: Dual system theories suggest that behavioral control is parsed between a deliberative "model-based" and a more reflexive "model-free" system. A balance of control exerted by these systems is thought to be related to dopamine neurotransmission. However, in the absence of direct measures of human dopamine, it remains unknown whether this reflects a quantitative relation with dopamine either in the striatum or other brain areas. Using a sequential decision task performed during functional magnetic resonance imaging, combined with striatal measures of dopamine using [(18)F]DOPA positron emission tomography, we show that higher presynaptic ventral striatal dopamine levels were associated with a behavioral bias toward more model-based control. Higher presynaptic dopamine in ventral striatum was associated with greater coding of model-based signatures in lateral prefrontal cortex and diminished coding of model-free prediction errors in ventral striatum. Thus, interindividual variability in ventral striatal presynaptic dopamine reflects a balance in the behavioral expression and the neural signatures of model-free and model-based control. Our data provide a novel perspective on how alterations in presynaptic dopamine levels might be accompanied by a disruption of behavioral control as observed in aging or neuropsychiatric diseases such as schizophrenia and addiction.
Full-text · Article · Jan 2015 · Proceedings of the National Academy of Sciences
[Show abstract][Hide abstract] ABSTRACT: Online: http://www.sciencedirect.com/science/article/pii/S0306453014004715
It is suggested that acute stress shifts behavioral control from goal-directed, model-based towards habitual, model-free strategies. Recent findings indicate that interindividual differences in cortisol stress response influence model-based decision-making. Although not yet investigated in humans, animal studies show that chronic stress also shifts decision-making towards more habitual behavior. Here, we ask whether acute stress and individual vulnerability factors, such as stress reactivity and previous exposure to stressful life events, impact the balance between the model-free and the model-based control systems. To test this, 39 male participants (21–30 years old) were exposed to a potent psychosocial stressor (Trier Social Stress Test) and a control condition in a within-subjects design before they performed a sequential decision-making task which evaluates the balance between the two systems. Physiological and subjective stress reactivity was assessed before, during, and after acute stress exposure. By means of computational modeling, we demonstrate that interindividual variability in stress reactivity predicts impairments in model-based decision-making. Whereas acute psychosocial stress did not alter model-based behavioral control, we found chronic and acute stress to interact in their detrimental effect on decision-making: subjects with high but not low chronic stress levels as indicated by stressful life events exhibited reduced model-based control in response to acute psychosocial stress. These findings emphasize that stress reactivity and chronic stress play an important role in mediating the relationship between stress and decision-making. Our results might stimulate new insights into the interplay between chronic and acute stress, attenuated model-based control, and the pathogenesis of various psychiatric diseases.
Full-text · Article · Jan 2015 · Psychoneuroendocrinology
[Show abstract][Hide abstract] ABSTRACT: Drugs of abuse elicit dopamine release in the ventral striatum, possibly biasing dopamine-driven reinforcement learning towards drug-related reward at the expense of non-drug-related reward. Indeed, in alcohol-dependent patients, reactivity in dopaminergic target areas is shifted from non-drug-related stimuli towards drug-related stimuli. Such ‘hijacked’ dopamine signals may impair flexible learning from non-drug-related rewards, and thus promote craving for the drug of abuse. Here, we used functional magnetic resonance imaging to measure ventral striatal activation by reward prediction errors (RPEs) during a probabilistic reversal learning task in recently detoxified alcohol-dependent patients and healthy controls (N = 27). All participants also underwent 6-[18F]fluoro-DOPA positron emission tomography to assess ventral striatal dopamine synthesis capacity. Neither ventral striatal activation by RPEs nor striatal dopamine synthesis capacity differed between groups. However, ventral striatal coding of RPEs correlated inversely with craving in patients. Furthermore, we found a negative correlation between ventral striatal coding of RPEs and dopamine synthesis capacity in healthy controls, but not in alcohol-dependent patients. Moderator analyses showed that the magnitude of the association between dopamine synthesis capacity and RPE coding depended on the amount of chronic, habitual alcohol intake. Despite the relatively small sample size, a power analysis supports the reported results. Using a multimodal imaging approach, this study suggests that dopaminergic modulation of neural learning signals is disrupted in alcohol dependence in proportion to long-term alcohol intake of patients. Alcohol intake may perpetuate itself by interfering with dopaminergic modulation of neural learning signals in the ventral striatum, thus increasing craving for habitual drug intake.
No preview · Article · Dec 2014 · European Journal of Neuroscience
[Show abstract][Hide abstract] ABSTRACT: [(11)C]P943 is a novel, highly selective 5-HT1B PET radioligand. The aim of this study was to determine the test-retest reliability of [(11)C]P943 using two different modeling methods and to perform a power analysis with each quantification technique.
Seven healthy volunteers underwent two PET scans on the same day. Regions of interest (ROIs) were the amygdala, hippocampus, pallidum, putamen, insula, frontal, anterior cingulate, parietal, temporal and occipital cortices, and cerebellum. Two multilinear radioligand quantification techniques were used to estimate binding potential: MA1, using arterial input function data, and the second version of the multilinear reference tissue model analysis (MRTM2), using the cerebellum as the reference region. Between-scan percent variability and intraclass correlation coefficients (ICC) were used to assess test-retest reliability. We also performed power analyses to determine the method that would allow the least number of subjects using within-subject or between-subject study designs. A voxel-wise ICC analysis for MRTM2 BPND was performed for the whole brain and all the ROIs studied.
Mean percent variability between two scans across regions ranged between 0.4 % and 12.4 % for MA1 BPND, 0.5 % and 11.5 % for MA1 BPP, 16.7 % and 28.3 % for MA1 BPF, and between 0.2 % and 5.4 % for MRTM2 BPND. The power analyses showed a greater number of subjects were required using MA1 BPF compared with other outcome measures for both within-subject and between-subject study designs. ICC values were the highest using MRTM2 BPND and the lowest with MA1 BPF in ten ROIs. Small regions and regions with low binding had lower ICC values than large regions and regions with high binding.
Reliable measures of 5-HT1B receptor binding can be obtained using the novel PET radioligand [(11)C]P943. Quantification of 5-HT1B receptor binding with MRTM2 BPND and with MA1 BPP provided the least variability and optimal power for within-subject and between-subject designs.
No preview · Article · Nov 2014 · European journal of nuclear medicine and molecular imaging
[Show abstract][Hide abstract] ABSTRACT: Background:
Human and animal work suggests a shift from goal-directed to habitual decision-making in addiction. However, the evidence for this in human alcohol dependence is as yet inconclusive.
Twenty-six healthy controls and 26 recently detoxified alcohol-dependent patients underwent behavioral testing with a 2-step task designed to disentangle goal-directed and habitual response patterns.
Alcohol-dependent patients showed less evidence of goal-directed choices than healthy controls, particularly after losses. There was no difference in the strength of the habitual component. The group differences did not survive controlling for performance on the Digit Symbol Substitution Task.
Chronic alcohol use appears to selectively impair goal-directed function, rather than promoting habitual responding. It appears to do so particularly after nonrewards, and this may be mediated by the effects of alcohol on more general cognitive functions subserved by the prefrontal cortex.
Full-text · Article · Oct 2014 · Neuropsychobiology
[Show abstract][Hide abstract] ABSTRACT: Impulsivity is a multidimensional construct that has been suggested as a vulnerability factor for several psychiatric disorders, especially addiction disorders. Poor response inhibition may constitute one facet of impulsivity. Trait impulsivity can be assessed by self-report questionnaires such as the widely used Barratt Impulsiveness Scale (BIS-11). However, regarding the multidimensionality of impulsivity different concepts have been proposed, in particular the UPPS self-report questionnaire (‘Urgency’, ‘Lack of Premeditation’, ‘Lack of Perseverance’, ‘Sensation Seeking’) that is based on a factor analytic approach. The question as to which aspects of trait impulsivity map on individual differences of the behavioral and neural correlates of response inhibition so far remains unclear.
[Show abstract][Hide abstract] ABSTRACT: In experimental psychology different experiments have been developed to assess goal-directed as compared to habitual control over instrumental decisions. Similar to animal studies selective devaluation procedures have been used. More recently sequential decision-making tasks have been designed to assess the degree of goal-directed vs. habitual choice behavior in terms of an influential computational theory of model-based compared to model-free behavioral control. As recently suggested, different measurements are thought to reflect the same construct. Yet, there has been no attempt to directly assess the construct validity of these different measurements. In the present study, we used a devaluation paradigm and a sequential decision-making task to address this question of construct validity in a sample of 18 healthy male human participants. Correlational analysis revealed a positive association between model-based choices during sequential decisions and goal-directed behavior after devaluation suggesting a single framework underlying both operationalizations and speaking in favor of construct validity of both measurement approaches. Up to now, this has been merely assumed but never been directly tested in humans.
Full-text · Article · Aug 2014 · Frontiers in Human Neuroscience
[Show abstract][Hide abstract] ABSTRACT: The fronto-limbic network interaction, driven by glutamatergic and dopaminergic neurotransmission, represents a core mechanism of motivated behavior and personality traits. Reward seeking behavior undergoes tremendous changes in adolescence paralleled by neurobiological changes of this network including the prefrontal cortex, striatum and amygdala. Since fronto-limbic dysfunctions also underlie major psychiatric diseases beginning in adolescence, this investigation focuses on network characteristics separating adolescents from adults. To investigate differences in network interactions, the brain reward system activity (slot machine task) together with frontal glutamate concentration (anterior cingulate cortex, ACC) was measured in 28 adolescents and 26 adults employing functional magnetic resonance imaging and magnetic resonance spectroscopy, respectively. An inverse coupling of glutamate concentrations in the ACC and activation of the ventral striatum was observed in adolescents. Further, amygdala response in adolescents was negatively correlated with the personality trait impulsivity. For adults, no significant associations of network components or correlations with impulsivity were found. The inverse association between frontal glutamate concentration and striatal activation in adolescents is in line with the triadic model of motivated behavior stressing the important role of frontal top-down inhibition on limbic structures. Our data identified glutamate as the mediating neurotransmitter of this inhibitory process and demonstrates the relevance of glutamate on the reward system and related behavioral traits like impulsivity. This fronto-limbic coupling may represent a vulnerability factor for psychiatric disorders starting in adolescence but not in adulthood.
No preview · Article · Jul 2014 · Brain Structure and Function
[Show abstract][Hide abstract] ABSTRACT: Background: Substance addiction (e.g. Alcohol addiction) and other psychiatric conditions (e.g. Binge Eating Disorder, BED) might be characterized by the loss of behavioral control particularly when behavioral adaptation is required. It has been hypothesized that Alcohol addiction and BED share as a common ground a failure of behavioral adaptation, which potentially results from aberrant Reinforcement Learning and its neural teaching signals. However, studies including both patient groups are scarce which renders direct comparisons difficult and speculative.
Methods: We present behavioral and fMRI data from 18 alcohol dependent patients, 10 patients diagnosed with BED and 20 healthy, matched controls. Participants performed an operant volatility task designed to assess behavioral adaptation in changing environmental conditions. We use computational modeling of Reinforcement Learning to describe changes in learning dynamically and to derive quantitative parameters for behavioral and fMRI analyses.
Results: We found significantly (p <.05) impaired task performance shown by less correct choices for both patient groups compared to controls indicating impaired behavioral adaptation over the course of the experiment. In all participants, fMRI revealed activation of a fronto-striatal network (ventral and dorsal striatum, anterior cingulate cortex, medial and lateral prefrontal cortices) during prediction error (PE) modulated feedback. However, coding of PEs appeared to be reduced in the patient groups which might underlie the
observed behavioral impairment.
Conclusions: Impaired behavioral adaptation seems to be a common pathway in nosologically different psychiatric conditions both characterized by the loss of behavioral control. Preliminary analyses indicate that a disrupted neural coding of RL-signals like PEs might be the neural basis of this impairment.
Keywords: Transdiagnostic approach, Reinforcement Learning, Addiction/ Addiction-like Disorders, Computional Modeling, fMRI
Supported By: SCHL1969/1-1