Fabian Grabenhorst’s research while affiliated with University of Oxford and other places

Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied a set of metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of entropy-based metrics in studying adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.

Significance Preferences for foods high in sugar and fat are near universal and major contributors to obesity. Additionally, human food choices are sophisticated and individualistic: we choose by evaluating a food’s nutrients and sensory features and trading them against quantity and cost. To understand the mechanisms behind human-like food choices, we developed an experimental paradigm in which monkeys chose nutrient rewards offered in varying quantities. Resembling human suboptimal eating, the monkeys’ fat and sugar preferences shifted their nutrient balance away from dietary reference points. Formally defined economic values for specific nutrients and food textures explained the monkeys’ preferences and individual differences. Our findings show how human-like preferences derive from biologically critical food components and open up investigations of underlying neural mechanisms.

Animals make adaptive food choices to acquire nutrients that are essential for survival. In reinforcement learning (RL), animals choose by assigning values to options and update these values with new experiences. This framework has been instrumental for identifying fundamental learning and decision variables, and their neural substrates. However, canonical RL models do not explain how learning depends on biologically critical intrinsic reward components, such as nutrients, and related homeostatic regulation. Here, we investigated this question in monkeys making choices for nutrient-defined food rewards under varying reward probabilities. We found that the nutrient composition of rewards strongly influenced monkeys' choices and learning. The animals preferred rewards high in nutrient content and showed individual preferences for specific nutrients (sugar, fat). These nutrient preferences affected how the animals adapted to changing reward probabilities: the monkeys learned faster from preferred nutrient rewards and chose them frequently even when they were associated with lower reward probability. Although more recently experienced rewards generally had a stronger influence on monkeys' choices, the impact of reward history depended on the rewards' specific nutrient composition. A nutrient-sensitive RL model captured these processes. It updated the value of individual sugar and fat components of expected rewards from experience and integrated them into scalar values that explained the monkeys' choices. Our findings indicate that nutrients constitute important reward components that influence subjective valuation, learning and choice. Incorporating nutrient-value functions into RL models may enhance their biological validity and help reveal unrecognized nutrient-specific learning and decision computations.

Long implicated in aversive processing, the amygdala is now recognized as a key component of the brain systems that process rewards. Beyond reward valuation, recent findings from single-neuron recordings in monkeys indicate that primate amygdala neurons also play an important role in decision-making. The reward value signals encoded by amygdala neurons constitute suitable inputs to economic decision processes by being sensitive to reward contingency, relative reward quantity and temporal reward structure. During reward-based decisions, individual amygdala neurons encode both the value inputs and corresponding choice outputs of economic decision processes. The presence of such value-to-choice transitions in single amygdala neurons, together with other well-defined signatures of decision computation, indicate that a decision mechanism may be implemented locally within the primate amygdala. During social observation, specific amygdala neurons spontaneously encode these decision signatures to predict the choices of social partners, suggesting neural simulation of the partner's decision-making. The activity of these 'simulation neurons' could arise naturally from convergence between value neurons and social, self-other discriminating neurons. These findings identify single-neuron building blocks and computational architectures for decision-making and social behavior in the primate amygdala. An emerging understanding of the decision function of primate amygdala neurons can help identify potential vulnerabilities for amygdala dysfunction in human conditions afflicting social cognition and mental health.

Rewarding choice options typically contain multiple components, but neural signals in single brain voxels are scalar and primarily vary up or down. In a previous study, we had designed reward bundles that contained the same two milkshakes with independently set amounts; we had used psychophysics and rigorous economic concepts to estimate two-dimensional choice indifference curves (IC) that represented revealed stochastic preferences for these bundles in a systematic, integrated manner. All bundles on the same ICs were equally revealed preferred (and thus had same utility, as inferred from choice indifference); bundles on higher ICs (higher utility) were preferred to bundles on lower ICs (lower utility). In the current study, we used the established behavior for testing with functional magnetic resonance imaging (fMRI). We now demonstrate neural responses in reward-related brain structures of human female and male participants, including striatum, midbrain and medial orbitofrontal cortex that followed the characteristic pattern of ICs: similar responses along ICs (same utility despite different bundle composition), but monotonic change across ICs (different utility). Thus, these brain structures integrated multiple reward components into a scalar signal, well beyond the known subjective value coding of single-component rewards.SIGNIFICANCE STATEMENT:Rewards have several components, like the taste and size of an apple, but it is unclear how each component contributes to the overall value of the reward. While choice indifference curves of economic theory provide behavioural approaches to this question, it is unclear whether brain responses capture the preference and utility integrated from multiple components. We report activations in striatum, midbrain and orbitofrontal cortex that follow choice indifference curves representing behavioral preferences over and above variations of individual reward components. In addition, the concept-driven approach encourages future studies on natural, multi-component rewards that are prone to irrational choice of normal and brain-damaged individuals.

Expected Utility Theory (EUT), the first axiomatic theory of risky choice, describes choices as a utility maximization process: decision makers assign a subjective value (utility) to each choice option and choose the one with the highest utility. The continuity axiom, central to EUT and its modifications, is a necessary and sufficient condition for the definition of numerical utilities. The axiom requires decision makers to be indifferent between a gamble and a specific probabilistic combination of a more preferred and a less preferred gamble. While previous studies demonstrated that monkeys choose according to combinations of objective reward magnitude and probability, a concept-driven experimental approach for assessing the axiomatically defined conditions for maximizing subjective utility by animals is missing. We experimentally tested the continuity axiom for a broad class of gamble types in four male rhesus macaque monkeys, showing that their choice behavior complied with the existence of a numerical utility measure as defined by the economic theory. We used the numerical quantity specified in the continuity axiom to characterize subjective preferences in a magnitude-probability space. This mapping highlighted a trade-off relation between reward magnitudes and probabilities, compatible with the existence of a utility function underlying subjective value computation. These results support the existence of a numerical utility function able to describe choices, allowing for the investigation of the neuronal substrates responsible for coding such rigorously defined quantity.SIGNIFICANCE STATEMENTA common assumption of several economic choice theories is that decisions result from the comparison of subjectively assigned values (utilities). This study demonstrated the compliance of monkey behavior with the continuity axiom of Expected Utility Theory, implying a subjective magnitude-probability trade-off relation which supports the existence of numerical subjective utility directly linked to the theoretical economic framework. We determined a numerical utility measure able to describe choices, which can serve as a correlate for the neuronal activity in the quest for brain structures and mechanisms guiding decisions.

Realistic, everyday rewards contain multiple components. An apple has taste and size. However, we choose in single dimensions, simply preferring some apples to others. How can such single-dimensional preference relationships refer to multicomponent choice options? Here, we measured how stochastic choices revealed preferences for 2-component milkshakes. The preferences were intuitively graphed as indifference curves that represented the orderly integration of the 2 components as trade-off: parts of 1 component were given up for obtaining 1 additional unit of the other component without a change in preference. The well-ordered, nonoverlapping curves satisfied leave-one-out tests, followed predictions by machine learning decoders and correlated with single-dimensional Becker-DeGroot-Marschak (BDM) auction-like bids for the 2-component rewards. This accuracy suggests a decision process that integrates multiple reward components into single-dimensional estimates in a systematic fashion. In interspecies comparisons, human performance matched that of highly experienced laboratory monkeys, as measured by accuracy of the critical trade-off between bundle components. These data describe the nature of choices of multicomponent choice options and attest to the validity of the rigorous economic concepts and their convenient graphic schemes for explaining choices of human and nonhuman primates. The results encourage formal behavioral and neural investigations of normal, irrational, and pathological economic choices. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

The sixth edition of the foundational reference on cognitive neuroscience, with entirely new material that covers the latest research, experimental approaches, and measurement methodologies. Each edition of this classic reference has proved to be a benchmark in the developing field of cognitive neuroscience. The sixth edition of The Cognitive Neurosciences continues to chart new directions in the study of the biological underpinnings of complex cognition—the relationship between the structural and physiological mechanisms of the nervous system and the psychological reality of the mind. It offers entirely new material, reflecting recent advances in the field, covering the latest research, experimental approaches, and measurement methodologies. This sixth edition treats such foundational topics as memory, attention, and language, as well as other areas, including computational models of cognition, reward and decision making, social neuroscience, scientific ethics, and methods advances. Over the last twenty-five years, the cognitive neurosciences have seen the development of sophisticated tools and methods, including computational approaches that generate enormous data sets. This volume deploys these exciting new instruments but also emphasizes the value of theory, behavior, observation, and other time-tested scientific habits. Section editorsSarah-Jayne Blakemore and Ulman Lindenberger, Kalanit Grill-Spector and Maria Chait, Tomás Ryan and Charan Ranganath, Sabine Kastner and Steven Luck, Stanislas Dehaene and Josh McDermott, Rich Ivry and John Krakauer, Daphna Shohamy and Wolfram Schultz, Danielle Bassett and Nikolaus Kriegeskorte, Marina Bedny and Alfonso Caramazza, Liina Pylkkänen and Karen Emmorey, Mauricio Delgado and Elizabeth Phelps, Anjan Chatterjee and Adina Roskies

Rewarding choice options typically contain multiple components, but neural signals in single brain voxels are scalar and primarily vary up or down. In a previous study, we had designed reward bundles that contained the same two milkshakes with independently set amounts; we had used psychophysics and rigorous economic concepts to estimate two-dimensional choice indifference curves (IC) that represented revealed stochastic preferences for these bundles in a systematic, integrated manner. All bundles on the same ICs were equally revealed preferred (and thus had same utility, as inferred from choice indifference); bundles on higher ICs (higher utility) were preferred to bundles on lower ICs (lower utility). In the current study, we used the established behavior for testing with functional magnetic resonance imaging (fMRI). We now demonstrate neural responses in reward-related brain structures of human female and male participants, including striatum, midbrain and medial orbitofrontal cortex that followed the characteristic pattern of ICs: similar responses along ICs (same utility despite different bundle composition), but monotonic change across ICs (different utility). Thus, these brain structures integrated multiple reward components into a scalar signal, well beyond the known subjective value coding of single-component rewards. Significance Statement Rewards have several components, like the taste and size of an apple, but it is unclear how each component contributes to the overall value of the reward. While choice indifference curves of economic theory provide behavioural approaches to this question, it is unclear whether brain responses capture the preference and utility integrated from multiple components. We report activations in striatum, midbrain and orbitofrontal cortex that follow choice indifference curves representing behavioral preferences over and above variations of individual reward components. In addition, the concept-driven approach encourages future studies on natural, multi-component rewards that are prone to irrational choice of normal and brain-damaged individuals.

Expected Utility Theory (EUT), the first axiomatic theory of risky choice, describes choices as a utility maximization process: decision makers assign a subjective value (utility) to each choice option and choose the one with the highest utility. The continuity axiom, central to EUT and its modifications, is a necessary and sufficient condition for the definition of numerical utilities. The axiom requires decision makers to be indifferent between a gamble and a specific probabilistic combination of a more preferred and a less preferred gamble. While previous studies demonstrated that monkeys choose according to combinations of objective reward magnitude and probability, a concept-driven experimental approach for assessing the axiomatically defined conditions for maximizing subjective utility by animals is missing. We experimentally tested the continuity axiom for a broad class of gamble types in four male rhesus macaque monkeys, showing that their choice behavior complied with the existence of a numerical utility measure as defined by the economic theory. We used the numerical quantity specified in the continuity axiom to characterize subjective preferences in a magnitude-probability space. This mapping highlighted a trade-off relation between reward magnitudes and probabilities, compatible with the existence of a utility function underlying subjective value computation. These results support the existence of a numerical utility function able to describe choices, allowing for the investigation of the neuronal substrates responsible for coding such rigorously defined quantity. SIGNIFICANCE STATEMENT A common assumption of several economic choice theories is that decisions result from the comparison of subjectively assigned values (utilities). This study demonstrated the compliance of monkey behavior with the continuity axiom of Expected Utility Theory, implying a subjective magnitude-probability trade-off relation which supports the existence of numerical subjective utility directly linked to the theoretical economic framework. We determined a numerical utility measure able to describe choices, which can serve as a correlate for the neuronal activity in the quest for brain structures and mechanisms guiding decisions.