John P O'Doherty

California Institute of Technology | CIT · Division of the Humanities and Social Sciences

Learning occurs across multiple timescales, with fast learning crucial for adapting to sudden environmental changes, and slow learning beneficial for extracting robust knowledge from multiple events. Here we asked if miscalibrated fast vs slow learn­ing can lead to maladaptive decision-making in individuals with problem gambling. We recruited parti...

To navigate our complex social world, it is crucial to deploy multiple learning strategies, such as learning from directly experiencing action outcomes or from observing other people’s behavior. Despite the prevalence of experiential and observational learning in humans and other social animals, it remains unclear how people favor one strategy over...

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to...

The value and uncertainty associated with choice alternatives constitute critical features relevant for decisions. However, the manner in which reward and risk representations are temporally organized in the brain remains elusive. Here we leverage the spatiotemporal precision of intracranial electroencephalography, along with a simple card game des...

Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus and amygdala while patients of both sexes performed an appetitive Pavlovia...

Learning occurs across multiple timescales, with fast learning crucial for adapting to sudden environmental changes, and slow learning beneficial for extracting robust knowledge from multiple events. Here we asked if miscalibrated fast vs slow learning can lead to maladaptive decision-making in individuals with gambling disorder. Participants with...

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectatio...

Over the last two decades, the model-based approach to analysing functional magnetic resonance imaging (fMRI) data has been adopted across the cognitive neurosciences to study how computations are implemented in the brain. In this time, methods have advanced along both computational modelling and neuroimaging domains. This chapter aims to provide a...

One's ability to infer the goals and intentions of others is crucial for social interactions, and such social capabilities are broadly distributed across individuals. Autism-like traits (i.e., traits associated with autism spectrum disorder (ASD)) have been associated with reduced social inference, yet the underlying computational principles and so...

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across d...

When encountering a novel situation, an intelligent agent needs to find out which actions are most beneficial for interacting with that environment. However, the range of possible actions that could be selected is virtually unlimited, making the problem of determining which subset of actions should be drawn from to begin exploration extremely chall...

The value and uncertainty associated with choice alternatives constitute critical features along which decisions are made. While the neural substrates supporting reward and risk processing have been investigated, the temporal organization by which these computations are encoded remains elusive. Here we leverage the high spatiotemporal precision of...

To navigate our complex social world, it is crucial for people to deploy multiple learning strategies, such as learning from directly experiencing the outcomes of one’s actions – experiential learning (EL) – as well as learning from observing the behavior of other people – observational learning (OL). Despite the prevalence of EL and OL in humans a...

Adaptive behavior in real-world environments demands that choices integrate over several variables, including the novelty of the options under consideration, their expected value, and uncertainty in value estimation. We recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex (vmPFC) and dorsal anterior...

Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus and amygdala neurons while patients performed a sequential Pavlovian condi...

Pavlovian learning depends on multiple and parallel associations leading to distinct classes of conditioned responses that vary in their flexibility following changes in the value of an associated outcome. Here, we aimed to differentiate brain areas involved in learning and encoding associations that are sensitive to changes in the value of an outc...

Little is known about how the brain computes the perceived aesthetic value of complex stimuli such as visual art. Here, we used computational methods in combination with functional neuroimaging to provide evidence that the aesthetic value of a visual stimulus is computed in a hierarchical manner via a weighted integration over both low and high lev...

The dual-process theory of action control postulates that there are two competitive and complementary mechanisms that control our behavior: a goal-directed system that executes deliberate actions, explicitly aimed toward a particular outcome, and a habitual system that autonomously execute well-learned actions, typically following an encounter with...

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model...

Both novelty and uncertainty are potent features guiding exploration; however, they are often experimentally conflated, and an understanding of how they interact to regulate the balance between exploration and exploitation has proved elusive. Using a task designed to decouple the influence of novelty and uncertainty, we identify separable mechanism...

Background Anorexia nervosa (AN) is a chronic and disabling psychiatric condition characterized by low hedonic drive towards food, and is thought to be inclusive of altered dimensions of reward processing. Whether there exists a fundamental aberrancy in the capacity to acquire and maintain de novo hedonic associations—a critical component of hedoni...

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision making across d...

Diminished motivation to pursue and obtain primary and secondary rewards has been demonstrated in anorexia nervosa (AN). However, the neurobehavioral mechanisms underlying the behavioral activation component of aberrant reward motivation remains incompletely understood. This work aims to explore this underexplored facet of reward motivation in AN....

Among the most challenging questions in the field of neuroaesthetics concerns how a piece of art comes to be liked in the first place. That is, how can the brain rapidly process a stimulus to form an aesthetic judgment even for stimuli never before encountered? In the article under discussion in this chapter, by leveraging computational methods in...

Neuroscience joins the long history of discussions about aesthetics in psychology, philosophy, art history, and the creative arts. In this volume, leading scholars in this nascent field reflect on the promise of neuroaesthetics to enrich our understanding of this universal yet diverse facet of human experience. The volume will inform and stimulate...

Adaptive behavior in real-world environments demands that choices integrate over several variables, including the novelty of the options under consideration, their expected value, and uncertainty in value estimation. We recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex (vmPFC) and dorsal anterior...

Recent evidence suggests that both novelty and uncertainty act as potent features guiding exploration. However, these variables are often conflated with each other experimentally, and an understanding of how these attributes interact to regulate the balance between exploration and exploitation has proved elusive. Using a novel task designed to deco...

Recent evidence suggests that both novelty and uncertainty act as potent features guiding exploration. However, these variables are often conflated with each other experimentally, and an understanding of how these attributes interact to regulate the balance between exploration and exploitation has proved elusive. Using a novel task designed to deco...

Here we argue that the assignment of subjective value to potential outcomes at the time of decision-making is an active process, in which individual features of a potential outcome of varying degrees of abstraction are represented hierarchically and integrated in a weighted fashion to produce an overall value judgment. We implicate the lateral orbi...

Anorexia nervosa (AN) is a difficult to treat, pernicious psychiatric disorder that has been linked to decision-making abnormalities. We examined the structural characteristics of habitual and goal-directed decision-making circuits and their connecting white matter tracts in 32 AN and 43 healthy controls across two independent data sets of adults a...

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning—whether it be learning about the value of stimuli or about the value of actions; the natu...

It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Here, we developed and tested a computational framework to investigate how aesthetic values are formed. We show that it is possible to explain human preferences for a visual art piece based on a mixture of low-...

In order to make decisions, we often seek and integrate information coming from other people, while at times also keeping track of the knowledge other people acquire from observing our own actions. In this chapter, we examine the computational mechanisms and the involvement of mentalizing when we learn from observing other people and when we engage...

Over the past three decades, functional MRI (fMRI) has become key to study how cognitive processes are implemented in the human brain. However, the question of whether participants recruited into fMRI studies differ from participants recruited into other study contexts has received little to no attention. This is particularly pertinent when effects...

It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a “Mixture of Experts” in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavi...

We review progress and highlight open questions in neuroaesthetics. We argue that computational methods can provide mechanistic insight into how aesthetic judgments are formed, while advocating for deeper collaboration between neuroscientists studying aesthetics and those in the arts and humanities.

Most of our waking time as human beings is spent interacting with other individuals. In order to make good decisions in this social milieu, it is often necessary to make inferences about the internal states, traits and intentions of others. Recently, some progress has been made to uncover the neural computations underlying human social decision-mak...

When individuals learn from observing the behavior of others, they deploy at least two distinct strategies. Choice imitation involves repeating other agents’ previous actions, whereas emulation proceeds from inferring their goals and intentions. Despite the prevalence of observational learning in humans and other social animals, a fundamental quest...

It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Moreover, little is known about how such preferences are actually constructed in the brain. Here we developed and tested a computational framework to gain an understanding of how the human brain constructs aesth...

It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Moreover, little is known about how such preferences are actually constructed in the brain. Here we developed and tested a computational framework to gain an understanding of how the human brain constructs aesth...

It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. However, the role of task complexity in the arbitration between these two strategies remains largely unknown. Here, using a combination of novel task design,...

The amygdala plays an important role in many aspects of social-cognition and reward-learning. Here we aimed to determine whether human amygdala neurons are involved in the computations necessary to implement learning through observation. We performed single-neuron recordings from the amygdalae of human neurosurgical patients (male and female) while...

The amygdala plays an important role in many aspects of social-cognition and reward-learning. Here we aimed to determine whether human amygdala neurons are involved in the computations necessary to implement learning through observation. We performed single-neuron recordings from the amygdalae of human neurosurgical patients (male and female) while...

In observational learning (OL), organisms learn from observing the behavior of others. There are at least two distinct strategies for OL. Imitation involves learning to repeat the previous actions of other agents, while in emulation, learning proceeds from inferring the goals and intentions of others. While putative neural correlates for these form...

In observational learning (OL), organisms learn from observing the behavior of others. There are at least two distinct strategies for OL. Imitation involves learning to repeat the previous actions of other agents, while in emulation, learning proceeds from inferring the goals and intentions of others. While putative neural correlates for these form...

In this issue of Neuron, Vikbladh et al. (2019) provide evidence to suggest that the human hippocampus, long known to support spatial memory, also plays a causal role in model-based planning.

Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can be more gratifying than the experience of reward itself. Theories of anticipation have described how it causes behaviors ranging from beneficial information-seeking to harmful addiction. Here, we investigated how the brai...

Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can be more gratifying than the experience of reward itself. Theories of anticipation have described how it causes behaviors ranging from beneficial information-seeking to harmful addiction. Here, we investigated how the brai...

Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can often be more gratifying than the very experience itself. Theories of anticipation have described how it induces behaviors ranging from beneficial information-seeking through to harmful addiction. However, it remains uncl...

While it is established that humans use model-based (MB) and model-free (MF) reinforcement learning in a complementary fashion, much less is known about how the brain determines which of these systems should control behavior at any given moment. Here we provide causal evidence for a neural mechanism that acts as a context-dependent arbitrator betwe...

Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavl...

There is a dichotomy in instrumental conditioning between goal-directed actions and habits that are distinguishable on the basis of their relative sensitivity to changes in outcome value. It is less clear whether a similar distinction applies in Pavlovian conditioning, where responses have been found to be predominantly outcome-sensitive. To test f...

Adolescence is a period of life in which social influences-particularly if they come from peers-play a critical role in shaping learning and decision preferences. Recent studies in adults show evidence of a risk contagion effect; that is, individual risk preferences are modulated by observing and learning from others' risk-related decisions. In thi...

It has been observed that the pressure of performing for high stakes can, paradoxically, lead to uncharacteristically poor performance. Here we investigate a novel approach to attenuating such 'choking under pressure' by instructing participants performing a demanding motor task that rewards successful performance with a monetary gain, to reapprais...

A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, an...

A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, an...

Traditionally, financial market participation has been treated as analogous to playing games of chance with a physical device such as roulette. Here, we propose that humans treat financial markets as intentional agents, with own beliefs and aspirations. As a result, the capacity to infer the intentions of others, Theory of Mind, explains behaviour....

The valuation of food is a fundamental component of our decision-making. Yet little is known about how value signals for food and other rewards are constructed by the brain. Using a food-based decision task in human participants, we found that subjective values can be predicted from beliefs about constituent nutritive attributes of food: protein, f...

In inverse reinforcement learning an observer infers the reward distribution available for actions in the environment solely through observing the actions implemented by another agent. To address whether this computational process is implemented in the human brain, participants underwent fMRI while learning about slot machines yielding hidden prefe...

areas exhibiting significant changes in BOLD associated with entropy signals. Pre-SMA: pre-supplementary motor area. TPJ: temporo-parietal junction. dlPFC: dorsolateral prefrontal cortex. x y z in MNI coordinates.

areas exhibiting significant changes in BOLD associated with predicted outcome in similar and dissimilar. OFC: orbitofrontal cortex, dmPFC: dorsomedial prefrontal cortex. x y z in MNI coordinates.

Prediction-error signals consistent with formal models of “reinforcement learning” (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. Here, we created a novel paradigm optimized...

Model predictions. Representative dynamics of value signals and learning signals as generated by the ACQ(λ) model are Illustrated with the final subject from the Good-learner group. Fitted parameters were assigned as follows for this subject: α = 0.639, λ = 0.322, wQ = 0.857, τ = 0.197, β0 = -0.046, λβ = 0.976, and βR = 0.193. (a-b) The model’s est...