John P O'Doherty

John P O'Doherty
California Institute of Technology | CIT · Division of the Humanities and Social Sciences

D.Phil

About

220
Publications
72,081
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
44,619
Citations
Additional affiliations
January 2008 - November 2013
Trinity College Dublin
Position
  • Professor (Full)
October 2004 - present
California Institute of Technology
Position
  • Professor (Full)
October 2000 - September 2004
University College London
Education
October 1997 - April 2001
University of Oxford
Field of study
  • Experimental Psychology

Publications

Publications (220)
Article
Both novelty and uncertainty are potent features guiding exploration; however, they are often experimentally conflated, and an understanding of how they interact to regulate the balance between exploration and exploitation has proved elusive. Using a task designed to decouple the influence of novelty and uncertainty, we identify separable mechanism...
Article
The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model...
Article
Full-text available
Background Anorexia nervosa (AN) is a chronic and disabling psychiatric condition characterized by low hedonic drive towards food, and is thought to be inclusive of altered dimensions of reward processing. Whether there exists a fundamental aberrancy in the capacity to acquire and maintain de novo hedonic associations—a critical component of hedoni...
Article
Full-text available
Diminished motivation to pursue and obtain primary and secondary rewards has been demonstrated in anorexia nervosa (AN). However, the neurobehavioral mechanisms underlying the behavioral activation component of aberrant reward motivation remains incompletely understood. This work aims to explore this underexplored facet of reward motivation in AN....
Chapter
Among the most challenging questions in the field of neuroaesthetics concerns how a piece of art comes to be liked in the first place. That is, how can the brain rapidly process a stimulus to form an aesthetic judgment even for stimuli never before encountered? In the article under discussion in this chapter, by leveraging computational methods in...
Article
Adaptive behavior in real-world environments demands that choices integrate over several variables, including the novelty of the options under consideration, their expected value, and uncertainty in value estimation. We recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex (vmPFC) and dorsal anterior...
Preprint
Adaptive behavior in real-world environments demands that choices integrate over several variables, including the novelty of the options under consideration, their expected value, and uncertainty in value estimation. We recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex (vmPFC) and dorsal anterior...
Article
Recent evidence suggests that both novelty and uncertainty act as potent features guiding exploration. However, these variables are often conflated with each other experimentally, and an understanding of how these attributes interact to regulate the balance between exploration and exploitation has proved elusive. Using a novel task designed to deco...
Preprint
Full-text available
Recent evidence suggests that both novelty and uncertainty act as potent features guiding exploration. However, these variables are often conflated with each other experimentally, and an understanding of how these attributes interact to regulate the balance between exploration and exploitation has proved elusive. Using a novel task designed to deco...
Article
Here we argue that the assignment of subjective value to potential outcomes at the time of decision-making is an active process, in which individual features of a potential outcome of varying degrees of abstraction are represented hierarchically and integrated in a weighted fashion to produce an overall value judgment. We implicate the lateral orbi...
Article
Full-text available
Anorexia nervosa (AN) is a difficult to treat, pernicious psychiatric disorder that has been linked to decision-making abnormalities. We examined the structural characteristics of habitual and goal-directed decision-making circuits and their connecting white matter tracts in 32 AN and 43 healthy controls across two independent data sets of adults a...
Article
We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning—whether it be learning about the value of stimuli or about the value of actions; the natu...
Article
Full-text available
It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Here, we developed and tested a computational framework to investigate how aesthetic values are formed. We show that it is possible to explain human preferences for a visual art piece based on a mixture of low-...
Chapter
In order to make decisions, we often seek and integrate information coming from other people, while at times also keeping track of the knowledge other people acquire from observing our own actions. In this chapter, we examine the computational mechanisms and the involvement of mentalizing when we learn from observing other people and when we engage...
Article
Full-text available
Over the past three decades, functional MRI (fMRI) has become key to study how cognitive processes are implemented in the human brain. However, the question of whether participants recruited into fMRI studies differ from participants recruited into other study contexts has received little to no attention. This is particularly pertinent when effects...
Article
It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a “Mixture of Experts” in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavi...
Article
We review progress and highlight open questions in neuroaesthetics. We argue that computational methods can provide mechanistic insight into how aesthetic judgments are formed, while advocating for deeper collaboration between neuroscientists studying aesthetics and those in the arts and humanities.
Article
Most of our waking time as human beings is spent interacting with other individuals. In order to make good decisions in this social milieu, it is often necessary to make inferences about the internal states, traits and intentions of others. Recently, some progress has been made to uncover the neural computations underlying human social decision-mak...
Article
When individuals learn from observing the behavior of others, they deploy at least two distinct strategies. Choice imitation involves repeating other agents’ previous actions, whereas emulation proceeds from inferring their goals and intentions. Despite the prevalence of observational learning in humans and other social animals, a fundamental quest...
Preprint
Full-text available
It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Moreover, little is known about how such preferences are actually constructed in the brain. Here we developed and tested a computational framework to gain an understanding of how the human brain constructs aesth...
Article
It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Moreover, little is known about how such preferences are actually constructed in the brain. Here we developed and tested a computational framework to gain an understanding of how the human brain constructs aesth...
Article
Full-text available
It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. However, the role of task complexity in the arbitration between these two strategies remains largely unknown. Here, using a combination of novel task design,...
Preprint
Full-text available
The amygdala plays an important role in many aspects of social-cognition and reward-learning. Here we aimed to determine whether human amygdala neurons are involved in the computations necessary to implement learning through observation. We performed single-neuron recordings from the amygdalae of human neurosurgical patients (male and female) while...
Article
The amygdala plays an important role in many aspects of social-cognition and reward-learning. Here we aimed to determine whether human amygdala neurons are involved in the computations necessary to implement learning through observation. We performed single-neuron recordings from the amygdalae of human neurosurgical patients (male and female) while...
Article
In observational learning (OL), organisms learn from observing the behavior of others. There are at least two distinct strategies for OL. Imitation involves learning to repeat the previous actions of other agents, while in emulation, learning proceeds from inferring the goals and intentions of others. While putative neural correlates for these form...
Preprint
Full-text available
In observational learning (OL), organisms learn from observing the behavior of others. There are at least two distinct strategies for OL. Imitation involves learning to repeat the previous actions of other agents, while in emulation, learning proceeds from inferring the goals and intentions of others. While putative neural correlates for these form...
Article
In this issue of Neuron, Vikbladh et al. (2019) provide evidence to suggest that the human hippocampus, long known to support spatial memory, also plays a causal role in model-based planning.
Article
Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can be more gratifying than the experience of reward itself. Theories of anticipation have described how it causes behaviors ranging from beneficial information-seeking to harmful addiction. Here, we investigated how the brai...
Preprint
Full-text available
Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can be more gratifying than the experience of reward itself. Theories of anticipation have described how it causes behaviors ranging from beneficial information-seeking to harmful addiction. Here, we investigated how the brai...
Article
Full-text available
Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can often be more gratifying than the very experience itself. Theories of anticipation have described how it induces behaviors ranging from beneficial information-seeking through to harmful addiction. However, it remains uncl...
Article
While it is established that humans use model-based (MB) and model-free (MF) reinforcement learning in a complementary fashion, much less is known about how the brain determines which of these systems should control behavior at any given moment. Here we provide causal evidence for a neural mechanism that acts as a context-dependent arbitrator betwe...
Article
Full-text available
Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavl...
Article
Full-text available
There is a dichotomy in instrumental conditioning between goal-directed actions and habits that are distinguishable on the basis of their relative sensitivity to changes in outcome value. It is less clear whether a similar distinction applies in Pavlovian conditioning, where responses have been found to be predominantly outcome-sensitive. To test f...
Article
Adolescence is a period of life in which social influences-particularly if they come from peers-play a critical role in shaping learning and decision preferences. Recent studies in adults show evidence of a risk contagion effect; that is, individual risk preferences are modulated by observing and learning from others' risk-related decisions. In thi...
Article
Full-text available
It has been observed that the pressure of performing for high stakes can, paradoxically, lead to uncharacteristically poor performance. Here we investigate a novel approach to attenuating such 'choking under pressure' by instructing participants performing a demanding motor task that rewards successful performance with a monetary gain, to reapprais...
Preprint
A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, an...
Article
A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, an...
Article
Full-text available
Traditionally, financial market participation has been treated as analogous to playing games of chance with a physical device such as roulette. Here, we propose that humans treat financial markets as intentional agents, with own beliefs and aspirations. As a result, the capacity to infer the intentions of others, Theory of Mind, explains behaviour....
Article
Full-text available
The valuation of food is a fundamental component of our decision-making. Yet little is known about how value signals for food and other rewards are constructed by the brain. Using a food-based decision task in human participants, we found that subjective values can be predicted from beliefs about constituent nutritive attributes of food: protein, f...
Article
Full-text available
In inverse reinforcement learning an observer infers the reward distribution available for actions in the environment solely through observing the actions implemented by another agent. To address whether this computational process is implemented in the human brain, participants underwent fMRI while learning about slot machines yielding hidden prefe...
Data
areas exhibiting significant changes in BOLD associated with entropy signals. Pre-SMA: pre-supplementary motor area. TPJ: temporo-parietal junction. dlPFC: dorsolateral prefrontal cortex. x y z in MNI coordinates.
Data
areas exhibiting significant changes in BOLD associated with predicted outcome in similar and dissimilar. OFC: orbitofrontal cortex, dmPFC: dorsomedial prefrontal cortex. x y z in MNI coordinates.
Article
Full-text available
Prediction-error signals consistent with formal models of “reinforcement learning” (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. Here, we created a novel paradigm optimized...
Data
Model predictions. Representative dynamics of value signals and learning signals as generated by the ACQ(λ) model are Illustrated with the final subject from the Good-learner group. Fitted parameters were assigned as follows for this subject: α = 0.639, λ = 0.322, wQ = 0.857, τ = 0.197, β0 = -0.046, λβ = 0.976, and βR = 0.193. (a-b) The model’s est...
Data
Good-learner group: Action-value signals. In addition to the separate types of RPE signals, separate types of value signals were evoked by the current paradigm. Among the Good-learner group, action-value signals were identified bilaterally in vmPFC (xyz = [1, 33.5, -17.5], t19 = 3.87, p < 10-3, k = 21, SVC pFWE = 0.086) as anticipated with marginal...
Data
Good-learner group versus Poor-learner group. (a) The aforementioned lack of dorsal-striatal RPE signals among Poor learners was confirmed as part of direct contrasts of the Good-learner and Poor-learner groups with respect to the different parametric effects. First, the between-group contrast of SVPE signals revealed a cluster in the left dorsal s...
Data
Action-value-prediction-error signals. (a) For the Good-learner group, AVPE signals were identified throughout both the ventral striatum and the dorsal striatum. As with the aggregate analysis, the global peak of a cluster also within the ROI for the right ventral striatum (xyz = [8.5, 11, -2.5], t19 = 4.02, p < 10-3, k = 71, SVC pFWE = 0.064) was...
Data
Model discriminability. The model comparison reported in Fig 2A was replicated using artificial data that were simulated with the ACQ(λ) model as fitted for each subject but otherwise yoked to the empirical data set. Average goodness of fit relative to the outcome-insensitive hysteresis model across performing subjects is shown for each model teste...
Data
Poor-learner group. (a) For the Poor-learner group, the relevant neural signals were expected to be weaker as a reflection of the less robust learning evident in behavior. In line with this expectation, SVPE signals were only identified in the right ventral striatum (xyz = [19, 11, -11.5], t14 = 4.92, p = 10-4, k = 13). (b) Correspondingly, AVPE si...
Article
During competitive interactions, humans have to estimate the impact of their own actions on their opponent's strategy. Here we provide evidence that neural computations in the right temporoparietal junction (rTPJ) and interconnected structures are causally involved in this process. By combining inhibitory continuous theta-burst transcranial magneti...
Article
See Manohar and Akam (doi:10.1093/brain/awx119) for a scientific commentary on this article. Neuroimaging studies suggest that the human ventromedial prefrontal cortex is a key region for goal-directed behaviour. However, it remains unclear whether the ventromedial prefrontal cortex is necessary for such behaviour. Here we used a canonical test fro...
Article
Full-text available
In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditionin...
Article
Although popular discussion of testosterone's influence on males often centers on aggression and antisocial behavior, contemporary theorists have proposed that it instead enhances behaviors involved in obtaining and maintaining a high social status. Two central distinguishing but untested predictions of this theory are that testosterone selectively...
Article
This paper offers an active inference account of choice behaviour and learning. It focuses on the distinction between goal-directed and habitual behaviour and how they contextualise each other. We show that habits emerge naturally (and autodidactically) from sequential policy optimisation when agents are equipped with state-action policies. In acti...
Article
Full-text available
A major open question is whether computational strategies thought to be used during experiential learning, specifically model-based and model-free reinforcement-learning, also support observational learning. Furthermore, the question of how observational learning occurs when observers must learn about the value of options from observing outcomes in...
Article
Our attitude toward risk plays a crucial role in influencing our everyday decision-making. Despite its importance, little is known about how human risk-preference can be modulated by observing risky behavior in other agents at either the behavioral or the neural level. Using fMRI combined with computational modeling of behavioral data, we show that...
Article
Full-text available
Author Summary When making decisions in our everyday life (e.g. where to eat) we first have to identify a set of environmental features that are relevant for the decision (e.g. the distance to the place, current time or the price). Although we are able to make such inferences almost effortlessly, this type of problems is computationally challenging...
Article
Full-text available
Unlabelled: The role of neurons in the substantia nigra (SN) and ventral tegmental area (VTA) of the midbrain in contributing to the elicitation of reward prediction errors during appetitive learning has been well established. Less is known about the differential contribution of these midbrain regions to appetitive versus aversive learning, especi...
Article
In this chapter, we will review evidence about the role of multiple distinct systems in driving the motivation to perform actions in humans. Specifically, we will consider the contribution of goal-directed action selection mechanisms, habitual action selection mechanisms and the influence of Pavlovian predictors on instrumental action selection. We...
Article
Symptoms of attention deficit hyperactivity disorder (ADHD) in children often persist into adulthood and can lead to severe antisocial behavior. However, to-date it remains unclear whether neuro-functional abnormalities cause ADHD, which in turn can then provide a marker of persistent ADHD. Using event-related functional magnetic resonance imaging...
Article
Full-text available
Considerable behavioral data indicate that operant actions can become habitual, as demonstrated by insensitivity to changes in the action-outcome contingency and in subjective outcome values. Notably, although several studies have investigated the neural substrates of habits, none has clearly differentiated the areas of the human brain that support...
Article
Consensus building in a group is a hallmark of animal societies, yet little is known about its underlying computational and neural mechanisms. Here, we applied a computational framework to behavioral and fMRI data from human participants performing a consensus decision-making task with up to five other participants. We found that participants reach...