No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior
Abstract
It is proposed that a cognitive map encoding the relationships between entities in the world supports flexible behavior, but the majority of the neural evidence for such a system comes from studies of spatial navigation. Recent work describing neuronal parallels between spatial and non-spatial behaviors has rekindled the notion of a systematic organization of knowledge across multiple domains. We review experimental evidence and theoretical frameworks that point to principles unifying these apparently disparate functions. These principles describe how to learn and use abstract, generalizable knowledge and suggest that map-like representations observed in a spatial context may be an instance of general coding mechanisms capable of organizing knowledge of all kinds. We highlight how artificial agents endowed with such principles exhibit flexible behavior and learn map-like representations observed in the brain. Finally, we speculate on how these principles may offer insight into the extreme generalizations, abstractions, and inferences that characterize human cognition. Behrens et al. review an emerging field building formalisms for understanding the neural basis of flexible behavior. The authors extend these ideas toward representations useful for generalization and structural abstraction, allowing rapid inferences and flexible behavior with little direct experience.
... This has led to the idea of the amygdala as a multidimensional processor that integrates cognitive and emotional functions by way of its interactions with the extensive number of regions it has anatomical connections to, including PFC 15,16 . Similarly, the hippocampus is now believed to not only contain maps of the spatial environment but also more generally maps of task spaceabstract representations of different features of a task such as stimuli, responses, and values, how they unfold in time and space and how they depend on each other 17,18 . Indeed, hippocampal neurons have now been shown to code different points in a value space determined by the relative reward values of a set of stimuli 19 . ...
... Most research has focused on the contribution of the PFC to these processes due to its role in reversal learning and executive function [3][4][5] . However, rule use may be broken down into a range of sub-processes which may be coded in brain regions such as amygdala and hippocampus which neurophysiological recording studies in primates have shown to be involved in multi-dimensional coding 15,16 and cognitive maps of task space 17,18 . Therefore, we examined the role of the human amygdala and hippocampus in ED and ID rule shifting using intracranial recordings. ...
... The increased hippocampal activity for win-stay, is consistent with the idea that the hippocampus maintains cognitive maps of task space [17][18][19] . An important question about shifting is when after reinforcement change does a shift occur? ...
Adaptive behavior requires the ability to shift responding within (intra-dimensional) or between (extra-dimensional) stimulus dimensions when reward contingencies change. Studies of shifting in humans have focused mainly on the prefrontal cortex and/ or have been restricted to indirect measures of neural activity such as fMRI and lesions. Here, we demonstrate the importance of the amygdala and hippocampus by recording local field potentials directly from these regions intracranially in human epilepsy patients. Reward signals were coded in the high frequency gamma activity (HFG; 60-250 Hz) of both regions and synchronised via low frequency (3-5 Hz) phase-locking only after a shift when patients did not already know the rule and it signalled to stop shifting (“Win-Stay”). In contrast, HFG punishment signals were only seen in the amygdala when the rule then changed and it signalled to start shifting (“Lose-Shift”). During decision-making, hippocampal HFG was more inhibited on non-shift relative to shift trials, suggesting a role in preventing interference in rule representation and amygdala HFG was sensitive to stimulus novelty. The findings expand our understanding of human amygdala-hippocampal function and shifting processes, the disruption of which could contribute to shifting deficits in neuropsychiatric disorders.
... Importantly, grid cells maintain their cell-to-cell activity covariance structure across different spatial environments (Barry et al., 2012;Fyhn et al., 2007; Yoon et al., 2013), suggesting they encode the statistical relationships common to all spatial environments, irrespective of specific sensory details. Complementary theoretical work suggests grid cells represent an efficient compression of the transition structure of 2D environments (Banino et al., 2018;Behrens et al., 2018;Dordek et al., 2016; George et al., 2021; Mathis et al., 2015;Stachenfeld et al., 2017;Whittington et al., 2020). Such a representation is highly useful for generalisation to novel tasks with the same 2D structure. ...
... Luckily, our world is replete with statistical structure, and similar relationships hold across related experiences (Shepard, 1987). If the brain effectively extracts and organizes information about the statistical relationships between experiences -forming what is known as a "cognitive map" (Behrens et al., 2018;Tolman, 1948) -it can generalize knowledge from one experience to similar situations. ...
... One particularly effective way of achieving this abstraction is by separating (or "factorizing") the representation of the relational structure of the environment from its specific sensory details (Behrens et al., 2018;Whittington et al., 2020). A potential locus for such an abstracted representation is the medial prefrontal cortex (mPFC), which has been implicated in recognising commonalities across experiences in the schema literature (Baldassano et al., 2018;El-Gaby et al., 2023;Gilboa and Marlatte, 2017;van Kesteren et al., 2013;Tse et al., 2007Tse et al., , 2011 and using these for inference (Zeithamova and Preston, 2010), as well as in concept learning (Kumaran et al., 2009;Mack et al., 2020), structural generalisation of reinforcement learning problems (Baram et al., 2021;Samborska et al., 2022) and generalisation of spatial tasks across different paths (Kaefer et al., 2020;Morrissey et al., 2017;Tang et al., 2023;Yu et al., 2018). ...
Understanding the structure of a problem, such as the relationships between stimuli, supports fast learning and flexible reasoning. Recent theoretical suggestions have highlighted the usefulness of explicit structural representations that are fully divorced from sensory details for generalisation. Rodent work has suggested that abstraction of structure occurs gradually, over time, in cortex. However, direct evidence of such explicit relational representations in humans is scarce, and its relationship to consolidation mechanisms is underexplored. Here, we use a graph-learning paradigm to find such a relational map in the human medial prefrontal cortex. Importantly, this representation was absent early after learning but emerged on the time scale of days. These results shed new light on neural representations underlying the remarkable human ability to draw accurate inferences from little data.
... In this Review we outline advances in cognitive neuroscience that concern how the brain forms structured internal representations of the world -cognitive maps -that organize knowledge to guide learning, inference, and behaviour [9][10][11][12]. While much of this knowledge concerns hippocampalentorhinal cortex representations of physical space, growing evidence indicates that conserved algorithmic principles are at play across diverse task domains. ...
... The study of cognitive maps began with a careful observation of behaviour. In the first half of the 20 th century, psychologists such as Tolman, Harlow, and Craik speculated on the existence of structured internal representations on the basis of observing animal behaviour that was difficult to reconcile with contemporary behaviourist theories of decision making [1,10,13,14]. Tolman described experiments in which rats appeared to learn detailed "field maps" of spatial environments in the absence of reward ("latent learning"), update these maps following surprises, and use them when engaging in deliberative behaviour ("vicarious trial and error") [13]. ...
... What makes a good task representation? The best task representations capture environmental features that maximally facilitate prediction, inference, and decision-making, and promote re-use of knowledge in new environments [9,10,12,19] ("solving problems in representation, not by exhaustive computation" [12]). This often necessitates tracking latent states, which are inferred as opposed to directly observed (Box 1). ...
Structured internal representations (cognitive maps) shape cognition, from imagining the future and counterfactual past, to transferring knowledge to new settings. Our understanding of how such representations are formed and maintained in biological and artificial neural networks has grown enormously. The cognitive mapping hypothesis of schizophrenia extends this enquiry to psychiatry, proposing that diverse symptoms - from delusions to conceptual disorganisation - stem from abnormalities in how the brain forms structured representations. These abnormalities may arise from a confluence of neurophysiological perturbations (excitation-inhibition imbalance, resulting in attractor instability and impaired representational capacity), and/or environmental factors such as early life psychosocial stressors (which impinge on representation learning). This proposal thus links knowledge of neural circuit abnormalities, environmental risk factors, and symptoms.
... Among the defining features of human cognition is an exceptional ability to adapt rapidly to novel contexts. One set of proposals suggests this relies on abstraction and generalization of past experiences (Allen et al., 2020;Dekker et al., 2022;Kumar et al., 2022;Lake et al., 2015Lake et al., , 2017Lehnert et al., 2020;Tsividis et al., 2021), enabling a transfer of past experiential knowledge to entirely new situations (Behrens et al., 2018;Lake et al., 2017;Mark et al., 2020;Shepard, 1987). Much previous research has been devoted to understanding how humans detect (Turk-Browne et al., 2008), generalize (Reber, 1967) and neurally represent a single dynamical, and sequentially, unfolding process (Garvert et al., 2017;Henin et al., 2021;Schapiro et al., 2013;Sherman et al., 2020) or do so from piecemeal presentation of a graph structure (Rmus et al., 2022). ...
... We conjectured that compositional generalization of structural knowledge could be enabled by a previously described mechanism that proposed abstraction of structural representations from sensory experiences within the hippocampal-entorhinal system for efficient generalization to novel situations (Behrens et al., 2018;Whittington et al., 2020Whittington et al., , 2022. In other words, knowledge generalization involves mapping of inferred generative causes of an ongoing experience onto a structural scaffolding that supports abstractions over concrete events Pesnot Lerousseau & Summerfield, 2024;Whittington et al., 2020). ...
... The question of how such a scaffolding might be realized has been addressed in computational accounts of hippocampal and entorhinal cortex function, where grid cell activity is proposed to provide a basis set representation of abstract structure inferred from sensory experiences (Behrens et al., 2018;Mark et al., 2020;Whittington et al., 2020Whittington et al., , 2022. We hypothesized that compositional generalization in our task might be achieved by extracting, and neurally representing, abstracted relational units of prior experience ("building blocks") that facilitates a flexible recombination in new contexts. ...
An exceptional human ability to adapt to the dynamics of novel environments relies on abstracting and generalizing past experiences. While previous research has examined how humans generalize isolated sequential processes, we know little concerning the neural mechanisms that enable adaptation to the more complex dynamics that govern everyday experience. Here, we deployed a novel sequence learning task based on graph factorization, coupled with simultaneous magnetoencephalography (MEG) recordings, to ask whether reuse of experiential “building blocks” provides an abstract structural scaffolding that enables inference and generalization. We provide behavioral evidence that participants decomposed task experience into subprocesses, abstracted dynamical subprocess structures away from sensory specifics, and transferred these to a new task environment. Neurally we show this transfer is underpinned by a representational alignment of abstract subprocesses across task phases, where this included enhanced neural similarity among stimuli that adhered to the same subprocess, a temporally evolving mapping between predictive representations of subprocesses and a generalization of the precise dynamical roles that stimuli occupy within graph structures. Crucially, decoding strength for dynamical role representations predicted behavioral success in transfer of subprocess knowledge, consistent with a role in supporting behavioral adaptation in new environments. We propose a structural scaffolding mechanism enables compositional generalization of dynamical subprocesses that facilitate efficient adaptation within new contexts.
... The format of these spatial maps allows objects to be placed within two-dimensional mental spaces 6,8 , affording representation of the longer-range relationships between those objects. Outside of spatial navigation, recent work demonstrates that humans also represent abstract maps of conceptual spaces [9][10][11] , including social traits such as competence and popularity 12,13 . However, relationships in social networks are poorly characterized by two-dimensional spaces, and it is not known what alternative format(s) might instead be used to build abstract cognitive maps of social networks. ...
... A related question revolves around the neural instantiation of multi-step abstraction. Although we leverage the SR in this work, we note that multi-step abstraction is a much more general representational strategy that could be implemented using many mechanisms with varying degrees of biological plausibility 9,19,20 . Despite the SR depending heavily on the temporal dynamics of experience 14,50 , multi-step abstraction appears to describe how people represent social networks even when observations of social interaction are temporally disjointed 4 . ...
... computation. A particularly intriguing possibility is that the brain may encode components of a network's structure (that is, basis sets) that afford greater flexibility in assembling useful representations when navigating a variety of social environments 9 . There are many ways that the brain could perform inference over graphs 20,[51][52][53] , and it may be useful for future work to examine what kinds of basis sets are afforded by various methods of graph inference, including multi-step abstraction. ...
To make adaptive social decisions, people must anticipate how information flows through their social network. While this requires knowledge of how people are connected, networks are too large to have first-hand experience with every possible route between individuals. How, then, are people able to accurately track information flow through social networks? Here we find that people immediately cache abstract knowledge about social network structure as they learn who is friends with whom, which enables the identification of efficient routes between remotely connected individuals. These cognitive maps of social networks, which are built while learning, are then reshaped through overnight rest. During these extended periods of rest, a replay-like mechanism helps to make these maps increasingly abstract, which privileges improvements in social navigation accuracy for the longest communication paths that span distinct communities within the network. Together, these findings provide mechanistic insight into the sophisticated mental representations humans use for social navigation.
... For instance, when presented with the ambiguous image in Figure 1A, some describe only the character in the foreground and discuss themes related to education while others focus on the character in the background and discuss the emotional toll entailed by farm labor. The cognitive processes supporting such variations in interpretation have been discussed extensively in the psychology literature, using terms such as cognitive distortions (1), mental representations (2), cognitive maps (3)(4)(5), schemas (6)(7)(8)(9), and object relations (10). It is generally believed that mental health disorders are linked to variations in mental representations and subjective experience. ...
... Furthermore, psychometric inventories are notoriously prone to report biases (16). Since 3 virtually all research conducted on subjective symptoms in mental health is quantified using such coarse indicators, developing a tool to capture the finer nuances of subjective experience in controlled experimental settings would greatly improve our capacity to understand and develop treatments for mental health disorders. ...
... These latent spaces capture broad semantic dimensions acquired during network training, such as complex emotions (e.g., rejection, persecution, anxiety), personality traits, politics, religion, etc. (21). Such latent spaces are conceptually related to, yet different from "maps of experience" that have often been discussed by psychologists and philosophers alike, sometimes under the terms of quality space (22)(23)(24)(25) or state space (3,26). Whereas subjective similarity space can be determined empirically from behavior, the latent spaces in ANNs are a product of data-driven learning processes that abstract patterns from large datasets. ...
The way that individuals subjectively experience the world greatly influences their own mental well-being. However, it remains a considerable challenge to precisely characterize the breadth and depth of such experiences. One persistent problem is the lack of objective tools for directly quantifying and comparing narrative reports of subjective experiences. Here, we develop a new approach to map and compare reports of experience using the latent space of artificial neural networks (ANNs). Using a series of 31 prompts, including 30 images and 1 open-ended question, we quantified how the verbal reports provided by participants (n=210, 50% female) deviate from one another and how these variations are linked to subjective experience and mental health. We found that latent space embeddings of experience can accurately predict subjective judgments of valence and arousal in a series of emotional pictures. Furthermore, we show that narrative reports to ambiguous images can accurately predict transdiagnostic factors of mental health. While distortions in the latent space of ANNs are notoriously difficult to interpret, we propose a new approach to synthesize visual stimuli with generative artificial intelligence that can be used to explore semantic distortions in reported experiences. In sum, latent space cartography could offer a promising avenue for objectively quantifying distortions of subjective experience in mental health and could ultimately help identify new therapeutic targets for clinical interventions.
... To achieve abstract hierarchical organization, structure and content are proposed to be represented in a disentangled manner, known as factorization 29,30 . This view has been proposed in computational models for sequence memory 31,32 and supported by recent empirical findings [33][34][35][36] . ...
... First, the fundamental component of such a hierarchical organization, chunking, has been found across various domains [66][67][68][69] . Moreover, structure and contents are dissociated from each other in neural representations, known as factorization 29 . Therefore, the observed neural geometry could serve as a general scaffold to accommodate a variety of WM items in both linguistic and non-linguistic contexts. ...
Working memory (WM) is constructive in nature. Instead of passively retaining information, WM reorganizes complex sequences into hierarchically embedded chunks to overcome capacity limits and facilitate flexible behaviour. Here, to investigate the neural mechanisms underlying hierarchical reorganization in WM, we performed two electroencephalography and one magnetoencephalography experiments, wherein humans retain in WM a temporal sequence of items, that is, syllables, which are organized into chunks, that is, multisyllabic words. We demonstrate that the one-dimensional sequence is represented by two-dimensional neural representational geometry in WM arising from left prefrontal and temporoparietal regions, with separate dimensions encoding item position within a chunk and chunk position in the sequence. Critically, this two-dimensional geometry is observed consistently in different experimental settings, even during tasks not encouraging hierarchical reorganization in WM and correlates with WM behaviour. Overall, these findings strongly support that complex sequences are reorganized into factorized multidimensional neural representational geometry in WM, which also speaks to general structure-based organizational principles given WM’s involvement in many cognitive functions.
... 42,91,92,93 Furthermore, cortical connections of the human hippocampalentorhinal system play an important role in cognitive mapping, abstract structure learning, and reinforcement learning. [94][95][96][97][98][99][100] Outside of the hippocampal-entorhinal system, these cognitive processes were shown to be associated with a broad set of distributed cortical regions, such as the prefrontal cortex, posterior cingulate, and inferior parietal lobule. Most of these cortical regions overlap with the canonical default network, 80 which was proposed to support the construction of internal mental models, potentially corresponding to the construction of cognitive maps facilitating flexible behavior in spatial and non-spatial domains. ...
... Most of these cortical regions overlap with the canonical default network, 80 which was proposed to support the construction of internal mental models, potentially corresponding to the construction of cognitive maps facilitating flexible behavior in spatial and non-spatial domains. 96,97 Our findings indicate that both subdivisions of the canonical default network (DN-A and DN-B) are associated with the hippocampal-entorhinal system, thus potentially supporting the formation of internal cognitive models by integrating spatial (DN-A) and non-spatial (DN-B) information. ...
... Different domains thus may necessitate different prior constraints, and successful generalization requires getting these constraints right 10 . Organizing world knowledge efficiently [11][12][13] (e.g. cognitive map) might therefore require a modular approach to internal model construction, leveraging domainappropriate inductive biases. ...
... There are strong computational reasons to assume that humans possess multiple distinct cognitive maps 11 or model spaces 17 , specialized for particular domains of the world. In this study, we introduce three such models, tuned to different domains of the world: states, agents and actions (illustrated in Fig 1a). ...
Reliance on internal predictive models of the world is central to many theories of human cognition. Yet it is unknown whether humans acquired multiple separate internal models, each evolved for a specific domain, or maintain a globally unified representation. Using fMRI, we show that during naturalistic experiences (during movie watching or narrative listening), adult participants selectively engage three topographically distinct midline prefrontal cortical regions, for different forms of predictions. Regions responded selectively to abstract spatial, referential (social), and temporal domains during model updates implying separate representations for each. Prediction-error-driven neural transitions in these regions, indicative of model updates, preceded subjective belief changes in a domain-specific manner. We find these parallel top-down predictions are unified and selectively integrated with sensory streams in the Precuneus, shaping participants' ongoing experience. Results generalized across sensory modalities and content, suggesting humans recruit abstract, modular predictive models for both vision and language. Our results highlight a key feature of human world modeling: fragmenting information into abstract domains before global integration.
... A priori, an organism's energy-efficient policy may imply idle existence. However, the ability to actively engage with one's environment, process the sensory information it provides, and integrate it into our internal model of the world is a fundamental feature of the brain, allowing for flexible and adaptive behavior (Friston, 2010;Summerfield and de Lange, 2014;Behrens et al., 2018). In this context, mechanisms to actively seek information are thought to be essential to learning, expanding individual knowledge, and maximizing the reward obtained from an environment (Gottlieb, 2012;Gottlieb et al., 2013;Gottlieb and Oudeyer, 2018;Monosov, 2024). ...
Boredom and curiosity are common everyday states that drive individuals to seek
information. Due to their functional relatedness, it is not trivial to distinguish
whether an action, for instance in the context of a behavioral experiment, is
driven by boredom or curiosity. Are the two constructs opposite poles of the
same cognitive mechanism, or distinct states? How do they interact? Can they
co-exist and complement each other? Here, we systematically review similarities
and dissimilarities of boredom and curiosity with respect to their subjective
experience, functional role, and neurocognitive implementation. We highlight
the usefulness of Information Theory for formalizing information-seeking in the
context of both states and provide guidelines for their experimental investigation.
Our emerging view is that despite their distinction on an experiential level,
boredom and curiosity are closely related on a functional level, providing
complementary drives on information-seeking: boredom, similar to hunger,
arises from a lack of information and drives individuals to avoid contexts with
low information yield, whereas curiosity constitutes a mechanism similar to
appetite, pulling individuals toward specific sources of information. We discuss
predictions arising from this perspective, concluding that boredom and curiosity
are independent, but coalesce to optimize behavior in environments providing
varying levels of information.
... The two-stream hypothesis involves many brain regions, and rigorous modeling is very challenging. At the framework level, we can extend our model to include more regions, in particular the hippocampal formation, which has long been considered as the basis for memory formation, flexible decision-making and reasoning (Behrens et al. 2018;Whittington et al. 2020Whittington et al. , 2022. In (Bakhtiari et al. 2021), the functional specialization of the visual cortex emerges from training parallel pathways with self-supervised predictive learning; the potential to incorporate such functionality into DRNet for similar tasks is still under exploration. ...
reasoning tasks present challenges for deep neural networks, exposing limitations in their capabilities. In this work, we present a neural network model that addresses the challenges posed by Raven's Progressive Matrices (RPM). Inspired by the two-stream hypothesis of visual processing, we introduce the Dual-stream Reasoning Network (DRNet), which utilizes two parallel branches to capture image features. On top of the two streams, a reasoning module first learns to merge the high-level features of the same image. Then, it employs a rule extractor to handle combinations involving the eight context images and each candidate image, extracting discrete abstract rules and utilizing an multilayer perceptron (MLP) to make predictions. Empirical results demonstrate that the proposed DRNet achieves state-of-the-art average performance across multiple RPM benchmarks. Furthermore, DRNet demonstrates robust generalization capabilities, even extending to various out-of-distribution scenarios. The dual streams within DRNet serve distinct functions by addressing local or spatial information. They are then integrated into the reasoning module, leveraging abstract rules to facilitate the execution of visual reasoning tasks. These findings indicate that the dual-stream architecture could play a crucial role in visual abstract reasoning.
... The re-ordering of objects to different temporal positions can be achieved by independently representing the specific object items and the latent ordinal structure to which the items could be assigned (Tian et al., 2024). In computational models the latent temporal structure of experienced environments can be inferred and represented as a non-spatial cognitive map of item locations that enables the flexible reindexing of objects to different positions on this cognitive map (Behrens et al., 2018;Whittington et al., 2022). ...
Intelligent behavior involves mentally arranging learned information in novel ways and is particularly well developed in humans. While nonhuman primates (NHP) will learn to arrange new items in complex serial order and re-arrange neighboring items within that order, it has remained contentious whether they are capable to re-assign items more flexibly to non-adjacent positions. Such mental re-indexing is facilitated by inferring the latent temporal structure of experiences as opposed to learning serial chains of item-item associations. Here, we tested the ability for flexible mental re-indexing in rhesus macaques. Subjects learned to serially order five objects. A change of the background context indicated when the object order changed, probing the subjects to mentally re-arrange objects to non-adjacent positions of the learned serial structure. Subjects successfully used the context cue to pro-actively re-index items to new, non-adjacent positions. Mental re-indexing was more likely when the initial order had been learned at a higher level, improved with more experience of the re-indexing rule and correlated with working memory performance in a delayed match-to-sample task. These findings suggest that NHPs inferred the latent serial structure of experiences beyond a chaining of item-item associations and mentally re-arrange items within that structure. The pattern of results indicates that NHPs form non-spatial cognitive maps of their experiences, which is a hallmark for flexible mental operations in many serially ordered behaviors including communication, counting or foraging.
... However, the role of the EC in this framework is unclear. At present the majority of models of EC function have focused on how grid cells could support generalization (Behrens et al., 2018;Gustafson & Daw, 2011;Momennejad, 2020), but very few models have touched on the issue of pattern separation (Kerdels & Peters, 2017). This contradiction makes it computationally interesting to examine hypotheses behind theories, how they connect, and where they differ. ...
The entorhinal cortex (EC) provides the major input to the hippocampus (HPC). Numerous computational models on the EC propose that its grid cells serve as a spatial metric, supporting path integration and efficient generalization. However, little is known about how these cells could contribute to episodic memory, which emphasizes episode-specific representations that align with pattern separation. Taking into consideration anatomical specifications of EC inputs to the HPC and computational principles underlying the EC-HPC memory system, we argue that EC layer IIa (EC2a) supports pattern separation and EC layer IIb/III (EC2b/3) supports generalization. Utilizing recurrent inhibition and the nature of single EC2a neurons binding converging inputs from the neocortex (i.e., conjunctive coding), we built a biologically-based neural network model of the EC-HPC system for episodic memory. By examining how EC2a transformed its cortical inputs and output them to the trisynaptic pathway (EC2a - Dentate Gyrus - CA3 - CA1), we found that instead of systematically generalizing across similar inputs, recurrent inhibition and conjunctive coding in EC2a neurons support strong pattern separation and increase mnemonic discrimination. Furthermore, lesioning EC2a neurons in the model resembled memory impairments found in people with Alzheimer’s Disease, suggesting an intertwined relationship between memory and the majority of pure grid cells in the EC. On the other hand, the topographically organized monosynaptic pathway (EC2b/3 - CA1) is computationally more suitable for efficient factorization and generalization. This model provides novel anatomically-based predictions regarding the computational roles of EC cells in pattern separation and generalization, which together form a critical computational framework for both episodic memory and spatial navigation.
... Animals actively sample their environment and explore space, even in lab experiments without experimenter-controlled stimuli and rewards (Berlyne, 1966;Buzsáki, 2019;Crowcroft, 1973;DeBose & Nevitt, 2008;Land & Tatler, 2009;Osborne et al., 1999;Renner, 1990;Wang & Hayden, 2021). Sampling sensory stimuli provides the raw material for constructing and updating internal models of self and the environment (Behrens et al., 2018;Keller & Mrsic-Flogel, 2018;O'Keefe & Nadel, 1978;Tolman, 1948;Weber et al., 2019;S. C.-H. Yang et al., 2016). ...
Vertebrates sniff to control the odor samples that enter their nose. These samples can not only help identify odorous objects, but also locations and events. However, there is no receptor for place or time. Therefore, to take full advantage of olfactory information, an animal′s brain must contextualize odor-driven activity with information about when, where, and how they sniffed. To better understand contextual information in the olfactory system, we captured the breathing and movements of mice while recording from their olfactory bulb. In stimulus- and task-free experiments, mice structure their breathing into persistent rhythmic states which are synchronous with statelike structure in ongoing neuronal population activity. These population states reflect a strong dependence of individual neuron activity on variation in sniff frequency, which we display using ″sniff fields″ and quantify using generalized linear models. In addition, many olfactory bulb neurons have ″place fields″ that display significant dependence of firing on allocentric location, which were comparable with hippocampal neurons recorded under the same conditions. At the population level, a mouse′s location can be decoded from olfactory bulb with similar accuracy to hippocampus. Olfactory bulb place sensitivity cannot be explained by breathing rhythms or scent marks. Taken together, we show that the mouse olfactory bulb tracks breathing rhythms and self-location, which may help unite internal models of self and environment with olfactory information as soon as that information enters the brain.
... Readout from these modules can be combined to efficiently encode current location 3 and to support inferences such as shortest path to goal 4 . The grid cells code also generalises: it abstracts over the sensory particularities of spatial environments [5][6][7] . Recent research has argued grid cells also encode non-spatial and abstract 2D spaces [8][9][10][11][12] , raising the possibility that the same neural system might function as a coordinate system for multiple 2D domains. ...
Grid cells in rodent entorhinal cortex (EC) support a coordinate system for space, enabling robust memory and powerful flexibility in spatial behaviour. This coordinate system is abstract - with the same grid cells encoding position across different sensory environments; and hierarchical - with grid modules of increasing spatial scale occupying increasingly ventral locations in the EC. Recent theories suggest that a similar abstract coordinate system could offer the same benefits to general memories that are not sequences drawn from a 2D surface. Here we show that an abstract hierarchical coordinate system supports arbitrary sequences in the human medial temporal lobe (MTL). In single-unit recordings from MTL, we find abstract, coordinate-like coding of a simple sequential memory task. In an fMRI experiment with more complex hierarchical sequences, we discover an abstract hierarchical representation in EC: the coordinate representations at distinct levels in the hierarchy are arranged on an anatomical gradient along the EC's anterior-posterior axis, homologous to the ventro-dorsal axis in rodents. These results therefore mirror the anatomical gradient of grid cells in the rodent EC but now for arbitrary non-spatial sequences. Together they suggest that memories are scaffolded on a hierarchical coordinate system using common neuronal coding principles, aligned to preserved anatomy, across domains and species.
... Despite tremendous advances in machine learning, a large gap still exists between humans and machines in terms of learning efficiency and generalization (Tenenbaum et al., 2011;Behrens et al., 2018;. A key reason for this is thought to be that machines lack the ability to generalize compositionally, which humans heavily rely on (Fodor and Pylyshyn, 1988;Lake et al., 2017;Battaglia et al., 2018;Goyal and Bengio, 2022;Greff et al., 2020). ...
Learning representations that generalize to novel compositions of known concepts is crucial for bridging the gap between human and machine perception. One prominent effort is learning object-centric representations, which are widely conjectured to enable compositional generalization. Yet, it remains unclear when this conjecture will be true, as a principled theoretical or empirical understanding of compositional generalization is lacking. In this work, we investigate when com-positional generalization is guaranteed for object-centric representations through the lens of identifiability theory. We show that autoencoders that satisfy structural assumptions on the decoder and enforce encoder-decoder consistency will learn object-centric representations that provably generalize compositionally. We validate our theoretical result and highlight the practical relevance of our assumptions through experiments on synthetic image data.
... Memory allows us to use past experience to generate expectations about the future. Integration of past information to predict future events enables efficient planning and flexible behavior in complex environments [1][2][3][4] and has been proposed to be a primary function of memory systems 5 and of the brain itself 6,7 . For predictions to usefully impact behavior, they should be represented on multiple timescales, allowing us to anticipate not just immediately upcoming events but also events further in the future. ...
Memory for temporal structure enables both planning of future events and retrospection of past events. We investigated how the brain flexibly represents extended temporal sequences into the past and future during anticipation. Participants learned sequences of environments in immersive virtual reality. Pairs of sequences had the same environments in a different order, enabling context-specific learning. During fMRI, participants anticipated upcoming environments multiple steps into the future in a given sequence. Temporal structure was represented in the hippocampus and across higher-order visual regions (1) bidirectionally, with graded representations into the past and future and (2) hierarchically, with further events into the past and future represented in successively more anterior brain regions. In hippocampus, these bidirectional representations were context-specific, and suppression of far-away environments predicted response time costs in anticipation. Together, this work sheds light on how we flexibly represent sequential structure to enable planning over multiple timescales.
... The firing phase of place cells can encode distinct task-related information within the same place field Beyond encoding spatial information, the hippocampus has been shown to carry information about episodic memories 44,45 , sequences 46,47 , and abstract relations 48,49 . Thus, hippocampal assemblies may encode task-relevant information beyond purely spatial parameters. ...
Spatial information is encoded by location-dependent hippocampal place cell firing rates and sub-second, rhythmic entrainment of spike times. These rate and temporal codes have primarily been characterized in low-dimensional environments under limited cognitive demands; but how is coding configured in complex environments when individual place cells signal several locations, individual locations contribute to multiple routes and functional demands vary? Quantifying CA1 population dynamics of male rats during a decision-making task, here we show that the phase of individual place cells’ spikes relative to the local theta rhythm shifts to differentiate activity in different place fields. Theta phase coding also disambiguates repeated visits to the same location during different routes, particularly preceding spatial decisions. Using unsupervised detection of cell assemblies alongside theoretical simulation, we show that integrating rate and phase coding mechanisms dynamically recruits units to different assemblies, generating spiking sequences that disambiguate episodes of experience and multiplexing spatial information with cognitive context.
... Decision making potentially relies on direct learning of action policies rather than a comparison of stored values (Hayden & Niv 2021). Furthermore, the OFC, one of the main putative value-related areas, may play a wider role in cognition by representing cognitive maps of state or task space (Behrens et al. 2018, Schuck et al. 2016, Wilson et al. 2014). This perspective suggests that the effects of OFC and vmPFC lesions on decision making (Bechara et al. 1994, Fellows & Farah 2007, Yu et al. 2022) might stem from broader cognitive impairments rather than a specific problem in value encoding. ...
Subjective value is a core concept in neuroeconomics, serving as the basis for decision making. Despite the extensive literature on the neural encoding of subjective reward value in humans, the neural representation of punishment value remains relatively understudied. This review synthesizes current knowledge on the neural representation of reward value, including methodologies, involved brain regions, and the concept of common currency representation of diverse reward types in decision-making and learning processes. We then critically examine existing research on the neural representation of punishment value, highlighting conceptual and methodological challenges in human studies and insights gained from animal research. Finally, we explore how individual differences in reward and punishment processing may be linked to various mental illnesses, with a focus on stress-related psychopathologies. This review advocates for the integration of both rewards and punishments within value-based decision-making and learning frameworks, leveraging insights from cross-species studies and utilizing ecological gamified paradigms to reflect real-life scenarios.
... The first originates from Tolman's (1948) pioneering notion of a "cognitive map". Research in this domain has extensively studied spatial navigation in the hippocampal-entorhinal system (Whittington et al. 2022;Epstein et al. 2017;Moser et al. 2014), which has since been extended to a wide range of nonspatial modalities and domains (Behrens et al. 2018). The second tradition, known as Bayesian structure induction (Kemp & Tenenbaum 2008), builds on a similar formalism as Bayesian concept learning (Tenenbaum & Griffiths 2001), where explicit, rule-like hypotheses about structure can be inferred from observed data, reflecting our ability to discern patterns and regularities in the environment. ...
Generalization, defined as applying limited experiences to novel situations, represents a cornerstone of human intelligence. Our review traces the evolution and continuity of psychological theories of generalization, from its origins in concept learning (categorizing stimuli) and function learning (learning continuous input-output relationships) to domains such as reinforcement learning and latent structure learning. Historically, there have been fierce debates between approaches based on rule-based mechanisms, which rely on explicit hypotheses about environmental structure, and approaches based on similarity-based mechanisms, which leverage comparisons to prior instances. Each approach has unique advantages: Rules support rapid knowledge transfer, while similarity is computationally simple and flexible. Today, these debates have culminated in the development of hybrid models grounded in Bayesian principles, effectively marrying the precision of rules with the flexibility of similarity. The ongoing success of hybrid models not only bridges past dichotomies but also underscores the importance of integrating both rules and similarity for a comprehensive understanding of human generalization.
... The orbitofrontal cortex (OFC) and hippocampus (HC) are both implicated in learning the states, and relationships between them, that define the world around us [1][2][3] . These so-called cognitive or task maps are fundamental to the process by which we predict impending events, particularly valuable outcomes 4,5 . Both areas are also well-positioned to provide information, either directly or indirectly, to the midbrain dopamine neurons, which generate teaching signals that reflect discrepancies between actual and expected outcomes [6][7][8] , i.e., prediction errors. ...
The orbitofrontal cortex (OFC) and hippocampus (HC) both contribute to the cognitive maps that support flexible behavior. Previously, we used the dopamine neurons to measure the functional role of OFC. We recorded midbrain dopamine neurons as rats performed an odor-based choice task, in which expected rewards were manipulated across blocks. We found that ipsilateral OFC lesions degraded dopaminergic prediction errors, consistent with reduced resolution of the task states. Here we have repeated this experiment in male rats with ipsilateral HC lesions. The results show HC also shapes the task states, however unlike OFC, which provides information local to the trial, the HC is necessary for estimating upper-level hidden states that distinguish blocks. The results contrast the roles of the OFC and HC in cognitive mapping and suggest that the dopamine neurons access rich information from distributed regions regarding the environment’s structure, potentially enabling this teaching signal to support complex behaviors.
... Our results corroborate and extend this by indicating a differential recruitment of the two subsystems according to social layer. The association of distant social layers with the DMN-MT subsystem may be attributed to episodes memory; nonetheless, its presence across all participant groups, irrespective of judgment strategy (Supplementary Note 1), in tandem with the association of distant social layers with larger spatial scales, reinforces the view that DMN-MT supports the organization and manipulation of relational knowledge, beyond the representation of spatial or temporal scales 45,[87][88][89] . Concurrently, the DMN-Core subsystem, associated with mentalizing 37 , exhibited preference for the support clique. ...
Healthy social life requires relationships in different levels of personal closeness. Based on ethological, sociological, and psychological evidence, social networks have been divided into five layers, gradually increasing in size and decreasing in personal closeness. Is this division also reflected in brain processing of social networks? During functional MRI, 21 participants compared their personal closeness to different individuals. We examined the brain volume showing differential activation for varying layers of closeness and found that a disproportionately large portion of this volume (80%) exhibited preference for individuals closest to participants, while separate brain regions showed preference for all other layers. Moreover, this bipartition reflected cortical preference for different sizes of physical spaces, as well as distinct subsystems of the default mode network. Our results support a division of the neurocognitive processing of social networks into two patterns depending on personal closeness, reflecting the unique role intimately close individuals play in our social lives.
... Knowledge graphs can integrate and utilize massive amounts of data to mine high-value information in order to provide important support for decision-making and many other aspects. Behrens, T.E et al. discussed the theoretical concepts and the formation process of knowledge mapping, providing insights into the extreme generalization, abstraction, and inference of human cognitive features, and through in-depth exploration of the hidden mysteries behind the principles of cognitive mapping, a better understanding of the functioning mechanism of the human mind and drawing inspiration from it to guide the path of future scientific and technological development [13]. Zhang, W et al. explored how to combine rule-based and embedding-based methods to be applied in knowledge graphs to give full play to their respective advantages and make up for the shortcomings through iterative learning, pruning strategies, and other means, successfully improved the sparse entity representation and link prediction results, and this research provides effective ideas and methods for further improving the construction and application of knowledge graphs [14]. ...
Career planning is of great importance to this special group of athletes, but most athletes do not have a better personal development plan for their professional sports career, so it is necessary to analyze the career development of soccer players. In this paper, firstly, we constructed the knowledge map of soccer players’ career development using the GAT mechanism and RNN model classification to construct the GGNN inference model and then analyzed the group characteristics of soccer players. Finally, the construction of a career development assistance system for soccer players was implemented. The main results are as follows: the cultural characteristics of soccer players from 2000 to 2020 found that the percentage of their university education increased by 13%. While the percentage of education in other stages decreased, especially the percentage of education in junior high school and elementary school decreased significantly by 11% and 33%. In the survey on the influencing factors of soccer players’ future re-selection, it was found that the influence of parents and friends on soccer players’ view of career choice reached 71%, of which 43% believed that their parents had the greatest influence on them, and 28% believed that it was their friends. Obviously, the model of soccer players’ career development that integrates knowledge graphs can better help soccer players implement career planning and improve the security of retired soccer players’ re-choice of career.
... Sometimes these mappings can be completely arbitrary like in video games, where an "up" press on a video game controller can lead a virtual character to move or jump. The advantage of learning and using these mappings, as opposed to simple state-action associations, is that the mappings can be used for planning and generalization to novel contexts [61][62][63]. ...
A fundamental feature of the human brain is its capacity to learn novel motor skills. This capacity requires the formation of vastly different visuomotor mappings. Using a grid navigation task, we investigated whether training variability would enhance the flexible use of a visuomotor mapping (key-to-direction rule), leading to better generalization performance. Experiments 1 and 2 show that participants trained to move between multiple start-target pairs exhibited greater generalization to both distal and proximal targets compared to participants trained to move between a single pair. This finding suggests that limited variability can impair decisions even in simple tasks without planning. In addition, during the training phase, participants exposed to higher variability were more inclined to choose options that, counterintuitively, moved the cursor away from the target while minimizing its actual distance under the constrained mapping, suggesting a greater engagement in model-based computations. In Experiments 3 and 4, we showed that the limited generalization performance in participants trained with a single pair can be enhanced by a short period of variability introduced early in learning or by incorporating stochasticity into the visuomotor mapping. Our computational modeling analyses revealed that a hybrid model between model-free and model-based computations with different mixing weights for the training and generalization phases, best described participants’ data. Importantly, the differences in the model-based weights between our experimental groups, paralleled the behavioral findings during training and generalization. Taken together, our results suggest that training variability enables the flexible use of the visuomotor mapping, potentially by preventing the consolidation of habits due to the continuous demand to change responses.
... Such abstraction has been described as a cognitive map in the context of spatial navigation 6,7 and 'schemas' or 'learning sets' in the context of human behaviour and memory research 32,33 . Recent computational research suggests that the brain implements similar neural mechanisms to extract the underlying structure of spatial as well as non-spatial problems and that the integration of 'what', 'where' and 'when' is essential for this process 7,34,35 . The temporal relational neurons that we identify here in human participants during a non-spatial task, have important implications for the hippocampal-entorhinal system as a neural substrate of the cognitive map. ...
Extracting the underlying temporal structure of experience is a fundamental aspect of learning and memory that allows us to predict what is likely to happen next. Current knowledge about the neural underpinnings of this cognitive process in humans stems from functional neuroimaging research1–5. As these methods lack direct access to the neuronal level, it remains unknown how this process is computed by neurons in the human brain. Here we record from single neurons in individuals who have been implanted with intracranial electrodes for clinical reasons, and show that human hippocampal and entorhinal neurons gradually modify their activity to encode the temporal structure of a complex image presentation sequence. This representation was formed rapidly, without providing specific instructions to the participants, and persisted when the prescribed experience was no longer present. Furthermore, the structure recovered from the population activity of hippocampal–entorhinal neurons closely resembled the structural graph defining the sequence, but at the same time, also reflected the probability of upcoming stimuli. Finally, learning of the sequence graph was related to spontaneous, time-compressed replay of individual neurons’ activity corresponding to previously experienced graph trajectories. These findings demonstrate that neurons in the hippocampus and entorhinal cortex integrate the ‘what’ and ‘when’ information to extract durable and predictive representations of the temporal structure of human experience.
... Some argue for the primacy of space in mapping daily experience, suggesting that the role of the hippocampus in memory stems from its role in processing spatial information (35). Conversely, others propose that the hippocampus performs a general computation, encoding relationships among stimuli, thereby contributing to both spatial and nonspatial tasks (36,37). Some human researchers, on the other hand, posit that memory and navigation represent distinct processes, with the primate hippocampus potentially evolving to play a more notable role in declarative memory (38). ...
The primate hippocampus, crucial for both episodic memory and spatial navigation, remains an enigma regarding whether these functions share the same neural substrates. We investigated how identical hippocampal neurons in macaque monkeys dynamically shifted their representations between tasks. In a recognition memory task, a notable fraction of hippocampal neurons showed that rate modulation strongly correlated with recognition performance. During free navigation in an open arena, spatial view, rather than position, predominantly influenced the spatial selectivity of hippocampal neurons. Neurons selective for recognition memory displayed minimal spatial tuning, while spatially tuned neurons exhibited limited memory-related activity. These neural correlates of recognition memory and space were more pronounced in the anterior and posterior portions of the hippocampus, respectively. These opposing gradients extended further into the anterior and posterior neocortices. Overall, our findings suggest the presence of orthogonal long-axis gradients between recognition memory and spatial navigation in the hippocampal-neocortical networks of macaque monkeys.
... Further, an intracranial electroencephalogram study showed that temporal self-projection activates brain regions within the lateral temporal lobe [23], also a hub of the core network. Taken together, these results suggest that the core network may not only support temporal self-projection but may also play an expanded role, flexibly enabling the mental representation of a continuous axis and the projection of oneself to different points on this axis in different domains [24,25]. ...
Mental time travel (MTT), a cornerstone of human cognition, enables individuals to mentally project themselves into their past or future. It was shown that this self-projection may extend beyond the temporal domain to the spatial and social domains. What about higher cognitive domains? Twenty-eight participants underwent functional magnetic resonance imaging (fMRI) while self-projecting to different political, moral and temporal perspectives. For each domain, participants were asked to judge their relationship to various people (politicians, moral figures, personal acquaintances) from their actual or projected self-location. Findings showed slower, less accurate responses during self-projection across all domains. fMRI analysis revealed self-projection elicited brain activity at the precuneus, medial and dorsolateral prefrontal cortex, temporoparietal junction and anterior insula, bilaterally and right lateral temporal cortex. Notably, 23.5% of active voxels responded to all three domains and 27% to two domains, suggesting a shared brain system for self-projection. For ordinality judgement (self-reference), 52.5% of active voxels corresponded to the temporal domain specifically. Self-projection activity overlapped mostly with the frontoparietal control network, followed by the default mode network, while self-reference showed a reversed pattern, demonstrating MTT’s implication in spontaneous brain activity. MTT may thus be regarded as a ‘mental-experiential travel’, with self-projection as a domain-general construct and self-reference related mostly to time.
This article is part of the theme issue ‘Elements of episodic memory: lessons from 40 years of research’.
... In contrast, the changes leading to the abstraction over irrelevant features might require latent consolidation mechanisms, such as those associated with the formation of abstract schemas in memory (Behrens et al., 2018;Preston & Eichenbaum, 2013;Rasch & Born, 2013), statistical regularizations of the learned weights (Löwe et al., 2024), or replay (Preston & Eichenbaum, 2013) and synaptic downscaling (Norimoto et al., 2018) that are often regulated by over night of sleep. Future work manipulating the conditions of practice or overnight consolidation might provide insights into these mechanisms of change. ...
Extensive practice makes task performance more efficient and precise, leading to automaticity. However, theories of automaticity differ on which levels of task representations (e.g., low-level features, stimulus-response mappings, or high-level conjunctive memories of individual events) change with practice, despite predicting the same pattern of improvement (e.g., power law of practice). To resolve this controversy, we built on recent theoretical advances in understanding computations through neural population dynamics. Specifically, we hypothesized that practice optimizes the neural representational geometry of task representations to minimally separate the highest-level task contingencies needed for successful performance. This involves efficiently reaching conjunctive neural states that integrate task-critical features nonlinearly while abstracting over non-critical dimensions. To test this hypothesis, human participants (n = 40) engaged in extensive practice of a simple, context-dependent action selection task over 3 days while recording EEG. During initial rapid improvement in task performance, representations of the highest-level, context-specific conjunctions of task-features were enhanced as a function of the number of successful episodes. Crucially, only enhancement of these conjunctive representations, and not lower-order representations, predicted the power-law improvement in performance. Simultaneously, over sessions, these conjunctive neural states became more stable earlier in time and more aligned, abstracting over redundant task features, which correlated with offline performance gain in reducing switch costs. Thus, practice optimizes the dynamic representational geometry as task-tailored neural states that minimally tesselate the task space, taming their high-dimensionality.
... Preserving the memory of an important life event requires successful integration of several modalities that constitute a multisensory trace, often referred to as the memory engram [1]. In particular, spatial memory is defined as the brain's ability to encode key features of the external environment and to navigate within the boundaries of this mental representation, also known as cognitive map [2,3]. On the physiological level, it manifests as populations of neurons exhibiting activity tuned to specific aspects of the external spatial context, in particular, firing correlated with an animal's presence in a certain, unique location within the environment [4,5]. ...
The formation of memories is a complex, multi-scale phenomenon, especially when it involves integration of information from various brain systems. We have investigated the differences between a novel and consolidated association of spatial cues and amphetamine administration, using an in situ hybridisation method to track the short-term dynamics during the recall testing. We have found that remote recall group involves smaller, but more consolidated groups of neurons, which is consistent with their specialisation. By employing machine learning analysis, we have shown this pattern is especially pronounced in the VTA; furthermore, we also uncovered significant activity patterns in retrosplenial and prefrontal cortices, as well as in the DG and CA3 subfields of the hippocampus. The behavioural propensity towards the associated localisation appears to be driven by the nucleus accumbens, however, further modulated by a trio of the amygdala, VTA and hippocampus, as the trained association is confronted with test experience. Moreover, chemogenetic analysis revealed central amygdala as critical for linking appetitive emotional states with spatial contexts. These results show that memory mechanisms must be modelled considering individual differences in motivation, as well as covering dynamics of the process.
The rise of generative artificial intelligence (GAI) applications, epitomized by ChatGPT, has reshaped design processes by enhancing idea generation and conceptual depth for designers. However, the facilitating effects on novice designers' thinking remain uncertain, particularly in the context of sustainable service concept generation. This study examines these impacts of ChatGPT on design thinking process and outcomes through controlled experiments with 36 novice designers with ChatGPT, Tiangong AI, and no tools under a sustainable service design task. Through the protocol analysis, this study visualizes the design thinking by network-based cognitive maps, then evaluates design outcomes and systematically analyzes characteristics of design thinking development under different tool interventions. Findings indicate that ChatGPT enhances design concept novelty and systematicity but has limited impact on originality and sustainability. Furthermore, ChatGPT plays an active role in fostering thinking divergence and fluency, especially in providing relevant guides for developed ideas and accelerating the evaluation and creation process. The network-based cognitive maps reveal distinct shifts and styles influenced by ChatGPT, providing references for novice designers using such tools to enhance inspiration and design fluency, and also effectively employ diverse tools during specific concept generation stage. The study also provides insights for enhancing the relevance of educational curricula and enabling bottom-up sustainable service innovations.
The ability to use subjective, latent contextual representations to influence decision-making is crucial for everyday life. The hippocampus is hypothesized to bind together otherwise abstract combinations of stimuli to represent such latent contexts, to support the process of hidden state inference. Yet evidence for a role of the hippocampus in hidden state inference remains limited. We found that the ventral hippocampus is required for mice to perform hidden state inference during a two-armed bandit task. Hippocampal neurons differentiate the two abstract contexts required for this strategy in a manner similar to the differentiation of spatial locations, and their activity is essential for appropriate dopamine dynamics. These findings offer insight into how latent contextual information is used to optimize decisions, and they emphasize a key role for the hippocampus in hidden state inference.
Movies captivate groups of individuals (the audience), especially if they contain themes of common motivational interest to the group. In drug addiction, a key mechanism is maladaptive motivational salience attribution whereby drug cues outcompete other reinforcers within the same environment or context. We predicted that while watching a drug-themed movie, where cues for drugs and other stimuli share a continuous narrative context, fMRI responses in individuals with heroin use disorder (iHUD) will preferentially synchronize during drug scenes.
Thirty inpatient iHUD (24 male) and 25 healthy controls (16 male) watched a drug-themed movie at baseline and at follow-up after 15 weeks.
Results revealed such drug-biased synchronization in the orbitofrontal cortex (OFC), ventromedial and ventrolateral prefrontal cortex, and insula. After 15 weeks during ongoing inpatient treatment, there was a significant reduction in this drug-biased shared response in the OFC, which correlated with a concomitant reduction in dynamically-measured craving, suggesting synchronized OFC responses to a drug-themed movie as a neural marker of craving and recovery in iHUD.
A striking feature of human cognition is an exceptional ability to rapidly adapt to novel situations. It is proposed this relies on abstracting and generalizing past experiences. While previous research has explored how humans detect and generalize single sequential processes, we have a limited understanding of how humans adapt to more naturalistic scenarios, for example, complex, multisubprocess environments. Here, we propose a candidate computational mechanism that posits compositional generalization of knowledge about subprocess dynamics. In two samples ( N = 238 and N = 137), we combined a novel sequence learning task and computational modeling to ask whether humans extract and generalize subprocesses compositionally to solve new problems. In prior learning, participants experienced sequences of compound images formed from two graphs’ product spaces (group 1: G1 and G2, group 2: G3 and G4). In transfer learning, both groups encountered compound images from the product of G1 and G3, composed entirely of new images. We show that subprocess knowledge transferred between task phases, such that in a new task environment each group had enhanced accuracy in predicting subprocess dynamics they had experienced during prior learning. Computational models utilizing predictive representations, based solely on the temporal contiguity of experienced task states, without an ability to transfer knowledge, failed to explain these data. Instead, behavior was consistent with a predictive representation model that maps task states between prior and transfer learning. These results help advance a mechanistic understanding of how humans discover and abstract subprocesses composing their experiences and compositionally reuse prior knowledge as a scaffolding for new experiences.
To flexibly adapt to new situations, our brains must understand the regularities in the world, as well as those in our own patterns of behaviour. A wealth of findings is beginning to reveal the algorithms that we use to map the outside world1, 2, 3, 4, 5–6. However, the biological algorithms that map the complex structured behaviours that we compose to reach our goals remain unknown. Here we reveal a neuronal implementation of an algorithm for mapping abstract behavioural structure and transferring it to new scenarios. We trained mice on many tasks that shared a common structure (organizing a sequence of goals) but differed in the specific goal locations. The mice discovered the underlying task structure, enabling zero-shot inferences on the first trial of new tasks. The activity of most neurons in the medial frontal cortex tiled progress to goal, akin to how place cells map physical space. These ‘goal-progress cells’ generalized, stretching and compressing their tiling to accommodate different goal distances. By contrast, progress along the overall sequence of goals was not encoded explicitly. Instead, a subset of goal-progress cells was further tuned such that individual neurons fired with a fixed task lag from a particular behavioural step. Together, these cells acted as task-structured memory buffers, implementing an algorithm that instantaneously encoded the entire sequence of future behavioural steps, and whose dynamics automatically computed the appropriate action at each step. These dynamics mirrored the abstract task structure both on-task and during offline sleep. Our findings suggest that schemata of complex behavioural structures can be generated by sculpting progress-to-goal tuning into task-structured buffers of individual behavioural steps.
Humans build mental models of the world and utilize them for various cognitive tasks. The exact form of cognitive maps is not fully understood, especially for novel and complex environments beyond the flat Euclidean environment. To address this gap, we investigated path integration—a critical process underlying cognitive mapping—and spatial-memory capacity on the spherical (non-Euclidean) and planar (Euclidean) environments in young healthy adults ( N = 20) using immersive virtual reality. We observed a strong Euclidean bias during the path-integration task on the spherical surface, even among participants who possessed knowledge of non-Euclidean geometry. Notably, despite this bias, participants demonstrated reasonable navigation ability on the sphere. This observation and simulation suggest that humans navigate nonflat surfaces by constructing locally confined Euclidean maps and flexibly combining them. This insight sheds light on potential neural mechanisms and behavioral strategies for solving complex cognitive tasks.
Transitive inference (TI) is a cognitive task that assesses an organism’s ability to infer novel relations between items based on previously acquired knowledge. TI is known for exhibiting various behavioral and neural signatures, such as the serial position effect (SPE), symbolic distance effect (SDE), and the brain’s capacity to maintain and merge separate ranking models. We propose a novel framework that casts TI as a probabilistic preference learning task, using one-parameter Mallows models. We present a series of simulations that highlight the effectiveness of our novel approach. We show that the Mallows ranking model natively reproduces SDE and SPE. Furthermore, extending the model using Bayesian selection showcases its capacity to generate and merge ranking hypotheses as pairs with connecting symbols. Finally, we employ neural networks to replicate Mallows models, demonstrating how this framework aligns with observed prefrontal neural activity during TI. Our innovative approach sheds new light on the nature of TI, emphasizing the potential of probabilistic preference learning for unraveling its underlying neural mechanisms.
In the last twenty years, knowledge-centered approaches have become increasingly popular in analytic epistemology. Rather than trying to account for knowledge in other terms, these approaches take knowledge as the starting point for the elucidation of other epistemic notions (such as belief, justification, and rationality). Knowledge-centered approaches have been so influential that it now looks as if epistemology is undergoing a factive turn. However, relatively little has been done to explore how knowledge-centered views fare in new fields inside and beyond epistemology strictly understood. This volume aims to remedy this situation by putting together contributions that investigate the significance of knowledge in debates where its roles have been less explored. The goal is to see how far knowledge-centered views can go by exploring new prospects and identifying new trends of research for the knowledge-first program. Extending knowledge-centered approaches in this way promises not only to deliver novel insights into these neglected fields but also to revisit more traditional debates from a fresh perspective. As a whole, the volume develops and evaluates the knowledge-first program in original and fertile ways.
The cognitive ability to go beyond the present to consider alternative possibilities, including potential futures and counterfactual pasts, can support adaptive decision making. Complex and changing real-world environments, however, have many possible alternatives. Whether and how the brain can select among them to represent alternatives that meet current cognitive needs remains unknown. We therefore examined neural representations of alternative spatial locations in the rat hippocampus during navigation in a complex patch foraging environment with changing reward probabilities. We found representations of multiple alternatives along paths ahead and behind the animal, including in distant alternative patches. Critically, these representations were modulated in distinct patterns across successive trials: alternative paths were represented proportionate to their evolving relative value and predicted subsequent decisions, whereas distant alternatives were prevalent during value updating. These results demonstrate that the brain modulates the generation of alternative possibilities in patterns that meet changing cognitive needs for adaptive behavior.
Transfer learning, the reuse of newly acquired knowledge under novel circumstances, is a critical hallmark of human intelligence that has frequently been pitted against the capacities of artificial learning agents. Yet, the computations relevant to transfer learning have been little investigated in humans. The benefit of efficient inductive biases (meta-level constraints that shape learning, often referred as priors in the Bayesian learning approach), has been both theoretically and experimentally established. Efficiency of inductive biases depends on their capacity to generalize earlier experiences. We argue that successful transfer learning upon task acquisition is ensured by updating inductive biases and transfer of knowledge hinges upon capturing the structure of the task in the inductive bias that can be reused in novel tasks. To explore this, we trained participants on a non-trivial visual stimulus sequence task (Alternating Serial Response Times, ASRT); during the Training phase, participants were exposed to one specific sequence for multiple days, then on the Transfer phase, the sequence changed, while the underlying structure of the task remained the same. Our results show that beyond the acquisition of the stimulus sequence, our participants were also able to update their inductive biases. Acquisition of the new sequence was considerably sped up by earlier exposure but this enhancement was specific to individuals showing signatures of abandoning initial inductive biases. Enhancement of learning was reflected in the development of a new internal model. Additionally, our findings highlight the ability of participants to construct an inventory of internal models and alternate between them based on environmental demands. Further, investigation of the behavior during transfer revealed that it is the subjective internal model of individuals that can predict the transfer across tasks. Our results demonstrate that even imperfect learning in a challenging environment helps learning in a new context by reusing the subjective and partial knowledge about environmental regularities.
Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice : the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning : expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here we employ a recently-developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.
The hippocampus and the medial entorhinal cortex are part of a brain system that maps self-location during navigation in the proximal environment1,2. In this system, correlations between neural firing and an animal’s position or orientation are so evident that cell types have been given simple descriptive names, such as place cells³, grid cells⁴, border cells5,6 and head-direction cells⁷. While the number of identified functional cell types is growing at a steady rate, insights remain limited by an almost-exclusive reliance on recordings from rodents foraging in empty enclosures that are different from the richly populated, geometrically irregular environments of the natural world. In environments that contain discrete objects, animals are known to store information about distance and direction to those objects and to use this vector information to guide navigation8–10. Theoretical studies have proposed that such vector operations are supported by neurons that use distance and direction from discrete objects11,12 or boundaries13,14 to determine the animal’s location, but—although some cells with vector-coding properties may be present in the hippocampus¹⁵ and subiculum16,17—it remains to be determined whether and how vectorial operations are implemented in the wider neural representation of space. Here we show that a large fraction of medial entorhinal cortex neurons fire specifically when mice are at given distances and directions from spatially confined objects. These ‘object-vector cells’ are tuned equally to a spectrum of discrete objects, irrespective of their location in the test arena, as well as to a broad range of dimensions and shapes, from point-like objects to extended surfaces. Our findings point to vector coding as a predominant form of position coding in the medial entorhinal cortex.
Over the past 20 years, neuroscience research on reward-based learning has converged on a canonical model, under which the neurotransmitter dopamine 'stamps in' associations between situations, actions and rewards by modulating the strength of synaptic connections between neurons. However, a growing number of recent findings have placed this standard model under strain. We now draw on recent advances in artificial intelligence to introduce a new theory of reward-based learning. Here, the dopamine system trains another part of the brain, the prefrontal cortex, to operate as its own free-standing learning system. This new perspective accommodates the findings that motivated the standard model, but also deals gracefully with a wider range of observations, providing a fresh foundation for future research.
Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go. Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex. Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space and is critical for integrating self-motion (path integration) and planning direct trajectories to goals (vector-based navigation). Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types12. We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments—optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation, demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments.
The representation of others in space
Different sets of neurons encode the spatial position and orientation of an organism. However, social animals need to know the position of other individuals for social interactions, observational learning, and group navigation. Surprisingly, very little is known about how the position of other animals is represented in the brain. Danjo et al. and Omer et al. now report the discovery of a subgroup of neurons in hippocampal area CA1 that encodes the presence of conspecifics in rat and bat brains, respectively.
Science , this issue p. 213 , p. 218
When participants performed a visual search task, functional MRI responses in entorhinal cortex exhibited a sixfold periodic modulation by gaze-movement direction. The orientation of this modulation was determined by the shape and orientation of the bounded search space. These results indicate that human entorhinal cortex represents visual space using a boundary-anchored grid, analogous to that used by rodents to represent navigable space.
Entorhinal grid cells map the local environment, but their involvement beyond spatial navigation remains elusive. We examined human functional MRI responses during a highly controlled visual tracking task and show that entorhinal cortex exhibited a sixfold rotationally symmetric signal encoding gaze direction. Our results provide evidence for a grid-like entorhinal code for visual space and suggest a more general role of the entorhinal grid system in coding information along continuous dimensions.
ELife digest
To help us navigate, the brain encodes information about the positions of landmarks in space in a series of maps. These maps are housed by two neighbouring brain regions called the hippocampus and entorhinal cortex. These regions also encode information about non-spatial relationships, for example, between two events that often occur close together in time. However, it was not known whether such non-spatial relationships may also be encoded as a map.
To address this question, Garvert et al. showed volunteers a series of objects on a screen. Unbeknown to the volunteers, the order of the objects was not entirely random. Instead, each object could only follow certain others. The objects were thus connected to one another by a network of non-spatial relationships, broadly comparable to the spatial relationships that connect physical locations in the environment. The next day, the volunteers viewed some of the objects again, this time while lying inside a brain scanner. Although the volunteers still believed that the objects had been presented at random, the activity of their hippocampus and entorhinal cortex reflected the non-spatial relationships volunteers had experienced between the objects. The relationships were organised in an abstract map.
This suggests that the brain organises knowledge about abstract non-spatial relationships into maps comparable to those used to represent spatial relationships. The brain can use these maps of non-spatial relationships to guide our behaviour, even though we have no conscious awareness of the information they contain. The maps may also enable us to make new inferences, just as we can use our spatial maps to find short cuts or navigate around obstacles. Future studies should investigate the mechanisms underlying our ability to create maps of non-spatial relationships and how we use them to guide decision making.
DOI: http://dx.doi.org/10.7554/eLife.17086.002
ELife digest
To help someone learn a new task, we might give them a reward after they have performed well. However, these rewards tend to be given several seconds or minutes after the behavior they are supposed to promote. Therefore, it is unclear how the rewards affect the brain and help accelerate the learning process.
Information is processed and sent around the brain by networks of cells called neurons. These networks are constantly remodeled because learning changes the connections—called synapses—that neighboring neurons signal across. Synapses can be strengthened so that signals are sent across them more easily in the future. Synapses can also be weakened, making it harder for the neurons to subsequently communicate.
A chemical called dopamine is often produced in the brain when a reward is received. If dopamine is present in a synapse whilst a neuron is signaling to its neighbor, it can affect how effectively this communication occurs. Brzosko et al. have now investigated whether dopamine can also change the synapses if it is applied after signaling has already happened.
The strengthening or weakening of synapses can be triggered by electrically stimulating the neurons on either side of a synapse at particular times. Brzosko et al. did this to neurons in slices of mouse brain, and then applied dopamine to the neurons. The results suggest that dopamine can reverse synaptic weakening and can even cause the synapses to strengthen. However, the dopamine had to be applied immediately after stimulation to be able to strengthen the synapse. The next challenge is to establish whether this change in synaptic strength is responsible for the change in behavior.
DOI: http://dx.doi.org/10.7554/eLife.09685.002
Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether offline integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.
A cognitive map has long been the dominant metaphor for hippocampal function, embracing the idea that place cells encode a geometric representation of space. However, evidence for predictive coding, reward sensitivity and policy dependence in place cells suggests that the representation is not purely spatial. We approach this puzzle from a reinforcement learning perspective: what kind of spatial representation is most useful for maximizing future reward? We show that the answer takes the form of a predictive representation. This representation captures many aspects of place cell responses that fall outside the traditional view of a cognitive map. Furthermore, we argue that entorhinal grid cells encode a low-dimensionality basis set for the predictive representation, useful for suppressing noise in predictions and extracting multiscale structure for hierarchical planning.
The hippocampus and orbitofrontal cortex (OFC) both make important contributions to decision making and other cognitive processes. However, despite anatomical links between the two, few studies have tested the importance of hippocampal-OFC interactions. Here, we recorded OFC neurons in rats performing a decision making task while suppressing activity in a key hippocampal output region, the ventral subiculum. OFC neurons encoded information about expected outcomes and rats' responses. With hippocampal output suppressed, rats were slower to adapt to changes in reward contingency, and OFC encoding of response information was strongly attenuated. In addition, ventral subiculum inactivation prevented OFC neurons from integrating information about features of outcomes to form holistic representations of the outcomes available in specific trial blocks. These data suggest that the hippocampus contributes to OFC encoding of both concrete, low-level features of expected outcomes, and abstract, inferred properties of the structure of the world, such as task state.
Planning can be defined as action selection that leverages an internal model of the outcomes likely to follow each possible action. Its neural mechanisms remain poorly understood. Here we adapt recent advances from human research for rats, presenting for the first time an animal task that produces many trials of planned behavior per session, making multitrial rodent experimental tools available to study planning. We use part of this toolkit to address a perennially controversial issue in planning: the role of the dorsal hippocampus. Although prospective hippocampal representations have been proposed to support planning, intact planning in animals with damaged hippocampi has been repeatedly observed. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide causal evidence directly linking dorsal hippocampus with planning behavior. Our results and methods open the door to new and more detailed investigations of the neural mechanisms of planning in the hippocampus and throughout the brain.
Transitive inference (TI) is a classic learning paradigm for which the relative contributions of experienced rewards and representation-based inference have been debated vigorously, particularly regarding the notion that animals are capable of logic and reasoning. Rhesus macaque subjects and human participants performed a TI task in which, prior to learning a 7-item list (ABCDEFG), a block of trials presented exclusively the pair FG. Contrary to the expectation of associative models, the high prior rate of reward for F did not disrupt subsequent learning of the entire list. Monkeys (who each completed many sessions with novel stimuli) learned to anticipate that novel stimuli should be preferred over F. We interpret this as evidence of a task representation of TI that generalizes beyond learning about specific stimuli. Humans (who were task-naïve) showed a transitory bias to F when it was paired with novel stimuli, but very rapidly unlearned that bias. Performance with respect to the remaining stimuli was consistent with past reports of TI in both species. These results are difficult to reconcile with any account that assigns the strength of association between individual stimuli and rewards. Instead, they support sophisticated cognitive processes in both species, albeit with some species differences.
The brain’s ability to associate different stimuli is vital for long-term memory, but how neural ensembles encode associative memories is unknown. Here we studied how cell ensembles in the basal and lateral amygdala encode associations between conditioned and unconditioned stimuli (CS and US, respectively). Using a miniature fluorescence microscope, we tracked the Ca²⁺ dynamics of ensembles of amygdalar neurons during fear learning and extinction over 6 days in behaving mice. Fear conditioning induced both up- and down-regulation of individual cells’ CS-evoked responses. This bi-directional plasticity mainly occurred after conditioning, and reshaped the neural ensemble representation of the CS to become more similar to the US representation. During extinction training with repetitive CS presentations, the CS representation became more distinctive without reverting to its original form. Throughout the experiments, the strength of the ensemble-encoded CS–US association predicted the level of behavioural conditioning in each mouse. These findings support a supervised learning model in which activation of the US representation guides the transformation of the CS representation.
Significance
Deep neural networks are currently the most successful machine-learning technique for solving a variety of tasks, including language translation, image classification, and image generation. One weakness of such models is that, unlike humans, they are unable to learn multiple tasks sequentially. In this work we propose a practical solution to train such models sequentially by protecting the weights important for previous tasks. This approach, inspired by synaptic consolidation in neuroscience, enables state of the art results on multiple reinforcement learning problems experienced sequentially.
Theories of reward learning in neuroscience have focused on two families of algorithms thought to capture deliberative versus habitual choice. ‘Model-based’ algorithms compute the value of candidate actions from scratch, whereas ‘model-free’ algorithms make choice more efficient but less flexible by storing pre-computed action values. We examine an intermediate algorithmic family, the successor representation, which balances flexibility and efficiency by storing partially computed action values: predictions about future events. These pre-computation strategies differ in how they update their choices following changes in a task. The successor representation’s reliance on stored predictions about future states predicts a unique signature of insensitivity to changes in the task’s sequence of events, but flexible adjustment following changes to rewards. We provide evidence for such differential sensitivity in two behavioural studies with humans. These results suggest that the successor representation is a computational substrate for semi-flexible choice in humans, introducing a subtler, more cognitive notion of habit.
Transitive inference (the ability to infer that “B>D” given that “B>C” and “C>D”) is a widespread characteristic of serial learning, observed in dozens of species. Despite these robust behavioral effects, reinforcement learning models reliant on reward prediction error or associative strength routinely fail to perform these inferences. We propose an algorithm called betasort, inspired by cognitive processes, which performs transitive inference at low computational cost. This is accomplished by (1) representing stimulus positions along a unit span using beta distributions, (2) treating positive and negative feedback asymmetrically, and (3) updating the position of every stimulus during every trial, whether that stimulus was visible or not. Performance was compared for rhesus macaques, humans, the betasort algorithm, and Q-learning (an established RPE model). Of these, only Q-learning failed to respond above chance during critical test trials. Implications for cognitive/associative rivalries, as well as for the model-based/model-free dichotomy, are discussed.
Neuronal reward valuations provide the physiological basis for economic behavior. Yet, how such valuations relate to economic decisions remains unclear. Here we show that the dorsolateral prefrontal cortex (DLPFC) implements a flexible value code based on object-specific valuations by single neurons. As monkeys perform a reward-based foraging task, individual DLPFC neurons signal the value of specific choice objects derived from recent experience. These neuronal object values satisfy principles of competitive choice mechanisms, track performance fluctuations, and follow predictions of a classical behavioral model (Herrnstein’s matching law). Individual neurons dynamically encode both, the updating of object values from recently experienced rewards, and their subsequent conversion to object choices during decision-making. Decoding from unselected populations enables a read-out of motivational and decision variables not emphasized by individual neurons. These findings suggest a dynamic single-neuron and population value code in DLPFC that advances from reward experiences to economic object values and future choices.
The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.
The hippocampus and the orbitofrontal cortex (OFC) both have important roles in cognitive processes such as learning, memory and decision making. Nevertheless, research on the OFC and hippocampus has proceeded largely independently, and little consideration has been given to the importance of interactions between these structures. Here, evidence is reviewed that the hippocampus and OFC encode parallel, but interactive, cognitive 'maps' that capture complex relationships between cues, actions, outcomes and other features of the environment. A better understanding of the interactions between the OFC and hippocampus is important for understanding the neural bases of flexible, goal-directed decision making.
Recent progress in artificial intelligence (AI) has renewed interest in building systems that learn and think like people. Many advances have come from using deep neural networks trained end-to-end in tasks such as object recognition, video games, and board games, achieving performance that equals or even beats humans in some respects. Despite their biological inspiration and performance achievements, these systems differ from human intelligence in crucial ways. We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn, and how they learn it. Specifically, we argue that these machines should (a) build causal models of the world that support explanation and understanding, rather than merely solving pattern recognition problems; (b) ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned; and (c) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations. We suggest concrete challenges and promising routes towards these goals that can combine the strengths of recent neural network advances with more structured cognitive models.
Dominant theories of hippocampal function propose that place cell representations are formed during an animal's first encounter with a novel environment and are subsequently replayed during off-line states to support consolidation and future behaviour. Here we report that viewing the delivery of food to an unvisited portion of an environment leads to off-line pre-activation of place cells sequences corresponding to that space. Such 'preplay' was not observed for an unrewarded but otherwise similar portion of the environment. These results suggest that a hippocampal representation of a visible, yet unexplored environment can be formed if the environment is of motivational relevance to the animal. We hypothesise such goal-biased preplay may support preparation for future experiences in novel environments.
When an organism receives a reward, it is crucial to know which of many candidate actions caused this reward. However, recent work suggests that learning is possible even when this most fundamental assumption is not met. We used novel reward-guided learning paradigms in two fMRI studies to show that humans deploy separable learning mechanisms that operate in parallel. While behavior was dominated by precise contingent learning, it also revealed hallmarks of noncontingent learning strategies. These learning mechanisms were separable behaviorally and neurally. Lateral orbitofrontal cortex supported contingent learning and reflected contingencies between outcomes and their causal choices. Amygdala responses around reward times related to statistical patterns of learning. Time-based heuristic mechanisms were related to activity in sensorimotor corticostriatal circuitry. Our data point to the existence of several learning mechanisms in the human brain, of which only one relies on applying known rules about the causal structure of the task.
Complex cognitive processes require sophisticated local processing but also interactions between distant brain regions. It is therefore critical to be able to study distant interactions between local computations and the neural representations they act on. Here we report two anatomically and computationally distinct learning signals in lateral orbitofrontal cortex (lOFC) and the dopaminergic ventral midbrain (VM) that predict trial-by-trial changes to a basic internal model in hippocampus. To measure local computations during learning and their interaction with neural representations, we coupled computational fMRI with trial-by-trial fMRI suppression. We find that suppression in a medial temporal lobe network changes trial-by-trial in proportion to stimulus-outcome associations. During interleaved choice trials, we identify learning signals that relate to outcome type in lOFC and to reward value in VM. These intervening choice feedback signals predicted the subsequent change to hippocampal suppression, suggesting a convergence of signals that update the flexible representation of stimulus-outcome associations.
The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of state-of-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.
Mammalian grid cells fire when an animal crosses the points of an imaginary hexagonal grid tessellating the environment. We show how animals can navigate by reading out a simple population vector of grid cell activity across multiple spatial scales, even though neural activity is intrinsically stochastic. This theory of dead reckoning explains why grid cells are organized into discrete modules within which all cells have the same lattice scale and orientation. The lattice scale changes from module to module and should form a geometric progression with a scale ratio of around 3/2 to minimize the risk of making large-scale errors in spatial localization. Such errors should also occur if intermediate-scale modules are silenced, whereas knocking out the module at the smallest scale will only affect spatial precision. For goal-directed navigation, the allocentric grid cell representation can be readily transformed into the egocentric goal coordinates needed for planning movements. The goal location is set by nonlinear gain fields that act on goal vector cells. This theory predicts neural and behavioral correlates of grid cell readout that transcend the known link between grid cells of the medial entorhinal cortex and place cells of the hippocampus.
Mammals are able to navigate to hidden goal locations by direct routes that may traverse previously unvisited terrain. Empirical evidence suggests that this "vector navigation" relies on an internal representation of space provided by the hippocampal formation. The periodic spatial firing patterns of grid cells in the hippocampal formation offer a compact combinatorial code for location within large-scale space. Here, we consider the computational problem of how to determine the vector between start and goal locations encoded by the firing of grid cells when this vector may be much longer than the largest grid scale. First, we present an algorithmic solution to the problem, inspired by the Fourier shift theorem. Second, we describe several potential neural network implementations of this solution that combine efficiency of search and biological plausibility. Finally, we discuss the empirical predictions of these implementations and their relationship to the anatomy and electrophysiology of the hippocampal formation.
Copyright © 2015 The Authors. Published by Elsevier Inc. All rights reserved.
Deciphering the neural mechanisms of social behavior has propelled the growth of social neuroscience. The exact computations of the social brain, however, remain elusive. Here we investigated how the human brain tracks ongoing changes in social relationships using functional neuroimaging. Participants were lead characters in a role-playing game in which they were to find a new home and a job through interactions with virtual cartoon characters. We found that a two-dimensional geometric model of social relationships, a "social space" framed by power and affiliation, predicted hippocampal activity. Moreover, participants who reported better social skills showed stronger covariance between hippocampal activity and "movement" through "social space." The results suggest that the hippocampus is crucial for social cognition, and imply that beyond framing physical locations, the hippocampus computes a more general, inclusive, abstract, and multidimensional cognitive map consistent with its role in episodic memory.
Copyright © 2015 Elsevier Inc. All rights reserved.
Mammals use distances and directions from local objects to calculate trajectories during navigation but how such vectorial operations are implemented in neural representations of space has not been determined. Here we show in freely moving mice that a population of neurons in the medial entorhinal cortex (MEC) responds specifically when the animal is at a given distance and direction from a spatially confined object. These ‘object-vector cells’ are tuned similarly to a spectrum of discrete objects, irrespective of their location in the test arena. The vector relationships are expressed from the outset in novel environments with novel objects. Object-vector cells are distinct from grid cells, which use a distal reference frame, but the cells exhibit some mixed selectivity with head-direction and border cells. Collectively, these observations show that object locations are integrated in metric representations of self-location, with specific subsets of MEC neurons encoding vector relationships to individual objects.
It is proposed that a cognitive map encoding the relationships between objects supports the ability to flexibly navigate the world. Place cells and grid cells provide evidence for such a map in a spatial context. Emerging evidence suggests analogous cells code for non-spatial information. Further, it has been shown that grid cells resemble the eigenvectors of the relationship between place cells and can be learnt from local inputs. Here we show that these locally-learnt eigenvectors contain not only local information but also global knowledge that can provide both distributions over future states as well as a global distance measure encoding approximate distances between every object in the world. By simply changing the weights in the grid cell population, it is possible to switch between computing these different measures. We demonstrate a simple algorithm can use these measures to globally navigate arbitrary topologies without searching more than one step ahead. We refer to this as intuitive planning.
The hippocampus plays a critical role in goal-directed navigation. Across different environments, however, hippocampal maps are randomized, making it unclear how goal locations could be encoded consistently. To address this question, we developed a virtual reality task with shifting reward contingencies to distinguish place versus reward encoding. In mice performing the task, large-scale recordings in CA1 and subiculum revealed a small, specialized cell population that was only active near reward yet whose activity could not be explained by sensory cues or stereotyped reward anticipation behavior. Across different virtual environments, most cells remapped randomly, but reward encoding consistently arose from a single pool of cells, suggesting that they formed a dedicated channel for reward. These observations represent a significant departure from the current understanding of CA1 as a relatively homogeneous ensemble without fixed coding properties and provide a new candidate for the cellular basis of goal memory in the hippocampus.
Decades of research on the neural code underlying spatial navigation have revealed a diverse set of neural response properties. The Entorhinal Cortex (EC) of the mammalian brain contains a rich set of spatial correlates, including grid cells which encode space using tessellating patterns. However, the mechanisms and functional significance of these spatial representations remain largely mysterious. As a new way to understand these neural representations, we trained recurrent neural networks (RNNs) to perform navigation tasks in 2D arenas based on velocity inputs. Surprisingly, we find that grid-like spatial response patterns emerge in trained networks, along with units that exhibit other spatial correlates, including border cells and band-like cells. All these different functional types of neurons have been observed experimentally. The order of the emergence of grid-like and border cells is also consistent with observations from developmental studies. Together, our results suggest that grid cells, border cells and others as observed in EC may be a natural solution for representing space efficiently given the predominant recurrent connections in the neural circuits.