ArticlePublisher preview available

Top-down models in biology: Explanation and control of complex living systems above the molecular level

The Royal Society
Journal of the Royal Society Interface
Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

It is widely assumed in developmental biology and bioengineering that optimal understanding and control of complex living systems follows from models of molecular events. The success of reductionism has overshadowed attempts at top-down models and control policies in biological systems. However, other fields, including physics, engineering and neuroscience, have successfully used the explanations and models at higher levels of organization, including least-action principles in physics and controltheoretic models in computational neuroscience. Exploiting the dynamic regulation of pattern formation in embryogenesis and regeneration requires new approaches to understand how cells cooperate towards large-scale anatomical goal states. Here, we argue that top-down models of pattern homeostasis serve as proof of principle for extending the current paradigm beyond emergence and molecule-level rules. We define top-down control in a biological context, discuss the examples of how cognitive neuroscience and physics exploit these strategies, and illustrate areas in which they may offer significant advantages as complements to the mainstream paradigm. By targeting system controls at multiple levels of organization and demystifying goal-directed (cybernetic) processes, top-down strategies represent a roadmap for using the deep insights of other fields for transformative advances in regenerative medicine and systems bioengineering. © 2016 The Author(s) Published by the Royal Society. All rights reserved.
rsif.royalsocietypublishing.org
Perspective
Cite this article: Pezzulo G, Levin M. 2016
Top-down models in biology: explanation and
control of complex living systems above the
molecular level. J. R. Soc. Interface 13:
20160555.
http://dx.doi.org/10.1098/rsif.2016.0555
Received: 12 July 2016
Accepted: 11 October 2016
Subject Category:
Life SciencesEngineering interface
Subject Areas:
systems biology, synthetic biology, biophysics
Keywords:
top-down, integrative, cognitive modelling,
developmental biology, regeneration,
remodelling
Author for correspondence:
Michael Levin
e-mail: michael.levin@tufts.edu
Top-down models in biology: explanation
and control of complex living systems
above the molecular level
Giovanni Pezzulo2and Michael Levin1
1
Biology Department, Allen Discovery Center at Tufts, Tufts University, Medford, MA 02155, USA
2
Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
GP, 0000-0001-6813-8282; ML, 0000-0001-7292-8084
It is widely assumed in developmental biology and bioengineering that opti-
mal understanding and control of complex living systems follows from
models of molecular events. The success of reductionism has overshadowed
attempts at top-down models and control policies in biological systems.
However, other fields, including physics, engineering and neuroscience,
have successfully used the explanations and models at higher levels
of organization, including least-action principles in physics and control-
theoretic models in computational neuroscience. Exploiting the dynamic
regulation of pattern formation in embryogenesis and regeneration requires
new approaches to understand how cells cooperate towards large-scale
anatomical goal states. Here, we argue that top-down models of pattern
homeostasis serve as proof of principle for extending the current paradigm
beyond emergence and molecule-level rules. We define top-down control
in a biological context, discuss the examples of how cognitive neuroscience
and physics exploit these strategies, and illustrate areas in which they may
offer significant advantages as complements to the mainstream paradigm.
By targeting system controls at multiple levels of organization and demysti-
fying goal-directed (cybernetic) processes, top-down strategies represent a
roadmap for using the deep insights of other fields for transformative
advances in regenerative medicine and systems bioengineering.
1. Introduction
If you want to build a ship, don’t herd people together to collect wood, and don’t
assign them tasks and work, but teach them to long for the endless immensity of
the sea.
—Antoine de Saint-Exupery, ‘Wisdom of the Sands’.
1.1. Levels of explanation: the example of pattern regulation
Most biological phenomena are complex—they depend on the interplay of many
factors and show adaptive self-organization under selection pressure [1]. One of
the most salient examples is the regulation of body anatomy. A single fertilized
egg gives rise to a cell mass that reliably self-assembles into the complex three-
dimensional structure of a body. Crucially, however, bioscience needs to understand
more than the feedforward progressive emergence of a stereotypical pattern. Some
animals have the remarkable ability to compensate for huge external perturbations
during embryogenesis, and as adults can regenerate amputated limbs or heads,
remodel whole organs into other organs if grafted to ectopic locations (figure 1a),
and reprogramme-induced tumours into normal structures (reviewed in [4,5]).
These capabilities reveal that biological structures implement closed-loop controls
that pursue shape homeostasis at many levels, from individual cells to the entire
body plan.
Biologists work towards two main goals: understanding the system to make
predictions and inferring manipulations that lead to desired changes. The
former is the province of developmental and evolutionary biology, whereas
&2016 The Author(s) Published by the Royal Society. All rights reserved.
... While 'white box' approaches are transparent, 'black box' ones are not. These two varieties also differ in their sequences: 'white box' approaches are bottom-up (they start identifying the variables to be investigated), while 'black box' perspectives are top-down (they are data-driven, lacking a preestablished theory on what they may find, why or how (35). Hence, a conundrum: while 'white box' approaches may fail to inform more or better, 'black box' alternatives may be invalid (36). ...
... It is also designed to capture personalized dimensions (19) which, to be unambiguously detected, require comparisons against populations. It addresses the bottom-up−top-down controversy as well as the 'white box'/ 'black box' AI problem (35,36). Because it recognizes patterns after data collection, it prevents the bias and self-limited consequences of approaches that define research questions before data collection. ...
Article
Full-text available
Introduction Data overlapping of different biological conditions prevents personalized medical decision-making. For example, when the neutrophil percentages of surviving septic patients overlap with those of non-survivors, no individualized assessment is possible. To ameliorate this problem, an immunological method was explored in the context of sepsis. Methods Blood leukocyte counts and relative percentages as well as the serum concentration of several proteins were investigated with 4072 longitudinal samples collected from 331 hospitalized patients classified as septic (n=286), non-septic (n=43), or not assigned (n=2). Two methodological approaches were evaluated: (i) a reductionist alternative, which analyzed variables in isolation; and (ii) a non-reductionist version, which examined interactions among six (leukocyte-, bacterial-, temporal-, personalized-, population-, and outcome-related) dimensions. Results The reductionist approach did not distinguish outcomes: the leukocyte and serum protein data of survivors and non-survivors overlapped. In contrast, the non-reductionist alternative differentiated several data groups, of which at least one was only composed of survivors (a finding observable since hospitalization day 1). Hence, the non-reductionist approach promoted personalized medical practices: every patient classified within a subset associated with 100% survival subset was likely to survive. The non-reductionist method also revealed five inflammatory or disease-related stages (provisionally named ‘early inflammation, early immunocompetence, intermediary immuno-suppression, late immuno-suppression, or other’). Mortality data validated these labels: both ‘suppression’ subsets revealed 100% mortality, the ‘immunocompetence’ group exhibited 100% survival, while the remaining sets reported two-digit mortality percentages. While the ‘intermediary’ suppression expressed an impaired monocyte-related function, the ‘late’ suppression displayed renal-related dysfunctions, as indicated by high concentrations of urea and creatinine. Discussion The data-driven differentiation of five data groups may foster early and non-overlapping biomedical decision-making, both upon admission and throughout their hospitalization. This approach could evaluate therapies, at personalized level, earlier. To ascertain repeatability and investigate the dynamics of the ‘other’ group, additional studies are recommended.
... Specifically, it is increasingly realized that the level of competency (and thus the appropriate level of control) often cannot be guessed by inspection of a system's components, and that its position on a spectrum ranging from passive matter to complex metacognition must be determined empirically (Rosenblueth et al., 1943;Levin, 2021Levin, 2022), . This is critical not only for fundamental understanding of evolution of bodies and minds (Lyon, 2006; Barandiaran and Moreno, 2006;Müller and Lengeler, 2000;Baluška and Levin, 2016;McGivern, 2020;Levin, 2023b), but also for the design of interventions in biomedicine and synthetic morphology contexts (Pezzulo and Levin, 2015;Pezzulo and Levin, 2016). Yet, a common property in many of these systems is that it is expensive in time and energy to conduct experiments: empirical exploration needs to be made under limited resources. ...
Article
Full-text available
Many applications in biomedicine and synthetic bioengineering rely on understanding, mapping, predicting, and controlling the complex behavior of chemical and genetic networks. The emerging field of diverse intelligence investigates the problem-solving capacities of unconventional agents. However, few quantitative tools exist for exploring the competencies of non-conventional systems. Here, we view gene regulatory networks (GRNs) as agents navigating a problem space and develop automated tools to map the robust goal states GRNs can reach despite perturbations. Our contributions include: (1) Adapting curiosity-driven exploration algorithms from AI to discover the range of reachable goal states of GRNs, and (2) Proposing empirical tests inspired by behaviorist approaches to assess their navigation competencies. Our data shows that models inferred from biological data can reach a wide spectrum of steady states, exhibiting various competencies in physiological network dynamics without requiring structural changes in network properties or connectivity. We also explore the applicability of these ‘behavioral catalogs’ for comparing evolved competencies across biological networks, for designing drug interventions in biomedical contexts and synthetic gene networks for bioengineering. These tools and the emphasis on behavior-shaping open new paths for efficiently exploring the complex behavior of biological networks. For the interactive version of this paper, please visit https://developmentalsystems.org/curious-exploration-of-grn-competencies .
... In systems biology, master regulators such as the Hox gene family orchestrate numerous complex developmental processes [5]. The ubiquity of top-down control may arise from robustness and parsimony, because it reduces the number of dynamical interactions needed to steer a high-dimensional dynamical system [6]. However, top-down control introduces experimental challenges when the driving variable is not known a priori, or is not directly measurable. ...
Article
Full-text available
Unmeasured causal forces influence diverse experimental time series, such as the transcription factors that regulate genes or the descending neurons that steer motor circuits. Combining the theory of skew-product dynamical systems with topological data analysis, we show that simultaneous recurrence events across multiple time series reveal the structure of their shared unobserved driving signal. We introduce a physics-based unsupervised learning algorithm that reconstructs causal drivers by iteratively building a recurrence graph with glasslike structure. As the amount of data increases, a percolation transition on this graph leads to weak ergodicity breaking for random walks—revealing the shared driver’s dynamics, even from strongly corrupted measurements. We relate reconstruction accuracy to the rate of information transfer from a chaotic driver to the response systems, and we find that effective reconstruction proceeds through gradual approximation of the driver’s dynamical attractor. Through extensive benchmarks against classical signal processing and machine learning techniques, we demonstrate our method’s ability to extract causal drivers from diverse experimental datasets spanning ecology, genomics, fluid dynamics, and physiology. Published by the American Physical Society 2025
... This conclusion is consistent with acknowledging that cellular memory can be fruitfully modelled 'as if' it was a cognitive process (cf. Pezzulo and Levin, 2016). However, due to the evolutionary path that somatic cells have followed, favouring the suppression of their metabolic independence in favour of the autonomy of the larger metabolic entity they constitute (such as the organism), treating them as cognitive agents, if I am correct, bottoms out as a useful fiction (e.g., helping to identify and highlight similar underlying mechanisms in use). ...
Book
Physarum polycephalum, also known more colloquially as 'the blob', 'acellular slime mould', or just 'slime mould', is a unicellular multinucleate protist that has continued to attract the interest of biologists over the past century because of its complex life cycle, unique physiology, morphology, and behaviour. More recently, attention has shifted to Physarum as a model organism for investigating putative cognitive capacities such as decision making, learning, and memory in organisms without nervous systems. The aim of this Element is to illustrate how Physarum can be used as a valuable tool for approaching various topics in the philosophy of biology. Physarum and its behaviour not only pose a challenge to some of the received views of biological processes but also, I shall argue, provide an opportunity to clarify and appropriately sharpen the concepts underlying such received views.
... The body-wide subthreshold membrane potential circuits play an essential role in both the development and maintenance of the overall body structure. They have been suggested as a possible "top-down" mechanism that organisms employ to coordinate their many parts [39,40]. ...
Article
Full-text available
An important part of studying living systems is figuring out the complicated steps that lead to order from chaos. Spontaneous oscillations are a key part of self-organisation in many biological and chemical networks, including kombucha and proteinoids. This study examines the spontaneous oscillations in kombucha and proteinoids, specifically exploring their potential connection to the origin of life. As a community of bacteria and yeast work together, kombucha shows remarkable spontaneous oscillations in its biochemical parts. This system can keep a dynamic balance and organise itself thanks to metabolic processes and complex chemical reactions. Similarly, proteinoids, which may have been primitive forms of proteins, undergo spontaneous fluctuations in their structure and function periodically. Because these oscillations happen on their own, they may play a very important part in the development of early life forms. This paper highlights the fundamental principles governing the transition from chaos to order in living systems by examining the key factors that influence the frequency and characteristics of spontaneous oscillations in kombucha and proteinoids. Looking into these rhythms not only helps us understand where life came from but also shows us ways to make self-organising networks in synthetic biology and biotechnology. There is significant discussion over the emergence of biological order from chemical disorder. This article contributes to the ongoing discussion by examining at the theoretical basis, experimental proof, and implications of spontaneous oscillations. The results make it clear that random oscillations are an important part of the change from nonliving to living matter. They also give us important information about what life is all about.
... These tools have shed light on the intricate virus-host interactions, revealing novel insights into how viruses manipulate host cellular machinery to establish infection, evade plant immune responses, and spread within the host. The molecular underpinnings of these processes is critical for the development of effective control strategies (Pezzulo and Levin, 2016). This review aims to consolidate recent findings in the field, emphasizing breakthroughs in our viral pathogenesis. ...
Article
Plant viral diseases pose a significant threat to global agriculture, leading to substantial yield losses and affecting food security. Advances in viral pathogenesis and control have been driven by a combination of traditional and modern approaches. This review highlights key areas of progress, including virus-host interactions, molecular mechanisms of viral replication, and host immune responses. Traditional control methods, such as crop rotation, sanitation, and vector management, remain foundational but face limitations due to the evolving nature of plant viruses. Breeding for genetic resistance, while effective, is challenged by the rapid adaptation of viral pathogens. The emergence of biotechnological strategies, such as RNA interference (RNAi), CRISPR/Cas systems, and the development of transgenic plants, has provided novel tools for enhancing resistance. Furthermore, molecular diagnostics, including PCR and next-generation sequencing (NGS), have revolutionized virus detection, enabling precise and early diagnosis. The integration of omics approaches-genomics, transcriptomics, proteomics, and metabolomics-has facilitated a plant-virus interactions, while synthetic biology and systems biology are opening new frontiers in engineering virus-resistant crops. Climate change exacerbates the challenge by altering virus spread, vector dynamics, and host susceptibility, necessitating adaptive strategies. Emerging and re-emerging plant viruses underscore the need for robust surveillance and biosecurity measures, emphasizing the role of international collaboration in controlling these threats.
... Computational techniques such as novelty search [Lehman and Stanley, 2011] and quality-diversity algorithms [Pugh et al., 2016] have made progress in addressing these questions, though gaps in our understanding persist. Recent efforts have started drawing connections between evolutionary processes and broader concepts of intelligence [Mitchell and Cheney, 2024, Levin, 2023b, Pezzulo and Levin, 2016 and learning theory [Kouvaris et al., 2017, Watson and Szathmáry, 2016, Power et al., 2015, Watson et al., 2014. ...
Preprint
Full-text available
Diffusion Models represent a significant advancement in generative modeling, employing a dual-phase process that first degrades domain-specific information via Gaussian noise and restores it through a trainable model. This framework enables pure noise-to-data generation and modular reconstruction of, images or videos. Concurrently, evolutionary algorithms employ optimization methods inspired by biological principles to refine sets of numerical parameters encoding potential solutions to rugged objective functions. Our research reveals a fundamental connection between diffusion models and evolutionary algorithms through their shared underlying generative mechanisms: both methods generate high-quality samples via iterative refinement on random initial distributions. By employing deep learning-based diffusion models as generative models across diverse evolutionary tasks and iteratively refining diffusion models with heuristically acquired databases, we can iteratively sample potentially better-adapted offspring parameters, integrating them into successive generations of the diffusion model. This approach achieves efficient convergence toward high-fitness parameters while maintaining explorative diversity. Diffusion models introduce enhanced memory capabilities into evolutionary algorithms, retaining historical information across generations and leveraging subtle data correlations to generate refined samples. We elevate evolutionary algorithms from procedures with shallow heuristics to frameworks with deep memory. By deploying classifier-free guidance for conditional sampling at the parameter level, we achieve precise control over evolutionary search dynamics to further specific genotypical, phenotypical, or population-wide traits. Our framework marks a major heuristic and algorithmic transition, offering increased flexibility, precision, and control in evolutionary optimization processes.
Article
The Diverse Intelligence research seeks to understand commonalities in behavioral competencies across a wide range of implementations. Especially interesting are simple systems that provide unexpected examples of memory, decision-making, or problem-solving in substrates that at first glance do not appear to be complex enough to implement such capabilities. We seek to develop tools to determine minimal requirements for such capabilities, and to learn to recognize and predict basal forms of intelligence in unconventional substrates. Here, we apply novel analyses to the behavior of classical sorting algorithms—short pieces of code studied for many decades. To study these sorting algorithms as a model of biological morphogenesis and its competencies, we break two formerly ubiquitous assumptions: top-down control (instead, each element within an array of numbers can exert minimal agency and implement sorting policies from the bottom up), and fully reliable hardware (instead, allowing elements to be “damaged” and fail to execute the algorithm). We quantitatively characterize sorting activity as traversal of a problem space, showing that arrays of autonomous elements sort themselves more reliably and robustly than traditional implementations in the presence of errors. Moreover, we find the ability to temporarily reduce progress in order to navigate around a defect, and unexpected clustering behavior among elements in chimeric arrays consisting of two different algorithms. The discovery of emergent problem-solving capacities in simple, familiar algorithms contributes a new perspective showing how basal forms of intelligence can emerge in simple systems without being explicitly encoded in their underlying mechanics.
Preprint
Full-text available
Understanding how populations of cells collectively coordinate activity to produce the complex structures and behaviors that characterize multicellular organisms is a fundamental issue in modern biology. Here we show how mathematical techniques from complex systems science and multivariate information theory can provide a rigorous framework for inferring the structure of collective organization in non-neural tissue. Many of these techniques were developed and refined in the context of theoretical neuroscience, a field well-used to the problem of inferring coordinated activity in high-dimensional data. In neuroscience, these statistics (functional connectivity network structure, modularity, higher-order information, etc) have been found to be altered during different cognitive, clinical, or behavioral states and are generally thought to be informative about the underlying dynamics linking biology to cognition. Here we show that these same patterns of coordinated activity are also present in the aneural tissues of evolutionarily distant biological systems: preparations of self-motile embryonic Xenopus tissue (colloquially known as "basal Xenobots". When analyzing calcium recordings from basal Xenobots and comparing them to fMRI recordings from a sample of adult human brains, we find that the bots have a "brain-like functional information architecture, complete with positive and negative functional connections, meso-scale communities, higher-order redundant and synergistic interactions, and integrated information that is "greater than the sum of its parts". By comparing each recording (brain and bot) to a personalized null model that preserves all first-order statistical structures (autocorrelation, frequency spectrum, etc. ) while disrupting all higher-order interactions, we show that these are genuine higher order interactions and not trivially reducible to lower-order features of the data. These similarities suggest that such patterns of activity and information structures either: arose independently in these two systems epithelial constructs and brains, are epiphenomenological byproducts of other dynamics conserved across vastly different configurations of life; or somehow directly support adaptive behavior across diverse living systems.
Article
The dominant paradigm in biomedicine focuses on genetically‐specified components of cells and their biochemical dynamics, emphasizing bottom‐up emergence of complexity. Here, I explore the biomedical implications of a complementary emerging field: diverse intelligence. Using tools from behavioral science and multiscale neuroscience, we can study development, regenerative repair, and cancer suppression as behaviors of a collective intelligence of cells navigating the spaces of possible morphologies and transcriptional and physiological states. A focus on the competencies of living material—from molecular to organismal scales—reveals a new landscape for interventions. Such top‐down approaches take advantage of the memories and homeodynamic goal‐seeking behavior of cells and tissues, offering the same massive advantages in biomedicine and bioengineering that reprogrammable hardware has provided information technologies. The bioelectric networks that bind individual cells toward large‐scale anatomical goals are an especially tractable interface to organ‐level plasticity, and tools to modulate them already exist. This suggests a research program to understand and tame the software of life for therapeutic gain by understanding the many examples of basal cognition that operate throughout living bodies.
Article
Full-text available
The central nervous system (CNS) underlies memory, perception, decision-making, and behavior in numerous organisms. However, neural networks have no monopoly on the signaling functions that implement these remarkable algorithms. It is often forgotten that neurons optimized cellular signaling modes that existed long before the CNS appeared during evolution, and were used by somatic cellular networks to orchestrate physiology, embryonic development, and behavior. Many of the key dynamics that enable information processing can, in fact, be implemented by different biological hardware. This is widely exploited by organisms throughout the tree of life. Here, we review data on memory, learning, and other aspects of cognition in a range of models, including single celled organisms, plants, and tissues in animal bodies. We discuss current knowledge of the molecular mechanisms at work in these systems, and suggest several hypotheses for future investigation. The study of cognitive processes implemented in aneural contexts is a fascinating, highly interdisciplinary topic that has many implications for evolution, cell biology, regenerative medicine, computer science, and synthetic bioengineering.
Article
Living systems exhibit remarkable abilities to self-assemble, regenerate, and remodel complex shapes. How cellular networks construct and repair specific anatomical outcomes is an open question at the heart of the next-generation science of bioengineering. Developmental bioelectricity is an exciting emerging discipline that exploits endogenous bioelectric signaling among many cell types to regulate pattern formation. We provide a brief overview of this field, review recent data in which bioelectricity is used to control patterning in a range of model systems, and describe the molecular tools being used to probe the role of bioelectrics in the dynamic control of complex anatomy. We suggest that quantitative strategies recently developed to infer semantic content and information processing from ionic activity in the brain might provide important clues to cracking the bioelectric code. Gaining control of the mechanisms by which large-scale shape is regulated in vivo will drive transformative advances in bioengineering, regenerative medicine, and synthetic morphology, and could be used to therapeutically address birth defects, traumatic injury, and cancer.
Article
Balancing habitual and deliberate forms of choice entails a comparison of their respective merits-the former being faster but inflexible, and the latter slower but more versatile. Here, we show that arbitration between these two forms of control can be derived from first principles within an Active Inference scheme. We illustrate our arguments with simulations that reproduce rodent spatial decisions in T-mazes. In this context, deliberation has been associated with vicarious trial and error (VTE) behavior (i.e., the fact that rodents sometimes stop at decision points as if deliberating between choice alternatives), whose neurophysiological correlates are "forward sweeps" of hippocampal place cells in the arms of the maze under consideration. Crucially, forward sweeps arise early in learning and disappear shortly after, marking a transition from deliberative to habitual choice. Our simulations show that this transition emerges as the optimal solution to the trade-off between policies that maximize reward or extrinsic value (habitual policies) and those that also consider the epistemic value of exploratory behavior (deliberative or epistemic policies)-the latter requiring VTE and the retrieval of episodic information via forward sweeps. We thus offer a novel perspective on the optimality principles that engender forward sweeps and VTE, and on their role on deliberate choice.
Article
A key problem in evolutionary developmental biology is identifying the sources of instructive information that determine species-specific anatomical pattern. Understanding the inputs to large-scale morphology is also crucial for efforts to manipulate pattern formation in regenerative medicine and synthetic bioengineering. Recent studies have revealed a physiological system of communication among cells that regulates pattern during embryogenesis and regeneration in vertebrate and invertebrate models. Somatic tissues form networks using the same ion channels, electrical synapses, and neurotransmitter mechanisms exploited by the brain for information-processing. Experimental manipulation of these circuits was recently shown to override genome default patterning outcomes, resulting in head shapes resembling those of other species in planaria and Xenopus. The ability to drastically alter macroscopic anatomy to that of other extant species, despite a wild-type genomic sequence, suggests exciting new approaches to the understanding and control of patterning. Here, we review these results and discuss hypotheses regarding non-genomic systems of instructive information that determine biological growth and form.
Article
This paper offers an active inference account of choice behaviour and learning. It focuses on the distinction between goal-directed and habitual behaviour and how they contextualise each other. We show that habits emerge naturally (and autodidactically) from sequential policy optimisation when agents are equipped with state-action policies. In active inference, behaviour has explorative (epistemic) and exploitative (pragmatic) aspects that are sensitive to ambiguity and risk respectively, where epistemic (ambiguity-resolving) behaviour enables pragmatic (reward-seeking) behaviour and the subsequent emergence of habits. Although goal-directed and habitual policies are usually associated with model-based and model-free schemes, we find the more important distinction is between belief-free and belief-based schemes. The underlying (variational) belief updating provides a comprehensive (if metaphorical) process theory for several phenomena, including the transfer of dopamine responses, reversal learning, habit formation and devaluation. Finally, we show that active inference reduces to a classical (Bellman) scheme, in the absence of ambiguity.