Adaptive Behavior

The Iterated Classification Game (ICG) combines the Classification Game with the Iterated Learning Model (ILM) to create a more realistic model of the cultural transmission of language through generations. It includes both learning from parents and learning from peers. Further, it eliminates some of the chief criticisms of the ILM: that it does not study grounded languages, that it does not include peer learning, and that it builds in a bias for compositional languages. We show that, over the span of a few generations, a stable linguistic system emerges that can be acquired very quickly by each generation, is compositional, and helps the agents to solve the classification problem with which they are faced. The ICG also leads to a different interpretation of the language acquisition process. It suggests that the role of parents is to initialize the linguistic system of the child in such a way that subsequent interaction with peers results in rapid convergence to the correct language.

The Dyna class of reinforcement learning architectures enables the creation of integrated learning, planning and reacting systems. A class of strategies designed to enhance the learning and planning power of Dyna systems by increasing their computational efficiency is examined. The benefit of using these strategies is demonstrated on some simple abstract learning tasks. It is proposed that the backups to be performed in Dyna be prioritized in order to improve its efficiency. It is demonstrated with simple tasks that use some specific prioritizing schemes can lead to significant reductions in computational effort and corresponding improvements in learning performance

This article describes a biomimetic control architecture affording an animat both action selection and navigation functionalities. It satisfies the survival constraint of an artificial metabolism and supports several complementary navigation strategies. It builds upon an action selection model based on the basal ganglia of the vertebrate brain, using two interconnected cortico-basal ganglia-thalamo-cortical loops: a ventral one concerned with appetitive actions and a dorsal one dedicated to consummatory actions. The performances of the resulting model are evaluated in simulation. The experiments assess the prolonged survival permitted by the use of high level navigation strategies and the complementarity of navigation strategies in dynamic environments. The correctness of the behavioral choices in situations of antagonistic or synergetic internal states are also tested. Finally, the modelling choices are discussed with regard to their biomimetic plausibility, while the experimental results are estimated in terms of animat adaptivity.

This paper develops generalizations of empowerment to continuous states. Empowerment is a recently introduced information-theoretic quantity motivated by hypotheses about the efficiency of the sensorimotor loop in biological organisms, but also from considerations stemming from curiosity-driven learning. Empowemerment measures, for agent-environment systems with stochastic transitions, how much influence an agent has on its environment, but only that influence that can be sensed by the agent sensors. It is an information-theoretic generalization of joint controllability (influence on environment) and observability (measurement by sensors) of the environment by the agent, both controllability and observability being usually defined in control theory as the dimensionality of the control/observation spaces. Earlier work has shown that empowerment has various interesting and relevant properties, e.g., it allows us to identify salient states using only the dynamics, and it can act as intrinsic reward without requiring an external reward. However, in this previous work empowerment was limited to the case of small-scale and discrete domains and furthermore state transition probabilities were assumed to be known. The goal of this paper is to extend empowerment to the significantly more important and relevant case of continuous vector-valued state spaces and initially unknown state transition probabilities. The continuous state space is addressed by Monte-Carlo approximation; the unknown transitions are addressed by model learning and prediction for which we apply Gaussian processes regression with iterated forecasting. In a number of well-known continuous control tasks we examine the dynamics induced by empowerment and include an application to exploration and online model learning.

This work presents a novel learning method in the context of embodied artificial intelligence and self-organization, which has as few assumptions and restrictions as possible about the world and the underlying model. The learning rule is derived from the principle of maximizing the predictive information in the sensorimotor loop. It is evaluated on robot chains of varying length with individually controlled, non-communicating segments. The comparison of the results shows that maximizing the predictive information per wheel leads to a higher coordinated behavior of the physically connected robots compared to a maximization per robot. Another focus of this paper is the analysis of the effect of the robot chain length on the overall behavior of the robots. It will be shown that longer chains with less capable controllers outperform those of shorter length and more complex controllers. The reason is found and discussed in the information-geometric interpretation of the learning process.

In this study, we investigate the adaptation and robustness of a packet switching network (PSN), the fundamental architecture of the Internet. We claim that the adaptation introduced by a transmission control protocol (TCP) congestion control mechanism is interpretable as the self-organization of multiple attractors and stability to switch from one attractor to another. To discuss this argument quantitatively, we study the adaptation of the Internet by simulating a PSN using ns-2. Our hypothesis is that the robustness and fragility of the Internet can be attributed to the inherent dynamics of the PSN feedback mechanism called the congestion window size, or \textit{cwnd}. By varying the data input into the PSN system, we investigate the possible self-organization of attractors in cwnd temporal dynamics and discuss the adaptability and robustness of PSNs. The present study provides an example of Ashby's Law of Requisite Variety in action.

The term "nexting" has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to "next" constitutes a basic kind of awareness and knowledge of one's environment. In this paper we present results with a robot that learns to next in real time, predicting thousands of features of the world's state, including all sensory inputs, at timescales from 0.1 to 8 seconds. This was achieved by treating each state feature as a reward-like target and applying temporal-difference methods to learn a corresponding value function with a discount rate corresponding to the timescale. We show that two thousand predictions, each dependent on six thousand state features, can be learned and updated online at better than 10Hz on a laptop computer, using the standard TD(lambda) algorithm with linear function approximation. We show that this approach is efficient enough to be practical, with most of the learning complete within 30 minutes. We also show that a single tile-coded feature representation suffices to accurately predict many different signals at a significant range of timescales. Finally, we show that the accuracy of our learned predictions compares favorably with the optimal off-line solution.

Language is often regarded as the hallmark for human intelligence. So, what design features make humans' linguistic behavior so special? First, human language is largely symbolic, which means that the communicative signals have either an arbitrary relationship to their meaning or reference, or this relationship is conventionalized (Peirce, 1931-58). As a result, the relationship between signal and reference must be learnt. Second, the number of words that make up a typical language is extremely large. There have been estimates that humans by the age of 18 have acquired approximately 60,000 words (Anglin, 1993). Third, the human vocal apparatus and auditory system allow us to produce and distinguish many different sounds, which we can combine in a controlled fashion to make even more distinctive vocalizations. Fourth, human language is an open system, so we can easily invent new words and communicate about previously unseen objects or events, thus allowing for language to grow and change. Finally, language has a complex grammar, which allows us to combine words in a different order and inflect words to give utterances different meanings. In effect, this allows humans to produce an infinite number of utterances given a finite number of means(Chomsky, 1956). This special issue includes computation studies which have provided major insight into the underlying principles of adaptive systems--biological evolution, individual learning, and cultural evolution--that interact with each other to account for language evolution. (PsycINFO Database Record (c) 2012 APA, all rights reserved)

In order to study learning as an adaptive process it is necessary to take into consideration the role of evolution which is the primary adaptive process. In addition, learning should be studied in (artificial) organisms that live in an independent physical environment in such a way that the input from the environment can be at least partially controlled by the organisms' behavior. To explore these issues we used a genetic algorithm to simulate the evolution of a population of neural networks each controlling the behavior of a small mobile robot that must explore efficiently an environment surrounded by walls. Since the environment changes from one generation to the next each network must learn during its life to adapt to the particular environment it happens to be born in. We found that evolved networks incorporate a genetically inherited predisposition to learn that can be described as: (a) the presence of initial conditions that tend to canalize learning in the right direc...

This paper concerns the relationship between the detectable and useful structure in an environment and the degree to which a population can adapt to that environment. We explore the hypothesis that adaptability will depend unimodally on environmental variety, and we measure this component of environmental structure using the information-theoretic uncertainty (Shannon entropy) of detectable environmental conditions. We define adaptability as the degree to which a certain kind of population successfully adapts to a certain kind of environment, and we measure adaptability by comparing a population's size to the size of a non-adapting, but otherwise comparable, population in the same environment. We study the relationship between adaptability and environmental structure in an evolving artificial population of sensorimotor agents that live, reproduce, and die in a variety of environments. We find that adaptability does not show a unimodal dependence on environmental variety alone...

We present an approach to support effective learning and adaptation of behaviors for autonomous agents with reinforcement learning algorithms. These methods can identify control systems that optimize a reinforcement program, which is, usually, a straightforward representation of the designer's goals. Reinforcement learning algorithms usually are too slow to be applied in real time on embodied agents, although they provide a suitable way to represent the desired behavior. We have tackled three aspects of this problem: the speed of the algorithm, the learning procedure, and the control system architecture. The learning algorithm we have developed includes features to speed up learning, such as niche-based learning, and a representation of the control modules in terms of fuzzy rules that reduces the search space and improves robustness to noisy data. Our learning procedure exploits methodologies such as learning from easy missions and transfer of policy from simpler environments to the more complex. The architecture of our control system is layered and modular, so that each module has a low complexity and can be learned in a short time. The composition of the actions proposed by the modules is either learned or predefined. Finally, we adopt an anytime learning approach to improve the quality of the control system on-line and to adapt it to dynamic environments. The experiments we present in this article concern learning to reach another moving agent in a real, dynamic environment that includes nontrivial situations such as that in which the moving target is faster than the agent and that in which the target is hidden by obstacles.

This paper proposes the concept of basis behaviors as ubiquitous general building blocks for synthesizing artificial group behavior in multi--agent systems, and for analyzing group behavior in nature. We demonstrate the concept through examples implemented both in simulation and on a group of physical mobile robots. The basis behavior set we propose, consisting of avoidance, safe--wandering, following, aggregation, dispersion, and homing, is constructed from behaviors commonly observed in a variety of species in nature. The proposed behaviors are manifested spatially, but have an effect on more abstract modes of interaction, including the exchange of information and cooperation. We demonstrate how basis behaviors can be combined into higher--level group behaviors commonly observed across species. The combination mechanisms we propose are useful for synthesizing a variety of new group behaviors, as well as for analyzing naturally occurring ones. Key words: group behavior, robotics, eth...

Introduction Psychologists have long paid lip service to Darwin, conceding that the human brain did arise through the course of evolution (for whatever, often unspecified, reason). But the full power of Darwinian theory is almost never used in day-to-day psychology research. This is peculiar, given the successful, integrated nature of evolutionary biology, and the typically fragmented and incomplete visage of modern psychology: one would think that a theory that explains the origin and maintenance of complex behavioral adaptations across all species (evolution) could inform and unify the study of human behavior (psychology) just as productively as it does the study of animal behavior (ethology and comparative cognition). But the emergence of a genuinely evolutionary psychology of humans (HEP) has been a slow, painful, and quite recent occurrence, marked at last by the publication of a flagship volume, The Adapted Mind. This work is of great importance not only for researchers in all br

The paper presents a neural network architecture (MAXSON) based on second-order connections that can learn a multiple goal approach/avoid task using reinforcement from the environment. It also enables an agent to learn vicariously, from the successes and failures of other agents. The paper shows that MAXSON can learn certain spatial navigation tasks much faster than traditional Q-learning, as well as learn goal directed behavior, increasing the agent's chances of long-term survival. The paper shows that an extension of MAXSON (V-MAXSON) enables agents to learn vicariously, and this improves the overall survivability of the agent population.

The study of adaptive behavior, including learning, usually centers on the effects of natural selection for individual survival. But because reproduction is evolutionarily more important than survival, sexual selection through mate choice (Darwin, 1871), can also have profound consequences on the evolution of creatures' bodies and behaviors. This paper shows through simulation models how one type of learning, parental imprinting, can evolve purely through sexual selection, to help in selecting appropriate mates and in tracking changes in the phenotypic makeup of the population across generations. At moderate mutation rates, when populationtracking becomes an important but still soluble problem, imprinting proves more useful and evolves more quickly than at low or high mutation rates. We also show that parental imprinting can facilitate the formation of new species. In reviewing the biological literature on imprinting, we note that these results confirm some previous s...

Monitoring is an important activity for anyembedded agent. To operate effectively, agents must gather information about their environment. The policy by whichtheydo this is called a monitoring strategy.Ourwork has focussed on classifying differenttypes of monitoring strategies, and understanding how strategies depend on features of the task and environment. Wehave discovered only a few general monitoring strategies, in particular periodic and interval reduction, and speculate that there are no more.

This paper describes a novel method of achieving load balancing in telecommunications networks. A simulated network models a typical distribution of calls between nodes; nodes carrying an excess of traffic can become congested, causing calls to be lost. In addition to calls, the network also supports a population of simple mobile agents with behaviours modelled on the trail laying abilities of ants. The ants move across the network between randomly chosen pairs of nodes; as they move they deposit simulated pheromones as a function of their distance from their source node, and the congestion encountered on their journey. They select their path at each intermediate node according the distribution of simulated pheromones at each node. Calls between nodes are routed as a function of the pheromone distributions at each intermediate node. The performance of the network is measured by the proportion of calls which are lost. The results of using the ant-based control (ABC) are compa...

The traditional explanation of delayed maturation age, as part of an evolved life history, focuses on the increased costs of juvenile mortality due to early maturation. Prior quantitative models of these trade-offs, however, have addressed only morphological phenotypic traits, such as body size. We argue that the development of behavioral skills prior to reproductive maturity also constitutes an advantage of delayed maturation and thus should be included among the factors determining the trade-off for optimal age at maturity. Empirical support for this hypothesis from animal field studies is abundant. This paper provides further evidence drawn from simulation experiments. "Latent Energy Environments" (LEE) are a class of tightly controlled environments in which learning organisms are modeled by neural networks and evolved according to a type of genetic algorithm. An advantage of this artificial world is that it becomes possible to discount all non-behavioral costs of early maturity in ...

Complete physically embodied agents present a powerful medium for the investigation of cognitive models for spatial navigation. This paper presents a maze solving robot, called a micromouse, that parallels many of the behaviours found in its biological counterpart, the rat. A cognitive model of the robot is presented and its limits investigated. Limits are found to exist with respect to biological plausibility and robot applicability. It is proposed that the fundamental representations used to store and process information are the limiting factor. A review of the literature of current cognitive models finds a lack of models suitable for implementation in real agents, and proposes that these models fail as they have not been developed with real agents in mind. A solution to this conundrum is proposed in a list of guidelines for the development of future spatial models. 1 Introduction This paper presents a complete physically embodied agent for a complex spatial navigation task; namely ...

This paper describes how an animat endowed with the MonaLysa control architecture can build a cognitive map that merges into a hierarchical framework not only topological links between landmarks, but also higher-level structures, control information, and metric distances and orientations. The paper also describes how the animat can use such a map to locate itself, even if it is endowed with noisy dead-reckoning capacities. MonaLysa's mapping and self-positioning capacities are illustrated by results obtained in three different environments and four noise-level conditions. These capacities appear to be gracefully degraded when the environment grows more challenging and when the noise level increases. In the discussion, the current approach is compared to others with similar objectives, and directions for future work are outlined. Keywords Hierarchical map. Topological information. Metric information. Landmarks. Self-positioning. Dead-reckoning. Robustness to noise. 1 Introdu...

A constructivist approach is applied to characterising social embeddedness. Social embeddedness is intended as a strong type of social situatedness. It is defined as the extent to which modelling the behaviour of an agent requires the inclusion of other agents as individuals rather than as an undifferentiated whole. Possible consequences of the presence of social embedding and ways to check for it are discussed. A model of co-developing agents is exhibited which demonstrates the possibility of social embedding. This is an extension of Brian Arthur's `El Farol Bar' model, with added learning and communication. Some indicators of social embedding are analysed and some possible causes of social embedding are discussed. It is suggested that social embeddedness may be an explanation of the causal link between the social situatedness of the agent and it employing a constructivist strategy in its modelling.

This article examines the relationship between environmental and cognitive structure. One of the key tasks for any agent interacting in the real world is the management of uncertainty; because of this the cognitive structures which interact with real environments, such as would be used in navigation, must effectively cope with the uncertainty inherent in a constantly changing world. Despite this uncertainty, however, real environments usually afford structure that can be effectively exploited by organisms. The article examines environmental characteristics and structures that enable humans to survive and thrive in a wide range of real environments. The relationship between these characteristics and structures, uncertainty, and cognitive structure is explored in the context of PLAN, a proposed model of human cognitive mapping, and R-PLAN, a version of PLAN that has been instantiated on an actual mobile robot. An examination of these models helps to provide insight into environmental characteristics which impact human performance on tasks which require interaction with the world.

Achieving tasks with a multiple robot system will require a control system that is both simple and scalable as the number of robots increases. Collective behavior as demonstrated by social insects is a form of decentralized control that may prove useful in controlling multiple robots. Nature 's several examples of collective behavior have motivated our approach to controlling a multiple robot system using a group behavior. Our mechanisms, used to invoke the group behavior, allow the system of robots to perform tasks without centralized control or explicit communication. We have constructed a system of five mobile robots capable of achieving simple collective tasks to verify the results obtained in simulation. The results suggest that decentralized control without explicit communication can be used in performing cooperative tasks requiring a collective behavior. 1 Introduction Can useful tasks be accomplished by a homogeneous team of mobile robots without communication using decentral...

Research on exploratory and searching behavior of animals and robots has attracted an increasing amount of interest recently. Existing works have focused mostly on exploratory behavior guided by vision and audition. Research on smell-guided exploration has been lacking, even though animals may use the sense of smell more widely than sight or hearing to search for food and to evade danger. This article contributes to the study of smell-guided exploration. It describes a series of increasingly complex neural networks, each of which allows a simulated creature to search for food and to evade danger by using smell. Other behaviors such as obstacle negotiation and risk taking emerge naturally from the creature's interaction with the environment. Comparative studies of these networks show that there is no significant performance advantage for a creature to have more than two sensors. This result may help to explain why real animals have only one or two smell-sensing organs.

Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning (Watkins, 1989) in that it allows an agent to adapt its actions to gain maximally from the environment while only being rewarded for correct performance. But animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process that attempts to capture some of the aspects that are missing from simple reinforcement learning: conditioned reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space refinement. We apply our model to a task commonly used to study working memory in rats and monkeys: the DMTS (Delayed Match to Sample) task. Animals learn this task in stages. In simulation, our model also acquires the task in stages, in a similar manner. We have used the model to train an RWI B21 robot.

w sense, it is not sufficient that the system's faculties determine what constitutes its environment; more than this, the organic system must actually intervene causally in the external world. This narrow conception of constructivism allows Godfrey-Smith to make a sharp contrast between, on the one hand, an organism constructing its environment and, on the other hand, an organism changing itself rather than its environment and so merely accommodating its environment (p. 147). (Hereafter I will always use "construct" and its cognates in Godfrey-Smith's preferred narrow sense.) Classifying explanations into these categories involves some subtleties. For one thing, although the definitions might suggest that the distinction between externalist and internalist explanations is dichotomous, Godfrey-Smith is clear that the distinction defines the poles of a continuous range of positions. The explanations of most organic systems invoke both internal and external factors (p. 51), so the degree

It has been reported recently that learning has a beneficial effect on evolution even if the learning involved the acquisition of an ability which is different from the ability for which individuals were selected (Nolfi, Elman & Parisi, 1994). This effect was explained as the result of the interaction between learning and evolution. In a successive paper, however, the effect was explained as a form of recovery from weight perturbation caused by mutations (Harvey, 1996, 1997). In this paper I provide additional data that show how the effect, at least in the case considered in the paper, can only be explained as a result of the interaction between learning and evolution as originally hypothesized. In a recent article Jeffrey Elman, Domenico Parisi, and I reported the results of a set of simulations in which neural networks that evolve (to become fitter at one task) at the population level may also learn (a different task) at the individual level (Nolfi, Elman & Parisi, 1994). In ...

Sutton's Dyna framework provides a novel and computationally appealing way to integrate learning, planning, and reacting in autonomous agents. Examined here is a class of strategies designed to enhance the learning and planning power of Dyna systems by increasing their computational efficiency. The benefit of using these strategies is demonstrated on some simple abstract learning tasks. 1 Introduction Many problems faced by an autonomous agent in an unknown environment can be cast in the form of reinforcement learning tasks. Recent work in this area has led to a clearer understanding of the relationship between algorithms found useful for such tasks and asynchronous approaches to dynamic programming (Bertsekas & Tsitsiklis, 1989), and this understanding has led in turn to both new results relevant to the theory of dynamic programming (Barto, Bradtke, & Singh, 1991; Watkins & Dayan, 1991; Williams & Baird, 1990) and the creation of new reinforcement learning algorithms, such as Qlearn...

If behavior networks, which use spreading activation to select actions, are analogous to connectionist methods of pattern recognition, then we suggest that recurrent behavior networks, which use energy minimization, are analogous to Hopfield networks. Hopfield networks memorize patterns by making them attractors. We argue that, similarly, each behavior of a recurrent behavior network should be an attractor of the network, to inhibit fruitless, repeated switching between different behaviors in response to small changes in the environment and in motivations. We demonstrate that the performance in a test domain of the Do the Right Thing recurrent behavior network is improved by redesigning it to create desirable attractors and basins of attraction. We further show that this performance increase is correlated with an increase in persistence and a decrease in undesirable behavior-switching.

We explore the use of behavior-based architectures within the context of reinforcement learning and examine the effects of using different behavior-based architectures on the ability to learn the task at hand correctly and efficiently. In particular, we study the task of learning to push boxes in a simulated 2D environment originally proposed by Mahadevan and Connell [Mahadevan and Connell, 1992]. We examine issues such as effectiveness of learning, flexibility of the learning method to adapt to new environments, effect of the behavior architecture on the ability to learn, and we report results obtained on a large number of simulation runs. Keywords: Reinforcement learning, behavior-based architectures, robot learning. 1 Introduction Behavior-based architectures [Brooks, 1986] are extremely popular for robotics. In this paper we examine the use of behavior-based architectures within the context of reinforcement learning and examine the effects of using different behavior-base...

A new way of building control systems, known as behavior based robotics, has recently been proposed to overcome the difficulties of the traditional AI approach to robotics. This new approach is based upon the idea of providing the robot with a range of simple behaviors and letting the environment determine which behavior should have control at any given time. We will present a set of experiments in which neural networks with different architectures have been trained to control a mobile robot designed to keep an arena clear by picking-up trash objects and releasing them outside the arena. Controller weights are selected using a form of genetic algorithm and do not change during the lifetime (i.e. no learning occurs). We will compare, in simulation and on a real robot, five different network architectures and will show that a network which allows for fine-grained modularity achieves significantly better performance. By comparing the functionality of each network module and its interactio...

this paper (Parisi, Nolfi, & Cecconi, 1992). The performance of the elite did not improve when lifetime learning of the second task was introduced, whereas average performance did improve. It seems clear that the effect of lifetime learning was merely to go some way towards restoring performance of networks which had had their weights perturbed (by mutation) away from trained (through evolution) values --- a form of relearning. The extreme convergence of the population around the clustered elite members of the previous generation should be borne in mind when reading from (Nolfi et al., 1994), p. 22: The offspring of a reproducing individual occupy initial positions in weight space that are deviations (due to mutations) from the position occupied by their parent at birth (i.e., prior to learning). One form of relearning in networks was analysed in (Hinton & Plaut, 1987). In that case a network is first trained by some learning algorithm on a set of input/output pairs; the weights are then perturbed. After retraining on a subset of the original training set, it is found that performance improves also on the balance of the original training set. The present case differs from this, in that the lifetime learning is on a fresh task, rather than on a subset of the original task. Recently just such an effect was predicted and observed in networks (Harvey & Stone, 1995). When good performance on one task is degraded by random perturbations of the weights, then in general training on any unrelated second task can be expected to improve, at least initially, the performance on the first task. C P Q B B A 1 2 Figure 1: A two-dimensional sketch of weight space. To briefly summarise the reasons for this, consider the diagram, which represents the weight space of a network in just 2 ...

Several researchers have demonstrated how complex behavior can be learned through neuro-evolution (i.e. evolving neural networks with genetic algorithms). However, complex general behavior such as evading predators or avoiding obstacles, which is not tied to specific environments, turns out to be very difficult to evolve. Often the system discovers mechanical strategies (such as moving back and forth) that help the agent cope, but are not very effective, do not appear believable and would not generalize to new environments. The problem is that a general strategy is too difficult for the evolution system to discover directly. This paper proposes an approach where such complex general behavior is learned incrementally, by starting with simpler behavior and gradually making the task more challenging and general. The task transitions are implemented through successive stages of delta-coding (i.e. evolving modifications), which allows even converged populations to adapt to the new task. The...

The paper describes simulations on populations of neural networks that both evolve at the population level and learn at the individual level. Unlike other simulations, the evolutionary task (finding food in the environment) and the learning task (predicting the next position of food on the basis of present position and planned network's movement) are different tasks. In these conditions both learning influences evolution (without Lamarckian inheritance of learned weight changes) and evolution influences learning. Average but not peak fitness has a better evolutionary growth with learning than without learning. After the initial generations individuals that learn to predict during life also improve their food finding ability during life. Furthermore, individuals which inherit an innate capacity to find food also inherit an innate predisposition to learn to predict the sensory consequences of their movements. They do not predict better at birth but they do learn to predict bett...

Exploration and localization are two of the capabilities necessary for mobile robots to navigate robustly in unknown environments. A robot needs to explore in order to learn the structure of the world, and a robot needs to know its own location in order to make use of its acquired spatial information. However, a problem arises with the integration of exploration and localization. A robot needs to know its own location in order to add new information to its map, but a robot may also need a map to determine its own location. We have addressed this problem with ARIEL, a mobile robot system that combines frontier-based exploration with continuous localization. ARIEL is capable of exploring and mapping an unknown environment while maintaining an accurate estimate of its position at all times. In this paper, we describe frontier-based exploration and continuous localization, and we explain how ARIEL integrates these techniques. Then we show results from experiments performed in the explorati...

Many animals, including insects, successfully engage in visual homing. We describe a system that allows a mobile robot to home. Specifically, we propose a simple, yet robust, homing scheme that only relies upon the observation of the bearings of visible landmarks. However, this can easily be extended to include other visual cues. The homing algorithm allows a mobile robot to home incrementally by moving in such a way as to gradually reduce the discrepancy between the current view and the view obtained from the home position. Both simulation and mobile robot experiments are used to demonstrate the feasibility of the approach. Keywords: Robotic Homing, Insect Behavior, Visual Navigation Robotic Homing 3 1 Introduction Getting robots to navigate autonomously has proven to be a difficult task. It is intriguing, then, that small living organisms such as insects have evolved effective solutions to this problem despite carrying relatively simple nervous systems and restricted pro...

This paper is concerned with autonomous learning of plans in probabilistic domains without a priori domain-specic knowledge. In contrast to existing reinforcement learning algorithms that generate only reactive plans and existing probabilistic planning algorithms that require a substantial amount of a priori knowledge in order to plan, a two-stage bottom-up process is devised, in which rst reinforcement learning/dynamic programming is applied, without the use of a priori domain-specic knowledge, to acquire a reactive plan and then explicit plans are extracted from the reactive plan. Several options for plan extraction are examined, each of which is based on a beam search that performs temporal projection in a restricted fashion, guided by the value functions resulting from reinforcement learning/dynamic programming. Some completeness and soundness results are given. Examples in several domains are discussed that together demonstrate the working of the proposed model. 1 I...

We describe the results of a set of evolutionary experiments in which a simulated robotic arm provided with a two-fingered hand has to reach and grasp objects with different shapes and orientations on the basis of simple tactile information. The results that we obtained are encouraging and demonstrate that the problem of grasping objects with characteristics that vary within a certain range can be solved by producing rather simple forms of behavior. These forms of behavior exploit emergent characteristics of the interaction between the body of the robot, its control system, and the environment. In particular we show that evolved individuals do not try to keep the environment stable, but rather push and pull the objects; thus, they produce a dynamic in the environment and exploit the interaction between the body of the robot and the dynamic environment to master different environmental conditions with similar control strategies.

A fundamental characteristic of human speech is that it uses a limited set of basic building blocks (phonemes, syllables), that are put to use in many different combinations to mark differences in meaning. This article investigates the evolution of such combinatorial phonology with a simulated population of agents. We first argue that it is a challenge to explain the transition from holistic to combinatorial phonology, as the first agent that has a mutation for using combinatorial speech does not benefit from this in a population of agents that use a holistic signaling system. We then present a solution for this evolutionary deadlock. We present experiments that show that when a repertoire of holistic signals is optimized for distinctiveness in a population of agents, it converges to a situation in which the signals can be analyzed as combinatorial, even though the agents are not aware of this structure. We argue that in this situation adaptations for productive combinatorial phonology can spread.

The kinematics of human walking are largely driven by passive dynamics, but adaptation to varying terrain conditions and responses to perturbations require some form of active control. The basis for this control is often thought to take the form of entrainment between a neural oscillator (i.e., a central pattern generator and/or distributed counterparts) and the mechanical system. Here we use techniques in evolutionary robotics to explore the potential of a purely reactive, linear controller to control bipedal locomotion over rough terrain. In these simulation studies, joint torques are computed as weighted linear sums of sensor states, and the weights are optimized using an evolutionary algorithm. We show that linear reactive control can enable a seven-link 2D biped and a nine-link 3D biped to walk over rough terrain (steps of ∼5% leg length or more in the 2D case). In other words, the simulated walker gradually learns the appropriate weights to achieve stable locomotion. The results indicate that oscillatory neural structures are not necessarily a requirement for robust bipedal walking. The study of purely reactive control through linear feedback may help to reveal some basic control principles of stable walking.

Sense-making, within enactive theories, provides a novel way of understanding how a comprehensible and manageable world arises for a subject. Elaboration of the concept of sense-making allows a fundamental reframing of the notion of perception that does not rely on the pick up of information about a pre-given world. In rejecting the notion of the subject as an input/output system, it is also necessary to reframe the scientific account of skilled action. Taking speech as an exemplary domain, I here present the outline of an enactive account of skilled action that is continuous with the concept of sense-making. Extending this account to the rich domain of joint or synchronous speaking allows many of the principal themes of the emerging enactive account to be considered as they relate to a familiar and important human practice.

Social coordination is studied in a simulated model of autonomous embodied agents that interact acoustically. Theoretical concepts concerning social behavior are presented from a systemic per spective and their usefulness is evaluated in interpreting the results obtained. Two agents moving in an unstructured arena must locate each other, and remain within a short distance of one anoth er for as long as possible using noisy continuous acoustic interaction. Evolved dynamical recurrent neural networks are used as the control architecture. Acoustic coupling poses nontrivial problems like discriminating 'self' from 'non-self' and structuring production of signals in time so as to minimize interference. Detailed observation of the most frequently evolved behavioral strategy shows that interacting agents perform rhythmic signals leading to the coordination of movement. During coordination, signals become entrained in an anti-phase mode that resembles turn-taking. Perturbation techniques show that signalling behavior not only performs an external function, but it is also integrated into the movement of the producing agent, thus showing the difficulty of separating behavior into social and non-social classes. Structural congruence between agents is shown by exploring internal dynamics as well as the response of single agents in the presence of signalling beacons that reproduce the signal patterns of the interacting agents. Lack of entrainment with the signals produced by the beacons shows the importance of transient periods of mutual dynamic perturbation wherein agents achieve congruence.

In this paper we investigate whether selective attention enables the development of action selection (i.e. the ability to select among conflicting actions afforded by the current agent/environmental context). By carrying out a series of experiments in which neuro-robots have been evolved for the ability to forage so to maximize the energy that can be extracted from ingested substances we observed that effective action and action selection capacities can be developed even in the absence of internal mechanisms specialized for action selection. However, the comparison of the results obtained in different experimental conditions in which the robots were or were not provided with internal modulatory connections demonstrate how selective attention enables the development of a more effective action selection capacity and of more effective and integrated action capacities.

The concept of agency is of crucial importance in cognitive science and artificial intelligence, and it is often used as an intuitive and rather uncontroversial term, in contrast to more abstract and theoretically heavily weighted terms such as intentionality , rationality, or mind. However, most of the available definitions of agency are too loose or unspecific to allow for a progressive scientific research program. They implicitly and unproblematically assume the features that characterize agents, thus obscuring the full potential and challenge of modeling agency. We identify three conditions that a system must meet in order to be considered as a genuine agent: (a) a system must define its own individuality, (b) it must be the active source of activity in its environment (interactional asymmetry), and (c) it must regulate this activity in relation to certain norms (normativity). We find that even minimal forms of proto-cellular systems can already provide a paradigmatic example of genuine agency. By abstracting away some specific details of minimal models of living agency we define the kind of organization that is capable of meeting the required conditions for agency (which is not restricted to living organisms). On this basis, we define agency as an autonomous organization that adaptively regulates its coupling with its environment and contributes to sustaining itself as a consequence. We find that spatiality and temporality are the two fundamental domains in which agency spans at different scales. We conclude by giving an outlook for the road that lies ahead in the pursuit of understanding, modeling, and synthesizing agents.

An abstract is not available.

This paper reviews two main strategies for dealing with the threat posed by radically enactive/embodied cognition to traditional cognitive science. Both strategies invoke action oriented representations (AORs). They differ in emphasizing different features of AORs in their attempt to answer the REC threat – focusing on their contents and vehicles, respectively. The first two sections review the central motivations and rationales driving the ‘content’ and ‘format’ strategies in turn and raise initial concerns about the tenability of each. With respect to the ‘content’ strategy, these worries ought to make us suspicious about the explanatory value of positing AORs. Although the ‘format’ strategy has a way of answering this concern, it raises a more fundamental worry about the motivation for even believing in AORs in the first place. Although these worries cast doubt on the feasibility of invoking AORs as a means of dealing with the REC threat, they do not constitute conclusive reasons for eliminating AORs altogether. There are other, stronger reasons for supposing that we should. The third section provides a sketch of a master argument, developed elsewhere, which makes that case in full dress fashion. The final section – ‘Resurrection?’ – considers and rejects the possibility that AORs might be resurrected, even if it is agreed that the master argument cited in the third section succeeds.

Long-term human-robot interaction, especially in the case of humanoid robots, requires an adaptable and varied behavior base. In this work we present a method for capturing, or learning, sequential tasks by transferring serial behavior execution from deliberative to routine control. The incorporation of this approach leads to natural development of complex and varied behaviors, with lower demands for planning, coordination and resources. We demonstrate how this process can be performed autonomously as part of the normal function of the robot, without the need for an explicit learning stage or user guidance. The complete implementation of this algorithm on the Sony QRIO humanoid robot is described.

Visual homing is a short-range robot navigation method which can lead an agent to a position with accuracy, provided that the majority of the scene visible from the home position is also visible from the current robot position. Recently Zeil, Hoffmann and Chahl (2003) showed that a simple calculation— the root mean square (RMS) difference between the current image and the home image—produces a monotonic function leading to the home position for natural images. In this article we propose a gradi ent descent algorithm based on Caenorhabditis elegans chemotaxis (Ferree & Lockery, 1999) for hom ing with the RMS signal. The parameters for this algorithm are evolved for a simulated agent, and the resulting homing behavior compared with alternative algorithms in simulation and using a real robot. A simulated agent using this algorithm in an environment constructed from real world images homes effi ciently and shows generalization to variations in lighting and changes in the scene. In the real robot this algorithm is affected by noise resulting from imperfect sensors, and alternative algorithms appear more robust. However, the best performing algorithm for unchanging environments, image warping (Franz, Schölkopf, Mallot, & Bülthoff, 1998), is completely disabled by scene changes that do not affect algo rithms utilizing the RMS difference.

Vision did not arise and evolve to just "see" things, but rather to act on and interact with the habitat. Thus it might be misleading to study vision without its natural coupling to vital action. Here we investigate this problem in a simulation study of the simplest kind of visually-guided foraging by a species of 2D hypothetical animal called the (diurnal) paddler. In a previous study, we developed a hypothetical animal called the archaepaddler, which used positive phototaxis to forage for autoluminescent prey in a totally dark environment (the deep-sea). Here we discuss possible visual mechanisms that allow (diurnal) paddlers to live in shallower water, foraging for light-reflecting prey in ambient light. The modification consists of two stages. In the first stage Weber adaptation compresses the retinal illumination into an acceptable range of neural firing frequencies. In the second stage highpass filtering with lateral inhibition separates background responses from foreground responses. We report on a number of parameter-studies conducted with the foraging diurnal paddler, in which the influence of dark/light adaptation and lateral inhibition on foreground/background segregation and foraging performance ("fitness") are quantified. It is shown that the paddler can survive adequately for a substantial range of parameters that compromises between discarding as much unwanted visual (background) information as possible, whilst retaining as much information on potential prey as possible. Parameter values that optimise purely visual performance like foreground/background segregation are not always optimal for foraging performance and vice versa. This shows that studies of vision might indeed require more serious consideration of the goals of vision and the ethogram of the studied organisms than has been customary.

This study presents an extended dynamic neural network model of homeostatic adaptation as the first step toward constructing a model of mental imagery. In the homeostatic adaptation model, higher-level dynamics internally self-organized from sensorimotor dynamics are associated with desired behaviors. These dynamics are regenerated when drastic changes occur, which might break the internal dynamics. Due to the weak link between desired behavior and internal homeostasis in the original homeostatic adaptation model, adaptivity is limited. In this paper, we improve on the homeostatic adaptation model to create a stronger link between desired behavior and internal homeostasis by introducing a metabolic causation in a plasticity mechanism and show that it becomes more adaptive. Our results show that our model has three different time scales in the adaptive behaviors, which are discussed with our cognition and mental imagery.