François Rivest

• PhD
• Professor (Associate) at Royal Military College of Canada

Significant progress has been made recently in deep reinforcement learning in the development of options. This idea consists in learning policies (or macro of actions) for sub-goals. An important bottleneck of this approach is that these options are often available as actions everywhere in the state space, hence, potentially enlarging the action sp...

Animals can quickly learn the timing of events with fixed intervals and their rate of acquisition does not depend on the length of the interval. In contrast, recurrent neural networks that use gradient based learning have difficulty predicting the timing of events that depend on stimulus that occurred long ago. We present the latent time-adaptive d...

Smart homes are becoming increasingly popular as a result of advances in machine learning and cloud computing. Devices such as smart thermostats and speakers are now capable of learning from user feedback and adaptively adjust their settings to human preferences. Nonetheless, these devices might in turn impact human behavior. To investigate the pot...

Animal interval timing is often studied through the peak interval (PI) procedure. In this procedure, the animal is rewarded for the first response after a fixed delay from the stimulus onset, but on some trials, the stimulus remains and no reward is given. The standard methods and models to analyse the response pattern describe it as break-run-brea...

Human Activity Recognition (HAR) plays a critical role in applications such as security surveillance and healthcare. However, existing methods, particularly two-stream models like Inflated 3D (I3D), face significant challenges in real-time applications due to their high computational demand, especially from the optical flow branch. In this work, we...

Efficient utilization of satellite resources in dynamic environments remains a challenging problem in satellite scheduling. This paper addresses the multi-satellite collection scheduling problem (m-SatCSP), aiming to optimize task scheduling over a constellation of satellites under uncertain conditions such as cloud cover. Leveraging Monte Carlo Tr...

It is well known that optimistically initializing a Q-table in table-based Reinforcement Learning can be a useful technique in reward-sparse environments. In this paper, we propose an approach to initialize a Deep Q-network optimistically. The traditional approach of initializing the Q-networks results in very small near zero initial values but by...

Machine learning and deep learning have made tremendous progress over the last decade and have become the de facto standard across a wide range of image, video, text, and sound processing domains, from object recognition to image generation [...]

Artificial Intelligence of Things (AIoT) combines the power of artificial intelligence, computing power, and IoT infrastructure. With AIoT, artificial intelligence (AI) is embedded in computing devices, all connected to one or more IoT networks. The mutual benefit of these two technologies allows for a different vision and a broader scope of action...

In reinforcement learning, agents have successfully used environments modeled with Markov decision processes (MDPs). However, in many problem domains, an agent may suffer from noisy observations or random times until its subsequent decision. While partially observable Markov decision processes (POMDPs) have dealt with noisy observations, they have...

The technologies used in smart homes have recently improved to learn the user preferences from feedback in order to enhance the user convenience and quality of experience. Most smart homes learn a uniform model to represent the thermal preferences of users, which generally fails when the pool of occupants includes people with different sensitivitie...

Hazardous situations such as house, car, or forest fires may be recorded by cameras long before they are identified by people. To test whether deep learning could be used to quickly detect fires, we performed a series of experiments to detect the presence of fire or smoke in images and labeled them with bounding boxes. Two custom datasets were crea...

The technology used in smart homes have improved to learn the user preferences from feedbacks in order to provide convenience to the user in the home environment. Most smart homes learn a uniform model to represent the thermal preference of user which generally fails when the pool of occupants includes people having different age, gender, and locat...

Weapon-target assignment (WTA) is a well-studied problem in operations research, with a wide range of applications in simple task assignments. However, in situations involving complex task scheduling, WTA fails to capture many important planning constraints. In this work, we present a new problem formulation called Spatio-Temporal WTA which incorpo...

Animal interval timing is often studied through the peak interval (PI) procedure. In this procedure, the animal is rewarded for the first response after a fixed delay from the stimulus onset, but on some trials, the stimulus remains and no reward is given. The common methods and models to analyse the response pattern describe it as break-run-break,...

We aim to investigate the potential impacts of smart homes on human behavior. To this end, we simulate a series of human models capable of performing various activities inside a reinforcement learning-based smart home. We then investigate the possibility of human behavior being altered as a result of the smart home and the human model adapting to o...

Multi-satellite scheduling often involves generating a fixed number of potential task schedules, evaluating them all, and selecting the path that yields the highest expected reward. Unfortunately, this approach, however accurate, is difficult to scale up and applied to large realistic problems due to combinatorial explosion. Furthermore, it is cost...

With the release of large-scale bone age assessment datasets and competitions looking at solving the problem of bone age estimation, there has been a large boom of machine learning in medical imaging which has attempted to solve this problem. Although many of these approaches use convolutional neural networks, they often include some specialized fo...

The ability to make predictions is central to the artificial intelligence problem. While machine learning algorithms have difficulty in learning to predict events with hundreds of time step dependencies, animals can learn event timing within tens of trials across a broad spectrum of time scales. This suggests strongly a need for new perspectives on...

Integrating a sense of time into reinforcement learning is one of the next challenges in both machine learning, and computational neuroscience. In this paper, we first show that the standard approach of minimizing the sum of squared error (SSE) at each time step does not provided the temporal information needed to solve this problem. We then propos...

Drift-diffusion models (DDMs) are a popular framework for explaining response times in decision-making tasks. Recently, the DDM architecture has been used to model interval timing. The Time-adaptive DDM (TDDM) is a physiologically plausible mechanism that adapts in real-time to different time intervals while preserving timescale invariance. One key...

The ability to represent temporal information and to learn the timing of recurring, instantaneous events is central to the artificial intelligence problem, in particular for embodied agents such as robots. In this paper, we introduce a learning algorithm that provides a new way to learn the value function of a given policy in a partially observable...

Dopaminergic models based on the temporal-difference learning algorithm usually do not differentiate trace from delay conditioning. Instead, they use a fixed temporal representation of elapsed time since conditioned stimulus onset. Recently, a new model was proposed in which timing is learned within a long short-term memory (LSTM) artificial neural...

Pacemaker-accumulator (PA) systems have been the most popular kind of timing model in the half-century since their introduction by Treisman (1963). Many alternative timing models have been designed predicated on different assumptions, though the dominant PA model during this period –Scalar Expectancy Theory (SET; Church, Meck, & Gibbon, 1984) – inv...

Animals readily learn the timing between salient events. They can even adapt their timed responding to rapidly changing intervals, sometimes as quickly as a single trial. Recently, drift-diffusion models-widely used to model response times in decision making-have been extended with new learning rules that allow them to accommodate steady-state inte...

Animals learn the timing between consecutive events very easily. Their precision is usually proportional to the interval to time (Weber's law for timing). Most current timing models either require a central clock and unbounded accumulator or whole pre-defined populations of delay lines, decaying traces or oscillators to represent elapsing time. Cur...

Tout au long de la vie, le cerveau développe des représentations de son environnement permettant à l’individu d’en tirer meilleur profit. Comment ces représentations se développent-elles pendant la quête de récompenses demeure un mystère. Il est raisonnable de penser que le cortex est le siège de ces représentations et que les ganglions de la base...

Dopaminergic neuron activity has been modeled during learning and appetitive behavior, most commonly using the temporal-difference (TD) algorithm. However, a proper representation of elapsed time and of the exact task is usually required for the model to work. Most models use timing elements such as delay-line representations of time that are not b...

We previously measured human performance on a complex problem-solving task that involves finding which ball in a set is lighter or heavier than the others with a limited number of weightings. None of the participants found a correct solution within 30 minutes without help of demonstrations or instructions. In this paper, we model human performance...

The new field of developmental robotics faces the formidable challenge of implementing effective learning mechanisms in complex, dynamic environments. We make a case that knowledge-based learning algorithms might help to meet this challenge. A constructive neural learning algorithm, knowledge-based cascade-correlation (KBCC), autonomously recruits...

While both dopamine (DA) fluctuations and spike-timing-dependent plasticity (STDP) are known to influence long-term corticostriatal plasticity, little attention has been devoted to the interaction between these two fundamental mechanisms. Here, a theoretical framework is proposed to account for experimental results specifying the role of presynapti...

A constructive learning algorithm, knowledge-based cascade-correlation (KBCC), recruits previously-learned networks in addition to the single hidden units recruited by ordinary cascade-correlation. This enables learning by analogy when adequate prior knowledge is available, learning by induction from examples when there is no relevant prior knowled...

Computational learning rules considered to be biologically realistic are not only rare but are also known to be seriously underpowered in the sense that they cannot, by themselves, implement the learning that humans and other mammals are capable of. We show mathematically that the computationally-powerful learning rules used in the cascade-correlat...

KBCC is an extension of the cascade-correlation algorithm that treats functions encapsulating prior knowledge as black-boxes which, like simple sigmoidal neurons, can be recruited in the network topology. KBCC has been studied on artificial and real tasks and it has successfully reused various kinds of knowledge. This paper surveys the work on KBCC...

Successful application of reinforcement learning algorithms often involves considerable hand-crafting of the necessary non-linear features to reduce the complexity of the value functions and hence to promote convergence of the algorithm. In contrast, the human brain readily and autonomously finds the complex features when provided with sufficient t...

A knowledge-based constructive learning algorithm, KBCC, simplifies and accelerates the learning of parity and chessboard problems. Previously learned knowledge of simpler versions of these problems is recruited in the service of learning more complex versions. A learned solution can be viewed as a composition in which the components are not altere...

In this report we discuss using the Cascade-correlation architecture in reinforcement learning tasks. By using a cache to store training patterns and then training the Cascade-correlation neural network at set intervals we can hope to obtain online learning for this algorithm. We first review the various elements used in our system. Following this,...

Artificial neural networks typically ignore the role of knowledge in learning by starting from random connection weights. A new algorithm, knowledge-based cascade-correlation (KBCC), finds, adapts, and uses its relevant knowledge to speed learning. We demonstrate its performance on small, clear problems involving decisions about whether a two-dimen...

An algorithm for performing simultaneous growing and pruning of cascade-correlation (CC) neural networks is introduced and tested. The algorithm adds hidden units as in standard CC, and removes unimportant connections by using optimal brain damage (OBD) in both the input and output phases of CC. To this purpose, OBD was adapted to prune weights acc...

Using neural networks to represent value functions in reinforcement learning algorithms often involves a lot of work in hand-crafting the network structure, and tuning the learning parameters. In this paper, we explore the potential of using constructive neural networks in reinforcement learning. Constructive neural network methods are appealing be...

Most neural network learning algorithms cannot use knowledge other than what is provided in the training data. Initialized using random weights, they cannot use prior knowledge such as knowledge stored in previously trained networks. This manuscript thesis addresses this problem. It contains a literature review of the relevant static and constructi...

Neural network algorithms are usually limited in their ability to use prior knowledge automatically. A recent algorithm, a knowledge-based cascade-correlation (KBCC), extends the cascade-correlation by evaluating and recruiting previously learned networks in its architecture. In this paper, we describe KBCC and illustrate its performance on the pro...

Research with neural networks typically ignores the role of knowledge in learning by initializing the network with random connection weights. We examine a new extension of a well-known generative algorithm, cascade-correlation. Ordinary cascade-correlation constructs its own network topology by recruiting new hidden units as needed to reduce networ...

Neural network modeling typically ignores the role of knowledge in learning by starting from random weights. A new algorithm extends cascade-correlation by recruiting previously learned networks as well as single hidden units. Knowledge-based cascade-correlation (KBCC) finds, adapts, and uses its relevant knowledge to speed learning. In this paper,...

Cognitive modeling with neural networks unrealistically ignores the role of knowledge in learning by starting from random weights. It is likely that effective use of knowledge by neural networks could significantly speed learning. A new algorithm, knowledge-based cascade- correlation (KBCC), finds and adapts its relevant knowledge in new learning....