Ana L. C. Bazzan

• Doctor of Engineering
• Professor (Full) at Federal University of Rio Grande do Sul

Providing timely information to drivers is proving valuable in urban mobility applications. There has been several attempts to tackle this question, from transportation engineering, as well as from computer science points of view. In this paper we use reinforcement learning to let driver agents learn how to select a route. In previous works, vehicl...

Multi-objective reinforcement learning algorithms (MORL) extend standard reinforcement learning (RL) to scenarios where agents must optimize multiple---potentially conflicting---objectives, each represented by a distinct reward function. To facilitate and accelerate research and benchmarking in multi-objective RL problems, we introduce a comprehens...

We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zero-shot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are...

The thought-provoking analogy between AI and electricity, made by computer scientist and entrepreneur Andrew Ng, summarizes the deep transformation that recent advances in Artificial Intelligence (AI) have triggered in the world. This chapter presents an overview of the ever-evolving landscape of AI, written in Portuguese. With no intent to exhaust...

Multi-objective reinforcement learning (MORL) algorithms tackle sequential decision problems where agents may have different preferences over (possibly conflicting) reward functions. Such algorithms often learn a set of policies (each optimized for a particular agent preference) that can later be used to solve problems with novel preferences. We in...

Even though many real-world problems are inherently distributed and multi-objective, most of the reinforcement learning (RL) literature deals with single agents and single objectives. While some of these problems can be solved using a single-agent single-objective RL solution (e.g., by specifying preferences over objectives), there are robustness i...

Using reinforcement learning (RL) to support agents in making decisions that consider more than one objective poses challenges. We formulate the problem of multiple agents learning how to travel from A to B as a reinforcement learning task modeled as a stochastic game, in which we take into account: (i) more than one objective, (ii) non-stationarit...

New technologies have the potential to transform urban mobility. Among the contributions, providing timely information to drivers via, e.g., apps, is proving valuable. However, providing the same information to nearly everyone is counterproductive. In this paper we extend previous works in which vehicles and the road infrastructure exchange informa...

We introduce MO-Gym, an extensible library containing a diverse set of multi-objective reinforcement learning environments. It introduces a standardized API that facilitates conducting experiments and performance analyses of algorithms designed to interact with multi-objective Markov decision processes. Importantly, it extends the widely-used OpenA...

In many real-world applications, reinforcement learning (RL) agents might have to solve multiple tasks, each one typically modeled via a reward function. If reward functions are expressed linearly, and the agent has previously learned a set of policies for different tasks, successor features (SFs) can be exploited to combine such policies and ident...

In many real-world applications, reinforcement learning (RL) agents might have to solve multiple tasks, each one typically modeled via a reward function. If reward functions are expressed linearly, and the agent has previously learned a set of policies for different tasks, successor features (SFs) can be exploited to combine such policies and ident...

Traffic congestion is ubiquitous in cities across the globe resulting in great economic and environmental costs. Although real-time traffic updates are now available, the tendency of drivers to make uncoordinated routing decisions exacerbates the known problems of selfish routing including traffic congestion and flow oscillation. Existing solutions...

Traffic signal control (TSC) is a practical solution to the major problem of congestion in metropolitan areas. Reinforcement Learning (RL) techniques present powerful frameworks for optimizing traffic signal controllers that learn to respond to real-time traffic changes. Multiagent RL (MARL) techniques have been showing better results over centrali...

Improvement of traffic signal control (TSC) efficiency has been found to lead to improved urban transportation and enhanced quality of life. Recently, the use of reinforcement learning (RL) in various areas of TSC has gained significant traction; thus, we conducted a systematic literature review as a systematic, comprehensive, and reproducible revi...

The task of choosing a route to move from A to B is not trivial, as road networks in metropolitan areas tend to be over crowded. It is important to adapt on the fly to the traffic situation. One way to help road users (driver or autonomous vehicles for that matter) is by using modern communication technologies.In particular, there are reasons to be...

Reinforcement learning is an efficient, widely used machine learning technique that performs well in problems with a reasonable number of states and actions. This is rarely the case regarding control-related problems, as for instance controlling traffic signals, where the state space can be very large. One way to deal with the curse of dimensionali...

In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time an...

Non-stationary environments are challenging for reinforcement learning algorithms. If the state transition and/or reward functions change based on latent factors, the agent is effectively tasked with optimizing a behavior that maximizes performance over a possibly infinite random sequence of Markov Decision Processes (MDPs), each of which drawn fro...

Non-stationary environments are challenging for reinforcement learning algorithms. If the state transition and/or reward functions change based on latent factors, the agent is effectively tasked with optimizing a behavior that maximizes performance over a possibly infinite random sequence of Markov Decision Processes (MDPs), each of which drawn fro...

Improvement of traffic signal control (TSC) efficiency has been found to lead to improved urban transportation and enhanced quality of life. Recently, the use of reinforcement learning (RL) in various areas of TSC has gained significant traction; thus, we conducted a systematic literature review as a systematic, comprehensive, and reproducible revi...

With the increase in the use of private transportation, developing more efficient ways to distribute routes in a traffic network has become more and more important. Several attempts to address this issue have already been proposed, either by using a central authority to assign routes to the vehicles, or by means of a learning process where drivers...

Multi-agent systems can be used for modelling large-scale distributed systems in real world applications. In intelligent transportation system (ITS), many interacting entities influence the performance of the system. As part of ITS, traffic signal control can be modelled using a multi-agent system. In this paper, a hierarchical multi-agent system i...

Reinforcement learning is an efficient, widely used machine learning technique that performs well when the state and action spaces have a reasonable size. This is rarely the case regarding control-related problems, as for instance controlling traffic signals. Here, the state space can be very large. In order to deal with the curse of dimensionality...

In recent years, due to the drastic rise in the number of vehicles and the lack of sufficient infrastructure, traffic jams, air pollution, and fuel consumption have increased in cities. The optimization of timing for traffic lights is one of the solutions for the mentioned problems. Many methods have been introduced to deal with these problems, inc...

Navigation apps have become more and more popular, as they give information about the current traffic state to drivers who then adapt their route choice. In commuting scenarios, where people repeatedly travel between a particular origin and destination, people tend to learn and adapt to different situations. What if the experience gained from such...

Traffic assignment is an important stage in the task of modeling a transportation system. Several methods for solving the traffic assignment problem (TAP) were proposed, mostly based on iterative procedures. However, little was done in the direction of analyzing the difficulty of such procedures. For instance, why is it that some networks require o...

Traffic congestion is a significant issue in urban areas and can cause adverse effects. In this paper, our proposal categorizes the traffic activity from video through three steps: vehicle monitoring, feature extraction, and classification. The vehicle monitoring step comprises an object detector based on a convolutional neural network and multi-ob...

Preventive maintenance of offshore units has proven to be crucial in reducing downtime and maintenance costs. In this work, Big Data analytics, applied in the industry to collect, process and analyze data, is used to identify a system abnormal behavior. This knowledge allows the adoption of a proactive maintenance approach instead of the convention...

Mobile devices and Internet-based applications are producing a significant volume of data that may be used to, at least partially, replace some of the hardware necessary to sense traffic systems. However, there are several issues related to such an agenda: data are heterogeneous, unstructured, may appear in natural language, are normally not geoloc...

How to choose a route that takes you from A to B? This is an issue that is turning more and more important in modern societies. One way to address this agenda is through the use of communication between the infrastructure (network), and the demand (vehicles). In this paper, we use car-to-infrastructure (C2I) communication to investigate whether the...

Many multi-agent reinforcement learning (MARL) scenarios lead towards Nash equilibria, which is known to not always be socially efficient. In this study we aim to align the social optimization objective of the system with the individual objectives of the agents by adopting a central controller which can interact with the agents. In details, our app...

Network science has proved useful in analyzing structure and dynamics of social networks in several areas. This paper aims at analyzing the relationships of characters in Friends, a famous sitcom. In particular, two important aspects are investigated. First, the structure of the communities (groups) is examined to shed light on how different method...

The agent-based modeling and simulation (ABMS) paradigm has been used to analyze, reproduce, and predict phenomena related to many application areas. Although there are many agent-based platforms that support simulation development, they rely on programming languages that require extensive programming knowledge. Model-driven development (MDD) has b...

The agent-based modeling and simulation (ABMS) paradigm has been used to analyze, reproduce, and predict phenomena related to many application areas. Although there are many agent-based platforms that support simulation development, they rely on programming languages that require extensive programming knowledge. Model-driven development (MDD) has b...

Since the 1970s, scholars have begun to pay attention to the presentation of women in Bede’s Ecclesiastical History of the English People, the main source for the early history of Britain (from the first century BC to the eighth century AD). Vastly different conclusions have been drawn, ranging from positivist approaches which saw the period as a g...

Network theory has been used to analyze structures of narratives in works of fiction. Indeed, previous works have shed light on issues related to role detection, for instance. However, few comparative works exist that deal with TV shows. Since these shows are very popular, there are several Internet forums that suggest how similar some of them are,...

In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time an...

It is increasingly common applications where data are naturally generated in a distributed fashion, especially after the emergence of technologies like the Internet of Things (IoT). In sensor networks, in collaborative health or genomic projects, in credit risk analysis, among other domains, distinct features are collected from multiple sources, in...

The problem of traffic congestion incurs numerous social and economical repercussions and has thus become a central issue in every major city in the world. For this work we look at the transportation domain from a multiagent system perspective, where every driver can be seen as an autonomous decision-making agent. We explore how learning approaches...

One of the defining representations of women from medieval times is in the role of peace-weaver, that is, a woman was expected to 'weave' peace between warring men. The underlying assumption in scholarship on this topic is that female mediation lessens male violence. This stance can however be questioned since it may be the result of gender-based p...

Resumo publicado nos Anais da Feira de Iniciação Científica da Universidade FEEVALE, entre os dias 21 e 26 de outubro de 2019. Anais da Feira de Iniciação Científica - FIC. Novo Hamburgo: Universidade FEEVALE, 2019. v. 11. p. 787. Disponível em: https://www.feevale.br/Comum/midias/e8786f81-38c3-446e-87c9-d60d84f64f69/Anais%20FIC%202019.pdf

Resumo publicado nos Anais do XXXI Salão de Iniciação Científica (SIC) da Universidade Federal do Rio Grande do Sul (UFRGS), 2019, Porto Alegre. Disponível em: <https://lume.ufrgs.br/handle/10183/208583>

Pôster apresentado no XXXI Salão de Iniciação Científica (SIC) da Universidade Federal do Rio Grande do Sul (UFRGS), entre os dias 21 e 25 de outubro de 2019. Porto Alegre - RS / Brasil. Disponível em: https://lume.ufrgs.br/handle/10183/208583

Social networks have been recently employed as a source of information for traffic event detection, with particular reference to traffic incidents. However, extracting meaningful and actionable knowledge from user-generated content is a complex endeavor, because social media texts are usually short, informal, with a lot of abbreviations, jargon, sl...

Search for equilibria in games is a hard problem and many games do not have a pure Nash equilibrium (PNE). Incentive mechanisms have been shown to secure a PNE in certain families of games. The present study utilizes the similarity between Asymmetric Distributed Constraints Optimization Problems (ADCOPs) and games, to construct search algorithms fo...

Learning from data streams requires efficient algorithms capable of constructing a model according to the arrival of new instances. These data stream learners need a quick and real-time response, but mainly, they must be tailored to adapt to possible changes in the data distribution, a condition known as concept drift. However, recent works have sh...

Pôster apresentado na Feira de Iniciação Científica da Universidade FEEVALE, entre os dias 21 e 26 de outubro de 2019. Novo Hamburgo - RS / Brasil. Disponível em: https://www.feevale.br/ConteudoVirtual/Inovamundi/Area/Listar/8294

Current estimates show that in Brazil traffic congestion has a huge impact in the economy. Although Intelligent Transportation Systems (ITS) techniques can contribute to reduce this figure, many of the proposed ITS-related techniques have not yet been fully developed and/or deployed. This is especially the case of techniques that use the potential...

Traffic congestions present a major challenge in large cities. Consid- ering the distributed, self-interested nature oftraffic we tackle congestions using multiagent reinforcement learning (MARL). In this thesis, we advance the state- of-the-art by delivering the first MARL convergence guarantees in congestion- like problems. We introduce an algori...

This article presents a multi-agent framework for optimization using metaheuristics, called AMAM. In this proposal, each agent acts independently in the search space of a combinatorial optimization problem. Agents share information and collaborate with each other through the environment. The goal is to enable the agent to modify their actions based...

In this article, we present a decentralized convention formation framework for creating social conventions within large multiagent convention spaces. We study the role of the topological characteristics of the network in forming conventions with an emphasis on scale-free topologies. We hypothesize that contextual knowledge encapsulated in the topol...

In complex socio-technical systems it is not easy to find a balance between the welfare state (i.e., a state where the overall performance of a system is optimal) and a situation in which individual components act selfishly to optimize their own utilities. This is even harder when individuals compete for scarce resources. In order to deal with this...

The effects of traffic congestion can be mitigated with a range of different methods. This paper addresses multiagent reinforcement learning (MARL) as a contribution to this effort. Specifically, while most of the literature on MARL applied to traffic assumes that either only traffic signals learn (while drivers do not change their behaviors) or vi...

We consider the route choice problem using multiagent reinforcement learning. In this problem, agents individually learn which routes minimise their expected travel costs. Such a selfish behaviour results in the so-called User Equilibrium (UE), which is inefficient from the system's perspective. In order to reduce the impact of selfishness , we dev...

Data stream classification poses many challenges for the data mining community when the environment is non-stationary, among which adaptation to the concept drifts, i.e., changes in the underlying concepts, is a major one. Two main ways to develop adaptive approaches are ensemble methods and incremental algorithms. Ensemble methods play an importan...

The envisaged usage of multiple Unmanned Aerial Vehicles (UAVs) to perform cooperative tasks is a promising concept for future autonomous military systems. An important aspect to make this usage a reality is the solution of the task allocation problem in these cooperative systems. This paper addresses the problem of tasks allocation among agents re...

Network science has proved useful in analyzing structure and dynamics of social networks in several areas. This paper aims at analyzing the relationships of characters in Friends, a famous sitcom. In particular, not only static structure and causality relationships are investigated, but also temporal aspects. After all, this show was aired for ten...

In systems composed by a high number of highly coupled components, aligning the optimum of the system with the optimum of those individual components can be conflicting, especially in situations in which resources are scarce. In order to deal with this, many authors have proposed forms of biasing the optimization process. However, mostly, this work...

In the route choice problem, self-interested drivers aim at choosing routes that minimise travel costs between their origins and destinations. We model this problem as a multiagent reinforcement learning scenario. Here, since agents must adapt to each others’ decisions, the minimisation goal is seen as a moving target. Regret is a well-known perfor...

In this chapter, we introduce the use of self-organised coalitions in smart grid scenarios for finding a coalition structure that maximises the systems’ utility. The complexity of such a task is exponential with the number of agents, and optimal coalition formation has been considered impractical. Several heuristic alternatives have been proposed i...

Model-driven development (MDD) is an approach for supporting the development of software systems, in which high-level modeling artifacts drive the production of time and effort-consuming low-level artifacts, such as the source code. Previous studies of the MDD effectiveness showed that it significantly increases development productivity, because th...

To reduce the air pollution and improve the energy efficiency, many countries and cities (e.g., Singapore) are on the way of introducing electric vehicles (EVs) to replace the vehicles serving in current traffic system. Effective placement of charging stations is essential for the rapid development of EVs, because it is necessary for providing conv...

The traffic assignment problem (TAP) consists of assigning routes to trips, in order to minimize the travel time of all these trips. Classical methods assume the existence of a central authority that computes and assigns routes to road users. In this paper, we present two multiagent reinforcement learning approaches for the TAP. They solve the prob...

This book constitutes the refereed proceedings of the 20th International Conference on Principles and Practice of Multi-Agent Systems, PRIMA 2017, held in Nice, France, in October/November 2017. The 24 revised full papers presented together with one abstract of a keynote talk and 11 short papers were carefully reviewed and selected from 88 submiss...

This paper presents a non-prioritized belief change operator, designed specifically for incorporating new information from many heterogeneous sources in an uncertain environment. We take into account that sources may be untrustworthy and provide a principled method for dealing with the reception of contradictory information. We specify a novel Data...

The Braess’ paradox is a well-known problem associated with route choice and traffic distribution. Agent-based simulations that investigate this paradox typically model driver’s behaviour using reactive agent architectures, which simplify and abstract an inherently complex behaviour. The BDI architecture is an alternative widely used in multi-agent...

In many real-world situations in which resources are scarce, aligning the optimum of the system with the optimum of agents can be conflicting. For instance, in traffic assignment, the system's and the agents' welfare may not be aligned. In order to deal with this, in this paper a new approach is proposed, based on a synergy between: (i) a global op...

Esse trabalho investiga uma maneira de diminuir o tempo médio de viagem dos veículos. Para isso, uma abordagem baseada em aprendizado por reforço para a escolha de rotas dos veículos foi utilizada. Cada veículo recebe um conjunto de rotas pré-computadas e antes de cada viagem o veículo deve escolher entre essas rotas. Para validar a abordagem foi u...

This paper discusses the use of a coupling metric to characterize traffic networks for an agent based solution to the traffic assignment problem (we call it route choice problems). This metric is based on how often routes share edges with others and how interconnected they are. Since the choice of learning agents may interfere with each other, we e...

Reinforcement learning (RL) is a challenging task, especially in highly competitive multiagent scenarios. We consider the route choice problem, in which self-interested drivers aim at choosing routes that minimise their travel times. Employing RL here is challenging because agents must adapt to each others' decisions. In this paper, we investigate...

Evaluating multiagent reinforcement learning (MARL) approaches in real world problems, such as traffic, is a challenging task. In general, such approaches cannot be deployed before extensive validation. Hence, simulating the impact of these approaches represents an essential step towards its deployment. Existing MARL tools make this process easier...

Defining a reward function that, when optimized, results in the rapid acquisition of an optimal policy, is one of the most challenging tasks involved in applying reinforcement learning algorithms. The behavior learned by agents is directly related to the reward function they are using. Existing works on Optimal Reward Problem (ORP) propose mechanis...

In the context of historical research, clustering of different groups into warring factions can lead to a better understanding of how conflicts arise or can be avoided. Using a spin-glass-based community detection algorithm, we study the crisis of 1225 between the Emperor of the Holy Roman Empire Frederick II and his son Henry VII, which almost led...

Many social networks exhibit some underlying community structure. In particular, in the context of historical research, clustering of different groups into warring or friendly factions can lead to a better understanding of how conflicts may arise, and whether they could be avoided or not. In this work we study the crisis that started in 1225 when t...

The impact of an increasing share of electric vehicles (EVs) is not fully known. Issues related to engine and battery point out that drivers of EVs may consider different route choices in order to take advantage of energy recovery by regenerative braking, for instance. Because not only travel time but also energy consumption may matter, this work p...

It is well-known that selfish routing, where individual agents make uncoordinated greedy routing decisions, does not produce a socially desirable outcome in transport and communication networks. In this paper, we address this general problem of the loss of social welfare that occurs due to uncoordinated behavior in networks and model it as a multia...

Since in many cities transport infrastructure is operating at or beyond capacity, novel approaches to organize urban mobility are gaining attraction. However, assessing the benefits of a measure that has disruptive capacity in a complex system requires a carefully designed research. This paper takes a recent idea for urban mobility - flexible road...

The notion of regret has been extensively employed to measure the performance of reinforcement learning agents. The regret of an agent measures how much worse it performs following its current policy in comparison to following the best possible policy. As such, measuring regret requires complete knowledge of the environment. However, such an assump...