Javier González-Jiménez

• PhD in Robotics
• Professor (Full) at University of Malaga

Large Language Models (LLMs) provide cognitive capabilities that enable robots to interpret and reason about their workspace, especially when paired with semantically rich representations like semantic maps. However, these models are prone to generating inaccurate or invented responses, known as hallucinations, that can produce an erratic robotic o...

Semantic scene understanding allows a robotic agent to reason about problems in complex ways, using information from multiple and varied sensors to make deductions about a particular matter. As a result, this form of intelligent robotics is capable of performing more complex tasks and achieving more precise results than simpler approaches based on...

Large Language Models (LLMs) provide cognitive capabilities that enable robots to interpret and reason about their workspace, especially when paired with semantically rich representations like semantic maps. However, these models are prone to generating inaccurate or invented responses, known as hallucinations, that can produce an erratic robotic o...

Robots in human-centered environments require accurate scene understanding to perform high-level tasks effectively. This understanding can be achieved through instance-aware semantic mapping, which involves reconstructing elements at the level of individual instances. Neural networks, the de facto solution for scene understanding, still face limita...

Con el objetivo de lograr una interacción humano-robot lo más natural posible, es fundamental que el robot se oriente hacia su interlocutor. Este trabajo presenta un sistema multimodal que usa información visual y de sonido para lograr una orientación precisa incluso en situaciones complejas con múltiples personas, personas fuera del campo de visió...

La aparición de los modelos a gran escala permite abordar algunas de las principales limitaciones que presentan las técnicas de mapeo semántico tradicional en robótica móvil. Sin embargo, estos modelos son propensos a generar respuestas incorrectas, incoherentes o incluso inventadas, pudiendo ocasionar comportamientos erróneos del robot. Para poder...

El SLAM visual se basa comúnmente en la optimización de un grafo de keyframes, imágenes clave en una secuencia de vídeo, para la construcción de mapas 3D y la localización de la cámara. La creación de este grafo requiere de un proceso front-end eficiente que seleccione keyframes con suficiente solape entre observaciones pero con bajo coste computac...

El desarrollo y despliegue de aplicaciones robóticas en investigación involucra desafíos como la gestión eficiente de hardware heterogéneo, especialmente GPUs, o la elaboración de configuraciones software con requisitos incompatibles, por ejemplo, conflictos de librerías y versiones. A menudo, estos problemas se convierten en una limitación para lo...

Este trabajo aborda el problema de la localización eficiente de emisiones de metano en espacios abiertos mediante el uso de robótica móvil. En contraposición a los métodos convencionales que emplean detectores puntuales, o que empleando medidores de rango dependen del suelo como reflector natural (los cuales conllevan largos tiempos e ineficientes...

The early detection of mild cognitive impairment, a condition of increasing impact in our aging society, is a challenging task with no established answer. One promising solution is the deployment of robotic systems and ambient assisted living technology in the houses of older adults for monitoring and assistance. In this work, we address and discus...

Gas source localization ( GSL ) with an autonomous robot is a problem with many prospective applications, from finding pipe leaks to emergency-response scenarios. In this work, we present a new method to perform GSL in realistic indoor environments, featuring obstacles and turbulent flow. Given the highly complex relationship between the source po...

There are many potential applications for an autonomous robotic agent capable of sensing gases in the environment, from locating leaks in pipes to monitoring air quality. However, the current state of the art in the field of robotic olfaction is not mature enough for most real-world applications. Due to the complexity of gas dispersion phenomena an...

We present a new fast Ground Decoupled 3D Lidar Odometry (GND-LO) method. The particularity of GND-LO is that it takes advantage of the distinct spatial layout found in urban settings to efficiently recover the lidar movement in a decoupled manner. For that, the input scans are reduced to a set of planar patches extracted from the flat surfaces of...

We present a novel 3D odometry method that recovers the full motion of a vehicle only from a Doppler-capable range sensor. It leverages the radial velocities measured from the scene, estimating the sensor's velocity from a single scan. The vehicle's 3D motion, defined by its linear and angular velocities, is calculated taking into consideration its...

The primitives used to model objects in semantic maps heavily influence their suitability for certain robot tasks, as well as the computational load required to process them. This paper contributes a semantic mapping framework that incrementally and efficiently builds a voxelized representation of the robot workspace, providing a balanced trade-off...

Representing the scene appearance by a global image descriptor (BoW, NetVLAD, etc.) is a widely adopted choice to address Visual Place Recognition (VPR). The main reasons are that appearance descriptors can be effectively provided with radiometric and perspective invariances as well as they can deal with large environments because of their compactn...

Released in 2017, Robot@Home is a dataset captured by a mobile robot during indoor navigation sessions in apartments. This paper presents Robot@Home2, an enhanced version of the Robot@Home dataset, aimed at improving usability and functionality for developing and testing mobile robotics and computer vision algorithms. Robot@Home2 consists of three...

The ability to sense airborne pollutants with mobile robots provides a valuable asset for domains such as industrial safety and environmental monitoring. Oftentimes, this involves detecting how certain gases are spread out in the environment, commonly referred to as a gas distribution map, to subsequently take actions that depend on the collected i...

Gas source localization (GSL) with an autonomous robot is a problem with many prospective applications, from finding pipe leaks to emergency-response scenarios. In this work we present a new method to perform GSL in realistic indoor environments, featuring obstacles and turbulent flow. Given the highly complex relationship between the source positi...

Cutting-edge field robotic systems, such as UAV or autonomous cars, demand fast and optimal solutions for any component at the core of their critical navigational tasks. Among them, we focus on the triangulation of image points from multiple views, which is a cornerstone for more complex tasks such as visual localization and SLAM. In this letter we...

This work presents GadenTools, a toolkit designed to ease the development and integration of mobile robotic olfaction applications by enabling a convenient and user-friendly access to Gaden’s realistic gas dispersion simulations. It is based on an easy-to-use Python API, and includes an extensive tutorial developed with Jupyter Notebook and Google...

Planar scenes predominate in man-made environments, e.g.interior or facades of buildings and in ground images from aerial vehicles. Points lying on those surfaces can be reconstructed from their observations in two images. However, generic reconstruction algorithms output 3D points not lying on the plane, thus obtaining inaccurate reconstructions....

This work presents Sigma-FP, a novel 3D reconstruction method to obtain the floor plan of a multi-room environment from a sequence of RGB-D images captured by a wheeled mobile robot. For each input image, the planar patches of visible walls are extracted and subsequently characterized by a multivariate Gaussian distribution in the convenient Plane...

In this paper we present a new way to compute the odometry of a 3D lidar in real-time. Due to the significant relation between these sensors and the rapidly increasing sector of autonomous vehicles, 3D lidars have improved in recent years, with modern models producing data in the form of range images. We take advantage of this ordered format to eff...

This paper presents an autonomous navigation method for an agricultural mobile robot "AgriEco Robot", with four-wheel-drive and embedded perception sensors. The proposed method allows an accurate guidance between strawberry crop rows while automatically spraying pesticides, as well as detecting the end and switching to the next rows. The main contr...

Appearance-based Localization (AL) focuses on estimating the pose of a camera from the information encoded in an image, treated holistically. However, the high-dimensionality of images makes this estimation intractable and some technique of dimensionality Reduction (DR) must be applied. The resulting reduced image representation, though, must keep...

This paper proposes LTC-Mapping, a method for building object-oriented semantic maps that remain consistent in the long-term operation of mobile robots. Among the different challenges that compromise this aim, LTC-Mapping focuses on two of the more relevant ones: preventing duplicate instances of objects (instance duplication) and handling dynamic...

Simulations and synthetic datasets have historically empower the research in different service robotics-related problems, being revamped nowadays with the utilization of rich virtual environments. However, with their use, special attention must be paid so the resulting algorithms are not biased by the synthetic data and can generalize to real world...

Visual Place Recognition (VPR), the task of identifying the place where an image has been taken from, is at the core of important robotic problems as relocalization, loop-closure detection or topological navigation. Even for indoors, the focus of this work, VPR is challenging for a number of reasons, including real-time performance when dealing wit...

This paper tackles the resolution of the Relative Pose problem with optimality guarantees by stating it as an optimization problem over the set of essential matrices that minimizes the squared epipolar error. We relax this non-convex problem with its Shor’s relaxation, a convex program that can be solved by off-the-shelf tools. We follow the empiri...

In an ageing society, the at-home use of Socially Assistive Robots (SARs) could provide remote monitoring of their users’ well-being, together with physical and psychological support. However, private home environments are particularly challenging for SARs, due to their unstructured and dynamic nature which often contributes to robots’ failures. Fo...

En este trabajo se propone un método que combina descriptores de imágenes de intensidad y de profundidad para detectar de manera robusta el problema de cierre de bucle en SLAM. La robustez del método, proporcionada por el empleo conjunto de información de diversa naturaleza, permite detectar lugares revisitados en situaciones donde métodos basados...

The integration of Ambient Assisted Living (AAL) frameworks with Socially Assistive Robots (SARs) has proven useful for monitoring and assisting older adults in their own home. However, the difficulties associated with long-term deployments in real-world complex environments are still highly under-explored. In this work, we first present the MoveCa...

In this paper, we present an attention mechanism for mobile robots to face the problem of place categorization. Our approach, which is based on active perception, aims to capture images with characteristic or distinctive details of the environment that can be exploited to improve the efficiency (quickness and accuracy) of the place categorization....

This paper proposes a method to enhance video object detection for indoor environments in robotics. Concretely, it exploits knowledge about the camera motion between frames to propagate previously detected objects to successive frames. The proposal is rooted in the concepts of planar homography to propose regions of interest where to find objects,...

Images of a given environment, coded by a holistic image descriptor, produce a manifold that is articulated by the camera pose in such environment. The correct articulation of such Descriptor Manifold (DM) by the camera poses is the cornerstone for precise Appearance-based Localization (AbL), which implies knowing the correspondent descriptor for a...

In this paper, we present an algorithm for the detection of line segments directly on the original, distorted images captured by calibrated wide-angle, fisheye and omnidirectional cameras. Distorted line segments are detected as convex polygonal chains of connected straight lines and then validated as the projection of 3D lines. This last validatio...

This work contributes an efficient algorithm to compute the relative pose problem (RPp) between calibrated cameras and certify the optimality of the solution, given a set of pair-wise feature correspondences affected by noise and probably corrupted by wrong matches. We propose a family of certifiers that is shown to increase the ratio of detected o...

A human motion capture system using an RGB-D camera could be a good option to understand the trunk limitations in spondyloarthritis. The aim of this study is to validate a human motion capture system using an RGB-D camera to analyse trunk movement limitations in spondyloarthritis patients. Cross-sectional study was performed where spondyloarthritis...

Most mobile robots are powered by batteries, which must be charged before their level become too low to continue providing services. This paper contributes a novel method based on Reinforcement Learning (RL) for the autonomous docking of mobile robots at their charging stations. Our proposal considers a RL network that is fed with images to visuall...

Semantic maps augment traditional representations of robot workspaces, typically based on their geometry and/or topology, with meta-information about the properties, relations and functionalities of their composing elements. A piece of such information could be: fridges are appliances typically found in kitchens and employed to keep food in good co...

The fusion of visual and inertial measurements is becoming more and more popular in the robotics community since both sources of information complement each other well. However, in order to perform this fusion, the biases of the Inertial Measurement Unit (IMU) as well as the direction of gravity must be initialized first. In case of a monocular cam...

This paper addresses appearance-based robot localization in 2D with a sparse, lightweight map of the environment composed of descriptor–pose image pairs. Based on previous research in the field, we assume that image descriptors are samples of a low-dimensional Descriptor Manifold that is locally articulated by the camera pose. We propose a piecewis...

The simulation of how a gas disperses in a environment is a necessary asset for the development of olfaction-based autonomous agents. A variety of simulators already exist for this purpose, but none of them allows for a sufficiently convenient integration with other types of sensing (such as vision), which hinders the development of advanced, multi...

The fusion of visual and inertial measurements is becoming more and more popular in the robotics community since both sources of information complement well each other. However, in order to perform this fusion, the biases of the Inertial Measurement Unit (IMU) as well as the direction of gravity must be initialized first. Additionally, in case of a...

Gas source localization ( GSL ) by an olfactory robot is a research field with a great potential for applications but also with numerous unsolved challenges, particularly when the search must take place in realistic, indoor environments that feature obstacles and turbulent airflows. In this work, we present a new probabilistic GSL method for a ter...

In this paper we present the first fast optimality certifier for the non-minimal version of the Relative Pose problem for calibrated cameras from epipolar constraints. The proposed certifier is based on Lagrangian duality and relies on a novel closed-form expression for dual points. We also leverage an efficient solver that performs local optimizat...

The Relative Pose problem (RPp) for cameras aims to estimate the relative orientation and translation (pose) given a set of pair-wise feature correspondences between two central and calibrated cameras. The RPp is stated as an optimization problem where the squared, normalized epipolar error is minimized over the set of normalized essential matrices...

Featured Application A novel tool to successfully bolster underpinning concepts and techniques in robotics courses in the context of STEM education. Abstract Jupyter notebooks are recently emerging as a valuable pedagogical resource in academy, being adopted in educational institutions worldwide. This is mainly due to their ability to combine the...

Face recognition is a technology with great potential in the field of robotics, due to its prominent role in human-robot interaction (HRI). This interaction is a keystone for the successful deployment of robots in areas requiring a customized assistance like education and healthcare, or assisting humans in everyday tasks. These unconstrained enviro...

Low back pain (LBP) can lead to motor control disturbance which can be one of the causes of reoccurrence of the complaint. It is important to improve our knowledge of movement related disturbances during assessment in LBP and to classify patients according to the severity. The aim of this study is to present differences in kinematic variables using...

Computer vision is a trending and innovative field that grows in parallel with IA, particularly since the appearance of deep learning. Intending to facilitate the learning of concepts in Computer Vision and to bring together practice and theory in a natural way, this paper presents a collection of interactive documents introducing the main computer...

This article presents a visual–inertial dataset gathered in indoor and outdoor scenarios with a handheld custom sensor rig, for over 80 min in total. The dataset contains hardware-synchronized data from a commercial stereo camera (Bumblebee®2), a custom stereo rig, and an inertial measurement unit. The most distinctive feature of this dataset is th...

This work addresses 2D gas and wind distribution mapping with a mobile robot for real-time applications. Our proposal seeks to estimate how gases released in the environment are distributed from a set of sparse and uncertain gas-concentration and wind-flow measurements; such that by exploiting the high correlation between these two magnitudes we ma...

In this paper we present the first fast optimality certifier for the non-minimal version of the Relative Pose problem for calibrated cameras from epipolar constraints. The proposed certifier is based on Lagrangian duality and relies on a novel closed-form expression for dual points. We also leverage an efficient solver that performs local optimizat...

The Industry 4.0 paradigm is being increasingly adopted in the production, distribution and commercialization chains worldwide. The integration of the cutting-edge techniques behind it entails a deep and complex revolution – changing from scheduled-based processes to smart, reactive ones – that has to be thoroughly applied at different levels. Aimi...

In domestic robotics, passing through narrow areas becomes critical for safe and effective robot navigation. Due to factors like sensor noise or miscalibration, even if the free space is sufficient for the robot to pass through, it may not see enough clearance to navigate, hence limiting its operational space. An approach to facing this is to inser...

Human–Robot interaction represents a cornerstone of mobile robotics, especially within the field of social robots. In this context, user localization becomes of crucial importance for the interaction. This work investigates the capabilities of wide field-of-view RGB cameras to estimate the 3D position and orientation (i.e., the pose) of a user in t...

Depth cameras, typically in RGB-D configurations, are common devices in mobile robotic platforms given their appealing features: high frequency and resolution, low price and power requirements, among others. These sensors may come with significant, non-linear errors in the depth measurements that jeopardize robot tasks, like free-space detection, e...

Olfaction is a valuable source of information about the environment that has not been sufficiently exploited in mobile robotics yet. Certainly, odor information can contribute to other sensing modalities, e.g., vision, to accomplish high-level robot activities, such as task planning or execution in human environments. This paper organizes and puts...

Depth cameras, typically in RGB-D configurations, are common devices in mobile robotic platforms given their appealing features: high frequency and resolution, low price and power requirements, among others. These sensors may come with significant, non-linear errors in the depth measurements that jeopardize robot tasks, like free-space detection, e...

In order to fuse measurements from multiple sensors mounted on a mobile robot, it is needed to express them in a common reference system through their relative spatial transformations. In this letter, we present a method to estimate the full 6-DoF extrinsic calibration parameters of multiple heterogeneous sensors (Lidars, depth, and RGB cameras) su...

In order to fuse measurements from multiple sensors mounted on a mobile robot, it is needed to express them in a common reference system through their relative spatial transformations. In this paper, we present a method to estimate the full 6DoF extrinsic calibration parameters of multiple heterogeneous sensors (Lidars, Depth and RGB cameras) suita...

In robotics, semantic mapping refers to the construction of a rich representation of the environment that includes high level information needed by the robot to accomplish its tasks. Building a semantic map requires algorithms to process sensor data at different levels: geometric, topological and object detections/categories, which must be integrat...

Object recognition is a cornerstone task in autonomous and/or assistance systems like robots, autonomous vehicles, or those assisting to visually impaired, aiming to achieve a certain level of understanding of their surroundings. Probabilistic models, such as Conditional Random Fields (CRFs), have been successfully applied to this end given their a...

MoveCare develops and field tests an innovative multi-actor platform that supports the independent living of the elder at home by monitoring, assist and promoting activities to counteract decline and social exclusion. It is being developed under H2020 framework and it comprises 3 hierarchical layers: (1) A service layer provides monitoring and inte...

In this paper we propose a solution to endow a mobile robot with the ability to approach humans in a safe and socially acceptable way. Our proposal focuses on real world indoor environments where the usual presence of multiple humans and obstacles notably rise the complexity of the approach action. We first deal with the problem of accurately estim...

Holistic Image Descriptors (HIDs) are compact representations of a whole image that, being suitable for Place Recognition, are not appropriate for accurate Visual Localization. The most successful HIDs are those extracted from Convolutional Neural Networks (CNNs) like VGG, ResNet, InceptionV4 or NetVLAD. Very recently, the equivariance property has...

This paper presents an interactive game system integrated in a mobile robot. Our proposal enables assistant robots to provide cognitive and entertainment games to elders with the aim of improving skills like memory, reflexes, or hand-eye coordination, which are jeopardised in aged people. To gain in user acceptance, the game is virtually recreated...

Out of all the components of a mobile robot, its sensorial system is undoubtedly among the most critical ones when operating in real environments. Until now, these sensorial systems mostly relied on range sensors (laser scanner, sonar, active triangulation) and cameras. While electronic noses have barely been employed, they can provide a complement...

A world model representing the elements in a robot's environment needs to maintain a correspondence between the objects being observed and their internal representations, which is known as the anchoring problem. Anchoring is a key aspect for an intelligent robot operation, since it enables high-level functions such as task planning and execution. T...

Olfactory telerobotics consists in augmenting the sensing capabilities of a conventional teleoperated mobile-robot to acquire information about the surrounding air (i.e. smell, wind-speed, etc.) in addition to the usual audio and video streams. Conceptually, this allows for new and improved applications, among which the most relevant are those rela...

This paper addresses the localization of a gas emission source within a real-world human environment with a mobile robot. Our approach is based on an efficient and coherent system that fuses different sensor modalities (i.e., vision and chemical sensing) to exploit, for the first time, the semantic relationships among the detected gases and the obj...