Project

MDL4EO: Machine and Deep Learning for Earth Observation data

Goal: Modern Earth Observation systems provide huge amount of data from different sensors at different temporal, spatial and spectral resolutions. Such amount of information is commonly represented by means of multispectral imagery and, due to its complexity, it requires new techniques and method to be correctly exploited to extract valuable knowledge.

The MDL4EO team (Machine and Deep Learning for Earth Observation) at the UMR TETIS (Montpellier, France) has the objective to scientifically contribute to this new era providing AI methods and algorithms to extract valuable knowledge from modern Earth Observation Data. The amount of data being collected by remote sensors is accelerating rapidly and we cannot manage them manually, this is why machine/deep learning lends itself well to remote sensing. More in detail, some of the research questions of the MDL4EO team are the follows:

- How to intelligently exploit Time Series of Satellite Images to leverage temporal dynamics
- How to combine/fusion together multi spectral/temporal/resolution/sensor information with the objective to add value to the information thanks to the combination of multi source
- How to transfer knowledge from different geographical Area: transfer land cover classification model from one site (i.e. France) to another one geographically distant (i.e. Africa).

It’s time to fill the gap between Remote Sensing and AI. MDL4EO is working on that direction bringing together different expertises: Data Science, Computer Vision, Machine Learning, Remote Sensing and Geoinformatics.

Updates
0 new
1
Recommendations
0 new
1
Followers
0 new
119
Reads
5 new
1326

Project log

Dino Ienco
added a research item
Forest ecosystems play a fundamental role in natural balances and climate mechanisms through their contribution to global carbon storage. Their sustainable management and conservation is crucial in the current context of global warming and biodiversity conservation. To tackle such challenges, earth observation data have been identified as a valuable source of information. While earth observation data constitute an unprecedented opportunity to monitor forest ecosystems, its effective exploitation still poses serious challenges since multimodal information needs to be combined to describe complex natural phenomena. To deal with this particular issue in the context of structure and biophysical variables estimation for forest characterization, we propose a new deep learning-based fusion strategy to combine together high density three-dimensional (3-D) point clouds acquired by airborne laser scanning with high-resolution optical imagery. In order to manage and fully exploit the available multimodal information, we implement a two-branch late fusion deep learning architecture taking advantage of the specificity of each modality. On the one hand, a 2-D CNN branch is devoted to the analysis of Sentinel-2 time series data, and on the other hand, a multilayer perceptron branch is dedicated to the processing of LiDAR-derived information. The performance of our framework is evaluated on two forest variables of interest: total volume and basal area at stand level. The obtained results underline that the availability of multimodal remote sensing data is not a direct synonym of performance improvements but, the way in which they are combined together is of paramount importance.
Dinh HO TONG MINH
added a research item
Recently, the frequency of natural and environmental disasters has increased significantly, causing constant changes on the Earth's surface. Synthetic Aperture Radar (SAR) data have been proved to be useful for operational change monitoring tasks. The multiscale framework presented in this paper aims at detecting and analyzing changes using SAR image time series composed of large-size images. Spatio-temporal changes are initially detected at the subimage scale analysis stage to determine regions and image acquisition dates related to the change occurrence. Detailed changes are then identified at the pixel scale analysis stage between selected acquisitions at each recognized region. This framework was used for flood monitoring over a large area along the central coast of Vietnam (from Thua Thien Hue province to Quang Nam province). We exploited a Sentinel-1 image time series acquired during two rainy seasons and typhoon seasons in the Western Pacific (from September to December of the two years 2017 and 2018). The proposed framework detected flooded areas with a high overall accuracy of 90.4% and could analyze different types of changes that occurred in this time series, i.e. dirac, periodic, chaotic changes, and temporal stability.
Dino Ienco
added a research item
In Sub-Saharan Africa, smallholder farms play a key role in agriculture, occupyingmost of the agricultural land. Design policies for increasing smallholder productivityremains a safe way to establish sustainable food systems and boost local economies.However, efforts are still needed in order to achieve accurate and timely monitor-ing in smallholder farming systems. With the advent of modern Earth Observationprogrammes such as the Sentinel satellites, that provide quasi-synchronous and high-resolution multi-source information over any area of the continental surfaces, newopportunities are opened up to accurately map crop yields in smallholder farmingsystems. This study intends to estimate and forecast millet yields in central Senegal,making the use of multi-source (Synthetic-aperture radar (SAR) and optical) imagetime series and state-of-the-art machine learning models. A Random Forest (RF)model explained up to 50% of the millet yield variability while deep learning modelssuch as Convolutional Neural Network (CNN) showed promise results but performedlower. We also found that the concatenation of SAR polarizations and vegetationindices improve our crop yield modelling but such improvement was tightly relatedto the modelling approach, namely RF and CNN. Using RF to forecast millet yields,we achieved stable and satisfactory accuracy two weeks before the harvest period.
Dino Ienco
added a research item
Extensive research studies have been conducted in recent years to exploit the complementarity among multi-sensor (or multi-modal) remote sensing data for prominent applications such as land cover mapping. In order to make a step further with respect to previous studies which investigate multi-temporal SAR and optical data or multi-temporal/multi-scale optical combinations, here we propose a deep learning framework that simultaneously integrates all these input sources, specifically multi-temporal SAR/optical data and fine scale optical information at their native temporal and spatial resolutions.Our proposal relies on a patch-based multi-branch convolutional neural network (CNN) that exploits different per source encode rsto deal with the specificity of the input signals. In addition, we introduce a new self-distillation strategy to boost the per source analyses and exploit the interplay among the different input sources. This new strategy leverages the final prediction of the multi-source framework to guide the learning of the per sourceCNN encoders supporting the network to learn from itself.Experiments are carried out on two real world benchmarks, namely theReunion island(a french overseas department) and the Dordogne study site (a southwest department in France)where the annotated reference data were collected under operational constraints (sparsely annotated ground truth data).Obtained results, providing an overall classification accuracy of about 94% (resp. 88%) on theReunion island (resp. theDordogne) study site highlight the effectiveness of our framework based on CNNs and self-distillation to combine heterogeneous multi-sensor remote sensing data and confirm the benefit of multi-modal analysis for downstream tasks such as land cover mapping
Dino Ienco
added 2 research items
Full waveform (FW) LiDAR systems have proven their effectiveness to map forest biophysical variables in the last two decades, owing to their ability of measuring, with high accuracy, forest vertical structures. The Global Ecosystem Dynamics Investigation (GEDI) system on board the International Space Station (ISS) is the latest FW spaceborne LiDAR instrument for the continuous observation of Earth's forests. FW systems rely on very sophisticated pre-processing steps to generate a priori metrics in order to leverage their capabilities for the accurate estimation of the aforementioned forest characteristics. The ever-expanding volume of acquired GEDI data, which to date comprises more than 25 billion acquired unfiltered shots, and along with the pre-processed data, amounting to more than 90 TB of data, raises new challenges in terms of adapted preprocessing methods for the suitable exploitation of such a huge and complex amount of LiDAR data. To overcome the issues related to the generation of relevant metrics from GEDI data, we propose a new metric-free approach to estimate canopy dominant heights (Hdom) and wood volume (V) of Eucalyptus plantations over five different regions in Brazil. To avoid metric computation, we leverage deep learning techniques and, more in detail, convolutional neural networks with the aim to analyze the GEDI Level 1B geolocated waveforms. Performance comparisons were conducted between four convolutional neural network (CNN) variants using GEDI waveform data (either untouched, or subsetted) and a metric based Random Forest regressor (RF). Additionally, we tested if our framework can improve the generalization of the models to different distant regions. First, the models were trained using data from all the study regions. Cross validated results showed that the CNN based models compared well against their RF counterpart for both Hdom and V. The RMSE on the estimation of Hdom from the CNN based models varied between 1.54 and 1.94 m with a coefficient of determination (R2) between 0.86 and 0.91, while the RF model produced an accuracy on Hdom estimates of 1.45 m (R2 = 0.92). For V, CNN based estimations ranged from 27.76 to 33.33 m3.ha−1 (R2 between 0.82 and 0.88), while for RF, the RMSE was 27.61 m3.ha−1 (R2 = 0.88). Next, model generalization was assessed by means of a spatial transfer experiment. For Hdom, both the CNN and RF approaches showed similar performances to a global model, however, the CNN based approach showed higher variability on the estimation accuracy, and the variability was related to the forest structure between the trained and tested data (similar tree heights yield better accuracies). For the estimation of V, considering both approaches, the accuracy was dependent on the allometric relationship between Hdom and V in the training and testing regions while lower accuracies on V were obtained when the testing and training regions exhibited a different allometric relationship.
Deep Neural Networks (DNNs) are getting increasing attention to deal with Land Cover Classification (LCC) relying on Satellite Image Time Series (SITS). Though high performances can be achieved, the rationale of a prediction yielded by a DNN often remains unclear. An architecture expressing predictions with respect to input channels is thus proposed in this paper. It relies on convolutional layers and an attention mechanism weighting the importance of each channel in the final classification decision. The correlation between channels is taken into account to set up shared kernels and lower model complexity. Experiments based on a Sentinel-2 SITS show promising results.
Dino Ienco
added a research item
Satellite image time series (SITS) collected by modern Earth Observation (EO) systems represent a valuable source of information that supports several tasks related to the monitoring of the Earth surface dynamics over large areas. A main challenge is then to design methods able to leverage the complementarity between the temporal dynamics and the spatial patterns that characterize these data structures. Focusing on land cover classification (or mapping) tasks, the majority of approaches dealing with SITS data only considers the temporal dimension, while the integration of the spatial context is frequently neglected. In this work, we propose an attentive spatial temporal graph convolutional neural network that exploits both spatial and temporal dimensions in SITS. Despite the fact that this neural network model is well suited to deal with spatio-temporal information, this is the first work that considers it for the analysis of SITS data. Experiments are conducted on two study areas characterized by different land cover landscapes and real-world operational constraints (i.e., limited labeled data due to acquisition costs). The results show that our model consistently outperforms all the competing methods obtaining a performance gain, in terms of F-Measure, of at least 5 points with respect to the best competing approaches on both benchmarks.
Dino Ienco
added a research item
The unprecedented possibility to acquire high resolution Satellite Image Time Series (SITS) data is opening new opportunities to monitor the different aspects of the Earth Surface but, at the same time, it is raising up new challenges in term of suitable methods to analyze and exploit such huge amount of rich image data. One of the main tasks associated to SITS data analysis is related to land cover mapping. Due to operational constraints, the collected label information is often limited in volume and obtained at coarse granularity level carrying out inexact and weak knowledge that can affect the whole process. To cope with such issues, in the context of object-based SITS land cover mapping, we propose a new deep learning framework, named TASSEL (aTtentive weAkly Supervised Satellite image time sEries cLassifier), to deal with the weak supervision provided by the coarse granularity labels. Our framework exploits the multifaceted information conveyed by the object-based representation considering object components instead of aggregated object statistics. Furthermore, our framework also produces an additional outcome that supports the model interpretability. Quantitative and qualitative experimental evaluations are carried out on two real-world scenarios. Results indicate that not only TASSEL outperforms the competing approaches in terms of predictive performances, but it also produces valuable extra information that can be practically exploited to interpret model decisions.
Louise Leroux
added a research item
European satellite missions Sentinel-1 (S1) and Sentinel-2 (S2) provide at high spatial resolution and high revisit time, respectively, radar and optical images that support a wide range of Earth surface monitoring tasks, such as Land Use/Land Cover mapping. A long-standing challenge in the remote sensing community is about how to efficiently exploit multiple sources of information and leverage their complementarity, in order to obtain the most out of radar and optical data. In this work, we propose to deal with land cover mapping in an object-based image analysis (OBIA) setting via a deep learning framework designed to leverage the multi-source complementarity provided by radar and optical satellite image time series (SITS). The proposed architecture is based on an extension of Recurrent Neural Network (RNN) enriched via a modified attention mechanism capable to fit the specificity of SITS data. Our framework also integrates a pretraining strategy that allows to exploit specific domain knowledge, shaped as hierarchy over the set of land cover classes, to guide the model training. Thorough experimental evaluations, involving several competitive approaches were conducted on two study sites, namely the Reunion island and a part of the Senegalese groundnut basin. Classification results, 79% of global accuracy on the Reunion island and 90% on the Senegalese site, respectively, have demonstrated the suitability of the proposal.
Dino Ienco
added a research item
Nowadays, modern Earth Observation systems continuously collect massive amounts of satellite information. The unprecedented possibility to acquire high resolution Satellite Image Time Series (SITS) data (series of images with high revisit time period on the same geographical area) is opening new opportunities to monitor the different aspects of the Earth Surface but, at the same time, it is raising up new challenges in term of suitable methods to analyze and exploit such huge amount of rich and complex image data. One of the main task associated to SITS data analysis is related to land cover mapping where satellite data are exploited via learning methods to recover the Earth Surface status aka the corresponding land cover classes. Due to operational constraints, the collected label information, on which machine learning strategies are trained, is often limited in volume and obtained at coarse granularity carrying out inexact and weak knowledge that can affect the whole process. To cope with such issues, in the context of object-based SITS land cover mapping, we propose a new deep learning framework, named TASSEL (aTtentive weAkly Supervised Satellite image time sEries cLassifier), that is able to intelligently exploit the weak supervision provided by the coarse granularity labels. Furthermore, our framework also produces an additional side-information that supports the model interpretability with the aim to make the black box gray. Such side-information allows to associate spatial interpretation to the model decision via visual inspection.
Dino Ienco
added 2 research items
Nowadays, huge volume of satellite images, via the different Earth Observation missions, are constantly acquired and they constitute a valuable source of information for the analysis of spatio-temporal phenomena. However, it can be challenging to obtain reference data associated to such images to deal with land use or land cover changes as often the nature of the phenomena under study is not known a priori. With the aim to deal with satellite image analysis, considering a real world scenario where reference data cannot be available, in this paper, we present a novel end-to-end unsupervised approach for change detection and clustering for satellite image time series (SITS). In the proposed framework, we firstly create bi-temporal change masks for every couple of consecutive images using neural network autoencoders. Then, we associate the extracted changes to different spatial objects. The objects sharing the same geographical location are combined in spatio-temporal evolution graphs that are finally clustered accordingly to the type of change process with gated recurrent unit (GRU) autoencoderbased model. The proposed approach was assessed on two realworld SITS data supplying promising results.
Due to the proliferation of Earth Observation programmes, information at different spatial, spectral and temporal resolution is collected by means of various sensors (optical, radar, hyperspectral, LiDAR, etc.). Despite such abundance of information, it is not always possible to obtain a complete coverage of the same area (especially for large ones) from all the different sensors due to: (i) atmospheric conditions and/or (ii) acquisition cost. In this context of data (or modalities) misalignment, only part of the area under consideration could be covered by the different sensors (modalities). Unfortunately, standard machine learning approaches commonly employed in operational Earth monitoring systems require consistency between training and test data (i.e., they need to match the same information schema). Such a constraint limits the use of additional fruitful information, i.e., information coming from a particular sensor that may be available at training but not at test time. Recently, a framework able to manage such information misalignment between training and test information is proposed under the name of Generalized Knowledge Distillation (GKD). With the aim to provide a proof of concept of GKD in the context of multi-source Earth Observation analysis, here we provide a Generalized Knowledge Distillation framework for land use land cover mapping involving radar (Sentinel-1) and optical (Sentinel-2) satellite image time series data (SITS). Considering that part of the optical information may not be available due to bad atmospheric conditions, we make the assumption that radar SITS are always available (at both training and test time) while optical SITS are only accessible when the model is learnt (i.e., it is considered as privileged information). Evaluations are carried out on a real-world study area in the southwest of France, namely Dordogne, considering a mapping task involving seven different land use land cover classes. Experimental results underline how the additional (privileged) information ameliorates the results of the radar based classification with a main gain on the agricultural classes.
Dino Ienco
added a research item
Recurrent Neural Networks (RNNs) can be seriously impacted by the initial parameters assignment, which may result in poor generalization performances on new unseen data. With the objective to tackle this crucial issue, in the context of RNN based classification, we propose a new supervised layer-wise pretraining strategy to initialize network parameters. The proposed approach leverages a data-aware strategy that sets up a taxonomy of classification problems automatically derived by the model behavior. To the best of our knowledge, despite the great interest in RNN-based classification, this is the first data-aware strategy dealing with the initialization of such models. The proposed strategy has been tested on four benchmarks coming from two different domains, i.e., Speech Recognition and Remote Sensing. Results underline the significance of our approach and point out that data-aware strategies positively support the initialization of Recurrent Neural Network based classification models.
Dino Ienco
added a research item
This letter proposes a deep learning model to deal with the spatial transfer challenge for the mapping of irrigated areas through the analysis of Sentinel-1 data. First, a convolutional neural network (CNN) model called "Teacher Model" is trained on a source geographical area characterized by a huge volume of samples. Then, this model is transferred from the source area to a target area characterized by a limited number of samples. The transfer learning framework is based on a distill and refine strategy in which the teacher model is firstly distilled into a student model and, successively, refined by data samples coming from the target geographical area. The proposed strategy is compared to different approaches including a random forest (RF) classifier trained on the target dataset, a CNN trained on the source dataset and directly applied on the target area as well as several CNN classifiers trained on the target dataset. The evaluation of the performed transfer strategy shows that the "distill and refine" framework obtains the best performance compared to other competing approaches. The obtained findings represent a first step towards the understanding of the spatial transferability of deep learning models in the Earth Observation domain.
Dino Ienco
added a research item
European satellite missions Sentinel-1 (S1) and Sentinel-2 (S2) provide at highspatial resolution and high revisit time, respectively, radar and optical imagesthat support a wide range of Earth surface monitoring tasks such as LandUse/Land Cover mapping. A long-standing challenge in the remote sensingcommunity is about how to efficiently exploit multiple sources of information and leverage their complementary. In this particular case, get the most out ofradar and optical satellite image time series (SITS). Here, we propose to dealwith land cover mapping through a deep learning framework especially tailoredto leverage the multi-source complementarity provided by radar and opticalSITS. The proposed architecture is based on an extension of Recurrent NeuralNetwork (RNN) enriched via a customized attention mechanism capable to fitthe specificity of SITS data. In addition, we propose a new pretraining strategythat exploits domain expert knowledge to guide the model parameter initial-ization. Thorough experimental evaluations involving several machine learningcompetitors, on two contrasted study sites, have demonstrated the suitabilityof our new attention mechanism combined with the extend RNN model as wellas the benefit/limit to inject domain expert knowledge in the neural networktraining process.
Dino Ienco
added a research item
The huge amount of data currently produced by modern Earth Observation (EO) missions has allowed for the design of advanced machine learning techniques able to support complex Land Use/Land Cover (LULC) mapping tasks. The Copernicus programme developed by the European Space Agency provides, with missions such as Sentinel-1 (S1) and Sentinel-2 (S2), radar and optical (multi-spectral) imagery, respectively, at 10 m spatial resolution with revisit time around 5 days. Such high temporal resolution allows to collect Satellite Image Time Series (SITS) that support a plethora of Earth surface monitoring tasks. How to effectively combine the complementary information provided by such sensors remains an open problem in the remote sensing field. In this work, we propose a deep learning architecture to combine information coming from S1 and S2 time series, namely TWINNS (TWIn Neural Networks for Sentinel data), able to discover spatial and temporal dependencies in both types of SITS. The proposed architecture is devised to boost the land cover classification task by leveraging two levels of complementarity, i.e., the interplay between radar and optical SITS as well as the synergy between spatial and temporal dependencies. Experiments carried out on two study sites characterized by different land cover characteristics (i.e., the Koumbia site in Burkina Faso and Reunion Island, a overseas department of France in the Indian Ocean), demonstrate the significance of our proposal.
Remi Cresson
added a research item
Deep learning (DL) techniques are becoming increasingly important to solve a number of image processing tasks. Among common algorithms, convolutional neural network- and recurrent neural network-based systems achieve state-of-the-art results on satellite and aerial imagery in many applications. While these approaches are subject to scientific interest, there is currently no operational and generic implementation available at the user level for the remote sensing (RS) community. In this letter, we present a framework enabling the use of DL techniques with RS images and geospatial data. Our solution takes roots in two extensively used open-source libraries, the RS image processing library Orfeo ToolBox and the high-performance numerical computation library TensorFlow. It can apply deep nets without restriction on image size and is computationally efficient, regardless of hardware configuration.
Dino Ienco
added 3 research items
Modern Earth Observation (EO) systems produce huge volumes of images with the objective to monitor Earth surface. Due to the high revisit time of EO systems such the Sentinel-2 constellation, satellite image time series (SITS) are continuously produced allowing to improve the monitoring of spatio-temporal phenomena. How to efficiently analyze SITS considering both spectral and spatial information is still an open question in the remote sensing field. To deal with SITS classification, in this letter we propose a spatio-spectral classification framework that leverages mathematical morphology to extract spatial characteristics from SITS data and combines them with the already available spectral and temporal information. Experiments carried out on two study sites characterized by different heterogeneous land cover have demonstrated the significance of our proposal and the value to combine spatial as well as spectral information in the context of SITS land cover classification.
In this work we introduce and evaluate a deep learning model, mbCNN, that combines together satellite imagery and Volunteer Geographical Information (VGI) data to deal with different types of built-up surfaces. Differently from most of the previous works that only consider Urban/Non-Urban settings involving only one urban LULC class, here, we investigate the possibility to go a step further and distinguish among several urban land use classes: residential, industrial, sport fields and non-urban. Experiments on a real-world dataset covering the City of Montpellier (South of France) site are reported. Such results demonstrate the quality of Deep Learning approaches to deal with several types of Urban LULC mapping as well as the positive influence to integrate VGI knowledge in the process.
A major issue affecting optical imagery is the presence ofclouds. The need of cloud-free scenes at specific date is cru-cial in a number of operational monitoring applications. Onthe other hand, the cloud-insensitive SAR sensors are a solidasset and they provide orthogonal information with respect tooptical satellite, that enable the retrieval of information lostin optical images due to cloud cover. In the context of an in-creasing availability of both optical and SAR images, thank tothe Sentinel constellation, we propose a deep learning methodto reconstruct (gap-fill) optical data, polluted by cloud phe-nomena, exploiting multi-temporal SAR and optical images.
Dino Ienco
added 2 research items
Radar and Optical Satellite Image Time Series (SITS) are sources of information that are commonly employed to monitor earth surfaces for tasks related to ecology, agriculture, mobility, land management planning and land cover monitoring. Many studies have been conducted using one of the two sources, but how to smartly combine the complementary information provided by radar and optical SITS is still an open challenge. In this context, we propose a new neural architecture for the combination of Sentinel-1 (S1) and Sentinel-2 (S2) imagery at object level, applied to a real-world land cover classification task. Experiments carried out on the Reunion Island, a overseas department of France in the Indian Ocean, demonstrate the significance of our proposal.
The use of Very High Spatial Resolution (VHSR) imagery in remote sensing applications is nowadays a current practice whenever fine-scale monitoring of the earth’s surface is concerned. VHSR Land Cover classification, in particular, is currently a well-established tool to support decisions in several domains, including urban monitoring, agriculture, biodiversity, and environmental assessment. Additionally, land cover classification can be employed to annotate VHSR imagery with the aim of retrieving spatial statistics or areas with similar land cover. Modern VHSR sensors provide data at multiple spatial and spectral resolutions, most commonly as a couple of a higher-resolution single-band panchromatic (PAN) and a coarser multispectral (MS) imagery. In the typical land cover classification workflow, the multi-resolution input is preprocessed to generate a single multispectral image at the highest resolution available by means of a pan-sharpening process. Recently, deep learning approaches have shown the advantages of avoiding data preprocessing by letting machine learning algorithms automatically transform input data to best fit the classification task. Following this rationale, we here propose a new deep learning architecture to jointly use PAN and MS imagery for a direct classification without any prior image sharpening or resampling process. Our method, namely M u l t i R e s o L C C , consists of a two-branch end-to-end network which extracts features from each source at their native resolution and lately combine them to perform land cover classification at the PAN resolution. Experiments are carried out on two real-world scenarios over large areas with contrasted land cover characteristics. The experimental results underline the quality of our method while the characteristics of the proposed scenarios underline the applicability and the generality of our strategy in operational settings.
Dino Ienco
added a research item
Nowadays, modern Earth Observation systems continuously generate huge amounts of data. A notable example is represented by the Sentinel-2 mission, which provides images at high spatial resolution (up to 10m) with high temporal revisit period (every 5 days), which can be organized in Satellite Image Time Series (SITS). While the use of SITS has been proved to be beneficial in the context of Land Use/Land Cover (LULC) map generation, unfortunately, machine learning approaches commonly leveraged in remote sensing field fail to take advantage of spatio-temporal dependencies present in such data. Recently, new generation deep learning methods allowed to significantly advance research in this field. These approaches have generally focused on a single type of neural network, i.e., Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), which model different but complementary information: spatial autocorrelation (CNNs) and temporal dependencies (RNNs). In this work, we propose the first deep learning architecture for the analysis of SITS data, namely \method{} (DUal view Point deep Learning architecture for time series classificatiOn), that combines Convolutional and Recurrent neural networks to exploit their complementarity. Our hypothesis is that, since CNNs and RNNs capture different aspects of the data, a combination of both models would produce a more diverse and complete representation of the information for the underlying land cover classification task. Experiments carried out on two study sites characterized by different land cover characteristics (i.e., the \textit{Gard} site in France and the \textit{Reunion Island} in the Indian Ocean), demonstrate the significance of our proposal.
Dino Ienco
added 8 research items
Modern Earth Observation systems provide sensing data at different temporal and spatial resolutions. Among optical sensors, today the Sentinel-2 program supplies high-resolution temporal (every 5 days) and high spatial resolution (10m) images that can be useful to monitor land cover dynamics. On the other hand, Very High Spatial Resolution images (VHSR) are still an essential tool to figure out land cover mapping characterized by fine spatial patterns. Understand how to efficiently leverage these complementary sources of information together to deal with land cover mapping is still challenging. With the aim to tackle land cover mapping through the fusion of multi-temporal High Spatial Resolution and Very High Spatial Resolution satellite images, we propose an End-to-End Deep Learning framework, named M 3 F usion, able to leverage simultaneously the temporal knowledge contained in time series data as well as the fine spatial information available in VHSR information. Experiments carried out on the Reunion Island study area asses the quality of our proposal considering both quantitative and qualitative aspects.
Nowadays, modern earth observation programs produce huge volumes of satellite images time series (SITS) that can be useful to monitor geographical areas through time. How to efficiently analyze such kind of information is still an open question in the remote sensing field. Recently, deep learning methods proved suitable to deal with remote sensing data mainly for scene classification (i.e. Convolutional Neural Networks - CNNs - on single images) while only very few studies exist involving temporal deep learning approaches (i.e Recurrent Neural Networks - RNNs) to deal with remote sensing time series. In this letter we evaluate the ability of Recurrent Neural Networks, in particular the Long-Short Term Memory (LSTM) model, to perform land cover classification considering multi-temporal spatial data derived from a time series of satellite images. We carried out experiments on two different datasets considering both pixel-based and object-based classification. The obtained results show that Recurrent Neural Networks are competitive compared to state-of-the-art classifiers, and may outperform classical approaches in presence of low represented and/or highly mixed classes. We also show that using the alternative feature representation generated by LSTM can improve the performances of standard classifiers.
Mapping winter vegetation quality is a challenging problem in remote sensing. This is due to cloud coverage in winter periods, leading to a more intensive use of radar rather than optical images. The aim of this letter is to provide a better understanding of the capabilities of Sentinel-1 radar images for winter vegetation quality mapping through the use of deep learning techniques. Analysis is carried out on a multitemporal Sentinel-1 data over an area around Charentes-Maritimes, France. This data set was processed in order to produce an intensity radar data stack from October 2016 to February 2017. Two deep recurrent neural network (RNN)-based classifiers were employed. Our work revealed that the results of the proposed RNN models clearly outperformed classical machine learning approaches (support vector machine and random forest).
Dino Ienco
added an update
Welcome to the MDL4EO working group at UMR TETIS (https://tetis.teledetection.fr/index.php/fr/) laboratory in Montpellier, South of France. The main objective of this working group is to fill the gap between modern Machine learning techniques and Remote Sensing analysis.
Here, you can find the website of our group:
 
Dino Ienco
added a project goal
Modern Earth Observation systems provide huge amount of data from different sensors at different temporal, spatial and spectral resolutions. Such amount of information is commonly represented by means of multispectral imagery and, due to its complexity, it requires new techniques and method to be correctly exploited to extract valuable knowledge.
The MDL4EO team (Machine and Deep Learning for Earth Observation) at the UMR TETIS (Montpellier, France) has the objective to scientifically contribute to this new era providing AI methods and algorithms to extract valuable knowledge from modern Earth Observation Data. The amount of data being collected by remote sensors is accelerating rapidly and we cannot manage them manually, this is why machine/deep learning lends itself well to remote sensing. More in detail, some of the research questions of the MDL4EO team are the follows:
- How to intelligently exploit Time Series of Satellite Images to leverage temporal dynamics
- How to combine/fusion together multi spectral/temporal/resolution/sensor information with the objective to add value to the information thanks to the combination of multi source
- How to transfer knowledge from different geographical Area: transfer land cover classification model from one site (i.e. France) to another one geographically distant (i.e. Africa).
It’s time to fill the gap between Remote Sensing and AI. MDL4EO is working on that direction bringing together different expertises: Data Science, Computer Vision, Machine Learning, Remote Sensing and Geoinformatics.