Conference PaperPDF Available

Abstract and Figures

As the increasing volume of Earth Observation data being produced risks outpacing the capacity to downlink said data and storing it back on Earth, much effort has been poured into compression algorithms or on-board applications. Both are capable of significantly decreasing the amount of data to be downlinked and stored, but, until the publication of a paper earlier this year, no solution combining both ideas had been proposed. The paper in question introduces CORSA, a compression algorithm based on deep learning, having also the benefit of producing meaningful semantic representations usable in downstream AI-tasks. The compression and reconstruction capabilities of the algorithm, as well as the usability of the representations it produces for classification had been established in the original publication, but there had yet to be a quantitative assessment of the representation's usability for a downstream task such as semantic segmentation. The aim of the present paper is therefore to asses the usability of CORSA's representations for water detection applications. The paper compares tried methods, such as the NDWI, U-Net on multispectral data, to new methods using CORSA representations as a prior, such as U-Net on reconstructed multispectral data and neural networks using CORSA's representations directly as input. Sentinel-2 data and manual annotations were used during the exercise to ensure quality. The ensuing results show that the compression performed by CORSA does not negatively affect the efficiency of a U-Net, moreover we have created new networks using CORSA representations directly and exploiting their underlying hierarchical structure, which also perform water detection successfully. Hence, with this paper, we have demonstrated that a versatile and generic algorithm such as CORSA, efficient for both on-board compression and on-board processing in the realms of classification and semantic segmentation, is a viable solution to the increasing demands in data downlink and storage.
Content may be subject to copyright.
OBPDC 2022
8th INTERNATIONAL WORKSHOP ON ON-BOARD PAYLOAD DATA COMPRESSION (OBPDC 2022)
28-30 SEPTEMBER 2022
CORSA DEEP EARTH OBSERVATION SEMANTIC COMPRESSION APPLIED TO FLOOD DETECTION
Xenia Ivashkovych(1), Lisa Landuyt(2), Tanja Van Achteren(3)
(1) Flemish institute for technological Research (VITO),
Boeretang 200 | 2400 Mol | Belgium
Email: xenia.ivashkovych@vito.be
(2) Flemish institute for technological Research (VITO),
Boeretang 200 | 2400 Mol | Belgium
Email: lisa.landuyt@vito.be
(3) Flemish institute for technological Research (VITO),
Boeretang 200 | 2400 Mol | Belgium
Email: tanja.vanachteren@vito.be
ABSTRACT
As the increasing volume of Earth Observation data being produced risks outpacing the capacity to downlink said data
and storing it back on Earth, much effort has been poured into compression algorithms or on-board applications. Both are
capable of significantly decreasing the amount of data to be downlinked and stored, but, until the publication of a paper
earlier this year, no solution combining both ideas had been proposed. The paper in question introduces CORSA, a
compression algorithm based on deep learning, having also the benefit of producing meaningful semantic representations
usable in downstream AI-tasks. The compression and reconstruction capabilities of the algorithm, as well as the usability
of the representations it produces for classification had been established in the original publication, but there had yet to
be a quantitative assessment of the representation’s usability for a downstream task such as semantic segmentation. The
aim of the present paper is therefore to asses the usability of CORSA’s representations for water detection applications.
The paper compares tried methods, such as the NDWI, U-Net on multispectral data, to new methods using CORSA
representations as a prior, such as U-Net on reconstructed multispectral data and neural networks using CORSA’s
representations directly as input. Sentinel-2 data and manual annotations were used during the exercise to ensure quality.
The ensuing results show that the compression performed by CORSA does not negatively affect the efficiency of a U-
Net, moreover we have created new networks using CORSA representations directly and exploiting their underlying
hierarchical structure, which also perform water detection successfully. Hence, with this paper, we have demonstrated
that a versatile and generic algorithm such as CORSA, efficient for both on-board compression and on-board processing
in the realms of classification and semantic segmentation, is a viable solution to the increasing demands in data downlink
and storage.
1. INTRODUCTION
As the amount of remote sensing observations and their quality keeps increasing, downlinking and exploiting this vast
amount of data becomes more and more challenging. One of the most obvious answers to this challenge is reducing the
amount of data downlinked by going beyond lossless compression. The CCSDS reflects this change in focus introducing
the CSDSS 123.0-B-2[1], which is a new standard covering not only lossless but also near lossless compression for
hyperspectral and multispectral satellites.
Since then, many research teams have attempted to increase the compression ratio by various means, including machine
learning and deep learning (DL). Diego Valsesia and Enrico Magli have attempted to use a convolutional neural network
(CNN) on ground in order to decrease the noise distortion induced by lossy compression upon distortion to great effect
for lower rates[2]. Dimitri Lebedeff’s team have elaborated an on-board hyperspectral selective spatial/spectral
compression targeting clouded pixels with higher compression rates, which are unexploitable in the Earth Observation
domain, using an support vector machine (SVM)[3]. A DL-based cloud-detection network has been tested on Phi-Sat
2021[4] in order to transmit only non-clouded pixels down to Earth. All these algorithms however only have compression
as their target and are not exploiting the full range of possibilities that Artificial Intelligence (AI) offers.
OBPDC 2022
On the other hand, some teams have explored the possibility of unsupervised deep learning to enable change detection
for extreme events without the need for analysis on the ground[5]; but without explicitly addressing compression. In a
recent work, Bart Beusen and his team at VITO have attempted to elaborate a deep-learning algorithm that allows for
simultaneously competitive data compression and for direct down-stream AI tasks: CORSA[6]. CORSA can be
understood as both a compression algorithm and a generic model producing meaningful semantic representations for
downstream applications. One of the biggest drawbacks of DL-based applications is the need for vast amounts of
annotated data. Since CORSA already produces meaningful semantic representations, the labelled dataset necessary to
produce an downstream DL-based application can be divided by ten without affecting performance negatively, which
results in huge saving in terms of development time and labour. Moreover, since the algorithm can do both compression
and generic data pre-processing, CORSA can save space on-board for missions that require both downlinking and no-
board processing.
In this paper, we will showcase CORSA’s usability as a pretrained generic model for a specific downstream task: water
detection. First, we will show that the usage of compressed images does not negatively impact the efficiency of the
downstream task. Second, we will show that the representations generated by CORSA themselves are usable in this same
down-stream task.
2. RELATED WORKS
2.1. Water detection
Water presence, being a key surface parameter, both from an ecological and socio-economic point of view, has been
studied extensively and is included in a vast amount of literature. Feyisa et al.[7]identified four categories of common
water classification methods for multispectral imagery: thematic classification, linear unmixing, single -band thresholding
and two-band spectral water indices. The latter is by far the most established, and different indices have been developed
throughout the years[8], [9].
More recently, new DL based methods have been applied to the problem and proven to be even more successful than the
aforementioned water indices. Isikdogan et al.[10] reported an F1 score of 0.93 for their DeepWaterMapV2 approach
applied on Landsat 8 imagery compared to 0.72 using the Modified Normalized Difference Water Index (MNDWI), while
Wieland et al.[11]reported an overall accuracy of 0.99 compared to 0.93 for the Normalized Difference Water Index
(NDWI) on a test dataset of Landsat TM, ETM+, OLI and Sentinel-2 (S2) imagery. Finally, Mateo Garcia et al.[12]
significantly outperformed the NDWI using both the default and an optimized threshold (Jaccard score of 0.40 and 0.65
respectively) with both a U-Net and a simple convolutional neural network (Jaccard score of 0.72 and 0.71) on Sentinel-
2 imagery resampled to 10 m.
The data available on water detection makes it a suitable problem to assess the usability of the representations produced
by CORSA in a downstream semantic segmentation task.
2.2. CORSA
CORSA is an unsupervised compression deep learning algorithm developed by VITO earlier in this year 2022. It uses a
multi-level variational auto-encoder architecture to train an encoder and a decoder simultaneously. This training has the
effect of not only producing a decoder capable of reconstructing the original images with a high amount of fidelity but
also producing meaningful representations at the bottleneck.
The three-layer architecture, showcased in Fig. 1, offers a significant boost in reconstruction quality with little effect on
the compression ratio. This gain in performance can be understood by having seen these different levels of representation
not as an arbitrary division but as conceptual hierarchy of information. The highest level of representation, corresponding
to the smallest spatial map, contains information relevant to the background of an image, its larger structures. The lowest
level of representation, corresponding to the largest spatial map, contains information more relevant to the foreground of
an image, its details and finer structures. The middle level of the representation contains information lying somewhere
in-between.
OBPDC 2022
Fig. 1: CORSA architecture
CORSA has achieved a compression ratio of 24.38 for a peak signal-to-noise ratio (PSNR) or 69.80, a structural similarity
index measure (SSIM) of 0.95 and a mean squared error (MSE) of 0.01 on BGRNIR (blue, green, red, near infra-red) S2
images upon reconstruction. The high compression rate is achieved by quantizing the representations at the bottle-neck,
or more precisely mapping them to a code-book. Once every vector has been mapped, the indices are sent back to Earth,
meaning the codebook is needed both on-board and down on Earth for encoding and decoding. This code-book is trained
together with the encoders and decoders in order to retain a high degree of expressivity for quantized representations. It’s
these quantized representations that will be used further in the paper as inputs for water detection algorithms.
3. METHODOLOGY
In this paper, we have opted for a smaller but high-quality dataset on which multiple benchmarks and networks were
trained. In order to stay consistent with the literature, U-Net on uncompressed images and NDWI served as benchmarks
to measure our different networks against.
3.1. Dataset
As part of a wider project on the digitalisation of Flanders, it is this region, for which multiple high-resolution and
medium-resolution datasets are easily available, that has been chosen to test the expressivity of CORSA representations.
The Flemish Institute for Nature and Forest Research (INBO) manages a dataset of natural closed water bodies, hence
excluding industrial water surfaces and water courses, last updated in 2020 (Leyssen et al., 2020). The Basemap Flanders
or Grootschalig Referentie Bestand (GRB) also comprises water bodies and courses at very high resolution (scale 1/250
to 1/5000). In order to produce a ground truth of the highest possible quality for S2 imagery, we have chosen to rely on
not only the aforementioned static high-resolution datasets, but have also performed manual annotations based on the
INBO and GRB datasets, topographic information, high resolution aerial imagery acquired yearly in winter and 3-yearly
in summer as well as the S2 imagery itself.
The ground truth was generated for a selection of cloud-free acquisitions over seven Areas Of Interest (AOIs), shown in
Fig. 2. For AOIs used for testing (in orange) only timeframes with overlap between aerial imagery and cloud-free Sentinel-
2 images were selected. For AOIs used for training and validation, ten cloud-free S-2 images uniformly spread across the
year were selected. Note that since the ground truth was generated using high-resolution imagery, some water bodies
present on these datasets are invisible on the S-2 acquisitions, either due to being covered by vegetation, too shallow or
too small. These water bodies do exist in the dataset but are labelled as ‘difficult water’ and were considered as land
during this exercise.
OBPDC 2022
Fig. 2: AOIs selected across Flanders.
In order to match CORSA’s input specifications, 2100 120x120x4 px patches were selected for the training and validation
datasets. The maximal overlap between tiles was set to 80% and 90% of the patches were required to comprise water
pixels. For the latter, the minimal fraction of water pixels was set at 0.5%.
3.2. Network Architectures
In this paper, our goal is to answer two questions: does the compression of the data negatively affect the efficiency of the
downstream DL algorithm, and can the generic representations generated by CORSA be usable in downstream DL
algorithms?
To answer the first question, a U-Net on reconstructed images was benchmarked against a U-Net on uncompressed images
and the NDWI. The U-Net on reconstructed images is, in practice, equivalent to a pipeline consisting of a frozen CORSA
decoder and U-Net which takes CORSA representations as inputs. This network is called ‘Reconstruction + U-Net’.
To answer the second question, two types of networks taking CORSA representations as inputs have been tested. The
first type is called ‘Decoder + U-Net’, as it is equivalent to a pipeline consisting of a trainable CORSA decoder and a U-
Net, as shown in Fig. 3. Both are trained simultaneously as one network, which reflects a naïve approach to crafting a
downstream algorithm with the aforementioned inputs.
The second type of network aims to leverage the underlying hierarchical structure of CORSA representations in addition
to their semantic meaningfulness, as shown in Fig. 4. For this type of network, the architectural possibilities are very
numerous and only two examples, chosen for their simplicity, are showcased in this paper. The structures are still U-Net-
like but contain separate different input blocks’, ensemble of operations and layers, for different levels of representation.
These input blocks can have differing architectures depending on the level, but even given the same architecture are very
likely to have differing weights after training, depending on which level of representation (background, middle-ground,
foreground) the application leverages the most. The two networks used in this paper, respectively called ‘Crow-Net’ and
‘Shrike-Net’, each use the same input block across the three levels, the difference mainly lying in the type of input blocks
they use. These two networks aim to show the variety of ways in which the generic representations produced by CORSA
can be leveraged for downstream DL applications. They reflect a more sophisticated and flexible approach to our problem.
Figure 3: Architecture schematics for the ‘Decoder + U-Net’ network.
OBPDC 2022
Fig. 4: Architecture schematics blueprint for networks such as Crow-Net and Shrike-Net.
3.3. Training and Testing Procedures
All networks were trained with the same parameters and with the same training data to enable later comparisons. Power
scaling was applied to the input data to match the distribution on which CORSA was trained, no data augmentations were
applied, a decaying learning rate starting at 0.0001, early stopping and batch size of 128 were used during training.
For each network, the testing was performed as follows. Each test AOI was cut into overlapping patches. These patches
were predicted by a network and woven back together into the AOI by discarding the padding to minimize border effects.
The predicted image was then sharpened with a threshold of 0.5, anything above this threshold would be considered
water. The ground truth was then compared to the aforementioned image by means of 3 metrics: the Jaccard score (also
known as the Intersection over Union), precision and recall. As for the training, the pixels belonging to the ‘difficult
water’ class in the ground truth were considered as land. The scores were then aggregated using area-based weights.
In addition to assessing the proposed networks in this manner, they were benchmarked against two other methods: a U-
Net on the uncompressed S2 BGRNIR imagery and NDWI with the general threshold of 0.
4. EXPERIMENTAL RESULTS AND DISCUSSION
To give the reader a visual idea of each network’s performance, contingency maps for the AOI2 dated of the 1st of April
2019, the hardest or second-hardest region to predict, depending on the algorithm, will be discussed in this section. Fig.
5 shows the original inputs in BGR.
Fig. 5: S2 BGR uncompressed image
OBPDC 2022
Table 1. Result comparison between NDWI, U-Net on uncompressed images and U-Net on reconstructed images.
Algorithm
Metrics
Weight
(in parameters)
Input type
Jaccard
Precision
Recall
NDWI
55.8%
96.2%
56.3%
0
Original Image
U-Net
70.2%
83.2%
82.1%
353 665
Original Image
Reconstruction + U-Net
70.3%
82.6%
82.4%
1 212 165
CORSA Representations
First, we were concerned with whether or not the CORSA compression algorithm would negatively impact the water
detection exercise. As seen in Table 1, the result of the U-Net is significantly better than the NDWI, as expected from
reading the literature. More importantly however, the compression appears to have no significant impact on the efficiency
of the network. The bigger number of parameters necessary for the ‘Reconstruction + U-Net’ pipeline compared to the
regular U-Net is explained by the addition of the CORSA decoder. Both the U-Net that operated on uncompressed images
and the U-Net component in the ‘Reconstruction + U-Net’ pipeline are exactly the same network, with 3 levels and 32
starting filters. As further seen on Fig. 6, both networks have similar contingency maps, suggesting that the compression
and decompression of the images leaves the most relevant information for this water detection task intact.
Second, we wanted to assess how well the CORSA representations themselves could be used in different networks for
water detection. Table 2 shows that not only algorithms with CORSA representations as an input can perform just as
well as a U-Net on uncompressed images, but the algorithms that allow for separate encoding for different levels of
representation allow for significantly better performance. Adding separate pipelines for each level of representation
induces extra parameters however, which must partially account for the gain in accuracy. Here again, each network
comprises 3 levels and 32 starting parameters for better comparison.
Fig. 6: Contingency maps for ‘U-Net' (left) and ‘Reconstructed + U-Net’ (right)
OBPDC 2022
Table 2. Result comparison between different downstream DL algorithms.
Algorithm
Metrics
Weight
(in parameters)
Input type
Jaccard
Precision
Recall
Decoder + U-Net
70.3%
81.6%
83.5%
1 212 165
CORSA representations
Crow-Net
71.4%
81.6%
85.2%
1 791 165
CORSA representations
Shrike-Net
72.2%
83.8%
83.8%
3 168 345
CORSA representations
These experiments demonstrate that CORSA is both useful for compression and for producing generic semantically
meaningful representations for downstream DL applications. These results are likely to be reproductible for many other
applications.
The first aspect of CORSA that has remained unexplored in this paper, since the focus was on the application of water
detection, is the size of the representations produced by the algorithm. Larger representations’ positive effects on quality
has been documented in the original paper, but its effects on downstream applications have remained unexplored. It is
possible that a CORSA network producing larger representations would result in a boost in performance in the task of
water detection.
The second aspect of CORSA which might also have an impact on downstream applications is initial scaling. During
experiments, we have hesitated between different types of input scaling, an operation necessary for the inputs of neural
networks. We have settled for power scaling, more specifically the same one used for the training of CORSA in the
original paper, because the inputs to the CORSA pipeline needed to be consistent with those applied for the training of
the algorithm. Since consistency is also necessary across all networks in the experiment, the same power scaling was
chosen for every network. However, the most characteristic feature of water surfaces is the high energy absorption at NIR
wavelengths and beyond, leading to very low reflectance values, Power scaling leads to a more Gaussian-like distribution
and, as such, reduces the extremity of these values and consequently the difference between water and other surfaces.
Preliminary tests on visible water tend to show that the impact of scaling might be significant on the performance of the
downstream water detection algorithm. CORSA could benefit from having this effect assessed and quantified.
Fig. 7: Contingency map for ‘Decoder + U-Net' and ‘Shrike-Net
OBPDC 2022
5. CONCLUSION
We have shown that the representations that CORSA produces can be directly used in a downstream water detection task
without any loss in efficiency compared to the same task performed on raw images. With the compression capabilities of
CORSA being established, our observation further demonstrates that CORSA is a versatile framework that can be used
for multiple purposes simultaneously.
REFERENCES
[1] The Consultative Committee for Space Data Systems, Image Data Compression, vol. 1, 1 vols. 2017.
[2] D. Valsesia and E. Magli, “High-throughput Onboard Hyperspectral Image Compression with Ground-based CNN
Reconstruction,” IEEE Trans. Geosci. Remote Sensing, vol. 57, no. 12, pp. 95449553, Dec. 2019, doi:
10.1109/TGRS.2019.2927434.
[3] D. Lebedeff, M. F. Foulon, R. Camarero, R. Vitulli, and Y. Bobichon, ON-BOARD CLOUD DETECTION AND
SELECTIVE SPATIAL/SPECTRAL COMPRESSION BASED ON CCSDS 123.0-B-2 FOR HYPERSPECTRAL
MISSIONS,” p. 9.
[4] G. Giuffrida et al., “The Φ-Sat-1 Mission: The First On-Board Deep Neural Network Demonstrator for Satellite
Earth Observation,” IEEE Trans. Geosci. Remote Sensing, vol. 60, pp. 114, 2022, doi:
10.1109/TGRS.2021.3125567.
[5] V. Růžička et al., “Unsupervised Change Detection of Extreme Events Using ML On-Board.” arXiv, Nov. 04,
2021. Accessed: Sep. 22, 2022. [Online]. Available: http://arxiv.org/abs/2111.02995
[6] B. Beusen, X. Ivashkovyc, and T. V. Achteren, “Image compression using vector-quantized auto-encoders with
semantically meaningful feature extraction,” 2022.
[7] G. L. Feyisa, H. Meilby, R. Fensholt, and S. R. Proud, “Automated Water Extraction Index: A new technique for
surface water mapping using Landsat imagery,” Remote Sensing of Environment, vol. 140, pp. 2335, Jan. 2014,
doi: 10.1016/j.rse.2013.08.029.
[8] S. McFeeters, “Using the Normalized Difference Water Index (NDWI) within a Geographic Information System
to Detect Swimming Pools for Mosquito Abatement: A Practical Approach,” Remote Sensing, vol. 5, no. 7, pp.
35443561, 1996, doi: 10.3390/rs5073544.
[9] H. Xu, “Modification of normalised difference water index (NDWI) to enhance open water features in remotely
sensed imagery,” International Journal of Remote Sensing, vol. 27, no. 14, pp. 30253033, Jul. 2006, doi:
10.1080/01431160600589179.
[10] L. F. Isikdogan, A. Bovik, and P. Passalacqua, Seeing Through the Clouds With DeepWaterMap,” IEEE Geosci.
Remote Sensing Lett., vol. 17, no. 10, pp. 16621666, Oct. 2020, doi: 10.1109/LGRS.2019.2953261.
[11] M. Wieland and S. Martinis, “A Modular Processing Chain for Automated Flood Monitoring from Multi -Spectral
Satellite Data,” Remote Sensing, vol. 11, no. 19, p. 2330, Oct. 2019, doi: 10.3390/rs11192330.
[12] G. Mateo-Garcia et al., “Towards global flood mapping onboard low cost satellites with machine learning,” Sci
Rep, vol. 11, no. 1, p. 7249, Dec. 2021, doi: 10.1038/s41598-021-86650-z.
... CORSA, as a multi-purpose framework, does not only allow for compressed representations to be downlinked to Earth more efficiently, but extracts meaningful semantic information while processing its inputs, meaning its features can be used for a variety of downstream AI applications. Indeed, its compressed representations have been successfully used as is for water detection in [7]and for image classification in [2]. Meaning, it can reasonably be used to softly annotate data when compressing acquired imagery on-board, in addition to any other application which might be relevant in the context of its usage. ...
... Figure 7 shows the comparisons between the exercise performed on reconstructed images and by leveraging latent features of CORSA representations. More details on the method and crafting of these custom networks in [7]. Figure 6. ...
Conference Paper
Full-text available
To tackle both the lack of downlink capacity and the lack of labelling in the data currently generated by Earth Observation missions, the Flemish Institute for Technological Research (VITO) has developed an ecosystem of multipurpose foundation models for onboard compression and downstream Artificial Intelligence (AI) applications. This papers showcases CORSA’s performance in compressing 80 bands of ENMAP hyperspectral data (418-903nm), as well as dealing with resource constraints through network quantization and transfer learning. The paper also introduces the diverse downstream applications build on-top of CORSA. Keywords: AI, Compression, Vector-Quantized Variational Auto-Encoder, Earth Observation, Hyperspectral, Quantization-Aware Training, Transfer Learning, Foundation model.
... In the CORSA project [1,9,10], an ESA PhiLab EO Science for Society project, Beusen et al. demonstrated an AI method for lossy image compression using the concept of vector quantized variational auto-encoders [11]. The model used was based on the hierarchical VQVAE-2 [12], with some changes in the model architecture to achieve higher reconstruction accuracies, as described in [1]. ...
... semantic segmentation or classification, may be performed directly on the semantically-rich feature vectors, without the need to reconstruct the original image. Ivashkovych et al. [9] assessed the usability of CORSA's compressed image representations as direct input for water detection applications. Their results showed that the representations that CORSA produces can be directly used in a downstream water detection task without any loss in efficiency compared to the same task performed on raw images. ...
Conference Paper
Full-text available
In the CORSA project [1] we demonstrated an AI method for near-lossless image compression for Sentinel-2 data using the concept of vector quantized auto-encoders. As part of the MOVIQ project, this compression model was adapted to the domain of hyperspectral data [2] and optimized to run on-board. In these previous works, the train dataset and test dataset belonged to the same overall dataset, being BigEarthNet (Sentinel-2) for [1] and HyspecNet-11k (EnMAP) for [2]. We now test different variants of the model, including a quantized int8 version optimized for on-board processing. Furthermore, we investigate the transferability of a trained model for compression of EnMAP data to be used directly on PRISMA data that was not part of the training data set.
... CORSA is based on the VQ-VAE architecture [10] and designed for onboard compression of data. The feature maps generated by CORSA have shown themselves to be useful in other downstream tasks as well [11]. Our main contributions. ...
... Convolutions can also be implemented on edge devices in a straightforward manner. We hold on to the convolutional backbone for CORSA with 2 levels as used in previous works [1,2,11]. ...
... An innovative approach to addressing these challenges is the CORSA system [2], which introduces an AI-driven compression technique that preserves the integrity of data while producing efficient representations. CORSA demonstrates its ability to compress EO data, such as water detection networks, without sacrificing accuracy, highlighting its adaptability for on-board satellite processing. ...
Preprint
Full-text available
Earth observation (EO) systems are essential for mapping, catastrophe monitoring, and resource management, but they have trouble processing and sending large amounts of EO data efficiently, especially for specialized applications like agriculture and real-time disaster response. This paper presents a novel framework for semantic communication in EO satellite networks, aimed at enhancing data transmission efficiency and system performance through cognitive processing techniques. The proposed system leverages Discrete Task-Oriented Joint Source-Channel Coding (DT-JSCC) and Semantic Data Augmentation (SA) integrate cognitive semantic processing with inter-satellite links, enabling efficient analysis and transmission of multispectral imagery for improved object detection, pattern recognition, and real-time decision-making. Cognitive Semantic Augmentation (CSA) is introduced to enhance a system's capability to process and transmit semantic information, improving feature prioriti-zation, consistency, and adaptation to changing communication and application needs. The end-to-end architecture is designed for next-generation satellite networks, such as those supporting 6G, demonstrating significant improvements in fewer communication rounds and better accuracy over federated learning.
... Fundamentally, semantic EO data cubes enable a wider range of users to utilize comprehensive EO data. This study assesses CORSA [23] and [34], a novel compression technique driven by artificial intelligence that produces valuable data representations. The authors contrast conventional water detection methods with innovative techniques that use the compressed data and representations generated by CORSA. ...
Preprint
Full-text available
Earth Observation (EO) systems are crucial for cartography, disaster surveillance, and resource administration. Nonetheless, they encounter considerable obstacles in the processing and transmission of extensive data, especially in specialized domains such as precision agriculture and real-time disaster response. Earth observation satellites, outfitted with remote sensing technology, gather data from onboard sensors and IoT-enabled terrestrial objects, delivering important information remotely. Domain-adapted Large Language Models (LLMs) provide a solution by enabling the integration of raw and processed EO data. Through domain adaptation, LLMs improve the assimilation and analysis of many data sources, tackling the intricacies of specialized datasets in agriculture and disaster response. This data synthesis, directed by LLMs, enhances the precision and pertinence of conveyed information. This study provides a thorough examination of using semantic inference and deep learning for sophisticated EO systems. It presents an innovative architecture for semantic communication in EO satellite networks, designed to improve data transmission efficiency using semantic processing methodologies. Recent advancements in onboard processing technologies enable dependable, adaptable, and energy-efficient data management in orbit. These improvements guarantee reliable performance in adverse space circumstances using radiation-hardened and reconfigurable technology. Collectively, these advancements enable next-generation satellite missions with improved processing capabilities, crucial for operational flexibility and real-time decision-making in 6G satellite communication.
... S2 bands are compressed simultaneously. The same CORSA models have been shown to be useful foundation models for tasks such as super-resolved parcel delineation [4], flood detection [7], landcover classification and change detection [8]. ...
Conference Paper
Full-text available
The paper discusses the implementation of CORSA, a data compression model, on Nvidia Jetson hardware, specifically targeting applications in remote sensing. The need for onboard data compression is driven by the vast amount of data captured by hyperspectral sensors, which often exceeds satellite downlink capacities. CORSA, a lightweight, vector-quantized variational autoencoder (VQVAE), was initially developed for the European Space Agency (ESA) to compress Earth observation (EO) data, while also functioning as a foundation model for other AI applications like super-resolution, flood detection, and landcover classification.
... An innovative approach to addressing these challenges is the CORSA system [2], which introduces an AI-driven compression technique that preserves the integrity of data while producing efficient representations. CORSA demonstrates its ability to compress EO data, such as water detection networks, without sacrificing accuracy, highlighting its adaptability for on-board satellite processing. ...
Preprint
Full-text available
Earth observation (EO) systems are essential for mapping, catastrophe monitoring, and resource management, but they have trouble processing and sending large amounts of EO data efficiently, especially for specialized applications like agriculture and real-time disaster response. This paper presents a novel framework for semantic communication in EO satellite networks, aimed at enhancing data transmission efficiency and system performance through cognitive processing techniques. The proposed system leverages Discrete Task-Oriented Joint Source-Channel Coding (DT-JSCC) and Semantic Data Augmentation (SA) integrate cognitive semantic processing with inter-satellite links, enabling efficient analysis and transmission of multispectral imagery for improved object detection, pattern recognition, and real-time decision-making. Cognitive Semantic Augmentation (CSA) is introduced to enhance a system's capability to process and transmit semantic information, improving feature prioritization, consistency, and adaptation to changing communication and application needs. The end-to-end architecture is designed for next-generation satellite networks, such as those supporting 6G, demonstrating significant improvements in fewer communication rounds and better accuracy over federated learning.
... Fundamentally, semantic EO data cubes enable a wider range of users to utilize comprehensive EO data. This study evaluates CORSA [23] and [24], an innovative compression method powered by artificial intelligence that produces useful data representations at the same time. The authors compare traditional methods for detecting water with innovative ways that use the compressed data and representations of CORSA. ...
Preprint
Full-text available
Earth Observation (EO) systems play a crucial role in achieving Sustainable Development Goals by collecting and analyzing vital global data through satellite networks. These systems are essential for tasks like mapping, disaster monitoring, and resource management, but they face challenges in processing and transmitting large volumes of EO data, especially in specialized fields such as agriculture and real-time disaster response. Domain-adapted Large Language Models (LLMs) provide a promising solution by facilitating data fusion between extensive EO data and semantic EO data. By improving integration and interpretation of diverse datasets, LLMs address the challenges of processing specialized information in agriculture and disaster response applications. This fusion enhances the accuracy and relevance of transmitted data. This paper presents a framework for semantic communication in EO satellite networks, aimed at improving data transmission efficiency and overall system performance through cognitive processing techniques. The proposed system employs Discrete-Task-Oriented Source-Channel Coding (DT-JSCC) and Semantic Data Augmentation (SA) to focus on relevant information while minimizing communication overhead. By integrating cognitive semantic processing and inter-satellite links, the framework enhances the analysis and transmission of multispectral satellite imagery, improving object detection, pattern recognition, and real-time decision-making. The introduction of Cognitive Semantic Augmentation (CSA) allows satellites to process and transmit semantic information, boosting adaptability to changing environments and application needs. This end-to-end architecture is tailored for next-generation satellite networks, such as those supporting 6G, and demonstrates significant improvements in efficiency and accuracy.
Article
Full-text available
Applications such as disaster management enormously benefit from rapid availability of satellite observations. Traditionally, data analysis is performed on the ground after being transferred—downlinked—to a ground station. Constraints on the downlink capabilities, both in terms of data volume and timing, therefore heavily affect the response delay of any downstream application. In this paper, we introduce RaVÆn, a lightweight, unsupervised approach for change detection in satellite data based on Variational Auto-Encoders (VAEs), with the specific purpose of on-board deployment. RaVÆn pre-processes the sampled data directly on the satellite and flags changed areas to prioritise for downlink, shortening the response time. We verified the efficacy of our system on a dataset—which we release alongside this publication—composed of time series containing a catastrophic event, demonstrating that RaVÆn outperforms pixel-wise baselines. Finally, we tested our approach on resource-limited hardware for assessing computational and memory limitations, simulating deployment on real hardware.
Conference Paper
Full-text available
Modern earth observation missions collect imagery with ever more spatial and spectral detail, which causes the amount of data to increase very rapidly. Limitations in data downlink capacity have become a major bottleneck hindering the exploitation of the rich information in current and future satellite missions. In the CORSA project, an ESA PhiLab EO Science for Society project, we developed an AI method for lossy image compression which learns optimal data reduction tuned to the desired quality for the intended use. It generates a compact image representation containing deep feature vectors that are part of a codebook. The vectors can be represented by their codebook indices, which can be written to file as compact bit-arrays, suitable for downlinking. The full vectors are recreated afterwards from the indices using the same codebook, and the image can be reconstructed from the vectors using the decoder part of the model. The method is very suitable for satellite onboard data reduction, especially for smaller remote sensing missions capable of acquiring large volumes of multispectral and/or high spatial resolution image data, but limited in power and downlink budget. The compressed format can also be used to store image data efficiently in the ground segment.
Article
Full-text available
Artificial intelligence (AI) is paving the way for a new era of algorithms focusing directly on the information contained in the data, autonomously extracting relevant features for a given application. While the initial paradigm was to have these applications run by a server hosted processor, recent advances in microelectronics provide hardware accelerators with an efficient ratio between computation and energy consumption, enabling the implementation of AI algorithms “at the edge.” In this way only the meaningful and useful data are transmitted to the end-user, minimizing the required data bandwidth, and reducing the latency with respect to the cloud computing model. In recent years, European Space Agency (ESA) is promoting the development of disruptive innovative technologies on-board earth observation (EO) missions. In this field, the most advanced experiment to date is the Φ\Phi -sat-1, which has demonstrated the potential of artificial intelligence (AI) as a reliable and accurate tool for cloud detection on-board a hyperspectral imaging mission. The activities involved included demonstrating the robustness of the Intel Movidius Myriad 2 hardware accelerator against ionizing radiation, developing a Cloudscout segmentation neural network (NN), run on Myriad 2, to identify, classify, and eventually discard on-board the cloudy images, and assessing the innovative Hyperscout-2 hyperspectral sensor. This mission represents the first official attempt to successfully run an AI deep convolutional NN (CNN) directly inferencing on a dedicated accelerator on-board a satellite, opening the way for a new era of discovery and commercial applications driven by the deployment of on-board AI.
Article
Full-text available
Spaceborne Earth observation is a key technology for flood response, offering valuable information to decision makers on the ground. Very large constellations of small, nano satellites— ’CubeSats’ are a promising solution to reduce revisit time in disaster areas from days to hours. However, data transmission to ground receivers is limited by constraints on power and bandwidth of CubeSats. Onboard processing offers a solution to decrease the amount of data to transmit by reducing large sensor images to smaller data products. The ESA’s recent PhiSat-1 mission aims to facilitate the demonstration of this concept, providing the hardware capability to perform onboard processing by including a power-constrained machine learning accelerator and the software to run custom applications. This work demonstrates a flood segmentation algorithm that produces flood masks to be transmitted instead of the raw images, while running efficiently on the accelerator aboard the PhiSat-1. Our models are trained on WorldFloods: a newly compiled dataset of 119 globally verified flooding events from disaster response organizations, which we make available in a common format. We test the system on independent locations, demonstrating that it produces fast and accurate segmentation masks on the hardware accelerator, acting as a proof of concept for this approach.
Conference Paper
Full-text available
Future hyperspectral satellite Earth observation missions will provide numerous bands in VNIR and SWIR domain, with wide swath and small spatial sampling distance. The amount of data to be transmitted is large, and image compression becomes mandatory. Considering the significant presence of clouds hiding the ground in the acquisitions, this work explores some possibilities to increase the reduction of on-board data volume, and downlink data rate, with a selective compression applied to the clouds. CCSDS SLS-MHDC Working Group has established the CCSDS 123.0-B-2 recommended standard for a low-complexity data compression applied to multispectral and hyperspectral sensors. The recommendation provides an effective method for performing lossless or near-lossless compression, with a control of the error that can be band-dependent; however, this standard does not include a possibility for a selective compression in the spatial dimension. Three different selective schemes have been established, all based on the CCSDS recommendation, but with potential adaptations to get a lower bit rate on clouds. The on-board compression chain includes a cloud detection algorithm selected for its performance, but also for its high adaptability for future evolutions. The data reduction and impact on cloud radiometry have been assessed on hyperspectral AVIRIS images and using simulated scene representative of the future European Copernicus Hyperspectral Imaging Mission for the Environment (CHIME).
Article
Full-text available
Emergency responders frequently request satellite-based crisis information for flood monitoring to target the often-limited resources and to prioritize response actions throughout a disaster situation. We present a generic processing chain that covers all modules required for operational flood monitoring from multi-spectral satellite data. This includes data search, ingestion and preparation, water segmentation and mapping of flooded areas. Segmentation of the water extent is done by a convolutional neural network that has been trained on a global dataset of Landsat TM, ETM+, OLI and Sentinel-2 images. Clouds, cloud shadows and snow/ice are specifically handled by the network to remove potential biases from downstream analysis. Compared to previous work in this direction, the method does not require atmospheric correction or post-processing and does not rely on ancillary data. Our method achieves an Overall Accuracy (OA) of 0.93, Kappa of 0.87 and Dice coefficient of 0.90. It outperforms a widely used Random Forest classifier and a Normalized Difference Water Index (NDWI) threshold method. We introduce an adaptable reference water mask that is derived by time-series analysis of archive imagery to distinguish flood from permanent water. When tested against manually produced rapid mapping products for three flood disasters (Germany 2013, China 2016 and Peru 2017), the method achieves ≥ 0.92 OA, ≥ 0.86 Kappa and ≥ 0.90 Dice coefficient. Furthermore, we present a flood monitoring application centred on Bihar, India. The processing chain produces very high OA (0.94), Kappa (0.92) and Dice coefficient (0.97) and shows consistent performance throughout a monitoring period of one year that involves 19 Landsat OLI ( μKappa=0.92 and σKappa=0.07 ) and 61 Sentinel-2 images ( μKappa=0.92 , σKappa=0.05 ). Moreover, we show that the mean effective revisit period (considering cloud cover) can be improved significantly by multi-sensor combination (three days with Sentinel-1, Sentinel-2, and Landsat OLI).
Article
Full-text available
Compression of hyperspectral images onboard of spacecrafts is a tradeoff between the limited computational resources and the ever-growing spatial and spectral resolution of the optical instruments. As such, it requires low-complexity algorithms with good rate-distortion performance and high through-put. In recent years, the Consultative Committee for Space Data Systems (CCSDS) has focused on lossless and near-lossless compression approaches based on predictive coding, resulting in the recently published CCSDS 123.0-B-2 recommended standard. While the in-loop reconstruction of quantized prediction residuals provides excellent rate-distortion performance for the near-lossless operating mode, it significantly constrains the achievable throughput due to data dependencies. In this paper, we study the performance of a faster method based on prequantization of the image followed by a lossless predictive compressor. While this is well known to be suboptimal, one can exploit powerful signal models to reconstruct the image at the ground segment, recovering part of the suboptimality. In particular, we show that convolutional neural networks can be used for this task and that they can recover the whole SNR drop incurred at a bitrate of 2 bits per pixel.
Article
Full-text available
Mosquito-borne diseases affect millions of people worldwide. In the United States, since 1999, West Nile Virus (WNV) has infected 36,801 people and has caused the deaths of 1,580. In California, since 2002, nearly 3,600 people have been infected with WNV with an additional 124 fatalities. Analyses of remotely- and spatially-based data have proven to facilitate the study of mosquito-borne diseases, including WNV. This study proposes an efficient procedure to identify swimming pools that may serve as potential mosquito habitat. The procedure derives the Normalized Difference Water Index (NDWI) from high resolution, multi-spectral imagery to detect the presence of surface water, and then incorporates vector-based data layers within a GIS to identify residential land parcels with detectable water. This study compared the parcels identified as having water (535) with parcels known to have swimming pools (682) resulting in an accuracy of 78.4%. Nineteen of the 147 land parcels with swimming pools had backyards with enough vegetation to obscure the presence of a swimming pool from the satellite. The remaining 128 parcels lacked enough surface water for the NDWI to indicate them as actually having surface water. It is likely then that swimming pools, associated with such parcels, may have enough water in them to provide adequate habitat for mosquitoes, and so field inspection by mosquito abatement personnel would be justified.
Article
We present our next-generation surface water mapping model, DeepWaterMapV2, which uses improved model architecture, data set, and a training setup to create surface water maps at lower cost, with higher precision and recall. We designed DeepWaterMapV2 to be memory efficient for large inputs. Unlike earlier models, our new model is able to process a full Landsat scene in one-shot and without dividing the input into tiles. DeepWaterMapV2 is robust against a variety of natural and artificial perturbations in the input, such as noise, different sensor characteristics, and small clouds. Our model can even "see" through the clouds without relying on any active sensor data, in cases where the clouds do not fully obstruct the scene. Although we trained the model on Landsat-8 images only, it also supports data from a variety of other Earth observing satellites, including Landsat-5, Landsat-7, and Sentinel-2, without any further training or calibration. Our code and trained model are available at https://github.com/isikdogan/deepwatermap.