Conference PaperPDF Available

Machine Learning in Earth Observation Operations: A review

October 2021

Conference: 72nd International Astronautical Congress
At: Dubai

Authors:

Pablo Miralles

GTD International

Antonio F. Scannapieco

D-Orbit S.p.A.

Prerna Baranwal

National University of Singapore

Show all 19 authorsHide

Analysis of downlinked satellite imagery has undeniably benefited greatly from the ongoing Machine Learning (ML) revolution. Other aspects of the Earth Observation industry, despite being less prone to an extensive application of ML, are also following this trend. This work aims at presenting-in the form of a review of Machine Learning applied to Earth Observation Operations-such applications, the existing use cases, potential opportunities and pitfalls, and perceived gaps in research. A wide range of topics are discussed including mission planning, diagnosis, prognosis, and repair of faults, optimization of telecommunications, enhanced GNC, on-board image processing, and usage of Machine Learning models within platforms with limited compute and power capabilities. The review tackles all on-board and off-board applications of machine learning to Earth Observation with one notable exception: it omits all post-processing of payload data on the ground, a topic that has been studied extensively by past authors. This research was produced by a team of volunteers from the Small Satellite Project Group of the Space Generation Advisory Council.

Hardware Accelerators

…

Figures - uploaded by Pablo Miralles

Content may be subject to copyright.

Content uploaded by Pablo Miralles

Content may be subject to copyright.

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

IAC-21-B.1.4.4

Machine Learning in Earth Observation Operations: A review

Pablo Miralles*, Antonio Fulvio Scannapieco*, Nitya Jagadam*, Prerna Baranwal*, Bhavin Faldu*, Ruchita

Abhang*, Sahil Bhatia*, Sebastien Bonnart*, Ishita Bhatnagar*, Beenish Batul*, Pallavi Prasad*, Héctor

Ortega-González*, Harrish Joseph*, Harshal More*, Sondes Morchedi*, Aman Kumar Panda*, Marco

Zaccaria Di Fraia*, Daniel Wischert*, Daria Stepanova*

(*) In respective order:

GTD International, GTD Group, 99 Route d’Espagne, 31100, Toulouse, France, pmirallesr@gmail.com

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, antonio.scannapieco@spacegeneration.org

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, jagadamnitya@gmail.com

Department of Electrical & Electronics/Instrumentation & Department of Mathematics, Birla Institute of Technology

and Science (BITS), Pilani, Rajasthan - 333031, India, f2016568@pilani.bits-pilani.ac.in

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, bhavinfaldu6474@gmail.com

Department of Computer Science Engineering, Savitribai Phule Pune University, Pune, Maharashtra - 411052,

India, rabhang09@gmail.com

Department of Aerospace Engineering, University of Petroleum and Energy Studies, Energy Acres, Bidholi via

Premnagar, Dehradun, Uttarakhand - 248007, India, sahil2112.b@gmail.com

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, sgac@sbonnart.fr

Department of Chemical Engineering & Department of Physics, Birla Institute of Technology and Science (BITS),

Pilani, Rajasthan - 333031, India, f20180306@pilani.bits-pilani.ac.in

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, 23.prasad@gmail.com

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, ortegagonzalez.hector@gmail.com

Sapienza University of Rome, Via Eudossiana 18, 00184, Rome, Italy, harrishjagan@gmail.com

Sapienza University of Rome, Via Eudossiana 18, 00184, Rome, Italy, harshal.more@spacegeneration.org

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, sondesmorchedi@gmail.com

ISAE-École Nationale Supérieure de Mécanique et d’Aérotechnique, Téléport 2 – 1 avenue Clément Ader BP 40109

86961 Futuroscope Chasseneuil Cedex, Poitiers, France. aman.panda143@gmail.com

Centre for Electronic Warfare Information and Cyber, Cranfield University, The Defence Academy Of The UK,

Shrivenham, SN6 8LA, UK, marco.di-fraia@cranfield.ac.uk

Space Generation Advisory Council (SGAC), c/o European Space Policy Institute, Schwarzenbergplatz 6, 1030

Vienna, Austria, daniel.wischert@spacegeneration.org

German Orbital Systems GmbH, Manfred-Fuchs-Platz 2-4, 28359 Bremen, Germany,

daria.stepanova@skolkovotech.ru

Abstract

Analysis of downlinked satellite imagery has undeniably benefited greatly from the ongoing Machine Learning

(ML) revolution. Other aspects of the Earth Observation industry, despite being less prone to an extensive application

of ML, are also following this trend. This work aims at presenting - in the form of a review of Machine Learning

applied to Earth Observation Operations - such applications, the existing use cases, potential opportunities and

pitfalls, and perceived gaps in research. A wide range of topics are discussed including mission planning, diagnosis,

prognosis, and repair of faults, optimization of telecommunications, enhanced GNC, on-board image processing, and

usage of Machine Learning models within platforms with limited compute and power capabilities. The review

tackles all on-board and off-board applications of machine learning to Earth Observation with one notable exception:

it omits all post-processing of payload data on the ground, a topic that has been studied extensively by past authors.

IAC-21-B.1.4.4 Page 1 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

This research was produced by a team of volunteers from the Small Satellite Project Group of the Space Generation

Advisory Council.

Keywords: Machine Learning; Artificial Intelligence; Earth Observation; Remote Sensing; Neural Networks;

Computing

Acronyms / Abbreviations

Artificial Intelligence (AI); Machine Learning (ML);

Deep Learning (DL); Fault Detection, Isolation and

Recovery (FDIR); Guidance, Navigation and Control

(GNC); Neural Network (NN); Convolutional Neural

Network (CNN); Deep Neural Network (DNN);

Artificial Neural Network (ANN); Binarized Neural

Network (BNN); Bayesian Network (BN); Dynamic

Bayesian Network (DBN); National Aeronautics and

Space Administration (NASA); European Space

Agency (ESA); On-Board Computer (OBC); Earth

Observation (EO); Random Decision Forest (RDF);

Bayesian Thresholding (BT); Support Vector Machine

(SVM); Commercial off-the-shelf (COTS); Size,

Weight and Power (SWaP); Light Detection and

Ranging (LIDAR); System on a Chip (SoC); False

Positives (FP); Consultative Committee for Space Data

Systems (CCSDS); Global Navigation Satellite System

(GNSS); Global Positioning System (GPS);

Proportional - Integral (PI); Proportional - Integral -

Derivative (PID); Attitude Orbital Determination

System (AODS); Reinforcement Learning (RL);

Extended Kalman Filter (EKF); Random Forest (RF);

Attitude and Orbit Control System (AOCS); k-Nearest

Neighbour (k-NN); Self-Organizing Map (SOM); On

Orbit Servicing (OOS); Anomaly Resolution and

Prognostic Health Management for Autonomy

(ARPHA); Density-Based Spatial Clustering of

Applications with Noise (DBSCAN); Electrical Power

System (EPS); Software and Sensor Health

Management (SSHM); Regularized Discriminant

Analysis (RDA); Adaptive Regularization of Weight

Vector (AROW); Soft Confidence-Weighted (SCW);

Centre national d'études spatiales (The National Centre

for Space Studies) (CNES); European Space Operations

Centre (ESOC); Radial Basis Function (RBF);

Deutsches Zentrum für Luft- und Raumfahrt (German

Aerospace Center) (DLR); One-Class Support Vector

Machine (OC-SVM); Normal Gaussian Herding

(NHERD); Thermal EMission Imaging System

(THEMIS); Intelligent Payload EXperiment (IPEX);

Hyperspectral Infrared Imager (HyspIRI);

Moderate-Resolution Imaging Spectroradiometer

(MODIS); Peak Signal to Noise Ratio (PSNR);

Structural Similarity Index (SSIM); Field

Programmable Gate Array (FPGA); Context-Based,

Adaptive, Lossless Image Codec (CALIC); Integral

Wavelet Transform (IWT); Peano-Hilbert (PH);

Learning Vector Quantization (LVQ); TensorFlow (TF);

Synthetic Aperture Radar (SAR); Ratio of Exponential

Weighted Average (ROEWA); Joint Photographic

Experts Group (JPEG); Space Test

Program-Houston-5-Cubesat Service protocol

(STP-H5-CSP); Neural Architecture Structure (NAS);

Central Processing Unit (CPU); Graphics Processing

Unit (GPU); Visual Processing Unit (VPU); Time

Processing Unit (TPU); Trained Ternary Quantization

(TTQ); Radiation Tolerant (RT); Mobile Neural

Architecture Search (MNAS); Knowledge Transfer

(KT); Knowledge Distillation (KD); Cubesat Service

Protocol (CSP); SpaceBorne Computer (SBC);

Modified National Institute of Standards and

Technology database (MNIST); National Information

Security Standardization Technical Committee

(NISSTC); Deutsches Institut für Normung (German

institute of Standardization) (DIN); Deutschen

Kommission Elektrotechnik Elektronik

Informationstechnik (German Commission for

Electrical, Electronic and Information Technologies)

(DKE); European Union (EU); European commission

(EC); Small and Medium Enterprises (SMEs);

DEpendable and Expandable Learning (DEEL); Earth

Observation Systems’ Data Information Systems

(EOSDIS); Space Generation Advisory Council

(SGAC); Small Satellite Project Group (SSPG); Deep

Reinforcement Learning (DRL); Geographic

Information Systems (GIS); Explainable AI (XAI).

Figures

Fig 1) On-board cloud segmentation from the CIAR

project.

Table

Table 1: Hardware Accelerators.

1. Introduction

Earth Observation (EO) satellites have allowed us to

look at our planet at a scale previously unattainable to

humankind. From the vantage point of space, it

becomes easier to monitor everything about our lives,

from the very large scale such as our impact on the

planet’s ecology [1] or the dynamics of the shifting

IAC-21-B.1.4.4 Page 2 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

Earth atmosphere[2] to the very small like the nature

and extent of specific facilities all around the world [3].

This capability has been and continues to be invaluable

to understanding the world around us and enforcing

regulations vital to the wellbeing of people all over the

globe.

However, as access to space becomes ever more

affordable, EO assets multiply at an increasingly faster

pace [4]. Moreover EO Operations - the sequence of

activities that take place in managing an EO spacecraft

from its launch to its demise - keep growing in number

and complexity as new assets are put into orbit. These

trends could soon lead to a situation where available

work-power becomes a limiting factor in the

deployment of EO systems. Orchestrating these

operations is, at its core, a control and data processing

problem - from taking in and analyzing large volumes

of telemetry from all EO platforms, to taking into

account their complex dynamics and evolving mission

profiles when utilizing them.

Machine Learning (ML) has taken the data

processing world by storm with one success story after

another. From object detection and classification [5] to

natural language processing [6] and nonlinear control

[7], the capacity of these algorithms to solve different

types of problems has been nothing short of

awe-inspiring.

For the purposes of the present review, we define

ML algorithms as those whose performance critically

depends and generally improves with exposure to

real-world data of the problem to be solved.

So far, these techniques have concentrated mostly on

analysis of downlinked imagery, due to the larger

availability of computing power and ease of deployment

relative to on-board applications, as well as its

relatedness to computer vision, one of the traditional

strong suits of ML.

But applications to other aspects of operations are

now starting to surface. The present review explores the

contexts for which these applications have been

proposed or in which they have been applied, exposes

the possibilities that they open up and risks that must be

avoided, and illustrates gaps in research that we believe

should be addressed by the Earth Observation

community.

A wide range of topics is discussed. We tackle

mission planning, a field in which the application of

sophisticated optimization algorithms has a long history

and that has seen some interesting progress recently;

Diagnosis, prognosis, and repair of faults, in which

either Bayesian Networks (BNs) or sequential neural

models seem poised to unlock great gains; enhanced

GNC, particularly through real-time guidance for

low-thrust trajectories and better processing of visual

data from star trackers and Sun/Earth sensors; and

on-board image processing, and how it enables

autonomous retasking and unlocks a tradeoff between

downlink capabilities and onboard compute power.

We also tackle some practical aspects of deploying

ML models in EO Operations. We survey how operators

can maximize the use of limited onboard capabilities by

appropriately optimizing the use of resources of ML

models, outlining a number of software and hardware

techniques devoted to that goal. We also review ongoing

efforts across the space industry to guide and

standardize the deployment of ML models, and the sort

of issues that a designer must keep in mind to avoid

common pitfalls of this technology.

Our goal is to provide EO Operators with a

comprehensive, if not exhaustive, an overview of the

current state of affairs of ML applications in their

domain of activity. The review aims to inspire

discussion and encourage further application proposals

and demonstrations, by connecting EO operators and

proponents of ML algorithms for EO Operations

problems.

The review tackles all on-board and off-board

applications of ML to EO with one notable exception: it

omits all post-processing of payload data on the ground,

a topic that has been studied extensively by past

authors.

One other topic that we do not dive into is the

optimization of tracking, telemetry, and command.

Although originally this was our intent, we found an

excellent and up-to-date review by Fourati and Alouini

[8]. Rather than needlessly repeat their work, we

encourage interested readers to check out the

outstanding article.

This research was produced by a team of 16

volunteers from the Small Satellite Project Group

(SSPG) of SGAC. SGAC is a non-profit

non-governmental organisation focused on the peaceful

uses of outer space with over 16,000 members

worldwide. It hosts 11 project groups, including the

SSPG, with more than 100 active volunteers. The SSPG

focuses on the uses of small satellites across the space

industry and how they can unlock the potential of space

for all of humankind.

2. Machine Learning in Earth Observation

Mission Planning

There are many constraints to mission planning.

Some relate to the target area: It needs to be under the

satellite and illuminated by the sun at capture time (orbit

and time-dependent); clouds are to be avoided (weather

dependent); the requester may set a deadline and/or a

priority. Others relate to the satellite such as limited

memory capacity; limited transmission capability;

reduced communication opportunities with the ground

antennas; multiple sensors to choose from; and limited

IAC-21-B.1.4.4 Page 3 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

manoeuvrability to skew the observation angle and

reach areas not directly flown over.

All of these parameters make optimal scheduling of

observations a highly combinational problem for a

mission that supports multiple independent requests and

it is even more complex when they are accomplished by

a constellation of satellites. Computational theory

literature does not provide deterministic algorithms that

can solve this problem in a timely and cost-effective

way. ML proposes a series of algorithms that may find

better solutions more efficiently.

There are many different formulations of the

observation scheduling problem, taking into account

different subsets of the constraints presented in the

previous section, adapted for different types of missions

and ground segments. The CCSDS’s mission planning

and scheduling [9] standardized definitions for the space

industry and provides an overview of the application of

the CCSDS’ Mission Operation service framework.

2.1. Current approaches

Non-ML algorithms to the satellite scheduling

problems can be classified into two categories: Exact

methods, and heuristic methods [10].

Exact methods typically consist of a combination of

branch and bound methods and mixed-integer linear

programming. These methods are computationally

costly and can become intractable for moderately sized

constellations.

Heuristic methods use an approximated rule to guide

the construction of a solution. Greedy algorithms

construct a solution by gradually choosing the best

action at every decision step according to some metric,

without regard as to how the overall sequence of

decisions plays out. Other heuristic methods include

backtracking through constraint programming and

search algorithms. Other forms of search include

hill-climbing or squeaky-wheel optimization, where the

geometry of the optimization functions is exploited to

accelerate the search process. Globus et al. [11]

compare multiple algorithms such as genetic, simulated

annealing, squeaky wheel and hill-climbing on a

problem with one or two satellites.

Evolutionary or genetic algorithms simulate

processes akin to biological evolution to optimize

candidate solutions according to a hand-crafted fitness

function. Mansour et al. [12] study the performances of

a genetic algorithm for a single satellite with limited

memory and multiple instruments and imaging modes.

Li et al. [13] explore genetic algorithms in order to

provide scheduling in real-time, optimizing the

transmission path towards the user.

Simulated annealing imitates the annealing

processes found in metals exposed to high temperatures,

and it forms the basis for another branch of heuristic

algorithms. The simulated annealing seems to provide

better results which are confirmed by Globus et al. [14]

in a very complete multi-satellite formulation of the

problem including satellite agility and priorities.

Lastly, multi-agent systems simulate interactions

between simple agents representing part of the systems

to determine an optimal policy. Bonnet et al. [15] use a

self-adaptive multi-agent system for real-time and

robust adaptation of a multi-satellite problem including

request priorities.

2.2. ML-based approaches

ML-based approaches can exploit the statistical

distribution of typical problem settings to accelerate the

finding of good solutions to the mission planning

problem.

Wang et al. [10] present a comprehensive review of

publications on the agile observation scheduling

problem, including ML and non ML approaches. The

authors classify approaches along multiple axes such as

time continuous and discrete-time model, type of

solving method and also other features such as

autonomy, and multi-objective profit function.

Neural networks (NNs) are explored by Wang et al.

[16] in order to provide immediate results for a

multi-satellite mission using deep reinforcement

learning (DRL).

Peng et al. [17] apply recursive NNs in a sequential

decision-making process in order to achieve low

scheduling computation time and low loss of profit

when compared to a deterministic resolution. Recursive

NNs allow the model to condition current decisions on

past inputs, instead of depending exclusively on the

present inputs to the system, providing the model with a

sort of memory.

We have not found any applications of Transformers

to this problem, a new sequence modelling technique

from the deep learning research field that has shown

excellent results in other sequential tasks like language

modelling and even in image processing tasks.

Neuroevolutionary techniques combine the

advantages of neural models and evolutionary

algorithms. Du et al. [18] leverage a prediction model

trained by a Cooperative Neuro-Evolution of

Augmenting Topologies algorithm in order to filter

tasks to be scheduled according to the probability to be

fulfilled before scheduling using genetic algorithms.

DRL uses NNs as function approximators to

approximate hard to determine functions in dynamic

programming. This has enabled groundbreaking

achievements in other control and scheduling problems

like playing Go or automated driving. Despite its

potential, it has not been extensively applied to this

problem set.

IAC-21-B.1.4.4 Page 4 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

Liu [19] applies Proximal Policy Optimization, a

method of the DRL literature, to mission planning for a

single satellite. Unfortunately, they do not compare

performance to other methods or extend it to a

multi-satellite setting.

Hadj-Salah et al. [20], [21] explore the application

of A2C, a DRL algorithm, to the mission planning

problem. They compare it to random planning and a

planning heuristic that compromises between greedy

and long-term planning. Their models are trained on a

simulated mission planning environment, then executed

on a real test scenario. Their long-term version of A2C

shows better performance than the heuristic algorithm,

albeit by a slim margin. In their later publication, they

augment the training process with techniques from the

domain randomization and transfer learning literature,

meant to increase robustness to the gap experienced

when passing from the simulated training scenario to

the real validation scenario. Their model beats the

heuristic approach by a slim margin.

2.3. Recommendations

Mission planning is a very rich problem that has

been explored for many years using machine learning

amongst other solutions.

Comparing the performances of algorithms

presented in different papers is not a suitable path

because each presents its own definition of the problem,

with a unique set of constraints, different mission

characteristics, variable satellite capabilities and

potentially incompatible metrics. For instance, a lot of

schedulers take into account satellite memory, limiting

the number of observations until a ground station is

visible, but few of them also make sure the ground

station is available for communication with the satellite

and not busy communicating with another one of the

constellations. Song et al. [22] introduce a framework in

order to facilitate future comparisons but additional

work on model standardization is needed before results

from different studies can be compared.

We observe a shifting trend in algorithms applied to

this problem over the years from genetic or annealing

to ML approaches such as NNs. Unfortunately, we

found no sources comparing the performances of

genetic and ML-based schedulers on a single problem

formulation.

Standardizing project formulations, constraints sets,

and optimization metrics seems to be a necessary step

for sustainable collaborative research in this field.

Relying on CCSDS’s published standards and models

could be a first step in the direction of a unified

approach.

3. Machine Learning in Earth Observation

Guidance, Navigation, and Control

Guidance, Navigation, and Control, often shortened

to GNC, describe the set of operations needed to move a

satellite platform or any other vehicle. Guidance relates

to the planning of paths from a current state to the

desired state. Navigation is the determination of the

present state. Control is the correct use of spacecraft

actuators, such as an engine, to execute the desired plan.

3.1. Current approaches

Two main tasks need to be achieved by a GNC

system: determination of the current state, which is an

estimation task, and use of the spacecraft’s actuators to

go from the current state to the desired state, which is a

control task.

Spacecraft control is typically subdivided into at

least two different granularity levels, where guidance is

the high-level control of the spacecraft from a current

dynamic state to a future one. A guidance module may

output the sequence of feasible dynamic states

necessary to achieve a new orbit from the required orbit.

EO Satellite maneuvers are often planned and optimized

on the ground, and the onboard guidance modules are

minimal. For the control task, a number of control

schemes are used, most notably controllers from the

robust control literature such as H∞ controllers.

As for navigation, spacecraft state is typically

determined via variants of the Kalman filter, such as the

Extended or the Unscented Kalman filters. These

methods are model-based, that is, they depend on an

explicit model of spacecraft dynamics for their

calculations.

Fuzzy controllers have been proposed as a possible

improvement to the classical approach. The literature

contains several works where GNC and AOCS

controllers based on fuzzy logic are compared to their

traditional counterparts. For instance, Wu et al. [23]

studied fitting the no longer in use X-38 re-entry vehicle

with fuzzy logic controllers. ESA also investigated the

usage of fuzzy logic controllers to carry out GEO-orbit

rendezvous autonomously [24] to aid in in-orbit

manufacturing. As another example, in [25], a

simulation of ROCSAT-1 / FORMOSAT-1’s attitude

controller is carried out, where the classical setup of a

PI pitch axis controller and PID roll/yaw axis controller

is replaced with two fuzzy controllers initially, and a

single consolidated fuzzy controller afterwards, yielding

considerable improvements against interference as well

as a lower steady-state error.

Nevertheless, despite the body of research backing

up their effectiveness, there is no widespread use of

fuzzy logic GNC controllers for space missions.

IAC-21-B.1.4.4 Page 5 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

3.2. ML-based approaches

Izzo et al. [26] present a survey of Artificial

Intelligence applied to GNC which, while not focused

on EO applications, can nonetheless be useful to

practitioners. The survey contains a section focusing on

ML approaches, on top of other AI approaches such as

evolutionary algorithms.

In another publication [27], Izzo and Öztürk

leveraged DRL to plan near-optimal real-time

computation of low-thrust transfers. They also suggest a

new method to generate training data for such problem

settings. Although originally designed for Earth-Venus

transfers, their solution is applicable to all low-thrust

transfers, but the data generation algorithm and

optimality comparisons are problem-dependent.

ML excels in problems where no structured

pre-existing model can be exploited. That is not the case

for the GNC problem, where the general form of the

dynamics governing spacecraft are well known and

solvable. It is however the case for visual-based GNC,

as no model exists for relating camera inputs to dynamic

state or control actions. For this reason, much research

on ML-powered GNC has focused on visual-based

GNC [28]–[30] for autonomous rendezvous. This,

however, is not directly relevant to the EO Operations

community, who are unlikely to engage in autonomous

docking as providers.

Hovell and Ulrich [31] propose a guidance module

for autonomous docking based on DRL that takes as

input the spacecraft’s state vector and distance to a

target spacecraft and outputs the desired velocity. A

classical controller takes over and transforms the

velocity output into control commands for the

spacecraft’s actuators. The algorithm, called Deep

Guidance, is trained for tracking and docking, and its

architecture is not easily convertible to other tasks,

which subtracts from its interest for the EO Operations

community. Nonetheless, it provides an example of

deep learning being used for spacecraft guidance.

An interesting streak of research looks into

applications of ML to processing visual navigation

sensors, particularly Earth and Sun sensors and star

trackers. Koizumi et al. [32] present a DL-powered

Earth sensor capable of determining the attitude of the

spacecraft by processing the images captured by a

COTS camera. It runs a real-time image processing

algorithm to extract features into the images separating

them into distinct feature sets using DL techniques. The

features sets are then compared to the preloaded data

sets to determine the position of the spacecraft relative

to Earth in the 3D plane. The primary advantage of the

system is the use of a COTS component and a single

board computer.

Another research thread explores the combination of

ML techniques and fuzzy controllers [33]. Classical

fuzzy controllers rely on manually set parameters that

define behaviour. This research thread attempts to

leverage ML techniques to learn the optimal value for

these parameters from a training dataset. These have the

advantage of interpretability - their reliance on

explicitly (if fuzzily enforced) rules means that they

remain grounded on human-interpretable system

models. Joghataie’ PhD. thesis [34] suggests the

development of a neuro-fuzzy controller, wherein the

tuning of the fuzzy logic is performed automatically by

using neural networks in a hybrid approach. Azarbad

[35] suggests a model applied to GPS systems that

outperforms the classical fuzzy controller. A simulation

study on MATLAB was done by Baranwal et al. in [36],

comparing the performance of a PID controller and a

fuzzy PID controller for a student satellite team. The

EKF based fuzzy controller outperformed the classical

controller. The study was done on a 3U CubeSat.

Further research can be done comparing these

controllers with ML-based approaches, paving the way

for future studies.

We have been unable to find a comparison between

the three types of controllers, i.e. neuro-based controller,

fuzzy controller and a hybrid model, as implementation

details in different studies differ, complicating their

comparison.

Wang et al. [37] have developed a DL framework

that stabilizes the spacecraft using a real-time torque

control. It is initially trained in a simulation

environment which enables it to learn the required

torque output and extrapolates it for unknown

disturbances. It performs better than a conventional PID

controller, as it can correct the attitude after unknown

disturbance rather than repeatable corrections.

A similar system is proposed by Yadava et al. [38].

They propose an AODS system that determines the

position of the spacecraft, taking inputs from the

magnetometers (magnetic vectors) and sun sensor (sun

vector) along with GPS data (position and velocity

vector), and determines the ideal attitude depending on

the position using a neural network. The required torque

calculations are made and sent to the RL based

controller to make the required adjustments. The system

performs better than classical PID controllers as it

consumes less computation power for subsequent cycles

as the algorithm learns.

3.3. Recommendations

Most ML for GNC applications in the space sector

seem to have been explored in the context of space

logistics and space exploration, rather than Earth

Observation. Although guidance and control for EO

platforms are simple compared to these applications, we

IAC-21-B.1.4.4 Page 6 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

believe there is a potential to adopt some of these

technologies.

Attitude determination is an area where EO

operations have high requirements. We believe that

vision processing applied to this area is just getting

started, and that use of more refined neural architectures

could enable improvements in performance or resource

consumption compared to current approaches.

4. Machine Learning in Earth Observation Fault

Detection, Isolation, and Recovery

Satellites performing EO tasks have stringent

requirements in terms of accuracy, continuity and

stability of payload operations. To this end, Fault

Detection, Isolation and Recovery (FDIR) is focused on

the development and improvement of tools to guarantee

and maintain reliable spacecraft operations. FDIR

describes a set of engineering disciplines focused on

safeguarding the spacecraft and maintaining it in

nominal operating conditions. The target of these

disciplines is represented by faults, irregular

occurrences and processes with the potential to disrupt

the mission, up to the point of failure.

ML can be an extremely powerful tool for FDIR.

Indeed, the core capability provided by ML is pattern

detection. Therefore, ML can be used both to detect

anomalies in the telemetry or outputs from any

subsystem (diagnosis), and identify signs indicating an

incipient fault (prognosis). This section presents

relevant ML literature for four significant sub-topics:

fault detection, fault diagnosis, recovery, and fault

avoidance

4.1. Fault Detection

Failure detection deals with identifying the presence

of faults and their rates of occurrence.

4.1.1. Current approaches

In classical approaches, the recognition of failures is

mainly based on constant thresholds and fixed logic

diagrams, defined during the design process. One of the

key issues with classical fault detection is model

brittleness. As fault detection schemes are based on

hardcoded thresholds, these models are easily disrupted

by noise and deviations from theoretical assumptions.

4.1.2. ML-based approaches

An example of an ML-based solution to the issue of

model brittleness can be found in a paper by Jaekel et al.

[39]. This work uses Self-Organizing Maps (SOM), an

unsupervised variant of Artificial Neural Networks

(ANNs), for the detection of failures in dexterous

manipulators for on-orbit servicing (OOS). SOMs

manage to adapt to the idiosyncrasies of incoming data

in a simulated environment and thus show increased

robustness to input variations with respect to traditional

methods. They can also deal with uncertainties and

noise in values. A client satellite is captured by a

servicing satellite with a dexterous manipulator having

7 degrees of freedom. They inject sensor failures,

including sensor outage and drift, during arm operations

and the results show that SOMs are a robust approach as

temporary fluctuations in the sensor, outliers and peaks

do not unnecessarily stop the current operation. But the

computational load is relatively high and needs to be

optimized to reduce system reaction time. The authors

suggest improving the precision and speed of the

method by adding more information from redundant

sensors.

Fuertes et al. [40] discuss ML-based fault detection

using NOSTRADAMUS, an algorithm developed by

the Centre National des Études Spatiales (CNES).

NOSTRADAMUS uses a One-Class - SVM

(OC-SVM), a common unsupervised learning algorithm

used to detect outliers, to detect the presence of an

anomaly in telemetry data. NOSTRADAMUS runs on

the ground segment, analyzing telemetry as it is

downlinked from the satellite. The performance of

NOSTRADAMUS is compared to algorithms inspired

by Novelty Detection (ESOC), Project Sybil [41], and

ATHMoS (DLR) [42]. If missing an anomaly is

regarded as unacceptable, NOSTRADAMUS is highly

preferred as it reaches a 100% detection rate with the

lowest false alarm rate (5%). For false alarm reduction,

the Novelty-inspired algorithm performs better with

85% of correct detections and less than 1% of false

alarms.

CNES is working on an on-board version of this

algorithm, as well as on extensions to the ground-based

variant for processing of multiple telemetry variables

based on dictionary learning approaches [43]. In

conversations during their collaboration with this

project, CNES teams signalled that explainability was a

crucial aspect of any technique. Being able to

understand the features of input data that signal a fault

lets the operational teams understand the context of

their satellite and know which actions must be taken to

remedy the situation - this is comparable in value to

being able to detect the anomaly in the first place.

Project Sybil is a joint initiative between the

Columbus Flight Control team by DLR and the

Advanced Mission Concept Section of ESA, and the

Ludwig Maximilians Universitat to apply an outlier

detection algorithm to the Columbus telemetry database.

It utilises the Density-Based Spatial Clustering of

Applications with Noise (DBSCAN) algorithm for

preprocessing the data after data segmentation and the

computation of its respective features. It is an

IAC-21-B.1.4.4 Page 7 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

unsupervised clustering algorithm that groups the data

according to their relative distances into a variable

number of clusters. Once this clustering is done, the

clusters containing less than 5 percent of the population

data are eliminated under the hypothesis that they may

represent non-nominal working modes in the learning

dataset. Project Sybill allows increasing mission success

rates by reducing the downtime due to failures of the

onboard systems.

4.2. Fault Diagnosis

While fault detection aims to identify the presence

of faults and performance degradations, fault diagnosis

aims to identify the root causes of these occurrences.

4.2.1. Current approaches

Though traditionally, fault diagnosis has been

achieved by human operators at the ground through

comparison with hardcoded hand-tuned thresholds. It is

difficult to deal with large amounts of data using this

approach. Iverson et al. [44] point out that for efficient

utilization of the data, there is a need for an autonomous

approach that eliminates the necessity of human experts

for diagnosis.

4.2.2. ML-based approaches

Ricks et al. [45] examine fault detection and

identification for a satellite Electrical Power System

(EPS) testbed using BNs compiled to arithmetic

circuits. BNs can be used to model partial knowledge

and uncertainty by identifying the system state based on

probabilistic relationships between a set of system

variables at a certain instance in time Meß et al. [46].

The proposed methods work for complex systems

exhibiting both continuous and discrete behaviour. The

discussed techniques can handle abrupt continuous

faults particularly well, which often pose problems. For

example, a nominal value region is not enough to detect

offset faults if they are small enough - the paper uses

cumulative sums to deal with these. Additionally,

“stuck” faults may be difficult to detect in low-noise

conditions since fluctuations might be infrequent. The

authors employ a tunable time interval which will mark

the sensor as working abnormally after it expires

without the readings having made any change.

Different types of nodes, modelling different

behaviours, are grouped to defined sensors and

components, which in turn are assembled to create the

entire EPS functional FDIR structure.

BNs have also been used by Schumann et al. [47] to

detect onboard failures and perform diagnoses. A

Software and Sensor Health Management (SSHM)

system is created for a simple GNC structure of a small

satellite using BNs that get information from hardware

sensors, software status signals, software quality

signals, and data from the Operating System to compute

if any failures are present, what are the most likely

causes behind them, and also give a statistically-sound

quality measure of the diagnoses. The developed SSHM

system requires no modification to the satellite

subsystems for which it performs FDIR - it just uses the

sensor data outputs. That way, model-level and

code-level Verification & Validation can be performed

independently on the SSHM system to certify that the

rate of false positives and false negatives is below a

selected threshold. This SSHM, applied to a simple

GNC system, was able to detect and diagnose both

hardware and software problems successfully.

Nevertheless, it remains a simplistic case and more

research into hierarchical SSHM systems is required in

order to apply them to large-scale BNs. The approach

can further be extended to failures that are not modelled

and unexpected and due to arising behaviour.

Although not specifically related to space systems,

Liu et al. [48] reviews the existing techniques for

ML-based fault diagnosis in rotating machinery. In

general, it presents useful research and conclusions

which we consider can be applied to reaction wheels in

the AOCS subsystem of spacecraft. K-Nearest

Neighbour (k-NN) is the simplest method reviewed,

which exhibits ease of implementation but necessitates

careful fine-tuning and large computation and storage

space. The authors cite BNs’ strong prior assumptions

as the biggest shortcoming of this family of algorithms

while mentioning as main advantages the fact that it

possesses a clear physical explanation of how it detects

faults as well as its reduced storage space requirement.

SVM is also reviewed and its high-dimension accuracy

is highlighted, even if the physical meaning is obscured,

unlike with the previous two techniques. Finally, DL

techniques have the potential to learn from data up to a

degree of complexity much higher than any of the other

techniques without the need for a manually crafted

feature extractor. However, the main drawback of this

approach is the need for large samples in order to train

the network, which is difficult to obtain unless the

spacecraft is a new iteration of previously flown models

for which data already exists. If the satellite is a one-off,

this can only be obtained in an approximate manner by

creating a simulation environment. The authors

underline that future ML-based fault diagnosis methods

should not be purely data-driven, but should take into

account possible failure mechanisms, system models

and prior knowledge in general, in order to increase

diagnostic performance.

Voss et al. [49] explore the use of DL for fault

detection and isolation in a simulation environment. A

NN is developed, trained offline and tested to detect and

IAC-21-B.1.4.4 Page 8 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

isolate single faults in the reaction wheels, GPS, star

tracker and magnetometer subsystems, as well as two

simultaneous faults. Additionally, a case study with

PROBA mission parameters is performed, for the

AOCS subsystem only. The implemented system

yielded mixed results: while there is a near-perfect

performance for some subsystems, the network fared

poorly regarding others, namely misalignment faults.

Also, fault isolation was much more reliable than fault

detection. On top of that, a large dataset is required for

this system to work, so creating a simulation

environment is mandatory, especially for one-off

spacecraft, in order to acquire enough data for adequate

training of the network. This study also assumes there is

enough electrical and computing power available on the

satellite to run this deep-learning-based solution. We

overview techniques for reducing resource consumption

of deep neural models and other techniques in section 6.

4.3. Recovery

Recovery in FDIR is a reconfiguration of the faulty

element and/or overall spacecraft in order to achieve

nominal system behaviour [39].

4.3.1. Current approaches

Traditional FDIR is able to respond to predefined

events by selecting a recovery path from the available

set of options. However, the status of the system and its

environment can exhibit various kinds of uncertain

behaviour due to their dependence on the internal

subsystem, component reliability factors, external

environment factors (e.g., illumination conditions,

thermal, radiation) and on system-environment

interactions (e.g., resource utilization profiles, stress

factors, degradation profiles) [46]. Due to these

uncertainties, the system and its environment cannot be

completely observed by traditional FDIR concepts that

pose limitations to autonomous isolation and recovery

[46]. One important example is Mars Express whose six

months of operation time was lost due to non-resolvable

memory failure that repeatedly transitioned it into safe

mode [39].

4.3.2. ML-based approaches

Raiteri et al. [50] discuss the use of Dynamic

Bayesian Networks (DBNs) to address issues like

partial observability, uncertain system evolution and

system-environment interaction, as well as the

prediction and mitigation of imminent failures. The BNs

do not model the relationship between variables at

previous points in time. DBNs are an extension to BNs

that refer to past values of certain variables to express

dynamic aspects of the system over discrete time [46].

The approach is applied by Raiteri et al. [50] onto the

power subsystem of a simulated ExoMars rover, by

simulating different failure scenarios. The DBNs can

infer whether the system is currently in a normal,

anomalous or failed state. On detection of a failure, a

suitable recovery plan is suggested. A preventive

recovery plan may be proposed in case an anomaly is

inferred. The FDIR presented in this paper also has the

capability of performing a prognostic state estimation

that can also be used for preventive recovery. The

proposed approach has been implemented in an

on-board software architecture called Anomaly

Resolution and Prognostic Health Management for

Autonomy (ARPHA). The results show that DBNs are

suitable for failure situations that require autonomous

(preventive as well as reactive) recovery.

AIKO Technologies [51] have developed a software

library, MiRAGE, that can enable the spacecraft to

make autonomous decisions for processing telemetry

and payload. The library is meant to be installed on the

satellites to enable functionalities such as event

detection, predictive maintenance and autonomous

re-planning.

4.4. Fault Avoidance

Fault avoidance methods are concerned with

preventing the occurrence of faults.

4.4.1. Current approaches

FDIR in past missions worked under the notion that

a fault is detected and then the algorithm will react,

according to predefined scenarios. In other words, the

classical FDIR approach responds only after failure

detection - failure avoidance cannot be achieved by such

a method. Regarding ML-based models, one of the

bottlenecks to having an on-board failure avoidance

system is that the models are trained on the ground with

limited data that does not represent actual behaviour in

space. This gives rise to the requirement of real-time

access to the data, which can be used to represent

multiple onboard scenarios, and closely represents

spacecraft behaviour during the mission.

4.4.2. ML-based approaches

Especially notable in the context of ML-enabled

fault avoidance is the work of Labrèche et al. [52]

discussing the OrbitAI experiment onboard the

OPS-SAT spacecraft. OPS-SAT is a special ESA

satellite deployed with the scope of being a testbench

for novel software technologies in orbit. OrbitAI uses

ML techniques to obtain intelligent FDIR algorithms

enabling the onboard camera to avoid direct exposure to

sunlight. Interestingly the ML model used is trained

IAC-21-B.1.4.4 Page 9 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

on-board, rather than offline. The model is trained with

five training algorithms tested of those natively

provided in the MochiMochi library [53] for online ML

training: Adam, RDA, AROW, SCW, and NHERD.

When using the figure of merit of balanced accuracy,

only one model appears to achieve values significantly

different from 0.5: the AROW algorithm in

three-dimensional input space.

4.5. Recommendations

FDIR innovation has been applied mainly to deep

space missions, which need a higher degree of

autonomy due to their long communication delays

inherent to the long distances travelled. However, the

analysis of the literature suggests that ML in EO FDIR

has promising prospects. The approach can be extended

to diagnose failures that are not modelled, unexpected

and due to arising behaviour, which offers a great

advantage in overcoming the model brittleness issues of

traditional FDIR. ML-based solutions for fault detection

and diagnosis can be integrated alongside the traditional

FDIR of the satellite. But ML-based recovery is

virtually unexplored and much research is needed in this

domain. The majority of the work in this field concerns

BNs, while other research avenues remain largely

unexplored, such as ANNs and DL. As it would be

shown and discussed in Section 6, power and

computational resources remain a big concern for

ML-based FDIR, especially for small satellites. The

benefits offered by ML-based FDIR can be further

researched to be implemented in future EO satellites to

perform FDIR on the AOCS subsystem, GNC,

On-Board Data Handling, Power subsystem as well as

detection of faulty sensors.

5. Machine Learning in Earth Observation

On-board Image Processing

5.1. Cloud and novelty detection

Clouds cover 66% of the Earth’s surface and are an

obstacle when observing the Earth’s surface in certain

wavelengths such as visible light. Removal of clouds

from satellite images is an important preprocessing

phase for most of the applications in remote sensing.

Researchers have explored various forms of Cloud

detection like “Cloud / No cloud”, “Snow / Cloud”, and

“Thin Cloud / Thick Cloud”, using various approaches

of ML and classical algorithms [54]. Cloud

detection/filtering can be used alongside novelty

detection. Novelty detection is to detect unexpected

features and it is especially important while looking into

new environments.

Good cloud detection algorithms are necessary to

optimize bandwidth and memory usage in EO missions

[55] and before the implementation of segmentation and

object detection methods. Convolutional neural

networks (CNN) have demonstrated excellent

performance in various visual recognition problems

such as image classification, and enable accurate

onboard cloud detection in small satellites.

With the increase in EO missions coupled with

high-resolution modern sensors, there is an increase in

bandwidth requirement that leads to the need to utilize

new techniques to efficiently manage the bandwidth

resources.

5.1.1. Current approaches

In the majority of missions, all images taken are

transmitted to the ground, which requires a significant

amount of bandwidth. Traditionally, data collection is

done by specifying in advance where and when to take

the measurements. Based on the content of the data,

there is no mechanism to tailor what is downlinked.

Other common approaches include novelty detection

based on spectral contrast, radiance spatial or temporal

contrast. But these methods are better used for dark

grounds like vegetation or deserts as clouds contrast in

colour compared to them. Furthermore, these methods

rely on manually chosen thresholds, which are

time-consuming to find and sometimes brittle.

Whereas spatial coherence is a better method of cloud

detection in areas with little contrast with the clouds (ice

sheets). NNs have also been shown to have greater

flexibility with classifying indistinct classes like clouds

on snow.

5.1.2. ML-based approaches to Cloud

Detection

For cloud detection, Zhang et al. [55] propose a

lightweight DNN based on U-Net. For performance

estimation of the proposed method, training and testing

of the red, green, blue and infrared waveband images

from Landsat-8 were used. The lightweight DNN is

based on U-Net and obtained better overall accuracy

while reaching the state-of-art inference speed by

applying the LeGall-5/3 wavelet transform on the

dataset which compresses the dataset and accelerates the

network for on-board use. Zhang et al. experimental

results illustrate that the proposed model maintains high

accuracy after four-level compression [55]. They reduce

processing time from 5.408s per million pixels to 0.12s

per million pixels, and average memory cost by around

30%. They observed that increasing batch size reduces

the time cost and increases peak memory. The largest

batch size experimented in the work, 1000, led to a time

cost reduced to 0.01s against around 20MB on peak

IAC-21-B.1.4.4 Page 10 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

memory cost. On the contrary, a smaller batch size, 100,

shows a time cost of 0.05s and a memory peak cost of

around 17MB. By taking advantage of mature image

compression systems in small satellites, the proposed

method provides a good possibility of onboard cloud

detection based on DL and hence helps in improving the

downlink data transmission efficiency and reducing the

memory cost. U-Net provides better accuracy on

compressed datasets. Also, the U-Net framework

showed great potential for pixel-wise classification of

remote sensing datasets [55].

Hinz et al. [56] also work on the detection of clouds,

in the framework of the H2020 EO-Alert project.

However, the EO-Alert project aims at keeping images

of clouds and enriching them with alert profiles in case

of severe storms for weather broadcasting. The

algorithm used is ML-based Gradient Boosted Decision

Trees and is embedded in a modular image processing

pipeline. Currently, tests of the pipeline are performed

in Matlab and are ported on hardware to be flown to

space.

Srivastava et al. [57] suggested using Kernel

methods for better onboard discovery computation of

cloud detection over snow and ice. This paper proposes

a Kernel method that can be used for both clustering

and classification of images on board any satellite. The

paper discusses a novel variant of the Probabilistic

Kernel (P-Kernels) with a mixture of Gaussian and

spherical covariance structures. It is very sensitive to

even the smallest of changes as it assumes all

observations to be independent. The results showed

great promise with clouds being differentiated much

better from Greenland ice sheets when compared to the

Gaussian and a Gaussian mixture model.

Giuffrida et al. [58] discuss a CNN deployed on the

PhiSat-1 reconfigurable nanosatellite to analyze

imagery from its Hyperscout-2 payload and select

images eligible for transmission to the ground. It is

implemented onboard the ESA Phisat-I mission to

classify cloud-covered images and clear ones. Only

images with less than 70% cloudiness are transmitted to

the ground. The network is trained and tested against an

extracted dataset from the Sentinel-2 mission, which

was appropriately pre-processed to emulate the

Hyperscout-2 hyperspectral sensor. On the test set, 92%

of accuracy is achieved with 1% of False Positives (FP).

The results showed a power consumption of 1.8 W,

requiring memory of 2.1 MB, keeping within the power

and the memory constraints.

Other solutions that have not flown yet and are in

the concept phase have been developed. Maskey et al.

[59] proposed an ultralight CNN algorithm, called

CubeSatNet, that prioritizes quality data over quantity

without changing the constraints of size, power, volume,

downlink and pointing requirements imposed by a 1U

CubeSat. The algorithm is trained over 48000

augmented images from CubeSats and validated against

12000 augmented images from CubeSats to classify

images as “bad” when cloudy, sunburnt, facing space or

saturated. Images are classified as “good” in all other

cases. If in orbit, the algorithm would select only

“good” images to be downlinked and discard images

that are covered in clouds or too bright or dark. Trained

on BIRDS3 satellite images, the algorithm reportedly

has an accuracy of 90% and can cut operation time by

about 2/3 while significantly improving the quality of

images received.

Murray [60] proposed a concept of on-board

processing with two cameras: the nadir-looking camera

performs the standard observation, whereas a

forward-looking camera observes if clouds are coming

in the trajectory of the satellite. A neural net

classification grid is used to identify clouds and an

algorithm then decides when to capture images with the

nadir-looking. This approach would be oriented towards

CubeSats.

Castaño et al. [61] trained an SVM for estimating

the opacity of atmospheric dust and water ice on Mars

on data from the THEMIS camera mounted on board

the Odyssey mission. The authors use both a regular

SVM and a reduced-set SVM. The reduced-set SVM is

trained on a reduced synthetic dataset maximizing the

similarity of the reduced-set SVM to the regular SVM.

The reduced amount of support vectors decreases

compute requirements. They then test both the full-size

SVM and reduced-set SVM on flight software, showing

the capability of such software to run the proposed

algorithms.

The authors mention two challenges related to the

analysis accuracy of onboard THEMIS data. Firstly, as

the onboard data is not calibrated, the deployed models

must be robust to significant noise. Secondly, due to

temperature fluctuations, the response function of the

camera can gradually increase or decrease its values,

even when there is no change in actual value. The

authors suggest characterizing the operation of the

algorithms in an environment as close as possible to that

of the spacecraft.

Lastly, the CIAR project from IRT Saint Exupéry

demonstrated cloud segmentation on board the

operational test-bed satellite OPS-SAT in 2021 [62].

Figure 1) showcases a visualization of their results.

IAC-21-B.1.4.4 Page 11 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

Figure 1) On-board cloud segmentation from the

CIAR project

5.1.3. ML-based approaches to Novelty

Detection

Wagstaff et al. [63] show the benefits of reduced

downlink data when performing cloud detection and

filtering for EO missions. Cloud detection is

demonstrated using Random Decision Forests (RDFs)

and Bayesian Thresholding (BT), while a third

saliency-based algorithm is used for novelty detection

onboard EO-1. For classifying the pixels, the RDF

method analyzes a window of values around the pixel.

In contrast, the BT independently performs the

classification of each pixel. BT uses the difference in

particular wavelengths between dark surface materials

and bright cloudy regions. The novelty detection

algorithm identifies such regions within an image that

may contain new features. EO-1’s primary science

instrument is Hyperion. It’s an Imaging Spectrometer

capable of data collection with high Spatial and Spectral

resolution. Using data from previous mission phases,

both cloud detection algorithms were trained to drop

useless images from the telemetry downstream. The

performance of the algorithms has been evaluated

onboard over a five-month period from November 2016

through March 2017. In comparison to ground testing,

the on-board performance showed similar or better

results on a diverse collection of targets. Both RDFs and

BT reached an accuracy of more than 90%. However,

in real-time, the RDFs were faster. The novelty

detection was able to detect new features in remote

locations such as small lakes and buildings and hence

such images could then be given priority for the

downlink. Such methods must be able to successfully

operate on board with limited resources while posing a

minimum risk to the overall spacecraft. With the

advancement in computing capabilities, more complex

models offering better accuracy can be used onboard

future EO missions.

Chien et al. [64] provide results of the IPEX, which

originated from a CubeSat that flew from December

2013 through January 2015 and validated autonomous

operations for the processing and the generation of

product onboard the platform hosting the Intelligent

Payload Module of the Hyperspectral Infrared Imager

(HyspIRI) mission concept. IPEX worked as a testbench

for on-board image classification, achieved through the

use of ML-based random decision forest algorithms.

Compared to other missions, the solution was enhanced

with the use of an ensemble of multiple trees to increase

the classifier’s reliability via statistical regularization

without the need for explicit pruning of the trees.

Additionally, the algorithm analyzes spatial

neighbourhoods within each image to incorporate local

morphology and texture, rather than single pixels.

Attention was also paid to reduce runtime by classifying

every 10th pixel along with the vertical and horizontal

directions and filling in the remainder with

nearest-neighbour interpolation. An extremely

interesting point is that the IPEX classifier was trained

before launch using just four hand-labelled images from

a high-altitude balloon flight that employed the same

type of camera used by IPEX. According to the authors

of that research, it was the first time an ML system has

been trained on a suborbital flight and then successfully

used in orbit.

IPEX also tested an unsupervised method to identify

images with potentially interesting content, to be paired

with supervised learning. In particular, a computer

vision visual salience software was employed to extract

interesting regions for downlink in acquired imagery.

The algorithm used a simple pixel-based measure of

visual salience for grayscale images that incorporate

local context to be compliant with limited resources

onboard the CubeSat. The algorithm is applied to a

downsampled version of the image using a 32 × 32

pixel window to identify the five most salient regions

within the image. The pipeline is completed with

thumbnails of salient regions and their salience scores,

which are being saved and made available for downlink

and examination on the ground. Images at full

resolution can be furthermore downlinked to ground

stations, in case needed.

5.1.4. Recommendations

With the strict limitation on bandwidth, onboard

filtering of useless data enables sending data to the

ground with minimum compromise on image quality

and the need for human intervention for decision

making. The results of ML algorithms can be improved

in terms of accuracy and precision, with the availability

of newly generated data.

5.2. Object/Image Classification

Image classification is a task of extracting

information on the basis of objects in the images instead

of individual pixels, where “objects” are referred to as

IAC-21-B.1.4.4 Page 12 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

meaningful scene components that distinguish an image

[65].

5.2.1. Current approaches

While current methods do extensively apply ML

algorithms to great success, image classification is done

on the ground instead of onboard a satellite.

5.2.2. ML-Based approaches

Arechiga et al. [66] give an example of an on-board

processing application where a CNN architecture is

used for object classification and trained using satellite

imagery of Planet’s Open California dataset. Nvidia

Jetson TX2 is used for implementing this application.

The authors suggest that more research can be done so

that the application can be enhanced to classify more

objects.

Machine intelligence is used to perform onboard

analysis of EO tasks such as hazard analysis (e.g. fire

and flood detection), target detection, area monitoring,

and weather forecasting [67]. Researchers at NASA

Goddard used ML to detect wildfires on MODIS

(Moderate-resolution imaging spectroradiometer) data.

In practice, the use of CNN’s is composed of two

main tasks: training and inference. Training is the

process of “learning” the optimal set of weights that

maximize the accuracy of the desired task (e.g. image

classification, object detection, semantic segmentation).

It is highly compute-intensive and often is accelerated

by GPUs. The inference is the process of using a trained

model (where parameters are no longer modified) to

make decisions on novel data. The inference is a less

computer-intensive process than training and has been

performed on CPUs, GPUs, and FPGAs.

5.2.3. Recommendations

Similarly to the case of onboard cloud detection,

moving object classification and detection onboard

satellite platforms allows operators to reduce the load of

ground-satellite communications links. EO Operators

can leverage the huge and quickly expanding research

field of computer vision.

The high-level information gained by using object

classification can then be used for other tasks, like

dynamic mission replanning.

5.3. On-board image compression

New, complex onboard sensors can rapidly saturate

both the downlink bandwidth of communication

transceivers and onboard data storage capability. More

efficient image-compression codecs are an emerging

requirement for spacecraft and can significantly reduce

transmitted or stored data volume. However, in the

tradeoff mission design, it is also necessary to consider

whether these are computationally intensive and require

rapid processing to sustain sensor data rates.

5.3.1. Current approaches

Systems to compress data in spaceborne operations

employed a wide variety of both lossless and lossy

compression schemes. Lossy compression is usually

used when the system bandwidth is too low and cannot

support lossless compression, when the science value is

not compromised by the distortion that would be

introduced by lossy compression, or when other sensors

that do not play a role in primary data products are

included. An example of this last case can be

scene-context cameras.

5.3.2. ML-Based approaches

Goodwill et al. [68] proposed an ML-based solution

to achieve good reconstruction fidelity after lossy

compression. The algorithm, CNN-JPEG, makes use of

a hybrid approach combining CNNs and JPEG

Compression. In the encoder, the image is fed to a

3-layer CNN to obtain a compact image representation,

which is then encoded with JPEG. The encoder, based

on previous work, is denoted by ComCNN and learns a

compact image representation that is half the size of the

original image. In the decoder, the resulting image is

upsampled to the original size and decoded with a

deeper 20-layer CNN, which reconstructs the original

image by learning a residual image and adding it to the

upsampled image.

Experimental results for CNN-JPEG show a 23.5%

and 33.5% increase in PSNR and SSIM over standard

JPEG, respectively, on an image dataset collected from

STP-H5-CSP compressed to the same file size. On the

same dataset, CNN-JPEG performed a 1.74 times

increase in average compression ratio at fixed PSNR. It

is interesting to also note that, according to the research,

the encoding segment of CNN-JPEG in TF Lite,

running on the Cortex-A9 cores of the Zynq-7020,

yielded an average execution time of 16.75s using a

single thread. Using the TF Lite interpreter to parallelize

operations was reportedly far from ideal linear speedup.

Authors also showed that leveraging the Zynq-7020

FPGA resources through SDSoC for hardware

acceleration helped in decreasing the average execution

time of the CNN-JPEG encoder to 2.293 s, with a 7.30

times speedup over the single-threaded TF Lite solution

and 6.87 times speedup over the single-threaded TF Lite

solution.

Vladimirova et al. [69] discuss the development of a

lossless compression method without the drawbacks of

low compression ratios using predictive NNs, coupled

IAC-21-B.1.4.4 Page 13 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

with integral wavelet transforms and the Peano-Hilbert

(PH) Scan algorithm. This is then benchmarked against

the CALIC Method using various image datasets. The

image is first sent through the IWT (Integral Wavelet

Transform) to produce a de-correlated image, which is

mapped and a PH scan is performed after which the NN

(a two-layer, 4x106x 1) scans and allocates a probability

distribution for the next incoming value. On the tested

data sets, using only the NN method achieved an

average compression ratio of 2.530, compared to the

CALIC method which achieved a ratio of 1.806.

Introducing the PH scan brought an 8.5% improvement

compared to the CALIC method at 2.747. The

IWT+PH+NN method overall achieved an improvement

of 13.1% compression ratio over the CALIC method.

The paper proposes potential applications of the

algorithm in previewing a satellite image before a full

image is transferred to assess features of the image and

hence would prevent bad images from being sent, such

as those affected by clouds or images suffering from

other distortions.

Cai et al. [70] proposed a novel method for LIDAR

image data compression. The method is called feature

indexing where specific features are assigned to a data

index system generated by DNNs. The whole program

is then uploaded to onboard hardware and it stores it as

a dictionary for reference. The OBC runs a feature

isolation program and identifies features and creates a

resultant dataset of pure indices based on the directory.

This data set is then transmitted with the location data

and then is decoded on the ground. Achieves a

compression level of 99.17% and works far better than

standard wavelet compression methods. The method

was tested against the LIDAR data of the Space Shuttle

program and achieved the above-mentioned results.

5.3.3. Recommendations

The exploitation of lossy compression to ease

downlink clearly represents a path to be explored. The

work by Goodwill et al. [68] also emphasizes the

importance of advancement in the field of hardware

acceleration and SoC FPGAs. Indeed, on-board

inference of CNNs is computationally expensive for

space platforms. Further advancements can possibly

support the application of more complex algorithms

even in constrained environments.

6. Machine Learning in resource-constrained

Earth Observation platforms

This section addresses the topic of ML in

resource-constrained spacecraft performing EO tasks.

These methods represent a powerful set of enabling

technologies, relevant both for the emerging interest in

small satellites and to preserve the operativity of large

platforms experiencing failures or operating with shared

resources. Moreover, the consistent technological lag of

space hardware makes considerations about reduced

available SWaP almost always necessary when

redeploying architectures developed for Earth-based

applications into orbit.

Within the scope of this work, the constraint on

resource availability will be limited to on-the-edge

computational and sensing capabilities, and not

extended to the data. It is also out of the scope of the

section to address scheduling approaches, which

optimize the availability of resources to multiple

subsystems or users. This variability, however, can be

also seen as a source of constraint over the available

budgets.

We investigate two ways in which this adaptation to

technological limitations can be implemented:

optimization of the AI architecture itself, and

optimization of the interplay between the model and the

hardware this operates on. In general,

resource-constrained platforms it is necessary to

maintain a holistic view of the architecture of the

software, the hardware, and the data at play.

It is worth noting that another emerging

technological field presenting similar constraints to the

space sector is represented by Internet-of-Things [71],

where the target platforms for AI are small, low-power

devices.

6.1. AI Architecture Optimization

6.1.1. Pruning

Pruning is the operation of removing or zeroing

parameters of a NN model thus reducing the network’s

size [72]. This process is generally performed by

associating scores with the network’s elements during

training in order to select the ones to prune. The lighter

model is then further trained and can be iteratively

re-pruned several times. Multiple pruning strategies

exist such as varying the number and nature of items

pruned, the number of iterations performed or changing

the scoring criteria [73]. There are also other emerging

pruning paradigms that do not rely on an iterative

process [74] [75].

Pruning’s main trade-off is to increase

computational efficiency at the cost of quality/accuracy

and increased training complexity. The objective is to

leverage compression rates of 4, 8 or even 32 while

costing at worst only a few percent of accuracy [73].

Performing pruning along this objective remains a

delicate task as literature demonstrates that keeping

good performances is dependent on the pruning method.

The main challenge of implementing pruning is thus to

determine and test which pruning methods to use in

IAC-21-B.1.4.4 Page 14 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
order to achieve the required compression while
keeping acceptable performances for a representative
type of datasets.
Although the lack of standards in evaluation
impedes the comparison of the multiple existing studies,
they all advertise significant compressing at low
accuracy cost, including several algorithms confirmed
by multiple papers [73]. Pruning has been successfully
applied in many image processing use cases but has also
been proven on voice processing [76], credit
classification [77], and multiple other types of datasets
[78].
Additional engineering and more complex training
on the ground in order to significantly reduce the
onboard execution constraints make pruning an
attractive trade-off and a strong technological enabler of
NN implementation in space.
Pruning is now developed enough to have
documented implementation and examples in ML
frameworks such as TF [79].
So far, pruning has been used as part of complex NN
applications for space but only on the ground with
applications such as image classification [80]–[83].
There are some applications aiming towards on-board
implementations like remote sensing image
classification [84], [85], vehicle detection in satellite
image [86] and image anomaly detection [87].
Unfortunately, the authors were unable to find
documented evidence of a pruned NN that flew on a
space mission.
6.1.2. Filter compression and matrix
factorization
In its section concerning “convolutional filter
compression and matrix factorization,” the paper by
Goel et al. [88] presents methods to adapt neural
networks to low-power platforms by operating at a
layer’s level. The distinction operated between the two
distinguishes between the types of network elements
that are being optimized.
Neural Networks can be algebraically represented as
n-dimensional matrices known as tensors. Matrix
factorization approaches reduce the complexity of these
underlying tensorial structures, to obtain compressed
networks without significant loss of accuracy. Filter
compression methods, on the other hand, reduce the
number of parameters in the network architecture by
acting on the structure of filters in the so-called
convolutional layers.
In particular, Goel et al. observe that filter
compression methods are capable of achieving a state of
the art accuracy in computer vision, albeit at times at a
high computational cost. As computer vision tasks are
essential in EO operations, this class of methods appears
to be the most significant within the scope of this paper.
Two architectures emerging as relevant for filter
compression are SqueezeNet [89] and MobileNets [90].
Both these architectures have found applications in the
EO community. For example, modified SqueezeNets
have been used by Alhichri [91], Alswayed et al. [92]
and Alhichri et al. [93] for the classification of remote
sensing images (both in drone and satellite images). In
particular, Alswayed et al. report results comparable to
or outperforming the state of the art at the time of
publication.
Poortinga et al. have used a MobileNet-based
architecture to map sugarcanes in satellite data of
Thailand [94], obtaining significant accuracy for the
task. Zhang et al. [95] also have used an architecture
capitalising on MobileNet, reporting results
outperforming the state of the art at the time. Similarly,
Yu et al. [96] present a MobileNet-based method to
classify remote sensing imagery and report
outperforming many state-of-the-art models while
requiring a smaller amount of training data. In their
report paper, Hoeser et al. [97] note that: “It is
important to note the small group of six items which use
MobileNets(...), of which five were published in 2019.
They describe an onset of interest in parameter efficient
models with high accuracy and they prove that such
models can compete in Earth observation studies.“
6.1.3. Architecture search
Neural Architecture Search (NAS) refers to a set of
tools and processes for the automatic generation of
optimal architectures for an ANN. NAS is a specific
instance of automated machine learning (AutoML), the
process of automating the overall ML construction
process [98]. As shown by Chan et. al [99], this process
can be specialized to address a constraint on available
resources.
Seminal developments in NAS emerged in late
2016, from the work of Zoph and Le [100] and Baker et
al. [101]. In a survey on the subject, Elsken et al. [102]
report three key parameters to operate a classification of
NAS processes. These are:
●Search space;
●Search strategy;
●Performance estimation strategy.
Being an approach to adapt the heavy computational
cost of NN to resource-constrained platforms, NAS has
naturally found application in many space-related use
cases. EO, there has been quite a research on
hyperspectral images classification using NAS, with
development performed by Liang et al. [103] have
employed NAS (and pruning) to detect aircraft in
remote sensing images. NAS Mobile neural architecture
IAC-21-B.1.4.4 Page 15 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

search (MNAS) [104] is a probable candidate in

implementing NAS to EO satellite inference on the edge

application.

6.1.4. Knowledge Transfer and distillation

In knowledge transfer (KT) and knowledge

distillation (KD) a small, lightweight network is trained

to reproduce the behaviour of a large, computationally

intensive network without having to fully duplicate the

architecture of the latter. This leads to small networks

both providing results comparable to those of large

networks and deployable on resource-constrained

platforms.

According to the paper of Goel et al. [88], in KT the

smaller network is trained using data labelled by the

larger network (defined as “synthetically labelled data”

by Ba and Caruana [105]), while in KD a small network

(student) is trained by a large network (teacher) to

replicate the latter’s output. Within the scope of this

section, it appears also relevant to discuss transfer

learning, which has attracted considerable interest from

the space community.

De Vieilleville et al. [106] proposed a distillation

method to perform DNN-mediated segmentation of EO

images on board of cubesats. In this work, they show

that a 10 to 30 fold reduction of the free parameters of

the network mediated through distillation leads to

weakly worse performance (+5/-10% accuracy).

Similarly, [107] provides a detailed distillation

implementation and results showing a strong reduction

of the NN execution load while keeping a steady

accuracy in remote sensing scene classification. [108]

applied distillation for mapping irrigated areas using

remote sensing data.

Since 2019, self-distillating networks are emerging

[109] with one successful implementation for cloud

detection in remote sensing by [110] achieving 200-fold

compression. Industrialization is not as developed as

pruning as there are only a few open access examples of

implementations but no widely developed library.

Unfortunately, the authors were unable to find

documented evidence of a distilled NN ever flown and

used on a space mission.

6.2. Hardware Acceleration

Computing limitations are demanding to ML-based

applications because of the significant amount of data to

be processed for DL. Many NN models require

high-end GPU devices to run in inference, and even

more so during training. In deploying ML to an EO

satellite it is widely acceptable to consider the

inferencing step of the process due to the

resource-constrained nature of a satellite system in orbit

in terms of volume, power, and mass especially under

the CubeSat standards. Progress in commercially

available off-the-shelf hardware in mobile edge

computing has a progressive effect in finding their way

to CubeSats in implementing DL algorithms for space

applications [111].

With CPUs considered to be general-purpose

computers, AI-specific hardware such as GPU’s, FPGA,

and ASIC takes the center stage which is designed to

accelerate the computation of linear algebra and

specializes in performing fast and matrix multiplications

with higher performance-per-watt ratios. Furthermore,

advanced next-generation architecture for onboard

computing which heavily depends on artificial

intelligence is developed like AI-OBC (Artificial

Intelligence-OnBoard Computing) [112] based on

distributed on-board architecture consisting of CPU,

Visual Processing Unit (VPU) emerging AI accelerator

class of microprocessor for running machine-learning

applications to train DNN and FPGA connected through

Cubesat Service Protocol (CSP) through which ML and

training are carried out in real-time with commercial

off-the-shelf COTS components to reduce cost and

development time.

Table 1. Hardware Accelerators

Name

Company

Description

Intel Movidius

Myriad 2

Vision

Processing

Unit (VPU)

Intel

Implemented with

DNN in Phisat-1 [113]

Myriad X

(VPU)

Intel

Active testing

[114]

Jetson Nano

(GPU)

Nvidia

Space Edge Zero

(2021) by Spiral blue

[115]

Tegra TX1

and TX2

(SoC)

Nvidia

Demonstrated AI

Image processing

capability [116], [117]

Coral TPU

Google

Used with SC-LEARN

Architecture for

Hyperspectral models

[118]

Apache 5

Almotive

In development

Neuromorphic

chip

Innatera

In development

IAC-21-B.1.4.4 Page 16 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

Spaceborne

Computer-2

(SBC-2)

Based on

Intel Xeon

Onboard ISS

Ultrascale

Radiation

Tolerant (RT)

Kintex FPGA

Xilinx

Prototype available

Xilinx

Zynq-7020

(ARM

Cortex-A9 +

FPGA)

Xilinx

Space Test Program

Houston 5/ CSP

(2017)

6.3. Quantization / BNNs

In quantized networks, the number of bits used to

represent numbers defining a model is reduced. This

provides a significant decrease in compute. memory,

and power requirements, for a comparatively low

decrease in performance. Quantization may be applied

to weights, activation functions or gradients of a

network, either during or after training. [119]–[121].

Quantization has been explored in research for remote

sensing image segmentation and processing but appears

to never have been flown in space.

Perhaps the most common established quantization

technique is reducing the bit-width of weights after

training. However, very low bit widths, typically of four

or less, usually incur heavy losses. This can be

mitigated by performing model training under the

reduced bit-width quantization. Good results have been

achieved with quantization, even going all the way to a

single bit.

Accuracy on par with full-precision NNs was

achieved for standard datasets in publications such as

Binary-Connect, XNOR-Net, and TTQ [122]–[124].

Quantization of already existing NNs such as AlexNet

[5] and VGGNet [125] applied to the ImageNet dataset

has been carried out without any accuracy loss while

reducing their sizes up to 50 times [126]. Quantization

both before and after model training is provided today

either as part of mainstream DL libraries [127], [128] or

third-party libraries such as Larq [129] respectively.

Although there exists no consensus on why

quantization works, a candidate explanation argues that

large amounts of pathway redundancy in NNs make the

expressivity loss a minor concern. Theoretical analysis

in that regard is still limited. Anderson and Berg [130]

found that statistical properties of the computation are

kept even when a network is binarized. Molchanov et al.

[131] indicate that nearly 99% of weights can be pruned

in certain NNs and achieved a 68-times sized reduction

on VGG-like networks without loss of accuracy.

Broadly speaking, quantization techniques can be

divided into two main categories: Deterministic and

Stochastic. Guo classifies deterministic quantization

methods [119] into:

●Rounding: Floating-point values are assigned

their nearest fixed-point representation.

●Vector Quantization: Weights are clustered

into groups, with the centroid of each group

replacing the real weights.

●Quantization as an optimization: Here, the

quantization is treated as an optimization

problem, which involves minimizing an error

function taking into account real and quantized

weight values.

Regarding stochastic quantization techniques, they

separate them into:

●Random Rounding: The quantized value is

obtained by sampling a discrete distribution

parameterized by the real values themselves.

●Probabilistic Quantization: Weights are

assumed to be discretely distributed, with the

methods trying to estimate which distribution

function it is.

Deterministic quantization has seen extensive

success, with rounding being the most commonly

successfully employed type of quantization, such as

Rastegari et al. [123] and [132], where a general

rounding function was introduced. In particular,

Binary-Connect Courbariaux et al. [122] used binary

rounding, achieving 98.8% accuracy on the MNIST

dataset. Also noteworthy is the use of vector

quantization in Gong et al. [133], where a network

compression ratio of 24 was obtained, losing only 1% of

accuracy on the ImageNet dataset. However, Stochastic

quantization has not experienced such a resounding

success, perhaps due to an over-reliance on statistical

assumptions which are not guaranteed to hold.

Quantization approaches may quantify several or all

of the following:

●Weights: The action of quantizing weights

yields a smaller network size and can

accelerate both the training as well as the

inference process. This comes at a price,

however: NNs will have a harder time

converging when training with quantized

weights, and a smaller learning rate is required.

Additionally, the gradient cannot

back-propagate through discrete neurons,

leading to the use of straight-through

estimators in order to estimate them, usually

with a high variance.

●Activations: The goal of quantized activations

is replacing inner products with binary

operations, reducing memory constraints since

the operation precision is reduced, all while

accelerating network training. In fact,

IAC-21-B.1.4.4 Page 17 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

activations may fill more memory than weights

[134]. Note that quantized activation will cause

what is called a “gradient mismatch”, where

the gradient of the activation function is

different from the one obtained from the

straight-through estimator used.

●Gradients: Quantizing the gradients is still a

relatively new avenue of research in NN

quantization. The main objective here is not

reducing the model size, but aiding in

distributed network training, where several

computing nodes need to share information of

the gradient values between them. The smaller

the size of the data the nodes need to share, the

faster parallel training can be performed.

Quantized gradients need to be carried out with

care since unsuitable implementations run the

risk of causing the gradient descent algorithm

not to converge.

7. Machine Learning standardization and issues in

Earth Observation Operations

Interest in AI and ML has increased in the past

years. Many organisations across different industries

and geographical locations are currently working on

creating guidelines, best practices, and standards, to

guide the correct implementation of these systems.

However, the process is far from being complete, and

we have found only one such example from the space

industry.

Standards, guidelines, and other documents

discussed in this section blur the line between

definitions of AI and ML. While we find this fact

misleading, we have kept the original usage from the

sources in order not to alter their message.

These bodies of work aim at aiding ML system

developers to avoid common pitfalls and problems

associated with these systems. We provide in this

section a cursory overview of what these problems are

in order to raise awareness amongst EO platform

operators.

7.1. Guidelines and Roadmaps

International Standards by ISO/IEC JTC 1/SC 42

committee [135] are currently available or under

development. These standards and projects represent the

united efforts of experts and entities in providing

guidance and focus on the standardization of Artificial

Intelligence, with currently more than 20 under

development and 6 already published. We found

ISO/IEC TR 24030:2021 to be particularly interesting

as it covers 132 use cases, as well as the projects under

development concerning Functional Safety and AI, data

quality and AI explainability.

The International Standardization Organization is

not alone in working on AI standardization, though.

The Chinese Big Data Security Standards Special

Working Group of the National Information Security

Standardization Technical Committee (NISSTC) wrote

the Artificial Intelligence Security Standardization

White Paper [136]. The focus of this White Paper

ranges from the security of AI to main security threats,

risks, and challenges. Seven recommendations have

been made on the importance of improving a system of

AI security standards, the need to speed up the

development of standards in key areas, promoting the

application of AI security standards, strengthening the

training of AI security standardization talent,

participating in international AI security

standardization, establishing an AI high-security risk

early warning mechanism, and improving AI security

supervision support capabilities.

Germany developed an Artificial Intelligence

Standardization Roadmap [137], continuously updated,

as a joint effort between DIN and DKE. The roadmap

strongly supports the idea that standardization would

help in getting the explainability and reliability of AI,

thus favouring its application. In the roadmap, five

recommendations are drafted. More specifically, they

deal with data reference models for the interoperability

of AI systems, development of an AI basic security

standard, practice-oriented initial criticality checking of

AI systems, national implementation program “Trusted

AI'' to strengthen the European quality infrastructure,

and analysis and evaluation of use cases for

standardization needs.

In addition, the work provides extensive analysis on

the definition of AI as well as classification schemes to

evaluate AI-based systems.

The work is particularly interesting also for

spotlighting issues as the risk-based assessment of

applications, trustworthiness, ethical approach and AI

application lifecycle. In addition, in each section of the

roadmap, specific needs in the direction of

standardization are pinpointed.

The European Commission (EC) shaped a white

paper [138] setting out policies to achieve the uptake of

AI in the European Union (EU) and to address risks

associated with the use of AI technology. Along the

sections of the document, which refrains from

addressing the development and use of AI for military

purposes, particular attention is given to the opportunity

to create an ecosystem of excellence. Six actions have

been highlighted, among which: focusing on SMEs and

ensuring that each member state has a digital hub highly

specialized in AI; strengthening public-private

partnerships in AI, data and robotics; and promoting the

use of AI in the public sector. An overview of the most

IAC-21-B.1.4.4 Page 18 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

significant risks is also provided, with more emphasis

on ethical and trustworthy AI.

The National Science and Technology Council from

the USA’s Executive Office developed an AI Research

Development Plan in 2016, later updated in 2019 [139].

The Plan does not define specific research agendas for

Federal agency investments but highlights strategies to

reach a given portfolio. While it must be noted that the

utmost focus of the strategies is not on the

standardization, strategy 4 "Ensure the Safety and

Security of AI Systems'' and Strategy 6 "Measure and

Evaluate AI Technologies through Standards and

Benchmarks" are covering aspects strictly related to

standards and certifiability. It is worth mentioning great

attention to the development of shared public datasets

and open source libraries, as means to keep the

technological lead.

Although slightly different in scope, as more

oriented towards certification rather than

standardization, it is worth mentioning the DEEL White

Paper [140]. The document aims at “sharing knowledge,

identifying challenges for the certification of systems

using ML, and fostering the research effort”. A

thorough discussion on the features that an ML-based

system should possess to be certified is carried on,

leading to the identification of seven challenges to

tackle: probabilistic assessment, resilience,

specifiability, data quality, explainability, robustness,

and verifiability.

7.2. Issues and Techniques

In this section, we offer a brief discussion of the

potential unique issues one may encounter when

developing and operating a system that incorporates

ML. Whenever possible, we discuss some current

approaches to bridge these issues. This discussion is

meant to be illustrative to the reader and an

encouragement to explore the topics in further detail,

but it attempts to be comprehensive on neither scope nor

depth. Furthermore, the topic is under active research

and is likely to expand in the coming years.

7.2.1. Explainability

ML models, and particularly large models with lots

of free parameters such as large decision trees or NNs,

can act as black boxes. The process by which they

arrive at the final output can be too complex to be

directly interpreted, thus becoming as inscrutable as if

the model’s internals had been inaccessible in the first

place.

However, transparency, explainability, and

interpretability are very important for any technical

system with a moderate or large impact, be it in terms of

dollars or human lives. Therefore model explainability

is very important in fields such as aerospace, medicine,

insurance, banking, and more.

Explainability is a hard problem because of several

reasons. Firstly, it is user-dependent: the type of

explanation expected for example by an average user

will differ from that expected by a regulator or an

engineer. This leads to the question “How detailed must

the explanation be, and what must it cover?”. Secondly,

the expected outcome of transmitting an explanation can

be hard to define: the questions to answers are “Must

the receiver become more able to predict model output

after receiving explanations? Must the explanation point

univocally to the features of the input data that had the

largest impact on the produced results, and is this

limited to input data or does it also include training

data? Perhaps it should illustrate a counterfactual -

«What would need to change for the decision to have

been different?»? Or perhaps something else entirely?

And are the previous goals mutually exclusive?”

There are a huge number of techniques to answer

some of these and related questions. The field of

explainable AI or XAI for short is huge and expanding

rapidly. Providing an overview of this field is not within

the scope of the current publication, but we recommend

our readers to consult the Interpretable Machine

Learning book [141] or one of the numerous reviews on

the topic to learn more [142], [143].

7.2.2. Robustness and reliability

Reliability is the rate of failure of a system when

operating in nominal conditions (e.g. 10-9 catastrophic

failures per flight hour [144]). Since a rate of system

error can be extremely challenging to calculate without

operating the system, heuristic development rules like

no single point-of-failure are accepted as valid ways to

achieve the goal. This acceptance stems from either a

competent authority, which implicitly accepts the risk of

not properly achieving the desired reliability level or

historical data when available. Neither is a possibility

for current ML-based systems, due to an absence of

historically validated, robust, and widely accepted

heuristic design rules.

For Machine Learning systems, reliability comes

from two distinct factors: accuracy and robustness. An

ML classifier with higher accuracy is less likely to

misclassify an input, hence is more reliable..

Performance does not usually come into play for

classical software system’s reliability as accuracy for a

valid set of inputs and execution path is 100%. This

section does not concern itself with increasing model

accuracy, a topic that is the main focus of each

application-specific research field mentioned so far.

Accuracy for an ML model is calculated over the

data points in the test dataset and only those. While this

IAC-21-B.1.4.4 Page 19 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

is also true for classical software testing, the notion of

input equivalence classes provides assurance that the

software system will continue to perform acceptably for

inputs outside the test set. Correctness equivalence

classes for ML models do not currently exist. A similar

notion of robustness can be used instead. A robust

model has bounded accuracy loss for inputs that are

within a bounded distance of the input distribution. This

fact can be used to construct arguments for the

reliability of an ML model.

Although classical software systems are also tested

on a discrete batch of test cases, equivalence classes are

used to demonstrate correctness against the entire range

of operating conditions. Equivalence classes are sets of

input data in which verification of a property

(correctness in this case) on one point implies (or is

assumed to imply) verification of said property on the

whole set. If a string processing system does not fail on

a generic string, it will not fail on other strings, save for

edge cases, which are also tested. That notion of input

equivalence does not currently exist for input to ML

systems. Equivalence class discovery for random forest

models is a topic under active research [136], [145].

When demonstrating model robustness, several

problems arise:

Firstly, how does one quantify the distance between

input data? Although several measures exist, they are

often hard to relate to the implicit notions of input

distance humans have. It is easier to qualitatively say to

what degree an image does not depict a cat than it is to

quantify it in a single measure. This only becomes

harder for more abstract forms of input such as satellite

telemetry data. Thus, relating system-level

specifications to notions of input distance is sometimes

complex. For a given distance definition, formal

verification methods attempt to formally prove certain

properties of DL models, including robustness

[146]–[149]. They allow a user to build a model tolerant

to a certain distance between inputs. Equivalent research

exists for other ML models, such as random forests

(RFs) [136], but the literature is significantly less

developed. Note that these approaches allow a designer

to fight adversarial examples, a specific and concerning

failure mode for ML models [150], [151]. Nonetheless,

the literature on the generation and defeat of adversarial

examples is highly active and ever-evolving, as

measures and counter-measures and

counter-counter-measures get deployed. It is out of

scope for this review to delve any deeper into that.

Secondly, there is the well-known issue of

generalization. A model may offer very good

performance on a dataset and very poor performance on

the actual population, in the phenomenon known as

overfitting. The PAC-Bayes approach offers

generalization bounds that specify a minimum number

of samples from distribution for a desired performance

and training process reliability levels within that

distribution. These bounds, however, are often

extremely conservative, and improving them is another

active field of research [152]. Since it is hard to

appropriately quantify these bounds, the only recourse

for organisations to ensure performance is to collect

massive amounts of data, which is prohibitively

expensive or downright impossible in many cases.Since

generalization and robustness shortcomings are highly

model-specific, one approach to tackle them focuses on

applying mixtures of models working in tandem, known

as ensemble models, and selecting an output based on

the collective response of the ensemble [153], [154].

Thirdly, and also related to the second issue, there is

the phenomenon of domain drift [155]. Models do not

just overfit to a given dataset but also to the current

population. And, as time goes by, systems change. An

FDIR system monitoring battery health will see its

voltage decrease over time as the battery ages, and the

statistical distribution of deviations around the nominal

value is likely to change too. The performance of the

ML model will thus decrease over time as the world

changes around it. Fine-tuning on new data causes the

phenomenon known as catastrophic forgetting [156],

where the model loses performance on old and new

data. A solution is to retrain it from scratch on new data,

but this entails capturing that data and retraining the

model, which increases operating costs and risks in hard

to predict ways. Alternative solutions exist but they

come with their own drawbacks. Training a model on a

dataset representative of the whole system’s life cycle

can mitigate the issue, but requires larger models and

better data capture at the project’s start.

Lastly, models also overfit the specifics of the

system they’re trained for. A model trained for one

specific satellite may have issues adapting to another

satellite instance, or model. Version improvements such

as equipment changes may bring performance hits with

them too. While research fields like transfer learning,

domain adaptation and domain generalization [157]

attempt to address the issue, they are far from

universally reliable at the moment. This is particularly

concerning for the space industry, where mass

manufacturing and standardized equipment is the

exception rather than the norm and can pose a serious

challenge to the industry’s adoption of ML

technologies. Sometimes, when adapting to new

platforms, new input data will be available or new

output data may be required. In this case, the field of

transfer learning is applicable, which includes both

domain adaptation and domain generalization.

In short, despite the aforementioned techniques, ML

models are extremely brittle to deviations in input data

from the training dataset, and it can be assumed that

deviations from the training dataset will break the

system. Therefore, building propper datasets is a key

IAC-21-B.1.4.4 Page 20 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

task of any ML system designer or operator, a topic

which we address in the next section.

7.2.3. Dataset Construction

Datasets are the lifeblood of ML. Therefore it is only

right to have standards assigned for data to avoid

anomalies and have a perfect collection that will help

produce the right results.

Cappi et al. [158] propose a Dataset Definition

Standard (DDS) which, while not specifically geared

towards space activities, can be applied to EO data from

either payload or satellites. It aims to provide a standard

for training, validating or testing datasets. It explains in

detail the recommendations to be followed not only

while collecting data but also how to annotate it and

perform functions. The paper talks about many

important aspects any dataset should possess, from how

it must cover as many situations as possible that could

be encountered during model development to how a

history of every single change to every data must be

kept to help with traceability and avoid discrepancy.

The paper provides clear recommendations for labelling

and annotation of data and how the dataset should be

segregated for training, validation and testing.

The US Geological Survey [159] provides dataset

standards for their various operations like Biological,

Climate and Forecast and Mapping. Cleansing “dirty

data” is mentioned as a common problem faced by data

scientists. They also take it a step further with

geological mapping by producing a set of Parameter

standards to be followed while collecting data which

define a set of rules for individual parameters within the

dataset. The parameter standards cover a wide range of

qualities like the date/time, geographic coordinates,

codes, etc. the satellite data should contain. Report

[159] explains how exactly a topographical map of

anything in the US should be produced and one

important aspect of it is the data standards including

which standards the file formats of the data should be

stored in.

For the data quality standards, they delve into it by

discussing various components like currency,

consistency, completeness, and accuracy. The paper

covers every aspect of mapping data from dealing with

off grid and oversized maps, data sources and

resolutions to how cartographic features should be

interpreted.

GIS, remote sensing and satellite imagery are the

primary space operations for Earth. Hence they are also

the primary data-producing activities. Therefore the

standards for remote sensing are relatively well

developed compared to any other ML operations in the

space industry. [160] and [161] go in-depth about all

standards of remote sensing including the dataset

standards.

Di and Kobler [161] introduce NASA’s well

developed EO Systems’ Data Information Systems

(EOSDIS). As the EOSDIS will process data from

various fields it is not feasible for the system to deal

with every single data collected one by one. This has led

to EOSDIS establishing standards to deal specifically

with remote sensing data.

7.3. Recommendations

As outlined above, ML systems face a number of

issues precluding their application in many scenarios

where they would otherwise be useful.

We believe the fundamental research being carried

out on ML model robustness is of great interest and

recommend that any practitioner follow it closely. For

certain small-scale problems, work on formal

verification of ML models may already be enough to

ascertain that the network responds appropriately within

the input regime, and input data outside of this regime

can be purged by data verification systems implemented

in classical software. Further, we recommend that any

practitioner keep a careful watch for ways in which the

lifecycle operation of a system will deviate from the

training scenarios, and mitigate the risks issued from

model brittleness to these differences.

ML explainability is another core issue,

explainability of model decisions can and does take

precedence over model performance in scenarios with

high impact decisions, or where (human) learning from

the model’s decisions is key. Current model

explainability methods can offer insight into the

relevant features of input data used for a model’s

decision, but they can also provide misleading or

unhelpful signals. For applications where explainability

is an important feature of the system, dictionary, tree, or

kernel-based models and other easily explainable

methods should be compared with harder to explain

models for a performance-explainability trade-off.

National recommendations, white papers and initial

official standards in the AI and ML field attest to the

growing interest in the subject. While the scope of these

is much broader than the space sector alone, some

considerations can be applied to ML for space

applications too.

Data quality and availability will play an important

role in the adoption of ML across EO Operations, and

will certainly be demanded by supervisory and

regulatory agencies performing standardization and

certification. This need goes beyond the mere

abundance of data. Relevance, cleanliness, and

useability will require careful attention and control. To

achieve this, the industry can leverage work from other

fields such as the aforementioned dataset standards.

IAC-21-B.1.4.4 Page 21 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.

8. Discussion & Conclusion

This review work covered different domains in the

field of ML in EO Operations, such as ground

operations, enhanced GNC, on-board image processing,

FDIR and standardization. It analyzed the field’s status,

which can be thought of as a baseline, and highlighted

interesting trends.

We have found that growing evidence across many

application fields show that EO missions can benefit

from onboard usage of ML. Found case studies have

shown improvements in both performance and

autonomy of platforms. New capabilities such as

automated filtering of payload data by transmitting to

ground only relevant images can reduce downlink

bandwidth requirements, which is important for small

satellites but also mitigates saturation of

radio-frequency bands. Enhanced visual processing also

enables better visual-based navigation for spacecraft,

and RL holds promise as the key to unlock better

nonlinear controllers. Autonomous FDIR operations

enable better autonomous decision making for EO

missions making it possible for existing teams to

manage more operations in a leaner way, reducing

satellite operating costs.

ML algorithms place high (and growing)

requirements on onboard computers. This increases the

size, weight, and power of the platform, leaving less

room available for payload. The issue of optimizing ML

models both at the software and hardware level for

space applications emerges as a key challenge.

Fortunately for space platform operators this mirrors

and is an instance of, the much larger challenge of

deploying ML on edge platforms. The community can

leverage a huge and expanding pool of research and

experience.

From the on-board EO application perspective, ML

has been applied mainly to cloud detection and

novelty/change detection. Many examples of these

applications rely on vision-based solutions.

Vision-based ML algorithms and solutions represent a

domain that has been extensively explored in other

technical fields, from which EO applications could

borrow technical knowledge. However, there are few

examples also for SAR-based images and this could

potentially suggest that there is room for improvement

and development with these sensors in all-weather

all-day conditions.

On a parallel, yet strongly intertwined, track to

research and applications there is standardization.

Specific standards for ML in the space sector are yet to

be set. Rigorous research is being carried out on key

topics such as explainability, robustness, and data

structure construction. EO Operators developing ML

applications should leverage this field for increased

reliability and performance. Organizations looking to

emit standards and guidelines should also take into

consideration these research areas while avoiding

overprescribing solutions that could hinder the

effectiveness of standardization development.

Data availability represents another key factor that

links together all the ML-based EO Operations. Having

a larger amount of data to use for ML in EO Operations

could have a tremendous impact in fostering

technological progress, yielding benefits to all players,

from industry to academia to space agencies. Currently,

though, there are few open datasets and most are

specifically tailored for image processing applications

or visual navigation applications. While disclosing

private data can be against a single organization’s

short-term interests, the technological advancement

favoured by open datasets in a broader range of

applications is a very valuable long-term goal for the

space sector as a whole. We believe the field should

focus on creating and publicizing such open datasets,

and we encourage actors not subject to the profit

motive, like space agencies, to take a leading role in

achieving such a goal.

One recurrent finding across the review’s subtopics

is that, at the moment, few missions have used ML in

orbit for EO missions. This contrasts with other

industries such as the automotive sector, that have

embraced the technology, and can be attributed to the

long lead times and slow cycles of the space industry.

With the development of New Space and faster access

to space, we hope that these cycles can be accelerated as

technology demonstrators move in a short time from

test benches to orbit. Additionally, increasing the

number of missions can lead to improved data

collection and platform standardization, further

improving the effectiveness of ML deployments in the

EO Operations sector and the space industry at large.

Last but not least we have noticed that the

application of ML for EO operations is, often, behind

the ML state of the art. A number of techniques that

have seen thorough success in both research and

operational environments have yet to be publicly

applied to EO Operations problems, such as

transformer models on sequential and other data.

Likewise, we have found little evidence for the adoption

of Neural Architecture Search techniques, despite being

well suited to adapt the computational cost of NN to

resource-constrained platforms and achieve higher

performance. Similarly, very few applications can be

found also for the use of resource reduction techniques

like pruning, distillation, or quantization and with

regards to formal verification and other certified

robustness techniques.

We hope this review will be a useful resource to

researchers and operators for further deployment and

experimentation of ML in future complex, demanding

IAC-21-B.1.4.4 Page 22 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
EO missions that are more autonomous, transmit only
useful data, and require less involvement.
Acknowledgements
We were supported by three partner organisations:
Mindseed, Satsure, and CNES.
Mindseed [162] is a company focused on bridging
the gap between space and non-space organisations and
showcasing the benefits of space for all.
Satsure [163] is an innovative decision analytics
company leveraging advances in satellite remote
sensing and Machine Learning to achieve the United
Nations Sustainable Development Goals.
CNES [164] is the French National Space Agency,
with activities all over the space value chain. Their
Earth Observation Operations teams provided
invaluable feedback for our research.
Our three partners reviewed our research, and we are
deeply thankful for their collaboration.
References
 
[1] Q. Guo, B. Fu, P. Shi, T. Cudahy, J. Zhang, and
H. Xu, ‘Satellite Monitoring the
Spatial-Temporal Dynamics of Desertification
in Response to Climate Change and Human
Activities across the Ordos Plateau, China’,
Remote Sens., vol. 9, no. 6, Art. no. 6, Jun.
2017, doi: 10.3390/rs9060525.
[2] ‘Copernicus air quality index’, euronews.
https://www.euronews.com/embed/weather/air-q
uality (accessed Sep. 18, 2021).
[3] G. Pan, Y. Xu, and J. Ma, ‘The potential of CO2
satellite monitoring for climate governance: A
review’, J. Environ. Manage., vol. 277, p.
111423, Jan. 2021, doi:
10.1016/j.jenvman.2020.111423.
[4] A. S. Belward and J. O. Skøien, ‘Who launched
what, when and why; trends in global
land-cover observation capacity from civilian
earth observation satellites’, ISPRS J.
Photogramm. Remote Sens., vol. 103, pp.
115–128, May 2015, doi:
10.1016/j.isprsjprs.2014.03.009.
[5] A. Krizhevsky, I. Sutskever, and G. E. Hinton,
‘ImageNet Classification with Deep
Convolutional Neural Networks’, in Advances
in Neural Information Processing Systems,
2012, vol. 25. Accessed: Sep. 18, 2021.
[Online]. Available:
https://proceedings.neurips.cc/paper/2012/hash/
c399862d3b9d6b76c8436e924a68c45b-Abstract
.html
[6] B. Wang, Mesh-Transformer-JAX:
Model-Parallel Implementation of Transformer
Language Model with JAX. 2021. [Online].
Available:
https://github.com/kingoflolz/mesh-transformer-
jax
[7] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves,
I. Antonoglou, D. Wierstra, and M. Riedmiller,
‘Playing Atari with Deep Reinforcement
Learning’, ArXiv13125602 Cs, Dec. 2013,
Accessed: Sep. 18, 2021. [Online]. Available:
http://arxiv.org/abs/1312.5602
[8] F. Fourati and M.-S. Alouini, ‘Artificial
Intelligence for Satellite Communication: A
Review’, ArXiv210110899 Cs Eess, Jan. 2021,
Accessed: Feb. 20, 2021. [Online]. Available:
http://arxiv.org/abs/2101.10899
[9] ‘2018 - Mission Planning and Scheduling.pdf’.
Accessed: Dec. 14, 2020. [Online]. Available:
https://public.ccsds.org/Pubs/529x0g1.pdf
[10] X. Wang, G. Wu, L. Xing, and W. Pedrycz,
‘Agile Earth observation satellite scheduling
over 20 years: formulations, methods and future
directions’, IEEE Syst. J., vol. 15, no. 3, pp.
3881–3892, Sep. 2021, doi:
10.1109/JSYST.2020.2997050.
[11] A. Globus, J. Crawford, J. Lohn, and A. Pryor,
‘Scheduling Earth Observing Satellites with
Evolutionary Algorithms’, Dec. 2003.
[12] M. A. A. Mansour and M. M. Dessouky, ‘A
genetic algorithm approach for solving the daily
photograph selection problem of the SPOT5
satellite’, Comput. Ind. Eng., vol. 58, no. 3, pp.
509–520, Apr. 2010, doi:
10.1016/j.cie.2009.11.012.
[13] J. Li, J. Li, H. Chen, and N. Jing, ‘A data
transmission scheduling algorithm for
rapid-response earth-observing operations’,
Chin. J. Aeronaut., vol. 27, no. 2, pp. 349–364,
Apr. 2014, doi: 10.1016/j.cja.2014.02.014.
[14] J. Crawford, J. Lohn, and A. Pryor, ‘A
Comparison of Techniques for Scheduling Earth
Observing Satellites’, p. 8.
[15] J. Bonnet, M.-P. Gleizes, E. Kaddoum, S.
Rainjonneau, and G. Flandin, ‘Multi-satellite
Mission Planning Using a Self-Adaptive
Multi-agent System’, in 2015 IEEE 9th
International Conference on Self-Adaptive and
Self-Organizing Systems, Cambridge, MA,
USA, Sep. 2015, pp. 11–20. doi:
10.1109/SASO.2015.9.
[16] H. Wang, ‘Online scheduling of image satellites
based on neural networks and deep
reinforcement learning’, p. 9.
[17] S. Peng, H. Chen, C. Du, J. Li, and N. Jing,
‘Onboard Observation Task Planning for an
Autonomous Earth Observation Satellite Using
IAC-21-B.1.4.4 Page 23 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
Long Short-Term Memory’, IEEE Access, vol.
6, pp. 65118–65129, 2018, doi:
10.1109/ACCESS.2018.2877687.
[18] Y. Du, T. Wang, B. Xin, L. Wang, Y. Chen, and
L. Xing, ‘A Data-Driven Parallel Scheduling
Approach for Multiple Agile Earth Observation
Satellites’, IEEE Trans. Evol. Comput., vol. 24,
no. 4, pp. 679–693, Aug. 2020, doi:
10.1109/TEVC.2019.2934148.
[19] X. Liu, ‘Mission schedule of agile satellites
based on Proximal Policy Optimization
Algorithm’, ArXiv200702352 Cs, Jul. 2020,
Accessed: Sep. 28, 2021. [Online]. Available:
http://arxiv.org/abs/2007.02352
[20] A. Hadj-Salah, R. Verdier, C. Caron, M. Picard,
and M. Capelle, ‘Schedule Earth Observation
satellites with Deep Reinforcement Learning’,
ArXiv191105696 Cs, Nov. 2019, Accessed: Sep.
28, 2021. [Online]. Available:
http://arxiv.org/abs/1911.05696
[21] A. Hadj-Salah, J. Guerra, M. Picard, and M.
Capelle, ‘Towards operational application of
Deep Reinforcement Learning to Earth
Observation satellite scheduling’. Aug. 30,
2020. Accessed: Sep. 28, 2021. [Online].
Available:
https://hal.archives-ouvertes.fr/hal-02925740
[22] Y. Song, Z. Zhou, Z. Zhang, F. Yao, and Y.
Chen, ‘A framework involving MEC: imaging
satellites mission planning’, Neural Comput.
Appl., vol. 32, no. 19, pp. 15329–15340, Oct.
2020, doi: 10.1007/s00521-019-04047-6.
[23] S.-F. Wu, C. J. H. Engelen, Q.-P. Chu, R.
Babuška, J. A. Mulder, and G. Ortega, ‘Fuzzy
logic based attitude control of the spacecraft
X-38 along a nominal re-entry trajectory’,
Control Eng. Pract., vol. 9, no. 7, pp. 699–707,
Jul. 2001, doi:
10.1016/S0967-0661(01)00036-3.
[24] G. Ortega, ‘Fuzzy logic techniques for
rendezvous and docking of two geostationary
satellites’, Telemat. Inform., vol. 12, no. 3, pp.
213–227, Jun. 1995, doi:
10.1016/0736-5853(95)00013-5.
[25] C.-H. Cheng, S.-L. Shu, and P.-J. Cheng,
‘Attitude control of a satellite using fuzzy
controllers’, Expert Syst. Appl., vol. 36, no. 3,
Part 2, pp. 6613–6620, Apr. 2009, doi:
10.1016/j.eswa.2008.08.053.
[26] D. Izzo, M. Märtens, and B. Pan, ‘A Survey on
Artificial Intelligence Trends in Spacecraft
Guidance Dynamics and Control’,
ArXiv181202948 Cs, Dec. 2018, Accessed: Dec.
12, 2020. [Online]. Available:
http://arxiv.org/abs/1812.02948
[27] D. Izzo and E. Öztürk, ‘Real-Time Guidance for
Low-Thrust Transfers Using Deep Neural
Networks’, J. Guid. Control Dyn., vol. 44, no. 2,
pp. 315–327, Feb. 2021, doi:
10.2514/1.G005254.
[28] L. Federici, B. Benedikter, and A. Zavoli, ‘Deep
Learning Techniques for Autonomous
Spacecraft Guidance During Proximity
Operations’, J. Spacecr. Rockets, vol. 0, no. 0,
pp. 1–12, doi: 10.2514/1.A35076.
[29] K. Hovell and S. Ulrich, ‘Deep Reinforcement
Learning for Spacecraft Proximity Operations
Guidance’, J. Spacecr. Rockets, vol. 58, no. 2,
pp. 254–264, 2021, doi: 10.2514/1.A34838.
[30] D. Lee, ‘Guidance, navigation and control
system for autonomous proximity operations
and docking of spacecraft’, p. 149.
[31] K. Hovell and S. Ulrich, ‘On Deep
Reinforcement Learning for Spacecraft
Guidance’, in AIAA Scitech 2020 Forum,
American Institute of Aeronautics and
Astronautics. doi: 10.2514/6.2020-1600.
[32] S. Koizumi, Y. Kikuya, K. Sasaki, Y. Masuda,
Y. Iwasaki, K. Watanabe, Y. Yatsu, and S.
Matsunaga, ‘Development of Attitude Sensor
using Deep Learning’, p. 8.
[33] S.-W. Kim, S.-Y. Park, and C. Park, ‘Spacecraft
attitude control using neuro-fuzzy
approximation of the optimal controllers’, Adv.
Space Res., vol. 57, no. 1, pp. 137–152, Jan.
2016, doi: 10.1016/j.asr.2015.09.016.
[34] A. Joghataie, ‘Neural Networks and Fuzzy
Logic for Structural Control’, University of
Illinois Engineering Experiment Station.
College of Engineering. University of Illinois at
Urbana-Champaign, 1994. [Online]. Available:
http://hdl.handle.net/2142/14218
[35] M. Azarbad, H. Azami, S. Sanei, and A.
Ebrahimzadeh, ‘New Neural Network-based
Approaches for GPS GDOP Classification
based on Neuro-Fuzzy Inference System, Radial
Basis Function, and Improved Bee Algorithm’,
Appl. Soft Comput., vol. 25, pp. 285–292, Dec.
2014, doi: 10.1016/j.asoc.2014.09.022.
[36] ‘Comparative study of Classical and Fuzzy PID
Attitude Control System with Extended Kalman
Filter feedback for nanosatellites. — IAF
Digital Library’.
https://dl.iafastro.directory/event/IAC-2018/pap
er/44553/ (accessed Sep. 14, 2021).
[37] Y. WANG, Z. MA, Y. YANG, Z. WANG, and L.
TANG, ‘A New Spacecraft Attitude
Stabilization Mechanism Using Deep
Reinforcement Learning Method’, p. 13 pages,
2019, doi: 10.13009/EUCASS2019-33.
[38] D. Yadava, R. Hosangadi, S. Krishna, P.
Paliwal, and A. Jain, ‘Attitude control of a
IAC-21-B.1.4.4 Page 24 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
nanosatellite system using reinforcement
learning and neural networks’, in 2018 IEEE
Aerospace Conference, Big Sky, MT, Mar. 2018,
pp. 1–8. doi: 10.1109/AERO.2018.8396409.
[39] S. Jaekel and B. Scholz, ‘Utilizing Artificial
Intelligence to achieve a robust architecture for
future robotic spacecraft’, in 2015 IEEE
Aerospace Conference, Big Sky, MT, Mar. 2015,
pp. 1–14. doi: 10.1109/AERO.2015.7119180.
[40] S. Fuertes, B. Pilastre, and S. D’Escrivan,
‘Performance assessment of NOSTRADAMUS
& other machine learning-based telemetry
monitoring systems on a spacecraft anomalies
database’, presented at the 2018 SpaceOps
Conference, Marseille, France, May 2018. doi:
10.2514/6.2018-2559.
[41] Ivano Verzola, Alessandro Donati, J.-A.
Martínez Heras, M. Schubert, and Laszlo
Somodi, ‘Project Sybil : A Novelty Detection
System for Human Spaceflight Operations, in :
Proc. Int. Conf. Space Operations’, presented at
the SpaceOps, 2016.
[42] C. O’Meara, L. Schlag, L. Faltenbacher, and M.
Wickler, ‘ATHMoS: Automated Telemetry
Health Monitoring System at GSOC using
Outlier Detection and Supervised Machine
Learning’, May 2016. doi:
10.2514/6.2016-2347.
[43] B. Pilastre, ‘Estimation parcimonieuse et
apprentissage de dictionnaires pour la détection
d’anomalies multivariées dans des données
mixtes de télémesure satellites’, phd, 2020.
Accessed: Sep. 13, 2021. [Online]. Available:
https://oatao.univ-toulouse.fr/27738/
[44] D. L. Iverson, ‘System Health Monitoring for
Space Mission Operations’, in 2008 IEEE
Aerospace Conference, Big Sky, MT, USA, Mar.
2008, pp. 1–8. doi:
10.1109/AERO.2008.4526646.
[45] B. W. Ricks, ‘METHODS FOR
PROBABILISTIC FAULT DIAGNOSIS: AN
ELECTRICAL POWER SYSTEM CASE
STUDY’, Mar. 2021, vol. Vol 1. No. 1.
[46] J.-G. Meß, ‘Techniques of Artificial Intelligence
for Space Applications - A Survey’, presented at
the European Workshop on On-Board Data
Processing (OBDP2019), Feb. 2019.
[47] J. Schumann, O. J. Mengshoel, and T. Mbaya,
‘Integrated Software and Sensor Health
Management for Small Spacecraft’, in 2011
IEEE Fourth International Conference on Space
Mission Challenges for Information Technology,
Palo Alto, CA, USA, Aug. 2011, pp. 77–84. doi:
10.1109/SMC-IT.2011.25.
[48] Ruonan Liu, Boyuan Yang, Enrico Zio, and
Xuefeng Chen, ‘Artificial intelligence for fault
diagnosis of rotating machinery: A review’.
[49] S.P.M (Sander) Voss, ‘Application of deep
learning for spacecraft fault detection and
isolation’, Master Thesis.
[50] D. Codetta-Raiteri and L. Portinale, ‘Dynamic
Bayesian Networks for Fault Detection,
Identification, and Recovery in Autonomous
Spacecraft’, IEEE Trans. Syst. Man Cybern.
Syst., vol. 45, no. 1, pp. 13–24, Jan. 2015, doi:
10.1109/TSMC.2014.2323212.
[51] ‘Periodic Reporting for period 1 - MiRAGE
(Enhanced mission autonomy through on-board
Artificial Intelligence for next generation
spacecraft) | H2020 | CORDIS | European
Commission’.
https://cordis.europa.eu/project/id/816442/report
ing (accessed Sep. 19, 2021).
[52] ‘Georges Labrèchea, Tanguy Sotoa, Evridiki
Ntagioua, David Evansa, In-Flight Training of a
FDIR Model with Online Machine Learning on
the OPS-SAT Spacecraft, 2021-
SpaceOps-2021,13,x1450’.
https://github.com/georgeslabreche/opssat-orbita
i/find/main (accessed Sep. 12, 2021).
[53] olanleed, MochiMochi. 2021. Accessed: Sep.
29, 2021. [Online]. Available:
https://github.com/olanleed/MochiMochi
[54] S. Mahajan and B. Fataniya, ‘Cloud detection
methodologies: variants and development—a
review’, Complex Intell. Syst., vol. 6, no. 2, pp.
251–261, Jul. 2020, doi:
10.1007/s40747-019-00128-0.
[55] Z. Zhang, A. Iwasaki, G. Xu, and J. Song,
‘Cloud detection on small satellites based on
lightweight U-net and image compression’, J.
Appl. Remote Sens., vol. 13, no. 2, p. 026502,
Apr. 2019, doi: 10.1117/1.JRS.13.026502.
[56] R. Hinz, J. I. Bravo, M. Kerr, C. Marcos, A.
Latorre, and F. Membibre, ‘EO-ALERT:
Machine Learning-Based On-Board Satellite
Processing for Very-Low Latency Convective
Storm Nowcasting’, p. 1.
[57] A. N. Srivastava, Onboard Detection of Snow,
Ice, Clouds and Other Geophysical Processes
Using Kernel Methods. 2003.
[58] G. Giuffrida, L. Diana, F. de Gioia, G. Benelli,
G. Meoni, M. Donati,and L. Fanucci,
‘CloudScout: A Deep Neural Network for
On-Board Cloud Detection on Hyperspectral
Images’, Remote Sens., vol. 12, no. 14, Art. no.
14, Jan. 2020, doi: 10.3390/rs12142205.
[59] A. Maskey and M. Cho, ‘CubeSatNet: Ultralight
Convolutional Neural Network designed for
on-orbit binary image classification on a 1U
CubeSat’, Eng. Appl. Artif. Intell., vol. 96, p.
103952, Nov. 2020, doi:
IAC-21-B.1.4.4 Page 25 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
10.1016/j.engappai.2020.103952.
[60] M. Ireland, ‘Integrating AI Techniques Into
Future Nanosatellite Onboard Data Processing’,
p. 30.
[61] R. Castaño, S. A. Chien, K. L. Wagstaff, and T.
M. Stough, ‘On-board analysis of uncalibrated
data for a spacecraft at mars’, presented at the
International Conference on Knowledge
Discovery and Data Mining, San Jose,
California, USA, Aug. 2007. doi:
10.1145/1281192.1281291.
[62] F. Férésin, E. Kervennic, Y. Bobichon, E.
Lemaire, N. Abderrahmane, G. Bahl, I. Grenet,
M. Moretti, and M. Benguigui, ‘In space image
processing using AI embedded on system on
module : example of OPS-SAT cloud
segmentation’, presented at the 2nd European
Workshop on On-Board Data Processing, 2021.
Accessed: Sep. 28, 2021. [Online]. Available:
https://www.irt-saintexupery.com/two-premieres
-in-on-board-artificial-intelligence-on-satellites/
[63] K. L. Wagstaff, S. Chien, A. Altinok, U.
Rebbapragada, D. Thompson, S. Schaffer, and
D. Tran, ‘Cloud Filtering and Novelty Detection
using Onboard Machine Learning for the EO-1
Spacecraft’, Int. Jt. Conf. Artif. Intell., p. 4,
2017.
[64] S. Chien, J. Doubleday, D. R. Thompson, K. L.
Wagstaff, J. Bellardo, C. Francis, E.
Baumgarten, A. Williams, E. Yee, E. Stanton
and J. Piug-Suari, ‘Onboard Autonomy on the
Intelligent Payload Experiment CubeSat
Mission’, J. Aerosp. Inf. Syst., vol. 14, no. 6, pp.
307–315, 2017, doi: 10.2514/1.I010386.
[65] P. Deepan and L. R. Sudha, ‘Object
Classification of Remote Sensing Image Using
Deep Convolutional Neural Network’, in The
Cognitive Approach in Cloud Computing and
Internet of Things Technologies for Surveillance
Tracking Systems, Elsevier, 2020, pp. 107–120.
doi: 10.1016/B978-0-12-816385-6.00008-8.
[66] A. P. Arechiga, A. J. Michaels, and J. T. Black,
‘Onboard Image Processing for Small
Satellites’, in NAECON 2018 - IEEE National
Aerospace and Electronics Conference, Jul.
2018, pp. 234–240. doi:
10.1109/NAECON.2018.8556744.
[67] J. Manning, D. Langerman, B. Ramesh, E.
Gretok, C. Wilson, A. George, J. MacKinnon,
and G. Crum, ‘Machine-Learning Space
Applications on SmallSat Platforms with
TensorFlow’, Small Satell. Conf., Aug. 2018,
[Online]. Available:
https://digitalcommons.usu.edu/smallsat/2018/al
l2018/458
[68] J. Goodwill, D. Wilson, S. Sabogal, A. D.
George, and C. Wilson, ‘Adaptively Lossy
Image Compression for Onboard Processing’, in
2020 IEEE Aerospace Conference, Mar. 2020,
pp. 1–15. doi:
10.1109/AERO47225.2020.9172536.
[69] T. Vladimirova and S. Atek, ‘A New Lossless
Compression Method for Small Satellite
On-Board Imaging’, University of Surrey 
Guildford, Surrey, GU2 7 XH  United Kingdom,
2002. doi: 10.1142/9789812776266_0038.
[70] Y. Cai, Y. Hu, M. Siegel, S. J. Gollapalli, A. R.
Venugopal, and U. Bardak, ‘Onboard Feature
Indexing from Satellite Lidar Images’, p. 4.
[71] N. D. Lane, S. Bhattacharya, P. Georgiev, C.
Forlivesi, and F. Kawsar, ‘An Early Resource
Characterization of Deep Learning on
Wearables, Smartphones and Internet-of-Things
Devices’, in Proceedings of the 2015
International Workshop on Internet of Things
towards Applications, Seoul South Korea, Nov.
2015, pp. 7–12. doi: 10.1145/2820975.2820980.
[72] H. Han, S. Lee, J. Im, M. Kim, M.-I. Lee, M. H.
Ahn, and S.-R. Chung, ‘Detection of Convective
Initiation Using Meteorological Imager Onboard
Communication, Ocean, and Meteorological
Satellite Based on Machine Learning
Approaches’, Remote Sens., vol. 7, no. 7, Art.
no. 7, Jul. 2015, doi: 10.3390/rs70709184.
[73] D. Blalock, J. J. G. Ortiz, J. Frankle, and J.
Guttag, ‘What is the State of Neural Network
Pruning?’, ArXiv200303033 Cs Stat, Mar. 2020,
Accessed: Aug. 06, 2021. [Online]. Available:
http://arxiv.org/abs/2003.03033
[74] H. Wang, C. Qin, Y. Zhang, and Y. Fu,
‘Emerging Paradigms of Neural Network
Pruning’, ArXiv210306460 Cs, Mar. 2021,
Accessed: Aug. 06, 2021. [Online]. Available:
http://arxiv.org/abs/2103.06460
[75] J. Frankle, G. K. Dziugaite, D. M. Roy, and M.
Carbin, ‘Pruning Neural Networks at
Initialization: Why are We Missing the Mark?’,
ArXiv200908576 Cs Stat, Mar. 2021, Accessed:
Aug. 06, 2021. [Online]. Available:
http://arxiv.org/abs/2009.08576
[76] T. He, Y. Fan, Y. Qian, T. Tan, and K. Yu,
‘Reshaping deep neural network for fast
decoding by node-pruning’, in 2014 IEEE
International Conference on Acoustics, Speech
and Signal Processing (ICASSP), Florence,
Italy, May 2014, pp. 245–249. doi:
10.1109/ICASSP.2014.6853595.
[77] Y. Tang, J. Ji, S. Gao, H. Dai, Y. Yu, and Y.
Todo, ‘A Pruning Neural Network Model in
Credit Classification Analysis’, Comput. Intell.
Neurosci., vol. 2018, pp. 1–22, 2018, doi:
10.1155/2018/9390410.
IAC-21-B.1.4.4 Page 26 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
[78] A. Lazarevic and Z. Obradovic, ‘Effective
pruning of neural network classifier ensembles’,
in IJCNN’01. International Joint Conference on
Neural Networks. Proceedings (Cat.
No.01CH37222), Washington, DC, USA, 2001,
vol. 2, pp. 796–801. doi:
10.1109/IJCNN.2001.939461.
[79] ‘Pruning in Keras example | TensorFlow Model
Optimization’, TensorFlow.
https://www.tensorflow.org/model_optimization
/guide/pruning/pruning_with_keras (accessed
Aug. 06, 2021).
[80] D. Browne, M. Giering, and S. Prestwich,
‘PulseNetOne: Fast Unsupervised Pruning of
Convolutional Neural Networks for Remote
Sensing’, Remote Sens., vol. 12, no. 7, Art. no.
7, Jan. 2020, doi: 10.3390/rs12071092.
[81] M. Castelluccio, G. Poggi, C. Sansone, and L.
Verdoliva, ‘Land Use Classification in Remote
Sensing Images by Convolutional Neural
Networks’, ArXiv150800092 Cs, Aug. 2015,
Accessed: Aug. 07, 2021. [Online]. Available:
http://arxiv.org/abs/1508.00092
[82] T. Kavzoglu and P. M. Mather, ‘Pruning
artificial neural networks: An example using
land cover classification of multi-sensor
images’, Int. J. Remote Sens., vol. 20, no. 14,
pp. 2787–2803, Jan. 1999, doi:
10.1080/014311699211796.
[83] E. Maggiori, Y. Tarabalka, G. Charpiat, and P.
Alliez, ‘Convolutional Neural Networks for
Large-Scale Remote-Sensing Image
Classification’, IEEE Trans. Geosci. Remote
Sens., vol. 55, no. 2, pp. 645–657, Feb. 2017,
doi: 10.1109/TGRS.2016.2612821.
[84] G. Pitsis, G. Tsagkatakis, C. Kozanitis, I.
Kalomoiris, A. Ioannou, A. Dollas, M. G.H.
Katevenis, and P. Tsakalides, ‘Efficient
Convolutional Neural Network Weight
Compression for Space Data Classification on
Multi-fpga Platforms’, in ICASSP 2019 - 2019
IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP),
Brighton, United Kingdom, May 2019, pp.
3917–3921. doi:
10.1109/ICASSP.2019.8682732.
[85] S. Zhang, G. Wu, J. Gu, and J. Han, ‘Pruning
Convolutional Neural Networks with an
Attention Mechanism for Remote Sensing
Image Classification’, Electronics, vol. 9, no. 8,
p. 1209, Jul. 2020, doi:
10.3390/electronics9081209.
[86] Q. Tan, J. Ling, J. Hu, X. Qin, and J. Hu,
‘Vehicle Detection in High Resolution Satellite
Remote Sensing Images Based on Deep
Learning’, IEEE Access, vol. 8, pp.
153394–153402, 2020, doi:
10.1109/ACCESS.2020.3017894.
[87] N. Ma, X. Yu, Y. Peng, and S. Wang, ‘A
Lightweight Hyperspectral Image Anomaly
Detector for Real-Time Mission’, Remote Sens.,
vol. 11, no. 13, p. 1622, Jul. 2019, doi:
10.3390/rs11131622.
[88] A. Goel, C. Tung, Y.-H. Lu, and G. K.
Thiruvathukal, ‘A Survey of Methods for
Low-Power Deep Learning and Computer
Vision’, in 2020 IEEE 6th World Forum on
Internet of Things (WF-IoT), Jun. 2020, pp. 1–6.
doi: 10.1109/WF-IoT48130.2020.9221198.
[89] F. N. Iandola, S. Han, M. W. Moskewicz, K.
Ashraf, W. J. Dally, and K. Keutzer,
‘SqueezeNet: AlexNet-level accuracy with 50x
fewer parameters and <0.5MB model size’,
ArXiv160207360 Cs, Nov. 2016, Accessed: Sep.
30, 2021. [Online]. Available:
http://arxiv.org/abs/1602.07360
[90] A. G. Howard, M. Zhu, B. Chen, D.
Kalenichenko, W. Wang, T. Weyand, M.
Andreetto, and H. Adam, ‘MobileNets: Efficient
Convolutional Neural Networks for Mobile
Vision Applications’, ArXiv170404861 Cs, Apr.
2017, Accessed: Sep. 30, 2021. [Online].
Available: http://arxiv.org/abs/1704.04861
[91] ‘Multitask Classification of Remote Sensing
Scenes Using Deep Neural Networks | IEEE
Conference Publication | IEEE Xplore’.
https://ieeexplore.ieee.org/document/8518874
(accessed Sep. 30, 2021).
[92] ‘SqueezeNet with Attention for Remote Sensing
Scene Classification | IEEE Conference
Publication | IEEE Xplore’.
https://ieeexplore.ieee.org/abstract/document/90
96876 (accessed Sep. 30, 2021).
[93] ‘Multi-Scale Convolutional Neural Network for
Remote Sensing Scene Classification | IEEE
Conference Publication | IEEE Xplore’.
https://ieeexplore.ieee.org/abstract/document/85
00107 (accessed Sep. 30, 2021).
[94] A. Poortinga, N. S. Thwal, N. Khanal, T. Mayer,
B. Bhandari, K. Markert, A. P. Nicolau, J.
Dilger, K. Tenneson, N. Clinton, and D. Saah,
‘Mapping sugarcane in Thailand using transfer
learning, a lightweight convolutional neural
network, NICFI high resolution satellite
imagery and Google Earth Engine’, ISPRS Open
J. Photogramm. Remote Sens., vol. 1, p.
100003, Oct. 2021, doi:
10.1016/j.ophoto.2021.100003.
[95] B. Zhang, Y. Zhang, and S. Wang, ‘A
Lightweight and Discriminative Model for
Remote Sensing Scene Classification With
Multidilation Pooling Module’, IEEE J. Sel.
IAC-21-B.1.4.4 Page 27 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
Top. Appl. Earth Obs. Remote Sens., vol. 12, no.
8, pp. 2636–2653, Aug. 2019, doi:
10.1109/JSTARS.2019.2919317.
[96] D. Yu, Q. Xu, H. Guo, C. Zhao, Y. Lin, and D.
Li, ‘An Efficient and Lightweight Convolutional
Neural Network for Remote Sensing Image
Scene Classification’, Sensors, vol. 20, no. 7, p.
1999, Apr. 2020, doi: 10.3390/s20071999.
[97] T. Hoeser, F. Bachofer, and C. Kuenzer, ‘Object
Detection and Image Segmentation with Deep
Learning on Earth Observation Data: A
Review—Part II: Applications’, Remote Sens.,
vol. 12, no. 18, Art. no. 18, Jan. 2020, doi:
10.3390/rs12183053.
[98] X. He, K. Zhao, and X. Chu, ‘AutoML: A
survey of the state-of-the-art’, Knowl.-Based
Syst., vol. 212, p. 106622, Jan. 2021, doi:
10.1016/j.knosys.2020.106622.
[99] M. Chan, D. Scarafoni, R. Duarte, J. Thornton,
and L. Skelly, ‘Learning Network Architectures
of Deep CNNs Under Resource Constraints’, in
2018 IEEE/CVF Conference on Computer
Vision and Pattern Recognition Workshops
(CVPRW), Salt Lake City, UT, USA, Jun. 2018,
pp. 1784–17847. doi:
10.1109/CVPRW.2018.00222.
[100] B. Zoph and Q. V. Le, ‘Neural Architecture
Search with Reinforcement Learning’,
ArXiv161101578 Cs, Feb. 2017, Accessed: Sep.
18, 2021. [Online]. Available:
http://arxiv.org/abs/1611.01578
[101] B. Baker, O. Gupta, N. Naik, and R. Raskar,
‘Designing Neural Network Architectures using
Reinforcement Learning’, ArXiv161102167 Cs,
Mar. 2017, Accessed: Sep. 18, 2021. [Online].
Available: http://arxiv.org/abs/1611.02167
[102] T. Elsken, J. H. Metzen, and F. Hutter, ‘Neural
Architecture Search: A Survey’,
ArXiv180805377 Cs Stat, Apr. 2019, Accessed:
Sep. 18, 2021. [Online]. Available:
http://arxiv.org/abs/1808.05377
[103] W. Liang, J. Li, W. Diao, X. Sun, K. Fu, and Y.
Wu, ‘FGATR-Net: Automatic Network
Architecture Design for Fine-Grained Aircraft
Type Recognition in Remote Sensing Images’,
Remote Sens., vol. 12, no. 24, p. 4187, Dec.
2020, doi: 10.3390/rs12244187.
[104] M. Tan, B. Chen, R. Pang, V. Vasudevan, M.
Sandler, A. Howard, and Q. V. Le, ‘MnasNet:
Platform-Aware Neural Architecture Search for
Mobile’, in 2019 IEEE/CVF Conference on
Computer Vision and Pattern Recognition
(CVPR), Long Beach, CA, USA, Jun. 2019, pp.
2815–2823. doi: 10.1109/CVPR.2019.00293.
[105] L. J. Ba and R. Caruana, ‘Do Deep Nets Really
Need to be Deep?’, Dec. 2013, Accessed: Jun.
28, 2021. [Online]. Available:
https://arxiv.org/abs/1312.6184v7
[106] F. de Vieilleville, A. Lagrange, R. Ruiloba, and
S. May, ‘Towards Distillation of Deep Neural
Networks for Satellite On-Board Image
Segmentation’, Int. Arch. Photogramm. Remote
Sens. Spat. Inf. Sci., vol. 43, pp. 1553–1559,
2020.
[107] G. Chen, X. Zhang, X. Tan, Y. Cheng, F. Dai, K.
Zhu, Y. Gong, and Q. Wang, ‘Training Small
Networks for Scene Classification of Remote
Sensing Images via Knowledge Distillation’,
Remote Sens., vol. 10, no. 5, p. 719, May 2018,
doi: 10.3390/rs10050719.
[108] H. Bazzi, D. Ienco, N. Baghdadi, M. Zribi, and
V. Demarez, ‘Distilling Before Refine:
Spatio-Temporal Transfer Learning for Mapping
Irrigated Areas Using Sentinel-1 Time Series’,
IEEE Geosci. Remote Sens. Lett., vol. 17, no.
11, pp. 1909–1913, Nov. 2020, doi:
10.1109/LGRS.2019.2960625.
[109] Y. Chen, Y. Bian, X. Xiao, Y. Rong, T. Xu, and
J. Huang, ‘On Self-Distilling Graph Neural
Network’, ArXiv201102255 Cs Stat, Apr. 2021,
Accessed: Aug. 07, 2021. [Online]. Available:
http://arxiv.org/abs/2011.02255
[110] Y. Chai, K. Fu, X. Sun, W. Diao, Z. Yan, Y.
Feng, and L. Wang, ‘Compact Cloud Detection
with Bidirectional Self-Attention Knowledge
Distillation’, Remote Sens., vol. 12, no. 17, p.
2770, Aug. 2020, doi: 10.3390/rs12172770.
[111] V. Kothari, E. Liberis, and N. D. Lane, ‘The
Final Frontier: Deep Learning in Space’,
ArXiv200110362 Cs Eess, Feb. 2020, Accessed:
Jul. 15, 2021. [Online]. Available:
http://arxiv.org/abs/2001.10362
[112] R. Huq, M. Islam, and S. Siddique, ‘AI-OBC:
Conceptual Design of a Deep Neural Network
based Next Generation Onboard Computing
Architecture for Satellite Systems’, p. 8.
[113] M. Esposito, S. S. Conticello, M. Pastena, and
B. C. Domínguez, ‘In-orbit demonstration of
artificial intelligence applied to hyperspectral
and thermal sensing from space’, in CubeSats
and SmallSats for Remote Sensing III, Aug.
2019, vol. 11131, p. 111310C. doi:
10.1117/12.2532262.
[114] F. C. Bruhn, N. Tsog, F. Kunkel, and O. Flordal,
‘Enabling radiation tolerant heterogeneous
GPU‑based onboard data processing in space’,
Vol01234567891 3CEAS Space Journal, vol. 12,
pp. 551–564, 2020.
[115] S. Mittal, ‘A Survey on optimized
implementation of deep learning models on the
NVIDIA Jetson platform’, J. Syst. Archit., vol.
97, pp. 428–442, Aug. 2019, doi:
IAC-21-B.1.4.4 Page 28 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
10.1016/j.sysarc.2019.01.011.
[116] N. Buonaiuto, M. Louie, R. Mital, D. Mateik, R.
Sivilli, A. Bhopale, C. Kief, and B. Zufelt,
‘Satellite Identification Imaging for Small
Satellites Using NVIDIA’, p. 12, 2017.
[117] J. J. Hernández-Gómez, G. A. Yañez-Casas, A.
M. Torres-Lara, C. Couder-Castañeda, M. G.
Orozco-del-Castillo, J. C. Valdiviezo-Navarro, I.
Medina, A. Solís-Santomé, D.
Vázquez-Álvarez, and P. I. Chávez-López,
‘Conceptual low-cost on-board high
performance computing in CubeSat
nanosatellites for pattern recognition in Earth’s
remote sensing’, pp. 114–104. doi:
10.29007/8d25.
[118] J. Goodwill, G. Crum, J. MacKinnon, C.
Brewer, M. Monaghan, T. Wise, and C. Wilson,
‘NASA SpaceCube Edge TPU SmallSat Card
for Autonomous Operations and Onboard
Science-Data Analysis’, p. 13.
[119] Y. Guo, ‘A Survey on Methods and Theories of
Quantized Neural Networks’, ArXiv180804752
Cs Stat, Dec. 2018, Accessed: Sep. 18, 2021.
[Online]. Available:
http://arxiv.org/abs/1808.04752
[120] H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N.
Sebe, ‘Binary Neural Networks: A Survey’,
Pattern Recognit., vol. 105, p. 107281, Sep.
2020, doi: 10.1016/j.patcog.2020.107281.
[121] T. Simons and D.-J. Lee, ‘A Review of
Binarized Neural Networks’, Electronics, vol. 8,
no. 6, p. 661, Jun. 2019, doi:
10.3390/electronics8060661.
[122] M. Courbariaux, Y. Bengio, and J.-P. David,
‘BinaryConnect: Training Deep Neural
Networks with binary weights during
propagations’, in Advances in Neural
Information Processing Systems, 2015, vol. 28.
[Online]. Available:
https://proceedings.neurips.cc/paper/2015/file/3
e15cc11f979ed25912dff5b0669f2cd-Paper.pdf
[123] M. Rastegari, V. Ordonez, J. Redmon, and A.
Farhadi, ‘XNOR-Net: ImageNet Classification
Using Binary Convolutional Neural Networks’,
ArXiv160305279 Cs, Aug. 2016, Accessed: Sep.
18, 2021. [Online]. Available:
http://arxiv.org/abs/1603.05279
[124] C. Zhu, S. Han, H. Mao, and W. J. Dally,
‘Trained Ternary Quantization’,
ArXiv161201064 Cs, Feb. 2017, Accessed: Sep.
18, 2021. [Online]. Available:
http://arxiv.org/abs/1612.01064
[125] K. Simonyan and A. Zisserman, ‘Very Deep
Convolutional Networks for Large-Scale Image
Recognition’, ArXiv14091556 Cs, Apr. 2015,
Accessed: Sep. 18, 2021. [Online]. Available:
http://arxiv.org/abs/1409.1556
[126] S. Han, H. Mao, and W. J. Dally, ‘Deep
Compression: Compressing Deep Neural
Networks with Pruning, Trained Quantization
and Huffman Coding’, ArXiv151000149 Cs,
Feb. 2016, Accessed: Sep. 18, 2021. [Online].
Available: http://arxiv.org/abs/1510.00149
[127] ‘Post-training quantization | TensorFlow Lite’.
https://www.tensorflow.org/lite/performance/pos
t_training_quantization (accessed Sep. 28,
2021).
[128] ‘Quantization — PyTorch 1.9.1 documentation’.
https://pytorch.org/docs/stable/quantization.html
(accessed Sep. 28, 2021).
[129] ‘Larq | Binarized Neural Network
development’. https://larq.dev/ (accessed Sep.
28, 2021).
[130] A. G. Anderson and C. P. Berg, ‘The
High-Dimensional Geometry of Binary Neural
Networks’, ArXiv170507199 Cs, May 2017,
Accessed: Sep. 18, 2021. [Online]. Available:
http://arxiv.org/abs/1705.07199
[131] D. Molchanov, A. Ashukha, and D. Vetrov,
‘Variational Dropout Sparsifies Deep Neural
Networks’, ArXiv170105369 Cs Stat, Jun. 2017,
Accessed: Sep. 18, 2021. [Online]. Available:
http://arxiv.org/abs/1701.05369
[132] A. Polino, R. Pascanu, and D. Alistarh, ‘Model
compression via distillation and quantization’,
ArXiv180205668 Cs, Feb. 2018, Accessed: Sep.
18, 2021. [Online]. Available:
http://arxiv.org/abs/1802.05668
[133] Y. Gong, L. Liu, M. Yang, and L. Bourdev,
‘Compressing Deep Convolutional Networks
using Vector Quantization’, ArXiv14126115 Cs,
Dec. 2014, Accessed: Sep. 18, 2021. [Online].
Available: http://arxiv.org/abs/1412.6115
[134] A. Mishra, J. J. Cook, E. Nurvitadhi, and D.
Marr, ‘WRPN: Training and Inference using
Wide Reduced-Precision Networks’,
ArXiv170403079 Cs, Apr. 2017, Accessed: Sep.
18, 2021. [Online]. Available:
http://arxiv.org/abs/1704.03079
[135] ‘ISO - ISO/IEC JTC 1/SC 42 - Artificial
intelligence’.
https://www.iso.org/committee/6794475/x/catal
ogue/p/1/u/1/w/0/d/0 (accessed Sep. 04, 2021).
[136] J. Törnblom and S. Nadjm-Tehrani, ‘Formal
Verification of Random Forests in
Safety-Critical Applications’, in Formal
Techniques for Safety-Critical Systems, Cham,
2019, pp. 55–71. doi:
10.1007/978-3-030-12988-0_4.
[137] W. Wahlster and Cristoph Winterhalter,
‘GERMAN STANDARDIZATION ROADMAP
ON ARTIFICIAL INTELLIGENCE’, p. 226,
IAC-21-B.1.4.4 Page 29 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
Nov. 2020.
[138] ‘White Paper on Artificial Intelligence: a
European approach to excellence and trust’,
European Commission - European Commission.
https://ec.europa.eu/info/publications/white-pap
er-artificial-intelligence-european-approach-exc
ellence-and-trust_en (accessed Sep. 11, 2021).
[139] Faisal D’Souza, ‘The National Artificial
Intelligence Research and Development
Strategic Plan: 2019 Update’, p. 50, 2019.
[140] Gregory Flandin, ‘White Paper Machine
Learning in Certified System’, p. 113, Feb.
2021.
[141] C. Molnar, Interpretable Machine Learning.
2021. Accessed: Sep. 28, 2021. [Online].
Available:
https://christophm.github.io/interpretable-ml-bo
ok/
[142] P. Linardatos, V. Papastefanopoulos, and S.
Kotsiantis, ‘Explainable AI: A Review of
Machine Learning Interpretability Methods’,
Entropy, vol. 23, no. 1, Art. no. 1, Jan. 2021,
doi: 10.3390/e23010018.
[143] E. Tjoa and C. Guan, ‘A Survey on Explainable
Artificial Intelligence (XAI): Towards Medical
XAI’, IEEE Trans. Neural Netw. Learn. Syst.,
pp. 1–21, 2020, doi:
10.1109/TNNLS.2020.3027314.
[144] ‘AC 25.1309-1A - System Design and Analysis
– Document Information’, Jun. 21, 1988.
https://www.faa.gov/regulations_policies/adviso
ry_circulars/index.cfm/go/document.informatio
n/documentid/22680 (accessed Sep. 11, 2021).
[145] C.-H. Cheng and R. Yan, ‘Testing Autonomous
Systems with Believed Equivalence
Refinement’, ArXiv210304578 Cs, Aug. 2021,
Accessed: Sep. 18, 2021. [Online]. Available:
http://arxiv.org/abs/2103.04578
[146] G. Katz, C. Barrett, D. Dill, K. Julian, and M.
Kochenderfer, ‘Reluplex: An Efficient SMT
Solver for Verifying Deep Neural Networks’,
ArXiv170201135 Cs, May 2017, Accessed: Sep.
04, 2021. [Online]. Available:
http://arxiv.org/abs/1702.01135
[147] S. Wang, K. Pei, J. Whitehouse, J. Yang, and S.
Jana, ‘Formal Security Analysis of Neural
Networks using Symbolic Intervals’,
ArXiv180410829 Cs, Jul. 2018, Accessed: Sep.
04, 2021. [Online]. Available:
http://arxiv.org/abs/1804.10829
[148] M. Mirman, T. Gehr, and M. Vechev,
‘Differentiable Abstract Interpretation for
Provably Robust Neural Networks’, in
International Conference on Machine Learning,
Jul. 2018, pp. 3578–3586. Accessed: Sep. 04,
2021. [Online]. Available:
https://proceedings.mlr.press/v80/mirman18b.ht
ml
[149] M. N. Müller, G. Makarchuk, G. Singh, M.
Püschel, and M. Vechev, ‘PRIMA: Precise and
General Neural Network Certiﬁcation via
Multi-Neuron Convex Relaxations’, p. 20, 2021.
[150] I. J. Goodfellow, J. Shlens, and C. Szegedy,
‘Explaining and Harnessing Adversarial
Examples’, ArXiv14126572 Cs Stat, Mar. 2015,
Accessed: Sep. 04, 2021. [Online]. Available:
http://arxiv.org/abs/1412.6572
[151] H. Chen, H. Zhang, D. Boning, and C.-J. Hsieh,
‘Robust Decision Trees Against Adversarial
Examples’, ArXiv190210660 Cs Stat, Jun. 2019,
Accessed: Sep. 04, 2021. [Online]. Available:
http://arxiv.org/abs/1902.10660
[152] S. Shalev-Shwartz, Understanding Machine
Learning: From Theory to Algorithms, 1st
edition. New York, NY, USA: Cambridge
University Press, 2014.
[153] Z. Yang, L. Li, X. Xu, B. Kailkhura, T. Xie, and
B. Li, ‘On the Certified Robustness for
Ensemble Models and Beyond’,
ArXiv210710873 Cs, Jul. 2021, Accessed: Sep.
29, 2021. [Online]. Available:
http://arxiv.org/abs/2107.10873
[154] T. Pang, K. Xu, C. Du, N. Chen, and J. Zhu,
‘Improving Adversarial Robustness via
Promoting Ensemble Diversity’, in Proceedings
of the 36th International Conference on
Machine Learning, May 2019, pp. 4970–4979.
Accessed: Sep. 29, 2021. [Online]. Available:
https://proceedings.mlr.press/v97/pang19a.html
[155] K. Shweta, ‘A Survey on Classification of
Concept Drift with Stream Data’, Mar. 2019.
Accessed: Sep. 11, 2021. [Online]. Available:
https://hal.archives-ouvertes.fr/hal-02062610
[156] C. V. Nguyen, A. Achille, M. Lam, T. Hassner,
V. Mahadevan, and S. Soatto, ‘Toward
Understanding Catastrophic Forgetting in
Continual Learning’, ArXiv190801091 Cs Stat,
Aug. 2019, Accessed: Sep. 26, 2021. [Online].
Available: http://arxiv.org/abs/1908.01091
[157] S. Zhao X. Yue, S. Zhang, B. Li, H. Zhao, B.
Wu, R. Krishna, J. E. Gonzalez, A. L.
Sangiovanni-Vincentelli, Sanjit A. Seshia, and
K. Keutzer, ‘A Review of Single-Source Deep
Unsupervised Visual Domain Adaptation’,
ArXiv200900155 Cs Eess, Sep. 2020, Accessed:
Sep. 04, 2021. [Online]. Available:
http://arxiv.org/abs/2009.00155
[158] C. Cappi, C. Chapdelaine, L. Gardes, E. Jenn,
B. Lefevre, S. Picard, and T. Soumarmon,
‘Dataset Definition Standard (DDS)’,
ArXiv210103020 Cs, Jan. 2021, Accessed: Sep.
19, 2021. [Online]. Available:
IAC-21-B.1.4.4 Page 30 of 31

72nd International Astronautical Congress (IAC), Dubai, United Arab Emirates, 25-29 October 2021.
Copyright © [IAC 2021] by Mr. Pablo Miralles. Published by the IAF, with permission and released to the IAF to publish in all forms.
http://arxiv.org/abs/2101.03020
[159] L. R. Davis, K. A. Fishburn, H. Lestinsky, L. R.
Moore, and J. L. Walter, ‘US Topo Product
Standard’, Techniques and Methods, 2019.
[Online]. Available:
https://pubs.usgs.gov/tm/tm11b2/
[160] L. Di, ‘Standards, Critical Evaluation of Remote
Sensing’, in Encyclopedia of GIS, S. Shekhar
and H. Xiong, Eds. Boston, MA: Springer US,
2008, pp. 1128–1135. doi:
10.1007/978-0-387-35973-1_1346.
[161] L. Di and B. Kobler, ‘NASA Standards for
Earth Remote Sensing Data’, Jan. 2000.
https://www.researchgate.net/publication/22895
3572_NASA_Standards_for_Earth_Remote_Se
nsing_Data (accessed Sep. 04, 2021).
[162] ‘Home’, Mindseed. https://www.mindseed.ie/
(accessed Sep. 18, 2021).
[163] ‘SatSure’. https://satsure.co/ (accessed Sep. 18,
2021).
[164] ‘cnes | Le site du Centre national d’études
spatiales’. https://cnes.fr/fr/ (accessed Sep. 18,
2021).
IAC-21-B.1.4.4 Page 31 of 31

CloudSatNet-1: FPGA-based Hardware-Accelerated Quantized CNN for Satellite On-Board Cloud Coverage Classification

Preprint

Full-text available

Mar 2022

div>CubeSats, the nanosatellites with a wet mass up to 10 kg, accompanied by the cost decrease of accessing the space, amplified the rapid development of the Earth Observation industry. Acquired image data serve as an essential source of information in various disciplines like environmental protection, geosciences, or the military. As the quantity of remote sensing data grows, the bandwidth resources for the data transmission (downlink) are exhausted. Therefore, new techniques that reduce the downlink utilization of the satellites must be investigated and developed. For that reason, we are presenting CloudSatNet-1: an FPGA based hardware-accelerated quantized convolutional neural network (CNN) for satellite on-board cloud coverage classification. We aim to explore the effects of the quantization process on the proposed CNN architecture. Additionally, the performance of cloud coverage classification by biomes diversity is investigated, and the hardware architecture design space is explored to identify the optimal FPGA resource utilization. Results of this study showed that the weights and activations quantization adds a minor effect on the model performance. Nevertheless, the memory footprint reduction allows the model deployment on low-cost FPGA Xilinx Zynq-7020. Using the RGB bands only, up to 90% of accuracy was achieved, and when omitting the tiles with snow and ice, the performance increased up to 94.4% of accuracy with a low false-positive rate of 2.23% for the 4-bit width model. With the maximum parallelization settings, the hardware accelerator achieved 15 FPS with 2.5W of average power consumption (0.2W increase over the idle state).</div

Brainsat: Hardware Development of a Neuromorphic On-Board Computer Applied to Methane Detection from Low Earth Orbit.

Conference Paper

Full-text available

Oct 2024

Machine learning applications for space require fast and intensive computation. The challenge however is that embedded computing is usually constrained in size, weight and power (SWaP) as well as thermal dissipation footprints. While compute efficiency gains have traditionally been made through smaller manufacturing nodes, this approach is reaching its limits, and new architectures are needed to meet payload computation needs in a Low-SWaP manner. Neuromorphic Computing (NC) mimics traits of biological brains, such as sparseness, spiking networks, and in-memory processing, thus offering better performance compared to traditional computing architectures, with drastically reduced footprints. The paper proposes an on-board computer design leveraging NC, with advanced processing capabilities within the size, mass and power constraints of a low earth orbit small satellite mission. We introduce and review current Neuromorphic computing approaches and define a set of requirements dictated by said space mission and its spacecraft design. We then perform a preliminary design of the hardware and software of the Neuromorphic-enabled computer for this mission, comparing its performance to that of a conventional digital processor on key metrics (size, weight and power consumption). We finally discuss aspects of production and testing (VV) for the novel design, closing out with recommendations charting a potential path towards implementation. The paper also details the challenges faced by Neuromorphic processing within the Space environment, and proposes a product-driven Systems Engineering Approach, paving the way for widespread future use of Neuromorphic hardware in Space. The paper aims to act as a reference point for other system integrators wishing to implement such hardware into future missions.

Deep Learning-based System for Change Detection On-Board Earth Observation Small Satellites

Article

Full-text available

Jan 2023

In recent years, the important evolution in the number, potentiality, and diversity of Earth Observation (EO) satellites, has resulted in dramatic increases in the payload data volume and rate. However, these exponential increases in the generated data volume are creating a significant bottleneck onboard EO satellites due to transmission bandwidth limits and communications delays. Onboard imaging payload data processing can provide an appropriate solution to alleviate the induced data bottleneck. It can also facilitate rapid response for decision-making operations. Change detection is one of the most significant functions in onboard payload data processing systems that enables a real-time reaction to natural disasters such as flooding, earthquakes, and volcanic eruptions. In this paper, we address the problem of automatic change detection onboard EO satellites. This research work aims to design an automatic onboard change detection system (OCDS) that can run on existing flight-proven hardware by taking advantage of the attractive features of a leading model in deep learning called Convolutional Neural Network. The contribution of this work is twofold. An efficient algorithmic solution for change detection based on Deep Learning that fulfills space environment-induced constraints is first proposed. Second, a preliminary hardware architecture of the proposed OCDS is designed based on payload data processing flight-proven hardware. The experimental results demonstrate the efficiency of the proposed deep learning-based change detection approach and the suitability of the designed OCDS for onboarding on EO small satellites.

A New Strategy of Satellite Autonomy with Machine Learning for Efficient Resource Utilization of a Standard Performance CubeSat

Article

Full-text available

Jan 2023

A mega constellation of Nano/microsatellites is the contemporary solution for global-level Earth observation demands. However, as most of the images taken by Earth-observing satellites are covered by clouds, storing and downlinking these images results in inefficient utilization of scarce onboard resources and bandwidth. In addition, the trend of making satellite task execution plans by ground operators demands the efforts of experts or simulators to predict the real-time situation of satellites and to decide which tasks should be executed next. Granting controlled autonomy to satellites to perform onboard tasks will boost mission effectiveness. We experimented with granting controlled autonomy for satellites in performing onboard image classification and task scheduling. We designed a convolutional neural network-based binary image classification model with more than 99% accuracy in classifying clear and cloudy images. The model is configured to perform inference in low-performance computers of ordinary Cubesats. Moreover, we designed an autonomous satellite task scheduling mechanism based on reinforcement learning. It performs better than a custom heuristic-based method in scheduling onboard tasks. As a result, the proposed classification and scheduling techniques with machine learning ensured efficient utilization of onboard memory, power, and bandwidth in the highly resource-constrained CubeSat platforms and mission accomplishment of Nano/microsatellite constellations.

CloudSatNet-1: FPGA-Based Hardware-Accelerated Quantized CNN for Satellite On-Board Cloud Coverage Classification

Article

Full-text available

Jul 2022

CubeSats, the nanosatellites and microsatellites with a wet mass up to 60 kg, accompanied by the cost decrease of accessing the space, amplified the rapid development of the Earth Observation industry. Acquired image data serve as an essential source of information in various disciplines like environmental protection, geosciences, or the military. As the quantity of remote sensing data grows, the bandwidth resources for the data transmission (downlink) are exhausted. Therefore, new techniques that reduce the downlink utilization of the satellites must be investigated and developed. For that reason, we are presenting CloudSatNet-1: an FPGA-based hardware-accelerated quantized convolutional neural network (CNN) for satellite on-board cloud coverage classification. We aim to explore the effects of the quantization process on the proposed CNN architecture. Additionally, the performance of cloud coverage classification by biomes diversity is investigated, and the hardware architecture design space is explored to identify the optimal FPGA resource utilization. Results of this study showed that the weights and activations quantization adds a minor effect on the model performance. Nevertheless, the memory footprint reduction allows the model deployment on low-cost FPGA Xilinx Zynq-7020. Using the RGB bands only, up to 90% of accuracy was achieved, and when omitting the tiles with snow and ice, the performance increased up to 94.4% of accuracy with a low false-positive rate of 2.23% for the 4-bit width model. With the maximum parallelization settings, the hardware accelerator achieved 15 FPS with 2.5 W of average power consumption (0.2 W increase over the idle state).

Machine Learning in Space: Surveying the Robustness of On-Board ML Models to Radiation

Conference Paper

Jul 2024

Artificial intelligence for satellite communication: A review

Article

Full-text available

Sep 2021

Satellite communication offers the prospect of service continuity over uncovered and under-covered areas, service ubiquity, and service scalability. However, several challenges must first be addressed to realize these benefits, as the resource management, network control, network security, spectrum management, and energy usage of satellite networks are more challenging than that of terrestrial networks. Meanwhile, artificial intelligence (AI), including machine learning, deep learning, and reinforcement learning, has been steadily growing as a research field and has shown successful results in diverse applications, including wireless communication. In particular, the application of AI to a wide variety of satellite communication aspects has demonstrated excellent potential, including beam-hopping, anti-jamming, network traffic forecasting, channel modeling, telemetry mining, ionospheric scintillation detecting, interference managing, remote sensing, behavior modeling, space-air-ground integrating, and energy managing. This work thus provides a general overview of AI, its diverse sub-fields, and its state-of-the-art algorithms. Several challenges facing diverse aspects of satellite communication systems are then discussed, and their proposed and potential AI-based solutions are presented. Finally, an outlook of field is drawn, and future steps are suggested.

Mapping sugarcane in Thailand using transfer learning, a lightweight convolutional Neural Network, NICFI high resolution satellite imagery and Google Earth Engine

Article

Full-text available

Jul 2021

Air pollution from burning sugarcane is an important environmental issue in Thailand. Knowing the location and extent of sugarcane plantations would help in formulating effective strategies to reduce burning. High resolution satellite imagery combined with deep-learning technologies can be effective to map sugarcane with high precision. However, land cover mapping using high resolution data and computationally intensive deep-learning networks can be computationally costly. In this study, we used high resolution satellite imagery from Planet that has been made available to the public through the Norway’s International Climate and Forest Initiative (NICFI). We tested a U-Net deep-learning algorithm with a lightweight MobileNetV2 network as the encoder branch using the Google Earth Engine computational platform. We trained a model using the RGB channels with pre-trained network (RGBt), a RGB model with randomly initialized weights (RGBr) and a model with randomly initialized weights including the NIR channel (RGBN). We found an F1-score of 0.9550, 0.9262 and 0.9297 for the RGBt, RGBr and RGBN models, respectively. For an independent model evaluation we found F1-scores of 0.9141, 0.8681 and 0.8911. We also found a discrepancy in the recall values reported by the model and those from the independent validation. We found that lightweight deep-learning models produce satisfactory results while providing effective means to apply mapping efforts at scale with reduced computational costs. We highlight the importance of central data repositories with labeled data as pre-trained networks were found to be effective in improving the accuracy.

FGATR-Net: Automatic Network Architecture Design for Fine-Grained Aircraft Type Recognition in Remote Sensing Images

Article

Full-text available

Dec 2020

Fine-grained aircraft type recognition in remote sensing images, aiming to distinguish different types of the same parent category aircraft, is quite a significant task. In recent decades, with the development of deep learning, the solution scheme for this problem has shifted from handcrafted feature design to model architecture design. Although a great progress has been achieved, this paradigm generally needs strong expert knowledge and rich expert experience. It is still an extremely laborious work and the automation level is relatively low. In this paper, inspired by Neural Architecture Search (NAS), we explore a novel differentiable automatic architecture design framework for fine-grained aircraft type recognition in remote sensing images. In our framework, the search process is divided into several phases. Network architecture deepens at each phase while the number of candidate functions gradually decreases. To achieve it, we adopt different pruning strategies. Then, the network architecture is determined through a potentiality judgment after an architecture heating process. This approach can not only search deeper network, but also reduce the computational complexity, especially for relatively large size of remote sensing images. When all differentiable search phases are finished, the searched model called Fine-Grained Aircraft Type Recognition Net (FGATR-Net) is obtained. Compared with previous NAS, ours are more suitable for relatively large and complex remote sensing images. Experiments on Multitype Aircraft Remote Sensing Images (MTARSI) and Aircraft17 validate that FGATR-Net possesses a strong capability of feature extraction and feature representation. Besides, it is also compact enough, i.e., parameter quantity is relatively small. This powerfully indicates the feasibility and effectiveness of the proposed automatic network architecture design method.

Testing Autonomous Systems with Believed Equivalence Refinement

Conference Paper

Aug 2021

Robust decision trees against adversarial examples

Article

Dec 2020

2019 by the Author(S). Although adversarial examples and model robustness have been extensively studied in the context of linear models and neural networks, research on this issue in tree-based models and how to make tree-bascd models robust against adversarial examples is still limited. In this paper, we show that tree based models are also vulnerable to adversarial examples and develop a novel algorithm to learn robust trees. At its core, our method aims to optimize the performance under the worst-case perturbation of input features, which leads to a max-min saddle point problem. Incorporating this saddle point objective into the decision tree building procedure is non-trivial due to the discrete nature of trees - a naive approach to finding the best split according to this saddle point objective will take exponential time. To make our approach practical and scalable, we propose efficient tree building algorithms by approximating the inner minimizer in this saddle point problem, and present efficient implementations for classical information gain based trees as well as state-of-the-art tree boosting models such as XG-Boost. Experimental results on real world datasets demonstrate that the proposed algorithms can substantially improve the robustness of tree-based models against adversarial examples.

On Self-Distilling Graph Neural Network

Conference Paper

Aug 2021

Recently, the teacher-student knowledge distillation framework has demonstrated its potential in training Graph Neural Networks (GNNs). However, due to the difficulty of training over-parameterized GNN models, one may not easily obtain a satisfactory teacher model for distillation. Furthermore, the inefficient training process of teacher-student knowledge distillation also impedes its applications in GNN models. In this paper, we propose the first teacher-free knowledge distillation method for GNNs, termed GNN Self-Distillation (GNN-SD), that serves as a drop-in replacement of the standard training process. The method is built upon the proposed neighborhood discrepancy rate (NDR), which quantifies the non-smoothness of the embedded graph in an efficient way. Based on this metric, we propose the adaptive discrepancy retaining (ADR) regularizer to empower the transferability of knowledge that maintains high neighborhood discrepancy across GNN layers. We also summarize a generic GNN-SD framework that could be exploited to induce other distillation strategies. Experiments further prove the effectiveness and generalization of our approach, as it brings: 1) state-of-the-art GNN distillation performance with less training cost, 2) consistent and considerable performance enhancement for various popular backbones.

Deep Learning Techniques for Autonomous Spacecraft Guidance During Proximity Operations

Article

Aug 2021

This paper investigates the use of deep learning techniques for real-time optimal spacecraft guidance during terminal rendezvous maneuvers, in presence of both operational constraints and stochastic effects, such as an inaccurate knowledge of the initial spacecraft state and the presence of random in-flight disturbances. The performance of two well-studied deep learning methods, behavioral cloning (BC) and reinforcement learning (RL), is investigated on a linear multi-impulsive rendezvous mission. To this aim, a multilayer perceptron network, with custom architecture, is designed to map any observation of the actual spacecraft relative position and velocity to the propellant-optimal control action, which corresponds to a bounded-magnitude impulsive velocity variation. In the BC approach, the deep neural network is trained by supervised learning on a set of optimal trajectories, generated by routinely solving the deterministic optimal control problem via convex optimization, starting from scattered initial conditions. Conversely, in the RL approach, a state-of-the-art actor–critic algorithm, proximal policy optimization, is used for training the network through repeated interactions with the stochastic environment. Eventually, the robustness and propellant efficiency of the obtained closed-loop control policies are assessed and compared by means of a Monte Carlo analysis, carried out by considering different test cases with increasing levels of perturbations.

Real-Time Guidance for Low-Thrust Transfers Using Deep Neural Networks

Article

Jan 2021
J GUID CONTROL DYNAM

Deep Reinforcement Learning for Spacecraft Proximity Operations Guidance

Article

Jan 2021

This paper introduces a guidance strategy for spacecraft proximity operations, which leverages deep reinforcement learning, a branch of artificial intelligence. This technique enables guidance strategies to be learned rather than designed. The learned guidance strategy feeds velocity commands to a conventional controller to track. Control theory is used alongside deep reinforcement learning to lower the learning burden and facilitate the transfer of the learned behavior from simulation to reality. In this paper, a proof-of-concept spacecraft pose tracking and docking scenario is considered, in simulation and experiment, to test the feasibility of the proposed approach. Results show that such a system can be trained entirely in simulation and transferred to reality with comparable performance.

AutoML: A survey of the state-of-the-art

Article

Jan 2021
KNOWL-BASED SYST

Deep learning (DL) techniques have obtained remarkable achievements on various tasks, such as image recognition, object detection, and language modeling. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering its wide application. Meanwhile, automated machine learning (AutoML) is a promising solution for building a DL system without human assistance and is being extensively studied. This paper presents a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. According to the DL pipeline, we introduce AutoML methods-covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS)-with a particular focus on NAS, as it is currently a hot sub-topic of AutoML. We summarize the representative NAS algorithms' performance on the CIFAR-10 and ImageNet datasets and further discuss the following subjects of NAS methods: one/two-stage NAS, one-shot NAS, joint hyperparameter and architecture optimization, and resource-aware NAS. Finally, we discuss some open problems related to the existing AutoML methods for future research.

Machine Learning in Earth Observation Operations: A review

Abstract and Figures

Recommended publications

A critical review on the state-of-the-art and future prospects of Machine Learning for Earth Observa...

Conceptual Design of a Mars Constellation for Global Communication Services using Small Satellites

DikpolaSat Mission: Improvement of Space Flight Performance and Optimal Control Using Trained Deep N...

DikpolaSat Mission: Improvement of Space Flight Performance and Optimal Control Using Trained Deep N...