Content uploaded by Zdravko Paladin
Author content
All content in this area was uploaded by Zdravko Paladin on Sep 26, 2022
Content may be subject to copyright.
COMBINED AI CAPABILITIES FOR ENHANCING
MARITIME SAFETY IN A COMMON
INFORMATION SHARING ENVIRONMENT
ZDRAVKO PALADIN,1 NEXHAT KAPIDANI,1 ŽARKO
L
UKŠIĆ
,
1
A
NDREJ
M
IHAILOVIĆ
,
1,2
P
IERO
S
CRIMA
,
3
C
HARLOTTE
J
ACOB
É
DE
N
AUROIS
,
4
C
LAIRE
L
AUDY
,
4
CONSTANTINOS RIZOGIANNIS,5 ALKIVIADIS
A
STYAKOPOULOS
5
& A
LEXIS
B
LUM
6
1
Administration for Maritime Safety and Port Management of Montenegro.
E-mail: zdravko.paladin@pomorstvo.me, nexhat.kapidani@pomorstvo.me,
zarko.luksic@pomorstvo.me
2
King College London, United Kingdom.
E-mail: andrej.mihailovic@kcl.ac.uk
3
Engineering I.I., S.p.A., ENG, Milano, Italy.
E-mail: piero.scrima@eng.it
4
Thales Group, Palaiseau, France.
E-mail: charlotte.jacobedenaurois@thalesgroup.com, claire.laudy@thalesgroup.com
5
Center for Security Studies, Ministry of Citizen Protection, Athens, Greece.
E-mail:
a.astyakopoulos@kemea-research.gr,
c.rizogiannis@kemea-research.gr
6 Secrétariat général de la mer - SGMER, Paris, France.
E-mail: alexis.blum@pm.gouv.fr
Abstract
The complexity of maritime traffic operations indicates
an unprecedented necessity for joint introduction and
exploitation of artificial intelligence (AI) technologies, that take
advantage of the vast amount of vessels’ data, offered by
disparate surveillance systems to face challenges at sea. This
paper reviews the recent Big Data and AI technology
implementations for enhancing the maritime safety level in the
common information sharing environment (CISE) of the
maritime agencies, including vessel behavior and anomaly
monitoring, and ship collision risk assessment. Specifically, the
trajectory fusion implemented with InSyTo module for soft
information fusion and management toolbox, and the Early
Notification module for Vessel Collision are presented within
EFFECTOR Project. The focus is to elaborate technical
architecture features of these modules and combined AI
capabilities for achieving the desired interoperability and
complementarity between maritime systems, aiming to provide
better decision support and proper information to be distributed
among CISE maritime safety stakeholders.
DOI
h
tt
p
s
:
//
do
i
.
or
g
/
10.
18690/
um
.
f
o
v.4.
2022
.9
ISBN 978-961-286-616-7
Keywords:
CISE,
maritime
safety
big data,
AI.
35TH BLED ECONFERENCE
146
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
1
Introduction
Nowadays, maritime safety agencies are faced with many challenges varying from
the high intensity of maritime traffic, vessel collisions in coastal areas, environmental
risks from ships accidents, irregular maritime border-crossing and illicit activities at
sea. Maintaining the required strategic and tactical level of maritime safety in a
complex environment calls for support of sophisticated and smart ICT technologies,
ready to assist in performing the operations of vessel traffic services and national
r
esc
u
e
c
oord
i
n
a
t
i
on
ce
n
t
e
r
s
(
VT
S
/
N
RCC
).
Th
e
eve
r
-
i
nc
r
e
a
s
i
ng
l
a
r
g
e
a
mou
nt
o
f
vessel data, collected through heterogeneous sensors and information sources
demands appropriate structuring for exchanging them among collaborative agencies
f
or
u
nd
e
r
tak
i
ng
j
oi
nt
ope
r
a
ti
ons
a
nd
s
a
fe
ty
/
sec
u
r
i
ty
m
i
ss
i
o
ns
a
t
se
a
a
nd bord
e
r
.
Therefore, in this paper we analyse the most important objectives that maritime
safety sector strives to:
1.
achieving greater maritime situational awareness through institutional
networking among relevant agencies for Common Operational Picture at
sea,
2.
full exploitation of the latest innovative achievements, automated ICT
technologies and big data science, capitalizing on versatile applications of
AI for maritime purposes, such as anomalies detection and navigation
predictions.
The goal of the paper is to present a case study EFFECTOR about maritime safety
and two specific solutions conmbined AI features and how these need to be adapted
for maritime context. Methological approach of this research reviews CISE as
maritime safety EU initiative, the Big Data collected from various maritime sensors
and shared among CISE network, with combined AI capabilities for the purpose of
efficient response of maritime operative systems. Consequently, the paper unfolds
as follows: Chapter 2 elaborates CISE in more details, while Chapter 3 reviews Big
Data impacts on development of AI technologies in maritime environments. In
Chapter 4 the case study presents EU project EFFECTOR with its specific solutions
base
d on
c
ombine
d
A
I
fe
a
t
u
r
es
f
or d
a
ta
/
i
nfor
mati
on
f
u
s
i
on
a
nd
vesse
l
col
li
s
i
on
prevention.
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
147
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
2
CISE EU initiative in maritime safety
Considering that maritime safety critically relies on vessel surveillance systems and
fast information flows networked via maritime authorities’ national competent
systems, the need for regional and international cooperation of European
stakeholders has led to the establishment of the concept of Common Information
Sharing Environment (CISE). The idea of establishing the CISE concept stems from
the EUCISE2020, a test-bed project that triggers a creation of a common network
for sharing and exchanging relevant maritime data and information between
collaborating authorities. This concept was developed and extended through further
innovation action projects supported by European Commission (EC) and aimed to
improve the current performance in information sharing. That is why CISE was used
in EFFECTOR project. Following the latest level of development concept, in Figure
1
we
d
e
p
i
c
t
g
e
n
e
r
a
l
C
I
S
E
Arc
h
i
t
ec
t
u
r
e
a
li
g
ne
d
w
i
th
the
mos
t
c
ommon d
a
ta
/
mess
a
g
e
flows, actors, and related software/AI components as decision support tools and
services. Based on documentation EC COM (2009) 538 and European Maritime
Safety Agency (EMSA) Guidelines for CISE [EMSA CISE Architecture document,
2012
]
,
w
hol
e
i
nfor
mati
on
sha
r
i
ng
/
r
e
t
r
i
ev
a
l
/
i
nterpr
etati
on pr
oc
ess
i
s
mana
g
e
d
v
i
a
CISE Data & Services Model, compliant with NATO Architectural Framework
NAFv3. It is structured in five main object blocks (Paladin et al., 2021): Legacy
S
y
s
tem
(L
S)
of
pa
r
tic
i
pati
n
g
a
g
e
nc
y
,
E
U
/
Re
g
i
ona
l
/
Na
ti
ona
l
C
I
S
E
Nod
e
,
C
I
S
E
A
d
a
ptor
,
C
I
S
E
Nod
e
/
Ga
t
ew
a
y
,
a
nd
C
I
S
E
Ne
tw
or
k
.
I
n d
e
t
a
il,
L
S
i
s
a
n
I
C
T
s
y
s
t
e
m
/
n
e
t
w
or
k
o
f
a
p
a
r
t
i
c
u
l
a
r
a
u
t
hor
i
t
y
,
i
n
t
e
g
r
a
t
e
d
w
i
t
h
s
u
r
ve
ill
a
n
ce
se
n
s
or
s
,
w
h
i
c
h
collects, integrates, stores and visualizes maritime Big Data received by their own
assets (radars, AIS systems, METOC data, NMSW, UxV) or received by EU Centers
(
L
RIT
,
A
I
S
/
MAR
E
Σ
,
I
M
S),
w
h
i
c
h
a
r
e
a
b
l
e
t
o
i
n
t
e
rop
e
r
a
t
e
w
i
t
h o
t
h
e
r
a
g
e
n
c
i
es
.
E
U
/
Re
g
i
ona
l
/
Na
ti
o
na
l
C
I
S
E
Nod
e
pr
ov
i
d
es
t
he
i
nte
g
r
a
ti
on of
one
or mor
e
national maritime authorities proxied via combined instances of the CISE adaptors
for each LS. Most usually, these LS-specific CISE adaptors for data stream sharing
are connected to the Command and Control (C2) platform accompanied with Data
F
u
s
i
on
a
nd
An
a
ly
t
i
c
S
e
r
v
i
ces
L
a
y
e
r
&
D
ec
i
s
i
on
S
u
ppor
t
S
e
r
v
i
ces
L
a
y
e
r
/
Too
l
s
.
Th
is
structure is mostly supported with Big Data infrastructure and specific AI
c
ompone
nt
s
,
lik
e
M
a
c
h
i
n
e
/
D
ee
p
Le
a
rn
i
n
g
L
i
br
a
r
i
es
,
t
r
a
j
ec
t
or
y
pr
e
d
i
c
t
i
on
a
nd
vesse
l
collision risk mitigation. Such combined AI capabilities in high-level operational C2
software provide an intelligent support for decision making based on comprehensive
maritime Common Operational Picture. Finally, via CISE Adaptor for data
tr
a
ns
l
a
ti
on
a
nd
C
I
S
E
N
od
e
/
G
a
t
ew
a
y
(
a
c
o
m
pon
e
n
t
g
i
v
i
n
g
t
h
e
a
ccess
t
o
t
h
e
E
U
/
Re
g
i
ona
l
Nod
e
c
o
ns
ol
i
d
a
te
d
i
nfor
mati
on
i
n
a
ce
ntr
a
l
d
a
tabase
),
t
he
C
I
S
E
Network facilitates the exchange of mentioned information in full compliance with
35TH BLED ECONFERENCE
148
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
the CISE message pattern among CISE Member states and EU agencies.
Accordingly, the structure of the maritime CISE Data & Service Model defines in
its vocabulary CISE Core and Auxiliary Entities concerning agents (person or
organization), objects (vessel, operational asset), event (action, anomaly, incident),
location, period, risks, documents (metadata), using XSD (XML Schema Definition)
or UML (Unified Modelling Language). Being enhanced, the CISE Model introduces
tasks, mission, operations, movement, maritime anomalies and sensors (AIS, radar,
camera). For instance, the maritime risk type identifies crisis, border crossings, areas,
vessels collisions, military and environmental risks (Mihailović et al., 2021a and
2021b).
Figure 1: CISE Architecture and Information Flows supported with AI components
Source: authors' adaptation
3
Big data impacts in the development of AI technologies applied in
maritime safety
In general, AI technologies, with their cognitive, forecasting and reasoning functions
are intensively developed toward providing greater software support to human
operators and agencies, increasing the level of automation in the maritime transport
sector. The aim is to strengthen the maritime safety domain by utilization of
prospective applications able to manage Big Data as: vessel route/paths control and
optimization, vessel traffic surveillance, prevention of collision, possible
fault/failure detection in ship operations, etc. Primarily, maritime AI applications
retrieve the vast amount of data from different data source types, such as: fixed
surveillance radar stations, patrolling and rescue ships, and most significantly, from
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
149
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
electronic tracking system with automatic identification (AIS) for vessels movement
and remote sensing systems. Furthermore, these Big Data, processed under Machine
Learning (ML) or more specific Deep Learning (DL) approaches/techniques with
opti
miz
a
ti
on mod
e
li
ng
ena
b
l
es
the
V
T
S
/
MRCC
ope
r
a
t
or
t
o
i
nc
r
e
a
se
the
c
ont
ro
l
efficiency on tactical level actions and assess the risks/accidents impacts at sea. Such
processed data enable the highly value information for sharing within CISE
Network. While regarding the vessel route data (such as AIS), ML is one of the
research trends for anomalies detection. In the following subchapters are taken in
consideration the works related on both of these research areas.
3.1
Maritime Big Data applications for vessels detection
The AIS is a cooperative information system that provides identification and
position of ships in real-time, but its coverage is limited by the structure of the
system itself. The most effective solution to cover the remote ocean areas are space-
based sensors, such as SAT-AIS (Helleren et al., 2012). The AIS and SAT-AIS are
the most used tracking systems in Maritime Surveillance, which have proven to help
and support the resolution of many problems in this area, but even with global
coverage, the AIS has its downside caused by the monitoring limitation of only
reporting vessels. Thus, AIS should be integrated by other vessel tracking data
sources. One of the shared Big Data sources alterative to AIS is provided by satellites
remote sensing, such as earth observation satellites and Synthetic Aperture Radar
(SAR). These images cover all the globe and contain also ships that do not share AIS
information. But unlike AIS, information in optical images is not explicit, and a
specific process is needed to be done to detect the vessel in the images. The vessel
information extraction from satellite imageries is driven by 3 main processes: object
or vessel recognition (finds a vessel in the image), vessel classification (the class of
the vessel) and vessel identification (Kanjir et al., 2018). The vessel recognition is the
first step to extract vessel information from the images, it can exploit different types
of algorithms, among these there is DL (Wang et al., 2018).
Even if in the past the image processing statistical techniques were more widespread,
today it seems that the use of Neural Networks (NN) is gaining ground (Bentes et
al., 2017), and in many works it is claimed that the latter provides advantages in terms
of performance, and compared to statistics or even computer vision (Kanjir et al.,
2018). In the next step, the classification of the vessels in almost all recent works
converge in the use of AI algorithms. Most of these classifiers seem to use Support
Vector Machines (SVM), and in recent years the trend is also in favor of using NN
here. Instead, other works focus on Bayesian networks and other statistics and AI
35TH BLED ECONFERENCE
150
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
algorithms (Soldi et al., 2021). The information extracted using satellite images can
be more effective in combination to those collaborative systems such as AIS, which
include identification and higher temporal resolution (Achiri et al., 2018). Based on
this information, it is possible to identify those vessels that omit the sending of AIS
data, or that falsify them. In order to develop these operations, there exist different
fusion techniques that have been studied (Fischer et al., 2010). An important point
of these technique is the usage of the interpolation on the AIS data, used to estimate
the AIS position at the moment in which the vessel is extracted from the satellite
image (Nguyen et al., 2015).
3.2
Big data and AI solutions for maritime surveillance
In the previous chapter a series of information extraction techniques have been
described, in this section the state of the art of anomaly detection algorithms are
considered, grouping them by type of algorithms. SVM is one of the simpler
machines learning methods, as it uses a separating hyperplane or a decision plane to
demarcate decision boundaries among a set of data points classified with different
labels. (Handayani et al. 2010) use SVM with Automated Identification System (AIS)
from Port Kelang vessel, tracked for 3 months period and involving 367 tracks
across 7 unique MMSI. By using these data, the paper assesses an accuracy of 90%
of its techniques. Also, in (De Vries et al. 2012) SVMs is applied for detecting the
outlying trajectories. The anomalies detection also takes advantage of Clustering,
which is often used to extract patterns from the route and identify waypoints and
classic routes. These routes, then, are used to describe the behavior of the vessels
and to store this information in a sematic graph, that can be queried to find anomaly
behavior as in (Varlamis el al. 2019).
Also (Dahlbom et al. 2007) explores trajectory clustering as a mean for representing
the normal behavior of vessels. The approach uses spline-based clustering to
overcome some issues in classical clustering. This approach breaks down the map
into small zones where behavior patterns are detected. The most recent work that
applies a similar approach is (Zhen et al. 2017) which executes a trajectory clustering,
and then applies a Naïve Bayes classifier to detect anomalous vessel behavior. (Liu
et al. 2015) separates the normal routes from AIS historical data and then extracts,
using clustering, the normal trajectories and normal behaviors from that one with
which the new data can be compared. The algorithms that have had the greatest
growth and development in recent years are certainly those concerning NN in all
variants, including also those that are defined as DL, which represents specific ML
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
151
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
model with multiple layers of non-linear processing units, referred to systems with
numerous serially connected layers of parallel connected neurons.
In (Nguyen et al. 2018) trajectory reconstruction, the anomaly detection and vessel
type identification are the tasks by which the deep framework proposed in the work
is demonstrated to be applied with effectiveness. The algorithm uses a Recurrent
Neural Network (RNN) with latent variables showing that this algorithm is
particular suited for time series processing. The RNN is also used in (Zhao et al.
2019) which adopts a hybrid approach using also clustering DBSCAN algorithm to
extract the traffic patterns and trains the RNN composed of Long Short-Term
Memory (LSTM) units. The combination of clustering and NN seems to be an
effective solution because the other works applied it, such as (Chen et al. 2019) where
firstly it executes an OPTICS clustering to extract trajectory, and then applies
convolutional NN in order to classify the trajectory. (Nguyen et al. 2021) uses a
probabilistic RNN-based representation of AIS tracks, and then a grid-based
threshold to assess the anomaly of the vessel. The grid threshold allows the
algorithms to adapt the global classified behavior analysis to the local route trend.
Also, the other approaches used within maritime surveillance to detect the vessel
anomaly are: Fuzzy ARTMAP NN, Gaussian Mixture Models (GMM), Bayesian
networks for false ship type, etc (Svenmarck et al. 2018). A very interesting approach
to identify the vessel are the Dynamic Bayesian Networks (DBN) that analyze the
traffic situations at sea and assess kind of relationship between them. Specifically, in
(Anneken et al., 2019) this algorithm for identification of anomaly behaviors of
vessels and reduction of unnecessary amount of data is elaborated according to the
corresponding probabilistic model with graphical representation of Bayesian
reasoning. In this approach, conditional probability is used with the time slices for
random variables, that over time can obtain new attributes by passing from “parent-
initial” to “child” situation. These changes can be abstracted as events with certain
dependency rate, and if one event is realized, the others related will also happen in
particular time interval. Applying this DBN to maritime environment and vessels as
objects, the abstracted situations with random variables correspond to constituent
events of vessel anomaly behavior as e.g. smuggling anomaly with particular
attributes like position, vessel type, course, distance and approaching (Anneken et
al., 2019).
35TH BLED ECONFERENCE
152
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
4
Case study: EU Project EFFECTOR
A significant EU research and innovation project related to maritime surveillance
strengthening is The End-to-end Interoperability Framework for Maritime Situational
Awareness at Strategic and Tactical Operations (EFFECTOR). This project gathers
national maritime safety and security institutions, vessel satellite surveillance and data
exchange software integrators, RTOs and academia with the aim to foster
collaboration among stakeholders, using a common Interoperability Framework for
Maritime Surveillance and Border Security. Some of important methods and tools
used to increase the situational awareness in maritime domain are the following:
multi-layered data lake platforms, data fusion and analytics, knowledge extraction
and semantics, collision notification, maritime ontologies and vessel surveillance AI
mod
u
l
es
ope
r
a
te
d
t
hrou
g
h
i
nte
g
r
a
te
d
C2
s
y
s
tem
s
/
pl
a
tfor
ms
(S
e
a
M
I
S,
E
NGA
GE
,
MUSCA) and in full compliance with CISE and EUROSUR standards. These
innovative technologies are deployed, tested and validated in three operational trials:
France, Portugal and Greece [EFFECTOR Grant Agreement, 2020]. Specifically,
the end-user group, composed of governmental maritime safety and border
authorities, provided relevant maritime data for Data Lakes collected from national
LS for participation in the French, Greek and Portuguese Operational Scenarios and
Trials, with final validation and evaluation of project technical solutions based on
Key Performance Indicators. In this part, a soft information fusion and management
toolbox and deployed in EFFECTOR project and then an Early Collision
Notification System will be described. These AI features are used cinematic of
vessels to take a decision.
4.1
A soft information fusion and management toolbox deployed in
EFFECTOR project
Data and information fusion refer to a set of scientific methods and artificial
intelligence algorithm to create or refine indicators by aggregating data from
heterogeneous sources. More specifically in EFFECTOR project, the main function
of fusion is enhancing situation awareness and reducing the number of information
to be shared between different systems, increasing the global coherence of the
information shared. Furthermore, as opposed to data, information embeds the
context needed to be understood and interpreted. Within EFFECTOR, and for
maritime safety in general, human operators are making decisions relying on the
information they have access to. This is why we claim that the situation awareness
of these operators should be improved thanks to semantic information, as it meaning
is easily accessible to human operators. In this section, we describe the approach
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
153
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
used in EFFECTOR for semantic information fusion. Specifically, in the project
EFFECTOR is deployed a soft information fusion and management toolbox,
InSyTo, providing core generic functions for high level information fusion (Laudy,
2010). It was used on several projects ranging from crisis management (Laudy et al.,
2017) to investigation and oceanography, and we chose to use it in EFFECTOR for
enhancing situational awareness and more particularly to detect meetings between
several ships. The framework uses bipartite graphs and more specifically Basic
Conceptual Graphs (Sowa, 1984; Chein and Mugnier, 2008) to represent
information and knowledge. An ontology is used to adapt the toolbox for specific
application domains. Basic conceptual graph are bipartite graphs containing concept
and relation nodes.
The combination of core functions from InSyTo may provide advanced semantic
information management functions. These core functions are depicted in Figure 2:
Information Synthesis, Query and information fusion. The rectangle boxes represent
concept node of conceptual graph and the circles represent relation of conceptual
graph (between two concepts). The core functions of the toolbox are generic
functions implemented over a generic maximal common subgraph (MCS) search
algorithm. Depending on the way the MCS search algorithm is used, and on which
parameters it is called, as illustrated in the Figure 2, several functions were developed
such as information synthesis, information fusion, sub-graph fusion, information
query, etc. To develop complex functionalities above InSyTo core function, one has
to assemble them, and use them together with fusion strategies.
Fusion Strategies are domain and application specific rules used to provide the
knowledge regarding compatibility of unit elements of the information graphs.
Indeed, the fusion strategies are used to detect and fuse information items that are
slightly different but describing the same situation. During an observation of an on-
going situation, these differences may appear from using different sources of
information with potentially different level of precision or points of view. The main
goal of sub-graph fusion is to detect and fuse compatible parts of two graphs. As
opposed to information Synthesis (top of Fig. 2), however, the result of the sub-
graph fusion is only the common and fused part of the two graphs. One may see
that, as the intersection of the two pieces of information.
35TH BLED ECONFERENCE
154
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
Figure 2: InSyTo core Function
Source: authors' adaptation
The information query function (middle of Fig. 2) can help to find all specified
graph patterns within a Big Data graph. It is based on the search for injective
homomorphism between the query graph and the data graph. The information
fusion
(bottom of Fig. 2) can help to find a specific situation model in observations.
Within all the core functions of InSyTo, we added a traceability capacity (Laudy and
Jacobé de Naurois, 2021). The aim is to keep records of all the fusion operations
that were achieved on each unitary component of an information graph. The lineage
graph records the initial source of each information item, as well as the succession
of fusion operations together with the fusion strategies used. Adding this capability
to the toolbox enabled us to improve the end user understanding and thus trust
toward the overall system. For a specific use case, these different functions can be
combined and specific strategy and similarity functions can be developed.
Also, trajectory fusion and abnormal vessel behavior is identified and managed in
the EFFECTOR project using the InSyTo framework. More specifically, the
suspicious vessel encounters are detected by a high-level fusion function,
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
155
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
implemented to reason on vessel trajectory data generated by AIS systems. The
InSyTo sub-graph fusion function is used to detect common sectors of different
vessel trajectories. Moreover, application and domain specific similarity functions
and fusion strategies are implemented to define what a vessel encounter is.. A vessel
encounter is considered suspicious if it lasts a minimum time duration and if the two
vessels are at less at a defined geographical and temporal distance. A human operator
further configures them to define the fusion conditions based on the specific
application requirements. For the EFFECTOR case study, the InSyTo framework
is connected to a Data Lake which contains vessels trajectories. After queries of
vessel trajectories, the InSyTo framefork searches for encounter between vessels.
Alerts in the CISE format are raised automatically to signal the beginning and the
end of a suspicious vessel encounter, accompanied by the time and location data for
each vessel pair involved. The two anomaly CISE types are “vessel approaching”
and “vessel moving away”. The InSyTo framework in EFFECTOR project is
instantiated for automatically detecting risks and incidents and more specifically
vessel encounter (or collisions). The goal is to enable better detection supports for
operative agencies in maritime safety domain and efficient collaboration based on
CISE network and architecture.
4.2
Early Collision Notification System architecture and deployment
Collisions at sea pose a significant threat with potential serious consequences for
human life, environment and economy and maritime safety in general. To avoid
these effects in an effective manner and reduce the implications of an imminent
collision, much research has been conducted to evaluate the collision risk (CR) of
two approaching vessels. Based on the value of this index early notifications can be
generated to help seafarers execute the International Regulations for Preventing
Collisions at Sea (COLREGS) avoidance maneuvers in time. Researchers have
proposed many CR evaluation methods including numerical [Liu and Liu, 2006] and
fuzzy comprehensive models [Feng and Li, 2012; Xu et al., 2009], ship domain
methods [Xu and Wang, 2014; Szlapczynski and Szlapczynska, 2017], fuzzy
reasoning methods [Kao et al., 2007; Rizogiannis and Thomopoulos, 2019] and
other.
35TH BLED ECONFERENCE
156
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
Rule based filtering
Ships are close to each other and
approaching on a collision course
Parameters calculation
Relative distance, DCPA, TCPA,
Relative bearing, Ship domain,
Domains and Routes intersection
Decision system
CR evaluation
Collision Risk
Yes
Navigational information,
vessel length, vessel type
for a pair of vessels
No
Figure 3: The high-level architecture of the ECNS
Figure 4: Block diagram of the ECNS decision module
In the context of the EFFECTOR project, the Early Collision Notification service
(ECNS) has been developed as part of the EFFECTOR Multi-level data fusion and
analytics services for knowledge extraction and provision of enhanced situational
awareness. ECNS aims at timely generation of notifications of imminent collisions
between ships that could cause death at sea in the area of operation. In this way
ECNS service contributes to an increased level of maritime safety by providing, at
an early stage, alerts and the necessary reaction time to avoid vessels collision. The
high-level architecture and the decision engine of the ECNS service are presented in
Figures 3 and 4 respectively. Compared to existing research, the proposed service
was built aiming to quickly discard pairs of ships that appear no collision risk and
minimize the number of variables used as input to the fuzzy system in order to
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
157
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
accelerate the decision process while at the same time achieve an efficient
performance.
The input to the ECNS module is a rich set of data, containing kinematics
information, (e.g. position, speed, course, turn rate, other) for the two most recently
reported positions of both vessels as well as vessels’ length and type. Using this
input, many new parameters (e.g. Distance to Closest Point of Approach (DCPA),
Time of Closest Point of Approach (TCPA), Relative bearing, other), are calculated,
as well as other useful intelligence (e.g. routes intersection point, determination of
encounter type, ships approaching or surpassing). At the Rule based filtering unit,
the speed, course, routes intersection point, and distance information are used to
determine whether vessels are close to each other and approaching on a collision
course. If both conditions are valid the processing flow moves to the decision system
where CR is evaluated. Otherwise, the ECNS service checks the next pair of vessels.
Finally, in the decision system unit, a type-1 Fuzzy inference system (FIS) uses as
input the set of variables (DCPA, TCPA, Relative distance) to evaluate the desired
CR index where the membership functions (MFs) of both the input and the output
variables are of the general form depicted in Figure 5.
Figure 5: General form of the input and output variables MFs
5
Conclusion
The paper discusses some of the most important recent AI capabilities based on Big
Data sources, and applied in maritime safety and surveillance in order to enhance
the
ov
e
r
a
ll
c
oo
pe
r
a
ti
on
a
nd
pe
r
f
or
ma
nc
e
o
f
in
t
e
r
/
n
a
t
io
n
a
l
a
g
e
n
c
ie
s
inv
o
lv
e
d
i
n
t
he
CISE network. We analyze the key features of AI approaches, that improve the
maritime surveillance using AIS and other data, and that, according to the
a
u
g
me
nted d
a
ta
/
i
nfor
mati
on
f
u
s
i
on pr
oc
esses
a
nd d
ec
i
s
i
on
s
u
ppor
t
tool
s
,
35TH BLED ECONFERENCE
158
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
significantly contribute to the higher interoperability among maritime ICT systems
and regional CISE cooperation of national agencies with purpose to enhance overall
maritime safety. Specifically, the InSyto and ECNS tools deployed in EFFECTOR
project concern the high level of development of AI-based fusion services for
trajectory and movement tracking, necessary to detect vessel anomalous behaviour
and assess the risk of possible vessel collision. Finally, we can conclude by saying
that, maritime safety environment will achieve greater resilience and operational
efficacy only by more intensive exploitation and combination of AI applications with
advanced algorithms for vessel behaviour, risk events identification, assessment and
control at sea and its timely, cost-effective exchange within CISE Network.
Acknowledgement
This work has received funding from the European Union’s Horizon 2020 Research and Innovation
Programme under Grant Agreement No 883374 (project EFFECTOR). This article reflects only the
author’s views and the Research Executive Agency (REA) is not responsible for any use that may be
made of the information it contains.
References
Achiri, L., Guida R., Iervolino, P (2018). SAR and AIS fusion for maritime surveillance. IEEE 4th
International Forum on Research and Technology for Society and Industry.
Anneken, M., de Rosa,
F., Jousselme, A.-L., Robert, S. (2019).
Modelling
Dynamic
Bayesian
Networks
to Identify Behaviour of Interest. In Proceedings of the Maritime Big Data Workshop, NATO-STO-
CMRE, CMRE-CP-2018-002, 31-35.
Bentes, C., Velotto, D., Tings, B. (2017). Ship classification in TerraSAR-X images with convolutional
neural networks. IEEE Journal of Oceanic Engineering 43.1, 258-266.
Chein M., Mugnier, M.-L. (2008). Graph-based Knowledge Representation: Com-putational Foundations of
Conceptual Graphs. Springer, 2008.
Chen R., et al. (2019). Mobility modes awareness from trajectories based on clustering and a
convolutional neural network. ISPRS International Journal of Geo-Information 8.5, 208.
Dahlbom A., Niklasson, L. (2007). Trajectory clustering for coastal surveillance. 10th International
Conference on Information Fusion. IEEE, 2007.
De Vries, G., Van Someren, M. (2012). Machine learning for vessel trajectories using compression,
a
li
g
n
men
t
s
a
nd do
m
a
i
n
k
no
w
l
ed
g
e.
Ex
p
e
r
t
S
yste
m
s
w
ith
A
pp
l
i
c
ati
o
n
s
39.18,
13426-13439.
EFFECTOR EU Project - “An End to end Interoperability Framework For MaritimE Situational
Awareness at StrategiC and TacTical OpeRations”, Grant agreement ID: 883374, 2020.
European Maritime Safety Agency (EMSA). CISE Architecture Document
(
http
:
//
www
.
em
s
a
.
eu
rop
a
.
eu
/
do
w
n
l
o
a
d
).
European Maritime Safety Agency (EMSA). (2021). Annual Overview of Marine Casualties and
Incidents 2021.
Feng, M., Li, Y. (2012) Ship intelligent collision avoidance based on Maritime Police Warships
Simulation System. IEEE Symposium on Electrical & Electronics Engineering, pp. 293–296.
Fischer, Y., Bauer, A. (2010). Object-oriented sensor data fusion for wide maritime surveillance. IEEE
International WaterSide Security Conference.
Z. Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. Jacobé de Naurois, C. Laudy,
C. Rizogiannis, A. Astyakopoulos & A. Blum:
159
Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment
Handayani, D., Dwi, O., Sediono, W., Shah, A. (2013). Anomaly detection in vessel tracking using
s
u
pport
ve
ct
or
m
a
ch
i
n
e
s
(
S
VMs
).
I
n
t
e
r
na
t
i
o
na
l
Con
f
e
r
e
n
ce
o
n
A
d
v
an
ce
d
Compu
t
e
r
S
c
i
e
n
ce
A
pp
l
i
c
a
t
i
o
n
s
and Technologies.
Helleren, Ø., Olsen, Ø., Narheim, B. T., Skauen, A. N., Olsen, R. B. (2012). AISSat-1–2 years of service.
In Proceedings of the 4S Symposium, Slovenia.
Kanjir, U., Greidanus, H., Oštir, K. (2018). Vessel detection and classification from spaceborne optical
images: A literature survey. Remote sensing of environment 207, 1-26.
Kao, S.-L., Lee, K.-T., Chang, K.-Y., Ko, M.-D. (2007). A Fuzzy Logic Method for Collision Avoidance
in Vessel Traffic Service. Journal of Navigation, 60,17–31.
Laudy, C., Jacobé de Naurois, C. (2021). Nested conceptual graphs for information fusion traceability.
In International Conference on Conceptual Structures, pp. 19–33. Springer.
Laudy, C., Ruini, F., Zanasi, A., Przybyszewski, M., Stachowicz, A. (2017). Using social media in crisis
management: SOTERIA fusion center for managing information gaps. In 20th IEEE
International Conference on Information Fusion, 2017, Xi’an, China, pp. 1–8.
Laudy, C. (2010). Introducing semantic knowledge in high level information fusion.
PhD
thesis,
Paris 6
University.
Liu, Y.-H., Liu, H.-X. (2006). Case Learning Base on Evaluation System for Vessel Collision Avoidance.
International Conference on Machine Learning and Cybernetics, pp. 2064–2069.
Liu B., et al. (2015). Ship movement anomaly detection using specialized distance measures. IEEE 18th
International Conference on Information Fusion.
Mihailović A., et al. (2021a). A Framework for Incorporating a National Maritime Surveillance System
into the European Common Information Sharing Environment. The 25th International Conference
on Information Technology (IT), p.1-6.
Mihailović, A., Kapidani, N., Kočan, E., Merino, D., Rasanen, J. (2021b). Analysing the prospect of the
maritime common information sharing environment’s implementation and feasibility in
Montenegro. In Scientific Journal of Maritime Research, No. 35, pp. 256-266.
Nguyen, D., et al. (2018). A multi-task deep learning architecture for maritime surveillance using ais
dat
a
s
t
r
eam
s
.
I
EEE
5
th
I
n
te
r
nati
o
na
l
C
o
n
f
e
r
e
n
ce
o
n
D
ata
S
c
i
e
n
ce
and
A
d
va
n
ce
d
A
na
l
y
ti
c
s
(
D
S
AA
)
.
Nguyen D., et al. (2021). GeoTrackNet--A Maritime Anomaly Detector Using Probabilistic Neural
Network Representation of AIS Tracks and A Contrario Detection. IEEE Transactions on
Intelligent Transportation Systems.
Nguyen, V.-S., Im, N.-K., Lee, S.-M. (2015). The interpolation method for the missing AIS data of
ship. Journal of navigation and port research 39.5, 377-384.
Paladin, Z., Mihailović, A., Kapidani, N., Merino, D., Grenner, J.-M., Vella, G., Moutzouris, M., Leuzzi,
R. (2021). Augmenting maritime Command and Control over a regional CISE implementation:
Montenegro Case. In NMIOTC 2021 Journal, 22 (1), 20-29.
Rizogiannis, C., Thomopoulos, S.C.A. (2019). A fuzzy inference system for ship-ship collision alert
g
en
era
t
i
on
.
In
P
r
o
cee
din
g
s
o
f
S
P
I
E
V
o
l
.
11018
,
S
i
g
na
l
P
r
o
ce
ss
in
g
,
S
e
n
so
r
/
I
n
f
o
r
ma
t
i
o
n
F
u
s
i
o
n
,
and
T
a
r
ge
t
Recognition.
Soldi, G., et al. (2021). Space-based global maritime surveillance. Part II: Artificial intelligence and data
fusion techniques. IEEE Aerospace and Electronic Systems Magazine 36.9, 30-42.
Sowa, J.-F. (1984). Conceptual Structures. Information Processing in Mind and Machine.
Svenmarck, P., Luotsinen, L., Nilsson, M., Schubert, J. (2018). Possibilities and Challenges for Artificial
Intelligence in Military Applications. In Proceedings STO-MP-IST-160, pp. S5 –15.
Szlapczynski, R., Szlapczynska, J. (2017). A Framework of a Ship Domain-based Collision Alert System, pp
183–189.
Varlamis, I., et al. (2019). A Network Abstraction of Multi-vessel Trajectory Data for Detecting
Anomalies. EDBT/ICDT Workshops, Vol. 2019.
Wang, R., et al. (2018). Study on the combined application of CFAR and deep learning in ship detection.
Journal of the Indian Society of Remote Sensing 46.9, 1413-1421.
Xu, Q, Wang, N. (2014). A Survey on Ship Collision Risk Evaluation. PROMET, 26(6), 475–486.
Xu, Q., Meng, X., Wang, N. (2009). Ship Manipulation Evaluation System. International Conference on
Measuring Technology and Mechatronics Automation, 3,859–862.
35TH BLED ECONFERENCE
160
DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION
Zhao, L., Shi, G. (2019). Maritime anomaly detection using density-based clustering and recurrent
neural network. The Journal of Navigation 72.4, 894-916.
Zhen, R., et al. (2017). Maritime anomaly detection within coastal waters based on vessel trajectory
clustering and Naïve Bayes Classifier. The Journal of Navigation 70.3, 648-670.
