XR4DRAMA Knowledge Graph: A Knowledge Graph for Disaster Management

Full text

XR4DRAMA Knowledge Graph: A Knowledge
Graph for Disaster Management
Alexandros Vassiliades
Information Technologies Institute
CERTH
Thessaloniki, Greece
valexande@iti.gr
Spyridon Symeonidis
Information Technologies Institute
CERTH
Thessaloniki, Greece
spyridons@iti.gr
Sotiris Diplaris
Information Technologies Institute
CERTH
Thessaloniki, Greece
diplaris@iti.gr
Georgios Tzanetis
Information Technologies Institute
CERTH
Thessaloniki, Greece
tzangeor@iti.gr
Stefanos Vrochidis
Information Technologies Institute
CERTH
Thessaloniki, Greece
stefanos@iti.gr
Nick Bassiliades
School of Informatics
Aristotle University of Thessaloniki
Thessaloniki, Greece
nbassili@csd.auth.gr
Ioannis Kompatsiaris
Information Technologies Institute
CERTH
Thessaloniki, Greece
ikom@iti.gr
Abstract—The evolution of Knowledge Graphs (KGs), during
the last two decades, has encouraged developers to create more
and more context related KGs. This advance is extremely
important because Artificial Intelligence (AI) applications can
access open domain specific information in a semantically rich,
machine understandable format. In this paper, we present the
XR4DRAMA KG which can represent information for disaster
management. More specifically, the XR4DRAMA KG can repre-
sent information about: (a) Observations and Events (e.g., data
collection of biometric sensors, information in photos and text
messages), (b) Spatio-temporal (e.g., highlighted locations and
timestamps), (c) Mitigation and response plans in crisis (e.g.,
first responder teams). Moreover, we offer a mechanism that can
create or update Points-Of-Interest (POIs), based on a visual or
textual messages received from users.
Index Terms—Knowledge Graphs, Disaster Management,
Points of Interest, POI Management Mechanism
I. INTRODUCTION
The creation of context related Knowledge Graphs (KGs),
i.e., KGs that can be used only in specific environments,
seems to be the next step for allowing KGs to become
the main knowledge representation format for the Web [1].
Our focus in this work is on representing information for
disaster management, more specifically, information about: (a)
Observations and Events (e.g., data collection of biometric
sensors, information in photos and text messages), (b) Spatio-
temporal features (e.g., highlighted locations and timestamps),
(c) Mitigation and response plans in crisis situations (e.g.,
for the first responder teams). For an individual in a disaster
management situation it is also important to access geospatial
data that contains information about the location that suffered
the destruction. For this reason, we provide a mechanism that
creates or updates Points-of-Interest (POIs)1. We will refer
throughout the paper to the POI creation/update mechanism
as POI management mechanism. The formal definition of a
POI is a specific place or location point inside a map that
someone may find useful or interesting. In our case, POIs also
include geospatial data which contain either textual or visual
information about the state of a location which has suffered a
destruction.
The XR4DRAMA KG was developed in order to work as
the knowledge representation of the XR4DRAMA project2.
XR4DRAMA is dedicated to improving situation awareness
via extended reality (XR) and a number of other technologies.
One of the main use cases of the XR4DRAMA project
focuses on disaster management. Therefore, the XR4DRAMA
KG can represent the structures and integrate the results
coming from multiple advanced analysis components that
process multimodal data (in this project we integrate visual,
textual, and stress level analysis messages). Additionally, the
XR4DRAMA KG through its POI management mechanism
offers an innovative mechanism that can create or update POIs,
which contain crucial geospatial information that is needed in
a case of emergency.
Our contribution in this paper, is on one hand the
XR4DRAMA KG which can represent multi-modal mea-
surements, by mapping textual, visual, and stress level mes-
sages/measurements, which in turn can aid citizens and first
1https://xr4drama.eu/2022/07/07/xr4drama-pois- virtual-whiteboards/
2https://xr4drama.eu/
262
2023 IEEE 17th International Conference on Semantic Computing (ICSC)
978-1-6654-8263-9/23/$31.00 ©2023 IEEE
DOI 10.1109/ICSC56153.2023.00051
2023 IEEE 17th International Conference on Semantic Computing (ICSC) | 978-1-6654-8263-9/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICSC56153.2023.00051
Authorized licensed use limited to: Aristotle University of Thessaloniki. Downloaded on March 24,2023 at 11:38:07 UTC from IEEE Xplore. Restrictions apply.

responders, in order to avoid crisis or tackle with the best
possible outcome a disaster that has already occurred. On
the other hand, the paper presents the POI management
mechanism for the XR4DRAMA KG, which could be helpful
in real-life scenarios by creating (or updating) POIs that will
further ease the work of first responders and notify the citizens.
The rest of this paper is organized as follows. Section II,
contains the related work. Next, in Section III we present the
XR4DRAMA KG, the POI management mechanism which
creates or updates POIs and the data upon which we con-
structed the KG. Section IV, contains the evaluation of the
KG and the POI management mechanism. We conclude our
paper with Section V.
II. RELATED WORK
The area of KGs for disaster management is rich to present
a handfull of studies, some papers that present a blueprint
of what a KG for disaster management should contain are
presented in [2] and [3], [4]. In the last two the focus is
mostly on geospatial information about a disaster, while the
first one is more general. The difference between these studies
and XR4DRAMA KG, is that they remain at a theoretical
level while we offer a complete KG with a POI management
mechanism.
In [5], [6], the authors present a deep learning model that
can generate a KG for disaster management, but as most data-
driven models it is restricted upon the data that is trained. This
means that if a new case needs to be inferred, for instance a
different type of disaster, new classifiers need to be trained.
Comparing this to the XR4DRAMA KG which is not restricted
to the information existing in some datasets, shows that our
KG might be more general than these models. Close to our
study is [7], where the authors present a KG for disaster
management, but they do not include a POI management
mechanism, for accessing the information in the KG. Similar
is the case of [8], as there is no POI management mechanism.
One can take a more detailed view for the KGs about
disasters and disaster management by reading the survey of
Mazimwe et al. [9].
III. XR4DRAMA KNOWLEDGE GRAPH
The XR4DRAMA KG is part of the back-end of the
project’s platform. For this reason, the multimodal mapping
mechanism which receives messages from the visual, textual,
and stress-level analysis components and passes their containt
into the XR4DRAMA KG will not be analyzed in detail. But
one can find a blueprint of these messages here3. Moreover,
the source code of the multimodal mapping mechanism can
be found here4. The idea of the pipeline is that after the
multimodal mapping mechanism has received the message
from a component, it will map the information into the KG.
Then, when the message arrives from the textual or visual
analysis component the POI management mechanism of the
3https://xr4drama.eu/wp-content/uploads/2021/12/d3.5 xr4drama
semanticrepresentationfusiondss 20211201 v1.2.pdf
4https://github.com/valexande/xr4drama-icsc-paper
XR4DRAMA KG, will create a new POI or update an existing
one, based on the information in the message and information
from the KG. In the second case, the idea is that the state
of a created POI changed, for example a flood has affected
more buildings, thus the information in the initially created
POI needs update. Figure 1, shows an outline of the pipeline,
where each number in the circles shows the order of steps.
Fig. 1. Pipeline of the XR4DRAMA KG
A. Nature of Data
The multi-modality and variety of data flowing in the system
and the necessity of homogenization and fusion mandated
the adoption of a semantic knowledge graph to address the
requirements of the project. The XR4DRAMA KG is not
responsible for archiving and storing raw data files, since
there is an underlying data storage facility for that purpose.
Instead, the XR4DRAMA KG hosts metadata of raw data,
analyses results and miscellaneous information with semantic
value among other candidates, for being mapped and fused
into the knowledge base.
The main categories of data needed to be captured in
the XR4DRAMA KG were: physiological analysis, visual
analysis, and textual analysis results and general information
about virtual reality experiments, but due to lack of space
further analysis can be found here4.
B. A Knowledge Graph for Disaster Management
In this subsection, we describe the structure of the KG
schema (i.e., the XR4DRAMA ontology) and the philosophy
of each class at a high level. The KG along with the codes
developed to populate it can be found here4. Figure 2 illustrates
a high level overview of the core XR4DRAMA ontology
classes.
•InformationOfInterest: The basic entities of interest to
facilitate the decision support.
•Location: This class represent the geographical area
where something happens. It can be presented with coor-
dinates or with the name of the location (e.g., Vicenza).
•Metadata: All the secondary information that comes with
the analysis results and can be used in the decision-
making process.
263
Authorized licensed use limited to: Aristotle University of Thessaloniki. Downloaded on March 24,2023 at 11:38:07 UTC from IEEE Xplore. Restrictions apply.

Fig. 2. XR4DRAMA KG high level illustration
•MultimediaObject: The type of the mean of transmis-
sion that is used to transform information, it can be either
Audio, Textual, or Video.
•Procedure: This class describes the process of analyzing
the date, and is used by each respective component.
•Project: This class describes each operation and some
data regarding them.
•RiskReport: This class describes the aggregated result
of all the risk levels that derive from the different com-
ponents.
•Sensor: Some information about the sensors that are used
during a project.
•User: A user can be a responder or a citizen. Each one of
them has different inputs of data to feed the knowledge
base.
C. Point of Interest Creation and Update
The idea behind POIs is to create some points in an
area (i.e., pins on a map) that contain crucial geospatial
information, that could ease the work of first responders, and
help citizens to avoid the emergency. The creation of a POI
is easily understandable, as if no POIs exist in the area a new
one needs to be created, if an emergency occurred. On the
other hand, the update of POIs is performed when other POI(s)
already exist in the area, and some information in them needs
to be updated, as the state of the emergency has changed. For
instance, a flood affected more buildings. Below we analyze
the information from a visual message that is passed to a POI
when is created or updated (see Table I).
One can notice that when a POI is created the information
passed from the visual analysis messages are: (i) how many
people are in danger, (ii) how many vehicles are in danger,
(iii) how many animals are in danger, (iv) what infrastructure
was affected, (v) what objects are affected, (vi) the type of
emergency, and if the emergency is a flood (vii) if a river
has overflown. The aforementioned data can be dynamic,
meaning that even if some are missing the POI will still
be created. Moreover, the current user is the name of the
user who sends the message, the category and subcategory
characterize the area which was affected. The necessary data
is the current user, the category, the subcategory, and the
TABLE I
INFORMATION PASSED FROM A VISUAL MESSAGE TO A POI WHEN
CREATED
Label Value Example
category string Education
subcategory string Universities
current user string citizen
peopleInDanger integer 0
vehiclesInDanger integer 0
animalsInDanger integer 0
riverOvertop boolean false
emergencyType string flood
objectsInDanger list of strings [car,bench]
infraInDanger list of strings [building]
type string Point
coordinates list of floats [11.5504,45.5499]
coordinates. On the other hand, if a POI already exists only
some information can be updated. The data which can be
updated are: (i-vii), blue colored lines in Table I.
We also analyze the information from a textual message
that is passed to a POI when is created or updated (see
Table II). Similarly, when a POI is created the information
passed from the textual analysis messages are: (i) which are
the affected objects, (ii) an auxiliary label that characterizes
the location, and (iii) the source text of the textual message.
The aforementioned data can be dynamic, meaning that even if
some are missing the POI will still be created. The necessary
data is the current user, the category, the subcategory, and
the coordinates. On the other hand, if a POI already exists
only some information can be updated. The data which can
be updated are: (i-iii), blue colored lines in Table II.
TABLE II
INFORMATION PASSED FROM A TEXTUAL MESSAGE TO A POI WHEN
CREATED
Label Value Example
category string Education
subcategory string Universities
current user string citizen
sourceText string a university was affected by flood
affected objects list of strings [car, man]
label string harbor
type string Point
coordinates list of floats [11.5504,45.5499]
IV. EVALUATION
The evaluation of the XR4DRAMA KG was twofold. On
the one hand, we evaluated the consistency and completeness
of the XR4DRAMA KG; we did this with two different
evaluation methods. Firstly, we evaluated the completeness
and consistency of the XR4DRAMA KG (subsection IV-A.
Secondly, the evaluation of the POI management mechanism
was performed by computing the precision-recall-F1 scores
used for information extraction systems (subsection IV-B).
A. Completeness and Consistency of Knowledge Graph
The completeness of the XR4DRAMA KG was evaluated
through a set of Competency Questions (CQs) assembled
264
Authorized licensed use limited to: Aristotle University of Thessaloniki. Downloaded on March 24,2023 at 11:38:07 UTC from IEEE Xplore. Restrictions apply.

during the formation of the official ontology requirements
specification document (ORSD) [10]. For this reason, before
constructing the XR4DRAMA KG, we asked from users to
define a set of questions that they would like to be answered
by the XR4DRAMA KG. The users were authority workers
from Autorita’ di bacino distrettuale delle alpi orientali5and
journalists from Deutsche Welle6. In total a number of 32 CQs
was collected; which can be found here4. The completeness of
the XR4DRAMA KG was found adequate, as each CQ when
translated into a SPARQL counterpart returned the desired
information.
The consistency of the XR4DRAMA KG was found ade-
quate, as out of 56 SHACL rules, from which 21 referred to
object type properties and 35 to data type properties, none
of them returned any invalidation of the rule. Moreover, we
checked if there exist instances which belong to intersection
of classes, as we did not desire such a case, and there were
not any.
B. POI Management Mechanism Evaluation
The evaluation of the POI management mechanism was
performed using the usual precision, recall and F1-score used
for information extraction systems (Equations 1, 2 and 3),
for the creation or update of POIs from visual and textual
messages.
precision =|{RelevantI nstance}∩{RetrievedI nstance}|
|{RetrievedI nstance}| (1)
recall =|{RelevantInstance}∩{RetrievedInstance}|
|{RelevantI nstance}| (2)
F1=2∗recall∗precision
recall+precision (3)
Retrieved Instances are considered all the visual (or textual)
messages for which the POI management mechanism, did not
return an error when we casted a message in order to create
or update a POI.
Relevant Instances are considered all the the visual (or
textual) messages for which the POI management mechanism,
managed to create or update a POI, when we casted a message
with them.
The dataset on which we evaluated our POI management
mechanism contains a set of 1201 textual messages and 600
visual messages, and can be found here4. Notice that in order
to tackle potential biases, the value of each label in each
message was randomly collected from a gold standard dataset
created from domain experts. The precision, recall and F1-
scores for both textual and visual messages can be found in
Table III. Notice, the results are rounded to four decimals.
5http://www.alpiorientali.it/
6https://www.dw.com/en/news/s-30701
TABLE III
PRECISION,RECALL AND F1-SCORES FOR TEXTUAL AND VISUAL
MESSAGES
Precision Recall F1
Textual Messages 0.88 1.0 0.9361
Visual Messages 0.89 1.0 0.9417
V. C ONCLUSION
In this paper we presented the XR4DRAMA KG, a KG
that can represent knowledge for disaster management, for
example, information such as: (a) Observations and Events,
(b) Spatio-temporal data, (c) Mitigation and response plans
in crisis. Additionally, the XR4DRAMA KG through its POI
management mechanism offers an innovative mechanism that
can create or update POIs.
For future work, we plan to develop a mechanism that
will make the POIs more helpful in decision making. More
specifically, we will add a severity score computed by a
decision support system in the POI, in order for the POI to
indicate the severity of the destruction. Additionally, we will
equip POIs with a sequence of tasks that need to be performed
when a disaster has occurred.
ACKNOWLEDGMENT
This work has been funded by XR4DRAMA Horizon 2020
project, grant agreement number 952133.
REFERENCES
[1] B. Abu-Salih, “Domain-specific knowledge graphs: A survey,” Journal
of Network and Computer Applications, vol. 185, p. 103076, 2021.
[2] S. Werder, “Knowledge representation for disaster management,” in
Proc. of International Symposium on Strong Vrancea Earthquakes and
Risk Mitigation, 2007, pp. 4–6.
[3] W. Xu and S. Zlatanova, “Ontologies for disaster management response,”
in Geomatics Solutions for Disaster Management. Springer, 2007, pp.
185–200.
[4] E. Klien, M. Lutz, and W. Kuhn, “Ontology-based discovery of geo-
graphic information services—an application in disaster management,”
Computers, environment and urban systems, vol. 30, no. 1, pp. 102–123,
2006.
[5] Y. Zhang, J. Zhu, Q. Zhu, Y. Xie, W. Li, L. Fu, J. Zhang, and J. Tan,
“The construction of personalized virtual landslide disaster environments
based on knowledge graphs and deep neural networks,” International
Journal of Digital Earth, vol. 13, no. 12, pp. 1637–1655, 2020.
[6] H. Purohit, R. Kanagasabai, and N. Deshpande, “Towards next gener-
ation knowledge graphs for disaster management,” in 2019 IEEE 13th
international conference on semantic computing (ICSC). IEEE, 2019,
pp. 474–477.
[7] J. L. Moreira, L. Ferreira Pires, M. van Sinderen, and P. D. Costa, “To-
wards ontology-driven situation-aware disaster management,” Applied
ontology, vol. 10, no. 3-4, pp. 339–353, 2015.
[8] G. Babitski, S. Bergweiler, O. Grebner, D. Oberle, H. Paulheim, and
F. Probst, “Soknos–using semantic technologies in disaster management
software,” in Extended Semantic Web Conference. Springer, 2011, pp.
183–197.
[9] A. Mazimwe, I. Hammouda, and A. Gidudu, “Implementation of fair
principles for ontologies in the disaster domain: A systematic literature
review,” ISPRS International Journal of Geo-Information, vol. 10, no. 5,
p. 324, 2021.
[10] M. C. Su ´
arez-Figueroa, A. G´
omez-P´
erez, and B. Villaz´
on-Terrazas,
“How to write and use the ontology requirements specification docu-
ment,” in OTM Confederated International Conferences” On the Move
to Meaningful Internet Systems”. Springer, 2009, pp. 966–982.
265
Authorized licensed use limited to: Aristotle University of Thessaloniki. Downloaded on March 24,2023 at 11:38:07 UTC from IEEE Xplore. Restrictions apply.