Content uploaded by Georgios Meditskos
Author content
All content in this area was uploaded by Georgios Meditskos on Dec 03, 2020
Content may be subject to copyright.
Supporting the Discovery and Reuse of Digital
Content in Creative Industries using Linked Data
Maria Rousi, Georgios Meditskos, Stefanos Vrochidis, Ioannis Kompatsiaris
Information Technologies Institute
Centre for Research and Technology Hellas, Thessaloniki, Greece
Email: {mariarousi, gmeditsk, stefanos, ikom}@iti.gr
Abstract—Creative industries play an important role in today’s
economy, being sources of growth that boost future development.
Their products have a huge potential to be re-used and re-
purposed, while at the same time they can enrich and improve
the design process and create newly exploitable content. This
short paper presents V4Link, a semantic annotation and linking
framework of multimodal annotations deriving from visual and
textual analysis of digital content. The framework reuses the
Web Annotation Data Model to capture the annotations, while
it semantically enriches the generated knowledge graphs by
defining links to Linked Datasets, offering to users advanced
query capabilities beyond the ones that can be supported on
top of the initial multimodal knowledge base. We illustrate the
capabilities of the framework presenting a number of use cases
that inspire and support the design, architecture, as well as the
3D and VR game creative industries.
Keywords—Semantic Representation, Linked Data, Semantic
Retrieval, Semantic Search.
I. INTRODUCTION
Architecture and Virtual Reality (VR) game design are
two of the most highly competing and demanding sectors
that can benefit from the available digital content. The high-
consumer demands in these sectors require from designers
to be constantly creative and multi-skilled, adapted to the
latest technology and coupled with strong presentational skills.
In this quest, the leverage of existing digital content can
drastically speed up the time to market and serve as sources of
inspiration towards innovative designs and new concepts. For
example, architects can be inspired by existing architecture and
related spatial elements of landscapes or historical buildings
to design surrounding elements, such as facades, during the
architectural design procedure in 3D environments.
In this context, the H2020 project V4Design1aims to exploit
state-of-the-art technological means so as to re-use and re-
purpose existing heterogeneous multimedia content and inspire
the design, architecture, as well as 3D and VR game industries.
More specifically, V4Design enables the re-use and re-purpose
of multimedia content by proposing innovative solutions to
extract 3D representations and VR game environments. This
allows architects, designers and video game creators to re-use
heterogeneous archives of already available digital content and
re-purpose it by making the wealth of 3D, VR, aesthetic and
textual information easily accessible and providing resources
1https://v4design.eu/
and tools to design and model outdoors and indoors environ-
ments of architecture and VR video game projects.
The enrichment of designers’ toolkits and workflows with
multimodal data (e.g. visual and textual) from various sources
can serve as a trigger for inspiration. However, the repre-
sentation and integration of such information by itself is not
adequate to serve as a catalyst for innovative creations and
assist the creative industries in sharing content and maximize
its exploitation. The retrieval and repurposing of content
demands intelligent mechanisms and novel approaches for
knowledge interlinking and contextual enrichment to facilitate
its discovery and integrate it into the design process.
In this short paper we propose V4Link, a framework for
combining and further linking the extracted metadata of the
V4Design technologies, so as to generate rich knowledge
graphs and support designers in discovering assets using
contextual information, such as architectural style, type of
building, distance from water (using Linked Datasets), creator,
height, number of walls, etc. To achieve this, the framework
maps incoming multimodal information to the V4Design con-
ceptual model that follows the Web Annotation Data Model
(WADM) [1]. The results are stored in the Knowledge Base,
where rules are applied for knowledge enrichment through
logical inferences. A semantic querying framework executes
the appropriate queries into the local Knowledge Base and
Linked Data interfaces (DBpedia and LinkedGeoData) to
detect buildings that meet the respective criteria.
The contribution of our research is summarized in the
following:
•We describe the annotation model that follows and ex-
tends WADM for semantically representing the data.
•We define a set of rules to extract useful inferences and
generate hidden knowledge.
•We propose a combination of Linked Data with local
knowledge to support useful domain-specific queries.
The rest of the paper is organized as follows. Section
II provides a literature review on issues related to seman-
tic representation and Linked Data. Section III presents the
methodology of our work, the heterogeneous data sources
and the way they are semantically represented. Section IV
describes example use cases. Section V presents evaluation
results and section VI concludes this work and reports possible
future steps.
II. RE LATE D WORK
Semantics and Linked Data are the appropriate means to
expand the knowledge over specific domains, like for instance
cultural heritage. Over the last decades, progress has been
made on retrieving knowledge over cultural heritage objects
by taking advantage of the capabilities that Linked Data offer
[2]. In [3] the scope is to represent geospatial resources that
change over time under the cultural heritage domain, while
in [4] a framework has been developed that applies semantic
annotation and search over cultural heritage resources. To this
end, they use the ClioPatria software, semantic web libraries,
SPARQL API and graph-search API. The mobile application
presented in [5] is based on the device GPS and the strength
of Linked Data, to detect cultural information that are in
close distance. Accessing a digital library by exploiting the
capabilities of semantics has been reported in [6]. Most studies
[7], [5] conclude that when information from different sources
are integrated, semantic retrieval performance is improved and
intelligent searching functionalities can be supported.
The big advantage of semantically representing and pub-
lishing data in the web is that one can easily retrieve data
using semantic queries. Useful interconnections can be created
between data, which are gaining value as they are semantically
enriched with information coming from different sources.
DBpedia2is a popular publicly available Knowledge Base
that contains an extremely wide range of information which
are composed from Wikipedia infoboxes. Another popular
database is GeoNames3, which focuses more on geospatial
information like country boundaries, population, etc. Linked-
GeoData4is a Knowledge Base that makes OpenStreetMap
data available using semantic queries. Data are mostly oriented
in the geolocation of different types of buildings. Another
popular source of information is the Europeana Data Model
(EDM)5. The model is based on cultural heritage objects
coming from digitised museums, libraries and archives.
V4Link aims to provide a practical content interlinking
framework on top of the V4Design multimodal platform (see
section III for more details on the available metadata), able
to facilitate advanced, context-aware discovery of 3D assets
taking into account: a) local relationships and interconnections
stemming from the multimedia analysis components, e.g.
aesthetics extraction, and b) links to external Linked Datasets,
semantically enriching the local knowledge graphs.
III. V4LINK KNOW LE DG E GRAPHS
The conceptual architecture of V4Link is depicted in Figure
1. Input is coming from multimodal analysis, such as Shot
Detection and Text Analysis. The incoming information is
semantically represented, following the annotation model that
we describe later in this section. Knowledge is further enriched
using domain-specific rules and external sources, namely DB-
pedia and LinkedGeoData, in the form of semantic queries.
2https://wiki.dbpedia.org/
3https://www.geonames.org/
4http://linkedgeodata.org/About
5https://pro.europeana.eu/page/edm-documentation
Fig. 1. The overall framework architecture
The output of this pipeline is the set of assets that fulfil the
searching criteria, such as 3D models of buildings.
V4Design integrates a number of multimodal analysis tech-
niques. A brief description for each component is found below:
•Text Analysis detects key entities (disambiguated) in
captions and descriptions. Information is mostly related
with the DBpedia URIs, language, additional triples from
Wikipedia analysis, etc.
•Aesthetics Extraction analyses the aesthetics of a paint-
ing and matches information like the creator, style and
the emotion that each painting cause to the viewer.
•Spatio-Temporal Building and Object Localisation
(STBOL) detects the type of building or object (e.g.
castle, furniture), along with masks.
•Building Information Model [8] (BIM) information is
made available regarding the 3D model topology, such as
floors count, ceilings count, roofs count, walls count, etc.
•3D Model Reconstruction reconstructs the 3D model
that comes from a video or set of images. V4Link
integrates and attaches to the 3D models all the relevant
metadata extracted through the above mentioned multi-
modal analysis techniques.
A key design choice underpinning the engineering of the
V4Link knowledge graphs is the adherence to an existing
standard so as to capitalise on a modular, extensible and inter-
operable framework for expressing annotations and achieve a
better degree of knowledge sharing, reuse and interoperability.
In particular, we use the Web Annotation Data Model, where
an annotation is considered to be a set of connected resources,
typically including a body and target, and conveys that the
body is related to the target. The exact nature of this rela-
tionship changes according to the intention of the annotation,
Fig. 2. The core Web Annotation Data Model pattern.
Fig. 3. Example of semantic representation of building localisation.
but the body is most frequently somehow “about” the target
(Figure 2).
As an example, we present the generated knowledge graph
for localising the Eiffel Tower in an image. More specif-
ically, the mask is generated for the building that is fur-
ther with a tag (tower). Following the WADM model, a
v4d:BuildingLocalisationAnnotation resource is generated that
is linked with the target of the annotation, i.e. the generated
mask (Mask 1) and the body annotation view (BuildingLocal-
isationView 1). The latter, defines property assertions relevant
to the original image where this mask has been extracted from
(Image 1), as well as the relevant tag which is the BabelNet
resource for the concept “tower”. The figure also depicts
additional descriptive properties relevant to id references in
the underlying data storage.
IV. CON TE XT-AWARE SEARCH
Since DBpedia entities are already available from the multi-
modal analysis pipeline, e.g. from text analysis, we capitalise
on this entity to extract more information of architectural
nature related to each asset, such as architectural style, creator,
etc. Additionally, we take advantage of the DBpedia entities to
access LinkedGeoData. Using LinkedGeoData further enriches
the Asset information with geolocation information as detect-
ing the proximity between an asset and other points of interest
is feasible. More specifically, each asset contains one main en-
tity from DBpedia. The following scenarios run for all the as-
set–entity couples and detect assets that comply with the given
criteria, taking advantage of the Linked Data dynamics. The
field < entity > represents the main entity which may be for
instance < http ://dbpedia.org/r esource/T oronto City H all >.
In this section we describe two different scenarios that
illustrate the way V4Link can assist in the design process.
1) Scenario 1: Proximity to Point of Interest (POI): The
aim in to detect assets that are close to a specific type of
POIs, such as Museums, Water objects, Archaeological Sites,
etc. This type of query has been specifically requested by the
users, as it allows them to retrieve assets and being inspired
based on surrounding information. To this end, we combine the
DBpedia entities that are extracted by analysing the captions
and descriptions of multimedia content, and LinkedGeoData
(lgdo). The query (Listing 1) uses the objects’ geometries
and the bif:st intersects function that detects whether the two
geometries (geom) are within a radius of one kilometer. The
first geometry is defined using the DBpedia entity that has
been detected in text analysis procedure, while the latter is
defined by a class which in this case is a lgdo:Museum. The
same query is applied to detect the proximity of a building
to hydrographic network objects or other types structures of
architectural interest i.e. Archaeological Sites, Art Centres,
Attractions, etc. using in each case the corresponding classes.
The type of POI and distance vary according to the criteria
that are defined by each user.
PREFI X rd f s : <h t t p : / / www. w3 . o r g / 2 0 0 0 / 0 1 / r d f −s c he ma#>
PREFI X lg d o : <h t t p : / / l i n k e d g e o d a t a . o r g / on t o l o g y />
PREFI X geom : <h t t p : / / g e o v oc a b . o r g / g e o m e t r y#>
PREFIX o gc : <h t t p : / / www. o p en g i s . n e t / o n t / g e o s p a r q l #>
PREFIX o wl : <h t t p : / / www. w3 . o r g / 2 0 0 2 / 0 7 / ow l#>
PREFI X bi f : <b i f :>
SELECT *WHERE{
? s ow l : sa meA s <entity >;
geom : ge o m e t r y [ o g c : asWKT ? s g ] .
? x a lg d o : Muse um ;
r d f s : l a b e l ? l ; geom : g e o m e t r y [ o gc : asWKT ? x g ] .
FILT ER ( b i f : s t i n t e r s e c t s ( ? sg , ? xg , 1 ) ) .
}LIMIT 1
Listing 1. Example of semantic query to decide whether a building has
Museums in an 1 kilometer distance using LinkedGeoData
2) Scenario 2: Architecture-related requirements: In this
scenario the scope is to detect assets by integrating multimodal
information, e.g. they have a specific architectural style or
creator, belong to a specific building type, have been renovated
or not. In the following query (Listing 2) we detect whether
a building has not been renovated, taking advantage of the
wide range of information and properties that DBpedia offers.
We detected several different properties such as dbp:restored,
that express the date of renovation for a DBpedia entity.
To implement this query, we took advantage of the different
properties that are provided by DBpedia and the FILTER NOT
EXISTS function of SPARQL.
PREFIX : <h t t p : / / d b pe d i a . o rg / r e s o u r c e />
PREFIX d bp : <h t t p : / / d b p e d ia . o r g / p r o p e r t y />
SELECT *WHERE {
FILTER NOT EXISTS {
<entity>db p : r e s t o r e d ? r e n d a t e 0 . }
FILTER NOT EXISTS {
<entity>db p : da t e R e n o v a t e d ? r e n d a t e 1 . }
FILTER NOT EXISTS {
<entity>db p : r e n o v a t i o n D a t e ? r e n d a t e 2 . }
<entity>? p r o p e r t y ? v a l u e .
}LIMIT 1
Listing 2. Example of querying DBpedia to detect non-renovated buildings
In cases that we want to detect a specific architectural style, we
combine information coming from the local Knowledge Base
with Linked Data. More specifically, as presented in Listing 3
the query searches among a list of different types of properties
that express the architectural style, whether a specific entity
follows the selected style. This query is running on DBpedia
data. Since not all DBpedia properties exist on each entity,
we use the OPTIONAL function of SPARQL to execute this
query and detect all the different architectural style values that
may exist for each entity.
PREFIX r d f s : <h t t p : / / www. w3 . o rg / 2 0 0 0 / 0 1 / rd f −s ch e ma#>
PREFIX : <h t t p : / / d b pe d i a . o r g / re s o u r c e />
PREFIX dbp : <h t tp : / / d b p ed i a . or g / p r o p e r t y/>
PREFIX dbo : <h t tp : / / d b p e d i a . or g / o n t o l o g y />
SELECT *WHERE {
OPTIONAL {
<entity>db p : a r c h i t e c t u r e S t y l e ? s t y l e 0 .
? s t y l e 0 r d f s : l a b e l ? a s t 1 . }
OPTIONAL {
<entity>db p : s t y l e ? s t y l e 1 .
? s t y l e 1 r d f s : l a b e l ? a s t 2 . }
OPTIONAL {
<entity>db o : a r c h i t e c t u r a l S t y l e ? s t y le 3 .
? s t y l e 3 r d f s : l a b e l ? a s t 3 . }
OPTIONAL {
<entity>db p : a r c h i t e c t u r e ? s t y l e4 .
? s t y l e 4 r d f s : l a b e l ? a s t 4 . }
OPTIONAL {
<entity>db p : a r c h i t e c t u r e ? a s t5 .
FILTE R NOT e x i s t s {?a s t 5 r d f s : l a b e l ? s t y l e 5 .} }
FILTE R ( r e g e x ( s t r (? a s t 1 ) , ” Ba r oq u e ”) | |
r e g e x ( s t r ( ? a s t 2 ) , ” Ba r oq u e ”) | | r e g e x ( s t r ( ? a s t 3 ) , ” Ba r oq u e ”) | |
r e g e x ( s t r ( ? a s t 4 ) , ” Ba r oq u e ”) | | r e g e x ( s t r ( ? a s t 5 ) , ” Ba r oq u e ”) )
FILTE R ( l a n g M a t c h e s ( la n g ( ? a s t 1 ) , ’ e n ’ ) && l a n g M a t c h e s ( l a n g ( ? a s t 2 ) , ’ en ’ )
&& l a n g M a t c h e s ( l a n g ( ? a s t 3 ) , ’ en ’ ) && l a n g M a t c h e s ( la n g ( ? a s t 4 ) , ’ en ’ ) )
}
Listing 3. Example of querying DBpedia to detect buildings with Baroque
architectural style
V. EVAL UATION
In Table I we display the execution time in seconds of
transforming incoming annotations into the V4Link annotation
model. The modality that needs the most time to be mapped
and stored in the RDF triple store (we have use the free
version of GraphDB6) is the 3D model reconstruction, while
BIM information is handled more efficiently. This is mainly
because BIM does not require any additional mapping logic
in the form of inference/integration rules, while the 3D model
reconstruction assets serve as main resources that need to be
associated with all the available metadata.
TABLE I
MEAN EXECUTION TIME OF SEMANTIC REPRESENTATION,STOR AGE A ND
REASONING PER COMPONENT
Component Execution time in seconds
Text analysis 1.87
Aesthetics extraction 0.92
STBOL 1.72
3D Model Reconstruction 1.99
BIM 0.55
In Table II we present the mean execution time per scenario
execution. The calculation has been applied in a set of 681
Assets, where 163,966 triples exist. The conclusion of this
process is that execution time increases rapidly when querying
in Linked Data is applied or there is a large amount of triples.
6https://graphdb.ontotext.com/
The number of properties that appear in the queries seems
to also increase execution time, as in cases like architecture-
related requirements, there is a large number of properties and
execution time is also high.
TABLE II
MEA N EXE CU TIO N TIM E OF S EMA NT IC SE AR CH BA SED O N EAC H
SC ENA RIO
Scenario Mean execution time in seconds
Proximity to POI 11.17
Architecture-related requirements 13.72
VI. CONCLUSION
In this short paper, we described V4Link, a framework that
takes advantage of Linked Data capabilities to support designer
(such as architects and video game designers) to intelligent
discovery of assets. Data coming from multimodal components
is mapped to RDF graphs following the Web Annotation Data
Model and is coupled with popular Linked Data datasets,
namely DBpedia and LinkedGeoData. User-driven queries are
then executed to assist the design process. The evaluation
showed that the semantic representation, reasoning and storage
has low execution time, while the execution time of semantic
search is strongly affected by the usage of Linked Data.
Future work includes the detection of different Linked Data
datasets which can be used to support scenarios that are not
included in the current version of this framework. Additional
properties of the existing datasets may also be examined. The
scenarios may be updated accordingly to meet the needs of
the users.
ACK NOW LE DG ME NT
This work was supported by the EC funded project
V4Design (H2020-779962)
REFERENCES
[1] R. Sanderson, P. Ciccarese, and B. Young, “Web Annotation
Data Model,” W3C, pp. 1–56, 2016. [Online]. Available:
https://www.w3.org/TR/annotation-model/
[2] K. N. Vavliakis, G. T. Karagiannis, and P. A. Mitkas, “Semantic web
in cultural heritage after 2020,” in Proceedings of the 11th International
Semantic Web Conference (ISWC), Boston, MA, USA, 2012, pp. 11–15.
[3] T. Kauppinen, J. V¨
a¨
at¨
ainen, and E. Hyv¨
onen, “Creating and using
geospatial ontology time series in a semantic cultural heritage portal,”
in European Semantic Web Conference. Springer, 2008, pp. 110–123.
[4] G. Schreiber, A. Amin, L. Aroyo, M. van Assem, V. de Boer, L. Hardman,
M. Hildebrand, B. Omelayenko, J. van Osenbruggen, A. Tordai et al.,
“Semantic annotation and search of cultural-heritage collections: The
multimedian e-culture demonstrator,” Journal of Web Semantics, vol. 6,
no. 4, pp. 243–249, 2008.
[5] C. Van Aart, B. Wielinga, and W. R. Van Hage, “Mobile cultural heritage
guide: location-aware semantic search,” in International Conference on
Knowledge Engineering and Knowledge Management. Springer, 2010,
pp. 257–271.
[6] A. Baruzzo, P. Casoto, P. Challapalli, A. Dattolo, N. Pudota, and C. Tasso,
“Toward semantic digital libraries: Exploiting web2. 0 and semantic
services in cultural heritage,” Journal of Digital Information, vol. 10,
no. 6, p. 4, 2009.
[7] R. Guha, R. McCool, and E. Miller, “Semantic search,” in Proceedings of
the 12th international conference on World Wide Web, 2003, pp. 700–709.
[8] R. Volk, J. Stengel, and F. Schultmann, “Building information modeling
(bim) for existing buildings—literature review and future needs,” Automa-
tion in construction, vol. 38, pp. 109–127, 2014.