Conference PaperPDF Available

Abstract and Figures

Creative industries play an important role in today's economy, being sources of growth that boost future development. Their products have a huge potential to be re-used and re-purposed, while at the same time they can enrich and improve the design process and create newly exploitable content. This short paper presents V4Link, a semantic annotation and linking framework of multimodal annotations deriving from visual and textual analysis of digital content. The framework reuses the Web Annotation Data Model to capture the annotations, while it semantically enriches the generated knowledge graphs by defining links to Linked Datasets, offering to users advanced query capabilities beyond the ones that can be supported on top of the initial multimodal knowledge base. We illustrate the capabilities of the framework presenting a number of use cases that inspire and support the design, architecture, as well as the 3D and VR game creative industries.
Content may be subject to copyright.
Supporting the Discovery and Reuse of Digital
Content in Creative Industries using Linked Data
Maria Rousi, Georgios Meditskos, Stefanos Vrochidis, Ioannis Kompatsiaris
Information Technologies Institute
Centre for Research and Technology Hellas, Thessaloniki, Greece
Email: {mariarousi, gmeditsk, stefanos, ikom}
Abstract—Creative industries play an important role in today’s
economy, being sources of growth that boost future development.
Their products have a huge potential to be re-used and re-
purposed, while at the same time they can enrich and improve
the design process and create newly exploitable content. This
short paper presents V4Link, a semantic annotation and linking
framework of multimodal annotations deriving from visual and
textual analysis of digital content. The framework reuses the
Web Annotation Data Model to capture the annotations, while
it semantically enriches the generated knowledge graphs by
defining links to Linked Datasets, offering to users advanced
query capabilities beyond the ones that can be supported on
top of the initial multimodal knowledge base. We illustrate the
capabilities of the framework presenting a number of use cases
that inspire and support the design, architecture, as well as the
3D and VR game creative industries.
Keywords—Semantic Representation, Linked Data, Semantic
Retrieval, Semantic Search.
Architecture and Virtual Reality (VR) game design are
two of the most highly competing and demanding sectors
that can benefit from the available digital content. The high-
consumer demands in these sectors require from designers
to be constantly creative and multi-skilled, adapted to the
latest technology and coupled with strong presentational skills.
In this quest, the leverage of existing digital content can
drastically speed up the time to market and serve as sources of
inspiration towards innovative designs and new concepts. For
example, architects can be inspired by existing architecture and
related spatial elements of landscapes or historical buildings
to design surrounding elements, such as facades, during the
architectural design procedure in 3D environments.
In this context, the H2020 project V4Design1aims to exploit
state-of-the-art technological means so as to re-use and re-
purpose existing heterogeneous multimedia content and inspire
the design, architecture, as well as 3D and VR game industries.
More specifically, V4Design enables the re-use and re-purpose
of multimedia content by proposing innovative solutions to
extract 3D representations and VR game environments. This
allows architects, designers and video game creators to re-use
heterogeneous archives of already available digital content and
re-purpose it by making the wealth of 3D, VR, aesthetic and
textual information easily accessible and providing resources
and tools to design and model outdoors and indoors environ-
ments of architecture and VR video game projects.
The enrichment of designers’ toolkits and workflows with
multimodal data (e.g. visual and textual) from various sources
can serve as a trigger for inspiration. However, the repre-
sentation and integration of such information by itself is not
adequate to serve as a catalyst for innovative creations and
assist the creative industries in sharing content and maximize
its exploitation. The retrieval and repurposing of content
demands intelligent mechanisms and novel approaches for
knowledge interlinking and contextual enrichment to facilitate
its discovery and integrate it into the design process.
In this short paper we propose V4Link, a framework for
combining and further linking the extracted metadata of the
V4Design technologies, so as to generate rich knowledge
graphs and support designers in discovering assets using
contextual information, such as architectural style, type of
building, distance from water (using Linked Datasets), creator,
height, number of walls, etc. To achieve this, the framework
maps incoming multimodal information to the V4Design con-
ceptual model that follows the Web Annotation Data Model
(WADM) [1]. The results are stored in the Knowledge Base,
where rules are applied for knowledge enrichment through
logical inferences. A semantic querying framework executes
the appropriate queries into the local Knowledge Base and
Linked Data interfaces (DBpedia and LinkedGeoData) to
detect buildings that meet the respective criteria.
The contribution of our research is summarized in the
We describe the annotation model that follows and ex-
tends WADM for semantically representing the data.
We define a set of rules to extract useful inferences and
generate hidden knowledge.
We propose a combination of Linked Data with local
knowledge to support useful domain-specific queries.
The rest of the paper is organized as follows. Section
II provides a literature review on issues related to seman-
tic representation and Linked Data. Section III presents the
methodology of our work, the heterogeneous data sources
and the way they are semantically represented. Section IV
describes example use cases. Section V presents evaluation
results and section VI concludes this work and reports possible
future steps.
Semantics and Linked Data are the appropriate means to
expand the knowledge over specific domains, like for instance
cultural heritage. Over the last decades, progress has been
made on retrieving knowledge over cultural heritage objects
by taking advantage of the capabilities that Linked Data offer
[2]. In [3] the scope is to represent geospatial resources that
change over time under the cultural heritage domain, while
in [4] a framework has been developed that applies semantic
annotation and search over cultural heritage resources. To this
end, they use the ClioPatria software, semantic web libraries,
SPARQL API and graph-search API. The mobile application
presented in [5] is based on the device GPS and the strength
of Linked Data, to detect cultural information that are in
close distance. Accessing a digital library by exploiting the
capabilities of semantics has been reported in [6]. Most studies
[7], [5] conclude that when information from different sources
are integrated, semantic retrieval performance is improved and
intelligent searching functionalities can be supported.
The big advantage of semantically representing and pub-
lishing data in the web is that one can easily retrieve data
using semantic queries. Useful interconnections can be created
between data, which are gaining value as they are semantically
enriched with information coming from different sources.
DBpedia2is a popular publicly available Knowledge Base
that contains an extremely wide range of information which
are composed from Wikipedia infoboxes. Another popular
database is GeoNames3, which focuses more on geospatial
information like country boundaries, population, etc. Linked-
GeoData4is a Knowledge Base that makes OpenStreetMap
data available using semantic queries. Data are mostly oriented
in the geolocation of different types of buildings. Another
popular source of information is the Europeana Data Model
(EDM)5. The model is based on cultural heritage objects
coming from digitised museums, libraries and archives.
V4Link aims to provide a practical content interlinking
framework on top of the V4Design multimodal platform (see
section III for more details on the available metadata), able
to facilitate advanced, context-aware discovery of 3D assets
taking into account: a) local relationships and interconnections
stemming from the multimedia analysis components, e.g.
aesthetics extraction, and b) links to external Linked Datasets,
semantically enriching the local knowledge graphs.
The conceptual architecture of V4Link is depicted in Figure
1. Input is coming from multimodal analysis, such as Shot
Detection and Text Analysis. The incoming information is
semantically represented, following the annotation model that
we describe later in this section. Knowledge is further enriched
using domain-specific rules and external sources, namely DB-
pedia and LinkedGeoData, in the form of semantic queries.
Fig. 1. The overall framework architecture
The output of this pipeline is the set of assets that fulfil the
searching criteria, such as 3D models of buildings.
V4Design integrates a number of multimodal analysis tech-
niques. A brief description for each component is found below:
Text Analysis detects key entities (disambiguated) in
captions and descriptions. Information is mostly related
with the DBpedia URIs, language, additional triples from
Wikipedia analysis, etc.
Aesthetics Extraction analyses the aesthetics of a paint-
ing and matches information like the creator, style and
the emotion that each painting cause to the viewer.
Spatio-Temporal Building and Object Localisation
(STBOL) detects the type of building or object (e.g.
castle, furniture), along with masks.
Building Information Model [8] (BIM) information is
made available regarding the 3D model topology, such as
floors count, ceilings count, roofs count, walls count, etc.
3D Model Reconstruction reconstructs the 3D model
that comes from a video or set of images. V4Link
integrates and attaches to the 3D models all the relevant
metadata extracted through the above mentioned multi-
modal analysis techniques.
A key design choice underpinning the engineering of the
V4Link knowledge graphs is the adherence to an existing
standard so as to capitalise on a modular, extensible and inter-
operable framework for expressing annotations and achieve a
better degree of knowledge sharing, reuse and interoperability.
In particular, we use the Web Annotation Data Model, where
an annotation is considered to be a set of connected resources,
typically including a body and target, and conveys that the
body is related to the target. The exact nature of this rela-
tionship changes according to the intention of the annotation,
Fig. 2. The core Web Annotation Data Model pattern.
Fig. 3. Example of semantic representation of building localisation.
but the body is most frequently somehow “about” the target
(Figure 2).
As an example, we present the generated knowledge graph
for localising the Eiffel Tower in an image. More specif-
ically, the mask is generated for the building that is fur-
ther with a tag (tower). Following the WADM model, a
v4d:BuildingLocalisationAnnotation resource is generated that
is linked with the target of the annotation, i.e. the generated
mask (Mask 1) and the body annotation view (BuildingLocal-
isationView 1). The latter, defines property assertions relevant
to the original image where this mask has been extracted from
(Image 1), as well as the relevant tag which is the BabelNet
resource for the concept “tower”. The figure also depicts
additional descriptive properties relevant to id references in
the underlying data storage.
Since DBpedia entities are already available from the multi-
modal analysis pipeline, e.g. from text analysis, we capitalise
on this entity to extract more information of architectural
nature related to each asset, such as architectural style, creator,
etc. Additionally, we take advantage of the DBpedia entities to
access LinkedGeoData. Using LinkedGeoData further enriches
the Asset information with geolocation information as detect-
ing the proximity between an asset and other points of interest
is feasible. More specifically, each asset contains one main en-
tity from DBpedia. The following scenarios run for all the as-
set–entity couples and detect assets that comply with the given
criteria, taking advantage of the Linked Data dynamics. The
field < entity > represents the main entity which may be for
instance < http :// esource/T oronto City H all >.
In this section we describe two different scenarios that
illustrate the way V4Link can assist in the design process.
1) Scenario 1: Proximity to Point of Interest (POI): The
aim in to detect assets that are close to a specific type of
POIs, such as Museums, Water objects, Archaeological Sites,
etc. This type of query has been specifically requested by the
users, as it allows them to retrieve assets and being inspired
based on surrounding information. To this end, we combine the
DBpedia entities that are extracted by analysing the captions
and descriptions of multimedia content, and LinkedGeoData
(lgdo). The query (Listing 1) uses the objects’ geometries
and the bif:st intersects function that detects whether the two
geometries (geom) are within a radius of one kilometer. The
first geometry is defined using the DBpedia entity that has
been detected in text analysis procedure, while the latter is
defined by a class which in this case is a lgdo:Museum. The
same query is applied to detect the proximity of a building
to hydrographic network objects or other types structures of
architectural interest i.e. Archaeological Sites, Art Centres,
Attractions, etc. using in each case the corresponding classes.
The type of POI and distance vary according to the criteria
that are defined by each user.
PREFI X rd f s : <h t t p : / / www. w3 . o r g / 2 0 0 0 / 0 1 / r d f −s c he ma#>
PREFI X lg d o : <h t t p : / / l i n k e d g e o d a t a . o r g / on t o l o g y />
PREFI X geom : <h t t p : / / g e o v oc a b . o r g / g e o m e t r y#>
PREFIX o gc : <h t t p : / / www. o p en g i s . n e t / o n t / g e o s p a r q l #>
PREFIX o wl : <h t t p : / / www. w3 . o r g / 2 0 0 2 / 0 7 / ow l#>
PREFI X bi f : <b i f :>
? s ow l : sa meA s <entity >;
geom : ge o m e t r y [ o g c : asWKT ? s g ] .
? x a lg d o : Muse um ;
r d f s : l a b e l ? l ; geom : g e o m e t r y [ o gc : asWKT ? x g ] .
FILT ER ( b i f : s t i n t e r s e c t s ( ? sg , ? xg , 1 ) ) .
Listing 1. Example of semantic query to decide whether a building has
Museums in an 1 kilometer distance using LinkedGeoData
2) Scenario 2: Architecture-related requirements: In this
scenario the scope is to detect assets by integrating multimodal
information, e.g. they have a specific architectural style or
creator, belong to a specific building type, have been renovated
or not. In the following query (Listing 2) we detect whether
a building has not been renovated, taking advantage of the
wide range of information and properties that DBpedia offers.
We detected several different properties such as dbp:restored,
that express the date of renovation for a DBpedia entity.
To implement this query, we took advantage of the different
properties that are provided by DBpedia and the FILTER NOT
EXISTS function of SPARQL.
PREFIX : <h t t p : / / d b pe d i a . o rg / r e s o u r c e />
PREFIX d bp : <h t t p : / / d b p e d ia . o r g / p r o p e r t y />
<entity>db p : r e s t o r e d ? r e n d a t e 0 . }
<entity>db p : da t e R e n o v a t e d ? r e n d a t e 1 . }
<entity>db p : r e n o v a t i o n D a t e ? r e n d a t e 2 . }
<entity>? p r o p e r t y ? v a l u e .
Listing 2. Example of querying DBpedia to detect non-renovated buildings
In cases that we want to detect a specific architectural style, we
combine information coming from the local Knowledge Base
with Linked Data. More specifically, as presented in Listing 3
the query searches among a list of different types of properties
that express the architectural style, whether a specific entity
follows the selected style. This query is running on DBpedia
data. Since not all DBpedia properties exist on each entity,
we use the OPTIONAL function of SPARQL to execute this
query and detect all the different architectural style values that
may exist for each entity.
PREFIX r d f s : <h t t p : / / www. w3 . o rg / 2 0 0 0 / 0 1 / rd f −s ch e ma#>
PREFIX : <h t t p : / / d b pe d i a . o r g / re s o u r c e />
PREFIX dbp : <h t tp : / / d b p ed i a . or g / p r o p e r t y/>
PREFIX dbo : <h t tp : / / d b p e d i a . or g / o n t o l o g y />
<entity>db p : a r c h i t e c t u r e S t y l e ? s t y l e 0 .
? s t y l e 0 r d f s : l a b e l ? a s t 1 . }
<entity>db p : s t y l e ? s t y l e 1 .
? s t y l e 1 r d f s : l a b e l ? a s t 2 . }
<entity>db o : a r c h i t e c t u r a l S t y l e ? s t y le 3 .
? s t y l e 3 r d f s : l a b e l ? a s t 3 . }
<entity>db p : a r c h i t e c t u r e ? s t y l e4 .
? s t y l e 4 r d f s : l a b e l ? a s t 4 . }
<entity>db p : a r c h i t e c t u r e ? a s t5 .
FILTE R NOT e x i s t s {?a s t 5 r d f s : l a b e l ? s t y l e 5 .} }
FILTE R ( r e g e x ( s t r (? a s t 1 ) , ” Ba r oq u e ”) | |
r e g e x ( s t r ( ? a s t 2 ) , ” Ba r oq u e ”) | | r e g e x ( s t r ( ? a s t 3 ) , ” Ba r oq u e ”) | |
r e g e x ( s t r ( ? a s t 4 ) , ” Ba r oq u e ”) | | r e g e x ( s t r ( ? a s t 5 ) , ” Ba r oq u e ”) )
FILTE R ( l a n g M a t c h e s ( la n g ( ? a s t 1 ) , ’ e n ’ ) && l a n g M a t c h e s ( l a n g ( ? a s t 2 ) , ’ en )
&& l a n g M a t c h e s ( l a n g ( ? a s t 3 ) , ’ en ) && l a n g M a t c h e s ( la n g ( ? a s t 4 ) , ’ en ) )
Listing 3. Example of querying DBpedia to detect buildings with Baroque
architectural style
In Table I we display the execution time in seconds of
transforming incoming annotations into the V4Link annotation
model. The modality that needs the most time to be mapped
and stored in the RDF triple store (we have use the free
version of GraphDB6) is the 3D model reconstruction, while
BIM information is handled more efficiently. This is mainly
because BIM does not require any additional mapping logic
in the form of inference/integration rules, while the 3D model
reconstruction assets serve as main resources that need to be
associated with all the available metadata.
Component Execution time in seconds
Text analysis 1.87
Aesthetics extraction 0.92
STBOL 1.72
3D Model Reconstruction 1.99
BIM 0.55
In Table II we present the mean execution time per scenario
execution. The calculation has been applied in a set of 681
Assets, where 163,966 triples exist. The conclusion of this
process is that execution time increases rapidly when querying
in Linked Data is applied or there is a large amount of triples.
The number of properties that appear in the queries seems
to also increase execution time, as in cases like architecture-
related requirements, there is a large number of properties and
execution time is also high.
Scenario Mean execution time in seconds
Proximity to POI 11.17
Architecture-related requirements 13.72
In this short paper, we described V4Link, a framework that
takes advantage of Linked Data capabilities to support designer
(such as architects and video game designers) to intelligent
discovery of assets. Data coming from multimodal components
is mapped to RDF graphs following the Web Annotation Data
Model and is coupled with popular Linked Data datasets,
namely DBpedia and LinkedGeoData. User-driven queries are
then executed to assist the design process. The evaluation
showed that the semantic representation, reasoning and storage
has low execution time, while the execution time of semantic
search is strongly affected by the usage of Linked Data.
Future work includes the detection of different Linked Data
datasets which can be used to support scenarios that are not
included in the current version of this framework. Additional
properties of the existing datasets may also be examined. The
scenarios may be updated accordingly to meet the needs of
the users.
This work was supported by the EC funded project
V4Design (H2020-779962)
[1] R. Sanderson, P. Ciccarese, and B. Young, “Web Annotation
Data Model,” W3C, pp. 1–56, 2016. [Online]. Available:
[2] K. N. Vavliakis, G. T. Karagiannis, and P. A. Mitkas, “Semantic web
in cultural heritage after 2020,” in Proceedings of the 11th International
Semantic Web Conference (ISWC), Boston, MA, USA, 2012, pp. 11–15.
[3] T. Kauppinen, J. V¨
ainen, and E. Hyv¨
onen, “Creating and using
geospatial ontology time series in a semantic cultural heritage portal,”
in European Semantic Web Conference. Springer, 2008, pp. 110–123.
[4] G. Schreiber, A. Amin, L. Aroyo, M. van Assem, V. de Boer, L. Hardman,
M. Hildebrand, B. Omelayenko, J. van Osenbruggen, A. Tordai et al.,
“Semantic annotation and search of cultural-heritage collections: The
multimedian e-culture demonstrator,Journal of Web Semantics, vol. 6,
no. 4, pp. 243–249, 2008.
[5] C. Van Aart, B. Wielinga, and W. R. Van Hage, “Mobile cultural heritage
guide: location-aware semantic search,” in International Conference on
Knowledge Engineering and Knowledge Management. Springer, 2010,
pp. 257–271.
[6] A. Baruzzo, P. Casoto, P. Challapalli, A. Dattolo, N. Pudota, and C. Tasso,
“Toward semantic digital libraries: Exploiting web2. 0 and semantic
services in cultural heritage,” Journal of Digital Information, vol. 10,
no. 6, p. 4, 2009.
[7] R. Guha, R. McCool, and E. Miller, “Semantic search,” in Proceedings of
the 12th international conference on World Wide Web, 2003, pp. 700–709.
[8] R. Volk, J. Stengel, and F. Schultmann, “Building information modeling
(bim) for existing buildings—literature review and future needs,Automa-
tion in construction, vol. 38, pp. 109–127, 2014.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
While BIM processes are established for new buildings, the majority of existing buildings is not maintained, refurbished or deconstructed with BIM yet. Promising benefits of efficient resource management motivate research to overcome uncertainties of building condition and deficient documentation prevalent in existing buildings. Due to rapid developments in BIM research, involved stakeholders demand a state-of-the-art overview of BIM implementation and research in existing buildings. This paper presents a review of over 180 recent publications on the topic. Results show scarce BIM implementation in existing buildings yet, due to challenges of (1) high modeling/conversion effort from captured building data into semantic BIM objects, (2) updating of information in BIM and (3) handling of uncertain data, objects and relations in BIM occurring in existing buildings. Despite fast developments and spreading standards, challenging research opportunities arise from process automation and BIM adaption to existing buildings' requirements.
Conference Paper
Full-text available
In this paper we explore the use of location aware mobile devices for searching and browsing a large number of general and cul- tural heritage information repositories. Based on GPS positioning we can determine a user's location and context, composed of physical nearby lo- cations, historic events that have taken place there, artworks that were created at or inspired by those locations and artists that have lived or worked there. Based on a geolocation, the user has three levels of refine- ment: pointing to a specific heading and selection and facets and sub- facets of cultural heritage objects. In our approach two types of knowl- edge are combined: general knowledge about geolocations an dp oints of interest and specialized knowledge about a particular domain, i.e. cul- tural heritage. We use a number of Linked Open Data sources an da number of general sources from the cultural heritage domain (including Art and Architecture Thesaurus, Union List of Artist Names) as well as data from several Dutch cultural institutions. We show thre ec oncrete scenarios where a tourist accesses localized information o nh is iPhone about the current environment, events, artworks or persons ,w hich are enriched by Linked Open Data sources. We show that Linked Ope nD ata sources in isolation are currently too limited to provide interesting se- mantic information but combined with each other and with a number of other sources a really informative location-based service can be created.
Conference Paper
Full-text available
Content annotations in semantic cultural heritage portals commonly make spatiotemporal references to historical regions and places using names whose meanings are different in different times. For example, historical administrational regions such as countries, municipalities, and cities have been renamed, merged together, split into parts, and annexed or moved to and from other regions. Even if the names of the regions remain the same (e.g., "Germany"), the underlying regions and their relationships to other regions may change (e.g., the regional borders of "Germany" at different times). As a result, representing and finding the right ontological meanings for historical geographical names on the semantic web creates severe problems both when annotating contents and during information retrieval. This paper presents a model for representing the meaning of changing geospatial resources. Our aim is to enable precise annotation with temporal geospatial resources and to enable semantic search and browsing using related names from other historical time periods. A simple model and metadata schema is presented for representing and maintaining geospatial changes from which an explicit time series of temporal part-of ontologies can be created automatically. The model has been applied successfully to represent the complete change history of municipalities in Finland during 1865-2007. The resulting ontology time series is used in the semantic cultural heritage portal CULTURESAMPO to support faceted semantic search of contents and to visualize historical regions on overlaying maps originating from different historical eras.
Developing and maintaining a digital library requires substantial investments that are not simply a matter of technological decisions, but include also organizational issues (user roles, workflows, types of contents, etc.). These issues are often handled by approaches based on a physical perspective that treats the stored information either in terms of data formats or physical space needed to archive them. All these perspectives completely ignore the semantic aspects of the digital contents. In this paper, we address such a semantic perspective. More specifically, we propose a service-oriented architecture that explicitly includes a semantic layer which provides primitive services to the applications built on top of the digital library. As part of this layer, a specific component is described: the PIRATES framework. This module assists end users to complete several tasks concerning the retrieval of the most relevant content with respect to a description of their information needs (a search query, a user profile, etc.). Techniques of user modeling, adaptive personalization, and knowledge representation are exploited to build the PIRATES services in order to fill the gap existing between traditional and semantic digital libraries.
In this article we describe a Semantic Web application for semantic annotation and search in large virtual collections of cultural-heritage objects, indexed with multiple vocabularies. During the annotation phase we harvest, enrich and align collection metadata and vocabularies. The semantic-search facilities support keyword-based queries of the graph (currently 20 M triples), resulting in semantically grouped result clusters, all representing potential semantic matches of the original query. We show two sample search scenario’s. The annotation and search software is open source and is already being used by third parties. All software is based on established Web standards, in particular HTML/XML, CSS, RDF/OWL, SPARQL and JavaScript.
Web Annotation Data Model
  • R Sanderson
  • P Ciccarese
  • B Young
R. Sanderson, P. Ciccarese, and B. Young, "Web Annotation Data Model," W3C, pp. 1-56, 2016. [Online]. Available: