Conference PaperPDF Available

Abstract and Figures

This demonstration paper describes the mobile application developed by the EAGLE project to increase the use and visibility of its epigraphic material. The EAGLE project (European network of Ancient Greek and Latin Epigraphy) is gathering a comprehensive collection of inscriptions (about 80 % of the surviving material) and making it accessible through a user-friendly portal, which supports searching and browsing of the epigraphic material. In order to increase the usefulness and visibility of its content, EAGLE has developed also a mobile application to enable tourists and scholars to obtain detailed information about the inscriptions they are looking at by taking pictures with their smartphones and sending them to the EAGLE portal for recognition. In this demonstration paper we describe the EAGLE mobile application and give an outline of its features and its architecture.
Content may be subject to copyright.
Searching the EAGLE Epigraphic Material through
Image Recognition via a Mobile Device
Paolo Bolettieri1, Vittore Casarosa1, Fabrizio Falchi1, Lucia Vadicamo1,
Philippe Martineau2, Silvia Orlandi3, Raffaella Santucci3
1CNR-ISTI, 2Eureva, 3Università di Roma La Sapienza
Abstract. This demonstration paper describes the mobile application developed
by the EAGLE project to increase the use and visibility of its epigraphic material.
The EAGLE project (European network of Ancient Greek and Latin Epigraphy)
is gathering a comprehensive collection of inscriptions (about 80% of the surviv-
ing material) and making it accessible through a user-friendly portal, which sup-
ports searching and browsing of the epigraphic material. In order to increase the
usefulness and visibility of its content, EAGLE has developed also a mobile ap-
plication to enable tourists and scholars to obtain detailed information about the
inscriptions they are looking at by taking pictures with their smartphones and
sending them to the EAGLE portal for recognition. In this demonstration paper
we describe the EAGLE mobile application and give an outline of its features
and its architecture.
Keywords: mobile application, image recognition, similarity search, epigraphy,
Latin and Greek inscriptions
1 The EAGLE Project
One of the main motivations of the project EAGLE (Europeana network of Ancient
Greek and Latin Epigraphy [1], a Best Practice Network partially funded by the Euro-
pean Commission) was to collect in a single repository information about the thousands
of Greek and Latin inscriptions presently scattered in a number of different institutions
(museums and universities) across all Europe. The collected information, about 1,5 mil-
lion digital objects (texts and images), representing approximately 80% of the total
amount of classified inscriptions in the Mediterranean area, is being ingested into Eu-
ropeana and is also made available to the scholarly community and to the general pub-
lic, for research and cultural dissemination, through a user-friendly portal supporting
advanced query and search capabilities.
In addition to the query capabilities (full text search a la Google, fielded search,
faceted search and filtering), the EAGLE portal supports two applications intended to
make the fruition of the epigraphic material easier and more useful. A Story Telling
application provides tools to assemble epigraphy-based narratives to be made available
at the EAGLE portal, intended for the fruition of the epigraphic material by less knowl-
edgeable users or young students. A Flagship Mobile Application (FMA) enables a user
to get information about one visible inscription by taking a picture with a mobile device,
and sending it to the EAGLE portal for recognition. This demo will show the EAGLE
Flagship Mobile Application (presently implemented on Android) and the next sections
will briefly describe the functionality and the architecture of the FMA.
2 The Flagship Mobile Application
The FMA enables a user to get information about one visible inscription by taking a
picture with a mobile device, and sending it to the EAGLE portal, specifying the recog-
nition mode. In “Similarity Search Mode” the result is a list of inscriptions (just thumb-
nails and some summary information) ranked in order of similarity to the image sent to
the EAGLE server; by clicking on one of the thumbnails the user will receive all the
information associated with that inscription. In Exact Match Mode” the result is all the
information associated with the image, if recognized, or a message saying that the im-
age was not recognized.
The Graphical User Interface (GUI) of the FMA, available on the touch screen of
the mobile device gives access to the functions listed below. The user can navigate
through the different functions with tabs, and at any moment has access to the initial
page.
Search EAGLE content using image recognition in Similarity Search mode
Search EAGLE content using image recognition in Exact Match mode
Search EAGLE content using text search
Login to the mobile application using an account already existing at the EAGLE
portal
For logged-in users, annotate and save queries and their results
For logged-in users, annotate and save pictures taken with the mobile device
For logged-in user, access and review the navigation history.
The mobile application communicates (through the Internet) with the Flagship Mo-
bile Application (FMA) server, which in turn communicates with the EAGLE server
using the specific APIs supporting the mobile application. Figure 1 shows the main
functionality blocks of the EAGLE portal and the communication APIs between the
FMA server and the EAGLE server. Complete details of the architecture and the mobile
application can be found in [2].
The Image Recognizer (middle block on the right in the EAGLE server) has three
main functions: (i) Image Feature Extractor, (ii) Image Indexer and support of Similar-
ity Search Mode, (iii) Support of Exact Match Mode.
Figure 1 Basic Architecture of the EAGLE server and the FMA server
2.1 Image Feature Extractor
The Image Feature Extractor analyses the visual content of the EAGLE images and
captures certain local visual properties of an image (features). Local features are low
level descriptions of Keypoints (or salient points), which are interest points in an image,
whose description is invariant to scale and orientation. The result of extraction of visual
features is a mathematical description of the image visual content that can be used to
compare different images, judge their similarity, and identify common content. The
Image Recognizer in EAGLE has a multi-threaded architecture for fast extraction of
features and for taking advantage of multicore processors. It has a plug-in architecture,
so that it is easy to add or delete the mathematical libraries supporting the many differ-
ent algorithms for the extraction of local visual features, such as SIFT, SURF, ORB,
etc. and their aggregations, such as BoF and VLAD [4].
2.2 Indexer and support of Similarity Search and Exact Match Modes
The Image Indexer leverages the functionality of the Melampo CBIR System.
Melampo stands for Multimedia Enhancement for Lucene to Advanced Metric PivOt-
ing [3]. It is an open source Content Based Image Retrieval (CBIR) library developed
at CNR-ISTI that allows efficient comparison of images by visual similarity through
the use of local features.
After the visual feature extraction, the local features are encoded using an approach
called “Bag of Features”, where a vocabulary of visual words is created starting from
all the local descriptors of the whole dataset. The set of all the local descriptors of all
the images is divided into a number of clusters (depending on the algorithms used, this
number can go from a few hundreds to tens of thousands) and a textual tag is assigned
to each cluster (usually in a random fashion). The set of all the textual tags becomes the
“vocabulary” of visual words related to the whole set of images. At this point each
image can be described by a set of “words” in this vocabulary, corresponding to the
clusters containing the visual features of the image.
The support of Similarity Search Mode is based on the use of the Lucene search
engine. Each image is represented by a set of words (the textual tags of the visual vo-
cabulary), and Lucene builds the index of those words. At query time, the query image
is transformed into a set of words, and then Lucene performs a similarity search, re-
turning a list of images ranked according to the similarity with the query image.
The support of Exact Match Mode is based on a set of classifiers, each one recog-
nizing a specific epigraph. The construction of the classifiers is done off-line, selecting
from the complete database those epigraphies for which several images are available.
The set of images representing the same epigraph is the training set used for building
the classifier of that epigraph. At query time, the recognizer performs a similarity search
for the image to be recognized and then takes from the result list the first k results for
which there is also a classifier. The recognizer uses the RANSAC algorithm to perform
geometry consistency checks [5] and assign a score to each class. We decided to assign
to each class the highest matching score (i.e., percentage of inliers after the RANSAC)
between the query image and all the image in the classifier. If the score is above a given
threshold, the image is recognized.
3 Results
The Flagship Mobile Application has been tested on a preliminary database of about
17 thousand images for Similarity Search and 70 training sets for Exact Match, using
different vocabulary size and visual features representation. Presently, the best results
have been obtained using VLAD for visual features aggregations, with a codebook size
of 256.
4 Bibliography
1. The EAGLE Project http://www.eagle-network.eu/
2. The EAGLE Project, Deliverable D4.1 Aggregation and Image Retrieval system (AIM)
Infrastructure Specification
3. Gennaro C., Amato G., Bolettieri P., Savino P.. An approach to content-based image re-
trieval based on the Lucene search engine library. In: ECDL 2010 - Research and Advanced
Technology for Digital Libraries. 14th European Conference (Glasgow (UK), , pp. 55 - 66.
4. Jégou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., & Schmid, C. (2012). Aggregat-
ing local image descriptors into compact codes. Pattern Analysis and Machine Intelligence,
IEEE Transactions on, 34(9), 1704-1716.
5. Amato, Giuseppe, Fabrizio Falchi, and Claudio Gennaro. "Geometric consistency checks
for kNN based image classification relying on local features." In Proceedings of the Fourth
International Conference on SImilarity Search and APplications, pp. 81-88. ACM, 2011.
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
Applications of image content recognition, as for instance landmark recognition, can be obtained by using techniques of kNN classifications based on the use of local image features, such as SIFT or SURF. Quality of image classification can be improved by defining geometric consistency check rules based on space transformations of the scene depicted in images. However, this prevents the use of state of the art access methods for similarity searching and sequential scan of the images in the training sets has to be executed in order to perform classification. In this paper we propose a technique that allows one to use access methods for similarity searching, such as those exploiting metric space properties, in order to perform kNN classification with geometric consistency checks. We will see that the proposed approach, in addition to offer an obvious efficiency improvement, surprisingly offers also an improvement of the effectiveness of the classification.
Article
Full-text available
This paper addresses the problem of large-scale image search. Three constraints have to be taken into account: search accuracy, efficiency, and memory usage. We first present and evaluate different ways of aggregating local image descriptors into a vector and show that the Fisher kernel achieves better performance than the reference bag-of-visual words approach for any given vector dimension. We then jointly optimize dimensionality reduction and indexing in order to obtain a precise vector comparison as well as a compact representation. The evaluation shows that the image representation can be reduced to a few dozen bytes while preserving high accuracy. Searching a 100 million image dataset takes about 250 ms on one processor core.
Conference Paper
Content-based image retrieval is becoming a popular way for searching digital libraries as the amount of available multimedia data increases. However, the cost of developing from scratch a robust and reliable system with content-based image retrieval facilities for large databases is quite prohibitive. In this paper, we propose to exploit an approach to perform approximate similarity search in metric spaces developed by [3,6]. The idea at the basis of these techniques is that when two objects are very close one to each other they ’see’ the world around them in the same way. Accordingly, we can use a measure of dissimilarity between the views of the world at different objects, in place of the distance function of the underlying metric space. To employ this idea the low level image features (such as colors and textures) are converted into a textual form and are indexed into the inverted index by means of the Lucene search engine library. The conversion of the features in textual form allows us to employ the Lucene’s off-the-shelf indexing and searching abilities with a little implementation effort. In this way, we are able to set up a robust information retrieval system that combines full-text search with contentbased image retrieval capabilities.
Deliverable D4.1 - Aggregation and Image Retrieval system (AIM) Infrastructure Specification
  • Eagle The
  • Project
The EAGLE Project, Deliverable D4.1 -Aggregation and Image Retrieval system (AIM) Infrastructure Specification
Aggregating local image descriptors into compact codes. Pattern Analysis and Machine Intelligence
  • H Jégou
  • F Perronnin
  • M Douze
  • J Sanchez
  • P Perez
  • C Schmid
Jégou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., & Schmid, C. (2012). Aggregating local image descriptors into compact codes. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(9), 1704-1716.