MOCHA 2017 as a Challenge for Virtuoso
Mirko Spasi´c1,2and Milos Jovanovik1,3
1OpenLink Software, United Kingdom
2Faculty of Mathematics, University of Belgrade, Serbia
3Faculty of Computer Science and Engineering,
Ss. Cyril and Methodius University in Skopje, Macedonia
Abstract. The Mighty Storage Challenge (MOCHA) aims to test the
performance of solutions for SPARQL processing, in several aspects rele-
vant for modern Linked Data applications. Virtuoso, by OpenLink Soft-
ware, is a modern enterprise-grade solution for data access, integration,
and relational database management, which provides a scalable RDF
Quad Store. In this paper, we present a short overview of Virtuoso with
a focus on RDF triple storage and SPARQL query execution. Further-
more, we showcase the ﬁnal results of the MOCHA 2017 challenge and its
tasks, along with a comparison between the performance of our system
and the other participating systems.
Keywords: Virtuoso, Social Network Benchmark, Mighty Storage Chal-
lenge, Benchmarks, Data Storage, Linked Data, RDF, SPARQL
Triple stores are the heart of a growing number of Linked Data applications. This
uncovers a growing need for representative benchmarks which will fairly summa-
rize their strengths and weaknesses , allowing stakeholders to choose between
technologies from diﬀerent vendors according to their needs and use-cases. The
HOBBIT project4aims to push the development of Big Linked Data process-
ing solutions by providing a family of industry-relevant benchmarks through a
generic evaluation platform – the HOBBIT Platform . In the scope of the
project, several challenges are being organized, with the goal of reaching system
providers, familiarizing them with the benchmarks of their interest, as well as
the platform itself. The Mighty Storage Challenge (MOCHA 2017)5is one of
these challenges: it aims to test the performance of systems capable of answer-
ing SPARQL SELECT queries and processing INSERT queries. Its goal is to
provide objective measures for how well current systems perform on real tasks
of industrial relevance and detect bottlenecks of existing systems to further their
development towards practical usage. The challenge was accepted and presented
2 Spasi´c and Jovanovik
in the Extended Semantic Web Conference (ESWC)6in 2017, held in Portoroz,
Slovenia. Even though four tasks were initially planned, MOCHA 2017 consisted
of three tasks in the end. OpenLink Software7, with our RDF Quad Store – Vir-
tuoso 8.0 Commercial Edition (beta release) – participated in all of three:
–Task 1: RDF Data Ingestion,
–Task 2: Data Storage, and
–Task 4: Browsing.
In Section 2, we will brieﬂy present our system, Virtuoso, putting our focus
on its quad storage, represented as a relational table, and its translation engine
for converting SPARQL queries to SQL. We will describe all preparatory actions
requested by the challenge organizers which the system had to fulﬁll in Section
3. After that, in Section 4, we will present the evaluation results of our system
achieved during the challenge, along with a comparison with the outcomes of the
other participants. Finally, Section 5 concludes the paper, and contains guidelines
for future work and further improvement of our system.
2 Virtuoso Universal Server
Virtuoso8is a modern enterprise-grade solution for data access, integration,
and relational database management. It is a database engine hybrid that com-
bines the functionality of a traditional relational database management sys-
tem (RDBMS), object-relational database (ORDBMS), virtual database, RDF,
XML, free-text, web application server and ﬁle server functionality in a single sys-
tem. It operates with SQL tables and/or RDF based property/predicate graphs.
Virtuoso was initially developed as a row-wise transaction oriented RDBMS with
SQL federation, i.e. as a multi-protocol server providing ODBC and JDBC ac-
cess to relational data stored either within Virtuoso itself or any combination
of external relational databases. Besides catering to SQL clients, Virtuoso has
a built-in HTTP server providing a DAV repository, SOAP and WS* protocol
end-points and dynamic web pages in a variety of scripting languages. It was
subsequently re-targeted as an RDF graph store with built-in SPARQL and
inference [4, 5]. Recently, the product has been revised to take advantage of
column-wise compressed storage and vectored execution .
The largest Virtuoso applications are in the RDF domain, with terabytes of
RDF triples which do not ﬁt into main memory. The excellent space eﬃciency of
column-wise compression was the greatest incentive for the column store transi-
tion . Additionally, this also makes Virtuoso an option for relational analytics.
Finally, combining a schemaless data model with analytics performance is at-
tractive for data integration in places with high schema volatility. Virtuoso has
a shared cluster capability for scale-out. This is mostly used for large RDF de-
MOCHA 2017 as a Challenge for Virtuoso 3
2.1 Triple Storage
The storage solution in Virtuoso is fairly conventional: a single table of four
columns, named RDF QUAD, holds one quad, i.e. a triple plus graph, per row.
The columns are Gfor graph, Pfor predicate, Sfor subject and Ofor object. P,
Gand Sare IRI IDs, for which Virtuoso has a custom data type, distinguishable
at runtime from integer, even though internally this is a 32 or 64-bit integer. Since
Ois a primary key part, it is not desired to have long Ovalues repeated in the
index. Hence, Os of string type which are longer than 12 characters are assigned
a unique ID and this ID is stored as the Oof the quad table, while the mapping
is stored in the RDF IRI and RDF PREFIX tables . By default, and with the
idea of faster execution, the table is represented as ﬁve covering indices, P SOG,
P OS G,SP ,GS, and O P . In the ﬁrst one, the quads are sorted primarily by
predicate, then subject and object, and ﬁnally by graph. The structures of the
other indices are analog to this one.
The compression is implemented at two levels. First, within each database page,
Virtuoso stores distinct values only once and eliminates common preﬁxes of
strings. Without key compression, there are 75 bytes per triple with a billion-
triple LUBM9dataset (LUBM scale 8000). With compression, only 35 bytes per
triple are present. Thus, when using 32-bit IRI IDs, key compression doubles
the working set while sacriﬁcing no random access performance. The beneﬁts of
compression are even better when using 64-bit IRI IDs .
The second stage of compression involves applying gzip to database pages,
which reduces their size to a third, even after key compression. This is expected,
since indices are repetitive by nature, even if the repeating parts are shortened
by key compression .
2.3 Translation of SPARQL Queries to SQL
Internally, SPARQL queries are translated into SQL at the time of query parsing.
If all triples are in one table, the translation is straightforward. In the next
paragraph, we give a couple of simple SPARQL queries, and their simpliﬁed
All triple patterns from the SPARQL query should be translated to SQL as a
self-join of the RDF QUAD table, with conditions if there are common subjects,
predicates and/or objects . For example, if a SPARQL query asks for ﬁrst and
last names of 10 people, as shown in the example on Figure 1, its SQL translation
will be similar to the query given at Figure 2. The functions __i2idn,__bft
and __ro2sq are used for translation of RDF IRIs to the internal datatypes
mentioned in the Subsection 2.1, and vice versa.
A SPARQL union becomes an SQL union (Figures 3 and 4) and optional
becomes a left outer join (Figures 5 and 6), while SPARQL group by,having,
4 Spasi´c and Jovanovik
Fig. 1: SPARQL Query 1 Fig. 2: SQL Translation of SPARQL Query 1
order by and aggregate functions are translated to their SQL corresponding
counterparts. Figure 4 shows an optimization trick: both members of a union
have limit clauses, as well as the main select, providing that both parts of the
query will not ﬁnd more than 10 results.
Fig. 3: SPARQL Query 2 Fig. 4: SQL Translation of SPARQL Query 2
In conclusion, a SPARQL query with ntriple patterns will result with n−1
self-joins. Thus, the correct join order and join type decisions are diﬃcult to
make given only the table and column cardinalities for the RDF triple or quad
table. Histograms for ranges of P,G,O, and Sare also not useful . The
solution is to go look at the data itself when compiling the query, i.e. do data
MOCHA 2017 as a Challenge for Virtuoso 5
Fig. 5: SPARQL Query 3 Fig. 6: SQL Translation of SPARQL Query 3
3 Challenge Prerequisites for Participation
In order to be a part of the challenge, the organizers proposed a set of require-
ments that participants had to conform to. The participants had to provide:
–A storage system that processes SPARQL INSERT queries
–A storage solution that can process SPARQL SELECT queries
–A solution as a Docker image that abides by the technical speciﬁcations, i.e.
the MOCHA API
Virtuoso has build-in SPARQL support, so we only had to pack it as a
Docker image and develop a System Adapter, a component of the HOBBIT
platform10 which implements the requested API and enables communication
between the benchmark and the Virtuoso instance. We developed an instance
of the System Adapter for the commercial version of Virtuoso 8.0, which shares
the same Docker container with it. Its code is publicly available on GitHub11.
After this component initializes itself, it starts receiving data from the Data
Generator, i.e. the ﬁles representing the benchmark dataset. When all ﬁles are
accepted, indicated by a signal from the Data Generator (the other part of
the platform that is in charge for creating the dataset for the benchmark), the
System Adapter starts loading the dataset into the Virtuoso instance. Upon
completion, it sends a signal to the other components indicating it is ready to
start answering the SPARQL queries, which are then sent by the Task Generator,
a component which creates the tasks, i.e. the SELECT and INSERT queries. All
6 Spasi´c and Jovanovik
accepted queries are then executed against our system, and their answers are
sent to the Evaluation Storage, for validation against the expected answers and
for measuring the achieved eﬃciency of the system.
In this section, we present the oﬃcial results of the challenge for all its tasks.
4.1 Task 1: RDF Data Ingestion
The aim of this task is to measure the performance of SPARQL query processing
systems when faced with streams of data from industrial machinery in terms of
eﬃciency and completeness. This benchmark, called ODIN (StOrage and Data
Insertion beNchmark), increases the size and velocity of RDF data used in order
to evaluate how well can a system store streaming RDF data obtained from the
industry. The data is generated from one or multiple resources in parallel and is
inserted using SPARQL INSERT queries. At some points in time, the SPARQL
SELECT queries check the triples that are actually inserted .
This task has three main KPIs:
–Triples per Second: For each stream, a fraction of the total number of triples
that were inserted during that stream divided by the total time needed for
those triples to be inserted.
–Average Answer Time: A delay between the time stamp that the SELECT
query has been executed and the time stamp that the results are send to the
–Correctness: A recall of each SELECT query by comparing the expected and
Micro-Average-Recall Micro-Average-Precision Micro-Average-F-Measure Macro-Average-Recall Macro-Average-Precision Macro-Average-F-Measure
Baseline for Task 1
Virtuoso Commercial 8.0
Fig. 7: Micro-Average-Recall, Micro-Average-Precision, Micro-Average-F-
Measure, Macro-Average-Recall, Macro-Average-Precision, Macro-Average-F-
Measure for Task 1 of MOCHA 2017.
MOCHA 2017 as a Challenge for Virtuoso 7
20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380
Baseline for Task 1 ONTOS Quad Virtuoso Commercial 8.0
Recall for Task 1
Fig. 8: Recall for Task 1 of MOCHA 2017.
20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380
Baseline for Task 1 ONTOS Quad Virtuoso Commercial 8.0
Precision for Task 1
Fig. 9: Precision for Task 1 of MOCHA 2017.
8 Spasi´c and Jovanovik
Baseline for Task 1 (25th task) ONTOS Quad (6th task)
Virtuoso Commercial 8.0 (358th task) MOCHA Baseline (105th task)
Fig. 10: Maximum Triples-per-Second for Task 1 of MOCHA 2017.
Results: Here, we give the oﬃcial results of our system, Virtuoso 8.0 Com-
mercial Edition (beta release), against ODIN, achieved during the challenge and
published by the organizers of the challenge. The task organizers at MOCHA
2017 speciﬁed the benchmark parameters for the actual challenge run, in order to
achieve the desired size and velocity of the RDF data. The values of the param-
eters were: number of insert queries per stream = 100, population of generated
data = 10,000, number of data generators - agents = 4.
Our system, Virtuoso Commercial 8.0, had by far the best performance com-
pared to the other systems, in terms of Macro and Micro-Average Precision,
Recall, and F-measure (Figure 7). By observing Figures 8 and 9, it is obvious
that our system was able to store and retrieve much more triples throughout the
whole benchmark, than the other systems. In terms of maximum Triples-per-
Second, based on Figure 10, our system has just conﬁrmed its convincing overall
victory in this task, with a one order of magnitude better score. This results
were announced by the organizers – our system had a best overall performance
in terms of data ingestion and retrieval.
4.2 Task 2: Data Storage
The goal of this task is to measure how data storage solutions perform with
interactive, simple, read, SPARQL queries as well as complex ones, accompanied
with a high insert data rate via SPARQL UPDATE queries, in order to mimic
real use-cases where READ and WRITE operations are bundled together. This
task also tests systems for their bulk load capabilities .
The main KPIs of this task are:
–Bulk Loading Time: The total time in milliseconds needed for the initial
bulk loading of the dataset.
–Throughput: The average number of tasks executed per second.
MOCHA 2017 as a Challenge for Virtuoso 9
–Correctness: The number of SPARQL SELECT queries whose result set is
diﬀerent from the result set obtained from the triple store used as a gold
Fig. 11: Loading Time for Task 2 of
Fig. 12: Throughput for Task 2 of
Fig. 13: Long Queries for Task 2 of MOCHA 2017.
Fig. 14: Short Queries and Updates for Task 2 of MOCHA 2017.
10 Spasi´c and Jovanovik
Results: Based on the results from the KPIs, shown in Figures 11, 12, 13
and 14, the winning system for the task was Virtuoso 7.2 Open-Source Edition
by OpenLink Software, that was used as a baseline system for all tasks in the
challenge. Our system, Virtuoso 8.0 Commercial Edition, was slightly slower,
while the third system was not able to ﬁnish the experiment in the requested
time, i.e. it exhibited a timeout, thus its scores are not present at the ﬁgures.
4.3 Task 4: Browsing
The task on faceted browsing checks existing solutions for their capabilities of
enabling faceted browsing through large-scale RDF datasets, that is, it analyses
their eﬃciency in navigating through large datasets, where the navigation is
driven by intelligent iterative restrictions. The goal of the task is to measure the
performance relative to dataset characteristics, such as overall size and graph
The evaluation is based on the following performance KPIs:
–Throughput: The time required by the system is measured for the two tasks
– facet count and instance retrieval – separately. The results are returned in
a score function computing number of returned queries per second.
–Correctness: The facet counts are being checked for correctness. For each
facet count, the distance of the returned count to the correct count in terms
of absolute and relative value is recorded. For each instance retrieval the
benchmark collects the true positives, the false positives and false negatives
to compute an overall precision, recall and F1-score.
Results: Similar to the ﬁrst task, the only two systems that managed to ﬁnish
the task within the requested time slot are shown on the Figures 15, 16, 17a and
17b, representing the main KPIs of the Faceted Browsing Benchmark. Based on
that, the organizers announced a tie between our system and the baseline system.
The Open-Source edition of Virtuoso was slightly faster, but the Commercial
edition performed better on the correctness of the facet counts queries.
4.4 Overall Winner
As a summary, our system had a signiﬁcant victory in Task 1; the baseline system
was slightly better in Task 2; and there was a tie in Task 4. Based on the results
in each task separately and the overall results, during the closing ceremony of
the ESWC 2017 conference, the challenge organizers declared our system as the
overall winner of the MOCHA 2017.
5 Conclusion and Future Work
This paper should be considered as an extended participant paper of MOCHA
2017, a challenge included in the Challenges Track of ESWC 2017, intended to
MOCHA 2017 as a Challenge for Virtuoso 11
Fig. 15: Instance Retrieval: Correctness for Task 4 of MOCHA 2017.
Fig. 16: Instance Retrieval: Speed for Task 4 of MOCHA 2017.
test RDF storage and SPARQL systems for the following tasks: RDF Data Inges-
tion, Data Storage and Browsing. Thus, a short overview of Virtuoso Universal
Server has been presented, with a focus on its RDF storage engine and the inter-
nal SPARQL to SQL translation. The evaluation part of the paper contains the
oﬃcial measurements from the challenge and its tasks. This section represents
an excellent guideline as to where our Virtuoso optimizer should be improved.
12 Spasi´c and Jovanovik
(a) Correctness (b) Speed
Fig. 17: Facet Counts for Task 4 of MOCHA 2017.
As future work, a further evaluation has been planned against newer versions
of the challenge benchmarks. For example, in Task 2, real-world workloads will be
used, consisting of speciﬁed query mixes, where reads and updates are bundled
together, and queries are run concurrently. Virtuoso will be tested with more
dataset sizes and especially larger datasets, stressing its scalability. For this
purpose, the HOBBIT platform will be used. We foresee improvements of the
query optimizer, driven by the current evaluation.
Acknowledgments. This work has been supported by the H2020 project HOB-
BIT (GA no. 688227).
1. Morsey M., Lehmann J., Auer S., Ngonga Ngomo AC.: DBpedia SPARQL Bench-
mark Performance Assessment with Real Queries on Real Data. In: Aroyo L. et
al. (eds.) The Semantic Web ISWC 2011. ISWC 2011. Lecture Notes in Computer
Science, vol 7031. Springer, Berlin, Heidelberg (2011)
2. Ngonga Ngomo AC., R¨oder M.: HOBBIT: Holistic Benchmarking for Big Linked
Data. In: ERCIM News 2016 - 105 (2016)
3. May, P., Ehrlich, H.C., Steinke, T.: ZIB Structure Prediction Pipeline: Composing
a Complex Biological Workﬂow through Web Services. In: Nagel, W.E., Walter,
W.V., Lehner, W. (eds.) Euro-Par 2006. LNCS, vol. 4128, pp. 1148–1158. Springer,
4. Erling O., Mikhailov I.: RDF Support in the Virtuoso DBMS. In: Pellegrini T. et
al. (eds.) Networked Knowledge - Networked Media: Integrating Knowledge Man-
agement, New Media Technologies and Semantic Systems 2009., pp. 7–24. Springer
Berlin Heidelberg (2009)
5. Erling O., Mikhailov I.: Virtuoso: RDF Support in a Native RDBMS. In: de Virgilio
R. at al. (eds.) Semantic Web Information Management: A Model-Based Perspective
2010., pp. 501–519. Springer Berlin Heidelberg (2010)
6. Erling O.: Virtuoso, a Hybrid RDBMS/Graph Column Store.
MOCHA 2017 as a Challenge for Virtuoso 13
7. Spasi´c M., Jovanovik M., Prat-P´erez A.: An RDF Dataset Generator for the Social
Network Benchmark with Real-World Coherence. In Proceedings of the Workshop
on Benchmarking Linked Data (BLINK 2016), pages 18-25. (2016)
8. Kaufmann M., Social Network Benchmark Interactive Workload Full Disclosure
9. Georgala K., Data Extraction Benchmark for Sensor Data. https:
10. Jovanovik M., Spasi´c M., First Version of the Data Storage Benchmark.
11. Petzka H., First Version of the Faceted Browsing Benchmark. https: