Conference PaperPDF Available

Unit Tests for Ontologies


Abstract and Figures

In software engineering, the notion of unit testing was successfully introduced and applied. Unit tests are easy manageable tests for small parts of a program – single units. They proved especially useful to capture unwanted changes and side effects during the maintenance of a program, and they grow with the evolution of the program. Ontologies behave quite differently than program units. As there is no information hiding in ontology engineering, and thus no black box components, at first the idea of unit testing for ontologies seems not applicable. In this paper we motivate the need for unit testing, describe the adaptation to the unit testing approach, and give use cases and examples.
Content may be subject to copyright.
Unit tests for ontologies
Denny Vrandeˇci´c1, Aldo Gangemi2
Institute AIFB, University of Karlsruhe, Germany
LOA Laboratory of Applied Ontology, Rome, Italy
Abstract. In software engineering, the notion of unit testing was suc-
cessfully introduced and applied. Unit tests are easy manageable tests
for small parts of a program – single units. They proved especially useful
to capture unwanted changes and side effects during the maintenance of
a program, and they grow with the evolution of the program.
Ontologies behave quite differently than program units. As there is no
information hiding in ontology engineering, and thus no black box com-
ponents, at first the idea of unit testing for ontologies seems not appli-
cable. In this paper we motivate the need for unit testing, describe the
adaptation to the unit testing approach, and give use cases and examples.
1 Introduction
In software engineering, the idea of unit testing [1] was introduced to counter
the complexities of modern software engineering efforts. Unit tests are meant
to facilitate the development of program modules or units, and to ensure the
interplay of such units in the combined system. It results in more loosely coupled
code, that is easier to refactor and simpler to integrate, and that has a formalized
documentation (although not necessarily complete). Unit tests can be added
incrementally during the maintenance of a piece of software, in order to not
accidentally stumble upon and old bug and hunt it down repeatedly.
Unit tests are not complete test suites: there are several types of errors that
unit tests will not catch, including errors that result from the integration of the
units to the complete system, performance problems, and, naturally, errors that
were not expected when writing the unit tests.
Unit tests in software engineering became popularized with the object ori-
ented language Smalltalk, and still to this today remain focused on languages
with strong possibilities to create smaller units of code. They are based on several
decomposition techniques, most important of all information hiding.
Ontologies are different. As of now, no form of information hiding or interfaces
are available – and it remains an open research issue in the field of ontology
modularization how this will be taken care of.
In this paper we will take a look at the benefits of unit testing applied to
ontologies, i.e. their possibilities to facilitate regression tests, to provide a test
framework that can grow incrementally during the maintenance and evolution
phase of the ontology, and that is reasonably simple to use. In order for the unit
testing for ontologies to be useful, they need to be reasonably easy to use and
mantain. This will depend heavily on the given implementation (which is un-
derway). The task of this paper is to investigate different ideas that are inspired
by the idea of unit testing, and to work out the intuitions of how these ideas
could be used in the context of the ontology lifecycle. Especially in the enterprise
application of ontologies, some easy to use form of ontology evaluation will be
required in order to let ontology based technologies become more widespread.
We will show a number of ideas and examples of how this goal can be achieved.
The paper will first show a typical use case, as encountered in a project
setting in Section 2. We will then discuss five different approaches, that all are
inspired by the unit testing frameworks in software engineering: first we look
at the idea of design by contract, i.e. of stating what statements should and
should not derive from an ontology being developed or maintained, either as
explicit ontology statements or as competency questions using a query language
(Sections 3 and 4). Then we investigate the relationship of heavy- to lightweight
ontologies, and how they can interplay with regards to ontology evaluation in
Section 5. Autoepistemic operators lend themselves also to be used in the testing
of ontologies, especially with regards to their (relative) completeness, since they
are a great way to formalize the introspection of ontologies (Section 6). We
also regard a common error in ontology modelling with description logics based
language, and try to turn this error into our favour in Section 7, before we discuss
related work and give an outlook on possible further work and open issues.
For this work, the term ontologies refers to web ontologies as defined by
the OWL DL standard [15]. This means that the ontologies are a variant based
on the description logics SH OIN (D), and, especially, that ontologies mean to
encompass both the so called TBox, where the vocabulary of the ontology is
defined (which some call the whole ontology), and the ABox, where facts using
the vocabulary defined are stated (which some call the knowledge base).
2 Motivation
In the SEKT project1, one of the case studies aims at providing an intelligent
FAQ system to help newly appointed judges in Spain [2]. The system depends on
an ontology for finding the best answers and to find references to existing cases
in order to provide the judge with further background information. The applied
ontology is built and maintained by legal experts with almost no experience in
formal knowledge representation [4].
As the ontology evolved and got refined (and thus changed), the legal experts
noticed that some of their changes had undesired side effects. To give a simplified
example, consider the class hierarchy depicted in Figure 1. Let’s take for granted
that this ontology has been used for a while already, before someone notices that
not every academic needs necessarily be a member of an university. So Academic
University member
University member
Fig. 1. Example class hierarchy
becomes a direct subclass of Person, instead of University member. But due to
this change, also Professor is no subclass of University member any more (a
change that maybe was hidden from the ontology engineer, as the ontology
development environment may not have displayed the subclasses of Academic).
The resulting ontology remains perfectly satisfiable. But a tool, that, for
example, creates a web page for all members of the university may now skip the
professors, since they are not classified as university members any more – an
error that would only become apparent in the use of the tool much later and will
be potentially hard to track down to that particular ontology change operation.
Unit testing for ontologies can discover such problems, and a few other ones
as well, as we will see in the following sections.
3 Affirming derived knowledge
We create two test ontologies T+(called the positive test ontology) and T
(the negative test ontology), and define that an ontology O, in order to fulfil the
constraints imposed by the test ontologies, needs to fulfil the following conditions:
each axiom A+
nT+must be derivable from O, i.e.
and each axiom A
nTmust not be derivable from O, i.e.
Note that T+trivially fulfils the first condition if Ois not satisfiable, whereas
an empty ontology trivially fulfils the second condition. So it is not hard to
come up with ontologies that fulfil the conditions, which shows that unit tests
are not meant to be complete formalizations of the requirements of an ontology,
but rather helpful indicators towards possible errors or omissions in the tested
ontologies. Note also that Tcould be unsatisfiable, i.e. there are two sets of
axioms (both subsets of Tthat contradict each other. This still makes sense,
as it means that Omust not make a decision about the truth of either of these
sets (thus formalizing the requirement that Omust be agnostic towards certain
To come back to our previous example in Section 2 a simple test ontology
T+that consists of the single axiom Professor vUniversitymember would have
been sufficient to discover the problem described. So after the discovered error,
this statement is added to the test ontology, and now this same error will be
detected by running the unit tests.
The test ontologies are meant to be created and grown during the mainte-
nance of the ontology. Every time an error is encountered in the usage of the
ontology, the error is formalized and added to the appropriate ontology (like
in the example above). Experienced ontology engineers may add appropriate
axioms in order to anticipate and counter possible errors in maintenance.
In software engineering it is often the case, that the initial development of a
program is done by a higher skilled, better trained, and more consistent team,
whereas the maintenance is then performed by a less expensive group, with less
experienced members, that change more frequently. So in software engineering,
the more experienced developers often anticipate frequent errors that can happen
during maintenance, and create unit tests accordingly in order to put appropriate
constraints on the future evolution of the software. We expect a similar devel-
opment in ontology engineering and maintenance, as soon as ontologies become
more common components of information systems. The framework proposed in
this paper offers the same possibilities to an ontology engineer.
Why should an ontology engineer not just add the axioms from T+to O,
and ¬A
ifor each A
iin T? There are several reasons: 1) not every axiom
ican be negated. For example, the simple statement R(a, b) stating a relation
Rbetween the individuals aand bcan not be negated in OWL DL. 2) adding
such axioms increases redundancy in the ontology, and thus makes it harder
to edit. 3) the axioms may potentially increase reasoning complexity, or else
use language constructs that are not meant to be used within the ontology, for
whatever reason. 4) as stated above, the axioms in Tmay be contradictory.
5) Finally, due to the open world assumption, O6|=A
iTis not the
same as O|=¬A
iT, so that the negative test ontology can actually
not be simulated with the means of OWL DL.
4 Formalized competency questions
Competency questions, as defined by some methodologies for ontology engineer-
ing (like OTK [17] or Methontology [6]), describe what kind of knowledge the
resulting ontology is supposed to answer. These questions can always be formal-
ized in a query language (or else the ontology will actually not be able to answer
the given competency question, and thus will not meet the given requirements).
Formalizing the queries, instead of writing them down in natural language, and
formalizing the expected answers as well, allows for a system that automati-
cally checks if the ontology meets the requirements stated with the competency
We consider this approach especially useful not for the maintenance of the
system, but rather for its initial build, in order to define the extent of the on-
tology. Note that competency questions usually are just exemplary questions –
answering all competency questions does not mean that the ontology is complete.
Also note that sometimes, although the question is formalizable, the answer does
not necessarily need to be known at the time of writing the question. This is es-
pecially true for dynamic ontologies, i.e. ontologies that reflect properties of the
world that keep changing often (like the song the user of the system is listening
to at query time). In that case we can define some checks if the answer is sensible
or even possible (like that the answer indeed needs to be a song). Often these
further checks will not go beyond the abilities defined by the other approaches
to unit testing in this paper.
5 Expressive consistency checks
Ontologies in information systems often need to fulfil the requirement of allowing
reasoners to quickly answer queries with regards to the ontology. Light weight
ontologies usually fulfil this task best. Also, many of the more complex con-
structors of OWL DL often do not add further information, but rather are used
to restrict possible models. This is useful in many applications, like ontology
mapping and alignment, or information integration from different sources.
For example, a minimal cardinality constraint will, due to the open world
assumption, hardly ever lead to any inferred statements in an OWL DL ontology
(this can only become the case if range of the minimal cardinality restricted
relation is a class consisting of nominals). Nevertheless the statement can be
useful as an indicator for tools that want to offer a user interface to the ontology,
or for mapping algorithms that can take this information into account.
Further expressive constraints on the ontology, like disjointness of classes, can
be used to check the ontology for consistency at the beginning of the usage, but
after this has been checked, a light weight version of the ontology, that potentially
enables reasoners to derive answers with a better response time, could be used
Also, for these pre-use consistency checks, more expressive logical formalisms
could be used, like reasoning over the ontology metamodel [13, 3], using SWRL [11],
or using the transformation of the ontology to a logic programming language like
datalog [8] and then add further integrity constraints to that resulting program
(that may easily go far beyond the expressivity available in the original ontology
language: with a logic programming language it would be easy to state that,
by company policy, a supervisor is required to have a higher income than the
persons reporting to her – which is impossible in OWL DL).
Formally, we introduce a test ontology TCfor an ontology O, that includes
the high axiomatization of the terms used in O, and check for the satisfiability
of the merged ontology OTC. In the case of using the logic programming
translation, we merge (concatenate) the translation of the ontology to datalog
(LP(O)) with the test program TC
LP, and test the resulting program for violation
of the integrity constraints.
Let us consider an example ontology Ox:
bestBuddy(Jim, J ack)
bestBuddy vhasNoSecrets
This example ontology is equivalent to the translated logic program LP(Ox):
bestBuddy(Jim, J ack).
hasNoSecrets(X, Y ) : bestBuddy(X, Y ).
The test program TC
LP (that checks for insider trading) may consist of the
following line, a single integrity constraint:
:hasNoSecrets(X, Y ),tradesStocks(Y, C),CEO(X, C ).
Evaluating the program should now raise a violated integrity constraint. We
could also name the integrity constraints (by putting insiderT rading into its
head and then explicitly evaluate insiderTrading to see if it evaluates to true
or false. We also could use the head insiderTrading(Y) and then query for the
same head, to get a list of people who actually do the insider trading (and thus
uncover problems in the program much faster).
6 Use of autoepistemic operators
In [7] an extension of OWL DL with autoepistemic operators is described. Espe-
cially the K-operator can be useful to check an ontology not only with regards to
its consistent usage, but also with regards to some explicitly defined understand-
ing of completeness. In a geographic ontology, we may define that every country
has a capital, Country v ∃capital.City. But stating the existence of a country
without stating its capital will not lead to an error in the ontology, because the
reasoner (correctly) assumes, that the knowledge is not complete. Using the K-
and Aoperators instead, we would define that KCountry v ∃Acapital.ACity,
i.e. for every known country the capital must also be known (i.e. either stated
explicitly or inferrable) in the ontology, or else the ontology will be not satisfiable
(the example follows ex. 3.3 in [5]). Thus we are able to state what should be
known, and a satisfiability check will check if, indeed this knowledge is present.
On the Semantic Web, such a formalism will prove of great value, as it allows
to simply discard data that does not adhere to a certain understanding of com-
pleteness. For example, a crawler may gather event data on the Semantic Web.
But instead of simply collecting all instances of event, it may decide to only
accept events that have a start and an end date, a location, a contact email,
and a classification with regards to a certain term hierarchy. Although this will
decrease the recall of the crawler, the data will be of a higher quality, i.e. of a
bigger value, as it can be sorted, displayed, and actually used by calendars, map
tools, and email clients in order to support the user.
The formalisation and semantics of autoepistemic operators for the usage in
web ontologies is described in [7], and thus will not be repeated here.
7 Domain and ranges as constraints
Users often complain that their intuitive usage of relation domain and ranges
contradict their actual semantics. They expect domain and ranges to be used like
constraints, i.e. if they say that brideOf is a relation with the domain Woman
and one applies it to John, who is a Man, instead of getting an inconsistency
John will be classified as a Woman by the reasoner (for the sake of the example,
we take it for granted that Man and Woman are not disjoint). In the following
we will try to use this error to the benefit of the user.
As the domain and range declarations in the ontology will usually render
these checks trivially true, we need to remove them first from the ontology.
Therefore we take an ontology, and delete all domain and range declarations that
are to be understood as constraints (in the example in the previous paragraph,
we would remove the declaration of brideOf ’s domain as Woman). Now we check
for all instantiations of the removed domain or range declaration’s relation if the
subject (in case of a removed domain declaration) or the object (in case of a
removed range declaration) indeed gets classified with the supposed class (in the
example, we ask if John is indeed a Woman).
Note that these removals may have further implications on the ontology’s
inferred statements, depending on the further axioms of the ontology, and its
planned usage. Experiments need to be performed to be able to judge the de-
scribed approach with regards to its impact on the inferred knowledge in real
world scenarios. This approach actually will not necessarily highlight errors, but
only indicate possible places for errors. It will probably make more sense to in-
troduce a new relation that let us define constraints for relations, and then to
check these explicitly. We expect to learn from the planned experiments how to
exactly bring this approach to use.
8 Related work
A Prot´eg´e Plug-In implementing an OWL Unit Test framework2exists, that
allows to perform what we have described with T+testing for affirming derived
knowledge in Section 3.
In [16] the theory and practice of ontology evolution is discussed. Ontology
change operations and ontology evolution strategies are introduced. Based on
this, [9] extends this work for OWL DL ontologies, and investigates the evo-
lution of ontologies with regards to consistency, implemented in the so called
evOWLution framework. As the theoretical work allows generic and user defined
consistency checks, the ideas presented here could be regarded as a number of
ways to formalize further aspects of the ontology, and enable more expressive
consistency checks beyond simple logical satisfiability.
Some parts of the presented ideas – especially the test ontologies in Section 3
and the consistency check against heavy weight descriptions in Section 5 – may
often lead to unsatisfiable ontologies when the unit testing uncover problems. In
this case, research done in debugging [14] and revising [12], especially evolving
ontologies [10], will provide the tools and techniques to actually resolve the
9 Outlook
The approaches described in this paper address problems in ontology engineering
and maintenance that have been discovered during the work with ontologies
within the SEKT case studies. As they often reminded us of problems that
occurred in software engineering, a solution that was successfully introduced to
software engineering was examined – unit testing. Although the notion of unit
testing needed to be changed, it inspired a slew of possible approaches, that
have been described in this paper. Also, examples for these approaches have
been given to illustrate how they can be used within the lifecycle of an ontology.
As of now, we are working on an implementation of the presented ideas in
order to experimentally verify their usefulness. Although we have shown that sev-
eral problems we have encountered can be solved with the presented approaches,
it is unclear if the idea behind them is simple enough to be understood by non-
experts in ontology engineering. Also it needs to be investigated, how often
certain classes of problems appear in real world ontologies, and which of the
ideas presented here are most effective to counter these problems.
Like in software engineering, we do not expect unit tests to cover the whole
field of ontology evaluation. But we expect it to become (and remain) an impor-
tant building block within an encompassing framework, that will cover regression
tests and (relative) completeness, and help to indicate further errors in the initial
development, and especially further maintenance of an ontology.
We expect modularization of ontologies and networked ontology to become
more important in the next generation of web ontology based technologies. Unit
testing provides a framework to formalize requirements about ontologies. We
expect the approaches described in this paper, and maybe further similar ap-
proaches, to become much more investigated and discussed in the close future.
Research reported in this paper was supported by the IST Programme of the Eu-
ropean Community under the SEKT project, Semantically Enabled Knowledge
Technologies (IST-1-506826-IP, We want to
thank our colleagues for fruitful discussions, especially Stephan Grimm, Boris
Motik, Peter Haase, Sudhir Agarwal, Jos Lehmann, and Malvina Nissim.
This publication reflects the authors’ view.
1. K. Beck. Simple Smalltalk testing: With patterns. http://www.xprogramming.
2. V. Benjamins, P. Casanovas, J. Contreras, J. L. Cobo, and L. Lemus. Iuriservice:
An intelligent frequently asked questions system to assist newly appointed judges.
In V. Benjamins, P. Casanovas, A. Gangemi, and B. Selic, editors, Law and the
Semantic Web, LNCS, Berlin Heidelberg, 2005. Springer.
3. S. Brockmans and P. Haase. A Metamodel and UML Profile for Rule-
extended OWL DL Ontologies –A Complete Reference. Technical report, Uni-
versit¨at Karlsruhe, March 2006.
4. P. Casanovas, N. Casellas, M. Poblet, J. Vallb´e, Y. Sure, and D. Vrandeˇci´c. Iuris-
ervice ii ontology development. In P. Casanovas, editor, Workshop on Artificial
Intelligence and Law at the XXIII. World Conference of Philosophy of Law and
Social Philosophy, May 2005.
5. F. M. Donini, D. Nardi, and R. Rosati. Description logics of minimal knowledge
and negation as failure. ACM Transactions on Computational Logic, 3(2):177–225,
6. M. Fern´andez-L´opez, A. G´omez-P´erez, J. P. Sierra, and A. P. Sierra. Building
a chemical ontology using Methontology and the Ontology Design Environment.
IEEE Intelligent Systems, 14(1), 1999.
7. S. Grimm and B. Motik. Closed world reasoning in the semantic web through
epistemic operators. In B. C. Grau, I. Horrocks, B. Parsia, and P. Patel-Schneider,
editors, OWL: Experiences and Directions, Galway, Ireland, Nov 2005.
8. B. Grosof, I. Horrocks, R. Volz, and S. Decker. Description Logic Programs: Com-
bining Logic Programs with Description Logic. In Proceedings of the Twelfth In-
ternational World Wide Web Conference, WWW2003, Budapest, Hungary, 20-24
May 2003, pages 48–57. ACM, 2003.
9. P. Haase and L. Stojanovic. Consistent evolution of OWL ontologies. In A. G´omez-
erez and J. Euzenat, editors, Proc. of the 2nd European Semantic Web Conf.
ESWC, volume 3532, pages 182–197, Heraklion, Crete, Greece, 2005. Springer.
10. P. Haase, F. van Harmelen, Z. Huang, H. Stuckenschmidt, and Y. Sure. A frame-
work for handling inconsistency in changing ontologies. In Y. Gil, E. Motta, V. R.
Benjamins, and M. A. Musen, editors, Proceedings of the ISWC2005, volume 3729
of LNCS, pages 353–367. Springer, NOV 2005.
11. I. Horrocks, P. Patel-Schneider, H. Boley, S. Tabet, B. Grosof, and M. Dean.
SWRL: a semantic web rule language combining OWL and RuleML, 2003.
12. S. J. Lam, D. Sleeman, and W. Vasconcelos. Retax++: a tool for browsing and
revising ontologies. In Proc. ISWC 2005 Demo Session, Galway, Ireland, 2005.
13. B. Motik. On the properties of metamodeling in OWL. In Proc. of the 4th Inter-
national Semantic Web Conference, Galway, Ireland, November 2005, NOV 2005.
14. B. Parsia, E. Sirin, and A. Kalyanpur. Debugging OWL ontologies. In Proc. of
the 14th World Wide Web Conference (WWW2005), Chiba, Japan, May 2005.
15. M. K. Smith, C. Welty, and D. McGuinness. OWL Web Ontology Language Guide,
2004. W3C Rec. 10 February 2004, avail. at
16. L. Stojanovic. Methods and Tools for Ontology Evolution. PhD thesis, University
of Karlsruhe, 2004.
17. Y. Sure and R. Studer. On-To-Knowledge methodology. In J. Davies, D. Fensel,
and F. van Harmelen, editors, On-To-Knowledge: Semantic Web enabled Knowl-
edge Management, chapter 3, pages 33–46. J. Wiley and Sons, 2002.
... They called it "Competency question-driven approach for ontology testing". Note that previously, [24] introduced the dynamic (unit) testing approach for ontologies too. ...
Full-text available
One of the Software Engineering areas that supports quality assurance is testing. Given that specific processes, artefacts, methods and ultimately strategies for software testing involve a large number of domain concepts, it is valuable to have a robust conceptual base, that is, a software testing ontology that defines the terms, properties, relationships and axioms explicitly and unambiguously. Ontologies for instance foster a clearer terminological understanding of process and method specifications for strategies, among many other benefits. After analyzing both the results of a conducted Systematic Literature Review of primary studies on conceptualized software testing ontologies and the state-of-the-art of testing-related standards, we decided to develop a software testing top-domain ontology named TestTDO that fits our goals. Therefore, this article specifies development, verification and validation aspects of the TestTDO, which was built following the Design Science Research approach.
... The implementation of the work has been done within the Semantic Me-diaWiki, an extension of the MediaWiki which provides collaborative creation and maintenance of ontologies. Another innovative idea is the introduction of unit tests for ontologies by the same authors [47]. [33] 2012 Ontology Methodology URL [WR24] Garcia et al. [13] 2010 Ontology Metrics Survey Fernandez et al. [40] 2009 Ontology Quality METHONTOLOGY [35] 1997 Ontology Methodology Tartir et al. [41] [42] 2007-2005 Ontology Quality OntoQA 3 Ontology Metrics 9 Instance Metrics Brank et al. [43] 2005 Ontology Evaluation Survey Burton et al. [44] 2005 10 ontology Metrics Lozano et al. [45] 2004 160 ontology Metrics OntoMetric Corcho et al. [32] 2003 Ontology Methodology Survey (Survey paper) Noy et al. [21] 2001 Ontology Methodology Ontology development 101 ...
... The implementation of the work has been done within the Semantic Me-diaWiki, an extension of the MediaWiki which provides collaborative creation and maintenance of ontologies. Another innovative idea is the introduction of unit tests for ontologies by the same authors [47]. [33] 2012 Ontology Methodology URL [WR24] Garcia et al. [13] 2010 Ontology Metrics Survey Fernandez et al. [40] 2009 Ontology Quality METHONTOLOGY [35] 1997 Ontology Methodology Tartir et al. [41] [42] 2007-2005 Ontology Quality OntoQA 3 Ontology Metrics 9 Instance Metrics Brank et al. [43] 2005 Ontology Evaluation Survey Burton et al. [44] 2005 10 ontology Metrics Lozano et al. [45] 2004 160 ontology Metrics OntoMetric Corcho et al. [32] 2003 Ontology Methodology Survey (Survey paper) Noy et al. [21] 2001 Ontology Methodology Ontology development 101 ...
Full-text available
Sensor-based applications are increasingly present in our everyday life. Due to the enormous quantity of sensor data produced, interpreting data and building interoperable sensor-based applications is needed. There are several problems to address the heterogeneity of (1) data format, (2) languages to describe sensor metadata, (3) models for structuring sensor datasets, (4) reasoning mechanisms and rule languages to interpret sensor datasets, and (5) applications. Semantic Web technologies (a.k.a, knowledge graphs), are immersed in an increasing number of online activities we perform today (e.g., search engines for gathering information). There is a need to find better ways to share data and distribute more meaningful and more accurate information. Innovative methodologies are needed to link and associate the data from different domains to improve knowledge discovery. Semantic knowledge graphs, made of datasets and ontologies, are intended to describe and organize heterogeneous data explicitly. If an ontology is widely used to structure data of a particular domain, the accessibility and the efficiency in sharing and reusing that information will increase. For this reason, we focused on the ontology quality used when building sensor-based applications. We designed PerfectO, a Knowledge Directory Services tool, focusing on ontology best practices, which: (1) improves knowledge quality, (2) leverages usability, accessibility, and classification of the information, (3) enhances engineering experience, and (4) promotes engineering best practices. PerfectO implementation is applied to the Internet of Things (IoT) domain because it covers more than 20 application domains (e.g., healthcare, smart building, smart farm) that use sensors. PerfectO enhances knowledge expertise quality implemented within any ontologies as demonstrated with the Linked Open Vocabularies for IoT (LOV4IoT) ontology catalog.
... There are several methodologies which deal with this testing activity, such as the work presented by Blomqvist et al. [3] or OntologyTest [20]. Having the requirements formalised allows for a system to automatically check if the ontology implementation meets them [53]. ...
Full-text available
The software engineering field is continuously making an effort to improve the effectiveness of the software development process. This improvement is performed by developing quantitative measures that can be used to enhance the quality of software products and to more accurately describe, better understand and manage the software development life cycle. Even if the ontology engineering field is constantly adopting practices from software engineering, it has not yet reached a state in which metrics are an integral part of ontology engineering processes and support making evidence-based decisions over the process and its outputs. Up to now, ontology metrics are mainly focused on the ontology implementation and do not take into account the development process or other artefacts that can help assessing the quality of the ontology, e.g. its requirements. This work envisions the need for a metrics-driven ontology engineering process and, as a first step, presents a set of metrics for ontology engineering which are obtained from artefacts generated during the ontology development process and from the process itself. The approach is validated by measuring the ontology engineering process carried out in a research project and by showing how the proposed metrics can be used to improve the efficiency of the process by making predictions, such as the effort needed to implement an ontology, or assessments, such as the coverage of the ontology according to its requirements.
... For a sub-set of CQs for ROoST, Souza et al. also used the "Ontology Testing approach" as per their words. Previously, in 2006, Vrandečić et al. [14] introduced the dynamic (unit) testing approach for ontologies as well. Table III. ...
Conference Paper
Verification and Validation (V&V) are undoubtedly two key aspects of the artifact construction process. V&V entails activities, methods, tools, and ultimately strategies for ensuring quality when developing artifacts. Software engineering production lines, stakeholders and researchers produce diverse artifacts. For example, researchers may build ontologies, which can be both conceptualized and implemented artifacts. Ontologies are playing a central role in many application domains in particular, as well as in software and knowledge engineering disciplines in general. To show the applicability of V&V approaches, this work considers a software testing ontology as an artifact. This was developed following the Design Science Research methodology in the context of a layered ontological architecture. This paper illustrates the V&V approaches used, involving activities and methods for static and dynamic testing, evaluation and assessment. It addresses V&V approaches for the ontology conceptualization, as well as a functional dynamic testing method for its implementation.
... As approach to ontology evaluation we took into account two aspects: (i) data-driven (Brank et al., 2005), to represent real situations, to validate BiGe-Onto, we instantiated concepts and relations using data from GBIF and OBIS, and (ii) ontology testing (Vrandečić and Gangemi, 2006), to answer the competence questions formulated by domain experts. Below, each of the proposed steps is described in detail. ...
Great progress to digitize the world’s available Biodiversity and Biogeography data have been made recently, but managing data from many different providers and research domains still remains a challenge. A review of the current landscape of metadata standards and ontologies in Biodiversity sciences suggests that existing standards, such as the Darwin Core terminology, are inadequate for describing Biodiversity data in a semantically meaningful and computationally useful way. As a contribution to fill this gap, we present an ontology-based system, called BiGe-Onto , designed to manage data together from Biodiversity and Biogeography. As data sources, we use two internationally recognized repositories: the Global Biodiversity Information Facility (GBIF) and the Ocean Biogeographic Information System (OBIS). BiGe-Onto system is composed of (i) BiGe-Onto Architecture (ii) a conceptual model called BiGe-Onto specified in OntoUML, (iii) an operational version of BiGe-Onto encoded in OWL 2, and (iv) an integrated dataset for its exploitation through a SPARQL endpoint. We will show use cases that allow researchers to answer questions that manage information from both domains.
Lack of methodologies for ontology quality evaluation causes a challenge in producing good quality ontologies. Thus, we developed an iterative quality methodology to address this gap by analyzing the existing quality theories defined in ontology engineering, as well as, the theories in software engineering. Accordingly, this paper presents the developed methodology including how the other ontology quality theories get associated with it. Moreover, a use case in the agriculture domain has been demonstrated in order to provide an understanding of how the methodology can be applied in a real context. In the future, many experiments are expected to be carried out to fine-tune the methodology and to illustrate its usefulness.
When ontologies reach a certain size and complexity, faults such as inconsistencies, unsatisfiable classes or wrong entailments are hardly avoidable. Locating the incorrect axioms that cause these faults is a hard and time-consuming task. Addressing this issue, several techniques for a semi-automatic fault localization in ontologies have been proposed and extensively studied. One class of these approaches involve a human expert who provides answers to system-generated queries about the intended (correct) ontology in order to reduce the possible fault locations. To suggest as informative questions as possible, existing methods draw on various algorithmic optimizations as well as heuristics. However, these computations are often based on certain assumptions about the interacting expert. In this work, we demonstrate that these assumptions might not always be adequate and discuss consequences of their violations. In particular, we characterize a range of expert types with different query answering behavior and show that existing approaches are far from achieving optimal efficiency for all of them. In addition, we find that the cost metric adopted by state-of-the-art techniques might not always be realistic and that a change of metric has a decisive impact on the best choice of query answering strategy. As a remedy, we suggest a new—and simpler—type of expert question that leads to a stable fault localization performance for all analyzed expert types and effort metrics, and has numerous further advantages over existing techniques. Moreover, we present an algorithm which computes and optimizes this new query type in worst-case polynomial time and which is fully compatible with existing concepts (e.g., query selection heuristics) and infrastructure (e.g., debugging user interfaces) in the field. Comprehensive experiments on faulty real-world ontologies attest that the new querying method is substantially and statistically significantly superior to existing techniques both in terms of the number of necessary expert interactions and in terms of the query computation time. We find that relying on the new querying method can save an interacting expert more than 80% of their work, and can reduce the expert’s waiting time for the next query by more than three orders of magnitude. Beside these findings, we demonstrate that the efficiency of existing query-based tools can be significantly boosted by suggesting an appropriate query answering strategy to an expert; we also make recommendations in this regard. Further, we suggest optimal configurations of a debugger for situations where the new type of query is used. Remarkably, the proposed approach is not only applicable to ontologies, but to any monotonic knowledge representation language, and can even be adopted to solve general model-based diagnosis problems expressible using Reiter’s theory.
Full-text available
The era of digitalization poses high demands on capturing and processing knowledge generated in everyday life in formal models. Ontologies provide common means for formal knowledge capturing and modelling for a universe of discourse. Developing ontologies, however, can be a complex, time-consuming and expensive process which requires a significant amount of resource investments. Different stakeholders, such as ontology engineers, domain experts and ultimately users, are usually involved in the development process; they may be geographically distributed and work independently in isolated environments while typically have to synchronize their contributions. Supporting the entire development life-cycle of ontology modelling places a number of challenges. Stakeholders may have different working behaviours and practices that should be accommodated. Concurrent ontology modifications performed using various authoring editors have to be tracked, integrated and may result in synchronization conflicts. Further, ensuring ontology quality according to the domain requirements is another challenge to be tackled. In the past years, several methodologies and tools have been created to enable ontology development for a number of different purposes and applications. Albeit designed to cover a range of development aspects, existing approaches lack comprehensive support of the ontology life-cycle, in particular independent work in disparate environments. In this thesis, we tackle the problem of collaborative ontology development in distributed and heterogeneous environments, and present a stakeholder-oriented approach able to holistically assist the development of ontologies in diverse and independent scenarios. First, we define Git4Voc, a lightweight methodology comprising a set of guidelines and practices to be followed by stakeholders while modelling ontologies. We then conceive VoCol, a flexible and integrated development platform to address critical requirements from a technical perspective. Moreover, techniques for reducing the number of conflicts and allowing the efficient evaluation of test cases have been designed and implemented. VoCol can be adopted in numerous scenarios and accommodate additional tools in a well-designed and semi-automatic ontology development workflow. The benefits of this flexibility are two-fold: 1) stakeholders do not need to strictly follow a specific methodology; in contrary, they can organize their work in small and incremental development steps; and 2) consumers may provide their feedback, even though they are not directly part of the active development team. VoCol can be utilized to efficiently ensure quality ontologies with respect to the pre-defined requirements. We conducted several empirical evaluations to assess the effectiveness and efficiency of our holistic approach. More importantly, ontologies for various domains, such as Industry 4.0, life sciences and education, have been successfully developed and managed following our approach. The results from the empirical evaluations and concrete applications provide evidence that the methodology and techniques presented in this thesis comply with stakeholders’ needs and effectively support the entire ontology development life-cycle in distributed and heterogeneous environments.
Full-text available
Achieving a comfortable thermal situation within buildings with an efficient use of energy remains still an open challenge for most buildings. In this regard, IoT (Internet of Things) and KDD (Knowledge Discovery in Databases) processes may be combined to address these problems, even though data analysts may feel overwhelmed by heterogeneity and volume of the data to be considered. Data analysts could benefit from an application assistant that supports them throughout the KDD process and aids them to discover which are the relevant variables for the matter at hand, or informing about relationships among relevant data. In this article, the EEPSA (Energy Efficiency Prediction Semantic Assistant) ontology which supports such an assistant is presented. The ontology is developed on the basis that a proper axiomatization shapes the set of admitted models better, and therefore, establishes the ground for a better interoperability. On the contrary, underspecification facilitates the admission of non-isomorphic models to represent the same state which hampers interoperability. This ontology is developed on top of three ODPs (Ontology Design Patterns) which include proper axioms in order to improve precedent proposals to represent features of interest and their respective qualities, as well as observations and actuations, the sensors and actuators that generate them, and the procedures used. Moreover, the ontology introduces six domain ontology modules integrated with the ODPs in such a manner that a methodical customization is facilitated.
Full-text available
Many existing ontology tools provide an integrated environment to browse and edit ontologies as well as inconsistency checking facilities. However, their visualization facilities are limited and guidance on how to correct the detected errors is not usually provided. We present our ontology editor, ReTAX++, a tool that facilitates browsing and revision of ontologies.
Conference Paper
Full-text available
One of the major problems of large scale, distributed and evolving on- tologies is the potential introduction of inconsistencies. In this paper we survey four different approaches to handling inconsistency in DL-based ontologies: con- sistent ontology evolution, repairing inconsistencies, reasoning in the presence of inconsistencies and multi-version reasoning. We present a common formal ba- sis for all of them, and use this common basis to compare these approaches. We discuss the different requirements for each of these methods, the conditions un- der which each of them is applicable, the knowledge requirements of the various methods, and the different usage scenarios to which they would apply.
On-To-Knowledge builds an ontology-based tool environment to improve know­ledge management, dealing with large numbers of heterogeneous, distributed, and semistructured documents typically found in large company intranets and the World-Wide Web. The project’s target results are: (i) a toolset for semantic information processing and user access; (ii) OIL, an ontology-based inference layer for the World-Wide Web; and (iii) an associated methodology and validation by industrial case studies. This chapter offers an overview of the On-To-Knowledge approach to knowledge management.
Conference Paper
We show how to interoperate, semantically and inferentially, between the leading Semantic Web approaches to rules (RuleML Logic Programs) and ontologies (OWL/DAML+OIL Description Logic) via analyzing their expressive intersection. To do so, we define a new intermediate knowledge representation (KR) contained within this intersection: Description Logic Programs (DLP), and the closely related Description Horn Logic (DHL) which is an expressive fragment of first-order logic (FOL). DLP provides a significant degree of expressiveness, substantially greater than the RDF-Schema fragment of Description Logic. We show how to perform DLP-fusion: the bidirectional translation of premises and inferences (including typical kinds of queries) from the DLP fragment of DL to LP, and vice versa from the DLP fragment of LP to DL. In particular, this translation enables one to "build rules on top of ontologies": it enables the rule KR to have access to DL ontological definitions for vocabulary primitives (e.g., predicates and individual constants) used by the rules. Conversely, the DLP-fusion technique likewise enables one to "build ontologies on top of rules": it enables ontological definitions to be supplemented by rules, or imported into DL from rules. It also enables available efficient LP inferencing algorithms/implementations to be exploited for reasoning over large-scale DL ontologies.
In this paper we present a MOF compliant metamodel and UML proflle for the Semantic Web Rule Language (SWRL) that inte- grates with our previous work on a metamodel and UML proflle for OWL DL. Based on this metamodel and proflle, UML tools can be used for visual modeling of rule-extended ontologies.
Conference Paper
A common practice in conceptual modeling is to separate the intensional from the extensional model. Although very intuitive, this approach is inadequate for many complex domains, where the borderline between the two models is not clear-cut. Therefore, OWL-Full, the most expressive of the Semantic Web ontology languages, allows combining the intensional and the extensional model by a feature we refer to as metamodeling. In this paper, we show that the semantics of metamodeling adopted in OWL-Full leads to undecidability of basic inference problems, due to free mixing of logical and metalogical symbols. Based on this result, we propose two alternative semantics for metamodeling: the contextual and the HiLog semantics. We show that SHOIQ\mathcal{SHOIQ} — a description logic underlying OWL-DL — extended with metamodeling under either semantics is decidable. Finally, we show how the latter semantics can be used in practice to axiomatize the logical interaction between concepts and metaconcepts.
In this paper, we describe the use of legal ontologies as a basis to improve IT support for professional judges. In the ontology, we emphasize the importance of professional knowledge and experience as an important pillar for constructing the ontology. We describe an intelligent FAQ system for junior judges that intensively uses the ontology.