Content uploaded by Christoph Benzmüller
Author content
All content in this area was uploaded by Christoph Benzmüller on Aug 20, 2019
Content may be subject to copyright.
Designing Normative Theories of Ethical Reasoning:
Formal Framework, Methodology, and Tool Support
Christoph Benzm¨ullera,b, Xavier Parenta, Leendert van der Torrea,c
aComputer Science and Communications, University of Luxembourg, Esch-sur-Alzette,
Luxembourg
bDepartment of Mathematics and Computer Science, Freie Universit¨at Berlin, Berlin,
Germany
cInstitute of Logic and Cognition, Zhejiang University, Hangzhou, China
Abstract
A framework and methodology—termed LogiKEy—for the design and engi-
neering of ethical reasoners, normative theories and deontic logics is presented.
The overall motivation is the development of suitable means for the control
and governance of intelligent autonomous systems. LogiKEy’s unifying formal
framework is based on semantical embeddings of deontic logics, logic combina-
tions and ethico-legal domain theories in expressive classic higher-order logic.
This meta-logical approach enables the provision of powerful tool support in
LogiKEy: off-the-shelf theorem provers and model finders for higher-order
logic are assisting the LogiKEy designer of ethical intelligent agents to flexi-
bly experiment with underlying logics and their combinations, with ethico-legal
domain theories, and with concrete examples—all at the same time. Continu-
ous improvements of these off-the-shelf provers, without further ado, leverage
the reasoning performance in LogiKEy. Case studies, in which the LogiKEy
framework and methodology has been applied and tested, give evidence that
HOL’s undecidability often does not hinder efficient experimentation.
Keywords: Thrustworthy and responsible AI; Knowledge representation and
reasoning; Automated theorem proving; Model finding; Normative reasoning;
Normative systems; Philosophical and ethical issues; Semantical embedding;
Higher-order logic
Email addresses: c.benzmueller@fu-berlin.de (Christoph Benzm¨uller),
xavier.parent@uni.lu (Xavier Parent), leon.vandertorre@uni.lu (Leendert van der Torre)
1Benzm¨uller is funded by the VolkswagenStiftung under grant CRAP (Consistent Rational
Argumentation in Politics). Parent and van der Torre are supported by the European Union’s
Horizon 2020 research and innovation programme under the Marie Sk lodowska-Curie grant
agreement MIREL (MIning and REasoning with Legal texts) No 690974.
Preprint submitted to Journal of L
A
T
E
X Templates August 18, 2019
arXiv:submit/2808959 [cs.AI] 18 Aug 2019
1. Introduction
The explicit representation-of and reasoning-with legal and ethical knowledge
capacitates ethical intelligent systems with increasing levels of autonomy in their
decision making [1, 2], guaranteeing sufficient degrees of reliability and account-
ability, and achieving human intuitive interaction means regarding explanability
and transparency of decisions. The ethico-legal theories can thereby be repre-
sented as normative systems formalized on top of suitably selected underlying
deontic logics [3, 4, 5] and logic combinations [6, 7]. In this article we introduce
our Logic and Knowledge Engineering Framework and Methodology, termed
LogiKEy, to apply off-the-shelf theorem proving and model finding technol-
ogy to the design and engineering of specifically tailored normative systems and
deontic logics for deployment in ethical intelligent reasoners.
An overall objective of LogiKEy is to enable and support the practical de-
velopment of computational tools for normative reasoning based on formal meth-
ods. To this end we introduce the concept of experimentation: by representing
examples, ethico-legal domain theories, deontic logics and logic combinations
in a computational system we enable predictions and their assessment, and ap-
ply formal methods. Given this motivation to experiment-with, explore and
assess different theories and logics in the design of ethical intelligent reasoners
we address the following research questions:
1. Which formal framework to choose,
2. which methodology to apply, and
3. which tool support to consult
for experimentation with ethical theories?
Several examples of computational tools to experiment with ethical reason-
ers, normative systems and deontic logics have been introduced in the recent
literature [8, 9, 10, 11, 12, 13] and are used in this article to illustrate the fea-
sibility of the proposed solutions. LogiKEy extracts the general insights from
these prior case studies and bundles them into a coherent approach.
1.1. Formal framework: expressive classical higher-order logic
We first explain why we use classical higher-order logic (HOL), i.e., Church’s
type theory [14], as our formal framework. To understand the challenge of the
choice of a formal framework in our first research question, consider the re-
quirements of a typical ethical agent architecture as visualized in Fig. 1. The
displayed architecture for an intelligent autonomous system [15] with explicit
ethical competency distinguishes an explicit ethical reasoner and ethico-legal
domain theories from an AI reasoner/planner and from other components, in-
cluding also application data and knowledge available to both reasoners. The
ethical reasoner takes as input suggested actions from the AI reasoner/planner,
hints to relevant application data and knowledge, and the ethico-legal domain
theories, and it produces as output accepted actions and explanations. That
2
Ethico-legal Domain Theories
Explicit Ethical
Reasoner
AI Reasoner
Application Data and Knowledge
Explanation
Action
Figure 1: Explicit ethical reasoner for intelligent autonomous systems.
is, the actions suggested by the AI reasoners in Fig. 1 are not executed imme-
diately, but additionally assessed by the ethical reasoner for compliance with
respect to the given ethico-legal domain theories. This assessment is intended
to provide an additional, explicit layer of control on top of the AI Reasoner,
which ideally already comes with solid own ethical competency. For the aims
in this article, the details of the ethical agent architecture are not important.
For example, it does not matter whether computations on the level of the AI
reasoner are based on sub-symbolic or symbolic techniques, or combinations
of them, since the suggested actions are not executed immediately—at least
not those considered most critical. Instead they are internally assessed, before
execution, against some explicitly modelled ethical and legal theories. These
theories govern and control the behaviour of the entire system, and they also
support—at this upper-level—system verification and intuitive user explana-
tions. What counts, in particular, in highly critical contexts, is not only how an
AI reasoner computed the critical action it wants to perform, but in particular
whether this action passes the additional assessment by the upper-level explicit
ethical reasoner before its execution.
The formal framework of the ethical reasoner has to be populated with
ethico-legal domain theories, be able to combine logics, reason about the en-
coded domain theories, and experiment with concrete applications and exam-
ples, as visualized in Fig. 2.
This leads to an apparent paradox: on the one hand, the area of knowledge
representation and reasoning in artificial intelligence is built on highly special-
3
L1- Logics and Logic Combinations
L2 - Ethico-Legal Domain Theories
L3- Applications
Embedding in meta-logic HOL
Figure 2: Logic and knowledge engineering in LogiKEy.
ized and efficient formalisms;2on the other hand to combine all these aspects
in a single formalism and to experiment with it, we need a highly expressive
language. For example, as we explain in more detail in Sect. 3, the handling of
normative concepts such as obligation, permission, prohibition, and moral com-
mitment is actually far more complex than one would initially think. This is il-
lustrated, among others, by the notorious paradoxes of normative reasoning [21],
and appropriate solutions to this challenge require sophisticated deontic logics.
In LogiKEy we therefore model and implement ethical reasoners, normative
systems and deontic logics in the expressive meta-logic HOL. This often trig-
gers scepticism, in particular, in the AI knowledge representation and reasoning
community, since HOL has known theoretical drawbacks, such as undecidability.
However, HOL, just like first-order logic, has decidable fragments, and our shal-
low semantical embedding (SSE) approach, cf. Sect. 2, is capable of translating
decidable proof problems, say in propositional modal logic, into corresponding
decidable proof problems in (a fragment of) HOL. For example, the modal for-
mula (ϕ∧ψ) is—assuming a request about its global validity—rewritten in the
SSE approach into the guarded fragment formula ∀w.∀v.¬(Rwv)∨(ϕv ∧ψv),
where Ris denoting the accessibility relation associated with . This formula
is decidable and it can be effectively attacked by Isabelle/HOL [19] and Leo-
III [20], which are the systems we currently prefer. The reason why these system
can attack such decidable problems effectively is simple: they internally collab-
orate with state-of-the-art SAT & SMT (satisfiability modulo theories) solvers,
2In knowledge representation in mathematics, in contrast, practical developments are in-
terestingly progressing already in the opposite direction; proof assistants such as Lean [16],
Coq [17], or Agda [18], for example, are being developed and deployed, which, due to their
rich type systems, are practically even more expressive than the HOL provers we utilize in
LogiKEy:Isabelle/HOL [19] and Leo-III [20]. The latter provers, however, provide a very
good degree of proof automation, and this, combined with a sufficient degree of expressivity
for our purposes, is the reason we prefer them so far.
4
and with first-order automated theorem proving (ATPs) systems; this ensures
that that fragments of HOL are indeed attacked with tools that are specialized
for these fragments. Moreover, the framework we provide is uniform, and this
makes it an ideal approach for experimentation.
1.2. Methodology: LogiKEy
The LogiKEy methodology, which constitutes our answer to the second
research question, is based on the simultaneous development of the three layers
depicted in Fig. 2: combining logics, ethico-legal theories, and concrete exam-
ples and applications. The black arrows between these three levels symbolize
dependencies. The normative governance applications developed at layer L3
depend on ethico-legal domain theories imported from layer L2, which in turn
are formalized within a specific logic or logic combination provided at layer
L1.3The engineering process at the different layers has backtracking points
and several work cycles maybe be required; thereby the higher layers may also
pose modification requests to the lower layers. The white arrows symbolize such
possible requests, and they may, unlike in most other approaches, also include
far-reaching modifications of the logical foundations engineered at layer L1; such
modifications at the logic layer are flexibly facilitated in our meta-logical ap-
proach.
As a concrete application, consider a smart home visualized in Fig. 3 [22].
This one uses an argumentation engine [23] in order to make ethical decisions
Argumentation-based
Ethical Reasoner
Legal
system
Manufacturer
Family
Sensors
Knowledge
Base
Action
Smart Home
NS1
NS2
NS3
Figure 3: Example of ethical reasoner in a smart home [22].
based on normative systems of stakeholders. The argumentation-based mech-
anism is used to find an agreement in case of moral dilemmas, and to offer an
3In this article we utilise a uniform coloring scheme in all our conceptual figures: Layers
L1 and L2 are displayed using dark red and light red coloring, respectively. Applications and
experiments are displayed in blue, and the meta-logical HOL framework that is supporting
the layers L1 and L2 is in given in green.
5
Logic A
Logic C
Logic B Theory 1
Theory 2
Theory 3
Experimentation
Simulation
Enabling Technology
Higher Order Theorem
Provers
Isabelle/HOL
Leo-III
Figure 4: Flexible tool support for normative reasoning is required in LogiKEy.
explanation as to how a specific morally sensitive decision has been made. Liao
et al. assume the existence of pre-defined systems of norms, coming from the
outside and constraining the agent’s behavior. Like humans, intelligent systems
evolve in a highly regulated environment. For instance, a smart home processes
personal data 24/7, and, as will be discussed in Sect. 6.1, there are legal rules
one must comply with when processing personal data; in the European Union
such rules include the General Data Protection Regulation (GDPR, Regulation
EU 2016/679). Given a description of the current situation, the smart home
should thus be able, for example, to check compliance with these rules, and act
accordingly in case of non-compliance.
1.3. Tool support: Isabelle/HOL and Leo-III
Our use of higher-order tool support, cf. our third research question, is vi-
sualized in Fig. 4. The LogiKEy methodology supports experimentation with
different normative theories, in different application scenarios, and it is not tied
to a specific deontic logic and also not restricted to decidable logics only. Since
ethico-legal theories as well as suitable normative reasoning formalisms are both
subject to exploration in LogiKEy, we introduce a flexible workbench to sup-
port empirical studies in which both can be varied, complemented, assessed and
compared.
The enabling technology are higher-order theorem proving systems via the
SSE technique [24]. We benefit from good improvements in the past decade in
interactive theorem proving (ITP) and automated theorem proving (ATP) for
HOL, and also from the overall coalescence of heterogeneous theorem proving
systems, as witnessed, in particular, by Isabelle/HOL,LEO-II [25] and Leo-
III, which fruitfully integrate other specialist ATPs. In this way—as a sort of
relevant byproduct of our research—we build a bridge between the classical and
non-classical logics communities, the deduction systems community, and the
6
formal ethics community;4cf. also Fig. 5, which will be explained in more detail
in Sect. 2.1.
Theorem proving was a major impetus for the development of computer sci-
ence, and ATP and ITP systems have been applied to a variety of domains,
including mathematics and software & hardware verification. However, compa-
rably little work has been invested so far to apply such systems to deontic logic
and normative reasoning.
1.4. Contributions
The core contributions of this article are on various levels:
1. Motivated by our position in the trustworthy AI debate—where we de-
fend the need for explicit ethico-legal governance of intelligent autonomous
systems—we provide a survey of various technical results pertaining to the
application of automated theorem proving to ethical reasoners, normative
systems and deontic logics.
2. This survey is targeted to a general audience in AI. Instead of present-
ing selected formal definitions and proofs, we discuss motivations, com-
monalities and applications. Moreover, we provide an entry-level user-
guide to our framework and methodology, called LogiKEy. Existing ap-
proaches [26, 27, 28, 29] are always tied to rather specific logical settings.
We introduce a flexible workbench to support empirical studies with such
theories in which the preferred logic formalisms themselves can be varied,
complemented, assessed and compared.
3. Our ambition is to build a library of ethical reasoners, normative systems
and deontic logics, and with this article we make a first step in that di-
rection by providing a collection of Isabelle source files that can be reused
by researchers and students interested in adopting LogiKEy in their own
work.5
We briefly clarify some terminology as used in the article. With experimen-
tation we refer to the action or process of trying out new ideas, methods, or
4The formal ethics community, among others, includes: Normative Multi-Agent Systems
(http://icr.uni.lu/normas), Deontic Logic and Normative Systems (http://deonticlogic.
org) and Formal Ethics (http://www.fe2019.ugent.be). With deduction systems community
we refer to the areas of Automated Reasoning (http://aarinc.org) and Interactive Theorem
Proving (https://itp2018.inria.fr); these are in turn composed of several subcommunities.
5We are working on the requested Data In Brief paper, which we intend to sub-
mit soon. Until then the Isabelle source files will remain available at the address http:
//www.christoph-benzmueller.de/papers/LogiKEy.zip; we are currently further extending
the content in this zip-file for the Data In Brief submission so that it contains all relevant
deontic logic encodings we have developed so far. The current archive contains 7 files present-
ing embeddings of logic or logic combinations and 8 files presenting exemplary applications
of such logics for the encoding of Chisholm’s paradox, the GDPR example, the PGC, etc.
We are considering to add also student contributions from our lecture courses, provided that
copyright issues and authorship questions can be resolved.
7
activities; this can be, for instance, the drafting of a new logic or combination
of logics, or a new ethical theory, where one may want to see what would be the
consequences of its adoption in a given application context. Methodology refers
to the principles underlying the organization and the conduct of a design and
knowledge engineering process; hereby design means the depiction of the main
features of the system we want to achieve, and (knowledge or logic) engineering
refers to all the technical and scientific aspects involved in building, maintaining
and using a knowledge-based, resp. logic-based, system. Infrastructure is the
framework that supports different components and functionalities of a system,
and by logic implementation we here mean an engineering process that returns
an executable computer program, a theorem prover, for a given logic; this can
be done in very different ways, for example, by coding software from scratch or,
as we prefer, by adopting the SSE approach, which utilizes HOL as a meta-level
programming language within existing reasoners such as Isabelle or Leo-III.
The structure of this article is as follows. Section 2 further motivates the
need for a flexible normative reasoning infrastructure and presents LogiKEy’s
expressive reasoning framework. Section 3 briefly surveys and discusses the chal-
lenging area of normative reasoning; this section also explains which particular
deontic logics have already been implemented. A concrete example implementa-
tion of a state-of-the-art deontic logic, ˚
Aqvist system E, is presented in further
detail in Sect. 4. Tool support is discussed in Sect. 5, and subsequently two case
studies are presented in Sect. 6. The first case study illustrates contrary-to-duty
compliant reasoning in the context of the general data protection regulation. A
second, larger case study shows how our framework scales for the formaliza-
tion and automation of challenging ethical theories (Gewirth’s “Principle of
Generic Consistency” [30]) on the computer. To enable the latter work an
extended contrary-to-duty compliant higher-order deontic logic has been pro-
vided and utilized in our framework. Section 7 provides a detailed description
of LogiKEy’s 3-layered logic and knowledge engineering methodology; it may
thus provide useful guidance to future LogiKEy users. Sections 8 and 9 discuss
related work and further research, and Sect. 10 concludes the article.
2. The LogiKEy expressive reasoning framework: meta-logic HOL
This section presents and discusses LogiKEy’s expressive reasoning frame-
work, cf. Fig. 5, which utilizes and adapts Benzm¨uller’s SSE approach [24].
The implementation of specialist theorem provers and model finders for, e.g.,
ambitious deontic logics and their extensions and combinations, is very tedious
and requires expertise. LogiKEy therefore focuses on reuse and adaptation of
existing technology rather than new implementations from first principles. A
particular interest is to enable novices to become acquainted with the area of
normative reasoning in short time and in a computer-assisted, hands-on fashion;
this should enable them to gradually acquire much needed background exper-
tise. An initial goal of our work therefore has been to build up a starting basis
of mechanized deontic logics at layer L1 and ethico-legal domain theories at
8
Ethico-legal Domain Theory
Deontic Logics &
Logic combinations
Isabelle/HOL
models unfolds into
unfolds into
embeds
(Counter-)Model Generation
Nitpick - Nunchaku
SAT-Solver
Proof Automation
Sledgehammer
SMT-Solver
FOL-ATP HOL-ATP
may-call-as-subsystem
interact with
Figure 5: The LogiKEy expressive reasoning framework is based on meta-logic HOL.
layer L2, and to make them accessible for experimentation to students and re-
searchers. Our approach pays much attention to intuitive interaction within
sophisticated user-interfaces to access, explore, assess and modify both the used
foundational logic and logic combination and the encoded domain theories. The
tools we reuse are state-of-the-art reasoning systems for classical logic that are
actively developed by the deduction systems community. That relieves us from
resource intensive investments in the implementation and maintenance of new
technology. However, to enable this reuse of technology, a bridge between de-
ontic logics (and their combinations with other modal logics) and classical logic
was needed. This bridge is provided by the SSE approach.
The framework, methodology and technology we contribute requires some
modest background knowledge to be acquired. The logic developers’ perspec-
tive, as relevant at LogiKEy layer L1, is a bit more ambitious and requires
some familiarity with meta-logic HOL (cf. Sect. 2.2), and also a good mastery
of the SSE technique (cf. Sect. 2.1). However, also at that level we support the
adoption of initial skills by providing a library of example encodings of deontic
logics, and other relevant logics and logic combinations, to start with; this is
helpful, since it, among others, enables copying and pasting from these encod-
ings. Several successful student projects at BSc, MSc and PhD level meanwhile
provide good evidence for the practical relevance of our approach at the devel-
opers level [9, 31, 11, 32].
2.1. Logic engineering and implementation methodology: SSEs in HOL
Our expressive reasoning framework, depicted graphically in Fig. 5, is based
on the SSE approach [33, 24]. HOL is utilized in this approach as a universal
9
meta-logic in which different deontic logics are semantically embedded in a shal-
low way by directly encoding their semantics in meta-logic HOL; commonalities
in the semantics of both logics are thereby typically shared as much as possible.6
We have extended this approach in our project for ambitious deontic log-
ics, their extensions and combinations with other relevant logics. This enables
the use of interactive proof assistants, such as Isabelle/HOL, which comes
with a sophisticated user-interface and, in addition, integrates various state-of-
the-art reasoning tools. The ATP systems integrated with Isabelle/HOL via
the sledgehammer [36] tool comprise higher-order ATP systems, first-order ATP
systems and satisfiability modulo theories (SMT) solvers, and many of these sys-
tems internally again employ efficient SAT solving technology. Isabelle/HOL
also provides two model finders, Nitpick [37] and Nunchaku [38].
The SSE approach utilizes the syntactic capabilities of the higher-order the-
orem prover (a) to represent the semantics of a target logic, and (b) to define the
original syntax of the target theory within the prover. The overall infrastruc-
ture, in combination with the SSE approach, meets our demanding requirements
regarding flexibility along different axes; cf. Sect. 5.
An initial focus in the SSE approach has been on quantified modal logics [33].
One core aspect is that the standard translation [39] from propositional modal
logic to first-order logic can be semantically embedded, in HOL without requir-
ing an external translation mechanism. The modal operator ♦, for example, can
be explicitly defined by the λ-term λϕ.λw.∃v.(Rwv ∧ϕv), where Rdenotes the
accessibility relation associated with ♦. This definition, however, can be hidden
from the user, who can construct now modal logic formulas involving ♦ϕand
use them to represent and prove theorems.
Most importantly, however, such an embedding of modal logic operators
in HOL can be extended to also include quantifiers. We briefly illustrate this
idea using an example (omitting types, as above, for better readability). First,
it is relevant to note that ∀x.φx is shorthand in HOL for Π(λx.φx), where
the logical constant symbol Π is given an obvious semantics, namely to check
whether the set of objects denoted by (λx.φx) is the set of all objects (of the
respective type). ∃x.φx is hence shorthand for ¬Π(λx.¬φx). The important
and interesting aspect thus is that additional binding mechanisms for universal
and existential quantifiers can be avoided in HOL by reusing λ-notation. This
principle can now be applied also to obtain SSEs for quantified modal logics
and, analogously, for many other quantified non-classical logics. For example,
♦∀x.P x is represented as ♦Π0(λx.λw.P xw), where Π0stands for the λ-term
λΦ.λw.Π(λx.Φxw) and where the ♦gets resolved as described above. The
6Shallow semantical embeddings are different from deep embeddings of a target logic. In
the latter case the syntax of the target logic is represented using an inductive data structure
(e.g., following the definition of the language). The semantics of a formula is then evaluated by
recursively traversing the data structure, and additionally a proof theory for the logic maybe
be encoded. Deep embeddings typically require technical inductive proofs, which hinder proof
automation, that can be avoided when shallow semantical embeddings are used instead. For
more information on shallow and deep embeddings we refer to the literature [34, 35].
10
following series of conversions explains this encoding in more detail:
♦∀x.P x ≡♦Π0(λx.λw.P xw)
≡♦((λΦ.λw.Π(λx.Φxw))(λx.λw .P xw))
≡♦(λw.Π(λx.(λx.λw.P xw )xw))
≡♦(λw.Π(λx.P xw))
≡(λϕ.λw.∃v.(Rwv ∧ϕv))(λw.Π(λx.P xw))
≡(λϕ.λw.¬Π(λv.¬(Rwv ∧ϕv)))(λw.Π(λx.P xw))
≡(λw.¬Π(λv.¬(Rwv ∧(λw.Π(λx.P xw))v)))
≡(λw.¬Π(λv.¬(Rwv ∧Π(λx.P xv))))
≡(λw.∃v.Rwv ∧ ∀x.P xv)
This illustrates the embedding of ♦∀xP x in HOL.7Moreover, this embedding
can be accompanied with different notions of validity. For example, we say
♦∀x.P x is globally valid (valid for all worlds w) if and only if ∀w.((♦∀x.P x)w)
holds. Local validity for a particular actual world, denoted by a constant symbol
aw, then amounts to checking whether ((♦∀x.P x)aw) is true in HOL.
What has been sketched above is an SSE for a first-order quantified modal
logic K with a possibilist notion of quantification. However, depending on the
type we assign to variable xin ♦∀x.P x, the sketched solution scales for arbitrary
higher-order types. Since provers such as Isabelle/HOL and Leo-III support
restricted forms of polymorphism, respective universal and existential quantifiers
for the entire type hierarchy can be introduced with a single definition.8Further
details on the SSE of quantified modal logics in HOL, including a proof of
faithfulness, are available in the literature [33]. Standard deontic logic (modal
logic KD) can easily be obtained from this work. To do so we simply postulate
in meta-logic HOL that the accessibility relation Runderlying the ♦operator
is serial. The corresponding Daxiom ϕ⊃ ¬¬ϕ, or, equivalently, ¬(ϕ∧
¬ϕ), then becomes derivable as a corollary from this postulate and the SSE
(and so does the K-schema and the necessitation rule, already in base logic K).
Further emendations of the presented framework to obtain multi-modal logics
and an actualist notion of quantification have been proposed by Benzm¨uller et
al.; cf. [24] and the references therein for further details.
2.2. Meta-logic HOL
HOL has its roots in the logic of Frege’s Begriffsschrift [40]. However, the
version of HOL as addressed here refers to a simply typed logic of functions,
which has been put foward by Church [41]. It provides λ-notation, as an elegant
and useful means to denote unnamed functions, predicates and sets. Types in
HOL eliminate paradoxes and inconsistencies.
7In the implementation of our approach in Isabelle/HOL such conversions are hidden by
default, so that the user may interact with the system at the level of the target logic and enter
formulas such as ♦∀x.Px. Definition unfolding is handled in the background, but can made
visible upon request by the user.
8See, e.g., lines 14–17 in Fig. 6, where respective polymorphic quantifier definitions, in-
cluding binder notation, are provided for ˚
Aqvist’s system E.
11
To keep this article sufficiently self-contained we briefly introduce HOL; the
reader may want to skip this subsection and get back again later. More detailed
information on HOL and its automation can be found in the literature [41, 42,
43].
Definition 1 (Types).The set Tof simple types in HOL is freely generated
from a set of basic types BT ⊇ {o, i}using the function type constructor .
Usually, type odenotes the (bivalent) set of Booleans, and idenotes a non-empty
set of individuals. Further base types may be added.
Definition 2 (Terms and Formulas).The terms of HOL are defined as follows
(where Cαdenotes typed constants and xαtyped variables distinct from Cα;
α, β, o ∈T):
s, t ::= Cα|xα|(λxα.sβ)αβ|(sαβtα)β
Complex typed HOL terms are thus constructed via λ-abstraction and func-
tion application, and HOL terms of type oare called formulas.
As primitive logical connectives we choose ¬oo,∨oooand Π(αo)o(for
each type α), that is, we assume that these symbols are always contained
in the signature. Binder notation ∀xα.sois used as an abbreviation for
Π(αo)oλxα.so. Additionally, description or choice operators (αo)α(for
each type α) or primitive equality =ααo(for each type α), abbreviated as
=α, may be added. From the selected set of primitive logical connectives, other
logical connectives can be introduced as abbreviations. Equality can also be
defined by exploiting Leibniz’ principle, expressing that two objects are equal
if they share the same properties. Type information as well as brackets may be
omitted if obvious from the context or irrelevant.
We consider two terms to be equal if the terms are the same up to the names
of bound variables (i.e., α-conversion is handled implicitly).
Substitution of a term sαfor a variable xαin a term tβis denoted by [s/x]t.
Since we consider α-conversion implicitly, we assume the bound variables of t
are disjunct from from the variables in s(to avoid variable capture).
Prominent operations and relations on HOL terms include βη-normalization
and βη-equality,β-reduction and η-reduction: a β-redex (λx.s)t β-reduces to
[t/x]s; an η-redex λx.(sx) where variable xis not free in s,η-reduces to s. It
is well known, that for each simply typed λ-term there is a unique β-normal
form and a unique βη-normal form. Two terms land rare βη-equal, denoted as
l=βη r, if their βη-normal forms are identical (up to α-conversion). Examples
of λ-conversions have been presented on p. 11 (types were omitted there).
The semantics of HOL is well understood and thoroughly documented in
the literature [42, 44]. The semantics of choice for our work is Henkin [45]’s
general semantics. The following sketch of standard and Henkin semantics for
HOL closely follows Benzm¨uller and Andrews [41].
Aframe is a collection {Dα}α∈Tof nonempty sets, called domains, such that
Do={T , F }, where Trepresents truth and Ffalsehood, Di6=∅and Du6=∅
12
are chosen arbitrary, and Dαβare collections of total functions mapping Dα
into Dβ.
Definition 3 (Interpretation).An interpretation is a tuple h{Dα}α∈T, I i,
where {Dα}α∈Tis a frame, and where function Imaps each typed constant
symbol cαto an appropriate element of Dα, which is called the denotation of
cα. The denotations of ¬,∨and Π(αo)oare always chosen as usual. A vari-
able assignment φmaps variables Xαto elements in Dα.
Definition 4 (Henkin model).An interpretation is a Henkin model (general
model) if and only if there is a binary valuation function V, such that V(φ, sα)∈
Dαfor each variable assignment φand term sα, and the following conditions are
satisfied for all φ, variables xα, constants Cα, and terms lαβ, rα, sβ(α, β ∈T):
V(φ, xα) = φ(xα),V(φ, Cα) = I(Cα),V(φ, lαβrα) = (V(φ, lαβ)V(φ, rα)),
and V(φ, λxα.sβ)represents the function from Dαinto Dβwhose value for
each argument z∈Dαis V(φ[z/xα], sβ), where φ[z/xα]is that assignment such
that φ[z/xα](xα) = zand φ[z/xα]yβ=φyβwhen yβ6=xα.
If an interpretation H=h{Dα}α∈T, Iiis an Henkin model the function V
is uniquely determined and V(φ, sα)∈Dαis called the denotation of sα.
Definition 5 (Standard model).H=h{Dα}α∈T, I iis called a standard model
if and only if for all αand β,Dαβis the set of all functions from Dαinto Dβ.
Obviously each standard model is also a Henkin model.
Definition 6 (Validity).A formula cof HOL is valid in a Henkin model Hif
and only if V(φ, c) = Tfor all variable assignments φ. In this case we write
H|=H OL c.cis (Henkin) valid, denoted as |=H OL c, if and only if H|=HOL c
for all Henkin models H.
The following theorem verifies that the logical connectives behave as in-
tended. The proof is straightforward.
Theorem 1. Let Vbe the valuation function of a Henkin model H. The follow-
ing properties hold for all variable assignments φ, terms so, to, lα, rα, and vari-
ables xα, wα:V(φ, ¬so) = Tif and only if V(φ, so) = F,V(φ, so∨to) = Tif and
only if V(φ, so) = Tor V(φ, to) = T,V(φ, ∀xα.so) = V(φ, Π(αo)oλxα.so) = T
if and only if for all v∈Dαholds V(φ[v/wα],(λxαso)wα) = T, and if lα=βη rα
then V(φ, lα) = V(φ, rα).
A HOL formula cthat is Henkin-valid is obviously also valid in all standard
models. Thus, when a Henkin-sound theorem prover for HOL finds a proof
for c, then we know that cis also theorem in standard semantics. More care
has to be taken when model finders for HOL return models or countermodels,
since theoretically these models could be non-standard. In practice this has not
been an issue, since the available model finders for HOL so far return finite
models only, and finite models in HOL are known to be standard [41]. Most
importantly, however, the returned models in Isabelle/HOL can always be
inspected by the user.
13
3. Theories of normative reasoning covered by the LogiKEy approach
Before explaining what theories of normative reasoning are covered by the
SSE approach, we briefly survey the area of deontic logic. The logics described
below are all candidates for LogiKEy layer L1.
3.1. Deontic logic
Deontic logic [46, 3, 5] is the field of logic that is concerned with normative
concepts such as obligation, permission, and prohibition. Alternatively, a de-
ontic logic is a formal system capturing the essential logical features of these
concepts. Typically, a deontic logic uses Op to mean that “it is obligatory that
p”, or “it ought to be the case that p”, and P p to mean that “it is permitted,
or permissible, that p”. Deontic logic can be used for reasoning about norma-
tive multiagent systems, i.e., about multiagent organizations with normative
systems in which agents can decide whether to follow the explicitly represented
norms, and the normative systems specify how, and to which extent, agents can
modify the norms. Normative multiagent systems need to combine normative
reasoning with agent interaction, and thus raise the challenge to relate the logic
of normative systems to aspects of agency.
There are two main paradigms in deontic logic, which we briefly describe in
the next two subsections.
3.1.1. Modal logic paradigm
Traditional (or “standard”) deontic logic (SDL) is a normal propositional
modal logic of type KD, which means that it extends the propositional tautolo-
gies with the axioms K:O(p→q)→(Op →Oq) and D:¬(Op ∧O¬p), and it is
closed under the inference rules modus ponens p, p →q/q and generalization or
necessitation p/Op. Prohibition and permission are defined by F p =O¬pand
P p =¬O¬p. SDL is an unusually simple and elegant theory. An advantage of
its modal-logical setting is that it can easily be extended with other modalities
such as epistemic or temporal operators and modal accounts of action.
Dyadic deontic logic (DDL) introduces a conditional operator O(p/q), to be
read as “it ought to be the case that p, given q”. A number of DDLs have been
proposed to deal with so-called contrary-to-duty (CTD) reasoning, cf. [21] for
an overview on this area. In brief, the CTD problem is mostly about how to
represent conditional obligation sentences dealing with norm violation, and an
example is provided Sect. 6.1. Two landmark DDLs are the DDL proposed by
Hansson [47], ˚
Aqvist [48, 49] and Kratzer [50], and the one proposed by Carmo
and Jones [21, 51]. A notable feature of the DDL by ˚
Aqvist is that it also
provides support for reasoning about so-called prima facie obligations [52]. A
prima facie obligation is one that leaves room for exceptions. A limitation of
the DDL by Carmo and Jones has been pointed out by Kjos-Hanssen [53].
To enable ethical agency a model of decision needs to be integrated in the
deontic frames. Horty’s deontic STIT logic [54], which combines deontic logic
with a modal logic of action, has been proposed as a starting point. The semantic
14
condition for the STIT-ought is a utilitarian generalization of the SDL view that
“it ought be that A” means that Aholds in all deontically optimal worlds.
3.1.2. Norm-based paradigm
The term “norm-based” deontic logic has been coined by Hansen [55] to refer
to a family of frameworks analysing the deontic modalities not with reference
to a set of possible worlds, some of them being more ideal than others, but with
reference to a set of explicitly given norms. In such a framework, the central
question is: given some input (e.g., a fact) and a set of explicitly given condi-
tional norms (a normative system), what norms apply? Thus, the perspective is
slightly different from the traditional setting, focusing on inference patterns [56].
Examples of norm-based deontic logics include the input/output (I/O) logic
of Makinson & van der Torre [57], Horty’s theory of reasons [58], which is
based on Reiter’s default logic, and Hansen’s logic of prioritized conditional
obligations [59, 55]. This proposed classification of paradigms (norm-based vs,
modal logic) is not meant to be exhaustive or exclusive. Some frameworks, like
adaptive deontic logic [60, 61], combine the two.
Our own work so far has focused on I/O logic. It can be viewed as a rule-
based system. The knowledge base takes the form of a set of rules of the form
(a,b) to be read as “if athen b”. The key feature of I/O logic is that it uses an
operational semantics, based on the notion of detachment, rather than a truth-
functional one in terms of truth-values and possible worlds. On the semantical
side, the meaning of the deontic concepts is given in terms of a set of procedures,
called I/O operations, yielding outputs (e.g., obligations) for inputs (facts). On
the syntactical side, the proof-theory is formulated as a set of inference rules
manipulating pairs of formulas rather than individual formulas. The framework
supports functionalities that are often regarded as characteristic of the legal
domain, and thus required to enable effective legal reasoning. We list below the
two most elementary requirements that can be expected of a framework, if it is
to deal with the legal domain; they are taken from Palmirani & colleagues [62].
1. Support for the modeling of constitutive rules, which define concepts or
constitute activities that cannot exist without such rules (e.g., legal defi-
nitions such as “property”), and prescriptive rules, which regulate actions
by making them obligatory, permitted, or prohibited.
2. Implementation of defeasibility [63, 64]; when the antecedent of a rule is
satisfied by the facts of a case (or via other rules), the conclusion of the
rule presumably holds, but is not necessarily true.
Other norm-based frameworks provide support for these functionalities. How-
ever, we have not covered them yet.
3.2. Theories of normative reasoning implemented
The following theories of normative reasoning have been “implemented” by
utilising the SSE approach.
15
SDL: All logics from the modal logic cube, including logic KD, i.e. SDL, have
meanwhile been faithfully implemented in the SSE approach [33]. These
implementations scale for first-order and even higher-order extensions.
DDL: the DDL by ˚
Aqvist [48, 49] and the DDL by Carmo and Jones [51]:
Faithful SSEs of these logics in Isabelle/HOL are already available [8,
10], and most recently the ATP system Leo-III has been adapted to
accept DDL as input [65, 66].
I/O logic [57]: The main challenge comes from the fact that the framework
does not have a truth-functional semantics, but an operational one. First
experiments with the SSE of the I/O-operator out 1(called simple-minded)
and out2(called basic) in Isabelle/HOL have been presented in the
literature [9, 67].
Some relevant I/O logic variants have very recently been studied [68, 69],
and we conjecture that some of these variants are related to certain non-normal
modal logics, e.g., conditional logics with a selection function semantics or sim-
ilar logics with a neighbourhood semantics. However, the embedding of such
logics has already been studied in the first authors previous work [70, 71]. It
should thus be possible to benefit in the exploration of the conjectured relation-
ship from these existing results in the given context.
4. Sample LogiKEy embedding: ˚
Aqvist’s system E in HOL
In this section, to illustrate our approach, we describe our embedding of
˚
Aqvist [48]’s dyadic deontic logic in HOL. The system is called E.
We give this example, because all the development steps foreseen at layer L1
of LogiKEy (cf. Sect. 7) have been caried out for this system. In particular, the
embedding has been shown to be faithfull—this is one of our key success criteria
at layer L1. Moreover, this logic has been tested against available benchmarks,
like those related to CTD reasoning. For sure, one does not need automated
theorem provers to see that the system can handle this type of reasoning while
SDL cannot. The treatment of CTD reasoning motived DDL in the first place.
However, from pen and paper results alone one cannot conclude that an imple-
mentation of a logic will result in a practically useful system. Such an orthogonal
question must be answered by empirical means, in particular, when first-order
or higher-order extensions, or combinations with other logics, are required. Our
SSE-based implementation of system Escales for such extensions; a first-order
extension will later be tested with a simple CTD example structure in Sect. 6.1.
4.1. Target logic: System E
Definition 7. The language of Eis generated by the following BNF:
φ::= p| ¬φ|φ∧φ|φ| (φ/φ)
16
φis read as “φis settled as true”, and (ψ/φ) as “ψis obligatory, given φ”.
φ(“φis unconditionally obligatory”) is short for (φ/>).
Traditionally so-called preference models are used as models for the language.
Definition 8. A preference model M= (W, , V )is a tuple where:
•Wis a (non-empty) set of possible worlds;
•is a binary relation over Wordering the worlds according to their bet-
terness or comparative goodness; stis read as “sis at least as good as
t”; is called a preference or betterness relation;9
•Vis a valuation assigning to each propositional letter pa set of worlds,
namely the set of those worlds at which pholds.
For E, no specific properties are assumed of the preference or betterness
relation . It is known that the assumptions of reflexivity and totalness (every
worlds are pairwise comparable) do not “affect” the logic; cf. Theorem 2.
Intuitively the evaluation rule for (ψ/φ) puts (ψ/φ) true if and only if
the best φ-worlds, according to , are all ψ-worlds. Formally:
Definition 9 (Satisfaction).Given a preference model M= (W, , V )and a
world s∈W, we define the satisfaction relation M, s φas usual, except for
the following two new clauses
•M, s φiff for all tin M,M , t φ
•M, s (ψ/φ)iff opt(||φ||)⊆ ||ψ||
where kφkis the set of worlds at which φholds and opt(||φ||)is the subset of
those that are optimal according to :
opt(kφk) = {s∈ kφk | ∀t(tφ→st)
Definition 10 (Validity).A formula φis valid in the class Pof all preference
models (notation: |=Pφ) if and only if, for all preference models Mand all
worlds sin M,M, s |=φ.
Definition 11. E is the proof system consisting of the following axiom schemata
9“Preference relation” is a generic name used in different areas like the areas of conditional
logic, rational choice theory, non-monotonic logic, and deontic logic. “Betterness relation” is
the name used in deontic logic. This choice of name is dictated by the intuitive interpretation
given to the preference relation.
17
and rule schemata (the names are taken from Parent [72]):
φ, where φis a tautology from PL (PL)
(φ→ψ)→(φ→ψ) (K)
φ→φ(4)
¬φ→¬φ(5)
(ψ→χ/φ)→((ψ/φ)→ (χ/φ)) (COK)
(φ/φ) (Id)
(χ/(φ∧ψ)) → ((ψ→χ)/φ) (Sh)
(ψ/φ)→(ψ/φ) (Abs)
ψ→ (ψ/φ) (Nec)
(φ↔ψ)→((χ/φ)↔ (χ/ψ)) (Ext)
If `φand `φ→ψthen `ψ(MP)
If `φthen `φ(N)
The notions of theorem and consistency are defined as usual.
The following theorem resolves a long-standing open problem in deontic
logic. In the work of ˚
Aqvist, cf. [73, pp. 179–181] and [48, pp. 247–249] the
question of whether Eis complete with respect to its intended modeling has
been left as an open problem; it has been recently been answered by Parent
[49]. See Goble [74] for a similar axiomatization result of Hansson’s original
models.
The second and third clauses in the statement of Theorem 2 mean that the
assumptions of reflexivity and totalness do not have any impact on the logic.
The fact that these two properties are idle remains largely unnoticed in the
literature.
Theorem 2. E is sound and complete with respect to the following three classes
of preference models:
1. the class of all preference models;
2. the class of preference models in which the betterness relation is required
to be reflexive;
3. the class of preference models in which the betterness relation is required
to be total (for all sand t, either stor ts).
Proof. The proof can be found in Parent [49].
Theorem 3. The theoremhood problem in E(“Is φa theorem in E?”) is de-
cidable.
Proof. The proof can be found in Parent [75].
Stronger systems may be obtained by adding further constraints on , like
transitivity and the so-called limit assumption, which rules out infinite sequences
of strictly better worlds.
18
4.2. Embedding of Ein HOL
The formulas of Eare identified in our SSE with certain HOL terms (predi-
cates) of type io, where terms of type iare assumed to denote possible worlds
and odenotes the (bivalent) set of Booleans.
That is, the HOL type iis now identified with a (non-empty) set of worlds.
Type iois abbreviated as τin the remainder. The HOL signature is assumed
to contain the constant symbol Riτ. Moreover, for each propositional symbol
Pjof E, the HOL signature must contain the corresponding constant symbol
Pj
τ. Without loss of generality, we assume that besides those symbols and the
primitive logical connectives of HOL, no other constant symbols are given in
the signature of HOL.
Definition 12. The mapping b·c translates a formula ϕof Einto a formula
bϕcof HOL of type τ. The mapping is defined recursively in the usual way [24],
except for the following two new clauses:
bφc=ττbφc
b(ψ/φ)c=τττbψcbφc
where ττand τττabbreviate the following formulas of HOL:
ττ=λφτ.λxi.∀yi.(φ y)
τττ=λφτ.λψτ.λxi.∀wi.((λvi.(φ v ∧(∀yi.(φ y →Riτv y)))) w→ψ w)
The basic idea is to make the modal logic’s possible worlds structure explicit
by introducing a distinguished predicate symbol Rto represent the preference
or betterness relation , and to translate a formula directly according to its
semantics. For instance (ψ/φ) translates into
λxi.((∀wi.(φw ∧(∀yi.(φy →Rwy)) →ψw))
Definition 13 (Validity of an embedded formula).Global validity (vld) of an
embedded formula φof Ein HOL is defined by the equation
vld bφc=∀zi.bφcz
For example, checking the global validity of (ψ/φ) in Eis hence reduced
to checking the validity of the formula
∀wi.(φw ∧(∀yi.(φy →Rwy)) →ψw)
in HOL.
This definition is hidden from the user, who can construct now deontic logic
formulas involving (ψ/φ) and use them to represent and prove theorems.
19
4.3. Faithfulness of the embedding
It can be shown that the embedding is faithful, in the sense given by Thm. 4.
Remember that the establishment of such a result is our main success criterium.
Intuitively, Thm. 4 says that a formula φin the language of Eis valid in the
class of all preference models if and only if its translation bφcin the language
of HOL is valid in the class of Henkin models in the sense of Def. 13.
Theorem 4 (Faithfulness of the embedding).
|=Pφif and only if |=HOL vld bφc
Proof. The proof can be found in Benzm¨uller et al. [8]. The crux of the ar-
gument consists in relating preference models with Henkin models in a truth-
preserving way.
4.4. Encoding in Isabelle/HOL
The practical employment of the above SSE for Ein Isabelle/HOL is
straightforward and can be done in a separate theory file. This way, for a con-
crete application scenario, we can simply import the embedding without dealing
with any technical details. The complete embedding is quite short (approx. 30
lines of code with line breaks) and is displayed in Fig. 6.
The embedding has been extended to include quantifiers as well. The possi-
bilist quantifiers (cf. Sect. 2.1) are introduced in lines 13–17—this amounts to
having a fixed domain of individuals rather than a world-relative domain (the
actualist semantics). If needs be, actualist quantifiers can also be introduced in
a few lines of code; see the study [24], where analogous definitions of both pos-
sibilist and actualist quantifiers are presented for higher-order conditional logic.
The betterness relation Ris introduced as an uninterpreted constant symbol in
line 20, and the conditional obligation operator is defined in line 26. Its defi-
nition mirrors the one given in Def. 12. The unconditional obligation operator
is introduced in line 27. It is defined in terms of its dyadic counterpart in the
usual way. Last, global and local validity (cf. Sect. 2.1) are introduced in lines
30–33. Lines 35–40 show sample queries. On line 35, consistency of the em-
bedding is confirmed. Lines 37–40 illustrate how Isabelle/HOL can be used
as a heuristic tool in correspondence theory. The focus is on the assumption of
transitivity of the betterness relation. One would like to know what its syntac-
tical counterpart is. Isabelle/HOL confirms that such an assumption has the
effect of validating the axiom Lewis [76] called CV (line 39). Isabelle/HOL
also confirms that transitivity is not equivalent to CV: the model finder Nitpick
[37] integrated with Isabelle/HOL finds a model validating CV in which the
betterness relation is not transitive (lines 42–44).
5. LogiKEy tool support
5.1. Support for different reasoning tasks
In a nutshell, a reasoner is a tool that can perform reasoning tasks in a
given application domain. Reasoning thereby refers to the process of deriving
20
Figure 6: Embedding of the semantics of system Ein Isabelle/HOL.
21
or concluding information that is not explicitly encoded in the knowledge base.
Which information is derivable and which is not is thereby dependent on the
particular choice of logic. The reasoning tasks that are particularly relevant in
our context, for example, include:
•Compliance checking: Is the current situation, respectevily, an intended
action by an IAS in a given context, compliant with a given regulation (a
set of formally represented norms)?
•Non-compliance analysis: If non-compliance is the result of a compliance
check, can the reasons be revealed and explained?
•Entailment checking: Does such-and-such obligation or legal interpreta-
tion follow from a given regulation?
•Non-entailment analysis: If entailment checking fails, can the reasons be
revealed and explained?
•Consistency checking: Is a given regulation consistent? Is such-and-such
norm, as part of a given regulation, consistent with this other set of norms,
stemming from another regulation? Is such-and-such legal interpretation
consistent with another one?
•Inconsistency analysis: If consistency checking fails, can a minimal set of
conflicting norms be revealed and the inconsistency be explained?
We take consistency in its usual sense in classical logic. A set of formulas
is consistent, if there exists a model that satisfies all of its members, and in-
consistent otherwise. Thus, {p, q }is consistent, but {p, ¬p}is not. Likewise,
{p, ¬p}is consistent in a system of deontic logic accommodating the exis-
tence of conflicts, but {p, ¬ p}is not.
Consistency checking, non-compliance analysis and non-entailment analysis
are well supported by model finders, respectively counter-model finders, while
the other tasks generally require theorem proving technology. A powerful de-
ontic reasoner should thus ideally provide both (counter-)model finding and
theorem proving. Moreover, intuitive proof objects and adequate presentations
of (counter-)models are desirable to enable user explanations.
5.2. Flexibility along different axes
While the above reasoning tasks are comparably easy to provide for many
decidable propositional fragments of deontic logics, it becomes much less so
for their quantified extensions. We are not aware of any system, besides the
work presented in this article, that still meets all these requirements for some
CTD compliant first-order or higher-order deontic logics; cf. the related work in
Sect. 8.
Our approach addresses challenges along different axes, which are motivated
by the following observations:
22
Different logics: The quest for “a most suitable deontic logic” is still open, and
perhaps it will remain so for quite some time. I/O logics are favoured in
our group, but we also study and implement alternative proposals, such
as the DDL by Hansson or the one by Carmo and Jones. Perhaps there is
no single best solution, and different deontic logics qualify best in different
application contexts.
Expressivity levels: It is highly questionable whether normative knowledge and
regulatory texts can often be abstracted and simplified to the point that
pure propositional logic encodings are feasible and justified. For example,
it can be doubted that the encoding of Gewirth’s “Principle of Generic
Consistency”, which we outline in Sect. 6.2, can be mechanized also in
a propositional setting without trivialising it to a point of vacuity. The
need for quantified deontic logics is also evidenced by related work such
as Govindarajulu and Bringsjord’s encoding of the “Doctrine of Double
Effect” [77].
Logic combinations: In concrete applications normative aspects often meet
with other challenges that can be addressed by suitable logic extensions,
respectively, by combining a deontic logic with other suitable logics as
required. An example is provided again by the encoding of the Gewirth’s
ethical theory as outlined in Sect. 6.2.
These issues should be addressed, utilising the LogiKEy methodology, in
empirical studies in which the different choices of logics, different expressivity
levels and different logic combinations are systematically compared and assessed
within well selected application studies. However, for such empirical work to be
feasible, implementations of the different deontic candidate logics, and its com-
binations with other logics, have to be provided first, both on the propositional
level and ideally also on the first-order and higher-order level. Moreover, it is
reasonable to ensure that these implementations remain comparable regarding
the technological foundations they are based on, since this may improve the
fairness and the significance of conceptual empirical evaluations.
Figure 4 well illustrates the different components and aspects that we con-
sider relevant in our work. Different target logics (grey circles) and their com-
binations are provided in the higher-order meta-logic of the host system. In
our case this one is either the proof assistant Isabelle/HOL or the higher-
order ATP system Leo-III. In provided target logics different ethical or legal
theories can then be encoded. Concrete examples are given in Sect. 6. This
set-up enables a two-way fertilization. On the one hand, the studied theories
are themselves assessed, e.g., for consistency, for entailed knowledge, etc. On
the other hand the properties of the different target logics are investigated. For
example, while one of the target logics might suffer from paradoxes in a con-
crete application context, another target logic might well be sufficiently stable
against paradoxes in the same application context. An illustration of this aspect
is given in Sect. 6.1. After arriving at consistent formalizations of an ethical
23
or legal theory in a suitable logic or suitable combination of logics, empirical
studies can be conducted.10
6. LogiKEy case studies
In this section we shift our attention to the encoding of ethico-legal theories
at LogiKEy layer L2. We outline and selectively discuss two respective case
studies.
6.1. Data protection
In our first case study the legal theory of interest is the General Data Protec-
tion Regulation (GDPR, Regulation EU 2016/679). It is a regulation by which
the European Parliament, the Council of the European Union and the European
Commission intend to strengthen and unify data protection for all individuals
within the European Union. The regulation became enforceable from 25 May
2018.
First-order logic has been identified so far as a good level of abstraction (in
the case study in 6.2, in contrast, higher-order logic is used). Given respec-
tive first-order extensions of deontic logics, the interest then is to practically
assess their correctness and reasoning performance in context, and to compare
the outcomes of such tests with our expectations. Below we illustrate how such
practical assessments can be conducted within the interactive user-interface of
Isabelle/HOL. As an illustrating example we present a concrete CTD struc-
ture that we revealed in the context of the GDPR; for readers not yet familiar
with CTD structures this example may also be useful from an educational view-
point.
The proper representation of CTD structures is a well-known problem in the
study of deontic logics. CTD structures refer to situations in which there is a
primary obligation and, additionally, what we might call a secondary, contrary-
to-duty obligation, which comes into effect when the primary obligation is vio-
lated.11 The paradox arises when we try to symbolize certain intuitively consis-
tent sets of ordinary language sentences, sets that include at least one CTD obli-
gation sentence, by means of ordinary counterparts available in various monadic
10 To this end the agent-based LeoPARD framework [78, 79], which is underlying the Leo-
III [20] prover, will be adapted and utilized in future work to, e.g., embody upper ethical
theories in virtual agents and to conduct empirical studies about the agents behaviour when
reasoning with such upper principles in a simulated environment. The prover Leo-III is
itself implemented as a proactive agent in the LeoPARD framework, which internally utilizes
another copy/instance of the LeoPARD framework for the organization and orchestration of
its distributed reasoning approach, where it collaborates, e.g., with other theorem provers that
are modelled as proactive agents. In other words, Leo-III already is a pro-active agent within
an agent-based framework, and it already comes with deontic logic reasoning support. The
idea is now to populate multiple instances of such pro-active agents with selected normative
theories within the LeoPARD framework, to initialize these agent societies with carefully
selected tasks, and to subsequently study their interaction behaviour within this controlled
environment.
11The problem was first pointed out by Chisholm [80] in relation with SDL.
24
deontic logics, such as SDL and similar systems. The formal representations of-
ten turn out to be inconsistent, in the sense that it is possible to deduce contra-
dictions from them, or else they might violate some other intuitively plausible
condition, for example that the sentences in the formalization should be inde-
pendent of each other. It is not the purpose of this article to discuss in any
greater depth the paradox. The interested reader should consult, e.g., Carmo
and Jones [21].
Here is the CTD structure we revealed in the context of the GDPR:
1. Personal data shall be processed lawfully (Art. 5). For example, the
data subject must have given consent to the processing of his or her
personal data for one or more specific purposes (Art. 6/1.a).
2. If the personal data have been processed unlawfully (none of the
requirements for a lawful processing applies), the controller has
the obligation to erase the personal data in question without de-
lay (Art. 17.d, right to be forgotten).
When combined with the following a typical CTD-structure is exhibited.
3. It is obligatory, e.g., as part of a respective agreement between a
customer and a company, to keep the personal data (as relevant to
the agreement) provided that it is processed lawfully.
4. Some data in the context of such an agreement has been processed
unlawfully.
The latter information pieces are not part of the GDPR. (3) is a norm coming
from another regulation, with which the GDPR has to co-exists. (4) is a factual
information—it is exactly the kind of world situations the GDPR wants to
regulate.
Figure 6.1 illustrates the problem raised by CTD scenarios, when the infer-
ence engine is based on SDL. The knowledge base is encoded in lines 7–25. The
relevant obligations and the assumed situation as described by (1)–(4) above are
formalized in lines 18–25. Subsequently, three different kinds of queries are an-
swered by the reasoning tools integrated with Isabelle/HOL. The first query
asks whether the encoded knowledge base is consistent, and Nitpick answers
negatively to this question in line 28. The failed attempt to compute a model
is highlighted in pink. The second query asks whether falsum is derivable, and
Isabelle’s prover metis returns a proof within a few milliseconds in line 29.
Notice that the proof depends on the seriality axiom Dof SDL, which is im-
ported from SDL’s encoding in file SDL.thy–that is not shown here. The query
in line 36 asks whether an arbitrarily weird and unethical conclusion such as the
obligation to kill Mary follows, and the prover answers positively to this query.
These results are clearly not desirable, and confirm the need to use a logic
other than SDL for application scenarios in which norm violation play a key
role.
25
Figure 7: Failed analysis of the GDPR example in SDL.
26
Figure 8: Successful analysis of the GDPR example scenario in system E.
Fig. 8 shows that our SSE based implementation of system Eis in contrast
not suffering from this effect. The prescriptive rules of the GDPR scenario are
modelled in lines 17–22:
A1:∀d(process lawfully d/is protected by GDPR d)
A2:∀d(erase d/is protected by GDPR d ∧ ¬process lawfully d)
Implicit: ∀d(¬erase d/is protected by GDPR d ∧process lawfully d)
The current situation, in which we have that Peter’s personal data d1 are not
processed lawfully, is defined in line 24:
S it u at io n : ¬process lawfully d1
The three same queries as before are run, but this time with success. In line
28 we are told by Nitpick that the knowledge base has a model. The computed
model can be inspected in full detail a separate window. In line 29 we are
told that falsum is no longer derivable and the theorem provers integrated
with sledghammer terminate with a time-out. In line 30 we are told that the
obligation to kill Mary no longer follows.
27
These practical experiments with theorem provers and model finders demon-
strate that a machine can indeed reason about norm violation in a first-order
extension of DDL at run-time, resp. on the fly. Future research will study the
practical deployment of the LogiKEy flexible reasoning technology, for exam-
ple, for ethico-legal governance of autonomous cars, smart homes, social robots,
etc. Given these ambitious goals, it is relevant to confirm early-on that the
LogiKEy solution, without further ado, can already handle simple examples
as discussed here.12 This, of course, provides only a relevant first assessment
step. The development of respective libraries of increasingly complex bench-
marks, see also Sect. 6.2, is therefore an important objective in our future work.
6.2. Gewirth’s Principle of Generic Consistency
In a second case study our framework has been adapted and utilized for the
exemplary mechanization and assessment [12, 13] of Alan Gewirth’s Principle
of Generic Consistency (PGC) [30, 82] on the computer; the PGC is an ethical
theory that can be understood as an emendation of the well known golden
rule. The formalization of the PGC, which is summarized below, demonstrates
that intuitive encodings of ambitious ethical theories and their mechanization,
resp. automation, on the computer are no longer antipodes.
Some further information on the PGC is given in Fig. 9, including a depic-
tion of the modeling decisions taken for its formalization. The self-contained
Isabelle/HOL sources of the formalization are available in the Archive of For-
mal Proofs [32], and an excerpt of the derivation and verification of the main
inference steps of the argument is presented in Fig. 10.
The argument used by Gewirth to derive the PGC is by no means trivial
and has stirred much controversy in legal and moral philosophy during the last
decades. It has also been discussed in political philosophy as an argument for
the a priori necessity of human rights. Perhaps more relevant for us, the PGC
has lately been proposed as a means to bound the impact of artificial general
intelligence [84].
Gewirth’s PGC is taken here as an illustrative showcase to exemplarily assess
its logical validity with our normative reasoning machinery, and the success of
Fuenmayor’s and Benzm¨uller’s work [12, 13] provides evidence for the claims
made in this article. It is the first time that the PGC has been formalized
and assessed on the computer at such a level of detail, i.e., without trivializing
it by abstraction means, and while upholding intuitive formula representations
and user-interaction means. As a side-effect, the ATP systems integrated with
Isabelle/HOL have helped in this work to reveal and fix some (minor) issues
in Gewirth’s argument.
12For example, Meder, in his MSc thesis [11], worked out an SSE for a variant of STIT-
logic put forth by Lorini [81], called T-STIT, for temporal STIT. It was found out that
model and (counter-)model finding wasn’t responsive for analogous experiments. The reason,
however, was quickly revealed: the T-STIT-logic, by nature, requires infinite models, while
the (counter-)model finding tools in Isabelle/HOL so far explore finite model structures only.
28
Alan Gewirth’s “Principle of Generic Consistency (PGC) [30, 82], consti-
tutes, loosely speaking, an emendation of the Golden Rule, i.e., the princi-
ple of treating others as one’s self would wish to be treated. Adopting an
agent perspective, the PGC expresses and deductively justifies a related up-
per moral principle, according to which any intelligent agent, by virtue of its
self-understanding as an agent, is rationally committed to asserting that it
has rights to freedom and well-being, and that all other agents have those
same rights. The main steps of the argument are (cf. [82] and [12]):
(1, premise) I act voluntarily for some purpose E, i.e., I am a prospective
purposive agent (PPA).
(2, derived) E is (subjectively) good (i.e. I value E proactively).
(3, derived) My freedom and well-being (FWB) are generically necessary
conditions of my agency (I need them to achieve any purpose whatsoever).
(4, derived) My FWB are necessary goods (at least for me).
(5, derived) I have (maybe nobody else) a claim right to my FWB.
(13, final conclusion) Every PPA has a claim right to their FWB.
Formalization of the argument is challenging; it faces complex linguistic ex-
pressions such as alethic and deontic modalities, quantification and index-
icals. The solution of Fuenmayor and Benzm¨uller [12, 13], cf. Fig. 10 for
an excerpt, is based on the following modeling decisions: FWB and Good are
introduced as unary uninterpreted predicate symbols; further uninterpreted
relation symbols are added: ActsOnPurpose,InterferesWith (both 2-ary)
and NeedsForPurpose (3-ary). PPA ais defined as ∃E. ActsOnPurpose a E ; an
additional axiom postulates that being a PPA is identity-constitutive for any
individual: b∀a.PPAa→D(PPAa)cD.bϕcDin there models indexical validity
of ϕ; it is defined following Kaplan’s logic of demonstratives [83] as being true
in all contexts. RightTo a ϕ is defined as Oi(∀b.¬InterferesWith b(ϕ a));
this captures that an individual ahas a (claim) right to some property ϕiff it
is obligatory that every (other) individual bdoes not interfere with the state
of affairs (ϕ a). Oiis defined as the ideal obligation operator from Carmo
and Jones [21] (their actual obligation operator Oacould be used as well).
The meaning of the uninterpreted constant symbols is constrained by adding
further axioms: e.g., axioms that interrelate the concept of goodness with
the concept of agency, or an axiom b∀P.∀a. NeedsForPurpose aFWB PcDthat
expresses that FWB is always required in order to act on any purpose, whereas
FWB is postulated to be a contingent property (b∀a.♦pFWB a∧♦p¬FWB acD).
Note that both first-order and higher-order quantifiers are required; cf. the
“∀P.∀a.”-prefix in the previous axiom, where Pranges over properties and a
over individuals.
Figure 9: Gewirth’s Priciple of Generic Conistency, and its formal encoding [12], in a nutshell
29
To encode the PGC an extended embedding of the DDL by Carmo and
Jones [21] in HOL was employed. Conditional, ideal and actual obligation has
been combined, among others, with further modalities, with Kaplan’s notion of
context-dependent logical validity, and with both first-order and higher-order
quantifiers.13
What can be seen in Fig. 10 is that readable formula presentations are sup-
ported, which in turn enable quite intuitive user-interactions in combination
with proof automation means. It is demonstrated that the interactive assess-
ment of Gewirth’s argument is supported at an adequate level of granularity:
the proof steps as presented in Fig. 10 correspond directly to actual steps in
Gewirth’s argument and further details in the verification process can be dele-
gated to automated reasoning tools provided in Isabelle.
Proof automation can be pushed further, but this has not been the focus
so far in this work. The result of this work is a formally verified derivation
of the statement “Every prospective purposive agent has a claim right to their
free will and well-being” from Gewirth’s elementary premises, axioms and defi-
nitions in the theorem prover Isabelle (at LogiKEy layer L2, cf. Fig. 2). The
formalization of this ethical theory in combination with the mechanization and
automation of its underlying non-trivial, logic combination (at LogiKEy layer
L1) enables further experiments with the theory and provides an entry for de-
vising applications of it (at LogiKEy layer L3). One possibility in this regard
is the population of this theory within virtual agents, so that the behaviour of
each agent is ultimately constraint, if no other regulatory rules apply, by this
very general moral principle; cf. Footote 10.
7. Logic & knowledge design and engineering methodology
Now that we have explained and illustrated various ingredients and aspects
of LogiKEy, we will, in this section, present and discuss the LogiKEy logic
& knowledge design and engineering methodology methodology in more detail.
Remember that LogiKEy distinguishes between the following three layers; they
were visualized earlier in Fig. 2.
L1 Logics and logic combinations. Logics and logic combinations are mod-
eled, automated and assessed at layer L1. The automation of CTD com-
pliant normative reasoning is thereby of particular interest, where logical
explosion is avoided even in situations when an intelligent autonomous
system (IAS) accidentally or deliberately violates some of its obligations.
We thus prefer a proper intra-logical handling of the CTD challenge in
our work over tackling this challenge, e.g., with extra-logical and ad hoc
13Kaplan’s context-sensitive logic of demonstratives covers the dimension of agency, among
others. Future work includes the study of alternative modelings, utilising, e.g., the modal logics
of agency or STIT logic. STIT logic, however, gives rise to interesting problems, including
infinite models as already noted. The infinite character is fixed in the axioms.
31
means. Sect. 3 presented respective examples of CTD compliant logics—
the dyadic deontic logic (DDL) of Carmo and Jones [21] and ˚
Aqvist’s
system E[48, 49]—and Sect. 4 explained how system Eis modelled and
automated adopting the LogiKEy methodology.
L2 Ethical & legal domain theories. At layer L2 concrete legal and ethical
theories are modeled, automated and assessed — exploiting the logic and
logic combinations from layer L1. Question answering, consistency and
compliance checking support is provided. Our particular interest is in legal
and ethical theories suited to govern the behaviour of IASs. Examples for
layer L2 developments were presented in Sect. 6, and tool support, for L2
and the other layers, was discussed in Sect. 5.
L3 Applications. Layer L3 addresses the deployment of the ethical & legal
domain reasoning competencies, e.g., for regulating the behaviour of IASs.
This requires the implantation of L1- and L2-reasoning capabilities within
a suitably designed governance component for the IAS to be regulated.
Concrete engineering steps for layers L1, L2 and L3 are presented below, and
pointers to directly related work and illustrating examples are provided. The
LogiKEy methodology has been developed, refined and tested in prior research
projects and also in lecture courses at Freie Universit¨at Berlin and at University
of Luxembourg. We have thus gained experience that LogiKEy well supports
both research and education [85] at the depicted layers.
While each layer in the methodology can serve as an entry point, we here
start our systematic exhibition at layer L1. In concrete research and education
projects, one may alternatively start from layers L2 or L3; the formalization of
Gewirth’s PGC, for example, started at layer L2. Entry at layers L3 or L2 will
increasingly become the default, when our library of reusable logic encodings at
layer L1 grows, so that extensive engineering work at layer L1 can largely be
avoided in future applications.
7.1. Layer L1 – Logics and logic combinations
Logic: Select a logic or logic combination. Logic examples include DDL or sys-
tem E, cf. Sect. 3. The need for a non-trivial logic combination was, e.g.,
identified in the case study on Gewirth’s PGC [12]; cf. Sect. 6.2 and Fig. 9. As
explained in detail by Fuenmayor and Benzm¨uller [12], in this work a higher-
order extension of DDL was combined with relevant parts of Kaplan’s logic of
demonstratives (LD) [83, 86]. Moreover, a combination of multi-epistemic logic
with an operator for common knowledge was used in an automation of the Wise
Men Puzzle [24, 87].
Semantics: Select a semantics for the chosen logic or logic combinations. This
step is preparing the automation of the logic or logic combination, using the SSE
approach, in the next step. Suitable semantics definitions for DDL and system E
are given in the mentioned original articles. To arrive at a semantics for the logic
32
combination used in the PGC formalization, DDL’s original semantics was ex-
tended and combined, utilising a product construction [6, §1.2.2], with Kaplan’s
semantics for LD. In the work on the Wise Men Puzzle, a logic fusion [6, §1.2.1]
was used, and the semantics for multi-epistemic logic and common knowledge
were adopted from Sergot [88].
Automate: Automate the selected logics and logic combinations in HOL. In
this step a selected logic or logic combination is shallowly embedded by di-
rectly encoding its semantics in meta-logic HOL. This is practically supported
in theorem provers and proof assistant systems for HOL, such as Isabelle [19]
or Leo-III [20]. SSEs thus constitute our preferred solution to the automation
challenge, since they enable the reuse of already existing reasoning machinery.14
Assess: Empirically assess the SSEs with model finders and theorem provers.
The consistency of an SSE can be verified with model finders, and ATP systems
can be employed to show that the original axioms and proof rules of an em-
bedded logic can be derived from its semantics. Respective examples have been
presented in the literature [13, 89]. If problems are revealed in this assessment,
modifications in previous steps maybe required.
Faithfulness: Prove the faithfulness of the SSEs. Part of this challenge is compu-
tationally addressed already in the previous step. In addition, explicit soundness
and completeness proofs should be provided using pen and paper methods. An
example is Thm. 4 in Sect. 4, which proves the faithfulness of the SSE of system
Ein HOL. If faithfulness cannot be established, modifications in the first three
steps maybe required.
Implications: Explore the implications of the embedded logics and logic combina-
tions. What theorems are implied by an SSE? Are the expectations matched?
For the SSE of the combination higher-order DDL with Kaplan’s LD in HOL
it has, e.g., been checked [13], whether model finders still report countermodels
to propositional and first-order variants of the Chisholm paradox [80]. If the
investigated implications do not match the expectations, modifications in the
first three steps maybe required.
Benchmarks: Test the logic automation against benchmarks. Benchmark tests
not only rank LogiKEy proof automations against competitor approaches, they
also provide further evidence for the practical soundness and robustness of the
14However, also deep logic embeddings in HOL can be utilized, and even combinations of
both techniques are supported. The abstraction layers used by Kirchner, Benzm¨uller and
Zalta [89], for example, support a deep embedding of a proof calculus for higher-order hyper-
intensional (relational) logic S5 in HOL on top of an SSE for the same logic in HOL; both
reasoning layers can then been exploited for proof automation and mutual verification; cf. also
Footnote 6 and Sect. 8.
33
implemented technology; this is most relevant since pen and paper faithfulness
proofs provide no guarantee that an implementation is bug free.15
Contribute: Contribute to the built-up and extension of benchmark suites.
7.2. Layer L2 – Domain Theories
Select an ethical & legal domain theory of interest. An example is Gewirth’s
PGC from Sect. 6.2. Another example is the German road traffic act, which
we have started to work on in student projects as part of our lecture courses.
Alternatively, or in addition, one might be interested in designing, from scratch,
new sets of ethical rules to govern the behaviour of autonomous cars.
Analyse the ethical & legal domain theory. Mutually related aspects are ad-
dressed in this step, and requirements for the logics and logic combinations
imported from layer L1 are identified.
1. Determine a suitable level of abstraction. Can relevant notions and con-
cepts be suitably abstracted, e.g., to a purely propositional level (as often
done in toy examples in AI), or would that cause an oversimplification
regarding the intended applications at layer L3?
2. Identify basic notions and concepts. What are the most essential concepts
addressed and utilized in a given domain theory. Which basic entities
need to be explicitly referred to, and which properties, relations and func-
tions for such entities must be explicitly modelled? For example, notions
identified as relevant for Gewirth’s PGC, cf. Fig. 9, include the relations
ActsOnPurpose,NeedsForPurpose and InterferesWith, the predicates
Good (for Goodness) and FWB (free will and well-being), and the defined
terms RightTo (has right to) and PPA (is a prospective purposive agent).
As a result of this analysis, a signature is obtained together with a set of
associated foundational axioms and definitions.
In this step dependencies on very generic notions, such as mereological
terms, may be revealed whose precise meanings are left implicit. For the
formalization of such notions one may consult other sources, including
existing upper ontologies.16
3. Identify notions of quantification. Domain theories may contain univer-
sal and/or existential statements that cannot or should not be abstracted
15Respective comparisons of SSE provers for first-order modal logics with competitor sys-
tems have been presented in the literature [31, 90]. Since we are not aware of any imple-
mentation of first-order or higher-order DDLs besides our own work, we cannot yet conduct
analogous evaluation studies for our SSE-based DDL provers.
16Upper ontologies formally define very generic terms that are shared across several domains.
Import from external upper ontologies requires some conceptual and logical fit which often is
not given.
34
away. Careful assessment of the precise characteristics of each of the iden-
tified quantifiers is then advisable. In particular, when quantifiers interact
with linguistic modalities, see below, the question arises whether, e.g., the
Barcan formulas [91] should hold or not. Different kinds of quantifiers
may thus have to be provided at layer L1.
4. Identify linguistic modalities. Ethico-legal domain theories are challenged,
in particular, by deontic modalities (e.g., ”an entity is permitted/obliged
to . .. ”), and they may occur in combination with other modalities. In
Gewirth’s PGC, for example, deontic and alethic modalities are relevant.
Notions of time or space are further examples that frequently need to be
addressed. Combinations of modalities may thus have to be provided at
layer L1.
Determine a suitable logic or logic combination. The previous step identifies
essential logical requirements for the formalization task at hand. Based on these
requirements a suitable base logic or logic combination must be determined or
devised and imported from layer L1; if not yet provided, further work at layer
L1 is required.
Formalize the ethical & legal domain theory. During the formalization process
regular sanity checks, e.g., for in-/consistency or logical entailment, with ATPs
and model finders are advisable, cf. Sect. 5.1. This serves two different purposes:
early detection of misconceptions and errors in the formalization process, and
early testing of the proof automation performance. If the reasoning perfor-
mance is weak early on, then countermeasures maybe taken, for example, by
considering alternative choices in the previous steps.17
Explore theory implications and check whether expectations are matched. If the
computationally explored and assessed implications of the formalized domain
theory are not matching the expectations, modifications in one of the previous
steps are required.18
Contribute: Contribute to the built-up of benchmark suites for domain theories.
7.3. Layer L3 – Applications
Layer L3 deploys the ethico-legal domain theories from layer L2 in practical
applications, e.g., to regulate the behaviour of an IAS.
17Remember Footnote 12, where respective issues for T-STIT-logic where identified by such
experiments.
18Respective experiments have been conducted, e.g., in the context of the formalisation of
G¨odel’s ontological argument. In these experiments it has been confirmed with automated
reasoning tools that monotheism and the modal collapse (expressing ”there are no contingent
truths”, ”there is no free will”) were implied by G¨odel’s theory; cf. [92]. Both implications
might not be in-line with our expectations or intentions, and the modal collapse has in fact
motivated further emendations of G¨odel’s theory and of the utilized foundational logic.
35
Select an application scenario and define the objectives. Already mentioned ex-
amples include the ethico-legal governance of autonomous cars and ethico-legal
reasoner in the smart home example.
Ethical governor component. A suitable explicit ethical governor architecture
must be selected and provided. This step connects with recent research area on
governing architectures for intelligent systems [93, 94, 95].
Populate the governor component with the ethico-legal domain theory and rea-
soner. Select the ethico-legal domain theory to be employed; if not yet formal-
ized and automated, first consult layer L2. Otherwise integrate the ethico-legal
domain reasoner obtained from layer L2 with the governor component and per-
form offline tests. Does it respond to example queries as expected and with
reasonable performance? In particular, can it check for compliance of a consid-
ered action in an IAS wrt. the formalized ethico-legal domain theory?
Properly test, assess and demonstrate the system in practice.
7.4. Note on open-texture and concept explication
A goal of LogiKEy at layer L2 is to support, among others, the systematic
exploration, formalization and automation of new regulatory theories for IASs.
Ideally, such a development is conducted by an expert team comprised of lo-
gicians, legal and ethical experts and practitioners representing the addressed
application domain. Alternatively, such a theory formalization process may
start from existing legislation. A known challenge in the latter case concerns
the open-texture of informal legal texts; e.g., relevant concepts may probably
be specified rather vaguely and deliberately left open for interpretation in con-
text. In prior work the open-texture challenge occurred also in other contexts,
e.g., in the analysis philosophical arguments. In a recent reconstruction and
verification of Lowe’s ontological argument [96], for example, underspecified no-
tions such as necessary being and concrete being had to be suitably interpreted
in context to finally arrive at a verified formalisation of the argument; cf. [97]
where a respective computer-supported, iterative interpretation process is pre-
sentation and explained. The LogiKEy methodology does not eliminate the
open-texture challenge. However, it provides suitable means to address it in
an interaction between human experts and computational tools. The idea is to
computationally explore suitable explications or emendations of vague concepts
for a given application context. The logical plurality, flexibility and expressivity
supported in LogiKEy thereby constitutes a distinctive virtue that is, as far
as we are aware of, unmatched in related work. Moreover, ongoing work on
computational hermeneutics [98] aims at reducing the need for user interaction
in future applications by automating the exploration of choice points in concept
explication and beyond.
36
8. Related work
Relevant own related work has already been mentioned in the previous sec-
tions; the referenced works contain technical details and evidence that we cannot
address in full depth in this article. Further references to own work and that of
others are provided below. Many of those contain illustrating examples and fur-
ther details that may well benefit researchers or students interested in adopting
the LogiKEy methodology in their own work. They also provide useful infor-
mation on various intellectual roots of the research presented in this article.
8.1. Machine ethics and deontic logic
The questions how transparency, explainability and verifiability can best be
achieved in future intelligent systems and whether bottom-up or top-down ar-
chitectures should be preferred are discussed in a range of related articles; cf.,
e.g., [99, 100, 101, 102, 1, 103, 104, 105] and the references therein. For exam-
ple, Dennis et al. [105] make a compelling case for the use of so-called formal
verification—a well-established technique for proving correctness of computer
systems—in the area of machine ethics. The idea is to use formal methods
to check whether an autonomous system (e.g., an unmanned civilian aircraft)
is compliant with some specific ethico-legal rules (e.g., the Rules of the Air)
when making decisions. An ethical rule is represented as a formula of the form
“do(a) ⇒c¬Eφ”, denoting that doing action a counts as a violation of ethical
principle φ. However, they do not specify in full the syntax and semantics of
their operator E. It may be valuable to further explore the relationship between
this work and the approach outlined in the present article. As the authors ob-
serve, on p. 6 of their article, the Emodal operator resembles the obligation
operator used in deontic logic.
Further related work includes a range of implemented theorem proving sys-
tems. A lean but powerful connection-based theorem prover for first-order
modal logics, covering also SDL, has been developed by Otten [106]. A
tableaux-based propositional reasoner is employed in the work of Furbach and
Schon [106, 27] and first-order resolution methods for modal logics have been
contributed by Schmidt and Hustadt [107]. Further related work includes a rea-
soner for propositional defeasible modal logic by Governatori and his team [26].
Their reasoner supports defeasible reasoning, but it is less flexible than ours,
because it does not allow the user to easily switch between different systems
of normative reasoning and explore their properties. A reasoner for expressive
contextual deontic reasoning was proposed by Bringsjord et al. [28]. Pereira
and Saptawijaya [29, 108, 109] present a solution that implements deontic and
counterfactual reasoning in Prolog.
We are not aware of any attempts to automate, within a single framework,
such a wide portfolio of CTD resistant propositional, first-order and higher-
order deontic logics as we report it in this article. Note that in addition to
the features of the above related systems our solution also supports intuitive
user-interaction and most flexible logic combinations.
37
The SSE approach has also been implemented in the Leo-III theorem
prover, so that the prover now provides native language support for a wide
range of modal logics and for DDL [65, 31]. A recent, independent study [110]
shows that Leo-III, due to its wide range of directly supported logics, has be-
come the most powerful and most widely applicable ATP system existent to
date.19
The flexibility of the SSE approach has been exploited and demonstrated
in particular in the case study on Gewirth’s PGC that we have presented in
Sect. 6.2. Related work on the mechanization of ambitious ethical theories
includes the already mentioned automation of the “Doctrine of Double Effect”
by Govindarajulu and Bringsjord [77].
8.2. Universal logical reasoning
Related experiments, at LogiKEy layers L1 and L2, with the SSE approach
have been conducted in metaphysics [92, 111]. An initial focus thereby has been
on computer-supported assessments of rational arguments, in particular, of mod-
ern, modal logic variants of the ontological argument for the existence of God.
In the course of these experiments, in which the SSE approach was applied for
automating different variants of higher-order quantified modal logics, the the-
orem prover LEO-II even detected an previously unnoticed inconsistency in
G¨odel’s [112] modal variant of the ontological argument, while the soundness of
the emended variant by Scott [113] was confirmed and all argument steps were
verified. Further modern variants of the ontological argument have subsequently
been studied with the approach, and theorem provers have even contributed to
the clarification of an unsettled philosophical dispute [114]. The good perfor-
mance of the SSE approach in previous work has been a core motivation for the
new application direction addressed here. In previous work, Benzm¨uller and
colleagues also studied actualist quantifiers [70, 71], and it should be possible
to transfer these ideas to our setting.
Another advantage of the SSE approach, when implemented within powerful
proof assistants such as Isabelle, is that proof construction, interactive or au-
tomated, can be supported at different levels of abstraction. For this note that
proof protocols/objects may generally serve two different purposes: (a) they
may provide an independently verifiable explanation in a typically well-defined
logical calculus, or (b) they may provide an intuitive explanation to the user
why the problem in question has been answered positively or negatively. Many
reasoning tools, if they are offering proof objects at all, do generate only objects
of type (a). The SSE approach, however, has already demonstrated its capabili-
ties to provide both types of responses simultaneously in even most challenging
logic settings. For example, a quite powerful, abstract level theorem prover
for hyper-intensional higher-order modal logic has been provided by Kirchner
19The assessment has included various variants of classical first-order and higher-order logic
benchmark problems. First-order and higher-order deontic logics and other non-classical logics
were still excluded though. Their inclusion would clearly further benefit the Leo-III prover.
38
and colleagues [115, 89]. He encoded, using abstraction layers, a proof calcu-
lus for this very complex logic as proof tactics and he demonstrated how these
abstract level proof tactics can again be automated using respective tools in Is-
abelle/HOL. Kirchner then successfully applied this reasoning infrastructure
to reveal, assess and intuitively communicate a non-trivial paradox in Zalta’s
“Principia Logico-Metaphysica” [116].
Drawing on the results and experiences from previous work, the ambition
of our ongoing project is to further extend the already existing portfolio of
deontic logics in Isabelle/HOL towards a most powerful, flexible and scalable
deontic logic reasoning infrastructure. A core motivation thereby is to support
empirical studies in various application scenarios, and to assess and compare
the suitability, adequacy and performance of individual deontic logic solutions
for the engineering of moral agents and explainable intelligent systems. It is
relevant to mention that proof automation in Isabelle, and also in related
higher-order ATP systems such as Leo-III [20], is currently improving at good
pace. These developments are fostered in particular by recently funded research
and infrastructure projects.20
8.3. Discussion
We propose higher-order logic as a uniform and highly expressive formal
framework to represent and experiment with normative theories of ethico-legal
reasoning. To some researchers, this may seem paradoxical for two reasons.
First of all, we do no longer aim for a unique and standard deontic logic which
can be used for all applications, but we do propose to use higher-order logic
as a unique and formal framework to represent normative theories. So what
exactly is the difference between a unique deontic logic and a unique formal
framework? The second apparent paradox is that we propose higher-order logic
for tool support, whereas it is well-known that higher-order logic has theoretical
drawbacks and is undecidable.
These two apparent paradoxes can be explained away by our methodology
of representing normative theories in higher-order logic. There are many ways
in which a normative theory can be represented in higher-order logic, and only
a few of them will be such that the formal methods of the tool support can
be suitably applied to them. Therefore, the representation of deontic logics in
higher-order logic is an art, and each new representation has to come with a
proof that the embedding is faithful. These proofs play a similar role in our
formal framework as soundness and completeness proofs play in most of the
traditional work of deontic logic.
20Prominent example projects include Matryoshka (http://matryoshka.gforge.inria.fr)
and ALEXANDRIA (http://www.cl.cam.ac.uk/~lp15/Grants/Alexandria/).
39
9. Further research
In future work, the range of normative theories must be extended, and the
currently represented theories must be further optimized. In particular, a wider
range of explicit ethical theories must be studied and formalized. We have made
historical and current developments in normative reasoning practically accessi-
ble for the use in machine ethics. We showed how our approach can support
research in normative reasoning itself. The use of computer-assisted exploration
and assessment of new deontic logics provides immediate feedback from systems
to property checks. This is particularly valuable for unifying and combining
different logics and for experimental studies. For example, since our approach
supports meta-logical investigations, conjectured relationships between I/O log-
ics and conditional logics can be formally assessed in future work.
Moreover, our approach can also be used for other relevant purposes. In
education, for example, the different logics discussed in this article can now be
integrated in computer-supported classroom education. First reassuring results
have been obtained in lecture courses at University of Luxembourg (Intelligent
Agents II ) and Freie Universit¨at Berlin (Universal Logical Reasoning and Com-
putational Metaphysics). Students start exploring existing deontic logics and
other non-classical logics without the need for a deep a priori understanding of
them. They can use the logics in small or larger examples, modify them, assess
their modifications, etc.
In agent simulation, we plan the embodiment/implementation of explicit
ethical theories in simulated agents. We can then investigate properties for single
agents, but beyond that also study agent interaction and the behaviour of the
agent society as a whole. To this end the agent-based LeoPARD framework
[78], which is underlying the Leo-III prover, will be adapted and utilized in
future work.
10. Conclusion
It is the availability of powerful systems such as Isabelle/HOL or Leo-III
for tool support that allows our approach to revolutionize the field of formal
ethics. Though the use of higher-order logic may come as a paradigm shift to
the field of ethical reasoning, it is an insight which is already well established in
the area of formal deduction. Whereas it is far from straightforward to represent
deontic logics in higher-order logic, once a deontic logic has been represented, it
becomes much easier to make small changes to them and see the effect of these
changes—and this is exactly how our approach supports the design of normative
theories of ethico-legal reasoning. It is in the ease in which the user can work
with and adapt existing theories, how the design of normative theories is made
accessible to non-specialist users and developers.
To validate our approach we have embedded the main strands of current
deontic logic within higher-order logic, and we have experimented with the
approach over the past two years.
40
Our LogiKEy normative reasoning framework and infrastructure supports
empirical studies on legal and ethical theories in which the underlying logic
formalisms itself can be flexibly varied, assessed and compared in context. This
infrastructure can fruitfully support the development of much needed logic based
approaches towards ethical agency. The solution we have presented supports a
wide range of specific deontic logic variants, and it also scales for their first-order
and higher-order extensions.
The use of tool support for ethico-legal reasoning is not only fruitful to de-
velop new normative theories, it is now being employed also in teaching, and
we plan to use it as a formal framework for simulation. Another promising
application is the use of our approach for the study of deontic modality in nat-
ural language processing. In linguistics the use of higher-order logic is already
adopted for the semantics of natural language, and we believe that our frame-
work can also support studies of the pragmatic aspects of the use of deontic
modality.
Acknowledgments:. We thank Ali Farjami, David Fuenmayor, Tobias Gleißner,
Alexander Steen and several further colleagues for their contributions to this
project.
References
References
[1] B. F. Malle, Integrating robot ethics and machine morality: The study and
design of moral competence in robots, Ethics and Information Technology
18 (4) (2016) 243–256.
[2] B. F. Malle, M. Scheutz, J. L. Austerweil, Networks of social and moral
norms in human and robot agents, in: M. I. Aldinhas Ferreira, J. Silva Se-
queira, M. O. Tokhi, E. E. Kadar, G. S. Virk (Eds.), A World with Robots:
International Conference on Robot Ethics: ICRE 2015, Springer Interna-
tional Publishing, Cham, 2017, pp. 3–17.
[3] D. Gabbay, J. Horty, X. Parent, R. van der Meyden, L. van der Torre
(Eds.), Handbook of Deontic Logic and Normative Systems, Vol. 1, College
Publications, London, UK, 2013.
[4] A. Chopra, L. van der Torre, H. Verhagen (Eds.), Handbook of Normative
Multi-Agent Systems, College Publications, London, UK, 2018.
[5] X. Parent, L. van der Torre, Introduction to Deontic Logic and Normative
Systems, College Publications, London, UK, 2018.
[6] W. Carnielli, M. Coniglio, D. M. Gabbay, G. Paula, C. Sernadas, Analysis
and Synthesis of Logics, no. 35 in Applied Logics Series, Springer, 2008.
41
[7] D. Gabbay, Many-dimensional Modal Logics: Theory and Applications,
Studies in logic and the foundations of mathematics, North Holland Pub-
lishing Company, 2003.
[8] C. Benzm¨uller, A. Farjami, X. Parent, Aqvist’s dyadic deontic logic E in
HOL, to appear in Journal of Applied Logics – IfCoLoG Journal of Logics
and their Applications, Special issue MIREL 2018 workshop on MIning
and REasoning with Legal texts, Url (preprint): http://orbilu.uni.
lu/handle/10993/37014 (2019).
[9] A. Farjami, P. Meder, X. Parent, C. Benzm¨uller, I/O logic in HOL, to
appear in Journal of Applied Logics – IfCoLoG Journal of Logics and
their Applications, Special issue MIREL 2018 workshop on MIning and
REasoning with Legal texts, URL (preprint): http://orbilu.uni.lu/
handle/10993/37013 (2019).
[10] C. Benzm¨uller, A. Farjami, X. Parent, A dyadic deontic logic in HOL,
in: J. Broersen, C. Condoravdi, S. Nair, G. Pigozzi (Eds.), Deontic Logic
and Normative Systems – 14th International Conference, DEON 2018,
Utrecht, The Netherlands, 3-6 July, 2018, College Publications, 2018, pp.
33–50.
[11] P. Meder, Deontic Agency and Moral Luck, Master’s thesis, Faculty of Sci-
ence, Technology and Communication, University of Luxembourg (2018).
URL http://orbilu.uni.lu/handle/10993/39770
[12] D. Fuenmayor, C. Benzm¨uller, Harnessing higher-order (meta-)logic to
represent and reason with complex ethical theories, in: PRICAI 2019:
Trends in Artificial Intelligence, Lecture Notes in Artificial Intelligence,
Springer International Publishing, 2019, pp. 1–14, in print, preprint http:
//arxiv.org/abs/1903.09818.
[13] D. Fuenmayor, C. Benzm¨uller, Mechanised assessment of complex natural-
language arguments using expressive logic combinations, in: Frontiers of
Combining Systems, 12th International Symposium, FroCoS 2019, Lon-
don, September 4-6, Lecture Notes in Artificial Intelligence, Springer,
2019, pp. 1–17, in print, preprint http://doi.org/10.13140/RG.2.2.
20803.45608/1.
[14] C. Benzm¨uller, P. Andrews, Church’s type theory, in: E. N. Zalta (Ed.),
The Stanford Encyclopedia of Philosophy, summer 2019 Edition, Meta-
physics Research Lab, Stanford University, 2019, pp. 1–62 (in pdf version).
[15] D. K. Pratihar, L. C. Jain (Eds.), Intelligent Autonomous Systems: Foun-
dations and Applications, Vol. 275 of Studies in Computational Intelli-
gence, Springer, 2010. doi:10.1007/978-3-642-11676-6.
URL https://doi.org/10.1007/978-3-642-11676- 6
42
[16] L. M. de Moura, S. Kong, J. Avigad, F. van Doorn, J. von Raumer, The
Lean theorem prover (system description), in: A. P. Felty, A. Middeldorp
(Eds.), Automated Deduction – CADE-25 – 25th International Conference
on Automated Deduction, Berlin, Germany, August 1–7, 2015, Proceed-
ings, Vol. 9195 of Lecture Notes in Computer Science, Springer, 2015, pp.
378–388.
[17] Y. Bertot, P. Casteran, Interactive Theorem Proving and Program Devel-
opment, Springer, 2004.
[18] A. Bove, P. Dybjer, U. Norell, A brief overview of Agda – A functional
language with dependent types, in: S. Berghofer, T. Nipkow, C. Urban,
M. Wenzel (Eds.), Theorem Proving in Higher Order Logics, 22nd In-
ternational Conference, TPHOLs 2009, Munich, Germany, August 17-20,
2009. Proceedings, Vol. 5674, Springer, 2009, pp. 73–78.
[19] T. Nipkow, L. Paulson, M. Wenzel, Isabelle/HOL: A Proof Assistant for
Higher-Order Logic, Vol. 2283 of Lecture Notes in Computer Science,
Springer, 2002.
[20] A. Steen, C. Benzm¨uller, The higher-order prover Leo-III, in:
D. Galmiche, S. Schulz, R. Sebastiani (Eds.), Automated Reasoning. IJ-
CAR 2018, Vol. 10900 of Lecture Notes in Computer Science, Springer,
Cham, 2018, pp. 108–116.
[21] J. Carmo, A. J. I. Jones, Deontic logic and contrary-to-duties, in: D. M.
Gabbay, F. Guenthner (Eds.), Handbook of Philosophical Logic: Volume
8, Springer Netherlands, Dordrecht, 2002, pp. 265–343.
[22] B. Liao, M. Slavkovik, L. van der Torre, Building Jiminy Cricket: An
architecture for moral agreements among stakeholders, in: AAAI/ACM
Conference on Artificial Intelligence, Ethics and Society, URL (preprint):
https://arxiv.org/pdf/1812.04741.pdf, 2019, pp. 1–15, forthcoming.
[23] P. M. Dung, On the acceptability of arguments and its fundamental role
in nonmonotonic reasoning, logic programming and n-person games, Ar-
tificial Intelligence 77 (2) (1995) 321–357.
[24] C. Benzm¨uller, Universal (meta-)logical reasoning: Recent successes, Sci-
ence of Computer Programming 172 (2019) 48–62.
[25] C. Benzm¨uller, N. Sultana, L. C. Paulson, F. Theiß, The higher-order
prover LEO-II, Journal of Automated Reasoning 55 (4) (2015) 389–404.
[26] E. Kontopoulos, N. Bassiliades, G. Governatori, G. Antoniou, A modal
defeasible reasoner of deontic logic for the semantic web, International
Journal on Semantic Web Information Systems 7 (1) (2011) 18–43.
43
[27] U. Furbach, C. Schon, Deontic logic for human reasoning, in: T. Eiter,
H. Strass, M. Truszczynski, S. Woltran (Eds.), Advances in Knowledge
Representation, Logic Programming, and Abstract Argumentation, Vol.
9060 of Lecture Notes in Computer Science, Springer, 2015, pp. 63–80.
[28] S. Bringsjord, N. S. G., B. F. Malle, M. Scheutz, Contextual deontic cogni-
tive event calculi for ethically correct robots, in: International Symposium
on Artificial Intelligence and Mathematics, ISAIM 2018, Fort Lauderdale,
Florida, USA, January 3-5, 2018., 2018, pp. 1–3.
[29] L. M. Pereira, A. Saptawijaya, Programming machine ethics, Vol. 26
of Studies in Applied Philosophy, Epistemology and Rational Ethics,
Springer, 2016.
[30] A. Gewirth, Reason and Morality, University of Chicago Press, 1981.
[31] T. Gleißner, A. Steen, C. Benzm¨uller, Theorem provers for every normal
modal logic, in: T. Eiter, D. Sands (Eds.), LPAR-21. 21st International
Conference on Logic for Programming, Artificial Intelligence and Reason-
ing, Vol. 46 of EPiC Series in Computing, EasyChair, Maun, Botswana,
2017, pp. 14–30.
[32] D. Fuenmayor, C. Benzm¨uller, Formalisation and evaluation of Alan
Gewirth’s proof for the Principle of Generic Consistency in Isabelle/HOL,
Archive of Formal Proofs (2018) 1–13.
URL http://isa-afp.org/entries/GewirthPGCProof.html
[33] C. Benzm¨uller, L. C. Paulson, Quantified multimodal logics in simple
type theory, Logica Universalis (Special Issue on Multimodal Logics) 7 (1)
(2013) 7–20.
[34] J. Gibbons, N. Wu, Folding domain-specific languages: deep and shal-
low embeddings (functional pearl), in: J. Jeuring, M. M. T. Chakravarty
(Eds.), Proceedings of the 19th ACM SIGPLAN international conference
on Functional programming, Gothenburg, Sweden, September 1-3, 2014,
ACM, 2014, pp. 339–347. doi:10.1145/2628136.2628138.
[35] J. Svenningsson, E. Axelsson, Combining deep and shallow embedding for
EDSL, in: H.-W. Loidl, R. Pe˜na (Eds.), Trends in Functional Program-
ming, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 21–36.
[36] J. C. Blanchette, S. B¨ohme, L. C. Paulson, Extending Sledgehammer with
SMT solvers, Journal of Automated Reasoning 51 (1) (2013) 109–128.
[37] J. C. Blanchette, T. Nipkow, Nitpick: A counterexample generator for
higher-order logic based on a relational model finder, in: M. Kaufmann,
L. C. Paulson (Eds.), Interactive Theorem Proving, First International
Conference, ITP 2010, Edinburgh, UK, July 11-14, 2010. Proceedings,
Vol. 6172 of Lecture Notes in Computer Science, Springer, 2010, pp. 131–
146.
44
[38] S. Cruanes, J. C. Blanchette, Extending nunchaku to dependent type
theory, in: J. C. Blanchette, C. Kaliszyk (Eds.), Proceedings First Inter-
national Workshop on Hammers for Type Theories, HaTT@IJCAR 2016,
Coimbra, Portugal, July 1, 2016., Vol. 210 of EPTCS, 2016, pp. 3–12.
[39] H. Ohlbach, A. Nonnengart, M. de Rijke, D. Gabbay, Encoding two-valued
nonclassical logics in classical logic, in: J. Robinson, A. Voronkov (Eds.),
Handbook of Automated Reasoning (in 2 volumes), Elsevier and MIT
Press, 2001, pp. 1403–1486.
[40] G. Frege, Begriffsschrift. Eine der arithmetischen nachgebildete Formel-
sprache des reinen Denkens, Halle, 1879.
[41] C. Benzm¨uller, P. Andrews, Church’s type theory, in: E. N. Zalta (Ed.),
The Stanford Encyclopedia of Philosophy, summer 2019 Edition, Meta-
physics Research Lab, Stanford University, 2019, pp. 1–62 (in pdf version).
URL https://plato.stanford.edu/entries/type-theory-church/
[42] C. Benzm¨uller, C. Brown, M. Kohlhase, Higher-order semantics and ex-
tensionality, Journal of Symbolic Logic 69 (4) (2004) 1027–1088.
[43] C. Benzm¨uller, D. Miller, Automation of higher-order logic, in: D. M.
Gabbay, J. H. Siekmann, J. Woods (Eds.), Handbook of the History of
Logic, Volume 9 – Computational Logic, North Holland, Elsevier, 2014,
pp. 215–254.
[44] P. B. Andrews, General models, descriptions, and choice in type theory,
Journal of Symbolic Logic 37 (2) (1972) 385–394.
[45] L. Henkin, Completeness in the theory of types, Journal of Symbolic Logic
15 (2) (1950) 81–91.
[46] G. H. von Wright, Deontic logic, Mind 60 (1951) 1–15.
[47] B. Hansson, An analysis of some deontic logics, Noˆus 3 (4) (1969) 373–398.
[48] L. ˚
Aqvist, Deontic logic, in: D. Gabbay, F. Guenthner (Eds.), Handbook
of Philosophical Logic, 2nd Edition, Vol. 8, Kluwer Academic Publishers,
Dordrecht, Holland, 2002, pp. 147–264.
[49] X. Parent, Completeness of ˚
Aqvist’s systems E and F, Review of Symbolic
Logic 8 (1) (2015) 164–177.
[50] A. Kratzer, Modals and Conditionals: New and Revised Perspectives,
Oxford University Press, New York, 2012.
[51] J. Carmo, A. J. I. Jones, Completeness and decidability results for a logic
of contrary-to-duty conditionals, Journal of Logic and Computation 23 (3)
(2013) 585–626.
45
[52] C. Alchourr´on, Philosophical foundations of deontic logic and the logic
of defeasible conditionals, in: J.-J. Meyer, R. Wieringa (Eds.), Deontic
Logic in Computer Science, John Wiley & Sons, Inc., New York, 1993,
pp. 43–84.
[53] B. Kjos-Hanssen, A conflict between some semantic conditions of Carmo
and Jones for contrary-to-duty obligations, Studia Logica 105 (1) (2017)
173–178.
[54] J. Horty, Agency and Deontic Logic, Oxford University Press, London,
UK, 2009.
[55] J. Hansen, Reasoning about permission and obligation, in: S. O. Hansson
(Ed.), David Makinson on Classical Methods for Non-Classical Problems,
Springer, Dordrecht, 2014, pp. 287–333.
[56] X. Parent, L. van der Torre, Detachment in normative systems: Exam-
ples, inference patterns, properties., Journal of Applied Logics – IfCoLog
Journal of Logics and their Applications 4 (2017) 2996–3039.
[57] D. Makinson, L. W. N. van der Torre, Input/output logics, Journal of
Philosophical Logic 29 (4) (2000) 383–408.
[58] J. Horty, Reasons as Defaults, Oxford University Press, 2012.
[59] J. Hansen, Prioritized conditional imperatives: problems and a new pro-
posal, Journal of Autonomous Agents and Multi-Agent Systems 17 (1)
(2008) 11–35.
[60] C. Straßer, M. Beirlaen, F. Van De Putte, Adaptive logic characterizations
of input/output logic, Studia Logica 104 (5) (2016) 869–916.
[61] C. Straßer, Adaptive Logics for Defeasible Reasoning: Applications in
Argumentation, Normative Reasoning and Default Reasoning, Springer,
2013.
[62] M. Palmirani, G. Governatori, A. Rotolo, S. Tabet, H. Boley, A. Paschke,
LegalRuleML: XML-based rules and norms, in: F. Olken, M. Palmirani,
D. Sottara (Eds.), Rule-Based Modeling and Computing on the Semantic
Web, Springer Berlin Heidelberg, Berlin, Heidelberg, 2011, pp. 298–312.
[63] T. Gordon, The Pleading Game: an Artificial Intelligence Model of Pro-
cedural Approach, Springer, New York, 1995.
[64] G. Sartor, Legal Reasoning: A Cognitive Approach to Law, Springer,
2005.
[65] A. Steen, Higher-order theorem proving and its applications, it Infor-
mation Technology * (2019) *–*, in print, online available at https:
//doi.org/10.1515/itit-2019-0001.doi:10.1515/itit-2019- 0001.
46
[66] A. Steen, C. Benzm¨uller, Extensional higher-order paramodulation in Leo-
III, submitted, preprint: https://arxiv.org/abs/1907.11501 (2019).
[67] C. Benzm¨uller, X. Parent, I/O logic in HOL – first steps, Tech. rep.,
CoRR, https://arxiv.org/abs/1803.09681 (2018).
[68] X. Parent, L. W. N. van der Torre, The pragmatic oddity in a norm-based
deontic logic, in: J. Keppens, G. Governatori (Eds.), Proceedings of the
16th edition of the International Conference on Articial Intelligence and
Law, ICAIL 2017, London, United Kingdom, June 12-16, 2017, ACM,
2017, pp. 169–178.
[69] X. Parent, L. W. N. van der Torre, I/O logics with a consistency check,
in: J. M. Broersen, C. Condoravdi, N. Shyam, G. Pigozzi (Eds.), Deontic
Logic and Normative Systems – 14th International Conference, DEON
2018, Utrecht, The Netherlands, July 3-6, 2018., College Publications,
2018, pp. 285–299.
[70] C. Benzm¨uller, Automating quantified conditional logics in HOL, in:
F. Rossi (Ed.), IJCAI-13, AAAI Press, Beijing, China, 2013, pp. 746–
753.
[71] C. Benzm¨uller, Cut-elimination for quantified conditional logic, Journal
of Philosophical Logic 46 (3) (2017) 333–353.
[72] X. Parent, Maximality vs. optimality in dyadic deontic logic, Journal of
Philosophical Logic 43 (6) (2014) 1101–1128.
[73] L. ˚
Aqvist, An Introduction to Deontic logic and the Theory of Normative
Systems, Bibliopolis, Naples, 1987.
[74] L. Goble, Axioms for Hansson’s dyadic deontic logics, Filosofiska Notiser
(2019) 13–61.
[75] X. Parent, Preference-based semantics for dyadic deontic logics in Hans-
son’s tradition: a survey of axiomatisation results, in D. Gabbay, J. Horty,
X. Parent, R. van der Meyden and L. van der Torre (eds.), Handbook of
Deontic Logic and Normative Systems, volume 2, College Publications,
London, UK (to appear).
[76] D. Lewis, Counterfactuals, Blackwell, Oxford, 1973.
[77] N. S. Govindarajulu, S. Bringsjord, On automating the doctrine of double
effect, in: C. Sierra (Ed.), Proceedings of the Twenty-Sixth International
Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Aus-
tralia, August 19-25, 2017, ijcai.org, 2017, pp. 4722–4730.
[78] M. Wisniewski, A. Steen, C. Benzm¨uller, LeoPARD – A generic plat-
form for the implementation of higher-order reasoners, in: M. Kerber,
J. Carette, C. Kaliszyk, F. Rabe, V. Sorge (Eds.), Intelligent Computer
47
Mathematics – International Conference, CICM 2015, Washington, DC,
USA, July 13-17, 2015, Proceedings, Vol. 9150 of Lecture Notes in Com-
puter Science, Springer, 2015, pp. 325–330.
[79] A. Steen, M. Wisniewski, C. Benzm¨uller, Agent-based HOL reasoning,
in: G.-M. Greuel, T. Koch, P. Paule, A. Sommese (Eds.), Mathematical
Software – ICMS 2016, 5th International Congress, Proceedings, Vol. 9725
of LNCS, Springer, Berlin, Germany, 2016, pp. 75–81, preprint: http:
//christoph-benzmueller.de/papers/C56.pdf.
[80] R. Chisholm, Contrary-to-duty imperatives and deontic logic, Analysis 24
(1963) 33–36.
[81] E. Lorini, Temporal STIT logic and its application to normative reasoning,
Journal of Applied Non-Classical Logics 2013 (2013) 372–399.
[82] D. Beyleveld, The Dialectical Necessity of Morality: An Analysis and De-
fense of Alan Gewirth’s Argument to the Principle of Generic Consistency,
University of Chicago Press, 1991.
[83] D. Kaplan, On the logic of demonstratives, Journal of Philosophical Logic
8 (1) (1979) 81–98.
[84] A. Kornai, Bounding the impact of AGI, Journal of Experimental & The-
oretical Artificial Intelligence 26 (3) (2014) 417–438.
[85] M. Wisniewski, A. Steen, C. Benzm¨uller, Einsatz von Theorembeweisern
in der Lehre, in: A. Schwill, U. Lucke (Eds.), Hochschuldidaktik der
Informatik: 7. Fachtagung des GI-Fachbereichs Informatik und Ausbil-
dung/Didaktik der Informatik; 13.-14. September 2016 an der Universit¨at
Potsdam, Commentarii informaticae didacticae (CID), Universit¨atsverlag
Potsdam, Potsdam, Germany, 2016, pp. 81–92.
URL https://publishup.uni-potsdam.de/opus4-ubp/frontdoor/
index/index/docId/9485
[86] D. Kaplan, Afterthoughts, in: J. Almog, J. Perry, H. Wettstein (Eds.),
Themes from Kaplan, Oxford University Press, 1989, pp. 565–614.
[87] C. Benzm¨uller, Universal (meta-)logical reasoning: The wise men puzzle,
Data in Brief 24 (103774).
[88] M. Sergot, Epistemic logic and common knowledge, Lecture Course
Notes, Department of Computing Imperial College, London,
https://www.doc.ic.ac.uk/~mjs/teaching/ModalTemporal499/
Epistemic_499_v0809_2up.pdf (2008).
[89] D. Kirchner, C. Benzm¨uller, E. N. Zalta, Computer science and meta-
physics: A cross-fertilization, Open Philosophy 2 (2019) 1–22, in print,
preprint: http://doi.org/10.13140/RG.2.2.25229.18403.doi:10.
1515/opphil-2019-0015.
URL https://arxiv.org/abs/1905.00787
48
[90] C. Benzm¨uller, J. Otten, T. Raths, Implementing and evaluating provers
for first-order modal logics, in: L. D. Raedt, C. Bessiere, D. Dubois,
P. Doherty, P. Frasconi, F. Heintz, P. Lucas (Eds.), ECAI 2012, Vol. 242
of Frontiers in Artificial Intelligence and Applications, IOS Press, 2012,
pp. 163–168.
[91] R. C. Barcan, A functional calculus of first order based on strict implica-
tion, The Journal of Symbolic Logic 11 (1) (1946) 1–16.
[92] C. Benzm¨uller, B. Woltzenlogel Paleo, The inconsistency in G¨odel’s on-
tological argument: A success story for AI in metaphysics, in: S. Kamb-
hampati (Ed.), IJCAI-16, Vol. 1-3, AAAI Press, 2016, pp. 936–942.
[93] N. S. Govindarajulu, S. Bringsjord, Ethical regulation of robots must be
embedded in their operating systems, in: R. Trappl (Ed.), A Construction
Manual for Robots’ Ethical Systems – Requirements, Methods, Implemen-
tations, Cognitive Technologies, Springer, 2015, pp. 85–99.
[94] R. C. Arkin, P. Ulam, A. R. Wagner, Moral decision making in au-
tonomous systems: Enforcement, moral emotions, dignity, trust, and de-
ception, Proceedings of the IEEE 100 (3) (2012) 571–589.
[95] R. C. Arkin, P. Ulam, B. Duncan, An ethical governor for constrain-
ing lethal action in an autonomous system, Tech. Rep. GIT-GVU-09-02,
Georgia Institute of Technology. (2009).
[96] E. J. Lowe, A modal version of the ontological argument., in: J. P. More-
land, K. A. Sweis, C. V. Meister (Eds.), Debating Christian Theism, Ox-
ford University Press, 2013, Ch. 4, pp. 61–71.
[97] D. Fuenmayor, C. Benzm¨uller, A case study on computational hermeneu-
tics: E. J. Lowe’s modal ontological argument, Journal of Applied Logic –
IfCoLoG Journal of Logics and their Applications (special issue on Formal
Approaches to the Ontological Argument) 5 (7) (2018) 1567–1603.
[98] D. Fuenmayor, C. Benzm¨uller, A computational-hermeneutic approach for
conceptual explicitation, in: A. Nepomuceno, L. Magnani, F. Salguero,
C. Bares, M. Fontaine (Eds.), Model-Based Reasoning in Science and
Technology. Inferential Models for Logic, Language, Cognition and Com-
putation, Sapere, Springer, 2019, pp. 1–29, in print, preprint http:
//doi.org/10.13140/RG.2.2.30869.78564.
[99] V. Dignum (Ed.), Special issue: Ethics and artificial intelligence, Ethics
and Information Technology 20 (1).
[100] A. F. T. Winfield, M. Jirotka, Ethical governance is essential to building
trust in robotics and artificial intelligence systems, Philosophical Trans-
actions of the Royal Society A: Mathematical, Physical and Engineering
Sciences 376 (2133) (2018) 20180085.
49
[101] V. Dignum, Responsible autonomy, in: C. Sierra (Ed.), Proceedings of
the Twenty-Sixth International Joint Conference on Artificial Intelligence,
IJCAI 2017, Melbourne, Australia, August 19-25, 2017, ijcai.org, 2017, pp.
4698–4704.
[102] M. Scheutz, The case for explicit ethical agents, AI Magazine 38 (2017)
57–64.
[103] M. Anderson, S. L. Anderson, Toward ensuring ethical behavior from au-
tonomous systems: a case-supported principle-based paradigm, Industrial
Robot 42 (4) (2015) 324–331.
[104] W. Wallach, C. Allen, I. Smit, Machine morality: bottom-up and top-
down approaches for modelling human moral faculties, AI & Society 22 (4)
(2008) 565–582.
[105] L. A. Dennis, M. Fisher, M. Slavkovik, M. Webster, Formal verification
of ethical choices in autonomous systems, Robotics and Autonomous Sys-
tems 77 (2016) 1–14.
[106] J. Otten, Non-clausal connection calculi for non-classical logics, in: R. A.
Schmidt, C. Nalon (Eds.), Automated Reasoning with Analytic Tableaux
and Related Methods – 26th International Conference, TABLEAUX 2017,
Bras´ılia, Brazil, September 25-28, 2017, Proceedings, Vol. 10501 of Lecture
Notes in Computer Science, Springer, 2017, pp. 209–227.
[107] R. A. Schmidt, U. Hustadt, First-order resolution methods for modal log-
ics, in: A. Voronkov, C. Weidenbach (Eds.), Programming Logics – Essays
in Memory of Harald Ganzinger, Vol. 7797 of Lecture Notes in Computer
Science, Springer, 2013, pp. 345–391.
[108] A. Saptawijaya, L. M. Pereira, Logic programming for modeling morality,
Logic Journal of the IGPL 24 (4) (2016) 510–525.
[109] L. M. Pereira, A. Saptawijaya, Counterfactuals, logic programming and
agent morality, in: R. Urbaniak, G. Payette (Eds.), Applied Formal/-
Mathematical Philosophy, Logic, Argumentation & Reasoning, Springer,
2017, pp. 25–53.
[110] C. E. Brown, T. Gauthier, C. Kaliszyk, G. Sutcliffe, J. Urban, GRUNGE:
A Grand Unified ATP Challenge, arXiv e-prints (2019) arXiv:1903.02539.
[111] C. Benzm¨uller, B. Woltzenlogel Paleo, Automating G¨odel’s ontological
proof of God’s existence with higher-order automated theorem provers,
in: T. Schaub, G. Friedrich, B. O’Sullivan (Eds.), ECAI 2014, Vol. 263 of
Frontiers in Artificial Intelligence and Applications, IOS Press, 2014, pp.
93–98.
[112] K. G¨odel, Appendix A: Notes in Kurt G¨odel’s hand, in: J. Sobel (Ed.),
Logic and Theism, Cambridge University Press, 1970, pp. 144–145.
50
[113] D. Scott, Appendix B: Notes in Dana Scott’s hand, in: J. Sobel (Ed.),
Logic and Theism, Cambridge University Press, 1972, pp. 145–146.
[114] C. Benzm¨uller, L. Weber, B. Woltzenlogel Paleo, Computer-assisted anal-
ysis of the Anderson-H´ajek controversy, Logica Universalis 11 (1) (2017)
139–151.
[115] D. Kirchner, C. Benzm¨uller, E. N. Zalta, Mechanizing principia logico-
metaphysica in functional type theory, Review of Symbolic Logic (2019)
1–13Preprint: https://arxiv.org/abs/1711.06542.doi:10.1017/
S1755020319000297.
[116] E. N. Zalta, Principia logico-metaphysica, draft version, preprint available
at https://mally.stanford.edu/principia.pdf (2018).
51
