PreprintPDF Available


Future intelligent autonomous systems (IAS) are inevitably deciding on moral and legal questions, e.g. in self-driving cars, health care or human-machine collaboration. As decision processes in most modern sub-symbolic IAS are hidden, the simple political plea for transparency, accountability and governance falls short. A sound ecosystem of trust requires ways for IAS to autonomously justify their actions, that is, to learn giving and taking reasons for their decisions. Building on social reasoning models in moral psychology and legal philosophy such an idea of »Reasonable Machines« requires novel, hybrid reasoning tools, ethico-legal ontologies and associated argumentation technology. Enabling machines to normative communication creates trust and opens new dimensions of AI application and human-machine interaction.
Reasonable Machines: A Research Manifesto
Christoph Benzmüller1[0000000233923093] and Bertram
1Institute of Computer Science, Freie Universität Berlin, Berlin, Germany
2Department of Law, Freie Universität Berlin, Berlin, Germany
Abstract. Future intelligent autonomous systems (IAS) are inevitably
deciding on moral and legal questions, e.g. in self-driving cars, health care
or human-machine collaboration. As decision processes in most modern
sub-symbolic IAS are hidden, the simple political plea for transparency,
accountability and governance falls short. A sound ecosystem of trust
requires ways for IAS to autonomously justify their actions, that is,
to learn giving and taking reasons for their decisions. Building on so-
cial reasoning models in moral psychology and legal philosophy such
an idea of »Reasonable Machines« requires novel, hybrid reasoning
tools, ethico-legal ontologies and associated argumentation technology.
Enabling machines to normative communication creates trust and opens
new dimensions of AI application and human-machine interaction.
Keywords: Trusthworthy and Explainable AI ·Ethico-Legal Governors
·Social Reasoning Model ·Pluralistic, Expressive Normative Reasoning
1 Introduction
Intelligent autonomous systems (IASs) are rapidly entering applications in in-
dustry, military, finance, governance, administration, healthcare, etc., leading to
a historical transition period with unprecedented dynamics of innovation and
change, and with unpredictable outcomes. Politics, regulatory bodies, indeed
society as a whole, are challenged not only with keeping pace with these poten-
tially disruptive developments, but also with staying ahead and wisely guiding
the transition. Fostering positive impacts, while preventing negative side effects,
is a balanced vision shared within most of the numerous ethical guidelines of the
last years on trustworthy AI, including the European Commission’s most recent
White Paper on AI [6], proposing the creation of an “ecosystem of excellence” in
combination with an “ecosystem of trust”.
We think that real “Trustworthy AI by Designdemands IASs, which are
able to give and take reasons for their decisions to act. Such »Reasonable
Machines« require novel, hybrid reasoning tools, upper ethico-legal ontologies
and associated argumentation technology to be utilised in practice for assess-
ing, justifying and controlling (externally and internally) the behaviour of IASs
with respect to explicitly encoded legal and ethical regulation. We envision this
2 C. Benzmüller and B. Lomfeld
technology to be integrated with an on-demand, cloud-based workbench for plu-
ralistic, expressive regulatory reasoning. This would foster knowledge transfer
with industry, research, and educational institutions, it would enable access to
critical AI infrastructure at scale with little risk and minimal costs, and, in the
long run, it could support dynamic adjustments of regulating code for IASs in
the cloud via politically and socially legitimated processes.
Paper structure: Section 2 formulates objectives for Reasonable Machines,
and section 3 provides models for them building on moral psychology and legal
philosophy. Section 4 outlines modular steps for research and implementation of
Reasonable Machines; this leverages own prior work such as the LogiKEy
methodology and framework for designing normative theories for ethical and
legal reasoning [4], which needs to be combined and extended with an upper-
level value ontology [17] and further domain-level regulatory theories for the
assessment and explanation of ethical and legal conflicts and decisions in IASs.
2Reasonable Machines: Objectives
The need for some form of “moral machines” [22] is no science fiction scenario
at all. With the rise of autonomous systems in all fields of life including highly
complex and ethically critical applications like self-driving cars, weapon sys-
tems, healthcare assistance in triage and pandemic plans, predictive policing,
legal judgement supports or credit scoring tools, involved AI systems are in-
evitably confronted with, and deciding on, moral and legal questions. One core
problem with ethical and legal accountability or even governance of autonomous
systems is the hidden decision process (black box) in modern (sub-symbolic)
AI technologies, which hinders transparency as well as direct intervention. The
simple plea for transparency disregards technological realities or even restrains
much needed further developments.3
Inspired by moral psychology and cognitive science, we envision the solution
in the development of independent, symbolic logic based safety-harnesses in fu-
ture AI systems [9]. Such “ethico-legal governors” encapsulate and interact with
black box AI systems, and they will use symbolic AI techniques in order to search
for possible justifications, i.e. reasons, for their decisions and (intended) actions
with regard to some formally encoded ethico-legal theories defined by regulating
bodies. The symbolic justifications computed at this abstract level thus provide
3While interpreting, modeling and explaining the inner functioning of black box AI
systems is relevant also with respect to our Reasonable Machines vision, such re-
search alone cannot completely solve the trust and control challenge. Sub-symbolic
AI black box systems (e.g. neural architectures) are suffering from various issues
(including adversarial attacks and influence of bias in data) which cannot be easily
eliminated by interpreting, modeling and explaining them. Offline, forensic processes
are then required such that the whole enterprise of turning black box AI systems
into fully trustworthy AI systems becomes a challenging multi-step engineering pro-
cess, and such an approach is significantly further complicated when online learning
capabilities are additionally foreseen.
Reasonable Machines: A Research Manifesto 3
a basis for generating explanations about why a decision/action (proposed by
an AI black box system) is ethico-legally legitimate and compliant with respect
to the encoded ethico-legal regulation.
Such an approach is complementary to, and as an additional measure more
promising than, explaining the inner (mis-)functioning of the black box AI sys-
tem itself. Symbolic justifications in turn enable the development of further
means towards a meaningful and robust control and towards human-understand-
able explanation and human-machine interaction. The Reasonable Machines
idea outlines a genuine approach of trustworthiness by design proposing, in psy-
chological terminology [14], a slow, rational (i.e. symbolic) “System 2” layer in
responsible IASs to justify and control their fast, “intuitive”, but opaque (sub-
symbolic), “System 1” layer computations.
Reasonable Machines research aims at analyzing and constructing ways
how intelligent machines could socially justify their actions at abstract level, i.e.
give and take moral and legal reasons for their decisions to act. Reason is based
on reasons. This is true as much for artificial as for human intelligent agents.
The “practical reasonableness” of intelligent agents depends on their moral abil-
ities to communicate socially acceptable reasons for their behavior [11]. Thus,
the exploration of methods and tools enabling machines to generate normative
reasons (which may be independent of underlying black box architectures and
opaque algorithms) smoothes the way for more comprehensive artificial moral
agency and new dimensions of human-machine communication.
The core objectives of Reasonable Machines technology are:
enabling argument-based explanations & justifications of IAS decisions,
enabling ethico-legal reasoning about, and public critique of, IAS decisions,
facilitating political and legal governance of IAS decision making,
evolving ethico-legal agency and communicative capacity of IASs,
enabling trustworthy human-interaction by normative communication,
fostering development of novel neuro-symbolic AI architectures.
3 Artificial Social Reasoning Model (aSRM)
The black box governance problem has an interesting parallel in human decision
making. Most actual models in moral psychology consider emotional intuition
to be the (or at least one) initial driving force of human action which is only
afterwards (or with a second significantly slower system) rationalized with rea-
sons [12, 14]. Within a social framework of giving and taking reasons (e.g. moral
convention or a legal system) the initial motivation of a single human agent
could be ignored if his actions and his post-hoc reasoning comply with given
social (moral or legal) standards [16]. Communicating reasons within such a
post-hoc “Social Reasoning Model” (SRM) is not superfluous, but essential, as
only they guarantee the coherence of a moral or legal order in an increasingly
pluralistic world. The remaining difference is the relative independence of ratio-
nal reasoning from the motivational impulse to act. Even so, in the long run the
4 C. Benzmüller and B. Lomfeld
inner-subjective or social feedback loop with rational reasons might also change
the agents’ motivational (emotional) disposition.
This post-hoc SRM is transferable to AI decision processes as “artificial Social
Reasoning Model” (aSRM). The black box of an opaque AI system functions
like an AI intuition. Following the SRM model, transparency is not needed as
long as the system generates post-hoc reasons for its action. Moral and legal
accountability and governance could instead be enabled through symbolic or
sub-symbolic aSRMs.
A symbolic solution would try to reconstruct (or justify with an alternative
argument) the intuitive decision of the black box with deontic logical reasoning
applying moral or legal standards. A pluralistic, expressive “normative reasoning
infrastructure”, such as LogiKEy [4], should e.g. be able to support this process.
A sub-symbolic solution could create an independent (second) neural network
to produce reasons for the output of the (first) decision network (e.g. autonomous
driving control). Of course, the structure of this “reasoning net” process is again
hidden. Yet, if the outcoming reasons coherently comply with prescribed social
and ethico-legal standards the lack of transparency in the second black box
constitutes less of a problem.
Robust solutions for aSRMs could even seek to integrate and align these
two options. Moreover, in both scenarios the introduced feedback loop of giving
and taking reasons could be integrated as learning environment (self-supervised
learning) for the initial, intuitive layer of autonomous decision making, with the
eventual effect that differences at both layers may gradually dissolve.
Allowing various kinds of reasons, SRMs & aSRMs advance normative plural-
ism and may integrate different (machine-)ethical traditions: deontological, con-
sequentialist and virtue ethics. “Reasonable pluralism” in recent moral and po-
litical philosophy defines reasonableness by meta-level procedures like “reflective
equilibrium” and “overlapping consensus” [20] or “rational discourse” [11]. Con-
temporary legal philosophy and theory has enfolded how law could act as demo-
cratic real-world implementation of these meta-procedures, structuring public
deliberation and argumentation over conflicting reasons [1, 15]. Constructing a
pluralist aSRM substantially widens the mostly consequentialist contemporary
approaches [5, 9] to machine ethics and moral IAS.
4Reasonable Machines: Implementation
The implementation of Reasonable Machines requires expertise from differ-
ent areas: pluralistic normative reasoning, formal ethics and legal theory, ex-
pressive ontologies and semantic web taxonomies, human-computer interaction,
rule-based systems, automated theorem proving, argumentation technology, neu-
ral architectures and machine learning. Acknowledging the complexity of each
field, Reasonable Machines research should complement top-down construc-
tion of responsible machine architecture with bottom-up developments starting
from existing works in different domains. More concretely, we propose a modu-
Reasonable Machines: A Research Manifesto 5
lar and stepwise implementation of our research scheme based on the following
M1: Responsible Machine Architecture. The vision of an aSRM and its
parallel to human SRM needs to be further explored to guide and refine the over-
all architectural design of Reasonable Machines based on respective system
components responsible for generating justifications, for conducting compliance
checks and for governing the action executions of an IAS.
M2: Ethico-Legal Ontologies. Ethico-legal ontologies constitute a core
ingredient to enable the computation, assessment and communication of aSRM-
based rational justifications in the envisioned ethico-legal governance compo-
nents for IASs, and they are also key for black box independent user-explanations
in form of rational arguments. We propose the development of expressive ethico-
legal upper-level ontologies to guide and connect the encoding of concrete ethico-
legal domain-level theories (regulatory codes) [13, 8]. Moreover, we propose the
concrete regulatory codes to be complemented with an abstract ethico-legal value
ontology, for example, as “discoursive grammar” of justification [17].
M3: Symbolic Reasoning Tools. For the implementation of pluralistic,
expressive and paradox-free normative reasoning at the upper-level, the LogiKEy
framework [4] can e.g. be adapted and further advanced. LogiKEy works with
shallow semantical embeddings (SSEs) of (combinations of) non-classical logics
in classical higher-order logic (HOL). HOL thereby serves as a meta-logic, rich
enough to support the encoding of a plurality of “object logics” (e.g. conditional,
deontic or epistemic logics and combinations thereof). The embedded “object
logics” are used for the iterative, experimental encoding of normative theories.
This generic approach shall ideally be integrated with specialized solutions based
e.g. on semantic web reasoning, logic programming, answer set programming,
and with formalized argumentation for ethical [21] or legal [3] systems design.
M4: Interpretable AI Systems. Sub-symbolic solutions to SRM-based
accountability and governance challenge could develop a hidden reasoning net,
which might be trained with legal and ethical use-cases. Moreover, techniques
in “explainable AI” [10] have to be assessed and, if possible, integrated with the
symbolic aSRM tools to be developed in M3 in order to provide guidance to
their computations and search processes. The more information can be obtained
about the particular information bits that trigger the decisions of the black box
systems we want to govern, the easier the corresponding reasoning tasks, i.e. the
search for justifications, should become in the associated, symbolic aSRM tool.
M5: Human-Machine Communication & Interaction. The intended
aSRM-based justifications generated by the tools developed in M3 and M4 re-
quire arguments and rational explanation which are understandable for different
AI ecosystems [19], including human users, collect decision scenarios between
machines and independent verification tools. Here, the development of respec-
tive techniques could build on argumentation theory in combination with recent
advances towards a computational hermeneutics [7]. An overarching objective
of Reasonable Machines is to contribute to trustful and fruitful interaction
between human and IASs.
M6: Cloud-based Reasoning Workbench. To facilitate access to the
proposed knowledge representation and reasoning solutions, and also to host the
ethico-legal theories, a cloud-based reasoning workbench should be implemented.
This workbench would (i) integrate the bottom-up construed components and
tools from M2-M5 and (ii) implement instances of the top-down governance
architecture(s) developed in M1 based on (i). This cloud-based solution could
be developed in combination with, or as an alternative to, more independent
solutions based e.g. on agent-based development frameworks [23].
M7: Use Cases and Empirical Studies. The overall system framework
needs to be adequately prepared to support changing use cases and empirical
studies. Concrete use cases with high ethical and legal potential must be defined
and employed to guide the research and development work, as for example the
representative issue on self-driving cars [5]. Empirical studies should support and
inform the constructive development process. For testing the ethico-legal value
ontology in M2, for example, we could try to demonstrate that it can make sense
out of the rich MIT Moral Machine experiment data [2]. When its architecture
evolves, it would be highly valuable to design a genuine aSRM experiment.
5 Conclusion
The Reasonable Machines vision and research requires the integration of
heterogeneous and interdisciplinary expertise to be fruitfully implemented. The
cloud-based framework we envision would ideally be widely available and reusable,
and it could become part of related, bigger initiatives towards the sharing of crit-
ical AI infrastructure (such as the vision towards a CERN for
AI). The implementation of the depicted program requires substantial resources
and investment in foundational AI research and in practical system development,
but it reflects the urgent and timely need for the development of trustworthy AI
The possible outreach of the Reasonable Machines idea is even far beyond
an ecosystem of trust. To enable machines to give normative reasons for their
decisions and actions means to capacitate them of communicative action [11],
or at least to engage in constitutive communication of social systems [18]. The
capacity to give and take reasons is a crucial step towards fully autonomous
normative (moral and legal) agency. Moreover, our research, in the long run,
paves way for interesting further studies and experiments on integrated neuro-
symbolic AI architectures and on the emergence of patterns of self-reflection in
intelligent autonomous machines.
Acknowledgement: We thank David Fuenmayor and the anonymous reviewers
for their helpful comments to this work.
1. Alexy, R.: Theorie der juristischen Argumentation. Suhrkamp, Frankfurt/M (1978)
2. Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F.,
Rahwan, I.: The Moral Machine experiment. Nature 563(7729), 59–64 (2018)
3. Benzmüller, C., Fuenmayor, D., Lomfeld, B.: Encoding Legal Balancing: Automat-
ing an Abstract Ethico-Legal Value Ontology in Preference Logic. In: MLR 2020,
Preprint: (2020)
4. Benzmüller, C., Parent, X., van der Torre, L.: Designing Normative Theories for
Ethical and Legal Reasoning: LogiKEy Framework, Methodology, and Tool Sup-
port. Artificial Intelligence 287(103348) (2020)
5. Bonnefon, J.-F., Shariff, A., Rahwan, I.: The social dilemma of autonomous vehi-
cles. Science 352(6293), 1573–1576 (2016)
6. European Commission, On Artificial Intelligence – A European approach to excel-
lence and trust. European Commission White Paper, COM(2020) 65 final (2020)
7. Fuenmayor, D., Benzmüller, C.: A Computational-Hermeneutic Approach for Con-
ceptual Explicitation. In: Nepomuceno, A.,, (eds.) Model-Based Reasoning
in Science and Technology. Inferential Models for Logic, Language, Cognition and
Computation, SAPERE, pp. 441–469. Springer, Cham (2019)
8. Fuenmayor, D., Benzmüller, C.: Harnessing Higher-Order (Meta-)Logic to Rep-
resent and Reason with Complex Ethical Theories. In: PRICAI 2019: Trends in
Artificial Intelligence, LNAI, vol. 11670, pp. 418–432. Springer, Heidelberg (2019)
9. Greene, J., Rossi, F., Tasioulas, J., Venable, K.B., Williams, B.C.: Embedding
Ethical Principles in Collective Decision Support Systems. In: Schuurmans, D.,
Wellman, M.P. (eds.) Proceedings of the Thirtieth AAAI Conference on Artificial
Intelligence, pp. 4147–4151. AAAI Press (2016)
10. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A
Survey of Methods for Explaining Black Box Models. ACM Comput. Surv. 51(5)
11. Habermas, J.: Theorie des kommunikativen Handelns. Frankf./M: Suhrkamp (1981)
12. Haidt, J.: The emotional dog and its rational tail: a social intuitionist approach to
moral judgment. Psychol. Rev. 108(4), 814–34 (2001)
13. Hoekstra, R., Breuker, J., Bello, M.D., Boer, A.: LKIF Core: Principled Ontology
Development for the Legal Domain. In: Breuker, J.,, (eds.) Law, Ontologies
and the Semantic Web - Channelling the Legal Information Flood, Frontiers in
Artificial Intelligence and Applications, pp. 21–52. IOS Press (2009)
14. Kahnemann, D.: Thinking, Fast and Slow. Farrar, Straus and Giroux (2013)
15. Lomfeld, B.: Die Gründe des Vertrages: Eine Diskurstheorie der Vertragsrechte.
Mohr Siebeck, Tübingen (2015)
16. Lomfeld, B.: Emotio Iuris. Skizzen zu einer psychologisch aufgeklärten Methoden-
lehre des Rechts. In: Köhler, Müller-Mall, Schmidt, Schnädelbach, (eds.) Recht
Fühlen, pp. 19–32. Fink, München (2017)
17. Lomfeld, B.: Grammatik der Rechtfertigung: Eine kritische Rekonstruktion der
Rechts(fort)bildung. Kritische Justiz 52(4) (2019)
18. Luhmann, N.: Soziale Systeme: Grund lage einer allgemeinen Theorie. Frankfurt/M:
Suhrkamp (1984)
19. Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal,
C., Crandall, J.W., Christakis, N.A., Couzin, I.D., Jackson, M.O., Jennings, N.R.,
Kamar, E., Kloumann, I.M., Larochelle, H., Lazer, D., McElreath, R., Mislove, A.,
Parkes, D.C., Pentland, A.‘, Roberts, M.E., Shariff, A., Tenenbaum, J.B., Wellman,
M.: Machine behaviour. Nature 568(7753), 477–486 (2019)
20. Rawls, J.: Justice as Fairness: A Restatement. Harvard University Press, Cam-
bridge/MA (2001)
21. Verheij, B.: Formalizing value-guided argumentation for ethical systems design.
Artif. Intell. Law 24(4), 387–407 (2016)
22. Wallach, W., Allen, C.: Moral machines: Teaching robots right from wrong. Oxford
University Press (2008)
23. Wisniewski, M., Steen, A., Benzmüller, C.: LeoPARD - A Generic Platform for the
Implementation of Higher-Order Reasoners. In: Intelligent Computer Mathematics
- CICM 2015, LNCS, vol. 9150, pp. 325–330. Springer, Heidelberg (2015)
Responsible Machine Architecture
ethico-legal governance of intelligent autonomous agents
Symbolic Reasoning Tools
pluralistic normative reasoning
rule-based normative reasoning
integrated and guided by M4
Ethico-Legal Ontologies
ethico-legal upper-level ontology
value ontology (moral “grammar”)
ethico-legal regulation (code)
Interpretable AI Systems
ethico-legal reasoning net
interpretable AI to inform M3
Human-Machine Communication & Interaction
human-understandable rational arguments
human-centered interaction
Cloud-based Reasoning Workbench
access at scale with little risk and minimal costs
Use Cases and Empirical Studies
grand vision (top-down) & module specific (bottom-up)
Fig. 1. Modular structure of Reasonable Machines research.
... In more general terms our goal is to work towards some form of legally and ethically reasonable machines [4], capable of exchanging rational justifications for the actions they take. Building up a capacity in such reasonable machines to engage in value-oriented ethico-legal argumentation is thus a relevant challenge to address. ...
Full-text available
Presentation at the AAAI Spring Symposium on Implementing AI Ethics
... Ethico-legal ontologies also constitute a core ingredient to enable the computation, assessment and communication of abstract-level justifications in the future ethico-legal governance of AI (Benzmüller and Lomfeld 2020). A sound implementation of any legally accountable "moral machine" requires the development of upper-level ontologies to guide and connect the encoding of concrete regulatory codes (Hoekstra et al. 2009;Fuenmayor and Benzmüller 2019b) or legal cases. ...
Full-text available
Enabling machines to legal balancing is a non-trivial task challenged by a multitude of factors some of which are addressed and explored in this work. We propose a holistic approach to formal modeling at different abstraction layers supported by a pluralistic framework in which the encoding of an ethico-legal value and upper ontology is developed in combination with the exploration of a formalization logic, with legal domain knowledge and with exemplary use cases until a reflective equilibrium is reached. Our work is enabled by a meta-logical approach to universal logical reasoning and it applies the recently introduced LogiKEy methodology for designing normative theories for ethical and legal reasoning. The particular focus in this paper is on the formalization and encoding of a value ontology suitable e.g. for explaining and resolving legal conflicts in property law (wild animal cases).
Full-text available
A framework and methodology—termed LogiKEy—for the design and engineering of ethical reasoners, normative theories and deontic logics is presented. The overall motivation is the development of suitable means for the control and governance of intelligent autonomous systems. LogiKEy's unifying formal framework is based on semantical embeddings of deontic logics, logic combinations and ethico-legal domain theories in expressive classic higher-order logic (HOL). This meta-logical approach enables the provision of powerful tool support in LogiKEy: off-the-shelf theorem provers and model finders for HOL are assisting the LogiKEy designer of ethical intelligent agents to flexibly experiment with underlying logics and their combinations, with ethico-legal domain theories, and with concrete examples—all at the same time. Continuous improvements of these off-the-shelf provers, without further ado, leverage the reasoning performance in LogiKEy. Case studies, in which the LogiKEy framework and methodology has been applied and tested, give evidence that HOL's undecidability often does not hinder efficient experimentation.
Full-text available
The computer-mechanization of an ambitious explicit ethical theory, Gewirth’s Principle of Generic Consistency, is used to showcase an approach for representing and reasoning with ethical theories exhibiting complex logical features like alethic and deontic modalities, indexicals, higher-order quantification, among others. Harnessing the high expressive power of Church’s type theory as a meta-logic to semantically embed a combination of quantified non-classical logics, our work pushes existing boundaries in knowledge representation and reasoning. We demonstrate that intuitive encodings of complex ethical theories and their automation on the computer are no longer antipodes.
Full-text available
With the rapid development of artificial intelligence have come concerns about how machines will make moral decisions, and the major challenge of quantifying societal expectations about the ethical principles that should guide machine behaviour. To address this challenge, we deployed the Moral Machine, an online experimental platform designed to explore the moral dilemmas faced by autonomous vehicles. This platform gathered 40 million decisions in ten languages from millions of people in 233 countries and territories. Here we describe the results of this experiment. First, we summarize global moral preferences. Second, we document individual variations in preferences, based on respondents’ demographics. Third, we report cross-cultural ethical variation, and uncover three major clusters of countries. Fourth, we show that these differences correlate with modern institutions and deep cultural traits. We discuss how these preferences can contribute to developing global, socially acceptable principles for machine ethics. All data used in this article are publicly available.
Full-text available
In the last years many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness sometimes at the cost of scarifying accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, delineating explicitly or implicitly its own definition of interpretability and explanation. The aim of this paper is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Full-text available
The persuasiveness of an argument depends on the values promoted and demoted by the position defended. This idea, inspired by Perelman’s work on argumentation, has become a prominent theme in artificial intelligence research on argumentation since the work by Hafner and Berman on teleological reasoning in the law, and was further developed by Bench-Capon in his value-based argumentation frameworks. One theme in the study of value-guided argumentation is the comparison of values. Formal models involving value comparison typically use either qualitative or quantitative primitives. In this paper, techniques connecting qualitative and quantitative primitives recently developed for evidential argumentation are applied to value-guided argumentation. By developing the theoretical understanding of intelligent systems guided by embedded values, the paper is a step towards ethical systems design, much needed in these days of ever more pervasive AI techniques.
Full-text available
Autonomous vehicles (AVs) should reduce traffic accidents, but they will sometimes have to choose between two evils, such as running over pedestrians or sacrificing themselves and their passenger to save the pedestrians. Defining the algorithms that will help AVs make these moral decisions is a formidable challenge. We found that participants in six Amazon Mechanical Turk studies approved of utilitarian AVs (that is, AVs that sacrifice their passengers for the greater good) and would like others to buy them, but they would themselves prefer to ride in AVs that protect their passengers at all costs. The study participants disapprove of enforcing utilitarian regulations for AVs and would be less willing to buy such an AV. Accordingly, regulating for utilitarian algorithms may paradoxically increase casualties by postponing the adoption of a safer technology.
Research on moral judgment has been dominated by rationalist models, in which moral judgment is thought to be caused by moral reasoning. The author gives 4 reasons for considering the hypothesis that moral reasoning does not cause moral judgment; rather, moral reasoning is usually a post hoc construction, generated after a judgment has been reached. The social intuitionist model is presented as an alternative to rationalist models. The model is a social model in that it deemphasizes the private reasoning done by individuals and emphasizes instead the importance of social and cultural influences. The model is an intuitionist model in that it states that moral judgment is generally the result of quick, automatic evaluations (intuitions). The model is more consistent than rationalist models with recent findings in social, cultural, evolutionary, and biological psychology, as well as in anthropology and primatology.