Moral Machines: Contradiction in Terms, or Abdication of Human Responsibility?

Full text

Preprint version of: Allen, C., Wallach, W. (2011) Moral Machines: Contradiction in Terms, or
Abdication of Human Responsibility? In P. Lin, K. Abney, G. Bekey (Eds.) Robot Ethics: The
Ethical and Social Implications of Robotics. Cambridge: MIT Press, 55-68.
Chapter 4
Moral Machines
4
Moral Machines:
Contradiction in Terms, or Abdication of Human Responsibility?
Colin Allen and Wendell Wallach
Over the past twenty years, philosophers, computer scientists, and engineers have begun
reflecting seriously on the prospects for developing computer systems and robots capable of
making moral decisions. Initially, a few articles were written on the topic (Gips 1991, 12;
Clarke 1993, 5; 1994, 6; Moor 1995, 17; Allen, Varner, and Zinser 2000, 1; Yudkowsky 2001,
23) and these were followed by preliminary software experiments (Danielson 1992, 8; 2003,
9; McLaren and Ashley 1995, 15; McLaren 2003, 16; Anderson, Anderson, and Armen 2006,
2; Guarini 2006, 13). A new field of inquiry directed at the development of artificial moral
agents (AMAs) began to emerge, but it was largely characterized by a scattered collection of
ideas and experiments that focused on different facets of moral decision making. In our
recent book, Moral Machines: Teaching Robots Right from Wrong (Wallach and Allen 2009,
!100

20), we attempted to bring these strands together and to propose a comprehensive framework
for this new field of inquiry, which is referred to by a number of names including machine
morality, machine ethics, artificial morality, and friendly AI. Two other books on related
themes, J. Storrs Hall’s Beyond AI: Creating the Conscience of the Machine (2007, 14) and
Ronald Arkin’s Governing Lethal Behavior in Autonomous Robots (2009, 3), have also been
published recently. Moral Machines (MM) has been well received, but a number of
objections have been directed at our approach and at the very project of developing machines
capable of making moral decisions. In this chapter, we provide a brief précis of MM. We then
list and respond to key objections that have been raised about our project.
4.1 Toward Artificial Moral Agents
The human-built environment increasingly is being populated by artificial agents, which
combine limited forms of artificial intelligence with autonomous (in the sense of
unsupervised) activity. The software controlling these autonomous systems is, to date,
“ethically blind” in two ways. First, the decision-making capabilities of such systems do not
involve any explicit representation of moral reasoning. Second, the sensory capacities of
these systems are not tuned to ethically relevant features of the world. A breathalyzer-
equipped car might prevent you from starting it, but it cannot tell whether you are bleeding to
death in the process. Nor can it appreciate the moral significance of its refusal to start the
engine.
In MM, we argued that it is necessary for developers of these increasingly
autonomous systems (robots and software bots) to make them capable of factoring ethical
!101

and moral considerations into their decision making. Engineers exploring design strategies
for systems sensitive to moral considerations in their choices and actions will need to
determine what role ethical theory should play in defining control architectures for such
systems.
There are many applications that underscore the need for AMAs. Among the most
dramatic examples that grab public attention are the development of military robots (both
land and airborne) for deployment in the theater of battle, and the introduction of service
robots in the home and for healthcare. However, autonomous bots within existing computer
systems are already making decisions that affect humans, for good or for bad. The topic of
morality for “(ro)bots” (a spelling convention we introduced in MM to represent both robots
and software bots within computer systems) has long been explored in science fiction by
authors such as Isaac Asimov, with his Three Laws of Robotics, in television shows, such as
Star Trek, and in various Hollywood movies. However, our project was not and is not
intended to be science fiction. Rather, we argued that current developments in computer
science and robotics necessitate the project of building artificial moral agents.
Why build machines with the ability to make moral decisions? We believe that AMAs
are necessary and, in a weak sense, inevitable; in a weak sense, because we are not
technological determinists. Individual actors could have chosen not to develop the atomic
bomb. Likewise, the world could declare a moratorium on the development and deployment
of autonomous (ro)bots. However, such a moratorium is very unlikely. This makes the
development of AMAs necessary since, as Rosalind Picard (1997, 19) so aptly put it, “The
greater the freedom of a machine, the more it will need moral standards.” Innovative
!102

technologies are converging on sophisticated systems that will require some capacity for
moral decision making. With the implementation of driverless trains—already common at
airports and beginning to appear in more complicated situations such as the London
Underground and the Paris and Copenhagen metro systems—the “runaway trolley cases”
invented by ethicists to study moral dilemmas ( Foot 1967) may represent actual challenges
for artificial moral agents.
Among the difficult tasks for designers of such systems is to specify what the goals
should be, that is, what is meant by a “good” artificial moral agent? Computer viruses are
among the software agents that already cause harm. Credit card approval systems (and
automated stock trading systems) are among the examples of autonomous systems that
already affect daily life in ethically significant ways, but these are “ethically blind” because
they lack moral decision-making capacities. Pervasive and ubiquitous computing, the
introduction of service robots in the home to care for the elderly, and the deployment of
machine-gun carrying military robots expand the possibilities of software and robots, without
sensitivity to ethical considerations harming people.
The development of AI includes both autonomous systems and technologies that
augment human decision making (decision support systems and, eventually, cyborgs), each
of which raises different ethical considerations. In MM, we focus primarily on the
development of autonomous systems.
Our framework for understanding the trajectory toward increasingly sophisticated
artificial moral agents emphasizes two dimensions: autonomy and sensitivity to morally
relevant facts (figure 4.1). Systems with very limited autonomy and sensitivity have only
!103

“operational morality,” meaning that their moral significance is entirely in the hands of
designers and users. As machines become more sophisticated, a kind of “functional morality”
is possible, where the machines themselves have the capacity for assessing and responding to
moral challenges. The creators of functional morality in machines face many constraints due
to the limits of present technology. This framework can be compared to the categories of
artificial ethical agents described by James Moor (2006, 18), which range from agents whose
actions have ethical impact (implicit ethical agents) to agents that are explicit ethical
reasoners (explicit ethical agents). As does Moor, we emphasize the near-term development
of explicit or functional moral agents. However, we do recognize that, at least in theory,
artificial agents might eventually attain genuine moral agency with responsibilities and
rights, comparable to those of humans.
[Figure 4.1: here]
!104
Autonomy
Ethical sensitivity
low
low
high
high
today's
(ro)bots
??
operational
morality
functional
morality
full
moral
agency

Do we want computers making moral decisions? Worries about whether it is a good
idea to build artificial moral agents are examples of more general concerns about the effects
of technology on human culture. Traditional philosophy of technology provides a context for
the more specific concerns raised by artificial intelligence and specifically AMAs. For
example, human anthropomorphism of robotic dolls, robopets, household robots, companion
robots, sex toys, and even military robots, raises questions of whether these artifacts
dehumanize people and substitute impoverished relationships for real human interactions.
Some concerns, such as whether AMAs will lead humans to abrogate responsibility to
machines, seem particularly pressing. Other concerns, such as the prospect of humans
becoming literally enslaved to machines, seem highly speculative. The unsolved problem of
technology risk assessment is how seriously to weigh catastrophic possibilities against the
obvious advantages provided by new technologies. Should, for example, a precautionary
principle be invoked when risks are fairly low? Historically, philosophers of technology have
served mainly as critics, but a new breed of philosophers see themselves as engaged in
engineering activism as they help introduce sensitivity to human values into the design of
systems.
Can (ro)bots really be moral? How closely could artificial agents, lacking human
qualities such as consciousness and emotions, come to being considered moral agents? There
are many people, including many philosophers, who believe that a “mere” machine cannot be
a moral agent. We (the authors) remain divided on whether this is true or not. Nevertheless,
we believe the need for AMAs suggests a pragmatically oriented approach. We accept that
full-blown moral agency (which depends on “strong” AI) or even “weak” AI that is
!105

nevertheless powerful enough to pass the Turing Test—the procedure devised by Alan
Turing (1950) by which a machine may be tested anonymously for its linguistic equivalence
to an intelligent human language user—may be beyond current or even future technology.
Only time will tell. Nevertheless, the more immediate project of developing AMAs can be
located in the space between operational morality and genuine moral agency (figure 4.1)—
the niche we labeled “functional morality.” We believe that traditional symbol-processing
approaches to artificial intelligence and more recent approaches based on artificial neural
nets and embodied cognition could provide technologies supporting functional morality.
4.2 Philosophers, Engineers, and the Design of Artificial Moral
Agents
Philosophers like to think in terms of abstractions. Engineers like to think in terms of
buildable designs. Bridging these two cultures is not a trivial task. Nevertheless, there are
benefits for each side to try to accommodate the concerns of the other. Theory can inform
design, and vice versa. How might moral capacities be implemented in (ro)bots? We
approach this question by considering possible architectures for AMAs, which fall within two
broad approaches: the top-down imposition of an ethical theory, and the bottom-up building
of systems that aim at goals or standards which may or may not be specified in explicitly
theoretical terms.
Implementing any top-down ethical theory of ethics in an artificial moral agent will
pose both computational and practical challenges. One central concern is framing the
background information necessary for rule- and duty-based conceptions of ethics and for
!106

utilitarianism. Asimov’s Three Laws come readily to mind when considering rules for
(ro)bots, but even these apparently straightforward principles are not likely to be practical for
programming moral machines. The high-level rules, such as the Golden Rule, the deontology
of Kant’s categorical imperative, or the general demands of consequentialism, for example,
utilitarianism, also fail to be computationally tractable. Nevertheless, the various principles
embodied in different ethical theories may all play an important guiding role as heuristics
before actions are taken, and during post hoc evaluation of actions.
Bottom-up approaches to the development of AMAs attempt to emulate learning,
developmental, and evolutionary processes. The application of methods from machine
learning, theories of moral development, and techniques from artificial life (Alife), and
evolutionary robotics may, like the various ethical theories, all contribute to the development
of AMAs and the emergence of moral capacities from more general aspects of intelligence.
Bottom-up approaches also hold out the prospect that moral behavior is a self-organizing
phenomenon, in which cooperation and a shared set of moral instincts (if not a “moral
grammar”) might emerge. (It remains an open question whether explicit moral theorizing is
necessary for such organization.) A primary challenge for bottom-up approaches is how to
provide sufficient safeguards against learning or evolving bad behaviors as well as good.
The difficulties of applying general moral theories in a completely top-down fashion
to AMAs motivate the return to another source of ideas for the development of AMAs: the
virtue-based conception of morality that can be traced back to Aristotle. Virtues constitute a
hybrid between top-down and bottom-up approaches, in that the virtues themselves can be
explicitly described (at least to some reasonable approximation), but their acquisition as
!107

moral character traits seems essentially to be a bottom-up process. Placing this approach in a
computational framework, neural network models provided by connectionism seem
especially well suited for training (ro)bots to distinguish right from wrong (DeMoss 1998,
10).
4.3 Early Research on the Development of AMAs, and Future
Challenges
A major goal of our book was not just to raise many questions, but also to provide a resource
for further development of AMAs. Software currently under development for moral decision
making by (ro)bots utilize a variety of strategies, including case-based reasoning or casuistry,
deontic logic, connectionism (particularism), and the prima facie duties of W. D. Ross (1930)
(also related to the principles of biomedical ethics). In addition to agent-based approaches
that focus on the reasoning of one agent, researchers are working with multiagent
environments and with multibots. Experimental applications range from ethical advisors in
healthcare to control architectures, for ensuring that (ro)bot soldiers won’t violate
international conventions.
The top-down and bottom-up approaches to artificial moral agents emphasize the
importance in ethics of the ability to reason. However, much of the recent empirical literature
on moral psychology emphasizes faculties besides rationality. Emotions, empathy, sociability,
semantic understanding, and consciousness are all important to human moral decision
making, but it remains an open question whether, or when, these will be essential to artificial
moral agents, and, if needed, whether they can be implemented in machines. Cutting-edge
!108

scientific investigation in the areas of affective computing, embodied cognition, and machine
consciousness that is aimed at providing computers and robots with the kinds of
“suprarational” capacities underlying those social skills, may be essential for sophisticated
human–computer interaction. However, to date, there are no working projects that combine
emotion-processing, social skills, or embodied cognition in (ro)bots with the moral capacities
of AMAs.
Recently, there has been a resurgence of interest in general, comprehensive models of
human cognition that aim to explain higher-order cognitive faculties, such as deliberation and
planning. Moral decision making is arguably one of the most challenging tasks for
computational approaches to higher-order cognition. We argue that this challenge can be
fruitfully pursued in the context of a comprehensive computational model of human
cognition. MM focuses specifically on Stan Franklin’s LIDA model (Franklin et al. 2005, 11;
Wallach, Franklin, and Allen 2010, 21). LIDA provides both a set of computational tools and
an underlying model of human cognition, which provides mechanisms that are capable of
explaining how an agent’s selection of its next action arises from bottom-up collection of
sensory data and top-down processes for making sense of its current situation. The LIDA
model also supports the integration of emotions into the human decision-making process, and
elucidates a process whereby an agent can work through an ethical problem to reach a
solution that takes account of ethically relevant factors.
The prospect of computers making moral decisions poses an array of future dangers
that are difficult to anticipate, but will, nevertheless, need to be monitored and managed.
Public policy and mechanisms of social and business liability management will both play a
!109

role in the safety, direction, and speed in which artificial intelligent systems are developed.
Fear is not likely to stop scientific research, but it is likely that various fears will slow it
down. Mechanisms for distinguishing real dangers from speculation and hype, fueled by
science fiction, are needed. Means of addressing the issues of rights and accountability for
(ro)bots and their designers will require attention to topics such as legal personhood, self-
replicating robots, the possibility of a “technological singularity” during which AI outstrips
human intelligence, and the transhumanist movement, which sees the future of humanity
itself as an inevitable (and desirable) march toward cyborg beings.
Despite our emphasis in the book on the prospects for artificial morality, we believe
that a richer understanding of human moral decision making is facilitated by the pursuit of
AMAs (Wallach 2010, 22). The project of designing AMAs feeds back into our
understanding of ourselves as moral agents and of the nature of ethical theory itself. The
limitations of current ethical theory for developing the control architecture of artificial moral
agents highlight deep questions about the purpose of such theories.
4.4 Challenges, Objections, and Criticisms
Since publishing MM, we have encountered several key critiques of the framework we
offered for why AMAs are needed, and the approaches for building and designing moral
machines. These fall into six categories, which we address in the sections that follow:
1. Full moral agency for machines requires capacities or features we either did not mention
in MM or whose centrality we did not emphasize adequately.
!110

2. Some features required for full moral agency cannot be implemented in a computer
system or robot.
3. The approaches we propose for developing AMAs are too humancentric. (Ro)bots will
need a moral code that does not necessarily duplicate human morality.
4. The work of researchers focused on ensuring that a technological singularity will be
friendly to humans (friendly AI) was not given its due in MM.
5. In focusing on the prospects for building AMAs, we imply that dangers posed by (ro)bots
can be averted, whereas many of the dangers cannot be averted easily. In other words,
MM contributes to the illusion that there is a technological fix, and thereby dilutes the
need to slow, and even stop, the development of harmful systems.
6. The claim that the attempt to design AMAs helps us understand human moral decision
making better could be developed more fully.
4.4.1 Full Moral Agency
In MM, we took what we consider to be an unusually comprehensive approach to moral
decision making by including the role of top-down theories, bottom-up development,
learning, and the suprarational capacities that support emotions and social skills. And yet the
most common criticisms we have heard begin with, “Full moral agency requires _______.”
The blank space is filled in with a broad array of capacities, virtues, and features of a moral
society that the speaker believes we either failed to mention, or whose centrality in moral
decision making we failed to underscore adequately. Being compassionate or emphatic,
!111

having a conscience, or being a member of virtuous communities, are among the many items
that have come up as critics fill in the blank space.
Some critics, coming especially from a Kantian perspective, believe that talk of
morality is misguided in connection with agents that lack the potential to choose to act
immorally. On this conception, central to human morality, is the struggle between acting in
self-interest and acting out of duty to others, even when it goes against self-interest. There are
several themes running through this conception of moral life, including the metaphysical
freedom to choose one’s principles and to accept responsibility for acting upon them. Such
critics maintain that machines, by their very nature, lack the kind of freedom required. We are
willing to grant the point for the sake of argument, but we resist what seems to be a corollary
for several critics: It is a serious conceptual mistake to speak of “moral agency” in
connection with machines. For reasons already rehearsed in MM, we think that the notion of
functional morality for machines can be described philosophically and pursued as an
engineering project. But if the words bother Kantians, let them call our project by another
name, such as norm-compliant computing.
We do not deny that it is intriguing to consider which attributes are required for
artificial agents to be considered full moral agents, the kinds of society in which artificial
agents would be accepted as a full moral agents, or the likelihood of (ro)bots ever being
embraced as moral agents. But there are miles to go before the full moral agency of (ro)bots
can be realistically conceived. Our focus has been on the steps between here and there. Moral
decision-making faculties will have to develop side by side with other features of
autonomous systems. It is still unclear which platforms or which strategies will be most
!112

successful in the development of AMAs. Full moral agency is a fascinating subject, but can
distract from the immediate task of making increasingly autonomous (ro)bots safer and more
respecting of moral values, given present or near-future technology.
4.4.2 Inherent Limits of Existing Computer Platforms
From John Searle’s Chinese Room thought experiment against the possibility of genuine
intelligence in a computer (Searle 1980), to Roger Penrose’s proposal that the human mind
depends essentially on quantum mechanical principles to exceed the capacities of any
computer ( Penrose 1989), there is no shortage of theorists who have argued that existing
computational platforms fail to capture essential features of intelligence and mental activity.
Some recent critics of our approach (Byers and Schleifer 2010, 4) have argued that the
inherent capacity of the human mind to intuitively comprehend mathematical notions and
work creatively with them is, at root, the same capacity that enables creative, intuitive, and
flexible understanding of moral issues. That human comprehension outstrips some rule-based
systems is uncontroversial. That it outstrips all rule-based, algorithmic systems is less
obvious to us. But even if true, it does not rule out moral machines—only full moral agents
that are rule based. Furthermore, even if we are stuck with rule-based systems for the
foreseeable future (which, depending on one’s definition of rule based, may or may not
include machines implementing the kinds of bottom-up and suprarational capacities we
surveyed), it doesn’t follow that there’s no advantage to trying to model successful moral
reasoning and judgment in such systems. Despite human brilliance and creativity, there are
rule-based, algorithmic systems capable of outperforming humans on many cognitive tasks,
!113

and which make perfectly useful tools for a variety of purposes. The fact that some tasks are
currently beyond our ability to build computers to do them well (Byers and Schleifer mention
the game of bridge) only shows that more work is necessary to build machines that are
sensitive to the “almost imperceptible” (but necessarily perceptible) cues that current
computational models fail to exploit, but to which humans are exquisitely attuned. As before,
however, even if we were to admit that there is a mathematically provable computational
limit to the capacity of machines to replicate human judgment, this does not undermine the
need to implement the best kind of functional morality possible.
4.4.3 AMAs Will Need a Moral Code Designed for Robots, Not a Facsimile
of Human Morality
By framing our discussion in MM in terms of the top-down implementation of ethical
theories or the bottom-up development of human-like moral capacities, we opened ourselves
to the criticism that our approach is too focused on the re-creation of human morality for
(ro)bots. Peter Danielson (2009, 9), for example, raises the quite reasonable possibility that
the particular situations in which machines are deployed might make the implementation of
more limited forms of morality for artificial agents more tractable and more appropriate. In
this we agree with Danielson, and although we did touch upon topics such as special virtues
for artificial agents, we concede that there is a difference of emphasis from what critics like
Danielson might have desired. At the very least, we are pleased that this discussion has been
sparked by MM, and it certainly opens up options for the design of AMAs that we did not
explore in detail. Nevertheless, given that technology will continue to race ahead, providing
!114

(ro)bots with sensory, computational, and motor capacities that humans may not have, we
believe it is important to pursue a less-limited version of artificial morality than our critics
have urged.
4.4.4 The Technological Singularity and Friendly AI
The project of building AMAs is bracketed by the more conservative expectations of
computer scientists, engaged with the basic challenges and thresholds yet to be crossed, and
the more radical expectations of those who believe that human-like and superhuman systems
will be built in the near future. There are a wide variety of theories and opinions about how
sophisticated computers and robotic systems will become in the next twenty to fifty years.
Two separate groups focused on ensuring the safety of (ro)bots have emerged around these
differing expectations: the machine ethics community and the singularitarians (friendly AI),
exemplified by the Singularity Institute for Artificial Intelligence (SIAI). Those affiliated
with SIAI are specifically concerned with the existential dangers to humanity posed by AI
systems that are smarter than humans. MM has been criticized for failing to give fuller
attention to the projects of those dedicated to a singularity in which AI systems friendly to
humans prevail.
SIAI has been expressly committed to the development of general mathematical
models that can, for example, yield probabilistic predictions about future possibilities in the
development of AI. One of Eliezer Yudkowsky’s projects is motivationally stable goal
systems for advanced forms of AI. If satisfactory predictive models or strategies for stable
goal architectures can be developed, their value for AMAs is apparent. But will they be
!115

developed, and what other technological thresholds must be crossed, before such strategies
could be implemented in AI? In a similar vein, no one questions the tremendous value
machine learning would have for facilitating the acquisition by AI systems of many skills,
including moral decision making. But until sophisticated machine learning strategies are
developed, discussing their application is speculative. That said, since the publication of MM,
there has been an increase in projects that could lead to further collaboration between these
two communities, a prospect we encourage.
4.4.5 The Illusion that There Is a Technological Fix to the Dangers AI Poses
Among our critics, Deborah Johnson has been the most forceful about the inadequacy of our
nearly exclusive focus on the technology involved in constructing AMAs themselves—the
autonomous artifacts presumed to be making morally charged decisions without direct human
oversight—rather than the entire technological system in which they are embedded. No
(ro)bot is an island, and yet we proceeded on the basis that the project of designing moral
machines should be centered on designing more and more sophisticated technological
artifacts. Johnson has patiently and persistently insisted at various conferences and
workshops that our focus on the capabilities of the (ro)bots considered as independent
artifacts carries potential dangers, insofar as it restricts attention to one kind of technological
fix instead of causing reassessment of the entire sociotechnological system in which (ro)bots
operate.
In a similar vein, David Woods and Erik Hollnagel maintain that robots and their
operators are best understood as joint cognitive systems (JCSs). The focus on isolated
!116

machine autonomy distorts the full appreciation for the kinds of systems design problems
inherent in JCSs. With the advent of artificial agents, when a JCS fails, there is a tendency to
blame the human as the weak link and to propose increased autonomy for the mechanical
devices as a solution. Furthermore, there is the illusion that increasing autonomy will allow
the designers to escape responsibility for the actions of artificial agents. But Woods and
Hollnagel argue that increasing autonomy will actually add to the burden and responsibility
of the human operators. The behavior of robots will continue to be brittle on the margins as
they encounter new or surprising challenges. The human operators will need to anticipate
what the robot will try to do under new situations in order to effectively coordinate their
actions with those of the robot. However, anticipating the robot’s actions will often be harder
to do as systems become more complex, leading to a potential increase in the failure of JCSs.
A focus on isolated autonomy can result in the misengineering of JCSs. Woods and Hollnagel
advocate more attention to coordination and resilience in the design of JCSs ( Woods and
Hollnagel 2006, 23).
To these critiques, we respond “guilty as charged.” We should have spent more time
thinking about the contexts in which (ro)bots operate and about human responsibility for
designing those contexts. We made a very fast jump from robots bolted to the factory floor to
free-roaming agents (hard and virtual), untethered from the surrounding sociotechnical
apparatus that makes their operation possible. AMAs cannot be designed properly without
attention to the systems in which they are embedded, and sometimes the best approach may
not be to design more sophisticated capacities for the (ro)bots themselves, but to rethink the
entire edifice that produces and uses them.
!117

Those roboticists who wish to ignore the dangers posed by autonomous systems are
likely to do so without hiding behind our suggestion that sensitivity to some moral
considerations can be engineered into (ro)bots. It should be apparent that it is not our intent
to mask the dangers. If on close inspection adequate safeguards cannot be implemented, then
we should turn our attention away from social systems that rely on autonomous systems.
4.4.6 (Ro)bot Ethics and Human Ethics
An implicit theme running throughout MM is the fragmentary character of presently available
models of human ethical behavior and the need for a more comprehensive understanding of
human moral decision making. In the book’s epilogue, we made that theme more explicit,
and proposed that a great deal can be learned about human ethics from the project of building
moral machines. While a number of critics have acknowledged this implicit theme, others
have advised that these comments were too cursory. A special edition of the journal Ethics
and Information Technology, edited by Anthony Beavers, is dedicated to what can be learned
about human ethics from robot ethics. Wallach’s contribution to that issue, “Robot Minds and
Human Ethics: The Need for a Comprehensive Model of Moral Decision Making” (2010,
22), explains how the task of assembling an AMA draws attention to a wider array of
cognitive, affective, and social mechanisms, contributing to human moral intelligence that is
usually considered by philosophers or social scientists, each working on their own particular
piece of the puzzle.
!118

4.5 Conclusion
The near future of moral machines is not and cannot be the attempt to recreate full moral
agency. Nevertheless, we are grateful to those critics who have emphasized the dangers of
too easily equating artificial and human moral agency. We always intended MM to be the
start of a discussion, not the definitive word, and we are thrilled to see the rich discussion
that has ensued.
Figure 4.1
Two Dimensions of AMA Development.
References
<jrn>Allen, Colin, Gary Varner, and Jason Zinser. 2000. Prolegomena to any future artificial
moral agent. Journal of Experimental & Theoretical Artificial Intelligence 12 (3):
251–261.</jrn>
<jrn>Anderson, Michael, Susan L. Anderson, and Chris Armen. 2006. An approach to
computing ethics. IEEE Intelligent Systems 21 (4): 56–63.</jrn>
<bok>Arkin, Ronald. 2009. Governing Lethal Behavior in Autonomous Robots. Boca Raton,
FL: Chapman and Hall.</bok>
<jrn>Byers, William, and Michael Schleifer. 2010. Mathematics, morality & machines.
Philosophy Now 78: 30–33.</jrn>
<jrn>Clarke, Roger. 1993. Asimov’s laws of robotics: Implications for information
!119

technology–Part I. IEEE Computer 26 (12): 53–61.</jrn>
<jrn>Clarke, Roger. 1994. Asimov’s laws of robotics: Implications for information
technology–Part II. IEEE Computer 27 (1): 57–66.</jrn>
<bok>Danielson, Peter. 1992. Artificial Morality: Virtuous Robots for Virtual Games. New
York: Routledge.</bok>
<conf>Danielson, Peter. 2003. Modeling complex ethical agents. Presented at the conference
on Computational Modeling in the Social Sciences, Seattle, Washington.</conf>
<jrn>Danielson, Peter. 2009. Can robots have a conscience? Nature 457: 540.</jrn>
<eref>DeMoss, David. 1998. Aristotle, connectionism, and the morally excellent brain. The
Paideia project online. Proceedings of the Twentieth World Congress of Philosophy
(American Organizing Committee Inc., Boston). <http://www.bu.edu/wcp/Papers/
Cogn/CognDemo.htm> (accessed May 18, 2010).</eref>
<jrn> Foot, Philippa. 1967. The Problem of Abortion and the Doctrine of Double Effect.
Oxford Review 5: 5-15.</jrn>
<jrn>Franklin, Stan, Bernard Baars, Uma Ramamurthy, and Matthew Ventura. 2005. The role
of consciousness in memory. Brains, Minds and Media 1: 1–38.</jrn>
<edb>Gips, James. 1991. Towards the ethical robot. In Android Epistemology, ed. Kenneth
G. Ford, Clark Glymour, and Patrick J. Hayes, 243–252. Cambridge, MA: MIT
Press.</edb>
<jrn>Guarini, Marcello. 2006. Particularism and classification and reclassification of moral
!120

cases. IEEE Intelligent Systems 21 (4): 22–28.</jrn>
<bok>Hall, J. Storrs. 2007. Beyond AI: Creating the Conscience of the Machine. Amherst,
NY: Prometheus Books.</bok>
<jrn>McLaren, Bruce. 2003 Extensionally defining principles of machine ethics: An AI
model. Artificial Intelligence Journal 150: 145–181.</jrn>
<conf>McLaren, Bruce, and Kevin D. Ashley. 1995. Case-based comparative evaluation in
truth-teller. In Seventeenth Annual Conference of the Cognitive Science Society, ed.
Johanna D. Moore and Jill F. Lehman, 72–77. San Diego, CA.</conf>
<jrn>Moor, James H. 1995. Is ethics computable? Metaphilosophy 26 (1–2): 1–21.</jrn>
<jrn> Moor, James H. 2006. The Nature, Importance, and Difficulty of Machine Ethics.
IEEE Intelligent Systems 21 (4): 18-21.</jrn>
<bok> Penrose, Roger 1989. The Emperor’s New Mind: Concerning Computers, Minds, and
the Laws of Physics. Oxford: Oxford University Press.</bok>
<bok> Picard, Rosalind. 1997. Affective Computing. Cambridge, MA: MIT Press.</bok>
<bok> Ross, W. D. 1930. The Right and the Good. Oxford: Clarendon Press.</bok>
<jrn> Searle, John R. 1980. Minds, Brains, and Programs. Behavioral and Brain Sciences 3
(3): 417-458.</jrn>
<jrn> Turing, Alan. 1950. Computing Machinery and Intelligence. Mind 59: 434-460. </jrn>
<jrn>Wallach, Wendell. 2010. Robot minds and human ethics: The need for a comprehensive
model of moral decision making. Ethics and Information Technology 12 (3): 243–
!121

250.</jrn>
<bok>Wallach, Wendell, and Colin Allen. 2009. Moral Machines: Teaching Robots Right
From Wrong. New York: Oxford University Press.</bok>
<jrn>Wallach, Wendell, Stan Franklin, and Colin Allen. 2010. A conceptual and
computational model of moral decision making in human and artificial agents.
TopiCS 2 (3): 454–485.</jrn>
<bok>Woods, David D., and Erik Hollnagel. 2006. Patterns in Cognitive Systems
Engineering. Boca Raton, FL: Taylor and Francis Group.</bok>
<eref>Yudkowsky, Eliezer. 2001. Creating friendly AI. <http:// singinst.org/upload/
CFAI.html> (accessed May 18, 2010).</eref>
!122