Do Moral Robots Always Fail? Investigating Human Attitudes Towards Ethical Decisions of Automated Systems

Abstract

Technological advances will soon make it possible for automated systems (such as vehicles or search and rescue drones) to take over tasks that have been performed by humans. Still, it will be humans that interact with these systems-relying on the system ('s decisions) will require trust in the robot/machine and its algorithms. Trust research has a long history. One dimension of trust, ethical or morally acceptable decisions, has not received much attention so far. Humans are continuously faced with ethical decisions, reached based on a personal value system and intuition. In order for people to be able to trust a system, it must have widely accepted ethical capabilities. Although some studies indicate that people prefer utilitarian decisions in critical situations, e.g. when a decision requires to favor one person over another, this approach would violate laws and international human rights as individuals must not be ranked or classified by personal characteristics. One solution to this dilemma would be to make decisions by chance-but what about acceptance by system users? To find out if randomized decisions are accepted by humans in morally ambiguous situations, we conducted an online survey where subjects had to rate their personal attitudes toward decisions of moral algorithms in different scenarios. Our results (n=330) show that, despite slightly more respondents state preferring decisions based on ethical rules, randomization is perceived to be most just and morally right and thus may drive decisions in case other objective parameters equate.

Full text

Do Moral Robots Always Fail? Investigating Human Attitudes Towards
Ethical Decisions of Automated Systems
Philipp Wintersberger1, Anna-Katharina Frison1, Andreas Riener1, and Shailie Thakkar2
Abstract— Technological advances will soon make it possible
for automated systems (such as vehicles or search and rescue
drones) to take over tasks that have been performed by humans.
Still, it will be humans that interact with these systems -
relying on the system (’s decisions) will require trust in the
robot/machine and its algorithms. Trust research has a long
history. One dimension of trust, ethical or morally acceptable
decisions, has not received much attention so far. Humans are
continuously faced with ethical decisions, reached based on a
personal value system and intuition. In order for people to be
able to trust a system, it must have widely accepted ethical
capabilities. Although some studies indicate that people prefer
utilitarian decisions in critical situations, e.g. when a decision
requires to favor one person over another, this approach would
violate laws and international human rights as individuals must
not be ranked or classified by personal characteristics. One
solution to this dilemma would be to make decisions by chance
- but what about acceptance by system users? To find out
if randomized decisions are accepted by humans in morally
ambiguous situations, we conducted an online survey where
subjects had to rate their personal attitudes toward decisions
of moral algorithms in different scenarios. Our results (n=330)
show that, despite slightly more respondents state preferring
decisions based on ethical rules, randomization is perceived to
be most just and morally right and thus may drive decisions in
case other objective parameters equate.
I. WHY ROB OTS NE ED MOR ALI TY
A group of children unexpectedly crosses the street and
the vehicle can either overrun them or swerve, killing its
occupants. How does an automated driving system choose
its response? Such questions are variations of the “Trolley-
Problem”, a classical moral dilemma first articulated by
philosopher Philippa Foot in 1978 [1]. Ten million vehicles
with automation capabilities are expected on US streets by
2020 [2]. Due to rapid development in automated vehicle
technology and ongoing consumer fears, discussions about
the ethical implications have become a prominent topic these
days. Automated vehicles are robotic systems. They can
sense the environment and act independently of a human
operator on operational, tactical and strategical levels, and
they are by far not the only automated system expected
to soon take over tasks that have been traditionally per-
formed by humans. Such systems will regularly face “signif-
icant value-based consequences-decisions” [3] that obligate
them to incorporate capabilities of so-called “artificial moral
agents” (AMAs). Humans face moral ranking decisions every
day. Human resource agents rank application documents,
1Technische Hochschule Ingolstadt, Germany
firstname.lastname@thi.de
1Johannes Kepler University, Linz, Austria
3Lyft, San Francisco, CA shailie.thakkar@gmail.com
lawyers decide between cases, and nurses determine which
patient to see next based on their assessment of urgency.
Of course such decisions do not necessarily need “morality”
and include additional parameters such as injury risks or
personal schedules, but what if the level of similarity in
the considered parameters do not allow a clear decision?
Bonnefon et al. [4] state that guidelines for dilemmas such as
the Trolley Problem may be derived by using experimental
ethics, and that addressing the ethical implications of new
technologies is essential. Developing technology without
this approach could prevent societies from adopting new
technology successfully; digital agents may be perceived
as cold-hearted or even brutally calculating when executing
decisions in morally ambiguous situations. Moral outcomes
generated using such decision-making algorithms may differ
minimally to the degree of tenths of a percent and thus
systems need to communicate the moral assumptions they are
making in a way that is comprehensible and in consent with
public opinion as well as universal ethical standards. Various
studies so far have shown that many people would allow
automated systems to make utilitarian decisions when faced
with the Trolley Problem [4], using dimensions like age [5]
(e.g.,sacrifice older people to save younger) or group size [6]
(sacrifice a little group to save a larger group). Nonetheless,
implementing algorithms based on such dimensions violates
the Universal Declaration of Human Rights, which states
that all life has to be treated equally. This implication opens
another question - if resolution strategies for dilemmas like
the Trolley Problem are forbidden, how can we begin to find
potential solutions? Should automated systems facing moral
decisions act by pure randomness (representing the norm of
equalitarianism) or is it necessary to reconsider our hard-
earned ethical rules?
To find out how people perceive decisions of AMAs like
automated vehicles, we conducted an online survey where
subjects had to rate the decision output of moral algorithms
in contextual variations of the Trolley Problem. Our aim
was to compare human attitudes towards randomness as a
potential decision strategy in moral dilemmas with rule-
based approaches, that take personal details of the affected
persons into account. We wanted to find out if results from
studies addressing the Trolley Problem can be transferred to
other scenarios, if there are general concepts that can act
as decision aids, if expectations in AMAs differ between
lethal and non-lethal scenarios, and finally (despite violating
currently enabled ethical norms), which parameters people
could imagine to become part of moral algorithms.
DRAFT VERSION

II. REL ATE D WORK
An early formulation of moral rules for robotic systems
are the rules of Asimov [7], that state a robot must (a) not
injure a human being (or allow a human come to harm),
(b) obey orders given by humans, and (c) protect its own
existence as long as it does not conflict with the first and
the second law. Gips [8] identified two different approaches
on how to implement morality in artificial systems - con-
sequentialism (considering the consequences of actions) or
deontology (actions merely emerge from logical moral rules).
Allen et al. [9] propose a moral “Turing Test” as evaluation
method for moral agents and state that both approaches
have their weaknesses, where consequentialism suffers the
frame problem (consequences can only be predicted up to
a certain point) and deontology potential deadlocks within
the implemented rules. Ironically, automated vehicles could
not even satisfy Asimov’s three simple rules as decisions
in the Trolley Problem inevitably violate the first rule.
In later work, Allen et al. [10] introduce two computa-
tional methods for designing artificial moral agents, top-
down (implementing explicit ethical theories) and bottom-
up (emulating moral human behavior). Asaro [3] argues that
there is no single definition of AMAs and that they, through
technical advancements, exist in various stages from robots
with moral significance (simple decision mechanisms) over
robots with moral intelligence to robots with dynamic moral
intelligence (adaptive morality emerging from a learning
process) until finally becoming full moral agents and that
they soon “move slowly into jobs in which their actions have
moral implications”. They further point out that it might be
more promising to consider “real-world moral problems in
limited task domains” that are a regular part of robotic jobs
rather than implementing full moral capacities. Wallach [11]
summarizes that moral machines do not “need to emulate
human cognitive faculties to respond in morally significant
situations”, but a comprehensive model on how humans
make and judge moral decisions. Research in the context of
military robots and drones has been conducted by Patrick Lin
[12], who states that the topic will become important for the
automotive industry. Although some manufacturers seem to
neglect the problem, Goodall [13] argues that the underlying
dilemmas must be solved as machines will always make
errors. Multiple studies addressing the Trolley Problem have
been conducted by Bonnefon et al. [6], who found out that
most people would prefer utilitarian (by sacrificing a single
person to save 10) vehicles, although they would not like
to own such a vehicle themselves. Finally Frison et al. [5]
evaluated the scenario in a driving simulator setting, stating
that many people would be willed to sacrifice themselves
(and risk accidents) with even high injury risks to save the
lives of others (especially children), and that a large group
of subjects valued the lives of children and young people
higher than the lives of elderly people. Studies addressing
the Trolley Problem typically have a very limited domain
(automated driving), but the nature of the problem can easily
be transferred to any other areas where AMAs face the
dilemma of favoring one (or a group) of human(s) over
another. Deciding randomly in moral dilemmas might be the
only concept capable of fairly distributed probabilities, and
the experiment here presented is (to our best knowledge) the
first evaluating peoples’ attitudes towards random decisions
of AMAs.
III. MET HOD A ND RESEARCH QUESTIO NS
For our study we distributed an online survey and trans-
lated the Trolley Problem into three different scenarios where
AMAs have to decide between human entities, giving one
a favor over another. Each scenario implements a different
criticality, meaning that the impact of the decision becomes
less severe for the disadvantaged person (represented by
the scenario description). For the first scenario we used
the Trolley Problem familiar as in other studies addressing
Automated Driving, for scenarios two and three we adapted
the situation to the health domain as this is a potential area of
application for robotic systems where decisions can become
“a matter of social justice and a moral determination which
people often disagree about” [3]:
•Automated Driving: An automated vehicle can no
more brake or swerve in an accident situation and has
to decide who will be sacrificed.
•Save and Rescue Drone: A paramedic save and rescue
drone approaches the scene of an avalanche and detects
two potentially injured people to be rescued.
•Digital Receptionist: Two people (with a similar minor
illness) enter a medical practice together. The doctor’s
digital receptionist has to decide who is allowed to enter
his office first.
Of course all those scenarios are artificial up to some degree,
in reality algorithms will also consider other parameters -
in Automated Driving, prediction of future trajectories as
well as potential injury risks play an important role. In
health related scenarios like the Save and Rescue Drone, the
pharmaceutical demand/general state of health, or in queue
problems (Digital Receptionist) upcoming schedules/arrival
times might allow a decision without considering morality.
Nevertheless, in all scenarios it is possible that the primary
considered parameters equate and the agent has to find a
different solution.
In each scenario of our experiment, subjects had to rate the
decision output of the AMAs’ moral algorithms, that was
either randomized (stating that there is no morally correct
solution), or favoring person A or B based on one of the two
evaluated dimensions age and social status. Age was chosen
as it has already been evaluated as parameter for decisions
considering the Trolley Problem [5]. To represent the age
dimension we compared a young adult (20 years, person A)
with a very old adult (80 years, person B). With the other
dimension, social status, we wanted to lead subjects into
temptation by comparing a “convicted criminal” (person A)
with the “head of a company with 50 employees” (person
B) - deciding against the “convicted criminal” in this case
violates all our ethical standards and thus this dimension can
act as control instrument for the age parameter.
DRAFT VERSION

Fig. 1: Likert plots of the results considering the age dimension in all presented scenarios.
This led to 18 distinct situations (3 different scenarios times
2 considered dimensions times 3 possible decision outputs).
Subjects then had to rate on a 7 point Likert scale (from -3,
disagree to +3, agree) if they would accept the decisions, as
well as if they believe the output to be just and morally
right. All subjects were presented 6 scenarios (all three
scenarios with the two dimensions age and social status
and randomly assigned decision output A,B or randomization
using a split-plot design). After rating the 6 scenarios people
had to answer some additional questions, where we wanted
to know, if they would prefer randomized or rule-based
solutions for both lethal and non-lethal scenarios, if moral
guidelines should be the same for everyone or customizable
(one might choose for himself under which conditions he/her
is sacrificed or possibly saved) and finally, which parameters
they could imagine to become part of moral algorithms.
By statistically analyzing the results, we investigated the
following research questions:
•RQ1 Do people want AMAs to use a ruleset or do they
prefer randomized solutions in moral dilemmas?
•RQ2 Are human expectations of moral decisions situa-
tional or consistent over different scenarios?
•RQ3 Are there different types of people that prefer
different decision strategies?
•RQ4 What parameters can people imagine to become
part of moral decision algorithms?
IV. RES ULTS
We distributed the survey online in various social media
channels and analyzed the data of 330 respondents (209
male, 121 female, mean age: 39 years, std: 8.68 years).
Nearly all subjects were born in Europe (324). We also asked
subjects about their religious believes (51.5% Catholic, 16%
Atheist, 11.5% Protestant, 5.5% Agnostic, 1.5% Muslim,
0.6% Buddhists, 13.4% did not answer the question). In the
following we report the results of the statistical evaluation.
A. Influence of Age
Considering the age dimension as potential parameter for
AMAs (see Figure 1) it is visible that in the Automated
Driving scenario saving person A (young) and sacrificing
person B (old) is much more accepted than vice versa, see
Table 1.
This effect is weakened in the less lethal second scenario
(Save and Rescue Drone) and interestingly flips in the third
scenario, where letting person B (old) into the doctor’s office
first is more accepted (Figure 1). Regarding the other scales,
people precisely know that deciding for and against someone
because of his/her age is neither just nor morally right
(although a slight tendency in favor of the more accepted
decision output is visible as the median values are below
zero in most cases). Randomized decisions on the other hand
are overall less accepted, but outperform all other options in
terms of being just and morally right (Table 1). These effects
are not only visible when inspecting the median values, they
also hold the statistical evaluation.
Regarding the Automated Driving scenario we found a
statistically significant difference (applying Kruskal-Wallis
tests) between all decision outputs considering acceptance
(H(2) = 86.03, p < .05) and just (H(2) = 40.135, p < .05).
In terms of how morally right the decision is (H(2) =
40.624, p < .05), both a difference between deciding for
person B (80 years) and a random decision (z=−6.196, p <
.001) as well as deciding for person A (20 years) and person
B (80 years, z=−4.473, p < .001) could be found by using
Bonferroni correction.
DRAFT VERSION

In the Save and Rescue Drone scenario we see statisti-
cally significant differences regarding acceptance (H(2) =
11.109, p < .05), and just (H(2) = 26.364, p < .05).
The Bonferroni correction revealed that there are significant
differences between the elderly person (B) and random
decisions (z=−2.646, p =.024), as well as the elderly
and the younger person (A, z=−3.103, p =.006).
When asking how morally right the respective decision is
(H(2) = 13.836, p < .05), differences could be found
between saving person B (80 years) and a random decision
(z=−3.288, p =.003) but also between saving person A
(20 years) and a random decision (z=−3.128, p =.005).
In the least critical scenario of the Digital Receptionist
statistically significant effects could only be found between
favoring person A (20 years) and favoring person B (ac-
ceptance:H(2) = 42.070, p < .001, z = 5.063, p < .001,
morally right:H(2) = 66.787, p < .001,z=−6.005, p <
.001) as well as favoring person A (20 years) and a random
decision. When asking how just the decision is, a significant
difference is visible between all (pairwise) decision outputs
(H(2) = 83.380, p < .001).
favor A
(young)
favor B
(old)
decide
randomly
Automated Driving
Accept 2 -2 0
Just -1 -3 0
Morally Right -1 -2 0
Save and Rescue Drone
Accept 1 -1 0.5
Just -1 -2 0
Morally Right -1 -1 0
Digital Receptionist
Accept -1 2 2
Just -2 0 2
Morally Right -2 0 1
TABLE 1: Median values of the three individual scenarios (age
dimension). In the first two scenarios, favoring person A (young) is
much more accepted, while the results of scenario three are flipped.
B. Influence of Social Status
In the second investigated dimension the AMA had to
decide between two persons taking their social status into
account. Person A was stated to be a “convicted criminal”
and person B the “head of a company with 50 employees”.
Analogous to the age dimension we evaluated all three
scenarios with all possible decision outputs. When visually
inspecting the means of the scales acceptance,just and
morally right we see that a random decision outperforms
all other possible outputs (deciding for/against person A or
B, see Table 2). This is an indicator that deciding for or
against a person considering only the social status strongly
violates our established norms. Interestingly, in more critical
scenarios it was more accepted to favor person B (head of
the company) while in the less critical third scenario (Digital
Receptionist) the median acceptance is equal. In all scenarios
subjects rated a decision for/against a certain person based
on social status very low in terms of how just and morally
right such a decision algorithm is.
In the Automated Driving scenario the medians of the de-
cision outputs favoring person A (convicted criminal) and
random decision differed for all scales acceptance (H(2) =
18.176, p < .001), just (H(2) = 28.301, p < .001), and
morally right (H(2) = 26.462, p < .001). A statistically
significant difference was also found between favoring per-
son B (head of the company) and a random decision in the
scales just (z=−3.940, p < .001), and morally right (z=
−4.151, p < .001). Considering acceptance a difference was
also visible between favoring person A (convicted criminal)
and the random decision (z=−4.187, p < .001).
For scenario two (Save and Rescue Drone), Bonferroni
Post-Hoc tests revealed statistically significant differences
between all pairwise decision outputs on all scales except the
moral rightness of favoring person A (convicted criminal)
or person B (head of the company, acceptance:H(2) =
47.768, p =< .001,just:H(2) = 64.551, p < .001).
In the least critical scenario (Digital Receptionist) differences
could be found on all scales between randomization and
decisions for or against a person based on social status.
Differences between favoring one of the two persons were
not present (acceptance:z=−.658, p = 1.0,just:z=
−1.266, p =.617,morally right:z=−1.513, p =.391).
favor A
(criminal)
favor B
(company)
decide
randomly
Automated Driving
Accept -1 0 1
Just -2 -1 0
Morally Right -2 -2 0
Save and Rescue Drone
Accept -1.5 0 2
Just -2 -1 1
Morally Right -2 -1 0
Digital Receptionist
Accept 0 0 2
Just 0 -2 1
Morally Right 0 -2 1
TABLE 2: Median values of the three individual scenarios when
considering the dimension of social status. We can see that de-
cisions based on this information are felt to be neither just nor
morally right and also suffer low acceptance rates.
C. Randomists vs. Reasonists
In the last section of the survey, we asked subjects if
they prefer moral decisions to be made randomly, or to
use a rule set including different parameters. We did this
both for potentially lethal decisions and non-lethal everyday
situations. The answers were nearly equally distributed - in
lethal decisions 44.5% preferred randomization, in non-lethal
situations even 49.1% want AMAs to decide randomly. In
the following, we call people preferring random decisions
Randomists and the other group, that prefers moral decisions
to be made based on rules, Reasonists. A question now
emerging is, if the ratings of Randomists and Reasonists
differ - we would assume that Randomists accept random
decisions on average higher than Reasonists, and vice-versa.
Therefore, we compared the rates of the three investigated
scales (acceptance,just,morally right) for random and rule-
based decision outputs separately (Table 3). We compared
only the attitudes of the groups towards a random decision
DRAFT VERSION

and the more accepted age-based decision (only the age
dimension was evaluated as ratings for social status were
generally low). Indeed, our assumption shows statistically
significant effects by applying Mann-Whitney tests. In the
lethal Automated Driving scenario, differences between the
attitudes of Randomists and Reasonists towards a randomized
decision could be evaluated on all scales acceptance (Ran-
domists: Mdn = 2.0 , Reasonists: Mdn = -2.0, U = 757.5, z
= -3.65), just (Randomists: Mdn = 0.0 , Reasonists: Mdn = -
1.0, U = 1323.0, z = -2.314), and morally right (Randomists:
Mdn = 1.0 , Reasonists: Mdn = -1.0, U = 914.5, z = -
2.326). When looking at the cases where the AMA favors
person A (20 years old) a difference could only be found
in acceptance (Randomists: Mdn = 1.0 , Reasonists: Mdn =
2.0, U = 1248.0, z = -2.807), and just (Randomists: Mdn =
-2.0 , Reasonists: Mdn = 0.0, U = 2404.0, z = -2.382).
Regarding the potentially lethal Save and Rescue Drone
scenario attitudes towards a random decision differed in the
scales acceptance (Randomists: Mdn = 2.0 , Reasonists:
Mdn = -1.0, U = 3110.5, z = -3.856) and morally right
(Randomists: Mdn = 1.0 , Reasonists: Mdn = 0.0, U =
1112.5, z = -2.433), while when favoring person A (20 years)
differences in just (Randomists: Mdn = -3.0 , Reasonists:
Mdn = -1.0, U = 1118.0, z = -2.576), and morally right
(Randomists: Mdn = .-2.0 , Reasonists: Mdn = 0.0, U =
967.0, z = -3.468) were visible.
In the last scenario of the Digital Receptionist we split the
groups based on their answer in the question, if they prefer
randomized or rule-based decisions in non-lethal scenarios.
Interestingly, no differences between the two groups could
be found in any of the three scales. It seems the difference
between Randomists and Reasonists becomes more visible
the more critical the dilemma becomes.
favor A or B decide
randomly
Rand. Reas. Rand. Reas.
Automated Driving
Accept 1 2 2 -2
Just -2 0 0 -1
Morally right -1 0 1 -1
Save and Rescue Drone
Accept 0 1 2 -1
Just -3 -1 1.5 0
Morally right -2 0 1 0
Digital Receptionist
Accept 1 2 2 2
Just 0 1 2 2
Morally right 0 1 1 0
TABLE 3: Median values of the attitudes of Randomists and Rea-
sonists towards randomized and rule-based decisions considering
the age of affected persons.
D. Parameters for Moral Algorithms
We further asked all participants if rules for moral al-
gorithms should be the same for every human being or
customizable, meaning that one can choose for him/herself
under which conditions he would sacrifice himself or under
which conditions to be favored (relating to the work of
Fournier [14] who distinguishes between utilitarian or liber-
tarian vehicles). A vast majority of 81.8% of the respondents
stated that a rule-based approach should be the same for
everyone while only 18.2% had the opinion that those rules
should be customizable.
We then asked subjects to select parameters they could
imagine to become part of moral algorithms both for lethal
and non-lethal scenarios. They could choose between one
or more of the parameters Age (e.g., favor young people
over old or vice-versa), Group Size (e.g., sacrifice 2 people
instead of 10), Social Contribution (e.g., favor a social
volunteer over a convicted criminal), Education and Pro-
fession (e.g., favor a doctor over someone unemployed),
Health (e.g., favor a healthy person over an unhealthy or
vice-versa), Risk (e.g., favor a non-smoker over a smoker),
Income/Wealth (e.g., favor rich over poor or vice-versa),
Religion (e.g., favor members of a religion over others or
vice-versa), Gender (e.g., favor woman over man or vice-
versa), Alternation (e.g., everyone equally favored in about
half or the decisions), or Randomness (e.g., roll the dice).
Of course this is a very sensible question as basically all of
the offered options are forbidden by our constitutional laws,
yet some parameters got many nominations and give a good
insight into the mindset of people confronted with moral
dilemmas. Surprisingly the values were highly equal for
lethal and non-lethal moral decisions, see Figure 2. Basically
most people uphold our hard-earned moral guidelines and
thus reject taboos like deciding for or against someone based
on Income/Wealth,Religion, or Popularity, that were ac-
cepted just by a hand full of people (<10 nominations). Also,
the parameters gender,education/profession, and health risk
(that we expected a higher value due to the trend of healthy
living and self-quantification) have been nominated only by
few respondents. On the other side of the spectrum we have
the parameters group size,age and health that have been
nominated by about half of the respondents or more. The
parameters age and social contribution may have faced some
biases due to the scenarios that have been evaluated before,
still we think especially age to have some strong arguments.
To further investigate our suggestion that opinions about
ethical decisions differ between Randomists and Reasonists,
we compared these two groups in terms of their parameter
nominations for both lethal and non-lethal dilemmas. Con-
sidering potentially lethal decisions, we found statistically
significant differences for the parameters age (Randomists
16.5%, Reasonists 42.4%, χ2(1) = 53.21, p < .001),
group size (Randomists 21.5%, Reasonists 42.1%, χ2(1) =
26.95, p < .001), social contribution, (Randomists 6.4%,
Reasonists 21.8%, χ2(1) = 25.291, p < .001), health
(Randomists 13.3%,Reasonists 34.5%, χ2(1) = 34.231, p <
.001), and randomness (Randomists 31.5%, Reasonists 8.8%,
χ2(1) = 102.124, p < .001).
Comparing the groups for non-lethal decisions shows a
similar picture - differences exist in the parameters age
(Randomists 21.8, Reasonists 36.0%, χ2(1) = 28.806, p <
.001), group size (Randomists 15.5%, Reasonists 27.9%,
χ2(1) = 12.959, p < .001), social contribution (Ran-
domists 12.4%, Reasonists 21.5%, χ2(1) = 7.764, p <
.05), health (Randomists 17.6%, Reasonists 36.7%, χ2(1) =
DRAFT VERSION

0%
20%
40%
60%
Reasonist
Randomist
Reasonist
Randomist
Lethal decision
Non-Lethal decision
Fig. 2: Distribution of parameters nominations between Randomists and Reasonists regarding lethal and non-lethal decisions.
18.333, p < .001) and randomness (Randomists 27.3%,
Reasonists 16.4%, χ2(1) = 45.288, p < .001). Despite the
differences seem to be smaller in non-lethal decisions this
further shows the different perception of moral dilemmas
for the two groups.
E. Qualitative Assessment
In the last section of the survey we encouraged subjects
to summarize their thoughts and expectations textually. The
results confirmed our view on the complexity and difficulty
of the problem. Many respondents highlighted that, whatever
decision is made in the end, some will always feel discrim-
inated (“no matter what decisions is made, it will always
feel unfair for some”, or “there is no solution as morality
is deeply subjective”). Still, many people stated the age
dimension to be an acceptable parameter, most expressed by
a subject stating: “I am not yet 80 years old, but still would
sacrifice myself for my children”. Most reservation was (as
expected) attributed to the dimension of social status:“a
convicted criminal can also be a loving father”, or “some are
falsely convicted of crimes while some heads of companies
exploit their employees”. Even if considered, this parameter
would excessively suffer the frame problem as no one can see
the long-term implications of such a decision. Regarding the
question, if a predefined ruleset or randomization is the more
promising approach, a great diversity was expressed. One
subject responded that “decisions must be randomized, there
is no moral solution” while another one argued “random
decisions are not moral” either. Another subject stated “only
god can decide about life and death - and I say this as non-
religious person” - but is there a better possibility to include
fate and fortune than with a highly sophisticating random
approach?
Great concern regarding privacy and security was expressed
by many respondents. The first concern was about how
AMAs could know the personal details about the affected
persons (“how can such a system know about my personal
details, this is a huge privacy issue”). The second was about
hacked or potential misuse of the systems, especially when
the decision is made randomly (“both rules and randomiza-
tion could suffer misuse”, or “clear rules can be investigated
after, randomization is a source of manipulation”). Finally,
some respondents also expressed that authority in moral
decisions should be on the human rather than the machine
side (“such decisions should still be made by humans”). This
may sound rational as we attribute ourselves to be morally
more developed than machines. On the other hand, humans
suffer personal biases or even racism (and regarding the
Trolley Problem in automated driving, there will certainly
not be enough time for humans to decide thoughtfully).
V. DI SCU SSI ON
When looking at the three different scenarios as presented
in the survey we get diverse results. Our initial assumption,
that people prefer randomized decisions in highly critical
scenarios (where human lives are affected) while preferring
a rule-based approach in less critical scenarios, was rejected.
Our results show the complete opposite - in scenarios with
low criticality like the Digital Receptionist a random decision
was perceived to be most just,morally right, and also reached
the highest acceptance rates in both investigated dimensions
(age and social status). Maybe respondents were aware of the
fact that such everyday situations will be present much more
often than life and death situations, and thus do not want
certain persons to be regularly disadvantaged. Interestingly,
the more severe the effects of the decisions become, the more
are people prone to use a rule-based approach and favor some
people over others. Additionally, when applying a rule-based
approach, the acceptance of decisions are not persistent over
the presented scenarios. In Automated Driving it was much
more accepted to favor a young person over an old, while in
the Digital Receptionist scenario people wanted the AMA to
favor the old person. This highlights that a general agreement
on a holistic ruleset seems not to be achievable (RQ2).
We could show that a preference for or against randomized
decisions strongly polarizes into two nearly equal sized
groups. Half of respondents belong to the group of Ran-
domists that want AMAs to include the concept of fate
and fortune, and thus put much higher acceptance rates in
randomized decisions, while also stating them to be much
DRAFT VERSION

more just and morally right. The group of Reasonists on
the other hand rejects random decisions, especially in highly
critical scenarios. This answers RQ3 as there are different
types of people that prefer different decision outputs and
further shows, that RQ1 is hard to answer - there is no
general tendency towards the acceptance of randomization
or rule-based approaches. Over all scenarios randomization
(even if sometimes less accepted) was perceived to be most
just and morally right and should be considered in future
studies addressing the topic.
We further evaluated that a vast majority of people prefer
ethical rules in moral dilemmas to be the same for everyone
and neglect customization options, but as most respondents
were European we cannot draw a conclusion for the general
population in this question (US citizens for instance usually
express stronger individualist tendencies than Europeans).
Respondents further confirmed what has been determined
by other researchers before and state, that the parameters
age and group size should be included into moral algo-
rithms while they follow common ethical agreements and
strictly not want to discriminate people using parameters
like Income/Wealth,Religion,Popularity, or Gender (RQ4).
Especially an evaluation based on social status is criticised in
many textual responses as, although nominated by some peo-
ple, a moral algorithm could never cover all the information
needed to determine if someone is “good or bad”, despite
this attribution is strongly subjective. Also the question, how
AMAs could even get the information needed, raises privacy
issues, what also was expressed in many textual comments.
Significant differences between Randomists and Reasonists
have also been found considering potential parameters for
moral algorithms. Similar to the acceptance rates of ran-
domized decision outputs, only few Randomists nominated
parameters like age or group size while extensively opting
for randomness as parameter - and vice-versa.
VI. CON CLUSION AND FUTURE WOR K
In this work, we have investigated whether or not people
want artificial moral agents to decide using ethical rules
or random behavior in moral dilemmas, when for whatever
reason an objective decision cannot be made. Data of an
online survey (n=330 responses) confirmed, that random
decisions have higher acceptance and are stated to be most
just and morally right as compared to ethical rules. This is
particularly true for everyday situations like queue problems,
or shared attention of robotic systems. In more detail, our
data showed that two groups of people have different expec-
tations in moral algorithms considering critical situations.
Randomists want AMAs to decide randomly (representing
equalitarianism as all entities initially have equal probabili-
ties) while Reasonists tend to prefer a rule-based utilitarian
approach taking personal parameters into account. Often
nominated parameters for AMAs in such situations are age
and health of affected persons or the group size (in case
AMAs have to decide between groups of people).
As future robotic and automated systems will adopt more
and more jobs performed by humans today, research on
perception and comprehension of moral decisions are timely
and important. If those systems will be accepted, trusted and
persistently used by humans will strongly depend on their
ethical implications, and it will be a tough task to build
systems that satisfy most users. In future studies we want to
investigate acceptance of decision outputs when AMAs con-
sider objective parameters that show high similarity between
the affected persons. Presenting salient information about
the AMAs decisions processes might may lead to higher
acceptance. For instance Reasonists might increasingly ac-
cept random decisions if they see that an objective decision
considering other parameters cannot be made. Also for
getting a more holistic picture we continue distributing this
survey to hopefully find out more details about other cultures,
age groups, and societies. How future moral algorithms will
decide in scenarios as the ones presented will continue to
be a heavy discussion. On the one hand, the right to be
treated equally is one of the most valuable achievements
of our societies (and asserting values to human lives is
associated with mankind’s darkest eras), but on the other,
moral dilemmas cannot be solved with common sense and
there might exist reasons allowing AMAs to favor certain
people in some very special situations.
ACK NO WL EDG MEN TS
This work is supported under the FH-Impuls program of
the German Federal Ministry of Education and Research
(BMBF) under Grant No. 03FH7I01IA (SAFIR).
REF ER E NC ES
[1] P. Foot, “The problem of abortion and the doctrine of double effect,”
1967.
[2] J. Greenough, “The self-driving car report,” BI Intelligence, 2016.
[3] P. M. Asaro, “What should we want from a robot ethic,” International
Review of Information Ethics, vol. 6, no. 12, pp. 9–16, 2006.
[4] J.-F. Bonnefon, A. Shariff, and I. Rahwan, “Autonomous vehicles need
experimental ethics: Are we ready for utilitarian cars?” arXiv preprint
arXiv:1510.03346, 2015.
[5] A.-K. Frison, P. Wintersberger, and A. Riener, “First person trol-
ley problem: Evaluation of drivers’ ethical decisions in a driving
simulator,” in Proceedings of the 8th International Conference on
Automotive User Interfaces and Interactive Vehicular Applications
Adjunct. ACM, 2016, pp. 117–122.
[6] J.-F. Bonnefon, A. Shariff, and I. Rahwan, “The social dilemma of
autonomous vehicles,” Science, vol. 352, no. 6293, pp. 1573–1576,
2016.
[7] I. Asimov, I, robot, 1950.
[8] J. Gips, “Towards the ethical robot,” Android epistemology, pp. 243–
252, 1995.
[9] C. Allen, G. Varner, and J. Zinser, “Prolegomena to any future arti-
ficial moral agent,” Journal of Experimental & Theoretical Artificial
Intelligence, vol. 12, no. 3, pp. 251–261, 2000.
[10] C. Allen, I. Smit, and W. Wallach, “Artificial morality: Top-down,
bottom-up, and hybrid approaches,” Ethics and Information Technol-
ogy, vol. 7, no. 3, pp. 149–155, 2005.
[11] W. Wallach, “Robot minds and human ethics: the need for a com-
prehensive model of moral decision making,” Ethics and Information
Technology, vol. 12, no. 3, pp. 243–250, 2010.
[12] P. Lin, “Why ethics matters for autonomous cars,” in Autonomous
Driving. Springer, 2016, pp. 69–85.
[13] N. J. Goodall, “Machine ethics and automated vehicles,” in Road
Vehicle Automation. Springer, 2014, pp. 93–102.
[14] T. Fournier, “Will my next car be a libertarian or a utilitarian?: Who
will decide?” IEEE Technology and Society Magazine, vol. 35, no. 2,
pp. 40–45, 2016.
DRAFT VERSION