ChapterPDF Available

Why Ethics Matters for Autonomous Cars



If motor vehicles are to be truly autonomous and able to operate responsibly on our roads, they will need to replicate—or do better than—the human decision-making process. But some decisions are more than just a mechanical application of traffic laws and plotting a safe path. They seem to require a sense of ethics, and this is a notoriously difficult capability to reduce into algorithms for a computer to follow.
Why Ethics Matters for Autonomous Cars
Patrick Lin
If motor vehicles are to be truly autonomous and able to operate responsibly on our roads,
they will need to replicateor do better thanthe human decision-maki ng process. But
some decisions are more than just a mechanical application of trafc laws and plotting a
safe path. They seem to require a sense of ethics, and this is a notoriously difcult
capability to reduce into algorithms for a computer to follow.
This chapter will explain why ethics matters for autonomous road vehicles, looking at
the most urgent area of their programming. Nearly all of this work is still in front of the
industry, which is to say that I will mainly raise the questions here and not presume to
have any denitive answers at such an early stage of the technology.
A brief note about terminology
I will use autonomous, self driving, driverless, and robot interchangeably.
These refer primarily to future vehicles that may have the a bility to operate without human
intervention for extended periods of time and to perform a broad range of actions. I will
also use cars to refer loosely to all motor vehicles, from a motorcycle to a freight truck;
those distinctions do not matter for the discussion here.
4.1 Why Ethics Matters
To start, let me offer a simple scenario that illustrates the need for ethics in autonomous
cars. Imagine in some distant future, your autonomous car encounters this terrible choice:
it must either swerve left and strike an eight-year old girl, or swerve right and strike an
80-year old grandmother [33]. Given the cars velocity, either victim would surely be
P. Lin (&)
Philosophy Department, California Polytechnic State University,
San Luis Obispo, CA 93442, USA
© The Author(s) 2016
M. Maurer et al. (eds.), Autonomous Driving,
DOI 10.1007/978-3-662-48847-8_4
killed on impact. If you do not swerve, both victims will be struck and killed; so there is
good reason to think that you ought to swerve one way or another. But what would be the
ethically correct decision? If you were programming the self-driving car, how would you
instruct it to behave if it ever encountered such a case, as rare as it may be?
Striking the grandmothe r could be the lesser evil, at least to some eyes. The thinking is
that the girl still has her entire life in front of hera rst love, a family of her own, a
career, and other adventures and happinesswhile the grandmother has already had a full
life and her fair share of exp eriences. Further, the little girl is a moral innocent, more so
than just about any adult. We might agree that the grandmother has a right to life and as
valuable a life as the little girls; but nevertheless, there are reasons that seem to weigh in
favor of saving the little girl over the grandmother, if an accident is unavoi dable. Even the
grandmother may insist on her own sacrice, if she were given the chance to choose.
But either choice is ethically incorrect, at least accordi ng to the relevant professional
codes of ethics. Among its many pledges, the Institute of Electrical and Electronics
Engineers (IEEE), for instance, commits itself and its 430,000+ members to treat fairly
all person s and to not engage in acts of discrimination based on race, religion, gender,
disability, age, national origin, sexual orientation, gender identity, or gender expression
[23]. The refore, to treat individuals differently on the basis of their age, when age is not a
relevant factor, seems to be exactly the kind of discrimination the IEEE prohibits [18, 33].
Age does not appe ar to be a relevant factor in our scenario as it might be in, say,
casting a young actor to play a childs character in a movie. In that movie scenario, it
would be appropriate to reject adult actors for the role. Anyway, a reason to discriminate
does not necessarily justify that discrimination, since some reasons may be illegitimate.
Even if we point to the disparity of life experiences between the old and the young, that
difference isnt automatically an appropriate basis for different treatment.
Discriminating on the basis of age in our crash scenario would seem to be the same evil
as discriminating on the basis of race, religion, gender, disability, national origin, and so
on, even if we can invent reasons to prefer one such group over another. In Germany
home to many inuential automotive companies that are working to develop self-driving
technologiesthe right to life and human dignity is basic and set forth in the rst two
articles of the very rst chapter in the nations constitution [9]. So it is difcult to see how
German law could even allow a company to create a product that is capable to making such
a horric and apparently illegal choice. The United States similarly strives to offer equal
protection to all persons, such as stipulated in the fourteenth amendment of its constitution.
If we cannot ethically choose a path forward, then what ought to be done? One solution
is to refuse to make a swerve decision, allowing both victims to be struck; but this seems
much worse than having only one victim die, even if we are prejudiced against her.
Anyway, we can force a decision by modifying the scenario: assume that 10 or 100 other
pedestrians would die, if the car continued forward; and swerving would again result in
only a single death.
Another solution could be to arbitrarily and unpredictably choose a path, without
prejudice to either person [34]. But this too seems ethically troubling, in that we are
70 P. Lin
choosing between lives without any deliberation at allto leave it to chance, when there
are potentially some reasons to prefer one over the other, as distasteful and uncomfortable
as those reasons may be. This is a dilemma that is not easily solvable and therefore points
to a need for ethics in developing autonomous cars.
4.1.1 Beyond Crash-Avoidance
Many readers may object right away that the dilemma above, as well as others that follow,
will never occur with autonomous cars. It may be suggested that future cars need not
confront hard ethical choices, that simply stopping the car or handing control back to the
human operator is the easy path around ethics. But I will contend here that braking and
relinquishing control will not always be enough. Those solutions may be the best we have
today, but if automated cars are to ever operate more broadly outside of limited highway
environments, they will need more response-options.
Current research already makes this case as a matter of physics [12, 13], but we can
also make a case from commonsense. Many ordinary scenarios exist today in which
braking is not the best or safest move, whether by human or self-driving car. A wet road or
a tailgater, for instanc e, may make it dangerous to slam the brakes, as opposed to some
other action such as steering around the obstacle or simply through it, if it is a small
object. Today, the most advanced self-driving cars cannot detect small objects such as
squirrels [7]; therefore, they presumably cannot also detect squirrel-sized rocks, potholes,
kittens, and other small but consequential hazards can cause equipment failure, such as tire
blowouts or sensor errors, or deviations from a safe path.
In these and many other cases, there may not be enough time to hand control back to the
driver. Some simulation experiments suggest that human drivers need up to 40 s to
regain situation awareness, depending on the distracting activity, e.g., reading or napping
far longer than the 12 s of reaction time required for typical accident scenarios [18, 38].
This means that the car must be responsible for making decisions when it is unreasonable to
expect a timely transfer of control back to the human, and again braking might not be the
most responsible action.
One possible reply is that, while imperfect, braking could successfully avoid the
majority of emergency situations a robot car may nd itself it, even if it regrettably makes
things worse in a small numbe r of cases. The benets far outweigh the risks, presumably,
and the numbers speak for themselves. Or do they? I will discuss the dangers of morality
by math throughout this chapter.
Braking and other responses in the service of crash-avoidance wont be enough,
because crash-avoida nce is not enough. Some accidents are unavoidablesuch as when
an animal or pedestrian darts out in front of your moving carand therefore autonomous
cars will need to engage in crash-optimization as well. Optimizing crashe s means to
choose the course of action that will likely lead to the least amount of harm, and this could
4 Why Ethics Matters for Autonomous Cars 71
mean a forced choice between two evils, for instance, choosing to strike either the
eight-year old girl or the 80-year old grandmother in my rst scenario above.
4.1.2 Crash-Optimization Means Targeting
There may be reasons, by the way, to prefer choosing to run over the eight-year old girl
that I have not yet mentioned. If the autonomous car were most interested in protecting its
own occupants, then it would make sense to choose a collision with the lightest object
possible (the girl). If the choice were between two vehicles, then the car should be
programmed to prefer striking a lighter vehicle (such as a Mini Cooper or motorcycle)
than a heavier one (such as a sports utility vehicle (SUV) or truck) in an adjacent lane
[18, 34].
On the other hand, if the car were charged with protecting other drivers and pedest rians
over its own occupantsnot an unreasonable imperativethen it should be programmed
to prefer a collision with the heavier vehicle than the lighter one. If vehicle-to-vehicle
(V2V) and vehicle-to-infrastructure (V2I) communications are rolled out (or V2X to refer
to both), or if an autonomous car can identify the specic models of other cars on the road,
then it seems to make sense to collide with a safer vehicle (such as a Volvo SUV that has a
reputation for safety) over a car not known for crash-safety (such as a Ford Pinto thats
prone to exploding upon impact).
This strategy may be both legally and ethically better than the previous one of jealously
protecting the cars own occupants. It could minimize lawsuits, because any injury to
others would be less severe . Also, because the driver is the one who introduced the risk to
societyoperating an autono mous vehicle on public roadsthe driver may be legally
obligated, or at least morally obligated, to absorb the brunt of any harm, at least when
squared off against pedestrians, bicycles, and perhaps lighter vehicles.
The ethical point here, however, is that no matter which strategy is adopted by an
original equipment manufacturer (OEM), i.e., auto manufacturer, programming a car to
choose a collision with any particular kind of object over another very much resembles a
targeting algorithm [33]. Somewhat related to the military sense of selecting targets,
crash-optimization algorithms may involve the deliberate and systematic discrimination
of, say, large vehicles or Volvos to collide into. The owne rs or operators of these targeted
vehicles bear this burden through no fault of their own, other than perhaps that they care
about safety or need an SUV to transport a large family.
4.1.3 Beyond Harm
The problem is starkly highlighted by the following scenario [1517, 34]: Again, imagine
that an autonomous car is facing an imminent crash, but it could select one of two targets
in adjacent lanes to swerve into: either a motorcyclist who is wearing a helmet, or a
motorcyclist who is not. It probably doesnt matter much to the safety of the car itself or
72 P. Lin
its occupants whether the motorcyclist is wearing a helmet; the impact of a helmet into a
car window doesnt introduce that much more risk that the autonomous car should want to
avoid it over anything else. But it matters a lot to the motorcyclist whether s/he is wearing
a helmet: the one without a helmet would probably not survive such a collision. Therefore,
in this dreadful scenario, it seems reasonable to program a good autonomous car to swerve
into the motorcyclist with the helmet.
But how well is justice and public policy served by this crash-optimization design?
Motorcyclists who wear helmets are essentially being penalized and discriminated against
for their responsible decision to wear a helmet. This may encourage some motorcyclists to
not wear helmets, in order to avoid targeting by autonomous cars. Likewise, in the
previous scenario, sales may decline for automotive brands known for safety, such as
Volvo and Mercedes Benz, insofar as customers want to avoid being the preferred targets
of crash-optimization systems.
Some readers may want to argue that the motorcyclist without a helmet ought to be
targeted, for instance, because he has acted recklessly and therefore is more deserving of
harm. Even if thats the correct design, notice that we are again moving beyond harm in
making crash-optimization decisions. Were still talking about justice a nd other such
ethical considerations, and thats the point: its not just a numbers game.
Programmers in such scenarios, as rare as they may be, would need to design
cost-functionsalgorithms that assign and calculate the expected costs of various possible
options, selecting the one with the lowest coststhat potentially determine who gets to
live and who gets to die. And this is fundamentally an ethics problem, one that demands
much more care and transparency in reasoning than seems currently offered. Indeed, it is
difcult to imagine a weightier and more profoundly serious decis ion a programmer
would ever have to make. Yet, there is little discussion about this core issue to date.
4.2 Scenarios that Implicate Ethics
In addition to the ones posited above, there are many actual and hypothetical scenarios
that involve judgments about ethics. I will describe some here to show how ordinary
assumptions in ethics can be challenged.
4.2.1 The Deer
Though difcult to quantify due to inconsistent and under- reporting, experts estimate that
more than a million car accidents per year in the US are caused by deer [6, 48]. Many, if
not most, drivers have been startled by an unexpect ed animal on the road, a dangerous
situation for both parties. Deconstructing a typical accident, or near-accident, involving an
animal illustrates the complexity of the decisions facing the driver [30]. While all this
happens within secondsnot enough time for careful deliberations by human driversan
4 Why Ethics Matters for Autonomous Cars 73
autonomous car could have the virtue of a (presumably) thoughtful decision-making script
to very quickly react in an opti mal way. If it is able to account for the many variables, then
it ought to, for the most informed decision possible.
First, suppose an object appears on the road directly in front of a car in autonomous
mode. Is there time to reasonably hand control back to the human behind the wheel?
(Probably not.) If not, is there time to stop the car? Would the car need to brake hard, or
would moderate braking be sufcient? The decision to brake depends, again, on road
conditions and whether a tailgater (such as a big-rig truck) is behind you, including its
speed to determin e the severity of a possible rear-end collision.
Second, what is the object? Is it an animal, a person, or something else? If it is an
animal, are some animals permissible to run over? It may be safer to continue ahead and
strike a squirrel, for instance, than to violently swerve around it and risk losing control of
the car. However, larger animals, such as deer and cows, are more likely to cause serious
damage to the car and injuries to occupants than a spun-out car. Other animals, still, have
special places in our hearts and should be avoide d if possible, such as pet dogs and cats.
Third, if the car should get out of the wayeither in conjunction wi th braking or not
should it swerve to the left or to the right? In the US and other nations in which drivers
must stay on the right side of the road, turning to the right may mean driving off the road,
potentially into a ditch or a tree. Not on ly could harm to the car and occupants be likely,
but it also matters how many occupants are in the car. The decision to drive into an
embankment seems different when only one adult driver is in the car, than when several
children are inside too.
On the other hand, turning to the left may mean driving into an opposite lane, potentially
into a head-on collision with incoming vehicles. If such a collision is unavoidable, then it
matters what kind of vehicle we would crash into (e.g., is it a compact car or SUV?), how
heavy incoming trafc is (e.g., would more than one vehicle be involved?), how many
persons may be involved (e.g., are there children in the other car?). Of course, here we are
assuming perfect sensing and V2X communi cations that can help answer these questions.
If we cannot answer the questions, then we face a possibly large unknown risk, which
makes driving into incoming trafc perhaps the worst option available.
Other factors relevant to the decision-points above include: the road-shoulder type
(paved, gravel, none, etc.), the condition of the cars tires and brakes, whether the cars
occupants are seat-belted, whether the car is transporting dangerous cargo that could spill
or explode, proximity to hospital or emergency rescue, damage to property such as houses
and buildings, and more. These variables inuence the probability of an accident as well
as expected harm, both of which are needed in selecting the best course of action.
From this short analysis of a typical crash (or possible crash) with an animal, we can
already see a daunting number of factors to account for. Sensing technologies today
cannot answer some or many of the questions above, but it is already unclear that braking
should be the safest default optionas a proxy for the most ethical optiongiven these
uncertain conditions, all things considered. Automated cars today can already detect
whether there is oncoming trafc in the opposite lane. Therefore, it is at least possible that
74 P. Lin
they can be programmed to maneuver slightly into the incoming lane under some con-
ditions, e.g., when there are no incoming cars and when it may be dangerous to slam on
the brakes.
Whether or not sensing technologies will improve enough to deliver answers to our
questions above, a programmer or OEM would still need to assign costs or weights to
various actions and objects as best as they can. Yet these values are not intrinsic to or
discoverable by science or engineering. Values are something that we humans must
stipulate and ideally agree upon. In constructing algorithms to control an autonomous car,
ethics is already implied in the design process. Any decision that involves a tradeoff such
as to strike object x instead of object y requires a value-judgment about the wisdom of the
tradeoff, that is, the relative weights of x and y. And the design process can be made better
by recognizing the ethical implications and by engaging the broader community to ensure
that those values are represented correctly or at least transparently. Working in a moral
bubble is less likely to deliver results that are acceptable to society.
Again, in a real-world accident today, a human driver usually has neither the time nor
the information needed to make the most ethical or least harmful decisions. A person who
is startled by a small animal on an otherwi se-uneventful drive may very well react poorly.
He mig ht drive into oncoming trafc and kill a family, or oversteer into a ditch and to his
own death. Neither of these results, however, is like ly to lead to criminal prosecution by
themselves, since there was no forethought, malice, negligence, or bad intent in making a
forced, split-s econd reaction. But the programmer and OEM do not operate under the
sanctuary of reasonable instincts; they make potentially life-and-death decisions under no
truly urgent time-constraint and therefore incur the responsibility of making better
decisions than human drivers reacting reexively in surprise situations.
4.2.2 Self-sacrifice
As we can see, real-world accidents can be very complicated. In philosophy and ethics, a
familiar method is to simplify the issues through hypothetical scenarios, otherwise known
as thought-experiments. This is similar to everyday science experiments in which
researchers create unusual conditions to isolate and test desired variables, such as sending
spiders into outer space to see how micro-gravity affects their ability to spin webs. It is not
a good objection to those experiments to say that no spiders exist naturally in space; that
misses the point of the experiment.
Likewise, it is no objection to our hypothetical examples that they are outlandish and
unlikely to happen in the real world, such as a car that can distinguish an eight-year old
from an 80-year old (though with improving biometrics, facial recognition technologies,
and linked databases, this doesnt seem impossible). Our thought-experiments are still
useful in drawing out certain ethical intuitions and principles we want to test.
With that understanding, we can devise hypothetical scenarios to see that reasonable
ethical principles can lead to controversial results in the context of autonomous driving.
4 Why Ethics Matters for Autonomous Cars 75
Digging into a standard philosophical toolbox for help with ethical dilemmas, one of the
rst principles we might reach for is consequ entialism: that the right thing to do is
whatever leads to the best results, especially in quanti ed terms [44]. As it applies here,
consequentialism suggests that we should strive to minimize harm and maximize whatever
it is that matters, such as, the number of happy lives.
In this thought-experiment, your future autonomous car is driving you on a narrow
road, alongside a cliff. No one and no technology could fores ee that a school bus with 28
children would appear around the corner, partially in your lane [29, 36]. Your car cal-
culates that crash is imminent; given the veloci ties and dist ance, there is no possible action
that can avoid harming you. What should your robot car do?
A good, standard-issue consequentialist would want to optimize results, that is, max-
imize the number of happy lives and minimize harm. Assuming that all lives in this
scenario are more or less equally happyfor instance, theres no super-happy or
super-depressed person, and no very important person who has unusual inuence over the
welfare of othersthey would each count for about the same in our moral calcul ation. As
you like, we may either ignore or account for the issue of whether there is extra value in
the life of innocent child who has more years of happiness ahead of her than an average
adult; that doesn t matter much for this scenario.
The robot cars two main choices seem to be: (1) to slam on the brakes and crash into
the bus, risking everyones lives, or (2) to drive off the cliff, sparing the lives of everyone
on the bus. Performin g a quick expected-utility calculation, if the odds of death to each
person (including the adult bus driver) in the ac cident averaged more than one in 30, then
colliding into the bus would yield the expected result of more than one death, up to all 30
persons. (Lets say the actual odds are one in three, which gives an expected result of 10
deaths.) If driving off a cliff meant certain death, or the odds of one in one, then the
expected result of that would be exactly one death (your own) and no more. The right
consequentialist decision for the robot carif all we care about is maximizing lives and
minimizing deathsis apparently to drive off the cliff and sacri ce the driver, since it is
better that only one person should die rather than more than one, especially 10 or all 30
This decision would likely be different if, instead of a school bus, your robot car were
about to collide with another passenger car carrying ve persons. Given the same average
odds of death, one in 10, the expected number of deaths in a collision would only be 0.6,
while the expected number of deaths in driving off a cliff remains at one. In that case, the right
consequentialist decision would be to allow the accident to occur, as long as the average odds
of death are less than one in six. If, instead of another vehicle, your car were about to collide
with a deer, then the decision to stay on the road, despite an ensuing accident, would be even
more obvious insofar as we value a deers life less than a human life.
Back to the school-bus scenario, programming an autonomous car with a conse-
quentialist framework for ethics would seem to imply your sacrice. But what is most
striking about this case might not even be your death or the moral mathematics: if you
were in a manually driven car today, driving off the cliff might still be the most ethical
76 P. Lin
choice you could make, so perhaps you would choose certain death anyway, had you the
time to consider the options. However, it is one thing for you to willingly make that
decision of sacrice yourself, and quite another matter for a machine to make that decision
without your consent or foreknowledge that self-sacrice was even a possibility. That is,
there is an astonishing lack of transparency and there fore consent in such a grave decision,
one of the most important that can be made about one s lifeperhaps noble if voluntary,
but criminal if not.
Thus, reasonable ethical principlese.g., aiming to save the greatest number of lives
can be stressed in the context of autonomous driving. An operator of an autonomous
vehicle, rightly or not, may very well value his own life over that of everyon e elses, even
that of 29 others; or he may even explicitly reject consequentialism. Even if conse-
quentialism is the best ethical theory and the cars moral calculations are correct, the
problem may not be with the ethics but with a lack of discussion about ethics. Industry,
therefore, may do well to have such a discussion and set expectations with the public.
Usersand news headlinesmay likely be more forgiving if it is explai ned in advance
that self-sacrice may be a justied feature, not a bug.
4.2.3 Ducking Harm
Other ethical principles can create dilemmas, too. It is generally uncont roversial that, if
you can easily avoid harm to yourse lf, then you should do it. Indeed, it may be morally
required that you save yourself when possible, if your life is intrinsically valuable or worth
protecting; and it is at least extrinsically valuable if you had a dependent family. Auto
manufacturers or OEMs seem to take this principle for granted as well: if an autonomous
car can easily avoid a crash, e.g., by braking or swerving, then it should. No ethical
problem hereor is there?
In another thought-experiment [15, 18, 33], your robotic car is stopped at an inter-
section and waits patiently for the children who are crossing in front of you. Your car
detects a pickup truck coming up behind you, about to cause a rear-end collision with you.
The crash would likely damage your car to some degree and perhaps cause minor injury to
you, such as whiplash, but certainly not death. To avoid this harm, your car is pro-
grammed to dash out of the way, if it can do so safely. In this case, your car can easily turn
right at the intersection and avoid the rear-end collision. It follows this programming, but
in doing so, it clears a path for the truck to continue through the intersection, killing a
couple children and seriously injuring others.
Was this the correct way to program an autonomous car? In most cases of an
impending rear-end collision, probably yes. But in this particular case, the design decision
meant saving you from minor injury at the expense of serious injury and death of several
children, and this hardly seems to be the right choice. In an importan t respect, you (or the
car) are responsible for their deaths: you (or the car) killed the children by removing an
obstruction that prevented harm from falling upon them, just as you would be responsible
4 Why Ethics Matters for Autonomous Cars 77
for a persons death if you removed a shield he was holding in front of a stream of gun re.
And killing innocent people has legal and moral ramications.
As with the self-sacrice scenario above, it might be that in the same situation today, in
a human -driven car, you would make the same decis ion to save yourself from injury, if
you were to see a fast-approaching vehicle about to slam into you. That is, the result might
not change if a human made the on-the-spot decision. But, again, it is one thing to make
such a judgment in the panic of the moment, but another less forgivable thing for a
programmerfar removed from the scene and a year or more in advanceto create a
cost-function that resulted in these deaths. Either the programmer did so deliberately, or
she did it unintentionally, unaware that this was a possibility. If the former, then this could
be construed as premeditated homicide; and if the latter, gross negligence.
Either way is very bad for the programmer and perhaps an inherent risk in the business,
when one atte mpts to replicate human decision-making in a broad range of dynamic
scenarios. Sometimes, an autonomous car may be faced with a no-win scenario, putting
the programmer in a difcult but all too real position. To mitigate this risk, industry may
do well to set expectations not only with users but also with broader society, educating
them that they could also become victims even if not operating or in a robot car, and that
perhaps this is justied by a great er public or overall good.
4.2.4 Trolley Problems
One of the most iconic thought-experiments in ethics is the trolley problem [4, 8, 11, 47],
and this is one that may now occur in the real world, if autonomous vehicles come to be.
Indeed, driverless trains are already operating in dozens of cities worldwide and could
bring this scene to life [24]. The classical dilemma involves a runaw ay trolley (or train)
that is about to run over and kill ve unaware people standing on the tracks. Looking at
the scene from the outside, you nd yourself standing next to a switch: if you pull the
switch, you can shunt the train to a right-hand set of tracks, thereby saving the ve
individuals on the track. Unfortunately, there is one person standing on the right-hand set
of tracks who would then be killed. What is the right decision?
The correct decision continues to be a subject of much debate in philosophy. Both
answers seem reasonable and defensible. A consequentialist might justify switching the
tracks to save ve people, even at the regrettable expense of one. But a non-consequentialist,
someone who considers more than just the math or results, might object on the grounds that
switching tracks constitutes an act of killing (the one person), while doing nothing is merely
allowing someone to die (the ve individuals); and that it is morally and legally worse to kill
than to let die.
Killing implies that you are directly responsible for a persons death: had you not done
what you did, the person would have lived. Letting die, however, involves much less
responsibility on your part, if any, since some causal process was already underway that
was not initiated or otherwise controlled by you. The question of whether it is wor se to kill
78 P. Lin
than to let die is also subject to debate in philosophy. But let us bracket that for the
moment, as a nal answer is not necessary for our discussion, only that it is reasonable to
believe that proposition.
Adapting the trolley problem to the technology at hand, let us suppose that you are
driving an autonomous car in manual mode; you are in control. Either intentionally or not
you could be homicidal or simply inattentiveyou are about to run over and kill ve
pedestrians. Your cars crash-avoidance system detects the possible accident and
activates, forcibly taking control of the car from your hands. To avoid this disaster, it
swerves in the only direction it can, lets say to the right. But on the right is a single
pedestrian who is unfortunately killed.
Was this the right decision for your car to make? Again, a consequentialist would say
yes: it is better that only one person dies than ve. But a non-consequentialist might
appeal to a moral distinction between killing and letting die, and this matters to OEMs for
liability reasons. If the car does not wrestle control from the human driver, then it (and the
OEM) would perhaps not be responsible for the deaths of the ve pedestrians while you
were driving the car; it is merely letting those victims die. But if the car does take control
and make a decision that results in the death of a person, then it (and the OEM) becomes
responsible for killing a person.
As with the trolley problem, either choice seems defensible. Results do matter, so it is
not ridiculous to think that the car should be programmed to act and save lives, even at the
expense of a fewer number of lives. Yet it also seems reasonable to think that killing is
worse than letting die, especially in the eyes of the law. What I want to highlight here is
not so much the answer but the process of deliberation that points us toward one answer
over another. To the extent that there could be many acceptable answers to any given
ethical dilemma, how well one answer can be defended is crucial toward supporting that
answer over others.
Industry again would do well to set expectations by debating and explaining in advance
its reasoning behind key algorithms that could result in life or death. Transparency, or
showing ones math, is an important part of doing ethics, not just the answer itself.
4.3 Next Steps
Notice that the ethical issues discussed in this paper do not depend on technology errors,
poor maintenance, imp roper servicing, security vulnerabilities, or other failingsand all
those will occur too. No complex technology we have created has been infallible. Even
industries with money directly at stake have not solved this problem. For instance, bank
ATMs continue to make headlines when they hemorrhage cashtens of thousands of
dollars more than the account holder actually hasbecause of software glitches alone
[2, 10], never mind hacking. And just about every computing device we have created has
been hacked or is hackable, including neural implants and military systems [3, 28].
4 Why Ethics Matters for Autonomous Cars 79
These vulnerabilities and errors certainly can cause harm in the context of autonomous
cars, and it would be unethically irresponsible to safeguard against them where we can.
Putting these technology issues aside and even assuming that perfect technology is
available, there are still many other safety and ethical questions to worry about, such as
the programming issues above.
4.3.1 Broader Ethical Issues
But programming is only one of many areas to reect upon as society begins to adopt
autonomous driving technologies. Assigning legal and moral responsibility for crashes is a
popular topic already [1, 14, 20, 22, 49, 51]. Here are a few others, as part of a much
longer list of possible questions:
Does it matter to ethics if a car is publicly owned, for instance, a city bus or re truck?
The owner of a robot car may reasonably expect that its property owes allegiance to the
owner and should value his or her life more than anonymous pedestrians and drivers. But
a publicly owned automated vehicle might not have that obligation, and this can change
moral calculations. Even for privately owne d autono mous vehicles, the occupants argu-
ably should bear more or all of the risk, since they are the ones introducing the machine
into public spaces in the rst place.
Do robot cars present an existential threat to the insurance industry? Some believe that
ultra-safe cars that can avoid most or all accidents will mean that many insurance com-
panies will go bankrupt, since there would be no or very little risk to insure against [40, 52].
But things could go the other way too: We could see mega-accidents as cars are networked
together and vulner able to wireless hackingsomething like the stock markets ash
crash in 2010 [5]. What can the insurance industry do to protect its elf while not getting in
the way of the technology, which holds immense benets?
How susceptible would robot cars be to hacking? So far, just about every computing
device we have created has been hacked. If authorities and owners (e.g., rental car
company) are able to remotely take control of a carwhich is reportedly under devel-
opment for law enforcement in the European Union [50]this offers an easy path for
cyber-carjackers. If under attack, whether a hijacking or ordinary break-in, what should
the car do: speed away, alert the police, remain at the crime scene to preserve evidence, or
maybe defend itself?
For a future suite of in-car apps, as well as sensors and persistent GPS/track ing, can we
safeguard personal information, or do we resign ourselves to a world with disappearing
privacy rights [27]? To the extent that online services bring online advertising, we could
see new, insidious advertising schemes that may allow third-party advertisers to have
some inuence on the autonomous cars route selection, e.g., steering the car past their
businesses [32].
What kinds of abuse might we see with autonomous cars? If the cars drive too con-
servatively, they may become a trafc hazard or trigger road-rage in human drivers with
80 P. Lin
less patience [26, 42]. If the crash-avoidance system of a robot car is generally known,
then other drivers may be tempted to game it, e.g., by cutting in front of it, knowing that
the automated car will slow down or swerve to avoid an accident. If those cars can safely
drive us home in a fully-auto mode, that may encourage a culture of more alcohol
consumption, since we wont need to worry so much about drunk-driving.
More distant concerns include: How will law-abiding robot cars affect city revenue,
which often depends on trafc nes imposed against law-breaking human drivers? Inas-
much as many organ transplants come from car-accident victims, how will society manage
a declining and already insufcient supply of donate d organs [41]?
Older-model autonomous cars may be unable to communicate with later models or
future road infrastructure. How do we get those legacy modelswhich may be less safe,
in addition to incompatible with newer technologyoff the roads [45]? Since 2009,
Microsoft has been trying to kill off its Windows XP operating system [39], a much less
expensive investment than an autonomous car; but many users still refuse to relinquish it,
including for critical military systems [37, 46]. This is a great security risk since Microsoft
will no longer offer software patches for the operating system.
4.3.2 Conclusions
We dont really know what our robot-car future will look like, but we can already see that
much work needs to be done. Part of the problem is our lack of imagination. Brookin gs
Institution director Pe ter W. Singer observ ed, We are still at the horseless carriage stage
of this technology, describing these technologies as what they are not, rather than wres-
tling with what they truly are [43].
As it applies here, robots arent merely replacing human drivers, just as human drivers
in the rst automobiles werent simply replacing horses: that would like mistaking
electricity as merely a replacement for candles. The impact of automating transportation
will change society in radical ways, and technology seems to be accelerating. As Singer
puts it, Yes, Moores Law is operative, but so is Murphys Law [43]. When technology
goes wrongand it willthinking in advance about ethical design and policies can help
guide us responsibility into the unknown.
In future autonomous cars, crash-avoidance features alone wont be enough. An
accident may be unavoidable as a matter of physics [12, 13], especially as autonomous
cars make their way onto city streets [19, 21, 25], a more dynamic environment than
highways. It also could be too dangerous to slam on the brakes, or not enough time to
hand control back to the unaware human driver, assuming theres a human in the vehicle
at all. Technology errors, misaligned sensors, malicious actors, bad weather, and bad luck
can also contribute to immin ent collisions. Therefore, robot cars will also need to have
crash-optimization strategies that are thoughtful about ethics.
If ethics is ignored and the robot ic car behaves badly, a powerful case could be made
that auto manufacturers were negligent in the design of their product, and that opens them
4 Why Ethics Matters for Autonomous Cars 81
up to tremendou s legal liability, should such an event happen. Today, we see activists
campaigning against killer military robots that dont yet exist, partly on the grounds that
machines should never be empowered to make life-and-death decisions [31, 35]. Its not
outside the realm of possibility to think that the same precautionary backlash wont
happen to the autonomous car industry, if industry doesnt appear to be taking ethics
The larger challenge, though, isnt just about thinking through ethical dilemmas. Its
also about setting accurate expectations with users and the general public who might nd
themselves surprised in bad ways by autonomous cars; and expectations matter for market
acceptance and adoption. Whatever answer to an ethical dilemma that industry might lean
towards will not be satisfying to everyone. Ethics and expectations are challenges com-
mon to all automotive manufacturers and tier-one suppliers who want to play in this
emerging eld, not just particular companies.
Automated cars promise great benets and unintended effects that are difcult to
predict, and the technology is coming either way. Change is inescapable and not neces-
sarily a bad thing in itself. But major disruptions and new harms should be anticipated and
avoided where possible. That is the role of ethics in innovation policy: it can pave the way
for a better future while enabling benecial technologies. Without looking at ethics, we
are driving with one eye closed.
Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0
International License (, which permits use, duplication,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit
to the original author(s) and the source, a link is provided to the Creative Commons license and any
changes made are indicated.
The images or other third party material in this chapter are included in the work's Creative Commons
license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative
Commons license and the respective action is not permitted by statutory regulation, users will need to
obtain permission from the license holder to duplicate, adapt or reproduce the material.
1. Anderson, J., Kalra, N., Stanley, K., Sorensen, P., Samaras, C., Oluwatola, O.: Autonomous
vehicle technology: a guide for policymakers. Report by RAND Corporation. http://www.rand.
org/pubs/research_reports/RR443-1.html (2014). Accessed 8 July 2014
2. Associated Press. ATM glitch gives $37,000 to lucky homeless man. Daily Mail. http://www.
requested-140.html (2014). Accessed 8 July 2014
3. Baldor, L.: China hacked the Pentagon to get weapons data. Talking Points Memo. http://
(2013). Accessed 8 July 2014
4. Cathcart, T.: The Trolley Problem, or Would You Throw the Fat Guy Off the Bridge? Workman
Publishing Company, New York (2013)
82 P. Lin
5. Commodity Futures Trading Commission and the Securities and Exchange Commission:
Findings regarding the market events of May 6, 2010. CFTC and SEC, Washington DC. http:// (2010). Accessed 8 July 2014
6. Curtis, P. and Hedlund, J.: Reducing deer-vehicle crashes. Report funded by the Insurance
Institute for Highway Safety. Cornell University, Ithaca.
Documents/Deer/Deer-Vehicle_factsheet1.pdf (2005). Accessed 8 July 2014
7. Davies, A.: Avoiding squirrels and other things Googles robot car cant do. Wired. http://www. (2014). Accessed 8 July 2014
8. Edmonds, D.: Would You Kill the Fat Man? The Trolley Problem and What Your Answer Tells
Us About Right and Wrong. Princeton University Press, Princeton (2014)
9. Federal Ministry of Justice and Consumer Protection: Basic Law for the Federal Republic of
Germany. (2014). Accessed 8
July 2014
10. Floro, Z.: Man goes on a casino bender after an ATM let him take out unlimited cash. Business
Insider. (2012). Accessed 8
July 2014
11. Foot, P.: The problem of abortion and the doctrine of the double effect. Oxford Review 5, 5-15
12. Fraichard, T.: Will the driver seat ever be empty? Report funded by Inria.
00965176 (2014). Accessed 8 July 2014
13. Fraichard, T., and Asama, H.: Inevitable collision states: a step towards safer robots? Advanced
Robotics 18(10), 10011024 (2004)
14. Garza, A.: Look Ma, no hands!: wrinkles and wrecks in the age of autonomous vehicles. New
England Law Review 46(3), 581-616 (2012)
15. Goodall, N.J.: Autonomous car ethics. Interview with CBC radio.
blog/2014/04/13/autonomous-car-ethics/ (2014). Accessed 8 July 2014
16. Goodall, N. J.: Machine ethics and automated vehicles. In: Meyer, G. and Beiker, S. (eds.) Road
Vehicle Automation. Springer, Cham (2014)
17. Goodall, N. J.: Ethical decision making during automated vehicle crashes. Transportation
Research Record: Journal of the Transportation Research Board (forthcoming)
18. Goodall, N. J.: Vehicle automation and the duty to act. In: Proceedings of the 21st World
Congress on Intelligent Transport Systems, 7-11 September 2014, Detroit, Michigan
19. Google: The latest chapter for the self-driving car: mastering city street driving. http:// (2014). Accessed
8 July 2014
20. Gurney, J.: Sue my car, not me: products liability and accidents involving autonomous vehicles.
Journal of Law, Technology and Policy 2, 247-277 (2013)
21. Hern, A.: Self-driving cars face a long and winding road to success. The Guardian. http://www. (2014). Accessed 8
July 2014
22. Hevelke, A. and Nida-Rumelin, J.: Responsibility for crashes of autonomous vehicles: an
ethical analysis. Science and Engineering Ethics (2014). doi: 10.1007/s11948-014-9565-5
23. IEEE: IEEE code of ethics. (2014).
Accessed 8 July 2014
24. International Association of Public Transport: Observatory of automated metros. http:// (2014). Accessed 8 July 2014
4 Why Ethics Matters for Autonomous Cars 83
25. Jaffe, E.: The rst look at how Googles self-driving car handles city streets. The
car-handles-city-streets/8977/ (2014). Accessed 8 July 2014
26. KPMG: Self-driving cars: are we ready? Report by KPMG. KPMG, Chicago. http://www.
ready.pdf (2013). Accessed 8 July 2014
27. Lee, T.: Self-driving cars are a privacy nightmare. And its totally worth it. The Washington
privacy-nightmare-and-its-totally-worth-it/ (2013). Accessed 8 July 2014
28. Leggett, H.: The new hacking frontier: your brain? Wired.
neurosecurity/ (2009). Accessed 8 July 2014
29. Lin, P.: The ethics of saving lives with autonomous cars is far murkier than you think. Wired. (2013). Accessed 8 July
30. Lin, P.: The ethics of autonomous cars. The Atlantic.
archive/2013/10/the-ethics-of-autonomous-cars/280360/ (2013). Accessed 8 July 2014
31. Lin, P.: Why the drone wars matter for automated cars. Lecture presented at Proceedings in
Automated Driving, Stanford Law School, 12 December 2013.
Accessed 8 July 2014
32. Lin, P.: What if your autonomous car keeps routing you past Krispy Kreme? The Atlantic.
routing-you-past-krispy-kreme/283221/ (2014). Accessed 8 July 2014
33. Lin, P.: Ethics and autonomous cars: why ethics matters, and how to think about it. Lecture
presented at Daimler and Benz Foundations Villa Ladenburg Project, Monterey, California, 21
February 2014
34. Lin, P.: The robot car of tomorrow might just be programmed to hit you. Wired. http://www. (2014).
Accessed 8 July 2014
35. Lin, P., Bekey, G., Abney, K.: Autonomous military robotics: risk, ethics, and design. Report
funded by the US Ofce of Naval Research. California Polytechnic State University, San Luis
Obispo. (2008). Accessed 8 July 2014
36. Marcus, G.: Moral machines. New Yorker.
2012/11/google-driverless-car-morality.html (2012). Accessed 8 July 2014
37. McCabe, R.: Navy, others still struggling to ditch Windows XP. The Virginian-Pilot. http:// (2014). Accessed 8
July 2014
38. Merat, N., Jamson, H., Lai F., and Carsten, O.: Human factors of highly automated driving:
results from the EASY and CityMobil projects. In: Meyer, G. and Beiker, S. (eds) Road Vehicle
Automation. Springer, Cham (2014)
39. Microsoft: Windows lifecycle fact sheet.
lifecycle (2014). Accessed 8 July 2014
40. Mui, C. and Carroll, P.: Driverless Cars: Trillions Are Up For Grabs. Cornerloft Press, Seattle
41. Park, A.: Driverless cars to kill organ transplantation. DriverlessCarHQ.
web/20120626151201/ (2012). Accessed 8 July
42. Roach, J.: Road rage at driverless cars? Its possible. NBC News. http://futureoftech-discuss. (2012).
Accessed 8 July 2014
84 P. Lin
43. Singer, P.W.: The robotic revolution. Brookings Institution.
opinions/2012/12/11-robotics-military-singer (2012). Accessed 8 July 2014
44. Sinnott-Armstrong, W.: Consequentialism. Stanford Encyclopedia of Philosophy. http://plato. (2011). Accessed 8 July 2014
45. Smith, B.W.: Planning for the obsolescence of technologies not yet invented. Stanford Law
Schools Center for Internet and Society.
obsolescence-technologies-not-yet-invented (2013). Accessed 8 July 2014
46. Sternstein, A.: Why Feds are still buying IT that works with Windows XP. Nextgov. http://
81667/ (2014). Accessed 8 July 2014
47. Thomson, J.J.: Killing, letting die, and the trolley problem. The Monist 59, 204-217 (1976)
48. Transportation Research Board: Animal-vehicle collision data collection. http://onlinepubs.trb.
org/onlinepubs/nchrp/nchrp_syn_370.pdf (2007). Accessed 8 July 2014
49. Villasenor, J.: Product liability and driverless cars: issues and guiding principles for legislation.
Report by The Brookings Institution.
products-liability-driverless-cars-villasenor (2014). Accessed 8 July 2014
50. Watereld, B. and Day, M.: EU has secret plan for police to remote stop cars. The Telegraph.
police-to-remote-stop-cars.html (2014). Accessed 8 July 2014
51. Yeomans, G.: Autonomous vehicles: handing over controlopportunities and risk for
insurance. Report by Lloyds. Lloyds, London.*/media/lloyds/
reports/emerging%20risk%20reports/autonomous%20vehicles%20nal.pdf (2014). Accessed
8 July 2014
52. Yeomans, G.: Autonomous vehicles: handing over controlopportunities and risk for
insurance. Report by Lloyds. Lloyds, London.*/media/lloyds/
reports/emerging%20risk%20reports/autonomous%20vehicles%20nal.pdf (2014). Accessed
8 July 2014
4 Why Ethics Matters for Autonomous Cars 85
... This makes it essential that decisions be taken when programming the software of highly and fully automated driving systems." As noted in [54], for " future autonomous cars, crash-avoidance features alone won't be enough", but when crash is inevitable, there needs to be a crash-optimization strategy. However, that strategy should aim not only to minimise damage since if that was the case, the vehicle might decide to crash into a cyclist wearing a helmet than a cyclist not wearing a helmet, thus, targeting people who chose to be safer-there is no clear resolution to this ethical issue, as of yet. ...
Full-text available
The Internet of Things is emerging as a vast, inter-connected space of devices and things surrounding people, many of which are increasingly capable of autonomous action, from automatically sending data to cloud servers for analysis, changing the behaviour of smart objects, to changing the physical environment. A wide range of ethical concerns has arisen in their usage and development in recent years. Such concerns are exacerbated by the increasing autonomy given to connected things. This paper reviews, via examples, the landscape of ethical issues, and some recent approaches to address these issues concerning connected things behaving autonomously as part of the Internet of Things. We consider ethical issues in relation to device operations and accompanying algorithms. Examples of concerns include unsecured consumer devices, data collection with health-related Internet of Things, hackable vehicles, behaviour of autonomous vehicles in dilemma situations, accountability with Internet of Things systems, algorithmic bias, uncontrolled cooperation among things, and automation affecting user choice and control. Current ideas towards addressing a range of ethical concerns are reviewed and compared, including programming ethical behaviour, white-box algorithms, black-box validation, algorithmic social contracts, enveloping IoT systems, and guidelines and code of ethics for IoT developers; a suggestion from the analysis is that a multi-pronged approach could be useful based on the context of operation and deployment.
... Concurrently, self-driving cars have been problematized as robots in public spaces, raising complex questions of safety assurance, human-machine interaction, and legal accountability. There has been a particular interest in the ethical dilemmas that self-driving cars might introduce (Lin, 2016). Most publications on the ethics of AVs have taken a very narrow view of ethics, treating self-driving cars as a case study for the application of practical ethics thought experiments, rather than offering analysis that would be of use to policymakers or engineers (JafariNaimi, 2018). ...
... Autonomous transportation systems (Lin 2016) and robotics (Alsegier 2016) have been considered as the main application fields of machine ethics in the field of technical engineering. In robotics, the word "Roboethics" has been coined and is considered as an emerging discipline in robotics (Westerlund 2020). ...
Full-text available
This paper addresses the engineering of the ethical behaviors of autonomous industrial cyber-physical human systems in the context of Industry 4.0. An ethical controller is proposed to be embedded into these autonomous systems, to enable their successful integration in the society and its norms. This proposed controller that integrates machine ethics is realized through three main strategies that utilize two ethical paradigms, namely deontology, and consequentialism. These strategies are triggered according to the type of event sensed and the state of the autonomous industrial cyber-physical human systems, their combination being potentially unknown or posing ethical dilemmas. Two case studies are investigated, that deal with a fire emergency, and two different contexts i.e. one with an autonomous train, and one with an autonomous industrial plant, are discussed to illustrate the controller utilization. The case studies demonstrate the potential benefits and exemplify the need to integrate ethical behaviors in autonomous industrial cyber-physical human systems already at the design phase. The proposed approach, use cases, and discussions make evident the need to address ethical aspects in new efforts to engineer industrial systems in the context of Industry 4.0.
... Ethical risks are some of the most strongly emphasized groups of the risks identified, since the most emphasized question is how the autonomous vehicle system reacts to a suspected accident situation (Blyth et al. 2015, Bonnefon et al. 2016. It is important to realize that it is not the autonomous vehicle engine and mechanics that provide a solution for these dangerous situations, but the algorithms that operate the vehicle, algorithms written by people (Bonnefon et al. 2016, Lin 2016, Yeomans 2014. The scenarios we need to examine already raise bafflingly complex issues and the individual elements that make up these scenarios offer almost infinite opportunities to discuss moral dilemmas, which touch upon areas of engineering, law, economics, philosophy and information technology. ...
Full-text available
Digitalization, a dominant megatrend in today's global world, offers numerous intriguing technological possibilities. Out of these novelties, self-driving cars have rapidly come to be a primary focus; the literature categorizes them as a radical innovation due to the possibility that the mass adoption of self-driving cars would not only radically change everyday life for members of industrialized societies, but calls into question the infrastructural, legal, and social ordering of towns and numerous aspects of transportation in the societies that adopt them. Meanwhile, the results of several international surveys with large samples show that public opinion of self-driving cars is ambivalent, indicating parallel signals of enthusiasm and concern. The aim of this paper is to develop key components of a general strategy for addressing the societal challenges associated with self-driving cars as identified in international surveys and relevant literature and using the framework of responsible innovation.
There are growing calls for more digital ethics, largely in response to the many problems that have occurred with digital technologies. However, there has been less clarity about exactly what this might mean. This chapter argues first that ethical decisions and considerations are ubiquitous within the creation of digital technology. Ethical analyses cannot be treated as a secondary or optional aspect of technology creation. This argument does not specify the content of digital ethics, though, and so further research is needed. This chapter then argues that this research must take the form of translational ethics: a robust, multi-disciplinary effort to translate the abstract results of ethical research into practical guidance for technology creators. Examples are provided of this kind of translation from principles to different types of practices.
Full-text available
Advocates of autonomous driving predict that the occupation of taxi driver could be made obsolete by shared autonomous vehicles (SAV) in the long term. Conducting interviews with German taxi drivers, we investigate how they perceive the changes caused by advancing automation for the future of their business. Our study contributes insights into how the work of taxi drivers could change given the advent of autonomous driving: While the task of driving could be taken over by SAVs for standard trips, taxi drivers are certain that other areas of their work such as providing supplementary services and assistance to passengers would constitute a limit to such forms of automation, but probably involving a shifting role for the taxi drivers, one which focuses on the sociality of the work. Our findings illustrate how taxi drivers see the future of their work, suggesting design implications for tools that take various forms of assistance into account, and demonstrating how important it is to consider taxi drivers in the co-design of future taxis and SAV services.
Full-text available
In 2017, the German ethics commission for automated and connected driving released 20 ethical guidelines for autonomous vehicles. It is now up to the research and industrial sectors to enhance the development of autonomous vehicles based on such guidelines. In the current state of the art, we find studies on how ethical theories can be integrated. To the best of the authors’ knowledge, no framework for motion planning has yet been published which allows for the true implementation of any practical ethical policies. This paper makes four contributions: Firstly, we briefly present the state of the art based on recent works concerning unavoidable accidents of autonomous vehicles (AVs) and identify further need for research. While most of the research focuses on decision strategies in moral dilemmas or crash optimization, we aim to develop an ethical trajectory planning for all situations on public roads. Secondly, we discuss several ethical theories and argue for the adoption of the theory “ethics of risk.” Thirdly, we propose a new framework for trajectory planning, with uncertainties and an assessment of risks. In this framework, we transform ethical specifications into mathematical equations and thus create the basis for the programming of an ethical trajectory. We present a risk cost function for trajectory planning that considers minimization of the overall risk, priority for the worst-off and equal treatment of people. Finally, we build a connection between the widely discussed trolley problem and our proposed framework.
For the first time in history, automated vehicles (AVs) are being deployed in populated environments. This unprecedented transformation of our everyday lives demands a significant undertaking: endowing complex autonomous systems with ethically acceptable behavior. We outline how one prominent, ethically relevant component of AVs-driving behavior-is inextricably linked to stakeholders in the technical, regulatory, and social spheres of the field. Whereas humans are presumed (rightly or wrongly) to have the "common sense" to behave ethically in new driving situations beyond a standard driving test, AVs do not (and probably should not) enjoy this presumption. We examine, at a high level, how to test the common sense of an AV. We start by reviewing discussions of "driverless dilemmas," adaptions of the traditional "trolley dilemmas" of philosophy that have sparked discussion on AV ethics but have limited use to the technical and legal spheres. Then, we explain how to substantially change the premises and features of these dilemmas (while preserving their behavioral diagnostic spirit) in order to lay the foundations for a more practical and relevant framework that tests driving common sense as an integral part of road rules testing.
Full-text available
As AI Systems become increasingly autonomous, they are expected to engage in decision-making processes that have moral implications. In this research we integrate theoretical and empirical lines of thought to address the matters of moral reasoning and moral uncertainty in AI Systems. We reconceptualize the metanormative framework for decision-making under moral uncertainty and we operationalize it through a latent class choice model. The core idea being that moral heterogeneity in society can be codified in terms of a small number of classes with distinct moral preferences and that this codification can be used to express moral uncertainty of an AI. Choice analysis allows for the identification of classes and their moral preferences based on observed choice data. Our reformulation of the metanormative framework is theory-rooted and practical in the sense that it avoids runtime issues in real time applications. To illustrate our approach we conceptualize a society in which AI Systems are in charge of making policy choices. While one of the systems uses a baseline morally certain model, the other uses a morally uncertain model. We highlight cases in which the AI Systems disagree about the policy to be chosen, thus illustrating the need to capture moral uncertainty in AI systems.
Full-text available
Road vehicle travel at a reasonable speed involves some risk, even when using computer-controlled driving with failure-free hardware and perfect sensing. A fully-automated vehicle must continuously decide how to allocate this risk without a human driver’s oversight. These are ethical decisions, particularly in instances where an automated vehicle cannot avoid crashing. In this chapter, I introduce the concept of moral behavior for an automated vehicle, argue the need for research in this area through responses to anticipated critiques, and discuss relevant applications from machine ethics and moral modeling research.
Full-text available
This chapter reports on a series of studies on driver behavior with a highly automated vehicle, conducted as part of the European project CityMobil and the UK project EASY. Using the University of Leeds Driving Simulator, a number of urban and highway scenarios were devised, where lateral and longitudinal control of the vehicle was managed by an automated controller. Drivers’ uptake of non-driving related tasks, their response to critical events, and their ability to resume control of driving, were some of the factors studied. Results showed some differences in performance based on the road environment studied, and suggest that whilst resuming control from automation was manageable when attention was dedicated to the road, diversion of attention by secondary tasks impaired performance when manual control resumed.
Full-text available
If motor vehicles are to be truly autonomous and able to operate responsibly on our roads, they will need to replicate – or do better than – the human decision-making process. But some decisions are more than just a mechanical application of traffic laws and plotting a safe path. They seem to require a sense of ethics, and this is a notoriously difficult capability to reduce into algorithms for a computer to follow.
Full-text available
Automated vehicles have received much attention recently, particularly the Defense Advanced Research Projects Agency Urban Challenge vehicles, Google's self-driving cars, and various others from auto manufacturers. These vehicles have the potential to reduce crashes and improve roadway efficiency significantly by automating the responsibilities of the driver. Still, automated vehicles are expected to crash occasionally, even when all sensors, vehicle control components, and algorithms function perfectly. If a human driver is unable to take control in time, a computer will be responsible for precrash behavior. Unlike other automated vehicles, such as aircraft, in which every collision is catastrophic, and unlike guided track systems, which can avoid collisions only in one dimension, automated roadway vehicles can predict various crash trajectory alternatives and select a path with the lowest damage or likelihood of collision. In some situations, the preferred path may be ambiguous. The study reported here investigated automated vehicle crashing and concluded the following: (a) automated vehicles would almost certainly crash, (b) an automated vehicle's decisions that preceded certain crashes had a moral component, and (c) there was no obvious way to encode complex human morals effectively in software. The paper presents a three-phase approach to develop ethical crashing algorithms; the approach consists of a rational approach, an artificial intelligence approach, and a natural language requirement. The phases are theoretical and should be implemented as the technology becomes available.
For the past hundred years, innovation within the automotive sector has created safer, cleaner, and more affordable vehicles, but progress has been incremental. The industry now appears close to substantial change, engendered by autonomous, or "self-driving," vehicle technologies. This technology offers the possibility of significant benefits to social welfare — saving lives; reducing crashes, congestion, fuel consumption, and pollution; increasing mobility for the disabled; and ultimately improving land use. This report is intended as a guide for state and federal policymakers on the many issues that this technology raises. After surveying the advantages and disadvantages of the technology, RAND researchers determined that the benefits of the technology likely outweigh the disadvantages. However, many of the benefits will accrue to parties other than the technology's purchasers. These positive externalities may justify some form of subsidy. The report also explores policy issues, communications, regulation and standards, and liability issues raised by the technology; and concludes with some tentative guidance for policymakers, guided largely by the principle that the technology should be allowed and perhaps encouraged when it is superior to an average human driver.
The act of driving always carries some level of risk. With the introduction of vehicle automation, it is probable that computer-driven vehicles will assess this changing level of risk while driving, and make decisions as to the allowable risk for itself and other road users. In certain situations, an automated vehicle may be forced to select whether to expose itself and its passengers to a small risk in order to protect other road users from an equal or greater amount of cumulative risk. In legal literature, this is known as the duty to act. The moral and legal responsibilities of an automated vehicle to act on the behalf of other road users are explored.
A runaway train is racing toward five men who are tied to the track. Unless the train is stopped, it will inevitably kill all five men. You are standing on a footbridge looking down on the unfolding disaster. However, a fat man, a stranger, is standing next to you: if you push him off the bridge, he will topple onto the line and, although he will die, his chunky body will stop the train, saving five lives. Would you kill the fat man?. The question may seem bizarre. But it's one variation of a puzzle that has baffled moral philosophers for almost half a century and that more recently has come to preoccupy neuroscientists, psychologists, and other thinkers as well. In this book, David Edmonds, coauthor of the best-selling Wittgenstein's Poker, tells the riveting story of why and how philosophers have struggled with this ethical dilemma, sometimes called the trolley problem. In the process, he provides an entertaining and informative tour through the history of moral philosophy. Most people feel it's wrong to kill the fat man. But why? After all, in taking one life you could save five. As Edmonds shows, answering the question is far more complex--and important--than it first appears. In fact, how we answer it tells us a great deal about right and wrong.
In October 2010 Google unveiled that it had developed and successfully road-tested the world's first truly autonomous car; moreover, that car had already logged over 140,000 autonomous miles. Media outlets around the country were abuzz with the news. Some championed it as a giant leap forward for safety, and others their worst nightmare realized. In recent years automatic technologies have been increasingly incorporated into motor vehicles, and estimates suggest that the Google car could be available within the next eight years. While legal scholars once wrote extensively about intelligent vehicle highway systems (‚IVHS‛) in the 1990s, the IVHS concept has been all but abandoned. To date, legal scholars have made no attempts to assess the potential liabilities that would follow from autonomous vehicle implementation. While manufacturers have been historically reluctant to incorporate safety technologies because of liability concerns, they have ultimately benefitted from implementation. Products liability law is capable of handling the advent of autonomous vehicles just as it handled seat belts, air bags, and cruise control. And, despite the catastrophization of critics, increased manufacturer liability will not be a dire concern. Autonomous vehicles will make driving safer, leading to a net decrease in manufacturer liability and the cost of insurance and litigation. Additionally, because modern consumers demand safety, economic pressures will necessitate manufacturer adoption and enable loss spreading.