ArticlePDF Available
Case of the Killer Robot
Author(s): Richard G. Epstein
Richard G. Epstein, Westchester University of Pennsylvania
Mike Melamed, CWRU 2000
The Case of the Killer Robot is a detailed scenario that combines elements of software
engineering and computer ethics.
The scenario consists of fictitious articles that touch on specific issues in software engineering
and computer ethics. The articles discuss programs such as programmer psychology, team
dynamics, user interfaces, software process models, software testing, the nature of requirements,
software theft, and privacy. A major consideration is "when is the software good enough?"
The articles in the scenario begin with the indictment for manslaughter of a programmer who
wrote faulty code that caused the death of a robot operator. Slowly, over the course of many
articles, students are introduced to factors within the software company that also contributed to
the accident. They are shown software development as a social process. It is hoped that students
will begin to realize the complexity of the task of building real-world software and to see some
of the ethical issues intertwined in that complexity.
Introduction and Cast of Characters
Author(s): Richard G. Epstein
The Case of the Killer Robot
The Case of the Killer Robot consists of seven newspaper articles, one journal article and one
magazine interview. This scenario is intended to raise issues of computer ethics and software
engineering.
The people and institutions involved in this scenario are entirely fictitious (except for references
to Carnegie Mellon and Purdue universities and to the venerable computer scientists Ben
Shneiderman and Jim Foley). Silicon Valley was chosen as the location for the accident because
it is an icon of high technology. All of the persons and institutions named in Silicon Valley are
purely fictitious.
The Cast of Characters
Alex Allendale
Attorney, hired to defend Randy Samuels.
Jan Anderson
Former programmer and analyst at Silicon Techtronics. She opposed the use of the waterfall
model on the robot project and was fired for her honesty.
Turina Babbage
President of the Association for Computing Machinery (ACM). She announces an investigation
by the ACM into violations of the ACM Code of Ethics by employees at Silicon Techtronics.
Robert Franklin
Reporter for the Silicon Valley Sentinel-Observer. He interviewed Professor Harry Yoder in order
to see how an ethicist would view the developments in the killer robot case. The interview was
published in the Sentinel-Observer's Sunday magazine.
Horace Gritty
Professor of computer science and related concerns at Silicon Valley University. He sees poor
interface design as a primary cause of the killer robot tragedy.
Sandra Henderson
Graduate student at Silicon Valley University. She assisted in an investigation into quality
assurance procedures at Silicon Techtronics.
Ray Johnson
Robotics division chief at Silicon Techtronics. He was driven by the division's need for a
successful robot.
Martha
Anonymous newspaper source. She is the insider at Silicon Techtronics who gave the Silicon
Valley Sentinel-Observer information about the group dynamics on the Robbie CX30 robot
project.
Bart Matthews
Robot operator. A faulty computer program caused a Robbie CX30 robot to strike him dead.
Roberta Matthews
Widow of Bart Matthews.
Jane McMurdock
Prosecuting attorney for the city of Silicon Valley. She brought the manslaughter charges against
Randy Samuels.
Mabel Muckraker
Reporter for the Silicon Valley Sentinel-Observer. She was put on the killer robot story because
of her reputation as an effective investigative reporter.
Bill Park
Professor of physics at Silicon Valley University. He confirmed that Randy Samuels
misinterpreted the robot dynamics equations.
Randy Samuels
Programmer. He wrote the program code that caused the Robbie CX30 robot to oscillate wildly,
killing the robot operator, Bart Matthews.
Sam Reynolds
CX30 Project Manager. Although background was in data processing, he was put in charge of the
Robbie CX30 project, much to Ray Johnson's chagrin. He was committed to the waterfall model
of software development.
Robbie CX30
The robot. Robbie never had an unkind thought about anyone, yet he turned into a savage killer.
Wesley Silber
Professor of software engineering at Silicon Valley University. He conducted a review of
software quality assurance procedures at Silicon Techtronics.
Sharon Skinner
Professor of software psychology at Silicon Valley University. She saw Randy Samuels as a task-
oriented person who was overly sensitive to criticism.
Valerie Thomas
Attorney, hired by Sam Reynolds
Michael Waterson
President and CEO of Silicon Techtronics. He placed Sam Reynolds in charge of the Robbie CX30
project as a cost-saving measure. He contributed generously to Jane McMurdock's re-election
campaign, and hired Dr. Wesley Silber to conduct an investigation into software quality
assurance at Silicon Techtronics.
Max Worthington
Chief security officer for Silicon Techtronics. He monitored electronic mail communications
among the employees and thus exposed Cindy Yardley.
Ruth Witherspoon
Programmer analyst and spokesperson for the Justice for Randy Samuels committee. She
defends Randy Samuels on the grounds that Silicon Techtronics was legally obligated to deliver a
safe robot.
Cindy Yardley
Silicon Techtronics employee and software tester. She admitted to faking software tests in order
to save the jobs of her co-workers.
Harry Yoder
Samuel Southerland Professor of Computer Technology and Ethics at Silicon Valley University.
He examines the tension between individual and corporate responsibilities in an interview
published by the Sentinel-Observer's Sunday magazine.
Silicon Valley Programmer Indicted For
Manslaughter: Program Error Caused Death
by Robot
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
Jane McMurdock, prosecuting attorney for the city of Silicon Valley, announced today the
indictment of Randy Samuels on charges of manslaughter. Samuels was formerly employed as a
programmer at Silicon Techtronics Inc., one of Silicon Valley's newest entries into the high-tech
arena. The charge involves the death of Bart Matthews, who was killed last May by an assembly-
line robot.
Matthews worked as a robot operator at Cybernetics Inc., in Silicon Heights. He was crushed to
death when the robot he was operating malfunctioned and started to wave its "arm" violently.
The robot arm struck Matthews, throwing him against a wall and crushing his skull. Matthews
died almost instantly. The case has shocked and angered many in Silicon Valley. According to
the indictment, Samuels wrote the particular piece of computer program responsible for the robot
malfunction.
"There's a smoking gun!" McMurdock announced triumphantly at a press conference held in the
Hall of Justice. "We have the handwritten formula, provided by the project physicist, which
Samuels was supposed to program. But, he negligently misinterpreted the formula, leading to
this gruesome death. Society must protect itself against programmers who make careless
mistakes or else no one will be safe, least of all our families and our children."
The Sentinel-Observer has obtained a copy of the handwritten formula in question. There are
actually three similar formulas, scrawled on a piece of yellow legal pad paper. Each formula
describes the motion of the robot arm in one direction: east-west, north-south and up-down. The
Sentinel-Observer showed the formulas to Bill Park, a professor of physics at Silicon Valley
University. He confirmed that these equations could be used to describe the motion of a robot
arm. The Sentinel-Observer then showed Park the program code, written by the accused in the C
programming language. We asked Park, who is fluent in C and several other languages, whether
the program code was correct for the given robot-arm formulas.
Park's response was immediate. He exclaimed, "By Jove! It looks like he misinterpreted the y-
dots in the formulas as y-bars, and he made the same mistake for the x's and the z's. He was
supposed to use the derivatives, but he took the averages instead. He's guilty as hell, if you ask
me."
The Sentinel-Observer was unable to contact Samuels for comment. "He is deeply depressed
about all this," his live-in girlfriend told us over the phone, "but Randy believes he will be
acquitted when he gets a chance to tell his side of the story."
Developers of "Killer Robot" Worked Under
Enormous Stress
The Sentinel-Observer learned today that Randy Samuels and others who worked on the "killer
robot" project at Silicon Techtronics Inc. were under tremendous pressure to finish the robot
software by January 1 of this year. According to an informed source, top level management
warned killer robot project staff that "heads would roll" if the January 1 deadline was not met.
Randy Samuels, a Silicon Techtronics programmer, was indicted last week on charges of
manslaughter in the now famous killer robot case. Samuels wrote the flawed software that
caused a Silicon Techtronics Robbie CX30 industrial robot to crush and fatally injure its
operator, Bart Matthews. Matthews was a robot operator at Cybernetics Inc. According to Silicon
Valley Prosecuting Attorney Jane McMurdock, Samuels misinterpreted a mathematical formula,
"turning harmless Robbie into a savage killer."
Our informed source, who wishes to remain anonymous, called "Martha" for the rest of this
article, has intimate knowledge of all aspects of the Robbie CX30 project. In an exclusive
interview, Martha told the Sentinel-Observer that there was an enormous amount of friction
between robotics division chief Ray Johnson and the Robbie CX30 project manager Sam
Reynolds.
"They hated each other's guts," Martha said. "By June of last year the robot project had fallen six
months behind schedule, and Johnson went through the roof. There were rumors that the entire
robotics division, which he headed, would be terminated if Robbie (the CX30 robot) didn't prove
a commercial success. He called Sam (Reynolds) into his office, and he really chewed Sam out. I
mean, you could hear the yelling all the way down the hall. Johnson told Sam to finish Robbie by
the first of January or heads would roll."
"I'm not saying that Johnson was ordering Sam to cut corners," Martha added. "I think the idea of
cutting corners was implicit. The message was, cut corners if you want to keep your job."
According to documents provided by Martha, twenty new programmers were added to the
Robbie CX30 project on June 12 of last year. This was just several days after the stormy meeting
between Johnson and Reynolds.
Martha reported the new hires were a disaster: "Johnson unilaterally arranged for these new
hires, presumably by shifting resources from other aspects of the Robbie project. Reynolds was
vehemently opposed to this. Johnson only knew about manufacturing hardware. That was his
background. He couldn't understand the difficulties that we were having with the robotics
software. You can't speed up a software project by adding more people. It's not like an assembly
line."
According to Martha and other sources inside the project, the hiring of the twenty new
programmers led to a staff meeting attended by Johnson, Reynolds, and all members of the
Robbie CX30 software project. At this meeting, it was Reynolds who was upset. He complained
that the project did not need more people, and he argued that the main problem was that Johnson
and others in Silicon Techtronics management did not understand that the Robbie CX30 was
fundamentally different from earlier versions of the robot. These sources told the Sentinel-
Observer that the new programmers were not fully integrated into the project even six months
later, when ten Robbie CX30 robots, including the robot that killed Bart Matthews, were shipped
out.
Martha explained, "Sam just wanted to keep things as simple as possible. He didn't want the new
people to complicate matters. They spent six months reading manuals. Most of the new hirees
didn't know diddly about robots and Sam wasn't about to waste his time trying to teach them."
Martha said the June 12 meeting has become famous in Silicon Techtronics corporate lore
because it was at that meeting that Ray Johnson announced his "Ivory Snow Theory" of software
design and development. She recounted, "Ray gave us a big multimedia presentation, with slides
and everything. The gist of his Ivory Snow Theory is simply that Ivory Snow is 99 and 44/100
percent pure and there was no reason why robotics software had to be any purer than that. He
stated repeatedly that 'Perfect software is an oxymoron.'"
Martha and the other insiders who came forward with information consistently portrayed
Johnson as a manager in desperate need of a successful project. Earlier versions of Robbie, the
CX10 and the CX20, were experimental in nature, and no one had expected them to be
commercial successes. In fact, the robotics division of Silicon Techtronics has operated heavily
in the red since its inception six years ago. If the CX30 did not succeed, Silicon Techtronics was
going to drop out of the industrial robotics business altogether.
"The earlier Robbie robots got a lot of press, especially here in Silicon Valley," said another
source, who also wishes to remain anonymous. "Robbie CX30 was going to capitalize on the
good publicity generated by the earlier projects. The only thing was that Robbie CX30 was more
revolutionary than Johnson wanted to admit. CX30 represented a gigantic step forward in terms
of sophistication. There were a lot of questions about the industrial settings that the CX30 would
be working in. Much of what we had to do was entirely new, but Johnson couldn't bring himself
to understand that. He just saw us as unyielding perfectionists. One of his favorite quotes was
'Perfection is the enemy of the good.'"
"Killer Robot" Programmer Was Prima
Donna, Co-Workers Claim
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
Randy Samuels, the former Silicon Techtronics programmer who was indicted for writing the
software responsible for the gruesome "killer robot" incident last May, was apparently a prima
donna who found it very difficult to accept criticism, several of his co-workers claimed today.
In a freewheeling interview with several of Samuels' co-workers on the Robbie CX30 robot
project, the Sentinel-Observer was able to gain important insights into the psyche of the man
who may have been criminally responsible for the death of Bart Matthews, a robot operator and
father of three small children.
With the permission of those interviewed, the Sentinel-Observer allowed Professor Sharon
Skinner of the department of software psychology at Silicon Valley University to listen to a
recording of the interview. Skinner studies the psychology of programmers and other
psychological factors that have an impact on software development.
Skinner said, "I would agree with the woman who called him a prima donna. This is a term used
to refer to a programmer who just cannot accept criticism, or more accurately, cannot accept his
or her own fallibility."
"Randy Samuels has what we software psychologists call a task-oriented personality, bordering
on self-oriented. He likes to get things done, but his ego is heavily involved in his work. In the
programming world this is considered a no-no," Skinner added, sitting in her book-lined office.
Skinner went on to explain more about programming teams and programmer personalities:
"Basically, we have found that a good programming team requires a mixture of personality
types, including a person who is interaction-oriented, who derives a lot of satisfaction from
working with other people, someone who can help keep the peace and keep things moving in a
positive direction. Most programmers are task-oriented, and this can be a problem if one has a
team in which everyone is task-oriented."
Samuels' co-workers were very reluctant to lay blame for the robot disaster at his feet, but
several important facts emerged when the co-workers were pressed to comment on Samuels'
personality and work habits. Samuels worked on a team that consisted of a dozen analysts,
programmers and software testers. (This does not include twenty programmers who were later
hired and who never became actively involved in the development of the robotics software.)
Although individual team members had definite specialties, almost all were involved in the entire
software process from beginning to end.
Referring to the manager of the Robbie CX30 project, one of the team members said, "Sam
Reynolds has a background in data processing. He's managed several software projects of that
nature, but his role in this project was mostly managerial. He attended all important meetings and
he kept Ray (Ray Johnson, the robotics division chief) off our backs as much as possible."
It was reported in yesterday's Sentinel-Observer that Sam Reynolds was under severe pressure to
deliver a working Robbie CX30 robot by January 1 of this year. Reynolds could not be reached
for comment about his own role in the incident or about Samuels and his work habits.
"We were a democratic team, except for the managerial guidance provided by Sam," another
team member observed. In the world of software development, a democratic team is a team in
which all members have an equal say in the decision-making process. The team member
continued, "Unfortunately, we were a team of very ambitious, very talented -- if I must say so
myself -- and very opinionated individualists. Randy (Samuels) was just the worst of the lot. I
mean we have two guys and one gal with masters degrees from C.M.U. who weren't as arrogant
as Randy." C.M.U. refers to Carnegie Mellon University, a national leader in software
engineering education.
One co-worker told of an incident in which Samuels stormed out of a quality-assurance meeting.
This meeting involved Samuels and three "readers" of a software module he had designed and
implemented. Such a meeting is called a code review. One of the readers mentioned that Samuels
had used a very inefficient algorithm (program) for achieving a certain result and Samuels
"turned beet red." He then yelled a stream of obscenities before he left the meeting and never
returned.
"We sent him a memo about the faster algorithm, and he eventually did use the more efficient
algorithm in his module," the co-worker added.
The software module in the quality-assurance incident was the very one found to be at fault in
the death of robot operator Bart Matthews. However, this co-worker was quick to point out that
the efficiency of the algorithm was not an issue in the malfunctioning of the robot:
"It's just that Randy made it very difficult for people to communicate their concerns to him. He
took everything very personally. He graduated tops in his class at college and later graduated
with honors in software engineering from Purdue. He's definitely very bright."
"Randy had this big computer-generated banner on his wall," this co-worker continued. "It said,
'You give me the specification and I'll give you the computation.' That's the kind of arrogance he
had, and it also shows that he had little patience for developing and checking the specifications.
He loved the problem-solving aspect, the programming itself."
"It doesn't seem that Randy Samuels caught on to the spirit of egoless programming," Professor
Skinner observed upon hearing this part of the interview with Samuels' co-workers. "The idea of
egoless programming is that a software product belongs to the team and not to the individual
programmers. The idea is to be open to criticism and to be less attached to one's work. Code
reviews are certainly consistent with this overall philosophy."
A female co-worker spoke of another aspect of Samuels' personality -- his helpfulness. She said,
"Randy hated meetings, but he was pretty good one on one. He was always eager to help. I
remember one time when I ran into a serious roadblock and instead of just pointing me in the
right direction, he took over the problem and solved it himself. He spent nearly five entire days
on my problem."
"Of course, in retrospect, it might have been better for poor Mr. Matthews and his family if
Randy had stuck to his own business," she added after a long pause.
"Killer Robot" Project Mired in Controversy
Right from Start
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
Sources close to the development of Silicon Techtronics' Robbie CX30, the so-called killer robot
that caused the death of an assembly line worker last May, have told the Sentinel-Oberver that
the project was controversial from the beginning. Two groups, committed to different software
development philosophies, nearly came to blows during the initial planning meetings for the
Robbie CX30, nearly two years ago. At issue was whether the Robbie CX30 project should
proceed according to the "waterfall model" or the "prototyping model" of software development.
The waterfall and prototyping models are two common methods for organizing a software
project. In the waterfall model, the software goes through definite stages of development. In the
first stage, called requirements analysis and specification, an attempt is made to agree upon the
detailed functionality of the system. As the project passes from one stage to the next, there are
limited opportunities to change earlier decisions. A drawback of this approach is that potential
users are not able to interact with the system until very late in the process.
In the prototyping model, great emphasis is placed on producing a working model or prototype
early in the process. The prototype is built to help arrive at a final specification of the
functionality of the proposed system. Potential users interact with the prototype early and often
until the requirements are agreed upon. This approach allows potential users to interact with the
prototype system long before the final system is designed and coded.
In a memo dated December 12 of the year before last, Jan Anderson, a member of the original
Robbie CX30 project team, bitterly attacked project manager Sam Reynolds' decision to employ
the waterfall model. The Sentinel-Observer has obtained a copy of Anderson's memo, which is
addressed to Reynolds, and Anderson verified the authenticity of the memo for this reporter.
Reynolds fired Anderson on December 24 of the same year, two weeks after she wrote the
memo.
The Anderson memo refers to a meeting on December 11, during which an angry exchange
occurred relating to software development philosophy. Anderson underlined the following
passage in her memo: "I did not intend to impugn your competence at our meeting yesterday, but
I must protest most vehemently against the idea that we complete the Robbie CX30 software
following the waterfall model, which you have used in previous projects. I need not remind you
that those were data processing projects involving the processing of business transactions. The
Robbie CX30 project will involve a high degree of interaction, both between robot components
and between the robot and the operator. Since operator interaction with the robot is so important,
the interface cannot be designed as an afterthought."
Randy Samuels, the programmer who has been charged with manslaughter in the death of robot
operator Bart Matthews, was in attendance at the December 11 meeting. In a conversation in the
living room of her suburban townhouse, Anderson said that Samuels had not had much to say
about the waterfall-prototyping controversy, and that she would give her "eye teeth" to have
Samuels exonerated.
"The project was doomed long before Samuels misinterpreted those formulas," Anderson said
emphatically.
Anderson did her best to explain the waterfall-prototyping controversy in lay terms: "The main
issue was really whether we could agree on the system requirements without allowing actual
robot operators to get a feel for what we had in mind. Reynolds has been in the data processing
business for three decades and he's good at that, but he never should have been made manager of
this project."
According to records obtained by the Sentinel-Observer, Silicon Techtronics moved Sam
Reynolds from the data processing division, which takes care of inventory and payroll, to the
robotics division just three weeks before the December 11 meeting referred to in Anderson's
memo.
Reynolds was transferred to the robotics division by Silicon Techtronics president Michael
Waterson. Reynolds replaced John Cramer, who managed the earlier Robbie projects, CX10 and
CX20. Cramer was placed in charge of the CX30 project, but he died unexpectedly in a sky-
diving accident. By placing Reynolds in charge of the CX30 project, Waterson was going against
the advice of the robotics division chief Ray Johnson, sources say. According to these sources,
Johnson strongly opposed the choice of Reynolds to head the Robbie CX30 project. These
sources tell the Sentinel-Observer that Waterson's choice of Reynolds was purely a cost-saving
decision. It was cheaper to move Reynolds to the robotics division than to hire a new project
leader from outside the corporation.
The anonymous source that the Sentinel-Observer calls "Martha" described the situation in this
way: "Waterson thought it would be cheaper to move Reynolds to robotics rather than try to find
a new manager for the Robbie project from outside. Also, Waterson tended to be suspicious of
people from the outside. He often sends down memos about how long it takes people to master
the Silicon Techtronics way of doing things. In Waterson's view, Reynolds was a manager and
he was moved to his new position in robotics as a manager and not as a technical expert. Clearly,
Reynolds saw himself as both a manager and as a technical expert. Reynolds was not aware of
his own technical limitations."
According to Martha, Reynolds was very reluctant to manage a project that would not use the
waterfall model, which had served him so well in data processing: "He attacked prototyping as a
fad at the meeting on December 11, and after a few verbal exchanges back and forth things got
pretty personal."
"Anderson was especially vocal," Martha recalled. "She had lots of experience with user
interfaces, and from her perspective the operator-robot interface was critical to the success of
CX30 since operator intervention would be frequent and at times critical."
In her interview with the Sentinel-Observer, Jan Anderson also commented on this aspect of the
December 11 meeting: "Reynolds was vehemently opposed to wasting time -- to use his words --
on any kind of formal analysis of the user interface and its human factors properties. To him,
user interfaces were a peripheral issue."
"Anything new was a fad to him," Anderson added. "Computer interfaces were a fad, object-
oriented design was a fad, formal specification and verification techniques were a fad, and most
of all, prototyping was a fad."
Exactly one week after the December 11 meeting, the Robbie team received a memo from Sam
Reynolds concerning the project plan for the Robbie CX30. "It was the waterfall model, right out
of a textbook," Anderson said as she reviewed a copy of the project plan memo. "Requirements
analysis and specification, then architectural design and detailed design, coding, testing, delivery
and maintenance. In Reynolds' view of things, there was no need to have any user interaction
with the system until very, very late in the process."
The Sentinel-Observer has learned that the first operator actually to use the Robbie CX30 robot
in an industrial setting was Bart Matthews. This initial use of the Robbie CX30 in an industrial
setting was covered by the media, including this newspaper. In a great irony, the Silicon
Techtronics Annual Report for Shareholders, published last March, has a picture of a smiling
Bart Matthews on its glossy front cover. Matthews is shown operating the very same Robbie
CX30 robot that crushed him to death barely two months after the photograph was taken.
Silicon Techtronics Promised To Deliver a
Safe Robot
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
At a news conference this afternoon, a ragtag group of programmers who call themselves the
Justice for Randy Samuels Committee distributed documents that show Silicon Techtronics
obligated itself to deliver robots that would "cause no bodily injury to the human operator."
Randy Samuels is the programmer charged with manslaughter in the infamous "killer robot"
case.
"We cannot understand how the prosecuting attorney could charge Randy with manslaughter
when, in fact, Silicon Techtronics was legally bound to deliver a safe robot to Cybernetics," said
committee spokesperson Ruth Witherspoon. "We believe that there is a cover-up going on and
that there is some kind of collusion between SiliTech (Silicon Techtronics) management and the
prosecuting attorney's office. Michael Waterson was a major contributor to Ms. McMurdock's re-
election campaign last year." Michael Waterson is president and CEO of Silicon Techtronics.
Jane McMurdock is the prosecuting attorney for the city of Silicon Valley. The Sentinel-
Observer has confirmed that Waterson made several large contributions to the McMurdock re-
election campaign last fall.
"Randy is being made the scapegoat for a company which had lax quality-control standards, and
we are not going to stand for it!" Witherspoon shouted in an emotional statement to reporters.
"We believe that politics has entered this case."
The documents distributed by the Justice for Randy Samuels Committee were portions of what is
called a "requirements document." According to Witherspoon and other committee members,
this document proves that Samuels was not legally responsible for the death of Bart Matthews,
the unfortunate robot operator who was killed by a Silicon Techtronics robot at Cybernetics Inc.,
in Silicon Heights, last May. The requirements document amounts to a contract between Silicon
Techtronics and Cybernetics Inc. It spells out in complete detail the functionality of the Robbie
CX30 robot Silicon Techtronics promised to deliver to Cybernetics.
According to Witherspoon, the Robbie CX30 robot was designed to be an "intelligent" robot
capable of operating in a variety of industrial settings. Separate requirements documents were
necessary for each corporate customer because the Robbie CX30 was not an "off-the-shelf" robot
but a robot that needed to be programmed differently for each application. However, all
requirements documents for the Robbie CX30 project, including the agreement between Silicon
Techtronics and Cybernetics, contain the following important statements:
"The robot will be safe to operate and even under exceptional conditions (see Section 5.2) the
robot will cause no bodily injury to the human operator . . . . In the event of the exceptional
conditions which potentially contain the risk of bodily injury (see Section 5.2.4 and all of its
subsections), the human operator will be able to enter a sequence of command codes, as
described in the relevant sections of the functional specification (see Section 3.5.2), which will
arrest robot motion long before bodily injury can actually occur."
Exceptional conditions include unusual events such as bizarre data from the robot sensors, erratic
or violent robot motion or operator error. It was exactly such an exceptional condition that led to
the death of Bart Matthews. These passages were extracted from the portion of the requirements
document that deals with "non-functional requirements." The non-functional requirements
present in complete detail the constraints under which the robot will operate. For example, the
requirement that the robot be incapable of harming its human operator is one such constraint, and
Silicon Techtronics, according to Witherspoon, was legally obligated to satisfy this constraint.
Elsewhere, the "functional requirements" portion of the requirements document covers, again in
complete detail, the behavior of the robot and its interaction with its environment and its human
operator. In particular, the functional requirements specified the behavior of the robot under each
and every anticipated exceptional condition. Exceptional conditions that require operator
intervention cause an error message to be generated at the operator console. In her statement to
reporters, Witherspoon explained that Bart Matthews was killed when exceptional condition
5.2.4.26 arose. This involved an exceptionally violent and unpredictable robot arm motion. This
is a condition that requires operator intervention, namely the entering of the command codes
mentioned in the document, but apparently Bart Matthews became confused and could not enter
the codes successfully. Silicon Valley Police confirm that when Bart Matthews was killed, the
reference manual at his console was opened to the page of the index which contained entries for
"errors."
"Although Randy Samuels' program was in error -- he did misinterpret the robot dynamics
formulas, as reported in the media -- exceptional condition 5.2.4.26 was designed to protect
against just this sort of contingency," Witherspoon told reporters. "The robot motion values
generated by Randy's program correctly set off this exceptional condition, and the robot operator
received due warning that something was wrong."
Witherspoon claimed that she has a signed affidavit from another Cybernetics robot operator to
the effect that the training sessions offered by Silicon Techtronics never mentioned this nor many
other exceptional conditions . According to Witherspoon, the robot operator has sworn that
neither she nor any other robot operator was ever told that the robot arm could oscillate violently.
Witherspoon quoted the affidavit at the news conference: "Neither I nor Bart Matthews was ever
trained to handle this sort of exceptional condition. I doubt that Bart Matthews had any idea what
he was supposed to do when the computer screen started flashing the error message on the
screen".
Witherspoon then quoted sections of the requirements document that obligated Silicon
Techtronics, referred to as "the vendor," to adequately train robot operators: "The vendor shall
provide forty (40) hours of operator training. This training shall cover all aspects of robot
operation, including exhaustive coverage of the safety procedures which must be followed in the
case of exceptional conditions which potentially contain the risk of bodily injury . . . . The
vendor shall provide and administer appropriate test instruments which shall be used to certify
sufficient operator understanding of robot console operations and safety procedures. Only
employees of the customer who have passed this test shall be allowed to operate the Robbie
CX30 robot in an actual industrial setting . . . . The reference manual shall provide clear
instructions for operator intervention in all exceptional situations, especially and including those
which potentially contain the risk of bodily injury."
According to Witherspoon, sworn affidavits from several robot operators at Cybernetics state
that only one work day of approximately eight hours was spent in operator training. Furthermore,
almost no time was spent discussing potentially dangerous exceptional conditions.
"The written test developed by Silicon Techtronics to certify a robot operator was considered a
joke by Cybernetics employees," Witherspoon asserted. "Silicon Techtronics obviously did not
give much thought to the training and testing procedures mandated by the requirements
document, according to the evidence in our possession."
The Killer Robot Interface
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
Abstract:
The Robbie CX30 industrial robot was supposed to set a new standard for industrial robot
intelligence. Unfortunately, one of the first Robbie CX30 robots killed an assembly-line worker,
leading to the indictment of one of the robot's software developers, Randy Samuels. This paper
propounds the theory that it is the operator-robot interface designer who should be on trial in this
case. The Robbie CX30 robot violates nearly every rule of interface design. This paper focuses
on how the Robbie CX30 interface violated every one of Shneiderman's eight golden rules.
1. Introduction
On May 17, 1992 a Silicon Techtronics Robbie CX30 industrial robot killed its operator, Bart
Matthews, at Cybernetics, Inc., in Silicon Heights, a suburb of Silicon Valley. An investigation
into the cause of the accident led authorities to the conclusion that a software module, written
and developed by Randy Samuels, a Silicon Techtronics programmer, was responsible for the
erratic and violent robot behavior which in turn lead to the death by decapitation of Bart
Matthews.1
As an expert in the area of user interfaces,2, 3, 4 I was asked to help police reconstruct the
accident. In order to accomplish this, Silicon Techtronics was asked to provide me with a Robbie
CX30 simulator which included the complete robot operator console. This allowed me to
investigate the robot's behavior without actually risking serious harm. Due to my extensive
understanding of user interfaces and human factors I was able to reconstruct the accident with
uncanny accuracy. On the basis of this reconstruction, I came to the conclusion that it was the
interface design and not the admittedly flawed software which should be viewed as the culprit in
this case.
Despite my finding, Prosecuting Attorney Jane McMurdock insisted on pursuing the case against
Randy Samuels. I believe that any competent computer scientist, given an opportunity to interact
with the Robbie CX30 simulator, would also conclude that the interface designer and not the
programmer should be charged with negligence, if not manslaughter.
2. Shneiderman's eight golden rules
My evaluation of the Robbie CX30 user interface is based upon Shneiderman's 'eight golden
rules' 5. I also used other techniques to evaluate the interface, but those will be published in
separate papers. In this section, I offer a brief review of Shneiderman's eight golden rules, a
subject which would be more familiar to computer interface experts such as myself as opposed to
the robot hackers who read this obscure journal.
The eight golden rules are:
1. Strive for consistency.
2. Enable frequent users to use shortcuts.
3. Offer informative feedback.
4. Design dialogues to yield closure.
5. Offer simple error handling.
6. Permit easy reversal of actions.
7. Support internal locus of control.
8. Reduce short-term memory load.
Strive for consistency.
As we shall see below, it is important for a user interface to be consistent on many levels. For
example, screen layouts should be consistent from one screen to another. In an environment
using a graphical user interface (GUI), this also implies consistency from one application to
another.
Enable frequent users to use shortcuts.
Frequent users (or power users) may be turned off by overly tedious procedures. The interface
should allow those users a less tedious procedure for accomplishing a given task.
Offer informative feedback.
Users need to see the consequences of their actions. It can be confusing and disorienting for the
user if the computer does not show either that it is processing or has processed a command that
the user has entered.
Design dialogues to yield closure
Interacting with a computer is somewhat like a dialogue or conversation. The dialogue for every
task should have a beginning, a middle, and an end, and it is important for the user to know when
a task is at its end. The user needs to have the feeling that a task has reached closure.
Offer simple error handling
User errors should be designed into the system. To state this another way: no user action should
be considered an error beyond the ability of the system to manage. If the user makes a mistake,
he or she should receive useful, concise, and clear information about the nature of the mistake,
and it should be easy for the user to undo the mistake.
Permit easy reversal of actions
More generally, users must be permitted to undo what they have done, whether it is an error or
not.
Support internal locus of control
User satisfaction is high when the user feels that he or she is in control, and user satisfaction is
low when the user feels that the computer is in control. Interfaces should be designed to reinforce
the feeling that the user is the focus of control in the human-computer interaction.
Reduce short-term memory load
Human short-term memory is remarkably limited. Psychologists often quote Miller's law to the
effect that short-term memory is limited to seven discrete pieces of information. The interface
should do everything possible to lessen the user's memory burden. For example, instead of being
asked to type in the name of a file to be retrieved, the user might be presented with a list of files
currently available.
3. Robot console overview
The Robbie CX30 operator interface violated each and every one of Shneiderman's rules. Several
of these violations were directly responsible for the accident that ended in the death of the robot
operator.
The robot console was an IBM PS/2 model 55SX with a 80386 processor and an EGA color
monitor with 640x480 resolution. The console had a keyboard, but no mouse. The console was
embedded in a workstation which included shelves for manuals and an area for taking notes and
for reading manuals. However, the reading/writing area was quite a distance from the computer
screen, so it was quite awkward and tiresome for the operator to manage any task that required
looking something up in the manual and then acting quickly at the console keyboard. The
operator's chair was poorly designed and much too high relative to the console and the
reading/writing area. This placed much strain on the operator's back and also caused excessive
eye strain.
I cannot understand why a sophisticated system such as this would not include a better device for
input. One can only conclude that Silicon Techtronics did not have much experience with user
interface technology. The requirements document6 specified a menu-driven system, which was a
reasonable choice. However, in an application where speed was of the essence, especially when
operator safety was at issue, the use of a keyboard for all menu selection tasks was an extremely
poor choice, requiring many keystrokes to achieve the same effect which could be achieved
almost instantaneously with a mouse. (See the paper by Foley et. al.)7. (Actually, I had most of
these ideas before Foley published them, but he beat me to the punch.)
The robot operator interacted with the robot and made an impact on its behavior by making
choices in a menu system. The main menu consisted of 20 items -- too many, in my opinion --
and each main menu item had a pull-down submenu associated with it. Some of the submenus
contained as many as 20 items -- again, too many. Furthermore, there seemed to be little rhyme
or reason as to why the menu items were listed in the order in which they appeared. A functional
or alphabetical organization would have been better.
In the pull-down submenus, some items had up to four pop-up menus associated with them.
These would appear in sequence as submenu choices were made. Occasionally, a submenu
choice would cause a dialogue box to appear at the screen. A dialogue box requires some kind of
interaction between the operator and the system to resolve some issue, such as the diameter of
the widgets being lowered into the acid bath.
A menu system presents a strict hierarchy of menu choices. On this system, the operator could
backtrack up the hierarchy by pressing the escape key. The escape key could also terminate any
dialogue. In addition, the use of color in the interface was very unprofessional: There were too
many colors in too small a space. The contrasts were glaring and the result, for this reviewer, was
severe eye strain in just fifteen minutes. There was also excessive use of screen flashing and silly
musical effects when erroneous choices or erroneous inputs were made.
One has to wonder why Silicon Techtronics did not attempt a more sophisticated approach to the
interface design. After a careful study of the Robbie CX30 applications domain, I have come to
the conclusion that a direct manipulation interface, which literally displayed the robot at the
operator console, would have been ideal. The very visual domain that the robot operated within
would lend itself naturally to the design of appropriate screen metaphors for that environment,
metaphors the operator could easily understand. This would allow the operator to manipulate the
robot by manipulating the graphical representation of the robot at the computer console. I have
asked one of my doctoral students, Susan Farnsworth, to give up her personal life for the better
part of a decade in order to investigate this possibility a bit further.
4. How the Robbie CX30 interface violated the eight golden rules
The Robbie CX30 user interface violated each and every golden rule in multitudinous ways. I
shall only discuss a few instances of rule violation in this paper, leaving a more detailed
discussion of these violations for future articles and my forthcoming book8. I will emphasize
those violations which were relevant to this particular accident.
4.1 Strive for consistency
There were many violations of consistency in the Robbie CX30 user interface. Error messages
could appear in almost any color and could be accompanied by almost any kind of musical
effect. Error messages could appear almost anywhere at the screen.
When Bart Matthews saw the error message for the exceptional condition that occurred, an
exceptional condition that required operator intervention, it was probably the first time he had
seen that particular message. In addition, the error message appeared in a green box, without any
audio effects. This is the only error message in the entire system which appears in green and
without some kind of orchestral accompaniment.
4.2 Enable frequent users to use shortcuts
This principle did not appear in any way in the entire interface design. For example, it would
have been a good idea to allow frequent users to enter the first letter of a submenu or menu
choice to effect a choice, in lieu of requiring the use of the cursor keys and the enter key. The
menu selection mechanism in this system must have been quite a mental strain on the operator.
Furthermore, a form of type-ahead should have been supported, which would have allowed a
frequent user to enter a sequence of menu choices without having to wait for the actual menus to
appear.
4.3 Offer informative feedback
There are many cases in which a given sequence of keystrokes represents one holistic idea, one
complete task, but the operator is left without the kind of feedback which would confirm that the
task has been completed. For example, there was a fairly complicated dialogue necessary to
remove a widget from the acid bath. However, upon completion of this dialogue, the robot
operator was led into a new, unrelated dialogue, without being informed that the widget removal
dialogue had been completed.
4.4 Design dialogues to yield closure
There are many cases in which a given sequence of keystrokes represents one holistic idea, one
complete task, but the operator is left without the kind of feedback which would confirm that the
task has been completed. For example, there is a fairly complicated dialogue which is necessary
in order to remove a widget from the acid bath. However, upon completion of this dialogue, the
user is led into a new, unrelated dialogue, without being informed that the widget removal
dialogue has been completed.
4.5 Offer simple error handling
The system seems to be designed to make the user regret any erroneous input. Not only did the
system allow numerous opportunities for error, but when an error actually occurred, it was
something that was not likely to be repeated for some time. This is because the user interface
made recovery from an error a tedious, frustrating and at times infuriating ordeal. Some of the
error messages were downright offensive and condescending.
4.6 Permit easy reversal of actions
As mentioned in the previous paragraph, the user interface made it very difficult to recover from
erroneous inputs. In general, the menu system did allow easy reversal of actions, but this
philosophy was not carried through to the design of dialogue boxes or to the handling of
exceptional conditions. From a practical (as opposed to theoretical) point of view, most actions
were irreversible when the system was in an exceptional state, and this helped lead to the "killer
robot" tragedy.
4.7 Support internal locus of control
Many of the deficiencies discussed in the previous paragraphs diminished the feeling of an
"internal locus of control," for example: not providing feedback, not bringing interactions to
closure, and not allowing easy reversal of actions when exceptions arose. All of these things
acted to diminish the user's feeling of being in control of the robot. There were many features of
this interface that could make the user feel as though there was an enormous gap between the
operator console and the robot itself, whereas a good interface design would have made the user
interface transparent and would have given the robot operator a feeling of being in direct contact
with the robot. In one case, I commanded the robot to move a widget from the acid bath to the
drying chamber and it took 20 seconds before the robot seemed to respond. Thus, I did not feel
as though I was controlling the robot. The robot's delayed response, along with the lack of
informative feedback at the computer screen, made me feel that the robot was an autonomous
agent -- an unsettling feeling, to say the least.
4.8 Reduce short-term memory load
A menu driven system is generally good in terms of the memory burden it places upon users.
However, there is great variation among particular implementations of menu systems. The
Robbie CX30 user interface had very large menus without any obvious internal organization.
These placed a great burden upon the operator in terms of memory, and also in terms of scan
time, the time it takes the operator to locate a particular menu choice.
In addition, many dialogue boxes required the user to enter part numbers, file names, and other
information from the keyboard. This greatly increased the memory burden upon the user. The
system could easily have been designed to present the user with part numbers, and so forth, so
the user would not have to recall these things from his or her own memory.
Finally, and this is really unforgivable, incredible as it may seem, there was no on-line, context-
sensitive help facility! Although I was taken through the training course offered by Silicon
Techtronics, I often found myself leafing through the reference manuals in order to find the
answer to even the most basic questions, such as: "What does this menu choice mean? What will
happen if I make this choice?"
5. A reconstruction of the "killer robot" tragedy
Police photographs of the accident scene are not a pleasant sight. The operator console was
splattered with a considerable amount of blood. However, the photographs are of exceptional
quality and using blow-up techniques, I was able to ascertain the following important facts about
the moment when Bart Matthews was decapitated:
1. The NUM LOCK light was on.
2. The IBM keyboard contains a calculator pad that can operate in two modes. When the
NUM LOCK light is on, it behaves like a calculator. Otherwise, the keys can be used to
move the cursor at the screen.
3. Blood was smeared on the calculator pad.
4. Bloody fingerprints indicate that Bart Matthews was using the calculator pad when he
was struck and killed.
5. A green error message was flashing.
6. This tells us the error situation in force when the tragedy occurred. The error message
said, "ROBOT DYNAMICS INTEGRITY ERROR - 45 ".
7. A reference manual was open and laid flat in the workstation reading/writing area.
8. This one volume of the four volume reference manual was open to the index page that
contained the entry "ERRORS/MESSAGES."
9. A message giving operator instructions was also showing on the screen.
10. This message was displayed in yellow at the bottom of the screen. It read "PLEASE
ENTER DYNAMICAL ERROR ROBOT ABORT COMMAND SEQUENCE
PROMPTLY!!!"
On the basis of this physical evidence, plus other evidence contained in the system log, and
based upon the nature of the error that occurred ("robot dynamics integrity error - 45," an error
caused by Randy Samuels' program), I have concluded that the following sequence of events
occurred on the fateful morning of the "killer robot" tragedy:
10:22.30. "ROBOT DYNAMICS INTEGRITY ERROR - 45" appears on the screen. Bart Matthews
does not notice this because there is no beep or audio effect such as occurs with every other
error situation. Also, the error message appears in green, which in all other contexts means that
some process is proceeding normally.
10:24.00. Robot enters state violent enough for Bart Matthews to notice.
10:24.05. Bart Matthews notices the error message, but does not know what it means and does
not know what to do. He tries the "emergency abort" submenu, a general purpose submenu for
turning off the robot. This involves SIX separate menu choices, but Mr. Matthews does not
notice that the NUM LOCK light is lit. The menu choices aren't registering, because the cursor
keys are operating as calculator keys.
10:24.45. Robot turns from acid bath and begins sweep towards the operator console, its jagged
robot arms flailing wildly. No one anticipated when the console was designed that the operator
might have to flee a runaway robot, so Bart Matthews is cornered in his work area by the
advancing robot. At about this time, Bart Matthews retrieves the reference manual and starts
looking for a reference to ³robot dynamics integrity error - 45² in the index. He successfully
locates an entry for error messages in the index.
10:25.00. Robot enters the operator area. Bart Matthews gives up on trying to find the operator
procedure for the robot dynamics integrity error. Instead, he tries once again to enter the
"emergency abort" sequence from the calculator keypad. As he does this, the robot strikes him.
6. Summary and conclusions
While the software module written by Randy Samuels did cause the Robbie CX30 robot to
oscillate out of control and attack its human operator, a good interface design would have
allowed the operator to terminate the erratic robot behavior. Based upon an analysis of the robot
user interface using Shneiderman's eight golden rules, this interface design expert has come to
the conclusion that the interface designer, and not the programmer, was the more guilty party in
this unfortunate fiasco.
7. Footnotes
1.The media were misled to believe that Bart Matthews was crushed by the robot, but the
photographic evidence given to this author shows otherwise. Perhaps authorities were
attempting to protect public sensibilities.
2.Gritty, Horace. "The Only User Interface Book You'll Ever Need." Oshkosh, WI: Vanity Press,
1990. pg 212
3.Gritty, Horace. "What We Can Learn from the Killer Robot." 1992. Invited talk given at the
Silicon Valley University International Symposium on Robot Safety and User Interfaces, March
1991. Also to appear in Silicon Valley University Alumni Notes.
4.Gritty, Horace. "CODEPENDENCY: How Computer Users Enable Poor User Interfaces." New
"York: Angst Press, expected 1993.
5.Shneiderman, Ben. "Designing the User Interface." Reading, Massachusetts: Addison-Wesley,
1987. pg 448 (see footnote 9)
6.Robbie CX30 INTELLIGENT INDUSTRIAL ROBOT REQUIREMENTS DOCUMENT : Cybernetics Inc.
Version, Technical Document Number 91-0023XA, Silicon Techntronics Corporation, Silicon
Valley, USA, pg 1245
7.Foley, J. P., L.V. Wallace, and P. Chan. "The Human Factors of Computer Graphics Interaction
Techniques." IEEE Computer Graphics and Applications, 4(11). 1984. pg 13-48 (see footnote 9)
8.Gritty, Horace (expected 1993). CODEPENDENCY: How Computer Users Enable Poor User
Interfaces, Angst Press, New York. This book presents a radically new theory concerning the
relationship between people and their machines. Essentially, some people need a poor interface
in order to avoid some unresolved psychological problems in their lives.
9. Foley and Shneiderman are actual authors. The other references are fictitious.
Software Engineer Challenges Authenticity of
"Killer Robot" Software Tests
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
The "killer robot" case took a significant turn yesterday when a Silicon Valley University
professor issued a report questioning the authenticity of software tests that were reportedly
performed on the "killer robot" software by Silicon Techtronics. Dr. Wesley Silber, professor of
software engineering, told a packed news conference held at the university that the test results
reflected in Silicon Techtronics internal documents were not consistent with test results obtained
when he and his associates tested the actual robot software.
Silicon Valley is still reacting to Professor Silber's announcement, which could play an important
role in the trial of Randy Samuels, the Silicon Techtronics programmer who has been charged
with manslaughter in the now infamous killer robot incident. Pressed for her reaction to
Professor Silber's report, Prosecuting Attorney Jane McMurdock reiterated her confidence that a
jury will find Randy Samuels guilty. McMurdock shocked reporters, however, when she added,
"But this does raise the possibility of new indictments."
Ruth Witherspoon, spokesperson for the Justice for Randy Samuels Committee, was almost
exultant when she spoke to this reporter. She said, "McMurdock cannot have it both ways. Either
the programmer is responsible for this tragedy, or management must be held responsible. We
believe that the Silber report exonerates our friend and colleague, Randy Samuels."
Silicon Techtronics CEO Michael Waterson issued a terse statement concerning the Silber
report: "Soon after the indictment of Randy Samuels was announced, I personally asked the
esteemed software engineer Dr. Wesley Silber to conduct an impartial inquiry into quality
assurance procedures at Silicon Techtronics. As the chief executive of this corporation, I have
always insisted on quality first, despite what you might have read in the press. I promised
Professor Silber that he would have access to all information relevant to this unfortunate
situation. I told him in a face-to-face meeting in my office that he should pursue his investigation
wherever it might lead, regardless of the implications. It never occurred to me, based upon the
information that I was getting from my managers, that there might be a problem in which
software quality assurance procedures were either lax or deliberately circumvented. I want the
public to be reassured that the person or persons who were responsible for the failure of software
quality assurance within the robotics division of Silicon Techtronics will be asked to find
employment elsewhere."
Roberta Matthews, widow of Bart Matthews, the robot operator who was killed in the incident,
spoke to the Sentinel-Observer by telephone from her home. She said, "I still want to see Mr.
Samuels punished for what he did to my husband. I don't understand what all the commotion is
about. The man who murdered my husband should have tested his own software!"
The Sentinel-Observer interviewed Professor Silber in his office shortly after his news
conference. On his office wall were numerous awards he has received because of his work in the
field of software engineering and software quality assurance. We began the interview by asking
Professor Silber to explain why it is that software is sometimes unreliable. He answered our
question by citing the enormous complexity of software.
"Large computer programs are arguably the most complex artifacts ever fashioned by the human
mind," Professor Silber explained, seated in front of a large computer monitor. "At any point in
time, a computer program is in one of an extremely large number of possible states, and it is a
practical impossibility to assure that the program will behave properly in each of those states. We
do not have enough time to do that kind of exhaustive testing. Thus, we use testing strategies or
heuristics that are very likely to find bugs, if they exist."
Professor Silber has published numerous papers on software engineering. He made headlines last
year when he published his list of "Airlines to Avoid As If Your Life Depended Upon It." That
list named domestic airlines he deemed irresponsible because of their purchase of airplanes that
are almost completely controlled by computer software.
Professor Silber told the Sentinel-Observer about his work at Silicon Techtronics: "Mike
[Waterson] told me to go in there [into the company] and conduct an impartial review of his
software testing procedures and to make my findings public. Mike seemed confident, perhaps
because of what his managers had told him, that I would find nothing wrong with quality
assurance at Silicon Techtronics."
Professor Silber explained that "quality assurance" refers to those methods a software developer
uses to assure that the software is reliable. These methods are applied throughout the
development life-cycle of the software product. For example, when a programmer writes code,
one quality assurance measure is to test the code by actually running it against test data. Another
would be to run special programs, called static analyzers, against the new code. A static analyzer
is a program that looks for suspicious patterns in programs, which might indicate an error or bug.
These two forms of quality assurance are called dynamic testing and static testing, respectively.
Software consists of discrete components or units that are eventually combined to create larger
systems. The units themselves must be tested, and this process of testing individual units is
called unit testing. When the units are combined, the integrated subsystems must be tested and
this process of testing the integrated subsystems is called integration testing.
Soon after arriving at Silicon Techtronics, Professor Silber focused his attention on procedures
for dynamically testing software at the high tech company. Assisted by a cadre of graduate
students, Professor Silber discovered a discrepancy between the actual behavior of the section of
program code (written by Randy Samuels) that caused the Robbie CX30 robot to kill its operator
and the behavior of that code as recorded in test documentation at Silicon Techtronics. This
discovery was actually made by Sandra Henderson, a graduate student in software engineering
who is completing her doctorate under Professor Silber.
We interviewed Henderson in one of the graduate computer laboratories at Silicon Valley
University. "We found a problem with the unit testing," Henderson explained. "Here are the test
results, given to us by Mr. Waterson at Silicon Techtronics, which are purported to be for the C
[programming language] code which Randy Samuels wrote and which caused the killer robot
incident. As you can see, everything is clearly documented and organized. There are two test
suites: one based upon white box testing and another based upon black box testing. Based upon
our own standards for testing software, these test suites are well designed, complete, and
rigorous."
She explained that black box testing involves viewing the software unit (or component) as a
black box that has expected input and output behaviors. Test suites are designed to cover all
"interesting" behaviors that the unit might exhibit but without any knowledge of the structure or
nature of the actual code. If the component demonstrates the expected behaviors for all inputs in
the test suite, then it passes the test. On the other hand, white box testing involves covering all
possible paths through the unit. Thus, white box testing is done with thorough knowledge of the
unit's structure. In white box testing, the test suite must cause each program statement to execute
at least once so that no program statement escapes execution.
Henderson went on to explain the significance of software testing: "Neither black box nor white
box testing proves that a program is correct. However, software testers, such as those employed
at Silicon Techtronics, can become quite skillful at designing test cases so as to discover new
bugs in the software. The proper attitude is that a test succeeds when a bug is found. Basically,
the tester is given a set of specifications and does his or her best to show that the code being
tested does not satisfy its specifications."
Henderson then showed this reporter the test results that she actually obtained when she ran the
critical "killer robot" code using the company's test suites for white box and black box testing. In
many cases, the outputs recorded in the company's test documents were not the same as those
generated by the actual killer robot code that Henderson tested.
During his interview with the Sentinel-Observer yesterday, Professor Silber discussed the
discrepancy: "You see, the software that was actually delivered with the Robbie CX30 robot was
not the same as the software that was supposedly tested -- at least according to these documents!
We have been able to determine that the actual killer code, as we call it, was written after the
software tests were supposedly conducted. This suggests several possibilities: First, the software
testing process, at least for this critical part of the software, was deliberately faked. We all know
that there was enormous pressure to get this robot out the door by a date certain. Another
possibility is that there was some kind of version management difficulty at Silicon Techtronics,
so that correct code was written, successfully tested, but the wrong code was inserted into the
delivered product."
We asked Professor Silber to explain what he meant by "version management." He said, "In a
given project, a given software component might have several versions: version 1, version 2, and
so forth. These reflect the evolution of that component as the project progresses. Some kind of
mechanism needs to be in place to keep track of versions of software components in a project as
complex as this one. Perhaps the software testers tested a correct version of the robot dynamics
code, but an incorrect version was actually delivered. However, this raises the question as to
what happened to the correct code."
Professor Silber sat back in his chair and sighed. "This really is a great tragedy. If the killer code
had gone through the testing process in an honest manner, the robot would never have killed Bart
Matthews. So, the question becomes, what was going on at Silicon Techtronics that prevented
the honest testing of the critical code?"
The Sentinel-Observer asked Professor Silber whether he agreed with the notion that the user
interface was the ultimate culprit in this case. He responded, "I don't buy the argument, being put
forth by my colleague Professor Gritty, that all of the culpability in this case belongs to the user
interface designer or designers. I agree with some of what he says, but not all of it. I have to ask
myself whether Silicon Techtronics was placing too much emphasis on the user interface as a
last line of defense against disaster. That is, perhaps they knew there was a problem, but they felt
that the user interface could allow the operator to handle that problem."
The Sentinel-Observer then asked Professor Silber about the charge made against him that he
should never have accepted Waterson's appointment to conduct an impartial investigation into
the accident. Critics have pointed out that Silicon Valley University, and Professor Silber in
particular, has many business ties with Silicon Techtronics, and thus he could not be counted on
to conduct an impartial investigation.
"I think my report speaks for itself," Professor Silber replied, visibly annoyed by our question. "I
have told you reporters over and over again that this was not a government investigation but a
corporate investigation, so I believe that Silicon Techtronics had the right to choose whomever
they desired. I believe I was known to them as a person of integrity."
Late yesterday, Robbie CX30 project manager Sam Reynolds hired attorney Valerie Thomas,
and Thomas issued this statement on his behalf: "My client is shocked that someone at Silicon
Techtronics has misled Professor Silber concerning the software tests for the Robbie CX30
robot. Mr. Reynolds asserts that the software was tested and that he and others were well aware
of the fact that there was something wrong with the robot dynamics software. However, Mr. Ray
Johnson, my client's immediate superior at Silicon Techtronics, decided that the robot could be
delivered to Cybernetics Inc., based upon Mr. Johnson's Ivory Snow Theory. According to that
theory, the software was nearly bug free and thus could be released. According to Mr. Johnson,
the risk of failure was very small and the cost of further delaying delivery of the robot was very
great. According to my client, Mr. Johnson felt that the environmental conditions that could
trigger erratic and violent robot behavior were extremely unlikely to occur. Furthermore, Mr.
Johnson felt that the robot operator would not be in danger because the user interface was
designed so as to permit the operator to stop the robot dead in its tracks in the case of any life-
threatening robot motion."
Johnson, robotics division chief at Silicon Techtronics, could not be reached for comment.
Randy Samuels will be placed on trial next month at the Silicon Valley Court House. When
contacted by phone, Samuels referred all questions to his attorney, Alex Allendale.
Allendale had this to say concerning Professor Silber's findings: "My client submitted the
software in question in the usual way and with the usual documentation and with the usual
expectation that his code would be thoroughly tested. He was not aware until Professor Silber's
report came out that the code involved in this terrible tragedy had not been tested properly or that
the test results might have been faked.
"Mr. Samuels wants to again express the great sorrow he feels about this accident. He, more than
anyone else, wants to see justice done in this case. Mr. Samuels once again extends his heartfelt
condolences to Mrs. Matthews and her children."
Silicon Techtronics Employee Admits Faking
Software Tests
Author(s): Richard G. Epstein
Mabel Muckraker
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
Cindy Yardley, a software tester at Silicon Techtronics, admitted today that she was the person
who created the fraudulent "killer robot" software tests. The fraudulent tests were revealed
earlier this week by Silicon Valley University professor Wesley Silber in what has come to be
known as the "Silber Report".
At issue are quality assurance procedures that were performed on the program code written by
Randy Samuels, the programmer charged with manslaughter in the killer robot incident. The
Silber Report asserted that the test results reflected in internal Silicon Techtronics documents
were inconsistent with the test results obtained when the actual killer robot code was tested.
In a startling development at noontime yesterday, Max Worthington, Chief Security Officer for
Silicon Techtronics, announced his resignation at a packed news conference that was broadcast
live by CNN and other news organizations.
Worthington stunned the assembled reporters when he began his news conference with the
announcement, "I am Martha," referring to the fictional name this paper has used to identify an
anonymous source inside Silicon Techtronics.
Worthington described his responsibilities at Silicon Techtronics in this way: "Basically, my job
was to protect Silicon Techtronics from all enemies, domestic and foreign. By foreign I mean
adversaries from outside the corporation. My role was mostly managerial. Those working under
me had many responsibilities, including protecting the physical plant, watching out for industrial
spying and even sabotage. I was also responsible for keeping an eye out for employees who
might be abusing drugs or who might be disloyal in some way to Silicon Techtronics."
Worthington then pointed to a stack of bound volumes which were on a table to his left. "These
volumes represent just some of the electronic surveillance of employees that I conducted over the
years for my superior, Mr. Waterson. These are print outs of electronic mail messages that
Silicon Techtronics employees sent to one another and to persons at other sites. I can say with
great certainly that no employee was ever told that this kind of electronic surveillance was being
conducted. However, I think the evidence shows that some employees suspected that this might
be going on."
Several reporters shouted questions asking who at Silicon Techtronics knew about the electronic
surveillance.
Worthington replied, "No one knew about this except Mr. Waterson, myself, and one of my
assistants, who was responsible for conducting the actual monitoring. My assistant produced a
special report, summarizing e-mail activity once a week, and that report was for Waterson's eyes
and my eyes, only. Upon request, my assistant could produce a more detailed accounting of
electronic communications."
Worthington explained that he was making the e-mail transcripts available to the press because
he wanted the whole truth to come out concerning Silicon Techtronics and the killer robot
incident.
The e-mail messages between employees at Silicon Techtronics indeed revealed new facets of
the case. A message from Cindy Yardley to Robotics Division Chief Ray Johnson indicates that
she faked the test results at his request. Here is the text of that message:
To: ray.johnson
From: cindy.yardlay
Re: samuels software
I have finished creating the software test results for that troublesome robot software, as per your
idea of using a simulation rather than the actual software. Attached you will find the modified
test document, showing the successful simulation.
Should we tell Randy about this?
- Cindy
Johnson's response to Yardley's message suggests that he suspected that electronic mail might
not be secure:
In-reply-to: cindy.yardley
From: ray.johnson
Re: samuel's software
I knew I could count on you! I am sure that your devotion to Silicon Techtronics will be repaid in
full.
Please use a more secure form of communication in the future when discussing this matter. I
assure you that the way we handled this was completely above board, but I have my enemies
here at good ol' SiliTech.
- Ray
These communications were exchanged just a few days before the Robbie CX30 robot was
shipped out to Cybernetics, Inc. This fact is important because the fake software tests were not
part of a cover-up of the killer robot incident. These facts seem to indicate that the purpose of the
fake software tests was to make sure that the Robbie CX30 robot was delivered to Cybernetics
by the deadline that was negotiated between Silicon Techtronics and Cybernetics.
The e-mail transcripts reveal repeated messages from Ray Johnson to various people to the effect
that the Robotics Division would definitely be closed down if the Robbie CX30 project was not
completed on time. In one message, he lectures project leader, Sam Reynolds, on the "Ivory
Snow Theory":
To: sam.reynolds
From: ray.johnson
Re: don't be a perfectionist!
Sam:
You and I have had our differences, but I must tell you that I like you personally. Please
understand that everything I am doing is for the purpose of SAVING YOUR JOB AND THE JOB
OF EVERYONE IN THIS DIVISION. I view you and all of the people who work with me in the
Robotics Division as my family.
Waterson has made it clear: he wants the robot project completed on time. That's the bottom
line. Thus, we have no recourse but "Ivory Snow". You know what I mean by that. It doesn't have
to be perfect. The user interface is our fall back if this version of the robot software has some
flaws. The robot operator will be safe because the operator will be able to abort any robot
motion at any time.
I agree with you that the non-functional requirements are too vague in places. Ideally, if this
weren't crunch time, it would be good to quantify the amount of time it would take the operator
to stop the robot in case of an accident. However, we cannot renegotiate those now. Nor, do we
have time to design new tests for new, more precise nonfunctional requirements.
I cannot emphasize enough that this is crunch time. It's no sweat off Waterson's back if he lops
off the entire Robotics Division. His Wall Street friends will just say, "Congratulations!" You see,
to Waterson, we are not a family, we are just corporate fat.
- Ray
In this message, Ray Johnson seems to be less concerned with the security of communicating by
e-mail.
Neither Ray Johnson nor Sam Reynolds could be reached for comment. However, the Sentinel-
Observer interviewed Cindy Yardley at her home yesterday evening.
Yardley was obviously upset that her private e-mail messages had been released to the press. "I
am relieved in some ways. I felt tremendous guilt when that guy was killed by a robot that I
helped to produce. Tremendous guilt."
The Silicon-Observer asked Ms. Yardley whether she felt that she had made an ethical choice in
agreeing to fake the software test results. She responded with great emotion: "Nothing, nothing
in my experience or background prepared me for something like this. I studied computer science
at a major university and they taught me about software testing, but they never told me that
someone with power over me might ask me to produce a fake software test!"
"When Johnson asked me to do this, he called me to his office, as if to show me the trappings of
power. You see, someday I would like to be in a managerial position. I sat down in his office and
he came right out and said, 'I want you to fake the test results on that Samuels software. I don't
want Reynolds to know anything about this.'"
Yardley fought back tears. "He assured me that no one would probably ever see the test results
because the robot was perfectly safe. It was just an internal matter, a matter of cleanliness, in
case anyone at Cybernetics or higher up in the corporation got curious about our test results. I
asked him whether he was sure about the robot being safe and all that and he said, 'It's safe! The
user interface is our line of defense. In about six months we can issue a second version of the
robotics software and by then this Samuels problem will be solved.'"
Yardley leaned forward in her chair as if her next remark needed special emphasis. "He then told
me that if I did not fake the software tests, then everyone in the Robotics Division would lose
their job. On that basis I decided to fake the test results. I was trying to protect my job and the
jobs of my co-workers."
Yardley is currently pursuing an MBA degree at night at Silicon Valley University.
The Sentinel-Observer then asked Ms. Yardley whether she still felt that she had made an ethical
decision, in view of the death of Bart Matthews. "I think I was misled by Ray Johnson. He told
me that the robot was safe."
Another revelation, contained in the released e-mail transcripts, was the fact that Randy Samuels
stole some of the software that he used in the killer robot project. This fact was revealed in a
message Samuels sent to Yardley when she first tested his software and got erroneous results:
In-reply-to: cindy.yardley
From: randy.samuels
Re: damned if I know
I cannot for the life of me figure out what is wrong with this function, swing_arm(). I've checked
the robot dynamics formula over and over again, and it seems to be implemented correctly. As
you know, swing_arm() calls 14 different functions. I lifted five of those from the PACKSTAT 1-
2-3 statistical package verbatim. Please don't tell a soul! Those couldn't be the problem, could
they?
- Randy
Experts tell the Silicon-Observer that lifting software from a commercial software package such
as the immensely popular PACKSTAT 1-2-3 is a violation of the law, because software is
protected by the same kind of copyright that protects printed materials.
Mike Waterson, CEO of Silicon Techtronics issued an angry statement concerning Max
Worthington's release of "confidential" e-mail transcripts. Waterson's statement said, in part, "I
have asked our attorneys to look into this matter. We consider those transcripts the exclusive
property of Silicon Techtronics. Our intent is to pursue either civil or criminal charges against
Mr. Worthington."
In reaction to yesterday's developments in the killer robot case, Association for Computing
Machinery (ACM) announced its intention to investigate whether any ACM members at Silicon
Techtronics have violated the ACM code of ethics. The ACM is an international association of
computer scientists with 85,000 members.
ACM president Dr. Turina Babbage issued a statement from the ACM's Computer Science
Conference, which is held every winter, this winter in Duluth, Minnesota. She said:
"All members of the ACM are bound by the ACM Code of Ethics and Professional Conduct.
This code states, in part, that ACM members have the general moral imperative to contribute to
society and human well-being, to avoid harm to others, to be honest and trustworthy, to give
proper credit for intellectual property, to access computing and communication resources only
when authorized to do so, to respect the privacy of others, and to honor confidentiality.1
Beyond that, there are professional responsibilities, such as the obligation to honor contracts,
agreements, and assigned responsibilities, and to give comprehensive and thorough evaluations
of computing systems and their impacts, with special emphasis on possible risks.
Several of the people involved in the killer robot case are ACM members and there is cause to
believe that they have acted in violation of our association's code of ethics. Therefore, I am
asking the ACM Board to appoint a task force to investigate ACM members who might be in
gross violation of the code.
We do not take this step lightly. This sanction has been applied only rarely, but the killer robot
incident has not only cost a human life, but it has done much to damage the reputation of the
computing profession.
1.A draft of this code was reported in Communications of the ACM, May 1992. Please note
that the statement by the fictitious Dr. Babbage contains verbatim quotes from the actual ACM
code.
A Conversation with Dr. Harry Yoder
by Robert Franklin
The Sunday Sentinel-Observer Magazine
Harry Yoder is a well-known figure on the Silicon Valley University campus. The Samuel
Southerland Professor of Computer Technology and Ethics, he has written numerous articles and
texts on ethics and the social impact of computers. His courses are very popular, and most are
closed long before the end of the registration period. Dr. Yoder received his Ph. D. in electrical
engineering from the Georgia Institute of Technology in 1958. In 1976 he received a Master of
Divinity degree from the Harvard Divinity School. In 1983 he received an MS in Computer
Science from the University of Washington. He joined the faculty at Silicon Valley University in
1988.
I interviewed Dr. Yoder in his office on campus. My purpose was to get his reaction to the case
of the killer robot and to "pick his brain" about the ethical issues involved in this case.
Sentinel-Observer: Going from electrical engineering to the study of religion seems like quite a
jump.
Yoder: I was an electrical engineer by profession, but all human beings have an inner life. Don't
you?
Sentinel-Observer: Yes.
Yoder: What is your inner life about?
Sentinel-Observer: It's about doing the right thing. Also, it's about achieving excellence in what I
do. Is that what sent you to Harvard Divinity School? You wanted to clarify your inner life?
Yoder: There was a lot going on at the Divinity School, and much of it was very compelling.
However, most of all I wanted to understand the difference between what was right and what
was wrong.
Sentinel-Observer: What about God?
Yoder: Yes, I studied my own Christian religion and most of the major world religions, and they
all had interesting things to say about God. However, when I discuss ethics in a forum such as
this, which is secular, or when I discuss ethics in my computer ethics courses, I do not place that
discussion in a religious context. I think religious faith can help a person to become ethical, but
on the other hand, we all know that certain notorious people who have claimed to be religious
have been highly unethical. Thus, when I discuss computer ethics, the starting point is not
religion, but rather a common agreement between myself and my students that we want to be
ethical people, that striving for ethical excellence is a worthwhile human endeavor. At the very
least, we do not want to hurt other people, we do not want to lie, cheat, steal, maim, murder and
so forth.
Sentinel-Observer: Who is responsible for the death of Bart Matthews?
Yoder: Please forgive me for taking us back to the Harvard Divinity School, but I think one of
my professors there had the correct answer to your question. He was an elderly man, perhaps
seventy, from Eastern Europe, a rabbi. This rabbi said that according to the Talmud, an ancient
tradition of Jewish law, if innocent blood is shed in a town, then the leaders of that town must go
to the edge of the town and perform an act of penance. This was in addition to any justice that
would be meted out to the person or persons who committed the murder.
Sentinel-Observer: That's an interesting concept.
Yoder: And a truthful one! A town, a city, a corporation -- these are systems in which the part is
related to the whole and the whole to the part.
Sentinel-Observer: You are implying that the leaders at Silicon Techtronics, such as Mike
Waterson and Ray Johnson, should have assumed responsibility for this incident right from the
start. In addition, perhaps other individuals, such as Randy Samuels and Cindy Yardley, bear
special burdens of responsibility.
Yoder: Yes, responsibility, not guilt. Guilt is a legal concept and the guilt or innocence of the
parties involved, whether criminal or civil, will be decided in the courts. I guess a person bears
responsibility for the death of Bart Matthews if his or her actions helped to cause the incident --
it's a matter of causality, independent of ethical and legal judgments. Questions of responsibility
might be of interest to software engineers and managers, who might want to analyze what went
wrong, so as to avoid similar problems in the future.
A lot of what has emerged in the media concerning this case indicates that Silicon Techtronics
was a sick organization. That sickness created the accident. Who created that sickness?
Management created that sickness, but also employees who did not make the right ethical
decisions contributed to the sickness.
Randy Samuels and Cindy Yardley were both right out of school. They received degrees in
computer science and their first experience in the working world was at Silicon Techtronics. One
has to wonder whether they received any instruction in ethics. Related to this is the question as to
whether either of them had much prior experience with group work. Did they, at the time that
they were involved in the development of the killer robot, did they see the need to become
ethical persons? Did they see that success as a professional requires ethical behavior? There is
much more to being a computer scientist or a software engineer than technical knowledge and
skills.
Sentinel-Observer: I know for a fact that neither Samuels nor Yardley ever took a course in
ethics or computer ethics.
Yoder: I suspected as much. Let's look at Randy Samuels. Based upon what I've read in your
newspaper and elsewhere, he was basically a hacker type. He loved computers and
programming. He started programming in junior high school and continued right through
college. The important point is that Samuels was still a hacker when he got to Silicon
Techtronics and they allowed him to remain a hacker.
I am using the term "hacker" here in a somewhat pejorative sense and perhaps that is not fair.
The point that I am trying to make is that Samuels never matured beyond his narrow focus on
hacking. At Silicon Techtronics, Samuels still had the same attitude toward his programming as
he had in junior high school. His perception of his life and of his responsibilities did not grow.
He did not mature. There is no evidence that he was trying to develop as a professional and as an
ethical person.
Sentinel-Observer: One difficulty, insofar as teaching ethics is concerned, is that students
generally do not like being told "this is right and that is wrong".
Yoder: Students need to understand that dealing with ethical issues is a part of being a
professional computer scientist or software engineer.
One thing that has fascinated me about the Silicon Techtronics situation is that it is sometimes
difficult to see the boundaries between legal, technical and ethical issues. Technical issues
include computer science and the management issues. I have come to the conclusion that this
blurring of boundaries results from the fact that the software industry is still in its infancy. The
ethical issues loom large in part because of the absence of legal and technical guidelines.
In particular, there are no standard practices for the development and testing of software. There
are standards, but these are not true standards. A common joke among computer scientists is that
the good thing about standards is that there are so many to choose from.
In the absence of universally accepted standard practices for software engineering, there are
many value judgments, probably more than in other forms of production. For example, in the
case of the killer robot there was a controversy concerning the use of the waterfall model versus
prototyping. Because there was no standard software development process, this became a
controversy, and ethical issues are raised by the manner in which the controversy was resolved.
You might recall that the waterfall model was chosen not because of its merits but because of the
background of the project manager.
Sentinel-Observer: Did Cindy Yardley act ethically?
Yoder: At first, her argument seems compelling: she lied, in effect, to save the jobs of her
coworkers and, of course, her own job. But, is it ever correct to lie, to create a falsehood, in a
professional setting?
One book I have used in my computer ethics course is Ethical Decision Making and Information
Technology by Kallman and Grillo.1 This book gives some of the principles and theories behind
ethical decision making. I use this and other books to help develop the students' appreciation for
the nature of ethical dilemmas and ethical decision making.
Kallman and Grillo present a method for ethical decision making and part of their method
involves the use of five tests: the mom test -- would you tell your mother what you did; the TV
test -- would you tell a national TV audience what you did; the smell test -- does what you did
have a bad smell to it; the other person's shoes test -- would you like what you did to be done to
you; and the market test -- would your action be a good sales pitch?
What Yardley did fails all of these tests, I think nearly everyone would agree. For example, can
you imagine Silicon Techtronics using an ad campaign that runs something like this:
"At Silicon Techtronics, the software you get from us is bug free, because even if there is a bug,
we will distort the test results to hide it, and you will never know about it. Ignorance is bliss!"
This shows that apparent altruism is not a sufficient indicator of ethical behavior. One might
wonder what other unstated motives Ms. Yardley had. Could it be that personal ambition led her
to accept Ray Johnson's explanation and his assurance that the robot was safe?
Sentinel-Observer: Are there any sources of ethical guidance for people who are confronted with
an ethical dilemma?
Yoder: Some companies provide ethical guidelines, in the form of corporate policies, and there is
such a document at Silicon Techtronics, or so I am told. I haven't seen it. An employee could
also refer to ethical guidelines provided by professional societies, such as the ACM. Beyond that,
he or she could read up on the subject to get a better feel for ethical decision making. Of course,
one must always consult with one's conscience and innermost convictions.
Sentinel-Observer: Did Randy Samuels act ethically?
Yoder: Stealing software the way that he did was both unethical and illegal.
I think the most important issue with Randy Samuels has never been discussed in the press. I
truly doubt that Samuels had the requisite knowledge that his job required. This kind of
knowledge is called domain knowledge. Samuels had a knowledge of computers and
programming, but not a very strong background in physics, especially classical mechanics. His
lack of knowledge in the application domain was a direct cause of the horrible accident. If
someone knowledgeable in mathematics, statistics and physics had been programming the robot
instead of Samuels, Bart Matthews would probably be alive today. I have no doubt about that.
Samuels misinterpreted the physics formula because he didn't understand its meaning and import
in the robot application. It may be that management is partly responsible for the situation.
Samuels might have told them his limitations and management might have said, "What the hell!"
Samuels had difficulty with group work, peer reviews and egoless programming. It is possible
that he was trying to hide his lack of expertise in the application domain.
Sentinel-Observer: Did Ray Johnson act ethically?
Yoder: This 'Ivory Snow' business! The trouble with the Ivory Snow theory is that it was just a
theory. If it were more than a theory and an actual methodology for keeping the likelihood of
failure within statistically determined limits, like what is called "clean room software
engineering", then there would be less culpability here.
Based upon the information that I have, the Ivory Snow theory was just a rationalization for
getting flawed software out the door to customers on time. The Ivory Snow theory is only valid,
ethically and professionally, if the customer is told of known bugs, or impurities, if we can use
the soap jargon. In the case of Silicon Techtronics, the Ivory Snow Theory worked like this: we
know it's not pure, but the customer thinks it is!
Of course, coercing Cindy Yardley the way Ray Johnson did was also not ethical. Did he believe
what he told Ms. Yardley, namely that the robot was safe, or was that an out and out lie? If he
believed that the robot was safe, why cover up with the false tests? If the user interface were so
important as a last line of defense, why avoid more rigorous tests of the user interface?
Sentinel-Observer: What is your view of Mike Waterson in all this?
Yoder: If Johnson is the father of the Ivory Snow theory, Waterson is the grandfather. His
demand that the robot be completed by a certain date or heads would roll might have caused
Johnson to formulate the Ivory Snow theory. You see, it is apparent that Johnson thought that the
delivery of Robbie to Cybernetics by the specified date was impossible unless the robot software
had bugs.
In many regards I feel that Waterson acted unethically and irresponsibly. He placed Sam
Reynolds in charge of the robot project, yet he, Reynolds, lacked experience with robots and
modern user interfaces. Reynolds rejected the idea of developing a prototype, which might have
allowed for the development of a better user interface.
Waterson created an oppressive atmosphere for his employees, which is unethical in itself. Not
only did he threaten to fire everyone in the Robotics Division if the robot was not completed on
time, he eavesdropped on private electronic mail communications throughout the corporation, a
controversial right that some companies do claim.
My personal belief is that this kind of eavesdropping is unethical. The nature of e-mail is
somewhat of a hybrid of normal mail and a telephone conversation. Monitoring or spying on
someone else's mail is considered unethical, as is tapping a telephone. Indeed, these activities are
also illegal under almost most circumstances. So, I believe it is an abuse of power to monitor
employees the way that Waterson did.
Sentinel-Observer: Does the prosecutor have a case here?
Yoder: Against Randy Samuels?
Sentinel-Observer: Yes.
Yoder: I doubt it, unless she has information that has not been made public thus far.
Manslaughter, to my understanding, implies a kind of reckless and irresponsible act, causing
death of another. Does this description apply to Samuels? I think the prosecutor's best bet is to
stress his lack of knowledge in the application domain if it can be shown that he did engage in a
deliberate deception.
I read last week that 79% of the people favor acquittal. People are inclined to blame the
corporation and its managers. Last night, one of the network news anchors said, "Samuels isn't a
murderer, he's a product of his environment".
Sentinel-Observer: Could you restate your position on the matter of ultimate responsibility in the
case of the killer robot?
Yoder: In my mind, the issue of individual versus corporate responsibility is very important. The
corporation created an environment in which this kind of accident could occur. Yet, individuals,
within that system, acted unethically and irresponsibly, and actually caused the accident. A
company can create an environment that brings out the worst in its employees, but individual
employees can also contribute to the worsening of the corporate environment. This is a feedback
loop, a system in the classical sense. Thus, there is some corporate responsibility and some
individual responsibility in the case of the killer robot.
Sentinel-Observer: Thank you, Professor Yoder.
Footnotes
1.This is an actual text book from McGraw-Hill.
The Case of the Killer Robot is a fictional scenario for ethics teaching and discussion purposes.
... The scandal in Academia [32] [33] [34] [35] is an extended fictional case-study intended for use as a teaching and discussion aid for educational practitioners looking to introduce elements of computer ethics into their curricula. Inspired by Epstein [17] [18] it is a full-cycle scenario involving many individuals which touches upon the complexity and interrelations of modern computer ethics. It has been trailed and evaluated as a teaching tool by the authors [36] and with multiple groups since then. ...
Article
Full-text available
The Scandal in Academia [32] [33] [34] [35] is an extended fictional case-study intended for use as a teaching and discussion aid for educational practitioners looking to introduce elements of computer ethics into their curricula. Inspired by Epstein [17] [18] it is a full-cycle scenario involving many individuals which touches upon the complexity and interrelations of modern computer ethics. It has been trailed and evaluated as a teaching tool by the authors [36] and with multiple groups since then. However its utility as a general resource is limited without the academic context that supports deeper investigation of the material. It is to address this issue that the authors offer this commentary on the Scandal, with a focus on the ninth and tenth newspaper items presented within. Specifically these are Culture of Fear and Nepotism at University and Witch-Hunts at the University - IT Crackdown Causes Criticisms.
... The exercise above is one example in which the technical details lead directly to questions about professional responsibility. Other resources that might be helpful to faculty wanting to develop their own exercises include Epstein's classic "Case of the Killer Robot" [5], and Collins and Miller's "Paramedic Ethics. " [2] The exercise was designed to encourage students to discover the importance of focusing on people affected by their decisions during software development. ...
Article
Full-text available
The integration of ethics into computer science instruction is not a new idea (for example, see [10]), but it is still useful to illustrate how this integration can be done in specific kinds of courses [14]. In this article, we present a hands-on ethics lesson for students involved in programming. We think this exercise can be used in any undergraduate class that engages students in software development. Although our examples are written using JavaScript, and can run in any modern browser, they could be easily adapted to other programming languages and platforms. Our goal in this exercise is to encourage students to think more carefully about the ethical implications of the kinds of work students are likely to be asked to do in their first software development jobs.
... However, if such machines are peered together, they can cause a massive destruction which might take a long time to reverse. One might ask; how can this happen, studied evidence shows that [21] if machines are connected together, there are high chances that they can exchange information that can be of help coordinate an attack on anything including human race. ...
Article
Full-text available
Humanoid and autonomous robots are well known globally not just for their intelligence, but also because they are highly automated and advanced. From the time of their first invention and usage till date, Man has been at the receiving end of their limitations, faults and/or destructiveness. They have caused greater harm than good in some senses, and hence, a rising agitation of their ban. This paper seeks to explore the challenges, risks, problems and drawbacks of the existence and usage of these robots in all aspects of human livelihood; in health, agricultural and social institutes, in attack or defense operations etc. while it dives through some reports by human right organizations and those of the general public, it would consider various revolting voices on why these innovations aren't the cutting edge of human civilization.
Book
This book explores how the design, construction, and use of robotics technology may affect today’s legal systems and, more particularly, matters of responsibility and agency in criminal law, contractual obligations, and torts. By distinguishing between the behaviour of robots as tools of human interaction, and robots as proper agents in the legal arena, jurists will have to address a new generation of “hard cases.” General disagreement may concern immunity in criminal law (e.g., the employment of robot soldiers in battle), personal accountability for certain robots in contracts (e.g., robo-traders), much as clauses of strict liability and negligence-based responsibility in extra-contractual obligations (e.g., service robots in tort law). Since robots are here to stay, the aim of the law should be to wisely govern our mutual relationships.
Chapter
Robots are affecting tenets of current legal systems in a twofold way. First, robotic technology is inducing a number of critical legal loopholes, which are proper of the criminal law field, e.g., the employment of autonomous robot soldiers in battle. Significantly, Christof Heyns, Special Rapporteur on extrajudicial executions, urged in his 2010 Report to the UN General Assembly that Secretary-General Ban Ki-moon convene a group of experts in order to address “the fundamental question of whether lethal force should ever be permitted to be fully automated.” On the other hand, we have to determine whether the behaviour of robots falls within the loopholes of the system, necessitating the intervention of lawmakers at both national and international levels, as they did in the early 1990s when establishing a new class of computer crimes. Besides the immunity of military and political authorities for the use of robots in battle, a second class of hard cases concerns how the growing autonomy of robots affects key notions of the system, such as reasonability, predictability, or foreseeability, on which an individual’s fault depends. This is the class of hard cases that criminal lawyers share with experts in tort law and contracts.
Chapter
L’obbiettivo di queste note bibliografiche è duplice. Il primo è di fornire al lettore ulteriore materiale in cui trovare resoconti più dettagliati sugli argomenti discussi. Il secondo è di dare il giusto credito alle persone responsabili della ricerca di cui si fa cenno nel libro, e di indirizzare il lettore, ove possibile, alle pubblicazioni originali in cui sono contenute le loro idee e i loro risultati.
Chapter
This chapter examines the impact of robotics technology on legal systems and how a new generation of robo-traders, AI chauffeurs, artificial pop singers and autonomous lethal weapons affect individual’s knowledge, environments and perceptions of the world. Although lawyers deem robots under the current state of law as legally and morally without responsibility as these artificial agents lack a set of preconditions for attributing liability to a party within the realm of criminal law, such machines are reshaping notions of agency and human liability in civil (as opposed to criminal) law. We already have a number of artificial agents that send bids, accept offers, request quotes, negotiate deals and make contracts, so that such machines can be held liable for certain of their actions through new types of accountability, authentication systems and insurance models. At least in the civil law-field, “only robots shall pay” at times may be the right answer.
ResearchGate has not been able to resolve any references for this publication.