Content uploaded by David D Woods
Author content
All content in this area was uploaded by David D Woods on Nov 14, 2024
Content may be subject to copyright.
Wrong and Strong
Wrong, Strong, and Silent:
What happens when automated systems with high autonomy and high authority misbehave?
Sidney W. A. Dekker David D. Woods
Griffith University The Ohio State University
Abstract
Warnings about the risks of literal-minded automation—a system that can’t tell if its model of
the world is the world it is actually in—have been sounded for over 70 years. The risk is that a
system will do the ‘right’ thing—its actions are appropriate given its model of the world, but it
is actually in a different world—producing unexpected/unintended behavior and potentially
harmful effects. This risk— wrong, strong and silent automation—looms larger today as our
ability to deploy increasingly autonomous systems and delegate greater authority to such
systems expands. It already produces incidents, outages of valued services, financial losses,
and fatal accidents across different settings. This paper explores this general and out-of-
control risk by examining a pair of fatal aviation accidents which revolved around wrong,
strong and silent automation.
Author Biographies
Sidney W. A. Dekker, is Professor and Director of the Safety Science Innovation Lab at Griffith University,
Brisbane, Australia, and Professor in Aerospace Engineering at Delft University, The Netherlands. A
prolific author on Human Factors and Safety including!The Field Guide to Understanding ‘Human Error;
Safety Differently;!Just Culture; The Safety Anarchist. He has flown Boeing 737 for an airline on the side,
is a trained mediator, and chaplain (see sidneydekker.com).
David D. Woods, Professor Emeritus in Integrated Systems Engineering at the Ohio State University and
Past-President of the Resilience Engineering Association and Human Factors and Ergonomics Society.
He pioneered in Resilience Engineering—how to support people as they adapt to cope with
complexity. His books include Behind Human Error, Resilience Engineering, Joint Cognitive Systems. He
started the SNAFU Catchers industry-university consortium to apply the new science to build resilience
in critical digital services.
Preprint of Dekker, S. W. A. and Woods, D. D. (2024). Wrong, Strong, and Silent: What Happens when
Automated Systems with High Autonomy and High Authority Misbehave? Journal of Cognitive
Engineering and Decision Making, 18(4) 339–34, DOI: 10.1177/15553434241240849.
1 of 13
Wrong and Strong
Literal-Minded Machines
Warnings about the limits of automata—algorithm(s) embodied to carry out activities on its
own when authorized by another party—began as soon as scientific progress led to tooling
that enabled widespread development and deployment of automata into dynamic and risky
worlds. Norbert Wiener (1950) highlighted the risks of literal-minded machines—a system that
can’t tell if its model of the world is the world it is actually in. As a result, the system will do the
right thing—in the sense that the actions are appropriate given its model of the world, when it
is in a different world—producing unexpected/unintended behavior and potentially harmful
effects (Woods and Hollnagel, 2006, chapters 10/11).
Technology advances our ability to deploy increasingly autonomous systems and our
willingness to delegate greater authority to these systems (to expand the scope of authority
where the system acts on its own). Risks associated with literal-minded automated systems
and subsystems loom larger every day, already having caused incidents, outages, financial
losses and fatal accidents across different industries. We see this in for example vehicle path
control, medication infusion, plan following, stock trading, and vehicle-to-vehicle separation.
When assumed and actual worlds do not match, automated systems will misbehave, taking
actions that are inappropriate and possibly dangerous. Some supervisory role has to
recognize the behavior as inappropriate or dangerous and intervene by redirecting the
automation. Stepping from a monitoring role into an active role in a developing non-normal
or abnormal situation is a difficult shift. If the behavior and configuration of automated
systems is opaque (What is it doing? What will it do next? What is the configuration currently
controlling key parameters or processes?), and if the mechanisms for re-directing parts of the
suite of automation are clumsy, then the integrated system has a built-in vulnerability to
breakdowns where the automation is strong, silent and wrong.
Of course, automata can function alone, but limits like literal-mindedness arise and
complexity factors produce surprising challenges that exceed an automata’s competence
envelope (Maguire, 2024). Suites of automata can be designed to coordinate with
supervisory roles to produce a more robust and resilient multi-agent integrated system
(Eraslan et al., 2020; Morey et al., 2020; Farjadian et al., 2021). Suites of automata can be
designed to be cooperating agents in a shared activity space with others human and machine
agents when disruptions occur and spread (Johnson et al., 2014; 2017; Maguire, 2024). How
this is done is beyond the scope of this paper. Why this knowledge is used so little remains a
troubling technical, psychological and social problem debated elsewhere (Woods, 2021).
Strong, Silent, Difficult to Direct Automation in Aviation
The adaptive pattern above played out in the introduction and expansion of flight deck
automation in the1980s. The effects and changes over time, led substantially by NASA, have
been well studied (see References:Flight Deck Automation Studies). Driven by reports of pilot
of 2 13
Wrong and Strong
difficulties, incidents, and accidents involving modern (or modernized), highly automated
flight decks, this research yielded insights into what has become known as ‘mode awareness,’
and ‘automation surprises.’ These denote a breakdown in human-machine coordination that
could be traced to the new ‘strong, silent, difficult to direct’ suite of automated subsystems on
the flight deck (Woods and Sarter, 2000, p. 329).
‘Strong’ is used to refer to the authority delegated to an automated (sub-)system. In the suite
of automated systems on modern aircraft, the envelope protection sub-system is ‘strong’ in
that it has the authority to take over control of the aircraft from the flight crew if it’s inputs/
internal model suggest the flight is exceeding a safe operating envelope. In this case the
delegation is from aircraft automation design and industry layers, not onboard pilots.
Traditionally in aviation this is called control authority—what automated subsystem is driving
what aspect of the aircraft, flightpath and flight plan—e.g., thrust, pitch, heading, altitude, and
changes in these. Other automated sub-systems scope of authority for control of path, plan,
power, etc. vary based on instructions or directions from the flight crew (via tactical modes or
more advanced modes where the automation will fly a maneuver on its own or in part). Within
the pilot-delegated scope of authority the suite of automation will do things on its own. One
example that has been a contributor to incidents and accidents is “indirect mode” changes
where automation changes the configuration of the automation, going beyond the specific
pilot supervisory input to the automation. This issue led to the discovery of mode awareness
as a contributor to automation surprises (Sarter and Woods, 1995).
‘Silent’ is used to refer to low observability—the form and quality of the feedback between
human (and machine) agents about how the automation is configured to fly the aircraft—
transitions in flight path control, flight maneuvers, flight plan—and how the aircrafts actual
behavior. Observability is about ability to see events and activities ahead—the near future,
especially transitions (external events, internal configuration changes). As Earl Wiener put it
famously in 1989: observability is the how smoothly human supervisors of automation can
answer questions—What’s it doing now? What is it going to do next? When unexpected events
occur (in automation configuration/control; aircraft behavior; external events), observability
refers to how smoothly human supervisors of automation can answer questions—Why did it
do that? How did we get into that mode? Past research has shown observability of the
behavior and changes in the configuration of the automation suite can be low, and deploying
high autonomy/high authority automata require more sophisticated design for observability
(Sarter, 2002).
Difficult to direct refers to how smoothly the design allows the flight crew to modify
automated system configuration/behavior as conditions and priorities change given the
tempo of operations—how the flightcrew manages the automation as the automation controls
the aircraft, e.g., instruct, program, configure, override, change scope of authority. Re-
directability refers to how smoothly human supervisors of automation can answer questions
such as—How do I get it to do what I want? How do I stop it from doing this? Effective
supervisory control requires mechanisms for re-directing automation prior to reversion to
of 3 13
Wrong and Strong
direct/manual control (note: as automation grows more powerful and central to operations
here and elsewhere, reversion to manual control becomes more difficult or even impossible)
When sensor inputs to strong, silent, difficult to direct automation go bad
The danger in how literal minded automata contributes to incidents and accidents can arise
from many sources (see References:Sample Accidents). One is hidden interdependencies in
software (e.g. as can be seen in radiation mis-administrations; medication mis-administrations
via automated infusion devices, runaway automation in financial trading). Another is bad
inputs to automated systems operating with high authority. Several commercial airliner
accidents in the past decade have been linked directly to automated system sensor failures. A
compelling example is formed by the twin Boeing 737 MAX accidents which together killed
346 people in 2018 and 2019.
In the latest modification to Boeing’s 737 series of aircraft, a maneuvering characteristics
augmentation system (MCAS) was added to the existing suite of automated systems and
given high command authority over pitch-up excursions (designed to respond to bring pitch
down to a normal range via control of the horizontal stabilizer). Another system already
existed to control pitch via the elevator. The design evolved to act twice as strong as the
human pilot—for every pilot input gain on the elevator, MCAS makes double that gain in the
opposite direction on the horizontal stabilizer. MCAS also adjusts the stabilizer at a rate faster
than can be countered by pilots using electric trim switches on the control column.
Repetitive activation was built into MCAS which represents persistent high control authority—
while it could deactivate after 10 seconds—it would reactivate again 5 seconds later. As long
as MCAS received sensor input in its trigger range, it would continue to reactivate trying to
push the nose of the aircraft down. Before the first accident in 2018—Lion Air FL610, pilots did
not know about the addition or operation of MCAS. MCAS was not described by name in
manuals or training.
MCAS control software operated based on input from Angle of Attack (AOA) sensors
(measures of pitch attitude). Except there was only one sensor. Individual AOA sensors, like all
sensors (especially those in harsh environments), can have reliability issues. Normally sensor
design adds some mix of redundancy, diversity and software checks to ensure accurate
inputs, detect when inputs may be inaccurate, and inform supervisors of the need to
intervene, redirect or takeover. But MCAS was not part of the safety case for the modified 737
MAX; why would the extra steps for extra reliability matter? [Note: Boeing was actively
minimizing the chance MCAS would become part of the safety case or need extra testing for
safety implications. However the same assumption has arisen before other accidents
occurred—evidence of a risk is downplayed or ignored for items not on the safety list—e.g.,
Columbia Space Shuttle Accident.]
of 4 13
Wrong and Strong
Another widespread assumption was present here and in the reactions to the twin accidents.
Engineering considered only the controller design/software as “the MCAS system” with
sensors, sensor reliability, alerts, much less supervisory control features as outside. This
confusion about system boundaries and what is the integrated whole for deployed
automation is common for autonomy and AI as well. In this case, the narrow perspective
meant many claimed “the automation” didn’t fail. This tendency to narrow what is “the
automation or the autonomy” hides the true integrated system, its complexity, overestimates
reliability and underestimates the need for resilient supervisory control (Woods, 2016).
The danger of a literal-minded machine was present—MCAS acting according to its model of
the world which was mismatched to the actual world. In the real world, aircraft attitude was
normal; in MCAS’s model, the aircraft was pitching up and dictated a vigorous response. In
the meantime, the situation in the cockpit followed the classic automation surprise pattern
with multiple lines of cognitive work interwoven as the tempo of operations increases,
uncertainty is high, danger is increasing while opportunities for recovery are vanishing (Sarter
et al., 1997; Woods and Sarter 2000; Woods and Paterson, 2001). Given the development/
design/testing policy that MCAS operation was not safety related and that single sensor
channel reliability was sufficient, flight crew were not provisioned with knowledge or signals
to determine that MCAS was driving the abnormal behavior. Nor did they have knowledge or
guidance on how to stop an automatic control system they did not know existed. In the first
accident, these decisions meant the flight crew had no help to understand and intervene
successfully to counter MCAS’ high command authority as the situation deteriorated.
As seen vividly in the 2nd MCAS driven accident, Ethiopian Airlines 302, stopping MCAS
activation and counteracting the effects of its mis-control was, in part, cumbersome and in
part, indirect. Ultimately, the limited alternative means for recovery/regaining control were
not strong enough to match the size of the disturbances created by the misbehavior of MCAS
—the pilots did not have sufficient control authority to recover without turning MCAS back on.
Once reactivated, the cycle of persistent over-control occurred. This ‘fight for control’
between the crew and the misbehaving automation has precedents in other aviation
accidents (including one where different parts of the suite of automation controlled the
aircraft at cross-purposes—DSB, 2010).
The recommended procedure, developed after the first accident, called for the pilots to use
the manual stabilizer trim wheel to recover control. This was not effective because of the
speed of the dive that MCAS had commanded (manual control inputs could not overcome
the aerodynamic forces on the horizontal tail). This meant the only option was to switch the
electric trim system back on which automatically re-activated MCAS—still programmed to take
‘strong and persistent’ actions given the erroneous input.
Information about angle of attack sensor problems were also kept from most crews. A Boeing
AR had questioned in 2015 whether MCAS was vulnerable to single AOA sensor failures, but
this was dismissed by Boeing test pilots and the aircraft was delivered with MCAS dependent
of 5 13
Wrong and Strong
only a single AOA sensor despite this. Even if MCAS software had been provisioned to two
AOA sensors, from the beginning of deliveries, 80% of the 737 MAX fleet worldwide was
flying around with an inoperable angle of attack disagree alert. Boeing had known this since
2015, but the FAA learned of it only after the first MAX accident (Lion Air).
Fundamentally, the design for supervisory management of MCAS as part of a suite of
automated systems for different aspects of flight was virtually non-existent. In other words, the
twin 737 Max accidents are a compelling exemplar of the risks from poor design of
supervisory management when high authority/high autonomy automation misbehaves. It
should not take 346 lives to get engineering and engineering management to understand
the fundamental vulnerability highlighted by these accidents. Why? Because similar events
have happened before and because much of the knowledge for joint system design has
already been developed and demonstrated (see References:Sample Accidents and, e.g.,
Johnson et al., 2014; 2017; Schraagen et al., 2022).
Conclusion: Misbehavior of Strong, Silent, Difficult to Direct Automation
is a General Risk Out-of-Control
Automated systems with high autonomy and high authority will misbehave when factors
combine to create a gap between the internal model of the world and the actual events/
context going on in the world where the automation is deployed. This risk is inescapable and
individual incidents or accidents involving misbehavior of strong, silent, difficult to direct
automation occur regularly as stakeholders deploy increasingly autonomous systems with
high authority in dynamic risky worlds (Woods, 2016). However, the risk is seen as an issue to
be handled on a case by case basis for the designers using tools tailored for a specific sub-
system/application. Organizational and financial pressures easily overwhelm engineering
teams’ ability to address the risk as in the 737 MAX accidents (Dekker et al., 2023).
Furthermore, fundamental research has shown there are hard limits to the achievable
robustness of high authority/high autonomy systems. Systems engineering could expand its
scope to engage techniques/models for design for supervisory management as part of joint
activity of systems of multiple automated and human roles (Johnson et al., 2014; 2017).
Furthermore, methods for designing resilient layered control have been developed recently
(Eraslan et al., 2020; Farjadian et al., 2021; Woods and Balkin, 2018).
Equally, stakeholders regularly discount the systemic and organizational lessons from these
breakdowns (Woods et al., 2010). Some see each individual incident or accident involving
misbehavior of strong, silent, difficult to direct automation as simply basic design failures
which have direct engineering solutions (MCAS design could have used standard methods
for sensor redundancy/diversity). In hindsight, other analysts determine there was some path
available to escape the deteriorating situation—therefore no systemic lessons or changes are
needed (as Boeing continues to assert on 737 MAX accidents). Ironically, these analysts
operate with knowledge/time/resources unavailable to supervisors responsible for a risky
of 6 13
Wrong and Strong
system. Supervisors in the scene face uncertainties, overload, and pressures after-the-fact
analysts miss or underplay.
The solution is straightforward. More robust and resilient control is indeed possible if and
only if stakeholders recognize the risk as fundamental and expand the systems engineering
concepts/techniques. All parties engaged in deploying high authority/high autonomy
systems/sub-systems should prioritize misbehavior of strong, silent, difficult to direct
automation as a top level failure mode in their risk analyses, testing programs, and assurance
processes.
Unless and until this shift is normative, formally and informally, the race to deploy high
authority/high autonomy systems will be accompanied by incidents/accidents driven by
misbehavior of strong, silent, difficult to direct automation. Underlying science has shown that
the risk arises from architecture/design decisions and has shown the way to architect more
resilient system architectures (Woods, 2018; 2024). There are pragmatic means available to
achieve gains from high authority/high autonomy systems while also provisioning safeguards
against wrong, silent and strong automation. But utilizing the knowledge requires a
substantial shift at organizational levels to re-balance/re-prioritize the trade-off between
maximizing short-term gains while discounting longer-term risks of autonomy. The short-term
financial costs of the joint systems and resilience engineering methods do not outweigh the
long-term risks as exemplified by massive financial losses in some cases (e.g, Knight Capital
1
case in References:Sample Accidents) and by the deaths of 346 people in the twin Boeing
737 MAX accidents.
References
General
Dekker, S. W. A., Layson, M. D., and Woods, D. D. (2022). Repentance as rebuke: Betrayal and
moral injury in safety engineering Science and Engineering Ethics, 28:56 https://
doi.org/10.1007/s11948-022-00412-2
Eraslan, E., Yildiz, Y. and Annswamy, A. M. (2020). Shared Control Between Pilots and
Autopilots: Illustration of a Cyber-Physical Human System. IEEE Control Systems
Magazine 40(6):77-97, December 2020. DOI: 10.1109/MCS.2020.3019721
Farjadian, A. B., Thomson, B., Annaswamy, A. M. and Woods, D. D. (2021). Resilient Flight
Control: An Architecture for Human Supervision of Automation. IEEE Transactions on
Control Systems Technology, 29(1), 29-42. DOI: 10.1109/TCST.2019.2959542
Johnson, M., J.M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. B. van Riemsdijk, and M. Sierhuis.
Coactive Design: Designing Support for Interdependence in Joint Activity. Journal of
Human-Robot Interaction, Vol. 3, No. 1, (2014), pp. 43-69.
Multiple outages of critical digital infrastructure have led to shutdowns for several airlines costing 10s-100s of
1
millions of dollars (see press reports on several Southwest, Several British Airways, plus American and Delta.
of 7 13
Wrong and Strong
Johnson, M., Bradshaw, J. M., & Feltovich, P. J. (2017). Tomorrow’s Human–Machine Design
Tools: From Levels of Automation to Interdependencies. Journal of Cognitive
Engineering and Decision Making, 12 (1). DOI:10.1177/1555343417736462
Maguire, L. (2024a).!Automation Doesn’t Work the Way We Think It Does. IEEE Software,!41(1),
Jan.-Feb.,138-141.DOI: 10.1109/MS.2023.3328746
Maguire, L. (2024b).!Joint Cognitive Systems: Ideas With Impact for Designing Safer, More
Resilient Automated Systems.!IEEE Software,!41(3), May-June, in press.
Morey, D. A., Marquisee, J. M., Gifford, R. C., Fitzgerald, M. C., & Rayo, M. F. (2020). Predicting
graceful extensibility of human-machine systems: a new analysis method for
evaluating extensibility plots to anticipate distributed system performance.
In!Proceedings of the Human Factors and Ergonomics Society Annual Meeting!(Vol. 64,
No. 1, pp. 313-318). Sage CA: Los Angeles, CA: SAGE Publications.
Nakahira, Y., Liu, Q., Sejnowski, T. J. and Doyle, J. C. (2021). Diversity-enabled sweet spots in
layered architectures and speed–accuracy trade-offs in sensorimotor control.
Proceedings of the National Academy of Science PNAS, 118, 22 https://doi.org/
10.1073/pnas.1916367118
Sarter, N. B., & Woods, D. D. (1995). “How in the world did we get into that mode?” Mode
error and awareness in supervisory control. Human Factors, 37(1), 5-19. https://
doi.org/10.1518/001872095779049516
Sarter, N. B., Woods, D. D., & Billings, C. E. (1997). Automation Surprises. In G. Salvendy (Ed.),
Handbook of Human Factors and Ergonomics, 2nd Edition (pp. 1926-1943). New York:
Wiley.
Schraagen, J.-M., Barnhoorn, J. S., van Schendel, J. & van Vught, W. (2022) Supporting
teamwork in hybrid multi-team systems, Theoretical Issues in Ergonomics Science,
23:2, 199-220, DOI: 10.1080/1463922X.2021.1936277
Wiener, N. (1950). The human use of human beings: Cybernetics and society. NewYork:
Doubleday.
Woods, D. D., & Sarter, N. B. (2000). Learning from Automation Surprises and Going Sour
Accidents. !In N. Sarter and R. Amalberti (Eds.), Cognitive Engineering in the Aviation
Domain, Erlbaum, Hillsdale NJ, pp. 327-354.
Woods, D. D., & Patterson, E. S. (2001). How unexpected events produce an escalation of
cognitive and coordinative demands. PA Hancock, & PA Desmond, Stress, Workload,
and Fatigue. Mahwah, NJ: L. Erlbaum.
Woods, D.D. and Hollnagel, E. (2006). Joint Cognitive Systems: Patterns in Cognitive Systems
Engineering. Boca Raton FL: Taylor & Francis.
Woods et al., 2010
Woods, D. D. (2016). The Risks of Autonomy: Doyle’s Catch. Journal of Cognitive Engineering
and Decision Making, 10(2), 131–133. https://doi.org/10.1177/1555343416653562
Woods, D. D. (2018). The Theory of Graceful Extensibility. Environment Systems and
Decisions, 38:433–457. https://doi.org/10.1007/s10669-018-9708-3
Woods, D. D. and Balkin, E. A. (2018). A Resiliency Trade Space Study of Detect and Avoid
Autonomy on Drones When Communications are Degraded. Report for NASA Ames
Research Center June 30, 2018.
of 8 13
Wrong and Strong
Woods, D. D. (2019). Cover Story: Misbehaving Automation. Air Traffic Management
Magazine, Issue 2, 2019, p. 22-25. Download from …
Woods, D. D. (2021). How to Kill Zombie Ideas: Why do people tenaciously believe myths
about the relationship between people & technology? Invited talk, Georgia Institute of
Technology,10/22/21. https://repository.gatech.edu/entities/publication/f362ff01-
a8b3-4734-89ca-95d0b2c1f647, Downloaded 02/22/24.
Woods, D. D. (2024). Limits of Automata—Then and Now: Challenges of Architecture,
Brittleness, and Scale. Journal of Cognitive Engineering and Decision Making, in press.
Sample of Accidents with Wrong, Strong, Silent Automation outside Aviation
Cook R. I., Woods, D. D. and Howie, M. B. (1992). Unintentional delivery of vasoactive drugs
with an electromechanical infusion device. Journal of Cardiothoracic and Vascular
Anesthesia, 6:238–244.
Leveson, N. G., & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer,
26(7), 18–41.
Cook R. I. and Woods, D. D. (1996). Implications of automation surprises in aviation for the
future of total intravenous anesthesia (TIVA). Journal of Clinical Anesthesia, 8:29s—37s
Leveson, N. (2001). Systemic factors in software-related spacecraft accidents. AIAA Space
2001 Conference and Exposition, 4763.
Knight Capital Runway Automation (2012) in Layered System, see Hochstein, L. (2014).
Counterfactual Thinking, Rules, and The Knight Capital Accident. Blog Post October
29, 2014 https://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/
Woods, D. D., ed. (2017). STELLA: Report from the SNAFU Catchers Workshop on Coping
With Complexity. SNAFU Catchers Consortium, October 4, 2017, Columbus, OH: The
Ohio State University, downloaded from!stella.report!10/05/2017.
Cook, R. I. and Long, B. A. (2021). Building and revising adaptive capacity sharing for
technical incident response: A case of resilience engineering. Applied Ergonomics 90,
DOI: 10.1016/j.apergo.2020.103240.
Void Report (2024). Exploring the Unintended Consequences of Automation in Software.
https://www.thevoid.community/report-2024
Aviation Accidents with sensor trouble & automation misbehavior (in addition to Lion Air 610
and Ethiopian Airlines 302)
BEA—Bureau d’Enquêtes et d’Analyses pour la Sécurité de l’Aviation Civil (2008). Report
Accident on 27 November 2008 off the coast of Canet-Plage (66) to the Airbus
A320-232 registered D-AXLA operated by XL Airways Germany.
Dutch Safety Board (2010). Turkish airlines, crashed during approach, Boeing 737-800, TC-JGE
Amsterdam Schiphol Airport, 25 February 2009 (D. S. Board Ed.). The Hague, The
Netherlands: Dutch Safety Board.
BEA. (2012). Final report on the accident on 1st June 2009 to the Airbus A330-203 registered
F-GZCP, operated by Air France flight AF 447 Rio de Janeiro - Paris. Paris: Bureau
d'Enquetes et d'Analyses pour la securité de l'aviation civile, Ministere de l'Ecologie,
du Developement durable, des Transports et du Logement.
of 9 13
Wrong and Strong
ATSB Australian Transport Safety Bureau Transport Safety Report (2011). In-flight upset 154
km west of Learmonth, WA 7 October 2008 VH-QPA Airbus A330-303 Aviation
Occurrence Investigation AO-2008-070.
Open source material related to Boeing 737 MAX Lion Air 610 and Ethiopian Airlines 302
accidents
US Congress (2020). Final Committee Report on the Design, Development & Certification of
the Boeing 737 Max, September 2020. https://transportation.house.gov/imo/media/
doc/
2020.09.15%20FINAL%20737%20MAX%20Report%20for%20Public%20Release.pdf
US Congress (2020). Transcript of Interview of Keith Leverkuhn. September Tuesday, May 19,
2020 Washington, D.C. https://transportation.house.gov/imo/media/doc/
FINAL%20Keith%20Leverkuhn%20(Boeing)
%20Transcript%20and%20Exhibits%20(9.9.20).pdf
US Congress (2020). Boeing Record Sets 1 & 2. https://transportation.house.gov/imo/media/
doc/Boeing%20Records%20First%20Set%20for%20Public%20Release.pdf https://
transportation.house.gov/imo/media/doc/
Boeing%20Records%20Second%20Set%20for%20Public%20Release.pdf
US Congress (2019). Testimony of Dennis Muilenburg, Boeing, CEO. Full Committee Hearing
on “The Boeing 737 MAX: Examining the Design, Development, and Marketing of the
Aircraft”. October 30, 2019. https://transportation.house.gov/imo/media/doc/
Muilenburg%20Testimony.pdf. https://www.congress.gov/116/meeting/house/
110066/documents/CHRG-116hhrg38282.pdf
US Congress (2019). Testimony of John Hamilton Vice President and Chief Engineer, Boeing
Commercial Airplanes. Full Committee Hearing on “The Boeing 737 MAX: Examining
the Design, Development, and Marketing of the Aircraft”. October 30, 2019. https://
www.congress.gov/116/meeting/house/110066/witnesses/HHRG-116-PW00-Wstate-
HamiltonJ-20191030.pdf
Aircraft Accident Investigation Preliminary Report Ethiopian Airlines Group B737-8 (MAX)
Registered ET-AVJ 28 NM South East of Addis Ababa, Bole International Airport March
10, 2019. http://www.ecaa.gov.et/Home/wp-content/uploads/2019/07/Preliminary-
Report-B737-800MAX-ET-AVJ.pdf.
National Transportation Safety Board, Safety Recommendation Report: Assumptions Used in
the Safety Assessment Process and the Effects of Multiple Alerts and Indications on
Pilot Performance, ASR-19-01 (Washington, DC, 2019), https://www.ntsb.gov/
investigations/AccidentReports/Reports/ASR1901.pdf.
Joint Authorities Technical Review, Boeing 737 MAXFlight Control System: Observations,
Findings, and Recommendations (Washington, DC, 2019), https://www.faa.gov/news/
media/attachments/Final_JATR_Submittal_to_FAA_Oct_2019.pdf.
Emails from Mark Forkner, Boeing, October 5, 2015, March 30, 2016, November 3, 2016,
November 9, 2016, January 17, 2017, February 9, 2018. https://
of 10 13
Wrong and Strong
www.commerce.senate.gov/services/files/40B117EA-4C4F-496B-91F0-
D7A4816DF71E
FAA emergency Airworthiness Directive 2018-23-51 https://www.govinfo.gov/content/pkg/
FR-2018-12-06/pdf/2018-26365.pdf
FAA Preliminary Summary of the FAA’s Review of the Boeing 737 MAXReturn to Service of the
Boeing 737 MAXAircraft, Version 1, August 3, 2020.
FAA Summary of the FAA’s Review of the Boeing 737 MAX Return to Service of the Boeing
737 MAXAircraft, Version 1, November 18, 2020.
U.S. Department of Transportation, Federal Aviation Administration, Preliminary Summary of
the FAA’s Review of the Boeing 737 MAX, version 1 (Washington, DC, 2020), https://
www.faa.gov/news/media/attachments/737-MAX-RTS-Preliminary-Summary-v-1.pdf.
Joint Authorities Technical Review, Boeing 737 MAXFlight Control System: Observations,
Findings, and Recommendations (Washington, DC, 2019). https://www.faa.gov/news/
media/attachments/Final_JATR_Submittal_to_FAA_Oct_2019.pdf.
National Transportation Safety Committee (NTSC) (2019). Komite Nasional Keselamatan
Transportasi of Indonesia, Final Aircraft Accident Investigation Report,
KNKT.18.10.35.04. 25 October 2019. PT. Lion Mentari Airlines Boeing 737-8 (MAX);
PK-LQP Tanjung Karawang, West Java Republic of Indonesia, 29 October 2018.
https://www.congress.gov/116/meeting/house/110066/documents/HHRG-116-
PW00-20191030-SD002.pdf
References:Flight Deck Automation Studies (samples) prior to 2018/2019 737MAX accidents
Wiener, E. L. (1989). Human factors of advanced technology ('glass cockpit') transport aircraft
(NASA CR-117528). NASA Ames Research Center, Moffett Field, CA: https://human-
factors.arc.nasa.gov/publications/HF_AdvTech_Aircraft.pdf
Sarter, N. B., & Woods, D. D. (1992). Pilot Interaction With Cockpit Automation I: Operational
Experiences With the Flight Management System. International Journal of Aviation
Psychology, 2(4), 303-322.
Sarter, N. B., & Woods, D. D. (1994). Pilot Interaction With Cockpit Automation II: An
Experimental Study of Pilots' Model and Awareness of the Flight Management System.
International Journal of Aviation Psychology, 4(1), 1-29.
FAA (1996). Federal Aviation Administration Human Factors Team Report on The Interfaces
Between Flightcrews and Modern Flight Deck Systems. Washington, DC: Federal
Aviation Administration. https://www.tc.faa.gov/its/worldpac/techrpt/hffaces.pdf
Sarter, N. B., & Woods, D. D. (1997). Teamplay with a powerful and independent agent: A
corpus of operational experiences and automation surprises on the Airbus A320.
Human Factors, 39, 553-569.
Billings, C. E. (1997). Aviation automation: The search for a human-centered approach.
Mahwah, N.J.: Lawrence Erlbaum Associates.
Singer, G., & Dekker, S. W. A. (2000). Pilot Performance During Multiple Failures: An Empirical
Study of Different Warning Systems. Transportation Human Factors, 2(1), 63-77.
of 11 13
Wrong and Strong
Sarter, N. B. and Woods, D.D. (2000). Team Play with a Powerful and Independent Agent: A
Full Mission Simulation. Human Factors, 42, 390—402.
Woods, D. D., & Sarter, N. B. (2000). Learning from Automation Surprises and Going Sour
Accidents. !In N. Sarter and R. Amalberti (Eds.), Cognitive Engineering in the Aviation
Domain, Erlbaum, Hillsdale NJ, pp. 327-354.
Sarter, N. B. (2002). Multimodal information presentation in support of human-automation
communication and coordination. In E. Salas (Ed.), Advances in human performance
and cognitive engineering research: Automation (pp. 13–35). Elsevier Science/JAI
Press. https://doi.org/10.1016/S1479-3601(02)02004-0
Burian, B., Barshi, I., & Dismukes, K. (2005). The Challenge of Aviation Emergency and
Abnormal Situations (No. NASA/TM-2005-213462). Moffett Field, CA: NASA.
Casner, S. M., Geven, R. W., & Williams, K. T. (2013). The Effectiveness of Airline Pilot Training
for Abnormal Events. Human Factors: The Journal of the Human Factors and
Ergonomics Society, 55(3), 477–485. https://doi.org/10.1177/0018720812466893
FAA. (2013). Operational use of flight path management systems: Final report of the
performance-based operations aviation rulemaking committee/commercial aviation
safety team flight deck automation working group (Report of the PARC/CAST Flight
Deck Automation WG) (F. A. Administration Ed.). Washington, DC: Federal Aviation
Administration. https://www.faa.gov/sites/faa.gov/files/aircraft/air_cert/design_approvals/
human_factors/OUFPMS_Report.pdf
Rankin, A., Woltjer, R., & Field, J. (2016). Sensemaking following surprise in the cockpit—A re-
framing problem. Cognition, Technology & Work, 18(4), 623–642. https://doi.org/
10.1007/s10111-016-0390-2
Landman, A., Groen, E. L., Van Paassen, M., Bronkhorst, A. W., & Mulder, M. (2017). Dealing
with unexpected events on the flight deck: A conceptual model of startle and surprise.
Human Factors, 59(8), 1161–1172.
Holbrook, J. B., Stewart, M. J., Smith, B. E., Prinzel, L. J., Matthews, B.L., Avrekh, I., Cardoza,
C.T., Ammann, O.C., Adduru, B., & Null, C.(2019). Human performance contributions to
safety in commercial aviation. NASA Langley Research Center. NASA/TM2019-220417.
of 12 13
