Content uploaded by John Bicknell
Author content
All content in this area was uploaded by John Bicknell on Oct 31, 2019
Content may be subject to copyright.
(Submitted to the InfraGard National Journal on 10/20/2019)
A Process Mining Approach for Protecting Self-Organizing
Critical Infrastructure Ecosystems
Authors:
John W. Bicknell, Jr.
CEO, More Cowbell Unlimited, Inc.
501 4th Street Unit 795, Lake Oswego OR 97034
Email: john@morecowbellunlimited.com
Werner G. Krebs, Ph.D.
CEO, Acculation, Inc.
5482 Wilshire Blvd., #342, Los Angeles, CA 90036
E-mail: wkrebs@acculation.com
Abstract: Self-organized ecosystems are found throughout the natural world, including
critical infrastructure. Self-organized criticality is the tendency of some systems to naturally
self-organize into critical states far from equilibrium and which are barely stable. Small
changes or stimuli can result in cascading chain reactions which may have unwanted or
devastating effects on economies or societies. Process technologies offer an expedient
method for simulating self-organized critical infrastructure ecosystems, understanding
criticality within these ecosystems, and adopting mitigation strategies analogous to controlled
avalanches. Several case studies from broad domains present naturally occurring criticality as
well deliberate ecosystem attacks designed to induce criticality or trigger catastrophic events.
Key Words: Critical Infrastructure, Process Mining, Self-organized criticality, Resilience
1
(Submitted to the InfraGard National Journal on 10/20/2019)
Introduction
Critical infrastructures and the larger societies that sustain them are engaged in never-ending
internal and external competition, which creates the conditions for self-organized criticality.
By analogy to Lewis Carroll, the “Red Queen Hypothesis” (Valen 1973; “Red Queen
Hypothesis” 2019) suggests an explanation where these assets must constantly evolve merely
to stay competitive. Indeed, many specific types of critical infrastructure (Lewis 2010;
Dobson 2015) and geo-political dynamics, such as financial markets, power grids, public
health/hospital systems, nuclear energy systems, Internet activity, and strategic arms races
(“Red Queen Hypothesis” 2019) are known to exhibit aspects of self-organized criticality.
In this paper, critical infrastructure is described as vast, interconnected dissipative self-
organizing ecosystems. Next, process technologies utilizing petabyte scalable explainable
artificial intelligence (AI) (“Explainable Artificial Intelligence (XAI)” 2016; Gandhi 2019;
“AI Black Box Horror Stories – When Transparency Was Needed More than Ever” 2019) are
discussed which offer, potentially, a novel and expedient method for understanding these
ecosystems temporally. After that, a framework for policy planners is presented which
combines process mining and process simulations in order to mitigate and monitor critical
infrastructure vulnerabilities, along with some case studies.
Dissipative Ecosystems and Self-Organized Criticality
1
The Department of Homeland Security (DHS) is the United States’ agency responsible to
“provide strategic guidance, promote a national unity of effort, and coordinate the overall
Federal effort to promote the security and resilience of the Nation's critical infrastructure”
(“Presidential Policy Directive -- Critical Infrastructure Security and Resilience” 2013).
There are sixteen critical infrastructure sectors whose assets, systems, and networks--whether
physical or virtual--are considered so vital that their incapacitation or destruction would have
a debilitating effect on security, national economic security, national public health or safety,
or any combination thereof (“Critical Infrastructure Sectors” 2013). DHS operates at the
intersection of the Federal Government, state and local governments, the private sector,
international partners, law enforcement, intelligence, and defense communities--enabling
network defenders to “share technical information at machine speed (Manfra 2019).”
Many people think of processes as intentional groups of activities which serve profit seeking
businesses or mission driven government agencies. While this is true, order and complex
processes are self-organizing with no human intention (Whitehead 1979; Prigogine, Nicolis,
and Babloyantz 1972; Prigogine, Stengers, and Toffler 1984; Hidalgo 2015). Processes
underlie all complex naturally occurring phenomena; indeed, the Earth is a process
(Whitehead 1979). Humans have famously been defined as “complexes [of] processes (Fuller
1970).” Self-organizing ecosystems have defining dynamics and characteristics relevant to
critical infrastructure and homeland security.
Self-organizing systems emerge naturally from chaos--or ever increasing entropy (Prigogine,
Nicolis, and Babloyantz 1972). In order for self-organizing ecosystems to form and persist,
they feed continually off of the energy, matter, or information contained in the surrounding
environment while transferring information and disorder to the surrounding environment, as
1
For a comprehensive discussion about dissipative systems and self-organized criticality in a homeland
security and critical infrastructure context, the reader is referred to Dobson, “Entropy and Self-Organization—
an Open System Approach to the Origins of Homeland Security Threats.”
2
(Submitted to the InfraGard National Journal on 10/20/2019)
well (Dobson 2015). They are called, aptly, dissipative systems. The idea of a dissipative
system is a foundational concept to understanding complex systems.
Self-organized criticality, another characteristic of dissipative systems, also has a major
impact on systems related to homeland security. Self-organized criticality is the natural
tendency of some systems to self-organize into a critical state far from equilibrium
(Prigogine, Nicolis, and Babloyantz 1972; Hidalgo 2015; Dobson 2015) and which are barely
stable (Bak, Tang, and Wiesenfeld 1987). Forest wildfires, power grids, the Internet, nuclear
energy systems, and financial systems all show self-organizing criticality. Small exogenous
additions to the ecosystem or changes within the ecosystem can result in cascading chain
reactions of different sizes (Kron and Grund 2009); however, ecosystem-level and node-level
measures of criticality enable monitoring and even control (Cajueiro and Andrade 2010).
Evolutionary studies of criticality suggest that complex systems compete with one another
and require increasing amounts of energy in order to gain or maintain advantage (“Red Queen
Hypothesis” 2019).
Self-organized criticality makes critical infrastructure vulnerable to both natural internal
fragility (Taleb 2014) and subtle hybrid warfare tactics such as weaponized interdependence
(Farrell and Newman 2019) in addition to other forms of cyber and information threats.
National security considerations dictate that policy planners adopt techniques to defuse self-
organized criticality by understanding these systems and adopting mitigating strategies, such
as solution analogues to controlled avalanches (Cajueiro and Andrade 2010; Lewis 2010;
“Avalanche Control” 2019) or hazard reduction burning in forests (“Controlled Burn” 2019).
Understanding these systems sufficiently to assess, quantify, and defuse criticality typically
requires sophisticated models and simulations.
In some critical infrastructure sectors, regulators already gather data to model these assets
(Cook 2017; Chen 2018; FINRA 2010; 2018a; 2018b; Bicknell and Krebs 2019b). For other
sectors, especially new or emerging types of critical infrastructure, such models either do not
exist, or at least are not widely-known to exist. Therefore, generalized techniques for rapidly
prototyping models and process simulations of critical infrastructure assets, especially
techniques that employ AI to reduce development costs, would appear to be needed.
Societies and the critical infrastructure which support them may be viewed through an
antagonistic co-evolutionary lens with relevance for homeland security and defense. All
aspects of critical infrastructure ecosystems may be modeled contextually and monitored
continually with temporal process technologies.
Process Technologies
Process technologies exist today which are an effective means to analyze time-domain
relationships. As Figure 1 suggests, process mining is inherently interdisciplinary--providing
additive capability to data scientists, process improvement experts, operations researchers,
organizational design practitioners, and national security analysts.
Process mining was conceived originally in a business context. Data science and management
science practitioners, Figure 1, use process mining to speed the discovery of business process
models while improving accuracy and preparing organizations for digital transformation
(Bicknell 2019). It is a highly versatile technique with vast utility beyond corporate process
improvement. Combined with other analytic disciplines such as game theory and complex
systems theory (again, Figure 1), process mining may inform critical infrastructure defense,
3
(Submitted to the InfraGard National Journal on 10/20/2019)
information warfare vulnerability detection, network fragility and resilience analysis,
geosurveillance, and many other domain analyses.
The goal of process mining is to turn event data into insights and actions (Aalst 2016).
Traditional process mining analyses use organizational enterprise resource planning system
logs to obtain event logs. For example, a talent hiring process might include the following
activities: Create Job Requisition; Post Job Announcement; Conduct Phone Screening;
Conduct Phone Interview; Conduct On-site Interview; Extend Offer; Accept Offer.
Process technologies elucidate ecosystem information flows, decision-making probabilities
and temporal measures. Machine-readable process outputs and models of complex natural
phenomena enable numerous applications. As the name implies, process mining AIs “mine”
data and surface (e.g. Markov or Bayesian) models of decision-making processes from
various input formats with no a priori knowledge. Process mining is also a human-
understandable, human-verifiable, and human-explainable AI, which is an increasingly
important consideration (“AI Black Box Horror Stories – When Transparency Was Needed
More than Ever” 2019) .
Structured system tables are suitable for process mining with relatively little pre-processing;
however, any structured, semi-structured, or unstructured data sources which chronicle events
is usable--cyber security logs, Internet of Things networks, satellite on-orbit characteristics,
hyperspectral imaging, social media data, etc. Three pieces of information are needed to
discover processes; additional features enrichen the analysis:
• Case ID: An identifier that represents a specific execution of a process.
• Activity: One of several steps performed within a process. For example, cyber exploit
attempts or corporate actions contained in regulatory documents.
• Time Stamp: This orders the activities within each case and enables sophisticated
modeling.
Figure 2 contains a trivial example where an event log is converted into a temporal process
model. Process cases 1 through 3 all contain the same activities, labeled “A” through “E.” In
Case 1, the activities happen in natural order. In Case 2, Activity “C” precedes “B.” Finally,
4
(Submitted to the InfraGard National Journal on 10/20/2019)
in Case 3, Activity “D” is repeated before concluding with Activity “E.” The discovered
process accounts for these process variations. Real world processes are significantly more
complicated.
Process models are temporal representations of events organized by process case. Software
outputs may include machine-readable business process model notation (BPMN) diagrams,
various Markov transition probability matrices, descriptive statistics associated with the
process, capacity estimates for process activities, and distribution probabilities associated
with activity state arrival data.
Framework and Case Studies
This section presents a red team-like framework for policy planners plus some motivating use
cases. DHS and critical infrastructure analysts may use this technique to understand self-
organized criticality thresholds, infer vulnerabilities which may be exploited by clever
adversaries, and apply controlled avalanches which lessen catastrophic event likelihood. Red
Team projects are intensive self-examinations from an adversary’s perspective and are a well-
known technique to build up defensive capabilities by having friendly forces simulate enemy
tactics to discover weaknesses (Zenko 2015). Naturally self-organizing critical infrastructure
ecosystems interact dynamically in the borderless, interconnected environment of the
Information Age. The resiliency, security and defense challenge is, therefore, not limited to
State- and non-State actors; natural laws of physics may also be considered adversarial in the
context of critical infrastructure stability. Macro-level critical infrastructure red team analyses
model, simulate, and understand vulnerabilities and inform homeland security and defense
efforts.
Process simulations of self-organizing criticality typically first require understanding of the
industrial process involved, such as through a BPMN process flow diagram (Bequette 2003;
“Process Simulation” 2017). Historically, such process flow diagrams were created through
extremely labor-intensive methods. However, today the explainable AI technique of process
mining can deduce process flow diagrams by analyzing event log data (Aalst 2016),
potentially greatly accelerating simulation development. Patent-pending implementations of
process mining combine Simple Queueing Theory with Monte Carlo simulations as well as
5
(Submitted to the InfraGard National Journal on 10/20/2019)
specialized data ingestion AI to enable detailed prototype simulations of complex industrial
processes to be automatically modeled and generated from event log data (Bicknell 2019;
Bicknell and Krebs 2019a; 2019b; 2019c; 2019d; Application 2019a; Application 2019b).
More Cowbell Unlimited’s process mining platform uses AI heuristic filters combined with
Simple Queueing Theory provide tools which allow users to conceptualize event logs as
various types of Markov models, including time-inhomogeneous, time-homogeneous, and
Continuous-Time Markov Chains (Bicknell and Krebs 2019d). Other researchers have
proposed Continuous Time Markov Chain and other Markov processes as models studying
for self-organizing criticality (Mao and Zhong 2007; Swart 2014). Statistics such as
relaxation times and range of time scales can be used to inspect the resulting rapidly-
prototyped models for self-organizing criticality-like behavior, such as a wide range of time-
spans for the different processes together with rapid relaxation times (Diamond and Zhang
2016). Thus, inspection of the standard statistics associated with time-homogenized Markov
matrices or Continuous Time Markov Chains can provide clues to identifying ecosystems
which are at or close to criticality. These various rapidly prototyped models can then be
further rapidly prototyped into automatically generated agent-based simulations and cellular
automata. Additional linkages between the automatically prototyped agent-based or cellular
automata models can then be manually added, as needed. When appropriate, with certain
types of self-organizing criticality, the individual cellular automata (or simulation agents)
have an associated geospatial interpretation, in which case real-world phenomena can
potentially be simulated and assigned GPS coordinates, indicating intervention points to
prevent criticality (such as controlled avalanches).
This framework may be deployed throughout critical infrastructure ecosystems and even as a
tool to understand social ecosystems.
Weaponized Interdependence Case Study
Weaponized interdependence of critical infrastructure networks is possible through economic
sanctions, trade policies, and reduced multilateral interstate negotiations (Farrell and
Newman 2019). States with political authority over the central nodes in the international
networked structures through which money, goods, and information travel are uniquely
positioned to impose costs on others. If they have appropriate domestic institutions, they can
weaponize networks to gather information or choke off economic and information flows,
discover and exploit vulnerabilities, compel policy change, and deter unwanted actions
(Farrell and Newman 2019).
To mitigate against weaponized interdependence, policy planners must first understand
critical global economic flows. Ideally, policy makers would use data-driven modeling and
simulation tools that can automatically update and keep pace with the ever-changing patterns
of the global economy. Process mining, and process simulation tools based off of process
mining, are one form of AI that can help automatically model global economic flows and
probe for weaponized interdependence.
Corporate Dynamics Case Study
Process models of email subject line data suggest ways information may be used as a
maneuver element in a larger cyber or information operations kill chain against critical
infrastructure firms (Bicknell and Krebs 2019d). Corporate email systems as a proxy for
corporate worker tasks have previously been studied as a model for self-organizing criticality
(Swart 2014). By virtue of their size and scope of operations, many Fortune 500 companies
6
(Submitted to the InfraGard National Journal on 10/20/2019)
and large corporations are considered critical infrastructure. The adversarial arms-race
between large corporations would suggest they exhibit naturally occurring self-organized
criticality. Customers depend on reliable services provided by large corporations, suggesting
self-organized criticality between these corporations and the large United States economy
(Sverdlik 2018; Daoudi 2019). For example, the failure of Enron severely undermined
investor confidence in United States regulators and markets, resulting in a market sell-off, as
well as the eventual failure of Arthur-Anderson, one of the former Big 5 Accounting firms
that play a vital role in auditing the United States markets (“Pub. No. GAO-03-864, Public
Accounting Firms: Mandated Study on Consolidation and Competition” 2003; McLean and
Elkind 2004; Cunningham 2006).
Tools that can help corporate managers, auditors, and regulators simulate corporate processes
as well as corporate interactions with the larger economy can potentially mitigate self-
organized criticality issues around large corporations and their global engagements. More
Cowbell Unlimited developed custom AI tools to ingest Enron email data to create process
models and process simulations (Bicknell and Krebs 2019d), a potential first step towards
understanding and mitigating corporate self-organized criticality.
Adversarial Propaganda Case Study
Information warfare detection and defense are active areas of process mining research
(Bicknell and Krebs 2019a; 2019b; 2019c). Complex feedback loops between social media
and critical infrastructure are already known. For example, some Wall Street algorithms trade
the financial markets based on Twitter sentiment or news items, which are, in turn, influenced
by those same financial markets (Sheridan et al. 2012; “Big Data Analytics: Articles, Movies,
Songs Robo-Written by Computer?” 2014). Tweets, “fake” or deceptive news items as well
as algorithmically-manipulative financial filings have been known to move markets (“SEC
Charges: False Tweets Sent Two Stocks Reeling in Market Manipulation” 2015; Carr 2013;
Arnoldi 2016).
Financial markets are a known example of critical infrastructure that displays self-organizing
criticality (Bak, Tang, and Wiesenfeld 1987; Cajueiro and Andrade 2010). Links between
social media and suspected adverse and potentially illicit influence on critical democratic
institutions have dominated mainstream media in recent years (Mueller 2019; “Russian
Active Measures Campaigns and Interference in the 2016 U.S. Election” 2019; Krasodomski-
Jones et al. 2019; Eustachewich 2019; Howard 2018). Both social media in general and
purveyors of adverse influence are believed to rapidly evolve their tactics, often themselves
employing AI (Waltzman 2017; Singer and Brooking 2018; Jones 2019). Automatically
generated process models and process simulation AI tools to both track illicit influence
operations as well as monitor the larger impact of potential criticality would seem to be
useful tools. Recently, More Cowbell Unlimited partnered with Texas A&M University to
add AI twitter propaganda monitoring capabilities to its cloud SaaS process mining
dashboard. ISIS Terrorist bot-driven propaganda, which would otherwise be difficult to
detect, latent within Twitter data were elucidated with a derived hidden Markov model
(Bicknell and Krebs 2019c).
Earth Pattern of Life Operations Case Study
AI is accelerating technological advancement (“Singularity University” n.d.), and in an
increasingly chaotic world AI tools should be used to rapidly prototype process simulations
that can defuse dangerous criticality. Process mining AI, and automated simulation
7
(Submitted to the InfraGard National Journal on 10/20/2019)
generation from process mining results, provide one form of rapid simulation prototyping
capability. Recently, More Cowbell Unlimited partnered with satellite AI provider
Spectrabotics to develop geospatial dashboards that ingest drone or satellite imagery, convert
them into event logs for process mining, automatically prototype simulations, and integrate
the resulting information into a geospatial dashboard to facilitate rapid action by on-the-
ground-teams.
The Federal government and geo-intensive industries are using satellite data to monitor
critical infrastructure assets, including agriculture, oil pipelines, electrical grids, as well as
environmental conditions that may have a large scale impact (“Improve the Quality of Your
Crop” 2017; “Vegetation Clearances Automatically Analyzed Across a Power Line Network”
2018; “Large Data Aggregation from Small Satellites to Determine Patterns of Life
Modifications” 2019). Natural environments such as wildfire-prone areas and snow-packed
mountains are also known to exhibit dangerous self-organized criticality (Bak, Tang, and
Wiesenfeld 1987; Lewis 2010; Cajueiro and Andrade 2010; Dobson 2015). Simulations
informed by real-world data are one way to detect and find ways to defuse criticality. For
example, simulations of mountain snowpacks can be used to plan controlled avalanches.
Conclusion
Resiliency has many definitions which vary from one end of the spectrum as a system’s
ability to recover from a significant stressful event to the other end as a system’s ability to
adapt to ever-changing environments (Fisher 2015). This paper examined critical
infrastructures as dissipative ecosystems which exhibit self-organized criticality. Systems at
or near criticality may, with very little warning or perturbation, fail with chain reactions that
cascade throughout the system and even into adjacent ecosystems. This natural phenomenon
may be exacerbated in the Information Age as nation states compete for dominance.
Developing methods to understand self-organized criticality dynamics empirically and
dissipate energy in a controlled manner is important for critical infrastructure ecosystem
resiliency and protection from natural internal fragilities and deliberate attacks.
Process technologies are proving to be useful in a critical infrastructure and national security
context--especially for discovering emergent process ecosystems from semi-structured data.
Temporal process models offer an expedient method for simulating self-organized critical
infrastructure ecosystems in order to better understand criticality and mitigation strategies.
Future research should test the hypotheses presented in this paper with goals to induce
controlled avalanches in simulated ecosystems and dissipate energy as moderate sized events.
If successful, controlled avalanches--like controlled forestry burns--may reduce the likelihood
of large critical infrastructure catastrophes, improve resilience, and mitigate information-
related vulnerabilities.
Acknowledgements
The authors are grateful to Mary Lasky of the Johns Hopkins University Applied Physics
Laboratory for her invaluable comments on earlier versions of this article.
8
(Submitted to the InfraGard National Journal on 10/20/2019)
References
Aalst, Wil M. P. van der. 2016. Process Mining: Data Science in Action. 2nd ed. 2016
edition. New York, NY: Springer.
“AI Black Box Horror Stories – When Transparency Was Needed More than Ever.” 2019.
Open Data Science - Your News Source for AI, Machine Learning & More (blog).
October 2, 2019. https://opendatascience.com/ai-black-box-horror-stories-when-
transparency-was-needed-more-than-ever/.
Arnoldi, Jakob. 2016. “Computer Algorithms, Market Manipulation and the
Institutionalization of High Frequency Trading.” Theory, Culture & Society 33 (1):
29–52. https://doi.org/10.1177/0263276414566642.
“Avalanche Control.” 2019. In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Avalanche_control&oldid=885052353.
Bak, Per, Chao Tang, and Kurt Wiesenfeld. 1987. “Self-Organized Criticality: An
Explanation of the 1/f Noise.” Physical Review Letters 59 (4): 381–84.
https://doi.org/10.1103/PhysRevLett.59.381.
Bequette, B. Wayne. 2003. Process Control: Modeling, Design and Simulation. 1 edition.
Upper Saddle River, N.J: Prentice Hall.
Bicknell, John W. 2019. “Process Mining Technologies.” ORMS Today 46 (5).
https://doi.org/10.1287/orms.2019.05.01.
Bicknell, John W, and Werner G Krebs. 2019a. “Process Mining: The Missing Piece in
Information Warfare.” ResearchGate, February.
https://doi.org/10.13140/RG.2.2.23584.94722/1.
———. 2019b. “FOCAL Information Warfare Defense Standard.” ResearchGate, June.
https://www.doi.org/10.13140/RG.2.2.12672.07687.
———. 2019c. “Detecting Botnet Signals Using Process Mining.” Manuscript submitted for
publication.
———. 2019d. “Process Mining Organization Email Data and National Security
Implications.” Manuscript submitted for publication.
———. Application 2019a. Methods and Systems for Estimating Process Capacity. United
States, filed May 2, 2019.
———. Application 2019b. Methods and Systems for Inferring Behavior and Vulnerabilities
from Process Models. United States, filed June 13, 2019.
“Big Data Analytics: Articles, Movies, Songs Robo-Written by Computer?” 2014. Acculation
(blog). March 19, 2014. https://www.acculation.com/blog/2014/03/18/big-data-
analytics-robo-generated-content/.
Cajueiro, Daniel O., and R. F. S. Andrade. 2010. “Controlling Self-Organized Criticality in
Sandpile Models.” Physical Review E 81 (1): 015102.
https://doi.org/10.1103/PhysRevE.81.015102.
Carr, David. 2013. “Using Twitter to Move the Markets.” The New York Times, October 6,
2013, sec. Business. https://www.nytimes.com/2013/10/07/business/media/using-
twitter-to-move-the-markets.html.
Chen, James. 2018. “Order Audit Trail System - OATS.” Investopedia. May 22, 2018.
https://www.investopedia.com/terms/o/order_audit_trail_system.asp.
“Controlled Burn.” 2019. In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Controlled_burn&oldid=921654267.
Cook, Robert. 2017. “Equity Market Surveillance Today and the Path Ahead | FINRA.Org.”
September 20, 2017. https://www.finra.org/newsroom/speeches/092017-equity-
market-surveillance-today-and-path-ahead.
9
(Submitted to the InfraGard National Journal on 10/20/2019)
“Critical Infrastructure Sectors.” 2013. Department of Homeland Security. March 5, 2013.
https://www.dhs.gov/cisa/critical-infrastructure-sectors.
Cunningham, Lawrence. 2006. “Too Big to Fail: Moral Hazard in Auditing and the Need to
Restructure the Industry Before It Unravels.” GW Law Faculty Publications & Other
Works, January. https://scholarship.law.gwu.edu/faculty_publications/504.
Daoudi, Mehdi. 2019. “2019’s Increasingly Fragile Internet: How Can Businesses Respond?”
Forbes. September 18, 2019.
https://www.forbes.com/sites/forbestechcouncil/2019/09/18/2019s-increasingly-
fragile-internet-how-can-businesses-respond/.
Diamond, P H, and Y Zhang. 2016. “1/ f Noise and Self-Organized Criticality.” Physics 235
Lecture Notes. University of California San Diego.
https://courses.physics.ucsd.edu/2019/Spring/physics235/1_f_noise_and_self_organiz
ed_criticality.pdf.
Dobson, Thomas Kirwan. 2015. “Entropy and Self-Organization—an Open System Approach
to the Origins of Homeland Security Threats.” Thesis, Monterey, California: Naval
Postgraduate School. https://calhoun.nps.edu/handle/10945/45839.
Eustachewich, Lia. 2019. “Russian Trolls Blamed for Spreading Anti-Vaccination
Propaganda.” New York Post, February 15, 2019.
https://nypost.com/2019/02/15/russian-trolls-blamed-for-spreading-anti-vaccination-
propaganda/.
“Explainable Artificial Intelligence (XAI).” 2016. August 10, 2016.
https://www.darpa.mil/program/explainable-artificial-intelligence.
Farrell, Henry, and Abraham L. Newman. 2019. “Weaponized Interdependence: How Global
Economic Networks Shape State Coercion.” International Security 44 (1): 42–79.
https://doi.org/10.1162/isec_a_00351.
FINRA. 2010. “SR-FINRA-2010-044 | FINRA.Org.” 2010.
http://www.finra.org/industry/rule-filings/sr-finra-2010-044.
———. 2018a. “FINRA Handles Record Volume of Market Activity through First Six
Months of 2018 | FINRA.Org.” 2018. http://www.finra.org/newsroom/2018/finra-
handles-record-volume-market-activity-through-first-six-months-2018.
———. 2018b. 13: How the Cloud and Machine Learning Have Transformed Market
Surveillance | Episode 13. Vol. 13. FINRA UNSCRIPTED.
https://embed.simplecast.com/d203ed4a?color=f5f5f5.
Fisher, Len. 2015. “More than 70 Ways to Show Resilience.” Nature 518 (7537): 35–35.
https://doi.org/10.1038/518035a.
Fuller, R. Buckminster. 1970. I Seem to Be a Verb: Environment and Man’s Future. 1st
edition. Bantam Books.
Gandhi, Preet. 2019. “Explainable Artificial Intelligence.” January 2019.
https://www.kdnuggets.com/2019/01/explainable-ai.html.
Hidalgo, Cesar. 2015. Why Information Grows: The Evolution of Order, from Atoms to
Economies. Basic Books.
Howard, Jacqueline. 2018. “Why Russian Trolls Stoked US Vaccine Debates - CNN.” CNN.
https://www.cnn.com/2018/08/23/health/russia-trolls-vaccine-debate-
study/index.html?utm_source=twCNN&utm_medium=social&utm_term=image&utm
_content=2018-08-23T21%3A12%3A56.
“Improve the Quality of Your Crop.” 2017. Spectrabotics (blog). June 6, 2017.
https://www.spectrabotics.com/precision-agriculture/.
Jones, Marc Owen. 2019. “The Gulf Information War | Propaganda, Fake News, and Fake
Trends: The Weaponization of Twitter Bots in the Gulf Crisis.” International Journal
of Communication 13 (0): 27.
10
(Submitted to the InfraGard National Journal on 10/20/2019)
Krasodomski-Jones, Alex, Josh Smith, Elliot Jones, Ellen Judson, and Carl Miller. 2019.
“Information Operations in the Digital Age.” Demos.
https://www.academia.edu/39161783/Information_Operations_in_the_Digital_Age.
Kron, Thomas, and T. Grund. 2009. “Society as a Self-Organized Critical System.” Text.
2009.
https://www.ingentaconnect.com/contentone/imp/chk/2009/00000016/f0020001/art00
005.
“Large Data Aggregation from Small Satellites to Determine Patterns of Life Modifications.”
2019. 2019. https://hyperspacechallenge.com/large-data-aggregation-from-small-
satellites-to-determine-pattern-of-life-modifications/.
Lewis, Ted. 2010. “Cause-and-Effect or Fooled by Randomness?” Homeland Security Affairs
6 (1). https://www.hsaj.org/articles/93.
Manfra, Jeanette. 2019. Role of the United States Government in Securing the Nation’s
Internet Architecture. Washington DC.
https://docs.house.gov/meetings/GO/GO06/20190910/109903/HHRG-116-GO06-
Wstate-ManfraJ-20190910.pdf.
Mao, De Tao, and Yisheng Zhong. 2007. “Markov Chain Hidden behind Power Law
Mechanism of Self-Organized Criticality.” ArXiv:0709.2404 [Cond-Mat], September.
http://arxiv.org/abs/0709.2404.
McLean, Bethany, and Peter Elkind. 2004. The Smartest Guys in the Room: The Amazing
Rise and Scandalous Fall of Enron. Reprint edition. New York: Portfolio Trade.
Mueller, Robert S. 2019. “Report On The Investigation Into Russian Interference In The 2016
Presidential Election.” https://www.hsdl.org/?abstract&did=824221.
“Presidential Policy Directive -- Critical Infrastructure Security and Resilience.” 2013. The
White House. https://obamawhitehouse.archives.gov/the-press-
office/2013/02/12/presidential-policy-directive-critical-infrastructure-security-and-
resil.
Prigogine, Ilya, Gregoire Nicolis, and Agnes Babloyantz. 1972. “Thermodynamics of
Evolution.” Physics Today 25 (11): 23–28. https://doi.org/10.1063/1.3071090.
Prigogine, Ilya, Isabelle Stengers, and Alvin Toffler. 1984. Order Out of Chaos. First Edition
edition. New York, NY: Bantam. https://archive.org/details/OrderOutOfChoas.
“Process Simulation.” 2017. In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Process_simulation&oldid=801531196.
“Pub. No. GAO-03-864, Public Accounting Firms: Mandated Study on Consolidation and
Competition.” 2003. United States General Accounting Office.
https://www.gao.gov/new.items/d03864.pdf.
“Red Queen Hypothesis.” 2019. In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Red_Queen_hypothesis&oldid=91400539
2.
“Russian Active Measures Campaigns and Interference in the 2016 U.S. Election.” 2019. S.
Rpt II6-XX. Russian Active Measures Campaigns and Interference in the 2016 U.S.
Election. Washington DC: Select Committee on Intelligence United States Senate.
https://www.intelligence.senate.gov/sites/default/files/documents/Report_Volume1.pd
f.
“SEC Charges: False Tweets Sent Two Stocks Reeling in Market Manipulation.” 2015. U.S.
Securities and Exchange Commission. November 5, 2015.
https://www.sec.gov/news/pressrelease/2015-254.html.
Sheridan, Cris, Senior Editor, Co-Host, and Financial Sense. 2012. “Feedback Loops: HFT,
Black-Scholes, and Cicadas.” Financial Sense. October 10, 2012.
11
(Submitted to the InfraGard National Journal on 10/20/2019)
https://www.financialsense.com/contributors/cris-sheridan/feedback-loops-black-
scholes-hft-cicadas.
Singer, P. W., and Emerson T. Brooking. 2018. LikeWar: The Weaponization of Social
Media. Eamon Dolan/Houghton Mifflin Harcourt.
“Singularity University.” n.d. Singularity University. Accessed October 19, 2019.
https://su.org/.
Sverdlik, Yevgeniy. 2018. “AWS Says It’s Never Seen a Whole Data Center Go Down.”
Data Center Knowledge. November 29, 2018.
https://www.datacenterknowledge.com/amazon/aws-says-it-s-never-seen-whole-data-
center-go-down.
Swart, Jan M. 2014. “A Simple Rank-Based Markov Chain with Self-Organized Criticality.”
ArXiv:1405.3609 [Math], May. http://arxiv.org/abs/1405.3609.
Taleb, Nassim Nicholas. 2014. Antifragile: Things That Gain from Disorder. Reprint edition.
New York: Random House Trade Paperbacks.
Valen, Leigh van. 1973. “A New Evolutionary Law.” In .
https://www.mn.uio.no/cees/english/services/van-valen/evolutionary-theory/volume-
1/vol-1-no-1-pages-1-30-l-van-valen-a-new-evolutionary-law.pdf.
“Vegetation Clearances Automatically Analyzed Across a Power Line Network.” 2018.
Enview (blog). August 16, 2018. https://enview.com/about/blog/vegetation-
clearances-automatically-analyzed-across-a-power-line-network/.
Waltzman, Rand. 2017. “The Weaponization of Information.” Product Page. 2017.
https://www.rand.org/pubs/testimonies/CT473.html.
Whitehead, Alfred North. 1979. Process and Reality. 2nd edition. New York: Free Press.
Zenko, Micah. 2015. Red Team: How to Succeed By Thinking Like the Enemy. 1 edition. New
York: Basic Books.