ArticlePDF Available

Red Team Performance for Improved Computer Security

Authors:

Abstract

This research attempts to develop a human factors understanding of red team assessment strategies in computer and information security. Red teaming is an advanced form of assessment that can be used to identify weaknesses in a variety of security systems. The purpose of this research is to identify and define the various dimensions of red team effectiveness with the aim of improving red team performance. A study of a red team was conducted in collaboration with Sandia National Laboratories Information Design Assurance Red Team (IDART). The design of the study included semi-structured individual interviews and focus groups with red team members and observation of red team practices. The analysis yielded various dimensions of red team effectiveness from the customer, management, individual, and team member perspectives.
RED TEAM PERFORMANCE FOR IMPROVED COMPUTER SECURITY
Sara Kraemer+, Pascale Carayon+ and Ruth Duggan*
+Department of Industrial Engineering
Center for Quality and Productivity Improvement
University of Wisconsin-Madison
610 Walnut Street 575 WARF
Madison, WI 53726
Tel: 1-608- 263-2520
Fax: 1-608-263-1425
Email: sbkraeme@wisc.edu / carayon@engr.wisc.edu
* Sandia National Laboratories
PO Box, 5800, MS-1375
Albuquerque, NM 87185-1375
Tel: 1-505-844-9320
Fax: 1-505-284-9043
Email: rduggan@sandia.gov
ABSTRACT
This research attempts to develop a human factors understanding of red team assessment strategies in
computer and information security. Red teaming is an advanced form of assessment that can be used to
identify weaknesses in a variety of security systems. The purpose of this research is to identify and define
the various dimensions of red team effectiveness with the aim of improving red team performance. A study
of a red team was conducted in collaboration with Sandia National Laboratories Information Design
Assurance Red Team (IDART). The design of the study included semi-structured individual interviews and
focus groups with red team members and observation of red team practices. The analysis yielded various
dimensions of red team effectiveness from the customer, management, individual, and team member
perspectives.
INTRODUCTION AND BACKGROUND
Red teaming in computer and information security
Adversaries of computer and information systems can and will
plan and execute strategic attacks campaigns against the
United States (Tinnel, Saydjari, & Farrel, 2002). The
Department of Defense has recognized that red teaming has
long been a valuable, if underutilized, tool for deepening the
understanding of the adversaries the United States faces in the
war on terrorism (Defense Science Board Task Force, 2003).
In particular, red teaming is valuable in understanding
adversary’s capabilities and potential responses to United
States’ initiatives (Defense Science Board Task Force, 2003).
In order to expand and improve the experience base of system
defenders, a developed understanding of the strategies and
tactics employed by red teams is critical to warding off attacks
on computer and information systems.
Red teams reveal weaknesses in computer and information
security systems. The use of red teams (Schudel & Wood,
2000a) and so-called “ethical hacking” (see, for example,
Palmer, 2001) are important mechanisms for detecting system
vulnerabilities and hence enhancing security, since they allow
system defenders to understand system weaknesses from the
adversary perspective. Whereas ‘daily security enforcement’
may work for a while, red team attacks and correction of the
defects that they reveal is necessary for organizations’
computer and information security systems (Computer
Science and Telecommunications Board-National Research
Council, 2002). This paper specifically examines Sandia
National Laboratories Information Design Assurance Red
Team (IDART). The objective of this study was to identify
measures of red team performance with the purpose of
improving red team performance.
The knowledge obtained by red teams is especially beneficial
when the target system is still in development and designers
can readily effect improvements (Wood & Duggan, 1999).
The red team approach is based on the premise that an analyst
who attempts to model an adversary can find systemic
vulnerabilities in a computer and information system that
would otherwise go undetected. They seek opportunities to
combine system, organizational, and architectural
vulnerabilities in order to execute a successful attack. Sandia’s
red team has developed a formal methodology of assessment
(Wood & Duggan, 1999). This method includes team building,
system assessing and attacking, and reporting to the customer.
A significant portion of a red team project is system
assessment. This includes gathering source information,
describing the system, creating an objective purpose,
identifying critical success factors, formulating functional,
spatial, temporal, system lifecycle, and consequence-based
views of the system, identifying candidate vulnerabilities for
attack, and formulating attack plans. Specific adversary goals
resulting in negative consequences are called “flags”. The red
team has “captured the flag” when they have successfully
accomplished those goals. The customers of Sandia’s red
teams are from the private sector, ranging from banking and
finance, information technology, manufacturing and e-
commerce, as well as the public sector, including the
Departments of Defense, Energy, Interior, Homeland Security,
and State.
There is a need for a more comprehensive understanding of
red teams. Not only is there a very high demand for a red team
assessment, little research has been done to understand the
effectiveness of red teams. This is due to their lack of
availability, accessibility, and funding for team performance
analysis. Teams are more likely to be successful and efficient
when they are proficient in recording and filing information
and function in a systematic process (Lynn & Reilly, 2000),
and often the knowledge obtained by red teams is recorded
only in project specific confidential reports or left to the
cognition of the people of the official red team. Some research
has been done to understand red team performance (Carayon,
Duggan, & Kraemer, 2003), but more and better information
is needed to define and measure red team performance in
order to have a more meaningful impact on the mitigation of
vulnerabilities, security breaches, and attacks. Specifically,
formalized measures of red team performance are needed in
order to monitor performance over time, track the varying
factors affecting performance, and indicate types of
interventions for performance improvement (e.g., training,
feedback). The purpose of this study was to describe
preliminary measures of red team performance in order to
improve red team performance.
Research on red teaming and teams
The existing human factors research in red teaming is
extremely limited. Cognitive task analyses of individual and
groups of hackers who have attacked networks and websites
have been used to reveal how hackers select targets, distribute
and share responsibilities, and conduct actual attacks
(McCloskey & Stanard, 1999). Experimental research has
measured the effects of deception defenses on attacks against
computer systems and networks (Cohen, Marin, Sappington,
Stewart, & Thomas, 2001). Experimentation conducted or
funded by the Defense Advanced Research Projects Agency
(DARPA) shows the use of red teams in evaluating different
defense mechanisms (Kewley & Bouchard, 2000; Schudel &
Wood, 2000b) such as survivability solutions (Pal, Atighetchi,
Webber, Schantz, & Jones, 2003). These experiments have not
directly evaluated red team performance, but have identified a
number of factors that may contribute to red team performance,
such as experience or proficiency (Pal, et. al, 2003) and
learning (Kewley & Bouchard, 2000; Pal et al., 2003). They
have also identified some of the red team behaviors, such as
usage of time, work process, and risk perception (Schudel &
Wood, 2000b). Research has been done to develop measures
of adversary work factor or red team work factor (Schudel &
Wood, 2000b; Wood & Bouchard, 2001). Red team work
factor measures the amount of effort required by a red team
(an adversary) to accomplish an attack (i.e. to capture a flag)
(Wood & Bouchard, 2001). Experiments conducted by
DARPA show that red team work factor can be useful in
comparing different system configurations (Schudel & Wood,
2000b), especially if red team capability varies. Capability
may include different red team behaviors, which depend upon
their preparations, training, and talents. However, red team
work factor may be more a measure of the red team capability
instead of system improvement (Schudel & Wood, 2000b).
Red team work factor may still be useful in measuring the
effectiveness of a red team. In particular, when faced with
multiple problems, comparing red team work factor between
different exercises by the same team may provide information
on how different team characteristics affect red team work
factor (Wood & Bouchard, 2001). When trying to identify
factors that contribute to red team performance, red team work
factor may be one measure of red team performance that can
be correlated with various team characteristics.
In order to understand what is required for effective teamwork,
it is first necessary to define a team. A team is a set of two or
more individuals who interact interdependently and adaptively
toward a common goal or objective (Cannon-Bowers & Salas,
1998). There are different types of teams and Sandia’s red
team fits most closely with the work team definition. Work
teams are continuing work units responsible for producing
goods or providing services, their membership is typically
stable and full time, and well-defined (Cohen & Bailey, 1997).
This definition also includes self-managing work teams with
members who are cross trained in a variety of skills relevant to
the tasks that they perform.
There is no single measure of team performance that is
appropriate for all purposes. A distinction in team
performance assessment concerns outcome versus process.
Although teams are valued in large part for their outcomes,
these measures often contain variance attributable to factors
other than team work (Brannick & Prince, 1997). Team
process measures may be closer to a true picture of team
functioning, but a comprehensive measure of team
performance needs to contain elements of both process and
outcome (Brannick & Prince, 1997). The primary types of
team performance measures are: (1) descriptive measures (i.e.
process), which describe what is happening at any given time
and seek to document individual and team behaviors; (2)
evaluative measures (i.e. outcome), which judge performance
against identifiable standards and serve to answer questions of
effectiveness; and (3) diagnostic measures (i.e. process),
which seek to identify causes of behavior and question how
and why things occurred as they did (Paris, Salas, & Cannon-
Bowers, 2000). Diagnostic measures contribute to inputs to
the feedback process necessary to improve subsequent
performance (Salas & Cannon-Bowers, 1997).
METHOD
Due to the lack of research in red team effectiveness and the
exploratory nature of this study, the study design was
qualitative in nature, consisting of the following elements:
fifteen semi-structured individual interviews and two focus
groups with red team members, observation of a red team
group training session, attendance of a Sandia technical
presentation, personal observation of site surroundings, and
analyses of documents pertaining to red team work. The data
was collected by the first author. Individual interviews and
focus groups used the same open-ended interview guide. See
Appendix 1 for interview guide. The individual interviews
lasted approximately one hour and the focus groups lasted
approximately two and one half hours. One focus group and
eleven interviews were audio-recorded and one focus group
and four individual interviews were not audio-recorded.
Personal notes of these interactions were taken by the first
author and the audio-recordings were transcribed.
The IDART program consists of core, non-core, and matrix
red team members. Core red team members are system
analysts who regularly participate in red team projects and
whose full-time job is within the IDART assessment
department. Non-core members are system analysts who semi-
regularly participate in red team projects and are not members
of the IDART assessment department. Matrix members rarely
participate in red team projects. They are accessed for their
specific expertise, which is needed for specific systems under
consideration. For example, a red team examining a biological
and chemical agent detection system could include experts on
biological and chemical warfare agents. These members are
accessed from the pool of experts within the Sandia
organization. Individual interviews were done with eleven
core members, three non-core members, and two matrix
members. The first focus group included six core members
and the second focus group consisted of seven core members.
The transcribed notes and interviews were analyzed by coding
the themes of interviews and observations using the qualitative
software package, QSR NVivo©. The coding structure
consisted of nodes, representing a defined category of red
team effectiveness. When coded, a node held references to
passages of text from the observation and interview data. In
this paper, analysis data on performance is reported and
discussed.
RESULTS
Findings are reported in measurement category totals and most
frequently cited dimensions of measurement. The coding
process resulted in 67 nodes and the total number of
comments coded was 95. The nodes were grouped into five
major categories. The first four categories were designated
into four perspectives: (1) individual team members (12
comments); (2) the team as a whole (27 comments); (3)
management (12 comments); and (4) customer (30 comments).
These perspective categories were further stratified into three
dimensions of team measurement: descriptive, evaluative, and
diagnostic. Refer to Table 1 for a summary of the number of
comments on the perspective categories of performance. The
fifth category consisted of comments regarding difficulties in
measuring red team effectiveness (14 comments). The
measurement dimensions with the largest number of
comments within each perspective category are reported.
Table 1. Comments on red team performance
Individual
Descriptive (Process)
Evaluative (Outcome)
Diagnostic (Process)
Total
Team
Man a geme nt
Customer
Total
Perspective
Meas urement T ype
0 0 11 22 33
014 1 8 23
12 13 0 0 25
12 27 12 30 81
The most regularly cited individual team member measure
was diagnostic (12 comments). Within this grouping, the
dimension of individual professional development (4
comments) was reported most frequently. Individual
professional development was defined as individual growth in
computer science and system analysis acumen. One core
member described this experience: “Especially in our
particular setting where we are not like auto mechanics where
you get trained to do something and every time you service an
engine. We need to be learning at every engagement…very
important.”
From the perspective of the team as a whole, the most
frequently cited measures of effectiveness were evaluative (14
comments) and diagnostic (13 comments). System
understanding (5 comments), the most frequently cited
dimension in the team evaluative measurement category, was
defined as the extent the red team synthesizes and
characterizes the system from different viewpoints. One core
member described system understanding: “The system
understanding is fundamental to the root cause of system
weaknesses. This is more than other auditing efforts that scan
for vulnerabilities; the red team looks at the hardware,
software, physical layout, organizational processes, and
policies. The red team is communicating across the functions
of the customer’s organization and they are creating a view
that not even the organization itself has. The red team
“marries” these elements into a larger understanding, which is
something that an adversary would take advantage of.” Team
dynamics (10 comments) was the most frequently cited
dimension in the team diagnostic measurement category, and
was defined as team behaviors and attitudes. A comment to
describe team dynamics: “No matter how much you plan or
prepare attack, there is a huge space of variable stuff that
changes when you go from preparation to action. This space
is void of any planning, and that’s where team dynamics come
in.” Red team members spend a considerable amount of time
in the planning phase (i.e. describing the system, identifying
vulnerabilities, planning attacks). When the team moves from
the planning phase to engaging in an attack on a system, time
is limited. Good team dynamics speaks to the ability of the red
team to address and resolve unplanned issues that arise in an
attack setting swiftly and accurately.
In the management perspective category, the most frequently
cited measures were in the descriptive dimension. Obtaining
all the system targets in a simulated attack (4 comments) was
the most frequently cited descriptive measure. An explanation
of the importance of red team’s system targets: “In the red
team engagements, we are very careful to define what the
goals or the “flags” are. Otherwise, at the end of the day, it is
very hard to determine whether or not you are successful.”
From the customer’s perspective of red teaming, performance
is largely descriptive (22 comments). The quality of the
communication with the customer (14 comments) was the
most frequently cited dimension of customer descriptive
measurement. Communication with the customer is related to
the ease of communication (e.g., accessibility of the customer
to the team and vice versa), the frequency of communications,
the formalism of the communication, and the feedback from
the customer which is elicited at multiple steps within the
assessment methodology. A comment to illuminate the
customer communication and feedback standpoint: “Were you
able to show the customer that they were well protected or
there were holes? It becomes a matter of how the customer
perceives the information you give them and what solutions
you provide them.”
Red team members expressed difficulty in how to measure red
team effectiveness (14 total comments). Among the various
topics mentioned, the difficulty in measuring how effective
their work was for the customer (9 comments) was the topic
mentioned the most. One red team member spoke about the
difficulty in assessing the effectiveness of the work from the
customer’s perspective: “Because every assessment we do is
so unique and so different, I think it’s really hard to come up
with a metric that you could use to quantify the effectiveness
of the red team. I think that’s what it comes down to really, is
the customer. It’s whether or not we’ve done a good job,
whether they think we’ve…taken a good shot at it, really.”
DISCUSSION
The current study focused on describing dimensions of red
team performance measures. The range of perspectives that
were reported by red team members was consistent with the
team literature that emphasizes the need for more than one
type of performance measurement (i.e. process and outcome
measures). In general, the red team placed an emphasis on the
team and customer perspectives. At the team performance
level, red team members tended to view it both diagnostically
and evaluatively, surmising that not only is team process an
important measure, but so is the output of the team. Team
dynamics stood out as an important measure. Sandia’s red
team spends a considerable amount of time assessing systems
and creating ‘multiple system views’. This includes
brainstorming and planning attacks, efforts that require
substantial group effort, usually under the pressure of limited
time. Red team members also emphasized team evaluative (i.e.
outcome) measures. They highlighted the ‘lessons learned’ at
the end of the project as a marker for the things that went well
in the project, as well as the problems encountered. In addition,
the extent that the red team understood and accurately
characterized the system under consideration was also stressed.
How well the team creates the system understanding is
directly associated to the ultimate measures of success,
capturing the flags with the planned attacks. The learning or
feedback processes that were emphasized in this study could
be another measure to examine when assessing future red team
performance.
Red team members viewed the customer’s perspective of
performance as largely descriptive, deeming communication
with the customer as a key measure. Since the customer is not
present in all of the team processes where team dynamics or
other key team factors play roles, the quality of the
interactions with the customer is important in order to address
how the team is working and meeting its project goals. In
previous red team studies, the customer perspective was not
addressed. Assessing that perspective may be beneficial in
validating red team members’ evaluation of performance.
Red team members described individual member effectiveness
in terms of professional learning and having fun at work.
Individual learning during projects was considered an
important measure of effectiveness because of the uniqueness
of each project. These dimension of individual team member
performance measurement may be useful in tracking how
increased professional learning is correlated with other
outcome measures, such as capturing flags or achieving other
statement of work goals (e.g., time and budget constraints).
In summary, red team members described numerous measures
of red team effectiveness. These ranged from team behaviors
and attitudes to more easily quantifiable measures such as
meeting deadlines and capturing system targets. In previous
red team studies (Schudel & Wood, 2000b; Wood & Bouchard,
2001) time to complete an engagement, such as red team work
factor, addressed a quantifiable measure that could be used to
measure some dimensions of red team performance. The
various measures addressed in this study could broaden the
understanding of red team performance in that other factors
affecting performance could be used for improvement of both
the systems under inspection as well as red team performance.
CONCLUSION
Comprehensive measures of red team effectiveness could
improve red team performance in several ways. Firstly, to
determine red team effectiveness for the purpose of
pinpointing team performance strengths and weaknesses, a set
of team performance measures must exist. Secondly, teams
evolve over time (Morgan, Glickman, Woodard, Blaiwes, &
Salas, 1986) and the length of time that they have worked
together can have a significant effect on group processes
(Foushee, Lauber, Baetge, & Acomb, 1986). Establishing
baseline and continuous measures of red team performance
could provide feedback and other mechanisms for self-
correction over time. This has significant implications for self-
managed work teams (Cannon-Bowers & Salas, 1998) and
may also be applied to Sandia’s red team. Further, red team
experiments, such as measuring red team work factor, require
establishing a baseline for red team and system performance
before engaging in multiple runs of a given experiment
(Schudel & Wood, 2000b). Thirdly, a set of performance
measures would help guide the team improvement efforts,
such as team training or resource allocation.
Limitations of this study include the fact that descriptions of
measures were based solely upon red team members’
perceptions. Further, IDART’s red teams represent one type of
red team. There are other red teams in existence and it would
be interesting to assess if the same or different dimensions of
performance would be identified in other groups. This
includes extending this preliminary work by interviewing
other red team members and leaders at Sandia, as well as other
red teams. Another area of future work is the investigation of
the factors that contribute to and hinder red team performance.
In addition to interviewing red team members, we may
conduct observations of red team interaction at the following
stages: team forming, brainstorming sessions, sessions
formulating candidate vulnerabilities and attacks, engagement
of the system, and wrap-up sessions. Identifying these factors,
such as team design or member composition, could help in
determining how to configure a high-performing red team.
There is much to be learned about the factors associated with
red team performance and understanding red team
performance measurement and its various facets is the first
step into this inquiry.
ACKNOWLEDGMENTS
Funding provided by Department of Defense on “Modeling
and Simulation for Critical Infrastructure Protection”
(#DAAD19-01-1-0502, PI: S. Robinson, UW-Madison).
REFERENCES
Brannick, M. T., & Prince, C. (1997). Overview of team performance
measurement. In M. T. Brannick & E. Salas & C. Prince (Eds.),
Team Performance Assessment and Measurement (pp. 3-16).
Mahwah, NJ: Lawrence Erlba um Associates.
Cannon-Bowers, J. A., & Salas, E. (1998). Team performance and training in
complex environments: Recent findings from applied research.
Current Directions in Psychological Science, 7(3), 83-87.
Carayon, P., Duggan, R., & Krae mer, S. (2003). A model of red team
performance. In K. J. Zink (Ed.), Seventh International Symposium
on Human Factors in Organizational Design and Management.
Aachen, Germany.
Cohen, F., Marin, I., Sappington, J., Stewart, C., & Thomas, E. (2001). Red
teaming experiments with deception technologies. Retrieved from
the Strategic Security Intelligence Website:
http://www.all.net/journal/deception/experiments/experiments.html.
Cohen, S. G., & Bailey, D. E. (1997). What makes teams work: Gr oup
effectiveness research from the shop floor to the executive suite.
Journal of Management, 23(3), 239-290.
Computer Science and Telecommunications Board-National Research Council.
(2002). Cybersecurity Today and Tomorrow: Pay Now or Pay
Later. Washington, DC: National Academy Press.
Defense Science Board Task Force. (2003). The role and status of DoD red
teaming activities. Washington, D.C.: Office of the Under
Secretary of Defense for Acquisition, Technology, and Logistics.
Foushee, H. C., Lauber, J., Baetge, M., & Acomb, D. (1986). Crew factors in
flight operations: III. The operational significance of exposure to
short-haul air transport operations (NASA Technical
Memorandum 88322). Sunnyvale, CA: National Aeronautics and
Space Administration-Ames Research Center.
Kewley, D. L., & Bouchard, J. F. (2000). DARPA Information Assurance
program dynamic defense experiment summary, Proceedings of
the 2000 IEEE Workshop on Information Assurance and Security
(pp. 117-122). United States Military Academy, West Point, NW.
Lynn, G. S., & Reilly, R. R. (2000). Measuring team performance. Research-
Technology Management, 43(2), 48-56.
McCloskey, M. J., & Stanard, T. (1999). A red team analysis of the electronic
battlefield: A cognitive approach to understanding how hackers
work in groups, Proceedings of the Human Factors and
Ergonomics Society 43rd Annual Meeting (pp. 179-183): Human
Factors and Ergonomics Society.
Morgan, B. B., Glickman, A. S., Woodard, E. A., Blaiwes, A. S., & Salas, E.
(1986). Measurement of team behaviors in a Navy en vironment
(Tech. Rep. No TR-86-014). Orlando, FL: Naval Training Center.
Pal, P., Atighetchi, M., Webber, F., Schantz, R., & Jones, C. (2003).
Reflections on evaluating survivability: The APOD experiments,
The 2nd IEEE International Symposium on Network Computing
and Applications (NCA-03). Royal Sonesta Hotel, Cambridge, MA.
Palmer, C. C. (2001). Ethical hacking. IBM Systems Journal, 40(3), 769-780.
Paris, C. R., Salas, E., & Cannon-Bowers, J. A. (2000). Teamwork in multi-
person systems: A review and a nalysis. Ergonomics, 43(8), 1052-
1075.
Salas, E., & Cannon-Bowers, J. A. (1997). Methods, tools, and strategies for
team training. In M. A. Quinones & A. Ehrenstein (Eds.), Training
for a Rapidly Changing Workforce: Applications of Psychological
Research (pp. 249-279). Washington, D.C.: American
Psychological Association.
Schudel, G., & Wood, B. (2000a). Modeling behavior of the cyber-terrorist,
Conference Proceedings: Research on Mitigating the Insider
Threat to Information Systems-#2. Santa Monica, California: Rand.
Schudel, G., & Wood, B. (2000b). Adversary work factor as a metric for
information assurance, Proceedings from the New Paradigms in
Security Workshop. Ballycotton, County Cork, Ireland:
Association of Computer Machinery.
Tinnel, L. S., Saydjari, O. S., & Farrell, D. (2002). Cyberwar strategy and
tactics: An analysis of cyber goals, strategies, tactics, and
techniques, Proceedings of the 2002 IEEE: Workshop on
Information Assurance. United States Military Academy, West
Point, NY.
Wood, B. J., & Dugga n, R. (1999). Red teaming of advanced information
assurance concepts, DISCEX2000 DARPA Information
Survivability Conference ( pp. SAND99-2590C). Hilton Head,
South Car olina.
Wood, B., & Bouchard, J. F. (2001). Red team work factor as a security
measurement, Proceedings of the Workshop on Information
Security Scoring and Ranking. Williamsburg, Virginia: Applied
Computer Security Associates.
APPENDIX 1
Individual and focus group interview guide
1. Various factors affect red team performance and red team
performance can be evaluated on different dimensions. What
are the various criteria for evaluating red team performance?
... There are several companies that have developed indigenous processes for executing red team -operations, but it's unusual to reveal the processes due to competitive edge of the business (Kraemer, Carayon & Duggan, 2004). A dissertation by James Michael Fleming (2010) examines different types of red teams and their processes in the commercial and defence sector. ...
... This has happened to red teaming research as well. Red team performance and effectiveness has been studied and problem has been the confidential nature of red teaming efforts which has prevented documentation of best practices (Kraemer et al., 2004). Team effectiveness in cyber exercises have been studied to better the achievements of red or blue teams (Granåsen & Andersson, 2016). ...
Thesis
Full-text available
The goal of red teaming is to create better plans, policies, procedures and products in any domain by challenging the current ones. This calls for assessment and critique of status quo. Red teaming is about mitigating future risks and communicating bad news. Red teaming research has focused in adversary emulation and penetration testing practices somewhat disregarding the remediations which are the key in building better security. Cyber threats are evolving and so should cyber red teaming research. Red teaming efforts should be conducted through a comprehensive planning and execution process which considers the complete information security lifecycle starting from planning of intelligence activities and ending to implementing remediations for security to the target organization. Red teaming should be a process that can be understood and adopted by organization and it should be also transparent and traceable. The research problem was to create a comprehensive agile red teaming framework by combining adaptive planning and execution framework in information security context. Design science research methodology was used to solve this challenge. Solid knowledge base and environment description about red teaming and information security was completed in accordance with information systems research framework. Adaptive planning and execution framework, intelligence, targeting and agile methodologies were introduced to support the creation of the framework. Challenges in red teaming were identified by a survey to five cyber security companies. Challenges were remediated by success factors identified from literature and survey. The framework was created, and it underwent two Delphi iterations with subject matter experts. Main result of the study is the comprehensive agile red teaming framework which incorporates the remediations drawn from subject matter experts, military and agile methods. The scope of this study was wide and therefore results can be considered general. The significance of the created framework lies in its novelty and possibilities to adapt it to any red teams’ purposes due to general outcome. Framework delivers a good basis for future work.
... There is literature supporting the value of teamwork for cyber offense (e.g. Kraemer, Carayon, & Duggan, 2004;McCloskey & Stanard, 1999). Much of the work has been in context of red (offensive) teams looking for vulnerabilites within corporate systems. ...
... However, there is not much information about qualitative processes and coordination of team efforts during these events. The measures of team effectiveness are typically the time it takes to find flags (targets), subjective feelings of the hacking teams, and whether the corporate client understood the results of the event (Kraemer et al., 2004). ...
Article
Full-text available
Effective team process is critical for the performance of cyber security teams. To examine this, we observed two comparably skilled cyber security teams participating in the International Capture the Flag (iCTF) competition held in December 2011. At the conclusion of the competition, we followed up with a focus group discussion with six members from the two teams. In this paper, we present our findings from the focus group interviews, on the relationship between team level factors and team performance. Findings from the focus group discussion indicate that team level factors such as team communication, coordination, team structure, and leadership play important roles in team performance.
... Leading up to the 2012 meeting, Cyber Security was represented in only 9 publications from all the previous years (Boyce et. al., 2011; D'Amico, Whitley, Tesone, O'Brien & Roth, 2005; Jariwala, Champion, Rajivan & Cooke, 2012; Kraemer, Carayon & Duggan, 2004; Mahoney et. al, 2010; Mancuso & McNeese, 2012; McCloskey & Stanard, 1999; McNeese et al. 2012; Stenard et al., 2004). ...
Article
Full-text available
Cyber operations offer a unique environment in which the lines between cognition and technology are constantly blurred. Within the greater research community, current work often focuses solely on the technology, often only acknowledging the human in passing, if at all. More recently, the Human Factors community has begun to address human-centered issues in cyber operations, but in comparison to technological communities, we have yet to scratch the surface. Even with publications on Cyber Human Factors gaining momentum, we still lack a complete and holistic understanding of the domain itself, creating a major gap in the field. The purpose of this panel is to continue to expand the role Human Factors in cyber research by introducing the community to current work being done, and to facilitate collaborations to drive future research. We have assembled a panel of scientists across multiple specializations in the Human Factors community to have an open discussion on how we can leverage previous work in human factors and current work in cyber operations to continue to push the bounds of the field.
... The result suggests that seniors are tougher targets than freshmen. In [34] interviews of members in a red team are used to investigate what the key factors are for their effectiveness. The result of these interviews suggests, among other things, that it is difficult to identify attributes that reflect the effectiveness of a red team as well as the relationships between the effectiveness of individuals and teams. ...
Conference Paper
This paper discusses the use of cyber security exercises and competitions to produce data valuable for security research. Cyber security exercises and competitions are primarily arranged to train participants and/or to offer competence contests for those with a profound interest in security. This paper discusses how exercises and competitions can be used as a basis for experimentation in the security field. The conjecture is that (1) they make it possible to control a number of variables of relevance to security and (2) the results can be used to study several topics in the security field in a meaningful way. Among other things, they can be used to validate security metrics and to assess the impact of different protective measures on the security of a system.
... Using the SAGEPub archives, a keyword search for cyber and security for all text, as of the writing of this paper, returns few results across HFES publications; 9 publications in the Annual Proceedings (Boyce et. al., 2011;D'Amico, Whitley, Tesone, O'Brien & Roth, 2005;Jariwala, Champion, Rajivan & Cooke, 2012;Kraemer, Carayon & Duggan, 2004;Mahoney et. al, 2010;Mancuso & McNeese, 2012;McCloskey & Stanard, 1999;McNeese et al. 2012;Stenard et al., 2004), 1 publication in the Journal of HFES (Dutt, Ahn & Gonzalez, 2012), and 0 in both the Journal of Cognitive Engineering and Decision Making or Ergonomics in Design. ...
Article
Full-text available
There has been a dramatic increase in the total number of reported cyber security breaches and attacks in recent years. In response, government, and corporate entities have invested billions of dollars in funding research and development efforts for cyber operations, including computer network defense (CND) and computer network attack (CNA). While cyber operations have become an important national priority over the last decade, the Human Factors community has yet to approach it with critical mass. In its purest form, cyber operations are a complex sociotechnical system that can have effects across all levels of an organization. Due to a consistent interplay between human cognition, technology, and organizational constraints within the environment, the Human Factors community is particularly well-suited to address the problem space. We have assembled a panel of six scientists, technologists, and subject matter experts across multiple specializations in the Human Factors community to help begin this increasingly important discussion. The goal of this panel is to have an open discussion on how we can leverage our specializations and expertise to address the cyber operations landscape as a community.
... Although often applied against cyberterrorism, red teams can in principle be applied against any type of terrorist threat. HF/E researchers have studied anti-cyberterrorism red teams, an activity that has yielded suggestions for improved red team effectiveness [17]. Similar research studies would likely improve the function of red teams deployed against other types of terrorism, as well. ...
Article
Full-text available
Various subdisciplines of psychology are relevant to the defence against terrorism, in terms of anti-terrorism, counter-terrorism, and terrorism consequence management. Anti-Terrorism: Psychological methods can be applied to reduce vulnerabilities to attack and to encourage the general public to identify infrastructure and other vulnerabilities. Counter-Terrorism: Psychological techniques are available to assess and improve terrorism awareness in the general population. The detection performance of counter-terrorism personnel can be improved: psychological methods can enhance situation awareness, situated cognition, detection capabilities, and decision-making; automated expert system tools employing fuzzy signal detection can assist personnel; other psychological techniques can enhance individual and team function, personnel selection and training. Psychological principles can also be applied to obstruct and impede terrorist functioning. Consequence Management: Psychological methods can be used to enhance capabilities of first responders, improve escape and evacuation procedures for civilians, promote resilience in the general population, and treat victims of terrorism more effectively. We propose possible configurations for psychological consulting teams who would help defence authorities use these strategies to address terrorist activity.
Chapter
Over the last decade, a noticeable growth in cyber-attacks has been recorded, pushing experts and specialists in cyber security for more advanced and sophisticated countermeasures to curb the spread of cyber-attacks. What needs to be addressed is the speed of which cybercrime is growing in the areas of critical national infrastructure; these can present multiple issues in allowing safe and secure operation for nations, states and countries. For example, Stuxnet—a highly sophisticated Advanced Persistent Threat (APT) attack directed at Supervisory Control and Acquisition Data (SCADA) systems responsible for impairing operations of a nuclear plant in Iran. As the domain of cyber security is becoming more high profile, it is notable to analyse how the landscape of power plants in the United Kingdom is shaping, focusing on the counter measures against cyber-attacks. One defence mechanism, designed by the Bank of England called CBEST, is a framework that provides an intelligence-led penetration testing solution to the financial sector of the UK. Consequently, these have formed questions for research as to what extent can the CBEST framework be effectively deployed for the energy sector and review as a potential defence solution. It seems that the CBEST framework could be successfully deployed in a sector other than financial, however, first some improvements have to be carried out e.g. cyber security training for a wide range of the personnel. This chapter aims to explore viability of this framework to be applicable to other sectors, such as the energy sector.
The nature of battle is evolving. As technology continues to evolve and the world becomes more interconnected, the potential for cyber-terrorism and hacker crime continues to grow. Malicious hackers have been developing tools and strategies for years that enable them to work in distributed teams and launch coordinated, synchronized attacks against targets of their choice. If we are to effectively combat the enemies of cyberspace, we need to first understand how they operate. When they work in groups, how do they form? How do they identify goals? How do they share information when they are a distributed team? Klein Associates is in the process of studying the behaviors and motivations of hackers and hacker groups. We can learn many lessons from the actions of these skilled individuals that will help network security professionals better understand and counter the threats they face.
Article
In this article, we summarize and review the research on teams and groups in organization settings published from January 1990 to April 1996. The article focuses on studies in which the dependent variables are concerned with various dimensions of effectiveness. A heuristic framework illustrating recent trends in the literature depicts team effectiveness as a function of task, group, and organization design factors, environmental factors, internal processes, external processes, and group psychosocial traits. The review discusses four types of teams: work, parallel, project, and management. We review research findings for each type of team organized by the categories in our heuristic framework. The article concludes by comparing the variables studied for the different types of teams, highlighting the progress that has been made, suggesting what still needs to be done, summarizing key leamings from the last six years, and suggesting areas for further research.
Article
Metrics can play an important role in helping companies to enhance their new product development efforts. A new approach applies metrics in several critical areas to measure and improve the performance of new product teams. Some of the metrics include: a team's ability to establish a clear vision, secure team member and management 'buy-in,' record project information, maintain adequate filing systems, conduct rigorous team review meetings, follow stage-gate processes, and retain a stable project team. If a team can excel in these seven areas, it will have a greater likelihood of capturing team knowledge and using that knowledge to speed development and improve new product success.
Article
In this article, we summarize and review the research on teams and groups in organization settings published from January 1990 to April 1996. The article focuses on studies in which the dependent variables are concerned with various dimensions of effectiveness. A heuristic framework illustrating recent trends in the literature depicts team effectiveness as a function of task, group, and organization design factors, environmental factors, internal processes, external processes, and group psychosocial traits. The review discusses four types of teams: work, parallel, project, and management. We review research findings for each type of team organized by the categories in our heuristic framework. The article concludes by comparing the variables studied for the different types of teams, highlighting the progress that has been made, suggesting what still needs to be done, summarizing key learnings from the last six years, and suggesting areas for further research.
Article
The research reported here represents the first year of a three-year effort to gain a better understanding of the processes that contribute to Team Evolution and Maturation (TEAM) in operational Navy contexts. The ultimate objective of this research is to provide a basis for enhancing the training, performance, and unit maintenance functions of Navy teams. The general focus of this effort is to measure team evolution and maturation as team members gain experience and knowledge about tasks, each other, and external environmental demands within the context of an operational training scenario. Existing models and methodologies have been synthesized from the team-performance-team-training literatures as a basis for the development of a working model of team evolution and maturation. Based on this model, prototype procedures and methods for measuring team development have been defined and developed. These measurement technologies have been tested at the Naval Gunfire Support Department (NGFS) at the Naval Amphibious School, Little Creek. Results of this effort indicate that the developing concepts, methods, and procedures are viable tools for the study of team training and performance. The results support the proposed stage model of team development and provide a sound foundation for the development of interventions for the enhancement of team training.
Article
Metrics can play an important role in helping companies to enhance their new product development efforts. A new approach applies metrics in several critical areas to measure and improve the performance of new product teams. Some of the metrics include: a team's ability to establish a clear vision, secure team member and management "buy-in," record project information, maintain adequate filing systems, conduct rigorous team review meetings, follow stage-gate processes, and retain a stable project team. If a team can excel in these seven areas, it will have a greater likelihood of capturing team knowledge and using that knowledge to speed development and improve new product success.