Conference Paper

Mixed method approach to identify analytic questions to be visualized for military cyber incident handlers

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... There were fewer papers addressing analysts' Jobs, with only 13 papers significantly relating to this theme. We used this theme to identify papers that took a holistic approach to describe the broader work environment of intelligence analysts and the ways that they perceive themselves in the context of this larger environment [37][38][39][40]. We found that the scope of this research area was larger than our planned interview study's allotted time would allow. ...
... For the study population, the main difference between the models was whether the study was performed using intelligence analysts [15,23,44], or a more generalized population [45,46]. For individual vs. general focus, we found that some models represented the process of an individual analyst [8,[45][46][47][48], while others were intended to describe generalizable processes [23,37,44,49]. For analysis methodology, some models were based on empirical research [15,23,47,48,50], while other models represented the author's synthesis of the space without performing empirical studies [8,42,45,51,52]. ...
Article
Full-text available
Analyst sensemaking research typically focuses on individual or small groups conducting intelligence tasks. This has helped understand information retrieval tasks and how people communicate information. As a part of the grand challenge of the Summer Conference on Applied Data Science (SCADS) to build a system that can generate tailored daily reports (TLDR) for intelligence analysts, we conducted a qualitative interview study with analysts to increase understanding of information passing in the intelligence community. While our results are preliminary, we expect that this work will contribute to a better understanding of the information ecosystem of the intelligence community, how institutional dynamics affect information passing, and what implications this has for a TLDR system. This work describes our involvement in and work completed during SCADS. Although preliminary, we identify that information passing is both a formal and informal process and often follows professional networks due especially to the small population and specialization of work. We call attention to the need for future analysis of information ecosystems to better support tailored information retrieval features.
... Some CTAs have resulted in knowledge representations that are workflow-oriented (e.g., Erbacher et al., 2010). The CTAs that are most aligned to the methodology resulted in a list of goals and subgoals of defenders and/or reflect decision making through the questions asked by defenders (e.g., Buchanan et al., 2016). It should be noted that, although outside the scope of this review, work has been done to understand processes at the team-level (e.g., Cooke et al., 2013;Nyre Yu, 2019;Tetrick et al., 2016). ...
... The result of the CTA was a list of nine preliminary categories of CCSA. Buchanan et al. (2016) conducted a goal-directed CTA to elicit the subgoals and decisions, also phrased as questions, under two high level goals: detecting threatening incidents and characterizing those incidents. Describing the content of CCSA as lists of questions can be understood as an answer to the question, what should a defender attend to? ...
Article
Full-text available
https://www.nationalcyberwatch.org/wp-content/uploads/2020/11/2020_CSJ_NICE_Special_Issue_Online.pdf
... It is important to identify those sources for which a sonified representation might add value in network monitoring; these might be raw network data sources such as packet captures, Netflow or Domain Name System (DNS) logs, or the sources might be monitoring systems such as IDSs or network firewalls. Buchanan et al. categorised the potential data sources used by security analysts in answering a number of different analytical questions (for example, in searching for the activities associated with a particular suspicious IP address) [49]. We hypothesise that raw network packet capture data is most suitable for network attack detection, because this constitutes a full representation of traffic on the network. ...
... Some prior work identifies network data features for network anomaly detection, and for the detection of particular classes of threat such as Advanced Persistent Threats (APTs) and Botnets [50][51][52]. Some of this work involves interviews with security analysts to identify the properties of data analysts search for in network security monitoring to enable attack detection [49,53]. The findings from attack characterisation and prior work can be bolstered through interviews with security analysts, to gather their views on the importance of particular network data features for network attack detection. ...
Article
Full-text available
Sonification systems, in which data are represented through sound, have the potential to be useful in a number of network-security monitoring applications in Security Operations Centres (SOCs). Security analysts working in SOCs generally monitor networks using a combination of anomaly-detection techniques, Intrusion Detection Systems and data presented in visual and text-based forms. In the last two decades significant progress has been made in developing novel sonification systems to further support network-monitoring tasks, but many of these systems have not been sufficiently validated, and there is a lack of uptake in SOCs. Furthermore, little guidance exists on design requirements for the sonification of network data. In this paper, we identify the key role that sonification, if implemented correctly, could play in addressing shortcomings of traditional network-monitoring methods. Based on a review of prior research, we propose an approach to developing sonification systems for network monitoring. This approach involves the formalisation of a model for designing sonifications in this space; identification of sonification design aesthetics suitable for realtime network monitoring; and system refinement and validation through comprehensive user testing. As an initial step in this system development, we present a formalised model for designing sonifications for network-security monitoring. The application of this model is demonstrated through our development of prototype sonification systems for two different use-cases within network-security monitoring.
... The challenge in creating meaningful visual tools for cybersecurity practitioners is in combining the expertise from specialists from the fields of data visualization and cybersecurity so that the resulting visualizations are effective and indeed useful for their intended users [10]. Further, creating visualizations useful for SMEs is not possible without an in-depth understanding of the tasks which the visualizations will support [11]. Hence, we describe here a multi-part, semi-structured interviewing method for extracting from an individual SME their internalized understanding of the dataset 4 that represents their protected environment, in order to create visualizations that align with their own understanding of that dataset and that will enhance the SMEs and their colleagues' ability to understand and work with that dataset. ...
Chapter
Visualizations can enhance the efficiency of Cyber Defense Analysts, Cyber Defense Incident Responders and Network Operations Specialists (Subject Matter Experts, SME) by providing contextual information for various cybersecurity-related datasets and data sources. We propose that customized, stereoscopic 3D visualizations, aligned with SMEs internalized representations of their data, may enhance their capability to understand the state of their systems in ways that flat displays with either text, 2D or 3D visualizations cannot afford. For these visualizations to be useful and efficient, we need to align these to SMEs internalized understanding of their data. In this paper we propose a method for interviewing SMEs to extract their implicit and explicit understanding of the data that they work with, to create useful, interactive, stereoscopically perceivable visualizations that would assist them with their tasks.
... The challenge in creating meaningful visual tools for cybersecurity practitioners is in combining the expertise from specialists from the fields of data visualization and cybersecurity so that the resulting visualizations are effective and indeed useful for their intended users [10]. Further, creating visualizations useful for SMEs is not possible without an in-depth understanding of the tasks which the visualizations will support [11]. Hence, we describe here a multi-part, semi-structured interviewing method for extracting from an individual SME their internalized understanding of the dataset 4 that represents their protected environment, in order to create visualizations that align with their own understanding of that dataset and that will enhance the SMEs and their colleagues' ability to understand and work with that dataset. ...
Preprint
Full-text available
Visualizations can enhance the efficiency of Cyber Defense Analysts, Cyber Defense Incident Responders and Network Operations Specialists (Sub-ject Matter Experts, SME) by providing contextual information for various cy-bersecurity-related datasets and data sources. We propose that customized, stere-oscopic 3D visualizations, aligned with SMEs internalized representations of their data, may enhance their capability to understand the state of their systems in ways that flat displays with either text, 2D or 3D visualizations cannot afford. For these visualizations to be useful and efficient, we need to align these to SMEs internalized understanding of their data. In this paper we propose a method for interviewing SMEs to extract their implicit and explicit understanding of the data that they work with, to create useful, interactive, stereoscopically perceivable visualizations that would assist them with their tasks.
... Users may be willing to accept high false alert rates when they perceive the expected damage from missing detections exceeds the cost of unnecessary protective actions following false alerts. In contrast, if alerts disturb the user's workflow, and the perceived risk of experiencing an attack is low, the user may prefer a low rate of alerts, even if this lowers the probability of detecting a threat (Buchanan, D′Amico, & Kirkpatrick, 2016;Schechter, Dhamija, Ozment, & Fischer, 2007). Previous studies on user adjustments of alerting thresholds show that users are sensitive to the quality of the alerting system but that they still tend to set nonoptimal alerting thresholds. ...
Article
Full-text available
Objective: We identify three risk-related behaviors in coping with cyber threats-the exposure to risk a person chooses, use of security features, and responses to security indications. The combinations of behaviors that users choose determine how well they cope with threats and the severity of adverse events they experience. Background: End users' coping with risks is a major factor in cybersecurity. This behavior results from a combination of risk-related behaviors rather than from a single risk-taking tendency. Method: In two experiments, participants played a Tetris-like game, attempting to maximize their gains, while exogenous occasional attacks could diminish earnings. An alerting system provided indications about possible attacks, and participants could take protective actions to limit the losses from attacks. Results: Variables such as the costs of protective actions, reliability of the alerting system, and attack severity affected the three behaviors differently. Also, users dynamically adjusted each of the three risk-related behaviors after gaining experience with the system. Conclusion: The results demonstrate that users' risk taking is the complex combination of three behaviors rather than the expression of a general risk-taking tendency. The use of security features, exposure to risk, and responses to security indications reflect long-term strategy, short-term tactical decisions, and immediate maneuvering in coping with risks in dynamic environments. Application: The results have implications for the analysis of cybersecurity-related decisions and actions as well as for the evaluation and design of systems and targeted interventions in other domains.
Article
Full-text available
Cyber security operators in the military and civilian sector face a lengthy repetitive work assignment with few critical signal occurrences under conditions in which they have little control over what transpires. In this sense, their task is similar to vigilance tasks that have received considerable attention from human factors specialists in regard to other operational assignments such as air traffic control, industrial process control, and medical monitoring. Accordingly, this study was designed to determine if cyber security tasks can be linked to more traditional vigilance tasks in regard to several factors known to influence vigilance performance and perceived mental workload including time on task, the probability of critical signal occurrence, and event rate (the number of stimulus events that must be monitored in order to detect critical signals). Consistent with the results obtained in traditional vigilance experiments, signal detection on a 40-minute simulated cyber security task declined significantly over time, was directly related to signal probability, and inversely related to event rate. In addition, as in traditional vigilance tasks, perceived mental workload in the cyber task, as reflected by the NASA Task Load Index, was high. The results of this study have potential meaning for designers of cyber security systems in regard to psychophysical factors that might influence task performance and the need to keep the workload of such systems from exceeding the information processing bounds of security operators.
Conference Paper
Full-text available
This paper offers insights to how cyber security analysts establish and maintain situation awareness of a large computer network. Through a series of interviews, observations, and a card sorting activity, we examined the ques-tions analysts asked themselves during a network event. We present the results of our work as a taxonomy of cyber awareness questions that represents a mental model of situation awareness in cyber security analysts.
Article
Full-text available
Analysts engaged in real-time monitoring of cybersecurity incidents must quickly and accurately respond to alerts generated by intrusion detection systems. We investigated two complementary approaches to improving analyst performance on this vigilance task: a graph-based visualization of correlated IDS output and defensible recommendations based on machine learning from historical analyst behavior. We tested our approach with 18 professional cybersecurity analysts using a prototype environment in which we compared the visualization with a conventional tabular display, and the defensible recommendations with limited or no recommendations. Quantitative results showed improved analyst accuracy with the visual display and the defensible recommendations. Additional qualitative data from a "talk aloud" protocol illustrated the role of displays and recommendations in analysts' decision-making process. Implications for the design of future online analysis environments are discussed.
Conference Paper
Full-text available
The goal of our project is to create a set of next-generation cyber situational-awareness capabilities with applications to other domains in the long term. The situational-awareness capabilities being developed focus on novel visualization techniques as well as data analysis techniques designed to improve the comprehensibility of the visualizations. The objective is to improve the decision-making process to enable decision makers to choose better actions. To this end, we put extensive effort into ensuring we had feedback from network analysts and managers and understanding what their needs truly are. This paper discusses the cognitive task analysis methodology we followed to acquire feedback from the analysts. This paper also provides the details we acquired from the analysts on their processes, goals, concerns, etc. A final result we describe is the generation of a task-flow diagram.
Chapter
In a survey of cyber defense practitioners, we presented 39 assertions about the work cyber operators do, data sources they use, and how they use or could use cyber security visual presentations. The assertions were drawn from prior work in cyber security visualization over 15 years. Our goal was to determine if these assertions are still valid for today’s cyber operators. Participants included industry, government and academia experts with real experience in the cyber domain. Results validated the assertions, which will serve as a foundation for follow-on security visualization research. Feedback also indicates that when analyzing a security situation, cyber operators inspect large volumes of data, usually in alpha-numeric format, and try to answer a series of analytic questions, expending considerable cognitive energy. Operators believe security visualizations could support their analysis and communication of findings, as well as training new operators.
Conference Paper
The Visualization for Cyber Security research community (VizSec) addresses longstanding challenges in cyber security by adapting and evaluating information visualization techniques with application to the cyber security domain. This research effort has created many tools and techniques that could be applied to improve cyber security, yet the community has not yet established unified standards for evaluating these approaches to predict their operational validity. In this paper, we survey and categorize the evaluation metrics, components, and techniques that have been utilized in the past decade of VizSec research literature. We also discuss existing methodological gaps in evaluating visualization in cyber security, and suggest potential avenues for future research in order to help establish an agenda for advancing the state-of-the-art in evaluating cyber security visualizations.
Article
Cyber Network degradation and exploitation can covertly turn an organization's technological strength into an operational weakness. It has become increasingly imperative, therefore, for an organization's personnel to have an awareness of the state of the Cyber Network that they use to carry out their mission. Recent high-level government initiatives along with hacking and exploitation in the commercial realm highlight this need for general Cyber Situational Awareness (SA). While much of the attention in both the military and commercial cyber security communities is on abrupt and blunt attacks on the network, the most insidious cyber threat to organizations are subtle and persistent attacks leading to compromised databases, processing algorithms, and displays. We recently began an effort developing software tools to support the Cyber SA of users at varying levels of responsibility and expertise (i.e., not just the network administrators). This paper presents our approach and preliminary findings from a CTA we conducted with an operational Subject Matter Expert to uncover the situational awareness requirements of such a tool. Results from our analysis indicate a list of preliminary categories of these requirements, as well as specific questions that will drive the design and development of our SA tool. Copyright 2010 by Human Factors and Ergonomics Society, Inc. All rights reserved.
Conference Paper
A Cognitive Task Analysis (CTA) was performed to investigate the workflow, decision processes, and cognitive demands of information assurance (IA) analysts responsible for defending against attacks on critical computer networks. We interviewed and observed 41 IA analysts responsible for various aspects of cyber defense in seven organizations within the US Department of Defense (DOD) and industry. Results are presented as workflows of the analytical process and as attribute tables including analyst goals, decisions, required knowledge, and obstacles to successful performance. We discuss how IA analysts progress through three stages of situational awareness and how visual representations are likely to facilitate cyber defense situational awareness.
Article
Visualization tools for cyber security often overlook related research from the information visualization domain. Cyber security data sets are notoriously large, yet many of the popular analysis tools use 3D techniques and parallel coordinates which have been shown to suffer issues of occlusion when applied to large data sets1,2. While techniques exist to ameliorate these issues they are typically not used. In this paper we evaluate several cyber security visualization tools based on established design principles and human-computer interaction research. We conclude by enumerating challenges, requirements, and recommendations for future work.
Article
APPLIED SECURITY VISUALIZATION Collecting log data is one thing, having relevant information is something else. The art to transform all kinds of log data into meaningful security information is the core of this book. Raffy illustrates in a straight forward way, and with hands-on examples, how such a challenge can be mastered. Let's get inspired. Andreas Wuchner, Head of Global IT Security, Novartis Use Visualization to Secure Your Network Against the Toughest, Best-Hidden Threats As networks become ever more complex, securing them becomes more and more difficult. The solution is visualization. Using todays state-of-the-art data visualization techniques, you can gain a far deeper understanding of whats happening on your network right now. You can uncover hidden patterns of data, identify emerging vulnerabilities and attacks, and respond decisively with countermeasures that are far more likely to succeed than conventional methods. In Applied Security Visualization, leading network security visualization expert Raffael Marty introduces all the concepts, techniques, and tools you need to use visualization on your network. Youll learn how to identify and utilize the right data sources, then transform your data into visuals that reveal what you really need to know. Next, Marty shows how to use visualization to perform broad network security analyses, assess specific threats, and even improve business compliance. He concludes with an introduction to a broad set of visualization tools. The books CD also includes DAVIX, a compilation of freely available tools for security visualization. You'll learn how to: Intimately understand the data sources that are essential for effective visualization Choose the most appropriate graphs and techniques for your IT data Transform complex data into crystal-clear visual representations Iterate your graphs to deliver even better insight for taking action Assess threats to your network perimeter, as well as threats imposed by insiders Use visualization to manage risks and compliance mandates more successfully Visually audit both the technical and organizational aspects of information and network security Compare and master todays most useful tools for security visualization Contains the live CD Data Analysis and Visualization Linux (DAVIX). DAVIX is a compilation of powerful tools for visualizing networks and assessing their security. DAVIX runs directly from the CD-ROM, without installation. Raffael Marty is chief security strategist and senior product manager for Splunk, the leading provider of large-scale, high-speed indexing and search technology for IT infrastructures. As customer advocate and guardian, he focuses on using his skills in data visualization, log management, intrusion detection, and compliance. An active participant on industry standards committees such as CEE (Common Event Expression) and OVAL (Open Vulnerability and Assessment Language), Marty created the Thor and AfterGlow automation tools, and founded the security visualization portal secviz.org. Before joining Splunk, he managed the solutions team at ArcSight, served as IT security consultant for PriceWaterhouseCoopers, and was a member of the IBM Research Global Security Analysis Lab.
Conference Paper
This paper reports on investigations of how computer network defense (CND) analysts conduct their analysis on a day-to-day basis and discusses the implications of these cognitive requirements for designing effective CND visualizations. The supporting data come from a cognitive task analysis (CTA) conducted to baseline the state of the practice in the U.S. Department of Defense CND community. The CTA collected data from CND analysts about their analytic goals, workflow, tasks, types of decisions made, data sources used to make those decisions, cognitive demands, tools used and the biggest challenges that they face. The effort focused on understanding how CND analysts inspect raw data and build their comprehension into a diagnosis or decision, especially in cases requiring data fusion and correlation across multiple data sources. This paper covers three of the findings from the CND CTA: (1) the hierarchy of data created as the analytical process transforms data into security situation awareness; (2) the definition and description of different CND analysis roles; and (3) the workflow that analysts and analytical organizations engage in to produce analytic conclusions.
Security Data Visualization: Graphical Techniques for Network Analysis
  • G Conti
Cyber Vigilance Effects of Signal Probability and Event Rate
  • B Sawyer
  • V Finomore
  • G Funke
  • V Mancuso
  • M Funke
  • G Matthews
  • J Warm
Goal Directed Task Analysis
  • D G Jones
  • M R Endsley
Understanding the Cyber Defender: A Cognitive Task Analysis of Information Assurance Analysts
  • A Amico
  • D Tesone
  • K Whitley
  • B O'brien
  • M Smith
  • E Roth