No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Visualization evaluation for cyber security
Abstract
The Visualization for Cyber Security research community (VizSec) addresses longstanding challenges in cyber security by adapting and evaluating information visualization techniques with application to the cyber security domain. This research effort has created many tools and techniques that could be applied to improve cyber security, yet the community has not yet established unified standards for evaluating these approaches to predict their operational validity. In this paper, we survey and categorize the evaluation metrics, components, and techniques that have been utilized in the past decade of VizSec research literature. We also discuss existing methodological gaps in evaluating visualization in cyber security, and suggest potential avenues for future research in order to help establish an agenda for advancing the state-of-the-art in evaluating cyber security visualizations.
... In another example, Happa et al. [15] conducted a user evaluation study of decision support tools for SOC analysts. Staheli et al. [71] surveyed the trends in research on visualization for cybersecurity and found out that usability testing, simulations, and surveys are the most common approaches to the evaluation. However, evaluations of cybersecurity tools are still rare [71], and the evaluations of tools supporting CSA are nearly non-existent [8]. ...
... Staheli et al. [71] surveyed the trends in research on visualization for cybersecurity and found out that usability testing, simulations, and surveys are the most common approaches to the evaluation. However, evaluations of cybersecurity tools are still rare [71], and the evaluations of tools supporting CSA are nearly non-existent [8]. Thus, there is a need to fill this gap. ...
The growing size and complexity of today’s computer network make it hard to achieve and maintain so-called cyber situational awareness, i.e., the ability to perceive and comprehend the cyber environment and be able to project the situation in the near future. Namely, the personnel of cybersecurity incident response teams or security operation centers should be aware of the security situation in the network to effectively prevent or mitigate cyber attacks and avoid mistakes in the process. In this paper, we present a toolset for achieving cyber situational awareness in a large and heterogeneous environment. Our goal is to support cybersecurity teams in iterating through the OODA loop (Observe, Orient, Decide, Act). We designed tools to help the operator make informed decisions in incident handling and response for each phase of the cycle. The Observe phase builds on common tools for active and passive network monitoring and vulnerability assessment. In the Orient phase, the data on the network are structured and presented in a comprehensible and visually appealing manner. The Decide phase opens opportunities for decision-support systems, in our case, a recommender system that suggests the most resilient configuration of the critical infrastructure. Finally, the Act phase is supported by a service that orchestrates network security tools and allows for prompt mitigation actions. Finally, we present lessons learned from the deployment of the toolset in the campus network and the results of a user evaluation study.
... Cyber-attack perception is an important research problem [12][13][14] which demands better techniques and methods to aid the perception and assessment of cyber-attacks. Quite often, observers find the analysis and understanding of complex patterns difficult to visualise [15,16]. Well-designed diagrams and graphical systems can aid this process [17,18]. ...
... AMTs enable observers to evaluate the salient information in a diagram [16,[21][22][23] and help remove the intellectual burden from security experts -who have to evaluate cyber-attack scenarios and evaluate potential mitigations [24]. Consequently, security problems can be presented in a manner that enables a decision maker -whether an expert or non-expert, to more quickly grasp the problem [22], better perceive risk landscapes [25], and easily perceive complex concepts [26]. ...
Perceiving and understanding cyber-attacks can be a difficult task, and more effective techniques are needed to aid cyber-attack perception. Attack modelling techniques (AMTs) - such as attack graphs, attack trees and fault trees, are a popular method of mathematically and visually representing the sequence of events that lead to a successful cyber-attack. These methods are useful visual aids that can aid cyber-attack perception.
This survey paper describes the fundamental theory of cyber-attack before describing how important elements of a cyber-attack are represented in attack graphs and attack trees. The key focus of the paper is to present empirical research aimed at analysing more than 180 attack graphs and attack trees to identify how attack graphs and attack trees present cyber-attacks in terms of their visual syntax.
There is little empirical or comparative research which evaluates the effectiveness of these methods. Furthermore, despite their popularity, there is no standardised attack graph visual syntax configuration, and more than seventy self-nominated attack graph and twenty attack tree configurations have been described in the literature — each of which presents attributes such as preconditions and exploits in a different way. The survey demonstrates that there is no standard method of representing attack graphs or attack trees and that more research is needed to standardise the representation.
... Visualizations often communicate raw-log insights for detection purposes, or output patterns of detection systems so analysts can prioritise their actions. Common visual variables in the literature include [9]- [14]: ...
... Staheli et al. [14], Harrison and Lu [13], and Shiravi et al. [12] all present in-depth discussions on the state of the art and trends in cybersecurity visualizations. Staheli et al. make a noteworthy point that from 10 years of VizSec literature: "(...) no papers used physiological methods for evaluating security visualizations (...). ...
Complex dependencies exist across the technology estate, users and purposes of machines. This can make it difficult to efficiently detect attacks. Visualization to date is mainly used to communicate patterns of raw logs, or to visualize the output of detection systems. In this paper we explore a novel approach to presenting cybersecurity-related information to analysts. Specifically, we investigate the feasibility of using visualizations to make analysts become anomaly detectors using Pattern-of-Life Visual Metaphors. Unlike glyph metaphors, the visualizations themselves (rather than any single visual variable on screen) transform complex systems into simpler ones using different mapping strategies. We postulate that such mapping strategies can yield new, meaningful ways to showing anomalies in a manner that can be easily identified by analysts. We present a classification system to describe machine and human activities on a host machine, a strategy to map machine dependencies and activities to a metaphor. We then present two examples, each with three attack scenarios, running data generated from attacks that affect confidentiality, integrity and availability of machines. Finally, we present three in-depth use-case studies to assess feasibility (i.e. can this general approach be used to detect anomalies in systems?), usability and detection abilities of our approach. Our findings suggest that our general approach is easy to use to detect anomalies in complex systems, but the type of metaphor has an impact on user’s ability to detect anomalies. Similar to other anomaly-detection techniques, false positives do exist in our general approach as well. Future work will need to investigate optimal mapping strategies, other metaphors, and examine how our approach compares to and can complement existing techniques.
... To the extent that the interface can increase or improve the presentation, understanding, or projection from information, then SA may also reap benefits from better visualizations. However a review in this area, though not focused on SA, has already shown a glut of interfaces that do not receive user evaluation or experimental examination (Staheli et al., 2014). ...
... However, the advantages must be documented, they must be shown to exist. Cyber situation awareness is rife with visualization efforts; but almost no evaluations at all (Staheli et al., 2014). The above findings indicate that visualizations are mostly ideations -some visualizations found in survey were essentially a proof of concept (e.g., Williams, Faithfull, and Roberts, 2012). ...
I review the literature for cyber situation awareness through 2015 (the review will be updated and refined in a formal publication, in prep). In particular, the review is focused on the human aspects of defensive cyber operations - and whether existing work has attempted to create measures of SA, or measure human performance in these environments.
... Actually the fourth type is not a formal case study. Although it is very important to evaluate the visualization system in practice with real users and real data, based on the Staheli et al. survey there are very few published papers that contain formal case studies ( Staheli et al., 2014 ). Scoring rubric . ...
... Conducting usability studies is very important ( Plaisant, 2004 ). The validity and utility of most of them must be viewed with skepticism as they are disconnected from real users or real data or both ( Staheli et al., 2014 ). This problem may arise because of the absence of requirements on providing a systematic evaluation. ...
Visualization helps to comprehend and analyse large amounts of data, a fundamental necessity for network security due to the large volume of audits traces produced each day. In this paper, we dissect the majority of recent work conducted in network security visualization and offer a taxonomy that provides a basis for classifying recently published works using nine criteria. Moreover, a comprehensive evaluation framework for comparing and ranking network security visualization systems and techniques is developed and presented. Finally, we present a taxonomy of network attacks, which covers most of the existing network attacks and provides a framework for the categorization of recent network security visualization systems.
... Various forms of visualization techniques have already been applied successfully in the cyber security community, ranging from 2-dimensional (2D) to 3-dimensional (3D) applications [11], [12]. Existing research in cyber security visualization covers a variety of domains, including for monitoring network traffic characteristics [13], [14] and for visualizing complex attack patterns [15], [16], [17]. ...
... (1, -1, 0) v 8 (1, 0, -1) v 9 (0, 1, 1) v 10 (1, 1, 1) v 11 (1, 1, -1) v 12 (1, -1, 1) v 13 (-1, 1, 1) ...
... A Visualization for Cyber Security research community (VizSec) reports enduring issues for cyber security by familiarizing and examining information visualization approaches for usage in the cyber security arena. Even though this research effort resulted in a large number of tools and approaches that may be utilized to enhance cyber security, the community has failed to set uniform standards for evaluating these methods to verify their practical validity [20,21]. The smart grid may be a very appealing target for attackers as a vital infrastructure because internet-based protocols and open-source software are used for controlling and monitoring [22]. ...
Cyber security is a practice to protect internet-based systems including software, hardware, and data such as networks, computers, mobile devices, electronics systems, and data from illegal attacks or cyber threats. One of the most focused and sensitive areas in today's world is cybersecurity. The main objective of this study is to emphasize the several cyber security attacks and threats under one umbrella. The goal of this work is to examine the literature review on cyber security approaches, datasets, threats, attacks, research trends, challenges, performance metrics, and software used to promote further research in this field. Based on a comprehensive review SWOT analysis is also performed on cyber security. The presented review paper aid researchers in both academia and industry in making advancements in their work in relevant application fields.
... Applying a mindset of capability development and preventive protection mechanisms is a fundamental component of the evolutionary approach to cybersecurity [23,24]. This involves transitioning from conventional static security measures to adaptive defense tactics against cyber threats [25]. ...
Amidst the rapid advancements in the digital landscape, the convergence of digitization and cyber threats presents new challenges for organizational security. This article presents a comprehensive framework that aims to shape the future of cyber security. This framework responds to the complexities of modern cyber threats and provides guidance to organizations to enhance their resilience. The primary focus lies in the integration of capabilities with resilience. By combining these elements into cyber security practices, organizations can improve their ability to predict, mitigate, respond to, and recover from cyber disasters. This article emphasizes the importance of organizational leadership, accountability, and innovation in achieving cyber resilience. As cyber threat challenges continue to evolve, this framework offers strategic guidance to address the intricate dynamics between digitization and cyber security, moving towards a safer and more robust digital environment in the future.
... gaming. The application of this technology to data analysis in the cybersecurity domain is ongoing, but is yet to see significant traction [96,97]. ...
The intent of this research is to develop and assess the application of 3D data visualisation to the field of computer security. The growth of available data relating to computer networks necessitates a more efficient and effective way of presenting information to analysts in support of decision making and situational awareness. Advances in computer hardware and display software have made more complex and interactive presentation of data in 3D possible.
While many attempts at creation of data-rich 3D displays have been made in the field of computer security, they have not become the tool of choice in the industry. There is also a limited amount of published research in the assessment of these tools in comparison to 2D graphical and tabular approaches to displaying the same data.
This research was conducted through creation of a novel abstraction framework for visualisation of computer network data, the Visual Interactive Network Analysis Framework (VINAF). This framework was implemented in software and the software prototype was assessed using both a procedural approach applied to a published forensics challenge and also through a human participant based experiment.
The key contributions to the fields of computer security and data visualisation made by this research include the creation of a novel abstraction framework for computer network traffic which features several new visualisation approaches. An implementation of this software was developed for the specific cybersecurity related task of computer network traffic analysis and published under an open source license to the cybersecurity community. The research contributes a novel approach to human-based experimentation developed during the COVID-19 pandemic and also implemented a novel procedure-based testing approach to the assessment of the prototype data visualisation tool.
Results of the research showed, through procedural experimentation, that the abstraction framework is effective for network forensics tasks and exhibited several advantages when compared to alternate approaches. The user participation experiment indicated that most of the participants deemed the abstraction framework to be effective in several task related to computer network traffic analysis. There was not a strong indication that it would be preferred over existing approaches utilised by the participants, however, it would likely be used to augment existing methods.
... A useable security product should also enable users to make informed decisions in a specific business context [17,18]. Staheli [19] discussed some dimensions for evaluating security visualization for human-machine collaborative systems, including user experience and preference, usability and learnability, effect on collaboration, insights generation, task demands, cognitive workload, and component interoperability. In the research reported in this paper we focused on whether the visualization supported a variety of user tasks, and the extent to which the tool had a positive impact on situation awareness in representative contexts. ...
Employees who have legitimate access to an organization's data may occasionally put sensitive corporate data at risk, either carelessly or maliciously. Ideally, potential breaches should be detected as soon as they occur, but in practice there may be delays, because human analysts are not able to recognize data exfiltration behaviors quickly enough with the tools available to them. Visualization may improve cybersecurity situation awareness. In this paper, we present a dashboard application for investigating file activity, as a way to improve situation awareness. We developed this dashboard for a wide range of stakeholders within a large financial services company. Cybersecurity experts/analysts, data owners, team leaders/managers, high level administrators, and other investigators all provided input to its design. The use of a co-design approach helped to create trust between users and the new visualization tools, which were built to be compatible with existing work processes. We discuss the user-centered design process that informed the development of the dashboard, and the functionality of its three inter-operable monitoring dashboards. In this case three dashboards were developed covering high-level overview, file volume/type comparison, and individual activity, but the appropriate number and type of dashboards to use will likely vary according to the nature of the detection task). We also present two use cases with usability results and preliminary usage data. The results presented examined the amount of use that the dashboards received as well as measures obtained using the Technology Acceptance Model (TAM). We also report user comments about the dashboards and how to improve them.
... A useable security product should also enable users to make informed decisions in a specific business context [17,18]. Staheli [19] discussed some dimensions for evaluating security visualization for human-machine collaborative systems, including user experience and preference, usability and learnability, effect on collaboration, insights generation, task demands, cognitive workload, and component interoperability. In the research reported in this paper we focused on whether the visualization supported a variety of user tasks, and the extent to which the tool had a positive impact on situation awareness in representative contexts. ...
... It is changing rapidly and each day new challenges occur. To design an IDS which is smart enough to work in this dynamic environment we make use of Computational Intelligence [4]. Computational intelligence is a field that uses different bio-inspired approaches to develop such a system that can solve complex problems in a dynamically changing environment [20]. ...
In this era of connectivity where billions of devices are interconnected for various purposes. As the network expands new threats are emerging requiring researchers to work for an effective Intrusion Detection System (IDS) that can mitigate those attacks. In this paper, we discuss common cyber-attacks and IDS used to cope with them. We discuss the use of Computational Intelligence in designing an IDS which can effectively work on unseen data. Because in today’s dynamically changing world, it is essential to have IDS that can work on unseen data (Schatz et al. in Journal of Digital Forensics, Security and Law 12:8, 2017; Kott in Towards fundamental science of cyber security. Springer, New York, NY, pp. 1–13).KeywordsCybersecurityAttacksThreatsComputational intelligence
... Staheli et al. [23] provide a survey of visualization evaluations for cyber security. The authors identify the most common evaluation types for security applications and discuss future directions. ...
The dynamics of cyber threats are increasingly complex, making it more challenging than ever for organizations to obtain in-depth insights into their cyber security status. Therefore, organizations rely on Cyber Situational Awareness (CSA) to support them in better understanding the threats and associated impacts of cyber events. Due to the heterogeneity and complexity of cyber security data, often with multidimensional attributes, sophisticated visualization techniques are needed to achieve CSA. However, there have been no previous attempts to systematically review and analyze the scientific literature on CSA visualizations. In this paper, we systematically select and review 54 publications that discuss visualizations to support CSA. We extract data from these papers to identify key stakeholders, information types, data sources, and visualization techniques. Furthermore, we analyze the level of CSA supported by the visualizations, alongside examining the maturity of the visualizations, challenges, and practices related to CSA visualizations to prepare a full analysis of the current state of CSA in an organizational context. Our results reveal certain gaps in CSA visualizations. For instance, the largest focus is on operational-level staff, and there is a clear lack of visualizations targeting other types of stakeholders such as managers, higher-level decision makers, and non-expert users. Most papers focus on threat information visualization, and there is a dearth of papers that visualize impact information, response plans, and information shared within teams. Interestingly, we find that only a few studies proposed visualizations to facilitate up to the
projection
level (i.e., the highest level of CSA), whereas most studies facilitated only the
perception
level (i.e., the lowest level of CSA). Most of the studies provide evidence of the proposed visualizations through toy examples and demonstrations, while only a few visualizations are employed in industrial practice. Based on the results that highlight the important concerns in CSA visualizations, we recommend a list of future research directions.
... Staheli et al. [23] provide a survey of visualization evaluations for cyber security. The authors identify the most common evaluation types for complex security applications and reveal trends and future directions. ...
The dynamics of cyber threats are increasingly complex, making it more challenging than ever for organizations to obtain in-depth insights into their cyber security status. Therefore, organizations rely on Cyber Situational Awareness (CSA) to support them in better understanding the threats and associated impacts of cyber events. Due to the heterogeneity and complexity of cyber security data, often with multidimensional attributes, sophisticated visualization techniques are often needed to achieve CSA. However, there have been no attempts to systematically review and analyze scientific literature on CSA visualizations until now. In this paper, we have systematically selected and reviewed 54 publications that discuss visualizations to support CSA. We extracted data from these papers to identify key stakeholders, information types, data sources, and visualization techniques. Furthermore, we analyze the level of CSA supported by the visualizations, maturity of the visualizations, challenges, and practices related to CSA visualizations to prepare a full analysis of the current state of CSA in the organizational context. Our results reveal certain gaps in CSA visualizations. For instance, the most focus is on operational-level staff and there is a clear lack of visualizations targeting other types of stakeholders such as managers, higher-level decision makers, and non-expert users. Most papers focus on threat information visualization and there is a lack of papers that visualize impact information, response plans, and information shared within teams. Interestingly, only a few studies proposed visualizations to facilitate up to projection level (i.e. the highest level of CSA) whereas most studies facilitated perception level (i.e. the lowest level of CSA). Based on the results that highlight the important concerns in CSA visualizations, we recommend a list of future research directions.
... Esto requiere de un estudio detallado del propósito y el alcance para el cual fue diseñado cada componente en la aplicación, además, de una correcta selección de las métricas y aplicación acertada de las técnicas de evaluación. Acorde con la clasificación brindada por Staheli [96] sobre las técnicas más utilizadas para validar herramientas de etiquetado de tráfico basado en soluciones visuales, las evaluaciones más comunes son las pruebas de usabilidad de la aplicación, la simulación de procesos y las pruebas de ejecución. ...
Las redes de computadoras se han vuelto indispensables para el intercambio de información entre personas y organizaciones. Por ello, su seguridad representa en la actualidad un gran desafío para la comunidad informática. Hoy en día, son varias las técnicas que se emplean para proteger la información de intrusos en la red. Dentro de las técnicas más utilizadas se encuentran, los conocidos sistemas de detección de intrusos (NIDS por sus siglas en inglés). Quienes, más allá de la autenticación de usuarios, el cifrado de datos y los cortafuegos, se utilizan ampliamente como defensa activa del entorno de la red. A grandes rasgos, un NIDS representa un proceso activo que supervisa el tráfico de la red para identificar las violaciones de la seguridad e iniciar las medidas pertinentes. Producto del crecimiento de la Internet y la interacción de esta con los usuarios, los NIDS requieren constantemente de formas de adaptarse al entorno cambiante de la red o corren el riesgo de convertirse en sistemas obsoletos. Entre las estrategias más utilizadas para hacer frente a los constantes entornos cambiantes se encuentra la generación de NIDS basados en modelos estadísticos de detección y aprendizaje de máquina. Estos sistemas han tenido éxitos sobre otros, debido a su capacidad para identificar patrones de comportamiento y poder predecir patrones similares a futuro, pudiendo seguir el proceso evolutivo de la red. Sin embargo, justo antes de desplegar este tipo de NIDS en cualquier entorno del mundo real, un NIDS basado en modelos estadísticos debe ser entrenado y evaluado usando trazas de tráfico de red etiquetadas que representen la diversidad real del tráfico. Es por ello que, una de las principales limitaciones de este tipo de NIDS es la carencia de conjuntos de datos etiquetados con las características apropiadas. Carencia que se encuentra asociada, entre otros factores, a la falta de recursos y la dificultad del proceso de creación de estos conjuntos de datos etiquetados. La generación y etiquetado de conjuntos de datos útiles para la seguridad de las redes, es un área de estudio relativamente joven que abarca no más allá de los últimos 20 años. En el ámbito de la seguridad, el proceso de etiquetado de un conjunto de datos de tráfico de red es especialmente difícil y se requieren conocimientos especializados para realizar la clasificación de las trazas de tráfico en eventos de índole malicioso o eventos normales. Por ello, la mayoría de las estrategias de etiquetado de tráfico están basadas en la generación automática de trazas de tráfico artificiales con conocimientos de comportamientos conocidos de antemano. Esto se debe, en gran medida, para evitar lidiar con las dificultades del etiquetado y las limitantes de seguridad de la información que conlleva el uso de tráfico real. Esto trae como consecuencia, que en la mayoría de las ocasiones los conjuntos de datos resultantes no expresan el verdadero flujo de la información en la red. Como alternativas, se han propuesto otras técnicas que incluyen el aprendizaje automático, la visualización y una combinación de ambos (método interactivo) para acelerar el proceso de generación de datos sobre el tráfico real. Estas estrategias tienen como objetivo trabajar con tráfico en redes auténticas y acelerar el proceso de etiquetado. Lo cierto es que gran parte del etiquetado que se realiza sobre trazas de red se lleva a cabo de manera manual por usuarios expertos en el análisis de tráfico. Justamente, el uso de expertos se considera un recurso escaso que requieren años de práctica y preparación pero su uso se justifica fundamental para lograr una mayor precisión de las etiquetas. Sin embargo, en muchas ocasiones no se les brinda soporte alguno que asista a los expertos en la tarea de diferenciar dentro del tráfico el comportamiento malicioso del normal. Lo cierto es que a pesar del empleo de varias técnicas, los conjuntos de datos etiquetados sobre el tráfico de red siguen siendo un recurso altamente demandado. Es por ello que, se requieren técnicas novedosas de etiquetado, puntualmente enfocadas a la clasificación de trazas sobre tráfico real. Por todo lo antes expuesto, el presente documento presenta una Tesis de Doctorado que aporta soluciones sobre las principales limitaciones presentes en el proceso de etiquetado de trazas red. Puntualmente, se realizan aportes a través de una estrategia de etiquetado de tráfico real basado en el uso de usuarios no necesariamente expertos. La estrategia propuesta, representa un etiquetado manual asistido a través de una herramienta con componentes visuales y de aprendizaje de máquina. El usuario, durante gran parte del proceso de etiquetado queda acoplado dentro de un ciclo hombre-máquina conocido como Aprendizaje Activo. Luego, para validar la eficacia de las herramientas desarrolladas, se desarrolló un estudio con usuarios y se planteó una metodología de evaluación sobre la solución de Aprendizaje Activo en entornos reales de etiquetado.
... In a context where ASes rely on cybersecurity specialists to make critical decisions regarding threats, it is necessary to structure and categorize data such that visualization "makes sense" to the analyst [230]. As a cooperative defense involves multi-disciplinary concepts and the decision-making process usually requires a low response time from the user, selecting an appropriate type of graphical representation and flow of interaction is not a straightforward task [126]. ...
... DK, for do not know, was also a possibility. These questions are based on the work of Staheli et al. [105] and Angelini et al. [106]. The first three questions evaluate the relevance of the addressed problem (Q1), VEGAS adequateness to this problem (Q2) and the relevance of interactiveness (Q3). ...
A security operations center, SOC, is a key element for the security of information systems. In this thesis, weexhibited the limitations of SOCs and proposed a process associated with two tools to answer them. Ourcontributions enable a better collaboration between the security analysts working in SOCs and facilitate securityevents triage thanks to visualization.
... We find almost no research measuring SA in the cybersecurity environment [Gutzwiller 2019]. Interestingly, this is despite a wealth of reports claiming that a new or unique tool or interface, often untested, could improve it ( [Gutzwiller 2019]; and see the important review by Staheli et al. [2014] of cyber visualizations in research). The claims in research literature mirror claims by industrial software solutions, which many cyber professionals abhor and abandon. ...
Demand is present among security practitioners for improving cyber situational awareness (SA), but capability and assessment have not risen to match. SA is an integral component of cybersecurity for everyone from individuals to business to response teams and threat exchanges. In this Field Note, we highlight existing research and our field observations, a recent review of cyber SA research literature, and call upon the research community to help address three research problems in situational awareness for cybersecurity. The gaps suggest the need to (1) understand what cyber SA is from the human operators? perspectives, then (2) measure it so that (3) the community can learn whether SA makes a difference in meaningful ways to cybersecurity, and whether methods, technology, or other solutions would improve SA and thus, improve those outcomes.
... Data processing algorithms and advanced analytics shall be the foundations of our cyber-security culture tool. Even among business that are intrinsically interlinked with information systems not all members of the organisations have the same knowledge set and even when dealing with sector experts novel cyber visualisations have been observed to be either too complex or too basic for the intended users (Diane Staheli, 2014). As such it is necessary to ensure that our results are presented with the appropriate complexity and tools corresponding to the knowledge level of each user, from the personal to the organisational level, to achieve the maximum impact. ...
... Many works have addressed the challenges related to the design or evaluation of cybersecurity tools and techniques [14]- [18]. A visual analytics approach to automated planning attacks has been discussed [19]. ...
Hands-on training is an effective way to practice theoretical cybersecurity concepts and increase participants' skills. In this paper, we discuss the application of visual analytics principles to the design, execution, and evaluation of training sessions. We propose a conceptual model employing visual analytics that supports the sensemaking activities of users involved in various phases of the training life cycle. The model emerged from our long-term experience in designing and organizing diverse hands-on cybersecurity training sessions. It provides a classification of visualizations and can be used as a framework for developing novel visualization tools supporting phases of the training life-cycle. We demonstrate the model application on examples covering two types of cybersecurity training programs.
... Computer generation of 3D-looking images goes back to computer games of the 1980's, such as Battlezone (Rotberg, 1980), and since then the gaming industry has driven significant advances in the realism and immersive quality of 3D gaming. The application of this technology to data analysis in the cyber security domain is ongoing, but is yet to see significant traction (Goodall, 2009, Staheli et al., 2014. ...
This paper outlines a novel approach to 3D visualization of network traffic. Existing approaches, which present node-graphs in 3D space may not be making the best use of the advantages of 3D. By combining the time component of network traffic data with nodal information and displaying these on separate planes it should be possible to provide analysts with insights that go beyond just the nodal information. The goal of allowing analysts to quickly form a mental map that corresponds with the network traffic ground truth may be achieved with this approach. The visualization approach is demonstrated through development of a tool which implements the approach and discusses its application to a recent network forensics challenge.
... Many works have addressed the challenges related to the design or evaluation of cybersecurity tools and techniques [14]- [18]. A visual analytics approach to automated planning attacks has been discussed [19]. ...
Hands-on training is an effective way to practice theoretical cybersecurity concepts and increase participants' skills. In this paper, we discuss the application of visual analytics principles to the design, execution, and evaluation of training sessions. We propose a conceptual model employing visual analytics that supports the sensemaking activities of users involved in various phases of the training life cycle. The model emerged from our long-term experience in designing and organizing diverse hands-on cybersecurity training sessions. It provides a classification of visualizations and can be used as a framework for developing novel visualization tools supporting phases of the training life-cycle. We demonstrate the model application on examples covering two types of cybersecurity training programs.
... To assess requirement R6, the ability of laymen (in terms of cyber security) to verify results of the anomaly detection, the evaluation was performed with 15 subjects, where 11 have a technical background (IT/electrical engineering) and none have experience in cyber security. This is also in consent with the findings of Staheli et al. in [26]. After a short introduction to the system, the users performed several tasks and filled the questionnaire on effectiveness along the way. ...
Operation technology networks, i.e. hard- and software used for monitoring and controlling physical/industrial processes, have been considered immune to cyber attacks for a long time. A recent increase of attacks in these networks proves this assumption wrong. Several technical constraints lead to approaches to detect attacks on industrial processes using available sensor data. This setting differs fundamentally from anomaly detection in IT-network traffic and requires new visualization approaches adapted to the common periodical behavior in OT-network data. We present a tailored visualization system that utilizes inherent features of measurements from industrial processes to full capacity to provide insight into the data and support triage analysis by laymen and experts. The novel combination of spiral plots with results from anomaly detection was implemented in an interactive system. The capabilities of our system are demonstrated using sensor and actuator data from a real-world water treatment process with introduced attacks. Exemplary analysis strategies are presented. Finally, we evaluate effectiveness and usability of our system and perform an expert evaluation.
... One of the most important reasons why cyber security stakeholders want to visualize data in cyber security domain is to understand and explain large amounts of data, which otherwise overwhelms an expert due to its huge size. To cope with the scale and complexity of the challenges posed by these data, cyber security analysts have powerful analytical and visualization tools [12]. ...
Visual Analytics is a complex sub-field of data analytics that concentrates on the use of the information visualization methods for facilitating effective analysis of data by employing visual and graphical representation. In cyber security domain, Effective visualization of the data allows to infer valuable insights that enable domain analysts to construct successful strategies to mitigate cyber attacks and provide decision support. We perform a survey of the state-of-the-art in the cyber security domain, analyze main challenges and discuss future trends. We summarize a large number of cyber security and digital forensics visualization works using the Five Question Method of Five W’s and How (Why, Who, What, How, When, and Where) approach as a methodological background. We perform analysis of the works using J. Bertin’s Semiotic Theory of Graphics, and VIS4ML ontology as a theoretical foundation of visual analytics. As a result, we formulate the main challenges for the development of this area of research in the future.
... This section presents two usage scenarios [27,38], showing how SymNav assists a human in making sensible decisions when applying symbolic execution techniques to the domains of malware analysis and vulnerability detection. In both scenarios we use BFS as search heuristic for the exploration, which is also the default one in angr. ...
Modern software systems require the support of automatic program analyses to answer questions about their correctness, reliability, and safety. In recent years, symbolic execution techniques have played a pivotal role in this field, backing research in different domains such as software testing and software security. Like other powerful machine analyses, symbolic execution is often affected by efficiency and scalability issues that can be mitigated when a domain expert interacts with its working, steering the computation to achieve the desired goals faster. In this paper we explore how visual analytics techniques can help the user to grasp properties of the ongoing analysis and use such insights to refine the symbolic exploration process. To this end, we discuss two real-world usage scenarios from the malware analysis and the vulnerability detection domains, showing how our prototype system can help users make a wiser use of symbolic exploration techniques in the analysis of binary code.
... For an evaluation to be useful, one must consider its purpose and scope, select the appropriate metrics and correctly apply assessment techniques. According to the classification given by Staheli et al. [22] for the commonly-used techniques for evaluating visualization the most common evaluations are Usability Testing, Simulation and Performing Testing. We present an evaluation framework to analyze application performance using one kind of Simulation and the Application Performance Testing technique, leaving the evaluation with users for a later work. ...
In the field of network security, the process of labeling a network traffic dataset is specially expensive since expert knowledge is required to perform the annotations. With the aid of visual analytic applications such as RiskID, the effort of labeling network traffic is considerable reduced. However, since the label assignment still requires an expert pondering several factors, the annotation process remains a difficult task. The present article introduces a novel active learning strategy for building a random forest model based on user previously-labeled connections. The resulting model provides to the user an estimation of the probability of the remaining unlabeled connections helping him in the traffic annotation task. The article describes the active learning strategy, the interfaces with the RiskID system, the algorithms used to predict botnet behavior, and a proposed evaluation framework. The evaluation framework includes studies to assess not only the prediction performance of the active learning strategy but also
the learning rate and resilience against noise as well as the improvements on other well known labeling strategies. The framework represents a complete methodology for evaluating the performance of any active learning solution. The evaluation results showed proposed approach is a significant improvement
over previous labeling strategies
... Furthermore, when compared with numeric data presentation, visual representation is more inspiring and intuitive [8]. Researchers have already investigated various forms of visualization for applications in network security, ranging from two-dimensional (2D) to three-dimensional (3D) visualization approaches [9,10]. In addition, research in network security visualization covers a variety of domains, including for monitoring network traffic characteristics [11,12] and for visualizing complex attack patterns [13,14,15]. ...
The threat of cyber-attacks is on the rise in the digital world today. As such, effective cybersecurity solutions are becoming increasingly important for detecting and combating cyber-attacks. The use of machine learning techniques for network intrusion detection is a growing area of research, as these techniques can potentially provide a means for automating the detection of attacks and abnormal traffic patterns in real-time. However, misclassification is a common problem in machine learning for intrusion detection, and the improvement of machine learning models is hindered by a lack of insight into the reasons behind such misclassification. This paper presents an interactive method of visualizing network intrusion detection data in three-dimensions. The objective is to facilitate the understanding of network intrusion detection data using a visual representation to reflect the geometric relationship between various categories of network traffic. This interactive visual representation can potentially provide useful insight to aid the understanding of machine learning results. To demonstrate the usefulness of the proposed visualization approach, this paper presents results of experiments on commonly used network intrusion detection datasets.
... The current prototypical implementation 3 of our visualization for IAM was developed in co-creation with experts as suggested by Staheli et al. [42]. larly conducted semi-structured interviews with the participating practitioners to ensure that the implementation fits their needs and requirements. ...
Enterprises have embraced identity and access management (IAM) systems as central point to manage digital identities and to grant or remove access to information. However, as IAM systems continue to grow, technical and organizational challenges arise. Domain experts have an incomparable amount of knowledge about an organization’s specific settings and issues. Thus, especially for organizational IAM challenges to be solved, leveraging the knowledge of internal and external experts is a promising path. Applying Visual Analytics (VA) as an interactive tool set to utilize the expert knowledge can help to solve upcoming challenges. Within this work, the central IAM challenges with need for expert integration are identified by conducting a literature review of academic publications and analyzing the practitioners’ point of view. Based on this, we propose an architecture for combining IAM and VA. A prototypical implementation of this architecture showcases the increased understanding and ways of solving the identified IAM challenges.
... Information visualization techniques can potentially bridge the gap between the performance of machine learning models and understanding factors that contribute to its performance. Visualization, whether in 2-dimensions (2D) or 3-dimensions (3D), also plays an important role in the cyber security domain [14]. In addition, previous work has shown that complex attack patterns in NIDS can be visualized in various forms [1] [9]. ...
Nowadays, network intrusion detection is researched extensively due to increasing global network threats. Many researchers propose to incorporate machine learning techniques in network intrusion detection systems since these techniques allow for automated intrusion detection with high accuracy. Furthermore, dimensionality reduction techniques can improve the performance of machine learning models, and as such, are widely used as a pre-processing step. Nevertheless, many researchers consider machine learning techniques as a black box because of its complex intrinsic mechanism. Visualization plays an important role in facilitating the understanding of such sophisticated techniques because visualization is able to offer intuitive meaning to the machine learning results. This research investigates the performance of two dimensionality reduction techniques on network intrusion detection datasets. In addition, this work also demonstrates visualizing the resulting data in 3-dimensional space. The purpose of this is to possibly gain insight into the results, which can potentially aid in the improvement of machine learning performance.
... Prior to this survey study, besides investigating existing survey papers (Staheli, et al., 2014) (Shiravi, et al., 2012), an extended literature work for the security visualization domain has been done by the authors. During this literature search in order to understand existing situation, different aspects of the designs are examined including design issues, display types, use-cases, common interactivity ways and common validation methods for the domain. ...
In order to find gaps or missing points in any domain, examination of the literature work is necessary and provides a good amount of information. Doing a requirement analysis on top of this literature search incorporating the domain experts is a convenient way to find out ideas to fill out the detected gaps. The security visualization domain has been popular for the latest twenty years. There have been many designs. However, our literature analyses work resulted with the conclusion that the majority of the earlier security visualization work focuses a known set of use-cases, and these are trying to be validated using these small sets of vulnerabilities and some commonly known threats through a few case studies or experimental results. In this work, a security visualization requirement analysis survey with 30 information security experts is done. The paper presents the qualitative and quantitative results of this survey.
Cyber security analysts use data visualizations to speed up ingestion of security data. These visualizations typically take the form of 2D graphics displayed on computer monitors. Virtual reality has the potential to improve these visualizations with immersive 3D environments and unique interaction mechanics. However, research into this newly synergised area lacks evaluation, leading to unfounded claims of effectiveness. A potential cause for these missing evaluations was identified as a lack of guidance detailing how evaluations should be conducted in this area. Additionally, the small amount of research that does include evaluation incorrectly relies on subjective participant opinions to objectively measure system effectiveness. An example of this misuse is asking participants which system they thought was quicker, rather than timing them. The objective of this paper was to propose a solution to these issues in the form of a surveyed, categorised, and analysed set of evaluation metrics. A total of 49 metrics were identified from 41 papers. The presented metrics detail which dependent variables should be considered when evaluating works in the combined fields of cyber security, data visualization, and virtual reality. These metrics can be used to produce more accurate evaluations in future works in this area.
Supervisory Control and Data Acquisition (SCADA) systems are widely adopted in critical infrastructures and prime targets of cyberattacks. Ecological Interface Design (EID) is postulated to be an invaluable framework for supporting operators to cope with cyber intrusions, particularly zero-day attacks because prior research has demonstrated effectiveness of ecological interfaces during unanticipated events. However, a suitable research platform is absent for studying user interface in cybersecurity of SCADA systems. This paper presents a SCADA system simulation being designed and implemented for the DURESS thermohydraulic process control simulation common in EID studies. Based on the open literature and industrial standards to ensure representativeness of industrial SCADA systems, the simulation includes two programable logical controllers, seven routers, and a server in a wired communication network. These components should be sufficient to study human response to common cyberattacks on SCADA systems and support future work in prototyping and evaluating user interfaces for SCADA cybersecurity.
Cybersecurity is an extremely important matter in the current business world due to increasing cyber threats. Cyber attackers have increased their cyber-attacks on almost all the business operations using various advanced techniques. All kind of business organization ranging from small scale to large organizations have been impacted. So, cybersecurity has become a necessity for all kinds of firms, and adopting the secured techniques of business transactions has become a prerequisite of the business. The authors wish to find the conceptual framework of cybersecurity, associated risks related to cybersecurity, ways for ensuring cybersecurity in businesses, emerging trends in cybersecurity, and different initiatives taken in India for ensuring cybersecurity.
In this contemporary era internet of things are used in every realm of life. Recent software’s (e.g., vehicle networking, smart grid, and wearable) are established in result of its use: furthermore, as development, consolidation, and revolution of varied ancient areas (e.g., medical and automotive). The number of devices connected in conjunction with the ad-hoc nature of the system any exacerbates the case. Therefore, security and privacy has emerged as a big challenge for the IoT. This paper provides an outline of IoT security attacks on Three-Layer Architecture: Three-layer such as application layer, network layer, perception layer/physical layer and attacks that are associated with these layers will be discussed. Moreover, this paper will provide some possible solution mechanisms for such attacks. The aim is to produce a radical survey associated with the privacy and security challenges of the IoT. The objective of this paper is to rendering possible solution for various attacks on different layers of IoT architecture. It also presents comparison based on reviewing multiple solutions and defines the best one solution for a specific attack on particular layer.KeywordsInternet of thingsSecurity and privacyIoT layersAttacks with solution mechanism
Computer vision has a great potential to deal with agriculture problems. It is crucial to utilize novel tools and techniques in the agriculture food industry. The focus of current studies is to automate the fruit harvesting, grading of fruits, fruit recognition, and identification of diseases in the agriculture domain using deep learning and computer vision. Integrating deep learning with computer vision facilitates the consistent, speedy and trustworthy classification of fruit and vegetables compared to the traditional machine learning algorithm. However, there are still some challenges, such as the need for expert farmers to develop large-scale datasets to recognize and identify the problems of agriculture production. This survey includes eighty papers relevant to deep learning and computer vision techniques in the agriculture field.KeywordsDeep learningObject detectionComputer visionYield estimation
In this paper, a comprehensive evaluation of social network analysis approaches performed with Cybersecurity prospect to analyze and visualize cybersecurity information. this paper help to understand the supporting features and their relevancy to security. However, these approaches are open source and supporting to many Operating system so these are easy to access and can be used by individuals to get their desired output.KeywordsSocial network analysisGephiTulipPajekCybersecuritySecurityVisualization of security
Cyberthreats are continually evolving and growing in numbers and extreme complexities with the increasing connectivity of the Internet of Things (IoT). Existing cyber-defense tools seem not to deter the number of successful cyberattacks reported worldwide. If defense tools are not seldom, why does the
cyber-chase
trend favor bad actors? Although cyber-defense tools monitor and try to diffuse intrusion attempts, research shows the required agility speed against evolving threats is way too slow. One of the reasons is that many intrusion detection tools focus on anomaly alerts’ accuracy, assuming that preobserved attacks and subsequent security patches are adequate. Well, that is not the case. In fact, there is a need for techniques that go beyond intrusion accuracy against specific vulnerabilities to the prediction of cyber-defense performance for improved proactivity. This article proposes a combination of cyberattack projection and cyber-defense agility estimation to dynamically but reliably augur intrusion detection performance. Since cyber-security is buffeted with many unknown parameters and rapidly changing trends, we apply a machine-learning (ML)-based hidden Markov model (HMM) to predict intrusion detection agility. HMM is best known for robust prediction of temporal relationships mid noise and training brevity corroborating our high prediction accuracy on three major open-source network intrusion detection systems, namely, Zeek, OSSEC, and Suricata. Specifically, we present a novel approach for combined projection, prediction, and cyber-visualization to enable precise agility analysis of cyber defense. We also evaluate the performance of the developed approach using numerical results.
Security visualization has been an issue, and it continues to grow in many directions. In order to give sufficient security visualization designs, information both in many different aspects of visualization techniques and the security problems is required. More beneficial designs depend on decisions that include use cases covering security artifacts and business requirements of the organizations, correct and optimal use of data sources, and selection of proper display types. To be able to see the big picture, the designers should be aware of available data types, possible use cases and different styles of displays. In this chapter, these properties of a large set of earlier security visualization work have been depicted and classified using both textual and graphical ways. This work also contains information related to trending topics of the domain, ways of user interaction, evaluation, and validation techniques that are commonly used for the security visualization designs.
Operation technology networks, i.e. hard- and software used for monitoring and controlling physical/industrial processes, have been considered immune to cyber attacks for a long time. A recent increase of attacks in these networks proves this assumption wrong. Several technical constraints lead to approaches to detect attacks on industrial processes using available sensor data. This setting differs fundamentally from anomaly detection in IT-network traffic and requires new visualization approaches adapted to the common periodical behavior in OT-network data. We present a tailored visualization system that utilizes inherent features of measurements from industrial processes to full capacity to provide insight into the data and support triage analysis by laymen and experts. The novel combination of spiral plots with results from anomaly detection was implemented in an interactive system. The capabilities of our system are demonstrated using sensor and actuator data from a real-world water treatment process with introduced attacks. Exemplary analysis strategies are presented. Finally, we evaluate effectiveness and usability of our system and perform an expert evaluation.
In the field of network security, the process of labeling a network traffic dataset is specially expensive since expert knowledge is required to perform the annotations. With the aid of visual analytic applications such as RiskID, the effort of labeling network traffic is considerable reduced. However, since the label assignment still requires an expert pondering several factors, the annotation process remains a difficult task. The present article introduces a novel active learning strategy for building a random forest model based on user previously-labeled connections. The resulting model provides to the user an estimation of the probability of the remaining unlabeled connections helping him in the traffic annotation task. The article describes the active learning strategy, the interfaces with the RiskID system, the algorithms used to predict botnet behavior, and a proposed evaluation framework. The evaluation framework includes studies to assess not only the prediction performance of the active learning strategy but also the learning rate and resilience against noise as well as the improvements on other well known labeling strategies. The framework represents a complete methodology for evaluating the performance of any active learning solution. The evaluation results showed proposed approach is a significant improvement over previous labeling strategies.
Visualization tools are critical components of cyber security systems allowing analyzers to better understand, detect and prevent security breaches. Security administrators need to understand which users accessed the database and what operations were performed in order to detect irregularities. The current work compares the Sankey diagram with the more commonly used node-link diagram as an alternative visualization technique for cyber security tasks in a controlled experiment. The results indicate, that the Sankey tool showed a consistent advantage in task completion time and was more effective (measured by the percent of correct answers) in synoptic tasks, while the Node-link diagram was more effective in basic, elementary tasks. Further results revealed that performance had only a small effect on user satisfaction and preferences. Our results suggest that the Sankey tool may be a viable option for cyber security visualization tools and strengthens the need to provide personalized visualization tools based on user preferences.
We show how brain sensing can lend insight to the evaluation of visual interfaces and establish a role for fNIRS in visualization. Research suggests that the evaluation of visual design benefits by going beyond performance measures or questionnaires to measurements of the user's cognitive state. Unfortunately, objectively and unobtrusively monitoring the brain is difficult. While functional near-infrared spectroscopy (fNIRS) has emerged as a practical brain sensing technology in HCI, visual tasks often rely on the brain's quick, massively parallel visual system, which may be inaccessible to this measurement. It is unknown whether fNIRS can distinguish differences in cognitive state that derive from visual design alone. In this paper, we use the classic comparison of bar graphs and pie charts to test the viability of fNIRS for measuring the impact of a visual design on the brain. Our results demonstrate that we can indeed measure this impact, and furthermore measurements indicate that there are not universal differences in bar graphs and pie charts.
Dynamic difficulty adjustments can be used in human-computer systems in order to improve user engagement and performance. In this paper, we use functional near-infrared spectroscopy (fNIRS) to obtain passive brain sensing data and detect extended periods of boredom or overload. From these physiological signals, we can adapt a simulation in order to optimize workload in real-time, which allows the system to better fit the task to the user from moment to moment. To demonstrate this idea, we ran a laboratory study in which participants performed path planning for multiple unmanned aerial vehicles (UAVs) in a simulation. Based on their state, we varied the difficulty of the task by adding or removing UAVs and found that we were able to decrease error by 35% over a baseline condition. Our results show that we can use fNIRS brain sensing to detect task difficulty in real-time and construct an interface that improves user performance through dynamic difficulty adjustment.
In this paper, we advocate the use of behavior-based methods for use in evaluating affective interactions. We consider behavior-based measures to include both measures of bodily movements or physiological signals and task-based performance measures.
Designing effective visual analytics systems is challenging. Not only must each component be well
understood and effectively designed on its own, but each must also operate in harmony with the rest. To a large extent,
the quality of the relationships among components determines how well visual analytic activities are supported. In
this paper, we define the quality of interaction among the components of visual analytics systems as interactivity.
This paper draws on research from the areas of cognitive and perceptual psychology, human-information interaction,
visualization sciences, and interaction design to examine some of the current challenges faced in discussing and
characterizing interactivity. In doing so, this paper attempts to contribute to a characterization of interactivity in
visual analytics.
Analysts engaged in real-time monitoring of cybersecurity incidents must quickly and accurately respond to alerts generated by intrusion detection systems. We investigated two complementary approaches to improving analyst performance on this vigilance task: a graph-based visualization of correlated IDS output and defensible recommendations based on machine learning from historical analyst behavior. We tested our approach with 18 professional cybersecurity analysts using a prototype environment in which we compared the visualization with a conventional tabular display, and the defensible recommendations with limited or no recommendations. Quantitative results showed improved analyst accuracy with the visual display and the defensible recommendations. Additional qualitative data from a "talk aloud" protocol illustrated the role of displays and recommendations in analysts' decision-making process. Implications for the design of future online analysis environments are discussed.
The goal of cyber security visualization is to help analysts in-crease the safety and soundness of our digital infrastructures by providing effective tools and workspaces. Visualization research-ers must make visual tools more usable and compelling than the text-based tools that currently dominate cyber analysts' tool chests. A cyber analytics work environment should enable multi-ple, simultaneous investigations and information foraging, as well as provide a solution space for organizing data. We describe our study of cyber-security professionals and visualizations in a large, high-resolution display work environment and the analytic tasks this environment can support. We articulate a set of design princi-ples for usable cyber analytic workspaces that our studies have brought to light. Finally, we present prototypes designed to meet our guidelines and a usability evaluation of the environment.
This chapter surveys methods, techniques, and practices in Participatory Design (PD) that can lead to hybrid experiences - that is, practices that take place neither in the workers' domain, nor in the software professionals' domain, but in an "in-between" region that shares attributes of both the workers' space and the software professionals' space. Recent work in cultural theory claims that this "in-between" region, or "third space," is a fertile environment in which participants can combine diverse knowledges into new insights and plans for action, to inform the needs of their organizations, institutions, products, and services. Important attributes of third space experiences include challenging assumptions, learning reciprocally, and creating new ideas, which emerge through negotiation and co-creation of identities, working languages, understandings, and relationships, and polyvocal (many- voiced) dialogues across and through differences. The chapter focuses on participatory practices that share these attributes, including: site-selection of PD work; workshops; story-collecting and story-telling through text, photography, and drama; games for analysis and design; and the co- creation of descriptive and functional prototypes. Introduction Participatory design (PD) is a set of theories, practices, and studies related to end- users as full participants in activities leading to software and hardware computer products
Information visualization research is becoming more established, and as a result, it is becoming increasingly important that
research in this field is validated. With the general increase in information visualization research there has also been an
increase, albeit disproportionately small, in the amount of empirical work directly focused on information visualization.
The purpose of this chapter is to increase awareness of empirical research in general, of its relationship to information
visualization in particular; to emphasize its importance; and to encourage thoughtful application of a greater variety of
evaluative research methodologies in information visualization.
As the first part of a Analyze-Visualize-Validate cycle, we have initiated a domain analysis of email computer forensics to
determine where visualization may be beneficial. To this end, we worked with police detectives and other forensics professionals.
However, the process of designing and executing such a study with real-world experts has been a non-trivial task. This paper
presents our efforts in this area and the lessons learned as guidance for other practitioners.
A new field of research, visual analytics, has been introduced. This has been defined as "the science of analytical reasoning facilitated by interactive visual interfaces" (Thomas and Cook, 2005). Visual analytic environments, therefore, support analytical reasoning using visual representations and interactions, with data representations and transformation capabilities, to support production, presentation, and dissemination. As researchers begin to develop visual analytic environments, it is advantageous to develop metrics and methodologies to help researchers measure the progress of their work and understand the impact their work has on the users who work in such environments. This paper presents five areas or aspects of visual analytic environments that should be considered as metrics and methodologies for evaluation are developed. Evaluation aspects need to include usability, but it is necessary to go beyond basic usability. The areas of situation awareness, collaboration, interaction, creativity, and utility are proposed as the five evaluation areas for initial consideration. The steps that need to be undertaken to develop systematic evaluation methodologies and metrics for visual analytic environments are outlined
In this paper, we report a study that examines the relationship between image-based computational analyses of web pages and users' aesthetic judgments about the same image material. Web pages were iteratively decomposed into quadrants of minimum entropy (quadtree decomposition) based on low-level image statistics, to permit a characterization of these pages in terms of their respective organizational symmetry, balance and equilibrium. These attributes were then evaluated for their correlation with human participants' subjective ratings of the same web pages on four aesthetic and affective dimensions. Several of these correlations were quite large and revealed interesting patterns in the relationship between low-level (i.e., pixel-level) image statistics and design- relevant dimensions. Author Keywords
Current practice in Human Computer Interaction as encouraged by educational institutes, academic review processes, and institutions with usability groups advocate usability evaluation as a critical part of every design process. This is for good reason: usability evaluation has a significant role to play when conditions warrant it. Yet evaluation can be ineffective and even harmful if naively done 'by rule' rather than 'by thought'. If done during early stage design, it can mute creative ideas that do not conform to current interface norms. If done to test radical innovations, the many interface issues that would likely arise from an immature technology can quash what could have been an inspired vision. If done to validate an academic prototype, it may incorrectly suggest a design's scientific worthiness rather than offer a meaningful critique of how it would be adopted and used in everyday practice. If done without regard to how cultures adopt technology over time, then today's reluctant reactions by users will forestall tomorrow's eager acceptance. The choice of evaluation methodology - if any - must arise from and be appropriate for the actual problem or research question under consideration.
Although both statistical methods and visualizations have been used by network analysts, exploratory data analysis remains a challenge. We propose that a tight integration of these technologies in an interactive exploratory tool could dramatically speed insight development. To test the power of this integrated approach, we created a novel social network analysis tool, SocialAction, and conducted four long-term case studies with domain experts, each working on unique data sets with unique problems. The structured replicated case studies show that the integrated approach in SocialAction led to significant discoveries by a political analyst, a bibliometrician, a healthcare consultant, and a counter-terrorism researcher. Our contributions demonstrate that the tight integration of statistics and visualizations improves exploratory data analysis, and that our evaluation methodology for long-term case studies captures the research strategies of data analysts.
Today's system administrators, burdened by rapidly increasing network activity, must quickly perceive the security state of their networks, but they often have only text-based tools to work with. These tools often provide no overview to help users grasp the big-picture. Our interviews with administrators have revealed that they need visualization tools; thus, we present VISUAL (Visual Information Security Utility for Administration Live), a network security visualization tool that allows users to see communication patterns between their home (or internal) networks and external hosts. VISUAL is part of our Network Eye security visualization architecture, also described in this paper. We have designed and tested a new computer security visualization that gives a quick overview of current and recent communication patterns in the monitored network to the users. Many tools can detect and show fan-out and fan-in, but VISUAL shows network events graphically, in context. Visualization helps users comprehend the intensity of network events more intuitively than text-based tools can. VISUAL provides insight for networks with up to 2,500 home hosts and 10,000 external hosts, shows the relative activity of hosts, displays them in a constant relative position, and reveals the ports and protocols used.
After an historical review of evaluation methods, we describe an emerging research method called Multi-dimensional In-depth Long-term Case studies (MILCs) which seems well adapted to study the creative activities that users of information visualization systems engage in. We propose that the efficacy of tools can be assessed by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users' success in achieving their professional goals. We summarize lessons from related ethnography methods used in HCI and provide guidelines for conducting MILCs for information visualization. We suggest ways to refine the methods for MILCs in modest sized projects and then envision ambitious projects with 3-10 researchers working over 1-3 years to understand individual and organizational use of information visualization by domain experts working at the frontiers of knowledge in their fields.
There is a growing recognition within the visual analytics community that interaction and inquiry are inextricable. It is through the interactive manipulation of a visual interface–the analytic discourse–that knowledge is constructed, tested, refined and shared. This article reflects on the interaction challenges raised in the visual analytics research and development agenda and further explores the relationship between interaction and cognition. It identifies recent exemplars of visual analytics research that have made substantive progress toward the goals of a true science of interaction, which must include theories and testable premises about the most appropriate mechanisms for human–information interaction. Seven areas for further work are highlighted as those among the highest priorities for the next 5 years of visual analytics research: ubiquitous, embodied interaction; capturing user intentionality; knowledge-based interfaces; collaboration; principles of design and perception; interoperability; and interaction evaluation. Ultimately, the goal of a science of interaction is to support the visual analytics and human–computer interaction communities through the recognition and implementation of best practices in the representation and manipulation of visual displays.
We take a new, scenario based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, evaluating visualization algorithms, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios distinguish different study goals and types of research questions and are illustrated through example studies. Through this broad survey and the distillation of these scenarios we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. Scenarios can be used to choose appropriate research questions and goals and the provided examples can be consulted for guidance on how to design one's own study.
With visual analytical tools becoming more sophisticated and prevalent in the analysis communities, it is now apparent that understanding how analysts utilize these tools is more important than ever. Such understanding can lead to improving the tools, but a more subtle and equally important aspect lies in the discovery of the analysts ’ reasoning process for solving complex problems through the use of these visual analytical tools. In this paper we demonstrate that we were able to identify several of the strategies, methods, and findings of an analysis process using a financial visual analytical tool through the examination of an analyst’s interaction log. In our study, we recorded the interactions and think-alouds of 10 financial analysts in a fraud detection task. By examining their interaction logs, we are able to quantitatively show that 60 % of strategies, 60% of methods, and 79 % of findings could be recovered through the use of two visual analytic log analysis tools. 1
In this position paper we discuss successes and limitations of current evaluation strategies for scientific visualizations and argue for embracing a mixed methods strategy of evaluation. The most novel contribution of the approach that we advocate is a new emphasis on employing design processes as practiced in related fields (e.g., graphic design, illustration, architecture) as a formalized mode of evaluation for data visualizations. To motivate this position we describe a series of recent evaluations of scientific visualization interfaces and computer graphics strategies conducted within our research group. Complementing these more traditional evaluations our visualization research group also regularly employs sketching, critique, and other design methods that have been formalized over years of practice in design fields. Our experience has convinced us that these activities are invaluable, often providing much more detailed evaluative feedback about our visualization systems than that obtained via more traditional user studies and the like. We believe that if design-based evaluation methodologies (e.g., ideation, sketching, critique) can be taught and embraced within the visualization community then these may become one of the most effective future strategies for both formative and summative evaluations.
User research is a broad term that encompasses many methodologies, such as usability testing, surveys, questionnaires, and site visits, that generate quantifiable outcomes. Usability testing is a central activity in user research and typically generates the metrics of completion rates, task times, errors, satisfaction data, and user interface problems. You can quantify data from small sample sizes and use statistics to draw conclusions. Even open-ended comments and problem descriptions can be categorized and quantified.
The natural, living world provides the backdrop for the practice of participatory ergonomics. By addressing the salient issues of complexity, context, distributed cognition, and team situational awareness in this world, the advocation of the Living Laboratory concept is presented. Socio-technical systems design is reflected through the joint integration of various Living-Lab outcomes such as Fields of practice, tools, technologies, qualitative models, scaled worlds, and in situ evaluations. A wholistic approach is derived by looking at these outcomes as a basis to bind together the cognitive, social, technological, and organizational constraints that design must consider to be effective.
In this position paper we discuss successes and limitations of current evaluation strategies for scientific visualizations and argue for embracing a mixed methods strategy of evaluation. The most novel contribution of the approach that we advocate is a new emphasis on employing design processes as practiced in related fields (e.g., graphic design, illustration, architecture) as a formalized mode of evaluation for data visualizations. To motivate this position we describe a series of recent evaluations of scientific visualization interfaces and computer graphics strategies conducted within our research group. Complementing these more traditional evaluations our visualization research group also regularly employs sketching, critique, and other design methods that have been formalized over years of practice in design fields. Our experience has convinced us that these activities are invaluable, often providing much more detailed evaluative feedback about our visualization systems than that obtained via more traditional user studies and the like. We believe that if design-based evaluation methodologies (e.g., ideation, sketching, critique) can be taught and embraced within the visualization community then these may become one of the most effective future strategies for both formative and summative evaluations.
This paper describes a web-based visualization system designed for network security analysts at the U.S. Army Research Laboratory (ARL). Our goal is to provide visual support to the analysts as they investigate security alerts for malicious activity within their systems. Our ARL collaborators identified a number of important requirements for any candidate visualization system. These relate to the analyst's mental models and working environment, and to the visualization tool's configurability, accessibility, scalability, and "fit" with existing analysis strategies. To meet these requirements, we designed and implement a web-based tool that uses different types of charts as its core representation framework. A JavaScript charting library (RGraph) was extended to provide the interface flexibility and correlation capabilities needed to support analysts as they explore different hypotheses about a potential attack. We describe key elements of our design, explain how an analyst's intent is used to generate different visualizations, and show how the system's interface allows an analyst to rapidly produce a sequence of visualizations to explore specific details about a potential attack as they arise. We conclude with a discussion of plans to further improve the system, and to collect feedback from our ARL colleagues on its strengths and limitations in real-world analysis scenarios.
My position is that improving evaluation for visualization requires more than developing more sophisticated evaluation methods. It also requires improving the efficacy of evaluations, which involves issues such as how evaluations are applied, reported, and assessed. Considering the motivations for evaluation in visualization offers a way to explore these issues, but it requires us to develop a vocabulary for discussion. This paper proposes some initial terminology for discussing the motivations of evaluation. Specifically, the scales of actionability and persuasiveness can provide a framework for understanding the motivations of evaluation, and how these relate to the interests of various stakeholders in visualizations. It can help keep issues such as audience, reporting and assessment in focus as evaluation expands to new methods.
In this short position paper, we explore three questions regarding cyber security visualization: (1) why cyber security visualization has not been more effective in the past, (2) how visualization can be utilized in cyber security, and (3) how to evaluate cyber security visualization.
Design studies are an increasingly popular form of problem-driven visualization research, yet there is little guidance available about how to do them effectively. In this paper we reflect on our combined experience of conducting twenty-one design studies, as well as reading and reviewing many more, and on an extensive literature review of other field work methods and methodologies. Based on this foundation we provide definitions, propose a methodological framework, and provide practical guidance for conducting design studies. We define a design study as a project in which visualization researchers analyze a specific real-world problem faced by domain experts, design a visualization system that supports solving this problem, validate the design, and reflect about lessons learned in order to refine visualization design guidelines. We characterize two axes - a task clarity axis from fuzzy to crisp and an information location axis from the domain expert's head to the computer - and use these axes to reason about design study contributions, their suitability, and uniqueness from other approaches. The proposed methodological framework consists of 9 stages: learn, winnow, cast, discover, design, implement, deploy, reflect, and write. For each stage we provide practical guidance and outline potential pitfalls. We also conducted an extensive literature survey of related methodological approaches that involve a significant amount of qualitative field work, and compare design study methodology to that of ethnography, grounded theory, and action research.
We present an assessment of the state and historic development of evaluation practices as reported in papers published at the IEEE Visualization conference. Our goal is to reflect on a meta-level about evaluation in our community through a systematic understanding of the characteristics and goals of presented evaluations. For this purpose we conducted a systematic review of ten years of evaluations in the published papers using and extending a coding scheme previously established by Lam et al. [2012]. The results of our review include an overview of the most common evaluation goals in the community, how they evolved over time, and how they contrast or align to those of the IEEE Information Visualization conference. In particular, we found that evaluations specific to assessing resulting images and algorithm performance are the most prevalent (with consistently 80-90% of all papers since 1997). However, especially over the last six years there is a steady increase in evaluation methods that include participants, either by evaluating their performances and subjective feedback or by evaluating their work practices and their improved analysis and reasoning capabilities using visual tools. Up to 2010, this trend in the IEEE Visualization conference was much more pronounced than in the IEEE Information Visualization conference which only showed an increasing percentage of evaluation through user performance and experience testing. Since 2011, however, also papers in IEEE Information Visualization show such an increase of evaluations of work practices and analysis as well as reasoning using visual tools. Further, we found that generally the studies reporting requirements analyses and domain-specific work practices are too informally reported which hinders cross-comparison and lowers external validity.
This paper analyzes trends in the approach to evaluation taken by CHI papers in the last 24 years. A set of papers was analyzed according to our schema for classifying type of evaluation. Our analysis traces papers' trend in type and scope of evaluation. Findings include an increase in the proportion of papers that include evaluation, and a decrease in the median number of subjects in quantitative studies. We also critique the types of subjects, in particular an over reliance on students, and lack of appropriately gender balanced samples. We contextualize these findings in historical trends as we move from machines intended for the technical elite in laboratories to computers integrated into the daily life of everyone.
Information Visualization systems have traditionally followed a one-size-fits-all model, typically ignoring an individual user's needs, abilities and preferences. However, recent research has indicated that visualization performance could be improved by adapting aspects of the visualization to each individual user. To this end, this paper presents research aimed at supporting the design of novel user-adaptive visualization systems. In particular, we discuss results on using information on user eye gaze patterns while interacting with a given visualization to predict the user's visualization tasks, as well as user cognitive abilities including perceptual speed, visual working memory, and verbal working memory. We show that such predictions are significantly better than a baseline classifier even during the early stages of visualization usage. These findings are discussed in view of designing visualization systems that can adapt to each individual user in real-time.
Measuring the User Experience was the first book that focused on how to quantify the user experience. Now in the second edition, the authors include new material on how recent technologies have made it easier and more effective to collect a broader range of data about the user experience. As more UX and web professionals need to justify their design decisions with solid, reliable data, Measuring the User Experience provides the quantitative analysis training that these professionals need. The second edition presents new metrics such as emotional engagement, personas, keystroke analysis, and net promoter score. It also examines how new technologies coming from neuro-marketing and online market research can refine user experience measurement, helping usability and user experience practitioners make business cases to stakeholders. The book also contains new research and updated examples, including tips on writing online survey questions, six new case studies, and examples using the most recent version of Excel.
In this position paper, we propose to investigate novel technologies for evaluating information visualization systems: physiological sensing. We review existing technologies and describe how advances in physiological sensing open a novel perspective for the evaluation of information visualization systems.
The goal of this chapter is to help authors recognize and avoid a set of pitfalls that recur in many rejected information
visualization papers, using a chronological model of the research process. Selecting a target paper type in the initial stage
can avert an inappropriate choice of validation methods. Pitfalls involving the design of a visual encoding may occur during
the middle stages of a project. In a later stage when the bulk of the research is finished and the paper writeup begins, the
possible pitfalls are strategic choices for the content and structure of the paper as a whole, tactical problems localized
to specific sections, and unconvincing ways to present the results. Final-stage pitfalls of writing style can be checked after
a full paper draft exists, and the last set of problems pertain to submission.
We present a nested model for the visualization design and validation with four layers: characterize the task and data in the vocabulary of the problem domain, abstract into operations and data types, design visual encoding and interaction techniques, and create algorithms to execute techniques efficiently. The output from a level above is input to the level below, bringing attention to the design challenge that an upstream error inevitably cascades to all downstream levels. This model provides prescriptive guidance for determining appropriate evaluation approaches by identifying threats to validity unique to each level. We also provide three recommendations motivated by this model: authors should distinguish between these levels when claiming contributions at more than one of them, authors should explicitly state upstream assumptions at levels above the focus of a paper, and visualization venues should accept more papers on domain characterization.
As the field of information visualization matures, the tools and ideas described in our research publications are reaching users. The reports of usability studies and controlled experiments are helpful to understand the potential and limitations of our tools, but we need to consider other evaluation approaches that take into account the long exploratory nature of users tasks, the value of potential discoveries or the benefits of overall awareness. We need better metrics and benchmark repositories to compare tools, and we should also seek reports of successful adoption and demonstrated utility.
Usage data logged from user interactions can be extremely valuable for evaluating software usability. However, instrumenting software to collect usage data is a time-intensive task that often requires technical expertise as well as an understanding of the usability issues to be explored. We have developed a new technique for software instrumentation that removes the need for programming. Interactive Usability Instrumentation (IUI) allows usability evaluators to work directly with a system's interface to specify what components and what events should be logged. Evaluators are able to create higher-level abstractions on the events they log and are provided with real-time feedback on how events are logged. As a proof of the IUI concept, we have created the UMARA system, an instrumentation system that is enabled by recent advances in aspect-oriented programming. UMARA allows users to instrument software without the need for additional coding, and provides tools for specification, data collection, and data analysis. We report on the use of UMARA in the instrumentation of two large open-source projects; our experiences show that IUI can substantially simplify the process of log-based usability evaluation.
We propose a new method for assessing the perceptual organization of information graphics, based on the premise that the visual structure of an image should match the structure of the data it is intended to convey. The core of our method is a new formal model of one type of perceptual structure, based on classical machine vision techniques for analyzing an image at multiple resolutions. The model takes as input an arbitrary grayscale image and returns a lattice structure describing the visual organization of the image. We show how this model captures several aspects of traditional design aesthetics, and we describe a software tool that implements the model to help designers analyze and refine visual displays. Our emphasis here is on demonstrating the model's potential as a design aid rather than as a description of human perception, but given its initial promise we propose a variety of ways in which the model could be extended and validated.
The complexity of large-scale scientific simulations often necessitates the combined use of multiple software packages developed by different groups in areas such as adaptive mesh manipulations, scalable algebraic solvers, and optimization. Historically, these packages have been combined by using custom code. This practice inhibits experimentation with and comparison of multiple tools that provide similar functionality through different implementations. The ALICE project, a collaborative effort among researchers at Argonne National Laboratory, is exploring the use of component-based software engineering to provide better interoperability among numerical toolkits. They discuss some initial experiences in developing an infrastructure and interfaces for high-performance numerical computing.
Insight provenance – a historical record of the process and rationale by which an insight is derived – is an essential requirement in many visual analytics applications. Although work in this area has relied on either manually recorded provenance (for example, user notes) or automatically recorded event-based insight provenance (for example, clicks, drags and key-presses), both approaches have fundamental limitations. Our aim is to develop a new approach that combines the benefits of both approaches while avoiding their deficiencies. Toward this goal, we characterize users' visual analytic activity at multiple levels of granularity. Moreover, we identify a critical level of abstraction, Actions, that can be used to represent visual analytic activity with a set of general but semantically meaningful behavior types. In turn, the action types can be used as the semantic building blocks for insight provenance. We present a catalog of common actions identified through observations of several different visual analytic systems. In addition, we define a taxonomy to categorize actions into three major classes based on their semantic intent. The concept of actions has been integrated into our lab's prototype visual analytic system, HARVEST, as the basis for its insight provenance capabilities.
Usability inspection is the generic name for a set of costeffective ways of evaluating user interfaces to find usability problems. They are fairly informal methods and easy to use.
The purpose of visualization is insight. The purpose of visualization evaluation is to determine whether visualizations are achieving their purpose. If these statements are true, then evaluating visualizations should seek to determine how well visualizations generate insight. But what, exactly, is insight? How can it be measured and evaluated? Do current approaches for evaluating visualizations provide measures of insight? This viewpoint identifies critical characteristics of insight, argues the fundamental reasons why traditional controlled experiments with benchmark tasks on visualizations do not effectively measure insight, and offers a new approach to controlled experiments that can better capture the notion of insight. The ultimate goal is a much richer view of how visualizations can achieve their purpose.
Patterns for visualization evaluation
- N Elmqvist
- J S Yi
Paper submission guidelines: Paper types. Online
- V P Committee
Usability inspection methods Conference Companion on Human Factors in Computing Systems
- Jakob Nielsen
Survey design and implementation in hci. Human-Computer Interaction: Development Process
- A A Ozok
Why ask why?: considering motivation in visualization evaluation. InProc. 2012 BELIV Workshop: Beyond Time and Errors-Novel Evaluation Methods for Visualization
- M Gleicher