Conference PaperPDF Available

Enhancing Cyber Defense Situational Awareness Using 3D Visualizations


Abstract and Figures

The human visual system is generally more adept at inferring meaning from graphical objects and natural scene elements than reading alphanumeric characters. Graphical objects like charts and graphs in cybersecurity dashboards often lack the requisite numbers of features to depict behaviors of complex network data. For example, bar charts afford few features to encode a panoply of parameters in network data. Furthermore, dashboard visualizations seldom support the transition of human work from situation awareness building to requisite responses during intrusion detection events. This research effort aims to identify how graphical objects (also referred as data-shapes) depicted in Virtual Reality tools, developed in accordance with an analyst’s mental model of an intrusion detection event, can enhance analyst’s situation awareness. We demonstrate the proposed approach using Locked Shields 16 CDX network traffic. Implications of this study and future case study are discussed.
Content may be subject to copyright.
Enhancing Cyber Defense Situational Awareness Using 3D
Kaur Kullman1, 2, Jennifer Cowley1 and Noam Ben-Asher1
1Computational and Information Sciences, US Army Research Laboratory, Adelphi, USA
2Tallinn University of Technology, Tallinn, Estonia
Abstract: The human visual system is generally more adept at inferring meaning from graphical objects and natural scene
elements than reading alphanumeric characters. Graphical objects like charts and graphs in cybersecurity dashboards often
lack the requisite numbers of features to depict behaviors of complex network data. For example, bar charts afford few
features to encode a panoply of parameters in network data. Furthermore, dashboard visualizations seldom support the
transition of human work from situation awareness building to requisite responses during intrusion detection events. This
research effort aims to identify how graphical objects (also referred as data-shapes) depicted in Virtual Reality tools,
developed in accordance with an analyst’s mental model of an intrusion detection event, can enhance analyst’s situation
awareness. We demonstrate the proposed approach using Locked Shields 16 CDX network traffic. Implications of this study
and future case study are discussed.
Keywords: visualization, decisionmaking, mental model, analysts, virtual reality, cybersecurity
The quantity of information collected by network monitoring tools has increased steadily in parallel to the
society’s growing dependence on information technology (Kaisler, et al., 2014). The use of monitoring tools to
maintain Cyber Defense Situational Awareness (CDSA) is a prominent task among cybersecurity analysts who
work in a Security Operations Center (SOC) or Network Operations Center (NOC). One way to potentially mitigate
the increasing workload of the cybersecurity analyst is to visualize the network architecture and types of data
traversing through it. Typical dashboard charts and graphs (e.g., line charts, node diagrams, etc.) could overlay
this architecture. However, visualizing data acquired from a wide range of sources on charts and graphs in
isolation may stymie the rapid acquisition of information about changing network behaviors because: (i) graphs
and charts have limited scalability for networks complexity and size, (ii) commonly user graphs/charts often fail
to account for the highly dynamic nature of the cyber environment, and (iii) detection of threats demand the
ability to notice and highlight small anomalies that tend to disappear when visualizing large volumes of data
(Schoenwaelder, et al., 2007). What is needed is a set of new types of graphs and charts, called visualizations
herein, which flex with the changing parameter space while pictorially representing of dynamic, evolving
network behavior. The purpose is to expedite analyst’s situation awareness by designing a visualization that
reflects the analyst’s mental model of the network environment.
Anecdotally, the common NOC/SOC analyst’s workstation often includes command line tools juxtaposed to
dashboard tools; some dashboards allow the user to interact (e.g., filter, drill, etc.) with the data depicted in the
chart or graph. Dashboards usually provide an array of two-dimensional (2D) graphs and charts that summarize
different types of network data. Network data has a high-parameter space and is often multidimensional, leaving
the dashboard designer to fit multidimensional network data into 2D visualizations. If NOC/SOC analysts have
multidimensional mental models network behaviors, then some conversions of 2D may occur. This conversion
renders a measurable perceptual lag and often augments subjective mental workload. The goal of this study is
to design a visualization aligned with analyst’s mental model of the network environment, which facilitates a
faster and more accurate detection of network behavioral change (Kandel, et al., 2012). Our approach is to
utilize a virtual environment to create new 3D visualizations with the capacity to encode a panoply of data
parameters into depth, spatial and temporal cues.
In this study, we describe the development of a 3-dimensional (3D) visualization technique for CDSA. First, we
describe the technique and application architecture. Then, we demonstrate how it can be used by network
operators to obtain and maintain CDSA using data from the Locked Shields 2016 cyber defense exercise as an
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
example. Finally, we discuss the integration of this visualization with a virtual reality, interactive environment as
well as plans for future evaluation.
2.Human visual perception
The human visual system has a finite amount of visual attention resources used to view the data presented on
a computer screen (Schneider, Dumais, & Shiffrin, 1982). Hence, the human visual system is one of the major
bottlenecks of information flows between a computer system to a human analyst (refer the Communication–
Human Information Processing (C–HIP) model for details (Dejoy, et al., 1999)). Because of this bottleneck, the
veracity and comprehensiveness of a human analyst’s mental model about a network event is directly impacted
by perceptual bottlenecks. A mental model is internal, cognitive representation of the environment based on
the acquired information. Then, these models can provide ways to describe, explain, predict, and, sometimes,
control the phenomena (Gentner & Stevens, 1983); (Johnson-Laird, 1983) and they are built through direct
perception of the environment. Hence, the design of data visualizations can impact the accuracy of an analyst’s
mental model development and sustainment (Paradice & Davis, 2008).
Coding information can be used to reduce the chances of perceptual bottlenecks. Visual information can be
efficiently augmented in reasoning processes based (See (Paivio, 1991) for an overview on dual-coding and
(Baddeley, 2012) for a review on working memory). Furthermore, information can be presented to the analyst
in more than one sensory modality to maximize the amount of information perceived in a time epoch. In Human
Computer Interaction research, codes are stimuli that represent the smallest unit of information communicated.
For example, visual codes within a scatterplot are size, color, shape, proximity, among others. These visual codes
are relevant to 2D visualizations but ‘depth’ may be an additional code in 3D environments that can be populated
with additional data parameters and impact the interpretation of other codes like proximity. A group of codes
representing a large amount of information can be configured to create a visual pattern, which can then be
perceived rapidly. This visual pattern, according to Gestalt Psychology, is called emergent features (Treisman &
Paterson, 1984) because the meta-data patterns emerge from the display of raw data. Meta-data patterns can
form the basis of human mental models that analysts use when searching for expected patterns of malicious
network behavior as well as represent normal network behavior. Gestalt laws of perception (e.g., Laws of
Proximity, Closure) (Ehrenstein, et al., 2003) characterize the natural ways humans perceive information
groupings that the interface designer can capitalize on. Poor designs that violate the Gestalt laws of perception
could force the analyst into controlled and deliberated processing that consumes attention resources
(Schneider, et al., 1982).
Depth perception is facilitated with a set of monocular and/or binocular visual cues that could provide additional
codes to depict network information. Monocular depth cues (i.e., light and shading, relative size, interposition,
blur, texture gradient, and aerial perspective linear perspective) (Lebreton, et al., 2012) allow for a 3D depiction
in a 2D plane (i.e., a page or photograph). These kinds of “3D in 2D” visualizations using monocular cueing were
called “perspective views” (Ellis, et al., 1987) or “pseudo 3D (Lange, et al., 2006). Binocular depth cues use
stereopsis to present objects to the viewer that seem to ‘pop out’ from the visual scene. Visualizations with
binocular depth cues are called “real 3D” (Lange, et al., 2006). Visual perception of natural 3D scenes is afforded
by monocular and binocular depth cues working together (Lee & Lee, 2015).
Perspective (Foyle, et al., 2005) is another 3D technique and design principle which can hamper perception in
3D visualization environments. Upon entering a 3D environment, the analyst is perceiving the environment
through the avatar’s eyes or perceiving via a top-down view looking at an avatar that represents themselves.
The terminology describing perspectives is non-standard and sometimes obtuse. Human factors research
defines allocentric views (Klatzky, et al., 1998), also called “through the window” (Brown, 1994) view, as one in
which the observer is watching themselves through a viewpoint outside of the body. For example, an avatar
representing a human has an allocentric view if the controller manipulates the avatar by watching it from behind.
Allocentric is used interchangeably with geocentric or exocentric views (Klatzky, et al., 1998) or plan vi ews (Fo yl e,
et al., 2005). Plan views are allocentric perspectives in which the human is looking down from a higher altitude.
Contrast this with egocentric views (Klatzky, et al., 1998), also called immersive (Brown, 1994) or inside
perspective (Bryant & Tversky, 1999), such that the controller is seeing the virtual environment through the eyes
of the avatar. The type of perspective used for a particular task has been shown to lead to human perceptual
and spatial memory errors (Klatzky et al., 1998).
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
The advantages and disadvantages of depicting data using 3D compared to 2D displays has been studied and
debated for decades with no clear resolution. With the rise of more sophisticated augmented reality
environments, modern visualization research has re-vamped. The advancement of computing provides some
explanations to the discrepancies between recent and dated studies of human performance with 3D
visualizations (Smallman, et al., 2001). In some cases, 3D is advantageous because of a lower interpretive effort
of perceived 3D information, given the human visualization system is designed to see in 3D (Dennehy, et al.,
1994); (Smallman, et al., 2001). Furthermore, 3D visualizations potentially can display more codes with depth
cueing compared to 2D displays. However, these benefits are couched in the type of work tasking required to
complete with the visualization. Tasks such as altitude extraction, geo-spatial maneuvering, and navigation
improve with 3D displays (Burnett & Barfield, 1991) while, there are tasks and environments for which the 2D
displays are more advantageous than 3D displays (see (St. John, et al., 2001).
In sum, prior research indicates that while 3D visualizations in virtual reality environments afford more codes to
use to depict data, the ways in which those codes are arranged using Gestalt’s laws, emergent features and
perspectives, determines how best to maximize the amount of data perceived. Communication from the
interface to the human analysis involves the clear mapping between a mental model of the data that is expected
to be reviewed, and the manner the data is depicted in the visualization (Ehrenstein, et al., 2003). The 3D objects
we design must fit the typical mental model building inherent in network defense job tasking. To our knowledge,
no prior research has identified whether computer network defense analysts are re-visualizing alphanumeric
network data in geospatial patterns in their minds. Furthermore, we have no clarity whether training an analyst
to build their mental models on 3D representations of alphanumeric network data will be advantageous to
performance. These are assumptions we are exploring in our research. Although prior research has described
basic Computer Network Defense (CND) operations and job tasking (DAmico, et al., 2016) (DAmico, et al.,
2005), their findings are relatively generic to ascertain analyst mental models to build 3D visualizations from.
Some preliminary research (Perl & Young, 2015) has attempted to document analyst’s mental models, but the
granularity of the models was too coarse to guide the development of 3D visualizations.
Based on discussions with subject matter experts, we hypothesize that akin to self-morphing graph structures
often used to make sense of new datasets, the relations in data that cybersecurity analysts are after with their
mental models to distill information, are more related to distinct data-shapes that arise while working with the
datasets that analysts are using to solve the task at hand. While analyzing different datasets during incident
response or other tasks. To verify this hypothesis a 3D environment and data-shapes was created, that and can
be observed and manipulated by analysts using devices providing them stereoscopic view of those shapes.
3.Virtual data explorer
Initially, the OpenGraphiti (Reuille, et al., 2015) ( platform was used to develop
3D visualizations, but due to lack of compatibility with motion controllers (i.e., input devices) Thus we used Unity
3D game engine to create a dedicated environment called Virtual Data Explorer (VDE,
which allow for motion controllers to interact as input devices for Oculus Touch controllers (Unity 3D, 2017) or
the Microsoft Mixed Reality headset with their appropriate controllers (Microsoft, 2017). VDE is currently an
academic prototype platform for building 3D data-shapes for data visualizations. The VDE affords 3D data
visualizations by exporting rendered stereoscopic images to a Virtual, Augmented or Mixed Reality Head
Mounted Display (HMD) to create an illusion for the user of immersion to virtual space, containing data-shapes
consisting of the data that the user wishes to analyze. Technically a HMD is a set of screens; each screen
rendering one of the pair of stereoscopic image per eye to provide vision for binocular depth cueing.
The type of data (network traffic, sessions and flows, but also application logs and process memory usage logs
among others) we visualize can be static (logs, forensic evidence) or live-wire data. In the case of a live ingest,
the characteristics of the data would be dynamic, however, our current prototype described herein is based off
a static repository, as the added complexity of live ingest was not deemed necessary for initial testing of the
usability of proposed 3D data-shapes. Note that VDE does not constrain the data-shape development to one
type of environment; ingested data could be visualized as data-shapes in virtual-, mixed-, or augmented reality
environments. The added benefit of using VR/AR is that 3D visualizations afford ample visual real estate to depict
high-parameter (but originally non-spatial) data, allowing perception of numerous variables for each unit of
observation. Each parameter can be encoded into visual codes like size, shape, color, and depth (Ware & Franck,
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
1996). For example, the perceived distance between observer and an object in the 3D virtual environment could
represent a continuous value like total bandwidth usage, or number of bytes transmitted per unit of time.
Although using stereoscopic vision and motion cues to encode data have been found useful (Ware & Franck,
1996), it can be challenging to provide analysts with such technical capability that will make good use of humans’
natural abilities. Unless analysts can immerse in and manipulate with the data-visualization environment
intuitively, it may not be helpful in accomplishing their tasks; in addition, it is assumed that users must be able
to use such environment without fatigue and simulator sickness (Kolasinski, 1995; Johnson, 2005), or as it’s
sometimes referred to – cybersickness.
Prior to current generation Graphics Processing Units (GPU), slow processing power created rendering lags that
yielded visually mismatched orienting cues in the environment – this mismatch often led to user nausea. With
the intensive development of the Interactive Entertainment Industry that has driven the market need, consumer
grade GPU-s have become powerful enough to provide users with non-nauseating VR experiences, while being
affordable enough to be used for our purposes. Recent GPUs with current generation HMDs significantly reduce
the occurrence of visual lag, therefore reducing the chances of users’ nausea.
There are also other factors that could cause unpleasant user experiences in VR. To minimize these effects,
we’ve implemented a few methods in our environment to avoid such experiences. For example, while the user
navigates the 3D space in avatar-less first-person perspective, we restrict the range of head movement during
motion such that the user can only move in a linear direction towards the point of gaze, or away from it,
sometimes referred to as “rudder head movement” (Unity 3D, 2017). The user of course has freedom to observe
360 degrees of the visual field (Kemeny, et al., 2017), only her movements are restricted to back-and-forth
This approach allows us to immerse the user rather conveniently into the VDE environment, where she can, with
hand and head gestures, roam around in 3D space to view the 3D data-shapesvisualizations from multiple
vantage points, grab and interact with the visualization, experience, manipulate, and explore the data
presentations that are dynamically created and adjusted to build situation awareness (in case of NOC/SOC
analysts). Akin to self-morphing graphs (as implemented for example in OpenGraphiti (Reuille, et al., 2015)), VDE
allows us to examine whether presenting data in 3D data-shapes and enabling interaction with its components
could help analysts detect changes in network traffic (this method could also be extended to application logs
and process’ memory usage, for example). Furthermore, VDE allows us to evaluate whether deliberately
structured visual data-shapes that are observed with stereoscopic HMD could enhance CDSA.
For the purposes of this study, we define:
Dataset – values (e.g. IP addresses, their relations, connections, sessions etc.) collected from sensors, log
files and network traffic monitors
Data-object – one instance from dataset, that may be a key-value pair, set of values related to an event that
caused a log-line or alert to be logged
Data-shape – a specific form of data visualization, where pixels (that in collections represent nodes,
connections etc.) are arranged so, that in the resulting visual data representation of the data-objects, visual
objects are positioned according to their logical topology so, that the resulting 3D structure would relate to
a specific task for which the NOC/SOC analyst is responsible for, and would be using that data-shape for
(e.g. relate to hers mental model of the problem/hypothesis/situation)
VDE scene – combined set of data-shapes, a meta-shape, that consists of spatially positioned data-shapes,
that in combination enable to user to view relations between different data-shapes’ nodes.
3.1Data preparation
The ingested network traffic data used to demonstrate the 3D visualization was not live-wire data but a
collection of data from the 2016 Locked Shields Cyber Defense Exercise (LS16) (NATO CCDCOE, 2016) (see This is an international cyber defense exercises with more than
550 participants from 26 nations. Participants were assigned roles in various teams, while most of them were
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
arranged into 20 defensive teams (Blue Teams) and one adversarial team (Red Team). In this exercise, the Blue
Teams’ goal was to maintain the availability and security of their networks during two days of the exercise.
Locked Shields’ dataset was selected as it is relatively well documented, reasonably large (~20TB of PCAP files),
has ground truth (what and when did Red Team members do) and there are 20 comparable networks that start
out as identical but change as respective Blue Teams adjust them. IPv6 visualization was chosen for first VDE
data-shape, as Blue Teams’ ability to monitor and secure their IPv6 addresses was a relevant topic during this
A valuable advantage of LS16 dataset to researchers is the availability of knowledge about the network topology
(of the Blue Teams’ assets) and ground truth of Red Team actions – e.g., what and when did the Red Team
members do during their attacks, and also the Blue Teams responses to adversarial activities (at least to some
To prepare that dataset (IPv6 network traffic information) for 3D visualization, the packets captured (into PCAP
files) during LS16 exercise were parsed with Bro IDS ( to get textual log files describing
network connections between nodes (servers, workstations, network devices in Blue Teams’ networks, and
elsewhere in the “game network”) that were observed in captured traffic. Textual log files were then queried
with SpectX (SpectX, 2017) ( to count the connections between devices to describe
relations of nodes by coloring the edges (connections) between nodes according to the number of times those
nodes were observed communicating – from transparent green to opaque green to red.
Based thet data, VDE would then generate a virtual environment, with LS16 data visualization in it. For our first
concept design we chose to visualize a network topology where the nodes (white spheres) represent devices
like computer desktops, servers, switches or routers, and the edges (lines connecting spheres) represent the
network traffic between those devices. VDE positions nodes according to their logical topology in their
respective networks, visualized as data-shapes that are generated per every Blue Team. Such environment
enables the user to immersively explore the data-shapes, its components using VR equipment. An example of a
user exploring VDE environment composed of data-shapes visualizing LS16 Blue Teams’ networks can be seen
in a brief video released with this article (
3.2Data-Shapes for visualizing logical topology of networked entities
When the user first enters the VDE, the viewer can look down at a ~30 degree angle at the scene that is
positioned at such a distance, as to fit in the view. The floor of the VDE environment (in VR) is a dark patterned
desert that continues until it meets a horizon line that delineates floor and skyline. The background environment
is chosen such, that it would be unobtrusive to the viewer’s task, while providing horizon for spatial orientation.
Visualized data-shapes are floating well above the floor and a little below the horizon line, to ease its
components’ visibility (brighter objects against darker background).
Contrary to self-organizing graphs which are useful for initial examination of unknown datasets, our goal is to
provide analysts with (the ability to create) data-shapes that would help them better comprehend datasets that
are depicted as structures they can learn to know well over time. We propose creating data-shapes where
networked entities (e.g. computers) are positioned according to their logical topology (e.g. computer or server
groups and not only physical or functional topology) so, that the resulting 3D structure(s) would relate to a
SOC/NOC analyst’s task, which in this LS16 example would be to detect prohibited connections between Blue
Team’s network devices. Data-shapes as such are nothing new (Hurter, 2016), but few have tried to use
stereoscopically perceivable 3D data-shapes for computer security (Payer & Trossbach, 2015).
One could consider following as prerequisite knowledge to the creation of the LS16 VDE scene, containing set of
proposed data-shapes:
Understand the principles of how does a computer network function; specifically, how such network is set
up for Locked Shields exercise;
Understand of the logical grouping of networked entities and their topology during Locked Shields; also
understand networks (virtual entities) and this game’s stakeholders’ (physical entities) goals, e.g. Red vs
Blue, but also Green, Yellow, White teams’ functions;
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
Understanding the expected behavior of the above actors and how it should reflect on network data
Search for indicators, validate, visualize and act.
To create data-shapes for other datasets, a different set of knowledge is required, but can be acquired by
mapping the mental models of the analysts who will be using these data-shapes.
To test the usefulness of using 3D data-shapes when encoding non-spatial data, networked entities were
spatially positioned, considering their position in network topology, and more importantly, entities’ affiliation
to logical groups (by functionality (e.g. SCADA components), purpose (e.g. DMZ servers), risk exposure, OS etc.).
This results in custom 3D data-shapes, that could be combined to a meta-shape (a VDE scene) representing
larger whole of the LS network(s) that are of interest in our scenario. A meta-shape, VDE scene depicted in Figure
2 is the overall view of the percept the LS16 network traffic visualization makes from a distance.
As we have three axises available to encode data (we are not using time in this visualization scenario), we chose
to use two of those to encode parts of network topology (subnet number (third octet in case of LS16) and entity’s
IP addresses’ last octet or position in its subgroup) while the third axis binds to the functional or logical group of
that entity. Using the common X, Y, Z referencing: within a data shape shown in Figure 1, the Y axis is the group
number, the X axis is the subnet number (the team number) and the Z axis has no relevant values other than the
IP addresses within a particular range cluster together (i.e., the IP addresses’ last octet or position in its
subgroup) along the Z-axis. A group number is assigned to a type of 6 functions the nodes or network devices
perform. These groups are:
1. DMZ servers for email, WWW, DNS, NTP, and others
2. Office network with Domain Controller and workstations
3. Lab network for research and development
4. Control network for drone operators
5. Secure devices
6. Incident Command System (ICS) systems as the high level objective of the Red Team attacks
Groups contain nodes in their respective subnets, grouped vertically according to their logical positions in their
functional groups (subnets). For example, Windows, Linux, OSX workstations are positioned onto separate layers
to distinguish them visually in subnets 2 and 3, while Windows, Linux and other servers, networks devices, etc.
are kept on the lowest group to distinguish intra-group traffic from inter-group traffic.
For example, to find suspicious connections inside a LS16 Blue Team’s network, entities were first positioned
according to their subnet and then by their functional groups—servers, network devices, workstations
(distinguished further by their type (Windows, Linux, OSX)) , and SCADA components among others (see Figure
1). The third dimension is entity’s sequential position inside of its subgroup (often the last octet of its IP address).
Because the designated functions (and therefore behavior) of the entities in same functional group should be
similar, it is beneficial for the analyst to have them close together, while still being spatially distinguishable to
quickly diagnose which group and which member to focus on.
At the start of the exercise, there were 20 functionally identical Blue Teams’ networks, whose entities should
have been communicating identically, but as the exercise advanced, the Blue Teams’ networks’ behavior (in this
case, entities’ activity and relative connections / edges) deteriorated from each other’s. Each Blue Team’s
network had 68 preconfigured nodes, and the teams could add two virtual machines per their specifications.
Entities that fell outside of the known functional groups were positioned to three cube-shaped matrixes: i)
entities with public IP addresses (simulated in-game internet); ii) entities that had IP addresses in Blue Teams’
internal address ranges, but which were not preconfigured prior to game; and iii) entities that had IP addresses
in Blue Teams’ internal address ranges, were not preconfigured before the game, and did not follow the Locked
Shields’ addressing logic (for example, those that had letters in IPv6 address). Before positioning entities to those
groups, these were sorted by their IP address.
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
Figure 1: Viewing same data-shape from three different angles
Figure 2: The analyst’s initial view of the meta-shape of the Locked Shield dataset in the VDE
Teams’ networks curved around three shapes containing external and/or potentially interesting hosts. All data -
shapes contain the Blue Teams’ systems with the same (or similar) layout as seen in Fig ure 1. From this broad
view, an analyst who knows the logical positioning and internal functions of the Blue Team’s networks’ groups,
subgroups and nodes may find anomalies to investigate further, which in this visualization scenario are
connections (green edges) between different Blue Teams’ systems. For example, the connections (green edges)
originating from a host in the 7th team’s network (7th data-shape from left) to other Blue Teams’ systems should
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
not be happening. In addition, connections from the 11th team’s network (11th cube from left) to hosts that
were named unconventionally (so that they appear to be in the 11th team’s IPv6 address range) should be
examined by taking a closer look at the visualization (selecting specific host that seems to be t rying to blend in
and exploring it’s relations) or continuing exploration in textual logs. For a better understanding of this process,
please see the videos (
Data-shapes were then spatially positioned to a meta-shape (as shown in Figure 2) to allow the user to take
advantage of stereoscopic viewing and the sense of binocular depth that it provides. Several layouts were
considered to minimize possible edge clutter and enable convenient distinguishability of intra- and extra-
network connections.
Edges connecting the nodes were then added to the data-shapes, as prepared using the process described in
subsection 3.1. From the initial vantage point user can distinguish edges that connect one Blue Teams’ node(s)
to those of other Blue Teams’ internal nodes (mostly horizontal edges as opposed to vertical ones); on the other
hand the edges connecting Blue Teams’ internal nodes with entities in either of the three matrixes have different
implications. Would an NOC/SOC operator have to evaluate a network from this view, she would want to see
only i) Blue Team’s first subnet (e.g. DMZ) nodes connecting to legitimate services located in “game internet”
and ii) Blue Team’s internal hosts communicating only to that same Blue Team’s hosts. All other edges might
need further examination.
3.3Detecting abnormalities in traffic
In Figure 2, the vertical and diagonal lines that connect one node with multiple nodes in other Blue Teams’
networks indicate a possible abnormality that should be investigated. Is it possible that the highlighted behaviors
represent a compromised node in the Blue Team’s network, which is used by a Red Team member to scan other
Blue Teams’ systems to find those that have not been correctly firewalled? Is it possible that some devices or
tasks (e.g., network scans) were misconfigured, e.g., SYN packets were found in the traffic but not ACK or RST,
meaning that the host did scan but could not connect to those hosts?
One could argue that this kind of anomalous behavior would be blocked by the network devices’ ACL rules, a
myriad of “cybersecurity appliances” endorsed by cyber-insurance providers, or at least detected by
conventional “cyber-devices” (e.g., IDS/IPS and firewalls). We argue, that while systems that help NOC/SOC
personnel to protect their networks are a necessity, our adversaries will always find functionalities (weaknesses)
in those systems that enable them to bypass those protections. Therefore NOC/SOC analysts will need to be able
to creatively approach their datasets to find their adversaries attacks in novel ways, and we need to provide
analysts with appropriate tools for those tasks. One such tool could be a system (ex. VDE) that would provide
analysts’ with environment where, using the same, similar, or improved structured data views to visualize
familiar but dynamic datasets, the analyst could have different views of relevant datasets to find anomalies,
which could be missed otherwise, would they rely on 2D and textual tools only.
4.Discussion and conclusion
This paper describes the theory and methodology used to develop a 3D visualization of network data. The
selection of attributes, data-shapes and display aims to capture cybersecurity analysts’ mental models enable
the analysts to better understand their respective datasets. Following the development of the visualization, we
are planning to conduct controlled validation study with experienced cybersecurity analysts and vulnerability
analysts. We will be using a mixed method that begins with a set of qualitative task analyses while the participant
is using the new visualization tool moving to quantitative behavioral studies. Our dependent measures are
situation awareness content and accuracy, speed of SA acquisition, mental model accuracy.
We argue that there is a need for structured evaluation of visualizations that are comparable with the analyst’s
mental model. Current technology is capable of delivering the basic 3D visualization needs and this preliminary
work demonstrates that through tight interaction with SMEs it is possible to identify core concepts in their
mental models and transform them into Data-shapes. Further research is needed on how general are the Data-
shapes over different types of networks, cyber operations, analyst past training and other individual differences.
However, the benefits of harnessing human superior visual-perception to cyber detection can provide a much
needed advantage to cyber defenders.
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
For all the hints, ideas and mentoring, authors thank Alexander Kott, Jaan Priisalu, Olaf Manuel Maennel and
Lee Trossbach. This research was partly supported by the Army Research Laboratory under Cooperative
Agreement Number W911NF-13-2-0045 (ARL Cyber Security CRA) and under Cooperative Agreement Number
W911NF-16-2-0113 and W911NF-17-2-0083. The views and conclusions contained in this document are those
of the authors and should not be interpreted as representing the official policies, either expressed or implied, of
the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and
distribute reprints for Government purposes notwithstanding any copyright notation herein.
Baddeley, A., 2012. Working Memory: Theories, Models, and Controversies. Annual Review of Psychology, Volume 63, pp.
Brown, M. A., 1994. Displays for Air Traffic Control: 2D, 3D and VR - A Preliminary Investigation, London: Queen Mary &
Westfield College.
Bryant, D. J. & Tversky, B., 1999. Mental Representations of Perspective and Spatial Relations from Diagrams and Models.
Journal of Experimental Psychology Learning Memory and Cognition, 25(1), pp. 137-156.
Burnett, M. S. & Barfield, W., 1991. Perspective versus plan view air traffic control (ATC) displays - Survey and empirical
results. Columbus, s.n.
D‘Amico, A., Buchanan, L., Kirkpatrick, D. & Walczak, P., 2016. Cyber Operator Perspectives on Security Visualization. In:
Advances in Human Factors in Cybersecurity. s.l.:Springer, pp. 69-81.
D’Amico, A. et al., 2005. Achieving Cyber Defense Situational Awareness: A Cognitive Task Analysis of Information
Assurance Analysts. s.l., s.n.
Dejoy, D. M., Laughery, K. R. & Wogalter, M. S., 1999. Organizing theoretical framework: a consolidated communication-
human information processing (C-HIP) model. Warnings and risk communication. In: s.l.:s.n., pp. 15-23.
Dennehy, M. T., Nesbitt, D. W. & Sumey, R. A., 1994. Real-Time Three-Dimensional Graphics Display for Antiair Warfare
Command and Control. Johns Hopkins APL Technical Digest, 15(2), pp. 110-119.
Ehrenstein, W. H., Spillmann, L. & Sarris, V., 2003. Gestalt Issues in Modern Neuroscience. In: Axiomathes. s.l.:Springer, pp.
Ellis, S. R., Mcgreevy, M. W. & Hitchcock, R. J., 1987. Perspective traffic display format and airline pilot traffic avoidance.
Human Factors, Volume 29, pp. 371-382.
Feltovich, P. J., Prietula, M. J. & Ericsson, K. A., 2006. Studies of expertise from psychological perspectives. In: The
Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press, pp. 41-67.
Foyle, D. C., Andre, A. D. & Hooey, B. L., 2005. Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints
and Cognitive Glue. Las Vegas, Proceedings of the 11th International Conference on Human Computer Interaction.
Gentner, D. & Stevens, A., 1983. Mental Models (Cognitive Science Series). s.l.:Lawrence Erlbaum Associates.
Hurter, C., 2016. Image-Based Visualization: Interactive Multidimensional Data Exploration. s.l.:Morgan & Claypool.
Johnson, D. M., 2005. Introduction to and Review of Simulator Sickness Research, Arlington: U.S. Army Research Institute
for the Behavioral and Social Sciences.
Johnson-Laird, P. N., 1983. Mental Models. s.l.:Cambridge University Press.
Kaisler, S., Armour, F., Espinosa, A. J. & Money, W., 2014. Big Data: Issues and Challenges Moving Forward. Wailea, s.n.
Kandel, S., Paepcke, A., Hellerstein, J. M. & Heer, J., 2012. Enterprise data analysis and visualization: An interview stud. IEEE
Transactions on Visualization and Computer Graphics, 18(12), pp. 2917-2926.
Kemeny, A., George, P. & Mérienne, F., 2017. New VR Navigation Techniques to Reduce Cybersickness. Electronic Imaging,
The Engineering Reality of Virtual Reality, pp. 48-53.
Klatzky, R. L. et al., 1998. Spatial Updating of Self-Position and Orientation during Real, Imagined, and Virtual Locomotion,
s.l.: Sage Publications, Inc..
Kolasinski, E. M., 1995. Simulator Sickness in Virtual Environments, Alexandria: United States Army Research Institute.
Lange, M., Dang, T. & Cooper, M., 2006. Interactive resolution of conflicts in a 3d stereoscopic environment for air traffic
control. Ho Chi Minh City, Vietnam, Vietnam, s.n.
Lebreton, P., Raake, A., Barkowsky, M. & Le Callet, P., 2012. Evaluating Depth Perception of 3D Stereoscopic Videos. IEEE
Journal of Selected Topics in Signal Processing, 6(6).
Lee, K. & Lee, S., 2015. 3D Perception Based Quality Pooling: Stereopsis, Binocular Rivalry, and Binocular Suppression. IEEE
Journal of Selected Topics in Signal Processing , 9(3), pp. 533-545.
Microsoft, 2017. Windows Dev Center, Motion controllers. [Online] Available at:
NATO CCDCOE, 2016. Locked Shields 2016. [Online] Available at:
Paivio, A., 1991. Dual Coding Theory: Retrospect And Current Status. Canadian Journal of Psychology/Revue canadienne de
psychologie, 45(3), pp. 255-287.
Paradice, D. & Davis, R. A., 2008. DSS and Multiple Perspectives of Complex Problems. s.l.:s.n.
Payer, G. & Trossbach, L., 2015. The Application of Virtual Reality for Cyber Information Visualization and Investigation. In:
Evolution of Cyber Technologies and Operations to 2035. s.l.:Springer, Cham, pp. 71-90.
Kaur Kullman, Jennifer Cowley and Noam Ben-Asher
Perl, S. J. & Young, R. O., 2015. A Cognitive Study of Incident Handling Expertise. Berlin, 27th Annual FIRST Conference.
Reda, K. et al., 2013. Visualizing large, heterogeneous data in hybrid-reality environments. IEEE Computer Graphics and
Applications, 33(4), pp. 38-48.
Reuille, T. et al., 2015. OpenDNS Data Visualization Framework. [Online] Available at:
Schneider, W., Dumais, S. T. & Shiffrin, R. N., 1982. Automatic and Control Processing and Attention, Illinois: University of
Schoenwaelder, P. J. et al., 2007. Key research challenges in network management. IEEE Communications Magazine,
45(10), p. 104–110.
Smallman, H. S., St. John, M., Oonk, H. M. & Cowen, M. B., 2001. Information availability in 2D and 3D displays. IEEE
Computer Graphics and Applications, 21(5), pp. 51-57.
SpectX, 2017. Inertia in Processing Machine Generated Data. [Online] Available at:
St. John, M., Cowen, M. B., Smallman, H. S. & Oonk, H. M., 2001. The Use of 2D and 3D Displays for Shape-Understanding
versus Relative-Position Tasks. Human Factors, Volume Spring, pp. 79-98.
The Bro Project, n.d. [Online]
Available at:
Treisman, A. & Paterson, R., 1984. Emergent features, attention, and object perception. Journal of Experimental
Psychology: Human Perception and Performance, 10(1)(12).
Unity 3D, 2017. Unity 3D Manual, Input for Oculus, Oculus Touch Controllers. [Online] Available at:
Unity 3D, 2017. Vision 2017 - Lessons from Oculus: Overcoming VR Roadblocks. [Online] Available at:
Ware, C. & Franck, G., 1996. Evaluating stereo and motion cues for visualizing information nets in three dimensions. ACM
Transactions on Graphics, March.15(2).
Wickens, C. D. & Hollands, J. G., 2000. Engineering psychology and human performance. Upper Saddle River: Prentice Hall.
Young, I., 2008. Mental Models: Aligning Design Strategy with Human Behavior. s.l.:Rosenfeld Media.
... Typical representations of network topology are in two dimensions (2D), which loses temporal and spatial relationships between nodes in the network, in addition to not scaling well with increased (but often necessary) complexity. Virtual Reality (VR) and Mixed Reality (MR) tools that are able to visualize CSA-relevant information such as network topology as 3D objects in space and time, may aid in the development of shared mental models for efficient RCP communication between technical and non-technical personnel (Kullman et al., 2018(Kullman et al., , 2019a(Kullman et al., ,b, 2020. For instance, SA level 3 is the most vital stage for decision-making and appears to . ...
... The Virtual Data Explorer (VDE; Kullman et al., 2018Kullman et al., , 2019a) was developed to visualize network topology in a manner that is idiosyncratic to the mental models that analysts use to conceptualize the network (Figure 2). Based on interviews with expert analysts, the VDE is able to visualize the relationship between nodes in an actual network in space and time (Kullman et al., 2018(Kullman et al., , 2019a(Kullman et al., ,b, 2020. ...
... The Virtual Data Explorer (VDE; Kullman et al., 2018Kullman et al., , 2019a) was developed to visualize network topology in a manner that is idiosyncratic to the mental models that analysts use to conceptualize the network (Figure 2). Based on interviews with expert analysts, the VDE is able to visualize the relationship between nodes in an actual network in space and time (Kullman et al., 2018(Kullman et al., , 2019a(Kullman et al., ,b, 2020. The visualizations produced by the VDE are interactive and can be shared between individuals, even remotely, thus allowing for collaborative development of shared mental models of events in the network. ...
Full-text available
Background Cyber defense decision-making during cyber threat situations is based on human-to-human communication aiming to establish a shared cyber situational awareness. Previous studies suggested that communication inefficiencies were among the biggest problems facing security operation center teams. There is a need for tools that allow for more efficient communication of cyber threat information between individuals both in education and during cyber threat situations. Methods In the present study, we compared how the visual representation of network topology and traffic in 3D mixed reality vs. 2D affected team performance in a sample of cyber cadets ( N = 22) cooperating in dyads. Performance outcomes included network topology recognition, cyber situational awareness, confidence in judgements, experienced communication demands, observed verbal communication, and forced choice decision-making. The study utilized network data from the NATO CCDCOE 2022 Locked Shields cyber defense exercise. Results We found that participants using the 3D mixed reality visualization had better cyber situational awareness than participants in the 2D group. The 3D mixed reality group was generally more confident in their judgments except when performing worse than the 2D group on the topology recognition task (which favored the 2D condition). Participants in the 3D mixed reality group experienced less communication demands, and performed more verbal communication aimed at establishing a shared mental model and less communications discussing task resolution. Better communication was associated with better cyber situational awareness. There were no differences in decision-making between the groups. This could be due to cohort effects such as formal training or the modest sample size. Conclusion This is the first study comparing the effect of 3D mixed reality and 2D visualizations of network topology on dyadic cyber team communication and cyber situational awareness. Using 3D mixed reality visualizations resulted in better cyber situational awareness and team communication. The experiment should be repeated in a larger and more diverse sample to determine its potential effect on decision-making.
... There have been numerous attempts to employ 3D visualizations for cybersecurity data that are displayed on 2D computer screens with varying degrees of success. Such visualizations sometimes use monocular depth cues (Lebreton, Raake, Barkowsky, & Le Callet, 2012) and object movement to convey the 3D shape of the visualization; advantages and disadvantages of which were thoroughly discussed in our previous paper (Kullman, Cowley, & Ben-Asher, 2018). VIDS (Shearer & Edwards, 2018) provides an interactive 3D environment for visualizing network and alert (or other) data in 3D shapes, whereby users can seamlessly switch styles and layouts to dynamically shape their data and easily adjust their viewpoint (Gaw, 2014). ...
... Virtual Data Explorer (VDE) is a VR tool that allows users to collaborate while investigating 3D data visualizations, to find anomalies in a variety of cybersecurity-related datasets (U.S. ARL, 2018). For our research herein, we used VDE (see 3.4), because it enables the user to perceive the spatial layout of the topology based on observed network traffic, while the resulting visualization can be augmented with additional data, like TCP/UDP session counts between network nodes (Kullman, Cowley, & Ben-Asher, 2018). Due to the 6DOF of Oculus Rift VR headset (OVR) used for this study, VDE also allows us to test the usefulness of stereoscopically perceived depth-cues (contrary to monocular depth-cues on flat screens) for encoding data. ...
... User could add additional sessions using VDE menu system in VR, in which case the added edges were colored red until a next set of edges was added. Additional information about VDE design decisions can be found in our previous paper (Kullman, Cowley, & Ben-Asher, 2018). ...
Conference Paper
Full-text available
Cybersecurity analysts ingest and process significant amounts of data from diverse sources in order to acquire network situation awareness. Visualizations can enhance the efficiency of analysts' workflow by providing contextual information, various sets of cybersecurity related data, information regarding alerts, among others. However, textual displays and 2D visualizations have limited capabilities in displaying complex, dynamic and multidimensional information. There have been many attempts to visualize data in 3D, while being displayed on 2D displays, but success has been limited. We propose that customized, stereoscopically perceivable 3D visualizations aligned with analysts' internal representations of network topology, may enhance their capability to understand their networks' state in ways that 2D displays cannot afford. These 3D visualizations may also provide a path for users who are trained and comfortable with textual and 2D representations of data to assess visualization methods that may be suitably aligned to implicit knowledge of their networks. Thus, the premise of custom data-visualizations forms the foundation for this study. Herein, we report on findings from a comparative, qualitative, within-subjects usability analysis between 2D and 3D representations of the same network traffic dataset. Study participants (analysts) provided information on: 1.) ability to create an initial understanding of the network, 2.) ease of finding task-relevant information in the representation, and 3.) overall usability. Results indicated that interviewees indicated a preference for 3D visualizations over the 2D alternatives and we discuss possible explanations for this preference.
... We acknowledge that the efficiency of 3D data visualization has been subject to controversy (as thoroughly explained in [14]) and that the usability of visualizations overall are hindered by biological factors of the user (e.g. impaired color vision, impaired vison): these and other concerns were covered in an earlier papers of our project [15] and [4]. Despite that, for the users who can use and who do find 3D visualizations useful, we should provide methods they can use to create, and suitable technical tools to use useful visualization of their data. ...
... The Virtual Data Explorer (VDE) software that may be employed for visualizing cybersecurity specific datasets was covered in previous research [15] and [4]. For a data-shape or their constellations to be useful, the SME must be able to readily map data into a data-shape and choose visual encoding for its attributes so that the resulting visualization will enhance their understanding of that data. ...
Visualizations can enhance the efficiency of Cyber Defense Analysts, Cyber Defense Incident Responders and Network Operations Specialists (Subject Matter Experts, SME) by providing contextual information for various cybersecurity-related datasets and data sources. We propose that customized, stereoscopic 3D visualizations, aligned with SMEs internalized representations of their data, may enhance their capability to understand the state of their systems in ways that flat displays with either text, 2D or 3D visualizations cannot afford. For these visualizations to be useful and efficient, we need to align these to SMEs internalized understanding of their data. In this paper we propose a method for interviewing SMEs to extract their implicit and explicit understanding of the data that they work with, to create useful, interactive, stereoscopically perceivable visualizations that would assist them with their tasks.
... We acknowledge that the efficiency of 3D data visualization has been subject to controversy (as thoroughly explained in [14]) and that the usability of visualizations overall are hindered by biological factors of the user (e.g. impaired color vision, impaired vison): these and other concerns were covered in an earlier papers of our project [15] and [4]. Despite that, for the users who can use and who do find 3D visualizations useful, we should provide methods they can use to create, and suitable technical tools to use useful visualization of their data. ...
... The Virtual Data Explorer (VDE) software that may be employed for visualizing cybersecurity specific datasets was covered in previous research [15] and [4]. For a data-shape or their constellations to be useful, the SME must be able to readily map data into a data-shape and choose visual encoding for its attributes so that the resulting visualization will enhance their understanding of that data. ...
Full-text available
Visualizations can enhance the efficiency of Cyber Defense Analysts, Cyber Defense Incident Responders and Network Operations Specialists (Sub-ject Matter Experts, SME) by providing contextual information for various cy-bersecurity-related datasets and data sources. We propose that customized, stere-oscopic 3D visualizations, aligned with SMEs internalized representations of their data, may enhance their capability to understand the state of their systems in ways that flat displays with either text, 2D or 3D visualizations cannot afford. For these visualizations to be useful and efficient, we need to align these to SMEs internalized understanding of their data. In this paper we propose a method for interviewing SMEs to extract their implicit and explicit understanding of the data that they work with, to create useful, interactive, stereoscopically perceivable visualizations that would assist them with their tasks.
... In addition, there are geo-referenced visualization charts for assets [41,43,44], risks [45][46][47] and threats [41,44]. Furthermore, there are also immersive visualization techniques using 3D models instead of 2D models which have been designed for optimum visualization with an ultra-wide high-definition screen, wrap-around screen or three-dimensional Virtual-Reality (VR) goggles, which allows the user to look around 360 degrees while moving [42,44,[48][49][50]. ...
Full-text available
The number and the diversity in nature of daily cyber-attacks have increased in the last few years, and trends show that both will grow exponentially in the near future. Critical Infrastructures (CI) operators are not excluded from these issues; therefore, CIs’ Security Departments must have their own group of IT specialists to prevent and respond to cyber-attacks. To introduce more challenges in the existing cyber security landscape, many attacks are unknown until they spawn, even a long time after their initial actions, posing increasing difficulties on their detection and remediation. To be reactive against those cyber-attacks, usually defined as zero-day attacks, organizations must have Threat Hunters at their security departments that must be aware of unusual behaviors and Modus Operandi. Threat Hunters must face vast amounts of data (mainly benign and repetitive, and following predictable patterns) in short periods to detect any anomaly, with the associated cognitive overwhelming. The application of Artificial Intelligence, specifically Machine Learning (ML) techniques, can remarkably impact the real-time analysis of those data. Not only that, but providing the specialists with useful visualizations can significantly increase the Threat Hunters’ understanding of the issues that they are facing. Both of these can help to discriminate between harmless data and malicious data, alleviating analysts from the above-mentioned overload and providing means to enhance their Cyber Situational Awareness (CSA). This work aims to design a system architecture that helps Threat Hunters, using a Machine Learning approach and applying state-of-the-art visualization techniques in order to protect Critical Infrastructures based on a distributed, scalable and online configurable framework of interconnected modular components.
... VR view of Locked Shields 16 network topology and traffic using VDE. Notice the slightly different constellation layout compared to Figures 2 -5[29]. ...
Full-text available
Interactive Data Visualizations (IDV) can be useful for cybersecurity subject matter experts (CSMEs) while they are exploring new data or investigating familiar datasets for anomalies, correlating events, etc. For an IDV to be useful to a CSME, interaction with that visualization should be simple and intuitive (free of additional mental tasks) and the visualization’s layout must map to a CSME’s understanding. While CSMEs may learn to interpret visualizations created by others, they should be encouraged to visualize their datasets in ways that best reflect their own ways of thinking. Developing their own visual schemes makes optimal use of both the data analysis tools and human visual cognition. In this article, we focus on a currently available interactive stereoscopically perceivable multidimensional data visualization solution, as such tools could provide CSMEs with better perception of their data compared to interpreting IDV on flat media (whether visualized as 2D or 3D structures).
... Consequently, analysts must either scale down the number of dimensions visible at a time for encoding into a 2D or 3D visualization, or they must combine multiple visualizations displaying different dimensions of that dataset into a dashboard. The inspiration for VDE was the hope that immersive visualization would enable the 3D encoding of data in ways better aligned to subject matter experts' (SMEs') natural understanding of their datasets' relational layout, better reflecting their mental models of the multilevel hierarchical relationships of groups of entities expected to be present in a dataset and the dynamic interactions between these entities [13]. ...
Full-text available
Cybersecurity practitioners face the challenge of monitoring complex and large datasets. These could be visualized as time-varying node-link graphs, but would still have complex topologies and very high rates of change in the attributes of their links (representing network activity). It is natural, then, that the needs of the cybersecurity domain have driven many innovations in 2D visualization and related computerassisted decision making. Here, we discuss the lessons learned while implementing user interactions for Virtual Data Explorer (VDE), a novel system for immersive visualization (both in Mixed and Virtual Reality) of complex time-varying graphs. VDE can be used with any dataset to render its topological layout and overlay that with time-varying graph; VDE was inspired by the needs of cybersecurity professionals engaged in computer network defense (CND). Immersive data visualization using VDE enables intuitive semantic zooming, where the semantic zoom levels are determined by the spatial position of the headset, the spatial position of handheld controllers, and user interactions (UIa) with those controllers. This spatially driven semantic zooming is quite different from most other network visualizations which have been attempted with time-varying graphs of the sort needed for CND, presenting a broad design space to be evaluated for overall user experience (UX) optimization. In this paper, we discuss these design choices, as informed by CND experts, with a particular focus on network topology abstraction with graph visualization, semantic zooming on increasing levels of network detail, and semantic zooming to show increasing levels of detail with textual labels.
The main goal of the field of augmented cognition is to research and develop adaptive systems capable of extending the information management capacity of individuals through computing technologies. Augmented cognition research and development is therefore focused on accelerating the production of novel concepts in human-system integration and includes the study of methods for addressing cognitive bottlenecks (e.g., limitations in attention, memory, learning, comprehension, visualization abilities, and decision making) via technologies that assess the user’s cognitive status in real time. A computational interaction employing such novel system concepts monitors the state of the user, through behavioral, psychophysiological, and neurophysiological data acquired from the user in real time, and then adapts or augments the computational interface to significantly improve their performance on the task at hand. The International Conference on Augmented Cognition (AC), an affiliated conference of the HCI International (HCII) conference, arrived at its 16th edition and encouraged papers from academics, researchers, industry, and professionals, on a broad range of theoretical and applied issues related to augmented cognition and its applications. The field of augmented cognition has matured over the years to solve enduring issues such as portable, wearable neurosensing technologies and data fusion strategies in operational environments. These innovations coupled with better understanding of brain and behavior, improved measures of brain state change, and improved artificial intelligence algorithms have helped expand the augmented cognition focus areas to rehabilitation, brain-computer interfaces, and training and education. The burgeoning field of human-machine interfaces such as drones and autonomous agents are also benefitting from augmented cognition research. This volume of the HCII 2022 proceedings is dedicated to this year’s edition of the AC conference and focuses on topics related to understanding human cognition and behavior, brain activity measurement and electroencephalography, human and machine learning, and augmented cognition in extended reality. Papers of this one volume are included for publication after a minimum of two single-blind reviews from the members of the AC Program Board or, in some cases, from members of the Program Boards of other affiliated conferences. We would like to thank all of them for their invaluable contribution, support, and efforts.
Conference Paper
Full-text available
US Army C5ISR Center Cyber Security Service Provider (CSSP) is a 24/7 Defensive Cyber Operations (DCO) organization that defends US Department of Defense and US Army networks from hostile cyber activity, as well as develops technologies and capabilities for use by DCO operators within the DoD. In recent years, C5ISR Center CSSP has been researching various advanced data visualization concepts and strategies to enhance the speed and efficiency of cybersecurity analyst's workflow. To achieve these goals Virtual and Mixed Reality (VR/MR) tools have been employed to investigate, whether these mediums would enable useful remote collaboration of DCO operators and whether stereoscopically perceivable 3D data visualizations would enable DCO operators to gain improved hindsight into their datasets. We'll be giving overview of the capabilities being developed as aligned to our research and operational requirements, our expected outcomes of using VR/MR in training and operational cyber environments and our planned path to accomplish these goals.
Full-text available
3D video quality of experience (QoE) is a multidimensional problem; many factors contribute to the global rating like image quality, depth perception and visual discomfort. Due to this multidimensionality, it is proposed in this paper, that as a complement to assessing the quality degradation due to coding or transmission, the appropriateness of the non-distorted signal should be addressed. One important factor here is the depth information provided by the source sequences. From an application-perspective, the depth-characteristics of source content are of relevance for pre-validating whether the content is suitable for 3D video services. In addition, assessing the interplay between binocular and monocular depth features and depth perception are relevant topics for 3D video perception research. To achieve the evaluation of the suitability of 3D content, this paper describes both a subjective experiment and a new objective indicator to evaluate depth as one of the added values of 3D video.
In a survey of cyber defense practitioners, we presented 39 assertions about the work cyber operators do, data sources they use, and how they use or could use cyber security visual presentations. The assertions were drawn from prior work in cyber security visualization over 15 years. Our goal was to determine if these assertions are still valid for today’s cyber operators. Participants included industry, government and academia experts with real experience in the cyber domain. Results validated the assertions, which will serve as a foundation for follow-on security visualization research. Feedback also indicates that when analyzing a security situation, cyber operators inspect large volumes of data, usually in alpha-numeric format, and try to answer a series of analytic questions, expending considerable cognitive energy. Operators believe security visualizations could support their analysis and communication of findings, as well as training new operators.
Control of air defense operations - antiair warfare - requires commanders to interpret and act on computer-generated graphical representations of aircraft traffic throughout a defended area. A watch officer's ability to interpret this virtual environment rapidly and correctly determines operational success. As part of a continuing effort to improve coordination of antiair warfare, the Applied Physics Laboratory is building a prototype system having as one of its functions a three-dimensional perspective display that can improve the commander's effectiveness in assessing a tactical situation. A distributed, object-oriented computer program has been developed to provide the performance and adaptability required for this approach to graphical display.
Performing the analysis of security data in the prevention of cyber-attacks on an organization’s information systems requires human analysts to make sense of ever-expanding amounts of information. In many security operation centers (SOCs), human analysts are presented with information through the use of multiple monitors. Information is processed using a number of commercial off-the-shelf and custom tools in order to carve information into sets of alerts that analysts can investigate. The amount and complexity of the data being presented to the analyst can significantly overwhelm a single or multiple displays. This avalanche of display information is alongside the additional research an analyst must perform in order to provide proper context to the alerts analysts may be investigating. Analyst investigations can include a number of competing interfaces. A non-exhaustive list includes web browsers with numerous tabs, documents, collaboration software, and both Graphical User Interface (GUI) and Command Line Interface (CLI) based command and control software. Adding additional monitors can lead to a diminishing rate of return in information processing as analysts now must physically observe multiple panels in fixed positions. With a virtual reality (VR) head-mounted (VRH) display, the display space for visualizing different information and data pertaining to cyber events becomes almost limitless. The information being displayed is no longer specifically restricted to a few small rectangular displays but is perceived as nearly infinite space. The OR can open the door to developing significantly more advanced VR experiences. Using more advanced VR technology not only can more information be displayed, but VR displays open the door to new and innovative visualization techniques, which enables us to model security information in new ways and which allows for the efficient identification of malicious behavior within information systems.
Our society has entered a data-driven era, one in which not only are enormous amounts of data being generated daily but there are also growing expectations placed on the analysis of this data. Some data have become simply too large to be displayed and some have too short a lifespan to be handled properly with classical visualization or analysis methods. In order to address these issues, this book explores the potential solutions where we not only visualize data, but also allow users to be able to interact with it. Therefore, this book will focus on two main topics: large dataset visualization and interaction.Graphic cards and their image processing power can leverage large data visualization but they can also be of great interest to support interaction. Therefore, this book will show how to take advantage of graphic card computation power with techniques called GPGPUs (general-purpose computing on graphics processing units). As specific examples, this book details GPGPU usages to produce fast enough visualization to be interactive with improved brushing techniques, fast animations between different data representations, and view simplifications (i.e. static and dynamic bundling techniques).Since data storage and memory limitation is less and less of an issue, we will also present techniques to reduce computation time by using memory as a new tool to solve computationally challenging problems. We will investigate innovative data processing techniques: while classical algorithms are expressed in data space (e.g. computation on geographic locations), we will express them in graphic space (e.g., raster map like a screen composed of pixels). This consists of two steps: (1) a data representation is built using straightforward visualization techniques; and (2) the resulting image undergoes purely graphical transformations using image processing techniques. This type of technique is called image-based visualization.The goal of this book is to explore new computing techniques using image-based techniques to provide efficient visualizations and user interfaces for the exploration of large datasets. This book concentrates on the areas of information visualization, visual analytics, computer graphics, and human-computer interaction. This book opens up a whole field of study, including the scientific validation of these techniques, their limitations, and their generalizations to different types of datasets.
One of the most challenging ongoing issues in the field of 3D visual research is how to interpret human 3D perception over virtual 3D space between the human eye and a 3D display. When a human being perceives a 3D structure, the brain classifies the scene into the binocular or monocular vision region depending on the availability of binocular depth perception in the unit of a certain region (coarse 3D perception). The details of the scene are then perceived by applying visual sensitivity to the classified 3D structure (fine 3D perception) with reference to the fixation. Furthermore, we include the coarse and fine 3D perception in the quality assessment, and propose a human 3D Perception-based Stereo image quality pooling (3DPS) model. In 3DPS we divide the stereo image into segment units, and classify each segment as either the binocular or monocular vision region. We assess the stereo image according to the classification by applying different visual weights to the pooling method to achieve more accurate quality assessment. In particular, it is demonstrated that 3DPS performs remarkably for quality assessment of stereo images distorted by coding and transmission errors.
Conference Paper
A Cognitive Task Analysis (CTA) was performed to investigate the workflow, decision processes, and cognitive demands of information assurance (IA) analysts responsible for defending against attacks on critical computer networks. We interviewed and observed 41 IA analysts responsible for various aspects of cyber defense in seven organizations within the US Department of Defense (DOD) and industry. Results are presented as workflows of the analytical process and as attribute tables including analyst goals, decisions, required knowledge, and obstacles to successful performance. We discuss how IA analysts progress through three stages of situational awareness and how visual representations are likely to facilitate cyber defense situational awareness.