Expert-interviews led analysis of EEVi — A model for effective visualization in cyber-security

Abstract

The area of visualization in cyber-security is advancing at a fast pace. However, there is a lack of standardized guidelines for designing and evaluating the resulting visualizations. Furthermore, limited end-user involvement in the design process leads to visualizations that are generic and often ineffective for cyber-security analysts. Thus, the adoption of the resultant cyber-security visualizations is low and this highlights a major research gap. This paper presents expert-interview based validation of EEVi - a model developed to aid in the design and evaluation process of cyber-security visualizations, with a view to make them more effective for cyber-security analysts. A visualization is considered effective if the characteristics of the visualization are essential for an analyst to competently perform a certain task. Thirteen experts were interviewed (six visualization designers and seven cyber-security analysts) and their feedback guided revisions to the model. The responses were subsequently transposed from qualitative data to quantitive data in order to perform statistical analyses on the overall data. This demonstrated that the perspectives of visualization designers and cyber-security analysts generally agreed in their views of effective characteristics for cyber- security visualization, however there was no statistically significant correlation in their responses.

Full text

To appear in an IEEE VGTC sponsored conference - VizSec 2017
Expert-Interviews Led Analysis of EEVi - A Model for Effective
Visualization in Cyber-Security
Aneesha Sethi*
Electronics and Computer Science
University of Southampton
Aneesha.Sethi@soton.ac.uk
Gary Wills
Electronics and Computer Science
University of Southampton
gbw@ecs.soton.ac.uk
Abstract
The area of visualization in cyber-security is advancing at a fast pace. However, there is a lack of standardized guidelines for designing and evaluating the resulting
visualizations. Furthermore, limited end-user involvement in the design process leads to visualizations that are generic and often ineffective for cyber-security analysts.
Thus, the adoption of the resultant cyber-security visualizations is low and this highlights a major research gap. This paper presents expert-interview based validation of
EEVi - a model developed to aid in the design and evaluation process of cyber-security visualizations, with a view to make them more effective for cyber-security analysts.
A visualization is considered effective if the characteristics of the visualization are essential for an analyst to competently perform a certain task. Thirteen experts were
interviewed (six visualization designers and seven cyber-security analysts) and their feedback guided revisions to the model. The responses were subsequently transposed
from qualitative data to quantitive data in order to perform statistical analyses on the overall data. This demonstrated that the perspectives of visualization designers
and cyber-security analysts generally agreed in their views of effective characteristics for cyber-security visualization, however there was no statistically significant
correlation in their responses.
Keywords: Cyber-Security; Data Visualization; Qualitative Evaluation; Task and Requirements Analysis; Human Factors
1 INTRODUCTION
In the field of cyber-security, cyber attacks and threats are increasing day-by-day but so is the dependence of humans on cyber networks.
According to a survey by the Department of Culture, Media and Sport [11], 61% of UK businesses hold personal data electronically and 46%
of all UK businesses have identified at least one cyber security breach in the past year. Thus, the public and private sectors rely on the expertise
and capabilities of cyber-security analysts to protect assets and resources connected via computer networks and it is getting increasingly
critical to have proficient cyber defense solutions. The area of cyber-security visualization aims to develop these solutions and these are being
intensively researched. However, these solutions focus on the technological aspects of the tools rather than considering the critical roles played
by humans that affect cyber operations [28]. Additionally, industries are turning towards cyber-security analysts to mitigate cyber threats as the
automated defenses are not enough [16]. Hence, it is even more important to arm the cyber-security analysts with competent and efficient tools
to defend against potential cyber attacks.
To address these issues, this paper presents the results of the evaluation of EEVi (Fig. 1b) which represents a cognitive model to build
and evaluate effective visualizations for cyber-security taking user requirements into consideration. The model was evaluated by thirteen
experts (six visualization designers and seven cyber-security analysts) via expert-interviews to revise the model and component roles on the
basis of end-user requirement. These revisions also made sure that the terminology and structure clearly conveyed the cyber-security analysts
requirements to the visualization designers to decrease the knowledge gap between them.
Sect. 2 presents the background literature that led to the identification of the research gap and introduces the model being reviewed. Sect. 3
explains the interview set-up followed to conduct the expert-interviews and Sect. 4 presents the results of the expert-interviews and the revisions
made to the model and the component roles on the basis of the expert-interviews. Finally, Sect. 6 concludes this article by discussing the
research introduced in the paper and the usefulness of EEVi along with a brief insight in to the future work being undertaken to develop this
research further.
2 BACKG ROUN D LITERATURE
Cyber-security visualization provides cyber-security analysts with visual data rather than textual data for analysis. Its main goal is to provide
effective tools [14] that help detect, monitor and mitigate attacks in a timely manner. To quote [19], “A picture is worth a thousand log records”.
Visual analytics have helped cyber-security analysts raise their level of awareness to a more holistic approach and thus, helps identify problems
and find solutions visually [14]. The vast amount of data to process along with the need for new methods and tools have made it a new and
popular approach in the domain of cyber security [19].
A research gap was identified by the authors while reviewing this literature: most cyber-security visualization tools were introduced with
minimal or no end-user evaluation of the resultant visualization. Additionally, there were no set of useful guidelines to follow for cyber-security
visualization. The following section illustrates the the background literature that led to the development of EEVi.
*©©
2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.
1

To appear in an IEEE VGTC sponsored conference - VizSec 2017
2.1 Current Cyber-Security Visualization Solutions and Issues
The current cyber-security visualization solutions can be broadly classified in three main categories: Network Analysis, Malware and Threat
Analysis and Situational Awareness:
Network Analysis
tools focus on detecting possible attacks by mapping the physical network. It comprises of many types of tools, those that
focus on network monitoring of dark-net to extract similar groups of packets, especially the malicious ones using topological graphs [8]
or proactive tools like PERCIVAL that highlights potential attack vectors based on the state of network using attack-graphs [2].
Malware and Threat Analysis
tools focus on detecting and eliminating malware and threats. It comprises of many types of tools like DAVAST
that detect malware attack and its effects using graphs [29] or tools like OwlSight that focus on real- time detection of cyber-attacks using
maps and graphs [5] or insider-threat analysis solutions that focus on analyzing attacks by malicious insiders like tools that use machine
learning to display attack-pattern trees for anomalous behaviour [1].
Situational Awareness
tools provide high-level abstract view [12] of a system which is beneficial to both technical and non-technical people.
It comprises of many types of tools like Dagger, a modelling and visualization framework to represent knowledge and information for
decision-makers by using layering and sunbursts [21] or tools like ePSA that focus on mapping internet data for novice users to enhance
their situational awareness using timeline, scatterplot and fixed-ring layout visualizations [17].
However, most of these aforementioned tools have not been evaluated to determine their effectiveness in terms of the task they aid in
performing. Staheli et al. [26] conducted a survey and it showed that 46% of 130 tools did not have any user-involvement in the evaluation
phase. Additionally, Sethi et al. [24] conducted a sample survey of nine tools which showed that two out of nine tools had no form of
evaluation and three out of nine tools did not have any user-involvement. In the field of visualization in cyber-security there are very few
studies that validate the efficiency of resultant visualizations while keeping the analysts’ needs in mind [15]. Additionally, the evaluation
techniques used are not standardized due to the lack of a common model or any guidelines [26]. There is a major gap between the technological
solutions and research to address the issues of humans as cyber-security analysts [18,28]. This leads to low adoption rates of the resultant
visualizations [3], as they are not very effective and they are usually designed and often evaluated without end-user involvement. Thus, there is
a need to clearly understand the end-user’s needs and requirements to develop successful visualizations and not just develop “...pretty picture
visualizations...” [5].
2.2 Background of EEVi
Following from the previous section, there is a need for a common model to standardise the development and evaluation of effective cyber-
security visualizations for cyber-security analysts. As a consequence, EEVi was developed whilst keeping the requirements of the end-users in
mind [24, 25]. The component roles described as a part of EEVi provide the visualization designers with a basis for a dialog with cyber-security
analysts which leads to effective visualization solutions. The process of development of EEVi is briefly explained below.
EEVi was developed using a qualitative inductive approach called Thematic Analysis. It emphasizes identifying, analyzing and reporting
patterns (or themes) within data [4]. Therefore, thematic analysis of five papers [9, 10, 12, 14, 20] was carried out. These papers presented results
from cognitive task analysis (CTA) of security analysts and gave insight into the roles and tasks security analysts’ perform and information
about how to make cyber-security visualizations effective for them as end-users. The results of thematic analysis are displayed in Table 1 and
Table 2, these represent the definitions of all identified codes segregated by the themes they fall under. The Links column of both the tables
refers to which component task uses the code.
The themes and codes helped recognize the cognitive relationships from the dataset. The cognitive relationships led to a similar generic
storyline of themes and defined the model and shaped the structure of EEVi, as shown in Fig. 1a.
The guidelines for effective visualization for cyber-security based on this model can be defined by the component roles performed by
security analysts. These component roles were identified from the thematic analysis, as they were the most common roles performed by
cyber-security analysts. The purpose of these guidelines is to guide visualization developers towards creating effective visualizations for
cyber-security in order to reduce the knowledge gap between cyber-security analysts (end-users) and visualization designers.
The eight component roles performed by cyber-security analysts and their goals are defined below:
• Triage Analysis (TA) is the first look at data, false positives are weeded out for further analysis, within an order of few minutes;
• Escalation Analysis (EA) is the investigation of suspicious activities and production of reports;
• Correlation Analysis (CA) is the search for previously unrecognised patterns and trends in data;
• Threat Analysis (ThA) is an intelligent analysis to profile attackers and their motivations using additional sources;
• Incident Response Analysis (IRA) is a recommendation or implementation of action against a confirmed incident;
• Forensic Analysis (FA) is when an analyst gathers and preserves data to inform and support law enforcement agencies;
•
Impact Assessment (IA) is the task to identify impact, damage and critical nodes that may be compromised or potentially reachable after
a breach;
•
Security Quality Management (SQM) is the task related to services, like tutorials or training, that maintain the quality of information
security in an organization [9, 10, 12, 24, 25].
EEVi represents cognitive relationships for each component role to determine the critical features of visualization that are required to allow
the security analyst to competently perform the task. The cognitive relationships represent guidelines for effective visualization for each
identified role, which is explained in [24]. These component roles are substituted with their respective themes in the model definition to create
the guideline for effective cyber-security visualization for the particular role.
According to the model, a visualization is effective for cyber-security when the characteristics of visualization, that are critical for a security
analyst to competently perform a specific task, are implemented. Thus, the characteristics for effective visualization represent the resources
required by a security analyst to perform a task effectively and not the aesthetics (like the type or color of graphs) that would be required by a
cyber-security analyst.
2

To appear in an IEEE VGTC sponsored conference - VizSec 2017
Table 1: Definitions of codes from the Type of Data and the Role of End-User
Theme Code Definition Links
Type of
Data
Raw Data
Most elemental data, usually in very large quantity and is passed
through automated process to filter. TA
Interesting Activity
Data flagged by automated processes and is inspected by an analyst,
usually consists of a large amount of false positives. TA
Suspicious Activity
Data that is anomalous after the initial TA and needs to be monitored.
EA
Incident
The point when the occurrence and seriousness of an activity is
confirmed and formally reported. EA
Intrusion Set
Sets of related Incidents that are given an increase in attention and
resources to detect, understand and respond. CA,ThA, IRA
Source Data
Data gathered from an intrusion used for further analysis or report-
ing. IA, FA, SQM
Security Regulations
Regulations defined by the government or organizations relating to
cyber security; also includes cyber law. FA, SQM
Role of
End-User
Real-Time Analyst Performs Triage Analysis TA
Lead Analyst Performs Escalation Analysis EA
Tactical Defender
Defends against current and immediate attacks by maintaining situ-
ational awareness and rapid remediation of problems.
EA, CA, ThA,
IRA
Site-Specific Analyst Performs Correlation Analysis CA
Threat Analyst Performs Threat Analysis ThA
Strategic Analyst
Works at the community level to understand implications of attack
and categorise it. ThA, IRA
Incident Handler/ Respon-
der Performs Incident Response Analysis IRA
Forensic Analyst Performs Forensic Analysis FA
IT Manager
Identifies impact damage after an intrusion and executes training
and development to maintain quality of workflow. IA, SQM
3 EX PE RT IN TE RVI EW SET UP
Expert-interviews of thirteen experts was conducted to update and validate the model. This review was conducted with approval from Ethics
and Research Governance (ERGO) under reference number ERGO/F PSE /23974.
Seven cyber-security analysts and six visualization designers were interviewed in a semi-structured format. The experts were asked some
general questions about their work and knowledge of their respective fields. This was followed by showing them the model structure and
discussing that in detail. Subsequently they were shown the structure and definitions of each component role and these were discussed in great
detail. Finally, they were asked about the usefulness of the model and any final thoughts. The interviews were analyzed using an abductive
mixed methods approach by first segregating the responses by question using question-based coding and then inductively coding the responses
in isolation for two purposes: to update the model and to quantify the qualitative responses for characteristics of visualization per component
role. The qualitative analysis were conducted by using NVivo
1
and the quantitative analysis ere conducted by using IBM SPSS
2
and GraphPad
Prism 3.
Both cyber-security analysts and visualization designers were interviewed to understand their individual points of view in order to update
the model to accommodate both of their perspectives to minimize the knowledge gap between the two fields.
4 EX PE RT IN TE RVI EW RES ULTS
Thirteen experts (six visualization designers and seven cyber-security analysts) were interviewed. They were from various geographical
locations with different experiences in different areas of their respective fields, which is represented in Table 3. Additionally, four out of seven
cyber security analysts and three out of six visualization designers had used cyber-security visualization tools prior to the interview.
Note-worthy critique from the expert-interviews are presented below:
•
All the experts unanimously agreed that the model and the component roles represented good fundamental guidelines for cyber-security
visualization.
•
All the experts unanimously agreed that the model and the component roles are useful to evaluate the effectiveness of cyber-security
visualization.
1https://www.qsrinternational.com/product/nvivo-mac
2https://www.ibm.com/analytics/us/en/technology/spss/
3https://www.graphpad.com/scientific-software/prism/
3

To appear in an IEEE VGTC sponsored conference - VizSec 2017
Table 2: Definitions of codes from the Characteristics of Visualization
Code Definition Links
Alerts A system to alert the user of the status of an activity being investigated. TA
Case-Building Capabilities
Provides support to user for the purpose of building a case. FA
Chain of Custody Maintains a log of users who have analyzed data or had access to data from an incident. FA
Collaboration
Enable users to communicate and collaborate with other analysts by sharing findings and
other information. EA, ThA, IRA
Color Highlighting Using color to highlight the the risk level of an activity to bring it to the attention of a user. TA
Correlation Displays relationships between different data dimensions. ThA
Feedback Provides feedback (to manager) for tasks performed, could be quantitive or qualitative. SQM
Filter
Allows the ability to easily filter, join or transform data without changing the original; Also
allows ability to filter noise to allow analyst to see trends. TA
Flexibility Gives the ability to manipulate the focal point and support the analytical process. CA
Impact Identification
The identification of vulnerabilities; malicious users or external source of attacks; intended
target of attacks; main resources of the system affected. IA
Interoperation Ability of tool to work efficiently with other tools, applications, utilities or data-sets.
EA, ThA, IRA,
FA, SQM
Investigatory Capabilities
Allow users to investigate data by supporting providing platform for rapid, open-ended
foraging activities. CA
Mitigation Performs clean-up and containment and provides support for mitigation activities. IRA
Priorities Using a priority system to inform user of the severity of attack. EA, ThA
Real-Time Access Viewing real-time data within seconds to minutes of an event. TA
Reporting Providing support for report building.
EA, FA, IRA,
IA
Situational Awareness
An accurate picture of external and internal information in an overview to understand the
state of all resources. TA, IRA, IA
Timeline
Order of events and activities that have taken placed over a period of time, used to coordinate
all views. CA
•
The experts observed that the characteristics: Reporting (six experts), Interoperation (four experts), Collaboration (three experts),
Flexibility (three experts), Situational Awareness (three experts) and Filter (two experts) are potential characteristics that are useful for all
component roles (Refer to Table 2 for definitions of these characteristics).
Additionally, the experts made some comments critiquing EEVi such as this model is the “...need of the hour...[ and this] can fill the
[knowledge] gap...[and] nobody has thought of this...”. It was also stated that “...we need this... [and its] going in the right direction...” and
“...[component roles are] covered in a good manner and across all controls we know...”. Finally some of the experts concluded their interview by
saying that the research shows “...great effort...” and “...its good [and] specific [to the task at hand]...”. Therefore, it can be concluded that the
experts believed the model and component roles to be useful for visualization in cyber-security.
The following sub-sections focus on revising the model and component roles based on the feedback received from the experts.
4.1 Revisions of EEVi on the basis of Expert-Interviews
The experts were presented with the original model (Fig. 1a) and on the basis of their assessment some revisions were made to the terminology
and structure of EEVi. The experts unanimously agreed with the logic of the model, however four experts (two cyber security analysts and two
visualization designers) did not agree with the model representing their organizations’ logic of how tasks are performed.
The feedback from the experts can be divided into two main areas, namely, the terminology used in EEVi and the structure of EEVi. The
revised version of EEVi is displayed in Fig. 1b.
4.1.1 Terminology used in EEVi
Most experts (especially the visualization designers) believed that the terminology used to define the model can be more distinct and clearer.
On the basis of their comments, the terminology was modified.
•
The term ‘Analysis of Data’ was replaced by ‘Goal of Task’ to make the distinction more apparent about what the goal of each task is,
which in turn sets the goal for the visualization for the task at hand.
• ‘Data’ was updated to ‘Type of Data’ to make the elements of this category more distinct.
•
The term ‘Role of Analyst’ was revised to ‘Role of End-User’ to clearly identify the end user who would be using the resultant
visualizations.
•
‘Features of Visualization’ was modified to ‘Characteristics of Visualization’ to focus on the aspects of visualization that aid security
analysts with their task rather than the features, which could be interpreted as aesthetics.
4

To appear in an IEEE VGTC sponsored conference - VizSec 2017
Table 3: Demographic Information concerning the experts
Country Current Job Description Experience
Cyber-Security
Analyst
UK System Analysis Engineer and PhD student - University of Southampton 7 years
India Cyber Crime Investigator 3 years
India Network Security Superintendent - CBI (Law Enforcement) 10+ years
India Cyber Law Enforcement Officer - Indian Police Service 9 years
Malaysia Cyber-Security Policy and Strategy Expert - PricewaterhouseCoopers 29 years
India Lead Cyber-Security Consultant 16 years
USA Security Operations Manager - HCL Americas 18 years
Visualization
Dessigners
USA PhD in Data Visualization Design - University of Utah 4+ years
UK PhD in HCI Design - University of Southampton 11 years
UK Research Staff in HCI - University of Southampton 6 years
Brazil Design Researcher 10+ years
UK Postdoctoral Researcher - Royal Holloway, University of London 6 years
UK Director of a UX Design Company 10+ years
Therefore, EEVi (Fig. 1b) can be defined as a model to aid in the design and evaluation process of cyber-security visualization to make the
resultant visualizations more effective for the end user (or security analyst), so that they can competently perform the task at hand with the
available resources. The ‘Goal of Task’ identifies the goals of each task at hand, which can be performed using the ‘Type of Data’ by the ‘Role
of End-User’ requiring the ‘Characteristics of Visualization’ to make the resultant visualization effective. Hence, the eight component roles
introduced in Sect. 2.2 are defined in the format of EEVi to determine the critical characteristics of visualization that make the task effective for
the end-user (or security analyst).
4.1.2 Structure of EEVi
The experts believed the representation of the model could be enhanced by changing the flow of logic and the structure of EEVi (Fig. 1).
However, most of the experts believed that ‘Goal of Task’ and ‘Type of Data’ should be the first aspects to consider, after which the ‘Role of
End-User’ was identified and lastly the ‘Characteristics of Visualization’ which represent the critical resources that were identified to perform
the task. This logic flow resonated more with the way the organizations performed tasks.
Additionally, the experts speculated about the structure of the model being sequential and not allowing for an iterative flow, if need be.
Accordingly, the structure of EEVi was revised from a sequential flow to a cyclic one (Fig. 1b). The initial aspect is clearly defined as the entry
point into a cyclic flow, and the final aspect can either iterate back to the initial task and continue the loop or the final aspect can close the loop
with ‘Effective Cyber-Security Visualization’.
4.2 Statistical Analysis Setup from Qualitative Data of Expert-Interviews
The experts gave qualitative feedback in their interviews about the model as well as the individual component roles. Following the revisions of
EEVi, the next stage was to analyze the data and update the component roles on the basis of the feedback received. Prior to these updates,
the authors conducted tests of statistical significance to assess the similarity in the responses of visualization designers and cyber-security
analysts about the ‘Characteristics of Visualization’ which make a task effective. However, to perform these tests, the data from each individual
component had to be quantified.
The quantification of the responses was performed using a combination of methodology for Integrative Mixed Methods (IMM) Studies
defined by Castro et al. [6] and the Indicator Measurements followed by Purwandari [22]. The first four steps of IMM approach were
implemented to quantify the qualitative feedback from the expert-interviews on the basis of the indicator measurements defined for each expert.
The IMM steps were: (a) Eliciting Responses, (b) Identifying Response Codes, (c) Creating Thematic Categories and (d) Scale Coding
(Fig. 2) [6]. In the first step, eliciting responses, the relevant responses of ‘Characteristics of Visualization’ for each component task of the
expert were identified. This was followed by identifying response codes wherein the relevant responses were encoded by responses codes,
which was represented by each characteristic of visualization defined. Subsequently, the third step, creating thematic categories, was followed at
which point the five thematic categories were created, namely, ‘very positive response’, ‘positive response’, ‘neutral or no response’, ‘negative
response’ and ‘very negative response’, similar to the indicator measurements defined by Purwandari [22]. Each response code was assigned
to one of these categories on the basis of the relevant responses encoded by it. Finally, in the Scale Coding, the thematic categories were
quantified with values
+2,+1,0,−1
and
−2
, respectively. Thus, each ‘Characteristic of Visualization’ was quantified, as can be seen in Fig. 2,
to perform statistical analysis.
The quantization of the component task TA for a Participant V6 is displayed in Fig. 2 as an example. As can be seen from Fig. 2, the relevant
responses for the ‘Characteristics of Visualization’ were coded to response codes (in this example, there was one relevant response for each
response code, however there can be more than one as well) which represented the three characteristics of visualization for triage analysis. The
response codes, on the basis of the type of responses are assigned to one thematic category. This thematic category represents a quantifiable
number, which is identified from the scale coding. Thus, TA Speed is quantified to value 2, TA SA is quantified to value 0 and TA Filter is
quantified to value 2. Similarly, each characteristic of each component task of each expert was quantified for the purpose of the statistical
analysis.
5

To appear in an IEEE VGTC sponsored conference - VizSec 2017
(a) Initial model - EEVi
(b) Revised and Validated EEVi
Figure 1: Transformation of EEVi from the initial sequential-flow model to a cyclic one with updated terminology
4.3 Statistical Analysis Results on the basis of Expert-Interviews
Following form the previous section, the authors conducted tests of statistical significance on the quantified data to compare the responses for
individual component roles between the visualization designers and the cyber security analysts. For this purpose, two tests were conducted, a
test for accuracy of results and a test for precision.
The test for accuracy, analyzes the difference between data sets. It does not check if the responses of data sets are good or not, however it
checks for the difference in responses between the cases. In this case, it is used to check if there is any statistical significant difference between
the responses of visualization designers and cyber security analysts for each characteristics of visualization for each component task. It is
performed using Mixed Design ANOVA in Sect. 4.3.1.
The test for precision, analyzes the relationship between different data elements. This test can be used to determine how good or bad the
responses are by using graphs to visualise the relationship between different data elements. In this case, precision is used to check if there is any
statistically significant correlation between the responses of visualization designers and cyber-security analysts for each of the characteristics of
visualization for each component roles. It is performed using Pearson’s r Correlation in Sect. 4.3.2.
4.3.1 Test for Accuracy: Mixed Design ANOVA
ANOVA (ANalysis Of VAriance) is used to analyze statistically significant differences between sample means used to determine what
proportion of variation in the dependent variable can be attributed to independent variable(s) or groups [23]. Mixed Design ANOVA combines
repeated measures (or repeated results) with the proportion of variation between independent variable(s) or groups [13]. For the purpose of this
analysis, the dependent variable is a quantitative variable which is represented by the means of each characteristic of visualization of each
component role. The independent variable is represented by the two groups: cyber-security analysts and visualization designers.
Table 4: Mixed Design ANOVA - Results of between and within subject effects for Greenhouse-Geisser correction
Source df Mean Square F-Value Sig (p)
Type of Expert 1 2.142 1.538 0.241
Characteristics of Visualization * Type of
Expert 5.870 1.695 0.836 0.544
Characteristics of Visualization 5.870 3.891 1.919 0.092
Error (Characteristics of Visualization) 64.567 2.027 - -
6

To appear in an IEEE VGTC sponsored conference - VizSec 2017
Figure 2: An example of IMM Process to quantify qualitative data
The results of the Mixed Design ANOVA are presented in Table 4. In the table, df stands for degrees of freedom and F-value gives a measure
of how much the means differ relative to the variability between groups [13]. Sig (p) represents the the significant value or pvalue, if this value
is greater than 0.05 then there is a non-significant effect. The results, from Table 4 are interpreted in the following way:
The null hypothesis (Ho) assumption:
Ho: “There is no significant difference between the responses of cyber- security analysts and visualization designers”
Mixed Design ANOVA was conducted to compare the differences between the responses of cyber-security analysts and vizualisation designers within the
seventeen characteristics of visualization.
There was a non-significant effect of the responses between cyber-security analysts on the tresponses of visualization designers at the p<0.05 level for the
conditions
[F(1,5.8)=1.538, p=0.241>0.05]
. This demonstrates that the responses of cyber-security analysts were not statistically significantly different
from the responses of visualisation designers for the seventeen characteristics of visualization.
There was a non-significant interaction within the responses of characteristics of visualization for the two types of experts for the Greenhouse-Geisser4
correction for the conditions
[F(5.8,64.5)=0.84, p=0.544>0.05]
. Therefore, the responses for characteristics of visualization for cyber-security analysts
were not statistically significantly different from the responses given by visualization designers.
These results demonstrate that there were no statistically significant differences between responses of cyber-security analysts and visualization
designers in terms of the characteristics that make visualization effective.
Therefore, it is safe to conclude that for Null Hypothesis (Ho) -
Ho: “There is no significant difference between the responses of cyber-security analysts and visualization designers”
=>Ho cannot be rejected for each characteristic of visualization.
Therefore, it can also be concluded that a model for effective visualization approved by cyber-security analysts, when applied by visulization
designers, leads to a non significant difference in their understanding of the problem. This point is further emphasised by the next calculation
which demonstrates that given a common model (EEVi), the two groups agreed which characteristics of visualization make it more effective for
cyber-security analysts.
Additionally, the results from Table 4 can be used to demonstrate if there were any statistically significant difference between the means of
the characteristics of visualization.
According to the table, the significant main effect for Characteristics of Visualization for
F(5.8,64.5)=1.92 where p=0.092>0.050
. This demonstrates that
the means of characteristics of visualization are not significantly different.Therefore, we can infer that the two groups generally agreed within their views of
which characteristics make a visualization effective for the given task, as the means do not have statistically significant differences.
Thus, it can be concluded that EEVi helps the visualization designers understand the requirements of cyber-seciurity analysts, as the two
groups generally agreed in their responses given the same information. The following section demonstrates if the agreement was towards
positive or negative responses.
7

To appear in an IEEE VGTC sponsored conference - VizSec 2017
Figure 3: Pearson’s r Correlation results displayed in the upper right quartile of the scatterplot to check for statistical significant correlation
4.3.2 Test for Precision: Pearson’s r Correlation
Pearson’s rCorrelation is also called Pearson’s Product-Moment Correlation because it is calculated by multiplying two variables (or product)
and then calculating the average (or moment) of the products based on group of ncases [7]. The interpretation of the results of Pearson’s r
correlation is explained below:
A Pearson product-moment correlation coefficient (r) was computed to assess the relationship between the responses of cyber-security analysts and visualization
designers. Pearson’s rdata analysis revealed no significant correlation between the two groups [r(15)=0.395,p=0.117>0.05]. A scatterplot summarizes
these results in Fig. 3. Overall, this means that high-valued or low-valued for responses of cyber-security analysts do not significantly relate to high-valued or
low-valued responses of visualization designers.
The scatterplot in Fig. 3 represents the upper right quartile of the correlation of each characteristic of visualization for each component task.
Although there is no statistically significant correlation between the two groups, the scatterplot shows the consolidation of data-points in the
upper right quartile, which means that the two groups generally agree within their views of which characteristics make a visualization effective
for the given task. On the basis of the quantization performed in Sect. 4.2, it can be concluded that the means of all the responses was positive
as the means fell between 0.3 and 2. Thus, the characteristics of visualizations received positive responses all in all. The regions represented in
Fig. 3 are described in more detail in Sect. 4.4, these are used to modify and update the component roles of EEVi.
4.4 Modifications of Component Roles of EEVi on the basis of Expert-Interviews
The analysis of the interviews from the expert-interviews led to the modification of the component roles, these roles are initially defined in
Sect. 2.2. EEVi has been developed using a qualitative inductive approach called Thematic Analysis, as explained in Sect. 2.2. The process of
thematic analysis involves the generation of codes and themes. Codes represent excerpts of related data that are used to intuitively identify
the aspect of data it represents and themes captures the significance of data in a patterned response by being attached to cluster of similar
codes [27]. To enhance the understanding of the component roles, Table 1 and Table 2 present definitions of codes of themes ‘Type of Data’,
‘Role of End-User’ and ‘Characteristics of Visualization’ that are used in the component task representations. The definitions of the codes (eg:
Triage Analysis...) under the theme ‘Goal of Task’ are already given in Sect. 2.2 and are not included in the table. The Links column of both the
tables refers to which component task uses the code.
The modification of the component task TA is represented in Fig. 4. The experts reviewed TA and their feedback directed the modification of
this component task.
All the experts unanimously agreed with task and the various aspects that represent the task. The six visualization designers unanimously
agreed that the characteristics of visualization were implementable. Three visualization designers believed Situational Awareness to be the
most difficult to implement, two of them said Real-Time Access was most difficult to implement and one of them assumed Filter to be the most
difficult to implement.
The original component task was a hierarchical structure as shown in Fig. 4a. This structure was modified to a circular list and the
terminology was altered on the basis of the feedback received: Speed was too ambiguous and was replaced by Real-Time Access to avoid
confusion.
The next step was to apply the regions from Fig. 3. As can be seen, TA-Filter and TA-SA fall within Region I as these characteristics received
a mean score greater than or equal to one (mean
>
=1) from cyber security analysts and a mean score greater than zero from visualization
designers (mean
>
0) and were moved into the confirmed category. However, TA-Speed falls within Region II as this characteristic received a
mean score greater than zero but less than one (0
<
mean
<
1) from cyber security analysts and a mean score greater than zero from visualization
designers (mean
>
0) and was moved into the unresolved category. In accordance with the feedback, three experts requested for an Alerts
characteristic and two experts requested a Color Highlighting characteristic, thus these were also moved to the unresolved category.
Therefore, the Fig. 4b represents the component task representation for Triage Analysis with the characteristics in the confirmed category.
Nonetheless, Real-Time Access,Alerts and Color Highlighting are in the unresolved category awaiting further analysis.
All the other component roles were similarly transformed from hierarchal structures to circular lists.
4
As the Sphericity condition is not met (due to having only two conditions: cyber-security analysts and visualization designers). Field [13] recommends
recommends using Greenhouse-Geisser correction if the Greenhouse-Geisser estimate is less than 0.75 or using Huynh-Feldt correction if the estimate is greater
than 0.75. In this case, Greenhouse-Geisser estimate
=0.367<0.750
. Therefore, Greenhouse-Geisser corrected significant values are used to interpret the results.
8

To appear in an IEEE VGTC sponsored conference - VizSec 2017
(a) Initial model - EEVi (b) Revised and Validated EEVi
Figure 4: Transformation of EEVi from the initial sequential-flow model to a cyclic one with updated terminology
5 DISCUSSION
The results of the expert-interviews demonstrated the validity of EEVi for effective cyber-security visualizations for cyber-security analysts.
The themes represented in the model (Fig. 1b) can be substituted by the aspects of the component roles (as shown in Fig. 4b) to represent the
guidelines for effective cyber-security visualization.
The use of the model can be threefold: it can be used by visualization designers to evaluate their work related to cyber-security; it can be used
by organisations or individuals, looking to purchase cyber-security visualization solutions, in terms of efficacy for the cyber-security analysts;
and finally, it can be used by stakeholders as a basis of communication between themselves, the visualization designers and cyber-security
analysts (end-users).
6 CONCLUSION AND FUTURE WORK
The literature draws attention to a major issue in the field of cyber-security visualization: the lack of end-user involvement in the development
process. Additionally, the evaluation techniques adopted lack end-user involvement which leads to a lack of clarity regarding the effectiveness
of the resultant cyber-security visualizations. Thus, there is need for a common model, within the field, which appreciates the requirements of
end-users (in this case, cyber-security analysts) and focuses on guidelines to design and evaluate visualizations that prove to be effective for the
end-users.
To address these issues, EEVi was developed from CTA results of cyber-security analysts and has been validated and revised on the basis of
the interviews of thirteen experts. These experts were six visualization designers and seven cyber-security analysts from academia and industry.
On the basis of their assessment, the component roles of the model were modified and updated as well. Thus, minimizing the knowledge gap
between the two groups by accommodating both their perspectives in the revised versions.
The experts generally agreed in regards to which characteristics make visualization effective for cyber-security analysts. Statistical analyses
were performed on the responses of visualization designers and cyber-security analysts. The qualitative responses were quantified and the
analyses were carried out. The statistical analysis also illustrated that there were no statistically significant differences between the responses
of the visualization designers and the cyber-security analysts. However, neither was there any statistically significant correlation between the
responses of the two groups.
Therefore, on the basis of the feedback from the expert-interviews it can be concluded that EEVi and the component roles represent a useful
model to design and evaluate cyber-security visualizations, with the intention of making them more effective for cyber-security analysts.
The future work for this research entails a further assessment to confirm the modified model and component roles, along with confirming the
critical characteristics of visualization that would make each task effective. This would be followed by a rigorous real-world evaluation of the
model to demonstrate the usefulness of EEVi by applying it to real-world scenarios and creating a form of quantitive evaluation to measure the
efficacy of cyber-security visualization solutions for different real-world scenarios.
ACK NOW LE DGM EN TS
The authors wish to thank the University of Southampton for sponsoring and providing the necessary resources to perform this research.
Additionally, the authors wish to thank Lester Gilbert for his useful advice and guidance in performing the statistical analysis.
9

To appear in an IEEE VGTC sponsored conference - VizSec 2017
REFERENCES
[1]
I. Agrafiotis, J. R. C. Nurse, O. Buckley, P. Legg, S. Creese, and M. Goldsmith. Identifying attack patterns for insider threat detection. Computer Fraud &
Security, 2015(7):9–17, July 2015.
[2]
M. Angelini, N. Prigent, and G. Santucci. PERCIVAL: proactive and reactive attack and response assessment for cyber incidents using visual analytics. In
Proc. VizSec, pages 1–8. IEEE, Oct 2015.
[3] D. M. Best, A. Endert, and D. Kidwell. 7 key challenges for visualization in cyber network defense. In Proc. VizSec, pages 33–40. ACM, nov 2014.
[4] V. Braun and V. Clarke. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101, Jul 2006.
[5]
V. S. Carvalho, M. J. Polidoro, and J. P. Magalhes. Owlsight: Platform for real-time detection and visualization of cyber threats. In Proc. BigDataSecurity,
pages 61–66. IEEE, April 2016.
[6]
F. G. Castro, J. G. Kellison, S. J. Boyd, and A. Kopak. A methodology for conducting integrative mixed methods research and data analyses. Journal of
Mixed Methods Research, 4(4):342–360, Dec 2010.
[7] P. Y. Chen and P. M. Popovich. Correlation: Parametric and Nonparametric Measures. SAGE, Thousand Oaks, CA, 2002.
[8]
M. Coudriau, A. Lahmadi, and J. Franois. Topological analysis and visualisation of network monitoring data: Darknet case study. In Proc. WIFS, pages
1–6. IEEE, Dec 2016.
[9] A. D’Amico and K. Whitley. The real work of computer network defense analysts. In Proc. VizSec, pages 19–37. Springer Berlin Heidelberg, Oct 2007.
[10]
A. D’Amico, K. Whitley, D. Tesone, B. O’Brien, and E. Roth. Achieving cyber defense situational awareness: A cognitive task analysis of information
assurance analysts. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 49(3):229–233, Sep 2005.
[11]
Department of Culture, Media and Sport. Cyber security breaches survey 2017 - main report. Klahr, R., Shah, J. N., Sheriffs, P., Rossington, T., Pestell, G.,
Button, M., and Wang, V.
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/609186/Cyber_Security_
Breaches_Survey_2017_main_report_PUBLIC.pdf, april 2017. Accessed: 5 May, 2017.
[12]
R. F. Erbacher, D. A. Frincke, P. C. Wong, S. Moody, and G. Fink. A multi-phase network situational awareness cognitive task analysis. Information
Visualization, 9(3):204–219, Jan 2010.
[13] A. Field. Discovering Statistics Using IBM SPSS Statistics. SAGE, London, 4th edition, 2013.
[14] G. A. Fink, C. L. North, A. Endert, and S. Rose. Visualizing cyber security: Usable workspaces. In Proc. VizSec, pages 45–56. IEEE, Oct 2009.
[15]
C. J. Garneau, R. F. Erbacher, R. E. Etoty, and S. E. Hutchinson. Results and lessons learned from a user study of display effectiveness with experienced
cyber security network analysts. In Proc. LASER, pages 33–42. USENIX Association, May 2016.
[16]
R. S. Gutzwiller, S. M. Hunt, and D. S. Lange. A task analysis toward characterizing cyber-cognitive situation awareness (CCSA) in cyber defense analysts.
In Proc. CogSIMA, pages 14–20. IEEE, March 2016.
[17] P. A. Legg. Enhancing cyber situation awareness for non-expert users using visual analytics. In Proc. CyberSA, pages 1–8. IEEE, June 2016.
[18]
V. F. Mancuso, A. J. Strang, G. J. Funke, and V. S. Finomore. Human factors of cyber attacks. Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 58(1):437–441, Oct 2014.
[19] R. Marty. Applied Security Visualization. Addison-Wesley Professional, 1 edition, 2008.
[20]
S. Mckenna, D. Staheli, and M. Meyer. Unlocking user-centered design methods for building cyber security visualizations. In Proc. VizSec, pages 1–8.
IEEE, Oct 2015.
[21] E. Peterson. Dagger: Modeling and visualization for mission impact situation awareness. In Proc. MILCOM, pages 25–30. IEEE, Nov 2016.
[22]
B. Purwandari. Developing a Model of Mobile Web Uptake in the Developing World. PhD thesis, School of Electronics and Computer Science, University
of Southampton, Southampton, UK, 2013.
[23] A. Rutherford. ANOVA and ANCOVA: A GLM Approach. John Wiley & Sons, New Jersey, 2nd edition, 2012.
[24]
A. Sethi, F. Paci, and G. Wills. EEVi - framework and guidelines to evaluate the effectiveness of cybersecurity visualization. International Journal of
Intelligent Computing Research, 7(4):761–770, Dec 2016.
[25]
A. Sethi, F. Paci, and G. Wills. EEVi - framework for evaluating the effectiveness of visualization in cyber-security. In Proc. ICITST, pages 340–345. IEEE,
Dec 2016.
[26]
D. Staheli, T. Yu, R. J. Crouser, S. Damodaran, K. Nam, D. O’Gwynn, S. McKenna, and L. Harrison. Visualization evaluation for cyber security: Trends
and future directions. In Proc. VizSec, pages 49–56. ACM, April 2014.
[27]
M. Vaismoradi, J. Jones, H. Turunen, and S. Snelgrove. Theme development in qualitative content analysis and thematic analysis. Journal of Nursing
Education and Practice, 6(5):100–110, 2016.
[28]
A. Vieane, G. Funke, R. Gutzwiller, V. Mancuso, B. Sawyer, and C. Wickens. Addressing human factors gaps in cyber defense. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, 60(1):770–773, Sep 2016.
[29] T. W¨
uchner, A. Pretschner, and M. Ochoa. Davast: Data-centric system level activity visualization. In Proc. VizSec, pages 25–32. ACM, Nov 2014.
10