Detection of Anomalies and Attacks in Container Systems: An Integrated Approach Based on Black and White Lists

Abstract

In this paper, we propose an approach for anomaly and attack detection based on the analysis of kernel logs obtained with enhanced Berkley Packet Filter (eBPF). Based on the logs we generate white and blacklists in form of rules for detection, that in comparison with machine learning models can be interpreted and adjected by specialist. In paper we propose four approaches for anomaly detection and attack detection. Paper also includes description of experiments where we evaluate proposed solution on three datasets with attacks. Experiments show that based on simple rules is possible to achieve a good attack and anomaly detection efficiency.

Full text

Detection of anomalies and attacks in container systems:
an integrated approach based on black and white lists
Igor Kotenko[0000-0001-6859-7120], Igor Saenko[0000-0002-9051-527 2],
Andrey Chechulin[0000-0001-7056-6972], Lidia Vitkova[00 00-0002-4945-6151],
Maxim Kolomeec [0000-0002-7873-2733], Igor Zelichenok[0000-0002-5 211-9025], Maxim Melnik,
Denis Makrushin, Nikita Petrevich
St. Petersburg Federal Research Center of the Russian Academy of Sciences,
Saint Petersburg 39, 14th Line V.O., 199178, Russia
Abstract. In this paper, we propose an approach for anomaly and attack detection
based on the analysis of kernel logs obtained with enhanced Berkley Packet Filter
(eBPF). Based on the logs we generate white and blacklists in form of rules for
detection, that in comparison with machine learning models can be interpreted
and adjected by specialist. In paper we propose four approaches for anomaly de-
tection and attack detection. Paper also includes description of experiments
where we evaluate proposed solution on three datasets with attacks. Experiments
show that based on simple rules is possible to achieve a good attack and anomaly
detection efficiency.
Keywords: cloud security, docker, eBPF, rule-based attack detection, anomaly
detection, whitelist, blacklist.
1 Introduction
The change of the classical approach for building distributed applications to micro-
service architecture, the emergence of IoT technologies, and the rapid development of
services in telecommunications contributed to the development of cloud computing and
containerization systems. In 2020, the global application container market, according
to statistics published by Statista in 2022, was $2.1 billion. According to forecasts, by
2023 the market volume will reach 5 billion US dollars, increasing by about 33 percent
per year [1]. One of the main features of container technologies is the possibility of
dynamic distribution of computing load on servers. Load distribution occurs
automatically and reduces the resources for service execution.
Scalability and cost-effectiveness of resource consumption are the main advantages
of container technologies from the point of view of a cloud service provider. The
required infrastructure changes are performed in hot-swap mode and do not affect
customers. The use of container technologies guarantees significant advantages to both
the provider and the user. However, the rapid development of containerization systems
leaves open the issue of security, which is a serious problem that requires proportionally
great attention at any time. In addition to the standard security threats in systems using
containerization technology, there are new threats, such as container escape, privilege
preprint
final version available at: https://link.springer.com/chapter/10.1007/978-3-031-19620-1_11

2
escalation, image compromise, etc. An attack of one service implemented in a container
allows the intruder to damage not only the attacked object but also other services
running in neighboring containers in the containerization system or on the host.
The existing approaches to the formation of attack detection systems [2, 3] can be
divided into three classes. The first class is systems for detecting attacks and anomalies
based on expert knowledge (signatures of attacks and anomalies). So, an operator or
developer, due to their knowledge of the behavior of the protected container and
possible attacks, can create a set of rules that allow detecting such attacks. An example
of this approach is most firewalls, which rules are set manually by the administrator of
the local network. The disadvantage of such systems is that the person responsible for
forming the rules may make mistakes or not know something, new attacks can appear
very quickly, and the protected system can be changeable, which leads to the rapid
obsolescence of the rules. The second class is automated attack detection systems based
on machine learning methods. Such systems can be trained on data sets containing
attacks and, thus, they learn to recognize attacks (including those similar to those on
which they were trained). Such systems can be pre-trained on new attacks, which allows
one to quickly update the list of detected attacks. The disadvantages of such systems
include the fact that the trained artificial intelligence model cannot be changed directly
by the operator, and the logic of its operation is hidden, which does not allow for fully
interpreting the results of its work. The third class is automated anomaly detection
systems based on machine learning methods. Such systems are trained on data sets
characterizing the normal operation of the protected container, which makes it possible
to detect deviations from normal functioning. The disadvantages of such systems can
also be attributed to the fact that the trained model of artificial intelligence cannot be
changed directly by the operator, and the logic of its operation is hidden, which does
not allow for fully interpreting the results of its work.
There is a contradiction, the rapid development of applications and services in
telecommunications and in IoT contributes to the active expansion of the market for
containerization protection technologies. Services and applications running in
containers become a potential target for intruders. But existing approaches to ensuring
the information security of containers based on expert knowledge or machine learning
methods are not always able to adapt to new attacks and, at the same time, maintain full
interpretability of the results. It is necessary to develop intrusion detection systems,
taking into account the emerging contradiction.
The paper proposes an integrated approach that provides detection of attacks and
anomalies by automated generation of white and black lists of security events for
containers. Such lists are generated automatically and, at the same time, can be easily
read and changed by the operator of the protection system. In addition, this approach
allows one to adapt the protection system for existing services and applications in the
system using containerization technologies. Thus, the main contribution and scientific
novelty of the integrated approach is that, unlike existing analogs, this approach
provides the creation of new human-readable rules, generation of white/black lists,
container behavior profiles, and adaptation of attack signatures for services and
applications operating in the system.

3
The article consists of five sections. The first section is an introduction, which
discusses the relevance and significance of the research topic. The second section
presents the results of the analysis of relevant works. The third section describes the
proposed comprehensive approach to detecting attacks and anomalies based on
automating the creation of black and white lists of security events in container systems.
The fourth section presents the results of the experiments. The fifth section analyzes
the advantages and disadvantages of the proposed approach. The article concludes with
a section with a conclusion and a brief description of possible further directions of
research.
2 Related works
One of the new technologies for collecting security event data is the eBPF filter. In
[4], an overview of applications for system introspection of distributed systems is
presented, the emphasis is on solutions based on eBPF under the Linux kernel. Also,
special attention is paid by the authors [4] to Log-Based Analysis as a concept of using
system logs, logs generated by various applications, or some combination of both to
obtain conclusions about the state of the system. In fact, collecting data about security
events using the eBPF filter is the creation of a tuple with the value of the event and its
arguments. In this case, the event is not equal to the system call but often has a similar
name. For example, the "OPENAT" event includes arguments of system calls: read,
write, close, etc. The list of system calls, and their arguments depends on the setting of
the eBPF filter. That is why there is a reduction in overhead costs. The article [4] does
not mention this, although it is very important for understanding eBPF, but in general
it provides a useful overview of the technology for developers.
In [5], a network monitoring solution is proposed below the Kubernetes application,
at the host level. At the same time, the load on the orchestration system is less than 1%.
The proposed solution also uses machine learning models. But the solution itself is
rather aimed at network diagnostics with subsequent load balancing. There are quite a
lot of such studies, and this is the most popular direction for approaches based on eBPF
[6, 7, 8]. For example, in [9] the concept of a mobile gateway for the 5G architecture
based on eBPF/XDP is proposed, and in [10] the methodology of communication
management with content for SDN networks is considered. In the article [11], the
filtering parameters configured in eBPF for container security events are proposed to
be used to protect the system from a wide range of attacks. The authors offer their own
framework called SNAPPY. The framework allows one to create new policies, which
are then attached to policy_NS. Processes are monitored via Linux Security Modules
(LSM), but policies are written on eBPF. If any eBPF policy prohibits an operation,
that last one will eventually be rejected. In [12], the authors propose a security
architecture concept for orchestration and container systems, according to which a
single agent is used to detect attacks on the host. An agent is launched on the host, it
collects data using the eBPF probe and detects attacks based on blacklists. The article
[13] presents an approach to analyzing the stability of the container application runtime
environment, which is based on neural networks. The approach uses variational

4
autocoders (VAE) to train and form a container profile, and then a trained neural
network is used to register anomalies.
In addition, a number of articles on approaches and methods of rule generation were
reviewed. The works [14] and [15] seemed to be the most interesting. In [14], genetic
algorithms are used, which are fed a marked-up data set at the input, and IF-THEN rules
are obtained at the output. In [15] profiles are used, that is, logs of network devices are
analyzed and if the conditions for including it in the anomaly profile are met for the
session, a new rule is created.
As a result of the analysis, the following conclusions can be drawn: (1) eBPF
technology is just beginning to be actively used to collect security data that allows
detecting attacks and anomalies; (2) systems based on machine learning show good
results in the quality of detecting attacks and anomalies, however, the issue of
Adversarial machine learning and Explicable AI is acute in the field of application of
such systems, which makes it difficult to use them in protection systems; (3) methods
of automatic rule generation for detecting attacks and anomalies are actively used,
however, there are a few publications on the use of these methods for container systems;
(4) when detecting attacks and anomalies, particular approaches based on black and
white lists are used, however, no approach using the combination of black and white
lists to adapt the attack detection rules for a specific container was found.
3 Proposed approach
The proposed approach is based on the analysis of kernel logs from eBPF. Based on the
logs we generate rules that are used in anomaly and attack detection. In this section we
provide the description of input data for analysis and detection approaches.
To collect logs from inside the kernel, the eBPF technology and special collectors
that can work with it are used. The concept of operation of this technology is quite
simple. Inside the kernel, a sandbox space is created within which you can use your
code. Thanks to this, it is possible to significantly expand the capabilities of the kernel
without the need to rebuild it.
With the ability to execute user code inside the kernel, the programs using eBPF can
get information directly from the kernel or add additional functionality that will be
executed by the system much faster. eBPF is widely used in the field of information
security, for example, in programs that function as firewalls at the kernel level or event
analyzer systems. For example, the Tracee collector can obtain information about the
user, the actions taken by the user, the process ID, and the system call. However, in
many cases, such detailed information will be redundant. For example, system calls
such as syscall_enter and syscall_exit do not provide useful information in this case.
Therefore, the logs need to be further filtered.
The input of the proposed approach is information about the time and event type, the
name and event identifier, additional arguments that led to the appearance of this event,
as well as process IDs, user IDs, and information about the system response.
For anomaly and attack detection we proposed a rule-based approach. For anomaly
detection, a rules set is the whitelist that is generated from log with normal behavior –

5
and any log event that do not appear in whitelist we interpret as an anomaly. For attack
detection, a rules set is the blacklist that is generated from log that contains attack – and
any log event that appear in blacklist we interpret as an attack. In this paper, we evaluate
4 approaches for generating such rules. Approach 1 is based on the event type
occurrence. Here, the whitelist and the blacklist are the sets of unique event types that
appear in corresponding log. Approach 2 is based on occurrence of the event type with
the specific argument. And the whitelist and the blacklist are sets of unique combination
of event types and arguments. Approach 3 is based on the event type sequences. In
order not to iterate over all possible sequences, for sequence extraction from log we
specify the length L of the sequence and the number N. After, we N times select a
random event from log and L following events. Approach 4 is based on the minimum
and maximum occurrence of the specific event type over the time window T. For that,
we split log on subsets according to time window T, calculate the number of events of
specific event type, and get the minimum and maximum occurrence of this event over
all time windows.
The rules examples in form of JSON for these approaches are presented in Table 1.
The pipeline for generating rules for specific container looks like: (1) record the log of
normal behavior, (2) generate the whitelist on this log, (3) generate the blacklist on log
with an attack, (4) exclude from the blacklist the rules that appear in the whitelist.
Table 1. Rules examples for approaches 1-4
N
Based on
example of rules as elements of JSON
1
event type
openat, access, …
2
event type & argument
(openat, pathname: /etc/default/su), …
3
event type sequence
(openat, openat, execve), …
4
event type min/max oc-curence
(openat, 0, 42), …
4 Experiments
To evaluate the developed approach, a software prototype of the anomaly and attack
detection system based on the approach described in section 3 was developed. This
prototype was divided into several software components (Fig. 1): (1) information
security event data collection component; (2) data preprocessing and storage
component; (3) anomaly and attack detection component. The Ebpf-Tracer solution
from Aqua company was used as a collector [16]. This solution allows one to use only
the data collection system, excluding the built-in data analysis component.
To create a normal activity, a set of microservices with load simulation was used.
The test bench included containers with network interaction, for example: a container
that sends packets with unimportant data to the network and receives get-requests from
there, a web server that hosts a simple one-page website, as well as containers with an
operating system, a file manager and client-server application. A more detailed
description is given in Table 2.

6
Fig. 1. Framework design scheme
Table 2. Test bench for creating data sets
Container
Description
kali
Kali linux container. kali_container.sh
goping
Simple client-server application written in GoLang that ex-changes encryption keys
website
Nginx server that hosts a simple two-page website
website-database
Redis database
tomcat
Tomcat
cpp
Simple container which sends network packets and get-requests
Normal activity scenario includes the work of containers on the PC with the collector
in the background during a period of 10 minutes with the following additions: (1)
launching new functions in a target container; (2) executing user requests in a container.
Abnormal activity scenario (refers to malicious activity) is described in Table 3.
Table 3. Datasets description.
Container
Description
Dataset A
Dataset A contains an attack based imitated backdoor. Inside the source code of the attacked
container, a block, that downloads part of the malicious code from the Internet and starts it,
was inserted.
Dataset B
Dataset B exploited a configuration flaw of privileged flag, resulting in access to directories
that are accessible only to the root user. Based on this attack, information such as the contents
of the /etc/passwd file was retrieved and dumped.
Dataset C
To create an attack on dataset C, an exploit cve-5736 to gain root access to the home PC was
used. As a result, malicious libraries were installed on the home PC and some directories and
files were created.
As a result of the collection, we formed three datasets that contain more than 600
thousand records of activity on the test bench. Each of the datasets contains labeled

7
normal and abnormal events. Both simple system configuration errors and more
complex ones, such as CVE-5736 [17], were used to carry out attacks.
For experiment we use the schema presented in Fig. 2 to split out the dataset to train
sample (used for rules generation) and test sample (used for evaluation).
Fig. 2. Scheme of splitting the training/testing of the dataset for the experiment
At first step we have a labeled dataset, where log events are labeled in their
correspondence to the attack and normal behavior. At second step, we split our dataset
to normal and attack samples ordered by event time. At third step we split our dataset
for rules generation and evaluation, where: train_A is sample of first 80% of events
with normal behavior and that is used for generating the whitelist; train_B is sample of
first 80% of events with an attack and that is used for generating the blacklist; for the
blacklist we also exclude events that appears in the whitelist; test_A and test_B –
samples of last 20% of events with normal behavior and an attack, that merged and
used for test evaluation. The results of the whitelists evaluation in experiments for
datasets A, B, and C are presented in table 4, where we highlight (a) non-zero true
positives with blue, (b) non-zero false positives with red and (c) non-zero false
negatives with red. The results of blacklists evaluation in experiments for datasets A,
B, and C are outlined in table 5 with the same color scheme. As one can see in result
tables, the best efficiency was shown by the approach 2 (based on event type and
argument), as it can detect attacks without false positives. Approach 1 (based on event

8
type) was not able to detect anomalies and attacks. While approaches 3 (based on event
type sequences) and 4 (based on min/max event type occurrence) have some false
positives events.
Table 4. Results of experiment for evaluation of the whitelist efficiency in anomaly detection
Appr.
Dataset
Container
True
False
Positive
Negative
Positive
Negative
1
A
1
0
7504
0
4278
2
0
1765
0
40
B
1
0
7754
0
470
2
0
1856
0
20
C
1
0
105101
0
2789
2
A
1
275
7504
0
4003
2
15
1765
0
25
B
1
123
7754
0
347
2
8
1856
0
12
C
1
113
105101
0
2676
3
A
1
113
24
276
187
2
37
231
69
2
B
1
225
33
267
75
2
18
206
94
1
C
1
26
182
118
274
4
A
1
6
104
2
259
2
148
17
36
64
B
1
0
84
20
0
2
14
38
14
38
C
1
0
156
9
0
Table 5. Results of experiment for evaluation of the blacklist efficiency in attack detection
Appr.
Dataset
Container
True
False
Positive
Negative
Positive
Negative
1
A
1
0
7504
0
4278
2
0
1765
0
40
B
1
0
7754
0
470
2
0
1856
0
20
C
1
0
105101
0
2789
2
A
1
275
7504
0
4003
2
15
1765
0
25
B
1
123
7754
0
347
2
8
1856
0
12
C
1
113
105101
0
2676
3
A
1
13
173
127
287
2
4
293
7
35
B
1
75
258
42
225
2
9
298
2
10
C
1
104
299
1
300
4
A
1
25
12
94
240
2
43
12
41
169
B
1
0
34
70
0
2
1
1
51
51
C
1
0
0
165
0

9
5 Discussion
The experiments conducted have shown that the best quality of detecting attacks (i.e.,
highlighting the maximum number of events related to the attack and the minimum
number of events not related to the attack) was shown by approach number two.
However, it should be noted that other approaches have their advantages. So, the first
approach uses less data and thus works faster and requires less memory than the second.
The third approach, unlike the second, considers the sequence of security events and,
due to this, can more accurately identify complex attacks or anomalies disguised as
normal activity. The fourth approach, by analyzing the statistics of security events,
allows one to identify such attacks as, for example, illegal mining, in which the attack
is to increase the intensity of the container without an obvious violation of the security
policy.
In general, the approaches proposed in this article differ in the possibility of
automated generation of rules - black and white lists in a human-interpreted form
(unlike machine learning models) in which the operator can make changes. Due to this,
it is possible to combine operator knowledge and automatic rule generation. It is worth
noting the approach to adapt blacklists to container when the whitelist rules are
removed from the blacklist. This approach reduces the number of false positives.
The disadvantages of the proposed approaches include the fact that the use of black
and white lists can reduce the speed of the detection system compared to modern
machine learning methods, and the generation of a large (often excessive) number of
rules can complicate the operator's work with them. In general, it is advisable to test the
effectiveness of the proposed approaches on a larger number of data sets. Nevertheless,
experiments show that even on the basis of simple rules, it is possible to detect attacks
and anomalies quite effectively.
6 Conclusions
In this paper we proposed 4 approaches for generating white and blacklists for
anomalies and attack detection based on EBPF kernel logs. We conducted experiments
where we evaluated 4 approaches on 3 datasets with attacks. Experiments demonstrated
that based on simple rules it is possible to achieve a good attack and anomaly detection
efficiency (ability to detect attacks with a low false positive rate). The biggest
advantage, that in comparison with machine learning models, the rules can be
interpreted and adjusted by operator. However, the proposed approaches should also be
evaluated on bigger number of datasets that will be the goal of our further research. In
the future, we plan to improve the developed approaches, increase the accuracy and
reduce the speed of detecting anomalies and attacks. It is also planned to expand the
complex of algorithms for new areas [18] and by the algorithm for detecting sequential
attacks and by signatures. It is assumed that signatures will be created based on the
second approach.

10
References
1. Global application container market revenue 2018-2021, https://www.statista.com/sta-
tistics/752647/worldwide-application-container-market-revenue, last accessed
2022/06/04.
2. Branitskiy, A.A., Kotenko, I.V.: Analysis and classification of methods for network attack
detection. SPIIRAS Proceedings 2(45), 207–244 (2016)
3. Branitskiy, A., Kotenko, I.: Hybridization of computational intelligence methods for attack
detection in computer networks. Journal of Computational Science 23, pp.145–156 (2017)
4. Findlay, W.: System Introspection for Improving Performance, Reliability, and Security in
Distributed Systems. COMP5102 Final Project. 2019.
5. Liu, C., Cai, Z., Wang, B., Tang, Z., Liu, J.: A protocol-independent container network
observability analysis system based on eBPF. 2020 IEEE 26th International Conference on Par-
allel and Distributed Systems (ICPADS) 697-702 (2020)
6. Hong, J., Jeong, S., Yoo, J., Hong, J. W.: Design and Implementation of eBPF-based Vir-
tual TAP for Inter-VM Traffic Monitoring. 2018 14th International Conference on Network and
Service Management (CNSM) 402-407 (2018)
7. Miano, S., Risso, F., Bernal, M. V., Bertrone, M., Lu, Y.: A Framework for eBPF-Based
Network Functions in an Era of Microservices. IEEE Transactions on Network and Service Man-
agement 18 (1) 133-151 (2021)
8. Xhonneux, M., Bonaventure, O.: Flexible failure detection and fast reroute using eBPF
and SRv6. 2018 14th International Conference on Network and Service Management (CNSM)
408-413 (2018)
9. Parola, F., Risso, F., Miano, S.: Providing Telco-oriented Network Services with eBPF:
the Case for a 5G Mobile Gateway. 2021 IEEE 7th International Conference on Network Soft-
warization (NetSoft), 221-225 (2021)
10. Baidya, S., Chen, Y., Levorato, M.: eBPF-based content and computation-aware commu-
nication for real-time edge computing. IEEE INFOCOM 2018 - IEEE Conference on Computer
Communications Workshops (INFOCOM WKSHPS) 865-870 (2018)
11.Belair, M., Laniepce, S., & Menaud, J. M.: SNAPPY: programmable kernel-level policies
for containers. In Proceedings of the 36th Annual ACM Symposium on Applied Computing
1636-1645 (2021)
12.Gantikow, H., Reich, C., Knahl, M., Clarke, N. L.: Rule-based Security Monitoring of
Containerized Workloads. CLOSER 543-550 (2019)
13.Sharma, P., Porras, P., Cheung, S., Carpenter, J., Yegneswaran, V.: Scalable Microservice
Forensics and Stability Assessment Using Variational Autoencoders. arXiv preprint
arXiv:2104.13193 (2021)
14.Dehuri, S., Patnaik, S., Ghosh, A., Mallc, R.:Application of elitist multi-objective genetic
algorithm for classification rule generation. Applied Soft Computing 8 (1) 477-487 (2008)
15.Breier, J., Branišová, J.: A Dynamic Rule Creation Based Anomaly Detection Method for
Identifying Security Breaches in Log Records. Wireless Pers. Commun. 94, 497–511 (2017)
16.Aqua Tracee: Runtime eBPF threat detection engine, https://www.aq-
uasec.com/products/tracee/, last accessed 2022/06/04.
17.CVE-2020-5736, https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-
5736, last accessed 2022/06/04.
18.Berger, I. Rieke, R., Kolomeets, M., Chechulin, A., Kotenko, I.: Comparative study of
machine learning methods for in-vehicle intrusion detection. 4th Workshop on the Security of
Industrial Control Systems and Cyber-Physical Systems held in conjunction with 23rd European
Symposium on Research in Computer Security (ESORICS 2018), 85-101 (2019).
View publication stats