Conference PaperPDF Available

Vulnerabilities in Continuous Delivery Pipelines? A Case Study


Abstract and Figures

More and more companies are in the process of adopting modern continuous software development practices and approaches like continuous integration (CI), continuous delivery (CD), or DevOps. These approaches can support companies in order to increase the development speed, the frequency of product increments, and the time to market. To be able to get these advantages , especially the tooling and infrastructure need to be reliable and secure. In case CI/CD is compromised or even unavailable, all mentioned advantages are at stake. Potentially, this could also even hinder the forthcoming of the software development. Therefore, our goal was to identify which vulnerabilities are present in industry CD pipelines and how they can be detected. In this paper, we present our results of an industry case study which includes a qualitative survey of agile project teams regarding the awareness of security in CI/CD, the analysis and abstraction of two CD pipelines, and a threat analysis based on the deducted CD pipeline to identify vulnerabilities. In this case study, we found that the team members that work with the CD pipeline in different roles do not have a strong security background but are aware of security attributes in general. Furthermore, two CD pipelines from industry projects were analyzed using the STRIDE threat analysis approach. In total, we identified 22 vulnerabilities that have been confirmed by the project teams.
Content may be subject to copyright.
Vulnerabilities in Continuous Delivery Pipelines?
A Case Study
Christina Paule
Novatec Consulting GmbH
Leinfelden-Echterdingen, Germany
Thomas F. D¨
ullmann, and Andr´
e van Hoorn
Institute of Software Technology, University of Stuttgart
Stuttgart, Germany
Abstract—More and more companies are in the process of
adopting modern continuous software development practices and
approaches like continuous integration (CI), continuous delivery
(CD), or DevOps. These approaches can support companies in
order to increase the development speed, the frequency of product
increments, and the time to market. To be able to get these advan-
tages, especially the tooling and infrastructure need to be reliable
and secure. In case CI/CD is compromised or even unavailable,
all mentioned advantages are at stake. Potentially, this could
also even hinder the forthcoming of the software development.
Therefore, our goal was to identify which vulnerabilities are
present in industry CD pipelines and how they can be detected. In
this paper, we present our results of an industry case study which
includes a qualitative survey of agile project teams regarding the
awareness of security in CI/CD, the analysis and abstraction of
two CD pipelines, and a threat analysis based on the deducted
CD pipeline to identify vulnerabilities. In this case study, we
found that the team members that work with the CD pipeline
in different roles do not have a strong security background but
are aware of security attributes in general. Furthermore, two CD
pipelines from industry projects were analyzed using the STRIDE
threat analysis approach. In total, we identified 22 vulnerabilities
that have been confirmed by the project teams.
By applying continuous delivery (CD), companies are able
to deploy application changes to the customer rapidly and reli-
ably from the software repository to the customer’s hands [1].
Based on the results of a study by Hurwitz & Associates [2],
one may assert with confidence that the CD process is an
essential component in software development projects.
However, Paulus [3] has shown that, as long as there are
no security issues, companies have little interest in application
vulnerabilities which could potentially result in a lack of secu-
rity. Also, they report that insecure applications can severely
damage the image of companies. Trend Micro [4] observed
that servers are often misconfigured and insecure, and that also
CD applications like Jenkins are becoming targets of attacks
(e.g., cryptojacking). According to Gruhn et al. [5], CI/CD
tools have more vulnerabilities than communicated through
vulnerability databases and communities (e.g., OWASP).
This strengthens our claim that industry CD systems need
more attention in terms of security. Therefore, the following
research question arises:
This work is partly funded by the Baden-W¨
urttemberg Stiftung (ORCAS
Which vulnerabilities are present in industry CD
pipelines and how can they be detected?
To investigate this question we conducted a case study [6].
With this study, we wanted to get insights into industry CD
pipelines, the related project teams, and vulnerabilities in
practice. In our case study consisting of two project teams
working with CD pipelines in industry, we did the following:
(i) Survey to identify roles team members have regarding
the CD pipelines in their projects, their experience with
security aspects, and their opinion on the most important
security attributes. The intention of the survey was to
focus on the most relevant security attributes for the fol-
lowing steps. Even though the survey did not investigate
the research question directly, it provided insights into
the context CD pipelines are operated in.
(ii) Analysis of two industry CD pipelines focusing on the
structure and the overall process. As well as the deduction
of an abstracted CD pipeline based on the CD pipelines
of the two projects supported by personal interviews with
experts for a detailed understanding.
(iii) Execution of a STRIDE threat analysis [7] focusing on
the identified most important security attributes (con-
fidentiality, integrity, availability) and mapping of the
identified threats based on NIST and OWASP project
methodologies for risk severities.
(iv) Manual vulnerability assessment based on the results of
the STRIDE threat analysis.
(v) Identification of tools suitable for detection and mitiga-
tion of the found threats.
(vi) Validation of results with the project teams.
In total, 22 security vulnerabilities could be identified and
confirmed. Furthermore, we could identify tools that poten-
tially could either detect or mitigate such vulnerabilities.
Section II presents foundations on CD pipelines, security,
and the STRIDE approach. The different parts of the case
study mentioned above are presented, i.e., survey (Section III),
CD pipeline analysis and abstraction (Section IV), threat anal-
ysis with STRIDE (Section V), vulnerability assessment (Sec-
tion VI), tool identification (Section VII), and verification
of the results (Section VIII). The findings are discussed in
Section IX and related work is presented in Section X. The
paper is concluded in Section XI. The thesis by Paule [8] is
the foundation of this paper and includes detailed resources.
A. CD pipelines & CD servers
According to Humble et al. [1], a deployment pipeline
is “an automated manifestation of your process for getting
software from version control into the hands of your users”.
In contrast to CD pipelines, CI pipelines do not include the
deployment of software, but only the automated preparation
of the artifacts. Even though this process does not have to be
automated, it is common to use a CD server (e.g., Jenkins)
to define the process and execute the steps automatically. So-
called stages allow a logical grouping of more fine-grained
steps. For example, all the steps that are required for testing
the artifact could be grouped in the test stage, which could
include the setup of a test environment, the execution of the
actual tests, and the tear-down of the test environment. If a
stage finished successfully, the next stage is triggered. In case
a stage failed, the whole process fails.
B. Security
The main objectives of software security are the attributes of
information security (confidentiality, integrity, and availability;
also known as the CIA triad) [3], [9] that need to be protected.
There are three additional attributes in literature, namely
authentication, authorization [10], and non-repudiation [11].
The NIST defines a vulnerability as a “weakness in an
information system [..] that could be exploited or triggered by
a threat source” [12]. Vulnerabilities (e.g., programming mis-
takes or software errors) can occur anywhere in software [10].
According to Farn et al. [13], vulnerabilities are connected
with threats, assets, values, and risks.
C. STRIDE threat modeling and analysis
Threat modeling is an approach to detect potential vulnera-
bilities, further risks, and threats in an application as early
as possible [14]. Threat modeling extracts the components
of a system and considers the possible entry points from an
attacker’s point of view [14].
Out of various threat modeling approaches, the STRIDE
threat modeling method is the most widely used method [15].
As reported by OWASP, threat modeling is a process that
can be executed in three steps [14]. In the first step, the
application has to be understood with all its components and
connections. The second step is to identify the threats and
vulnerabilities in the application. Threats can be assigned to
six different categories according to STRIDE, namely spoof-
ing, tampering, repudiation, information disclosure, denial of
service, and elevation of privilege [7]. In the third step, the
results are assessed. In this research, this assessment is done
with the OWASP risk rating methodology [14], the calculator
of Makovi´
c [16], and the definition of the NIST [17]. As a
result, the overall risk severity is the product of the likelihood
and an impact level. For example, a higher likelihood factor
is given if the attacker needs no resource access rights to
exploit the vulnerability. If the attacker needs full access rights
and detailed system knowledge then a low likelihood factor
exists because it is unlikely that an attacker can exploit the
Low Medium High
Low None Low Medium
Likelihood Medium Low Medium High
High Medium High Critical
TABLE I: Overall risk severity matrix based on OWASP [14]
vulnerability. The damage to the company and the amount
of data which is gained through the exploitation of the
vulnerability defines the impact factor. Table I shows that the
overall risk severity can have a level between none (no risk)
and critical [14].
Mack et al. [18] recommends a qualitative research method
to understand the present problem or the context of a topic.
In order to better understand the context and narrow down the
security attributes to look at, we conducted a survey. It was
conducted in project teams working with CD pipelines in a
DevOps context. The survey was set up as an online survey
that was sent via e-mail to approximately 100 employees
of a selected company which includes the investigated CD
pipeline project teams of the case study. The participation was
voluntary and was possible at any time. In total, there were 19
completed surveys containing nine questions about the roles
of the participants regarding CD pipelines, the participants’
opinion on the security of CD pipelines and the most prevalent
problems, and their background on security and tooling for CD
pipelines. The survey details and results can be found in the
thesis by Paule [8].
Question 1:In your opinion which security objectives should
be pursued to CD pipelines? Please do not focus on a specific
used pipeline. Think in general.
We manually aggregated the objectives into the following
categories, presented in no specific order: “Securing
source code, logs and artifacts”, “Build steps should not be
manipulated”, “No pipeline modification through unauthorized
persons”, “No triggering of the pipeline through unauthorized
persons”, “Securing environment properties such as login
data”, “Reduce human errors (storing password)”, “Securing
credentials (encrypt all sensitive data”, “No vulnerabilities
in dependencies”, “Secure transmission over HTTPS or
SSH”, “Use 4-eye-principle”, “Check access rights of the
components of the CD pipeline”.
Question 2:In your opinion which security attribute is the
most important one in respect to CD pipelines (artifacts,
files, scripts, connections, ...)? Order the following security
attributes (confidentiality, integrity, availability, authorization,
authentication, non-repudiation) according to their impor-
tance. The attribute on top is the most important for you.
Out of the six ranked attributes, the top three are used for
scoping the threat analysis. The results of this question help
to delimit the subject because it reflects the interest and the
necessity from the viewpoint of the employees. In the view
of the employees, the three top-ranked attributes are integrity,
availability, and confidentiality.
Question 3:In your opinion what are possible attack scenarios
for the pipeline you use? Against which attacks would you like
to protect your pipeline?
Based on the results from question 2, we narrowed down the
classification to the three security attributes of confidentiality
(C), integrity (I), and availability (A). All answers that could
not be assigned to any of these, were labeled with ”N/A”.
The answers and the results of the classification can be
found in Table II on the following page. Only typographical
errors were corrected in the listed answers.
Seven answers could be assigned to attack scenarios
related to confidentiality, 15 answers to integrity, and five
to availability. There were 27 answers in total that could, in
parts, be assigned to multiple security attributes.
Question 4:Which security objectives are pursued in your
project in respect to CD pipelines? Which are implemented?
The security objectives the participants mentioned are: re-
quiring authentication and authorization, securing credentials
and hiding critical data, reviewing the process, no information
should be included in the source code of applications, access
control, keeping the pipeline components and software up to
It can be seen that in the industrial projects, authentication
and authorization are implemented. In addition, securing
sensitive data and access rights also contribute to secure
the pipeline. If a project does not pursue security goals, it
cannot be guaranteed to the customer that the software will
be deployed securely.
Question 5:How many years of experience in software
development do you approximately have?
The results show that the participants have 10.32 years of
development experience on average.
Question 6:Which tools do you know and/or use? Response
options: (DevOps tools) Jenkins; Kubernetes; TeamCity; Spin-
naker; Travis; GoCD; Concourse CI; JFrog Artifactory; (Static
analysis tools) PMD; Checkstyle; FindBugs; FindBugs Se-
curity; (Security tools) OWASP ZAP; BDD Security; JFrog
Xray; Security Monkey; Black Duck; Snyk
The answers, which can be found in the thesis by
Paule [8], show that the CI server Jenkins is known by all
15 participants. The know-how in security testing tools is not
as well-known as static analysis tools. In summary, it can be
said that the competence of the company lies in the CI tool
Jenkins and in static analysis tools.
Question 7:In which role do you interact with your CD
pipeline? Response options: user (committing code to the
project, usage of the CD pipeline); installation and operation
of the pipeline; configuration of the pipeline; other.
The results show that 58% of the 19 participants interact
with all facets of a pipeline (using, installing, configuring,
and operating the pipeline). Only one person does not come
into contact with the pipeline because he is scrum master and
manages the team. In summary, it can be said that 95% of
the participants come into frequent contact with the pipeline.
Question 8:In your opinion how important is the topic
security vulnerabilities in CD pipelines? Response options:
1; 2; 3; 4; 5 (1: not important, 5: very important)
With a median of four, the results show that the employees
realize that it is necessary to do research on this topic.
Question 9:How often do you deal with security in your devel-
opment process? Response options: Never; only occasionally;
quite often; most of the time; no answer
The answers show that most employees only occasionally
deal with security topics during their development process.
Five employees deal with the security context quite often and
one is never concerned with it.
Question 10 (Only for the participants in the teams of the
analyzed CD pipelines): In the next step think about the
security of the <project name>CD pipeline. In your opinion
how secure is this pipeline? Response options: 1; 2; 3; 4; 5
(1: Many threatening vulnerabilities, 5: No vulnerabilities)
For both teams, the median value of the answers was 3.
This shows, that there is neither the expectation that no
vulnerabilities exist at all nor that there are many.
In the following, we follow established steps for designing
and planning a software engineering case study [6].
The rationale for and the objective of the study are
that the case study investigates the security level of the CD
pipelines of a selected software consulting company referred to
as Alpha and Beta in this paper, while the teams working with
these CD pipelines are organizationally separated and the CD
pipelines are related to different customers. The CD pipeline
details were identified by oral interviews with the developers.
The goal of the case study is to gain knowledge about how
team members assess the security level of the CD pipelines,
which security objectives are implemented in the CD pipelines,
and which overall risk severity the industrial CD pipelines
have. As methods of data collection, a qualitative survey
and interviews are used. The additional question of the survey
is used to find out how the team members assess the security
level. They have the response options 1; 2; 3; 4; 5 (1: means
CD pipeline is insecure, 5: means CD pipeline is secure
(pipeline has no vulnerabilities)). The vulnerabilities of each
CD pipeline were assessed either manually or with tools.
The methods of data analysis comprise collecting vulner-
abilities and assess their likelihood and impact levels which
lead to an overall risk severity.
Answer C I A N/A
Getting the source code as the intellectual property and publish and sell it under a different name. X
Stolen passwords / ssh keys X
Extraction of credentials from the pipeline configuration in order to use them to get access to sensitive systems connected to the pipeline. X
If the CD pipeline can push a software package directly to production and the hacker can also change software in the VCS, it may be possible to execute
bad code in production system. This should not be possible.
Unauthorized access: a non-malicious user can misconfigure a component of the pipeline X
Changes on CI/CD Server configuration X
Integrate ”bad code” like backdoors or special behaviours (like: don’t charge my account for shop orders) into the application going productive later. X
Try a manipulation with the input data a job requires when it is started manually X
Source code manipulation 7→ bad for integrity X
Malicious code alteration, either by commit directly to our version control system, or by modifying the final build artifact which is deployed to production. X
Modifying the pipeline configuration to make it perform additional operations, which normally would not be permitted. X
Possible attacks scenarios are running the pipeline to release an manipulated artifact. The code to create the artifact was manipulated by the attacker
beforehand. The only obstacle left to overcome to release a malicious artifact is the CD pipeline. Another scenario is that the attacker introduces
weaknesses over the pipeline. Maybe the attack has no access to the code but can introduce weak password or vulnerable dependencies over the CD
pipeline parameters.
Denial of Service X
Any availability hazard: an unavailable pipeline would prevent us from delivering software X
DoS attacks 7→ bad for availability X
Changes are being made by individuals that break the pipeline, Updates of the environment where the pipeline runs destroys the pipeline X X
Amok user deletes all Jenkins jobs X X
Grapping configuration files to directly access databases or backends and perform data manipulation there. X X
Anybody can change/access the pipeline X X
Manipulate files/artifacts, unauthorized use of resources X X
Plant malicious code, gain access to build hosts, retrieve credentials X X
Pipeline uses old libraries/applications with vulnerabilities X
Zero day exploits against the server running the CI/CD software X
Bruteforce use/passwords X
Trying to run scripts to gain root access to the filesystem where the CI/CD server runs or the agent X
All of the above mentioned apply for our toolchain, I think. X
I would like to protect our pipeline against all attacks. In practice, there is always a trusted circle of developers/operators which we cannot really protect
Summary 7 15 5 6
TABLE II: Question 3 answer categorization
(C = Confidentiality, I = Integrity, A = Availability, N/A = not assignable to any of C, I, or A)
A. Industry CD pipelines
We inspected two CD pipelines (Alpha and Beta) that are
actively used in industry projects.
1) CD pipeline Alpha: Figure 1 depicts the rough structure
of CD pipeline Alpha. At the time of the investigation, the
components of the CD pipeline were Bitbucket 4.14.2, Jenkins
2.89.3, Amazon Web Service (AWS), JFrog Artifactory 5.8.3,
Rundeck 2.10.2-1, HockeyApp store, and Puppet. The Jenkins
master can only be accessed from the company’s intranet. In
an external Cloud Foundry Bitbucket, JFrog Artifactory, and
a Rundeck server are installed. Puppet is used to set up the
deployment environment.
The first step of pipeline Alpha is that a developer commits
code changes to Bitbucket (1). After a developer has reviewed
that the code has no issues, it can be committed. Jenkins
triggers a new instance of the CD pipeline (2, 3). The 20
available EC2 container instances in the AWS cloud are
used to build the project. The build phase is successfully
completed when all tests (e.g., JUnit) passed. The built artifact
is stored in the JFrog Artifactory (6). For every testing type
(automatic acceptance test, load testing), a separate pipeline is
triggered. The deployment phase is triggered manually. Jenkins
notifies the Rundeck server which deployment environment
(e.g., production) it should use. The aim of the deployment
step is to deploy the desired artifact to Cloud Foundry or, if
it is an Android app, to the HockeyApp store (7, 8, 9).
The team of CD pipeline Alpha has specified security ob-
jectives that comprise performing regular updates, restricting
access of users, usage of authentication mechanisms, no or
restricted access from outside to the CD pipeline infrastructure.
Fig. 1: Structure of the investigated CD pipeline Alpha
2) CD pipeline Beta: Figure 2 depicts the rough structure
of the investigated CD pipeline Beta. At the time of the
investigation, the components of the CD pipeline are GitLab
Community 10.1.0, Jenkins 2.32.2, Sonatype Nexus Repos-
itory OSS 2.14.3, IBM WebSphere, SFTP server,
and HashiCorp Consul 0.75.5. All components of the CD
pipeline can only be reached over the customer intranet. The
results of each CD pipeline stage are stored in a key-value
database which is provided by HashiCorp Consul. The first
step of the CD pipeline is that the developer commits her
Fig. 2: Structure of the investigated CD pipeline Beta
changes to GitLab (1). Jenkins triggers a new instance of
this CD pipeline (2). After that, the application is built on
two Jenkins nodes (3). The build stage ends successfully if
all tests have been passed (e.g., SonarQube). The artifacts
are uploaded in the Sonatype Nexus Repository OSS (4).
For testing, the application has to be installed on an IBM
WebSphere application server (5). The production stage is
triggered manually. If this is done, the artifacts are deployed
on an SFTP server (6).
The team of CD pipeline Beta has not explicitly specified
security objectives but nonetheless implemented security mea-
sures like regular updates for components and plugins of the
CD pipeline, limited access to CD pipeline components (i.e.
intranet only), and restricted access and authentication with
limited permissions.
B. Abstracted CD pipeline
A generalized version of the investigated CD pipelines
Alpha and Beta is illustrated in Figure 3. The first pipeline’s
process step is that a developer pushes source code changes
to a source code repository (1). This event notifies the CI/CD
server (e.g., using a webhook) (2). The CI/CD server then
triggers the first stage (“build”) in the CD pipeline process
(3). The “build” stage comprises checking out the source
code from the source code repository (4), retrieving third-party
libraries from the library store (e.g., Maven Central1) (5) for
building the application and storing the resulting artifacts in an
artifact repository (6). If the build stage was successful, the test
stage is triggered (7), which executes tests on the artifacts (e.g.,
JUnit). Afterwards, the deploy stage is triggered (8), which
retrieves the latest successfully built artifact from the artifact
repository (9), and deploys it to a deployment server (10).
To detect vulnerabilities in the abstracted CD pipeline, the
aforementioned threat modeling approach STRIDE [19] is
applied. Because of the survey results and the three main
attributes (CIA triad) from literature, we focused the threat
analysis on the threats in the categories tampering (T), infor-
mation disclosure (I) and denial of service (D) of the STRIDE
Fig. 3: Generalized CD pipeline
Overall risk severity # Total
None 3
Low 3
Medium 1
Between None and Medium 1
Between Low and Medium 1
Between Low and Potentially High 1
Between Medium and Potentially High 1
TABLE III: Pipeline Alpha: Potential risk severity
method. These are investigated for each component and data
flow of this CD pipeline. Tampering hurts the attribute integrity
and means that the attacker modifies (sensible) data for which
she has no authorization. Confidentiality is hurt by information
disclosure which means that the software/application displays
information to unauthorized persons/attackers. The third at-
tribute availability is hurt by a denial of service attack which
means that the attacker slows down the application, causes it
to crash, or fills the memory.
In order to consider all possible threats, the cards of the
Elevation of Privilege Threat Modeling Card Game [19],
which are issued by Microsoft, are used as a basis.
For each identified component from the generalized CD
pipeline, the aspects of tampering, information disclosure, and
denial of service were considered to find possible vulnerabil-
ities. This resulted in 21 tables each referring to a component
and one aspect of T, I, and D with each table containing one
or multiple possible threats, their effect, and the corresponding
vulnerability. The STRIDE tables can be found in the thesis
by Paule [8].
These results were mapped to the CD pipelines Alpha and
Beta which resulted in 11 vulnerabilities for each of the two
CD pipelines with different risks (see Table III and Table IV).
Overall risk severity # Total
None 2
Low 1
Medium 3
Between None and Medium 2
Between Low and Medium 1
Between Low and Potentially High 1
Between Medium and Potentially High 1
TABLE IV: Pipeline Beta: Potential risk severity
Eventually, we identified the following set of general vul-
nerabilities that were the result of manual analysis based on
the results of the STRIDE analysis:
(1) Employees that may harm the security by introducing
problems either intentionally or unintentionally, (2) unen-
crypted connections between components of the CD pipeline,
(3) insecure environments of the CD pipeline components
in terms of their deployment, (4) none or few access re-
strictions, (5) the use of vulnerable versions of the CD
pipeline components, (6) vulnerable CD pipeline configu-
rations, (7) vulnerable code commits, CD pipeline scripts,
Docker images/containers, artifacts, and (8) no review of
changes to the CD pipeline.
Based on the identified vulnerabilities, we looked for tools
that could help to either detect or mitigate them. We found
tools that supposedly are able to satisfy these requirements
either for single or even multiple of the CD pipeline stages.
Of course, the tools to apply highly depend on the technolo-
gies and steps used in a CD pipeline. The list of identified tools
can be found in the thesis by Paule [8].
After having identified potential vulnerabilities, the project
teams were notified of the results. The teams operating the
CD pipelines were able to confirm all found vulnerabilities
and plan to resolve as many of them as possible. According
to them, there might be vulnerabilities that need further
discussion with the customer to make all needed changes.
Some critical vulnerabilities were fixed immediately.
The results show that only the project team working with
CD pipeline Alpha has declared explicit security objectives for
their CD pipeline. The project team working with CD pipeline
Beta indirectly pursues security objectives but their main aim
is automation and fast velocity of the delivery process. All
in all, both teams try to fulfill their objectives but the project
team working with CD pipeline Alpha does this with more
awareness. Both teams cannot fulfill the security objectives
completely, also because they are dependent on the network,
hardware, and cloud infrastructure of the customer. Often, they
would like to do more in this area but are limited in budget
and time for it or components are managed by a third party.
The investigation of the CD pipelines show that vulnerabilities
exist in both CD pipelines which pose potentially high threats.
In the end, the results were presented to the project teams
which confirmed the found vulnerabilities. According to the
teams, several of these vulnerabilities could be fixed by now.
This shows, that the awareness of vulnerabilities can solve
at least some of the potential threats. Though, some may be
harder to resolve as third parties need to be involved (e.g.,
customer) or can hardly be changed within a reasonable time.
Threats to validity
This section discusses threats to validity for our case study.
Internal validity. The case study was conducted in a single
company, which might pose a threat to validity. However, the
participants of the survey came from different organizational
units (i.e., teams). The CD pipelines that were investigated
are handled by different teams from the same company and
belonged to different customers. It needs to be emphasized
that it is difficult to get insights into real-world practices and
CD pipelines in the industry, whereas our exploratory work
can provide a first glimpse.
Construct validity. The details of the CD pipelines were
mainly deducted from information obtained in interviews. A
possible threat to validity could be that this information was
not completely accurate or not transferred correctly. Even
though the STRIDE approach helps to structure the steps of
threat analysis, aspects might have been overlooked.
External validity. As we only had a small sample size in the
survey (19) as well as for the CD pipelines (two), the results
are not generalizable to all industry CD pipelines. However,
this exploratory work gives first insights into possible prob-
lems in practice that could be further investigated.
A CD pipeline consists of tools that are in most cases web-
based applications or services (e.g., Jenkins). Approaches were
developed to detect vulnerabilities in such applications [20]–
[24]. The OWASP Top 10 list 2017 [14] postulates which
vulnerabilities occur in most web applications. In comparison
to our work, these tools and approaches do not specifically
focus on the CD pipeline context.
DevSecOps (Secure DevOps) [25], [26] tools help to auto-
mate security in the CD process. Rahman and Williams [25],
Bird [27], Schneider [28], Storms [29], and Kuusela [30]
presented tools and methods which detect vulnerabilities in
applications, partly also inside a CD pipeline. In addition to
the mentioned papers, Staˇ
c [31] addressed the addition of
security to agile development processes. Despite this work
addresses vulnerabilities in the context of CD pipelines, they
focus on artifacts that are built by the pipeline, while our work
focuses on the pipeline itself in an industry context.
Several approaches have been developed in the form of
tactics to increase the security level of CD pipelines [32]–
[34]. In a proof of concept, they mention that they want to
eliminate one class of attack vectors with visualization in
public CI services. Abomhara et al. [35] investigated threats in
a telehealth system and Lipke [36] used the STRIDE method
to investigate threats for a CD pipeline based on Docker. In
contrast to our work, this work did not apply their findings to
real-world industry cases.
By conducting a survey and inspecting two CD pipelines
from industry using STRIDE, we found that the security of
CD pipelines does not have high priority in development
teams. Additionally, though most of the team members have
access to the CD pipeline configuration, the lack of security
awareness and background in the teams pose a risk to this
increasingly business-critical development tool, both in terms
of infrastructure, as well as in terms of application.
The goal was to find out how secure specific CD pipelines
in a company are. In order to answer this, a case study was
conducted. The results of the case study show that both inves-
tigated CD pipelines have included vulnerabilities which have
an overall risk severity between medium and potentially high.
Some vulnerabilities in these CD pipelines occur because the
project teams are dependent on the customers’ infrastructure.
For both investigated CD pipelines the project teams have to
trust in the security of the foreign infrastructure and their own
team members. The research shows which vulnerabilities are
present in CD pipelines and how they can be detected.
Our findings could improve the awareness of security issues
in the teams, but as the circumstances (e.g., project focus,
budget, influence on third-party infrastructure) in a business
context may be complicated, it might not always be easy to
overcome all of the said issues. In order to address security
issues in CD pipelines, we aim to investigate ways (e.g.,
failure injection) to identify dependability and security issues
as described in D¨
ullmann et al. [37].
Future work
Since only two industry CD pipelines were investigated,
further CD pipelines could be analyzed to validate our find-
ings and to identify additional vulnerabilities in existing CD
pipelines. A more extensive case study with a bigger sample
size from different companies and multiple projects could
show whether the CD pipelines also show similar vulner-
abilities and similar patterns in team member background
towards security. Another opportunity may be to derive a CD
pipeline metamodel (e.g., based on the presented abstracted
CD pipeline) to better describe security properties on an
abstract level.
[1] J. Humble and D. Farley, Continuous Delivery: Reliable Software
Releases through Build, Test, and Deployment Automation, ser. A Martin
Fowler signature book. Addison-Wesley, 2011.
[2] H. Gilmore, “DevOps survey results: Why
enterprises are embracing continuous delivery,” 2017.
[Online]. Available:
[3] S. Paulus, Basiswissen sichere Software. dpunkt. Verlag, 2012.
[4] “Tesla and Jenkins servers fall victim to
cryptominers,” 2018. [Online]. Available: https://www. threats/
tesla-and- jenkins-servers-fall-victim-to-cryptominers
[5] V. Gruhn, C. Hannebauer, and C. John, “Security of public continuous
integration services,” in Proc. OpenSym, 2013.
[6] P. Runeson, M. Host, A. Rainer, and B. Regnell, Case Study Research in
Software Engineering: Guidelines and Examples. John Wiley & Sons,
[7] A. Shostack, Threat Modeling: Designing for Security. John Wiley &
Sons, 2014.
[8] C. Paule, “Securing DevOps – Detection of vulnerabilities in CD
pipelines,” Master’s thesis, University of Stuttgart, Germany, 2018.
[9] ISO/IEC 27000: Information technology–Security techniques–
Information security management systems–Overview and vocabulary,
5th ed., 2018.
[10] A. K. Talukder and M. Chaitanya, Architecting Secure Software Systems.
CRC Press, 2008.
[11] L. Pesante, “Introduction to information security,” in CERT, Software
Engineering Institute, 2008.
[12] Joint Task Force Transformation Initiative, “Security and privacy con-
trols for federal information systems and organizations,” in NIST Special
Publication, vol. 800.53, 2013.
[13] K.-J. Farn, S.-K. Lin, and A. R.-W. Fung, “A study on information se-
curity management system evaluation—assets, threat and vulnerability,”
in Computer Standards & Interfaces, vol. 26.6, 2004.
[14] “Open web application security project (OWASP).” [Online]. Available:
[15] S. Hussain, A. Kamal, S. Ahmad, G. Rasool, and S. Iqbal, “Threat
modelling methodologies: A survey,” Sci. Int. (Lahore), vol. 26, no. 4,
[16] I. Markovi´
c, “OWASP risk assessment calculator.” [Online]. Available:
[17] G. Stoneburner, A. Y. Goguen, and A. Feringa, “Risk management guide
for information technology systems,” in Special publication 800-30.,
[18] N. Mack and C. Woodsong, Qualitative Research Methods: A Data
Collector’s Field Guide. FLI and USAID, 2005.
[19] A. Shostack, “Elevation of Privilege (EoP) Threat Modeling Card
Game,” 2014. [Online]. Available:
[20] G. Deepa and P. S. Thilagam, “Securing web applications from injection
and logic vulnerabilities: Approaches and challenges,” in Information
and Software Technology, vol. 74, 2016.
[21] Y.-W. Huang, F. Yu, C. Hang, C.-H. Tsai, D.-T. Lee, and S.-Y. Kuo, “Se-
curing web application code by static analysis and runtime protection,”
in Proc. WWW, 2004.
[22] Y.-W. Huang, C.-H. Tsai, T.-P. Lin, S.-K. Huang, D. T. Lee, and S.-Y.
Kuo, “A testing framework for web application security assessment,” in
Computer Networks, vol. 48.5, 2005.
[23] Y. Kosuga, K. Kono, M. Hanaoka, M. Hishiyama, and Y. Takahama,
“Sania: Syntactic and semantic analysis for automated testing against
SQL injection,” in Proc. ACSAC, 2007.
[24] T. Lee, G. Won, S. Cho, N. Park, and D. Won, “Detection and mitigation
of web application vulnerabilities based on security testing,” in Proc.
NPC), 2012.
[25] A. A. Ur Rahman and L. Williams, “Software security in DevOps:
Synthesizing practitioners’ perceptions and practices,” in Proc. CSED.
ACM Press, 2016.
[26] V. Mohan and L. B. Othmane, “SecDevOps: Is it a marketing
buzzword?-mapping research on security in DevOps,” in Proc. ARES,
[27] J. Bird, DevOpsSec: Securing software through continuous delivery.
O’Reilly Media, 2016.
[28] C. Schneider, “Security devops: Free pentester’s time to focus on high-
hanging fruits,” in HackPra Allstars, 2015.
[29] A. Storms, “How security can be the next force multiplier in DevOps,
in RSA Conference, 2015.
[30] J. Kuusela, “Security testing in continuous integration processes,” Mas-
ter’s thesis, Aalto University, 2017.
[31] D. Staˇ
c, “Security DevOps: Konzeption einer Umgebung zur In-
tegration von Sicherheitstests in agile Softwareentwicklungsprozesse,
Master’s thesis, Reutlingen University, 2017.
[32] L. Bass, R. Holz, P. Rimba, A. B. Tran, and L. Zhu, “Securing a
deployment pipeline,” in Proc. RELENG, 2015.
[33] F. Ullah, A. J. Raft, M. Shahin, M. Zahedi, and M. Ali Babar,
“Security support in continuous deployment pipeline,” in Proc. ENASE.
SciTePress, 2017.
[34] P. Rimba, L. Zhu, L. Bass, I. Kuz, and S. Reeves, “Composing patterns
to construct secure systems,” in Proc. EDCC, 2015.
[35] M. Abomhara, M. Gerdes, and G. M. Køien, “A STRIDE-based threat
model for telehealth systems,” Proc. NISK, vol. 8, no. 1, 2015.
[36] S. Lipke, “Building a secure software supply chain,” Master’s thesis,
Stuttgart Media University, 2017.
[37] T. F. D¨
ullmann, C. Paule, and A. van Hoorn, “Exploiting DevOps
Practices for Dependable and Secure Continuous Delivery Pipelines,
in Proc. RCoSE@ICSE. ACM, 2018.
... We notice that many providers ignore recommendations about CI/CD server isolation. Additionally, Paul et. al in [9] revealed that developers who work with the CD pipeline are only familiar with the general security attributes and lack in-depth security knowledge. As described in [6], failures in a CI/CD pipeline are immediately visible and could halt the advancement of the affected release to the later stages of the cycle. ...
Full-text available
Over the past decade, software development has evolved from a rigid, linear process to a highly automated and flexible one, thanks to the emergence of continuous integration and delivery environments. Nowadays, more and more development teams rely on such environments to build their complex projects, as the advantages they offer are numerous. On the security side, however, most environments seem to focus on authentication, neglecting other critical aspects, such as the integrity of the source code and the compiled binaries. To ensure the soundness of a software project, its source code must be secured from malicious modifications. Yet, no method can accurately verify that the integrity of a project’s source code has not been breached. This paper presents P2ISE, a novel integrity-preserving tool that provides strong security assertions for developers against attackers. At the heart of P2ISE lies the TPM trusted computing technology, which is leveraged to ensure integrity preservation. We have implemented the P2ISE and quantitatively assessed its performance and efficiency.
... However, a plethora of attacks have undermined the integrity of entire applications by compromising one or more operations of the software supply chain [5,8,11,13,14,18,24,26,[30][31][32]. To protect this vital ecosystem, our design provides integrity for the supply chain by employing trusted execution environments (TEEs) (e.g., [2,20,21,25]) for their hardware-enforced code integrity and authentication features. ...
Full-text available
As cloud providers push multi-tenancy to new levels to meet growing scalability demands, ensuring that externally developed untrusted microservices will preserve tenant isolation has become a high priority. Developers, in turn, lack a means for expressing and automatically enforcing high-level application security requirements at deployment time. In this paper, we observe that orchestration systems are ideally situated between developers and the cloud provider to address these issues. We propose a security policy framework that enables security-oriented orchestration of microservices by capturing and auditing code properties that are incorporated into microservice code throughout the software supply chain. Orchestrators can leverage these properties to deploy microservices on a node that matches both the developer's and cloud provider's security policy and their resource requirements. We demonstrate our approach with a proof-of-concept based on the Private Data Objects [1] confidential smart contract framework, deploying code only after checking its provenance.
Full-text available
The widespread dependency on open-source software makes it a fruitful target for malicious actors, as demonstrated by recurring attacks. The complexity of today's open-source supply chains results in a significant attack surface, giving attackers numerous opportunities to reach the goal of injecting malicious code into open-source artifacts that is then downloaded and executed by victims. This work proposes a general taxonomy for attacks on open-source supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution. Taking the form of an attack tree, it covers 107 unique vectors, linked to 94 real-world incidents, and mapped to 33 mitigating safeguards. User surveys conducted with 17 domain experts and 134 software developers positively validated the correctness, comprehensiveness and comprehensibility of the taxonomy, as well as its suitability for various use-cases. Survey participants also assessed the utility and costs of the identified safeguards, and whether they are used.
Insecure cloud-native applications (CNAs) will continue to experience security compromises including data breaches due to their dynamic, complex, and varied threat landscape. We review current application security techniques, examine their benefits and shortcomings in the context of CNAs, and point out future research opportunities.
Full-text available
Deployed microservices must adhere to a multitude of application-level security requirements and regulatory constraints imposed by mutually distrusting application principals--software developers, cloud providers, and even data owners. Although these principals wish to enforce their individual security requirements, they do not currently have a common way of easily identifying, expressing and automatically enforcing these requirements at deployment time. CDI (Code Deployment Integrity) is a security policy framework that enables distributed application principals to establish trust in deployed code through high-integrity provenance information. We observe that principals expect the software supply chain to preserve certain code security properties throughout the creation of an executable bundle, even if the code is transformed or inspected through various tools (e.g., compilation inserts stack canaries for memory safety). Our key insight in designing CDI is that even if application principals do not trust each other directly, they can trust a microservice bundle to meet their security policies if they can trust the tools involved in creating the bundle.
Kumar, RakeshGoyal, RinkajIn the quest of velocity in time-to-market for the cloud applications, often, the security requirements are overlooked. It is mainly due to the preconceived notion that building security capabilities is very time-intensive and affects delivery timeline. However, the lack of required security capabilities in the applications have cascading impact on business objectives. The proposed research work addresses this issue by enabling automation in security activities. This work extends our previous work to analyze security challenges in cloud-enabled, cloud-native, cloud edge IoT, and cloud big data analytics applications and conceptualize a continuous security model for cloud applications using DevSecOps. The model proposes four inter-woven components: (1) principles to follow, (2) application lifecycle stages to implement, (3) practices to adopt, and (4) automation ecosystem to use. These components provide security elements to consider during cloud application development, deployment, and operation phases to bring velocity in building security capabilities and providing the continuous security assurance.
Full-text available
Continuous Deployment (CD) has emerged as a new practice in the software industry to continuously and automatically deploy software changes into production. Continuous Deployment Pipeline (CDP) supports CD practice by transferring the changes from the repository to production. Since most of the CDP components run in an environment that has several interfaces to the Internet, these components are vulnerable to various kinds of malicious attacks. This paper reports our work aimed at designing secure CDP by utilizing security tactics. We have demonstrated the effectiveness of five security tactics in designing a secure pipeline by conducting an experiment on two CDPs - one incorporates security tactics while the other does not. Both CDPs have been analyzed qualitatively and quantitatively. We used assurance cases with goal-structured notations for qualitative analysis. For quantitative analysis, we used penetration tools. Our findings indicate that the applied tactics improve the security of the major components (i.e., repository, continuous integration server, main server) of a CDP by controlling access to the components and establishing secure connections.
Conference Paper
Full-text available
At the RELENG 2014 Q&A, the question was asked, "What is your greatest concern?" and the response was "someone subverting our deployment pipeline". That is the motivation for this paper. We explore what it means to subvert a pipeline and provide several different scenarios of subversion. We then focus on the issue of securing a pipeline. As a result, we provide an engineering process that is based on having trusted components mediate access to sensitive portions of the pipeline from other components, which can remain untrusted. Applying our process to a pipeline we constructed involving Chef, Jenkins, Docker, Github, and AWS, we find that some aspects of our process result in easy to make changes to the pipeline, whereas others are more difficult. Consequently, we have developed a design that hardens the pipeline, although it does not yet completely secure it.
Full-text available
The security of software systems can be broadly divided into two categories namely external security and internal security. The internal security of software systems is the main issue for any secure system. This issue of software security depends on the design of software systems and integration of security features into the design. This process involves the identification of security threats to software systems, identification of relevant mitigation measures and their integration into the design of software systems. To identify security requirements of software systems, many techniques have been developed. Threat modelling is one of the approaches to identify security requirements and helps to add them into the design of the software systems. Threat modelling makes it possible to identify all potential threats to the software systems and hence helps software designers to add mitigations to make their software design more secure and reliable. Many threat modelling approaches had been developed over the time since its emergence. This paper presents a survey on various threat modelling techniques. In this paper, we have focussed on the techniques in the context of secure web applications.
Traditionally, software engineers have defined security as a non-functional requirement. As such, all too often it is only considered as an afterthought, making software applications and services vulnerable to attacks. With the phenomenal growth in cybercrime, it has become imperative that security be an integral part of software engineering so that all software assets are protected and safe. Architecting Secure Software Systems defines how security should be incorporated into basic software engineering at the requirement analysis phase, continuing this sharp focus into security design, secured programming, security testing, and secured deployment. Outlines Protection Protocols for Numerous Applications Through the use of examples, this volume defines a myriad of security vulnerabilities and their resultant threats. It details how to do a security requirement analysis and outlines the security development lifecycle. The authors examine security architectures and threat countermeasures for UNIX, .NET, Java, mobile, and Web environments. Finally, they explore the security of telecommunications and other distributed services through Service Oriented Architecture (SOA). The book employs a versatile multi-platform approach that allows users to seamlessly integrate the material into their own programming paradigm regardless of their individual programming backgrounds. The text also provides real-world code snippets for experimentation. Define a Security Methodology from the Initial Phase of Development Almost all assets in our lives have a virtual presence and the convergence of computer information and telecommunications makes these assets accessible to everyone in the world. This volume enables developers, engineers, and architects to approach security in a holistic fashion at the beginning of the software development lifecycle. By securing these systems from the project’s inception, the monetary and personal privacy catastrophes caused by weak systems can potentially be avoided.
Conference Paper
Continuous delivery (CD) pipelines recently gained wide adoption. They provide means for short and high-frequent development cycles in DevOps by automating many steps after a commit has been issued and bringing it into production. CD pipelines have become essential for development and delivery. Hence, they are crucial and business-critical assets that need to be protected from harm in terms of dependability and security. DevOps practices like canary releasing and A/B testing aim to improve the quality of the software that is built by CD pipelines while keeping a high pace of development. Although CD is a part of DevOps, the DevOps practices have primarily been applied to the artifacts that are processed but not on the pipelines themselves. We outline our vision of using these DevOps practices to improve the dependability and security of CD pipelines. The goal is to detect, diagnose, and resolve dependability and security issues in the CD pipeline behavior. In this paper, we outline our envisioned roadmap and preliminary results from an ongoing industrial case study.
Conference Paper
In organizations that use DevOps practices, software changes can be deployed as fast as 500 times or more per day. Without adequate involvement of the security team, rapidly deployed software changes are more likely to contain vulnerabilities due to lack of adequate reviews. The goal of this paper is to aid software practitioners in integrating security and DevOps by summarizing experiences in utilizing security practices in a DevOps environment. We analyzed a selected set of Internet artifacts and surveyed representatives of nine organizations that are using DevOps to systematically explore experiences in utilizing security practices. We observe that the majority of the software practitioners have expressed the potential of common DevOps activities, such as automated monitoring, to improve the security of a system. Furthermore, organizations that integrate DevOps and security utilize additional security activities, such as security requirements analysis and performing security configurations. Additionally, these teams also have established collaboration between the security team and the development and operations teams.