Conference PaperPDF Available

Towards Cloud Native Continuous Delivery: An Industrial Experience Report


Abstract and Figures

Several companies suffer from long delays between implementing and delivering software features. Continuous Integration, Delivery, and Deployment tools and practices are increasing their popularity since they can help companies decrease delivery costs and delays, and reduce bugs and errors in the delivery processes thanks to automation. Companies have difficulties in implementing Continuous Integration, Delivery, and Deployment. There is information available on the subject, but collecting, studying, and distributing that information can be very costly. The CI and CD pipelines are examples of well founded Cloud Native applications that run in a variety of different configurations and provide flexibility and speed for companies, but require elastic execution platforms and knowledge of modern cloud technologies for optimal use. In this experience report we present the lessons learned during the implementation of a cloud-based Continuous Delivery tool stack. Our experience includes the design and implementation of a pure cloud and hybrid cloud based Continuous Integration and Delivery solutions at Vincit.
Content may be subject to copyright.
Towards Cloud Native Continuous Delivery: An Industrial Experience Report
Aleksi H¨
Helsinki, Finland
Davide Taibi, Kari Syst¨
Tampere University of Technology
Tampere, Finland
Abstract Several companies suffer from long delays be-
tween implementing and delivering software features.
Continuous Integration, Delivery, and Deployment tools and
practices are increasing their popularity since they can help
companies decrease delivery costs and delays, and reduce bugs
and errors in the delivery processes thanks to automation.
Companies have difficulties in implementing Continuous
Integration, Delivery, and Deployment. There is information
available on the subject, but collecting, studying, and distribut-
ing that information can be very costly.
The CI and CD pipelines are examples of well founded
Cloud Native applications that run in a variety of different
configurations and provide flexibility and speed for companies,
but require elastic execution platforms and knowledge of
modern cloud technologies for optimal use.
In this experience report we present the lessons learned dur-
ing the implementation of a cloud-based Continuous Delivery
tool stack. Our experience includes the design and implemen-
tation of a pure cloud and hybrid cloud based Continuous
Integration and Delivery solutions at Vincit.
In 2015 Vincit, a Finnish software development company,
started a Continuous Integration and Delivery project as
part of the Finnish N4S project [2] with a the goal of
providing software developers build and test automation tools
that support complex Continuous Delivery pipelines which
contain multiple build, test, and deployment targets with
complex configurations.
Continuous Delivery practices can save time and reduce
costs in projects. Development teams can ensure that soft-
ware has been tested and ready for production when working
and verified builds are rolled from integration pipelines
into staging environments. Managers and sales personnel
can happily tell the customer that software is ready to be
deployed when the customer asks to see the latest sprint
result in action. Customers do not have to wait for a week
or a month of deployment delays [4].
Continuous Delivery can be implemented in stages, sup-
porting first the processes where the needs for automation
and orchestration are the greatest, or just implementing the
parts which offer the most returns for invested money and
time. The degrees of implementing automation have been
defined in the Continuous Delivery Maturity Model which
offers guidelines for what to implement in what order to be
more efficient [9], [10], [11].
We had previous experience in build and test automation
solutions from running tools such as Hudson, Jenkins, and
Travis CI, which we currently use in our repertoire as well.
However, the creation of configuration pipelines and delivery
was still complex in 2015. Since then, all these projects
have seen tremendous improvement in terms of features and
maturity, and nowadays, are much more viable for advanced
Continuous Delivery usage.
We consider Continuous Delivery to be a Cloud Native
application as it requires elastic computing capacity and
the capacity requirements change rapidly. This has financial
implications since it is hard to estimate the short- and long-
term financial implications of implementing systems. Con-
tinuous Delivery pipelines also require advanced monitoring
In this paper we report the lessons learned while im-
plementing a customized open-source Continuous Delivery
system. The different phases we look into include research,
requirements definition, design, implementation, and refac-
toring of a whole Continuous Delivery software system on
top of pure cloud infrastructure. This purely cloud based
system architecture later mutates into a hybrid cloud archi-
tecture, and we briefly discuss transforming from pure cloud
to hybrid cloud solutions.
The paper is structured into distinct sections that discuss
design, implementation and evaluation of a Continuous De-
livery system. Section 2 reports Background and Related
Works. Section 3 describes the different possibilities for im-
plementing a CD platform. Section 4 analyzes the difference
between self-hosted and cloud platforms. Section 5 discusses
cost of systems while Section 6 discusses the choice of
Continuous Delivery tools. Section 7 reports the case study
with an analysis of how the different steps of the CD pipeline
have been implemented. Section 8 evaluates the results of the
case study, and finally, Section 9 draws conclusions.
Continuous Integration [3] is related to the frequent auto-
matic software integration, which commonly means building
and testing changed source code. Frequency means that
software is built and tested periodically or, for example, on
every version control commit.
Continuous Delivery [4] includes Continuous Integration
and making sure that the software is always configurable
and deployable. This requirement is usually satisfied with an
automated staging environment deployment.
Continuous Deployment includes always automatically
deploying software to production when it is committed to
version control system branches corresponding to production
environments and qualified by the automatic tests to be
Continuous Delivery and Deployment are often used in
the same context and can be mistaken with each other, but
in academic context differentiating the terminology is impor-
tant [7]. Figure 1 shows the relations between Continuous
Integration, Delivery, and Deployment.
Fig. 1. Relations of Continuous Integration, Delivery and Deployment
Companies cannot hence implement Continuous Deploy-
ment without Continuous Integration and Delivery systems.
A. Terminology
For the sake of clarity, we define different terms adopted
in this work:
Continuous Integration is a group of practices that aims
to improve software development quality and speed
with build and test automation in order to improve
reproducibility and to remove the chance of errors
from manual build step execution or environment state
mutation in build processes. Continuous Integration is
a subset of Continuous Delivery;
Continuous Delivery is a group of practices that includes
Continuous Integration and adds to them the automated
end-to-end testing and delivery of software in such
a way that software builds are stateless, reproducible,
and proven deployable across target environments and
platforms. Continuous Delivery makes delivering recent
software iterations to production at any given time feasi-
ble and aims at guaranteeing deployability. Continuous
Delivery is a super-set of Continuous Integration, and a
subset of Continuous Deployment. Continuous Delivery
does not include automatically deploying software to
production environments;
Continuous Deployment is a group of practices that
includes the aforementioned Continuous Integration and
Delivery practices but adds to them the practice of
automatically deploying software to production envi-
ronments. Continuous Deployment ideally removes the
need of manual production environment updates and
aims to roll-forward only deployments. Continuous De-
ployment is a super-set of Continuous Delivery.
There are many tools for Continuous Integration and
Delivery written in many languages illustrated in Table I.
Software Implementation Published Maintainer
Buildbot Python 2003 Mitchell et. al
GoCD Java 2007 ThoughtWorks, Inc.
Jenkins Java 2011 Kawaguchi et. al
Travis CI Ruby 2011 Travis CI, GmbH
Strider CD Ruby 2012 Radchenko et. al
GitLab CI Ruby 2012 GitLab, Inc.
Drone Go 2014
Java is one of the most commonly used languages for
CD tools, powering services such as Jenkins [12] and GoCD
[6]. There are also alternatives for Java powered platforms
such as Buildbot [13] written in Python, Travis CI [14]
written in Ruby, and Strider [15] and Drone [16] written in
Node.js. These are just few examples of the tools available
for implementing Continuous Integration. Many of the listed
alternatives are available as partly or fully open-source
Equally, many architectural models exist for Continuous
Integration and Delivery systems. The systems range from
simple examples running bash or Python scripts to multi-
tiered enterprise solutions that can be hosted in multiple data
centres. For example, the simplest of build systems can be
implemented in a few hours on top of Buildbot on a single
computer. Some platforms such as Travis CI or Snap CI [17]
require a multiple machine set up just to operate on-premise.
Platforms and tools such as Make and Buildbot can be
perfectly viable for implementing Continuous Delivery for a
software product, but the set up of the pipeline from devel-
opment to production server deployments with configuration
management and source code builds can be more difficult
to master and scale. Some specifically tailored software
platforms intended for building a specific technology solution
such as Azure Pipelines for Microsoft products can be much
easier to utilize, because they might support the scenarios
that one can commonly run into when setting up Continuous
Delivery pipelines.
Developers have a myriad of options for adopting Contin-
uous Delivery into their work and project flows which can
be categorized as:
managed SaaS solutions such as Travis CI;
hostable enterprise solutions such as Azure DevOps, or;
self-hosted open-source solutions such as GoCD.
Each of the aforementioned options can be valid, depend-
ing on the current and future situation and conditions in the
company. Smaller companies should prefer to use lightweight
managed solutions and avoid over-committing to one path
unless there is a clear need for a heavyweight system.
Larger enterprises might need multiple different systems to
support their operations. In each case, an understanding of
the different alternatives and their service and cost models
is necessary in making the right choice.
One important factor in choosing the right alternative is the
solution’s extensibility. If in 5 years time we need a feature
that is not implemented, what would we do? Many of the
current systems do not offer extensibility. Travis CI offers
access to its deployment tools, but a lot of platforms offer
no access to their inner workings or source code, and cannot
be modified at all. We already knew some requirements for
the Continuous Delivery system we wanted to implement in
our Minimum Viable Product (MVP)[5].
The first of our requirements was extensibility. Another
one was the ability to support cross-platform builds. We
wanted the same tool to be usable on macOS, Windows,
and Linux environments. The last important requirement was
that we could host the Continuous Delivery service in private
or public data centers. Having someone else host the service
was simply too rigid of an option. Our customers have a need
for flexibility, so we wish to offer them as many options as
Hence we decided that we wanted to invest in an open-
source solution that we could extend and program ourselves,
and hopefully host ourselves, if needed. In the open-source
front there are a few options that have a community around
them offering support and tool-sets to each other. Narrowing
the search down, we found ourselves facing yet another
decision: choosing the right open-source option for our
Different Continuous Delivery solutions have different
cost models, which are of essential knowledge when making
management decisions regarding the implementation and
lifespan of Continuous Delivery systems. Understanding of
the cost implications of the different implementation choices
is an important decision criterion.
Operating expenses and capital expenses are two main
elements of the cost models [21], [33]. In short, capital
expenses are multiple-term costs that are tied to a system
for a long time, such as data centre investments and system
vendor acquisition costs, network infrastructure acquisition
and initial large licence purchases. Operating expenses are
single-term costs that are tied to running the system at a
certain load, for example network transfer and electricity
costs and manual maintenance labor. Using an operation-
ready SaaS platform has a pay-per-use cost model.
Implementing a platform with on-premise hardware or
cloud hardware can have very different cost models which
can include servers acquisition, management, power, cooling,
backups, maintenance, licences, and an assortment of other
things, which can be very hard to predict.
Most cloud service providers bill for networking, CPU,
RAM, and storage capacity. For example, Amazon Web Ser-
vices bills for network components such as Internet and VPN
connectivity and outbound traffic, server resource usage such
as CPU cores and memory, and storage capacity. In addition
to this, for the users of proprietary operating systems, such
as Windows or Red Hat Enterprise Linux, licence fees apply
on per-machine basis. In case of proprietary Continuous
Delivery tools, a licence is also applicable. [22], [32]
On-premise computing capacity, in addition to the cloud
platform components, adds the cost for power, cooling,
and staff work. Moreover, on-premise systems also require
resources for handling with power outages, loss of data and
other similar issues.
From these concepts we defined to fixed and floating costs.
Fixed costs are the baseline costs that are tied to the
running of the system and rarely change: data centers
and equipment are examples of fixed costs.
Floating costs are costs that change in the lifespan of
the system.
The ability and willingness to pay large fixed costs and
make purchases up-front affects the choice of service and
hosting model. If a company can predict capacity needs in
detail and has liquidity, then an upfront investment can be
wise. If capacity needs are not static and change over time,
making optimal investment choices can be hard. With cloud
computing platforms capital is not tied to fixed investments,
and risks are reduced. [33]
One of our requirements that was realized and refined
during the project was the ability to support and specify de-
pendencies for build steps and different projects [20]. Com-
plex dependency management is important when building,
for example, a microservice architecture or complex multiple
tier software where one wants to define the build, test, and
deployment pipeline as a dependency graph. For example,
it might be necessary to build the backend and frontend
software first, and, then test their component integration, and
finally test the end-to-end functionality of the system.
Considering the different requirements regarding tooling
support for multiple platforms, languages and tools we
decided to look further into options that offered script based
and non-opinionated architectures. The most prominent of
these systems was GoCD. GoCD has most of the things we
wanted our tool to have. It is:
open-source and has a permissive licensing;
platform agnostic and runs anywhere where Java is
non-opinionated and runs anything you can script to run
via system shell;
scalable, both horizontally and vertically, and lastly;
has a stable user community and good documentation.
All these factors combined, the only issue we had with the
project was its lack of an established plugin ecosystem, such
as the one in Jenkins. Jenkins CI has a myriad of different
extensions and supports most common tools because of
its age and community. GoCD was, in 2016, in middle
of implementation of some very central features such as
dynamic build agent provisioning. Small delays, however,
are things that we were willing to deal with when investing
into long-term tooling.
We implemented the CD system in AWS cloud based
computer system. Our implementation work for the private
cloud at Vincit began by creating a network layout.
We created a VPC (Virtual Private Cloud) in Frankfurt
with /16 CIDR block that was compatible with our existing
network layout, and created three different subnets in that
network segment. Our subnets consisted of a /24 manage-
ment subnet, a /24 GoCD server subnet, and a /24 GoCD
agent subnet. Once we had our network layout defined, we
set up a VPN gateway to it and opened a ticket to our ISP
(Internet Service Provider) requesting that our office network
be connected via our router with VPN to the AWS network
and routing policies be configured. This took about two
weeks and a few failed configuration attempts from our ISP,
but after the wait we had our networks defined and were able
to connect to the AWS cloud via private connection from our
office. During this waiting period we started setting up our
virtual server infrastructure and software components into
AWS to avoid downtime in the whole process.
We started our EC2 (Elastic Compute Cloud) virtual
server configuration by searching for the Ubuntu LTS AMI
(Amazon Machine Image) from the AWS Marketplace [23],
which houses software that can be run on the AWS. Most
Linux distributions can be found on the Marketplace free of
charge as they have permissive licensing schemes.
After finding and launching our Ubuntu instances, we
continued by configuring them with SSH keys and setting up
secure connectivity with them. After successfully connecting
to the instances we installed updates and provisioned the
instances with Salt. Salt then proceeded to automatically
install Sensu and GoCD software to the nodes. At this point
we had the architecture illustrated in Figure 2.
The system seemed to work in the beta testing environ-
ment, and we had everything running smoothly. Builds were
executed on the GoCD agents and we were managing nodes
with an integrated Salt solution. Our Salt scripts would install
packages and whole programming environments required in
builds, and fetch and configure the SSH keys and configura-
tions needed to interact with source code repositories.
We were also tracking the raw node statistics with Sensu,
which was running on every agent node.
Analyzing the degree of system usage is easy on most
IaaS cloud platforms. Most IaaS platforms are virtualized
and offer access to the virtualization system’s CPU usage
AWS offers numerous statistics of an instance that can be
gathered and stored for an arbitrary period of time.
Some of the statistics that AWS offers via its proprietary
CloudWatch system for an EC2 virtual machine instance
are: 1) CPU usage; 2) disk read and write statistics, and; 3)
network device usage. Memory usage statistics are not pro-
vided by the virtualization platform, but can be additionally
monitored with reporting scripts running in the virtualized
guest operating system. [24]
After running the service for a while we realized that
we were running workers that were not used during the
night time since all our developers were out-of-office. This
meant that we were running idle computing capacity but
paid for the full capacity. Because AWS supports capacity
scaling with ASGs (Auto Scaling Groups) and Autolaunch
Configurations, we created an automatically scaling cluster
that could increase capacity in the morning and decrease
capacity in the night time. The system would run zero
instances in the night and 2-4 instances between 6AM and
8PM, local time. This would total to 40% less running
time for worker instances, which reduced the total costs of
four worker instances and two management and instances
by over 25%. Since, we earlier saw that the EC2 running
costs constitute for about 80% of our overall costs, we could
reduce our overall AWS costs by about 20%.
Automatic scaling requires that each time an instance is
started, all necessary software and configuration is installed
to it. We configured our Linux instances to run a boot-
strapping script that installs a Salt Minion [26] to a node
each time a cluster machine is brought up, and Salt, our
orchestration tool, would configure the node as a GoCD
agent after that. All-in-all, our whole bootstrapping for the
instance constituted to a simple shell script that can easily
be modified to install Salt on any Linux distribution. It
also can be reconfigured largely on the same principles
to bootstrap a node that is running macOS or Windows
for Salt configuration. This removes the need for manually
configuring computers. More details on the script can be
found in the associated thesis work [1].
The systems implemented and discussed in this paper
offered data and subjective experiences that we would like to
further discuss and evaluate. Some interesting technical as-
pects of the system were cost factors, the technical evolution
of the tooling, improvements of processes and other tooling
we made with implementing Continuous Delivery tools, and
measurements of the systems.
A. Cost and labor factors
Work wise, we have invested about two weeks of time
in different stages of design and meetings throughout the
project. One person has also worked on the project and on
a Master thesis describing it for about three months [1]. In
grand total, we have about four months of work invested
in research and development and the implementation of the
system. It might be possible to implement a Continuous
Delivery system from scratch for a small amount of projects
much faster, but overall, the effort of studying the tooling
and theory associated to software automation, orchestration,
metrics and data gathering, and different details such as cloud
platform specifics is quite time consuming.
In early 2016 we were paying for a few medium sized
nodes in AWS. We had a master build node, an orchestration
node and a monitoring node for GoCD in AWS. These
amounted to three fixed cloud computing nodes. Rest of the
fleet was flexible build capacity that we were running as
needed. In addition to the computation capacity we are also
Fig. 2. Pure Cloud Continuous Integration and Delivery System Architecture
paying for approximately 200GB of fixed storage capacity
and low network transfer costs. We did not have fixed
dedicated capacity tied to the build system, but acquisition
and utilization of extra capacity was fairly easy. This setup
is illustrated in Figure 2.
The total cost of the system was in the few hundreds of
dollars per month range for the pure cloud solution. Most
notable individual cost of the system was an AWS VPN
gateway for a hybrid connection that cost USD 100 per
month, with the computing capacity costing between USD
100 to USD 300 per month, totaling to a well less than USD
500 per month in all cases for a small build fleet.
Maintenance and administration work has required an
average of one to two day investment of work per month
on the tools, resulting in notable continuous running costs.
These costs, are however, hard to define and attribute, as
some could be associated to project setup, and at least some
of the work used on maintenance of the tools would be
required in a managed tool such as Travis CI because projects
need to be kept up-to-date and build failures investigated.
B. Evolution of the technology stack
Later in 2016 and 2017 we moved into a hybrid and then
onto a on-premise hosting model due to hosting structure
changes, and moved the GoCD master node into a fixed
virtual machine and inside a Docker container. The build
agents are run as Docker containers, too. This seems to offer
excellent price-to-performance ratio due to lowered over-
heads. Containerization also corrects many of the problems
of running multiple build agents on the same physical or
virtual server. All agent workspaces are isolated because the
GoCD agent Java processes run inside containers, having
very limited and closely controlled access to the host system
resources aside from ephemeral file system and network
access. This has worked well for us for well over a two
At the time of writing in late 2018 we are running Jenkins
as our primary and GoCD as our secondary internal build
system with both doing Continuous Delivery for projects.
Jenkins has a larger amount of projects, slaves, and users due
to its more complete feature set. GoCD also has a distinct
group of projects, but most new projects are set up in Jenkins.
This is because we have previous history with Jenkins, and
its development for both core and and plugins have moved
forward with a faster pace than in GoCD. Most reasonable
technical requirements are also met by the existing plugins.
Jenkins has also introduced a large feature called pipelines
in 2016 and 2017. Pipelines enable writing build definitions
in either declarative or scripted manner, and enable shareable
libraries for defining common build operations. Pipelines also
enable easy parallel build steps and allow granular build
dependency definition in free-form graphs.
A major note we would like to make is that both Jenkins
and GoCD have access control and identity management
limitations. In our use they are currently only offered to
our developers internally in company intranet due to security
considerations. Both could be opened up to the Internet
with some restrictions, but little is to be gained in terms of
usability, and system security will need to regularly audited.
In terms of user management and authentication Jenkins
and GoCD both do feature LDAP based login backends,
allowing easy directory integration. Both also offer varying
quality SSO solutions based on SAML 2.0, OAuth 2.0 and
OpenID Connect 1.0. These plugins, however, are commu-
nity driven and rarely if ever audited security-wise, and the
responsibility of ensuring their security often falls to the
administrators of the tools.
C. Perceived and measured improvements
Our build duration dropped by approximately 25% due
to moving our builders to the AWS cloud and having them
less loaded. This is largely connected to a single builder
node only processing a single project and not taking any
additional load. We were able to select the correct build
machine sizes for various projects and select the optimal
amount of resources to host our systems, making it possible
to finely tune the offered build capacity to the needs of our
developers. This is also possible with our on-premise build
system that is based on virtualization.
Failure rates due to system errors have reduced, because
we are only executing a single build on a single system
or container, and are not introducing conflicts, caching
problems, or computing resource exhaustion into the build
processes. These are all things that are fairly expensive to
debug, because a person has to go and investigate the build
and system logs to determine an indeterministic reason for a
build failure. The exact reduction in errors is not transparent,
but early data suggests we have solved some of our concur-
rency, virtualization, and container based problems, moving
from platform problems to build node or job configuration
errors. The latter are much easier to locate and fix.
It also seems that we improved our build tooling on many
parts during the project. GoCD supports resource tagging,
build environment specification, heterogeneous builders,
management of project dependencies, and other features that
are hard to find in traditional build tools. We have not faced
any performance issues or instability from the tool.
In addition to improving our systems technically we have
introduced the concept of push-button deliveries and high
deployability. Only some projects are using push-button
deployments on GoCD, but we have implemented similar
features using Travis CI for Continuous Delivery with the
AWS Elastic Beanstalk platform using the dpl tool by Travis
CI [27]. We are currently introducing push-button delivery
to new projects, which has reduced the need for manual
deployments and saved work time in projects.
All-in-all, the perceived improvements are considerable.
The concrete measurable improvements which will save
our customers money will hopefully come apparent in the
upcoming months and years. Quantifying the project results
is hard at this stage, because we do not have extensive
data available yet. Many of the benefits we have achieved
were not expected to be immediately available though, and
will accumulate in time when an increasing number of
projects adopt the Continuous Delivery methodology and
gain confidence in rapidly available customer deliverables
and increased deployment rates.
In addition to us implementing continuous improvements
to the build tooling since 2016, we have gained a consid-
erable amount of knowledge in the Continuous Integration,
Delivery, and Deployment domain. The increased knowledge
has steadily improved the processes and tools in use, and
will hopefully enable us to move forward with software
development practices as well.
D. Metrics, data, and information
Getting features implemented and delivered to the cus-
tomer with less work, fewer errors, shorter development
cycle, and less downtime is the main thing that automation
enables [4]. To improve the rate of delivery we hope to
implement a comprehensive measuring system that could
give us insight on deltas between development, deployment,
and activation times. We want to measure features and
releases done per month in addition to other system statistics
such as build durations and frequencies. We also want to
make this information transparent to software development
teams and customers [29].
At this time the collection and evaluation of metrics and
data is hard due to the use of multiple different build systems
that build different kinds of software projects. The effort
of closely measuring and analyzing the systems and their
differences has not been yet undertook, but is an essential
step in developing more robust and sophisticated tooling.
Metrics and data enable improved decision making pro-
cesses which are based on scientific methods. Most build
and deployment tools offer some built-in data visualization
and metrics, but few offer simple APIs for exporting the
said metrics into usable formats. The implementation of a
metrics and measurement service that integrates into different
projects, their services, hosting platforms, and other tools is
therefore a task that will require more research efforts.
In this paper we report on our experience in implement-
ing a Continuous Delivery system based on open-source
technologies. We first described the different possibilities
to implement a Continuous Delivery platform, highlighting
pros and cons and we compared cost between hosted and
cloud solutions. In our case study we then implemented
a complete cloud-based Continuous Delivery pipeline, and
then we migrated into a hybrid solution. It is interesting to
notice how and when we maximized the benefits between
cloud and hosted solutions, and how we integrated them.
The same benefits could be transferred to any Cloud
Native application, not only Continuous Delivery. Companies
could learn from our experience when and why it could be
beneficial to migrate parts of their systems into the cloud.
We believe that a pure cloud hosting and containeriza-
tion offers the most flexibility for implementing Continuous
Integration, Delivery and Deployment systems. The current
trends in open-source build systems seem to moving towards
so called Cloud Native Continuous Integration. Quite re-
cently CloudBees and Kohsuke Kawaguchi have been driving
the Jenkins development into a more containerized direction,
even opting to redesign existing solutions and dropping
backwards compatibility to move development faster going
forwards [30].
Future work include the definition of a cost model to
support companies in understanding the most suitable alter-
natives between different cloud providers and a in-premise
solution. We will also analyze the reasons why companies
are migrating to Cloud Native pipelines [31], of patterns
and anti-patterns of the Cloud Native pipelines, following
the previous approaches applied in [8], [19], [31]. Moreover,
we are also planning to support companies in using data
collected from the CD platform to predict different software
characteristics. As an example, it we are planning to extend
data-driven models for software reliability [34], maintenance
[35] also considering dynamic measures [28] and suitability
of Cloud Native patterns [25].
[1] A. H¨
akli, ”Implementation of Continuous Delivery Systems”
Master Thesis. Tampere University of Technology, 2016.
[2] Need4Speed Project. DIMECC N4S-Program: Finnish Software Com-
panies Speeding Digital Economy.
[3] P. M. Duvall, S. Matyas, and A. Glover, Continuous Integration:
Improving Software Quality and Reducing Risk. Addison-Wesley
Professional, 2007.
[4] J. Humble and D. Farley, Continuous Delivery: Reliable Software
Releases Through Build, Test, and Deployment Automation. Addison-
Wesley, 2010.
[5] V. Lenarduzzi, D. Taibi, ”MVP explained: A systematic mapping
study on the definitions of minimal viable product.” 42nd Euromicro
Conference on Software Engineering and Advanced Applications,
SEAA 2016, 112-119. doi:10.1109/SEAA.2016.56
[6] ThoughtWorks et al., ”Go Continuous Delivery,”,
retrieved May 13, 2016.
[7] C. Caum, ”Continuous Delivery Vs. Continuous Deployment. What’s
the diff?”
deployment-whats-diff, retrieved Sept 2018.
[8] D. Taibi, V. Lenarduzzi, ”On the definition of microservice bad
smells.” IEEE Software, 35(3), 56-62. doi:10.1109/MS.2018.2141031
[9] A. Rehn, T. Palmborg, and P. B¨
ostrom, ”The Continuous Delivery Ma-
turity Model,” Continuous-Delivery-
Maturity-Model, Tech. Rep., 2013, retrieved Oct 2018.
[10] E. Minick, ”Continuous Delivery Maturity Model, urbancode/docs/continuous-delivery-
maturity-model/, Tech. Rep., 2014, retrieved Oct 2018.
[11] P. Bahrs, ”Adopting the IBM DevOps approach for continu-
ous software delivery: Adoption paths and the DevOps maturity
model, ” developerworks/library/d-adoption-
paths/, Tech. Rep., 2013, retrieved Oct 2018.
[12] K. Kawaguchi et al., ”Jenkins,”, retrieved Oct 2018.
[13] D. J. Mitchell et al., ”Buildbot,”, retrieved Oct
[14] Travis CI, GmbH, ”Travis CI,”, retrieved Oct
[15] I. Radchenko et al., ”Strider Continuous Delivery,”, retrieved Oct 2018.
[16], ”,”, retrieved Oct 2018.
[17] ThoughtWorks, Inc., ”Snap CI,”, retrieved Oct
[18] Microsoft, Inc., ”Team Foudation Server,” en-us/products/tfs-overview-vs.aspx,
retrieved Oct 2018.
[19] D. Taibi, A. Janes, V. Lenarduzzi, ”How developers perceive smells
in source code: A replicated study.” Information and Software Tech-
nology, 92. 2017. doi:10.1016/j.infsof.2017.08.008
[20] ThoughtWorks, Inc., ”Fan-out & Fan-in,”
go-fan-out-fan.html, retrieved Oct 2018.
[21] A. Damodaran, Applied Corporate Finance: A User’s
Manual, 4th ed. John Wiley and Sons, 1999. [Online].
Available: adamodar/New Home
[22] Amazon, Inc., ”How AWS Pricing Works, pricing overview.pdf,
Tech. Rep., 2016, retrieved Oct 2018.
[23] ——, ”AWS Marketplace,”, re-
trieved Oct 2018.
[24] ——, ”AWS CloudWatch,”, re-
trieved Oct 2018.
[25] D. Taibi, V. Lenarduzzi, C. Pahl, ”Architectural patterns for microser-
vices: A systematic mapping study.” 8th International Conference on
Cloud Computing and Services Science, CLOSER, 2018.
[26] SaltStack, Inc., ”SaltStack architecture for system command and con-
trol,”, retrieved Oct 2018.
[27] Docker, Inc., ”Docker,”, retrieved Oct 2018.
[28] L. Lavazza, S. Morasca, D. Taibi, D. Tosi, ”On the defini-
tion of dynamic software measures.” International Symposium on
Empirical Software Engineering and Measurement (ESEM) 2012.
[29] K. Haase, ”Dpl deployment tool,”, re-
trieved Oct 2018.
[30] K. Kawaguchi, ”Jenkins: Shifting Gears,, retrieved Oct
[31] D. Taibi, V. Lenarduzzi, C. Pahl, ”Processes, motivations, and
issues for migrating to microservices architectures: An empir-
ical investigation.” IEEE Cloud Computing, 4(5), 22-32. 2017
[32] P. Rosati, F. Fowley, C. Pahl, D. Taibi, T. Lynn, ”Making the cloud
work for software producers: Linking architecture, operating cost
and revenue.” 8th International Conference on Cloud Computing and
Services Science (CLOSER 2018), 2018-January 364-375
[33] S. Aldawood, F. Fowley, C. Pahl, D. Taibi, X. Liu, ”A coordination-
based brokerage architecture for multi-cloud resource markets.” 4th
International Conference on Future Internet of Things and Cloud
Workshops, W-FiCloud 2016, 7-14. doi:10.1109/W-FiCloud.2016.19
[34] L. Lavazza, S. Morasca, D. Taibi, D. Tosi ”An empirical in-
vestigation of perceived reliability of open-source Java programs.”
ACM Symposium on Applied Computing (SAC), 1109-1114.
[35] V. Lenarduzzi, A.C. Stan, D. Taibi, D. Tosi, G. Venters, ”A dynamical
quality model to continuously monitor software maintenance.” 11th
European Conference on Information Systems Management, ECISM
... In 2015, software configuration was still complex. Vincit, a software development company, took the initiative through their Need for Speed (N4S) project to bring tremendous developments and improvements in the features of software giving birth to continuous integration/delivery. Vincit brought out software build and development tools that are automated to support the complexity of the deployment and configurations to the software [6]. ...
... Factors range from the CI tools used, the organizational needs, the security of the data, reliability of the providers, and hosting licenses. These factors vary with the main factor being the consideration of capital costs of set up [6]. ...
... A subset of Continuous delivery improves software development in terms of quality and speed by building and testing how projects are automated to eradicate errors from developers' manual environment set up. This means that software is periodically tested [6] [4]. ...
Full-text available
Continuous Integration (CI) aids in ensuring the software development process is reliable and provides fast feedback on tests and builds. It helps to detect errors quickly and corrects them before the system gets complicated. The CI process comprises of tools that help in identifying the correctness and provision of feedback for the software under process. The CI tools cleanse the data of errors through constant build and tests. The CI tools assist in transforming and mapping of the data before Integration and quick detection of errors. As cloud computing grows and gets embraced by businesses, the choice to host CI tools on premise or on cloud becomes a critical management decision. To make this decision, businesses ought to get acquainted with various CI tools and models they wish to implement, the security of their data, the capital to invest, and the reliability of the providers. Analyzing these variables will give business management an understanding of 1) the right CI tools for their businesses and 2) the best platform to host the integration tools. This review of CI on the cloud versus on premise revealed that it is imperative when determining the right CI tool for your need, to consider the right platform to host it to ensure synchronization.
... In this paper, we consider proactive autoscaling in the scaling-per-query scenario of on-demand cloud service: When a query comes in, it will be served by one of the available idle instances (warm start) or a new instance will be initiated for that query (cold start); meanwhile, each instance is terminated after processing one query and is not reused/shared by other queries. The scaling-per-query scenario is common in cloud computing [7], [8]: One example is the container registry [9], [10] for storing, managing, and securing custom container images; another example can be found in Continuous Integration (CI) and Continuous Delivery (CD) [11] to avoid user's account/data being hijacked/spoofed. Note that scalingper-query is in contrast to other cloud services where an instance keeps running and processing new queries until it is terminated for other reasons (e.g., scaled in when traffic goes down). ...
... In this paper, we consider proactive autoscaling in the scaling-per-query scenario of on-demand cloud service: When a query comes in, it will be served by one of the available idle instances (warm start) or a new instance will be initiated for that query (cold start); meanwhile, each instance is terminated after processing one query and is not reused/shared by other queries. The scaling-per-query scenario is common in cloud computing [7], [8]: One example is the container registry [9], [10] for storing, managing, and securing custom container images; another example can be found in Continuous Integration (CI) and Continuous Delivery (CD) [11] to avoid user's account/data being hijacked/spoofed. Note that scalingper-query is in contrast to other cloud services where an instance keeps running and processing new queries until it is terminated for other reasons (e.g., scaled in when traffic goes down). ...
Full-text available
Autoscaling is a critical component for efficient resource utilization with satisfactory quality of service (QoS) in cloud computing. This paper investigates proactive autoscaling for widely-used scaling-per-query applications where scaling is required for each query, such as container registry and function-as-a-service (FaaS). In these scenarios, the workload often exhibits high uncertainty with complex temporal patterns like periodicity, noises and outliers. Conservative strategies that scale out unnecessarily many instances lead to high resource costs whereas aggressive strategies may result in poor QoS. We present RobustScaler to achieve superior trade-off between cost and QoS. Specifically, we design a novel autoscaling framework based on non-homogeneous Poisson processes (NHPP) modeling and stochastically constrained optimization. Furthermore, we develop a specialized alternating direction method of multipliers (ADMM) to efficiently train the NHPP model, and rigorously prove the QoS guarantees delivered by our optimization-based proactive strategies. Extensive experiments show that RobustScaler outperforms common baseline autoscaling strategies in various real-world traces, with large margins for complex workload patterns.
... Dos Santos Serralheiro and Morais (2018) measure the sensitivity of companies to a single certi¯able standard for integrated quality and environment and safety management systems. Häkli et al. (2018) present the lessons learned during the implementation of a cloud-based Continuous Delivery tool stack. Kuruwitaarachch (2018) explores how organizations should start moving to new advance communication technologies and how to address key technical challenges in the deployment process. ...
Industrial information integration engineering (IIIE) is a set of foundational concepts and techniques that facilitate the industrial information integration process. In recent years, many applications of the integration between Internet of Things (IoT) and IIIE have become available, including industrial Internet of Things (IIoT), cyber-physical systems, smart grids, and smart manufacturing. In order to investigate the latest achievements of studies on IIIE, this paper reviews literatures from 2016 to 2019 in IEEEXplore and Web of Science. Altogether, 970 papers related to IIIE are grouped into 27 research categories and reviewed. The results present up-to-date development of IIIE and provide directions for future research on IIIE.
Agile software development methodology and DevOps, together, have helped the business to achieve agility and velocity in delivering time-to-market applications and services. Open-source software (OSS) and cloud technologies are taking up business innovation and DevOps at new heights. However, in the quest of agility and velocity, user data security and privacy assurance often get lower priority as they are perceived as a time-consuming activity requiring specialized people, process, and technology. We see this problem being addressed by integrating security in DevOps processes. Security for DevOps has been institutionalized as DevSecOps with practical considerations for a given business context. In this work, we proposed a conceptual security model, ADOC, to facilitate adopting DevSecOps for the business processes capitalizing OSS over the cloud. This work contributes towards the following to integrate continuous security in application and service delivery: (i) A continuous security conceptual framework proposal based on the requirements elicited from the analysis of challenges in adopting DevSecOps using OSS over the cloud. (ii) An integrationist security model, ADOC, based on the proposed continuous security conceptual framework, integrating development, security, and operation activities through automation of security controls using OSS over the cloud. (iii) A set of inter-working OSS tools for automation of the proposed security controls in ADOC workflow and practices. (iv) A set of metrics for performance measurement of the ADOC model. (v) Mapping of the solutions for the analyzed challenges using the proposed security controls, followed by a use case scenario to adopt the ADOC workflow and continuous practices. The ADOC transforms security being adhoc compliance-oriented activities into continuous assurance-oriented activities by codifying security controls into an automated delivery workflow. Its practical adoption enables businesses to deliver time-to-market security ready applications and services with accelerated velocity and sustainable agility in a cost-effective way.
Full-text available
Continuous Integration, Delivery, and Deployment are subjects that have been on the table in the recent years. The books Continuous Integration by Duvall et. al (2007), and Continuous Delivery by Humble et. al (2010) are, however, the only extensive literature that has been published on the subject. In addition to the books there is information available based on miscellaneous conferences and scientific publishments, but this information is fairly scattered and hard to compile. A lot of companies suffer from long delays between implementing and delivering software features. Multiple parties ranging from foreign developers to Finnish leader projects have researched the benefits that can be gained from automating and making software delivery continuous. Benefits include among other things rapid customer feedback, lowered production delivery costs and delays, and smaller number of errors in processes. Companies do, however, have difficulties in implementing Continuous Integration, Delivery, and Deployment. This is mainly due to the arbitrarity of the work. There is information available on the subject, but collecting, studying, and distributing that information can be very costly. This Master of Science thesis researches the implementation a Continuous Delivery system in a Finnish software company. The research work is done in an internal project, and the aim is to implement a generic software build, test, and delivery system. The process involves gathering business, technical, and user requirements, compiling a requirements definition, designing a phased project plan, and executing that plan to implement a Continuous Delivery system on top of the AWS cloud platform.
Full-text available
Code smells and architectural smells (also called bad smells) are symptoms of poor design that can hinder code understandability and decrease maintainability. Several bad smells have been defined in the literature for both generic architectures and specific architectures. However, cloud-native applications based on microservices can be affected by other types of issues. In order to identify a set of microservice-specific bad smells, researchers collected evidence of bad practices by interviewing 72 developers with experience in developing systems based on microservices. Then, they classified the bad practices into a catalog of 11 microservice-specific bad smells frequently considered harmful by practitioners. The results can be used by practitioners and researchers as a guideline to avoid experiencing the same difficult situations in the systems they develop.
Conference Paper
Full-text available
Microservices is an architectural style increasing in popularity. However, there is still a lack of understanding how to adopt a microservice-based architectural style. We aim at characterizing different microservice architectural style patterns and the principles that guide their definition. We conducted a systematic mapping study in order to identify reported usage of microservices and based on these use cases extract common patterns and principles. We present two key contributions. Firstly, we identified several agreed microservice architecture patterns that seem widely adopted and reported in the case studies identified. Secondly, we presented these as a catalogue in a common template format including a summary of the advantages, disadvantages, and lessons learned for each pattern from the case studies. We can conclude that different architecture patterns emerge for different migration, orchestration, storage and deployment settings for a set of agreed principles.
Conference Paper
Full-text available
Cloud migration is concerned with moving an on-premise software system into the cloud. In this paper, we focus on software producers adopting the cloud to provide their solutions to enterprise customers. Their challenge is to migrate a software product, developed in-house and traditionally delivered on-premise, to an Infrastructure-as-a-Service or Platform-as-a-Service solution, while also mapping an existing traditional licensing model on to a cloud monetization model. The analysis of relevant cost types and factors of cloud computing generate relevant information for the software producers when deciding to adopt cloud computing, and defining software pricing. We present an integrated framework for informing cloud monetization based on operational cost factors for migrating to the cloud and test it in a real-life case study. Differences between basic virtualization of the software product and using fully cloud-native platform services for re-architecting the product in question are discussed.
Full-text available
Microservices have been getting more and more popular in recent years, and several companies are migrating monolithic applications to microservices. Microservices allow developers to independently develop and deploy services, and ease the adoption of agile processes. However, many companies are still hesitant to migrate because they consider microservice as a hype or because they are not aware of the migration process and the benefits and issues related to migration. For this purpose, we conducted a survey among experienced practitioners who already migrated their monoliths to microservices. In this paper, we identify a process framework based on the comparison of three different migration processes adopted by the interviewed practitioners, together with the common motivations and issues that commonly take place during migrations. In this work, we describe the results and provide an analysis of our survey, which includes a comparison of the migration processes, a ranking of motivations, and issues and some insights into the benefits achieved after the adoption. Maintainability and scalability were consistently ranked as the most important motivations, along with a few other technical and non-technical motivations. Although return on investment was expected to take longer, the reduced maintenance effort in the long run was considered to highly compensate for this.
Full-text available
Context. In recent years, smells, also referred to as bad smells, have gained popularity among developers. However, it is still not clear how harmful they are perceived from the developers’ point of view. Many developers talk about them, but only few know what they really are, and even fewer really take care of them in their source code. Objective. The goal of this work is to understand the perceived criticality of code smells both in theory, when reading their description, and in practice. Method. We executed an empirical study as a differentiated external replica- tion of two previous studies. The studies were conducted as surveys involving only highly experienced developers (63 in the first study and 41 in the second one). First the perceived criticality was analyzed by proposing the description of the smells, then different pieces of code infected by the smells were proposed, and finally their ability to identify the smells in the analyzed code was tested. Results. According to our knowledge, this is the largest study so far investi- gating the perception of code smells with professional software developers. The results show that developers are very concerned about code smells in theory, nearly always considering them as harmful or very harmful (17 out of 23 smells). However, when they were asked to analyze an infected piece of code, only few infected classes were considered harmful and even fewer were considered harmful because of the smell. Conclusions. The results confirm our initial hypotheses that code smells are perceived as more critical in theory but not as critical in practice. Keywords: Software Maintenance, Code Smells, Bad Smells, antipatterns, Refactoring
Conference Paper
Full-text available
With an increasing number of service providers in the cloud market, the competition between these is also increasing. Each provider attempts to attract customers by providing a high quality service with lowest possible cost and at the same time trying to make profit. Often, cloud resources are advertised and brokered in a spot market style, i.e., traded for immediate delivery. This paper proposes an architecture for a brokerage model specifically for multi-cloud resource spot markets that integrates the resource brokerage function across several cloud providers. We use a tuple space architecture to facilitate coordination. This architecture supports specifically multiple cloud providers selling unused resources in the spot market. To support the matching process by finding the best match between customer requirements and providers, offers are matched with regard the lowest possible cost available for the customer in the market at the time of the request. The key role of this architecture is to provide the coordination techniques built on a tuple space, adapted to the cloud spot market.
Full-text available
The quantification of several software attributes (e.g., size, complexity, cohesion, coupling) is usually carried out in a static fashion, and several hundreds of measures have been defined to this end. However, static measurement may only be an approximation for the measurement of these attributes during software use. The paper proposes a theoretical framework based on Axiomatic Approaches for the definition of sensible dynamic software measures that can dynamically capture these attributes. Dynamic measures based on this framework are defined for dynamically quantifying size and coupling. In this paper, we also compare dynamic measures of size and coupling against well-known static measures by correlating them with fault-pronenesses of four case studies.