Content uploaded by Abheeshta Putta
Author content
All content in this area was uploaded by Abheeshta Putta on Oct 26, 2021
Content may be subject to copyright.
Benefits and Challenges of Adopting the Scaled
Agile Framework (SAFe):
Preliminary Results from a Multivocal
Literature Review
Abheeshta Putta1, Maria Paasivaara2,1, and Casper Lassenius1,3
1Aalto University, Dept. of Computer Science, Espoo, Finland
abheeshta.putta, casper.lassenius@aalto.fi
2IT University of Copenhagen, Denmark
mpaa@itu.dk
3Simula Research Laboratory, Oslo, Norway
Abstract. Over the past few years, the Scaled Agile Framework (SAFe)
has been adopted by a large number of organizations to scale agile to
large enterprises. At the moment, SAFe seems to be the most predomi-
nant agile scaling framework. Despite the current popularity of SAFe in
the software intensive industry, there exists surprisingly little scientific
research on the benefits and challenges of SAFe adoption. To collect the
existing knowledge on this topic, we conducted a multivocal literature
review, which includes both peer-reviewed and non-peer reviewed case
studies and experience reports on organizations that have adopted SAFe.
We identified 52 unique organisations adopting SAFe, five from the sci-
entific literature and 47 from the grey literature. The most salient bene-
fit categories were: transparency, alignment, productivity, predictability
and time to market. The most frequently mentioned challenge categories
were:change resistance, challenges with the first program increment plan-
ning and moving away from agile.
Keywords: Agile Software Development, Large-Scale Agile Software
Development, Scaled Agile Framework, SAFe
1 Introduction
Agile development methods have become highly popular in software organiza-
tions since the early 2000. The principles and practices of agile development
were originally designed for small and co-located teams [5]. To leverage the po-
tential benefits also in larger enterprises, the agile practices have to be scaled
[33, 8]. Large-scale agile transformations has been a burning topic [26, 8], with
increased concerns for additional coordination mechanisms and integration of
non-development units, such as finance and human resource management [8]. To
support scaling, new frameworks such as the Scaled Agile Framework (SAFe)
[25], Large Scale Scrum (LeSS) [24], and Disciplined Agile Delivery (DAD) [3]
2 A. Putta et al
have been proposed by agile consultants. International workshops on large-scale
agile organized at XP2016 [27] and XP2017 [26] have highlighted the importance
and need for research into the adoption of scaling frameworks [26].
According to the 12th State of Agile Survey, which Version One conducts
yearly, SAFe continues to be the most popular scaling framework in large en-
terprises [29]. Moreover, a recent survey on software development approaches
indicated the predominance of SAFe over LeSS and DAD [23]. As the number
of organizations adopting scaling frameworks is increasing [29], this provides op-
portunities for researchers and software practitioners to accumulate knowledge
on the usage of these frameworks through case studies, technical reports and
experience reports. To our knowledge, no secondary studies on the benefits and
challenges of the scaling frameworks or their adoption process have been pub-
lished. This is striking, given their importance in the industry. In this paper, we
start filling this gap by summarizing the benefits and challenges of adopting the
SAFe framework in the form of a multi-vocal literature review.
2 Related Work
2.1 Scaled Agile Framework (SAFe)
The Scaled Agile Framework was designed by Dean Leffingwell to scale agile
to large enterprises [1, 36]. It incorporates practices from Scrum, Extreme Pro-
gramming, Kanban and Lean. It offers four levels: the Team, Program, Portfolio
and Value stream levels. The team level comprises the agile teams. Agile Release
Trains (ART’s) are introduced to scale a large number of teams and individu-
als at the program level. ART’s follow HIP (Hardening, Innovation, Planning)
iterations to develop Potential Shippable Increments (PSI) or Program Incre-
ments (PI). PI’s are planned during the release planning days. SAFe introduces
additional roles such as the Agile Release Train Engineer, system teams, release
management team and portfolio management team. The core values of SAFe
are: build in quality, transparency, alignment and program execution [16]. Figure
1 gives on overview of the SAFe framework.
2.2 Secondary Studies on Large-scale Agile
Secondary studies on large-scale agile have explored topics such as challenges and
success factors of large-scale agile transformations [8], organizational, managerial
and cultural aspects [32], scalability and adoptability [22], inter-team coordina-
tion [14], architectural roles [35] and quality requirements practices [2]. However,
systematic literature reviews on scaling frameworks have not been found in the
literature. The only review found on scaling frameworks, compares a few scaling
frameworks based on team size, practices and organization type [1]. Neither that
review, nor other previous reviews on large-scale agile have included the grey lit-
erature, e.g. case studies or experience reports published on the homepages of
the frameworks. While there are inherent problems with case studies published
Benefits and Challenges of Adopting the Scaled Agile Framework 3
Fig. 1. Scaled Agile Framework [19]
by the proponents of a particular framework, completely eliminating such stud-
ies from literature reviews unnecessarily excludes the voice of the practitioners
on the usage of scaling frameworks and implementation of agile at scale. Thus,
in particular given the lack of scientific literature, we consider it important to
study such cases, fully understanding the related problems, further discussed
below.
3 Research Method
We conducted a multivocal literature review on the adoption of the SAFe frame-
work to answer the following research questions:
RQ1: What are the reported benefits of adopting SAFe?
RQ2: What are the reported challenges of adopting SAFe?
3.1 Multivocal Literature Review
Systematic literature reviews and systematic mapping studies have been popular
in the field of software engineering. They help to summarize the existing studies
4 A. Putta et al
reported in a specific research domain [11]. According to the widely adopted
systematic literature review guidelines [21], a “fully systematic literature review”
should include both the grey and the peer reviewed literature. Grey literature
is defined as, “(the literature), produced on all levels of government, academics,
business and industry in print and electronic formats, but which is not controlled
by commercial publishers, i.e., where publishing is not the primary activity of the
producing body” [11].
However, most SLRs published in the software engineering literature have
not included the grey literature [12]. Including the grey literature is challenging,
and the search strategy for grey literature has not been systematically addressed
in the SLR guidelines [21]. This is unfortunate, as excluding this literature will
eliminate the voice and opinions of the practitioners who do not publish in the
academic forums [12, 4]. It has been evident that most of the software practi-
tioners do not publish in academic fora [13].
The situation seems to be slowly changing for the better. Recent literature
reviews including both peer reviewed literature, as well as grey literature from
blogs, websites, and white papers, have popularized the term ”multi-vocal re-
views”, or MLRs [12]. Several such reviews have been conducted in software
engineering to bridge the gap between the voice of practitioners and academics
[12, 34, 28, 4].
The inclusion of grey literature can also be considered as a threat, as the in-
formation reported is based on the opinions and experiences of the practitioners
rather than systematic data collection procedures and analysis [10]. Thus, there
are severe issues with, e.g., author and publication bias that needs to be ac-
counted for when analyzing such literature. However, several SLRs published in
the SE literature have already included peer-reviewed experience reports, writ-
ten by consultants and practitioners, and that rely on experiences rather than
systematic data collection and analysis (e.g.,[8, 31, 9]).
Table 1. Search Strings
Database Search strings # matches
Scopus (“safe” AND “scrum”) OR “Scaled agile framework” 33
Web of Science (“safe” AND “scrum”) OR “Scaled agile framework” 16
ACM (+safe +(scrum)) OR “Scaled agile framework” 6
IEEEXplorer (“safe” AND “scrum”) OR “Scaled agile framework” 8
Total 63
Benefits and Challenges of Adopting the Scaled Agile Framework 5
3.2 Study Selection
Databases: For identifying the peer-reviewed literature we searched scientific
databases by formulating keywords. The number of matches and the search
strings are given in Table 1.
The main source for the grey literature was the official SAFe website [19].
We included the case studies, including backlinks, published there. We used this
source, as it currently is the most notable source of SAFe studies available. The
case studies are based on a defined review and data collection procedure. Orga-
nizations initially answer a questionnaire [17] and thereafter, the “scaled agile
team” reviews the answers and supplements provided by the organizations. The
team contacts organizations for interviews with key members responsible dur-
ing the SAFe implementation to gather background information [18]. Drafts are
written with the help of case study specialists. These are reviewed and approved
by the organizations before being published on the website. The aim is to pub-
lish reports of mature SAFe organizations, i.e., the reports should reflect the
situation no earlier than after 18-24 months into the SAFe implementation [18].
The main benefits of this material are the standard format and questionnaire
used giving some opportunity for cross-case analysis. However, the review process
is likely built not only to guarantee the quality of the published case studies,
but also to ensure that the SAFe framework is put in a good light. Therefore,
the publication bias is extremely strong, and it can be questioned whether case
studies providing negative results would make it through.
Inclusion criteria: We used the following inclusion criteria:
1. Only articles related to the Scaled Agile Framework.
2. Only primary evidence: experience reports, case studies, action research.
3. Publication type: Conference papers, journal papers, workshop papers, white
papers from the Scaled Agile Framework’s homepage.
Search procedure: During the keyword search from scientific databases 63
matches from four databases were identified. After removing the duplicates, we
had a total of 41 papers. These were filtered based on the titles and abstracts
by two authors resulting in eight includes and 33 excludes. After full-text filter-
ing, six scientific papers were selected for the analysis. Finally, we selected five
papers, eliminating one paper, as the same case was published both as a con-
ference and a journal paper: we included only the journal paper. We backward
searched by snowballing through the references of selected five papers and also
forward searched by snowballing the citations. In the forward search we found
one primary study meeting the inclusion criteria. Thus, in total, we included six
primary studies from peer-reviewed sources.
Grey literature: For identifying the grey literature, we manually searched
the SAFe homepage [19] and identified 48 white paper reports. In addition, we
6 A. Putta et al
used backlinks and gathered additional supplements supporting the case studies,
such as downloadable presentations and external links published within each
white paper report (e.g: John Deere [G11, G12, G13, G14, G80]). We gathered
46 case study reports published on the SAFe homepage, nine reports, eleven
presentations and sixteen backlinks. In total, we included 82 reports from the
grey literature.
Search results: In total, we selected 884documents: 82 gathered from the grey
literature and six from the scientific databases. When the same organization was
described in multiple documents, we treated it as one case only if the documents
described an adoption in the same organizational unit. If the adoptions were
separate, e.g., coming from different organizational units of the the same com-
pany (e.g., AVL Gmbh: D6 and D75) they were treated as different cases coming
from the same organization. When the same paper described multiple cases (e.g.,
adoption in different units at different time frames), they were separated as dif-
ferent cases (e.g., Comptel: D3 and D4). Altogether, we got 54 unique SAFe
adoption cases from 52 organizations (see Table 2).
3.3 Analysis
The qualitative data from both peer-reviewed and non-peer reviewed sources
was imported into the coding tool NVivo 11 [20]. We followed the coding guide-
lines presented in [6]. The analysis started with open coding. The open codes
were constantly compared to each other based on similarities and differences
observed between them. They were grouped into higher code categories, called
axial codes. Both axial and open codes formed were thoroughly discussed by the
three authors constantly during the coding process that was performed by the
first author. We identified 23 codes for the benefits and 15 codes for the chal-
lenges of SAFe adoption. We clustered the benefits according the core values of
the SAFe framework: alignment, build in quality, transparency, as they are the
elementary beliefs that are claimed to be of primary importance for the effective
SAFe implementation [16]. We clustered the challenges into organizational and
cultural, roles, practices, as well scaled and distributed. Regarding each benefit
and challenge, we mention the number of cases to express the predominance
across organizations. However, we did not make any other quantitative analysis,
like ranking the benefits and challenges according to the most important and
least important, as even though very interesting, that was not possible with this
qualitative data.
4Due to space limitations, all the primary sources can be found using this link: https:
//figshare.com/s/608559b8dae49bf7b0ad
5D represents a peer reviewed case
Benefits and Challenges of Adopting the Scaled Agile Framework 7
4 Results and Discussion
We identified only six peer-reviewed primary studies on SAFe. The focus ar-
eas of these studies were: assessing the maturity of SAFe adoption [P6], SAFe
self-assessment [P5], the SAFe framework in testing [P2], a real-world example
on key elements of SAFe [P1], the adoption of SAFe in a globally distributed
organization [P3] and one partially focused on the challenges [P4]. Only three
studies focused on the adoption and usage of SAFe [P4, P3, P5]. We identified 47
unique cases (82 documents) from the SAFe homepage. These reports focused
on the adoption reasons, transformation steps, and benefits of SAFe. Neither
the peer-reviewed nor the grey literature had an explicit focus on the adoption
challenges. The grey literature provided deeper insights on the SAFe adoption
and usage compared to the peer-reviewed literature.
A total of 54 unique cases from 52 organizations (Table 2), were identified.
Out of 54 cases, seven6were identified from the peer-reviewed literature7and
47 cases8were identified from the grey literature. Organizations from various
domains have adopted SAFe such as financial (12 cases), software (9 cases),
manufacturing (6 cases), and telecommunications (6 cases). The most prominent
domain was the financial services. Moreover, SAFe has been popular in globally
distributed organizations.
4.1 Benefits of Adopting SAFe
The reported benefits achieved by adopting SAFe are summarized in Table 3.
The most common benefits identified are: transparency (22 cases), alignment
(19 cases), quality (17 cases), time to market (17 cases), predictability (16 cases)
and productivity (15 cases). The benefits marked by a star (*) in Table 3, are
common to both the peer-reviewed and the non-peer reviewed studies.
The core values of SAFe are: build in quality, alignment, program execution
and transparency [16]. A large proportion of the cases mentioned they had gained
these benefits by adopting SAFe. We compared our findings to the 12th state of
agile survey. 29% of respondents of that survey had adopted SAFe [29]. This
survey reported similar benefits of agile in general as our study found regard-
ing SAFe, visibility (66%), productivity (61%), alignment (65%), morale (61%),
predictability (49%), quality (47%), and time to market (62%).
According to our results, practitioners seem to think that SAFe can help to
bring several business benefits, such as improved time to market, and faster and
more frequent deliveries. Surprisingly, none of the business benefits were reported
in the peer-reviewed studies, but the majority of non-peer reviewed studies have
attributed their business success to the SAFe framework. This difference could
be due to the Scaled Agile Team insisting for business benefits, “most impor-
tantly, we look for specific business results, which may include time-to-market,
6* marked in Table 2
7marked with D in Table 2
8marked with C in Table 2
8 A. Putta et al
Table 2. Domain of the Case Organizations
Domain Organizations and cases
Financial Services
(12)
Standard Bank: C1 [G50], Capital One: C2 [G21], Northwestern
Mutual: C3 [G42], Nordea: C4 [G77, G41], SEI: C5 [G5, G6, G7],
Tradestation: C6 [G54], Westpac: C7 [G59, G68], Simcorp*: D1
[P4], Vantiv: C8 [G57], Fannie Mae: C9 [G29], Edge Verve: C10
[G27, G63], Seamless Payments: C11 [G65, G64],
Electronics (4) Thales: C12 [G15], Intel: C13 [G33, G82], Fitbit: C14 [G30],
TomTom: C15 [G53, G60]
Software (9) Sony PlayStation: C16 [G48], Amdocs: C17 [G17], HP: C18
[G31], Anonymous: C19 [G58, G76], BMC Software: C20 [G20,
G78], Mitchells: C21 [G38, G66], Censhare: C22 [G23], Accen-
ture: C23 [G16, G1], X company*: D2 [P6]
Telecommunications
(6)
Comptel*: D3 and D4 [P3], Amdocs: C16 [G17], Swisscom: C24
[G51], Telstra: C25 [G52, G3, G69], Big IT: C26 [G19, G70, G71,
G72, G73, G74]
Retail and
Distribution (3)
Kantar Retail: C27 [G34], Travis Perkins: C28 [G79, G55, G75],
DiscountTire: C29 [G26]
Medical and Pharma
(4)
NHS: C30 [G40], Philips: C31 [G46], Elekta: C32 [G28, G62],
AstraZeneca: C33 [G18]
Media and
Marketing (2)
Valpak: C34 [G56], Sproutland: C35 [G49].
Agriculture (1) LIC: C36 [G37]
Manufacturing (6) Cisco: C37 [G24], TomTom: C14 [G53, G60], SK Hynix: C38
[G47, G61], Lego: C39 [G36, G10], JohnDeere: C40 [G11, G12,
G13, G14, G80], Ocuco*: D5 [P5]
COT’s (1) RMIT University: C41 [G4, G44, G9, G45]
Customer Care (1) CSG: C42 [G25]
Outsourcing (1) Infogain: C43 [G32]
Government IT (2) PoleEmploi: C44 [G8, G43, G2], Australian Postal Services: C45
[G67, G22]
Maritime IT (1) NAPA: C46 [G39, G81]
Automobile (2) AVL Gmbh* (2): D6 [P1] and D7 [P2]
Aviation (1) Air France KLM: C47 [G35]
Benefits and Challenges of Adopting the Scaled Agile Framework 9
productivity, quality, and employee engagement” [18]. Moreover, non-peer re-
viewed studies are more inclined towards presenting the benefits of the SAFe
framework compared to peer-reviewed studies, e.g., only 8 (marked by *) out of
24 benefits were reported by peer-reviewed studies.
According to our results, practitioners clearly think that SAFe has brought
benefits, however, it is also important to look into how the organizations mea-
sured these benefits. Unfortunately, not much information is given related to
this. Only one study in the peer-reviewed literature focused on SAFe metrics
[P5]. Most grey literature cases attributed all the mentioned benefits to the
SAFe adoption. However, it would be interesting to learn, e.g., which practices
of SAFe brought the benefits. Only few cases had done that. Moreover, most of
the benefits mentioned were similar to the general benefits from implementing
agile. In the future it would be interesting to study what the unique benefits
provided by SAFe practices, such as Agile Release Trains, PI planning meetings
and value streams are.
4.2 Challenges of Adopting SAFe
The reported challenges of adopting SAFe are summarized in Table 4. The most
commonly mentioned challenges are: resistance to change (10 cases), moving
away from agile (7 cases), controversies with the framework (6 cases), staffing
roles (5 cases), Agile Release Train challenges (5 cases), PI challenges (4 cases)
and GSD challenges (4 cases). Change resistance, GSD challenges, integration
of the non-development units and test automation challenges found in our re-
view, were also mentioned in the Systematic Literature Review on Challenges
and Success Factors for Large-scale Agile Transformations [8]. Further, change
resistance, could be supported by the results from the 12th state of agile survey,
general resistance [29].
11 out of 15 challenges were common both for the peer-reviewed and non-
peer reviewed studies (marked by * in Table 4). It is notable that the majority of
the peer-reviewed studies reported challenges during the SAFe adoption, while
very few non-peer reviewed studies mentioned challenges.
Even though SAFe is a framework for scaling agile to large enterprises, several
organizations felt they were moving away from agile. This challenge is supported
by the arguments of several “agilists”, for example, Ken Schwaber (co-creator
of Scrum), says that “SAFe is based on RUP, rather than Scrum” [15], Ron
Jeffries (co-founder of XP) sees issues in centralized approaches and planning in
the framework [15] and Stephen Denning (board of directors of Scrum Alliance),
finds SAFe to enforce the horizontal ideology of agile into vertical structures by
saying [7] “they run the risk that the firm will emerge back in the unproductive
vertical world of hierarchical bureaucracy” [7]. Pancholi and Grover [30] argue
that SAFe “murders the spirit of agile development” and claim that SAFe is
sold to large organizations that fear change, but would like to increase their
productivity and reduce defects. According to them the framework portrays an
“agile fairy illusion” [30].
10 A. Putta et al
Both organizations previously using traditional methods, as well as those
having agile already in use, have shown resistance towards accepting SAFe. There
is also a need to draw attention towards the specific challenges of SAFe, such as
the challenges related to PI planning, value streams and agile release trains. Some
faced controversies within the framework itself, like overhead, and story point
normalization. Unlike the benefits, challenges have been mentioned only by 40%
of the cases. Consequently, there is a need for more research into the challenges
of the SAFe framework adoption and usage, as well as ways to overcome those
challenges.
Table 3: Benefits of Adopting SAFe
Benefit Description #
cases
Build in Quality
Quality Improved product quality [G17, G18, G15, G57, G51, G54,
G28, G31, G34, G26, G46, G25, G50, G35]9, higher quality
releases [G78, G31], higher code quality [G5, G7, G49].
17
Defect
Reduction
Drop in the defect rate [G26, G47, G61, G39, G38, G31, G52,
G66, G59, G37, G29], reduction of the patches [G39], increase
in the defect removal efficiency, reduction in quality assurance
defects [G24] and escaped defects [G27, G63], warranty ex-
penses down [G11], bug fixes reduced [G58, G76].
14
Continuous
Improvement*
Focus on continuous improvement [G20, G31, G60, G67, G7,
P4, P5]10.
7
Waste
elimination
Less duplicated work [G36], reduced rework [G36, G15], negli-
gible waste [G53, G60], reduced waste [G79, G73].
5
Alignment
Alignment* Increased alignment teams [G16, G1, G36, G7, G41, G51,
G57, G59, P1, G49], management and development teams
[G23, G28, G41], solved problems of misaligned teams [G36],
alignment of business [G7], customer expectations [G17], align-
ment between IT and business units [G56, G49], client and ven-
dor teams [G32], alignment towards organizational goals [G57],
processes and projects [G18, G2, G46], tools [G46], products
[G21, G27] and priorities [G32].
19
Collaboration* Enhanced collaboration [G9, G15, G80, P3, G6, G45], greater
collaboration between team members [G27, G41], multiple
teams [G41] international teams [G24, P5], diverse working
groups [G15], different units (IT, Business) [G56, G32, G35,
G6], cross site and cross functional collaboration [G28].
14
9Gdenotes non peer-reviewed sources or grey literature sources
10 Pdenotes peer-reviewed sources or scientific literature sources
Benefits and Challenges of Adopting the Scaled Agile Framework 11
Benefit Description #
cases
Dependencies* Improved dependency management [G48, G23, G50, G37, P5]
dependencies across trains are addressed [G43] less dependency
problems [G10, G2].
7
Vision Established shared vision [G34, G23, G24], shared goals [G10]
broader view on company wide strategies [G30, G48].
6
Transparency
Transparency* Enhanced transparency [G67, G47, G61, G39, G48, G15, G53,
G58, G57, G5, G7, G45, G35, G29], process transparency
[G28, G56, G34, G9, G41, G10, G57, G32], cross team de-
pendencies are transparent [G23, P1], transparency in commu-
nication [G33].
22
Visibility* Improved visibility [G73, G56, G30, G82, G56, G33, G34, P4,
G9, G48, G45, G27, G49, G29].
11
Organizational Benefits
Productivity Improved productivity [G20, G25, G26, G67, G54, G50, G27,
G35, G29], increase in productivity across teams and employees
[G34, G56], increased delivery of number of products, variants
and capabilities [G30, G57, G33, G48].
15
Team
Autonomy*
More empowered teams [G67, G20, G23, G56, G10, G41, G5,
G29, G49], self managing teams [G60, G37], self organizing
teams [G56, G6, P5], improved morale [P4, G21, G56], owner-
ship [G53, G35], control of own commitments [G60] and own
code [G60].
13
Engagement Improved employee engagement [G20, G21, G67, G30, G11,
G13, G57, G59, G15, G37, G49], improved employee retention
[G21, G38], decrease in attrition [G57, G34].
12
Employee
Satisfaction*
Improved employee satisfaction [G67, G23, G24, P3, G35],
happier teams [G19, G73, G52, G58, G76], happy employees
[G58, G76].
8
Predictability Greater predictability in the product delivery [G17, G47, G61,
G82, G39, G48, G53, G34, G21, G50, G57, G7, G67, G35, G37,
G49, G29].
16
Feedback Faster feedback from customers [G17, G20, G34, G29] and
greater feedback mechanisms [G57].
5
Business Benefits
Cost Financial benefits [G18], reduced costs [G20, G50], controlled
costs [G31], reduced cost per epic [G34, G27], reduced cost
point size [G15], decreased infrastructural costs [G67], decrease
in delivery costs [G19, G52], reduced quality costs [G16].
11
12 A. Putta et al
Benefit Description #
cases
Frequent
Deliveries
Increase in release frequency [G78, G15, G52, G44, G80, G31],
frequent deliverable’s [G21, G56, G17, G37, G29, G35] and
value [G56], increased deployments [G50].
13
Faster
Deliveries
Faster deliveries [G18, G21, G31, G15, G44], faster feature de-
livery [G78, G24], quick releases [G56, G80, G78], decrease
in feature cycle time [G46, G42] and release cycle down
[G46, G59, G24, G44].
12
On time
delivery
On time delivery [G19, G34, G31, G30, G57, G29], no missing
dates [G56] and schedule slips [G33].
8
Responsiveness More responsive towards market needs [G34] and customer
needs [G57] and decrease in time to respond to customer re-
quests [G38, G34].
4
Time to
market
Improved time to market [G17, G18, G21, G23, G26, G56, G58,
G31, G11, G13, G34, G50, G51, G57, G35, G27, G37, G66].
17
Customer
Satisfaction
Increase in the customer satisfaction [G17, G21, G58, G34,
G51, G57, G6, G5, G67, G2, G35, G27] delighted [G17] and
happy customers [G76].
13
Table 4: Challenges of Adopting SAFe
Challenge Description #
cases
Organizational
and cultural
Change
resistance*
Resistance [G31, G21, G35, G37, G49] towards accepting
change, experiencing change as negative [P3], initial hesitation
from teams [G57], individuals choose to leave [G42], teams re-
ject to take part in ART [G60], initial resistance towards ART
[G21] and reject the common ways of working, strong change
resistance from teams towards lack of SAFe knowledge and
need to change [P3].
10
Mindset* Difficult to implement agile mindset [P4, G72]. 2
Autonomy* Teams lacked autonomy [P3, G21]. 2
Plan driven or
traditional
culture*
Struggles to become iterative from fixed delivery cycles [P2,
G4, G44], struggle to shift from waterfall culture [G42, G44,
G4],
3
Roles
Benefits and Challenges of Adopting the Scaled Agile Framework 13
Challenge Description #
cases
Resistance for
new roles*
Resistance from traditional project managers roles during
adoption of SAFe [G15], challenges with change of roles [P4].
2
Staffing roles Trouble to find the Product Owners [G72, G14, G54, G6, G40]
and challenging to find someone with both technical and in-
dustrial experience [G54]. Product ownership is complex across
universities [G9]. Staffing scrum master was also difficult [G6]
6
Practices
Value streams Defining values streams [G62, G6, G14] 3
First Program
Increment*
Lack of knowledge on importance about PI [P3], chaotic event
[P3, G29], people were uncomfortable during PI [P3] and con-
sidered it as unpleasant [G72], clash of time slots to fix PI
planning [P3], surprises during PI planning [P3], hard to fo-
cus on PI [G10] and ambiguity about time allocation to event
[G34], lack of technical knowledge (how to code) during first
PI [G34], fail to implement effective PSI and find PSI cadence
[G72], teams resist towards PI [P3], feature shaping to PSI was
difficult [G72], logistic challenges [G40], technical dependencies
in PI’s [G8] and management reviews were chaotic [G10].
7
Backlog
Management
and feature
shaping
Feature grooming [G72] and backlog prioritization challenges
[G28], not finding right backlog [G9] feature prioritization did
not involve every one, mostly a solo effect (only product man-
ager) [G59], feature shaping clashed with deployment require-
ments [G72]
4
Test
automation*
Automated unit tests could not be applied to legacy systems
[P2], automated testing was challenging [G4, P5].
3
Controversies
with
framework*
More complex [G10, P6] and risky [P6], confusion with the
way of working [P4], framework as overhead [P3], controver-
sies regarding story point normalization [G72], difficulties with
release management in SAFe framework [G28], separation of
deployment cycle from PSI cycle was challenging [G72], frame-
work not suitable for organizations working on multiple prod-
ucts [G36].
6
Agile Release
Train*
Failure demand of ART’s due to ineffective PSI [G72], integra-
tion of teams with less dependencies into agile release trains
[P1], handling cross team dependencies across the ART’s [P1],
rearrangement of trains for distributed teams [G28], rejection
to take part into ART [G60], difficulties to define ART in or-
ganizational context [G8, G14].
6
14 A. Putta et al
Challenge Description #
cases
Moving away
from agile*
Moving to SAFe feels like moving back to plan driven methods
(such as waterfall and RUP like) [P6, P3, G72, G49], fixed in-
crements [P3], centralized planning [P6], not really incremental
[G9], loss of incremental and iterative development [G60], too
much detail [G10].
7
Scaling and
distribution
Large and
distributed
settings
Challenges of full scaling of agile to whole organization [P4] and
global organization [P2], integrating non development units
such as IT, HR and sales and marketing [P4].
2
GSD* Collaborative planning meeting and critical gatherings were
difficult due to distributed teams [P2], deriving global priorities
[G28], different time zones [G28][G4], scaling agile to global
organization [P2], rearrangement of ART’s was challenging due
to geographic distribution [G28], release planning challenges
due to distributed teams [P6].
4
5 Limitations
This section presents the threats to validity [37] and the steps that have been
taken to mitigate those threats.
Selection bias. This occurs during the selection of primary studies based
on the interpretation of inclusion and exclusion criteria. We mitigated this by
involving all authors in designing the criteria and two researchers filtered the ab-
stracts and titles of peer-reviewed articles independently. Regarding the non-peer
reviewed literature, we included all the case studies published on the homepage
of the Scaled Agile Framework, which mitigated the threat of selection bias.
Subjective bias. This threat occurs during the coding of qualitative data.
Coding was performed by the first author very meticulously. Coding was itera-
tive, and all authors had several discussions during the coding process regarding
the naming of the axial codes and categorization of the open codes into axial
codes. The process is traceable.
Restricted time span. In the database search, we included primary stud-
ies published in the selected databases until November 2017. For the non-peer
reviewed literature, we included all case studies published by May 4th, 2018.
Publication bias. Including grey, non-peer reviewed, literature can be seen
as a limitation. Non-peer reviewed articles usually present positive results [37].
This was also evident from our study, as majority of these cases gave attention
to the benefits of the framework. In addition, the Scaled Agile team reviewed
all case studies reported by the organizations. There might be a possibility for
them to influence the organizations to present only the positive elements of
Benefits and Challenges of Adopting the Scaled Agile Framework 15
the SAFe adoption process. However, out of 82 documents from grey literature
26 documents came directly from the organizations and other online websites,
as additional supplements. These supplements (e.g: presentations) reported the
same information as was published under the case studies on the website. This
threat of bias was partially mitigated by comparing the benefits from peer-
reviewed primary studies, identified in this MLR (6 studies) and the State of
Agile survey [29]. The challenges of adopting SAFe were compared to the findings
of the SLR on challenges of large-scale agile transformations [8] as well as to the
State of Agile survey [29]. However, to establish scientific evidence there is a
strong need for more empirical research on benefits and challenges of SAFe.
Information loss. The codes with only a few quotes and cases (3 cases or less
for benefits, 1 case for challenges) were not reported. The keyword search could
have missed some studies. We mitigated this by going through the references
and citations of all 5 selected studies. We found one additional case study [P5]
from the citations of already selected papers [P3].
6 Conclusions and Future Work
The number of organizations adopting scaling frameworks has increased tremen-
dously during the recent years. A few studies have given insights on agile usage in
large organizations, however, the literature on the adoption and usage of scaling
frameworks has not been systematically reviewed. Moreover, systematic liter-
ature reviews on large-scale agile, have not included the grey literature. This
means that most published information about the scaling frameworks has been
excluded, giving an incomplete picture, as current research literature on them
is very limited. Therefore, we included also grey literature in this multivocal
literature review.
We analyzed 54 peer and non-peer reviewed cases. The most salient benefit
categories were: transparency, alignment, productivity, predictability and time to
market. The most frequently mentioned challenges are: change resistance, con-
troversies with the framework and moving away from agile. The most important
difference between the peer-reviewed and grey literature was the bias in report-
ing benefits, especially with respect to business benefits received from the usage
of the framework. These benefits were mentioned only in the grey literature.
This emphasizes the need for validation of the claimed benefits reported in the
grey literature. The majority of the challenges were common for both the peer-
reviewed and grey literature.
Apart from the challenges related to scaling agile, SAFe has brought in new
challenges with respect to practices such as PI planning, value streams, and Ag-
ile Release Trains. Empirical research on how the SAFe framework is addressing
the existing challenges, that have been reported in the agile in the large lit-
erature, could be interesting for practitioners. Moreover, finding solutions for
the challenges reported in this MLR, would help organizations to address these
challenges.
16 A. Putta et al
We identified only six peer reviewed primary studies on SAFe since the intro-
duction of the framework (year 2011). Literature lacks in-depth primary studies
on the usage and adoption of SAFe. Some of the non-peer reviewed cases pub-
lished at the SAFe home page had deep insights on the rationale behind the SAFe
adoption, transformation steps, implementation of practices, as well as the ben-
efits of the adoption. Unfortunately, both peer-reviewed and grey literature lack
extensive information on challenges and the negative traits of SAFe in large en-
terprises, as there likely is an inherent positive bias in the cases published at the
SAFe home page. Hence, it is crucial to conduct more in-depth primary studies
on SAFe adoptions to establish scientific evidence on the SAFe framework usage
in large scale.
References
1. Alqudah, M., Razali, R.: A review of scaling agile methods in large software devel-
opment. International Journal on Advanced Science, Engineering and Information
Technology 6(6), 828–837 (2016)
2. Alsaqaf, W., Daneva, M., Wieringa, R.: Quality requirements in large-scale dis-
tributed agile projects–a systematic literature review. In: International Working
Conference on Requirements Engineering: Foundation for Software Quality. pp. 219–
234. Springer (2017)
3. Ambler, S.W., Lines, M.: Disciplined agile delivery: A practitioner’s guide to agile
software delivery in the enterprise. IBM Press (2012)
4. Ampatzoglou, A., Ampatzoglou, A., Chatzigeorgiou, A., Avgeriou, P.: The financial
aspect of managing technical debt: A systematic literature review. Information and
Software Technology 64, 52–73 (2015)
5. Boehm, B., Turner, R.: Management challenges to implementing agile processes in
traditional development organizations. IEEE software 22(5), 30–39 (2005)
6. Corbin, J.M., Strauss, A.L.: Basics of qualitative research: techniques and proce-
dures for developing grounded theory. Sage Publications, Inc., Los Angeles, Calif.,
3rd ed edn. (2008), http://www.loc.gov/catdir/toc/ecip0725/2007034189.html
7. Denning, S.: Agile: its time to put it to use to manage business complexity. Strategy
& Leadership 43(5), 10–17 (2015)
8. Dikert, K., Paasivaara, M., Lassenius, C.: Challenges and success factors for large-
scale agile transformations: A systematic literature review. Journal of Systems and
Software 119, 87–108 (2016)
9. Dyb˚a, T., Dingsøyr, T.: Empirical studies of agile software development: A system-
atic review. Information and software technology 50(9-10), 833–859 (2008)
10. Garousi, V., Felderer, M., M¨antyl¨a, M.V.: The need for multivocal literature re-
views in software engineering: complementing systematic literature reviews with
grey literature. In: Proceedings of the 20th International Conference on Evaluation
and Assessment in Software Engineering. p. 26. ACM (2016)
11. Garousi, V., Felderer, M., M¨antyl¨a, M.V.: Guidelines for including the grey liter-
ature and conducting multivocal literature reviews in software engineering. arXiv
preprint arXiv:1707.02553 (2017)
12. Garousi, V., M¨antyl¨a, M.V.: When and what to automate in software testing?
a multi-vocal literature review. Information and Software Technology 76, 92–117
(2016)
Benefits and Challenges of Adopting the Scaled Agile Framework 17
13. Glass, R.L.: Software Creativity 2.0. developer.* Books (2006)
14. Gustavsson, T.: Assigned roles for inter-team coordination in large-scale agile de-
velopment: a literature review. In: Proceedings of the XP2017 Scientific Workshops.
p. 15. ACM (2017)
15. van Haaster, K.: Agile in-the-large: Getting from paradox to paradigm
16. Inc., S.A.: Core Values. http://www.scaledagileframework.com/safe-core-
values/
17. Inc., S.A.: Questionnaire for safe adopters. https://www.scaledagile.com/
resources/submit-a-case-study/
18. Inc, S.A.: Review process. https://www.scaledagile.com/case-study- faqs/
19. Inc, S.A.: Scaled Agile Framework. http://www.scaledagileframework.com/case-
studies/
20. International, Q.: Coding Tool for Qualtitaive Analysis. http://
www.qsrinternational.com/nvivo/support-overview/downloads
21. Keele, S., et al.: Guidelines for performing systematic literature reviews in software
engineering. In: Technical report, Ver. 2.3 EBSE Technical Report. EBSE. sn (2007)
22. Khalid, H., Ahmed, M., Sameer, A., Arif, F.: Systematic literature review of agile
scalability for large scale projects. International Journal of Advanced Computer
Science and Applications (IJACSA) 6(9), 63–75 (2015)
23. Kuhrmann, M., Diebold, P., M¨unch, J., Tell, P., Garousi, V., Felderer, M., Trek-
tere, K., McCaffery, F., Linssen, O., Hanser, E., et al.: Hybrid software and system
development in practice: waterfall, scrum, and beyond. In: Proceedings of the 2017
International Conference on Software and System Process. pp. 30–39. ACM (2017)
24. Larman, C., Vodde, B.: Practices for scaling lean & Agile development: large, mul-
tisite, and offshore product development with large-scale scrum. Pearson Education
(2010)
25. Leffingwell, D.: Scaling software agility: best practices for large enterprises. Pearson
Education (2007)
26. Moe, N.B., Dingsøyr, T.: Emerging research themes and updated research agenda
for large-scale agile development: a summary of the 5th international workshop at
xp2017. In: Proceedings of the XP2017 Scientific Workshops. p. 14. ACM (2017)
27. Moe, N.B., Olsson, H.H., Dingsøyr, T.: Trends in large-scale agile development: A
summary of the 4th workshop at xp2016. In: Proceedings of the Scientific Workshop
Proceedings of XP2016. p. 1. ACM (2016)
28. Myrbakken, H., Colomo-Palacios, R.: Devsecops: A multivocal literature review.
In: International Conference on Software Process Improvement and Capability De-
termination. pp. 17–29. Springer (2017)
29. One, V.: State of Agile Survey. https://explore.versionone.com/state-of-
agile/versionone-12th-annual-state- of-agile-report
30. Pancholi, A., Grover, S.: Scaled agile framework: A blight. International Journal
of Innovative Research and Development 3(5) (2014)
31. Rafi, D.M., Moses, K.R.K., Petersen, K., M¨antyl¨a, M.V.: Benefits and limitations
of automated software testing: Systematic literature review and practitioner survey.
In: Proceedings of the 7th International Workshop on Automation of Software Test.
pp. 36–42. IEEE Press (2012)
32. Razavi, A.M., Ahmad, R.: Agile development in large and distributed environ-
ments: A systematic literature review on organizational, managerial and cultural
aspects. In: Software Engineering Conference (MySEC), 2014 8th Malaysian. pp.
216–221. IEEE (2014)
33. Rolland, K.H., Fitzgerald, B., Dingsøyr, T., Stol, K.J.: Problematizing agile in the
large: alternative assumptions for large-scale agile development (2016)
18 A. Putta et al
34. Tom, E., Aurum, A., Vidgen, R.: An exploration of technical debt. Journal of
Systems and Software 86(6), 1498–1516 (2013)
35. Uluda˘g, ¨
O., Kleehaus, M., Xu, X., Matthes, F.: Investigating the role of architects
in scaling agile frameworks. In: Enterprise Distributed Object Computing Confer-
ence (EDOC), 2017 IEEE 21st International. pp. 123–132. IEEE (2017)
36. Vaidya, A.: Does dad know best, is it better to do less or just be safe? adapting
scaling agile practices into the enterprise. PNSQC. ORG pp. 1–18 (2014)
37. Zhou, X., Jin, Y., Zhang, H., Li, S., Huang, X.: A map of threats to validity
of systematic literature reviews in software engineering. In: Software Engineering
Conference (APSEC), 2016 23rd Asia-Pacific. pp. 153–160. IEEE (2016)
Non-Peer-Reviewed Sources
1. Andrew Ball, Ajay Nair, M.H.: Accenture Case Study. http://
www.scaledagileframework.com/wp-content/uploads/delightful-downloads/
2018/09/Key Accenture Learning on Scaled and Distributed Agile August-
18-for-SAFe.pdf (2018), [Online; accessed 09-June-2018]
2. Auret, C.: PoleEmploi Case Study. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/12/Cambridge-conf-
V1.6 slides.pdf (2018), [Online; accessed 09-June-2018]
3. Blog, P.A.: Telstra Case Study. http://www.prettyagile.com/2013/05/the-
power-of-haka.html (2018), [Online; accessed 09-June-2018]
4. Catherine Haugh, Terence Le Grange, C.S.: RMIT Case Study. http://
www.scaledagileframework.com/wp-content/uploads/delightful-downloads/
2018/09/1434405237wpdm 1410-HEUG-Going-Agile- Presentation.pdf (2018),
[Online; accessed 09-June-2018]
5. Crudup, R.F.: SEI Case Study. http://www.scaledagileframework.com/wp-
content/uploads/delightful-downloads/2018/09/SEI Agile Case Study.pdf
(2018), [Online; accessed 09-June-2018]
6. Crudup, R.F.: SEI Case Study. http://www.scaledagileframework.com/wp-
content/uploads/delightful-downloads/2018/09/SEI Agile Case Study.pdf
(2018), [Online; accessed 09-June-2018]
7. Cudup, R.F.: SEI Case Study. https://capital-markets.cioreview.com/
cxoinsight/transformation-moving-a-large- organization-to-agile-
development--nid-13265- cid-8.html (2018), [Online; accessed 09-June-2018]
8. Ccile Auret, Jrme Froville, M.L.: Pole Emploi Case Study. http://
www.scaledagileframework.com/wp-content/uploads/delightful-downloads/
2018/09/Pole-Emploi-SAFe-Case- Study-V1.1.pdf (2018), [Online; accessed
09-June-2018]
9. Edu, R.: RMIT Case Study. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/09/
1434405111wpdm PRESENTATION-Alliance-2014-StAART@RMIT.pdf (2018), [On-
line; accessed 09-June-2018]
10. Henrik Kniberg, E.T.B.: Lego Case Study. http://
www.scaledagileframework.com/wp-content/uploads/delightful-downloads/
2018/09/LEGO update.pdf (2018), [Online; accessed 09-June-2018]
11. Holdorf, C.: John Deere Case Study. http://www.scaledagileframework.com/
john-deere-case-study/ (2018), [Online; accessed 09-June-2018]
12. Holdorf, C.: John Deere Case Study. http://www.scaledagileframework.com/
john-deere-case-study- part-1/ (2018), [Online; accessed 09-June-2018]
Benefits and Challenges of Adopting the Scaled Agile Framework 19
13. Holdorf, C.: John Deere Case Study. http://www.scaledagileframework.com/
john-deere-case-study- part-1/ (2018), [Online; accessed 09-June-2018]
14. Holdorf, C.: John Deere Case Study. http://www.scaledagileframework.com/
john-deere-case-study- part-3/ (2018), [Online; accessed 09-June-2018]
15. Inc, S.A.: Thales Case Study. http://www.scaledagileframework.com/thales-
case-study/ (2018), [Online; accessed 09-June-2018]
16. Inc, S.A.: Accenture Case Study. http://www.scaledagileframework.com/
accenture-case-study/ (2018), [Online; accessed 09-June-2018]
17. Inc, S.A.: Amdocs Case Study. http://www.scaledagileframework.com/amdocs-
case-study (2018), [Online; accessed 09-June-2018]
18. Inc, S.A.: AstraZeneca Case Study. http://www.scaledagileframework.com/
astra-zeneca-case-study/ (2018), [Online; accessed 09-June-2018]
19. Inc., S.A.: Big It Shop Case Study. http://www.scaledagileframework.com/big-
it-shop-down-under- case-study/ (2018), [Online; accessed 09-June-2018]
20. Inc., S.A.: BMC Case Study. http://www.scaledagileframework.com/bmc-
software-case-study/ (2018), [Online; accessed 09-June-2018]
21. Inc, S.A.: Capital One Case Study. http://www.scaledagileframework.com/
capital-one-case-study/ (2018), [Online; accessed 09-June-2018]
22. Inc., S.A.: Case Study Australian Post. http://www.scaledagileframework.com/
case-study-australia-post/ (2018), [Online; accessed 20-November-2018]
23. Inc, S.A.: Censhare Case Study. http://www.scaledagileframework.com/case-
study-censhare/ (2018), [Online; accessed 09-June-2018]
24. Inc, S.A.: Cisco Case Study. http://www.scaledagileframework.com/cisco-
case-study/ (2018), [Online; accessed 09-June-2018]
25. Inc., S.A.: CSG Case Study. http://www.scaledagileframework.com/csg-case-
study-2/ (2018), [Online; accessed 09-June-2018]
26. Inc., S.A.: Discount Tire Case Study. http://www.scaledagileframework.com/
discount-tire-case-study/ (2018), [Online; accessed 09-June-2018]
27. Inc, S.A.: EdgeVerve Case Study. https://www.scaledagileframework.com/case-
study-edgeverve-systems/ (2018), [Online; accessed 28-March-2018]
28. Inc., S.A.: Elekta Case Study. http://www.scaledagileframework.com/elekta-
case-study/ (2018), [Online; accessed 09-June-2018]
29. Inc., S.A.: Fannie Mae Case Study. https://www.scaledagile.com/case-study/
fannie-mae/ (2018), [Online; accessed 09-June-2018]
30. Inc, S.A.: FitBit Case Study. http://www.scaledagileframework.com/fitbit-
case-study/ (2018), [Online; accessed 09-June-2018]
31. Inc, S.A.: HPE Case Study. http://www.scaledagileframework.com/hpe-case-
study/ (2018), [Online; accessed 09-June-2018]
32. Inc, S.A.: Infogain Case Study. http://www.scaledagileframework.com/
infogain-case-study/ (2018), [Online; accessed 09-June-2018]
33. Inc, S.A.: Intel Case study. http://www.scaledagileframework.com/case-study-
intel/ (2018), [Online; accessed 09-June-2018]
34. Inc, S.A.: Kantar Retail Case Study. http://www.scaledagileframework.com/
kantar-retail-case-study/ (2018), [Online; accessed 09-June-2018]
35. Inc, S.A.: KLM Air France Case Study. https://www.scaledagileframework.com/
case-study-air-france- klm/ (2018), [Online; accessed 28-March-2018]
36. Inc., S.A.: Lego Case Study. http://www.scaledagileframework.com/lego-case-
study/ (2018), [Online; accessed 09-June-2018]
37. Inc, S.A.: LIC Case Study. https://www.scaledagileframework.com/case-study-
lic/ (2018), [Online; accessed 28-March-2018]
20 A. Putta et al
38. Inc., S.A.: Mitchell Case Study. http://www.scaledagileframework.com/
mitchell-international-case-study/ (2018), [Online; accessed 09-June-2018]
39. Inc., S.A.: NAPA Case Study. http://www.scaledagileframework.com/napa-
case-study/ (2018), [Online; accessed 09-June-2018]
40. Inc, S.A.: NHS Case Study. http://www.scaledagileframework.com/nhs- case-
study (2018), [Online; accessed 09-June-2018]
41. Inc., S.A.: Nordea Case Study. http://www.scaledagileframework.com/nordea-
case-study-2/ (2018), [Online; accessed 09-June-2018]
42. Inc, S.A.: Northwestern Mutual Case Study. http://
www.scaledagileframework.com/northwestern-mutual-case-study/ (2018),
[Online; accessed 09-June-2018]
43. Inc, S.A.: Pole Emploi Case Study. http://www.scaledagileframework.com/pole-
emploi-case-study/ (2018), [Online; accessed 09-June-2018]
44. Inc., S.A.: RMIT Case Study. http://www.scaledagileframework.com/rmit-
case-study-2/ (2018), [Online; accessed 09-June-2018]
45. Inc., S.A.: RMIT Case Study. https://www.slideshare.net/Cont3xtMatt3rs/
rmit-safe-case-study- the-art-of- accelerating-agility- 52956873 (2018),
[Online; accessed 09-June-2018]
46. Inc, S.A.: Royal Philips Case Study. http://www.scaledagileframework.com/
royal-phillips-case-study/ (2018), [Online; accessed 09-June-2018]
47. Inc, S.A.: SK Hynix Case Study. http://editor.scaledagileframework.com/
hynix-case-study/ (2018), [Online; accessed 09-June-2018]
48. Inc, S.A.: Sony Play Station Case Study. http://www.scaledagileframework.com/
sony-playstation-network-case- study/ (2018), [Online; accessed 09-June-2018]
49. Inc, S.A.: Sproutloud Case Study. https://www.scaledagileframework.com/case-
study-sproutloud/ (2018), [Online; accessed 08-June-2018]
50. Inc, S.A.: Standard Bank Case Study. http://www.scaledagileframework.com/
standard-bank-case-study/ (2018), [Online; accessed 09-June-2018]
51. Inc, S.A.: Swisscom Case Study. http://www.scaledagileframework.com/
swisscom-case-study (2018), [Online; accessed 09-June-2018]
52. Inc., S.A.: Telstra Case Study. http://www.scaledagileframework.com/telstra-
case-study/ (2018), [Online; accessed 09-June-2018]
53. Inc, S.A.: Tomtom Case Study. http://www.scaledagileframework.com/tomtom-
case-study/ (2018), [Online; accessed 09-June-2018]
54. Inc., S.A.: Tradestation Case Study. http://www.scaledagileframework.com/
tradestation-technologies-case-study/ (2018), [Online; accessed 09-June-
2018]
55. Inc., S.A.: Travis Perkins Case Study. http://www.scaledagileframework.com/
travis-perkins-case-study- 2/ (2018), [Online; accessed 09-June-2018]
56. Inc., S.A.: Valpak Case Study. http://www.scaledagileframework.com/wp-
content/uploads/2013/06/Final-Valpak-Case-Study- Short.pdf (2018), [On-
line; accessed 09-June-2018]
57. Inc, S.A.: Vantiv Case Study. http://www.scaledagileframework.com/vantiv-
case-study (2018), [Online; accessed 09-June-2018]
58. Inc, S.A.: Waterfall to Agile Case Study. http://www.scaledagileframework.com/
from-waterfall-to-enterprise- agility-case-study/ (2018), [Online; accessed
09-June-2018]
59. Inc, S.A.: Westpac Case Study. http://www.scaledagileframework.com/westpac-
case-study/ (2018), [Online; accessed 09-June-2018]
Benefits and Challenges of Adopting the Scaled Agile Framework 21
60. Janisse, J.: Tomtom Case Study. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/09/Driving-SAFe- at-
Tomtom1.pdf (2018), [Online; accessed 09-June-2018]
61. Lam, J.: SK Hynix Case Study. http://editor.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/09/SK-Hynix- Case-Study-
November-16-2015-Final.pdf (2018), [Online; accessed 09-June-2018]
62. Lars Gusch, P.H.: Elekta Case Study. http://
www.scaledagileframework.com/?ddownload=35393 (2018), [Online; accessed
09-June-2018]
63. Limited, E.S.: Edge Verve Case Study. https://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/05/Infosys-EV- Biz-
Agility-Casestudy.pdf (2018), [Online; accessed 09-June-2018]
64. Mikael Lundgren, T.P.: Seamless Payments Case Study. http://
www.scaledagileframework.com/seamless-case-study/ (2018), [Online; ac-
cessed 09-June-2018]
65. Mikael Lundgren, T.P.: Seamless Payments Case Study . https://www.infoq.com/
articles/downscaling-SAFe (2018), [Online; accessed 09-June-2018]
66. North, R.: Mitchell Case Study. http://www.scaledagileframework.com/wp-
content/uploads/2013/06/Mitchell-Case-Study-PPT- edited-down-from-
MitchellsBigLEAP-NORTH-0604-11AM.pptx (2018), [Online; accessed 09-June-
2018]
67. post, M.A.: Case Study Australian Post. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/11/Australia-Post- SAFe-
Case-Study-v1.0.pdf (2018), [Online; accessed 09-June-2018]
68. Pretty, C.: Westpac Case Study. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/09/AgileAus2016-
BankingOnAQuickStart-FINALwithNotes1.pdf (2018), [Online; accessed 09-
June-2018]
69. Em champbell Pretty, M.R.: Telstra Case Study. https://www.slideshare.net/
ScaledAgile/agile-aus-2013final (2018), [Online; accessed 09-June-2018]
70. Richards, M.: Big It Shop Case Study. http://www.agilenotanarchy.com/2013/
01/scaled-agile-framework-applied- part-15.html (2018), [Online; accessed
09-June-2018]
71. Richards, M.: Big It Shop Case Study. http://www.agilenotanarchy.com/2013/
02/scaled-agile-framework-applied- 25.html (2018), [Online; accessed 09-June-
2018]
72. Richards, M.: Big It Shop Case Study. http://www.agilenotanarchy.com/2013/
02/scaled-agile-framework-35- program-level.html (2018), [Online; accessed
09-June-2018]
73. Richards, M.: Big It Shop Case Study. http://www.agilenotanarchy.com/2013/
03/scaled-agile-framework-applied- 55.html (2018), [Online; accessed 09-June-
2018]
74. Richards, M.: Big It Shop Case Study. http://www.agilenotanarchy.com/2013/
03/scaled-agile-framework-applied- 45-in.html (2018), [Online; accessed 09-
June-2018]
75. Rossi, B.: Travis Perkins Case Study. http://www.information-age.com/story-
travis-perkinss-agile-it- transformation-123457817/ (2018), [Online; ac-
cessed 09-June-2018]
76. Rutzen, A.: Waterfall to Agile Case Study. http://
www.scaledagileframework.com/wp-content/uploads/delightful-downloads/
22 A. Putta et al
2018/09/1434405855wpdm Waterfall to Agile A Case-Study.pdf (2018), [Online;
accessed 09-June-2018]
77. Scaled Agile Inc., I.J.I.: Nordea Case Study. http://
www.scaledagileframework.com/wp-content/uploads/delightful-downloads/
2018/09/Nordea-Case-Study.pdf (2018), [Online; accessed 09-June-2018]
78. Software, R.: BMC Case Study. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/09/Case Study BMC10.pdf
(2018), [Online; accessed 09-June-2018]
79. software, R.: Travis Perkins Case Study. http://www.scaledagileframework.com/
wp-content/uploads/delightful-downloads/2018/09/Travis-Perkins- Case-
Study.pdf (2018), [Online; accessed 09-June-2018]
80. Thibodeau, P.: John Deere Case Study. https://www.computerworld.com/
article/2500298/app-development/john-deere-plows- into-agile.html
(2018), [Online; accessed 09-June-2018]
81. Vaje, T.: NAPA Case Study. http://www.scaledagileframework.com/?ddownload=
35408 (2018), [Online; accessed 09-June-2018]
82. Weltsch-Cohen, Y.: MDO Intel Case Study. http://
www.scaledagileframework.com/wp-content/uploads/2014/09/Implementing-
SAFe-MDO-test-case.pdf (2018), [Online; accessed 09-June-2018]
Peer-Reviewed Sources
1. Brenner, R., Wunder, S.: Scaled agile framework: Presentation and real world ex-
ample. In: Software Testing, Verification and Validation Workshops (ICSTW), 2015
IEEE Eighth International Conference on. pp. 1–2. IEEE (2015)
2. Korosec, R., Pfarrhofer, R.: Supporting the transition to an agile test matrix. In:
Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International
Conference on. pp. 1–2. IEEE (2015)
3. Paasivaara, M.: Adopting safe to scale agile in a globally distributed organization.
In: Proceedings of the 12th International Conference on Global Software Engineer-
ing. pp. 36–40. IEEE Press (2017)
4. Pries-Heje, J., Krohn, M.M.: The safe way to the agile organization. In: Proceedings
of the XP2017 Scientific Workshops. p. 18. ACM (2017)
5. Razzak, M.A., Noll, J., Richardson, I., Canna, C.N., Beecham, S.: Transition from
plan driven to safe R
: Periodic team self-assessment. In: International Conference
on Product-Focused Software Process Improvement. pp. 573–585. Springer (2017)
6. Turetken, O., Stojanov, I., Trienekens, J.J.: Assessing the adoption level of scaled
agile development: a maturity model for scaled agile framework. Journal of Soft-
ware: Evolution and Process 29(6) (2017)
