Scrum Maturity Model

Abstract

Within the agile development methodologies context, the topic of client relationship management is strongly focused, mainly due to the importance of collaboration between the development team and its clients. Most clients avoid or are unable to develop a close cooperation with vendor organizations, since it requires a motivation and close participation among key stakeholders in the development processes within and correct usage of the adopted software development methodology. Hence, software development projects fail and become unsuccessful because of this lack of communication. In order to increase the rate of successful projects, this paper will present the journey of the validation process for this roadmap to lead and aid software vendor organizations improve their development processes, concentrating mainly on the client's role throughout the process. This concept is called Scrum Maturity Model; therefore, our main goal is to validate this concept with organizations that use Scrum agile methodology as their main development process, which turns out to be an viable approach to reduce the rate failed development projects.

Full text

Scrum Maturity Model
Validation for IT organizations‟ roadmap to develop software centered on the client role
Alexandre Yin
Departamento Engenharia Informática (DEI)
Instituto Superior Técnico (IST)
Lisboa, Portugal
alexandre.yin@ist.utl.pt
Soraia Figueiredo; Miguel Mira da Silva
INESC . INOV
Instituto Superior Técnico (IST)
Lisboa, Portugal
soraia.figueiredo@inov.pt; mms@ist.utl.pt
Abstract—Within the agile development methodologies context,
the topic of client relationship management is strongly focused,
mainly due to the importance of collaboration between the
development team and its clients. Most clients avoid or are
unable to develop a close cooperation with vendor organizations,
since it requires a motivation and close participation among key
stakeholders in the development processes within and correct
usage of the adopted software development methodology. Hence,
software development projects fail and become unsuccessful
because of this lack of communication. In order to increase the
rate of successful projects, this paper will present the journey of
the validation process for this roadmap to lead and aid software
vendor organizations improve their development processes,
concentrating mainly on the client’s role throughout the process.
This concept is called Scrum Maturity Model; therefore, our
main goal is to validate this concept with organizations that use
Scrum agile methodology as their main development process,
which turns out to be an viable approach to reduce the rate failed
development projects.
Keywords-development methodologies; agile methodologies;
scrum development methodology; maturity model; action research
I. INTRODUCTION
According to a CHAOS Report [1], about 70% of IT
development projects fail to deliver functional software, mostly
due to a poor communication between stakeholders, who play
key roles in the development process. This problem of human
factors in software development collaboration is also
highlighted in these three following papers [2][3][4][5].
The fact that most clients spend an extremely small amount
of time and effort working closely with the software vendor
organization, that develops the solution, goes against the Agile
Manifesto values [6], which are the foundations for a
successful agile oriented development.
The failure of Information Technology (IT) projects caused
by mediocre software requirements engineering and other
human/client factors is a highly researched theme among
professionals and scholars. Therefore, this paper intends to
provide a different insight about the current issues concerning
this topic [7] [8][9][10].
The main concern that induced this research was precisely
the dilemma mentioned above: lack of cooperation among
stakeholders involved in an IT development project, focusing
on the type of communication between the development team
and the client. This problem in communication can result from:
(1) Human factors and resistance to changes; (2) Distance that
separates both vendors and clients or; (3) Inexistence of a
commitment that follows the definition of a contract of
collaboration.
Generally, both clients and software development
organization teams may fear and avoid the adoption of new
methods of collaboration with a new team [10]. This harms the
partnership between the two, thus resulting in inadequate
requirements engineering emphasized by agile methodologies,
which will, eventually, lead to an unsuccessful project.
Concerning human behavior, the distance that separates the
vendor organization and the client challenges the
accomplishment of a fluent and successful cooperation [11].
Apart from this exact physical distance, that hardens the
communication and occasionally blocks the possibility of face-
to-face meetings, a cultural distance must also be considered,
since this aspect may bring a negative impact, such as cultural
clashes, to the performance of the collaboration and influence
the project as a whole [10].
Another cause of this problem is the inexistence of
highlighted goals, such as market competition, which will
motivate all stakeholders to improve their processes and
maximize the outputs. According to a survey made by Gartner
[13], agile methodologies could use a maturity model as a
roadmap and market differential, so software development
organizations might explore their processes and reach higher
levels of maturity. Moreover, a paper from Software
Engineering Institute (SEI) [14] reveals that Capability
Maturity Model Integration (CMMI) can coexist with agile
methodologies and enhance these software development
organizations [15].
This paper will focus on the changes from the previous
proposal [16] and recent evaluation processes of the solution
for this lack of collaboration, usually, between vendor
organizations and clients. Moreover, it will conceive a roadmap
for improvement in order to create successful IT development
projects. Since Scrum development methodology emphasizes
such collaboration, the solution shall be molded as a roadmap
in the form of a maturity model so as to achieve the goal of this
paper.

Note that this topic of maturity models and other IT
governance frameworks on agile methodologies a highly
polemic among the agile community. Nevertheless, IT
governance mechanisms are necessary and welcomed in
organizations which are underproductive, and, thus, hold the
major slice of failed projects [14].
The chosen research method was Action Research (AR)
due to its success in various academic investigations in the
Information Systems area and for allowing the researcher to
interfere and observe introduced modifications on the studied
environment. AR is comprised by a five stages cycle [17]: (1)
Diagnosis – problem identification; (2) Action Planning –
planning and research phase to prepare the experiment and
alternative actions; (3) Action – implementation of planned
actions, introduction of changes and analysis of the outputs on
the environment; (4) Evaluation – it is determined if the
outcomes are expected or against odds and assures that
introduced actions are the only reason for the obtained success;
(5) Specifying Learning – Identify general findings.
Note that AR is carried out by individuals who are
interested parties in the research. This fact has led to criticisms
of the validity of the research process, with accusations of
inevitable researcher bias in data gathering and analysis. The
justification for AR counters this criticism by suggesting that
it is impossible to access practice without involving the
practitioner. Practice is action informed by values and aims
which are not fully accessible from the outside. The
practitioner may not even be wholly aware of the meaning of
his or her values until he or she tries to embody them in her
action.
Nevertheless, there are some limitations with this research
methodology, namely: the unfamiliarity with research methods
and the representations of the process of action research may
confuse, rather than enlighten.
As stated, this paper continues our previous research,
hence, the first two cycles of action research were already
previously applied. This paper will mainly focus on the
changes to the proposal, based on past learning, and iterate
more cycles of action research in order to achieve stronger
validation of the proposal.
Before the presentation of the improved proposition, a brief
introduction and review of the related work in this area of
research shall be developed in Sections II and III. After, the
changes in the proposition are detailed in Section VI; in the
next section (Section V) the results of newer and various
practical experimentations of the proposition will be presented.
Afterwards, in Section VI, the main lessons learned shall be
analyzed. Finally, Section VII will conclude with the summary
of this investigation, relating all mentioned topics as a whole.
In this section, some future works and approaches are given to
continue the research.
II. RELATED WORK
This section intends to make a brief review of the related
work in the field of agile development study.
A. Agile Methodologies
The origins of Agile methodologies are deeply connected
with the concepts of iterative and incremental development.
There were several ideas concerning the agile concept, hence
an Agile Manifesto [6] was established.
The set of values and inherent principles listed on this
Manifesto stress the importance of the clients‟ presence in
order to obtain a better collaboration outcome, working
software as the main goal and agility when facing a sudden
change in requirements [18][19].
Since this approach requires a high cooperation level
between the client and the development team, mainly through
face-to-face meetings, it has the drawback of being partially
obsolete in the current market, in which an ascending number
of projects are developed at a distance [20][21][22].
B. Scrum
Scrum is an agile methodology to manage development
projects through an iterative and incremental method
[23][24][25]. It is divided into three main key roles: (1) Scrum
Master – individual who is responsible for the Scrum process
and its correct usage maximizing its benefits; also known as the
facilitator of Scrum team; (2) Product Owner – individual who
is accountable for the alignment of the development and
business goals definition, and; (3) Team – team that is in
charge of delivering the product. A team comprises 5 to 9
members with cross-functional skills, who are self-organized
and self-led.
This methodology identifies four objects that are operated
by the Scrum team throughout the development cycle: (1)
Product Backlog – a prioritized list of everything necessary to
conclude the product; (2) Sprint Backlog – a list of tasks to
perform during a sprint, i.e., an up to four weeks development
iteration to introduce parts of the Product Backlog into working
software; (3) Release Burndown Charts – charts that show the
progress of the project over time, and; (4) Sprint Burndown
Charts – charts that show the progress of the sprint over time.
The interaction of the roles maneuvering these objects is set
for the following meeting: (1) Release Planning Meeting –
Scrum team gathers and fills in the Product Backlog; (2) Sprint
Planning Meeting – development team and client closely
discuss matters and define the goals for the next sprint; (3)
Daily Scrum – a brief meeting for developers to identify
personal issues and possible improvements in methodology
usage; (4) Sprint Review – demonstration of the working
software to the client and stakeholders; (5) Sprint Retrospective
– team performs a self-examination regarding the last sprint in
order to seek improvements on their use of Scrum
Methodology and collaboration in general.
Scrum methodology is an iterative and incremental
development methodology. The phase for planning and system
architecture takes place in Release Planning Meeting, while the
sprints are comprised by Sprint Planning Meetings, Daily
Scrum, Sprint Review and Sprint Retrospective.
Although Scrum has a wide definition of concepts, that,
when applied, may allow agile software development, it cannot
guarantee the success of IT projects. This methodology

emphasizes close collaboration between development teams
and their clients; still, most of the time this does not happen
and, thus, a supplementary solution to complement this
imperfection is needed.
C. Modified Agile
Modified Agile is an agile development methodology that
results from the analysis of the flaws in the Agile Manifesto [6]
opposing to distant outsourcing environment [11].
The main problems identified concerning this matter were
the poor communication among participants of the IT projects
and the exhaustive documentation needed for contract
negotiation. All other values and principles mentioned in Agile
Manifesto remain feasible in a distant outsourcing context.
Figure 1. Modified Agile communcation model proposal [11].
The solution recommended by the author of this paper is an
authentic communication model and team composition
structure, which will enhance the communication between
clients and developers and reduce the negative effects derived
from the distance factor that leads to a loss of knowledge.
In Figure 1, the introduction of two specific roles is
emphasized: (1) Coordination – an individual from the client-
side, who ensures the maximization of development outputs by
assigning the most important business goals to be developed as
a priority; (2) Ambassador – individual from the development
team-side who makes sure that the product developed is
aligned according to the customer‟s needs and wills. These two
roles must work closely as a formal communication channel,
while team members from both development and the client-
side might communicate among themselves through an
informal channel
Although this distributed agile concept is broadly used with
several case studies proving its success, there are also many
failed IT projects due to human factors and inadequate
collaboration between clients and vendor organizations
[26][27].
III. MATURITY MODELS
The maturity models from software development processes
enable the classification of the performance of the actual ones
and guide organizations to encourage process improvement
through a staged method, also known as maturity. These
maturity models are an interesting approach to solving the
problem described in Section I, since the presence of a
maturity classification can allow the comparison between
competitor organizations.
A. Capability Maturity Model Integration
Capability Maturity Model Integration (CMMI) was
introduced in 2002 and ever since, it has focused on process
improvement approaches, which assist organizations in
adopting the best type of practices from each process area and
make the processes performance evolve [28][29].
In the staged representation, CMMI presents different
levels that vary from one to five. One level of maturity is
characterized by a set of predefined process areas, evaluated by
the accomplishment of specific and generic goals applicable to
the various areas. Each of these is attached to a set of practices,
which reflect specific and generic goals [30]. This type of
approach is highly successful worldwide amongst enterprises
that wish to surpass competitors by providing improved and
better products and services.
Given its broad scope coverage, CMMI does not solve the
issue due to its non-focus on agile software development
processes, which are the area of the current study.
B. Agile Maturity Model
Agile Maturity Model (AMM) was introduced by two
researchers in an IT University in Leeds, and it was conceived
in order to provide future researchers a more in-depth agile
maturity model as a basis for their investigations [31].
Figure 2. Agile Maturity Model staged representation [31].
This model is shown in Figure 2. and it is somehow
inspired by CMMI, since it also has 5 levels, each with a set of
goals for their practices: (1) Level 1: Initial – organizations
belonging to this level of agile maturity do not have a clearly
defined process for agile development and eminent success
depends solely on the competence of individuals; (2) Level 2:
Explored – it gives particular focus to project planning and
requirements engineering for organizations; (3) Level 3:
Defined – it stresses the importance of frequent deliveries, pair
programming and customer relationship enhancement; (4)

Level 4: Improved – it focus on project management,
sustainable velocity of development and self-organizing teams;
(5) Level 5: Sustained – underlines the need for the
management of projects‟ performance, thus continuously
improving processes.
The AMM provides a first approach to classifying the
maturity of agile development processes, which comprises
practices from various agile methodologies. Therefore, it leads
us to a continuous research, since this model‟s set of practices
crosses too many agile methodologies that most organizations
do not apply, causing increased levels of entropy.
C. Agile Maturity
Agile Maturity paper appeared as a study case from the
British Telecom while developing an IT project [32]. Since it
was said that big organizations had increased the barrier for a
successful agile adoption, an agile maturity roadmap was
presented.
The agile maturity evaluates the agile performance in seven
dimensions within five levels of maturity: (1) Level 1 –
represents the appearance of software engineering best-
practices; (2) Level 2 – best-practices are continuous and
improve within small development teams; (3) Level 3 – there is
continuous integration within local component teams; (4) Level
4 – there is an incessant integration within global journey
teams, i.e., distributed teams, and; (5) Level 5 – on-demand
development maturity.
For each of these levels there shall be an evaluation of each
the seven existing dimension: (1) Automation of regression
tests; (2) Code quality metrics; (3) Automation of deployment;
(4) Automation of configurations and best-practices
management; (5) Interface integration tests; (6) Test driven
development, and; (7) Performance scalability tests.
The combination of these five maturity levels and the seven
dimensions allowed British Telecom to incrementally perform
a better agile development process. However, this approach is
generic and non-focused on the description of these levels and
their practices, which leads to one‟s need to seek another
solution for the major problem stated in Section I.
IV. PROPOSAL
Following the problem focused throughout the last
investigation and its various related work, the proposal of a
potential solution was introduced in the previous work.
Therefore, this section will present the improvements made, the
results from the previous proposal through the last two cycles
of action research, and propose an optimized roadmap for IT
organizations, with renewed validation, so as to develop
software with better quality, i.e., more focused on the client
role and motivated to self-improvement and market
competition.
The Scrum Maturity Model‟s main purpose is to aid and
guide IT software development organizations and encourage
self-improvement, giving special attention to the client‟s role,
which is mandatory on this fast moving, global and competitive
worldwide market. Furthermore, this proposal intends to help
organizations that are not familiar with Scrum and wish to
implement and adopt it on a staged and incremental approach.
This proposition introduces five levels for Scrum
development methodology with its respective goals, objectives,
specific and suggested practices. The number of levels is a
standard of maturity models; thus making it easier to be
measured up with other maturity models for comparison and
evaluation purposes.
Next, the main improvements made from the original
proposal will be presented. Note that the full details of the
proposition contain the complete goals, objectives, practices
and suggested metrics for each level of Scrum maturity.
A. Level 1 – Initial
This first and lowest level of maturity, which can be
assigned to an organization that uses Scrum, represents the
absence of goals for process improvement. The explicit
definition of agile development with Scrum methodology does
not exist within organizations classified as belonging to this
level.
The main issues of the organizations in this level are the
frequent over-time and over-budget projects, poor
communication among stakeholders and unsatisfactory quality
of the final product. These organizations operate on their own
and unique way depending on their particular situation which
makes their success highly reliant on competent and skilled
individuals rather than on standardized and capable teams. In
fact, organizations that do not comply with the goal defined for
level 2 of Scrum maturity are downgraded to level 1 until
further improvements are performed in order to achieve the
next level.
B. Level 2 – Managed
In level 2, software development practices appear more
structured and complete than in level 1, due to the fulfillment
of the two main goals set for this level also shown in Figure 3:
Figure 3. Goals and objectives for level 2 of Scrum Maturity.
 Basic Scrum Management – this goal dictates practices
that organizations in this level must accomplish, which
will ensure the minimum acceptable usage of the
Scrum methodology and structure. Note that, although
all Scrum roles, objects and meetings must exist in
these organizations, those Scrum objects might not be
correctly or effectively used, resulting on the need to
have further process improvement;

 Software Requirement Engineering – this goal
comprises a set of practices that the organizations must
comply with in order to achieve satisfaction from the
final product‟s quality created by the vendor
organization. Organizations in level 2 usually face
fewer problems in the development process than the
ones in level 1. However, they still have difficulty in
communicating with the client-side representatives and
delivering their projects as planned, concerning
schedule and budget.
According to the last evaluation of the proposal, this level
showed solid goals, objectives, practices and suggested
metrics. For this reason, level 2 presented minor changes in the
text of the practices, remaining the majority of this level intact.
C. Level 3 – Defined
Level 3 of this maturity model has its major focus on the
relationship with clients and on time deliveries. Hence, this
level also has two major goals, shown in Figure 4, to guide
organizations and improve their processes:
Figure 4. Goals and objectives for level 3 of Scrum Maturity.
 Customer Relationship Management – this goal
emphasizes the importance of the client and the efforts
required to maximize the collaboration with the
customer side, even considering the three main
difficulties mentioned in Section I. A set of practices
are defined and must be satisfied in order to solve the
core problem of this investigation.
 Iteration Management – this goal is indirectly linked to
the previous one, since both contribute to raise
customer satisfaction levels. In order to achieve this
goal, a set of practices must be fulfilled and
implemented so that the organizations always deliver
their projects and sprints on time, following their
budgets.
With the implementation of level 3 of maturity, an
organization can be successful on several projects. However,
this success is only partial due to the lack of standardized
management, which would guarantee the same quality and
performance in all development processes.
Again, the previous work evaluated this level as fairly
solid, and only minor changes within the description of the
practices were introduced.
D. Level 4 – Quantitativelty Managed
In level 4 of Scrum maturity, an organization can boost
their achievements by offering standardized and regular
software development process aided by the management of the
process performance through measurement and analysis
practices. In this level of maturity, there are two main fields:
 Standardized Project Management – this goal shall
lead organizations to use the same development
process for all projects and deliver significantly high
quality and performance levels. In order to achieve this
goal, an organization must complete the
standardization of the performed processes;
 Process Performance Management – this goal demands
the monitoring of all suggested practices up to level 4
of Scrum maturity. These metrics aim to provide
enough feedback about actual processes and manage
their performance.
Although this level seems very simple in Figure 5, it is
actually extremely hard to implement the management and
monitor all projects within an organization so as to fulfill all
specific practices and maintain the process‟ consistency. Note
that suggested metrics may be used and organizations are
encouraged to customize them to be more appropriate for each
enterprise‟s culture and best practices.
Figure 5. Goals and objectives for level 4 of Scrum Maturity.
Organizations in this level adopt appealing Scrum
development processes and the majority of their projects are
successful. The only and last improvement left is optimization
of the current processes.
With the previous evaluation process for this proposal it
was possible to identify the ambiguity within level 4 for many
organizations. In order to clarify it, the demand for
“Standardized Projects Management” is now only applied to all
agile Scrum projects within the organization, and not to all
projects, since in one organization both waterfall development
methodologies and agile, in different projects and clients, can
coexist.
E. Level 5 – Optimizing
Organizations in level 5 of the Scrum Maturity Model are
top class software developers using Scrum methodology. They
focus on continuous self-improvement to excel competition
and bring higher levels of satisfaction from client, development
team and all stakeholders. The only goal for this level is:
 Performance Management – this goal allows
organizations to measure and analyze their own actions
and processes to self-improve.

Organizations in this level have achieved a maximum level
and must not discard previous accomplishments and goals by
negligence which will block continuous process improvement.
In Figure 6, the four objectives for the main goal of this
Scrum maturity are illustrated, being “Causal Analysis
Resolution” a newly added objective to this level of Scrum
maturity.
Figure 6. Goals and objectives for level 5 of Scrum Maturity.
The main result from the previous work for the definition
and first approach experimentation was that the top levels of
this Scrum Maturity Model were slightly incomplete and
ambiguous. Therefore, the objective “Causal Analysis and
Resolution” was included to be used with Daily Scrum and
Scrum Retrospective Meeting as to analyze the occurred
impediments, differentiate them from incidents and problems,
make causal analysis retrospective and then take corrective
actions against them.
Note that the whole Scrum maturity model was constantly
aligned with similar and renowned best-practices such as
CMMI. This decision was based on the purpose of future
comparisons with CMMI assessments versus assessments
using the proposed model, in order to provided another form of
the validation.
Before the results from the practical experimentation of this
preposition, note that this Scrum Maturity Model is comprised
by its goals, objectives, specific practices and suggested
practices for each level. However, due to its size, the complete
list of specific practices was not presented. Therefore, only
instances from the set were given.
V. RESULTS
In order to evaluate and validate the usefulness and
effectiveness of this improved proposal, a third cycle of action
research was planned, which included two interviews with
Scrum, agile and CMMI experts to validate the concept and
details of the proposal as well as six appraisals and audits of
Scrum maturity in three different enterprises so as to evaluate
its usefulness, efficiency and impact made.
A. Interviews
In order to attain validation of this concept: maturity model
for agile Scrum development methodology, a few experts were
interviewed.
1) Expert A
Expert A, an international CMMI, Agile and Scrum expert
and also partner of an Agile coaching company, granted us
two interviews to present our previous proposal and discuss it
regarding its viability, usefulness and value created from it.
According to Expert A, the first three levels of Scrum
maturity have sufficient detail and acceptable approach.
However, although level 4 and 5 have proper goals and
objectives, they required some more detail, more specifically,
practices to enhance the quality of Scrum Retrospective
Meeting are lacking. For instance, practices such as “Question
five W‟s”, “Identify problems and incidents” and “Build
cause-effect diagram to identify problems” would enhance the
quality of the inner inspection from retrospective meeting to
seek continuous improvement.
Nevertheless, in her feedback, Expert A also stated that the
suggested metrics from level 4 of Scrum maturity presents an
excellent feature, since not even CMMI presents suggested
metrics that exists in COBIT. These suggested metrics allow
the monitoring of the current state of the process and discover
where to put efforts for improvement, apart from analyzing
quantitative statistics from the development process.
About the concept as a whole, Expert A accepts that
scattered Scrum loses integrity, however she also agrees that
Scrum Maturity Model is not intended to split Scrum into five
levels and areas, but rather to provide more emphasis on
different areas in each level. Furthermore, it was assured that
if this proposal does not become a standard worldwide, it will
at least be an extraordinary tool to be used in Scrum
Retrospective Meetings as self appraisal and assessment of
own maturity.
2) Expert B
Expert B, also an international Agile and Scrum expert as
well as a Scrum coach, works for a top five world largest IT
company, and conceded us an interview to present to him the
actual proposal and discuss about its viability, usefulness and
created value . He was pleased with the concept which
involves the evaluation of the maturity of the Scrum process,
and provided precious feedback for the definition of the
practices of each level and within each goal.
Most of the original proposal remains, while merely the
definition of the required practices changed, remaining the
goals and objectives intact.
B. Appraisals
Another way to validate this theoretical work is to apply it
to organizations with strong contact with real business
problems. To evaluate the proposal, the following process was
adopted:
 Pre-appraisal questionnaire – First, a brief presentation
of Scrum Maturity Model concept and its goals on
each level will take place. Then, the organization will
be asked to fill in the pre-appraisal evaluation form,
which will unfold its beliefs about the level in which
the organization should and will be classified;
 Appraisal – Later on, if it was never audited before, the
appraisal for level 2 of Scrum maturity will begin. If
they had obtained successful appraisals before, then the
next level of Scrum maturity will be appraised. This
process consists in auditing the organizations‟ practices
against the checklist of the Scrum ones which must be

accomplished in order to obtain the intended level
Scrum of maturity.
 Post-appraisal questionnaire – After the appraisal, the
assessed organization receives a post-appraisal
questionnaire to evaluate the proposal. This phase aims
to extract all feedback, both positive and negative,
about the proposal and the satisfaction level with
appraisal results, comparing it to the initial
expectations.
Next, we will present the action taken within three IT
development and consulting organizations while auditing the
maturity of their development process using Scrum. A number
of organizations provided more than one project in progress for
the audit process, therefore, in some, more than one project
manager was interviewed.
1) Organization X
Organization X, which is focused on cutting waste in
software delivery through the practice of lean and agile
concepts that they have been implementing for a year now,
allowed an audit of their software development process to
assess their maturity of Scrum usage.
They are comprised by around seven developers abroad in
Ukraine, who assume the Scrum role “Team”, and three project
managers in Portugal, that take on the role of “Scrum Master”
involved in two or three projects at a time. This enterprise is
the excellent example of distributed Scrum, which intends to
manage the resources wisely without creating waste and still
fulfills the needs of the client, considering the problems from
cooperation and distance.
Within the pre-appraisal questionnaire, the organization
predicted the possible outcome from the audit as level 2 or 3 of
Scrum maturity, since they were aware of the lack of
mechanisms to measure and monitor process metrics and
formal processes for continuous improvement.
As the appraisal occurred, the organization was confronted
with the checklist of the practices which had to be fulfilled in
order to achieve the first level of Scrum maturity – level 2.
According to the audit, they failed the “Basic Scrum
Management” goal by missing the objective of “Scrum
meetings occur and are participated”. Actually, they ignored
the need of a Scrum Retrospective Meeting and neglected the
importance of a formal Daily Scrum Meeting and Scrum
Review Meeting.
During the post-appraisal questionnaire, the organization
did not show any sight of disappointment and, instead,
appeared to be very excited with the results, displaying
motivation and critic analysis toward the results and
opportunities for future improvements for a better development
process. First, they argued that it is very difficult to
communicate with clients in this fast moving generation. It was
hard to convince the collaboration and their presence at the end
of each sprint, which caused them to fail practices such as
“Sprint Review Meeting occurs exactly once per Sprint” and
“Sprint Review Meeting is attended by Stakeholders, Scrum
Master, Product Owner and Team”. They also claimed against
the failed “Daily Scrum occurs exactly once per workday”
practice, since the organization affirms there are casing
meetings from lean development principles, for the nature of
these meetings is different.
Nevertheless, at the end, the organization will rethink these
failed practices, and the interviewed project manager planned
to immediately launch the implementation of Scrum
Retrospective Meeting, since it has great potential benefits that
had not yet been considered.
When the interview ended, the interviewee gave the
following feedback regarding the Scrum Maturity Model:
“This proposal provides a good roadmap for IT organizations
by offering goals and objectives per level to evolve and
gradually improve, attacking one goal at a time.”; “For higher
levels of maturity, it is required much more stability to see the
improvements and, although the existence of suggested metrics
is brilliant, it lacks how to implement the monitoring
mechanism.”. As a final word, the Scrum Master from the
organization stated: “Many organizations nowadays declare
themselves as agile, but how agile they can be when there are
no definitions or rules? The existence of this proposal can
surely differentiate the successful agile practitioners from the
others.”
2) Organization Y
Organization Y, a fast growing IT consultant enterprise
focused on satisfying the market needs through agile and
flexible principles, also accepted to be a part of this
investigation by providing three of their four project managers
to be audited with Scrum Maturity Model.
They are around forty employees, with about thirty in
headquarters and ten distributed in two other branches, being
one of these branches located abroad, in Vienna. Currently,
they employ four project managers and the CEO arranged three
meetings with three of them in order to receive some academic
research feedback within his company.
a) Project Manager Y1
Project Manager Y1 has been recently promoted to perform
the more technical oriented role of project manager. He has a
background in the business intelligence field, and now focuses
more on the leadership and management of the team of
developers for consulting projects.
During the pre-appraisal questionnaire phase, while
analyzing the goals required for each level, he determined
levels 1 or 2 as a possible result, for he was fully aware that the
organization is on the early stage of agile implementation and
several goals might not be fulfilled.
As the appraisal for level 2 of Scrum maturity occurred,
soon the missing practices was identified. They missed the
“Sprint Retrospective Meeting occurs exactly once per Sprint”
practice. Unfortunately, this missing feature made this
organization fail level 2, although many other practices were
accomplished.
Then, within the post-appraisal questionnaire, the project
manager agreed with the results, although slightly disappointed
with the obtained level. The grounds for this result, he said,
was that many unimplemented practices were not given the
importance they should have and, although it is possibly very

rewarding, they wanted to focus more on the current client
needs without having to worry about overworking their
employees. Another explanation is that, given the dimension of
his team, so much formality in the development process was
not really necessary, as long as the results show up and the
clients are satisfied.
For evaluation purposes, it was allowed for the project
manager to inspect the next level, which turned out to be
another failed appraisal, but this time for level 3, the
organization failed the “Sprint Backlog Items are split into
tasks” practice when all other practices were accomplished.
With this result, the project manager was relieved as he
believed that they could achieve up to level 3 of Scrum
maturity with a relative small amount of effort, even though it
required immense work in employees‟ culture to implement
them.
To conclude, he agreed that the concept itself has potential
to grown into a certification, which will provide more market
differentiation. Another interesting point is that it might not be
very expensive to concentrate efforts and obtain an acceptable
level 3 of Scrum maturity.
b) Project Manager Y2
Project Manager Y2 is in charge of four development
projects, each of them with only one or two developers located
in Vienna focusing on the improvement of applications for
smart phones. The main challenges for him are how to
coordinate and perform the role of middle man between the
client‟s needs and developers‟ performance with Scrum
methodology, since he has less than a year experience with this
development methodology.
Within the pre-appraisal questionnaire, given Manager Y2
relative inexperience, the project manager did not have high
expectations and pointed out level 2 as a possible outcome.
During the appraisal, they failed many practices such as:
“Release Burndown Chart exists”, “Sprint Burndown Chart
exists” and “Sprint Retrospective Meeting occurs exactly once
per Sprint”.
In the post-appraisal questionnaire phase, the project
manager explained that due to the unawareness of the technical
capabilities from the project management tools, it was not
possible to maintain updated and correct burndown charts.
Concerning the missing retrospective meeting, he stated that it
is very difficult to have a formal meetings with the distributed
team located in Vienna, seriously affecting the performance of
this communication.
Again, for evaluation purposes, it was allowed for the
interviewee to inspect the fully detailed Scrum Maturity
Model, and advanced to the next level‟s audit. They did not
accomplish practices like: “Definition of „Done‟ is achieved in
each iteration” and “During Sprint Review Meeting Product
Owner and other stakeholders provide feedback”.
In the end, the project manager was satisfied to learn more
about agile Scrum methodologies, and where he should
improve in further projects. He stated that this maturity model
might be an important tool to measure their current
performance and guide them to continuous improvement.
c) Project Manager Y3
Project Manager Y3, a very experienced and enthusiastic
Scrum and agile practitioner, is leading the company to
implement the backbone for Scrum adoption. It has been
almost a year since they started trying to reach this objective,
and, at the moment, they are in the final stage. For him,
continuous improvement is the core strategy to achieve a
competitive advantage. In order to achieve this goal, he leads
the implementation and integration of several support systems
to aid the development process, since he believes that no agile
is solid enough without the required backbone tools. Now, he is
in charge of a development project with three developers and a
three month length deadline.
The pre-appraisal questionnaire phase revealed that he had
high expectations and confidence in their maturity, choosing
the level 4 or 5 as the expected result from the appraisal.
When the appraisal began, they succeed to fulfill level 2
practices, and then level 3. No problems were encountered so
far. Surprisingly, level 4 was also achieved, because all his
previous projects were managed with a standard method and he
had a data mining module that defined, monitored and measure
their development process and metrics. At the last appraisal for
level 5, unfortunately, they failed the practices: “Successful
Retrospective Meetings result in concrete improvement
proposals” and “Successful Retrospective Meetings‟ lessons
learned are recorded to a knowledge base”.
Within the post-appraisal questionnaire, the project
manager was satisfied with the results, seeing his efforts
recognized by external parties and not totally disappointed with
the obtained level 4 of Scrum maturity, since they were
working on the quality of retrospective meetings now.
His final feedback for this proposal is the following: “This
proposal is an excellent tool for deeper insight, to rethink their
agile path. Moreover, this preposition motivates the adoption of
Scrum by separating several objectives via levels. Agile is easy
to learn, however very hard to master. Thus, it is very
important for prepositions like these to exist in order to aid
organizations to correctly adopt Scrum.”
3) Organization Z
Organization Z is a worldwide renowned company that
provides technology solutions and services around the world.
In their office located in Portugal, they employ around four
hundred professionals, delivering both consulting service and
software solutions. Their development projects are normally
very big involving more than forty people and a twelve-month
period per project.
a) Project Manager Z1
Project Manager Z1 is the senior software architect and
performs team coaching regularly. He worked for a leading and
pioneer company using agile methodologies, where he learned
a lot about agile best practices from the elite from that
generation. Currently, the project he is working with involves
forty people, three scrum teams and a year of schedule, and it
applies Scrum methodology with this particular client for the
first time. They are on the production and deployment phase.

In the pre-appraisal questionnaire assessment, Manager Z1
suggested level 3 as the possible result, since he was aware that
the company missed the goals “Measurement and Analysis
Management” and “Performance Management”.
During the appraisal for level 2 of Scrum maturity,
Manager Z1‟s project succeeded to accomplish all practices for
level 2, except “Sprint Burndown Chart exists” practice.
In the post-appraisal questionnaire phase, Manager Z1
intensely argued about the need of a sprint burndown chart,
which is only used to manage small two weeks sprints and
creates waste by joining efforts to manually build such a chart.
Note that the organization uses manual means to follow Scrum
methodology.
By analyzing the next levels, Manager Z1 felt frustrated
again, because he would fail level 3 due to the inexistence of
the sprint burndown chart stressed in the goal “Iteration
Management”. However, to achieve levels 4 and 5, he agreed
that more efforts were needed and that they intend to move
further in their question of continuous improvement as a
competitive advantage.
As final words, he said: “What I see here is a very
interesting approach in agile methodologies study. The
roadmap is very good for new enterprises to adopt Scrum and a
nice differentiation model for companies in the development
industry.”
b) Project Manager Z2
Project Manager Z2 is also a well experienced Scrum
practitioner within the organization, and is currently managing
a project with four years already, which involves three Scrum
teams. This project‟s particularity is that the client does not
collaborate as closely as the company would wish, so Scrum
was only applied as internal communication and work
methodology.
In the pre-appraisal questionnaire, after the overview of the
maturity mode, he selected level 2 as most likely result of the
appraisal.
As the appraisal started, “Sprint Burndown Chart exists”
practice was found to be missing just like in the last project
manager. Moreover, they did not have “Sprint Review
Meeting occurs exactly once per Sprint” practice formally
implemented, only some demonstrations once or twice a year.
Yet another missing practice was “Sprint Retrospective
Meeting occurs exactly once per Sprint”, as according to
company‟s culture, it only happens right after the Scrum
Review Meeting.
During the post-appraisal questionnaire, he commented as
the following: “Agile methodologies stress communication a
lot. Its qualities are not shown in tiny projects, but in large
scale projects in which real problems occur. In these big
projects, flexible and constant communication is needed to
maximize and optimize the work performed. This proposal
presents a staged maturity model to guide Scrum
implementation and Scrum performance and usage to
differentiate enterprises, which is a magnificent idea.”
VI. EVALUATION
Given the results previously presented, in this section, a
critic study for the Scrum Maturity Model will be analyzed
and presented.
Regarding the interviews, it was possible for us to realize
that the first three levels were well structured, while top levels
needed some rework, which is already done. Moreover, it was
stated by professionals that the preposition is a very good
approach for Scrum adoption, self-inspection and continuous
improvement.
This study considered the six performed appraisals in three
sample organizations from Portugal, represented by a small, a
medium and a large-sized company. Although the average
level of maturity is not very high, many of the audited
organizations were able to easily reach level 3 by focusing
efforts to implement the missing goals, objectives and
practices.
The most common missing practices for the first level of
Scrum maturity, level 2, were “Sprint Review Meeting occurs
exactly once per Sprint” and “Sprint Retrospective Meeting
occurs exactly once per Sprint”. In level 3, “Definition of
„Done‟ is achieved in each iteration” is the most commonly
failed practice. Top levels were scarcely achievable due to
their requirements for mechanisms and concepts for
measurement; analysis of process metrics; causal analysis;
resolution of problems; and, impediments identified, which
were not popular among IT development organizations.
Although many organizations define themselves as agile
and Scrum followers, another interesting finding is that many
of the basics were not taken into account, and only main and
popular values and principles were retained, resulting in these
low levels of Scrum maturity.
Through this assessment, it was possible to conclude that
the proposal provides a good roadmap for organizations that
want to implement Scrum methodology from scratch, align
their position for benchmarking purposes or for organizations
that want to self-improve.
All feedback collected from both interviews with experts
and professionals in the development industry gave us a great
deal of confidence and insight to continue our research, refine
it and possibly scale its usage and define it as a standard.
VII. CONCLUSION AND FUTURE WORK
In Section I, followed by some discussion and analysis, the
main problem was a visible lack of collaboration, in most
cases, between vendor organizations and clients as they tried to
achieve the development of a successful IT project. This
problem is a widely researched topic amongst IT experts, due
to its vital importance on the success of software development
projects [1][2][3][4][5][6][7][8].
Inspired by the related work and maturity models, the
improved proposition, from previous research, with five levels
of Scrum maturity presents a roadmap for organizations to
implement Scrum methodology and compare the performance
of software development process amongst competitors.
The main focus of this paper was the validation phase of
the current proposal within cycles of AR, which are comprised

by two interviews with two agile and CMMI experts and six
appraisals and post-appraisal assessment. The proposal was
evaluated and validated by them, and it is our intention to share
our findings with the scientific community. Since this
proposition is continuously evolving, the current research shall
be repeated until the community agrees on a final iteration and
accept it as standard.
We are aware that the evaluation process has limitations,
but despite credibility issues regarding this process, the
experienced validation phase is worthy to be share with the
scientific community, given the interest of the process and its
results.
Along with the analysis of the motivation for this research,
it was pointed out that further investigation on human factors
and on the change of management areas might benefit and
enhance the performance of this maturity model. Another
interesting research topic would be the classification of the
partnership and client maturity, since, as referred to in Section
I, clients are usually the major impediment for successful IT
projects.
In the end, and we once more stress, the proposition of
maturity model is highly polemic within agile community.
Nevertheless, the concept Scrum Maturity Model has proved
successful as the roadmap for organizations that seek self-
improvement and guidance.
ACKNOWLEDGMENT
We would like acknowledge the participation of all experts,
organizations and their projects manager for their validation,
feedback and experience sharing. A special thanks will go to
Microsoft and Tiago Andrade e Silva for their support and
guidance throughout this validation process.
REFERENCES
[1] Group, S.: The Chaos Report 2009 (2007) Retrieved from
http://www1.standishgroup.com/newsroom/chaos_2009.php last
accessed 25 August 2011
[2] Kraut, R. E. and Streeter, L. A.: Coordination in software development:
Communications of the ACM, 38(3), 69-81. ACM (1995)
[3] Herbsleb, J. D. and Moitra, D.: Global software development: IEEE
Software, 18(2), 16-20. IEEE (2001)
[4] Highsmith, J. and Cockburn, A.: Agile software development: the
business of innovation: Computer, 34(9), 120-127. IEEE (2001)
[5] Cockburn, A. and Highsmith, J.: Agile software development, the people
factor: Computer, 34(11), 131-133. IEEE (2001)
[6] Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A., Cunningham,
W., Fowler, M., Grenning, J., et al.: Agile Manifesto (2001) Retrieved
from http://agilemanifesto.org/principles.html last accessed 25 August
2011
[7] Leffingwell, D. and Widrig, D.: Managing Software Requirements: A
Unified Approach: AddisonWesley Longman Publishing Co Inc Boston
MA USA, 491. Addison Wesley (2000)
[8] Charette, R. N.: Why Software Fails: Ieee Spectrum, 42(9), 42-49. IEEE
INSTITUTE OF ELECTRICAL AND ELECTRONICS (2005)
[9] Reel, J. S.: Critical success factors in software projects: IEEE
Software 16(3), 18-23. IEEE (1999)
[10] Wilson S.: Failed IT Projects (The Human Factor) (1998) Retrieved
from http://ac-support.europe.umuc.edu/~meinkej/inss690/wilson.htm
last accessed 25 August 2011
[11] Batra, D.: Modified agile practices for outsourced software projects.:
Communications of the ACM 52(9), 143. AMCIS (2009)
[12] Holmström, H., Fitzgerald, B., Ågerfalk, P. J., and Conchúir, E. Ó.:
Agile Practices Reduce Distance in Global Software Development:
Information Systems Management, 23(3), 7-18. Taylor & Francis (2006)
[13] Norton, D.: The Current State of Agile Method Adoption. Analysis
(2008) Retrivied from
http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2
&PageID=3460702&resId=837321&ref=QuickSearch&sthkw=agile+m
ethods last accessed 25 August 2011
[14] Glazer, H., Dalton, J., Anderson, D., Konrad, M., and Shrum, S.:
CMMI® or Agile : Why Not Embrace Both!: Carnegie Mellon
University, Software Engineering Institute (2008)
[15] Laudon, K. C., Laudon, J. P.: Management Information Systems:
Pearson (2009)
[16] Yin, A., Figueiredo, S., and Mira da Silva, M.: Scrum Maturity Model:
Roadmap for IT organizations to develop software centering on the
client role: submitted to 23th Internation Software & Systems
Engineering and their Applications (2011)
[17] Baskerville, R. L.: Investigating information systems with action
research: October, 2(October), 1-32. Association for information
Systems (1999)
[18] Fowler, M., and Highsmith, J.: The Agile Manifesto: Software
Development, 9(August), 28-35. San Francisco, CA: Miller Freeman,
Inc (2001)
[19] Larman, C. and Basili, V. R.: Iterative and Incremental Development: A
Brief History: Computer, 36(6), 47-56. IEEE (2003)
[20] Layman, L., Williams, L., and Cunningham, L.: Motivations and
Measurements in an Agile Case Study: Journal of Systems Architecture
52(11) 654-667 Elsevier North-Holland, Inc. (2006)
[21] Chow, T. and Cao, D.: A Survey Study of Critical Success Factors in
Agile Software Projects: Journal of Systems and Software, 81(16), 961-
971. Elsevier Science Inc. (2008)
[22] Cockburn, A.:Crystal Clear: A Human-Powered Methodology for Small
Teams. (Series in Agile Software Development). Addison-Wesley
Professional (2004)
[23] Schwaber and K., Beedle, M.: Agile Software Development with Scrum
: (Series in Agile Software Development). Prentice Hall (2002)
[24] Beedle, M., Devos, M., Sharon, Y., Schwaber, K., and Sutherland, J.:
SCRUM: An Extension Pattern Language for Hyperproductive Software
Development: Pattern Languages of Program Design, 4, 637-651 (1999)
[25] Kircher, M., Jain, P., Corsaro, A., and Levine, D.: Distributed eXtreme
Programming: Challenges, 20-23. XP01 (2001)
[26] Sutherland, J., Viktorov, A., Blount, J., and Puntikov, N.: Distributed
Scrum: Agile Project Management with Outsourced Development
Teams. 40th Annual Hawaii International Conference on System
Sciences HICSS07 0, 274a-274a. Ieee (2007)
[27] Braithwaite, K. and Joyce, T.: XP Expanded: Distributed Extreme
Programming: Communication, 180-188. Springer Berlin / Heidelberg
(2005)
[28] Chrissis, M. B., Konrad, M., and Shrum, S.: CMMI Guidlines for
Process Integration and Product Improvement: Addison-Wesley
Longman Publishing Co., Inc. Boston, MA, USA (2003)
[29] Menezes, W.: To CMMI or Not to CMMI: Issues to Think About:
CrossTalk The Journal of Defense Software Engineering, (February
200), 9-11. (2002)
[30] Development, C.: CMMI® for Development, Version 1.3 CMMI-DEV,
V1.3:. Engineering. Carnegie Mellon University (2010) Retrieved from
http://www.sei.cmu.edu/library/abstracts/reports/10tr033.cfm last accessed
25 August 2011
[31] Patel, C. and Ramachandran, M.: Agile Maturity Model (AMM): A
Software Process Improvement framework for Agile Software
Development Practices: International Journal of Software Engineering,
2(1), 3-28. Software Engineering Competence Center (2009)
[32] Benefield, R.: Seven Dimensions of Agile Maturity in the Global
Enterprise: A Case Study: System Sciences HICSS 2010 43rd Hawaii
International Conference, 1-7. IEEE (2010)