Content uploaded by Jose A Calvo-Manzano
Author content
All content in this area was uploaded by Jose A Calvo-Manzano on Nov 22, 2017
Content may be subject to copyright.
IT Service Management Process Improvement based on ISO/IEC 15504: A
systematic review
Antoni Lluís Mesquida
a,
⇑
, Antonia Mas
a
, Esperança Amengual
a
, Jose A. Calvo-Manzano
b
a
Department of Mathematics and Computer Science, University of the Balearic Islands, Ctra. de Valldemossa, Km. 7.5, 07122 – Palma de Mallorca, Spain
b
Faculty of Computer Science, Technical University of Madrid, Campus de Montegancedo s/n, 28660 – Boadilla del Monte, Madrid, Spain
article info
Article history:
Received 3 February 2010
Received in revised form 14 September 2011
Accepted 10 November 2011
Available online 20 November 2011
Keywords:
Software Process Improvement (SPI)
ISO/IEC 15504 (SPICE)
IT Service Management (ITSM)
Systematic review
abstract
Context: In recent years, many software companies have considered Software Process Improvement (SPI)
as essential for successful software development. These companies have also shown special interest in IT
Service Management (ITSM). SPI standards have evolved to incorporate ITSM best practices.
Objective: This paper presents a systematic literature review of ITSM Process Improvement initiatives
based on the ISO/IEC 15504 standard for process assessment and improvement.
Method: A systematic literature review based on the guidelines proposed by Kitchenham and the review
protocol template developed by Biolchini et al. is performed.
Results: Twenty-eight relevant studies related to ITSM Process Improvement have been found. From the
analysis of these studies, nine different ITSM Process Improvement initiatives have been detected. Seven
of these initiatives use ISO/IEC 15504 conformant process assessment methods.
Conclusion: During the last decade, in order to satisfy the on-going demand of mature software develop-
ment companies for assessing and improving ITSM processes, different models which use the measure-
ment framework of ISO/IEC 15504 have been developed. However, it is still necessary to define a method
with the necessary guidelines to implement both software development processes and ITSM processes
reducing the amount of effort, especially because some processes of both categories are overlapped.
Ó2011 Elsevier B.V. All rights reserved.
Contents
1. Introduction . . . ...................................................................................................... 240
2. ITSM standards. ...................................................................................................... 240
3. Systematic review method . . . . . . . . . . ................................................................................... 240
3.1. Question formularization .......... ............................................................................... 241
3.2. Selection of sources. . . . . ...................................................... ................................... 241
3.3. Selection of studies . . . . . ............................................................. ............................ 241
3.4. Information extraction . . ............................................. ............................................ 242
3.5. Results summarization . . ................................ ......................................................... 242
4. Results and discussion . . . . . . . . . . . . . . ................................................................................... 242
4.1. Classification of studies. . ....................... .................................................................. 244
4.2. Results from the analysis of primary studies ............................................. ............................ 244
4.2.1. Standards used for ITSM Process Improvement . . . . . ........................................................... 244
4.2.2. New ITSM Process Improvement initiatives . . . . . . . . ........................................................... 244
4.3. Temporal view of primary studies . . . . . . . . . ......................................................... ................ 245
5. Conclusions and future work. . . . . . . . . ................................................................................... 245
Acknowledgments . . . . . . . . . . . . . . . . . ................................................................................... 246
Appendix A. List of primary studies in the systematic review . . . . . . . . . . . . . . . . . . ............................................. 246
A.1. List of primary studies in the systematic review . . . . . . . . . . . . . . . . ................ ...................................... 246
References . . . . ...................................................................................................... 247
0950-5849/$ - see front matter Ó2011 Elsevier B.V. All rights reserved.
doi:10.1016/j.infsof.2011.11.002
⇑
Corresponding author.
E-mail addresses: antoni.mesquida@uib.es (A.L. Mesquida), antonia.mas@uib.es (A. Mas), eamengual@uib.es (E. Amengual), joseantonio.calvomanzano@upm.es (J.A. Calvo-
Manzano).
Information and Software Technology 54 (2012) 239–247
Contents lists available at SciVerse ScienceDirect
Information and Software Technology
journal homepage: www.elsevier.com/locate/infsof
1. Introduction
Over the course of the last two decades, both large and small
software companies have made important efforts in order to
implement Software Process Improvement (SPI) programmes to
establish a process-oriented culture in the organization. A focus
on processes provides the stability and the management infra-
structure required to deal with the ever-changing and competitive
market. A well-defined management framework results in a better
monitoring of the processes so that they can reach higher maturity
levels.
Higher maturity enables a global understanding and a better vi-
sion of the workload and, as a result, a more consistent and repeat-
able work. Moreover, technical skills of the staff are improved and
the use of technology is maximized. When the productivity and
efficiency of development activities are improved, the organization
can develop, maintain and deliver high quality products, meeting
business objectives (usually focused on quality improvement and
cost and time reduction) and obtaining a higher customer
satisfaction.
The two most internationally used SPI models are Capability
Maturity Model Integration (CMMI) [1] and ISO/IEC 15504 (SPICE)
[2]. These models define a process improvement approach that
provide organizations with the essential elements to set process
improvement goals, establish a point of reference for assessing cur-
rent processes and support the improvement of their performance.
Nowadays, customers not only demand quality products
obtained from mature processes, but they also require quality in
the services they receive. In recent years, while companies have
been deploying their software development processes, there has
been an on-going demand for better IT services. In order to satisfy
this demand, SPI international standards have evolved and have
been adapted to incorporate IT Service Management (ITSM) best
practices. For mature software development companies, besides
the SPI initiative in which they are involved, the implementation
of an ITSM standard has currently become one of the main priori-
ties to assure their continuity and maximize the return of invest-
ment and business opportunities.
The main goal of our research is to analyse existent relations
between two disciplines that have traditionally been treated sepa-
rately: SPI and ITSM. With the intention of taking advantage of the
experience of the authors in the application of the ISO/IEC 15504
standard [3–7], the scope of the research is focused on this SPI
standard and pursues a double objective:
Facilitate the implementation of ITSM processes in software
companies already involved in an ISO/IEC 15504 SPI initiative.
Maximize the efficiency of a joint implementation of ISO/IEC
15504 and an ITSM standard. More concretely, the intention is
to consider ISO/IEC 20000 as the ITSM standard. The main rea-
son for this decision is that both standards have been developed
by the same organization and thus compatibility among them
should be feasible. Moreover, some processes of both ISO stan-
dards are overlapped.
As a first step of our research, a systematic review of the litera-
ture which deals with existing initiatives of ITSM Process Improve-
ment initiatives based on ISO/IEC 15504 for process assessment
and improvement is presented in this paper.
2. ITSM standards
ITSM is a process-oriented discipline which combines process
management and industry best practices into a standard approach
for optimizing IT services. ITSM provides a framework to structure
IT operations that enables organizations to deliver quality IT ser-
vices to meet business needs and adhere to service level
agreements.
Because ITSM is process-focused, it shares common interests
with the process improvement movement. ITSM provides specific
processes, frameworks, methodologies and guidance to manage
planning, implementation and assessment of IT service processes
to optimize tactical and strategic IT operations-related activities.
Some of the most internationally accepted ITSM standards are ITIL
(Information Technology Infrastructure Library), ISO/IEC 20000
and CMMI for Services (CMMI-SVC).
In spite of the diversity of fields that this work deals with, and
with the intention of making its comprehension easier, this section
offers a brief description of the different standards mentioned so
far: ISO/IEC 15504, ITIL, ISO/IEC 20000, and CMMI-SVC.
ISO/IEC 15504 Information Technology – Process Assessment
[2] is an International Standard for process assessment and
improvement. It can be used by any organization to determine
the current and potential capability of its own processes, and also
to define areas and priorities for process improvement. In order to
perform a process assessment conformant with the assessment
model defined in ISO/IEC 15504-2 [8], a Process Assessment Model
(PAM) based upon a suitable Process Reference Model (PRM) needs
to be properly defined. Regarding ITSM, a new part describing an
ITSM PAM is currently under development, called ISO/IEC NP TR
15504-8 Information Technology – Software process assessment
– Part 8: An exemplar process assessment model for IT Service
Management [9].
ITIL is a series of documents that contains a set of best practices
to aid the implementation of a framework for ITSM. ITIL has been
accepted in the industry as the de facto standard for ITSM around
the world. It is focused on the continual measurement and the
quality improvement of the services, from both the business and
the client perspectives. The current version of ITIL, ITIL V3, contains
five reference books [10–14] and its main contribution is the defi-
nition of a structure based on the services life cycle.
ISO/IEC 20000 is an ITSM quality standard that promotes the
adoption of an integrated process approach to effectively deliver
managed services to meet the business and customer require-
ments. This standard, consisting of two parts [15,16], is under a
review process in order to provide a better alignment with ITIL
V3 and other ISO standards. New part 4 [17] will describe an ITSM
PRM which is expected to be closely related to the new Part 8 of
ISO/IEC 15504 [9].
CMMI models are collections of best practices developed by the
Software Engineering Institute (SEI) for the application of process
improvement in the development of products and services cover-
ing the entire product lifecycle. Current CMMI version, CMMI
V1.3, was created with the main intention of defining constella-
tions for being applied in different areas of interest. Regarding to
Service Management, the CMMI-SVC [18] constellation was partic-
ularly created for Service Management process improvement.
CMMI-SVC includes a collection of services best practices that
provides guidance for the application of CMMI-SVC by a service
provider organization. CMMI-SVC best practices focus on activities
for providing quality services to the customer and end users.
3. Systematic review method
The research is undertaken as a systematic literature review
based on the guidelines proposed by Kitchenham [19,20] and the
review protocol template developed by Biolchini et al. [21] which
describes each phase of the systematic review process in terms
of template sections. Other systematic reviews in SPI [22–24] have
also been used as a basis for the work presented in this paper.
240 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247
The protocol used for the systematic review is composed of five
different stages: Question formularization, selection of sources,
selection of studies, information extraction and results summariza-
tion. These five stages are detailed in the next sections.
3.1. Question formularization
With the aim of defining the context in which the systematic re-
view is applied, the protocol suggests to specify a set of items [21].
In the particular case of our systematic review, each item has been
defined specifically for ITSM Process Improvement initiatives
based on ISO/IEC 15504 International Standard for process assess-
ment and improvement.
Problem: software companies have shown a growing interest in
providing quality services to their customers. SPI could be
extended in order to include ITSM processes. The context of this
research is particularly focused on ITSM Process Improvement
models based on the ISO/IEC 15504 assessment framework.
Question: what ITSM Process Improvement initiatives based on
ISO/IEC 15504 exist?
Keywords and synonyms: Software Process Improvement (SPI), IT
Service Management (ITSM), ISO/IEC 15504 (SPICE).
Intervention: analyse ITSM Process Improvement according to
ISO/IEC 15504.
Control: there are no initial data for this systematic review.
Effect: identify all the ITSM Process Improvement initiatives,
frameworks and models performed according to ISO/IEC 15504.
Outcome measure: the number of identified initiatives, frame-
works and models.
Population: the set of research proposals related to ITSM Process
Improvement and ISO/IEC 15504 which have been published in
the list of sources selected for conducting the systematic
review.
Application: companies interested in assessing and improving
their ITSM processes within an ISO/IEC 15504-based SPI initia-
tive. Researchers working on SPI models, more concretely on
ISO/IEC 15504, or on IT Service Management standards.
Experimental design: none statistical analysis methods will be
applied.
3.2. Selection of sources
To perform the selection of the sources where searches for pri-
mary studies will be executed, the systematic review protocol pro-
poses to address the following issues: definition of source selection
criteria, language of the studies, identification of sources, definition
of search strings, and selection of sources [21].
With regard to source selection criteria, the following criteria
have been defined:
Publishing companies or websites suggested by experts.
High-impact publications.
Availability of search mechanisms using keywords.
Non-variability in search results by using the same set of
keywords.
Availability on the web.
Concerning language studies, the obtained primary studies
must be written in English.
The sources have been identified on the basis of the judgement
of the authors of this paper. The list of sources includes relevant
journals in which SPI research area is widely dealt with, such as:
Information and Software Technology, Software Process: Improvement
and Practice, IEEE Software,IEEE Transactions on Software Engineer-
ing and ACM SIGSOFT Software Engineering Notes, Software Quality
Journal, among others. Moreover, the authors have also considered
papers published in the proceedings of different conferences, such
as: the EuroSPI Conference where experts discuss and exchange SPI
practices and the SPICE Conference which is one of the most rele-
vant events regarding the ISO/IEC 15504 standard, among others.
By taking the list of keywords defined in Section 3.1 and making
combinations with the logical operators ‘‘AND’’ and ‘‘OR’’, the
search strings shown in Table 1 have been obtained. To carry out
the searches, these search strings need to be adapted to each of
the search engines of the selected sources.
Taking into account the defined sources selection criteria the
initial list of sources is shown in Table 2. Each element of the initial
list of sources presented above has been evaluated according to the
source selection criteria. After this evaluation, all the elements
have been included in the final list of sources. The authors of the
paper have evaluated the list of sources obtained and approved
all the elements of this list.
3.3. Selection of studies
Once the sources are defined, it is necessary to describe the pro-
cess and the criteria for studies selection and evaluation [21]. The
criteria by which studies will be evaluated to decide if they must
be selected or excluded in the context of the systematic review
have been defined by the authors of the paper taking into account
Kitchenham’s proposals [19,20]. These criteria, Inclusion Criteria
(IC) and Exclusion Criteria (EC), are shown in Table 3.
The process performed to obtain and evaluate primary studies
according to the defined inclusion and exclusion criteria is illus-
trated as a flow diagram in Fig. 1. This flow diagram shows two
main groups of activities. The goal of the first group is the selection
of primary studies. The second group of activities aims to extract
the information of the selected primary studies. Information
extraction will be presented later in this paper in Section 3.4.
With regard to the selection of primary studies, the analysis of
the title and the keywords will be the main inclusion criteria. In
case this information is not enough to decide about the inclusion
or the exclusion of the study then the abstract will be also analysed
and the full text, if necessary. Initially all types of primary studies
related to ITSM Process Improvement according to ISO/IEC 15504
will be taken into account. More concretely, the focus will be on
studies presenting ITSM Process Improvement models based on
the ISO/IEC 15504 assessment framework and the results of the
application of these models to software companies.
Table 4 shows the distribution of the studies obtained from
each search source. The format of this table is adapted from other
systematic reviews in software engineering [22–24]. As a result of
the search execution 1944 studies have been obtained for further
evaluation. Table 4 shows the number of initial studies obtained
from each source (see the column ‘‘Discovered’’).
After applying the inclusion criteria IC1, IC2, IC3 and IC4,
defined in Table 3, only 82 of the 1944 discovered articles have
been considered as relevant articles. Applying the criterion EC2
for the exclusion of duplicated articles, only 49 articles have been
obtained. From these, applying the criterion EC1, finally 28 articles
have been selected as primary studies. These results are shown in
Table 1
Search strings.
Search strings
1 (15504 OR SPICE) AND (ITSM OR ‘‘IT service’’ OR ‘‘service’’)
2 (ITSM OR ‘‘IT service’’ OR ‘‘service’’) AND (improvement OR assessment)
AND (15504 OR SPICE)
3 (15504 OR SPICE) AND ‘‘service’’ AND (ITIL OR 20000 OR ‘‘CMMI-SVC’’)
A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 241
the last row of Table 4. The complete list of selected primary
studies is presented in Appendix A of this article. This primary
study selection has been reviewed by the authors in order to guar-
antee the quality of the included studies.
3.4. Information extraction
Once primary studies are selected, the extraction of relevant
information begins [21]. The criteria by which the information ob-
tained from the studies has been included were defined. These
Information Inclusion Criteria (IC
inf
) are presented in Table 5.
To analyze the data obtained from the selected primary studies
and to standardize the way in which information should be regis-
tered, a data extraction form has been designed in order to meet
our particular research goals. This form, which is shown in Table
6, has been used to record comments, impressions and the most
important ideas from each primary study. The structure and con-
tents of this form are based on the information extraction format
proposed in [25] and other systematic reviews [22–24]. The con-
tents are grouped into objective and subjective results. The first
category, objective results, refers to those results that can be
directly extracted from the selected primary studies. These results
have been organized in the form into the following four sections:
Study identification: the main data to identify the primary
study: a consecutive number assigned to the paper, the publica-
tion title, its authors, the author’s contact information, the
journal or conference in which it was published, the date of
publication and the source from which it was obtained.
Study methodology: the type of the study, the country in which
it was developed or applied, the improvement strategy goals
and the ITSM Process Improvement models used or developed,
among other information.
Study results: information about the improved processes, the
key factors for successful improvement and the conclusions.
Study problems: study limitations and the deviation between
the expected and the obtained results.
The second category, subjective results, refers to those results
that cannot be extracted directly from the study. The ‘‘Results
Extraction’’ section of the form shows the study outcomes, the
study contribution, as well as other issues.
After a complete and detailed reading of each primary study
and an unbiased evaluation of the information, the objective and
subjective results have been extracted. During the information
extraction execution, the authors’ perception of some aspects var-
ied. Nevertheless, none of them was considered an important
divergence. The authors reached a consensus and different percep-
tions were considered complementary to obtain a comprehensive
analysis of the study.
3.5. Results summarization
The last stage of the systematic review protocol aims to present
the data resulting from the selected primary studies [21].
Results statistical calculus: with the purpose of showing the rel-
evant findings of this systematic review, a statistical analysis
has been conducted with the information extracted from the
selected primary studies. A more detailed discussion of these
results is further presented in Section 4of this article.
Results presentation in tables: with the aim of facilitating the
analysis, the results obtained from the systematic review have
been summarized in tables and displayed in graphics under dif-
ferent perspectives. These results are shown in Tables 7–10 and
in Fig. 2 in Section 4.
Sensitivity analysis: it was not applied.
Plotting: it was not applied.
Final comments:
–Number of studies: 1944 studies were found and 28 of them
were considered primary studies.
–Search, selection and extraction bias: none search, selection or
information extraction biases that could invalidate the sys-
tematic review results were identified.
–Publication bias: none was defined.
–Inter-reviewers variation: there were no divergences between
reviewers regarding the systematic review results.
–Results application: the obtained results suggest that new
ITSM Process Improvement models have arisen to satisfy
the growing demand of software companies for managing
the services they provide.
–Recommendations: the results of the systematic review pre-
sented in this paper should be applied to get an idea of the
state of the art of ITSM Process Improvement based on the
ISO/IEC 15504 standard.
4. Results and discussion
This section presents a summary of the results after the system-
atic review execution.
Table 2
List of sources.
Source Name Web site
1 ACM Portal (Digital Library &
Guide)
http://portal.acm.org/portal.cfm
2 IEEE Computer Society Digital
Library
http://www.computer.org/portal/
web/csdl
3 IEEE Xplore http://ieeexplore.ieee.org
4 Springer Link http://www.springerlink.com
5 ScienceDirect http://www.sciencedirect.com
6 Wiley InterScience http://
www.interscience.wiley.com
7 CiteSeerX http://citeseerx.ist.psu.edu
8 IET Digital Library http://www.ietdl.org
9 ISI Web of Knowledge http://www.isiknowledge.com
10 Google Scholar http://scholar.google.com
11 EuroSPI (Proceedings of) (since
2004)
http://www.eurospi.net
12 SPICE (Proceedings of) (since
2003)
http://www.spiceconference.com
Table 3
Definition of studies inclusion and exclusion criteria.
Criterion Description
IC1 Include papers whose title is related to ITSM Process Improvement according to ISO/IEC 15504
IC2 Include papers that contain keywords that match with those defined in the search string
IC3 Include papers whose abstract is related to the topic under consideration
IC4 Include papers that contain information related to the definition or application of ITSM Process Improvement models
EC1 Exclude those papers that refer to ITSM Process Improvement and ISO/IEC 15504 separately, without showing any kind of relationship between both topics
EC2 Exclude all duplicated papers
242 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247
Fig. 1. Procedure for executing the systematic review.
Table 4
Distribution of studies by source.
Source Search date Discovered Relevant Not repeated Primaries %
ACM Portal 25/07/2011 15 0 0 0 0
IEEE Computer Society Digital Library 25/07/2011 98 3 2 2 7
IEEE Xplore 25/07/2011 43 2 0 0 0
Springer Link 26/07/2011 178 9 3 3 11
ScienceDirect 25/07/2011 106 0 0 0 0
Wiley InterScience 25/07/2011 11 1 0 0 0
CiteSeerX 26/07/2011 99 2 1 1 4
IET Digital Library 26/07/2011 0 0 0 0 0
ISI Web of Knowledge 26/07/2011 39 4 2 2 7
Google Scholar 26/07/2011 778 42 30 9 32
SPICE 27/07/2011 211 8 4 4 14
EuroSPI 27/07/2011 366 11 7 7 25
Total 1944 82 49 28 100
Table 5
Definition of information inclusion criteria.
Criterion Description
IC1
inf
Identify existent methodologies, techniques, methods and procedures for ITSM Process Improvement
IC2
inf
Collect information about proposed initiatives or models used to assess or improve ITSM processes
IC3
inf
Collect information about ITSM Process Improvement strategies followed by software organizations
IC4
inf
Collect information about improved processes and key factors for successful ITSM Process Improvement in software organizations
A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 243
4.1. Classification of studies
After the extraction of relevant information from each primary
study, it was possible to determine that these studies could be clas-
sified into the four categories presented in Table 7.
According to Table 7, most of the studies provide a new ISTM
Process Improvement model based on ISO/IEC 15504. In addition,
there is an interest in analysing the need of ITSM Processes
Improvement, as well as combining ISO/IEC 15504 and ITSM Pro-
cess Improvement standards. Finally, five primary studies offer a
summary of the results of the application of new ITSM Process
Improvement models.
4.2. Results from the analysis of primary studies
This section details the results obtained after analysing the pri-
mary studies. These results have been classified into two different
groups.
4.2.1. Standards used for ITSM Process Improvement
Table 8 presents a classification of the primary studies regard-
ing each different standard used for ITSM Process Improvement.
Regarding process reference models, the results from the anal-
ysis of the primary studies reveal that ISO/IEC 20000 is the ITSM
process reference model most frequently used. ITIL V2 and ITIL
V3 follow in order of importance. ITIL V1, which is out of market,
is only mentioned in two primary studies. Moreover, ISO new work
item proposals, specifically ISO/IEC 15504-8 and ISO/IEC 20000-4,
define an ITSM process assessment model and an ITSM process ref-
erence model, respectively. Finally, CMMI models, including the
CMMI-SVC constellation, have also been used for guidance in ITSM
Process Improvement initiatives.
4.2.2. New ITSM Process Improvement initiatives
Table 9 presents the initiatives specifically created with the aim
of developing new ITSM Process Improvement models.
Table 10 shows the distribution of these new ITSM Process
Improvement models depending on the process assessment meth-
od and the process reference model they use.
As Table 10 shows, both SCAMPI and ISO/IEC 15504-2 have been
used as process assessment methods. Then, the feasibility of using
these measurement frameworks for ITSM process assessment and
improvement has already been tested.
Starting with the first row, MITO and IT Service CMM are two
ITSM Process Improvement initiatives with self-developed process
reference models which use SCAMPI as assessment method.
Regarding ITSM initiatives based on ISO/IEC 15504, the assess-
ment method defined in ISO/IEC 15504 Part 2 has been used to
assess both ITIL and ISO/IEC 20000 processes, as well as to assess
new developed process reference models:
SPINI+ defines an ITSM process library from ITIL and ISO/IEC
20000.
The TIPA, NOEMI and SPICE Lite [ITSM] models use ITIL (V2) as
process reference model.
TickIT Plus core includes, among others, ISO/IEC 20000 as
optional standard in its certification model.
SPICE 1-2-1 for ISO 20000 uses the ISO/IEC 15504-2 assessment
method for ISO/IEC 20000 process assessment.
The NiCE model uses the ISO/IEC 15504-2 assessment method
to assess the processes provided by NOVE-IT, a set of IT procure-
ment, development, operation and service provision processes.
Table 6
Information extraction form.
Information extraction form
Study identification
Consecutive number:
Publication title:
Authors:
Contact information:
Journal/Conference:
Date:
Source:
Study methodology
Type of Study:
Country:
Improvement strategy goals:
Models/Standards used:
Models/Standards developed:
Study results
Improved processes:
Improvement success key factors:
Conclusions:
Study problems
Limitations:
Deviation between expected and obtained results:
Results extraction
Study outcomes:
Study contribution:
Other issues:
Table 7
Classification of primary studies.
Category Primary studies Percentage of primary studies
1 Analysis of the need of assessing and improving ITSM processes. 7, 8, 14, 17, 20, 25 21.4% (6/28)
2 Combining ISO/IEC 15504 and ITSM Process Improvement frameworks. 2, 5, 12, 15, 16, 24 21.4% (6/28)
3 Development of an ISO/IEC 15504-based model, framework
or approach for ITSM Process Improvement.
1, 3, 4, 6, 9, 11, 18, 19, 21, 22, 28 39.3% (11/28)
4 Results of the application of an ITSM Process Improvement model. 10, 13, 23, 26, 27 17.9% (5/28)
Table 8
Standards used for ITSM.
Standard Primary studies Percentage of primary studies
1 ISO/IEC 20000 1, 3, 4, 5, 7, 8, 12, 14, 15, 16, 18, 19, 21, 22, 25, 26, 28 60.7% (17/28)
2 ITIL V1 20, 25 7.1% (2/28)
3 ITIL V2 1, 2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 15, 22, 26, 27 53.6% (15/28)
4 ITIL V3 4, 8, 11, 17, 19, 23, 27, 28 28.6% (8/28)
5 ISO/IEC 15504–8 & 20000–4 4, 5, 7, 8, 19, 21, 24, 26 28.6% (8/28)
6 CMM/CMMI 7, 11, 16, 17, 20, 25 21.4% (6/28)
7 CMMI-SVC 7, 8, 16, 19 14.3% (4/28)
244 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247
4.3. Temporal view of primary studies
Fig. 2 shows the temporal distribution of primary studies result-
ing from the review carried out. Since most of the models were
developed from 2006 onwards, there is a notable increment from
this year in the number of published studies showing the
experiences of their development and usage. This recent increase
may be a reflection of a growing awareness of the importance of
ITSM Process Improvement.
5. Conclusions and future work
In this systematic review we have formulated a research ques-
tion which aims to identify existing ITSM Process Improvement
initiatives based on the ISO/IEC 15504 standard for process assess-
ment and improvement.
By examining the results obtained after the systematic review
execution, we have detected nine different ITSM Process Improve-
ment initiatives in which new models for the assessment of this
kind of processes have already been developed. We have observed
that all these models are composed of an ITSM process reference
model and an assessment method. This review suggests that ISO/
IEC 20000, ITIL (V2 and V3) and CMMI-SVC are the ITSM process
reference models most frequently used. Both SCAMPI and ISO/IEC
15504-2 have been used by these new ITSM Process Improvement
models as process assessment methods.
If we focus on the models based on ISO/IEC 15504, seven of the
nine detected models use ISO/IEC 15504 conformant process
assessment methods to assess the ITSM processes defined by ITIL
or ISO/IEC 20000. Then, the results suggest that the feasibility of
using the ISO/IEC 15504 measurement framework for ITSM process
assessment and improvement has already been examined and
proved during the last decade. With that same purpose, ISO started
the development of an ITSM Process Reference Model (ISO/IEC
20000-4) and its Process Assessment Model (ISO/IEC 15504-8)
conformant to the ISO/IEC 15504 assessment method.
Bearing in mind the final goal of our research (the development
of a new method with the necessary guidelines for the assessment
according to ISO/IEC 15504 of both software lifecycle and ITSM
processes reducing the amount of effort), further work would have
to be performed. To meet this goal the first step would consist of
examining SPINI+, TickIT Plus and SPICE 1-2-1 for ISO 20000, the
three models which use the ISO/IEC 15504 assessment method to
assess the ISO/IEC 20000 processes. These models will be
Table 9
ITSM process improvement initiatives.
Initiative Primary studies Year Description
TIPA (formerly
called AIDA)
1, 2, 3, 4, 7, 13, 22,
23, 26, 27
2007 ‘‘The TIPA model was inspired by ITIL best practices, with the goal to enable objective ITSM capability assessments.
The references used to create the PRM and the PAM were the Service Support and Service Delivery books published
by OGC.’’ (Primary study 1)
SPINI+ 4, 28 2007 ‘‘The SPINI-methodology was supplemented with process modeling methods and the process library was extended
with IT Service Management processes (ITIL, ISO/IEC 20000).’’ (Primary study 28)
NiCE(NOVE-IT) 6 2002 ‘‘A process model for IT procurement, development, operation, and service provision (also called NOVE-IT) was
developed.’’ ‘‘For the NOVE-IT project, it was decided to create an assessment model consisting of a process
dimension adopted as-is from Part 2 of ISO/IEC TR 15504. The model is called NOVE-IT Capability dEtermination, or
NiCE.’’ (Primary study 6)
MITO 6, 25 2000 ‘‘A maturity model for the assessment of companies or parts of companies providing IT Operations services. The
assessment is process based thus able to reveal the potentials for improvement. The model combines features of the
SEI CMM and the assessment method of the EFQM.’’ (Primary study 25)
NOEMI 9, 10 2003 ‘‘The development and experimentation of an IT process assessment methodology especially designed to be used in
very small enterprises (VSEs).’’ ‘‘The processes themselves are based on a combined approach of ISO/IEC 15504 and
the IT Infrastructure Library.’’ (Primary study 9)
SPiCE Lite [ITSM] 11 2008 ‘‘SPiCE Lite [ITSM] supports the guided assessment of ITIL IT organization processes. SPiCE applies its own maturity
level model to ITIL processes. It thus provides a qualitative evaluation of process maturity in accordance to the
SPiCE-process maturity model (ISO/IEC 15504).’’ (Primary study 11)
TickIT Plus 14 2008 ‘‘ISO/IEC 15504, ISO/IEC 15288 and ISO/IEC 12207 will form the core around which TickIT Plus is designed: the
capability model and the process structure. Each of other three standards: ISO/IEC 20000, ISO/IEC 27001 and ISO/IEC
25030, termed ‘Requirements Standards’, are all optional standards that can be included under the TickIT Plus
certification.’’ (Primary study 14)
SPICE 1-2-1 for
ISO 20000
18 2007 ‘‘So ISO/IEC 15504 can be used as universal model for process assessment and process improvement. Following this
idea I have defined an ‘ISO 20000 – PAM’, a process assessment model for IT Service Management according to ISO/
IEC 20000–1:2005. Based on the ISO 20000 – PAM we have implemented an Assessment Tool for IT Service
Management: SPICE 1-2-1 for ISO 20000.’’ (Primary study 18)
IT service CMM 20 1998 ‘‘We propose an Information Technology Service Capability Maturity Model (IT Service CMM) that can be used to
assess the maturity of IT service processes and identify directions for improvement.’’ (Primary study 20)
Table 10
Classification of ITSM Process Improvement models.
Assessment
method
Process reference model
ITIL ISO/IEC 20000 Self-developed
SCAMPI – – MITO
IT service CMM
NiCE (NOVE-IT)
ISO/IEC 15504-2 SPINI+ SPINI+
TIPA TickIT plus
NOEMI SPICE 1-2-1
for ISO 20000SPICE lite [ITSM]
Fig. 2. Number of primary studies included in the review.
A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 245
considered as a starting point for the assessment of the ISO/IEC
20000 processes. Then, our work will focus on the development
of a new process assessment model based on the ISO/IEC 15504
measurement framework for the assessment and improvement of
both the ISO/IEC 15504-5 software development processes and
the ITSM processes provided by ISO/IEC 20000-4. As some pro-
cesses of both process reference models are overlapped, the com-
monalities between them should be carefully analysed.
Acknowledgments
This work is supported by CICYT-TIN2010-20057-C03-03 ‘‘Sim-
ulación aplicada a la gestión de equipos, procesos y servicios’’,
Sim4Gest.
Appendix A. List of primary studies in the systematic review
Next, selected primary studies in the development of this sys-
tematic review are presented.
A.1. List of primary studies in the systematic review
[1] B. Barafort, A. Rousseau, ‘‘Sustainable Service Innovation
Model: A Standardized IT Service Management Process
Assessment Framework’’, Software Process Improvement:
EuroSPI 2009. Communications in Computer and Information
Science, vol. 42, pp. 69–80, Springer-Verlag, Heidelberg, 2009.
[2] B. Barafort, B. Di Renzo, O. Merlan, ‘‘Benefits resulting from
the combined use of ISO/IEC 15504 with the Information
Technology Infrastructure Library (ITIL)’’, International Con-
ference on Product Focused Software Process Improvement
(PROFES 2002). Lecture Notes in Computer Science, vol.
2559, pp. 314–325, Springer-Verlag, 2002.
[3] B. Barafort, B. Di Renzo, V. Lejeune, S. Prime, J.-M. Simon,
‘‘ITIL Based Service Management measurement and ISO/IEC
15504 process assessment: a win–win opportunity’’,
Proceedings of the 5th International SPICE Conference on
Process Assessment and Improvement (SPICE 2005),
Klagenfurt, Austria, 2005.
[4] B. Barafort, D. Jezek, T. Mäkinen, S. Stolfa, T. Varkoi, I. Vond-
rak, ‘‘Modeling and Assessment in IT Service Process
Improvement’’, Software Process Improvement: EuroSPI
2008. Communications in Computer and Information Sci-
ence, vol. 16, pp. 117–128, Springer-Verlag, 2008.
[5] B. Barafort, A. Renault, M. Picard, S. Cortina, ‘‘A transforma-
tion process for building PRMs and PAMs based on a collec-
tion of requirements - Example with ISO/IEC 20000’’,
Proceedings of the International SPICE Conference on Pro-
cess Improvement and Capability dEtermination (SPICE
2008), Nuremberg, Germany, 2008.
[6] A. Cass, C. Völcker, P. Sutter, A. Dorling, H. Stienen, ‘‘SPiCE in
Action - Experiences in Tailoring and Extension’’, Proceed-
ings of the 28th Euromicro Conference (EUROMICRO 2002),
Dortmund, Germany, 2002.
[7] Cater-Steel, ‘‘Integration of service management with
CMMI
Ò
and SPICE’’, 5th Annual SEPG Australia Conference,
Gold Coast, Australia, August 2007.
[8] Cater-Steel, ‘‘IT Service Departments Struggle to Adopt a Ser-
vice-Oriented Philosophy’’, International Journal of Informa-
tion Systems in the Service Sector, vol. 1, issue 2, pp. 69–77,
2009.
[9] Di Renzo, C. Feltus, ‘‘Process assessment for use in very small
enterprise: the NOEMI assessment methodology’’, Proceed-
ings of the European Software Process Improvement Confer-
ence (EuroSPI 2003), Graz, Austria, December 2003.
[10] Di Renzo, C. Feltus, S. Prime, ‘‘Collaborative management for
ICT process improvement in SME: experience report’’, Pro-
ceedings of the European Software Process Improvement Con-
ference (EuroSPI 2004), Trondheim, Norway, November 2004.
[11] T. Goldschmidt, A. Dittrich, M. Malek, ‘‘Quantifying Critical-
ity of Dependability-Related IT Organization Processes in
CobiT’’, IEEE Proceedings of 15th Pacific Rim International
Symposium on Dependable Computing (PRDC 2009), Shang-
hai, China, 2009.
[12] Grandry, E. Dubois, M. Picard, A. Rifaut, ‘‘Managing the
Alignment between Business and Software Services Require-
ments from a Capability Model Perspective’’, Towards a Ser-
vice-Based Internet (ServiceWave 2008), Lecture Notes In
Computer Science, vol. 5377, pp. 171–182, 2008.
[13] R. Hilbert, A. Renault, ‘‘Assessing IT Service Management
Processes with AIDA - Experience Feedback’’, Proceedings
of the European Systems & Software Process Improvement
and Innovation Conference (EuroSPI 2007), Potsdam, Ger-
many, September 2007.
[14] Irving, ‘‘TickIT Plus - the Future of TickIT!’’, TickIT Interna-
tional, issue 2Q08, pp. 3–7, 2008.
[15] A. Kramer, ‘‘ISO/IEC 15504 and ITIL’’, International SPICE
Days 2008, Prague, Czech Republic, June 2008.
[16] D. Malzahn, ‘‘A service extension for SPICE?’’, Proceedings of
the International SPICE Conference on Process Assessment
and Improvement (SPICE 2007), Seoul, South Korea, May
2007.
[17] D. Malzahn, ‘‘Assessing - learning - improving, an integrated
approach for self assessment and process improvement sys-
tems’’, Proceedings of the fourth International Conference on
Systems (ICONS 2009), pp. 126–130, Cancun, Mexico, March
2009.
[18] A. Nehfort, ‘‘SPICE Assessments for IT Service Management
according to ISO/IEC 20000–1’’, International SPICE Days
2007, Frankfurt, Germany, June 2007.
[19] R. Nevalainen, M. Johansson, ‘‘Comparison of CMMI-SVC and
ISO20000 – A Case Study’’, Proceedings of the European Sys-
tems & Software Process Improvement and Innovation Con-
ference (EuroSPI 2008), Dublin, Ireland, September 2008.
[20] Niessink, H. Van Vliet, ‘‘Towards Mature IT Services’’, Soft-
ware Process - Improvement and Practice, vol. 4, issue 2,
pp. 55–71, June 1998.
[21] M. Picard, A. Renault, S. Cortina, ‘‘How to Improve Process
Models for Better ISO/IEC 15504 Process Assessment’’, Sys-
tems, Software and Services Process Improvement (EuroSPI
2010), CCIS 99, pp. 130–141, 2010.
[22] Public Research Centre Henri Tudor: B. Barafort, V. Betry, S.
Cortina, M. Picard, M.St-Jean, A. Renault, O. Valdés, ‘‘ITSM
Process Assessment Supporting ITIL
Ò
’’, Van Haren Publish-
ing, Zaltbommel, December 2009.
[23] A. Renault, B. Barafort, ‘‘TIPA: 7 years experience with SPICE
for IT Service Management’’, Proceedings of the European
System & Software Process Improvement and Innovation
Conference (EuroSPI 2011), Roskilde, Denmark, June 2011.
[24] A. Rifaut, ‘‘Goal-Driven Requirements Engineering for Sup-
porting the ISO 15504 Assessment Process’’, Software Pro-
cess Improvement: EuroSPI 2005. Lecture Notes in
Computer Science, vol. 3792, pp. 151–162, Springer-Verlag,
2005.
[25] Q. Scheuing, K. Frühauf, W. Schwarz, ‘‘Maturity model for IT
operations (MITO)’’, Proceeding of the 2nd World Congress
on Software Quality, Yokohama, Japan, September 2000.
[26] M. St-Jean, ‘‘TIPA to keep ITIL going and going’’, Proceedings
of the European Systems & Software Process Improvement
and Innovation Conference (EuroSPI 2009), Alcalá de Hen-
ares, Spain, September 2009.
246 A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247
[27] M. St-Jean, A.-L. Mention, ‘‘How to evaluate benefits of
Tudor’s ITSM Process Assessment?’’, Proceedings of the
International SPICE Conference on Process Improvement
and Capability dEtermination (SPICE 2009), Turku, Finland,
June 2009.
[28] T. Varkoi, T. Makinen, ‘‘Proactive elicitation of software pro-
cess improvements’’, Proceedings of the Portland Interna-
tional Conference on Management of Engineering &
Technology (PICMET 2008), pp. 1576–1579, Cape Town,
South Africa, July 2008.
References
[1] Software Engineering Institute (SEI), CMMI-DEV, CMMI
Ò
for Development
(CMMI-DEV) Version 1.3, 2010.
[2] International Organisation for Standardization (ISO/IEC), ISO/IEC 15504-
1:2004 Information Technology – Process Assessment – Part 1: Concepts and
Vocabulary, 2004.
[3] A. Mas, E. Amengual, La mejora de los procesos de software en las pequeñas y
medianas empresas (pyme). Un nuevo modelo y su aplicación en un caso real,
Revista Española de Innovación, Calidad e Ingeniería del Software (REICIS) 1 (2)
(2005) 7–29.
[4] A. Mas, E. Amengual, A Method for the implementation of a quality
management system in software SMEs, in: Proceedings of the Twelfth
International Conference on Software Quality Management, British Computer
Society, March 2004, pp. 61–74.
[5] E. Amengual, A. Mas, Software process improvement in small companies: an
experience, in: Proceedings of the European Systems & Software Process
Improvement and Innovation Conference (EuroSPI 2007), Potsdam, Germany,
September 2007.
[6] A. Mas, B. Fluxà, E. Amengual, Lessons learned from an ISO/IEC 15504 SPI
programme in a company, in: Proceedings of the European Systems & Software
Process Improvement and Innovation Conference (EuroSPI 2009), Alcalá de
Henares, Spain, September 2009.
[7] A.L. Mesquida, A. Mas, E. Amengual, La madurez de los servicios TI, Revista
Española de Innovación, Calidad e Ingeniería del Software (REICIS) 5 (2) (2009)
77–87.
[8] International Organisation for Standardization (ISO/IEC), ISO/IEC 15504-
2:2004 Software Engineering – Process Assessment – Part 2: Performing an
Assessment, 2003.
[9] International Organisation for Standardization (ISO/IEC), ISO/IEC NP TR 15504-
8 Information Technology – Process Assessment – Part 8: An Exemplar Process
Assessment Model for IT Service Management, Standard Under Development,
ISO/IEC JTC1/SC7 3798, NWI Proposal, 2007.
[10] Office of Government Commerce (OGC), ITIL Version 3 Service Strategy, 2007.
[11] Office of Government Commerce (OGC), ITIL Version 3 Service Design, 2007.
[12] Office of Government Commerce (OGC), ITIL Version 3 Service Transition,
2007.
[13] Office of Government Commerce (OGC), ITIL Version 3 Service Operation,
2007.
[14] Office of Government Commerce (OGC), ITIL Version 3 Continual Service
Improvement, 2007.
[15] International Organisation for Standardization (ISO/IEC), ISO/IEC 20000-
1:2005 Information Technology – Service Management – Part 1:
Specification, 2005.
[16] International Organisation for Standardization (ISO/IEC), ISO/IEC 20000-
2:2005 Information Technology – Service Management – Part 2: Code of
practice, 2005.
[17] International Organisation for Standardization (ISO/IEC), ISO/IEC CD TR 20000-
4 Information technology – Service Management – Process Reference Model,
Standard Under Development, ISO/IEC JTC1/SC7 3797, NWI Proposal, 2007.
[18] Software Engineering Institute (SEI), CMMI-SVC, CMMI
Ò
for Services (CMMI-
SVC) Version 1.3, 2010.
[19] B. Kitchenham, Guidelines for Performing Systematic Literature Reviews in
Software Engineering Version 2.3, Technical Report EBSE-2007-01, Software
Engineering Group, School of Computer Science and Mathematics, Keele
University, and Department of Computer Science, University of Durham, July
2007.
[20] B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El
Emam, J. Rosenberg, Preliminary guidelines for empirical research in software
engineering, IEEE Transactions on Software Engineering 28 (8) (2002) 721–
734.
[21] J. Biolchini, P. Gomes, A.C. Cruz, G. Horta, Systematic Review in Software
Engineering, Systems Engineering and Computer Science Department, COPPE/
UFRJ, Technical Report RT-ES679/05, Rio de Janeiro, Brasil, May 2005.
[22] J.A. Calvo-Manzano, G. Cuevas, G. Gasca, T. San Feliu, State of the art for risk
management in software acquisition, ACM SIGSOFT Software Engineering
Notes 34 (4) (2009) 1–20.
[23] J.A. Calvo-Manzano, G. Cuevas, G. Gómez, J. Mejía, M. Muñoz, F. Rabbi, T. San
Feliu, Service level management for IT services in small settings: a systematic
review, in: Proceedings of the European Software Process Improvement
Conference (EuroSPI 2009), Alcalá de Henares, Spain, September 2009.
[24] F.J. Pino, F. García, M. Piattini, Software process improvement in small and
medium software enterprises: a systematic review, Software Quality Journal
16 (2) (2008) 237–261.
[25] D. Cruzes, M. Mendonça, V. Basili, F. Shull, M. Jino, Extracting Information from
Experimental Software Engineering Papers, in: Proceedings of the XXVI
International Conference of the Chilean Society of Computer Science (SCCC
2007), 2007, pp. 105–114.
A.L. Mesquida et al. / Information and Software Technology 54 (2012) 239–247 247
