Content uploaded by Marco Kuhrmann
Author content
All content in this area was uploaded by Marco Kuhrmann on Jun 10, 2015
Content may be subject to copyright.
How do Artifact Models help direct SPI Projects?
Marco Kuhrmann
University of Southern Denmark
Mærsk Mc-Kinney Møller Institute
Odense, Denmark
kuhrmann@mmmi.sdu.dk
Ita Richardson
University of Limerick
Lero - the Irish Software Research Centre
Limerick, Ireland
Ita.Richardson@lero.ie
ABSTRACT
To overcome shortcomings associated with software process
improvement (SPI), we previously recommended that pro-
cess engineers focus on the artifacts to be developed in SPI
projects. These artifacts should define desired outcomes,
rather than specific methods. During this prior research,
we developed a model for Artifact-based Software Process
Improvement & Management (ArSPI). We are now carrying
out studies to confirm our claims that ArSPI will provide
benefits such as quality assurance. In this paper, we re-
port on an experimental setting in which we developed and
analyzed a strategy to use artifact models to direct pro-
cess model improvement. We analyzed a process specifica-
tion, the realized model, and the generated electronic pro-
cess guide. We used ArSPI v0.9 as our process model and
the Capability Maturity Model Integration (CMMI) as an
external reference to provide a set of overall improvement
goals. We propose an effective approach to analyze and im-
prove a process model. In addition, the analysis revealed
issues with ArSPI realization, which will be corrected in the
next major release.
Categories and Subject Descriptors
D.2.9 [Software Engineering Management]: Software
process models
General Terms
Management, Experimentation
Keywords
software process improvement, software process management,
SPI, SPM, artifact-orientation
1. INTRODUCTION
A variety of software process improvement (SPI) models is
presented in literature. Yet, within these papers, the feasi-
bility of these approaches is oftentimes shown by identifying
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
the realized improvements for the clients and not for other
stakeholders. Therefore, as appropriateness is assumed by
sufficient customer satisfaction, the quality of the realized
model is usually not subject to discussion.
However, SPI is a long-term endeavor in which learning
organizations have to deal with evolving processes. For in-
stance, if a software process was initially specified and re-
alized as a process model, it would be unusual for users to
implement this model by the letter. Rather, they normally
implement it in a situation-specific manner [11]. When it
comes to process improvement, process engineers also face
the problem of harmonizing originally defined processes with
the processes which are actually being used.. SPI requires
that all parts of a process such as process specification, pro-
cess realization, and all deployed process assets, e.g., elec-
tronic process guides, templates, and collected data and ex-
periences, are considered. Consequently, process engineers
must take a broader perspective, as, otherwise, they will en-
counter the risk of improving a process that is never applied
and, eventually, is neglected.
Problem Statement. Since the process model’s quality
is often out of scope within SPI projects, companies risk
the introduction of “technical debt” into their process mod-
els. McConnell [9] defines this as “a design or construction
approach that’s expedient in the short term, but that cre-
ates a technical context in which the same work will cost
more to do later than it would cost to do now”. Further-
more, if processes evolve over time, analyzing the process,
locating gaps and the improvement-relevant parts generates
extra effort [13]. Hence, process engineers do not only have
to improve the process according to new requirements, but
also have to deal with previously introduced flaws. This in-
creases the complexity of SPI projects thus increasing cost.
Objectives. To overcome shortcomings associated with
SPI, we recommended concentrating on the artifacts to be
developed in SPI projects. For this, we developed a model
for Artifact-based Software Process Improvement & Man-
agement (ArSPI; [8]). By relying on artifact models, this
approach explicitly addresses process-model-related quality
questions. The goal of this research is to study instruments,
which analyze and improve the quality of process models
(especially a process model’s completeness and consistency)
to support structured SPI.
Contribution. In this paper, we present an approach that
helps process engineers to determine the quality of a soft-
ware process and to systematically identify and prioritize
improvement candidates. The approach relies on the identi-
© ACM. PREPRINT. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution.
The definitive version was published in the conference/workshop proceedings.
fication of process modeling artifacts and their justification
with (general) quality attributes, process requirements and
external standards, and other complementary material. We
report on an exploratory case study, which applied this ap-
proach to ArSPI. The case study revealed about 80 techni-
cal as well as content-related issues in the studied process
model, which were not found in previously conducted quality
assurance studies. Initial results suggest that the proposed
approach is a useful tool, which can support process engi-
neers in critically reviewing process models and in improving
the quality of process models to support long-term process
evolution.
Outline. The remainder of the paper is organized as fol-
lows: Section 2 summarizes related work. In Sect. 3, we
outline the approach, and discuss the results from the case
study in Sect. 4. We conclude the paper in Sect. 5.
2. RELATED WORK
In SPI research, the quality of the process model is of-
ten not in scope. For example, Unterkalmsteiner et al. [14]
present a comprehensive review of evaluation and measure-
ment in SPI. They list 12 evaluation strategies and 14 suc-
cess indicators. Among 148 publications studied by the first
author, no one addresses the quality of the process model as
a success factor. Moreover, (standardized) approaches, such
as CMMI [2] or ISO/IEC 15504 [4] assess a process by us-
ing only content- the structure and the underlying model(s),
apparently, do not play a role. We also note that most SPI
initiatives and approaches are evaluated by conducting com-
prehensive case study research with industry, e.g., Raninen
et al. [12], to look for those attributes as collected in [14].
The research we present in this paper thus aims at closing
a gap in literature by adding another perspective to SPI.
3. APPROACH
The construction procedure of ArSPI [7] motivates the
research presented in this paper. We looked for an instru-
ment taking advantage of the artifact-oriented design prin-
ciple to provide support, which would easily detect issues in
the process model and would derive and prioritize improve-
ment measures.
ArSPI Context. Our focus in this paper is the ArSPI arti-
fact Actual Process, which is, together with an overall Vision
and a set of Changes, the major input for an improvement
project (iteration) [8]. An actual process has a variety of
facets: it comprises a process specification, a process realiza-
tion, and a process in use (Figure 1). A process specifica-
tion can be represented, for example by standards, norms,
and company-wide blueprints. A process realization is often
represented by some kind of model1, for example, through a
Software a Systems Process Engineering Metamodel Speci-
fication [10] and complementing tools. Finally, the process
in use is represented by a variety of artifacts, for example,
electronic process guides, documents from past projects, and
process user experiences. Calling for a balanced improve-
ment towards user requirements combined with the require-
ment to consider process model quality improvements, each
facet needs to be accounted for in SPI projects as such. Most
1In this stage of our research, we focus on model-based soft-
ware processes only.
challenging is the process in use. In addition, over time, pro-
cesses evolve and the actual used process can contradict the
-process as originally defined.
Idea and Goals. To analyze a software process for com-
pleteness and consistency of the process model and to aid
the development and prioritization of improvement require-
ments, our approach aims to achieve the following goals:
Goal 1: Determine a process model’s quality in terms of
completeness and consistency, thus providing a sound
and consistent basis for SPI projects.
Goal 2: Determine the current state of a process under in-
vestigation in relation to overall improvement goals.
Goal 3: Determine to which extent “cheap” improvements,
help to address the improvement goals in general. An
example of such improvements would be updating the
model’s structure, e.g., regarding consistency.
In a nutshell, the basic idea is to analyze a process for com-
pleteness and consistency which can be considered at two
levels: 1) the (process) model is complete and consistent
based on Bass et al. [1], 2) the resulting description is com-
plete (is the process description missing something) and con-
sistent (is the description free of contradictions). We ana-
lyze to which extent “just fixing” completeness and consis-
tency issues helps to address improvement requirements. In
addition, we relate found issues to (general) improvement
requirements and their prioritization. As we rely on an
artifact-based approach, finding all parts of the Actual Pro-
cess is straightforward; to a large extent, this information
can be generated from the artifact models.
3.1 Analysis Strategy
The approach to our research is illustrated in Figure 1.
Process
Specification
Process
Realization
Process
in Use
ArSPI:Actual Process
Process
Requirements
Artifact 1
Artifact 1.1
Artifact …
Artifact 1.n
Spec. Artifacts Real. inUse
yes/no/? yes/no/? yes/no/?
Artifact is complete…? yes/no/?
Artifact has a responsible role…? yes/no/?
All required artifacts are implemented? 1-5
All dependencies are implemented? 1-5
<topic> is adequately addressed…? 1-5
<topic> is described with sufficient detail…? 1-5
…
Requirements-based checklist…
1
2
3
Setup…
1. Gap: Location, impact on the model, … on the requirements
2. Gap: Location, impact on the model, … on the requirements
3. Gap: Location, impact on the model, … on the requirements
4. …
Figure 1: Overview of our approach.
Figure 2: Example artifact list generated from the ArSPI model. The snapshot shows the general structure
of the checklist and the parts of the model (specification, realization, etc.) to be analyzed.
3.1.1 Analysis Setup
When setting up an SPI project, process engineers collect
all information regarding the Actual Process from specifi-
cations, realizations/models, and all complementing process
material in use (e.g., process guides, template, and so forth).
Furthermore, all (relevant) process requirements need to be
collected, e.g., change/feature requests, general (strategic)
improvement requirements, and external standards, such as
Capability Maturity Model Integration (CMMI).
3.1.2 Step 1: The Process Checklist
In doing this, the process checklist is created and the pro-
cess is analyzed for structural issues. For example, the pro-
cess specification provides information such as UML mod-
els or textual description of the process to be improved. The
process realization contributes further information, such as
particular artifacts and roles realized as a “real” model (e.g.,
using SPEM/EPF [3]). Finally, material, process documen-
tation and templates from the practical implementation of
the process to be considered is analyzed.
Figure 2 shows an example of the stepwise created check-
list. The checklist was generated from the ArSPI specifica-
tion [6] and stepwise completed by the other material. Each
(sub-)artifact and (sub-)activity is represented by a row of
the spreadsheet. The columns represent the sources of the
information and the criteria/measures. The checklist helps
to evaluate the model and its structure, outworking incon-
sistencies between model specification and deployment pack-
ages. The goal is to analyze if all specified model elements
are present correctly in the model’s realization as well as in
the process used. Furthermore, the inspection aims to iden-
tify elements in the model’s realization that are not part of
the specification (e.g., implementation “hacks” required to
address specialties of the used modeling environment).
The outcome of this analysis step is a list of model-related
issues and an investigation of rationale. This information
serves as input for the analysis of the requirements checklist,
and eventually, for the development of a prioritized list of
requirements improvement tasks.
3.1.3 Step 2: The Requirements Checklist
In the second step, the external requirements are collected
and structured, and the Actual Process is initially analyzed
to get a reference measurement. External requirements can
be defined by the process-owning organization, and can also
include external standards and norms. For instance, if a
requirement is to achieve a high maturity level, CMMI or
ISO/IEC 15504 would be possible inputs.
For every requirement, detailed criteria and measures need
to be defined to answer the question: how well does the
current process address and/or fulfill the respective require-
ments. Figure 3 shows an example of questions based on
CMMI. It can be seen that each requirement is addressed
individually to collect detailed information about gaps and
issues.
The outcome of this analysis step is an initial measure-
ment showing to which extent the Actual Process fulfills the
general process/improvement requirements. The checklist
contains detailed information about which content-related
requirements are not yet properly addressed.
3.1.4 Step 3: Requirements Prioritization
The third step aims to derive detailed information about
the gaps and/or issues by combining the findings from the
structural analysis (step 1) and the content-related analysis
(step 2). The overall goal is to investigate whether content-
related requirements are caused by structure-related issues,
or whether fixing a structural issue helps to address one or
more content-related requirements. In particular, the fol-
lowing questions are investigated:
•Are there structural flaws in the model that negatively
influence the process?
•Which of the structural improvement requirements also
address content-related improvements?
•Which of the identified improvement requirements de-
liver the “quick wins”?
The outcome of this analysis step is a prioritized list in
which, for each gap, information is provided regarding where
the gap is located and what impact a gap has on the overall
requirements. This list aids the prioritization of the im-
provement requirements.
4. INITIAL VALIDATION USING ARSPI
We carried out an exploratory case study in which the
aforementioned procedure was implemented.
© ACM. PREPRINT. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution.
The definitive version was published in the conference/workshop proceedings.
Figure 3: Example CMMI-based checklist for analyzing a process for the purpose of studying how well the
Actual Process addresses a particular topic (here CMMI process area Requirements Management).
4.1 Study Setup
To provide an initial application, we opted for the ArSPI
model v0.9 [8] for the purpose of defining the requirements
for the next version. The study was conducted as a con-
trolled experiment in a Master’s Thesis [5]. The student was
provided with the specification of ArSPI [6], the process’s re-
alization as an EPF model, and the deployment package as
HTML web-sites and document templates. CMMI v1.3 [2]
was selected as an external reference. The following require-
ments were defined for the study:
R1: The ArSPI model/process consistency and complete-
ness must be ensured.
R2: The ArSPI model/process shall be improved to achieve
CMMI level 5 or equivalent.
R3: Gaps in the ArSPI model/process that hamper the
achievement of CMMI level 5 shall be identified, and
respective improvement proposals shall be developed
and evaluated.
The expected outcomes of the study were an overview of the
“hidden” flaws of the model, issues regarding the model’s
ability to implement high-maturity processes, and a con-
solidated and prioritized list of (technical) improvement re-
quirements to improve the overall quality of the model.
4.2 Selected Study Results
Figure 2 shows an excerpt from the artifact list generated
from the different input. All elements were pair-wise com-
pared to each other to work out the degree of completeness
of the different parts in relation to the process specification,
and the consistency between the different parts of the ac-
tual process. The detailed analysis revealed gaps and issues,
which are summarized in Table 1. In the second part of
the analysis, CMMI was used to develop a checklist for a
content-related analysis (Figure 3). Using this checklist, in
a first step, an initial assessment was conducted to generate
a baseline. Based on the analysis of the model and the gaps
found in step 1, 29 improvement requirements were defined.
The requirements were then used to repeat the CMMI-based
assessment. This allowed an understanding off which re-
quirements have the most impact on the CMMI-based as-
sessment. From this, technical improvement requirements
were developed and prioritized (goal: minimal effort – max-
imum impact).
Table 1: Summary of found issues in ArSPI v0.9.
Category Description
Roles Missing roles and responsibilities were iden-
tified across the whole model, e.g., project-
and quality managers, testers, and so forth
Artifacts 16 artifacts from the specification were not
properly realized
Artifacts 10 artifact realizations were not in tune
with the respective specifications
Activities 37 activities were identified that were nei-
ther properly specified nor realized
Activities 17 activities from the specification were not
properly realized
An Example. Due to space limitations, we only provide a
small example: In the analysis, we found that ArSPI v0.9
was focused on the process engineer, which was the only role
specified in the model. Taking the CMMI-based question-
naire, which—among others—asked for responsibilities, we
found a major gap resulting in a low rating. In order to ad-
dress this issue, a requirement was developed, which calls for
defining a role model and providing explicit responsibilities.
A set of 4 roles (1 from the original model and 3 new roles)
was defined and a RACI matrix was developed to provide
the requested responsibility assignments. Having defined
this requirement, the CMMI-based questionnaire was revis-
ited to analyze the impact of this requirement. Figure 4
exemplary shows the outcomes for a selected CMMI process
area and illustrates the dramatic impact the improvement
requirement has on the model.
5. CONCLUSION & FUTURE WORK
In this paper, we presented an approach to support pro-
cess engineers who need to analyze process models. This
can emphasize the quality of the process model, especially,
the completeness and consistency of the process model un-
der consideration. The approach can determine deviations
among the different parts of a process (model), analyze the
© ACM. PREPRINT. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution.
The definitive version was published in the conference/workshop proceedings.
0
2
3
0
0
0
0
7
4
7
7
7
5
5
0 1 2 3 4 5 6 7
Completeness of Roles
Completeness of Activities
Completeness of Artifacts
Completeness Role-Artifact
dependencies
Completeness Role-Activity
dependencies
Process Area sufficiently addressed
General level of detail is sufficient
Requirements Assessment Initial Assessment
Figure 4: Exemplary results from initial evaluating
ArSPI and analyzing the impact of a requirement.
technical requirements for harmonization, and to work out
which technical issues found can also help to improve the
quality of the process in general. In particular, we aim to
reduce the risk of introducing “technical debt” into a process
model, which encounters the risk of missing the improvement
targets, as improvement goals may address aged processes
that are just not used in practice—this would be considered
to be specification-based improvement rather than improve-
ment of the actual process.
The approach presented in this paper utilizes the artifact
type Actual Process as provided by the ArSPI model. The
actual process comprises a process specification, a process
realization, and process in use components. Using this in-
formation, we generate checklists that we use to conduct
a pair-wise comparison to find and locate issues regarding
completeness and consistency. As second part of the analy-
sis procedure, we use (external) improvement requirements
and analyze whether “just fixing” the previously found gaps
helps to address the improvement requirements.
Although tentative, we found that the approach using
artifact-based improvement procedures to be beneficial. Us-
ing artifact models as input simplified the analysis proce-
dures, as all relevant information for the analysis steps could
be easily collected and structured. Analysis results were eas-
ily linked to tangible objects, allowing us to locate issues in
the model and to estimate the impact of potential changes.
This also enabled us to provide a prioritized requirements
list, to improve the improvement realization procedures and,
at the same time, to improve the model’s quality by improv-
ing its completeness and consistency.
Limitations and Future Work. Further research is nec-
essary to validate the approach, to demonstrate its feasibil-
ity, to improve the approach presented in this paper, and
its evaluation in practice. Furthermore, we will make use of
the findings gathered so far and implement the requirements
found into the next version of ArSPI.
ACKNOWLEDGMENTS
We thank Anastasia Kadnikova for implementing the ap-
proach in practice. This work was supported, in part, by
Science Foundation Ireland grant 13/RC/2094 to Lero - the
Irish Software Research Centre (www.lero.ie)
6. REFERENCES
[1] L. Bass, P. Clements, and R. Kazman. Software
Architecture in Practice. The SEI Series in Software
Engineering. Addison-Wesley, 3rd edition, 2013.
[2] CMMI Product Team. CMMI for Development,
Version 1.3. Technical Report CMU/SEI-2010-TR-033,
Software Engineering Institute, CMU, 2010.
[3] Eclipse Foundation. Eclipse Process Framework
(EPF). Online, http://www.eclipse.org/epf, 2010.
[4] ISO. Software Process Assessment - Part 4: Guidance
on use for process improvement and process capability
determination. Technical Report ISO/IEC
15504-4:2004, International Organization for
Standardization, 2004.
[5] A. Kadnikova. Assessing and improving an spi method
using the arspi framework. Master’s thesis, Technische
Universit¨
at M¨
unchen, 2014.
[6] M. Kuhrmann. Arspi: An artifact model for software
process improvement and management. Research
Report TUM-I1337, TU M¨
unchen, 2013.
[7] M. Kuhrmann. Crafting a software process
improvement approach—a retrospective
systematization. Journal of Software: Evolution and
Process, 27(2):114–145, 2015.
[8] M. Kuhrmann and S. Beecham. Artifact-based
software process improvement and management: A
method proposal. In International Conference on
Software and Systems Process, pages 165–169. ACM
Press, 2014.
[9] S. McConnell. Managing technical debt. Online
http://www.youtube.com/watch?v=lEKvzEyNtbk,
2011.
[10] OMG. Software & Systems Process Engineering
Metamodel Specification (SPEM) Version 2.0.
Technical report, Object Management Group, 2008.
[11] D. L. Parnas and P. C. Clements. A Rational Design
Process: How And Why To Fake It. IEEE
Transactions on Software Engineering, 12(2):1–10,
1986.
[12] A. Raninen, J. J. Ahonen, H.-M. Sihvonen,
P. Savolainen, and S. Beecham. LAPPI: A light-weight
technique to practical process modeling and
improvement target identification. Journal of
Software: Evolution and Process, 25(9):915–933, 2012.
[13] M. Sot´o and J. M¨
unch. Process model difference
analysis for supporting process evolution. In European
System & Software Process Improvement and
Innovation, pages 123–134. Springer, 2006.
[14] M. Unterkalmsteiner, T. Gorschek, A. Islam, C. K.
Cheng, R. Permadi, and R. Feldt. Evaluation and
measurement of software process improvement - a
systematic literature review. IEEE Transactions on
Software Engineering, 38(2):398–424, 2012.
© ACM. PREPRINT. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution.
The definitive version was published in the conference/workshop proceedings.
