Field Study on Requirements Engineering Artefacts and Patterns

Abstract

Requirements Engineering (RE) constitutes a critical discipline within Software Engineering. The quality of requirements is the backbone of project execution since the following phases strongly rely on it. Nowadays, industries are more then ever facing the problem that the RE process is highly volatile because it depends on the customer's capabilities, on the used process models, and on the type of specifications developed. This paper describes a study on the RE process in a specific company in the application domain of business information systems. While existing surveys often analyse the general impact of RE processes on project success, we investigate and discuss different influences that arose in 12 real-life projects and the effects of these influences onto produced RE artefacts. We infer different artefact patterns and probable project execution strategies that cause these patterns. The strategies are performed in order to tackle the different project influences. The identification enables us to get a more detailed understanding of RE in practice for the future elaboration of tailoring approaches that customise RE efforts to volatile project environments.

Full text

Field Study on
Requirements Engineering Artefacts and Patterns
Daniel M
´
endez Fern
´
andez, Stefan Wagner, Klaus Lochmann
Institut f
¨
ur Informatik
Technische Universit
¨
at M
¨
unchen
Boltzmannstr. 3, 85748 Garching, Germany
mendezfe, wagnerst, lochmann@in.tum.de
Andrea Baumann
1
Fakult
¨
at f
¨
ur Elektrotechnik und Technische Informatik
Universit
¨
at der Bundeswehr M
¨
unchen
Werner-Heisenberg-Weg 39, 85577 Munich, Germany
andrea.baumann@unibw.de
Requirements Engineering (RE) constitutes a critical discipline within Software Engineering. The quality of
requirements is the backbone of project execution since the following phases strongly rely on it. Nowadays,
industries are more then ever facing the problem that the RE process is highly volatile because it depends
on the customer’s capabilities, on the used process models, and on the type of specifications developed.
This paper describes a study on the RE process in a specific company in the application domain of business
information systems. While existing surveys often analyse the general impact of RE processes on project
success, we investigate and discuss different influences that arose in 12 real-life projects and the effects
of these influences onto produced RE artefacts. We infer different artefact patterns and probable project
execution strategies that cause these patterns. The strategies are performed in order to tackle the different
project influences. The identification enables us to get a more detailed understanding of RE in practice for
the future elaboration of tailoring approaches that customise RE efforts to volatile project environments.
Requirements Engineering, Strategies, Artefacts, Business Information Systems, Field Study
1. INTRODUCTION
Requirements Engineering (RE) lays the foundation for
successful development projects regarding cost and
quality (Broy 2006). In literature as well as in practice
a rich set of methods for RE is available. However, in
practice these methods are still not integrated into the
development process (Boehm 2006).
Even if a company defines and integrates an RE process
for a broad use in different projects, it still does not
address the various influences that engineers have to
face in volatile project environments. The possibilities
and necessities of applying available RE techniques
are often limited by different project parameters such
as time, budget or the availability of end users.
These parameters hamper a standardised and efficient
RE phase that fits individual project needs. In fact,
the dependence on customer’s capabilities and used
process models render the process highly variable and
increase the demand for a systematic customisation of
RE efforts according to individual project environments.
Problem Statement. The high variability of the
development processes makes it unclear for the
practitioner what methods to choose and consequently
how to design an appropriate RE process. Indeed, the
1
Was at Capgemini sd&m AG at the time of the study.
understanding of what project parameters take effect
on the project execution and how these effects should
be reflected by the RE process is fundamental for
customising the process in terms of specific strategies.
Although this problem is recognised, there still exists little
guidance on how to tackle it.
Research Objective. The overall research objective is
to investigate how RE is performed within successful
projects in order to discover strategies that allow the
customisation of RE efforts to individual project envi-
ronments. Since the variability in the process definition
complicates a comparison and categorisation of project
execution strategies, the investigation has to be per-
formed in a process-agnostic manner.
Hence, we want to understand what problems arise in in-
dividual project environments, what consequences these
problems have on produced specification documents
(artefacts) and finally how volatile project situations can
be reflected by artefact patterns and project execution
strategies.
Contribution. Based on the analysis of real-life
projects at the company Capgemini sd&m AG we
contribute
• the degree of completeness in which requirements
artefacts are produced.
c
 The Authors. Published by the British
Informatics Society Ltd. 1
Proceedings of 14th International Conference on Evaluation and Assessment in Software Engineering (EASE ’10)

M
´
endez Fern
´
andez • Wagner • Lochmann • Baumann
• project parameters influencing the artefact com-
pleteness.
• artefact patterns reflecting probable execution
strategies for RE.
A major part of the contribution is the analysis of
the interrelations between these three parts. The
discussion which RE execution pattern should be chosen
when distinctive project parameters are given supports
practical project decisions.
Context. This study is performed as part of the re-
search cooperation between the Technische Univer-
sit
¨
at M
¨
unchen (TUM) and the research department of
Capgemini sd&m AG, a major German software and
consultancy company. We perform the survey for un-
derstanding problems and necessities of RE in practice.
To reach this aim, we conduct content analyses of
RE artefacts produced in 12 real-life projects as well
interviews with project participants.
2. RELATED WORK
Related studies can be categorised into two major
areas. They either analyse the processes in order to get
detailed insights into activities, used techniques and the
relation to projects’ success in general or they focus on
assessment and improvement of RE processes.
Process Understanding. Related work analysing the
RE process for a better understanding and extraction
of best practices often emphasises used techniques in
practice (Boehm and Alexander 1998) and the impact
of used techniques on the projects’ failures in general.
A widely known empirical evaluation of such project
failures and their causes is presented in the chaos report
of the Standish Group International (1995). The chaos
report examines project failures and related causes like
missing user involvement. However, the report does not
give detailed insights into the study design. Instead, the
success study of Buscherm
¨
ohle et al. (2006) gives a
similar investigation of German companies including a
description of how the study exactly has been performed.
Still, both surveys exclusively investigate failed projects
and general causes, thus, give not necessarily an
understanding on how specific problems are tackled in
successful projects.
Nikula et al. (2000) present a survey on requirements
engineering at organisational levels of small and medium
size companies in Finland. They concluded that even
basic knowledge about RE is often missing in such
companies and that more effort on technology transfer
should be spent. This study was restricted to interviews
and thereby did not give fine-grained views onto the
documentation of single projects. Such a view as part
of project success stories is given in Kamata and Tamai
(2007) presenting an analysis of used documentation
parts of the IEEE software requirements specification
Std. 830-1998 in general relation to project success.
An investigation of project-specific influences was not in
scope of their work.
Process Assessment. Prominent approaches that
emphasise capability aspects of RE processes are,
for example, CMMI and those that are specifically
elaborated for RE issues. For instance, Beecham et al.
(2003) performed a survey on twelve companies and
identified relations between the process capability and
experienced problems within those companies. They
developed – based on those results – a capability
maturity model specifically for requirements engineering.
Similar work is presented by Sommerville and Ransom
(2005). They performed a survey on requirements
engineering process capabilities and their impact on the
business success in general.
Summarised, earlier studies show that basic knowledge
about RE is often missing at organisational levels of
companies (Nikula et al. 2000). A consequence is that
projects have no reference approach for a systematic
process in projects. Hence, the process capability often
suffers in projects (Beecham et al. 2003) and can be
directly put in relation to project failures (Standish Group
International 1995).
However, while the capability of RE processes strongly
depends on the maturity of the underlying work products,
the investigation of the overall processes and their failure
rate alone gives no detailed understanding on how
specific problems are tackled in practice. In order to get
such an insight into RE, it has to be analysed what has
been produced and why it has been produced instead of
focusing on how it was produced.
3. FIELD STUDY DESIGN
We organise the study according to Runeson and H
¨
ost
(2009). We formulate the research questions, describe
the case and subject selection as well as the data
collection procedures. We then define the analysis
procedure before we conclude with a description of how
we ensure the validity.
3.1. Research Questions
The study aims at identifying three main factors in RE
processes and their relationships. Figure 1 shows this
triangle of the factors artefact completeness, project
parameters, and pattern & strategies. The project
parameters are conditions from inside or outside the
project that influence its execution. They have an
influence on which artefacts are created in which
completeness. The complete set of artefacts and their
completeness shows which pattern or execution strategy
was used in the RE process. Therefore, it can also
be analysed which project parameters lead to which
execution strategies. To establish this triangle, we
formulate 3 research questions.
RQ 1 Which artefacts exist in which completeness?
2

Field Study on Requirements Engineering Artefacts and Patterns
Pattern &
Strategies
Artefact
Completeness
Project
Parameters
Figure 1: The triangle of factors and relationships analysed
We focus on the produced artefacts in the requirements
engineering process and investigate the completeness
of each artefact. It is not sufficient to just create a
document and add some text, but appropriate content
is needed.
RQ 2 Which project parameters have an influence on
the individual process and the artefact completeness?
For a better understanding of the context of the process,
influencing project parameters need to be found and put
into relation to the completeness of the RE artefacts.
Having the completeness of the artefacts in individual
projects as well as influencing project parameters, they
are set into relation. The project parameters influence
the completeness of the RE artefacts and, hence, the
individual project execution since the artefacts are a
direct outcome of performed processes. While RQ 1
addresses what has been produced, RQ 2 addresses
why this has been produced.
RQ 3 Are there artefact patterns and corresponding
execution strategies?
Having the project parameters that influence individual
projects in relation to the completeness of the RE
artefacts, we analyse whether specific patterns can
be distilled. This can be examined by comparing and
categorising the overall artefact completeness of single
projects in dependence to the project parameters. As
a whole, this concluding analysis enables a better
understanding in which contexts which RE strategy is
followed in practice.
3.2. Case and Subjects Selection
We analyse documented and ongoing projects
2
of
Capgemini sd&m AG, the German technology service
entity of the Capgemini Group. The company has
its focus on custom software development within the
application domain of business information system.
Therefore, the following study subjects and objects all
come from corresponding project environments.
Regarding the study subjects (the participants), we focus
on two roles: project leaders and chief analysts. Further
roles like customers, could not be taken into account
for reasons of confidentiality. The project leaders have
the responsibility for project management issues, such
2
With ongoing projects we refer to projects that have further releases
or increments beyond the ones analysed.
as control or risk management, but also acquisition.
They are the source for initial detailed information on
how the development project was set up and upon
what documents the project scope was defined. After
the successful acquisition and scoping phase of a
project, the chief analysts have the responsibility for
analysing and documenting the business processes,
the requirements, and the initial (overall) system
specifications.
Regarding the study objects, we distinguish between
the company-wide process definition (RE definition at
organisational level) and the actual analysed projects
that followed the process. The first serves as a
preparation of the study in order to get an understanding
of the used terminology and the process. The study
objects include reference process descriptions, role
models, training material and finally standard operating
procedures. Once we understand the process definition,
we select different projects and analyse corresponding
documents. As a whole, 18 projects were available and
ready to participate within the study, but we concentrate
on 12 projects (see section 4.1) as these were able to
offer sufficient data.
The documentation of the projects includes acquisition
documents, so-called study documents, requirements
specifications and system specifications. Acquisition
documents capture background information and possible
objectives of each project. Study documents usually
capture information on the customer’s domain and
the organisation, the actual business processes, and
a sketch about the future business processes to
be supported and realised by the system under
consideration. These documents mostly are produced
as a preparation for a development project. In some
cases such documents are produced in parallel to a
maintenance phase, so that they also include a detailed
description of the main use cases and technical aspects
of the current IT infrastructure. The requirements
specification contains the textual requirements. Finally,
the system specification contains the external and
internal functional specification, which defines the logical
architecture of the system.
3.3. Data Collection Procedures
We collect the reference process definition as a
preparation before analysing the single projects. The
corresponding documentation was made available at the
beginning of the study. We then prepare and set up the
data collection from the single projects within two steps
that take approximately two days for each project.
Step 1: The first step includes a short open interview
of approximately one hour. This allows to get a brief
overview on the project background, its relevance for the
survey and an overview of the existing documentation.
If the project has no relevance or if the project offers
no possibility of accessing the documentation because
of, e.g., non-disclosure agreements with third parties, we
change the project.
3

M
´
endez Fern
´
andez • Wagner • Lochmann • Baumann
Step 2: In a second step, project participants send
the documentation for a brief content analysis. This
enables an initial understanding of the artefacts and
allows the identification of further information needs for
requesting additional documents. Within a second short
open interview the preparation of the analysis is finalised
and an appointment for a structured interview for the
actual content analysis is made (see the next section).
This way, the efforts for the study subjects within the
survey can be kept to a minimum.
3.4. Analysis Procedures
The analysis of the projects was conducted in three
steps:
Step 1 addresses RQ 1 by content analyses.
Step 2 addresses RQ 2 by expert interviews.
Step 3 addresses RQ 3 by cluster analysis.
Step 1: This step answers RQ 1 by content analyses
of the collected documents, which we perform isolated
without any discussions with related study subjects. This
ensures an objective review of the documentation and
furthermore a neutral comparison of the documentation
of different projects. Within projects of a high complexity
that have several thousands of requirements, we carry
out single spot checks for each of the documented
(software) releases. This reduces the efforts and
gives an average view on the completeness of each
artefact. Finally, requirement’s attributes (such as the
documented priority of requirements and at which basis
this priority is calculated) are not explicitly taken into
account.
This content analysis is performed by comparing the
documents and their content on the basis of a neutral
artefact-based Requirements Engineering Reference
Model (REM) (Berenbach et al. 2009). It defines a
taxonomy of the core RE artefacts with a description of
recommended contents and an abstract description of
the contents’ interdependencies (traceability relations).
REM consists of three major artefacts: the Busi-
ness Needs Specification, the Requirements Specifica-
tion and the System Specification; each of the artefacts
includes a list of content items.
On the basis of the included content items, we compare
the content of each document of the projects. As
REM has been developed for the application domain
of embedded systems, it contains several domain-
specific artefacts (e.g., hardware-specific constraints).
We tailored REM according to the domain of business
information systems considering also domain-specific
frameworks like the Zachmann framework so that no
content items where missing.
For each content item in REM completeness criteria
are defined. This allows the usage of this model as
a reference for a detailed analysis of a specification’s
content and its completeness according to the criteria:
• Completely specified: The content item can be
unambiguously mapped onto a specific element of
the analysed documents.
• Incompletely specified: The content item can be
identified but exhibits major deficiencies to either
the defined content within REM or compared to the
content of other projects.
• Missing: The content item cannot be identified.
The content analysis using REM gives a detailed view
on the produced artefacts and their completeness (what
has been documented). Still, since it allows no deeper
understanding about the process and the underlying
motivation (why has it been documented), we perform
step 2 of the analysis.
Step 2:
In order to elaborate different project parameters
that have an influence on individual projects and their
artefacts, a structured interview is performed. Within this
interview, we ask the study subjects directly for specific
requirement artefacts that we found exceptional in step
1. This allows a discussion on why specific artefacts
are documented or not and furthermore the extraction
of specific project parameters that are directly related to
these artefacts. The interview guides through the results
of the content analysis in order to (1) detect individual
project parameters that took effect on the RE process
and (2) detect the dependencies of those parameters to
specific content items. Therefore, it addresses RQ 2 and
analyses why the requirements are documented in this
way. The interview is prepared by identifying the content
items that exhibit differences compared to other projects
and items that have not been specified at all. Based on
these content items, the project-specific reasons for why
they are specified in corresponding intensity are directly
elaborated during a structured interview and put into
relation to those content items.
Step 3: After performing the content analyses and the
interviews within all the projects, we make a concluding
analysis in order to address RQ 3. We analyse the RE
artefacts of all projects for similarities in order to identify
common patterns. The analysis groups the projects into
clusters with similar artefact completeness. Hence, it is
probable that the same project execution strategy was
performed to create the artefacts of projects in one
group. We thereby connect artefact patterns with project
execution strategies that cause the patterns. We perform
the clustering using the k-means cluster analysis, which
minimises the distance from each project to the mean,
called centre, in its cluster. We code the three possible
verdicts for an artefact on an ordinal scale with 0 =
missing, 1 = incompletely specified, and 2 = completely
specified. We experiment with different values for the
number of clusters to get a useful grouping for the
relation to project execution strategies. We identify
suitable strategies by comparing the completeness of
artefacts in the cluster and analysing the differences
between them.
4

Field Study on Requirements Engineering Artefacts and Patterns
3.5. Validity Procedures
The selection of projects from different industrial sectors,
with different sizes and different project participants
maximises the external validity of the results. Initial
discussions with the study subjects lower barriers and
ensure the collection of accurate and appropriate data.
The neutral artefact model REM serves as basis for
classification and the classifications are checked by
other researchers. We present and discuss the results
of the overall analysis with project participants. This
clarifies open questions and excludes wrong results that
could arise from incomplete documentation or wrong
assumptions.
4. RESULTS
In this section the results of the study are presented.
They are structured according to (1) the case and subject
description and (2) according to the defined research
questions.
4.1. Case and Subject Description
The analysed development projects can all be allocated
to custom software development projects within the
application domain of business information systems.
Although they are restricted to this application domain,
they exhibit different characteristics. Table 1 gives an
overview of the analysed projects. We distinguish for
each project between the industrial sectors of the
customers and the project size that for reasons of
confidentiality is approximated by 3 categories
3
. The
projects are labelled with numbers and it is also
described whether they are finished or are still ongoing
(in terms of further releases or increments). All the
projects focus either on the displacement of legacy
systems, on the development of new systems or
on consultancy in which application landscapes are
analysed and re-designed. Consequently, all projects
have in common that they define at least requirements
and system specification documents independently of
the following phases.
Regarding the company-wide reference process defini-
tion and the conformance of the projects to it, nearly
all projects followed initially a waterfall model as a
consequence of multi-staged bidding procedures. The
only exception is P4 that followed an iterative incremental
process model right from the beginning as part of a
follow-up project with no explicit bidding procedure. The
followed reference process definition offers a collection
of architecture-driven design methods based on the
Integrated Architecture Framework (IAF) of Capgemini.
These methods were employed by all projects as part of
3
small: 0 to 19 person years
medium: 20 to 119 person years
large: more than 120 person years
a so-called analysis and specification discipline accord-
ing to the process framework RUP, but with no explicit
assignment of an RE phase.
Table 1: Analysed projects
ID Industrial Sector Size
2
P1 (finished) Finance Small
P2 (finished) Finance Small
P3 (finished) Finance Small
P4 (ongoing) Retail sale Medium
P5 (ongoing) Contr. authority Medium
P6 (ongoing) Telecommunication Large
P7 (ongoing) Logistics Large
P8 (ongoing) Logistics Large
P9 (finished) Aerospace Medium
P10 (ongoing) Contr. authority Medium
P11 (finished) Finance Medium
P12 (ongoing) Automotive Large
4.2. Documented Requirements Artefacts (RQ 1)
For each analysed project, the existence and complete-
ness of the artefacts proposed by REM is compared
to the content of the analysed documents. Table 2
summarises the results of the content analysis. Going
from top to bottom, the table structures the content
items of REM according to the three major artefacts:
the business needs, the requirements specification, and
the system specification. Within the content items of
the requirements specification we distinguish between
functional aspects and non-functional aspects. The first
includes, for example, requirements-specific application
scenarios such as use case models. The non-functional
models include, for example, the specification of quality
requirements, models of the application’s future environ-
ment, or process constraints regarding the delivery of the
application (“release strategy”). Within the system speci-
fication we distinguish between the design concepts and
test case specifications.
The bottom part of the table highlights traceability as-
pects, describing the maintained interdependencies be-
tween specific content items. We distinguish between
traceability from the contents of the business needs
to the requirements specification and from the require-
ments to the system specifications. We do not distin-
guish between forward tracing and backward tracing. In
general, it can be observed that the content items of the
initial specifications are expressed in a very different in-
tensity. The closer we come to the system specification,
the more homogeneous and detailed they are specified.
In fact, the content items of the system specifications are
nearly all completely specified.
Within the business needs, it can be seen that nearly all
projects specified the business objectives of a customer
and corresponding high-level requirements, respectively
the goals (the “Customer Requirements”). However, we
observe a highly variable handling of the further content
5

M
´
endez Fern
´
andez • Wagner • Lochmann • Baumann
Table 2: Completeness of artefacts in the analysed projects
Project P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12
Business Needs Artefacts
Business Objectives H# H# H# H# H#
Customer/Market Requirements H#
Value to the Customer H# H# H# H# # H# # H#
Main Features H# H# H# H# H# H# H# H# H# H#
Assumptions Dependencies H# H# H# H# H# H# H# # H#
Scope and Limitations # # H# # #
ROI Calculation H# H# H# H# H# # # # H# H#
Business Risk Analysis H# H# H# # # # # # H# H#
Risk Calculation H# # H# H# H#
System Success Factors # # # H# # H# H# H# H# H# # H#
Requirements Specification Artefacts
Functional
Application Scenarios H# H# H# H# H# H# H# H#
User Interface # # # # H#
User Classes H# H#
System Interaction H# H# H# # H# H# H# H# H# H#
Non-Functional
Release Strategy
Domain Model
Environment Model H#
System Boundaries H# H# H# H#
Quality Requirements H# H# H# # H# H# H# H# H# H# H# H#
Assumptions # # # H# H# H# # H# H# # #
SW Design Constraints # # H# # H#
Acceptance Criteria H# H# H# # H# # # # H# #
Acceptance Test Cases # # # # # # # # # #
System Specification Artefacts
Design Concept
Release Planning
Behaviour Model
System Interaction
Service Interaction
Data Model
User Interface
Communication Interfaces
Interfaces to Service Components
Architecture Constraints
Deployment Constraints #
Coding Standards
Test
Functional Test Criteria # H# H#
Integrations Test Criteria # H# H# H#
Design Constraints Test Criteria # # # # H# H# H#
Traceability Needs to Requirements # # # # # # # # #
Traceability Requirements to Sys.Spec # # # # # # H# H# H#
Legend: completely specifed H# incompletely specifed # missing
6

Field Study on Requirements Engineering Artefacts and Patterns
Table 3: Content items in relation to influencing project
parameters
Content Items Project Parameter
ROI Calculation Industrial Sector & Confi-
dentiality
Value to the Customer Confidentiality
Scope and Limitations Relation to Customer
System Success Factors Knowledge to Customer’s
Domain,
Degree of Innovation
Application Scenarios Objectives, Process Model
& Availability of End Users
Quality Requirements Technical Ability &
Availability of Customers
Assumptions Technical Knowledge of
Customers
Design Concept Standardised Design Pro-
cess
Test Criteria External Suppliers
items of the business needs. Especially the “Return of
Investment (ROI) calculations” and the elaboration of
the customer’s value is documented very variably. We
observe the same for the “Scope and Limitations” that
either have been completely specified or are missing.
The “System Success Factors”, including the basis for
prioritising the requirements, are either incomplete or
missing.
Within the requirements specification, we also observe
a difference in the handling of the functional analysis
models, especially within the “Application Scenarios”.
Considering the non-functional analysis models it is eye-
catching that the system environment and the related
content items are of high intensity. Instead, the rest of
the content items, such as the quality requirements or
the assumptions, are often incomplete.
Finally, the system specifications are nearly all
completely specified, only the test case specifications
are documented highly differently.
4.3. Influencing Project Parameters (RQ 2)
Within the structured interviews, we discuss the
influencing parameters for specific content items that
take our attention. The stated influencing project
parameters vary in their project-specific details but can
be summarised as done in table 3.
Whether and how the chief analysts documented
ROI calculations strongly depends on the industrial
sector and the objectives of the customers. In
particular, most customers do not share enough
details of their organisation (e.g., their business
processes) for performing such calculations, mostly
for reasons of confidentiality. This is especially true
when elaborating the value of single requirements
to the customers. Consequently, the prioritisation of
requirements (expressing the relevance of requirements
for customers) can often not be performed. The
only exception are projects that are performed with
German contracting authorities. The reason is that
those customers and related development projects are
required to conform to the V-Modell XT, a German
standard for development projects. Within this standard
a ROI calculation is demanded before performing a
requirements analysis.
The definition of the scope and limitations item mostly
is influenced by the relation to customers. The more
familiar a customer is, for example within a follow-up
order, the less specific limitations are documented to
improve efficiency. Similarly, the system success factors
have a strong dependency on the knowledge about
the customer and his domain. In particular, the less
familiar customers (e.g., within the first development
project for this customer), the higher the probability of
defining system success factors. Dependencies are also
related to the degree of innovation of the application. The
higher the degree, the more specific success factors are
documented.
The application scenarios within the functional analysis
model exhibit dependencies to the objectives of a project
in general and the process model in particular. These
models are explicitely documented within a requirements
specification if either a requirements phase with an
acceptance phase of corresponding specifications is
assigned (instead of a continuous development project)
or if a change management process is set up. In the latter
case the customers are often the ones to state functional
requirements by the means of such analysis models, for
example by the means of use cases. Another reason to
be mentioned is the low availability of end users. In such
cases the projects have no possibility of accessing their
needs and hence the functional analysis models (e.g.,
the use cases) are often incomplete.
The quality requirements strongly depend on the
technical ability and temporal availability of customers
to state these requirements in detail. When specifying
quality requirements, reference models and reference
values often are missing. They remain on a high level
of abstraction. For example, instead of stating specific
security requirements, customers often refer to security
standards, such as the Common Criteria. This project
parameter has also a strong relation to the assumptions
that often have to be made. Assumptions are in general
used in order to compensate risks that are implied by
incomplete or missing quality requirements. The less
quality requirements are precisely defined, the higher the
probability of defining assumptions.
Finally, the completeness of the content items of the
design concept has its rationale in the standardised
design process of the company according to the IAF
and RUP. The completeness of the test criteria mostly
depends on the integration of external suppliers, as
many customers distributed the testing phase to third
parties.
7

M
´
endez Fern
´
andez • Wagner • Lochmann • Baumann
Table 4: Artefact patterns and corresponding artefacts for
which the completeness differ between patterns
Pattern
Solution Oriented
P1–P4
Functional Oriented
P5, P9
Problem Oriented
P6–P8, P10–P12
Business Objectives H#
Assumptions H# H#
Scope #
Business Risk H# # H#
Risk Calculation H# H#
System Success Factors # H# H#
Application Scenarios H# H#
User Interface #
System Boundaries H#
Assumptions # H# H#
SW Design Constraints H#
Design Const. Test Crit. #
Acceptance Criteria H# #
Acceptance Test Cases # #
Tracing: Needs to Req # # H#
Tracing: Req to Sys.Spec # H# H#
4.4. Artefact Patterns (RQ 3)
To identify artefact patterns, we identify clusters with
similar completeness of artefacts. The k-means cluster
analysis gives the most useful grouping into three
artefact patterns. The Eucledian distances from the
projects to their cluster centres range from 1 to 3.3 based
on the coding from 0 to 2. A higher number of clusters
results in very small clusters that cannot be reasonably
interpreted, but have similar distances. Table 4 shows
the mapping of projects to the artefact patterns as well as
the cluster centres from the k-means cluster analysis for
those artefacts that have differing values between these
patterns. The analysis of the three artefact patterns
shows different tendencies in the artefacts. One cluster
has more emphasis on the solution description, one
on the functional description, and one on the problem
description. This leads us to the conjecture that the
artefact patterns are caused by a solution oriented,
a functional oriented, and a problem oriented project
execution strategy.
Solution Orientation. This pattern reflects a solution-
biased approach and implicates a weak description
of the content of the business specification. The
corresponding project parameter show that this pattern
results from circumstances at the customer such as
the relation to customers and the knowledge about
their domain. The consequences are, for example,
that the business objectives and the future system
environment are incompletely specified and that the
system success factors are missing. Instead, the
projects emphasise risk calculations and the initial
scope. Another consequence is that the traceability is
missing due to the incompleteness of the content that
would be traced.
Functional Orientation. The projects within this pat-
tern emphasise the functional analysis models of the
requirements specification including the application sce-
narios and the user interfaces. These circumstances are
also reflected in the traceability from the requirements
to the system specification. This mainly arises from the
project parameter concerning the used process model,
the objectives and the availability of end users. Corre-
sponding projects have set up a change management
process and end users were able to contribute to the
definition of functional analysis models (like use cases).
Since the functional demands requested by the cus-
tomers can be linked up with the elements of the system
specification this positively affects the traceability and
the acceptance criteria. In contrast to the solution ori-
ented approach, the business objectives are completely
specified.
Problem Orientation. The projects in this pattern have
a profound specification of the business needs, still if
the requirements specification is incompletely specified.
The stronger focus on business needs is also reflected in
the traceability that includes in contrast to the functional
oriented pattern the linkage between the requirements
and the business needs. A major reason for this resulting
strategy are project parameters like a high degree of
(product) innovation, the relation to customers and low
confidentiality issues enabling insights into customers’
organisations and business processes, but also the
industrial sector.
Summarised, as seen only half of the projects act
problem oriented, while a third act solution oriented and
a sixth functional oriented. Even if solution orientation
is an established phenomenon in practice (Boehm
2006), we can, however, depict the strategy with
clear artefact patterns and its clear relation to project
parameters. For instance, we can observe that project
parameters resulting from the customers’ domain take
strong influence on the choice of a solution oriented
approach. The reason is that the parameters reflect
the possibilities of accessing business knowledge
and whether customers can contribute to a clear
requirements analysis. Once, a customer does not give
detailed insights into his organisation and his business
processes, solution orientation can be chosen as a way
to successfully tackle this problem. This circumstance is
then mainly reflected by the low degree of intensity of the
content items within the business needs specification. At
the same time, one can observe a high rising intensity of
risk calculations and of the initial scope.
Furthermore, none of the interviewees showed during
the last feedback meeting awareness of having made an
8

Field Study on Requirements Engineering Artefacts and Patterns
explicit decision on whether to follow solution orientation
or not. This unawareness of possibilities and necessities
in RE could arise from mentioned historically grown
reference process models and underlying solution-
biased architecture frameworks as it is the case with the
used IAF. Both still neglect the discipline of RE to be a
vital part of software engineering. Hence, no integrated
RE approach was available for its use as an orientation
in the considered projects.
4.5. Evaluation of Validity
We subsequently discuss the construct validity, the
internal validity and finally the external validity of the
study.
Construct Validity. The major threat to the construct
validity of our study is that we analyse the different
project execution strategies only after the fact. We have
not observed the study subjects in their actual execution
but analysed the existing artefacts and interviewed
them in retrospect. Because of that, our analysis might
differ from a direct observation. However, analysing
the existing artefacts gives at least a basic level of
objectiveness. Furthermore, missing information was
collected directly from participating study subjects to
complete that picture. Another threat associated with
that is that artefacts are not the only indicator for a
specific project execution strategy. They do not directly
tell us how the artefacts were developed. In order to
mitigate this threat additional interviews and feedback
meetings were held to add this additional information.
Finally, for the categorisation of the artefact patterns, we
had to abstract from many details of each project. Every
project has its specifics because of e.g. its business
contexts, or the persons that work in it that all can have
a decisive influence. These detailed specifics had to be
disregarded for a more general analysis. We mitigated
this threat by considering these details in the interviews.
Internal Validity. In terms of the internal validity, it
is possible that there were more artefacts developed
in the projects that we have not analysed but which
would change the classification of the project. There
were, for example, specific feature lists in some projects
that could not be analysed because of confidentiality.
A similar threat comes from the interviews, in which
the interviewees could not give all information because
of confidentiality. We mitigated this in the feedback
meetings in which the study subjects had the chance to
comment on analysis and classification.
Furthermore, it is a threat that a large share of the
information is based on the interviews and information
given by people that participated in the projects.
Hence, there is the possibility that the information is
embellished. We mitigated this threat by the preliminary
interviews in which we emphasised that the analysis
is not an assessment that results in statements about
which projects are good and which are bad. This lowered
the barrier to giving complete and accurate information.
Finally, the analysis of the content items and their
classification was done subjectively by us. This holds the
threat that it is not repeatable. To mitigate that threat we
employed a neutral artefact model (REM), classified the
artefacts by more than one researcher, and discussed
the results with study subjects.
External Validity. The results of this study can only
be generalised to some extent because the major threat
is that we only analysed a single company. Hence, the
results can depend to a large degree on company-
specific context parameters such as the development
project or the corporate culture. Moreover, all analysed
projects built some kind of business information system.
Hence, it is not clear to what extent the results can be
transferred to other kinds of software. These threats are
mitigated by analysing projects from different industrial
sectors and by including the company-wide reference
process definitions into the investigation. Furthermore,
the analysed company is large enough that the overlap
of study subjects working on more than one analysed
project is low.
5. CONCLUSIONS AND FUTURE WORK
5.1. Summary of Conclusions
We analysed 12 industrial projects with regards to their
requirements engineering project execution strategy. For
this, we investigated the existence and completeness
of RE artefacts based on a generic reference model
as well as used interviews for further information. This
resulted in an analysis of the produced artefacts, project
parameters influencing the elaboration of these artefacts
and a categorisation of the projects into 3 main patterns.
We found that half of the projects act problem oriented, a
third act functional oriented and a sixth solution oriented.
5.2. Relation to Existing Evidence
In contrast to earlier studies (see section 2) we analysed
projects that were successful in terms of analysing
software releases being deployed to the customer and
being used in production. A finding was that only 50%
of these successful projects acted problem oriented in
requirements engineering. First, this manifests the idea
that there still is a tendency to solution orientation as it
was already observed during the late 1970’s by Boehm
(2006). Second, we also showed that many project
parameters that are considered to be the main cause
for project failures (1) arise from the customer domain
and often cannot be avoided, (2) can be tackled and
finally (3) are reflected by specific artefact patterns. The
analysis was performed in a process-agnostic manner
investigating which RE artefacts had been produced
and what influencing project parameters took effect
in the production of each single artefact. Hence, we
get a more fine-grained view onto the parameters that
extend existing ones. For instance, the Standish Group
9

M
´
endez Fern
´
andez • Wagner • Lochmann • Baumann
International (1995) states that 12.4% of project failures
are caused by missing user involvement. We showed not
only that these problems can be tackled, we also detailed
this parameter by the availability of end users, their
technical ability, confidentiality issues, and the general
relation to the customers, each having different impacts
on different artefacts. Further discovered parameters
that are not in scope of current studies are, for example,
the industrial sector, the degree of innovation (of the
product), or the involvement of external suppliers.
5.3. Impact/Implications
What is a successful artefact pattern in requirements
engineering depends on the parameters that influence
the project. We found in the study that in successful
projects these parameters, such as confidentiality or
technical ability, lead to differences in artefacts and
finally artefact patterns. The identified project execution
strategies that are reflected by the artefact patterns
can all be applied in real-life projects in order to adapt
the RE process to individual needs of volatile project
environments.
5.4. Limitations
The survey has only been performed within one
company. If extending this survey to other companies,
this could affect the results and consequently the criteria
for deriving the patterns. We doubt that the basic findings
would change extremely but we would be able to make
more detailed assertions on details of the patterns based
on elaborated trends. Furthermore, we only analysed
successful projects and, hence, cannot investigate the
differences in patterns.
5.5. Future Work
As we concentrate on a single company, it is important
that this study is extended by further projects and other
companies. In addition, all projects have been a success
in the sense that they resulted in systems that are now
in production at the customers. Hence, the different
project execution strategies were no principle success
factor. Therefore, as a future work we investigate the
detailed economic effects of these different strategies,
especially in combination with the identified influencing
project parameters.
Acknowledgement
We acknowledge the effort of all employees of
Capgemini sd&m that participated in the study. In
particular, we thank Gerhard Koller for the discussions
and fruitful remarks on the paper. Finally, we are grateful
to Eva Geisberger for the discussions on REM’s usage
for the content analyses.
REFERENCES
S. Beecham, T. Hall, and A. Rainer. Software process
improvement problems in twelve software companies:
An empirical analysis. Empirical software engineering,
8(1):7–42, 2003.
B. Berenbach, D. Paulish, J. Kazmeier, and A. Rudorfer.
Software & Systems Requirements Engineering: In
Practice. McGraw-Hill Osborne Media, 2009.
B. Boehm. A view of 20th and 21st century software
engineering. In Proc. 28th International Conference on
Software Engineering (ICSE 06), pages 12–29. ACM
Press, 2006.
B. Boehm and E. Alexander. Software requirements
negotiation: some lessons learned. In Proc. 20th
International Conference on Software engineering
(ICSE ’98), pages 503–506. IEEE Computer Society,
1998.
M. Broy. Requirements engineering as a key to
holistic software quality. In Proc. 21th International
Symposium on Computer and Information Sciences
(ISCIS 2006), volume 4263 of LNCS, pages 24–34.
Springer-Verlag, 2006.
R. Buscherm
¨
ohle, H. Eckhoff, and B. Josko. Success –
Erfolgs- und Misserfolgsfaktoren bei der Durchf
¨
uhrung
von Hard- und Softwareentwicklungsprojekten in
Deutschland. BIS-Verlag, 2006.
Capgemini. Integrated architecture framework.
www.capgemini.com/iaf.
M. Itakura Kamata and T. Tamai. How does requirements
quality relate to project success or failure? In Proc.
15th IEEE Requirements Engineering Conference,
pages 67–78. IEEE Computer Society, 2007.
U. Nikula, J. Sajaniemi, and H. K
¨
alvi
¨
ainen. A state-
of-the-practice survey on requirements engineering
in small-and medium-sized enterprises. Research
Report 1, Telecom Business Research Center
Lappeenranta, 2000.
P. Runeson and M. H
¨
ost. Guidelines for conducting
and reporting case study research in software
engineering. Empirical Software Engineering, 14(2):
131–164, 2009.
I. Sommerville and J. Ransom. An empirical
study of industrial requirements engineering process
assessment and improvement. ACM Transactions
on Software Engineering and Methodology, 14(1):85–
117, 2005.
Standish Group International. The Chaos Report.
http://net.educause.edu/ir/library/pdf/NCP08083B.pdf,
accessed on 2009/03/15, 1995.
10