Estimation of the business value of software modernizations
ABSTRACT ELTIS is a research project aiming at supporting the process of selecting software modernization strategies and estimating the business value of modernizations. The project is funded by TEKES (National Technology Agency of Finland) and Finnish software industry. Modernizations are large-scale changes of information systems which are performed in order to keep it operational and competitive. Determination of the business value of software and its modernizations is an important but difficult problem. Both software suppliers and their user organizations enter decision making processes while considering possible modernizations. Decisions are typically made by groups of experts in a complex organizational setting. Relevant decision criteria of the software supplier include software maintenance benefits, risks, and costs. These issues include the effects of possible related reengineering and other factors. This position paper shortly describes our project, work done so far, and our research agenda: we aim at empirical data gathering and long-term iterative process improvement in co-operation with software industry.
Article: Why Reengineering Projects Fail
- [show abstract] [hide abstract]
ABSTRACT: Empirical software engineering research needs research guidelines to improve the research and reporting processes. We propose a preliminary set of research guidelines aimed at stimulating discussion among software researchers. They are based on a review of research guidelines developed for medical researchers and on our own experience in doing and reviewing software engineering research. The guidelines are intended to assist researchers, reviewers, and meta-analysts in designing, conducting, and evaluating empirical studies. Editorial boards of software engineering journals may wish to use our recommendations as a basis for developing guidelines for reviewers and for framing policies for dealing with the design, data collection, and analysis and reporting of empirical studies.IEEE Transactions on Software Engineering 09/2002; 28(8):721- 734. · 2.59 Impact Factor
- [show abstract] [hide abstract]
ABSTRACT: This work is based on a retrospective analysis of a renewal process applied to a very aged, highly degraded software system. Some parts of the results are generalized to define a method for determining the technical and economic qualities of software system components. The work also presents a decision model for identifying the most suitable renewal process to be applied, based on the quality of the components and the aims of the renewal process. With the model presented, decision‐making on the renewal process is specialized to each component of the software system, thus ensuring greater benefits from the process as a whole. The same model can be used to monitor software system quality decay and thus avoid the need to use the most costly renewal processes.Annals of Software Engineering 01/2000; 9:215-233.
Estimation of the Business Value of Software Modernizations
Jussi Koskinen, Jarmo Ahonen, Heikki Lintinen, Henna Sivula, Tero Tilus
Information Technology Research Institute
University of Jyväskylä
P.O.Box 35, FIN-40014 Jyväskylä, Finland
firstname.lastname@example.org, email@example.com, firstname.lastname@example.org
Date: 19th April 2004
ELTIS is a research project aiming at supporting the process of selecting software
modernization strategies and estimating the business value of modernizations. The project is
funded by TEKES (National Technology Agency of Finland) and Finnish software industry.
Modernizations are large-scale changes of information systems which are performed in order
to keep it operational and competitive. Determination of the business value of software and
its modernizations is an important but difficult problem. Both software suppliers and their
user organizations enter decision making processes while considering possible
modernizations. Decisions are typically made by groups of experts in a complex
organizational setting. Relevant decision criteria of the software supplier include software
maintenance benefits, risks, and costs. These issues include the effects of possible related
reengineering and other factors. This position paper shortly describes our project, work done
so far, and our research agenda: we aim at empirical data gathering and long-term iterative
process improvement in co-operation with software industry.
According to Lehman’s first law (Lehman et al., 1998) software must be continually adapted
or it will become progressively less satisfactory in “real-world” environments. Many legacy
systems have been very large investments, and they contain invaluable business logic and
knowledge. Thus, the modernization of these applications (instead of a complete rewrite), is
often potentially desirable option.
ELTIS (Extending the LifeTime of Information Systems) project started in 2003 and is
carried out at the University of Jyväskylä’s Information Technology Research Institute. The
project is outlined to last three years. The first year has been completed (with a budget of
about 200,000 euros) and the second year started (with a similar additional budget of 200,000
euros). The project is funded by TEKES (National Technology Agency of Finland), and
Finnish software companies: TietoEnator Inc., Tietokarhu Inc., and IBS (International
Business Systems) Inc. ELTIS aims at two main goals: 1) supporting the determination of the
profitability of extending information system lifetimes, and 2) providing a framework for
organizing software modernization support.
We’ve finished the background theory survey phase of the project reported as
(Koskinen et al., 2003a). We’ve also compared different methods aimed at supporting
software modernization estimation reported as (Koskinen et al., 2003b). Modernizations
should be performed by using technically sound and practical methods and tools, which
support the actual information needs of software maintainers (Koskinen et al., 2003c). We
have planned and started a data gathering phase, and have received some preliminary
empirical data. Our targets are our partner enterprises, and some of their customers. We have
also elaborated a preliminary model for the factors to be taken into account while deciding on
software modernizations (Koskinen et al., 2004). Qualitative data on software modernizations
have also been gathered (Lintinen et al., 2004). We will extend the data gathering in near
future, especially software modernization cases are interesting targets for research. Method
development will also be done related to pilot projects within our partner companies or their
customer organizations. In the following we will consider the possibilities and applicable
approaches for determining the business value of software modernizations.
2. Business value of software modernizations
2.1. Nature of modernizations
Generally modernizations mean large changes to the system, typically due to the major
changes in the technical context, e.g. related to the user interface, operating system,
programming language, system architecture, or major changes in business processes of the
organization using the software. These sort of large modernizations are generally both
technically, and humanly, hardest, and economically most critical maintenance situations.
2.2. Modernization options
First of all, the software supplier and the user organization may have different notions, and
possibly conflicting interests, on the issues of modernizations. The strategic options of
software suppliers are discussed e.g. by Bennett et al. (1999). The software user (i.e. client)
organization and the software supplier have the following main options as listed in Tables 1
1) Continuing using the currently used software product.
2) Replacement - purchasing a competing product (if such exists on markets).
Table 1. Main options of the software user organization.
1) Termination of system maintenance (e.g. warrants, corrections of errors, upgrades,
or user support).
2) Complete rewrite of the system.
3) Modernization in required extent (this may include e.g. reengineering, and/or
Table 2. Main options of the software supplier organization.
2.3. Determination of the Return on Investment (ROI) -ratio
For each involved organization it would (in principle) be valuable and sufficient to receive a
reliable Return on Investment (ROI) -ratio for each option. Our industrial partner enterprises
especially would appreciate this sort of information. One of the main goals of our project
during the second phase is to study this question. However, very few authors have actually
tried to determine ROI-estimates for software maintenance projects/tasks. Sneed (1995a) and
Verhoef (2002) have discussed the issue. In ideal (theoretical) situation there would probably
exist a decision support system taking into account all the relevant decision criterias,
producing the ROI-estimate, and explicating the argumentation (in style of expert support).
ROI reduces the effects of multiple underlying decision criteria into a single variable.
It is calculated (at the basic level) based on probable costs and benefits of an investment. The
reliability of the ROI-calculations naturally depends on the degree that relevant factors are
taken into account while determining it. For software suppliers, the potential criteria for
making a modernization decision are numerous. These include: direct modernization cost,
changed future maintenance cost, customer satisfaction (e.g. their ROI), and competitive
advantage. We have charted potential factors extensively in (Koskinen et al., 2003a).
In practice, the decision making is complicated by the “real-world” considerations.
First, making of software modernization decisions is a process within some organizational
context. “Real world” decision making in business organizations often has to be made based
on “bounded rationality” (Simon, 1983). This means that decisions have to be made without
complete information. Besides that there exists multiple (and possibly conflicting) decision
criteria, the certainty, completeness, and availability of useful information (as a basis for the
decision) is often limited. There may also exist major restrictions, such as already made
maintenance contracts or warrants, for the viable options. Especially organizational group
decision making is complex, see e.g. (March, 1981). Usually there are strategic-, political-,
legislative-, and regulatory considerations complicating the decision making. The made
decisions are not necessarily correct, and the evaluation of their correctness may be difficult
or impossible. Thus, the nature of this process should be taken into account while planning its
support (DeSanctis & Gallupe, 1987).
2.4. Modernization strategies and benefits
There exists the approches described in Table 3 for determining proper (high-level)
modernization strategies. Some of these approaches are also partly applicable for the actual
determination of the related, potential benefits.
SABA Bennett et al. (1999)
maintenance strategies based on aspired
customer satisfaction level.
High-level framework for planning the
evolution and migration of legacy systems
taking into account both organizational and
Method for iteratively evaluating legacy
systems, from technical, business, and
Measurement framework based on GQM
Method and decision model for determining
suitable software renewal processes at
component-level based on the technical and
economic qualities of those components.
Formal model for determining optimal
software rewrite and replacement timings
based on versatile metrics data.
& Zahedi for choosing appropriate
Renaissance Warren & Ransom
Model to Software
Table 3. Main methods of determining modernization strategies and benefits.
Strategic considerations are outlined most notably by Sahin & Zahedi (2001). Warranty-,
maintenance-, and upgrade (WMU) decisions can be made based on the suggestions of their
model during the system upgrade cycle. Also the probable system lifetime, and market
Chan et al. (1996)
Sneed (1995b) Process model for estimating costs and
benefits of reengineering.
volatility should be considered. All systems are not necessarily even meant to have long
lifetime (cf. prototyping, RAD, agile software development, strategic ventures etc.).
According to Lehman’s second law (Lehman et al., 1998), the complexity of software
systems increases as they evolve (unless additional effort, such as reengineering, is directed
to reducing it). Reengineering potentially enables continued system operation (and may
support flexible future changes). Sneed (1995b) has represented a 5-step method for
reengineering planning process (RPP). Sneed is one of the few researchers who has
considered both costs, benefits (measured via ROI), priorirization (based on portfolio
analysis) and risks (Sneed, 1999) of maintenance and RPP-processes. Our hypothesis is that
long-term improvements in system quality should be justifiable in case that system lifetime is
long, but those improvements should be justified economically. Teng et al. (1998) believe
that the profitability of reengineering is considerably better while there exists profound and
drastic (business) needs of change.
2.5. Modernization risk management
RMM (Risk-Managed Modernization, Seacord et al., 2003) is a new, general software
modernization management approach taking risks (and both technological and business
objectives) explicitly into account. It is aimed at disciplined risk-management, and creative
problem solving related to the selection of incremental modernization strategies. There exist
also approaches for risk analysis of reengineering projects (Sneed, 1999). We feel that this is
an important area and should be studied more extensively, especially empirically. There are,
e.g., risks related both to ignoring the improvement of the system quality and in
reengineering it (especially without a sufficiently well-thought-out long-term plan). In
addition, there exists RPFA (Reengineering Project Failure Analysis, Bergey et al., 1999)
which basically is a check-list of potential problems related to reengineering projects, and of
the corresponding appropriate technical and other means to react to the situation.
2.6. Modernization cost estimation
There exists the established general software cost estimation methods as well as methods
tailored for software maintenance. Most renown general methods are COCOMO, and FPA.
COCOMO II (Constructive Cost Model II, Boehm et al., 2000), is the new version of the
established and relatively widely used general method for software effort and cost estimation,
including many extensions for different kind of software and evaluation situations.
COCOMO II is well validated (with 161 industrial reference points). Like COCOMO, FPA
(Function Point Analysis, Albrecht & Gaffney, 1983) is an established and relatively widely
used general method for software effort and cost estimation, having many variants for
different kind of software, but also some limitations (Kitchenham, 1997).
Softcalc (Sneed, 1995a) is a 7-step method and tool for estimating costs of incoming
maintenance requests (developed based on COCOMO and FPA). The approach requires a
relatively large input data set. We believe that at least, the size of the maintenance task, its
potential side-effects (reflected e.g. by impact domain), and maintainability should be
considered. These models can (and potentially should) be applied at sub-system level. EMEE
(Early Maintenance Effort Estimation, De Lucia et al., 2002) is a new approach for quick
maintenance effort estimation before starting the actual maintenance. There exists also recent
efforts for applying FPA to software maintenance task evaluation (Ahn et al., 2003).
2.7. Theoretical and practical limitations of the existing models
There are three underlying main problems related to most of these models (which are in fact
typical problems in SE-field also more generally):
1) Lack of really reliable basic-level metrics. E.g. source lines of code is not an optimal
metric (since the length of program lines depends on the used programming language,
programming style, and amount of comments).
2) Many of the existing software maintenance estimation models (especially software
maintenance cost models) assume availability of unrealistic amount of input data, see
(Koskinen et al., 2003b). Most organizations are not at such capability maturity levels
that they would have established procedures for gathering this sort of extensive metrics
3) Software engineering is a research area where empirical validation is important, however
the empirical validation of most of the existing models typically is weak or non-existent.
We’ve described the industry-related ELTIS-project and the problem of determining the
business value of software modernizations. We’ve earlier charted the existing methods for
determining related software benefits, risks, and costs. We’ve also brought forth some of the
problems related to the use of these methods. Because of the limitations of the existing
models (and general problems of SE-field mentioned above), it is (in this case) more effective
first to focus on gathering versatile empirical data than trying to optimize the presented
models technically. We share the views of Kitchenham et al. (2002) of the criterias for
successful empirical software engineering research including the following: 1) a large data
source of programs and their versions, 2) willingness of a good commercial partner to
participate in the research project, and 3) highly disciplined research approach with desire to
expand the previous research.
We aim at: 1) iterative, step-wise long-term process improvement in the software
industry partner enterprises, regarding: software modernization estimation decision making
process by systematically enhancing the awareness of important issues, plausible risks,
options and possibilities, and introducing an approach best suited to the particular
organizational and technical characteristics of software suppliers and the specific problem. 2)
Gathering of (both quantitative and qualitative) empirical data.
Because the existing set of available methods has its limitations, there is a clear need
to gather empirical data on both: 1) actual system portfolios of software developing
organizations (including such attributes as average system lifetimes, change-history and
reasons for large-scale modernizations), as well as 2) actual criteria for modernization
decisions of their customers. More specifically we aim at gathering: both versatile
quantitative and qualitative data using: questionnaires (concerning system portfolios, and
performed modernization projects, focusing on software suppliers) and semi-structured
interviews (concerning expert decision criterias, focusing on user organizations).
Ahn, Y., Suh, J., Kim, S. & Kim, H. (2003). “The software maintenance project effort estimation model based
on function points”. Journal of Software Maintenance and Evolution: Research and Practice 15 (2), 71-85.
Albrecht, A. & Gaffney, J. (1983). “Software function, source lines of code, and development effort prediction:
a software science validation”. IEEE Transactions on Software Engineering SE-9 (6), 639-648.
Aversano, L., Esposito, R., Mallardo, T. & Tortorella, M. (2004). “Supporting decisions on the adoption of re-
engineering technologies”. CSMR 2004 Proceedings: Eight European Conference on Software
Maintenance and Reengineering, 95-104. IEEE Computer Soc.
Bennett, K., Ramage, M. & Munro, M. (1999). “Decision model for legacy systems”. IEE Proc.-Softw. 146 (3),
Bergey, J., Smith, D., Tilley, S., Weiderman, N. & Woods, S. (1999). “Why reengineering projects fail”.
Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, Report: CMU/SEI-99-TR-