Analysis of the Functional Size Measurement Methods Usage by Polish Business Software Systems Providers

Abstract

This paper analyses the level of using the software functional size measurement methods by the Polish providers of dedicated
business software systemsas well as the reasons behind this status quo. The surveys were conducted against a background of author’s own research concerning the usage of software development and
enhancement projects effort estimation methods. The use of both types of methods was examined in two cycles: at the turn of
the year 2005/2006, being the time of economic prosperity, and next at the turn of the year 2008/2009, that is in the initial
stage of crisis and increased investment uncertainty associated with it. This paper presents the most significant conclusions
coming from the results of both surveys as well as from comparative analysis of the two.

Full text

Analysis of the Functional Size Measurement Methods
Usage by Polish Business Software Systems Providers
Beata Czarnacka-Chrobot
Faculty of Business Informatics, Warsaw School of Economics,
Al. Niepodleglosci 164, 02-554 Warsaw, Poland
bczarn@sgh.waw.pl
Abstract. This paper analyses the level of using the software functional size
measurement methods by the Polish providers of dedicated business software
systems as well as the reasons behind this status quo. The surveys were
conducted against a background of author’s own research concerning the usage
of software development and enhancement projects effort estimation methods.
The use of both types of methods was examined in two cycles: at the turn of the
year 2005/2006, being the time of economic prosperity, and next at the turn of
the year 2008/2009, that is in the initial stage of crisis and increased investment
uncertainty associated with it. This paper presents the most significant
conclusions coming from the results of both surveys as well as from
comparative analysis of the two.
Keywords: software engineering, software development and enhancement
projects, business software systems, effort estimation methods, software
functional size measurement methods, ISO/IEC standards, IFPUG method,
COSMIC method, benchmarking data.
1 Introduction
As in the whole world, in Poland too the effectiveness of software projects execution
leaves a lot to be desired. The Standish Group [26, p. 1] estimates that now only 32%
of application software development projects worldwide turn out successful. The
Panorama Consulting Group [25, pp. 1-2] surveys on the effectiveness of the world’s
ERP systems implementations indicate that as much as 93% of them are completed
after the planned time, 65% go over the assumed costs while only 13% of the
respondents declare strong satisfaction with functionality being implemented in the
end product. Similar – as to the general conclusion – data result from the analysis of
IT projects being accomplished in Poland, which was carried out by M. Dyczkowski
[8, pp. 470-472], indicating that in 2006-2007 approx. 48% of such projects went over
the planned completion time while approx. 40% exceeded the estimated budget.
Low effectiveness of Software Development and Enhancement Projects (SD&EP)
is one of the fundamental reasons why for a few dozens of years the software
engineering has been in search of sufficiently objective and reliable approaches to the
software processes and products measurement. Some of the undertakings have only
recently gained recognition: for instance, the latest version of the CMMI model

(CMMI for Development) is strongly focused on measurement [3, pp. 178-197] as
well as that the ISO and IEC have recently established a dozen or so international
standards on measurement, regarding software products quality and Functional Size
Measurement (FSM) in particular, as discussed in detail in [6].
The set of rules for software FSM was normalised in the six-part ISO/IEC 14143
series [15] which, among others, provides key definitions, characteristics and
requirements for FSM, and also defines Functional Size Measurement Method
(FSMM) as a specific FSM implementation defined by a set of rules, which conforms
to the mandatory features of such measurement.
After about 30 years of improving various software FSM techniques five of them
(out of over 20) have been now acknowledged by the ISO/IEC as conforming to the
rules laid down in the ISO/IEC 14143, namely:
•Function Point Method developed by the International Function Point Users Group
(IFPUG) [171]
•Function Point Method in the Mk II version proposed by the United Kingdom
Software Metrics Association (UKSMA) [18]
•Function Point Method in the version developed by the Netherlands Software
Metrics Association (NESMA) [19]
•COSMIC-FFP Method proposed by the Common Software Measurement
International Consortium (COSMIC) [16]
•Functional Size Measurement Method developed by the Finnish Software Metrics
Association (FiSMA) [20].
The first three methods listed above are accepted by the ISO/IEC not in full
versions, as proposed by the organizations developing them, but in part – however in
the most important part with respect to the software functional size measurement [15,
Part 6, p. 5]. On the other hand the COSMIC and FiSMA methods were recognized as
international standard entirely ([15, Part 6, pp. 4-5], [20]).
2 Key Research Assumptions
The author of this paper for many years has been conducting studies concerning the
effective SD&EP scope management issue, which as a result has led her to the
interest in FSM methods (see e.g. [5]). These studies evidently indicate that objective
and reliable effort estimation of SD&EP, whose products are dedicated Business
Software Systems (BSS), still appears to be a great challenge to the software
engineering; therefore, clients commissioning them many times do not find grounds
to make rational decision on investment. On the other hand, her observations, backed
by the relevant desk research (see e.g. [7]), allow to advance a hypothesis that proper
FSMM usage could increase accuracy level of the such projects’ effort estimation
results.
Thus at the turn of the year 2005/2006 the author undertook research, whose aim,
generally speaking, was to analyse chiefly the level of using the SD&EP effort
estimation methods and their products’ FSM methods by the Polish providers of
1 This standard is now being revised.

dedicated BSS as well as the reasons behind this status quo2. Originally the research
was intended to examine also the FSMM reliability however sparse resources of the
data obtained made it impossible to formulate binding conclusions in this area.
It was assumed that the discussed studies will be continued after 5 years however
due to the change in the economic situation both worldwide and in Poland they were
repeated at the turn of the year 2008/2009, which enabled to gather some data for
comparative analysis.
Both research cycles were completed using the method of diagnostic survey: the
first cycle analysed responses given in 44 questionnaires (52 questionnaires were sent
out) while the second cycle – responses given in 53 questionnaires (62 questionnaires
were sent out). Questionnaires were distributed to various Polish dedicated BSS
providers, both internal (IT departments in organisations) as well as external (for the
most part from SME sector since there are only few large Polish IT companies
operating on the market), providing systems for the needs of financial institutions
(banks, insurance) departments, trading companies and public administration
institutions, all varying in size. In both cycles the overwhelming majority of responses
were answered by IT managers or project managers. Each questionnaire included
about 30 questions validated by experts; most questions were of open or semi-open
character and were divided into two main groups: concerning the usage of the effort
estimation methods (answered by all respondents) and concerning the usage of the
FSM methods (more numerous group of questions, answered only by the respondents
familiar with FSMM).
Due to the fact that the size of both samples may seem low it should be stressed
that the research was limited only to organisations dealing with SD&EP, whose
products are dedicated BSS – analysis included neither software maintenance and
support projects, software package acquisition and implementation projects, nor other
software products types. Each of the surveyed respondents declared completion of
several to a dozen or so projects in their organisation, meeting such requirements.
The adopted research perspective results from the following facts:
•BSS are one of the fundamental IT application areas
•BSS development or enhancement often constitutes serious investment undertaking
•In practice, COTS (Commercial-Off-The-Shelf) BSS rarely happen to be fully
tailored to the particular client business requirements therefore their customisation
(in smaller or bigger part) appears vital
•Rational ex ante and ex post valuation of unique (at least partially) BSS, being of
key significance to clients, encounters serious problems in practice
•From the provider’s perspective, the discussed type of IT projects is particularly
difficult in terms of management, which basically results in their exceptionally low
effectiveness as compared to other types of IT projects ([25, pp. 1-2], [26, p. 1]).
2 The author doesn’t know such surveys previously conducted in Poland.

3 Usage of Software FSMM by Polish Dedicated BSS Providers –
Conclusions from First Research Cycle
Level of using the functional size measurement methods by the surveyed Polish
dedicated BSS providers, together with the reasons behind this status quo, in both
cycles were analysed on the basis of the findings concerning the level of using the
SD&EP effort estimation methods by such providers. As mentioned above, the author
intended to investigate also FSMM popularity and reliability, but with the responses
delivered by the first research cycle the assumed goal could have been met only
partially, namely in the part concerning methods’ popularity. It resulted from the fact
that possession of own data sufficient for reliability analysis was declared only by a
non significant percentage of the respondents. The remaining ones declared they did
not collect such data at all nor had enough information to draw meaningful
conclusions on the applied methods reliability.
It comes as no surprise if we take into account that at the turn of the year
2005/2006 approx. 55% of the respondents declared they did not use commonly any
of the analysed SD&EP effort estimation methods, in most cases pointing to the
“price-to-win” technique as being the preferred project estimation approach. Basically
this situation is caused by the fact that Polish providers use this estimation technique
commonly when providing dedicated BSS commissioned by domestic government
institutions, because legal regulations reward the cheapest offers. Meanwhile the
surveys purposely did not analyse the level of using the two SD&EP effort estimation
ways being quite often employed in the SD&EP practice, that is the so-called “price-
to-win” technique and the so-called Parkinson rule [24, p. 6] as the author believes
both approaches may hardly be considered as having methodical grounds.
The studies analysed the usage of the following SD&EP effort estimation methods:
•Analogous estimating
•Decomposition based on Work Breakdown Structure (WBS)
•Expert methods (e.g. brain-storming, Delphi method)
•Algorithmic methods/models based on software size expressed in programming
units (e.g. source lines of code, which are applied, among others, in the COCOMO
method [2])
•Algorithmic methods/models based on software size expressed in functionality
units (e.g. function points, which are the base, among others, for Abran-Robillard
model [1, p. 83], ISBSG model [13, p. 6], Jiang-Naudé model [21]).
The level of using the analysed SD&EP effort estimation methods by the
remaining approx. 45% of the respondents, having been delivered by the first research
cycle, is displayed in Fig. 1.
The relatively lowest popularity of algorithmic methods, confirmed also by the
second research cycle (see Fig. 2 in section 4), is the result of:
•Difficulty in applying them, which concerns both types of such methods,
•Doubts as to their usefulness, which on the other hand applies mostly to the
methods based on product size expressed in programming units,
•Little familiarity with methods enabling to calculate product size in functionality
units.

11%
20%
27%
30%
36%
25%
45%
60%
67%
80%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Algorithmic -
programming units
Algorithmic -
functionality units
Decomposition – WBS Analogous estimating Expert methods
Types of projects effort estimation methods
Percentage
Total respondents
Estimation methods users
Fig. 1. The level of using the analysed SD&EP effort estimation methods by the surveyed
Polish dedicated BSS providers at the turn of the year 2005/2006 (sample n = 44)
Hence providers keep preferring other methods, particularly those of expert
character, regardless of the high risk associated with getting accurate estimates with
the usage of non-systematic approach: tests showed that the ratio of the effort
estimates, being calculated by project managers from different business areas for the
same project may be 1:6 or even 1:12 at the worst [12].
In the first research cycle it was also observed that providers employing effort
estimation methods usually do not follow only one such method. The most often used
combination is to apply expert methods along with decomposition based on WBS (at
the turn of 2005/2006 declared by 40% of those using methods); as a rule, they are
used in the same project at different execution stages. It happens relatively often that
expert methods are used along with models based on functionality units (in the first
research cycle declared by 25% of those using effort estimation methods). It mainly
originates either in the need to sort out significant disagreements between diverse
estimations of several experts – in this case, using general (mean) benchmarking data
usually is a solution, or possibly in client’s requirements concerning necessity to
provide justification for estimation results. While lack of sufficient own relevant
benchmarking data collection is the main reason why functionality units models are
employed by providers along with expert methods, and possibly with other estimation
methods.
Results obtained with the usage of the employed effort estimation methods are
designed for estimating SD&EP costs and time frame while relatively rarely they are
used to estimate projects’ economic efficiency (at the turn of 2005/2006 such use of
these methods was indicated by only 25% of those using effort estimation methods).
In companies dealing with delivery of externally-commissioned dedicated BSS,
clients rarely require to be presented with quantitative evidence of benefits to be
brought by product implementation, which becomes understandable if we take into
account that adequate calculation of software development or enhancement project
profitability requires knowing the specificity of client’s activity. On the other hand,

heads of IT departments in Polish companies, for which SD&EP are executed, still
explain the sporadically required calculation of this type of investments profitability
mostly by the necessity to undertake them – most often due to the fact that without
such solutions they lack possibility to match competition from foreign companies
where they constitute basis of functioning, as well as to match foreign business
partners requirements. While Polish public administration institutions still do not see
in practice the need for the SD&EP economic efficiency evaluation, in most cases as
an argument giving the non-economic purposes of IT systems being implemented in
this type of organisations.
The results of the first cycle of the research on the SD&EP effort estimation
methods usage indicated that approx. 20% of the Polish dedicated BSS providers
under study declared that for this very purpose they commonly employed algorithmic
methods based on functionality units (see Fig. 1). It means that 45% of domestic
providers employing effort estimation methods to determine product size used
FSMM. What’s interesting, the research indicated that basically this was only one of
such methods being used, namely the IFPUG Function Point (FP) method [11].
Additionally, in one case rather superficial awareness of COSMIC method [4] was
declared whereas in two cases possibility to employ the use case points method
instead of IFPUG method was considered. Familiarity with the IFPUG method was
declared by approx. 27% of all respondents, that is 60% of the respondents using
effort estimation methods, which means that 75% of those familiar with the method
also employed it.
The key findings concerning the usage of the IFPUG method resulted from the first
research cycle are summarized in Table 1. They need some additional comments:
•In Poland the IFPUG method in most cases has been employed from the first years
of this century; using it since 1999 was declared in one case. Since this moment
each of the providers employing this method has used it – in nearly 100% in full
version, i.e. together with Value Adjustment Factor (VAF) - to determine the size
of several to a dozen or so software products. As a rule this method was mostly
used in case of business applications – some respondents used to stress even its
very high reliability in estimating the effort for this product type. In several cases it
was stated that the discussed method usage was limited to rather relatively small
business applications (up to 500 IFPUG FPs), at the same time pointing out that
these applications dominate on the Polish market. In other cases, it was maintained
that the method was successfully used also, and quite often first of all, for large or
very large BSS (up to several thousands of IFPUG FPs).
•As pointed out by the respondents already in the first cycle, key benefit coming
from the IFPUG approach usage is in helping to increase effectiveness of
delivering the required functionality on time and within the planned budget.
Fundamental purposes for using the discussed method indicated by Polish
dedicated BSS providers analysed at the turn of 2005/2006 are featured in detail in
Table 4 in section 4 together with purposes delivered by the research held 3 years
later.
•As early as at the turn of 2005/2006 over half the surveyed Polish providers using
the IFPUG method declared they employed it together with expert methods
(usually with regard to one project). This caution mostly results from the lack of
own relevant benchmarking data that would allow for deriving dependencies

specific to organisation, although collecting of such data for some time has been
declared. Providers often explain the usage of both methods by the verification
need of the approach based on function points, not excluding the possibility that it
might be the only method used in the future. In case of using both the above
methods general (mean) benchmarking data as a rule are used for the effort
estimation, sometimes after being corrected with standard factors affecting the
effort. Such approach usually proves satisfactory in case of estimation results
justification required by a client or when it is necessary to sort out considerable
differences in various experts estimations.
•The IFPUG method happened to be used also together with decomposition based
on WBS or in order to use the COCOMO II method later on. In the first research
cycle two providers declared they used it alone, at the same time pointing out they
rather employed own benchmarking data – being used to calculate project
activities’ productivity, which proves middle level of the effort estimation model’s
adjustment to the organisation’s specificity [22]. It is worth mentioning here that
the research carried out at the time revealed no case of high estimation model
adjustment level, being the level at which provider’s own regression equation
would be used. It results from the lack of sufficient relevant organizational
benchmarking data collection as well as from the perception of such approach
effort as excessive in relation to the prospective profits.
•Among advantages of the IFPUG method, most providers using it (either alone or
together with other approach) indicated method’s objectivity and its high
usefulness, including most of all possibility to employ it at initial project stages at
sufficient, or even high/very high estimates accuracy level in relation to BSS,
which allows for effective SD&EP effort, costs and time estimation. Other
advantages of the method indicated by the respondents already in the first research
cycle include: the approach’s repeatability and regularity, getting better project
understanding, greater client engagement, easier project activities and their results
control as well as independence of the used technology, life cycle model and
project developing methodologies, what ensure results comparability. What’s
more, among advantages was also indicated possibility to make use of out-of-
organisation knowledge and experience as well as possibility to refer the
estimation results obtained by using this method to the outside statistics, which is a
considerable argument in negotiations with a client.
•Respondents declaring familiarity with the IFPUG method (both its users and non-
users) noted its disadvantages, too. Three years ago majority in this group pointed
first of all to the high difficulty level in applying it, including problems with
interpretation, which in the case of providers who are familiar with yet do not use
the discussed method happens to be the reason why they quit using it (what seems
interesting, however, is that these respondents declared at most low awareness of
the IFPUG method supporting tools). The above makes the method both work- and
time-consuming. In addition, providers familiar with the discussed method used to
stress subjectivity in the assessment of the General System Characteristics’ (GSC)
influence on the functional size (this step of the IFPUG method is not accepted by
the ISO/IEC) as well as the fact that it not sufficiently takes into account the high
processing complexity although quite often the method is found having
universality level being sufficient from the viewpoint of the providers needs.

Table 1. The key findings on the IFPUG method usage from the research cycle undertaken at
the turn of the year 2005/2006 (sample n = 44)
Issue Finding
Familiarity with the
IFPUG method
-27% of all respondents
-60% of the respondents using effort
estimation methods
The IFPUG method usage
-20% of all respondents
-45% of the respondents using effort
estimation methods
-75% of the respondents familiar with the
IFPUG method
Familiarity with tools
supporting the IFPUG
method 50% of the respondents familiar with the IFPUG method
The IFPUG method usage
along with other methods
In most cases with expert methods (over half the
respondents using the IFPUG method), because of the lack
of own relevant benchmarking data
The form of the IFPUG
method usage In nearly 100% in full version (with VAF)
The products measured
with the IFPUG method Mostly BSS, different in size
The key benefit of the
IFPUG method usage The increase of effectiveness in delivering products
The main IFPUG method
advantages
Objectivity, possibility to employ at initial project stages,
repeatability, regularity, getting better project
understanding, greater client engagement, easier project
control, independence of the used technology, life cycle
model and project developing methodologies
The main IFPUG method
disadvantages High difficulty, GSC assessment subjectivity, not sufficient
taking into account the high processing complexity
As already mentioned, in order to observe changes in the approach of Polish
providers of BSS towards SD&EP estimation methods and software FSM methods,
the author originally intended the research to be repeated at the turn of 2010 and
2011. However radical change in the economic situation worldwide and in Poland
persuaded her to undertake it at the turn of 2008 and 2009, although not without
doubts as to the possibility to observe changes within such a short time. The results
obtained in the second research cycle are presented and compare with the findings of
the first cycle in the next section, whereas main concluding remarks from the both
cycles can be found in section 5.
4 Usage of Software FSMM by Polish Dedicated BSS Providers –
Conclusions from Second Research Cycle
In the second research cycle the range of analysis was extended by 9 Polish dedicated
BSS providers. At this time the respondents were pointing to the need to increase

caution when investing in SD&EP, which entailed lower number of such projects
being undertaken, in addition being usually projects of smaller size.
Among 53 Polish dedicated BSS providers surveyed in this cycle, approx. 53%
declared they commonly used the analysed SD&EP effort estimation methods, which
means increase by 8 percentage points comparing to the previous research cycle.
Also, the number of providers indicating ‘price-to-win’ technique as the only method
used to estimate such projects attributes has slightly lowered. Level of using particular
SD&EP effort estimation methods by the respondents surveyed at the turn of 2008
and 2009 is displayed on Fig. 2.
In Table 2 it is contrasted with the relevant level revealed by the preceding
research cycle.
8%
26%
34%
30%
43%
14%
50%
64%
57%
82%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Algorithmic -
programming units
Algorithmic -
functionality units
Decomposition – WBS Analogous estimating Expert methods
Types of projects effort estimation methods
Percentage
Total respondents
Estimation methods users
Fig. 2. The level of using the analysed SD&EP effort estimation methods by the surveyed
Polish dedicated BSS providers at the turn of the year 2008/2009 (sample n = 53)
Table 2. The level of using the analysed SD&EP effort estimation methods by the surveyed
Polish dedicated BSS providers at the turn of the year 2005/2006 and at the turn of the year
2008/2009*
2005/2006 2008/2009
Methods Percentage
Sample
n = 44
Percentage
Sample
k = 20
Percentage
Sample
n = 53
Percentage
Sample
k = 28
Expert methods 36% 80% 43% 82%
Analogous estimating 30% 67% 30% 57%
Decomposition based on WBS 27% 60% 34% 64%
Algorithmic methods/models
based on functionality units 20% 45% 26% 50%
Algorithmic methods/models
based on programming units 11% 25% 8% 14%
* n - number of surveyed Polish dedicated BSS providers
k - number of surveyed Polish dedicated BSS providers declaring the analysed SD&EP effort
estimation methods usage

Data presented in Table 2 indicate:
•Relative increase of use: expert methods, decomposition based on WBS, and
algorithmic methods based on functionality units. Considering all respondents, this
increase is lowest for functionality units models, while these models slightly takes
the lead with regard to Polish dedicated BSS providers declaring usage of the
analysed SD&EP effort estimation methods.
•Relative decrease of use: analogous estimating (with regard to providers declaring
usage of the analysed SD&EP effort estimation methods) and algorithmic methods
based on programming units.
Results obtained with the usage of the analysed SD&EP effort estimation methods
are now more often used to estimate projects’ efficiency (increase from 25% to
approx. 36% of those using effort estimation methods) – this applies to internal IT
departments of Polish companies yet still it does not comprise Polish public
administration institutions.
Like in the previous research cycle, Polish providers employing effort estimation
methods declare they usually do not follow only one such method. No significant
changes were noted in this respect, except in the increased frequency of using expert
methods along with functionality units models (increase from 25% to approx. 29% of
those using effort estimation methods). The reasons behind this status quo have not
changed, either; they still include: need to sort out significant disagreements between
diverse estimations of several experts, or client’s requirements concerning necessity
to provide justification for estimation results. Lack of sufficient own relevant
benchmarking data continues to be the main reason why functionality units models
are employed by Polish providers along with expert methods (now approx. 57% of the
surveyed Polish providers using the FSMM). Still, as a rule, general benchmarking
data are used for the effort estimation, sometimes after correction with appropriate
factors.
Although there is an increase in the level of using models based on functionality
units, it hardly can be considered satisfactory since these methods still rank at 4th
position out of total 5 positions in this respect. The reasons why remain the same;
they are: difficulty in applying these methods and still relatively low familiarity with
them. Providers keep on preferring the expert methods, involving high risk, or
decomposition based on WBS, which may be effectively used relatively late in the
software development or enhancement project lifecycle, or analogous estimating,
allowing for very approximate prognosis only with regard to the projects that are very
similar to each other. Although the latter ones dropped to the third position in the
2008/2009 study; they, however, are still employed more often than functionality
units models.
As far as FSM methods are concerned, the IFPUG method continues to hold
considerable advantage although switching it to the COSMIC method due to the
easier usage of the latter was noted in two cases while other two respondents reported
high probability of such a switch. This time, one respondent declared employing the
use case points method, which replaced the feature points method due to development
technique changing from traditional into object-oriented one. Familiarity with FSM
methods has grown: it was declared by approx. 34% of all respondents (2005/2006:
approx. 27%), that is by approx. 65% of the respondents using effort estimation

methods (2005/2006: 60%), which means that approx. 78% of those familiar with
these methods also employ them (2005/2006: 75%).
These key findings on the FSM methods usage from the second survey cycle,
which are different from the first cycle results (see Table 1) are summarized in Table
3. They need some additional comments:
•Similarly to the research conducted 3 years earlier, the FSM methods happen to be
used, besides expert methods, also together with the COCOMO II or with
decomposition based on WBS. This time four providers declared they used them
alone (IFPUG method: 3, COSMIC method: 1) with employing own benchmarking
data to calculate own productivity (3) or to draw own regression equation (1).
•FSMM evaluation did not change considerably, either – respondents declaring
familiarity with the FSM methods used to point out similar advantages and
disadvantages as three years earlier with regard to the IFPUG method. Among
disadvantages the respondents stress first of all high difficulty in applying all FSM
methods as well as GSC assessment subjectivity in relation to IFPUG method.
•However in the majority of cases the IFPUG method continues to be used in full
version, i.e. with VAF. Although the VAF usage has being criticized in the
literature, as it doesn’t cause the better correlation to effort estimation (see e.g. [9,
p. 98]), in the opinion of the surveyed providers using this method the VAF
adjustment step enables to take into account other, non-functional requirements.
•Awareness of tools dedicated to support FSM methods has increased.
Table 3. The key findings on the FSM methods usage from the research cycle undertaken at
the turn of the year 2008/2009 - only differences with regard to the survey conducted 3 years
earlier (sample n = 53)
Issue Finding
Familiarity with FSM
methods
-34% of all respondents
-65% of the respondents using effort
estimation methods
FSM methods usage
-26% of all respondents
-45% of the respondents using effort
estimation methods
-78% of the respondents familiar with
FSMM
Familiarity with tools
supporting FSM methods 67% of the respondents familiar with the FSMM
Fundamental purposes for using the FSM methods were analysed on the basis of
purposes indicated in the first research cycle for the IFPUG method; the respondents
were asked to supplement them whenever they felt it was applicable. Table 4 displays
the obtained results, which are related to the purposes for using FSM described in the
ISO/IEC 14143 norm.
Data presented in Table 4 indicate that:
•In both research cycles higher importance is assigned to the purposes of Project
Management group
•Fundamental purpose of using FSMM indicated in both research cycles is product
size estimation in order to effectively estimate the effort, costs and time frame for

the initiated project, which is the purpose belonging to the Project Management
group
•Among purposes belonging to the Performance Management group, productivity
management was indicated as the most important one in both research cycles
•At the turn of 2008 and 2009, three new items appeared on the list of purposes for
using FSMM, namely: managing organization maturity and determining degree to
which the supplied dedicated product or the COTS meets functional user
requirements – in the first cycle they were indicated by none of the surveyed Polish
dedicated BSS providers.

Table 4. Basic purposes for using the FSM methods indicated by the surveyed Polish
dedicated BSS providers at the turn of the year 2005/2006 and at the turn of the year 2008/2009
Purpose indicated by Polish dedicated BSS
providers Percent.
in 2006 Percent.
in 2009 ISO/IEC 14143
purpose
Estimation of product size and, based on
this, estimation of the effort, costs and time
frame for the project being initiated – in
order to design own offer as well as for the
commissioned applications
100% 100% Project
resource
forecasting
Project Management
Supporting decisions about rationality of
initiating the projects and way of
completing projects (e.g. using own
resources or by outsourcing)
56% 64%
Monitoring progress, costs and time in the
project execution 67% 64% Tracking the
progress of a
project
Managing the changes in the required
product size and their influence on project
work effort 44% 36% Managing
scope change
Determining degree to which the
Commercial-Off-The-Shelf meets
functional user requirements 0% 7% Package
functionality
fit
Comparing attributes of the finished
project with other projects 44% 50% Post-mortem
analysis
Managing software development,
enhancement or maintenance productivity 78% 86% Productivity
management
Performance Management
Managing software reliability 44% 50% Quality
management
Managing organization’s maturity 0% 7%
Organizational
maturity and
process
capability
Measuring existing applications in order to
determine their value to estimate costs of
its potential replacement, reengineering, or
outsourcing
56% 64%
Accounting for
an
organization’s
software asset
Making prognosis on the budget necessary
to maintain software 33% 29% Budgeting
for
maintenance
Managing the product size and project
scope in the client-provider relations 67% 78%
Contract
management
Valuation of applications being executed
by other companies 56% 57%
Determining degree to which the supplied
dedicated product meets functional user
requirements 0% 14%
Source: Author’s own study with the use of: [15, Part 6, pp. 9-10].
One of the fundamental differences between the surveys carried out in the second
cycle and those carried out 3 years earlier is acquisition of data, which in the case of
three providers allowed for the IFPUG method reliability analysis (in all these cases

being used along with expert effort estimation methods). For this method analysis of
prognoses accuracy was made in comparison with actual end product’s size based on
the number of Unadjusted Function Points (UFPs):
•estimated on the basis of data model and function model, with average complexity
being assumed for the function, depending on its type (FP1)
•calculated in accordance with method’s recommendations on the basis of
requirements specification (FP2).
When analysing reliability, prediction accuracy indicator PRED(RE) was
employed, which serves to express what in the surveyed cases is the percentage share
of these projects/products whose estimates are contained within the assumed
estimation Relative Error (RE) related to the actually received value [10]. Thus in
order to consider a method reliable the PRED(30) was assumed on the level not lower
than 80% [1, p. 81].
What also was calculated is the PRED(10), in order to compare prediction
accuracy level with surveys conducted by the ISBSG (International Software
Benchmarking Standards Group), in which allowable estimation error was assumed
on the level of ±10%. On the other hand, level of prediction accuracy for the effort
estimated on the basis of UFPs was not analysed at all since the obtained data proved
being not sufficient to do so.
Evaluation of the IFPUG method reliability resulting from the obtained data is
displayed in Table 5. All products considered in the analysis are rather relatively
small business applications (up to 600 IFPUG UFPs). As indicated by these data, the
IFPUG method limited to unadjusted FPs meets the assumed reliability condition in
the case of calculations made, according to method’s recommendations, on the basis
of requirements specification (FP2). If estimates are made on the basis of data model
and function model with average complexity being assumed for the function, this
method does not meet the prediction accuracy condition in any of the analysed cases.
Author’s observations indicate that fundamental reason behind this status quo is the
skipping of a significant number of External Outputs (EO) and/or External inQuiries
(EQ) at this project stage.
Thus the research results confirm that better effects may be achieved if calculations
are made on the basis of requirements specification, which is consistent with the
conclusion coming from the ISBSG analyses [14, pp. 5-6]. Yet the obtained results
appear, generally speaking, worse in comparison with this institution’s report where
estimates calculated on the basis of requirements specification with allowable RE
assumed on the level of ±10% were not lower than actual product functional size in
the case of approx. 70% of projects whereas those calculated on the basis of data
model and function model – in the case of approx. 62% of projects. This may result
from the fact that in the discussed survey providers presented the author with data
coming from SD&EP chosen by chance, without prior thorough analysis being made
(not from the best projects, which was probably the case of ISBSG) as well as from
scantier experience in using FSM methods in our country.

Table 5. Evaluation of the IFPUG method (UFPs) reliability on the basis of data delivered by
3 Polish dedicated BSS providers – FP1 and FP2 calculations
FP1
Polish
provider
Sample
(number of
products
analysed)
Number of
products
within the
assumed RE
= ±30%
PRED(30)
Number of
products
within the
assumed RE
= ±10%
PRED(10)
Provider 1
(small IT
company) 11 8 73% 6 55%
Provider 2
(IT
department
in a bank)
11 7 64% 5 45%
Provider 3
(medium-
sized IT
company)
14 10 71% 7 50%
FP2
Polish
provider
Sample
(number of
products
analysed)
Number of
products
within the
assumed RE
= ±30%
PRED(30)
Number of
products
within the
assumed RE
= ±10%
PRED(10)
Provider 1
(small IT
company) 11 9 82% 7 64%
Provider 2
(IT
department
in a bank)
11 9 82% 7 64%
Provider 3
(medium-
sized IT
company)
14 12 86% 10 71%
5 Concluding Remarks
Summary of the key findings delivered by both cycles of survey, which aimed to
attempt to analyse the level of using FSM methods in the Polish BSS development
and enhancement projects reality as well as to diagnose the status quo causes, is
presented in Table 6.

Table 6. Summary of the results of surveys on using the SD&EP effort estimation methods
and their products’ FSM methods by the surveyed Polish dedicated BSS providers
2005/2006 2008/2009
Number of surveyed Polish dedicated BSS
providers 44 53
SD&EP effort estimation methods usage 45% 53%
Expert effort estimation methods usage 36% 43%
Familiarity with FSM methods 27% 34%
FSM methods usage 20% 26%
Expert methods along with FSM methods usage 11% 15%
Familiarity with tools dedicated to support FSM
methods 14% 23%
When summing up conclusions it should be stated that:
•Considerable part of the respondents declares they do not commonly employ any
of the methodology-based approaches to the SD&EP effort estimation; however,
the level of using such methods has increased.
•In both research cycles the respondents declared rather widespread usage of at least
one of the effort estimation methods, mostly pointed to the expert methods.
•Relatively lowest popularity of two algorithmic methods, in both research cycles,
mostly results from doubts as to the usefulness of models based on programming
units and from insufficient familiarity with the methods enabling to estimate
software size in functionality units.
•Familiarity with the methods based on functionality units as well as the level of
using them have increased over the analysed time. Percentage of the respondents
using these methods versus those familiar with the methods has increased slightly
too, which means that the overwhelming majority of those familiar with discussed
methods are also employing them. Yet these methods still place at the penultimate
position among all methods being used for SD&EP effort estimation by the
surveyed Polish dedicated BSS providers.
•It happened, and happens relatively often, that expert methods, burdened with high
risk, are employed along with functionality units models. It mainly comes from the
need to sort out significant discrepancies in differing several experts estimations, or
from client’s requirements regarding necessity to justify estimation results.
•The above means that over half the surveyed Polish dedicated BSS providers
declaring usage of the FSM methods employ it along with expert methods. This
caution is caused mostly by the lack of own relevant benchmarking data.
•In both research cycles, the main purpose of using the FSM methods is product
size estimation in order to effectively estimate the effort, costs and time frame for
the initiated project. This corresponds to the ISO/IEC 14143 purposes from the
Project Management group. Among purposes of the Performance Management
group, productivity management in both research cycles is regarded as the most
important one.
•In both research cycles the main advantages of the FSM methods are the methods
objectivity and high usefulness, including most of all possibility to employ them at
initial project stages at sufficient accuracy level of estimates, which helps to

increase the effectiveness of delivering the required functionality on time and
within the planned budget.
•Disadvantages of the FSM methods include high difficulty level in using them (all
FSMM), GSC assessment subjectivity and not sufficient taking into account the
high processing complexity (IFPUG method).
•Among FSM methods significant advantage is still being held by the IFPUG
method (with VAF), in case of which some respondents used to stress very high
reliability with regard to business applications; however in several cases there has
been a switch to the COSMIC method or such a switch was considered, this being
due to easier application. Other FSMM standardized by the ISO/IEC are practically
unknown in Poland.
•Analysis of data received in the second research cycle from 3 Polish dedicated BSS
providers indicates that the IFPUG method on the level of unadjusted FPs meets
the assumed reliability condition if calculation is based on requirements
specification. Yet it fails to meet this condition if estimation is made on the basis of
data model and function model with average complexity being assumed for the
functions.
•In the case of all respondents in the two studies, the main reason for relatively low
popularity of the FSM methods is that none of the SD&EP effort estimation
methods is used commonly as well as insufficient familiarity with the methods,
whereas among respondents using estimation methods – insufficient the FSMM
awareness and at the same time familiarity with other approaches. Among
providers declaring familiarity with the FSM methods the main reason why they
quitted using them is, next to their high difficulty, lack of relevant organizational
benchmarking data and at the same time lack of trust in general data.
•Over a span of three years the awareness of tools dedicated for supporting the FSM
methods has increased.
The FSM methods stayed practically unknown in Poland until the recession that
took place in the first years of the 21st century. Although the level of using these
methods can be hardly considered high, increase in their popularity, however, may be
possibly explained by four main factors, namely:
•Stronger care about financial means in the times after recession mentioned above,
including current crisis where it appears even somewhat stronger;
•Growing competition on the market and increasing market globalization level;
•Growing awareness of clients therefore greater requirements concerning providing
justification for the project costs and completion time offered by providers;
•Standardization of the FSM conception and its several methods by the ISO/IEC.
The main conclusion coming from the above analysis agrees with the general
conclusion drawn by the Software Engineering Institute (SEI) on the basis of the
research attempted to answer the question about today’s approach to the measurement
of software processes and products: “From the perspective of SEI's Software
Engineering Measurement and Analysis (SEMA) Group, there is still a significant
gap between the current and desired state of measurement practice. (…) Generally
speaking, based on the results of this survey, we believe that there is still much that
needs to be done so that organizations use measurement effectively to improve their
processes, products, and services” [23, p. 29].

The research will be continued to keep observing the changes while the research
area will be extended as much as possible to other Polish dedicated BSS providers
and other SD&EP scope management aspects, with particular consideration of the
software functional size measurement methods reliability.
References
1. Abran, A., Robillard, P.N.: Reliability of Function Points Productivity Models for
Enhancement Projects (A Field Study). Conference on Software Maintenance 1993-CSM-
93, Montreal, pp. 80--97. IEEE Computer Society Press, Los Alamitos (1993)
2. Boehm, B., et al.: Software cost estimation with COCOMO II. Prentice Hall, Upper
Saddle River, Englewood Cliffs, NJ (2000)
3. CMMI Product Team: CMMI for Development, Version 1.2. Software Engineering
Institute, Carnegie Mellon University, Pittsburgh (2006)
4. Common Software Measurement International Consortium: The COSMIC Functional Size
Measurement Method, Version 3.0, Measurement Manual. COSMIC, Québec (2007)
5. Czarnacka-Chrobot B., O uzytecznosci metod wymiarowania funkcjonalnego
informatycznych przedsiewziec projektowych [About IT Projects Functional Measurement
Methods Usefulness]. In: Szyjewski, Z., Grabara, J.K., Nowak, J.S. (eds.) Strategie
informatyzacji [IT Implementation Strategies], pp. 107-134. Polish Information Processing
Society, Katowice (2006).
6. Czarnacka-Chrobot B.: The ISO/IEC Standards for the Software Processes and Products
Measurement. In: Frontiers in Artificial Intelligence and Applications. Proceedings of
the 8th International Conference on Software Methodologies, Tools & Techniques
SOMET’2009. IOS International Publisher, Amsterdam (2009) (in press)
7. Czarnacka-Chrobot B.: Wiarygodnosc metod szacowania pracochlonnosci przedsi wziecę
rozwoju systemow oprogramowania wspomagajacych zarzadzanie [Reliability of the
Business Software Systems Development and Enhancement Projects Effort Estimation
Methods]. In: Informatyka ekonomiczna, Informatyka w zarzadzaniu [Business
Informatics, Informatics in Management]. Prace Naukowe Uniwersytetu Ekonomicznego
we Wroclawiu, Wroclaw (2009) (in press)
8. Dyczkowski, M.: Ocena przebiegu i efektow przedsiewziec informatycznych. Wybrane
wyniki badan porownawczych z lat 2004–2007 [Evaluation of the course and effects of IT
projects. Selected results of comparative studies over the years 2004-2007]. In: Porebska-
Miac, T., Sroka, H. (eds.) Systemy wspomagania organizacji SWO'2007 [Organisation
Support Systems SWO’2007], pp. 465-474. Prace Naukowe Akademii Ekonomicznej w
Katowicach, Katowice (2007)
9. Fenton N., E.: Zapewnienie jako ci i metryki oprogramowania [Ensuring software qualityś
and software metrics]. In: Górski, J. (ed.) In ynieria oprogramowania w projekcież
informatycznym [Software Engineering in IT Project], 2nd edition. Mikom, Warsaw (2000)
10. Ferens, D.V., Christensen, D.S.: Does Calibration Improve Predictive Accuracy? In:
CrossTalk. The Journal of Defence Software Engineering, April 2000, pp. 14--17 (2000)
11. International Function Point Users Group: IFPUG Function Point Counting Practices
Manual, Release 4.2, Part 1-4. IFPUG, Princeton Junction, NJ (2004)
12. International Software Benchmarking Standards Group (ISBSG),
http://www.isbsg.org/Isbsg.Nsf/weben/Functional%20Sizing%20Methods (4/21/2008)
13. International Software Benchmarking Standards Group: The ISBSG Special Analysis
Report: Early Lifecycle Software Estimation. ISBSG, Hawthorn VIC, Australia (2005)
14. International Software Benchmarking Standards Group: The ISBSG Report: Software
Project Estimates – How accurate are they? ISBSG, Hawthorn VIC, Australia (2005)

15. ISO/IEC 14143 Information Technology – Software measurement – Functional size
measurement – Part 1-6. ISO, Geneva (1998-2007)
16. ISO/IEC 19761:2003 Software engineering – COSMIC-FFP – A functional size measurement
method. ISO, Geneva (2003)
17. ISO/IEC 20926:2003 Software engineering - IFPUG 4.1 Unadjusted functional size
measurement method - Counting practices manual. ISO, Geneva (2003)
18. ISO/IEC 20968:2002 Software engineering – Mk II Function Point Analysis - Counting
practices manual. ISO, Geneva (2002)
19. ISO/IEC 24570:2005 Software engineering – NESMA functional size measurement
method version 2.1 - Definitions and counting guidelines for the application of Function
Point Analysis. ISO, Geneva (2005)
20. ISO/IEC 29881:2008 Information Technology – Software and systems engineering –
FiSMA 1.1 functional size measurement method. ISO, Geneva (2008)
21. Jiang, Z., Jiang, B., Naudé, P.: The Effects of Software Size on Development Effort and
Software Quality. In: International Journal of Computer and Information Science and
Engineering, vol. 1, no. 4, pp. 230--234 (2007)
22. Jørgensen, M.: Estimation of Software Development Work Effort: Evidence on Expert
Judgment and Formal Models. In: International Journal of Forecasting 23(3), pp. 449--462
(2007)
23. Kasunic, M.: The State of Software Measurement Practice: Results of 2006 Survey.
Software Engineering Institute, Carnegie Mellon University, Pittsburgh (2006)
24. Leung, H., Fan, Z.: Software Cost Estimation. The Hong Kong Polytechnic University,
Hong Kong (2006)
25. Panorama Consulting Group: 2008 ERP Report, Topline Results. Denver (2008)
26. Standish Group: CHAOS Summary 2009. West Yarmouth, Massachusetts (2009)