The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment

Simula Res. Lab., Univ. of Oslo, Lysaker, Norway
IEEE Transactions on Software Engineering (Impact Factor: 2.29). 11/2011; DOI: 10.1109/TSE.2010.78
Source: IEEE Xplore

ABSTRACT Studies in laboratory settings report that software development effort estimates can be strongly affected by effort-irrelevant and misleading information. To increase our knowledge about the importance of these effects in field settings, we paid 46 outsourcing companies from various countries to estimate the required effort of the same five software development projects. The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows: 1) reduced length of requirement specification with no change of content, 2) information about the low effort spent on the development of the old system to be replaced, 3) information about the client's unrealistic expectations about low cost, and 4) a restriction of a short development period with start up a few months ahead. We found that the effect sizes in the field settings were much smaller than those found for similar manipulations in laboratory settings. Our findings suggest that we should be careful about generalizing to field settings the effect sizes found in laboratory settings. While laboratory settings can be useful to demonstrate the existence of an effect and better understand it, field studies may be needed to study the size and importance of these effects.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, a literature review and classification scheme for selected software engineering researches is presented. The study shows that an increasing volume of software engineering researches have been conducted in diverse range of areas. The articles are classified and results of these are presented based on classification scheme that consist of five main categories: software development process, software management, software engineering techniques, software re-engineering, and software applications. Analyses of the selected researches are carried out and gaps in the research are identified. A comprehensive list of references is presented. This review is intended to provide impetus in research and help simulate further interest.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: In formal experiments on software engineering, the number of factors that may impact an outcome is very high. Some factors are controlled and change by design, while others are are either unforeseen or due to chance. This paper aims to explore how context factors change in a series of formal experiments and to identify implications for experimentation and replication practices to enable learning from experimentation. We analyze three experiments on code inspections and structural unit testing. The first two experiments use the same experimental design and instrumentation (replication), while the third, conducted by different researchers, replaces the programs and adapts defect detection methods accordingly (reproduction). Experimental procedures and location also differ between the experiments. Contrary to expectations, there are significant differences between the original experiment and the replication, as well as compared to the reproduction. Some of the differences are due to factors other than the ones designed to vary between experiments, indicating the sensitivity to context factors in software engineering experimentation. In aggregate, the analysis indicates that reducing the complexity of software engineering experiments should be considered by researchers who want to obtain reliable and repeatable empirical measures.
    Empirical Software Engineering 12/2014; 19(6). DOI:10.1007/s10664-013-9262-z · 1.64 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: The meaning of an effort or cost estimate should be understood and communicated consistently and clearly to avoid planning and budgeting mistakes. Results from two studies, one of 42 software companies and one of 423 individual software developers, suggest that this is far from being the case. In both studies we found a large variety in what was meant by an effort estimate and that the meaning was frequently not communicated. To improve the planning and budgeting of software projects we recommend that the meaning of effort estimates is understood and communicated using a probability-based terminology.

Preview (2 Sources)

Available from