The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment

Simula Res. Lab., Univ. of Oslo, Lysaker, Norway
IEEE Transactions on Software Engineering (Impact Factor: 1.61). 11/2011; 37(5):695 - 707. DOI: 10.1109/TSE.2010.78
Source: IEEE Xplore


Studies in laboratory settings report that software development effort estimates can be strongly affected by effort-irrelevant and misleading information. To increase our knowledge about the importance of these effects in field settings, we paid 46 outsourcing companies from various countries to estimate the required effort of the same five software development projects. The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows: 1) reduced length of requirement specification with no change of content, 2) information about the low effort spent on the development of the old system to be replaced, 3) information about the client's unrealistic expectations about low cost, and 4) a restriction of a short development period with start up a few months ahead. We found that the effect sizes in the field settings were much smaller than those found for similar manipulations in laboratory settings. Our findings suggest that we should be careful about generalizing to field settings the effect sizes found in laboratory settings. While laboratory settings can be useful to demonstrate the existence of an effect and better understand it, field studies may be needed to study the size and importance of these effects.

30 Reads
  • Source
    • "It is well known that expert judgments are subject to contextual biases. For example, in software cost estimation, questioning format and irrelevant information have been shown to significantly affect the estimates [1] [2]. Due to the cognitive commonality between cost estimation and value judgments, it is reasonable to assume that assessments of value and priorities of product features are subject to similar biases as those found for cost estimation. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Context: Being able to select the essential, non-negotiable product features is a key skill for stakeholders of software projects. Such selection relies on human judgment, sometimes supported by structured prioritization techniques and associated tools. Goal: Our goal was to investigate whether certain attributes of prioritization techniques affect stakeholders' threshold for judging product features as essential. The four investigated techniques reflect four combinations of granularity (low, high) and cognitive support (low, high). Method: In one experiment, 94 subjects in four treatment groups indicated the features (from a list of 16) that would be essential in their decision to buy a new cell phone. With a similar setup in a controlled field experiment, 44 domain experts indicated the software product features that were essential for the fulfillment of the project's vision. The effects of granularity and cognitive support on the number of essential ratings were analyzed and compared between the experiments. Result: With lower granularity, significantly more features were rated as essential. The effect was large in the first experiment and extreme (Cohen's d=2.40) in the second. Added cognitive support had medium effect (Cohen's d=0.43 and 0.50), but worked in opposite directions in the two experiments, and was not statistically significant in the second. Implications: The results of the study imply that software projects should avoid taking stakeholders' judgments of essentiality at face value. Practices and tools should be designed to counteract potentially harmful biases; however, more empirical work is needed to obtain more insight into the causes of these biases.
    01/2012; DOI:10.1145/2372251.2372300
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, a literature review and classification scheme for selected software engineering researches is presented. The study shows that an increasing volume of software engineering researches have been conducted in diverse range of areas. The articles are classified and results of these are presented based on classification scheme that consist of five main categories: software development process, software management, software engineering techniques, software re-engineering, and software applications. Analyses of the selected researches are carried out and gaps in the research are identified. A comprehensive list of references is presented. This review is intended to provide impetus in research and help simulate further interest.
  • [Show abstract] [Hide abstract]
    ABSTRACT: The main determinant of many types of software-related investments is the amount of development effort required. The ability of software clients to make investment decisions based on cost estimates is consequently strongly tied to the software providers’ ability to estimate the effort accurately. Similarly, the ability of project managers to plan a project and ensure efficient development frequently depends on accurate effort estimates. The importance of accurate effort estimates is illustrated by the findings of a 2007 survey of more than 1,000 IT professionals. The survey reports that two out of the three-most-important causes of IT project failure were related to poor resource estimation, that is, inaccurate effort estimates.
    Simula Research Laboratory, 12/2009: pages 381-403;
Show more

Preview (2 Sources)

30 Reads
Available from