The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment

Simula Res. Lab., Univ. of Oslo, Lysaker, Norway
IEEE Transactions on Software Engineering (Impact Factor: 1.61). 11/2011; 37(5):695 - 707. DOI: 10.1109/TSE.2010.78
Source: IEEE Xplore


Studies in laboratory settings report that software development effort estimates can be strongly affected by effort-irrelevant and misleading information. To increase our knowledge about the importance of these effects in field settings, we paid 46 outsourcing companies from various countries to estimate the required effort of the same five software development projects. The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows: 1) reduced length of requirement specification with no change of content, 2) information about the low effort spent on the development of the old system to be replaced, 3) information about the client's unrealistic expectations about low cost, and 4) a restriction of a short development period with start up a few months ahead. We found that the effect sizes in the field settings were much smaller than those found for similar manipulations in laboratory settings. Our findings suggest that we should be careful about generalizing to field settings the effect sizes found in laboratory settings. While laboratory settings can be useful to demonstrate the existence of an effect and better understand it, field studies may be needed to study the size and importance of these effects.

Full-text preview

Available from:
  • Source
    • "Newer approaches like min-max approach [16] improve the accuracy in COCOMO model. A new concept was introduced to outline the failure of effort estimation due to misleading information [17] on account of irrelevant data. However, later results have confirmed that instance selection [18] and retrieval are automatically done to reduce the unrelated data but still a unified and integrated approach is invariably essential to avoid any perception of errors. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Rapid industrialization in the past few decades has necessitated the ever increasing demand for newer technologies leading to the dramatic development of sophisticated software for cost estimation and is expected to grow manifold in the forthcoming years. The improper understanding of software requirements has often resulted in inaccurate cost estimation. In analogy concept, there is deficiency in handling the datasets containing categorical variables though there are innumerable methods to estimate the cost. The proposed fuzzy analogy method is a new approach based on reasoning by analogy for handling both numerical and categorical variables where the uncertainty and imprecision solution is ascertained by studying the behaviour pattern of linguistic values utilized in the software projects. The performance of linguistic values in fuzzy sets has improved in the proposed method. The performance of this method analyzed using Mean Absolute Relative Error (MARE) and Variance Absolute Relative Error (VARE) criteria indicates that the fuzzy analogy outperforms other techniques in terms of both quality and accuracy of the results in software cost estimation.
    Full-text · Article · Feb 2012 · International Journal of Computer Applications
  • Source
    • "It is well known that expert judgments are subject to contextual biases. For example, in software cost estimation, questioning format and irrelevant information have been shown to significantly affect the estimates [1] [2]. Due to the cognitive commonality between cost estimation and value judgments, it is reasonable to assume that assessments of value and priorities of product features are subject to similar biases as those found for cost estimation. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Context: Being able to select the essential, non-negotiable product features is a key skill for stakeholders of software projects. Such selection relies on human judgment, sometimes supported by structured prioritization techniques and associated tools. Goal: Our goal was to investigate whether certain attributes of prioritization techniques affect stakeholders' threshold for judging product features as essential. The four investigated techniques reflect four combinations of granularity (low, high) and cognitive support (low, high). Method: In one experiment, 94 subjects in four treatment groups indicated the features (from a list of 16) that would be essential in their decision to buy a new cell phone. With a similar setup in a controlled field experiment, 44 domain experts indicated the software product features that were essential for the fulfillment of the project's vision. The effects of granularity and cognitive support on the number of essential ratings were analyzed and compared between the experiments. Result: With lower granularity, significantly more features were rated as essential. The effect was large in the first experiment and extreme (Cohen's d=2.40) in the second. Added cognitive support had medium effect (Cohen's d=0.43 and 0.50), but worked in opposite directions in the two experiments, and was not statistically significant in the second. Implications: The results of the study imply that software projects should avoid taking stakeholders' judgments of essentiality at face value. Practices and tools should be designed to counteract potentially harmful biases; however, more empirical work is needed to obtain more insight into the causes of these biases.
    Full-text · Article · Jan 2012
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, a literature review and classification scheme for selected software engineering researches is presented. The study shows that an increasing volume of software engineering researches have been conducted in diverse range of areas. The articles are classified and results of these are presented based on classification scheme that consist of five main categories: software development process, software management, software engineering techniques, software re-engineering, and software applications. Analyses of the selected researches are carried out and gaps in the research are identified. A comprehensive list of references is presented. This review is intended to provide impetus in research and help simulate further interest.
    Full-text · Article ·
Show more