Article

Jorgensen, M.: A Review of Studies on Expert Estimation of Software Development Effort. Journal of Systems and Software 70, 37-60

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

This paper provides an extensive review of studies related to expert estimation of software development effort. The main goal and contribution of the review is to support the research on expert estimation, e.g., to ease other researcher’s search for relevant expert estimation studies. In addition, we provide software practitioners with useful estimation guidelines, based on the research-based knowledge of expert estimation processes. The review results suggest that expert estimation is the most frequently applied estimation strategy for software projects, that there is no substantial evidence in favour of use of estimation models, and that there are situations where we can expect expert estimates to be more accurate than formal estimation models. The following 12 expert estimation “best practice” guidelines are evaluated through the review: (1) evaluate estimation accuracy, but avoid high evaluation pressure; (2) avoid conflicting estimation goals; (3) ask the estimators to justify and criticize their estimates; (4) avoid irrelevant and unreliable estimation information; (5) use documented data from previous development tasks; (6) find estimation experts with relevant domain background and good estimation records; (7) Estimate top-down and bottom-up, independently of each other; (8) use estimation checklists; (9) combine estimates from different experts and estimation strategies; (10) assess the uncertainty of the estimate; (11) provide feedback on estimation accuracy and development task relations; and, (12) provide estimation training opportunities. We found supporting evidence for all 12 estimation principles, and provide suggestions on how to implement them in software organizations.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... • Jørgensen [13] reviewed 15 studies comparing model-based to expert-based estimation. Five of those studies found in favour of expert-based methods, five found no difference, and five found in favour of model-based estimation. ...
... The motivation for this paper has been the difficulties in forming a consistent picture of the relative performance of competing software engineering prediction systems. Not only does no one technique dominate, but also different researchers' validation studies have often produced inconsistent results as highlighted by systematic reviews such as [13,25,21]. Until researchers gain a better understanding of the underlying reasons for this state of affairs, it is unclear that devising new prediction systems and conducting more and more primary validation studies will be particularly illuminating. ...
... For example, Myrtveit and Stensrud [31] found that a combination of expert and formal prediction system led to the most accurate predictions. Jørgensen [13] has consistently argued that there is need to understand the human element of software engineering predictions in practice. Mair and Shepperd [26] suggest that in order to unlock real improvement in predictive practice (i.e. ...
Preprint
Context: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random 'predictions', that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.
... S OFTWARE development effort estimation is a crucial activity for project planning and monitoring, specifically for ensuring that the product is delivered on time and within budget [1], [2]. Studies have shown that engineers make inaccurate effort estimations [3], [4], [5], [6], which can negatively affect the outcome of software projects leading to great losses [2], [7]. ...
... Moreover, all datasets contain two more features: the hours to complete a software project as estimated by a human expert (i.e. Expert Estimated Effort) and the number of hours actually required to complete it as recorded at the end 4. FSM methods have obtained world-wide acceptance [1] and allow software size measurement in terms of the functionality with which users are provided. ...
... Surveys on estimation practice in the software industry found that human effort estimates are over-optimistic [3], [4] and there is a strong over-confidence in their accuracy [5]. A recent survey on agile practice also revealed that half of the respondents believe that their effort estimates on average are under/over estimated by an error of 25% or more [6]. ...
Article
In this paper, we introduce a novel approach to predictive modeling for software engineering, named Learning From Mistakes (LFM). The core idea underlying our proposal is to automatically learn from past estimation errors made by human experts, in order to predict the characteristics of their future misestimates, therefore resulting in improved future estimates. We show the feasibility of LFM by investigating whether it is possible to predict the type, severity and magnitude of errors made by human experts when estimating the development effort of software projects, and whether it is possible to use these predictions to enhance future estimations. To this end we conduct a thorough empirical study investigating 402 maintenance and new development industrial software projects. The results of our study reveal that the type, severity and magnitude of errors are all, indeed, predictable. Moreover, we find that by exploiting these predictions, we can obtain significantly better estimates than those provided by random guessing, human experts and traditional machine learners in 31 out of the 36 cases considered (86%), with large and very large effect sizes in the majority of these cases (81%). This empirical evidence opens the door to the development of techniques that use the power of machine learning, coupled with the observation that human errors are predictable, to support engineers in estimation tasks rather than replacing them with machine-provided estimates.
... Besides expert judgment, algorithmic model estimation is another predominant method discussed in the literature. Jørgensen [14] did an extensive review of 15 accuracy comparison studies between expert judgment and formal models of software development effort. The results indicate that there is no substantial evidence in preference the use of formal estimation models. ...
... The expert judgment technique is generally applied in forecasting domains. In the "software estimation" context, several definitions can be found [14,[19][20][21][22]. In all, expert judgment can generally be seen as the solicitation of estimates from the experts in the domains. ...
... He concluded that "there is no evidence to support that the Delphi is more accurate than other judgment methods". However, it is reported in [14] that a group-based process led to the highest accuracy in the area of human judgment and forecasting. Moløkken-Østvold et al., [10] have not found the support for the hypothesis --"Group estimates were more accurate than statistical groups". ...
Article
Literature review shows that more accurate software effort and cost estimation methods are needed for software project management success. Expert judgment and algorithmic model estimation are two predominant methods discussed in the literature. Both are reported almost at the comparable level of accuracy performance. The combination of the two methods is suggested to increase the estimation accuracy. Delphi method is an encouraging structured expert judgment method for software effort group estimation but surprisingly little was reported in the literature. The objective of this study is to test if the Delphi estimates will be more accurate if the participants in the Delphi process are exposed to the algorithmic estimates. A Delphi experiment where the participants in the Delphi process were exposed to three algorithmic estimates –Function Points, COCOMO estimates, and Use Case Points, was therefore conducted. The findings show that the Delphi estimates are slightly more accurate than the statistical combination of individual expert estimates, but they are not statistically significant. However, the Delphi estimates are statistically significant more accurate than the individual estimates. The results also show that the Delphi estimates are slightly less optimistic than the statistical combination of individual expert estimates but they are not statistically significant either. The adapted Delphi experiment shows a promising technique for improving the software cost estimation accuracy.
... The overall perceived success of a software project depends heavily on meeting the time and cost estimates [3]. Improving effort estimation is therefore a critical goal for software organizations: it can help companies reduce delays and improve customer satisfaction, while enabling them to efficiently allocate resources, reduce costs and optimize delivery [4], [5]. In spite of the availability of many estimation methods and guidelines [6], [7], on-time delivery in software development remains a major challenge. ...
... Prior work analyzed a variety of factors influencing the software development effort (so-called effort drivers). Trendowicz et al. [20] reviewed and divided the most commonly used effort drivers into four categories: personnel factors (i.e., team capabilities and experience [5], [9], [21], [22]), process factors (i.e., quality of methods, tools and technologies applied [23], [24], [25]), project factors (i.e., project resources and management activities [26], [27]) and product factors (i.e., effort for requirements analysis, design and coding [28]). Personnel factors and project factors are the top mentioned effort drivers in agile projects [12]. ...
... As observed in previous studies [20], [29], the accuracy of effort estimation methods depends on the selection of relevant factors and the elimination of irrelevant and misleading factors. Existing estimation methods can generally be classified into expert-based and model-based approaches [5], [30]. Expert-based methods rely on human expertise to select relevant factors, and are the most popular technique in both agile and traditional (waterfall-like) projects [5], [31]. ...
Article
Full-text available
Late delivery of software projects and cost overruns have been common problems in the software industry for decades. Both problems are manifestations of deficiencies in effort estimation during project planning. With software projects being complex socio-technical systems, a large pool of factors can affect effort estimation and on-time delivery. To identify the most relevant factors and their interactions affecting schedule deviations in large-scale agile software development, we conducted a mixed-methods case study at ING: two rounds of surveys revealed a multitude of organizational, people, process, project and technical factors which were then quantified and statistically modeled using software repository data from 185 teams. We find that factors such as requirements refinement, task dependencies, organizational alignment and organizational politics are perceived to have the greatest impact on on-time delivery, whereas proxy measures such as project size, number of dependencies, historical delivery performance and team familiarity can help explain a large degree of schedule deviations. We also discover hierarchical interactions among factors: organizational factors are perceived to interact with people factors, which in turn impact technical factors. We compose our findings in the form of a conceptual framework representing influential factors and their relationships to on-time delivery. Our results can help practitioners identify and manage delay risks in agile settings, can inform the design of automated tools to predict schedule overruns and can contribute towards the development of a relational theory of software project management.
... Story Point (SP) is commonly used to measure the effort needed to implement a user story [2], [4] and agile teams mainly rely on expert-based estimation [1], [5]. However, similar to traditional software project effort estimation [6], [7], task-level effort estimation is not immune to the expert's subjective assessment [4]. Subjective assessment may not only lead to inaccurate estimations but also, and more importantly to an agile team, it may introduce inconsistency in estimates throughout different sprints. ...
... Surprisingly, for 11 out of 16 experiments, more than 97% of the issues used to train the models in RQ3.1 were created after the start date of the target project, thus making the use of Deep-SE in practice infeasible. 6 Therefore, we extend the original analysis in order to assess Deep-SE's performance in a realistic scenario, which takes into account the chronological order of the data used for cross-project estimation. This motivates, our second sub-question: RQ3.2. ...
... In particular, we include in the training set the issues from other projects (belonging to the same repository as the target project) that are created before the earliest created issue in the validation set. Therefore, we augment the target project training data with every issue that a company already had in their repository 6. Among the five remaining projects, for two of them, 75% and 37% of the issues were created after the start date of the target projects, and for three (in which MULESOFT was used as the source or target project) we could not determine the percentage as the start/end time of the MULESOFT project is not known as this project repository is no longer accessible. ...
Preprint
In the last decade, several studies have proposed the use of automated techniques to estimate the effort of agile software development. In this paper we perform a close replication and extension of a seminal work proposing the use of Deep Learning for agile effort estimation (namely Deep-SE), which has set the state-of-the-art since. Specifically, we replicate three of the original research questions aiming at investigating the effectiveness of Deep-SE for both within-project and cross-project effort estimation. We benchmark Deep-SE against three baseline techniques (i.e., Random, Mean and Median effort prediction) and a previously proposed method to estimate agile software project development effort (dubbed TF/IDF-SE), as done in the original study. To this end, we use both the data from the original study and a new larger dataset of 31,960 issues, which we mined from 29 open-source projects. Using more data allows us to strengthen our confidence in the results and further mitigate the threat to the external validity of the study. We also extend the original study by investigating two additional research questions. One evaluates the accuracy of Deep-SE when the training set is augmented with issues from all other projects available in the repository at the time of estimation, and the other examines whether an expensive pre-training step used by the original Deep-SE, has any beneficial effect on its accuracy and convergence speed. The results of our replication show that Deep-SE outperforms the Median baseline estimator and TF/IDF-SE in only very few cases with statistical significance (8/42 and 9/32 cases, respectively), thus confounding previous findings on the efficacy of Deep-SE. The two additional RQs revealed that neither augmenting the training set nor pre-training Deep-SE play a role in improving its accuracy and convergence speed. ...
... As presented in (Boehm et al., 2000), some are based on previous projects analysis, other are based on expert judgment. Work as (Jørgensen, 2004) shows that expert judgements are relevant when processes are supporting the evaluation to reduce the human biases. In (Jørgensen, 2004), 12 principles to support the cost estimation process are proposed based on empirical evidence. ...
... Work as (Jørgensen, 2004) shows that expert judgements are relevant when processes are supporting the evaluation to reduce the human biases. In (Jørgensen, 2004), 12 principles to support the cost estimation process are proposed based on empirical evidence. ...
Article
Model-based systems engineering (MBSE) has been proposed as an approach to manage the complexity of modern product development through the continuous use of models. The use of model simulation is a core principle of the MBSE approach. In the early stages of projects, it for example supports defining the expected system features, when in the later phases it can be used to estimate the dynamic behavior. Simulation is pushed to obtain results earlier and cheaper than with testing and prototyping. However, the development of simulation can be a very tedious and expensive task. Simulation opportunities are numerous, but the project managers must identify the more relevant for their project. This paper aims at documenting the current state of practice on the usage of simulation in MBSE processes. Then it aims at exploring decision support opportunities for simulation use in MBSE projects. The paper presents a survey conducted amongst French companies, on how they apply MBSE, Verification Validation & Testing (VVT), and simulation. The perceived benefits, barriers, and the parameters influencing VVT strategies, and the use of simulation are alternately analyzed. The results of the survey are used to propose a Prioritization of Simulation Efforts Methodology (PSEM) to assists managers in choosing the right functions, or systems' elements to be simulated.
... Meanwhile, estimating the most realistic amount of effort in the early stage of software development is difficult since the information available at that stage is usually incomplete and uncertain. Although construction of formal software effort estimation models started in the very early times of the industrialization of software production, expert judgement still remains the dominant strategy for effort prediction in practice where the accuracy of the estimate is sensitive to the practitioner's expertise and thus prone to bias [53], [87]. Early work to build an estimation technique tried to find a set of factors related to the software size and cost by using regression analysis [9]. ...
... To answer RQ4, we compare the performance of CoGEE (and other state-of-the-art estimators) against claims made for best human-expert-based results achievable in the industry [53], [75], as done in the original study. In particular, we investigate the magnitude of relative error (compared to claimed industrial best practice). ...
Article
Full-text available
Replication studies increase our confidence in previous results when the findings are similar each time, and help mature our knowledge by addressing both internal and external validity aspects. However, these studies are still rare in certain software engineering fields. In this paper, we replicate and extend a previous study, which denotes the current state-of-the-art for multi-objective software effort estimation, namely CoGEE. We investigate the original research questions with an independent implementation and the inclusion of a more robust baseline (LP4EE), carried out by the first author, who was not involved in the original study. Through this replication, we strengthen both the internal and external validity of the original study. We also answer two new research questions investigating the effectiveness of CoGEE by using four additional evolutionary algorithms (i.e., IBEA, MOCell, NSGA-III, SPEA2) and a well-known Java framework for evolutionary computation, namely JMetal (rather than the previously used R software), which allows us to strengthen the external validity of the original study. The results of our replication confirm that: (1) CoGEE outperforms both baseline and state-of-the-art benchmarks statistically significantly (p < 0.001); (2) CoGEEs multi-objective nature makes it able to reach such a good performance; (3) CoGEEs estimation errors lie within claimed industrial human-expert-based thresholds. Moreover, our new results show that the effectiveness of CoGEE is generally not limited to nor dependent on the choice of the multi-objective algorithm. Using CoGEE with either NSGA-II, NSGA-III, or MOCell produces human competitive results in less than a minute. The Java version of CoGEE has decreased the running time by over 99.8% with respect to its R counterpart. We have made publicly available the Java code of CoGEE to ease its adoption, as well as, the data used in this study in order to allow for future replication and extension of our work.
... Note that the most likely cost estimates for epics are those in Fig. 3.16. 1 For example, for epic E3, the most likely estimate is 1.8 million (corresponding to Fig. 3.16). This epic also has a bad-case estimate of four and a good-case estimate of one. ...
... It is more important that your interval is not too narrow. According to evidence [1], you should fix the low and high values first and then assess the probability of staying within these bounds, rather than fix a probability first and then find an interval in which there is that probability of staying within the interval. Research is ongoing on how best to elicit people's perceptions of uncertainty. ...
Chapter
Full-text available
Agile methodology purports to deal with uncertainty through continuous monitoring and learning. To do so, we need to see how productivity is faring against our plans, as in the previous chapter. But we also need to communicate what our uncertainty is realistically. This is regularly done for cost, but must also be done for benefit to obtain a complete picture. In this chapter, we show how both benefit points and size points can be instantiated with values reflecting different levels of uncertainty.
... Ahmed et al. [16] also identified the fact that many organizations face serious problems because project managers are unable to understand the project's requirements and the skills required to accomplish the project. Their work provides a theoretical basis of the skill required in different phases of software development [16,32]. The theoretical basis is not adequate and requires skill management system. ...
... The human resource department also uses this system to match the skills of employees for certain tasks and then assign tasks accordingly. Moreover, it saves time and costs of the overall project [16,32]. ...
Article
Full-text available
Skills Management is an essential concept of human resource management in which a skill inventory may be created for each employee and managers can assign tasks to workers based on worker’s abilities. This concept is not fully practiced for two reasons: i) employee’s skills are not effectively evaluated and documented, ii) tool support is deficient to manage this complex task. Ineffective skill management of an organization fizzle tasks assigned to the incompetent employees and this may lead to project failure. To fill up this gap, a survey is conducted across various software organizations to find out the best practices for the skill management and to gather requirements for skills management framework. Based on survey findings, a mathematical framework is proposed that calculates the soft and hard skills of employees automatically based on time and achievements as skill increases or decreases over time. In this framework, the Skills Calculation Engine (SCE) is developed for the managers to enhance the capacity of appropriate decisions making in assigning tasks to the rightly skilled workers. This framework is also useful for organizations as it can increase profitability as tasks are assigned to the most appropriate employees. The SCE is implemented as a Windows-based application to calculate skills, store skills in skills inventory, and assign tasks based on an employee’s skills. The skills management tool is evaluated in a facilitated workshop; furthermore, a feature-wise comparison of the tool is also made with existing tools.
... Expert judgement is another well-known approach for estimation (Jorgensen, 2004;Helmer, 1966;Baird, 1989). This approach captures knowledge, experiences, and expertise of practitioners who are recognized as experts within a domain of interest, and derives estimates based on historical data that they are well aware of, or past projects that they participated. ...
... The overhead cost of the migration tasks can be achieved by comparing the time spent on each migration task category with the development time of the application. The application was not developed by us; hence, the development time can be estimated using an effort estimation approach in the literature (either analogy, expert judgement, or algorithmic models (Shepperd & Schofield, 1997;Jorgensen, 2004;Finnie et al., 1997)). ...
Thesis
Full-text available
Cloud computing has been a buzz word over the last decade - it offers great potential benefits for enterprises who migrate their computing systems from local data centers to a Cloud environment. One major obstacle to enterprise adoption of Cloud technologies has been the lack of visibility into migration effort and cost. Currently, there is very limited existing work in the literature. This thesis improves our understanding of this matter by identifying critical indicators of Cloud migration effort. A taxonomy of migration tasks to the Cloud has been proposed, outlining possible migration tasks that any migration project to the Cloud may encounter. It enables Cloud practitioners to gain an understanding of the specific tasks involved and its implication on the amount of effort required. A methodology, called Cloud Migration Point (CMP), is presented for estimating the size of Cloud migration projects, by recasting a well-known software size estimation model, Function Point, into the context of Cloud migration. The CMP value implies how large the migration project is, and it can be used as an indicator for Cloud migration effort estimation. The process of calculating CMP also assists one in itemizing the migration tasks, and identifying the complexity of each task. This is useful for project planning and management. The empirical validation on the set of data points collected from our survey shows that, with some calibrations, the CMP metric is practically useful as a predictor for effort estimation under a defined set of assumptions. Besides size measurement, other factors also influence the migration effort. We propose a list of external cost factors, which do not affect how migration tasks are designed, but may affect how fast migration tasks can be done, such as development team's experience in software engineering, or experience with the Cloud. Our overall contribution is to shed light into Cloud migration and the tasks involved, which enables Cloud practitioners to estimate the amount of effort required for the migration of legacy systems into the Cloud. This contributes towards the cost-benefit analysis of whether the benefits of the Cloud exceed the migration effort and other Cloud costs.
... Jørgensen [12] presented a review of expert estimation model. They suggested that there are situations when expert estimation is more precise than the former estimation models. ...
... The magnitude of relative error is used for finding the relative error between actual effort and estimated effort for each project of dataset as given in (12). ...
Article
The software industry is highly competitive, and hence, it is imperative to have an accurate method to estimate the effort needed in the key phases of software development. Accurate estimates ensure efficient allocation of human and machine resources for the project. This paper proposes a technique for software development effort estimation using deep belief network (DBN). For fine-tuning of DBN, Whale Optimization Algorithm (WOA) is used which mimics the social behaviour of humpback whales. The proposed technique DBN-WOA has been experimentally evaluated on four promise datasets—COCOMO81, NASA93, MAXWELL and CHINA. The results from DBN-WOA are compared with the results from fine-tuning of DBN with backpropagation (DBN-BP) and it is observed that the proposed technique outscores DBN-BP. The proposed approach is also empirically validated through a statistical framework.
... According to Jørgensen [14], the expert estimation accuracy is better than model-based estimation because the information available with humans can be used flexibly than model-based. The expert judgment can become an ad-hoc activity without using explicit support structure. ...
... The expert judgment can become an ad-hoc activity without using explicit support structure. The experts may forget relevant activities and tasks which are the key reasons for underestimating the development effort [14]. According to Furulund and Molkken-stvold [15], the use of checklists improves effort estimation accuracy. ...
Conference Paper
Full-text available
Software effort estimation is an essential feature of software engineering for effective planning, controlling and delivering successful software projects. The overestimation and underestimation both are the key challenges for future software development. The failure to acknowledge the effort estimation accuracy may lead to customer disappointment, inaccurate estimation and hence, contribute to either poor software development process or project failure. The main aim of this research is to optimize the estimation accuracy prediction of software development effort to support software development firms and practitioners. In this paper, we propose an ensemble software effort estimation model based on Use Case Points (UCP), expert judgment and Case-Based Reasoning (CBR) techniques. This research is conducted through primary (a multi-case involving software companies) study to make an ensemble model. The estimation accuracy prediction of the proposed model will be evaluated by selecting projects from primary studies as case selections in applying a quantitative approach through industrial experts, archival data about estimates and evaluation metrics. The proposed model produced at the end of this research will be used by software development firms and practitioners as an instrument to estimate the effort required to develop new software projects at an earlier stage.
... However, despite the vast number of studies on agile software effort estimation, there is a lack of empirical evidence on the improvement of estimation accuracy over time in agile development. Studies on effort estimation of software development, on the other hand, report that experts' previous experience in estimation does not lead to better judgment on similar tasks [4], [18], [35], [36]. As most of these studies are based on lab experiments with a small number of estimation tasks, there is a call for research to collect more data [19] to investigate estimation-learning processes. ...
... Jørgensen and Gruschke [4] argue that humans' capabilities for improving their own expert judgment are poor. One study found that experts' ability to learn from prior estimation experience is ''disappointingly low'' in both software maintenance and development [36]. Jørgensen and Sjøberg [37] conducted a study of 54 software professionals' work on extending and maintaining large administrative applications, finding that the amount of application-specific experience from a previous estimation task does not lead to more accurate assessments of a similar task. ...
Article
Full-text available
Effort estimation is an important practice in agile software development. The agile community believes that developers’ estimates get more accurate over time due to the cumulative effect of learning from short and frequent feedback. However, there is no empirical evidence of an improvement in estimation accuracy over time, nor have prior studies examined effort estimation in different development activities, which are associated with substantial costs. This study fills the knowledge gap in the field of software estimation in agile software development by investigating estimations across time and different development activities based on data collected from a large agile project. This study investigated effort estimation in various development activities, including feature development, bug fixing, and refactoring in agile software development. The results indicate that estimation of agile development does not improve over time, as claimed in the literature. Our data also indicate that no difference exists in the magnitude of estimation errors between feature tasks and bug-fixing/refactoring tasks, while bug-fixing and refactoring tasks are overestimated more frequently than feature tasks. This study also contributes to our knowledge about overestimation and underestimation patterns in agile software development.
... [380, 596,47,271,359,403,587,32,302,444,493,536,667,671,7,178,222,532,636,5,274,402,401,452,451,633,728,736,515,26,51,65,105,183,333,340,361,434,460,571,582,709,13,531] Named entity recognition [286,357,287, 665] IE [779] Information extraction -Relation extraction [60,220,291] Open information extraction 11 [280,283] A. 1 ...
... IoT [50, 60, 71, 131, 170, 38, 47, 180, 191, 237, 298, 22, 26, 29, 55, 74, 73, 120, 157, 238, 250, 278, 4, 16, 19, 30, 77, 92, 142, 147, 153, 159, 162, 168, 183, 201, 217, 251, 261, 264, 283, 289, 295, 5, 48, 87, 95, 106, 109, 112, 114, 126, 135, 138, 149, 143, 145, 150, 155, 160, 175, 176, 184, 198, 223, 230, 234, 239, 249, 263, 275, 276, 282, 284, 288, 294, 303, 305, 306, 3, 13, 10, 24, 25, 28, 43, 45, 83, 84, 89, 107, 163, 195, 197, ?, 210, 214, 235, 245, 247, 265, 266, 274, 279, 281, 293, 300, 308, 76, 94, 103, 116, 158] IoE 42 [144,242] e-commerce [272,246,7,79,204,218,61,88,132,141,82,301,1,70,93,115,133,219,292,33,62,68,97,113,195,212,216,229,235] Supply chain [54,125,36,64,110,174,193,235,296] e-business [302,310,220] e-commerce -electronic services -eservice [271] e-government 43 [17,232,256,50,123,286,8,122,186,189,177,190,257,124,178,269,105,166,203,206,221,254,40,41,81,99,252,267,270] e-logistics 44 [227] Other internet [58,181,192,209 [532,422,534,533,594,648,1102,25,140,1085,764,172,206,1050,342,488,344,415,490,621,1090,21,192,784,947,968,824,1086,92,554,70,71,97,555,746,904,1064,83,338,370,659,747,47,141,195,416,567,568,87] Software project -Software development [532, 534, 533, 578, 579, 707, 1039, 341, 595, 20, 543, 611, 688, 876, 925, 42, 153, [528,838,337,529,579,269,365,837,270,323,537,677,200,252,676,1002,206,234,283,131,628,729,618] Global software engineering [514,244,603,642] Virtual software development 49 [945,1115] 47 The concept does not exists in the CSO. 48 The concept does not exists in the CSO. ...
... In-view of the vitality of an accurate software cost estimation process, a big volume of software cost estimation techniques from last so many decades have been proposed. All Human-based techniques [28] are the ones that are hard to review. Furthermore, human-based techniques are most often biased as being prone to human judgements as humans hardly update their estimation knowledge [29]. ...
... In this study, experimentation results of many forms were obtained as given in various tables and are providing answers to many questions. that the best configuration is not only data set dependant but also training set of a particular data set too. tests [28]. Furthermore, the lesser the rank value, the better is the accuracy. ...
Preprint
Software cost estimation is the prediction of development effort and calendar time required to develop any software project. It is considered to be the very fundamental task for successful execution of an ongoing project as well as budgetary requirements of futuristic projects. As accuracy in software cost estimation is very hard because of the availability of vague information at the time of inception of software project, it prompted many researchers to explore in this domain from last so many decades. Their pioneer works suggest a bulk of techniques for this purpose. However, because of the availability of large number of estimation techniques, it becomes hard for any software practioners to select an appropriate one. To help the industry practitioners in these situations, a novel analogy centred model based on differential evolution exploration process is proposed in this research study. The proposed model has been assessed on 676 projects from 05 different datasets and the results achieved are significantly better when compared with other bench mark analogy based estimation studies. Furthermore, being the very less computational cost of the proposed model, it is suggested that the proposed model be considered as the preliminary stage of any analogy based software estimation technique.
... In part, this may well be because it is much more tractable to teach about using models than about using experience when teaching students about software engineering, and so greater emphasis has been placed upon the former. Anyway, whatever the reason, this belief is clearly challenged by the findings of Jørgensen (2004), who, from a set of 15 primary studies comparing models with expert judgement, found that: ...
... For quantitative systematic reviews, completeness is crucial. If we look at the examples described in chapter 3, which relate to methods of estimating software development effort (Jørgensen 2004), pair programming versus solo programming ) and perspective-based reading (PBR) compared to other approaches to reading (Basili et al. 1996) we see that 15 studies are included in the cost estimation review, 18 in the pair programming review and 21 in the PBR review (with many of the primary studies in the last of these considered to be replications rather than independent studies). Given the highly focused nature of these reviews and the small numbers of included studies, missing only a few of these could substantially affect the outcomes of the reviews. ...
... La estimación del esfuerzo de desarrollo de software ( ) intenta predecir el trabajo (en términos de horas-persona o dinero) necesario para desarrollar un software de calidad [162]. En particular, en esta investigación, se utiliza el esfuerzo de desarrollo de software para determinar cuál es el trabajo necesario para producir un producto P derivado de IRArduino-SPL. ...
... Se define como la cantidad de trabajo necesario para desarrollar o mantener un software, normalmente expresado en términos de horas-persona o dinero[162]. ...
Thesis
Full-text available
En la actualidad, los sistemas robóticos industriales han tomado gran importancia en la sociedad, debido a que se utilizan en muchos dominios como la robótica de servicios, la industria de manufactura y las ciencias de la salud, entre otros. Sin embargo, hay un aumento en la complejidad del software requerido por parte de estos sistemas electromecánicos. Como respuesta, las universidades han diseñado programas de formación relacionados con esta área de conocimiento. En particular, la construcción de robots en el ámbito académico se ha centrado en la realización de prototipos, que permiten a los estudiantes comprender el dominio y sus principales bases teóricas y prácticas. Estos prototipos suelen utilizar microcontroladores (Arduino o Raspberry Pi) para dotar de inteligencia a los dispositivos electrónicos, lo que permite emular el desarrollo de software en la industria y cómo influye en el hardware subyacente. Aunque se han realizado esfuerzos para incorporar metodologías de reutilización de software en el dominio de Arduino, no se reportan muchas investigaciones que lo hagan en robots industriales que utilicen estos microcontroladores. Por lo tanto, se hace evidente la necesidad de fomentar y aplicar enfoques de reutilización que mejoren el desarrollo de software para robots industriales con Arduino, de manera que los desarrolladores (estudiantes) puedan beneficiarse de la reutilización planificada, además, de entender y familiarizarse con estos enfoques de ingeniería de software desde su formación académica, permitiendo así que la reutilización en la industria sea más factible en estos dispositivos cuando sea necesaria. Para resolver el problema planteado, se propuso como solución una línea de productos de software (IRArduino-SPL) enfocada en robots industriales con Arduino, desarrollada a través de dos iteraciones dentro de este enfoque de reutilización, la primera para observar la viabilidad de la propuesta en el dominio y la segunda para refinar la línea en base a la experiencia adquirida e incrementar el nivel de abstracción. Posteriormente, IRArduino-SPL demostró su viabilidad en el dominio mediante una prueba de concepto y su utilidad en términos de reutilización a través de un estudio de caso, logrando reutilizar aproximadamente entre el 38 y el 41% del total de las líneas de código necesarias para el funcionamiento de un robot industrial con Arduino.
... Several studies have carried out vast review of the promising work done by researchers in the past [13], [14], [15]. As per these review studies, some researchers assume expert based SDEE methods to be better than the model based SDEE, some favour model based estimation while some found no difference in them [16]. As per a comparative study done in [17], nine studies found analogy-based SDEE model outperforming regression based SDEE, while four studies found regression based SDEE model to be the best performing. ...
Article
Full-text available
The software engineering researchers have worked on different dimensions to facilitate better software effort estimates, including those focusing on dataset quality improvement. In this research, we specially investigated the effectiveness of outlier removal to improve estimation performance of 5 machine learning (ML) methods (Support Vector Regression, Random Forest, Ridge Regression, K-Nearest Neighbor, and Gradient Boosting Machines) for software development effort estimation (SDEE). We propose a novel discretization method based on Golden Section (dubbed as Golden Section based Adaptive Discretization, GSAD) to identify optimal number of outliers for SDEE dataset. The results signify the importance of optimal number of outliers’ removal to improve estimations. Moreover, the results obtained after applying GSAD technique have been compared with IQR and Cooks’ distance based outlier identification methods over 4 datasets: ISBSG Release 2021, UCP, NASA93 and China. The empirical results confirm that the performance of ML based SDEE methods is generally improving by employing GSAD and the proposed GSAD method has the ability to compete with the other prevalent outlier identification methods.
... Delphi method can be considered as the refinement of expert's judgment [5]. It deals with the standardization of the expert's judgement model. ...
Article
Cost estimation of software projects is risky task in project management field. It is a process of predicting the cost and effort required to develop a software applications. Several cost estimation models have been proposed over the last thirty to forty years. Many software companies track and analyse the current project by measuring the planed cost and estimate the accuracy. If the estimation is not proper then it leads to the failure of the project. One of the challenging tasks in project management is how to evaluate the different cost estimation and selecting the proper model for the current project. This paper summarizes the different cost estimation model and its techniques. It also provides the proper model selection for the different types of the projects.
... In addition, PES success factors have been studied, but no paper actually talks about customization in PES. For example, Alves et al. [21] focus on requirement engineering in the PES development, Sheppard [22] studies PES development costs, and Jørgensen [23] reviews the experts' estimation on the development efforts. Other papers, not being systematic literature reviews, evaluate the success and failure of PES implementation [24], or are case studies on ERP implementation cases [25]. ...
... Another common technique for predicting effort is expert estimation, which is suitable when the domain knowledge is not leveraged by the models [18]. Despite its popularity, expert systems exhibit considerable human bias. ...
Chapter
Full-text available
Machine Learning (ML) has become a ubiquitous tool for predicting and classifying data and has found application in several problem domains, including Software Development (SD). This paper reviews the literature between 2000 and 2019 on the use the learning models that have been employed for programming effort estimation, predicting risks and identifying and detecting defects. This work is meant to serve as a starting point for practitioners willing to add ML to their software development toolbox. It categorises recent literature and identifies trends and limitations. The survey shows as some authors have agreed that industrial applications of ML for SD have not been as popular as the reported results would suggest. The conducted investigation shows that, despite having promising findings for a variety of SD tasks, most of the studies yield vague results, in part due to the lack of comprehensive datasets in this problem domain. The paper ends with concluding remarks and suggestions for future research.
... Faced with the need for a software solution (e.g., a new client database) this manager can either purchase an existing solution for a known cost or employ a team of developers to build something new for an uncertain cost. Estimating software development cost is a commonly studied forecasting problem that in practice is frequently accomplished with subjective "gut" judgements by managers (Jørgensen 2004). The decision itself comes down to a simple determination of which one is more expensive, an example of a more general class of decisions made by comparing numeric estimates to a benchmark threshold. ...
Preprint
Full-text available
Decades of research suggest that information exchange in groups and organizations can reliably improve judgment accuracy in tasks such as financial forecasting, market research, and medical decision-making. However, we show that improving the accuracy of numeric estimates does not necessarily improve the accuracy of decisions. For binary choice judgments, also known as classification tasks--e.g. yes/no or build/buy decisions--social influence is most likely to grow the majority vote share, regardless of the accuracy of that opinion. As a result, initially inaccurate groups become increasingly inaccurate after information exchange even as they signal stronger support. We term this dynamic the "crowd classification problem." Using both a novel dataset as well as a reanalysis of three previous datasets, we study this process in two types of information exchange: (1) when people share votes only, and (2) when people form and exchange numeric estimates prior to voting. Surprisingly, when people exchange numeric estimates prior to voting, the binary choice vote can become less accurate even as the average numeric estimate becomes more accurate. Our findings recommend against voting as a form of decision-making when groups are optimizing for accuracy. For those cases where voting is required, we discuss strategies for managing communication to avoid the crowd classification problem. We close with a discussion of how our results contribute to a broader contingency theory of collective intelligence.
... It is usually used as a basis for bidding for a contract for the company or developer team and for resources to be allocated and plays an important role in project and decision making. Software prediction methods cover a wide range of methods that are used to predict effort, time and cost of the program [10]. ...
Article
Full-text available
Software project cost prediction is a very important task during building and developing software projects. This process helps software project engineers to accurately manage and plan their resources in terms of cost estimation. However, Need for accurate cost development prediction model for a software project is not a simple procedure. Predicting the cost required while developing software engineering projects is the most difficult challenge that attracts the attention of researchers and practitioners. This paper adopts a new model in estimating the cost of building or developing software engineering projects using a machine learning approach. The results proves that machine learning methods can be used to predict program cost with high accuracy rate compared with traditional software estimation techniques. The proposed model in this research was trained on the NASA (National Aeronautics and Space Administration) data set, which contains the characteristics of 60 projects in addition to the real cost of the projects. An analysis of the results of the implementation for the proposed methods showed that the cost Predicting process using K-Nearest Neighbours algorithm (KNN), Cascade Neural Networks (CNN) and Elman Neural Networks (ENN) It has the ability to predict the costs required to build or develop software engineering projects, K-Nearest Neighbours algorithm has shown high accuracy for Predict the required cost to develop Software Engineering projects Compared to Cascade Neural Networks and Elman Neural Networks ENN.
... Decisions are based on pervious projects. [5]. ...
Conference Paper
Full-text available
... They are based on the estimation of different variables [1] that, in the case of fuzzy techniques, they are linguistic quantifiers for many or most of their variables [2][3] [4]. In many cases the experts' opinion is essential [5]. Regarding risk analysis, the experts' contribution is crucial, especially on the risk hierarchy definition, and the probabilities of occurrence. ...
Preprint
Full-text available
Protocol for a Systematic Mapping of the Literature, which aims to identify and classify the estimations techniques used in software development agile methodologies based on the results found, and to compare their estimation accuracies against those obtained in traditional software development methodologies.
... Numerous related works [6] [24] [25] show that the expert estimation is the dominant method used for software development effort estimation. The first estimation of software effort in the 1960s relied on expert judgment [26]. ...
Article
Full-text available
Abstract The estimation of software development efforts has become a crucial activity in software project management. Due to this significance, a few models have been proposed so far to build a connection between the required efforts to be employed, and the software size, time schedule, budget and similar requirements. However, various holes and slips can still be noticed in software effort’s estimation processes due to the lack of enough data available in the initial stage of project’s lifecycle. In order to improve the accuracy of time estimation in the software industry, this work used NASA projects dataset to train and validate the proposed model, which is based on Feedforward Artificial Neural Network. Moreover, Dragonfly Algorithm was used to provide optimal training, which in consequence offered more enhanced and accurate software estimation model. Randomly selected project datasets were used to test the proposed model, which resulted in clear enhanced results in comparison to similar estimation models. Different performance criteria were used to validate and accept the hypothesis suggested by this paper that the proposed model could be used in predicting the efforts required for various types of software projects.
... After conducting several field studies, although not particularly in SSC, Jorgensen [10,14] recommends to combine the use of expert judgment with other simple techniques like analogy [11,12] or checklist-based [14]. However, the suitability of these advises is still uncertain when used by SSC. ...
Article
The software engineering research recognizes that small software companies are different to medium-sized and large organizations. Therefore, they require particular implementations of already known software practices to support their developments; this includes the estimation of the software development effort. The literature reports that estimation techniques based on expert judgement are preferred by the software industry in general, however, it is not clear what practices are used by small software companies and why they use them. Trying to clarify these aspects, particularly for small software companies in Chile, this article presents an empirical study based on semi-structured interviews that explores the effort estimation practices used by ten business stable organizations. The study results show the practices preferred by them, the reasons behind their preferences, and the expected accuracy of their estimates. The study also shows that using an appropriate estimation technique is not enough to keep the estimates under control.
... Software size estimate represents the set of deliverables that should be implemented, it also consider calibration size adjustment and the other software that will be integrated with the new system. Software size can be driven from several techniques; the main two methods are: 1) expert judgment/analogy, in which an expert gives an estimate for the software based on his previous knowledge and experience [42][43]; 2) functional analysis which is based on requirements specifications and it represents the number of functionalities with several techniques IFPUG [44], NESMA [45] and COSMIC FFP [46]. ...
... • For many years, it was an article of faith among SE researchers and tool vendors that models were better than humans at estimating the effort required for software development. Jørgensen's systematic review pointed out that aggregated empirical evidence did not support this view (see [22] and the subsequent update [23]). About a third of the empirical studies suggested model estimates were better than those made by humans, a third suggested not much difference in estimate accuracy, and the remaining studies suggested estimates from humans outperformed those from models. ...
Article
Full-text available
Context: Recent papers have proposed the use of grey literature (GL) and multivocal reviews. These papers have raised issues about the practices used for systematic reviews (SRs) in software engineering (SE) and suggested that there should be changes to the current SR guidelines. Objective: To investigate whether current SR guidelines need to be changed to support GL and multivocal reviews. Method: We discuss the definitions of GL and the importance of GL and of industry-based field studies in SE SRs. We identify properties of SRs that constrain the material used in SRs: a) the nature of primary studies b) the requirements of SRs to be auditable, traceable, and reproducible, and explain why these requirements restrict the use of blogs in SRs. Results: SR guidelines have always considered GL as a possible source of primary studies and never supported excluding field studies that incorporate the practitioners’ viewpoint. However, GL which was meant to refer to documents that were not formally published, is now being extended to information from blogs/tweets/Q&A posts. Thus, it might seem that SRs do not make full use of GL because they do not include such information. However, the unit of analysis in SR is the primary study. Thus, it is not the source but the type of information that is important. Any report describing a rigorous empirical evaluation is a candidate primary study. Whether it is actually included in a SR depends on the SR eligibility criteria. However, any study that cannot be guaranteed to be publicly available in the long term should not be used as a primary study in an SR. This does not prevent such information being aggregated in surveys of social media and used in the context of evidence-based software engineering (EBSE). Conclusions: Current guidelines for SRs do not require extensions, but their scope needs to be better defined. SE researchers require guidelines for analysing social media posts (e.g., blogs, tweets, vlogs), but these should be based on qualitative primary (not secondary) study guidelines. SE researchers can use mixed-methods SRs and/or the fourth step of EBSE to incorporate findings from social media surveys with those from SRs and to develop industry-relevant recommendations.
... Table 4 summarizes significant waiting times per week. For this study, we considered a significant time of 60 minutes, given that a developer needs a time of understanding, preparation, construction, and testing of a requirement [Jørgensen, 2004]. ...
Article
Full-text available
Data-driven storytelling helps to communicate facts, easing comprehension and decision making, particularly in crisis settings such as the current COVID-19 pandemic. Several studies have reported on general practices and guidelines to follow in order to create effective narrative visualizations. However, research regarding the benefits of implementing those practices and guidelines in software development is limited. In this article, we present a case study that explores the benefits of including data visualization best practices in the development of a software system for the current health crisis. We performed a quantitative and qualitative analysis of sixteen graphs required by the system to monitor patients' isolation and circulation permits in quarantine due to the COVID-19 pandemic. The results showed that the use of storytelling techniques in data visualization contributed to an improved decision-making process in terms of increasing information comprehension and memorability by the system stakeholders.
... Temas asociados al uso de estas metodologías también se encuentran como los Casos de estudio, Gerencia de proyectos y la Gestión del conocimiento, así como el proceso de Estimación de esfuerzo, propio del campo de desarrollo de software en términos de determinar tiempo, personal y dinero que se requiere para el desarrollo y mantenimiento de software (Jørgensen, 2004). ...
Article
Full-text available
Pages: 28-40 Resumen: El desarrollo ágil de software y sus diversas metodologías se han convertido en protagonistas en la tendencia de mejora de procesos para el impulso del software y aplicación de sus principios para resolver retos en diferentes campos: académicos e industriales. Esta investigación presenta algunos hallazgos de un estudio bibliométrico, pretendiendo mostrar el comportamiento del mismo en el tiempo, reflejando su crecimiento y vigencia, evidenciados desde el 2001, fecha importante en la formalización del enfoque de desarrollo de proyectos tecnológicos. Se analizan indicadores de cantidad y calidad para exhibir la difusión e impacto que tienen las indagaciones a través de autores y revistas reconocidos. Además, se relacionan instituciones y países destacados en el desarrollo del tópico, tipo de publicaciones y áreas del conocimiento abordadas. Los temas crecientes y emergentes son un punto clave al ofrecer claridad sobre el panorama investigativo en los últimos años y sectores de aplicación de este conocimiento. Palabras-clave: Agenda; Desarrollo Ágil de Software; Scrum; Bibliometría. Research trends in agile software development Abstract: Agile software development and its various methodologies have become protagonists in the trend of process improvement for the promotion of software and application of its principles to solve challenges in different fields: academic and industrial. This research presents some findings of a bibliometric study, trying to show its behavior in time, reflecting its growth and validity, evidenced since 2001, an important date in the formalization of the development approach of technological projects. Quantity and quality indicators are analyzed to show the diffusion and impact that the investigations have through recognized authors and
... Jorgensen [11] reported the method for choosing the effort unit that affects the expert judgementbased effort estimation. Jorgensen et al. [12] performed an extensive review of the expert estimation of software development effort. They suggested that expert estimation is the most frequently applied. ...
Article
Full-text available
Abstract Most of the software development organisations frequently use an appreciable amount of resources to estimate the effort in the beginning of the development process. In most of the cases, inaccurate estimates tend to wastage of these resources. Very few generalised models have been found in the literature. These models have been developed using the prototype dataset of the organisation. The project management team of an organisation tries to predict the effort needed for the development of software using various mathematical techniques. These techniques are mostly based on statistical methods (viz. simple linear regression (SLR), multi linear regression, support vector machine, cascade correlation neural network (CCNN) etc.) and some probability‐based models. They use historical data of similar projects. The work presented in this article envisages the use of Support Vector Regression (SVR) and constructive cost model (COCOMO), where SVR can be used for both linear and non‐linear models and COCOMO can be used as a regression model. The proposed hybrid model has been tested on the International Software Benchmarking Standards Group dataset. The data has been grouped according to the size of man power. It has been found that the proposed model yields better results than the SVR or SLR for each group of data in general.
... Por ejemplo, Jorgesen et al. en 2004 no encontraron evidencia que apoye que los modelos de estimación son mejores que la estimación por expertos. Notaron que hay situaciones en las cuales los modelos no incluyen información importante del dominio de aplicación que si es tenida en cuenta por los expertos [5]. ...
Technical Report
Full-text available
Resumen-Este documento presenta una guía básica a la Ingeniería de Software Basada en Evidencias. Se describen sus objetivos, su métodos más utilizados y además se incluyen referencias para profundizar en cada tema específico. I. INTRODUCCIÓN Este reporte 1 tiene como objetivo presentar los fundamentos de la Ingeniería de Software Basada en Evidencias (EBSE, por sus siglas en inglés), y en particular se incluye una breve guía para realizar revisiones sistemáticas en ingeniería de software. Este documento puede ser utilizado como guía inicial o referencia por estudiantes o investigadores que se encuentren realizando trabajos de EBSE con el Grupo de Ingeniería de Software (GrIS). Este documento se actualizó para tener en cuenta el libro de referencia de EBSE publicado en 2015 [1], en algunos temas se indican qué capítulos del libro se pueden consultar para profundizar más. En la sección II se presenta la definición de la EBSE así como sus motivaciones. En la sección III se describen las revisiones sistemáticas de literatura, en la sección IV los estudios de mapeo. En la sección V se introducen las revisiones terciarias. Más adelante, en la sección VI se presentan las revisiones rápidas (rapid reviews). Porúltimo, la sección VII incluye una breve introducción a la traducción de conocimiento. II. INGENIERÍA DE SOFTWARE BASADA EN EVIDENCIAS La ingeniería de software basada en evidencias tiene co-mo propósito mejorar la toma de decisiones relacionada al desarrollo y mantenimiento de software integrando la mejor evidencia actual de la investigación con experiencias prácticas y valores humanos [2]. En general, el conocimiento se deriva de la evidencia a partir de un proceso de interpretación [1]. Esto ocurre, por ejemplo, cuando un científico estudia registros médicos para demostrar que fumar tabaco causa cáncer de pulmón. Enáreas con investigación empírica o experimental la evidencia se obtiene mediante observaciones y mediciones, cuyos resultados son reportados en una o más publicaciones científicas. En el pasado relativamente cercano, para disponer del conocimiento más actual sobre un tema era común que un experto hiciera una reseña de las publicaciones que creía más relevantes y actuales. Estas revisiones tradicionales o reseñas 1 Versión 2.0-Julio 2019 tienen sesgos relacionados a la experiencia del investigador, a su subjetividad y además no son del todo reproducibles. Un caso muy famoso que muestra el sesgo del investigador es la revisión de Linus Pauling de 1970 sobre los beneficios de la vitamina C para combatir la gripe [1]. En esa revisión el autor tuvo en cuenta las publicaciones que apoyaban su teoría y descartó el resto sesgando fuertemente los resultados. Al agregar los resultados de varios estudios, pueden surgir problemas, por ejemplo, al tratar de decidir cuándo hay que descartar evidencia débil. Para mejorar la agregación de evidencia surge para medicina en la década de 1970 la práctica basada en evidencias (EBP, por sus siglas en inglés). La EBP busca utilizar un enfoque objetivo, riguroso y planificado para seleccionar estudios relevantes y realizar una síntesis de los resultados de esos estudios [1]. La rigurosidad metodológica hace que los resultados sean más confiables ya que es posible estudiar el procedimiento llevado a cabo para su obtención así como también reproducirlo. En medicina la EBP ha sido fundamental para ayudar a controlar los factores de riesgo de infarto de miocardio y accidente cardiovascular, para transfor-mar el VIH de una infección mortal a una crónica, para probar medicamentos para la hepatitis C y para mejorar tratamientos de algunos tipos de cáncer [3]. Las técnicas utilizadas en EBP son llamadas estudios se-cundarios, ya que realizan la agregación de evidencia a partir de estudios primarios (experimentos controlados, estudios de caso, encuestas, entre otros). El principal estudio secundario es la revisión sistemática de la literatura (SLR, por sus siglas en inglés). Las SLRs permiten recolectar y sintetizar evidencia de distintas fuentes. La característica clave que las distingue de las revisiones tradicionales narrativas (o clásicas) es su intento explícito de minimizar las posibilidades de llegar a conclusiones erradas, que puedan resultar del sesgo en los estudios primarios o en el proceso de revisión [4]. Para lograrlo se debe establecer un plan, llamado protocolo, con todas las actividades a realizar y criterios a utilizar previo a la ejecución de la SLR. La introducción de la EBP en elárea de la ingeniería de software comenzó en 2004 [2], llamándose EBSE, y tuvo una gran aceptación por parte de los investigadores. Se estima que se han publicado más de 200 estudios secundarios solamente en los primeros diez años [1]. Muchos de sus resultados se han utilizado como punto de partida para investigaciones más amplias. EBSE tiene tanta importancia en la investigación
... The questionnaire was distributed to the RPA creators of the twelve projects. Note that the data retrieved with the questionnaire are estimations made by experts to the best of their knowledge, which constitutes a widely accepted approach [12]. ...
Conference Paper
Full-text available
Robotic Process Automation (RPA) is the automation of rule-based routine processes to increase process efficiency and to reduce process costs. In practice, however, RPA is often applied without knowledge of the concrete effects its introduction will have on the automated process and the involved stakeholders. Accordingly, literature on the quantitative effects of RPA is scarce. The objective of this paper is to provide empirical insights into improvements and deteriorations of business processes achieved in twelve RPA projects in the automotive industry. The results indicate that the positive benefits promised in literature are not always achieved in practice. In particular, shorter case duration and better quality are not confirmed by the empirical data gathered in the considered RPA projects. These quantitative insights constitute a valuable contribution to the currently rather qualitative literature on RPA.
... Programming effort: Games' file size Assessing software development costs is deemed to be a difficult task (Jørgensen, 2004). In the IT project management literature, the most frequently mentioned measurable cost driver is software program size (Mislick and Nussbaum, 2015). ...
Article
Increasing numbers of consumers who engage in the development of new products are selling their innovations on online marketplaces. We contribute to the scarce research on the commercialization activities of consumer innovators by comparing the consumers’ price decisions with the pricing of firms. Our predictions build on the baseline assumption that the price decisions of consumers are influenced by the same motivations that originally prompt them to innovate. We use a sequential mixed-method approach with a quantitative main study and follow-up qualitative research. The quantitative results draw on a matched-pair analysis of 4,242 computer games released on the online game platform Steam. We find that consumer innovators charge lower prices than firms for comparable games and that consumers and firms show different inclinations in aligning prices with the games’ development costs and perceived quality. The subsequent interview study with 29 hobbyist game developers provides clear support for the motivational explanations of consumers’ pricing decisions. The findings contribute to research on consumer innovation marketing and nascent entrepreneurship. They also improve the understanding of welfare effects resulting from increasing commercial activities of consumers.
... To this end, teams mainly rely on expert estimation methods like Planning Poker and Delphi [3]. However, expert judgment has been shown to be prone to bias due to its reliance on subjective assessment [4]- [6]. This motivated several research endeavours to find automated ways to predict story points of a task given its features with the aim of avoiding inaccurate estimations by human judgement, in addition to, and more importantly to an agile team, producing consistent estimations throughout the project's lifecycle. ...
Conference Paper
Automated techniques to estimate Story Points (SP) for user stories in agile software development came to the fore a decade ago. Yet, the state-of-the-art estimation techniques’ accuracy has room for improvement. In this paper, we present a new approach for SP estimation, based on analysing textual features of software issues by employing latent Dirichlet allocation (LDA) and clustering. We first use LDA to represent issue reports in a new space of generated topics. We then use hierarchical clustering to agglomerate issues into clusters based on their topic similarities. Next, we build estimation models using the issues in each cluster. Then, we find the closest cluster to the new coming issue and use the model from that cluster to estimate the SP. Our approach is evaluated on a dataset of 26 open source projects with a total of 31,960 issues and compared against both baselines and state-of-the-art SP estimation techniques. The results show that the estimation performance of our proposed approach is as good as the state-of-the-art. However, none of these approaches is statistically significantly better than more naive estimators in all cases, which does not justify their additional complexity. We therefore encourage future work to develop alternative strategies for story points estimation. The experimental data and scripts we used in this work are publicly available to allow for replication and extension.
... In reviewing tools and techniques used to create the initial baseline, Jørgensen (2004a) researched the use of expert estimation to develop schedules for software based projects and found that the use of expert estimation was the most common. In a more recent paper, Jorgensen (2014) found that there is no one "Best Effort Estimation Model", but recommended the use of historical data to set minimum and maximum values, whilst still allowing expert estimation to develop the P50 case. ...
Chapter
Developing and delivering a project to an agreed schedule is fundamentally what project managers do. There is still an ongoing debate about schedule delays. This research investigates the development of schedules through semi-structured in-depth interviews. The findings reveal that half of the respondents believe that delays reported in the media are not real and should be attributed to scope changes. IT project managers estimating techniques include bottom-up estimates, analogy, and expert judgement. Impeding factors reported for the development of realistic schedules were technical (e.g. honest mistakes) and political (e.g. completion dates imposed by the sponsor). Respondents did not mention any psychological factors, although most were aware of optimism bias. However, they were not familiar with approaches to mitigate its impacts. Yet, when these techniques were mentioned, the overwhelming majority agreed that these mitigation approaches would change their schedule estimate.
Article
In the real world, many software projects have effort, schedule, and cost overrun. Estimation accuracy is vital for a successful software project. This is a challenge since mostly there is a tendency to over-estimate or under-estimate the size and effort needed in a software project. The prediction of required effort is a critical activity, and a greater focus is required by considering additional factors such as project risk. The importance of project risks and their management has been indicated in literature and its consideration in effort estimation is the focus of this research. This research paper has a focus on proposing a model based on machine learning techniques for improving the effort estimation through inclusion of risk score. The proposed solution considered aggregating the capability of various machine learning models for prediction. The methodology involved usage of extreme gradient boosting algorithm. Data for the research included projects from industry standard dataset and organizational projects. The analysis revealed a reduction in the root mean square error values over multiple iterations suggesting an improvement in model performance. This reveals a better estimation due to minimization of the gap between estimated and actual efforts in a project. This, in turn, would enhance the potential for success of the project through an improved estimation process integrating risk score along with other parameters for estimation of software development effort. The feature importance chart also revealed that project risk score is an important attribute to be considered for effort estimation.
Article
Software cost estimation is the prediction of development effort and calendar time required to develop any software project. It is considered to be the very fundamental task for successful execution of an on-going project as well as budgetary requirements of futuristic projects. As accuracy in software cost estimation is very hard because of the availability of vague information at the time of inception of the software project, it prompted many researchers to explore in this domain from past decades. Their pioneer works suggest a bulk of techniques for this purpose. However, because of the availability of large number of estimation techniques, it becomes hard for any software practitioners to select an appropriate one. To help the industry practitioners in these situations, a novel analogy-centered model based on differential evolution exploration process is proposed in this research study. The proposed model has been assessed on 676 projects from 5 different data sets and the results achieved are significantly better when compared with other benchmark analogy-based estimation studies. Furthermore, being the very less computational cost of the proposed model, it is suggested that the proposed model be considered as the preliminary stage of any analogy-based software estimation technique.
Article
Full-text available
The present research estimates the efficacy of a legacy program and the areas of its development. The research also intends to put forward as to what extent reengineering of a legacy program has to be done on the basis of the estimation approach. The study has tried to outline the current issues and trends in reengineering of a legacy program from various perspectives. An all-inclusive literature review reveals that a lot of work has already been piled up with legacy system estimation and the reengineering domain, yet the basic assumptions of Complexity, Quality and Effort have not been worked out collectively. Hence the present research underlines this very maxim and studies the reengineering of a legacy program on the paradigms of Quality, Complexity, and Effort Estimation collectively. The findings put forward an equation and reengineering scale which would be highly compatible with present technology for the feasibility of an effective reengineering.
Preprint
Full-text available
Now a days, software development organizations use machine learning techniques in different areas to improve decision-making process so that their performance is boosted. In this dissertation, with the goal of increasing the accuracy in effort estimates, we applied programming models in an environment of software development organizations. We collected empirical data from two organizations and constructed a consolidated data sets. The programming models applied in this study are K-Means clustering, Support Vector Machines using polynomial kernel, Random Forest, Linear Regression, K Nearest Neighbor and Neural Networks using ORANGE tool. The obtained results demonstrate the use of data mining and machine learning techniques in general increases the accuracy of predictions with lesser error magnitude as compared to experts. Moreover, we recommend application of programming models in comparable environment of software development organizations to get reliable and more generalized predictions for decision making.
Chapter
The Internet of Things (IoT) is an important emerging technology that enables (usually) pervasive ubiquitous devices to connect to the Internet. Medical and Healthcare Internet of Things (MHIoT) represents one of the application areas for IoT that has revolutionized the healthcare sector. In this study, a systematized literature review on the adoption of MHIoT for diabetes management is done to investigate the application of IoT in the monitoring of diabetes, key challenges, what has been done, in which context, and the research gap using Denyer and Transfield’s systematic literature review methodology. The key findings reveal that developing nations are lagging despite the greater benefits of MHIoT in such resource-constrained contexts. The findings suggest that infrastructure costs, security, and privacy issues are most important in the adoption of MHIoT for diabetes management. The opportunities presented by MHIoT surpass the challenges as healthcare costs are reduced in a resource-constrained context. Further research in infrastructural needs and privacy concerns is needed to take full advantage of these benefits and address the challenges.KeywordsHealth careDeveloping countriesDeveloped countriesSensorsGlucoseBlood sugarActuatorsRemote health monitoring
Article
Full-text available
Article
Decades of research suggest that information exchange in groups and organizations can reliably improve judgment accuracy in tasks such as financial forecasting, market research, and medical decision making. However, we show that improving the accuracy of numeric estimates does not necessarily improve the accuracy of decisions. For binary-choice judgments, also known as classification tasks—for example, yes/no or build/buy decisions—social influence is most likely to grow the majority vote share, regardless of the accuracy of that opinion. As a result, initially, inaccurate groups become increasingly inaccurate after information exchange, even as they signal stronger support. We term this dynamic the “crowd classification problem.” Using both a novel data set and a reanalysis of three previous data sets, we study this process in two types of information exchange: (1) when people share votes only, and (2) when people form and exchange numeric estimates prior to voting. Surprisingly, when people exchange numeric estimates prior to voting, the binary-choice vote can become less accurate, even as the average numeric estimate becomes more accurate. Our findings recommend against voting as a form of decision making when groups are optimizing for accuracy. For those cases where voting is required, we discuss strategies for managing communication to avoid the crowd classification problem. We close with a discussion of how our results contribute to a broader contingency theory of collective intelligence. This paper was accepted by Lamar Pierce, organizations.
Article
Full-text available
Analogy-based effort estimation is the major task of software engineering which estimates the effort required for new software projects using existing histories for corresponding development and management. In general, the high accuracy of software effort estimation techniques can be a non-solvable problem we named as multi-objective problem. Recently, most of the authors have been used machine learning techniques for the same process however not possible to meet the higher performance. Moreover, existing software effort estimation techniques are mostly affected by bias and subjectivity problems. Analogy based effort estimation (ABE) is the most extensively conventional technique because of its effortlessness and evaluation ability. We define five research questions are defined to get clear thoughts on ABE studies. Improvement of ABE can be done through supervised learning techniques and unsupervised learning techniques. Furthermore, the results can be knowingly affected by the different performance metrics in ABE configuration.
Article
Full-text available
Judgmental and statistical forecasts can each bring advantages to the forecasting process. One way forecasters can integrate these methods is to adjust statistical forecasts based on judgment. However, judgmental adjustments can bias forecasts and harm accuracy. Forecasters should consider six principles in deciding when and how to use judgment in adjusting statistical forecasts: (1) Adjust statistical forecasts if there is important domain knowledge; (2) adjust statistical forecasts in situations with a high degree of uncertainty; (3) adjust statistical forecasts when there are known changes in the environment; (4) structure the judgmental adjustment process; (5) document all judgmental adjustments made and periodically relate to forecast accuracy; (6) consider mechanically integrating judgmental and statistical forecasts over adjusting.
Article
Full-text available
But 'glory' doesn't mean "a nice knock -down argument," Alice objected. "When I use a word," Humpty Dumpty said, in a rather scornful tone, "it means just what I choose it to mean—neither more nor less." "The question is," said Alice, "whether you can make words mean so many different things." "The question is," said Humpty Dumpty, "which is to be master—that's all." Through the Looking Glass Lewis Carroll This dictionary defines terms as they are commonly used in forecasting. The aims, not always met, are to: ?? provide an accurate and understandable definition of each term, ?? describe the history of the term, ?? demonstrate how the term is used in forecasting, ?? point out how the term is sometimes misused, and ?? provide research findings on the value of the term in forecasting.
Article
Full-text available
Information systems cost estimating is an important management concern. An estimate helps to cost justify individual proposals, to schedule their development, to staff them, to control and monitor their progress, and to evaluate estimators and implementers. Through a case study of a chemical manufacturer, the investigation reported in this article facilitates a better understanding of the management of the cost estimating process. Interviews with 17 information systems managers and staff members, and four user managers confirm that the practice of cost estimating can be viewed in terms of both a Rational Model and a Political Model, can identify impediments to accurate estimating, and can provide suggestions and warnings for managers and future researchers.
Article
Full-text available
Direct comparisons were made of expert highway engineers' use of analytical, quasirational, and intuitive cognition on three different tasks, each displayed in three different ways. Use of a systems approach made it possible to develop indices for measuring the location of each of the nine information display conditions on a continuum ranging from intuition-inducing to analysis-inducing and for measuring the location of each expert engineer's cognition on a continuum ranging from intuition to analysis. Individual analyses of each expert's performance over the nine conditions showed that the location of the task on the task index induced the expert's cognition to be located at the corresponding region on the cognitive continuum index. Intuitive and quasirational cognition frequently outperformed analytical cognition in terms of the empirical accuracy of judgments. Judgmental accuracy was related to the degree of correspondence between the type of task and the type of cognitive activity on the cognitive continuum.
Article
Full-text available
Overconfidence is a common finding in the forecasting research literature. Judgmental overconfidence leads people (1) to neglect decision aids, (2) to make predictions contrary to the base rate, and (3) to succumb to “groupthink.” To counteract overconfidence forecasters should heed six principles: (1) Consider alternatives, especially in new situations; (2) List reasons why the forecast might be wrong; (3) In group interaction, appoint a devil’s advocate; (4) Make an explicit prediction and then obtain feedback; (5) Treat the feedback you receive as valuable information; (6) When possible, conduct experiments to test prediction strategies. These principles can help people to avoid generating only reasons that bolster their predictions and to learn optimally by comparing a documented prediction with outcome feedback.
Article
Full-text available
Existing algorithmic models fail to produce ac- curate software development effort estimates. To address this problem, a case-based reasoning model, called Estor, was developed based on the verbal protocols of a human expert solving a set of estimation problems. Estor was then presented with 15 software effort estimation tasks. The estimates of Estor were compared to those of the expert as well as those of the function point and CQCQMQ estimations of the projects. The estimates generated by the human expert and Estor were more accurate and consistent than those of the function point and CQCQMQ methods. In fact, Estor was nearly as accurate and consistent as the expert. These results sug-
Article
Full-text available
Expert opinion is often necessary in forecasting tasks because of a lack of appropriate or available information for using statistical procedures. But how does one get the best forecast from experts? One solution is to use a structured group technique, such as Delphi, for eliciting and combining expert judgments. In using the Delphi technique, one controls the exchange of information between anonymous panelists over a number of rounds (iterations), taking the average of the estimates on the final round as the group judgment. A number of principles are developed here to indicate how to conduct structured groups to obtain good expert judgments. These principles, applied to the conduct of Delphi groups, indicate how many and what type of experts to use (five to 20 experts with disparate domain knowledge); how many rounds to use (generally two or three); what type of feedback to employ (average estimates plus justifications from each expert); how to summarize the final forecast (weight all experts’ estimates equally); how to word questions (in a balanced way with succinct definitions free of emotive terms and irrelevant information); and what response modes to use (frequencies rather than probabilities or odds, with coherence checks when feasible). Delphi groups are substantially more accurate than individual experts and traditional groups and somewhat more accurate than statistical groups (which are made up of noninteracting individuals whose judgments are aggregated). Studies support the advantage of Delphi groups over traditional groups by five to one with one tie, and their advantage over statistical groups by 12 to two with two ties. We anticipate that by following these principles, forecasters may be able to use structured groups to harness effectively expert opinion.
Chapter
Principles designed to improve judgment in forecasting aim to minimize inconsistency and bias at different stages of the forecasting process (formulation of the forecasting problem, choice of method, application of method, comparison and combination of forecasts, assessment of uncertainty in forecasts, adjustment of forecasts, evaluation of forecasts). The seven principles discussed concern the value of checklists, the importance of establishing agreed criteria for selecting forecast methods, retention and use of forecast records to obtain feedback, use of graphical rather than tabular data displays, the advantages of fitting lines through graphical displays when making forecasts, the advisability of using multiple methods to assess uncertainty in forecasts, and the need to ensure that people assessing the chances of a plan’s success are different from those who develop and implement it.
Article
Many decisions are based on beliefs concerning the likelihood of uncertain events such as the outcome of an election, the guilt of a defendant, or the future value of the dollar. Occasionally, beliefs concerning uncertain events are expressed in numerical form as odds or subjective probabilities. In general, the heuristics are quite useful, but sometimes they lead to severe and systematic errors. The subjective assessment of probability resembles the subjective assessment of physical quantities such as distance or size. These judgments are all based on data of limited validity, which are processed according to heuristic rules. However, the reliance on this rule leads to systematic errors in the estimation of distance. This chapter describes three heuristics that are employed in making judgments under uncertainty. The first is representativeness, which is usually employed when people are asked to judge the probability that an object or event belongs to a class or event. The second is the availability of instances or scenarios, which is often employed when people are asked to assess the frequency of a class or the plausibility of a particular development, and the third is adjustment from an anchor, which is usually employed in numerical prediction when a relevant value is available.
Article
A summary is presented of the current state of the art and recent trends in software engineering economics. It provides an overview of economic analysis techniques and their applicability to software engineering and management. It surveys the field of software cost estimation, including the major estimation techniques available, the state of the art in algorithmic cost models, and the outstanding research issues in software cost estimation.
Article
A solution is suggested for an old unresolved social psychological problem.
Article
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of the participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
In this book we describe how to elicit and analyze expert judgment. Expert judgment is defined here to include both the experts' answers to technical questions and their mental processes in reaching an answer. It refers specifically to data that are obtained in a deliberate, structured manner that makes use of the body of research on human cognition and communication. Our aim is to provide a guide for lay persons in expert judgment. These persons may be from physical and engineering sciences, mathematics and statistics, business, or the military. We provide background on the uses of expert judgment and on the processes by which humans solve problems, including those that lead to bias. Detailed guidance is offered on how to elicit expert judgment ranging from selecting the questions to be posed of the experts to selecting and motivating the experts to setting up for and conducting the elicitation. Analysis procedures are introduced and guidance is given on how to understand the data base structure, detect bias and correlation, form models, and aggregate the expert judgments.
Article
In various decision contexts, a decision-maker has to combine the estimates or information supplied by several experts. Methods which may be used to combine different estimates are presented with examples. A new method is presented when the opinions of the experts have a particular form of dependence, namely a common overlap of knowledge base. The role of interaction among the experts, and the emergence of consensus, are discussed. New methods are also presented for combining expert estimates when a small or sketchy amount of observational data is available.
Article
The estimation of development effort and time is difficult for demanding real-time data communications software projects, where rapid response rates are critical in machine-machine communications. Applying popular cost models to estimate cost and time in three real-time data communications development projects showed that the development effort predictions varied from overestimation by a factor of 515 to an underestimation by a factor of 0.06. These results indicate that current cost estimation models may not be appropriate in estimating development effort for such software. Based on the distinctive features of data communications software projects, suggestions are made to modify the parameters of the function point model. Future research is essential to develop a more accurate model for these software projects.
Article
This paper presents results from two case studies and two experiments on how effort estimates impact software project work. The studies indicate that a meaningful interpretation of effort estimation accuracy requires knowledge about the size and nature of the impact of the effort estimates on the software work. For example, we found that projects with high priority on costs and incomplete requirements specifications were prone to adjust the work to fit the estimate when the estimates were too optimistic, while too optimistic estimates led to effort overruns for projects with high priority on quality and well specified requirements.Two hypotheses were derived from the case studies and tested experimentally. The experiments indicate that: (1) effort estimates can be strongly impacted by anchor values, e.g. early indications on the required effort. This impact is present even when the estimators are told that the anchor values are irrelevant as estimation information; (2) too optimistic effort estimates lead to less use of effort and more errors compared with more realistic effort estimates on programming tasks.
Article
In all studies of human performance, the experimenter must be certain that the subject is performing the task that the experimenter believes he has set; otherwise results become uninterpretable. Early studies of computer programming have shown such wide variations in individual performance that one might suspect that subjects differed in their interpretation of the task. Experiments are reported which show how programming performance can be strongly influenced by slight differences in performing objectives. Conclusions are drawn from these results regarding both future experimentation and management practices in computer programming.
Article
Questionnaires sent to maintenance managers is a frequently used approach to collect data on software maintenance. This paper reports findings from two studies, carried out at a large Norwegian maintenance organisation, investigating the quality of questionnaire based software maintenance studies.Interesting findings were, among others, that:- The definition of essential terms, for example of 'software maintenance', at the beginning of a questionnaire did not assure a consistent use of the terms by the questionnaire respondents.- Manager estimates of the proportion of effort spent on corrective maintenance were biased when based on best guesses instead of good data. For this reason, the frequently referred studies of Lientz and Swanson (1980) and Nosek and Palvia (1990) may have reported a too high proportion of effort spent on corrective maintenance.
Article
We report three experiments comparing confidence judgments made by actors and by observers. In Experiment 1, actors generated qualitative answers (countries of the world) in a country-identification task; in Experiment 2, actors generated quantitative answers (years) in a historical event-dating task. Both actors and observers indicated their confidence in the actors' answers. Actors were signifi- cantly less confident in their answers than were observers in the first experi- ment. This eÄect was substantially reduced in the second experiment, whether confidence was measured by judged probability or by credible interval width. Experiment 3 used a control task in which actors attempted to bring an outcome variable into a desired range. In contrast to the first two experiments, actors in the control task were more confident than observers. Because subjects were generally overconfident in all three experiments, the present results demonstrate that the use of observers can reduce or exacerbate overconfidence depending on the kind of task and the nature of the event or possibility under evaluation. #1997 by John Wiley & Sons, Ltd.
When a panel of experts is assembled to make predictions about some aspect of the future, they invariably disagree. The possible strategies for dealing with such disagreement include (1) taking the statistical average of the individual forecasts, (2) face-to-face discussion until consensus is achieved, (3) the Delphi procedure, and (4) the Estimate-Talk-Estimate procedure proposed by D. H. Gustafson, R. K. Shukla, A. Delbecq, and G. W. Walster (Organizational Behavior and Performance, 1973). This paper does two things. First, it very briefly reviews the literature relevant to opinion aggregation when forecasts are expressed as subjective probability distributions. Second, it describes an experimental comparison of the four procedures listed above using a subjective probability forecasting task. Together, the review and the experiment lead to two conclusions. First, subjective probability forecasts can be substantially improved by aggregating the opinions of a group of experts rather than relying on a single expert. Second, from a practical standpoint, there is no evidence to suggest that the method used to aggregate opinions will have a substantial impact on the quality of the resulting forecast.
Article
An empirical study that compares the relative effectiveness of groups vs. individuals in developing a project plan and focuses on two aspects of planning effectiveness (quality and acceptance) is presented. Members of 80 groups completed a simulation, the Project Planning Situation, first individually and then as interacting groups. The results show that the quality of the project plans developed by the groups was significantly higher than the average quality of the plans developed by members working independently. The groups' plans also were better than those that were derived through nominal techniques. It is open to question, however, whether the groups' plans were always superior to those of their best members. The effectiveness of the groups in planning is related to the two basic elements of group process: the rational and the interpersonal. The rational elements of process determined the quality of the plan and the interpersonal factor were associated with the groups' acceptance of the project plan. The management implications of these findings are discussed.
Article
We examine predictions and judgments of confidence based on one-sided evidence. Some subjects saw arguments for only one side of a legal dispute while other subjects (called 'jurors') saw arguments for both sides. Subjects predicted the number of jurors who favored the plaintiff in each case. Subjects who saw only one side made predictions that were biased in favor of that side. Furthermore, they were more confident but generally less accurate than subjects who saw both sides. The results indicate that people do not compensate sufficiently for missing information even when it is painfully obvious that the information available to them is incomplete. A simple manipulation that required subjects to evaluate the relative strength of the opponent's side greatly reduced the tendency to under- weigh missing evidence.
Decision-making (prediction) behavior under two types of conflict was experimentally examined within the Social Judgment Theory research paradigm. Interpersonal cognitive conflict (the degree of disagreement over the interpretation of a common stimulus), goal conflict (the degree of competition for payoffs), and trial blocks were independent variables. Prediction error was the dependent measure. Individuals made better predictions about task-criterion values under no-goal conflict than under goal conflict conditions. During the initial stage of a series of prediction trials, subjects made better predictions of task-criterion values under high cognitive conflict than under low cognitive conflict conditions. All groups of subjects were able to improve prediction performance significantly over time. These results are generally consistent with arguments stressing the potential benefits of minimal goal conflict over payoffs and high cognitive conflict on decision quality.
Article
Realistic estimation is a process by which the cost and time of software projects can be predicted. This enables management to set up attainable project objectives—that is software development organizations delivering what was promised, on time, and in budget. The main benefit is an enhancement of the professional credibility of these organizations. I have observed that some organizational aspects support deployment of software estimation while others block it. In this paper, I have defined these aspects as Driving Forces and Restraining Forces, as per Kurt Lewin's Force-Field Analysis. The purpose of this paper is to review these elements with a view to provoking new thinking.
Article
This paper examines the role of computer-based decision aids in reducing cognitive effort and therefore influencing strategy selection. It extends and complements the work reported in the behavioral decision theory literature on the role of effort and accuracy in choice tasks. The central proposition of the research is that if a decision aid makes a strategy that should lead to a more accurate outcome at least as easy to employ as a simpler, but less accurate, heuristic, then the use of a decision aid should induce that more accurate strategy and as a consequence improve decision quality. Otherwise, a decision aid may only influence decision-making efficiency. This occurs because decision makers use a decision aid in such a way as to minimize their overall level of effort expenditure. Results from a laboratory experiment support this proposition. When a more accurate normative strategy is made less effortful to use, it is used. This result is consistent with the findings of our prior studies, but more clearly demonstrates that decision aids can induce the use of normatively oriented strategies. The key to inducing these strategies is to make the normative strategy easier to execute than competing alternative strategies. Copyright © 2000 John Wiley & Sons, Ltd.
This note examines the number of experts to be included in a prediction group where the criterion of predictive ability is the correlation between the uncertain event and the mean judgment of the group members. It is shown that groups containing between 8 and 12 members have predictive ability close to the “optimum” under a wide range of circumstances but provided (1) mean intercorrelation of experts' opinions is not low (<.3, approximately) and/or (2) mean expert validity does not exceed mean intercorrelation. Evidence indicates these exceptions will not be common in practice. The characteristics needed by an additional expert to increase the validity of an existing group are also derived.
Article
A study of information systems managers and other information systems professionals at 112 different organizations confirmed that information systems software cost estimating is an important concern. Subjects reported the completion of only one of every four systems development projects within their estimates. According to them, the major cause of inaccurate estimates was changes in user requirements. Organizations using sophisticated cost estimating software packages were less successful at preventing large cost overruns than organizations not using them. However, the use of the estimator as system developer, the careful monitoring of systems development projects, and the inclusion in performance evaluations of success in meeting estimates were associated with more accurate cost estimating.
Article
The paper describes the results of an experiment in software costing. The Mark II Function Point method was used to predict the cost of a number of projects at the Inland Revenue's Information Technology. The results were compared with individual managers' estimates and the actual expenditure. The results give some cause for optimism in the use of the function point model that was used.