Figure 3 - uploaded by Maurice Dawson
Content may be subject to copyright.
IBM System Science Institute Relative Cost of Fixing Defects 

IBM System Science Institute Relative Cost of Fixing Defects 

Source publication
Article
Full-text available
This article examines the integration of secure coding practices into the overall Software Development Life Cycle (SDLC). Also detailed is a proposed methodology for integrating software assurance into the Department of Defense Information Assurance Certification & Accreditation Process (DIACAP). This method for integrating software assurance helps...

Citations

... After all, one of the main reasons why technology providers engage with auditors is that it is cheaper and easier to address system vulnerabilities early in the development process. For example, it can costs up to 15 times more to fix a software bug found during the testing phase than fixing the same bug found in the design phase [140]. This suggests that-despite the associated costs-businesses have clear incentives to design and implement effective EBA procedures. ...
Article
Full-text available
Ethics-based auditing (EBA) is a structured process whereby an entity’s past or present behaviour is assessed for consistency with moral principles or norms. Recently, EBA has attracted much attention as a governance mechanism that may help to bridge the gap between principles and practice in AI ethics. However, important aspects of EBA—such as the feasibility and effectiveness of different auditing procedures—have yet to be substantiated by empirical research. In this article, we address this knowledge gap by providing insights from a longitudinal industry case study. Over 12 months, we observed and analysed the internal activities of AstraZeneca, a biopharmaceutical company, as it prepared for and underwent an ethics-based AI audit. While previous literature concerning EBA has focussed on proposing or analysing evaluation metrics or visualisation techniques, our findings suggest that the main difficulties large multinational organisations face when conducting EBA mirror classical governance challenges. These include ensuring harmonised standards across decentralised organisations, demarcating the scope of the audit, driving internal communication and change management, and measuring actual outcomes. The case study presented in this article contributes to the existing literature by providing a detailed description of the organisational context in which EBA procedures must be integrated to be feasible and effective.
... In this work, we propose to apply the surprisal measure to software engineering artefacts, motivated by many researchers arguing that software developers need to be aware of unusual or surprising events in their repositories, e.g., when summarizing project activity [19], notifying developers about unusual commits [7,9], and for the identification of malicious content [26]. The basic intuition is that catching bad surprises early will save effort, cost, and time, since bugs cost significantly more to fix during implementation or testing than in earlier phases [17], and by extension, bugs cost more the longer they exist in a product after being reported and before being addressed. ...
Preprint
Full-text available
Background. From information theory, surprisal is a measurement of how unexpected an event is. Statistical language models provide a probabilistic approximation of natural languages, and because surprisal is constructed with the probability of an event occuring, it is therefore possible to determine the surprisal associated with English sentences. The issues and pull requests of software repository issue trackers give insight into the development process and likely contain the surprising events of this process. Objective. Prior works have identified that unusual events in software repositories are of interest to developers, and use simple code metrics-based methods for detecting them. In this study we will propose a new method for unusual event detection in software repositories using surprisal. With the ability to find surprising issues and pull requests, we intend to further analyse them to determine if they actually hold importance in a repository, or if they pose a significant challenge to address. If it is possible to find bad surprises early, or before they cause additional troubles, it is plausible that effort, cost and time will be saved as a result. Method. After extracting the issues and pull requests from 5000 of the most popular software repositories on GitHub, we will train a language model to represent these issues. We will measure their perceived importance in the repository, measure their resolution difficulty using several analogues, measure the surprisal of each, and finally generate inferential statistics to describe any correlations.
... This shows that security is one of the serious issues in the current era that need to be addressed carefully during SDLC. Further, the relative cost of addressing bugs and failure increase as the project progress as mentioned in the IBM system science institute report [26]. Therefore, handling security from the beginning of the project is necessary to save the software from future security breaches. ...
... It is time consuming, disrupts schedules, and hurts the reputation of software products. Moreover, it is generally accepted that fixing bugs costs more the later they are found and that maintenance is costlier than initial development (Boehm, 1981;Boehm & Basili, 2001;Boehm & Papaccio, 1988;Dawson et al., 2010;Hackbarth et al., 2016). The effort invested in bug fixing therefore reflects on the health of the development process. ...
Article
Full-text available
The effort invested in software development should ideally be devoted to the implementation of new features. But some of the effort is invariably also invested in corrective maintenance, that is in fixing bugs. Not much is known about what fraction of software development work is devoted to bug fixing, and what factors affect this fraction. We suggest the Corrective Commit Probability (CCP), which measures the probability that a commit reflects corrective maintenance, as an estimate of the relative effort invested in fixing bugs. We identify corrective commits by applying a linguistic model to the commit messages, achieving an accuracy of 93%, higher than any previously reported model. We compute the CCP of all large active GitHub projects (7,557 projects with 200+ commits in 2019). This leads to the creation of an investment scale, suggesting that the bottom 10% of projects spend less than 6% of their total effort on bug fixing, while the top 10% of projects spend at least 39% of their effort on bug fixing — more than 6 times more. Being a process metric, CCP is conditionally independent of source code metrics, enabling their evaluation and investigation. Analysis of project attributes shows that lower CCP (that is, lower relative investment in bug fixing) is associated with smaller files, lower coupling, use of languages like JavaScript and C# as opposed to PHP and C++, fewer code smells, lower project age, better perceived quality, fewer developers, lower developer churn, better onboarding, and better productivity.
... However, the relative cost of fixing defects grows significantly through the software development life-cycle. [1] found that resolving the defects in maintenance can cost 100 times more compared to early detection and fix. Software defect prediction (SDP) plays an important role of reducing the cost by recognizing defect-prone modules of a software prior to testing [2]. ...
... Finding and fixing bugs accounts for a significant portion of maintenance cost for software (Britton et al., 2013), and the cost of bug fixing increases exponentially with time (Dawson et al., 2010). Hence, automatic program repair has long been a focus of software engineering, with the goal of lowering down the cost and reducing the introduction of new bugs during bug fixing. ...
Preprint
Full-text available
The advance in machine learning (ML)-driven natural language process (NLP) points a promising direction for automatic bug fixing for software programs, as fixing a buggy program can be transformed to a translation task. While software programs contain much richer information than one-dimensional natural language documents, pioneering work on using ML-driven NLP techniques for automatic program repair only considered a limited set of such information. We hypothesize that more comprehensive information of software programs, if appropriately utilized, can improve the effectiveness of ML-driven NLP approaches in repairing software programs. As the first step towards proving this hypothesis, we propose a unified representation to capture the syntax, data flow, and control flow aspects of software programs, and devise a method to use such a representation to guide the transformer model from NLP in better understanding and fixing buggy programs. Our preliminary experiment confirms that the more comprehensive information of software programs used, the better ML-driven NLP techniques can perform in fixing bugs in these programs.
... Similarly, dependency analysis tools [treatment of E2.2, E2.3] are designed to be run together with code analysis tools. Hence, the code security analysis is often postponed to the prerelease stage (or even skipped altogether), which exponentially increases the cost of fixing discovered vulnerabilities [44]. ...
Preprint
Full-text available
Pushed by market forces, software development has become fast-paced. As a consequence, modern development projects are assembled from 3rd-party components. Security & privacy assurance techniques once designed for large, controlled updates over months or years, must now cope with small, continuous changes taking place within a week, and happening in sub-components that are controlled by third-party developers one might not even know they existed. In this paper, we aim to provide an overview of the current software security approaches and evaluate their appropriateness in the face of the changed nature in software development. Software security assurance could benefit by switching from a process-based to an artefact-based approach. Further, security evaluation might need to be more incremental, automated and decentralized. We believe this can be achieved by supporting mechanisms for lightweight and scalable screenings that are applicable to the entire population of software components albeit there might be a price to pay.
... If we take the time difference between the first and the last event, this will likely be close to the total development time. This can be done for all users of both the control and [112,113]. ...
... Security aspects should be integrated in all phases of the development cycle to avoid the costs which can increase exponentially. Relative defect repair costs are shown in Fig. 2. [16] A discovery of vulnerability showed that the software code could involve reconstruction and implementation. It will require a lot of time and money. ...
... The figure above has shown that the defects identified in the testing phase are 15 times more expensive than the design phase; the defects identified in the testing phase are 2 times more expensive than the implementation phase. The defects found in maintenance are most costly which is 7 times more expensive than the testing phase [16]. ...
Preprint
Full-text available
Information protection is becoming a focal point for designing, creating and implementing software applications within highly integrated technology environments. The use of a safe coding technique in the software development process is required by many industrial IT security standards and policies. Despite current cyber protection measures and best practices, vulnerabilities still remain strong and become a huge threat to every developed software. It is crucial to understand the position of secure software development for security management, which is affected by causes such as human security-related factors. Although developers are often held accountable for security vulnerabilities, in reality, many problems often grow from a lack of organizational support during development tasks to handle security. While abstract safe coding guidelines are generally recognized, there are limited low-level secure coding guidelines for various programming languages. A good technique is required to standardize these guidelines for software developers. The goal of this paper is to address this gap by providing software designers and developers with direction by identifying a set of secure software development guidelines. Additionally, an overview of criteria for selection of safe coding guidelines is performed along with investigation of appropriate awareness methods for secure coding.
... And since bugs are time consuming, disrupt schedules, and hurt the general credibility, lowering the bug rate has value regardless of other implications-thereby lending value to having a low CCP. Moreover, it is generally accepted that fixing bugs costs more the later they are found, and that maintenance is costlier than initial development [16], [17], [19], [29]. Therefore, the cost of low quality is even higher than implied by the bug ratio difference. ...
Preprint
We present a code quality metric, Corrective Commit Probability (CCP), measuring the probability that a commit reflects corrective maintenance. We show that this metric agrees with developers' concept of quality, informative, and stable. Corrective commits are identified by applying a linguistic model to the commit messages. Corrective commits are identified by applying a linguistic model to the commit messages. We compute the CCP of all large active GitHub projects (7,557 projects with at least 200 commits in 2019). This leads to the creation of a quality scale, suggesting that the bottom 10% of quality projects spend at least 6 times more effort on fixing bugs than the top 10%. Analysis of project attributes shows that lower CCP (higher quality) is associated with smaller files, lower coupling, use of languages like JavaScript and C# as opposed to PHP and C++, fewer developers, lower developer churn, better onboarding, and better productivity. Among other things these results support the "Quality is Free" claim, and suggest that achieving higher quality need not require higher expenses.