About
70
Publications
26,961
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
5,282
Citations
Publications
Publications (70)
Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded sys...
Communication devices such as routers and switches play a critical role in the reliable functioning of embedded system networks. Dozens of such devices may be part of an embedded system network, and they need to be tested in conjunction with various computational elements on actual hardware, in many different configurations that are representative...
Westermo Research and Development has developed SuiteBuilder, an automated tool to determine an effective ordering of regression test cases. The ordering is based on factors such as fault detection success, the interval since the last execution, and code modifications. SuiteBuilder has enabled Westermo to overcome numerous regression-testing proble...
We provide an overview of a decade-long research program aimed at identifying the most fault-prone files of large industrial software systems. We describe the motivation, approach used and results observed when we applied this technology to 170 releases of 9 different systems running continuously in the field. In all cases the files predicted to be...
In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artif...
Previous research has provided evidence that a combination of static code metrics and software history metrics can be used to predict with surprising success which files in the next release of a large system will have the largest numbers of defects. In contrast, very little research exists to indicate whether information about individual developers...
This chapter is a survey of close to ten years of software fault prediction research performed by our group. We describe our initial motivation, the variables used to make predictions, provide a description of our standard model based on Negative Binomial Regression, and summarize the results of using this model to make predictions for nine large i...
Previous studies have shown that software code attributes, such as lines of source code, and history information, such as
the number of code changes and the number of faults in prior releases of software, are useful for predicting where faults
will occur. In this study of two large industrial software systems, we investigate the effectiveness of ad...
In earlier studies of multiple-release systems, we observed that the number of changes and the number of faults in a file in the past release, the size of a file, and the maturity of a file are all useful predictors of the file's fault proneness in the next release. In each case the data needed to make predictions have been extracted from a configu...
Background: Several studies have examined code churn as a variable for predicting faults in large software systems. High churn is usually associated with more faults appearing in code that has been changed frequently.
Aims: We investigate the extent to which faults can be predicted by the degree of churn alone, whether other code characteristics oc...
Replication is a standard part of scientific experimentation. Unfortunately, in software engineering, replication of experiments is often considered an inferior type of research, or not even research at all. In this paper we describe four different types of replication that we have been performing as part of validating the effectiveness and applica...
Software developers and testers need realistic ways to measure the practical effects of using fault prediction models to guide software quality improvement methods such as testing, code reviews, and refactoring. Will the availability of fault predictions lead to discovery of different faults, or to more efficient means of finding the same faults? O...
Does the use of fault prediction models to help focus software testing resources and other development efforts to improve software reliability lead to discovery of different faults in the next release, or simply an improved process for finding the same faults that would be found if the models were not used? In this short paper, we describe the chal...
We compare the effectiveness of four modeling methods—negative binomial regression, recursive partitioning, random forests
and Bayesian additive regression trees—for predicting the files likely to contain the most faults for 28 to 35 releases of
three large industrial software systems. Predictor variables included lines of code, file age, faults in...
We compare two types of model that have been used to predict software fault-proneness in the next release of a software system. Classification models make a binary prediction that a software entity such as a file or module is likely to be either faulty or not faulty in the next release. Ranking models order the entities according to their predicted...
Background: Previous research has provided evidence that a combination of static code metrics and software history metrics can be used to predict with surprising success which files in the next release of a large system will have the largest numbers of defects. In contrast, very little research exists to indicate whether information about individua...
We have developed an interactive tool that predicts fault likelihood for the individual files of successive releases of large, long-lived, multi-developer software systems. Predictions are the result of a two-stage process: first, the extraction of current and historical properties of the system, and second, application of a negative binomial regre...
Software testing research has not kept up with modern software system designs and applications, and software engineering education falls short of providing students with the type of knowledge and training that other engineering specialties require. Testing researchers should pay more attention to areas that are currently relevant for practicing sof...
Previous studies have shown that software code attributes, such as lines of source code, and histo ry information, such as the number of code changes and the number of faults in prior releases of software, are useful for predicting where faults will occur. In this study of an industrial software system, we investig ate the effectiveness of adding i...
We have designed and implemented a tool that predicts files most likely to have defects in a future release of a large software
system. The tool builds a regression model based on the version and defect history of the system, and produces a list of the
next release’s most probable fault-prone files, sorted in decreasing order of the number of predi...
Fault prediction by negative binomial regression models is shown to be effective for four large production software systems
from industry. A model developed originally with data from systems with regularly scheduled releases was successfully adapted
to a system without releases to identify 20% of that system’s files that contained 75% of the fault...
It would obviously be very valuable to know in advance which files in the next release of a large software system are most
likely to contain the largest numbers of faults. This is true whether the goal is to validate the system by testing or formally
verifying it, or by using some hybrid approach. To accomplish this, we developed negative binomial...
A key problem when doing automated fault analysis and fault prediction from information in a software change management database is how to determine which change reports represent software faults. In some change management systems, there is no simple way to distinguish fault reports from changes made to add new functionality or perform routine main...
The PROMISE workshop seeks to deliver to the software engineering community useful, usable, verifiable, and repeatable models. To provide a sound and realistic basis for creating predictive models, and to allow researchers to conduct repeatable software engineering experiments, we maintain the PROMISE repository, a growing collection that now conta...
Two different software fault prediction models have been used to predict the N % of the files of a large software sys-tem that are likely to contain the largest numbers of faults. We used the same predictor variables in a negative binomial regression model and a recursive partitioning model, and compared their effectiveness on three large industria...
We have been investigating different prediction models to identify which files of a large multi-release industrial software system are most likely to contain the largest numbers of faults in the next release. To make predictions we considered a number of different file characteristics and change information about the files, and have built fully- au...
This research investigates ways of predicting which files would be most likely to contain large numbers of faults in the next release of a large industrial software system. Previous work involved making predictions using several different models ranging from a simple, fully-automatable model (the LOC model) to several different variants of a negati...
It is often wondered wh ys om uch of the software engineering research that appears in the literature is not adopted by practitioners. After all, there are lots of exciting ne wi deas that could potentially improve both the quality and efficienc yo f software production. So wh yi st his research ignored? Is it just am atter of ignorance or shortsig...
Many fault prediction models have been proposed in the software engineering literature, and their success evaluated according to various metrics that are widely used in the statistics community. To be able to make meaningful comparisons among the proposed models, it is important that the metrics assess meaningful properties of the predictions. We e...
Industrial experience assessing the stability of a large mission-critical software project is reported. We observed that the project incurred significant additional delays in resolving the types of problems usually uncovered when assessing mission-critical software stability. We present plausible hypotheses about the possible causes of these additi...
We continue investigating the use of a negative binomial regression model to predict which files in a large industrial software system are most likely to contain many faults in the next release. A new empirical study is described whose subject is an automated voice response system. Not only is this system's functionality substantially different fro...
This paper discusses the issues involved in building a practical automated tool to predict the incidence of software faults in future releases of a large software system. The possibility of creating such a tool is based on the authors' experience in analyzing the fault history of several large industrial software projects, and constructing statisti...
In earlier work we investigated ways of predicting which les would be most likely to contain large numbers of faults in the next release of a large industrial software system. To make predictions we considered several different models ranging from a simple, fully-automatable model requiring no statistical expertise (the LOC model) to several differ...
Attendees: M. Zelkowitz (chair), V. Basili, R. Glass, M. Host, M. Mueller, T. Ostrand, A. Rainer, C. Seaman and H. Sharp
If empirical software engineering is to prosper as a research domain, the field needs a mechanism for the sharing of data
and artifacts developed by one group that may be useful to another group desiring to work in the same area...
The goal of this research is to allow software developers and testers to become aware of which files in the next release of a large software system are likely to contain the largest numbers of faults or the highest fault densities in the next release, thereby allowing testers to focus their efforts on the most fault-prone files. This is done by dev...
Advance knowledge of which files in the next release of a large software system are most likely to contain the largest numbers of faults can be a very valuable asset. To accomplish this, a negative binomial regression model has been developed and used to predict the expected number of faults in each file of the next release of a system. The predict...
We present a numerical experimentation of the global optimization algorithm presented by Velázquez et. al. [3] applied to a nonlinear hyperboloid least squares problem. This problem arises when beta sheet residues from an allosteric enzyme are fitted ...
We investigated a different mode of using the prediction model to identify the files associated with a fixed percentage of the faults. The tester could ask the tool to identify which files are likely to contain the bulks of faults, with the tester selecting any desired percentage of faults. Again the tool would return a list ordered in decreasing o...
The ability to predict which files in a large software system are most likely to contain the largest numbers of faults in the next release can be a very valuable asset. To accomplish this, a negative binomial regression model using information from previous releases has been developed and used to predict the numbers of faults for a large industrial...
In earlier research we identified characteristics of files in large software systems that tend to make them particularly likely to contain faults. We then developed a statistical model that uses historical fault information and file characteristics to predict which files of a system are likely to contain the largest numbers of faults. Testers can u...
We perform static analysis and develop a negative binomial regression model to predict which files in a large software system are most likely to contain the largest numbers of faults that manifest as failures in the next release, using information from all previous releases. This is then used to guide the dynamic testing process for software system...
A case study is presented using thirteen releases of a large industrial inventory tracking system. Several types of questions are addressed in this study. The first involved examining how faults are distributed over the different files. This included making a distinction between the release during which they were discovered, the lifecycle stage at...
Software tests can be categorized according to a variety of criteria, including the purpose of the tests, the information source from which the tests are derived, and the life-cycle phase for which they are intended. Each of these criteria addresses a distinct, important aspect of software tests, and clarifies the role the tests play in helping ass...
Black-box testing of software is any method of generating testcases that is independent of the software's internal structure. Since black-box testing is always based, either directly or indirectly, on the software's specification, it is also called specification-based testing. The simplest concept of black-box testing is that the tester examines th...
White‐box testing refers to test methods that rely on the internal structure of the software. White‐box methods are based on executing or “covering” specific elements of the code. The essential rationale for these methods is that it is impossible to detect a fault in some piece of code by testing if that code is never executed. The ideal would be t...
A case study is presented using thirteen releases of a large industrial inventory tracking system. Several types of questions are addressed in this study. The first involved examining how faults are distributed over the different files. This included making a distinction between the release during which they were discovered, the lifecycle stage at...
We have implemented an experimental test development environment (TDE) intended to raise the effectiveness of tests produced for GUI systems, and raise the productivity of the GUI system tester.The environment links a test designer, a test design library, and a test generation engine with a standard commercial capture/replay tool. These components...
This paper reports an experimental study investigating the effectiveness of two code-based test adequacy criteria for identifying sets of test cases that detect faults. The all- edges and all-D Us (modified all-uses) coverage criteria were applied to 130 faulty program versions derived from seven moderate size base programs by seeding realistic fau...
The data f70w adequacy criteria, originally proposed for a simple language and Pascal, have been substantially modified to provide more thorough analysis for code wifh exfensive use of pointers and complex control sirucfures, such as code frequently writ fen in the C language. A prototype fool, TACTIC, has been builf fo exiracf clef-use association...
TSL is a language for writing formal test specifications of the functions of a software system. The test specifications are compiled into executable test scripts that establish test environments, assign values to input variables, perform necessary setup and cleanup operations, run the test cases, and check the correctness of test results. TSL is a...
A method for creating functional test suites has been developed in which a test engineer analyzes the system specification, writes a series of formal test specifications, and then uses a generator tool to produce test descriptions from which test scripts are written. The advantages of this method are that the tester can easily modify the test speci...
A study has been made of the software errors committed during development of an interactive special-purpose editor system. This product, developed for commercial production use, has been followed during nine months of coding, unit testing, function testing, and system testing. Detected problems and their fixes have been described by testers and deb...
Software errors detected during development of an interactive special purpose editor system were studied. This product was followed during nine months of coding, unit testing, function testing, and system testing. A new error categorization scheme was developed.
Not every unambiguous regular grammar can be parsed by a finite state machine, even if a lookahead facility is added to the machine's capabilities. Those which can be parsed with a fixed lookahead of k are said to be FL(k). If such a grammar has n non-terminals, it never needs more than (n(n-1)/2) + 1 lookahead, and there exist grammars which do re...
The theory of test data selection proposed by J. B. Goodenough and S. L. Gerhart is examined. In order to extend and refine this theory, the concepts of a revealing test criterion and a revealing subdomain are proposed. These notions are then used to provide a basis for constructing program tests. A subset of a program's input domain is revealing i...
Amoroso and Cooper have shown that for an arbitrary state alphabet A, one- and two-dimensional tessellation automata are definable which have the ability to reproduce any finite pattern contained in the tessellation space. This note shows that the same construction may be applied to tessellation spaces of any finite dimension.
We perform static analysis and develop a negative bino- mial regression model to predict which files in a large soft- ware system are most likely to contain the largest numbers of faults that manifest as failures in the next release, using information from all previous releases. This is then used to guide the dynamic testing process for software sy...
In earlier research we identied characteristics of les in large software systems that tend to make them particu- larly likely to contain faults. We then developed a statistical model that uses historical fault information and le char- acteristics to predict which les of a system are likely to contain the largest numbers of faults. Testers can use t...
