Forrest Shull's research while affiliated with Carnegie Mellon University and other places

Publications (186)

Article
Full-text available
The technical debt (TD) concept inspires the development of useful methods and tools that support TD identification and management. However, there is a lack of evidence on how different TD identification tools could be complementary and, also, how human-based identification compares with them. To understand how to effectively elicit TD from humans,...
Article
Full-text available
Many practitioners and academics believe in a delayed issue effect (DIE); i.e. the longer an issue lingers in the system, the more effort it requires to resolve. This belief is often used to justify major investments in new development processes that promise to retire more issues sooner. This paper tests for the delayed issue effect in 171 software...
Article
To celebrate the 200th issue of IEEE Software, Voice of Evidence uses data to examine whether it has helped bridge the gap between research and practice by extracting actionable lessons from the body of research.
Article
The guest editors of this special issue describe five articles that make up the second part of a series describing US Department of Defense software engineering efforts.
Conference Paper
Programming languages can restrict state change by preventing it entirely (immutability) or by restricting which clients may modify state (read-only restrictions). The benefits of immutability and read-only restrictions in software structures have been long-argued by practicing software engineers, researchers, and programming language designers. Ho...
Article
Every software system is potentially vulnerable in ways that aren't always imagined during development. White-collar crime involving data breaches are rampant, and governments are investigating the potential for terrorist attacks on power grids, airplanes, and other public services. Technology is a double-edged blade: although computers let us purs...
Article
This special issue offers a range of perspectives on software engineering's future from professionals working around the world in diverse areas of software. The content ranges from detailed technical articles about the research areas behind today's trends to shorter essays and opinion pieces from folks working to sharpen the focus of their own visi...
Article
Physics-based high-performance computing (HPC) engineering software applications are proving to highly effective for the development of complex innovative products such as automobiles, airplanes, and microprocessors. Over the next year and a half, CiSE will feature three issues describing the US Department of Defense (DoD) High Performance Computin...
Article
Software data analytics is key for helping stakeholders make decisions, and thus establishing a measurement and data analysis program is a recognized best practice within the software industry. However, practical implementation of measurement programs and analytics in industry is challenging. In this chapter, we discuss real-world challenges that a...
Article
Context The technical debt metaphor describes the effect of immature artifacts on software maintenance that bring a short-term benefit to the project in terms of increased productivity and lower cost, but that may have to be paid off with interest later. Much research has been performed to propose mechanisms to identify debt and decide the most app...
Conference Paper
Many programming languages provide features that express restrictions on which data structures can be changed. For example, C++ includes const and Java includes final. Languages that are in widespread use typically provide non-transitive immutability: when a reference is specified to be immutable or read-only, the object referenced can still refere...
Article
Forrest Shull writes his final column as IEEE Software editor in chief; outlining the magazine's accomplishments during his four-year tenure; introducing his successor, Diomidis Spinellis; and discussing the success of Software Experts Summit 2014 in Bangalore, India. The Web extra at http://youtu.be/yhBsX8OHqhY is a video overview of the multimedi...
Article
Editor-in-chief Forrest Shull talks about the practical application of software analytics.
Article
IEEE Software editor in chief Forrest Shull talks with author and consultant Linda Rising about the power of retrospectives for software teams. The Web extra at http://youtu.be/2Tgui-qr2AQ is an audio recording of IEEE Software editor in chief Forrest Shull talking with author and consultant Linda Rising about the power of retrospectives for softwa...
Conference Paper
Bug classification is a well-established practice which supports important activities such as enhancing verification and validation (V&V) efficiency and effectiveness. The state of the practice is manual and hence classification errors occur. This paper investigates the sensitivity of the value of bug classification (specifically, failure type clas...
Article
IEEE Software Editor-in-Chief Forrest Shull discusses his tenure and looks at the initiatives during this time. He also welcomes Robert Blumen as department editor for Software Engineering Radio and puts out a call for more hosts to contribute to the Software Engineering Radio podcast interviews.
Article
IEEE Software Editor-in-Chief Forrest Shull discusses privacy implications for mobile and cloud computing with the John Howie, chief operating officer of the Cloud Security Alliance. He also looks at the upcoming Software Experts Summit scheduled for 30 May 2014 in Bangalore, India, and discusses the 200th episode of Software Engineering Radio. The...
Article
Forrest Shull, division director at the Fraunhofer Center for Experimental Software Engineering in Maryland, shares his opinion on the profession of software engineering and the industry's current ability to credibly deliver quality systems within expected cost and schedule. He considers the many challenges faced by the Health Insurance Marketplace...
Chapter
In this chapter, we discuss recent progress and opportunities in empirical software engineering by focusing on a particular technology, Technical Debt (TD), which ties together many recent developments in the field. Recent advances in TD research are providing empiricists the chance to make more sophisticated recommendations that have observable im...
Article
IEEE Software editor-in-chief Forrest Shull discusses the software sustainability and his interview with Girish Seshagiri, the CEO of AIS, an organization that offers "firm fixed-price contracting with performance guarantees, including a lifetime warranty on software defects" in government contracts. In addition, he discusses the best paper award a...
Article
IEEE Software editor-in-chief Forrest Shull discusses the magazine's recent editorial board meeting and some of the discussions the group had about where software development is headed, including refactoring and reengineering, technical debt, measurement, cloud computing, user experiences, and effective project management. In addition, he describes...
Article
Full-text available
Background: Software systems accumulate technical debt (TD) when short-term goals in software development are traded for long term goals (e.g., quick-and-dirty implementation to reach a release date vs. a well-refactored implementation that supports the long term health of the project). Some forms of TD accumulate over time in the form of source co...
Article
IEEE Software Editor-in-Chief Forrest Shull discusses the importance of building reliable systems to interpret big data. In addition, he discusses the IBM Impact 2013 Unconference; the Software Engineering Institute's SATURN 2013 conference in which the IEEE Software Architecture in Practice Award went to Simon Brown of Coding the Architecture, for...
Article
Full-text available
Existing research is unclear on how to generate lessons learned for defect prediction and effort estimation. Should we seek lessons that are global to multiple projects or just local to particular projects? This paper aims to comparatively evaluate local versus global lessons learned for effort estimation and defect prediction. We applied automated...
Conference Paper
Developing a software product with a high level of quality that also meets budget and schedule is the main goal of any organization. This usually implies making tradeoffs among conflicting aspects like number of features to implement, user perceived quality, time-to-market, and the ability of the company to maintain and improve the system in a feas...
Article
IEEE Software Editor-in-Chief Forrest Shull discusses the value of experience reports and how they can bring practical advice and perspective that simple metrics are not always able to provide. In addition, he discusses Software Experts Summit 2013 and announces that the magazine is seeking a new multimedia editor. The first Web extra at http://you...
Conference Paper
Continual evolution of the available hardware (e.g. in terms of increasing size, architecture, and computing power) and software (e.g. reusable libraries) is the norm rather than exception. Our goal is to enable CSE developers to spend more of their time finding scientific results by capitalizing on these evolutions instead of being stuck in fixing...
Conference Paper
Full-text available
We identified and organized a number of statements about technical debt (TD Folklore list) expressed by practitioners in online websites, blogs and published papers. We chose 14 statements and we evaluated them through two surveys (37 practitioners answered the questionnaires), ranking them by agreement and consensus. The statements most agreed wit...
Conference Paper
Full-text available
Context: The technical debt (TD) concept describes a tradeoff between short-term and long-term goals in software development. While it is highly useful as a metaphor, it has utility beyond the facilitation of discussion, to inspire a useful set of methods and tools that support the identification, measurement, monitoring, management, and payment of...
Article
IEEE Software Editor-in-Chief Forrest Shull discusses the importance of having and applying professional principles in all facets of software development while also keeping them in perspective. He cites work by Philippe Kruchten, Ellen Gottesdiener and Mary Gorman to support his position. In addition, he welcomes Dr. Adam Welc to the IEEE Software...
Article
IEEE Software Editor in Chief Forrest Shull discusses his interviews with Ben Schneiderman on the social networking site the Nation of Neighbors and with Anne Rose on the International Children's Digital Library and lessons learned from both websites on how developers can best design social networks to award volunteer appreciation, thus ensuring co...
Conference Paper
Full-text available
Background: Automatic static analysis (ASA) tools examine source code to discover "issues", i.e. code patterns that are symptoms of bad programming practices and that can lead to defective behavior. Studies in the literature have shown that these tools find defects earlier than other verification activities, but they produce a substantial number of...
Article
IEEE Software Editor in Chief Forrest Shull discuss the state of research in software engineering, focusing on empirical software engineering (ESE) and the expanded goal—question—metric strategies (GQM+Strategies) to tie specific measurements to the technical goals that they address. He also welcomes Girish Suryanarayana as the magazine's newest me...
Conference Paper
A significant body of knowledge concerning software inspection practice indicates that the value of inspections varies widely both within and across organizations. Inspection effectiveness and efficiency may be affected by a variety of factors such as inspection planning, the type of software, the developing organization, and many others. In the ea...
Article
Editor in chief Forrest Shull describes the decisions made at the magazine's most recent editorial board meeting and the exciting changes to come for this publication's digital edition. He also highlights winners of recent IEEE Software-sponsored awards.
Article
Smart mobile devices have had a huge impact on the world today with new apps being produced at a prodigious rate. How we got to this point has a lot to do with the ease of use that manufacturers and app developers have achieved, which includes aspects such as quick response time, intuitive interfaces, and well-designed functionality. To explore how...
Conference Paper
When planning or controlling the system development process, a project leader needs to make decisions which take into account a number of aspects, including: availability of assets and competences, previously enacted processes in the organization, certifications the system is required to obtain, standards to comply with, interactions among process...
Conference Paper
Full-text available
To date, several methods and tools for detecting source code and design anomalies have been developed. While each method focuses on identifying certain classes of source code anomalies that potentially relate to technical debt (TD), the overlaps and gaps among these classes and TD have not been rigorously demonstrated. We propose to construct a sem...
Conference Paper
Full-text available
The management of technical debt ultimately requires decision making - about incurring, paying off, or deferring technical debt instances. This position paper discusses several existing approaches to complex decision making, and suggests that exploring their applicability to technical debt decision making would be a worthwhile subject for further r...
Article
This article presents a vision of future techniques and approaches for software assurance, based on interviews with researchers for NASA's Office of Safety and Mission Assurance, and their sense of research trends and future directions. Key components of this vision include a more constructive role for software assurance, based upon early and effec...
Article
The increasing pervasiveness of cloud computing is changing the state of the practice in software testing. In an interview with James Whittaker, an engineering director at Google, editor in chief Forrest Shull explores some of the important trends in cloud computing and their implications. The conversation covers key technology changes, such as mor...
Article
Many studies have shown that important factors and key relationships often don't hold up well when transferred from one project to another. To deal with this seeming lack of global truisms in software engineering, it helps to develop a healthy skepticism and find ways to test our beliefs in key development practices against measures collected withi...
Conference Paper
Full-text available
Background: Existing software defect classification schemes support multiple tasks, such as root cause analysis and process improvement guidance. However, existing schemes do not assist in assigning defects to a broad range of high level software goals, such as software quality characteristics like functionality, maintainability, and usability. Aim...
Article
The scientific community studying climate change uses a variety of strategies to assess the correctness of their models. These software systems represent large, sophisticated, fine-grained scientific tools. The validation practices described are thus tailored to a domain in which software and software engineering practices are useful but cannot be...
Article
EIC Forrest Shull interviews AI lead programmer Ed Beach to investigate the software engineering practices employed to create computer gaming software that is both high quality and fun. This interview focuses on the context of the best-selling Civilization series of games created by Firaxis Inc.
Article
One of the problems of software development is accurately assessing progress and the cost remaining to completion. Without a good sense of where the project is and how far it still has to go, it's just not possible to consistently manage people and resources well. And miscommunicating progress to stakeholders is among the surest ways to lose trust...
Article
Technical debt is a metaphor describing situations where developers accept sacrifices in one dimension of development (e.g. software quality) in order to optimize another dimension (e.g. implementing necessary features before a deadline). Approaches, such as code smell detection, have been developed to identify particular kinds of debt, e.g. design...
Article
Technical debt is the technical work developers owe a system, typically caused by speeding up development, e.g. before a software release. Approaches, such as code smell detection, have been developed to identify particular kinds of debt, e.g. design debt. Up until now, code smell detection has been used to help point to components that need to be...
Article
Keeping up to date with new software engineering methods, practices, and tools is challenging in the best of times, and made even more urgent by today's tough economic climate. This article discusses a survey of software developers and describes high-level themes related to the types of media that were deemed useful for staying up to date. Based on...
Article
The metaphor of technical debt, originally coined by Ward Cunninghamhas helped me recently get a handle on this type of issue. Almost invariably in software projects, developers can be so focused on accomplishing the needed functionality that the software itself grows less understandable, more complex, and harder to modify. Since this system deteri...
Article
The new editor in chief of IEEE Software magazine briefly outlines his vision for the future, including items regarding new digital content and possibilities for reader-author interaction. This EIC message also contains an erratum to the Nov./Dec. 2010 article, "Visual Tools for Software Architecture Understanding: A Stakeholder Perspective," by Al...
Article
Full-text available
What if someone argued that one of your basic conceptions about how to develop software was misguided? What would it take to change your mind? That's essentially the dilemma faced by advocates of test-driven development (TDD). The TDD paradigm argues that the basic cycle of developing code and then testing it to make sure it does what it's supposed...
Article
Watts Humphrey, one of the true pioneers of the software engineering discipline, passed away this past October. Watts Humphrey had a truly remarkable career, during which he developed or contributed to the Personal Software Process, Team Software Process, and Capability Maturity Model Integration (CMMI) framework, among many other contributions. Ma...
Article
Full-text available
Software inspections provide a proven approach to quality assurance for software products of all kinds, including requirements, design, code, test plans, among others. Common to all inspections is the aim of finding and fixing defects as early as possible, and thereby providing cost savings by minimizing the amount of rework necessary later in the...
Conference Paper
Full-text available
Identifying refactoring opportunities in software systems is an important activity in today's agile development environments. The concept of code smells has been proposed to characterize different types of design shortcomings in code. Additionally, metric-based detection algorithms claim to identify the "smelly" components automatically. This paper...
Conference Paper
Full-text available
Adapting new software processes and practices in organizational and academic environments requires training the developers and validating the applicability of the newly introduced activities. Investigating process conformance during training and understanding if programmers are able and willing to follow the specific steps are crucial to evaluating...
Article
Full-text available
peer-reviewed A seeming multitude of software process capability/maturity models (SPCMMs) have emerged, and many software engineers have had to worry about compliance with them at one time or another. Although using SPCMMs is a well-established practice, the ways they're used can vary widely. At best, they can pull together vast bodies of knowledge...
Article
This paper reports on an industrial case study in a large Norwegian Oil and Gas company (StatoilHydro ASA) involving a reusable Java-class framework and two applications that use that framework. We analyzed software changes from three releases of the reusable framework, called Java Enterprise Framework (JEF), and two applications reusing the framew...
Conference Paper
Code smells refer to commonly occurring patterns in source code that indicate poor programming practices or code decay. Detecting code smells helps developers find design problems that can cause trouble in future maintenance. Detection rules for code smells, based on software metrics, have been proposed, but they do not take domain-specific charact...
Article
As with many other industries today, software development must increasingly adapt to teams whose members work together but are geographically distributed. Many factors have contributed to this rise in distributed software development (DSD), including companies' desires to leverage skilled resources wherever they can be found and to reduce costs by...
Article
Full-text available
peer-reviewed Variability management (VM) is a fundamental activity of software product line engineering (SPLE). VM explicitly represents software artifact variations for managing dependencies among SPL variants and support their instantiations throughout the SPL life cycle. It involves complex and challenging tasks, which must be supported by effe...
Article
Full-text available
Our current research is focused on identifying system engineering approaches that address four key development challenges in a tightly constrained, rapid reaction environment: 1) changing and emerging requirements; 2) conflicting stakeholder priorities; 3) concurrent sustainment and development activities; and 4) integration of independently evolvi...
Article
Full-text available
The use of empirical studies with students in software engineering helps researchers gain insight into new or existing techniques and methods. However, due mainly to concerns of external validity, questions have been raised about the value of these types of studies. The authors of this paper draw on their experiences of conducting a large number of...
Article
Full-text available
There is an increasing emphasis on rapid development of products and services to meet immediate, mission-critical needs. In order to reduce the development cycle, "agile" approaches are needed. There are methods, proc-esses, and tools (MPTs) already in existence, and gaps where new solutions are needed that work within specific contexts. For a cont...
Article
Full-text available
A seeming multitude of software process capability/maturity models (SPCMMs) have emerged, and many software engineers have had to worry about compliance with them at one time or another. Although using SPCMMs is a well-established practice, the ways they're used can vary widely. At best, they can pull together vast bodies of knowledge about good so...
Article
The paper discusses the software metrics programs. Software metrics programs might rank among the all-time touchiest subjects in software development. Done well, a measurement program can prove an effective tool for keeping on top of development effort-especially for large, distributed projects. It can help developers feel that they have a fair and...
Article
For capturing and transferring knowledge between different projects and organizations, the concept of a Best Practice is commonly used. A similar but more general concept for knowledge capturing is often referred to as a Lesson Learned. Both best practices and lessons learned are frequently organized in the form of knowledge collections. Such colle...
Conference Paper
Full-text available
In the past decades the software engineering community has proposed a large collection of software development life cycles, models, and processes. The goal of a major set of these processes is to assure that the product is finished within time and budget, and that a predefined set of functional and nonfunctional requirements (e.g. quality goals) ar...
Article
Full-text available
There have been many claims about knowledge management's benefits in software engineering, such as decreased time and cost for development, increased quality, and better decision-making abilities. Although we can find some success stories illustrating these claims, particularly on aspects related to the systems and engineering schools, more researc...
Article
Is 25 years enough time to build up a coherent body of knowledge that can help point to useful principles? As a testbed for helping us answer this question, software testing techniques are a good place to start. Few software practices are as important as testing, and testing techniques are amenable to measurement and reasoning about their effective...
Article
Full-text available
One challenge in software engineering education is to give students sufficient hands-on experience in actually building software. This is necessary so that students can understand which practices and techniques are useful in various situations. Some researchers have advocated alternative teaching methods to help in this regard. If successful, such...
Article
The systems engineering research center university affiliated research center (SERC-UARC) at Stevens Institute has been tasked to evaluate the effectiveness of the systems and software engineering processes, methods and tools (MPTs) used in US department of defense acquisition and development programs. This paper presents the selection and evaluati...
Article
Effective teaching of software engineering is a challenging endeavor, and some researchers have suggested alternative teaching methods to help. This column looks at the empirical evidence for one such method, game-based learning, and reports on a literature review that looked at what games are available as well as the degree to which their efficacy...
Article
The classroom is a valuable resource for conducting software engineering experiments. However, coordinating a family of experiments in classroom environments presents a number of challenges to researchers. Understanding how to run such experiments, developing procedures to collect accurate data, and collecting data that is consistent across multipl...
Conference Paper
Full-text available
There is widespread belief in the computer science community that MPI is a difficult and time-intensive approach to developing parallel software. Nevertheless, MPI remains the dominant programming model for HPC systems, and many projects have made effective use of it. It remains unknown how much impact the use of MPI truly has on the productivity o...
Article
Software inspections, reviews, and walkthroughs have become a standard process component in many software development domains. Maturity level 3 of the CMM-I requires establishment of peer reviews [12] and substantial sustained improvements in quality and productivity have been reported as a result of using reviews ([16], [21], [22], [27]). The NSF...
Article
Software inspections, reviews, and walkthroughs have become a standard process component in many software development domains. Maturity level 3 of the CMM-I requires establishment of peer reviews [12] and substantial sustained improvements in quality and productivity have been reported as a result of using reviews ([16], [21], [22], [27]). The NSF...
Conference Paper
Full-text available
This paper describes our experience in aggregating a number of historical datasets containing inspection defect data using different categorization schemes. Our goal was to make use of the historical data by creating models to guide future development projects. We describe our approach to reconciling the different choices used in the historical dat...
Article
Full-text available
Studies of computational scientists developing software for high-performance computing systems indicate that these scientists face unique software engineering issues. Previous failed attempts to transfer SE technologies to this domain haven't always taken these issues into account. To support scientific-software development, the SE community can di...
Conference Paper
Full-text available
This paper reports on an industrial case study in a large Norwegian Oil and Gas company (StatoilHydro ASA) involving a reusable Java-class framework and an application that uses that framework. We analyzed software changes from three releases of the framework and the application. On the basis of our analysis of the data, we found that perfective an...
Article
Full-text available
A rich body of experiences hasn't yet been published on all the software development techniques researchers have proposed. In fact, by some estimates, the techniques for which we do have substantial experience are few and far between. When we started looking at the evidence on model-based testing (MBT), we thought we'd come across some strong studi...
Article
Full-text available
Getting the requirements right is one of the most important activities in software development. Making a crucial misstep at this phase can easily lead to large amounts of rework when the customer simply can't accept a system the way it was developed. When used correctly, approaches such as incremental development or agile methods can mitigate the r...
Article
Goals of replication in empirical software engineering. Experimental replications are performed for the same reasons that independent experiments are conducted, to better understand software engineering phenomena and how to improve the practice of software development. One important benefit of replications is that they help mature software engineer...
Conference Paper
Full-text available
Experimental replications are very important to the advancement of empirical software engineering. Replications are one of the key mechanisms to confirm previous experimental findings. They are also used to transfer experimental knowledge, to train people, and to expand a base of experimental evidence. Unfortunately, experimental replications are d...
Article
If we're going to have a column about evidence in software engineering, we're going to need to talk about inspections sooner or later. Inspections are among the most mature and perhaps best-studied practices in software engineering. In short, software inspection was one of those rare software engineering innovations that had the ability to effect r...
Book
Empirical studies have become an integral element of software engineering research and practice. This unique text/reference includes chapters from some of the top international empirical software engineering researchers and focuses on the practical knowledge necessary for conducting, reporting and using empirical methods in software engineering. Pa...
Chapter
As emphasized in other chapters of this book, useful results in empirical software engineering require a variety of data to be collected through different studies – focusing on a single context or single metric rarely tells a useful story. But, in each study, the requirements of the local context are liable to impose different constraints on study...
Conference Paper
Experiments have been conducted to investigate analysis, design, implementation, testing, maintenance, quality assurance and reuse techniques, but, a body of evidence has not yet been built that enables a project manager to know with confidence what software processes produce what product characteristics and under what conditions. This paper extend...
Article
Full-text available
Pair programming is a collaborative approach that makes working in pairs rather than individually the primary work style for code development. Because PP is a radically different approach than many developers are used to, it can be hard to predict the effects when a team switches to PP. Because projects focus on different things, this article conce...
Conference Paper
Full-text available
Providing robust decision support for software engineering (SE) requires the collection of data across multiple contexts so that one can begin to elicit the context variables that can influence the results of applying a technology. However, the task of comparing contexts is complex due to the large number of variables involved. This works extends a...