Robert V. Binder

• MBA, University of Chicago. MS, EECS U of Illinois at Chicago
• Consulting Software Engineer at RBSC Corporation

In 2002, Microsoft began the difficult process of verifying much of the technical documentation for its Windows communication protocols. The undertaking came about as a consequence of a consent decree Microsoft entered into with the U.S. Department of Justice and several state attorneys general that called for the company to make available certain...

Since I started working at the SEI, I've visited and profiled about two dozen Software Integration Labs (SIL) and stood up one for an SEI client. This report (in power point) presents an overall profile of these labs, with a focus on test automation. The aggregate profile is compared with automotive best practices. I define recommended practices f...

Thirty-six years after the first edition of IEEE standard 982.1, Measures of the Software Aspects of Dependability, the third edition focuses on the measurement of in-service software dependability. This article explains how this new point of view evolved and shaped the third edition's guidance for software dependability measurement. Download the...

Model-based testing has positive effects on efficiency and effectiveness, even if it only partially fulfills high expectations. Report on a 2014 survey of 100 users of model-based testing tools.

Although the bulk of software engineering is focused on development, nearly all of the impact and value of software is realized when it is placed into service. Likewise, the often-devastating scale and effects of software failures do not become evident unless and until the software is placed into service. These consequential software failures can o...

In project management, the iron triangle refers to a quality tradeoff, often expressed as “Good, fast, cheap. Choose two.” A similar relationship exists among the elements of dependability: reliability, availability, and supportability, i.e., the dependability triangle.

People usually speak of dependability as a whole and generally agree that more of it is a good thing. During the IEEE P982 working group’s research and discussions, we were intrigued by the idea of characterizing dependability as a single-number score derived from its constituent measurements (reliability, availability, supportability, and recovera...

Thirty-six years after the first edition of IEEE standard 982.1, Measures of the Software Aspects of Dependability, the third edition focuses on the measurement of in-service software dependability. This article explains how this new point of view evolved and shaped the third edition’s guidance for software dependability measurement.

To measure software dependability and its components—reliability, availability, supportability, and recoverability—certain basic operational questions must be answered.

Since 2021, I’ve been the chair of the working group revising IEEE standard 982.1 - Measures of the Software Aspects of Dependability. This talk presents some of the questions this has raised, how our understanding of Dependability has evolved, and summarizes our results to date. This talk is not sponsored by the IEEE – it is my own point of view....

This tutorial shows how to comply with Software Reliability Engineering (SRE) practices defined in IEEE Standard 1633, Recommended Practices on Software Reliability Engineering, when following Agile software development. The main topics are (1) a review of SRE basics, (2) a review of Agile Testing Practices, and (3) a presentation of Reliability Dr...

Test case ordering can have significant effects on the cost, duration, or safety of a test suite. As the total number of possible orderings is n! for n test cases, finding a cost-optimal ordering can be a non-trivial problem. Combinatorial algorithms that generate t-wise test suites either explicitly randomize sequence or order them as a side effec...

This presentation shows how classical integer programming can be used to determine the least cost (that is, the quickest) order of test configurations dictated by design-of-experiments or combinational coverage.

In the 1980s, testers and methodologists agreed that early involvement of testing with software development was a win-win. This led to the well-known V-model and its correlation of development and validation. The principle was carried forward in spiral and incremental process refinements. As we note in the following, Agile practices follow this tra...

The Model-based Assurance Lab is a reference environment expected to improve cyber-physical system reliability by 10x and cut testing cycle time by half. Although hardware-in-the-loop (HIL) is widely used for virtualization, testing is often limited to hand-coded test scripts and primitive test asset management. MBAL integrates MBT (model-based tes...

The relationship of software testing to software development may be likened to Sancho Panza and Don Quixote. Available evidence indicates that, compared with traditional practices, Agile testing practices have no significantly different effect on software quality.This talk analyzes common Agile testing practices to suggest why this occurs and sketc...

You have probably heard about MBT (model-based testing), but like many software-engineering professionals who have not used MBT, you might be curious about others' experience with this test-design method. From mid-June 2014 to early August 2014, we conducted a survey to learn how MBT users view its efficiency and effectiveness. The 2014 MBT User Su...

This invited talk compares the popular xUnit framework for automated devleoper testing with MTS, a test automation framework the author developed. Limitations of xUnit frameworks and TTCN are noted. The talk shows how the MTS framework addressed these limitations.

This survey of model-based testing (MBT) users presents data about the who, what, when, why, and how of MBT usage; views about previously reported obstacles; views about benefits and costs; assumptions held about success/fail factors; and views about how MBT and software process aspects are associated. The responses indicate that model-based testin...

In this paper, we present a conceptual framework and a system model for an intelligent assistant for requirement definition, KB/RMS. The requirement definition process is characterized by the Requirement Context Model. Informal and formal methods for requirement definition are considered in the light of this model, which serves as the logical schem...

This case study considers an approach to an enormous testing challenge and one of the testing methodologies used—model-based testing—and the primary challenges that have emerged in adopting that approach for a very large-scale project. Two lead engineers from the Microsoft team and an engineer who played a role in reviewing the Microsoft effort tel...

A perennial and critical question for software testing is: When can we stop? Apart from "when time is up," many test coverage measures have been proposed and used. Automated test generation from a Markov model can track model coverage. The reliability demonstration chart uses sequential sampling to evaluate when the failure intensity observed in te...

This presentation defines the testability of software systems and characterizes the factors that drive it. Strategies to improve testability are listed along with contributing factors in the organization of development teams.

The Advanced Mobile Application Testing Environment (AMATE) combines model-based test generation and evaluation, controllable RF airlink variation, and a robust standards-based distributed test harness for end-to-end testing of distributed mobile applications. This report summarizes AMATE capabilities.

The Advanced Mobile Application Testing Environment (AMATE) combines model-based test generation and evaluation, controllable RF airlink variation, and a robust standards-based distributed test harness for end-to-end testing of distributed mobile applications. This report summarizes AMATE capabilities.

Emerging information technology promises to take us from the wired web to a wireless “hive mind”. This talk considers how can we achieve the necessary reliability. Some lessons from testing object-oriented systems and my work in advanced automated testing are presented. A strategy to apply these lessons to new technical challenges concludes the tal...

Assertions cannot be inherited in Ada 95, C++, Java, Objective-C, or Smalltalk. Only Eiffel supports assertion inheritance. For example, although we can make a C++ base class invariant( ) function visible in a derived class, there is no built-in mechanism for checking base class assertions with the proper Boolean operators in a derived class. The P...

More than ever, mission-critical and business-critical applications depend on object-oriented (OO) software. Testing techniques tailored to the unique challenges of OO technology are necessary to achieve high reliability and quality. Testing Object-Oriented Systems: Models, Patterns, and Tools is an authoritative guide to designing and automating t...

Six-sigma (i.e. 6 standard deviations) is a parameter that is used in statistical models of the quality of manufactured goods (including computer hardware). It also serves as a slogan that suggests high quality. Some attempts have been made in the past to apply 6-sigma to software quality measurement. Software engineers often look to hardware analo...

Six-sigma (i.e. 6 standard deviations) is a parameter that is used in statistical models of the quality of manufactured goods (including computer hardware). It also serves as a slogan that suggests high quality. Some attempts have been made in the past to apply 6-sigma to software quality measurement. Software engineers often look to hardware analo...

The essential challenge in testing object-oriented software is to craft a practical test suite that exercises enough combinations of message sequence and state interactions to provide sufficient confidence. These interactions are often subtle and complex and therefore error-prone. If all classes were alike, it might be enough to devise tests based...

Research and practitioner literature on testing object-oriented software published up to the end of 1994 is summarized. The contribution of each source to eight topics presented: (1) abstract data type verification and testing as it relates to object-oriented testing; (2) testing theory—fault hypotheses for object-oriented software and adequate tes...

Software development is a difficult and time-consuming process. As software systems become larger and ever more complex, developers are searching for mechanisms to control that complexity. The main goals have always been to lower the cost and improve productivity, reuse, and maintainability. The procedure-oriented paradigm and structured programmin...

This paper addresses two questions: what is software testability, and how do the unique features of object-oriented programming languages and related approaches to analysis, design, and testing facilitate or hinder testability? It presents a model of software testability drivers: 1) object-oriented representations, 2) object-oriented implementation...

A conceptual framework and a system model for an intelligent assistant for requirement definition, KB/RMS, is presented. The requirement definition process is characterised by the requirements context model. Informal and formal methods for requirement definition are considered in light of this model, which serves as the logical schema for the KB/RM...

Recognized as a definitive reference for IBM mainframe programmers. Shows how to analyze run-time errors for programs written in Cobol, Assembly, PL/I, and Fortran. Featured selection of the Library of Computer and Information Sciences Book Club. Published by Prentice-Hall, 1985.

This month's column outlines an answer to a question I'm asked in every one of my testing seminars: how much time should we budget for testing in object-oriented development? There is no one-size-fits-all answer to this question. Ideally, we'd like to test until we have very good evidence that our system is sufficiently reliable for its intended us...