D. Richard Kuhn

National Institute of Standards and Technology | NIST · Computer Security Division

Distributed ledger technology (DLT), including blockchain, has a number of properties that make it useful for distributed systems. However, the immutability of blockchain and most forms of DLT make it impossible to delete data, as is required for compliance with many privacy rules regarding personally identifiable information. Thus, there is a nee...

Industry continues to be challenged when attempting to share data among organizations, especially when the data comes from different database management systems (DBMS) and different DBMS schemas. Another concern is that privacy laws may require some types of data to be protected under local access policies. We describe a secure data sharing solutio...

We briefly review properties of explainable AI proposed by various researchers. We take a structural approach to the problem of explainable AI, examine the feasibility of these aspects and extend them where appropriate. Afterwards, we review combinatorial methods for explainable AI which are based on combinatorial testing-based approaches to fault...

With the broader adoption of AI-enabled software systems, it is necessary to provide assurance to the layman user that the AI system will behave as intended. This interactive tutorial will provide an overview of AI assurance, introduce a new set of assurance goals for AI systems, discuss the open challenges in AI assurance, and present recommendati...

Testing Internet of Things (IoT) systems is challenging. This is not only because of the various aspects of IoT systems, such as software, hardware, and network that need to be tested, but also because of the unexpected issues caused by a large number of heterogeneous devices brought together by IoT systems. When an IoT system has hundreds, or even...

Wide-scale adoption of intelligent algorithms requires artificial intelligence (AI) engineers to provide assurances that an algorithm will perform as intended. In this article, we discuss the formalization of important aspects of AI assurance, including its key components.

In this paper, we report on applying combinatorial testing to Internet of Things (IoT) home automation hub systems. We detail how to create a dedicated input parameter model of an IoT home automation hub system for use with combinatorial test case generation strategies. Further, we developed an automated test execution framework and two test oracle...

In the twenty-first century, our life will increasingly depend on software-based products and complex interconnected systems. Thus, the quality and security of software-based systems is a world-wide concern. Combinatorial testing is a versatile methodology for finding errors (bugs) and vulnerabilities in software-based systems. This paper offers a...

A correction to this paper has been published: https://doi.org/10.1007/s11786-021-00502-7

We propose a two-step TLS-based fingerprinting approach using combinatorial sequences and properties of TLS handshake messages. Our approach combines fingerprinting based on attributes of the initial ClientHello message with the observed behavior of TLS clients when presented with permuted handshake messages in order to enhance the granularity of t...

Explainable artificial intelligence (XAI) is a crucially important aspect of improving trust and reliability of artificial intelligent and machine learning (AI/ML) systems. In this special issue, we have included a diverse group of articles addressing different aspects of XAI in a variety of domains. From real-time systems to human-in-the-loop faul...

The adequacy of a test suite is typically assessed with respect to a criterion such as, for example, requirements coverage or code coverage. This paper presents a metric for the adequacy of a test suite with respect to the modeled test space. Most failures in real-life systems involve only a few parameters. So, a useful criterion for the coverage o...

Recent advancements in the field of deep learning have enabled its application in Autonomous Driving Systems (ADS). A Deep Neural Network (DNN) model is often used to perform tasks such as pedestrian detection, object detection, and steering control in ADS. Unfortunately, DNN models could exhibit incorrect or unexpected behavior in real-world scena...

Machine Learning (ML) models, a core component to artificial intelligence systems, often come as a black box to the user, leading to the problem of interpretability. Explainable Artificial Intelligence (XAI) is key to providing confidence and trustworthiness for machine learning-based software systems. We observe a fundamental connection between XA...

Combinatorial testing typically considers a single input model and creates a single test set that achieves $t$ -way coverage. This paper addresses the problem of combinatorial test generation for multiple input models with shared parameters. We formally define the problem and propose an efficient approach to generating multiple test sets, one for...

Sharing data between organizations is difficult due to different database management systems imposing different schemas as well as security and privacy concerns. We leverage two proven NIST technologies to address the problem: Next Generation Database Access Control and the data block matrix.

There is an opportunity to solve the database sharing problem of clinical trial data, while protecting proprietary, PII and other sensitive data through the integration of two proven NIST technologies: Next Generation Database Access Control (NDAC) and the data block matrix.

Understanding fault types can lead to novel approaches to debugging and runtime verification. Dealing with complex faults, particularly in the challenging area of embedded systems, craves for more powerful tools, which are now becoming available to engineers.

Embedded systems are becoming ubiquitous companions in all our lives. This article reviews the terminology and modern understanding of complex anomalies and state-of-the-art debugging. It details sophisticated omniscient debugging and runtime verification and describes a novel technique to combine the benefits of those processes.

Studies have shown that combinatorial testing (CT) can be effective for detecting faults in software systems. By focusing on the interactions between different factors of a system, CT shows its potential for detecting faults, especially those that can be revealed only by the specific combinations of values of multiple factors (multi-factor faults)....

Testing is the most commonly used approach for software assurance, yet it remains as much judgment and art as science. We suggest that structural coverage measures must be supplemented with measures of input space coverage, providing a means of verifying that an adequate input model has been defined.

There are multiple options for communication of data to and from mobile sensors. For tracking systems, Global Navigation Satellite System (GNSS) is often used for localization and mobile-phone technologies are used for transmission of data. Low-power wide area networks (LPWAN) is a newer option for sensor networks including mobile sensors. We devel...

Presents the views of five experts who examine distributed ledger technology and blockchain, discussing their status in terms of adoption and success.

We present a combinatorial coverage measurement analysis for test vectors provided by the NIST Cryptographic Algorithm Validation Program (CAVP), and in particular for test vectors targeting the AES block ciphers for different key sizes and cryptographic modes of operation. These test vectors are measured and analyzed using a combinatorial approach...

In this paper, we report on the applicability of combinatorial sequence testing methods to the problem of fingerprinting browsers based on their behavior during a TLS handshake. We created an appropriate abstract model of the TLS handshake protocol and used it to map browser behavior to a feature vector and use them to derive a distinguisher. Using...

Combinatorial interaction testing (CIT) is a well-known technique, but the industrial experience is needed to determine its effectiveness in different application domains. We present a case study introducing a unified framework for generating, executing and verifying CIT test suites, based on the open-source Avocado test framework. In addition, we...

Combinatorial interaction testing (CIT) is a well-known technique, but the industrial experience is needed to determine its effectiveness in different application domains. We present a case study introducing a unified framework for generating, executing and verifying CIT test suites, based on the open-source Avocado test framework. In addition, we...

Distributed ledger technology (DLT) offers new and unique advantages for information systems, but some of its features are not a good fit for many applications. We review the properties of DLT and show how two recently developed ideas can be used to retain its advantages while simplifying design.

Distributed ledger technology (DLT) offers new and unique advantages for information systems, but some of its features are not a good fit for many applications. We review the properties of DLT and show how two recently developed ideas can be used to retain its advantages while simplifying design.

This draft white paper identifies seventeen technical trust-related issues that may negatively impact the adoption of IoT products and services. The paper offers recommendations for mitigating or reducing the effects of these concerns while also suggesting additional areas of research regarding the subject of “IoT trust.” This document is intended...

As big data, cloud computing, grid computing, and the Internet of Things reshape current data systems and practices, IT experts are keen to harness the power of distributed systems to boost security and prevent fraud. How can these systems’ capabilities be used to improve processing without inflating risk?

Describes a data structure that provides integrity protection for distributed ledger systems while also allowing deletion of records, preserving hash-based integrity assurance that other blocks are unchanged. The datablock matrix data structure could be useful in meeting privacy requirements such as the European Union General Data Protection Regula...

Combinatorial testing has been shown to be a very effective strategy for software testing. After a failure is detected, the next task is to identify one or more faulty statements in the source code that have caused the failure. In this paper, we present a fault localization approach, called BEN, which produces a ranking of statements in terms of th...

Cryptographic hash functions are security-critical algorithms with many practical applications, notably in digital signatures. Developing an approach to test them can be particularly difficult, and bugs can remain unnoticed for many years. We revisit the National Institute of Standards and Technology hash function competition, which was used to dev...

Will our smart devices betray us? Can we trust our smart beds, pet feeders, and watches to maintain the level of privacy we want and expect? As the numbers of devices coming online reach staggering levels, serious questions must be raised about the level of cybertrust we can reasonably expect to have in our modern lifestyle.

Most security vulnerabilities result from ordinary coding errors. What does this mean for the prospects of more secure software?

This note describes a data structure, which can be referred to as a block matrix, that supports the ongoing addition of hash-linked records while also allowing the deletion of arbitrary records, preserving hash-based integrity assurance that other blocks are unchanged. The block matrix data structure may have utility for incorporation into applicat...

In the Internet of Things (IoT), what can we measure? The authors explore how the field of metrology might be applicable to the IoT.

Successfully addressing the cybersecurity needs of new technologies is not an easy task, but advances in data analytics, forensics, threat modeling, and other techniques presented in this special issue on Cyberthreats and Security can help us meet the challenge.

This note describes a data structure that provides the capability of deleting specified blocks while retaining hash-based assurance that other blocks are unchanged. It is primarily designed to be implemented in a permissioned infrastructure, providing certain features of existing permissioned blockchains.

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost, but when are these methods practical and cost-effective? This tutorial comprises two parts. The first introductory part will briefly explain the background, process, and tools available for combinatorial testing, including illustrations based on...

The analysis reported in this poster developed from questions that arose in discussions of the Reducing Software Vulnerabilities working group, sponsored by the White House Office of Science and Technology Policy in 2016 [1]. The key question we sought to address is the degree to which vulnerabilities arise from ordinary program errors, which may b...

Key finding: About 2/3 of software vulnerabilities are errors that should be detectable in ordinary code reviews and testing

Analysis of the US National Vulnerability Database indicates that about two-thirds of vulnerabilities result from ordinary coding errors, not security-specific errors. This finding suggests that significant improvement could be achieved through greater use of static analysis, code inspection, and dynamic testing.

Introduction to combinatorial testing.

Six senior computer science educators answer questions about the current state of computer science education, software engineering, and licensing software engineers.

Six panelists (Phillip A. Laplante, Michael Lewis, Keith Miller, Jeff Offutt, Jon George Rokne, and Shiuhpyng Shieh) debate whether university computer science education is leading technology forward, or commercial technology demands are leaving these programs in the dust.

Given the large and impactful data breaches making headlines in recent years, Internet users naturally wonder: Why is this happening, and how much worse can it get? Here, the authors review trends in vulnerabilities, looking at earlier findings discussed in a previous installment of this column, as well as more recent trends. They also highlight pr...

A panel of seven experts discuss the past 40 years of software metrics, with a focus on evidence-based methods.