John Hughes

Chalmers University of Technology · Department of Computer Science and Engineering

Property-based testing uses randomly generated inputs to validate high-level program specifications. It can be shockingly effective at finding bugs, but it often requires generating a very large number of inputs to do so. In this paper, we apply ideas from combinatorial testing , a powerful and widely studied testing methodology, to modify the dist...

Property-based testing tools test software against a specification, rather than a set of examples. This tutorial paper presents five generic approaches to writing such specifications (for purely functional code). We discuss the costs, benefits, and bug-finding power of each approach, with reference to a simple example with eight buggy variants. The...

This book constitutes revised selected papers from the 21st International Symposium on Trends in Functional Programming, TFP 2020, which was held in Krakow, Poland, during February 13-14, 2020. The 11 full papers presented in this volume were carefully reviewed and selected from 22 submissions. They were organized in topical sections named: domain-...

Developing a static type system suitable for Erlang has been of ongoing interest for almost two decades now. The challenge with retrofitting a static type system onto a dynamically typed language, such as Erlang, is the loss of flexibility in programming offered by the language. In light of this, many attempts to type Erlang trade sound type checki...

Formal specifications of software applications are hard to understand, even for domain experts. Because a formal specification is abstract, reading it does not immediately convey the expected behaviour of the software. Carefully chosen examples of the software’s behaviour, on the other hand, are concrete and easy to understand—but poorly-chosen exa...

In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics known as branching processes, and show how they help to analytically predict (at compile-time) the ex...

In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics known as branching processes, and show how they help to analytically predict (at compile-time) the ex...

This paper is a foreword for the Special Section of Future Generation Computing Systems journal on Functional Paradigm for High-Performance Computing connected with Lambda Days 2017 Conference. In this paper the substance of the special section is located in the current state of the art and the overviews of the four papers constituting this special...

Property-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific language in which generators are conveniently expressed by decorating predicates with lightweight ann...

We demonstrate a specific method and technology for model-based testing of large software projects with the QuickCheck tool using property-based specifications. Our specifications are very precise, state-full models of the software under test (SUT). In our approach we define (a) formal descriptions of valid function call sequences (public API), (b)...

We demonstrate a specific method and technology for model-based testing of large software projects with the QuickCheck tool using property-based specifications. Our specifications are very precise, state-full models of the software under test (SUT). In our approach we define (a) formal descriptions of valid function call sequences (public API), (b)...

Property-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific language in which generators are conveniently expressed by decorating predicates with lightweight ann...

Property-based random testing a la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific language in which generators are conveniently expressed by decorating predicates with lightweight ann...

We present a framework for automatic grading of programming exercises using property-based testing, a form of model-based black-box testing. Models are developed to assess both the functional behaviour of programs and their algorithmic complexity. From the functional correctness model a large number of test cases are derived automatically. Executin...

Random testing is increasingly popular and successful, but tends to spend most time rediscovering the "most probable bugs" again and again, reducing the value of long test runs on buggy software. We present a new automated method to adapt random test case generation so that already-discovered bugs are avoided, and further test effort can be devoted...

We address the question: to what extent does covering requirements ensure that a test suite is effective at revealing faults? To answer it, we generate minimal test suites that cover all requirements, and assess the tests they contain. They turn out to be very poor—ultimately because the notion of covering a requirement is more subtle than it appea...

This is not a typical scientific paper. It does not present a new method, with careful experiments to evaluate it, and detailed references to related work. Rather, it recounts some of my experiences over the last 15 years, working with QuickCheck, and its purpose is as much to entertain as to inform. QuickCheck is a random testing tool that Koen Cl...

QuickCheck allows us to verify software against particular properties. A property can be regarded as an abstraction over many unit tests. QuickCheck uses generated random input data to test such properties. If a counterexample is found, it becomes immediately clear what we have tested. This is not the case when all tests pass, since we do not (and...

In 1989 when functional programming was still considered a niche topic, Hughes wrote a visionary paper arguing convincingly ‘why functional programming matters’. More than two decades have passed. Has functional programming really mattered? Our answer is a resounding ‘Yes!’. Functional programming is now at the forefront of a new generation of prog...

AUTOSAR (AUTomotive Open System ARchitecture) is an evolving standard for embedded software in vehicles, defined by the automotive industry, and implemented by many different vendors. A modern car may contain over 100 processors, running AUTOSAR software from a variety of different suppliers, which must work together for the car to function. On beh...

QuickCheck can test a system by specifying a state machine for the API exported by that system. This state machine specification includes a list of possible API calls. Each call is accompanied by a precondition, a postcondition, a generator for the arguments, and a description of how the state is changed. Based on this specification QuickCheck gene...

This paper presents a generic method for randomly generating well-typed expressions. It starts from a specification of a typing judgment in PLT Redex and uses a specialized solver that employs randomness to find many different valid derivations of the judgment form. Our motivation for building these random terms is to more effectively falsify conje...

We present a semantics of mocking, based on a process calculus-like formalism, and an associated mocking framework. We can build ex-pressive mocking specifications from a small, orthogonal set of operators. Our framework detects and rejects ambiguous specifications as a valida-tion measure. We report our experience testing software components for t...

Information-flow control mechanisms are difficult to design and labor intensive to prove correct. To reduce the time wasted on proof attempts doomed to fail due to broken definitions, we advocate modern random testing techniques for finding counterexamples during the design process. We show how to use QuickCheck, a property-based random-testing too...

Information-flow control mechanisms are difficult to design and labor intensive to prove correct. To reduce the time wasted on proof attempts doomed to fail due to broken definitions, we advocate modern random testing techniques for finding counterexamples during the design process. We show how to use QuickCheck, a property-based random-testing too...

In 2009, Claessen et al. presented a way of testing for race conditions in Erlang programs, using QuickCheck to generate parallel tests, a randomizing scheduler to provoke races, and a sequential consistency condition to detect failures of atomicity [1]. That work used a small industrial prototype as the main example, showing how two race condition...

Race conditions are notoriously frustrating to find, and good tools can help. The main difficulty is reliably provoking the race condition. In previous work we presented a randomising scheduler for Erlang that helps with this task. In a language without pervasive shared mutable state, such as Erlang, performing scheduling decisions at random uncove...

This paper considers random testing of a compiler, using randomly generated programs as inputs, and comparing their behaviour with and without optimisation. Since the generated programs must compile, then we need to take into account syntax, scope rules, and type checking during our random generation. Doing so, while attaining a good distribution o...

QuickCheck is a tool which tests software against a formal specification, reporting discrepancies as minimal failing examples. QuickCheck uses properties specified by the developer both to generate test cases, and to identify failing tests. A property such as ∀xs : list(int()). reverse(reverse(xs)) = xs is tested by generating random lists of integ...

This paper presents a tutorial, with extensive exercises, in the use of Quviq QuickCheck—a property-based testing tool for Erlang, which enables developers to formulate formal specifications of their code and to use them for testing. We cover the basic concepts of properties and test-data generators, properties for testing abstract data types, and...

We present QuickSpec, a tool that automatically generates algebraic specifications for sets of pure functions. The tool is based on testing, rather than static analysis or theorem proving. The main challenge QuickSpec faces is to keep the number of generated equations to a minimum while maintaining completeness. We demonstrate how QuickSpec can imp...

We present an unbiased method for measuring the relative quality of different solutions to a programming problem. Our method is based on identifying possible bugs from pro-gram behaviour through black-box testing. The main mo-tivation for such a method is its use in experimental evalu-ation of software development methods. We report on the use of o...

Asynchronous events are awkward to handle in specification-based testing. State machine specifications become very complex when variable event order, timing constraints, and timing uncertainties must all be captured. We propose an alternative formalism for specifying asynchronous behaviour based on temporal relations, designed to support more decla...

The ProTest project is an FP7 STREP on property based testing. The purpose of the project is to develop software engineering approaches to improve reliability of service-oriented networks; support fault-finding and diagnosis based on specified properties of the system. And to do so we will build automated tools that will generate and run tests, mo...

Abstract We address the problem of testing and debugging concurrent, dis- tributed Erlang applications. In concurrent programs, race condi- tions are a common,class of bugs and are very hard to find in prac- tice. Traditional unit testing is normally unable to help finding all race conditions, because their occurrence depends so much on tim- ing. T...

We address the problem of testing and debugging concurrent, distributed Erlang applications. In concurrent programs, race conditions are a common class of bugs and are very hard to find in practice. Traditional unit testing is normally unable to help finding all race conditions, because their occurrence depends so much on timing. Therefore, race co...

Protecting confidentiality of data has become increasingly important for computing systems. Information-flow techniques have been developed over the years to achieve that purpose, leading to special-purpose languages that guarantee information-flow security in programs. However, rather than producing a new language from scratch, information-flow se...

John Hughes shared his experiences of teaching functional programming at Chalmers University in Gothenburg along with the successes and the problems he faced. His highest priority was to convince students that they could write real, interesting programs in Haskell by the end of the first course. He eliminated all the course material directly irrele...

When creating software, data types are the basic bricks. Most of the time a programmer will use data types defined in library modules, therefore being tested by many users over many years. But sometimes, the appropriate data type is unavailable in the libraries and has to be constructed from scratch. In this way, new basic bricks are created, and p...

Protecting condentiality of data has become increasingly im- portant for computing systems. Information-ow techniques have been developed over the years to achieve that purpose, leading to special-purpose languages that guarantee information-o w secu- rity in programs. However, rather than producing a new language from scratch, information-o w secu...

Li and Zdancewic have recently proposed an approach to provide information-flow security via a library rather than producing a new language from the scratch. They have shown how to implement such a library in Haskell by using arrow combinators. However, their approach only works with computations that have no side-effects. In fact, they leave as an...

This paper describes the history of Haskell, including its g enesis and principles, technical contributions, implementation s and tools, and applications and impact.

One of the nice things about purely functional languages is that functions often satisfy simple properties, and enjoy simple algebraic relationships. Indeed, if the functions of an API satisfy elegant laws, that in itself is a sign of a good design—the laws not only indicate conceptual simplicity, but are useful in practice for simplifying programs...

A major difficulty for tracking information flow in multithreaded programs is due to the internal timing covert channel. Information is leaked via this channel when secrets affect the timing behavior of a thread, which, via the scheduler, affects the interleaving of assignments to public variables. This channel is particularly dangerous because, in...

We present a case study in which a novel testing tool, Quviq QuickCheck, is used to test an industrial implementation of the Megaco protocol. We considered positive and negative testing and we used our developed specification to test an old version in order to estimate how useful QuickCheck could potentially be when used early in development. The r...

We apply the categorical properties of polymorphic functions to compile-time analysis, specifically projection-based strictness analysis. First we interpret parameterised types as functors in a suitable category, and show that they preserve monics and epics. Then we define strong and weak polymorphism — the latter admitting certain projections that...

Program analyses are often presented as one of two brands: forwards or backwards. In this paper we explore the significance of the direction of analysis, and show how arbitrary abstract interpretations may be reversed.

In this chapter we have considered the design of combinator libraries. We saw how studying the algebraic properties of the combinators desired can both help to suggest natural choices of representation, and guide the implementation of the operators. We saw several examples — lists, monads, and a pretty-printing library. For this kind of program dev...

Functional programmers often reason about programs as if they were written in a total language, expecting the results to carry over to non-total (partial) languages. We justify such reasoning.Two languages are defined, one total and one partial, with identical syntax. The semantics of the partial language includes partial and infinite values, and a...

We justify reasoning about non-total (partial) functional languages using methods seemingly only valid for total ones; this permits "fast and loose" reasoning without actually being loose.

Proof assistants based on dependent type theory are closely related to functional programming languages, and so it is tempting to use them to prove the correctness of functional programs. In this paper, we show how Agda, such a proof assistant, can be used to prove theorems about Haskell programs. Haskell programs are translated into an Agda model...

Consider this simple Haskell definition, of a function which counts the number of occurrences of a given word w in a string: count w = length . filter (==w) . words This is an example of “point-free” programming style, where we build a function by composing others, and make heavy use of higher-order functions such as filter. Point-free programming...

We present the derivation of a space efficient parser combinator library: the constructed parsers do not keep unnecessary references to the input, produce online results and efficiently handle ambiguous grammars. The underlying techniques can be applied in many contexts where traditionally backtracking is used.We present two data types, one for kee...

We present the derivation of a space efficient parser combinator library: the constructed parsers do not keep unnecessary references to the input, produce online results and efficiently handle ambiguous grammars. The underlying techniques can be applied in many contexts where traditionally backtracking is used.We present two data types, one for kee...

Type specialisation is an approach to program specialisation that works with both a program and its type to produce specialised versions of each. As it combines many powerful features,it appears to be a good framework for automatic program production, -- despite the fact that it was designed originally to express optimal specialisation for interpre...

QuickCheck is a previously published random testing tool for Haskell programs. In this paper we show how to use it for testing monadic code, and in particular imperative code written using the ST monad. QuickCheck tests a program against a speci cation: we show that QuickCheck's speci - cation language is suciently powerful to represent common form...

It is a very undesirable situation that today's software often contains errors. One motivation for using a functional programming language is that it is more di cult (or even impossible) to make low-level mistakes, and it is easier to reason about programs. But even the most advanced functional programmers are not infallible; they misunderstand the...

Haskell today provides good support not only for a functional programming style, but also for an imperative one. Elements of imperative programming are needed in applications such as web servers, or to provide efficient implementations of well-known algorithms, such as many graph algorithms. But one element of imperative programming, the global var...

. A tension in language design has been between simple semantics

Offline partial evaluators specialise programs with annotations which distinguish specialisation-time (or static) computations from run-time ones. These annotations are generated by a binding-time analyser, via type inference in a suitable type-system. Henglein and Mossin developed a type system which allows polymorphism in binding-times, so that t...

Tag elimination is a program transformation for removing unnecessary tagging and untagging operations from automatically generated programs. Tag elimination was recently proposed as having immediate applications in implementations of domain specific languages (where it can give a two-fold speedup), and may provide a solution to the long standing pr...

A tension in language design has been between simple semantics on the one hand, and rich possibilities for side-effects, exception handling and so on on the other. The introduction of monads has made a large step towards reconciling these alternatives. First proposed by Moggi as a way of structuring semantic descriptions, they were adopted by Wadle...

Hitherto all partial evaluators have processed a complete program to produce a complete residual program. We are interested in treating programs as collections of modules which can be processed independently: ‘separate partial evaluation’, so to speak. In this paper we still assume that the original program is processed in its entirety, but we show...

QuickCheck is a tool which aids the Haskell programmer in formulating and testing properties of programs. Properties are discribed as Haskell functions, and can be automatically tested on random input, but it is also possible to define custom test data generators. We present a number of case studies, in which the tool was successfully used, and als...

Monads have become very popular for structuring functional programs since Wadler introduced their use in 1990. In particular, libraries of combinators are often based on a monadic type. Such libraries share (in part) a common interface, from which numerous benefits flow, such as the possibility to write generic code which works together with any li...

We present a natural semantics that models the untyped, normal order -calculus plus McCarthy's amb in the context of call-by-need parameter passing. This results in a singular semantics for amb. Previous work on singular choice has concentrated on erratic choice, a less interesting nondeterministic choice operator, and only in relation to callby -v...

Graph reduction underlies most implementations of lazy functional languages, allowing separate computations to share results when subterms are evaluated. Once a term is evaluated, the node of the graph representing the computation is updated with the value of the term. However, in many cases, no other computation requires this value, so the update...

QuickCheck is a tool which aids the Haskell programmer in formulating and testing properties of programs. Properties are described as Haskell functions, and can be automatically tested on random input, but it is also possible to define custom test data generators. We present a number of case studies, in which the tool was successfully used, and als...

Many useful monads can be designed in a systematic way, by successively adding facilities to a trivial monad. The capabilities that can be added in this way include state, exceptions, backtracking, and output. Here we give a brief description of the trivial monad, each kind of extension, and sketches of some interesting operations that each monad s...

array :: (Ix a) =? (a,a) -? [(a,b)] -? Array a b listArray :: (Ix a) =? (a,a) -? [b] -? Array a b (!) :: (Ix a) =? Array a b -? a -? b bounds :: (Ix a) =? Array a b -? (a,a) indices :: (Ix a) =? Array a b -? [a] elems :: (Ix a) =? Array a b -? [b] assocs :: (Ix a) =? Array a b -? [(a,b)] accumArray :: (Ix a) =? (b -? c -? b) -? b -? (a,a) -? [(a,c)...

Introduction The essence of partial evaluation is beautifully simple: we just take a program, together with values of some of its inputs; we perform the operations that depend only on known inputs, build a new program from the other operations, and finally obtain a residual program which solves the same problem as the original for a subclass of the...

We extend type specialisation to a computational lambda calculus with first-class references. The resulting specialiser has been used to specialise a self-interpreter for this typed computational lambda calculus optimally. Furthermore, this specialiser can perform operations on references at specialisation time, when possible. Keywords: program tra...

Type specialisation, like partial evaluation, is an approach to specialising programs. But type specialisation works in a very different way, using a form of type inference. Previous articles have described the method and demonstrated its power as a program transformation, but its correctness has not previously been addressed. Indeed, it is not eve...

The implementations of abstract type constructors must often restrict the type parameters: for example, one implementation of sets may require equality on the element type, while another implementation requires an ordering. Haskell has no mechanism to abstract over such restrictions, which can hinder us from replacing one implementation by another,...

We present a functional language with a type system such that well typed programs run within stated space-bounds. The language is a strict, first-order variant of ML with constructs for explicit storage management. The type system is a variant of Tofte and Talpin's region inference system to which the notion of sized types, of Hughes, Pareto and Sa...

We present a functional language with a type system such that well typed programs run within stated space-bounds. The language is a strict, first-order variant of ML with constructs for explicit storage management. The type system is a variant of Tofte and Talpin's region inference system to which the notion of sized types, of Hughes, Pareto and Sa...

As software becomes more and more complex, it is more and more important to structure it well. Well-structured software is easy to write, easy to debug, and provides a collection of modules that can be re-used to reduce future programming costs. Conventional languages place conceptual limits on the way problems can be modularised. Functional langua...

Categories and Subject Descriptors: D.1.1 [Software]: Programming Techniques|applicative (functional) programming; D.3.4 [Software]: Programming Languages|processors: translator writing systems and compiler generators ; F.3.3 [Theory of Computation]: Logics and Meanings of Programs|studies of program constructs: type structure; I.2.2 [Computing Met...

Phil Wadler's idea of using monads to structure purely functional programs has had a profound effect on the way libraries of combinators are designed in Haskell. By basing the signature of such a library on a monad, the designer gains access to a wide variety of possible implementations, and moreover enables the library to be used in conjunction wi...

Hitherto all partial evaluators have processed a complete program to produce a complete residual program. We are interested in treating programs as collections of modules which can be processed independently: 'separate partial evaluation', so to speak. In this paper we still assume that the is processed in its entirety, but we show how to specialis...

Hitherto all partial evaluators have processed a complete program to produce a complete residual program. We are interested in treating programs as collections of modules which can be processed independently: 'separate partial evaluation', so to speak. In this paper we still assume that the original program is processed in its entirety, but we show...

We extend type specialisation to a computational lambda calculus with first-class references. The resulting specialiser has been used to specialise a self-interpreter for this typed computational lambda calculus optimally. Furthermore, this specialiser can perform operations on references at specialisation time, when possible.

We extend type specialisation to a computational lambda calculus with first-class references. The resulting specialiser has been used to specialise a self-interpreter for this typed computational lambda calculus optimally. Furthermore, this specialiser can perform operations on references at specialisation time, when possible. Keywords: program tra...

We present an approach for specialising large programs, such as programs consisting of several modules, or libraries. This approach is based on the idea of using a compiler generator (cogen) for creating generating extensions. Generating extensions are specialisers specialised with respect to some input program. When run on some input data the gene...

We present an approach for specialising large programs, such as programs consisting of several modules, or libraries. This approach is based on the idea of using a compiler generator (cogen) for creating generating extensions. Generating extensions are specialisers specialised with respect to some input program. When run on some input data the gene...

this paper is to propagate static information via residual types. For example, when we specialise a static integer the residual expression is a dummy value, but the residual type tells us which static value it represents:

this paper is to propagate static information via residual types. For example, when we specialise a static integer the residual expression is a dummy value, but the residual type tells us which static value it represents: 3 : int ,! ffl : 3 Here `3' is a type with only one element, namely ffl, and which therefore has just the same elements as 4, 5,...

We have designed and implemented a type-based analysis for proving some basic properties of reactive systems. The analysis manipulates rich type expressions that contain information about the sizes of recursively defined data structures. Sized types are useful for detecting deadlocks, nontermination, and other errors in embedded programs. To establ...

In this paper we present a new paradigm for partial evaluation, based not on transforming terms into terms, but on transforming terms and types into terms and types. An immediate advantage is that residual programs need not in- volve the same types as source programs, so type specialisation can be accomplished naturally. Furthermore, while a conven...

Interpreters that detect some forms of non-termination have a variety of applications, from abstract interpretation to partial evaluation. A simple and often used strategy is to test for a repeated state, but this cannot handle infinite values (such as first-class functions) or unevaluated states such as arise in lazy programs. In this paper we pro...