Figure 1 - uploaded by Herman Venter
Content may be subject to copyright.
A screenshot of the Spec# IDE. Verification errors are indicated by squigglies, and tool tips display error messages. The example could be fixed by adding the precondition a.Length > 0 . 

A screenshot of the Spec# IDE. Verification errors are indicated by squigglies, and tool tips display error messages. The example could be fixed by adding the precondition a.Length > 0 . 

Source publication
Article
Full-text available
Spec # is a programming system that facilitates the development of correct software. The Spec # language extends C # with contracts that allow programmers to express their design intent in the code. The Spec # tool suite consists of a compiler that emits run-time checks for contracts, a static program verifier that attempts to mathematically prove...

Contexts in source publication

Context 1
... Purity. As mentioned earlier, we require contracts to be pure , that is, side-effect free, to ensure that dynamic contract checking does not interfere with the execution of the rest of the program and that contracts have a simple semantics that can be encoded in the static verifier. It would be easy to forbid the use of all side-effecting operations (such as field updates) in the body of a pure method, but doing so would be too restrictive. For instance, a method called in a specification might want to use an iterator to tra- verse a collection. Creating and advancing the iterator are side effects; however, these effects are not observable when the method returns. Spec# enforces this notion of purity, called weak purity , which forbids pure methods only from changing the state of existing objects. Method Admissibility. types in C#. Purity. That The is, As compiler instead mentioned of also T! earlier, and ensures T , this we that require options specifica- con- lets tracts tions one write follow to be T and the pure T? Spec# , , that respectively. methodology. is, side-effect This free, includes to ensure enforc- that dynamic ing The limits main contract on complication what checking can be in mentioned does enforcing not interfere in non-null an object with types invariant the arises exe- during cution and what of the the things initialization rest a of pure the program method of variables. is and allowed that To contracts make to read. sure have These that a a simple admissibility constructor semantics checks does that not are can access crucial be encoded the for object sound in the static being static verification. constructed verifier. before It would its non-null be easy to fields forbid are the initialized, use of all side-effecting we developed oper- two solutions. ations (such The as field simpler updates) solution in the is to body allow of the a pure constructor method, base but doing call to so occur would anywhere be too restrictive. within a For constructor instance, body a method (not just called first). in a specification The compiler might can then want perform to use an syntactic iterator to checks tra- to verse make a collection. sure initialization Creating occurs and advancing before the base the iterator call, which are means side effects; all non-null however, fields these have effects non-null are not values observable after the when base call. the method The other returns. solution Spec# caters enforces to legacy this code notion and cyclic of purity, data structures called weak by purity a more , which sophisticated forbids pure data-flow methods analysis only [10]. from changing Importantly, the state the of compiler existing is objects. smart enough to track the flow of null and non-null values. Therefore, non-null annotations are typically applied only to fields and parameters. Admissibility. The compiler also ensures that specifications follow the Spec# methodology. This includes enforcing limits on what can be mentioned in an object invariant and what things a pure method is allowed to read. These admissibility checks are crucial for sound static verification. Large portions of Spec# contracts can be checked at runtime. For instance, except for the non-null specifications and method purity, all of the specifications shown in Fig. 0 result in run-time checks generated by the Spec# compiler. To avoid the performance overhead of run-time checks, there are compiler options for turning off the dynamic checks for some of the contracts. The program verifier flags violations both of the explicit contracts and of the runtime semantics (e.g., null-dereference, array index out of bounds, or division by zero). It checks one method at a time. If the verification fails, it displays an error message, the location of the error, the trace through the method that contains the error, and possibly a counterex- ample. The problem can then be fixed by correcting the program or, almost equally often, the specification. Fig. 1 illustrates how the IDE reports verification errors to the programmer. Our static verifier is sound, but not complete. That is, it finds all errors in the programs, but it might also complain about methods that are actually correct (so-called spurious warnings). Such spurious warnings can often be fixed by providing more comprehensive specifications. In some cases, it is necessary to add an assumption to the program, a ...
Context 2
... index out of bounds, or division by zero). It checks one method at a time. If the verification fails, it displays an er- ror message, the location of the error, the trace through the method that contains the error, and possibly a counterex- ample. The problem can then be fixed by correcting the program or, almost equally often, the specification. Fig. 1 illustrates how the IDE reports verification errors to the pro- ...

Similar publications

Article
Full-text available
We present a development support tool, called LogChamber, which infers source-code locations by analyzing run-time logs of mobile applications. During development, developers insert log functions into applications calls in order to confirm that the applications correctly run as expected. After that, they need to have a process for estimating a prog...
Chapter
Full-text available
We present the RIVERtools integrated development environment for specifying Agent Interaction Protocols (AIPs) modelled as trace expressions, and for statically verifying some of their properties. In particular, this demonstration paper aims at showing why a “good” AIP can become a “bad” one because of unreliability of some communication channels,...
Conference Paper
Full-text available
Model transformations are an important aspect of Model-Driven Engineering as models throughout the software development process are transformed and refined until, finally, application code is generated. However, model transformations are complex to build, maintain, and verify for correctness. We propose the combination of visual contracts, an imple...
Conference Paper
Full-text available
Among various static and dynamic software verification techniques, runtime assertion checking traditionally holds a particular place. Commonly used by most software developers, it can provide a fast feedback on the correctness of a property for one or several concrete executions of the program. Quite easy to realize for simple program properties, i...

Citations

... For TraitCbC, tools for post-hoc verification can be reused. There are tools for many languages such as Java [3], C [16], C# [7,8]. Other languages are integrated with their verifier from the start, e.g., Spec# [8] and Dafny [32]. ...
... For C#, programs written in the similar language Spec# [8] are verified with Boogie. That is, code and specification are translated to an intermediate language and verified [7]. For C, the tool VCC [16] reuses the Spec# tool chain to verify programs. ...
Preprint
We demonstrate that traits are a natural way to support correctness-by-construction (CbC) in an existing programming language in the presence of traditional post-hoc verification (PhV). With Correctness-by-Construction, programs are constructed incrementally along with a specification that is inherently guaranteed to be satisfied. CbC is complex to use without specialized tool support, since it needs a set of refinement rules of fixed granularity which are additional rules on top of the programming language. In this work, we propose TraitCbC, an incremental program construction procedure that implements correctness-by-construction on the basis of PhV by using traits. TraitCbC enables program construction by trait composition instead of refinement rules. It provides a programming guideline, which similar to CbC should lead to well-structured programs, and allows flexible reuse of verified program building blocks. We introduce TraitCbC formally and prove the soundness of our verification strategy. Additionally, we implement TraitCbC as a proof of concept.
... According to Hoare's vision, a verifying compiler "uses automated mathematical and logical reasoning to check the correctness of the programs that it compiles" [80]. A variety of tools have blossomed in this space, including Spec# [16], Dafny [104], Why3 [65], OpenJML [46], ESC/Java [68], VeriFast [85], SPARK/Ada [121], AutoProof for Eiffel [163], Frama-C [52], KeY [2], SPARK/Ada [13,40] and Whiley [139,159]. Automated theorem provers are integral to such tools and are responsible for discharging proof obligations [16,45,68,85]. ...
... A variety of tools have blossomed in this space, including Spec# [16], Dafny [104], Why3 [65], OpenJML [46], ESC/Java [68], VeriFast [85], SPARK/Ada [121], AutoProof for Eiffel [163], Frama-C [52], KeY [2], SPARK/Ada [13,40] and Whiley [139,159]. Automated theorem provers are integral to such tools and are responsible for discharging proof obligations [16,45,68,85]. Various satisfiability modulo theory (SMT) solvers are typically used for this, such as Z3 [53], CVC4 [19,20], Yices2 [61], Alt-Ergo [49], Vampire [82,95] or Simplify [55]. ...
... Multiple clauses are simply conjoined together. We have found that allowing multiple requires and/or ensures clauses can help readability, and note that JML [48], Spec# [16] and Dafny [104] also permit this. -Loop invariants are given by where clauses. ...
Article
Full-text available
The quest to develop increasingly sophisticated verification systems continues unabated. Tools such as Dafny, Spec#, ESC/Java, SPARK Ada and Whiley attempt to seamlessly integrate specification and verification into a programming language, in a similar way to type checking. A common integration approach is to generate verification conditions that are handed off to an automated theorem prover. This provides a nice separation of concerns and allows different theorem provers to be used interchangeably. However, generating verification conditions is still a difficult undertaking and the use of more “high-level” intermediate verification languages has become commonplace. In particular, Boogie provides a widely used and understood intermediate verification language. A common difficulty is the potential for an impedance mismatch between the source language and the intermediate verification language. In this paper, we explore the use of Boogie as an intermediate verification language for verifying programs in Whiley. This is noteworthy because the Whiley language has (amongst other things) a rich type system with considerable potential for an impedance mismatch. We provide a comprehensive account of translating Whiley to Boogie which demonstrates that it is possible to model most aspects of the Whiley language. Key challenges posed by the Whiley language included: the encoding of Whiley’s expressive type system and support for flow typing and generics; the implicit assumption that expressions in specifications are well defined; the ability to invoke methods from within expressions; the ability to return multiple values from a function or method; the presence of unrestricted lambda functions; and the limited syntax for framing. We demonstrate that the resulting verification tool can verify significantly more programs than the native Whiley verifier which was custom-built for Whiley verification. Furthermore, our work provides evidence that Boogie is (for the most part) sufficiently general to act as an intermediate language for a wide range of source languages.
... • Deductive verification and first-order reasoning: Deductive program verification [14,60,81] is probably the oldest formal method technique, dating back to 1969 [81]. In this approach, programs are annotated with logical assertions, such as pre-and post-conditions for operations or loop invariants, then so-called proof obligations are automatically generated (e.g., by the weakest precondition algorithm) in such a way that proving (a.k.a. ...
Preprint
Full-text available
While recent progress in quantum hardware open the door for significant speedup in certain key areas (cryptography, biology, chemistry, optimization, machine learning, etc), quantum algorithms are still hard to implement right, and the validation of such quantum programs is achallenge. Moreover, importing the testing and debugging practices at use in classical programming is extremely difficult in the quantum case, due to the destructive aspect of quantum measurement. As an alternative strategy, formal methods are prone to play a decisive role in the emerging field of quantum software. Recent works initiate solutions for problems occurring at every stage of the development process: high-level program design, implementation, compilation, etc. We review the induced challenges for an efficient use of formal methods in quantum computing and the current most promising research directions.
... However, this approach requires the verifier to be (re-)implemented in an interactive theorem prover (ITP) such as Coq [14] or Isabelle [24]. This precludes the free choice of implementation language and paradigm, exploitation of concurrency, and possibility of tight integration with standard compilers and IDEs, which is often desirable for program verifiers [4,5,13,26]. Both verification approaches substantially impede software maintenance, which is problematic since verifiers are often rapidly-evolving software projects (for instance, the Boogie repository [1] contains more than 5000 commits). ...
Chapter
Full-text available
A program verifier produces reliable results only if both the logic used to justify the program’s correctness is sound, and the implementation of the program verifier is itself correct. Whereas it is common to formally prove soundness of the logic, the implementation of a verifier typically remains unverified. Bugs in verifier implementations may compromise the trustworthiness of successful verification results. Since program verifiers used in practice are complex, evolving software systems, it is generally not feasible to formally verify their implementation. In this paper, we present an alternative approach: we validate successful runs of the widely-used Boogie verifier by producing a certificate which proves correctness of the obtained verification result. Boogie performs a complex series of program translations before ultimately generating a verification condition whose validity should imply the correctness of the input program. We show how to certify three of Boogie’s core transformation phases: the elimination of cyclic control flow paths, the (SSA-like) replacement of assignments by assumptions using fresh variables (passification), and the final generation of verification conditions. Similar translations are employed by other verifiers. Our implementation produces certificates in Isabelle, based on a novel formalisation of the Boogie language.
... This research was supported by grants from WHJIL and NSERC CRDPJ 543583-19. 1 By continuous verification, we mean verification that is integrated with continuous integration (CI) and is checked during every commit. ...
Preprint
Full-text available
A recent case study from AWS by Chong et al. proposes an effective methodology for Bounded Model Checking in industry. In this paper, we report on a follow up case study that explores the methodology from the perspective of three research questions: (a) can proof artifacts be used across verification tools; (b) are there bugs in verified code; and (c) can specifications be improved. To study these questions, we port the verification tasks for $\texttt{aws-c-common}$ library to SEAHORN and KLEE. We show the benefits of using compiler semantics and cross-checking specifications with different verification techniques, and call for standardizing proof library extensions to increase specification reuse. The verification tasks discussed are publicly available online.
... However, this approach requires the verifier to be (re-)implemented in an interactive theorem prover (ITP) such as Coq [14] or Isabelle [24]. This precludes the free choice of implementation language and paradigm, exploitation of concurrency, and possibility of tight integration with standard compilers and IDEs, which is often desirable for program verifiers [4,5,13,26]. Both verification approaches substantially impede software maintenance, which is problematic since verifiers are often rapidly-evolving software projects (for instance, the Boogie repository [1] contains more than 5000 commits). ...
... A is given by j >= 0 ∧ (i = 0 ⇒ j > 0).(only) loop head here, and the edge from B 5 to it the only back-edge (completing looping paths via B 2 and B 3 or B 2 and B 4 ). An assert A statement starting a loop head (like B 1 ) is interpreted as declaring A to be the loop invariant.5 The CFG-to-DAG phase performs the following steps:1. ...
Preprint
A program verifier produces reliable results only if both the logic used to justify the program's correctness is sound, and the implementation of the program verifier is itself correct. Whereas it is common to formally prove soundness of the logic, the implementation of a verifier typically remains unverified. Bugs in verifier implementations may compromise the trustworthiness of successful verification results. Since program verifiers used in practice are complex, evolving software systems, it is generally not feasible to formally verify their implementation. In this paper, we present an alternative approach: we validate successful runs of the widely-used Boogie verifier by producing a certificate which proves correctness of the obtained verification result. Boogie performs a complex series of program translations before ultimately generating a verification condition whose validity should imply the correctness of the input program. We show how to certify three of Boogie's core transformation phases: the elimination of cyclic control flow paths, the (SSA-like) replacement of assignments by assumptions using fresh variables (passification), and the final generation of verification conditions. Similar translations are employed by other verifiers. Our implementation produces certificates in Isabelle, based on a novel formalisation of the Boogie language.
... Proof obligations frequently contain universal quantifiers, both in the specification and to encode the semantics of the programming language. Most program verifiers [20,38,11,14,5,8,4] rely on SMT solvers to discharge the proof obligations via E-matching [13]. This SMT algorithm requires syntactic matching patterns of ground terms (called patterns in the following), to control the instantiations. ...
... It is nevertheless important to detect it because it could surface if Boogie was extended to support quantifier instantiation algorithms that are not based on E-matching (such as MBQI [16]) or first-order provers. This example also shows that the problems tackled in this paper cannot be solved by simply switching to other instantiation strategies: these are not the preferred choices of most verifiers [20,38,11,14,5,8,4], and they might produce unsound results for verifiers that were designed to use E-matching. F2 : ∀m2 : U, k2 : U, x2 : U, v2 : U :: {Store(m2, k2, x2, v2)} typ(Store(m2, k2, x2, v2)) = Type(typ(k2), typ(v2)) Contributions. ...
... G) to automatically identify the subset that falls into this category. The results show that our algorithm is suited also for benchmarks from [8,33,11]. Sec. 4.3 illustrates that our triggering terms are simpler than unsat proofs, which are sometimes produced by quantifier instantation and refutation techniques. ...
Preprint
Full-text available
Universal quantifiers occur frequently in proof obligations produced by program verifiers, for instance, to axiomatize uninterpreted functions and to express properties of arrays. SMT-based verifiers typically reason about them via E-matching, an SMT algorithm that requires syntactic matching patterns to guide the quantifier instantiations. Devising good matching patterns is challenging. In particular, overly restrictive patterns may lead to spurious verification errors if the quantifiers needed for a proof are not instantiated; they may also conceal unsoundness caused by inconsistent axiomatizations. In this paper, we present the first technique that identifies and helps the users remedy the effects of overly restrictive matching patterns. We designed a novel algorithm to synthesize missing triggering terms required to complete a proof. Tool developers can use this information to refine their matching patterns and prevent similar verification errors, or to fix a detected unsoundness.
... For languages with first-class specifications (e.g. JML [32,33,50,59,85]) prior work has considered both static verification [11,12,22,31,40,52,57,61,68,86] and random testing [16,18,24,92,101,104]. An important question often raised here is: why test when you could statically verify? ...
... The ninth commandment of formal methods is "Thou shalt test, test and test again" [17]. Likewise, static verification tools remain difficult to use in practice [11,14,19,25,67]. Groce, Havelund, Holzmann, Joshi, and Xu found that, despite the considerable resources of NASA's Jet Propulsion Laboratory, it was "not feasible to produce a rigorous formal proof of correctness"" for mission-critical flight software and, instead, successfully employed random testing (amongst other techniques) [44]. ...
... The ultimate aim is that all programs written in Whiley will be verified at compile-time to ensure their specifications hold which, for example, has obvious application in safety-critical systems [23,75]. Comparable systems to Whiley include ESC/Java [40], Spec# [11], Dafny [60], Why3 [39], VeriFast [51], SPARK/Ada [10], Frama-C [55], 13:2 Viper [66], KeY [2,13], and RESOLVE [46,89] (amongst others). Whiley has, for example, been used for teaching large introductory classes in formal methods [76,79]. ...
... For languages with first-class specifications (e.g. JML [32,33,50,59,85]) prior work has considered both static verification [11,12,22,31,40,52,57,61,68,86] and random testing [16,18,24,92,101,104]. An important question often raised here is: why test when you could statically verify? ...
... The ninth commandment of formal methods is "Thou shalt test, test and test again" [17]. Likewise, static verification tools remain difficult to use in practice [11,14,19,25,67]. Groce, Havelund, Holzmann, Joshi, and Xu found that, despite the considerable resources of NASA's Jet Propulsion Laboratory, it was "not feasible to produce a rigorous formal proof of correctness"" for mission-critical flight software and, instead, successfully employed random testing (amongst other techniques) [44]. ...
... The ultimate aim is that all programs written in Whiley will be verified at compile-time to ensure their specifications hold which, for example, has obvious application in safety-critical systems [23,75]. Comparable systems to Whiley include ESC/Java [40], Spec# [11], Dafny [60], Why3 [39], VeriFast [51], SPARK/Ada [10], Frama-C [55], 13:2 Viper [66], KeY [2,13], and RESOLVE [46,89] (amongst others). Whiley has, for example, been used for teaching large introductory classes in formal methods [76,79]. ...
Preprint
Full-text available
Automated specification-based testing has a long history with several notable tools having emerged. For example, QuickCheck for Haskell focuses on testing against user-provided properties. Others, such as JMLUnit, use specifications in the form of pre- and post-conditions to drive testing. An interesting (and under-explored) question is how effective this approach is at finding bugs in practice. In general, one would assume automated testing is less effective at bug finding than static verification. But, how much less effective? To shed light on this question, we consider automated testing of programs written in Whiley -- a language with first-class support for specifications. Whilst originally designed with static verification in mind, we have anecdotally found automated testing for Whiley surprisingly useful and cost-effective. For example, when an error is detected with automated testing, a counterexample is always provided. This has motivated the more rigorous empirical examination presented in this paper. To that end, we provide a technical discussion of the implementation behind an automated testing tool for Whiley. Here, a key usability concern is the ability to parameterise the input space, and we present novel approaches for references and lambdas. We then report on several large experiments investigating the tool's effectiveness at bug finding using a range of benchmarks, including a suite of 1800+ mutants. The results indicate the automated testing is effective in many cases, and that sampling offers useful performance benefits with only modest reductions in bug-finding capability. Finally, we report on some real-world uses of the tool where it has proved effective at finding bugs (such as in the standard library).
... SRS programs are translated into verification conditions, i.e., logical formulae whose validity entails the correctness of the program. Following a popular approach initiated by Spec# [4], these conditions are not generated directly but instead obtained through a translation into Boogie [5], an intermediate language for program verification. Once a SRS program is translated into a Boogie program, it is up to the Boogie validator to generate the verification conditions and, resorting to an SMT solver, verify whether they hold. ...
Chapter
Consumption of REST services has become a popular means of invoking code provided by third parties, particularly in web applications. Nowadays programmers of web applications can choose TypeScript over JavaScript to benefit from static type checking that enables validating calls to local functions or to those provided by libraries. Errors in calls to REST services, however, can only be found at runtime. In this paper, we present SRS, a language that extends the support of static analysis to calls to REST services, with the ability to statically find common errors such as missing or invalid data in REST calls and misuse of the results from such calls. SRS features a syntax similar to JavaScript and is equipped with a rich collection of types and primitives to natively support REST calls that are statically validated against specifications of the corresponding APIs written in the HeadREST language.