Fig 11 - uploaded by Patrick Cousot
Content may be subject to copyright.
A screenshot of the error reporting with the precondition inference.
Source publication
We consider the problem of automatic precondition inference. We argue that the common notion of sufficient precondition inference (i.e., under which precondition is the program correct?) imposes too large a burden on callers, and hence it is unfit for automatic program analysis. Therefore, we define the problem of necessary precondition inference (...
Citations
... As such, SO makes an excellent source for our study, as the primary goal of this study is to derive ML contracts from peer-reviewed and well-vetted content for the reliability of the findings. To capture the contracts, analyzing the large code corpus of API usages Wasylkowski et al. (2007);Nguyen et al. (2009);Khairunnesa et al. (2017) or the implementation of the software itself Cousot et al. (2013) are both well-known techniques. Our chosen methodology is closer to the former. ...
Recent work has shown that Machine Learning (ML) programs are error-prone and called for contracts for ML code. Contracts, as in the design by contract methodology, help document APIs and aid API users in writing correct code. The question is: what kinds of contracts would provide the most help to API users? We are especially interested in what kinds of contracts help API users catch errors at earlier stages in the ML pipeline. We describe an empirical study of posts on Stack Overflow of the four most often-discussed ML libraries: TensorFlow, Scikit-learn, Keras, and PyTorch. For these libraries, our study extracted 413 informal (English) API specifications. We used these specifications to understand the following questions. What are the root causes and effects behind ML contract violations? Are there common patterns of ML contract violations? When does understanding ML contracts require an advanced level of ML software expertise? Could checking contracts at the API level help detect the violations in early ML pipeline stages? Our key findings are that the most commonly needed contracts for ML APIs are either checking constraints on single arguments of an API or on the order of API calls. The software engineering community could employ existing contract mining approaches to mine these contracts to promote an increased understanding of ML APIs. We also noted a need to combine behavioral and temporal contract mining approaches. We report on categories of required ML contracts, which may help designers of contract languages.
... This implies that is an overapproximation of the set of states that end up in , and corresponds to the notion of necessary preconditions studied by Cousot et al. [2013]. In particular, if an initial state ̸ |= , then is guaranteed to not terminate in ( could also diverge). ...
... Following the terminology from above, which is inspired from the naming necessary preconditions of Cousot et al. [2013], we can state that ...
... The latter connection is justified by the fundamental Galois connection between slp and wp, and the strong duality between total and partial correctness, but where we replace nontermination with unreachability. Finally, we notice that there are three additional Hoare-style triples that can be naturally defined using our transformers, and we identify a precise connection between partial incorrectness and the so-called necessary preconditions [Cousot et al. 2013]. ...
We present a novel strongest-postcondition-style calculus for quantitative reasoning about non-deterministic programs with loops. Whereas existing quantitative weakest pre allows reasoning about the value of a quantity after a program terminates on a given initial state, quantitative strongest post allows reasoning about the value that a quantity had before the program was executed and reached a given final state. We show how strongest post enables reasoning about the flow of quantitative information through programs. Similarly to weakest liberal preconditions, we also develop a quantitative strongest liberal post. As a byproduct, we obtain the entirely unexplored notion of strongest liberal postconditions and show how these foreshadow a potential new program logic - partial incorrectness logic - which would be a more liberal version of O'Hearn's recent incorrectness logic.
... Accordingly, these summaries must necessarily either be rather coarse or very expensive. An interesting approach is presented in [17] where soundness is traded for an improved false-positivity rate. There, the goal is to find necessary pre-conditions, i.e., pre-conditions that, when violated, are guaranteed to lead to errors. ...
To put static program analysis at the fingertips of the software developer, we propose a framework for interactive abstract interpretation. While providing sound analysis results, abstract interpretation in general can be quite costly. To achieve quick response times, we incrementalize the analysis infrastructure, including postprocessing, without necessitating any modifications to the analysis specifications themselves. We rely on the local generic fixpoint engine TD, which dynamically tracks dependencies, while exploring the unknowns contributing to answering an initial query. Lazy invalidation is employed for analysis results affected by program change. Dedicated improvements support the incremental analysis of concurrency deficiencies such as data-races. The framework has been implemented for multithreaded C within the static analyzer Goblint, using MagpieBridge to relay findings to IDEs. We evaluate our implementation w.r.t. the yard sticks of response time and consistency: formerly proven invariants should be retained - when they are not affected by the change. The results indicate that with our approach, a reanalysis after small changes only takes a fraction of from-scratch analysis time, while most of the precision is retained. We also provide examples of program development highlighting the usability of the overall approach.
... This implies that is an overapproximation of the set of states that end up in , and corresponds to the notion of necessary preconditions studied by Cousot et al. [2013]. In particular, if an initial state ̸ |= , then is guaranteed to not terminate in ( could also diverge). ...
... Following the terminology from above, which is inspired from the naming necessary preconditions of Cousot et al. [2013], we can state that • total correctness triples provide sufficient preconditions; ...
... The latter connection is justified by the fundamental Galois connection between slp and wp, and the strong duality between total and partial correctness, but where we replace nontermination with unreachability. Finally, we notice that there are three additional Hoare-style triples that can be naturally defined using our transformers, and we identify a precise connection between partial incorrectness and the so-called necessary preconditions [Cousot et al. 2013]. ...
We present a novel strongest-postcondition-style calculus for quantitative reasoning about non-deterministic programs with loops. Whereas existing quantitative weakest pre allows reasoning about the value of a quantity after a program terminates on a given initial state, quantitative strongest post allows reasoning about the value that a quantity had before the program was executed and reached a given final state. We show how strongest post enables reasoning about the flow of quantitative information through programs. Similarly to weakest liberal preconditions, we also develop a quantitative strongest liberal post. As a byproduct, we obtain the entirely unexplored notion of strongest liberal postconditions and show how these foreshadow a potential new program logic - partial incorrectness logic - which would be a more liberal version of O'Hearn's recent incorrectness logic.
... Given its interest, considerable effort has recently been invested in achieving automatic support for equipping programs with extensive contracts, yet the current contract inference tools are still often unsatisfactory in practice [13]. ...
... Let us briefly discuss those strands of research that have influenced our work the most. A detailed description of the related literature can be found [1,13,20,32,45]. Our axiomatic representation is inspired by [44], which relies on a model checker for (bounded) symbolic execution of .NET programs and generates either Spec# specifications or parameterized unit tests. ...
... ). However, this is achieved at the expense of some additional precision loss [13]. We achieve generality thanks to the K Framework. ...
In this article, we propose a symbolic technique that can be used for automatically inferring software contracts from programs that are written in a non-trivial fragment of C, called KERNELC, that supports pointer-based structures and heap manipulation. Starting from the semantic definition of KERNELC in the 𝕂 semantic framework, we enrich the symbolic execution facilities recently provided by 𝕂 with novel capabilities for contract synthesis that are based on abstract subsumption. Roughly speaking, we define an abstract symbolic technique that axiomatically explains the execution of any (modifier) C function by using other (observer) routines in the same program. We implemented our technique in the automated tool KINDSPEC 2.1, which generates logical axioms that express pre- and post-condition assertions which define the precise input/output behavior of the C routines. Thanks to the integrated support for symbolic execution and deductive verification provided by 𝕂, some synthesized axioms that cannot be guaranteed to be correct by construction due to abstraction can finally be verified in our setting with little effort.
... Our analysis does not aim to address this question. More generally, the result of our analysis can be used for a wide range of applications: from code specifications [14], to grammar-based testing [29], to automatically checking and guiding the cleaning of the data [1,38]. ...
... Finally, the main difference compared to the inference of necessary preconditions proposed by Cousot et al. [14] or the (bi-)abduction [6] used in tools like Infer [5] is that our analysis can also deal with inputs read at any point during the program (thus notably also inside loops whose execution may depend on other inputs -this is where the need for the stack comes from, cf. Section 4). ...
Nowadays, as machine-learned software quickly permeates our society, we are becoming increasingly vulnerable to programming errors in the data pre-processing or training software, as well as errors in the data itself. In this paper, we propose a static shape analysis framework for input data of data-processing programs. Our analysis automatically infers necessary conditions on the structure and values of the data read by a data-processing program. Our framework builds on a family of underlying abstract domains, extended to indirectly reason about the input data rather than simply reasoning about the program variables. The choice of these abstract domain is a parameter of the analysis. We describe various instances built from existing abstract domains. The proposed approach is implemented in an open-source static analyzer for Python programs. We demonstrate its potential on a number of representative examples.
... In this section, we assume a pre-image over-approximating backwards analysis of the program, e.g. [5], i.e., we assume a function pre |prg| : ℘(Y ) → ℘(X) such that pre |prg| (A) ⊆ pre |prg| (A). We want to use pre |prg| to provide upper and lower probability bounds for all output events. ...
We consider reusing established non-probabilistic output analyses (either forward or backwards) that yield over-approximations of a program's pre-image or image relation, e.g., interval analyses. We assume a probability measure over the program input and present two techniques (one for forward and one for backward analyses) that both derive upper and lower probability bounds for the output events. We demonstrate the most involved technique, namely the forward technique, for two examples and compare their results to a cutting-edge probabilistic output analysis.
... In this section, we assume a pre-image over-approximating backwards analysis of the program, e.g. [5], i.e., we assume a function pre |prg| : ℘(Y ) → ℘(X) such that pre |prg| (A) ⊆ pre |prg| (A). We want to use pre |prg| to provide upper and lower probability bounds for all output events. ...
We consider reusing established non-probabilistic output analyses (either forward or backwards) that yield over-approximations of a program's pre-image or image relation, e.g., interval analyses. We assume a probability measure over the program input and present two techniques (one for forward and one for backward analyses) that both derive upper and lower probability bounds for the output events. We demonstrate the most involved technique, namely the forward technique, for two examples and compare their results to a cutting-edge probabilistic output analysis.
... The concept of necessary preconditon [Cousot et al. 2013] is related. A necessary precondition for a program is a predicate which, whenever falsified, leads to divergence or an error, but never to successful termination. ...
... Curiously, flipping the subset relation gives two inequivalent notions, p ⊇ wlp(C)q and post (C)p ⊇ q. Replacing wlp with the weakest possible preconditon wpp(C)q leads to a relationship p ⊇ wpp(C)q which is as in Thresher and in necessary preconditions, the difference being that in Thresher q describes error states where Cousot et al. [2013] use q to describe success states. I am grateful to Benno Stein for discussions on Thresher, necessary preconditions, and wpp. ...
Program correctness and incorrectness are two sides of the same coin. As a programmer, even if you would like to have correctness, you might find yourself spending most of your time reasoning about incorrectness. This includes informal reasoning that people do while looking at or thinking about their code, as well as that supported by automated testing and static analysis tools. This paper describes a simple logic for program incorrectness which is, in a sense, the other side of the coin to Hoare's logic of correctness.
... We leverage the notion of "abstract testing" [19] to denote the process of abstract execution (i.e., forward analysis) of a program with an given abstract input (which represents a set of concrete inputs) and the checking of whether the abstract output satisfies the target property. The key idea of our approach is to perform abstract testing iteratively in a top-down manner, by refining the abstract input via partitioning, wherein the refinement process also makes use of the computed necessary precondition of violating the target assertion via the "inverse" abstract execution (i.e., backward analysis [21]). When the property has been checked to hold for all abstract sub-inputs in the partition, the iterative process stops and gives a proof. ...
... With the abstract semantics, sound program invariants can be computed automatically in finite steps by forward abstract interpretation [18] (denoted as Forward AI) and backward abstract interpretation [21] (denoted as Backward AI) respectively. The computation with abstract interpretation is parameterized by abstract domains specifying the considered approximated properties. ...
When applying abstract interpretation to verification, it may suffer from the problem of getting too conservative over-approximations to verify a given target property, and being hardly able to generate counter-examples when the property does not hold. In this paper, we propose iterative abstract testing, to create a property-oriented verification approach based on abstract interpretation. Abstract testing employs forward abstract executions (i.e., forward analysis) together with property checking to mimic (regular) testing, and utilizes backward abstract executions (i.e., backward analysis) to derive necessary preconditions that may falsify the target property, and be useful for reducing the input space that needs further exploration. To verify a property, we conduct abstract testing in an iterative manner by utilizing dynamic partitioning to split the input space into sub-spaces such that each sub-space involves fewer program behaviors and may be easier to verify. Moreover, we leverage bounded exhaustive testing to verify bounded small sub-spaces, as a means to complement abstract testing based verification. The experimental results show that our approach has comparable strength with several state-of-the-art verification tools.
