Content uploaded by Alcides Fonseca
Author content
All content in this area was uploaded by Alcides Fonseca on Oct 28, 2023
Content may be subject to copyright.
User-driven Design and Evaluation of Liquid Types in Java
CATARINA GAMBOA, LASIGE, Faculdade de Ciências da Universidade de Lisboa, Portugal
PAULO ALEXANDRE SANTOS, LASIGE, Faculdade de Ciências da Universidade de Lisboa, Portugal
CHRISTOPHER S. TIMPERLEY, School of Computer Science, Carnegie Mellon University, USA
ALCIDES FONSECA, LASIGE, Faculdade de Ciências da Universidade de Lisboa, Portugal
Bugs that are detected earlier during the development lifecycle are easier and cheaper to x, whereas bugs
that are found during production are dicult and expensive to address, and may have dire consequences.
Type systems are particularly eective at identifying and preventing bugs early in the development lifecycle
by causing invalid programs to result in build failure.
Liquid Types are more powerful than those found in mainstream programming languages, allowing the
detection of more classes of bugs. However, while Liquid Types were proposed in 2008 with their integration in
ML and subsequently introduced in C (2012), Javascript(2012) and Haskell(2014) through language extensions,
they have yet to become widely adopted by mainstream developers. This paper investigates how Liquid Types
can be integrated in a mainstream programming language, Java, by proposing a new design that aims to lower
the barrier to entry and adapts to problems that Java developers commonly encounter at runtime.
To promote accessibility, we conducted a series of developer surveys to design the syntax of LiquidJava, our
prototype. To evaluate the prototype’s usability, we conducted a user study of 30 Java developers, concluding
that users intend to use LiquidJava and that it helped to nd more bugs and debug faster.
ACM Reference Format:
Catarina Gamboa, Paulo Alexandre Santos, Christopher S. Timperley, and Alcides Fonseca. 2021. User-driven
Design and Evaluation of Liquid Types in Java. 1, 1 (October 2021), 9 pages. https://doi.org/10.1145/nnnnnnn.
nnnnnnn
1 INTRODUCTION
Software quality is a major concern throughout software development [
16
]. Given the increased
costs of nding and addressing bugs later in the development lifecycle, many developers and
organizations aim to identify issues as early as possible when they are cheaper and easier to address
by “shifting left” [
22
]. Unsurprisingly, validation techniques that are integrated into editors have
proved widely popular and been adopted by many developers.
Strong type systems have been introduced in modern programming languages (e.g., Haskell,
Java), allowing developers to specify the expected type of operations and verify, at compile time, if
those types are respected. Editors typically integrate this verication to provide developers with
immediate feedback about the incorrect use of variables and values, allowing them to debug and
resolve the cause of identied issues more quickly.
Authors’ addresses: Catarina Gamboa, cvgamboa@fc.ul.pt, LASIGE, Faculdade de Ciências da Universidade de Lisboa,
Lisboa, Portugal; Paulo Alexandre Santos, pacsantos@fc.ul.pt, LASIGE, Faculdade de Ciências da Universidade de Lisboa,
Lisboa, Portugal; Christopher S. Timperley, ctimperley@cmu.edu, School of Computer Science, Carnegie Mellon University,
Pittsburgh, USA; Alcides Fonseca, amfonseca@fc.ul.pt, LASIGE, Faculdade de Ciências da Universidade de Lisboa, Lisboa,
Portugal.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and
the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specic permission and/or a fee. Request permissions from permissions@acm.org.
©2021 Association for Computing Machinery.
XXXX-XXXX/2021/10-ART $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn
, Vol. 1, No. 1, Article . Publication date: October 2021.
arXiv:2110.05444v1 [cs.PL] 11 Oct 2021
2 Catarina Gamboa, Paulo Alexandre Santos, Christopher S. Timperley, and Alcides Fonseca
Renement types are more expressive than the relatively simple types of these programming
languages [
13
], as they extend the language with predicates that restrict the allowed values in
variables and methods. For example, Listing 1 shows a simple renement on the variable
r,
stating
that its value must be between 0 and 255. Renement types can be used to detect simple division
by zero errors and out-of-bounds array access bugs [
23
], but they have also been used to detect
security issues [6] and protocol violations [7].
@Renement("r >= 0 && r <= 255")
int r;
r = 90; // okay
r = 200 + 60; /∗okay in Java, Renement Type Error:
Type expected: (r >= 0 && r <= 255);
Renement found: (r == 200 + 60)∗/
Listing 1. Variable refinement in LiquidJava and verification of its assignments.
Although renement types have been introduced in programming languages such as ML [
11
],
Haskell [
21
], C [
19
] and Javascript[
8
], they have not yet been widely adopted by developers. There
are many plausible explanations for the lack of adoption, including, but not limited to, a lack of
awareness about renement types and the language extensions that provide them, developers
may lack experience in writing specications, or it could be that the rst languages to support
renement types were not widely used within the industry at the time.
The primary goal of this work is to explore how to promote the wide usage of renement types
by adding them to Java, one of the most popular programming languages in the world, and using
developer feedback to improve the usability of renement types in general.
2 LIQUIDJAVA DESIGN
LiquidJava represents the integration of Liquid Types in Java, having a design focused on usability
through the identication of system requirements (Section 2.1) and the guidance of the renements
language by Java developers (Section 2.2). The language features are presented in Section 2.3
and aim to adapt to problems that Java developers commonly encounter at runtime. Finally, we
implemented a prototype for LiquidJava and integrated it into an IDE to improve the usability of
the framework (Section 2.4).
2.1 Requirements
To promote the usability of renement types, we identied the following design requirements
based on popular characteristics of successful static verication techniques (e.g., the NonNull
annotation [9]) and feedback from developers [20]:
•Renements are optional.
A valid Java program without renements should type-check,
allowing developers to incrementally adopt stronger types. This requirement can be fullled
by using parametric annotations, such as the @Refinement annotation;
•Idiomatic renements.
The language of the renements should be as close to Java as
possible, so the developer can intuitively use and write the specication without learning a
dierent language. To gain exibility, we encoded the renements language as strings inside
the annotations. We created an online survey, detailed in Section 2.2, to access the best syntax
for the renements, and developed the grammar based on the survey results;
•Renement type-checking should be decidable
without introducing unnecessary over-
head to the compilation process and providing interactive feedback to developers as they
, Vol. 1, No. 1, Article . Publication date: October 2021.
User-driven Design and Evaluation of Liquid Types in Java 3
type. To this end, we restricted the renements language to Liquid Types [
18
] which are
veriable by SMT solvers;
•Renement type-checking works on top of an existing Java type-checker
to avoid
early deprecation as the Java release cycle increases. Therefore, we implemented the rene-
ment type checker on top of Spoon [
17
] which uses the Eclipse Compiler Kit, adopting Java
features as they are released.
These requirements guided the design of LiquidJava and the implementation of a prototype of
the renement type checker that was then integrated as an IDE plugin.
2.2 Syntax Survey
To assess the best syntax of the renements language, we created and shared an online survey
with multiple possible syntaxes for dierent LiquidJava features: type renements of variables
and methods, predicate aliases, and anonymous variables. The survey had 50 valid answers from
participants familiar with Java, from which the majority was "Not Familiar" with renement types.
Participants evaluated their preference for the syntax options, and the options that were most
preferable were used to guide the development of the renements grammar. For example, Figure 1
shows two possible syntaxes for renements in methods. The rst one adds the renements before
each basic type (before the parameters and the return type), while the second only adds one
annotation with all the renements and is inspired by renements of functional programming
languages. The answers show a preference for the rst syntax over the second since the rst has a
higher rate of "Preferable" answers and a lower rate of "Not Acceptable" answers. Similar analyses
were applied to the remaining features leading to the presented in Section 2.3.results
Fig. 1. Preferences on the Syntax for Methods Refinements.
2.3 LiquidJava features
LiquidJava supports the renement of variables (as shown in Listing 1), elds and methods (both
parameters and return types). Listing 2 shows an example of a method with the second parameter
and return types extended with renements.
Furthermore, renements can be used in classes to model the state of their objects. Classes are
considered the fundamental programming elements of the Java language [
15
]. Although classes
themselves do not have a specic value that can be rened, they can have methods that produce
changes to the internal state of the objects. In this view, we can rene the object state when a
method is called and when the method has ended with the annotation
@StateRefinement(
from="predicate", to = "predicate").
, Vol. 1, No. 1, Article . Publication date: October 2021.
4 Catarina Gamboa, Paulo Alexandre Santos, Christopher S. Timperley, and Alcides Fonseca
@Renement("_ >= a && _ <= b")
public static int inRange(int a, @Renement("b > a") int b){
return a + 1;
}
...
inRange(10, 20); //correct
inRange(10, 2); /∗Renement Type Error
Type expected: (b > a);
Renement found: (b == 2) && (a == 10)∗/
Listing 2. Annotation of method inRange with refinements and the verification of the method’s invocations.
@ExternalRenementsFor("java.net.Socket")
@StateSet({"unconnected", "bound", "connected", "closed"})
public interface SocketRenements {
@StateRenement(to="unconnected(this)")
public void Socket();
@StateRenement(from="unconnected(this)",to="bound(this)")
public void bind(SocketAddress add);
@StateRenement(from="bound(this)", to="connected(this)")
public void connect(SocketAddress add, int timeout);
@StateRenement(from="connected(this)")
public void sendUrgentData(int n);
@StateRenement(to="closed(this)")
public void close();
}
Listing 3. Socket class object state refinement
Fig. 2. DFA that represents the
Socket class states and transitions.
Listing 3 presents the modeling of the class Socket from the external library java.net and whose
methods follow an implicit order of invocations that the DFA of Figure 2 can describe. To model this
class, we can start by dening the possible disjoint states with the annotation @StateSet and then,
for each method, describe the allowed transitions. For example, the method
sendUrgentData
can only be invoked if the object is connected, which means that this method should only be called
after the
connect
method. The developers using the Socket class can now detect if the invocations
follow the correct protocol before executing the program.
2.4 Integration with IDE
Integrated development environments (IDEs) are indispensable tools for software development
nowadays. They integrate editor services that enhance developers productivity, such as content
completion, documentation popups, and real-time type checking [
14
]. Overall, IDEs provide instant
feedback to the developers while implementing the code, helping them correct errors before
executing the programs.
, Vol. 1, No. 1, Article . Publication date: October 2021.
User-driven Design and Evaluation of Liquid Types in Java 5
To enhance the usability of LiquidJava, we have created an IDE plugin to allow the developers
to use the lightweight verication integrated with the development environment. To create the
LiquidJava plugin, we used the Language Server Protocol [
2
] that decouples the creation of the
language server from the implementation of the interface for the target editor. By using LSP we
create a single implementation of the LiquidJava language server that can be paired with a client
for any editor that implements LSP (e.g., Visual Studio Code [3], Eclipse [10], Emacs [1]).
Fig. 3. IDE Plugin reporting an error in the incorrect usage of the Socket class.
For the implementation of the LiquidJava plugin, we chose to create a client for Visual Studio
Code, given that it is one most used IDEs for Java development [
4
,
5
]. The main features of the
plugin can be seen in Figure 3 and include:
•Error Reporting
– This informs the user, in real-time, of the code elements whose speci-
cation could not be proved and underlines their exact location;
•Error Information
– The problems tab shows the specication that LiquidJava failed to
verify. For more detailed information, the user can hover the error and get all the verication
conditions used to try to prove the specication.
The plugin is available for download on the project website [
12
], along with examples for the
usage of renements in variables, methods and classes.
3 EVALUATION
To evaluate the usability of our approach, we conducted a user study with volunteer Java developers,
to answer the following research questions:
RQ. 1 Are renements easy to understand?
RQ. 2 Is it easier and faster to nd implementation errors using LiquidJava than with plain Java?
RQ. 3 How hard is it to annotate a program with renements?
RQ. 4 Are developers open to using LiquidJava in their projects?
The study had 30 participants familiar with Java, of which 80% described themselves as being
only “Vaguely Familiar” or “Not Familiar” with renement types, and 90% had no previous contact
, Vol. 1, No. 1, Article . Publication date: October 2021.
6 Catarina Gamboa, Paulo Alexandre Santos, Christopher S. Timperley, and Alcides Fonseca
with LiquidJava. Each participant had an online synchronous study session with an interviewer
and followed the tasks described in 3.1. The results of the study are presented in 3.2.
3.1 Study Configuration
We conducted the synchronous sessions through the Zoom video platform, and gave the participants
a survey with the study guidelines and answer placeholders, and a GitHub repository with the
study les. The participants were also asked to share their screen with the VSCode editor and the
document with the answers.
The study was divided into six parts, as follows:
(1) Task 1: Find the error in plain Java
– Participants had to nd and x semantic errors in two
Java programs, where the implementation did not correspond to the informal documentation
presented in the associated Javadoc.
(2) Task 2: Understand Renements without prior explanation
– Participants had to in-
terpret the renements present in dierent sections of the code (variables, methods and
classes) and use them correctly and incorrectly. For example, the rst exercise had a variable
with a renement type annotation and the participants had to assign both a correct and then
an incorrect value. The results on this task aim to answer
RQ. 1
by counting the correct uses
of the renements.
(3) Overview of LiquidJava
– We introduced participants to LiquidJava using a 4-minute video
and a webpage [
12
] explaining the examples of the previous task. Both resources were then
available to be used within the remainder of the study. In the rest of the study, participants
used LiquidJava through an IDE extension created for Visual Studio Code;
(4) Task 3: Find the error with LiquidJava
– Similar to the Task 1, participants had to nd
and x the incorrect behaviour of the programs. However, for this task, they were aided by
the LiquidJava plugin. This task, paired with the Task 1, intend to answer
RQ. 2
. The exercises
were the same in both tasks, but they were split into two sets so that each participant could
have dierent exercises in each task. Hence, half the participants had one set of exercises
for Task 1 and a dierent one for Task 3, and the remaining half had the reverse set order.
Therefore, the plain Java results serve as a baseline for the LiquidJava results.
(5) Task 4: Annotate Java programs with LiquidJava
– Participants were asked to add Liquid-
Java annotations to three Java programs that targeted the LiquidJava features of renements
on variables, elds, methods and classes. This part targets
RQ. 3
, and includes a nal question
about the diculty of adding the annotations.
(6) Final Comments
– Participants had the opportunity to express their thoughts on the
overall experience of using LiquidJava by sharing what they most liked and disliked about the
framework and whether they would use LiquidJava in their projects. This overview aimed to
answer the last research question, RQ. 4, and provide feedback to improve the project.
3.2 Results
The answers were reviewed and evaluated, leading to the results presented in this section. We
evaluated the answers to the presented problems using the categories: Correct,Incorrect,Unanswered,
and Compiler Correct.Compiler correct represents the answers that lead to the expected behaviour
of the LiquidJava compiler without being completely correct according to the exercise.
Figure 4a shows that the renements on variables and methods were intuitive to use since a
large majority of participants answered correctly. The renements in the class were less intuitive,
and 26.7% of the participants even decided to leave this question unanswered. However, after the
short overview on LiquidJava, all participants could annotate the class with the protocol using
, Vol. 1, No. 1, Article . Publication date: October 2021.
User-driven Design and Evaluation of Liquid Types in Java 7
Variables Methods Class Protocol
Annotation Target
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
Number of Answers
86.7%
96.7%
46.7%
13.3%
3.3%
26.7%
26.7%
LiquidJava Usage before Overview
Correct Incorrect Unanswered
(a) Answers to Understanding Refinements without
prior explanation.
Variables Methods Class Protocol Class Fields
Annotation Target
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
Number of Answers
100.0%
80.0%
100.0%
43.3%
20.0%
56.7%
LiquidJava Annotation Results
Correct Compiler Correct Incorrect Unanswered
(b) Answers to Annotation with LiquidJava.
Fig. 4. Understandability of LiquidJava.
LiquidJava (Figure 5b), providing support for the notion that they are easy to learn and understand
(
RQ. 1
). When participants were asked to rate the ease of adding annotations on a scale of 0 (very
dicult) to 5 (very easy), 60% assigned it a value of 5, while the remaining 40% chose the value 4.
This result provides evidence that it is fairly easy to add the renement to the code (RQ. 3).
To Find To Fix
Answers
1
3
5
7
9
11
13
15
Number of Answers
6.7%
53.3%
53.3%
40.0% 46.7%
Using Java
To Find To Fix
Answers
100.0%
46.7%
53.3%
Using LiquidJava
Correct Compiler Correct Incorrect Unanswered
Answers to Socket
(a) Answers on finding and fixing bug in Socket exam-
ple.
100 200 300 400 500 600 700
Time (s)
Java
LiquidJava
Version
Comparison on Java and LiquidJava
time expent for bug finding
(b) Time for finding and fixing bug in ArrayDeque example.
Fig. 5. Using Java and LiquidJava to find and fix implementation bugs.
In Tasks 1 and 3, two programs were used. The rst one consisted of identifying the wrong order
in which methods are called (Listing 3). As shown in Figure 5a, almost no participant could nd or
x the error without the help of LiquidJava. In the second program, also with a wrong sequence
of invocations of ArrayDeque methods, every participant was able nd and x the bugs in both
Java and LiquidJava, but they were faster when using LiquidJava (Figure 5). This result shows that
LiquidJava might be more useful when added to lesser-known classes and protocols, and can reduce
the time spent on the localization of the bugs, answering to RQ. 2.
On the overview of the experience, participants reported that they mostly enjoyed LiquidJava’s
error reporting, state renement, and the syntax of its language. For the features that the participants
, Vol. 1, No. 1, Article . Publication date: October 2021.
8 Catarina Gamboa, Paulo Alexandre Santos, Christopher S. Timperley, and Alcides Fonseca
disliked, they referred to some aspects of the syntax and some plugin features, but the majority
answered that there was nothing they disliked in the usage of LiquidJava.
The last question of the study was Would you use LiquidJava in your projects? to which all the
participants answered yes, which gives condence that participants nd this tool accessible for its
gains (RQ. 4).
4 CONCLUSION
This work presents the design of LiquidJava, a project focused on promoting the usability of
renement types, specically in the Java language. To this end, we dened design requirements
and used the input of programmers familiar with Java to guide the syntax of the renements
language. The features of the LiquidJava join the existing renements concepts with the Java
features, introducing the renements of object state. An implementation of LiquidJava was created
and integrated within the Visual Studio Code editor. We developed a research study to evaluate
LiquidJava, and its results indicate that the renements are easy to understand and that developers
are interested in using LiquidJava in their projects.
REFERENCES
[1] [n.d.]. Gnu emacs. https://www.gnu.org/software/emacs/
[2] [n.d.]. Language Server Protocol. https://microsoft.github.io/language-server-protocol/
[3] 2016. Visual Studio Code. https://code.visualstudio.com/
[4] 2021. 2021 Java technology report. https://www.jrebel.com/blog/2021-java-technology-report
[5] 2021. JVM ecosystem REPORT 2021. https://snyk.io/jvm-ecosystem- report-2021/
[6]
Jesper Bengtson, Karthikeyan Bhargavan, Cédric Fournet, Andrew D. Gordon, and Sergio Maeis. 2008. Renement
Types for Secure Implementations. In Proceedings of the 21st IEEE Computer Security Foundations Symposium, CSF 2008.
IEEE Computer Society, 17–32. https://doi.org/10.1109/CSF.2008.27
[7]
Nuno Burnay, Antónia Lopes, and Vasco T. Vasconcelos. 2020. Statically Checking REST API Consumers. In Software
Engineering and Formal Methods - 18th International Conference, Frank S. de Boer and Antonio Cerone (Eds.), Vol. 12310.
265–283. https://doi.org/10.1007/978-3- 030-58768- 0_15
[8] Chugh R., Herman D., Jhala R. [n.d.]. Dependent Types for JavaScript. http://goto.ucsd.edu/~ravi/research/oopsla12-
djs.pdf.
[9]
Manuel Fähndrich and K. Rustan M. Leino. 2003. Declaring and checking non-null types in an object-oriented
language. In Proceedings of the 2003 ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages
and Applications, OOPSLA 2003, Ron Crocker and Guy L. Steele Jr. (Eds.). ACM, 302–312. https://doi.org/10.1145/
949305.949332
[10] Eclipse Foundation. [n.d.]. Eclipse IDE 2021-06: The Eclipse Foundation. https://www.eclipse.org/eclipseide/
[11]
Timothy S. Freeman and Frank Pfenning. 1991. Renement Types for ML. In Proceedings of the ACM SIGPLAN’91
Conference on Programming Language Design and Implementation (PLDI), David S. Wise (Ed.). ACM, 268–277. https:
//doi.org/10.1145/113445.113468
[12] Catarina Gamboa. 2021. LiquidJava - Project Website. https://catarinagamboa.github.io/liquidjava.html
[13]
Ranjit Jhala and Niki Vazou. 2020. Renement Types: A Tutorial. CoRR abs/2010.07763 (2020). arXiv:2010.07763
https://arxiv.org/abs/2010.07763
[14]
Lennart C. L. Kats, Richard G. Vogelij, Karl Trygve Kalleberg, and Eelco Visser. 2012. Software development environ-
ments on the web: a research agenda. In ACM Symposium on New Ideas in Programming and Reections on Software,
Onward! 2012, part of SPLASH ’12, Tucson, AZ, USA, October 21-26, 2012, Gary T. Leavens and Jonathan Edwards (Eds.).
ACM, 99–116. https://doi.org/10.1145/2384592.2384603
[15]
David Holmes Ken Arnold, James Gosling. 2005. THE Java
™
Programming Language, Fourth Edition. Addison Wesley
Professional.
[16]
Upadhyay P. 2012. The Role of Verication and Validation in System DevelopmentLife Cycle. IOSR Journal of Computer
Engineering (IOSRJCE) 5, 1 (2012), 17–20.
[17]
Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, Carlos Noguera, and Lionel Seinturier. 2016. SPOON: A
library for implementing analyses and transformations of Java source code. Softw. Pract. Exp. 46, 9 (2016), 1155–1179.
https://doi.org/10.1002/spe.2346
[18]
Patrick Maxim Rondon, Ming Kawaguchi, and Ranjit Jhala. 2008. Liquid types. In Proceedings of the ACM SIGPLAN
2008 Conference on Programming Language Design and Implementation, Rajiv Gupta and Saman P. Amarasinghe (Eds.).
, Vol. 1, No. 1, Article . Publication date: October 2021.
User-driven Design and Evaluation of Liquid Types in Java 9
ACM, 159–169. https://doi.org/10.1145/1375581.1375602
[19]
Rondon P., Bakst A., Kawaguchi M., Jhala R.,. [n.d.]. CSolve: Verifying C With Liquid Types. http://goto.ucsd.edu/
~rjhala/papers/csolve_verifying_c_with_liquid_types.pdf.
[20]
Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. 2018. Lessons from building
static analysis tools at Google. Commun. ACM 61, 4 (2018), 58–66. https://doi.org/10.1145/3188720
[21]
Niki Vazou, Eric L Seidel, Ranjit Jhala, Dimitrios Vytiniotis, and Simon Peyton-Jones. 2014. Renement types for
Haskell. In ACM SIGPLAN Notices, Vol. 49. ACM, 269–282.
[22] Hyrum Wright, Titus Delafayette Winters, and Tom Manshreck. 2020. Software Engineering at Google.
[23]
Hongwei Xi and Frank Pfenning. 1998. Eliminating Array Bound Checking Through Dependent Types. In Proceedings
of the ACM SIGPLAN ’98 Conference on Programming Language Design and Implementation (PLDI). ACM, 249–257.
https://doi.org/10.1145/277650.277732
, Vol. 1, No. 1, Article . Publication date: October 2021.