Content uploaded by Lisa Nguyen Quang Do
Author content
All content in this area was uploaded by Lisa Nguyen Quang Do on Jun 28, 2018
Content may be subject to copyright.
VISUFLOW: a Debugging Environment for Static Analyses
Lisa Nguyen Quang Do
Fraunhofer IEM
lisa.nguyen@iem.fraunhofer.de
Stefan Krüger
Paderborn University
stefan.krueger@upb.de
Patrick Hill
Paderborn University
pahill@campus.uni-paderborn.de
Karim Ali
University of Alberta
karim.ali@ualberta.ca
Eric Bodden
Paderborn University & Fraunhofer IEM
eric.bodden@upb.de
ABSTRACT
Code developers in industry frequently use static analysis tools to
detect and x software defects in their code. But what about defects
in the static analyses themselves? While debugging application
code is a dicult, time-consuming task, debugging a static analysis
is even harder. We have surveyed 115 static analysis writers to
determine what makes static analysis dicult to debug, and to iden-
tify which debugging features would be desirable for static analysis.
Based on this information, we have created V, a debugging
environment for static data-ow analysis. V is built as an
Eclipse plugin, and supports analyses written on top of the program
analysis framework S. The dierent components in V
provide analysis writers with visualizations of the internal computa-
tions of the analysis, and actionable debugging features to support
debugging static analyses. A video demo of V is available
online: https://www.youtube.com/watch?v=BkEfBDwiuH4
CCS CONCEPTS
•Software and its engineering →Software testing and debug-
ging; •Theory of computation →Program analysis; •Human-
centered computing →Empirical studies in visualization;
KEYWORDS
Debugging, Static analysis, IDE, Survey, User Study, Empirical
Software Engineering
1 INTRODUCTION
As more and more complex software is written every day, ensur-
ing its functionality, quality, and security becomes increasingly
important and dicult to achieve. Static analysis is particularly
useful in that regard, because it allows developers to reason even
about partial/incomplete programs. Researchers and practition-
ers are continuously contributing to various static analysis frame-
works [
3
,
4
,
9
,
13
]. Yet, as application code gets more sophisticated,
writing a static analysis for it becomes increasingly harder as well,
and, as we show in this paper, debugging the analysis can prove
more complicated than debugging the analyzed code.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ICSE ’18 Companion, Gothenburg, Sweden
© 2018 Copyright held by the owner/author(s). 978-1-4503-5663-3/18/05.. .$15.00
DOI: 10.1145/3183440.3183470
We have conducted a large-scale survey [
11
] with 115 analysis
writers to determine the diculties of debugging static analysis
compared to general application code. Our ndings show that bugs
found in static analyses are quite dierent from those generally
found in application code. Current debugging tools do not support
debugging those specic bugs (e.g., wrong assumptions on how the
static analysis framework interprets the analyzed code), making
it much harder for analysis writers to nd the cause of an error.
The main cause of this problem is the inability to visualize how the
analysis behaves at a given point in time.
Based on the responses we received from our survey participants,
we have identied the debugging features required to provide analy-
sis writers with helpful visuals. In this paper, we present V,
a debugging environment for static analysis built in Eclipse, an in-
tegrated development environment (IDE). V is designed to
help analysis writers debug their analyses through comprehensive
visualizations. In particular, it provides the following features:
•
Access to the intermediate representation of the analyzed code.
•Interactive graph visualizations of the analyzed code.
•Overview of the intermediate results of the analysis.
•
Breakpoint and stepping functionalities for both the analysis
code and the analyzed code.
2 MOTIVATION
Our survey explores how analysis writers debug static analysis
and which features are most useful when debugging static anal-
yses compared to general application code. The 115 participants
cover dierent branches of static analysis, including the analyzed
language, the types of static analysis (e.g., data-ow, abstract in-
terpretation), and the analysis frameworks. We have made the
anonymized responses available online [10].
We observed that 5.3
×
more participants nd static analyses
harder to debug than application code, because debugging static
analyses requires one to comprehend two dierent codebases (the
analysis code and the analyzed code) instead of just the application
code, resulting in more complex corner cases. Additionally, the
correctness of an analysis is not directly veriable, as opposed to
the output of application code:
“Static analysis code usually deals with
massive amounts of data. [...] It is harder to see where a certain state is
computed, or even worse, why it is not computed.”
Those specic properties of static analysis directly inuence the
types of bugs that are found in static-analysis code. In our study,
81.3% of the participants report that the main cause of bugs in
application code is programming errors such as
“wrong conditions,
wrong loops statements”. While those bugs exist in static analyses,
89
2018 ACM/IEEE 40th International Conference on Software Engineering: Companion Proceedings
ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden Lisa Nguyen ang Do, Stefan Krüger, Patrick Hill, Karim Ali,
Eric Bodden
Graph visuals
Other visuals
IR
Test generation
Quick updates
Breakpoints
Stepping
0%
25 %
50 %
75 %
100 %
Not Important Neutral Important
Very Important N/A
Figure 1: Importance of the features for debugging static
analysis. IR denotes Intermediate Representation.
they are only mentioned by 41.7% of the participants. Corner cases,
algorithmic errors, and handling the analysis semantics and infras-
tructure are signicantly more prevalent in static analysis than in
application code.
Since the bugs found in static analyses are dierent from the ones
found in application code, one would expect dierent debugging
tools to be used for those two types of code. However, our survey
showed that regardless of the type of code that participants write,
they use the same techniques: breakpoints and stepping, variable
inspection, printing intermediate results, debugging tools such as
gdb
or the Eclipse integrated debugger, and coding support such as
auto-completion. Interestingly, participants expressed dissatisfac-
tion with their debugging tools in general. This shows that existing
debugging tools are not sucient to fully support static analysis
writers:
“While the IDE can show a path through [my] code for a symbolic
execution run, it doesn’t show analysis states along that path.” Overall,
current debugging tools miss one crucial component to properly
support static analysis: visibility of what is (not) computed in the
analysis at a given point of its execution.
We asked the participants which debugging features would be
useful in a debugging environment and noticed a signicant dier-
ence between the features requested to debug static analysis and
application code ((
p=
0
.
04
≤
0
.
05) for a
X2
test). For application
code, participants requested better coding support and hot-code
replacement. For static analysis, participants requested better vi-
sualizations of the analysis constructs such as the intermediate
representation of the code or intermediate results of the analysis
(18.4%), graph visualizations of the analysis (23.7%), omniscient
debugging (13.2%) [
8
], and better breakpoints and stepping func-
tionalities that would allow them to step through both the analysis
code and the analyzed code at the same time. Figure 1 details the
relative importance of those debugging features.
Through our survey, we see that current debugging tools are
designed for application code, and while helpful, they are not su-
cient to fully support debugging static analysis. We identied the
following debugging features that a static analysis debugger should
provide: graph visualizations, access to analysis intermediate re-
sults, and better conditional breakpoints.
3 VISUFLOW
Based on the features identied in the survey, we present V,
a debugging environment for static analyses. We have implemented
V on top of the Eclipse IDE, and it supports debugging
static data-ow analyses written on top of the S [
14
] analysis
framework. Our survey shows the importance of displaying both
the analysis code and the analyzed code, and in particular, high-
lighting how the former handles the latter. With this goal in mind,
we have designed V provide such information, presented
in an understandable and usable manner.
We detail below the main functionalities of V, illustrated
in Figure 2, using the corresponding numbers.
1. Java Editor:
We used the standard Eclipse Java Editor and its
functionalities (e.g., Eclipse breakpoints) to display the analysis
code, since providing users with familiar views and functionali-
ties allows for a better integration of V into the Eclipse
IDE. We extended the Java Editor to add navigation functions
between this view and the other views, as detailed below.
2. Jimple View:
S converts Java code to an intermediate repre-
sentation called Jimple, which it then analyzes. To show analysis
writers how the analysis handles Jimple code, we introduced a
Jimple View that shows the analyzed code in Jimple format. The
Jimple code is not editable, because it is automatically gener-
ated by S. Similar to the Java Editor, the Jimple View oers
navigation functionalities to other views.
3. Unit Breakpoints:
In the Jimple View, analysis writers can set
special breakpoints called Unit Breakpoints that stop the execu-
tion of the analysis at a given unit (i.e., Jimple statement). Once
the execution stops, V highlights the Jimple statement
being currently analyzed, and the user may step through the
intermediate representation of the analyzed code to debug their
analysis. By jointly using the stepping functionalities of the Unit
Breakpoints and the Java Editor’s breakpoints, analysis writers
can step through both code bases at the same time, without need-
ing to write complex conditional breakpoints in the standard
Eclipse debugger.
4. Graph View:
By default, this view displays the call graph of the
analyzed code. The user can explore the control ow graph (CFG)
of a particular method by selecting its node in the call graph.
One can also navigate between the CFGs of dierent methods
through navigation menus as shown in Figure 2. On the edges of
the CFGs, V displays the information propagated by the
analysis. Such access to the intermediate results of the analysis
allows users to follow the data ows and locate miscalculations
more easily. When stepping through the Jimple code, V
highlights the corresponding unit in the Graph View. Tooltips
give access to more information about the dierent units.
The Graph View is intended to give the user a better un-
derstanding of the structure of the analyzed code, while also
providing a more visual approach to debugging. It combines dif-
ferent information about the analysis, the analyzed code, and the
intermediate results, thus providing an easy way to keep track of
multiple things without cluttering the interface with code. These
characteristics also cause the graphs to act as navigational hubs
from which the user can jump to the dierent views based on
what is observed in the graph to gather more information. The
90
VISUFLOW: a Debugging Environment for Static Analyses ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden
Figure 2: The graphical user interface of V. We describe the labeled views in Section 3.
graph information, retrieved from the data model, is displayed
using the open source framework GraphStream [
2
], which scales
to graphs with a large number of nodes.
5. Results View:
The intermediate results are also shown in the
Results View, along with additional information such as editable
tags that the developer can use to mark specic units. This view
also contains searching and ltering functionalities, which is
especially helpful for methods with a large number of statements.
6. Unit Inspection View:
This view enables analysis writers who
are unfamiliar with S and Jimple to inspect Jimple state-
ments and see how they are constructed. Since S analyses
manipulate Jimple statements, understanding Jimple units helps
analysis writers design appropriate analysis rules, or judge the
correctness of existing ones.
7. Synchronization and Navigation:
V allows users to
synchronize the Unit Inspection View, the Graph View and the
Results View so that statements that are selected in a view are
also shown in the other views. In order to connect the dierent
views, we enriched V with navigation features that allow
developers to switch between views through drop-down menus,
as illustrated in Figure 2. The synchronization functionalities
provide a smooth navigation between all views that does not
disrupt the analysis writer by forcing them to manually look for
a specic statement when switching between views.
The dierent views and breakpoints of V are imple-
mented as an extension of the standard Eclipse UI components. To
populate the views with analysis information, V maintains
an internal data model of the intermediate representation of the an-
alyzed code. The tool hooks into Eclipse’s builder and uses Eclipse’s
OSGi-Event Model to populate and update this data model at every
change in the code base. The intermediate results of the analysis
are updated every time the analysis is re-run, using a Java agent to
instrument the analysis to collect the information at runtime.
4 EVALUATION
To evaluate the usefulness of V, we conducted a user study
with 20 participants. We prepared faulty analyses that do not pro-
duce correct results for a given piece of analyzed code. Each par-
ticipant debugged two such analyses, each containing three errors,
and used V for one analysis, and the standard Eclipse de-
bugging environment (hereafter named Eclipse) for the other. Half
of the participants used V rst, and the other half Eclipse
rst. We recorded how many errors the participants could identify
and x, how long they spent using each feature of the tools, and
asked them to ll a comparative questionnaire and to discuss their
impressions of the two tools. The results are available online [10].
Figure 1 shows that the IDE features that were most used by the
participants include many of V’s features (i.e., the Graph
View, the Jimple View, the breakpoints, and the Results View). With
V, participants spent 25.6% less time using the Java Editor,
and 44.4% less time stepping through the code. Instead, they spent
this time using the Graph View and the Results View. This shows
that graphs, special breakpoints, and access to the intermediate
representation and to the intermediate results are desirable features
in a debugging environment for static analyses, conrming the
ndings of our survey. Moreover, Eclipse’s breakpoints editor was
used 88.2% less often in V than in Eclipse. We attribute
this to the special breakpoint features in V, which relieve
users from dening complex conditional breakpoints.
Overall, participants identied 25% and xed 50% more errors
with V than with Eclipse. For Task 1, they identied and
xed 1.4
×
more errors, and for Task 2, they identied 1.1
×
and xed
91
ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden Lisa Nguyen ang Do, Stefan Krüger, Patrick Hill, Karim Ali,
Eric Bodden
Table 1: Main features of V and Eclipse that partic-
ipants used, and the average time spent using each feature.
V Eclipse
#users Time (s) #users Time (s)
Java Editor 14 486 14 653
Graph View 14 201 n/a n/a
Jimple View 11 58 12 60
Breakpoints / Stepping 11 174 11 313
Variable Inspection 3 78 8 67
Results View 8 50 n/a n/a
Console 5 24 7 40
Drop Frame 5 12 3 5
Breakpoints View 3 13 2 110
Unit View 3 7 n/a n/a
1.6
×
more errors. Out of the 20 participants in our study, 12 partici-
pants are used to debugging their analyses using Eclipse. However,
7 of them still found and xed more errors using V.
In general, V was positively received by the participants
of our user study. In the questionnaire, they gave it a Net Promoter
Score [
12
] of 9.1/10 compared to Eclipse (standard deviation
σ
= 1.1),
and 8.3/10 compared to their own coding environment (
σ
= 1.7).
In the questionnaire and interviews, participants stressed the use-
fulness of the Graph, Jimple, and Results View, and V’s
special breakpoints, especially in understanding both the analysis
and the analyzed code, and how they interacted: “I think [V]
is helpful because of the linkage between the Java code, the Jimple code
and the graphic visualization: all that I had to keep in my mind [earlier].”.
5 RELATED WORK
While we are not aware of any debugging tool that targets writers
of static analyses as V does, a few tools have been pub-
lished that oer at least subsets of V’s features. By means
of integrated Eclipse views, the software-analysis platform Atlas [
5
]
is able to visualize data-ows through a given program. Path Pro-
jection [
6
] supports users of static analysis in understanding error
messages and locating the related lines of code. However, neither
of these tools target analysis developers, and therefore, they do not
have the comprehensive feature set that V provides for
debugging static analyses.
V is tightly integrated into Eclipse. Consequently, its
users can use all of Eclipse’s integrated debugging functionalities
for Java, such as breakpoints and stepping. While these are general-
purpose debugging features that lack the necessary focus on static
analysis, they still prove useful in any scenario involving debugging.
V does not support more complex paradigms for debugging
such as delta [
15
], omniscient [
8
], and interrogative debugging [
7
].
We leave the exploration of how V could benet from their
integration to future work.
6 CONCLUSION
We presented V, a debugging environment for static anal-
ysis. Through a user study, we demonstrated that the features in
V, designed to help visualize the internal computations of
the analysis, help analysis writers identify and x more errors than
with the standard Eclipse IDE. In future work, we plan to explore
better visualizations for the analysis of large code bases, especially
in terms of collapsable graphs and quick response to user modi-
cations in either of the code bases. V instantiates several
of the debugging features identied in our survey for S-based,
static data-ow analysis. It would be interesting to explore how to
adapt the features for other types of static analysis. Other features
suggested in the survey (e.g., omniscient debugging and quick up-
dates) are not currently oered by V. We plan to explore
adding those features to V in the future. The anonymized
answers to the survey and user study are available online [
10
]. V
is open source [
10
], and we welcome contributions under
the Apache 2.0 licence [1].
ACKNOWLEDGEMENTS
We thank Henrik Niehaus, Shashank Basavapatna Subramanya,
Kaarthik Rao Bekal Radhakrishna, Zafar Habeeb Syed, Nishitha
Shivegowda, Yannick Kouotang Signe, and Ram Muthiah Bose
Muthian for their work on the implementation of V. This
research was supported by a Fraunhofer Attract grant as well as
the Heinz Nixdorf Foundation. This work has also been partially
funded by the DFG as part of project E1 within the CRC 1119 CROSS-
ING, and was supported by the Natural Sciences and Engineering
Research Council of Canada.
REFERENCES
[1]
2017. Apache License 2.0. https://www.apache.org/licenses/LICENSE-2.0. (2017).
[2] 2017. GraphStream. http://graphstream-project.org/. (2017).
[3]
Eric Bodden. 2012. Inter-procedural data-ow analysis with IFDS/IDE and Soot.
In International Workshop on State of the Art in Java Program Analysis (SOAP).
3–8. https://doi.org/10.1145/2259051.2259052
[4]
Cristiano Calcagno and Dino Distefano. 2011. Infer: An Automatic Program
Verier for Memory Safety of C Programs. In NASA Formal Methods (NFM)
(Lecture Notes in Computer Science), Vol. 6617. 459–465.
[5]
Tom Deering, Suresh Kothari, Jeremias Sauceda, and Jon Mathews. 2014. Atlas:
A New Way to Explore Software, Build Analysis Tools (ICSE Companion 2014).
ACM, New York, NY, USA, 588–591. https://doi.org/10.1145/2591062.2591065
[6]
Yit Phang Khoo, Jerey S. Foster, Michael Hicks, and Vibha Sazawal. 2008. Path
Projection for User-centered Static Analysis Tools (PASTE ’08). ACM, New York,
NY, USA, 57–63. https://doi.org/10.1145/1512475.1512488
[7]
Andrew Jensen Ko and Brad A. Myers. 2004. Designing the whyline: a debugging
interface for asking questions about program behavior. In CHI 2004, Vienna,
Austria, April 24 - 29, 2004. 151–158.
[8]
Bil Lewis. 2003. Debugging Backwards in Time. CoRR cs.SE/0310016 (2003).
http://arxiv.org/abs/cs.SE/0310016
[9]
Lisa Nguyen Quang Do, Karim Ali, Benjamin Livshits, Eric Bodden, Justin Smith,
and Emerson Murphy-Hill. 2017. Just-in-time Static Analysis (ISSTA 2017). ACM,
New York, NY, USA, 307–317.
[10]
Lisa Nguyen Quang Do, Stefan Krüger, Patrick Hill, Karim Ali, and Eric Bodden.
2017. VisuFlow. https://blogs.uni-paderborn.de/sse/tools/visuow-debugging-
static-analysis/. (2017).
[11]
Lisa Nguyen Quang Do, Stefan Krüger, Patrick Hill, Karim Ali, and Eric Bodden.
2018. Debugging Static Analysis. Technical Report. arXiv:cs.SE/1801.04894
[12]
Frederick F Reichheld. 2003. The one number you need to grow. Harvard Business
Review 81, 12 (2003), 46–55.
[13]
Haihao Shen, Jianhong Fang, and Jianjun Zhao. 2011. Endbugs: Eective error
ranking for ndbugs. In International Conference on Software Testing, Verication
and Validation (ICST). 299–308.
[14]
Raja Vallée-Rai, Etienne Gagnon, Laurie J. Hendren, Patrick Lam, Patrice Pom-
inville, and Vijay Sundaresan. 2000. Optimizing Java Bytecode Using the Soot
Framework: Is It Feasible?. In CC. 18–34. https://doi.org/10.1007/3- 540-46423-9_2
[15]
Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and Isolating Failure-
Inducing Input. IEEE Trans. Software Eng. 28, 2 (2002), 183–200.
92
