ArticlePDF Available

Abstract

Because threads are a core feature of the Java language, the widespread adoption of Java has exposed a much wider audience to concurrency than previous languages have. Concurrent programs are notoriously di#cult to write correctly, and many subtle bugs can result from incorrect use of threads and synchronization. Therefore, finding techniques to find concurrency bugs is an important problem.
Finding Concurrency Bugs In Java
David Hovemeyer and William Pugh
Dept. of Computer Science, University of Maryland
College Park, Maryland 20742 USA
{daveho,pugh}@cs.umd.edu
ABSTRACT
Because threads are a core feature of the Java language, the
widespread adoption of Java has exposed a much wider audi-
ence to concurrency than previous languages have. Concur-
rent programs are notoriously difficult to write correctly, and
many subtle bugs can result from incorrect use of threads
and synchronization. Therefore, finding techniques to find
concurrency bugs is an important problem.
Through development and use of an automatic static anal-
ysis tool, we have found a significant number of concurrency
bugs in widely used Java applications and libraries. Inter-
estingly, we have found that race conditions abound in con-
current Java programs; underuse of synchronization is the
rule rather than the exception. We have also found many ex-
amples of other kinds of concurrency errors, suggesting that
many Java programmers have fundamental misconceptions
ab out how to write correct multithreaded programs.
This paper makes two main contributions. First, it de-
scrib e s simple analysis techniques that are effective at find-
ing concurrency errors in real programs. Second, it provides
evidence that threads and concurrency are widely misused
in Java, even in programs written by experienced program-
mers.
1. INTRODUCTION
When used correctly, threads are an elegant mechanism
to express concurrency and parallelism. Many applications,
especially network server applications, have significant con-
currency, and can benefit from parallel execution in multi-
processor systems. The Java language, by making threads
a core language feature, has achieved wide popularity for
writing concurrent programs.
However, threads and concurrency can be difficult to use
correctly. Concurrency bugs, such as race conditions and
deadlo cks, can result in program misbehavior that is very
difficult to diagnose. In general, reasoning ab out the possi-
ble behavior of a multithreaded program is difficult.
For some kinds of programs, events can be used in place
of threads to avoid some of the iss ues that make threaded
programming difficult [19]. However, events alone cannot be
used to express parallelism, which is an imp ortant require-
ment for many applications. Several race-free and deadlock-
free dialects of Java have been proposed [5, 6]; however,
these dialects have not yet been widely adopted.
To better understand the kinds of concurrency bugs af-
fecting Java programs, we have studied a variety of real ap-
plications and libraries using a static analysis tool which we
developed. The tool, called FindBugs, searches for instances
of bug patterns—code idioms that are likely to be errors.
Our experience in developing the tool and using it on real
software has lead us to several interesting conclusions:
1. Concurrency errors are very common, even in software
widely used in pro duction environments
2. Many programmers have fundamental misconceptions
ab out concurrency in Java
3. Many serious bugs can be found with simple analysis
techniques
The structure of this paper is as follows. In Section 2, we
give an overview of our findings. In Section 3, we describe
concurrency bug patterns, how our tool recognizes instances
of those patterns, and how we tuned those detectors to yield
more accurate results with fewer false positives. In Section 4,
we present empirical data on the effectiveness of the detec-
tors on several real Java applications and libraries, as well as
anecdotal experience using the tool. In Section 5, we discuss
related work. In Section 6, we present conclusions from this
study, and describe possibilities for future work.
2. CONCURRENCY IN JAVA PROGRAMS
Writing correct concurrent programs is difficult [19]. The
inherent nondeterminism of threaded programs, as well as
the complexity of the semantics governing multithreaded ex-
ecution, makes it difficult for programmers to reason about
possible program behaviors. Small errors or inconsistencies,
such as forgetting to obtain a lock before accessing a field,
can lead to runtime errors that are very hard to debug.
Static analysis techniques have been very successful at
finding many kinds of bugs in real software [16, 12, 10, 7],
including errors in multithreaded programs [21, 3, 13, 11].
Bug checkers based on static analysis represent an important
comp onent of quality assurance, and can be very effective
at finding potential bugs.
When we began working on techniques to find bugs in
Java programs, we assumed that we would need to use so-
phisticated analysis techniques, especially in the case of con-
currency bugs, where the prerequisites for a bug to manifest
may involve subtle interactions between multiple threads.
Instead, we found that simple analysis techniques were very
effective, because all of the multithreaded software we looked
at contained very obvious synchronization errors. We at-
tribute this to two main causes.
First, many programmers, especially beginning program-
mers, do not understand the fundamentals of thread syn-
chronization. We have seen many examples of code idioms,
such as spin loops and race-prone condition waits, that indi-
cate basic misconceptions about how threads work. Surpris-
ingly, we see these idioms even in widely-used software, sug-
gesting that these misconceptions are common even among
experienced programmers.
Second, many programmers tend to view synchronization
as something to be avoided whenever possible, presumably
due to the assumption that synchronization is “slow”. This
attitude is surprising. Recent research [4, 18] has greatly re-
duced the overhead of locking for the uncontended case. Like
any performance optimization, eliminating performance bot-
tlenecks caused by locks must be done using careful profil-
ing and analysis which takes the expected workload into
account. Many programmers, however, appear to believe
that good performance can be ensured by making the most
frequent accesses to shared mutable data structures unsyn-
chronized, even at the expense of correctness. In some cases,
we even see comments indicating that the programmer was
aware that he or she was writing incorrect code.
It should also be noted that our methods are not intended
to be complete. There are many kinds of concurrency errors
we do not attempt to detect, and our failing to detect con-
currency errors should give no confidence that concurrency
errors do not o ccur.
2.1 Why Java Concurrency is Difficult
Writing correct concurrent or multithreaded programs is
exceeding difficult in any language that allows for explicit
concurrency and admits the possibility of incorrect synchro-
nization [19]. However, there are perhaps more problems
in Java than in other languages, paradoxically because pro-
grammers are not as scared of writing multithreaded Java
programs. All of the problems we cite also arise in other lan-
guages, such as C, C++ or C#. However, few people write
multithreaded C programs unless they have carefully stud-
ied operating systems and/or concurrency, while high school
students write multithreaded Java programs. Also, the con-
sequences of incorrect programs in Java are not as severe.
For example, a race condition in a Java program cannot
cause the program to violate type safety, and faults arising
when objects are accessed in an inconsistent state, such as
null pointer dereferences and out of bound array accesses,
are guaranteed to trapped and propagated as exceptions.
A more fundamental problem is that programmers often
have an incorrect mental model of what behaviors are pos-
sible in concurrent Java programs. From the concurrency
errors we have found in the applications and libraries we ex-
amined, it is apparent that many programmers believe that
their program will execute in a sequentially consistent [1]
manner. In sequential consistency, there is a global total
order over all memory accesses, consistent with program or-
der for all threads. Sequential consistency would allow some
uses of data races to have useful semantics. For example, use
of the double checked locking idiom [8] to avoid acquiring a
lo ck works as expected under sequential consistency.
Unfortunately, the Java memory model [15] is not se-
quentially consistent. Processors and compiler optimiza-
tions may reorder memory accesses in ways that violate se-
quential consistency. This issue is compounded by the fact
that aggressive inlining of methods, performed by most mod-
ern JVMs, can make the scope of optimizations non-local.
This makes it very difficult to guess the possible behavior of
co de with a race condition by simply eyeballing the co de.
In this paper we will show some examples of code where it
is clear that the programmer had a serious misunderstanding
of the semantics of concurrency in Java. Many programmers
are trapped in the second order of ignorance [2] with respect
to concurrent programming in Java: they do not understand
how to write correct multithreaded programs, and they are
not aware that they do not understand.
3. CONCURRENCY BUG PATTERNS
Our work has focused on finding simple, effective analysis
techniques to find bugs in Java programs, including con-
currency bugs. We start by identifying bug patterns, which
are code idioms that are often errors. We have used many
sources to find bug patterns: some have come from books
and articles, while many have been suggested by other re-
searchers and Java developers, or have been found through
our own experience.
Detecting bug patterns is the automated equivalent of a
co de review—bug pattern detectors look for code that devi-
ates from good practice. While to goal is to find real errors,
bug pattern detectors can also produce warnings that do
not correspond to real problems. Sometimes, false warnings
arise because the analysis technique used by the bug pattern
detector is inherently imprecise. Other times, a detector will
make non-conservative assumptions about the likely behav-
ior of the program in order to retain precision in the face of
difficult program analysis problems, such as pointer aliasing
or heap modeling.
Once we have identified a new bug pattern, we start with
the simplest technique we can think of to detect instances
of the pattern in Java code. Often, we will start out using
simple state-machine driven pattern-matching techniques.
We then try applying the new detector to real programs. If it
pro duces too many false warnings, we either add heuristics,
or move to a more sophisticated analysis technique (such
as dataflow analysis). Our target for accuracy is that at
least 50% of the warnings produced by the detector should
represent genuine bugs.
We have implemented bug detectors for over 45 bug pat-
terns, including 21 multithreaded bug patterns, in a tool
called FindBugs. All of the detectors operate on Java byte-
co de, using the Apache Byte Code Engineering Library
1
.
FindBugs is distributed under an open source license, and
may be downloaded from the FindBugs website:
http://findbugs.sourceforge.net
This section describes some of the concurrency bug pat-
terns implemented in FindBugs, discusses some of the im-
plementation decisions and tradeoffs we made in writing de-
tectors for those patterns, and presents some examples of
bugs found by the tool.
1
http://jakarta.apache.org/bcel
3.1 Inconsistent Synchronization
When mutable state is accessed by multiple threads, it
generally needs to be protected by synchronization. A very
common technique in Java is to protect the mutable fields
of an object by locking on the object itself. A method may
be defined with the synchronized keyword, in which case a
lo ck on the receiver object is obtained for the scope of the
metho d. Or, if finer grained synchronization is desired, a
synchronized(this) block may be used to acquire the lock
within a block scope. Classes whose instances are intended
to be thread safe should generally only access shared fields
while the instance lock is held. Unsynchronized field accesses
often are race conditions that can lead to incorrect behavior
at runtime. We refer to unsynchronized accesses in classes
intended to be thread safe as inconsistent synchronization.
To detect inconsistent synchronization, the FindBugs tool
tracks the scop e of locked objects
2
. For every instance field
access, the tool records whether or not a lock is held on the
instance through which the field is accessed. Fields that are
not consistently locked are reported as potential bugs.
We use a variety of heuristics to reduce false positives.
Field accesses in object lifecycle methods, such as construc-
tors and finalizers, are ignored, because it is unlikely that the
object is visible to multiple threads in those methods. We
ignore non-final public fields, on the assumption users must
be responsible for guarding synchronization of such fields.
Volatile fields are also ignored, because under the proposed
Java memory model [15], reads and writes of volatile fields
can be used to enforce visibility and ordering guarantees be-
tween threads. Similarly, final fields are ignored, since they
are largely thread safe (the only exception being cases where
objects are made visible to other threads before construction
is complete).
Initially, we assumed that shared fields of objects intended
by programmers to be thread-safe would generally be syn-
chronized consistently, and that bugs would usually be the
result of oversight by the programmer. For example, a pro-
grammer might add a public method to a thread-safe class,
but forget to make the method synchronized. Under this
assumption, we used the frequency of unsynchronized ac-
cesses to prioritize the warnings generated for inconsistently
synchronized fields. Fields with 25% or fewer unsynchro-
nized accesses (but at least one unsynchronized access) were
assigned medium or high priority. Fields with 25-50% un-
synchronized accesses were assigned low priority, on the as-
sumption that such fields were likely to be only incidentally
(not intentionally) synchronized.
To evaluate the appropriateness of our ranking heuristic,
we manually categorized inconsistent synchronization warn-
ings for se veral applications and libraries. We used three
categories to describe the accuracy of the warnings:
Serious An accurate warning, where the unsynchronized
accesses might result in incorrect behavior at runtime.
Harmless An accurate warning, where the unsynchronized
accesses would be unlikely to result in incorrect b ehav-
ior. An example would be an unsy nchronized getter
metho d that returns the value of an integer field.
2
The analysis is intraprocedural, with the addition that calls
to non-public methods within a class are analyzed, and non-
public methods called only from locked contexts are consid-
ered to be synchronized as well.
False An inaccurate warning: either the analysis performed
by the tool was incorrect, or any unsynchronized ac-
cesses would be guaranteed to behave correctly.
Our decisions were based on manual inspection of the of
the code identified by each warning. While our judgment is
fallible, we tried to err on the side of classifying warnings as
false or harmless if we could not see a scenario that would
lead to unintended b ehavior.
We then studied the number of serious, harmless, and
false warnings that would b e reported by the tool for vary-
ing cutoff values for the minimum percentage of unsynchro-
nized field accesses. For example, for a cutoff value of 75%,
only fields whose accesses were sy nchronized at least 75%
of the time would be reported. By graphing the number of
warnings in these categories, we were able to evaluate the va-
lidity of the hypothesis that most of the serious bugs would
have a high percentage of synchronized accesses. We com-
bined the ‘harmless’ and ‘false’ categories because together
they represent the set of warnings we believed would not be
of interest to developers. Figure 1 shows these graphs for
several applications and libraries.
We were surprised to find that the likelihood of an in-
consistently synchronized field being a serious bug was not
strongly related to the percentage of synchronized accesses
for the range of cutoff values we ex amined. In other words,
the inconsistent synchronization bugs we found were not
generally the result of the programmer simply forgetting to
synchronize a particular field access or method. Instead,
we found that the lack of synchronization was almost al-
ways intentional—the programmer had deliberately used a
race condition to communicate b e tween threads. The data
suggests that we should try even lower cutoff values (below
50%), since many genuine bugs were found for fields syn-
chronized only 50% of the time. The message to take away
here is that lack of synchronization is not exceptional; for
many classes, it is the norm.
3.1.1 Forms of Inconsistent Synchronization
As we examined examples of inconsistent synchronization,
we noticed several common forms:
Synchronized field assignment, unsynchronized field
use. In this form, locking is used whenever a field is set, but
not when the field is read. There are two potential problems
here.
First, the value read is not guaranteed to be up to date
when it is eventually used; if the read is part of an operation
with multiple steps, the operation will not be guaranteed to
take place atomically. A more subtle problem is that if a
reference is read from a field without synchronization, there
is generally no guarantee that the object will be completely
initialized.
This was the most common form of inconsistent synchro-
nization. An example is shown in Figure 2.
Object pair operation. In this form, an operation is
performed involving two objects, each of which can be ac-
cessed by multiple threads. However, a lock is acquired on
only one of the objects, allowing the other to be vulnera-
ble to concurrent modification. This problem is especially
prevalent in equals() methods. It can be quite hard to
fix because of the p otential deadlock issues: if two threads
try to lock both objects, but use different lock acquisition
orders, a deadlock can result.
0
5
10
15
20
25
30
35
40
45
50 55 60 65 70 75 80 85 90 95 100
Number of warnings
Minimum accesses synchronized (percent)
Inconsistent synchronization false positive results for classpath-0.08
Serious
Harmless and false
0
20
40
60
80
100
120
50 60 70 80 90 100
Number of warnings
Minimum accesses synchronized (percent)
Inconsistent synchronization false positive results for rt-1.5-b42
Serious
Harmless and false
0
5
10
15
20
25
30
35
50 55 60 65 70 75 80 85 90 95 100
Number of warnings
Minimum accesses synchronized (percent)
Inconsistent synchronization false positive results for jboss-4.0.0DR3
Serious
Harmless and false
0
5
10
15
20
50 55 60 65 70 75 80 85 90 95 100
Number of warnings
Minimum accesses synchronized (percent)
Inconsistent synchronization false positive results for jacorb-2.1
Serious
Harmless and false
Figure 1: Serious bugs and false and harmless warnings for varying values for minimum percentage of
synchronized accesses. The applications are two implementations of the J2SE core libraries (rt-1.5-b42 and
classpath-0.08), a open source Java application server (jboss-4.0.0DR3), and an open source CORBA ORB
(jacorb-2.1). Our hypothesis that fields with lower (but nonzero) percentages of unsynchronized accesses
would be more likely to be errors was found to be incorrect.
// java.lang, StringBuffer.java, line 825
public int lastIndexOf(String str)
{
return lastIndexOf(str, count - str.count);
}
Figure 2: An atomicity bug in GNU Classpath 0.08.
The count field is read without synchronization and
then passed to a synchronized method. Because the
value may b e out of date, an ArrayIndexOutOfBounds
exception can result.
3.1.2 Limitations of the Inconsistent Synchronization
Detector
While it is effective at finding many concurrency bugs,
the inconsistent synchronization detector has some impor-
tant limitations. First, programs that are free of race con-
ditions may still have atomicity bugs. A na¨ıve approach to
synchronization is to make every metho d of a class synchro-
nized. However, successive calls to methods in such a class
will not occur atomically unless a lock on the receiver object
is explicitly held in a scope surrounding both calls. Another
limitation is that the detector only works when shared fields
are synchronized by locking the object instance. Although
this is a common technique in Java, it is also common to
use explicit lock objects. Modifying the detector to han-
dle arbitrary lock objects would require more sophisticated
analysis, including some form of heap analysis.
3.2 Double Checked Locking
Lazy initialization is a common performance optimization
used to create singleton objects only as they are needed. In
a multithreaded program, some form of synchronization is
needed to ensure that the singleton object is created only
once, and that the object is always fully initialized b e fore it
is used.
A common idiom for lazy initialization of singleton objects
is double checked locking. Synchronization is performed only
if the object has not yet been created:
static SomeClass field;
static SomeClass createSingleton() {
if (field == null) {
synchronized (lock) {
if (field == null) {
SomeClass obj = new SomeClass();
// ...initialize obj...
field = obj;
}
}
}
return field;
}
The intent of double checked locking is that the overhead
of lock acquisition is only incurred if the singleton object is
observed as not having been created yet.
Unfortunately, this form of double checked locking is not
correct. Although the idiom guarantees that the object is
created only once, the Java memory model does not guar-
antee that the threads that see a non-null field value (but
do not acquire the lock) will see all of the writes used to ini-
tialize the object. For example, the JIT compiler may inline
the call to the constructor and reorder some of the writes
initializing the object so that they occur after the write to
the field storing the object instance.
Double checked locking is a good illustration of the gulf
between how multithreaded code looks like it should be-
have and the behaviors actually allowed by the language.
Most programmers can easily understand how synchroniza-
tion can be used to guarantee the atomicity of a sequence
of operations. However, it is much harder to understand
the subtle interaction of compiler and processor optimiza-
tions. Memory model issues are challenging even for experts:
double checked locking has been advocated in a number of
books and articles (as mentioned in [8]), showing that even
experts do not always understand the consequences of omit-
ting proper synchronization.
Under the proposed Java Memory Mode l [15], it is pos-
sible to fix instances of double checked locking by making
the field volatile. The volatile qualifier causes the compiler
to insert the necessary optimization and memory barriers
needed to ensure that all threads will see a completely ini-
tialized object, even if they don’t acquire the lock. However,
volatiles must be used with caution; almost all programmers
should forego the use of volatiles in favor of using locking.
Our detector for this bug pattern looks for sequences of
bytecode containing an ifnull instruction, followed by a
monitorenter instruction, followed by another ifnull. This
implementation catches many instances of double checked
lo cking, with a low false positive rate. We have experi-
mented with more sophisticated detectors for this pattern,
but we found that they were not significantly more accurate.
An example of incorrect double checked locking found in
JBoss-4.0.0DR3 is shown in Figure 3.
3.3 Wait Not In Loop
Java monitors support notify and wait operations to al-
low threads to wait for a condition related to the state of
a shared object. For example, in a multithreaded blocking
queue class, the dequeue() method would wait for the queue
to become nonempty, and the enqueue() method would per-
form a notify to wake up any threads waiting for the queue
to become nonempty.
Often, a single Java monitor is used for multiple condi-
tions. For such classes, the correct idiom is to surround the
call to wait with a loop which repeatedly checks the condi-
tion. Without the loop, the thread would not know whether
the condition is actually true when the call to wait returns.
The condition can fail to be true for several reasons:
The monitor is being used for multiple conditions, so
the condition set by the notifying thread may not be
the one the waiting thread is waiting for.
In between the notification and the return from wait,
another thread obtains the lock and changes the con-
dition. For example, a thread might be waiting for a
queue to become non-empty. A thread inserts a new el-
ement into the queue and notifies the waiting thread.
Before the waiting thread acquires the lo ck, another
thread removes the element from the queue.
The specification for the wait method allows it to spu-
riously return for no reason. This can arise due to
// org.jboss.net.axis.server, JBossAuthenticationHandler.java,
// line 178
public void invoke(MessageContext msgContext) throws AxisFault {
// double check does not work on multiple processors, unfortunately
if (!isInitialised) {
synchronized (this) {
if (!isInitialised) {
initialise();
}
}
}
...
Figure 3: A doub le checked locking bu g in JBoss-4.0.0DR3. Not only does the comment indicate that the
programmer was aware the idiom is incorrect, the initialise() method writes a true value to the isInitialised
field before the object has been completely initialized, meaning that the code would not be correct on a single
processor system, even if the Java memory model were sequentially consistent.
if (!book.isReady()) {
DebugInfo.println("book not ready");
synchronized (book) {
DebugInfo.println("waiting for book");
book.wait();
}
...
Figure 4: An unconditional wait bug in an early
version of the International Children’s Digital Li-
brary. If the book becomes ready after isReady()
is called and before the lock is acquired, the notifi-
cation could be missed and the thread could block
forever.
special handling needed for the interaction of inter-
rupts and waiting, and because underlying operating
system synchronization primitives used by the JVM,
such as pthreads, allow spurious wakeups.
The detector for this bug pattern examines bytecode for a
call to wait() which is not in close proximity to the target
of a backwards branch (i.e., a loop head).
3.4 Unconditional Wait
The Unconditional Wait pattern is a special case of Wait
Not In Loop. In this bug pattern, a wait is performed imme-
diately (and unconditionally) upon entering a synchronized
blo ck. Often, this indicates that the programmer did not
include the test for the waited-for condition as part of the
scope of the lock, which could lead to a missed notification.
An example of this bug pattern is shown in Figure 4.
3.5 Other Concurrency Bug Patterns
This section presents several other concurrency bug pat-
terns. These detectors are generally not effective at finding
bugs in mature software. However, they are often useful in
finding bugs in code written by inexperienced programmers.
Mismatched Wait and Notify. In this bug pattern,
the programmer calls a wait or notify method without hold-
ing a lock on the monitor object. This will result in an
IllegalMonitorStateException being thrown. Our imple-
mentation intraprocedurally tracks the scopes of locks, and
emit a warning when a call to a wait or notify method is
seen without a lo ck being held on the receiver object.
Two Locks Held While Waiting. Signaling between
threads is often handled by using wait() and notify()/
notifyAll(). When wait() is invoked, the thread invoking
wait must hold a lock on the object on which wait is invoked,
and all locks on that object are released while the thread is
waiting. However, locks on other objects are not released.
This can cause poor performance, and can cause deadlo ck
if they thread that is trying to perform a notify needs to
acquire that lock.
This detector performs an intraprocedural analysis to find
the scope of locks, and emits a warning whenever a method
holds multiple locks whe n wait is invoked.
Calling notify() instead of notifyAll(). Whe n a shared
object is visible to multiple threads, it is possible for more
than one thread to wait on the object’s monitor. When
multiple conditions are associated with a single monitor,
the threads waiting on the monitor may be waiting for more
than one distinct condition. Because Java makes no guaran-
tee about which thread is woken w hen the notify() method
is called, it is goo d practice to use notifyAll() instead, so
that all threads are woken. Otherwise, the thread woken
might not b e the one waiting for the condition that the no-
tifier has made true.
Invoking Thread.run() instead of Thread.start().
The run() method of classes extending the Thread class de-
fines the code to be executed as the body of a new thread.
Inexperienced programmers sometimes call the run() method
directly, not realizing that it will not actually start the
thread.
Mutable lock. In order for lock objects to be use d cor-
rectly by multiple threads, they need to be created before
the threads that use them for synchronization. Therefore,
it is suspicious when a method performs synchronization on
an object loaded from a field that was assigned earlier in the
metho d.
Naked Notify. In general, calling a notify method on a
monitor is done b ecaus e some condition another thread is
waiting for has b ecome true. To correctly notify the waiting
thread (or threads) of a change in the condition, the noti-
// get next items from queue if any
while (listlock) {
;
}
listlock = true;
Figure 5: A spin loop in an early version of the
International Children’s Digital Library.
fying thread must make an update to shared state be fore
performing the notification. The detector for this bug pat-
tern looks for calls to notify methods which do not appear to
be associated with earlier updates to shared mutable state.
Because calls to notify methods always increase the live-
ness of threads in a program, instances of this pattern are
not always genuine bugs. However, misunderstanding the
correct use of wait and notify is common for inexperienced
programmers, and this detector helps inform them when
their code deviates from good practice.
Spin Wait. In this bug pattern, a method executes a
lo op which reads a non-volatile field until an expected value
is seen. Aside from the obvious waste of CPU time, there is
a more subtle issue. If the field is never as signed in the body
of the loop, the compiler may legally hoist the read out of
the loop entirely, resulting in an infinite loop. An example
of a spin lo op is shown in Figure 5.
4. EVALUATION
Because the FindBugs tool uses heuristics, it may report
warnings that are not real errors. In order to evaluate the
accuracy of the warnings pro duced by the tool, we applied
it to several applications and libraries:
rt.jar, from Sun JDK 1.5 beta build 42: this is Sun’s
implementation of the core J2SE libraries
classpath-0.08: an open source implementation of a
subset of the core J2SE libraries from the GNU project
jboss-4.0.0DR3: a popular open source application server
for Enterprise Java Beans
JacORB-2.1: an open source implementation of a CORBA
Object Request Broker (ORB)
International Children’s Digital Library 0.1: an early
version of a web-based digital library application from
the University of Maryland Human-Computer Interac-
tion Laboratory [9]
With the exception of ICDL, These are mature, production
quality applications and libraries. We manually classified
each warning as serious (a real bug), harmless (a bug un-
likely to cause incorrect behavior), and false (an inaccurate
warning). The results are shown in Table 1.
In evaluating the accuracy of the detectors, we tried to
err on the side of not marking a warning as serious unless
we felt confident that it could result in undesirable behavior
at runtime. For example, we marked some of the ‘Wait not
in loop’ warnings as harmless because they were used to
implement an infinite wait, so liveness was not an issue.
3
3
Interestingly, spurious wakeups will be allowed in the re-
vised Java memory model[15], meaning that an uncondi-
In general, the detectors achieved our target of no more
than 50% false and harmless warnings. Some detectors were
very accurate: for example, the warnings generated by the
Double Checked Locking detector were almost always accu-
rate. The Inconsistent Synchronization detector was some-
what less accurate, although still within the acceptable range,
especially considering that our tool operates without explicit
specifications of which classes and methods are intended to
be thread safe. The Unconditional Wait and Wait Not In
Loop detectors were less accurate than desired. However,
they produce only a small number of warnings, and genuine
instances of these bug patterns tend to be critical bugs that
can be very hard to debug.
4.1 Anecdotal Experience
This section describes some of our experience in applying
the FindBugs tool.
One of the applications we studied as we were develop-
ing FindBugs was an early version of the International Chil-
dren’s Digital Library [9]. In conversations with the authors,
we found out that they had spent several months tracking
down a threading bug. When we applied FindBugs to the
buggy version, it immediately found the problem. A similar
problem in a different version of the ICDL software is shown
in Figure 4.
4.1.1 Reporting Bugs Found by FindBugs
We have submitted some of the most serious bugs found
by our tool in the Java core libraries to Sun’s Java bug
database.
Using our detector for Two Lock Wait, we found a serious
potential deadlock in the com.sun.corba.se.impl.orb.ORBImpl
class of Sun’s JDK 1.5 build 32. In the get next response()
metho d, two locks are held when wait is called. Only one
of these locks is released while the threads is waiting. T he
thread can be notified by calling the notifyORB() method.
Unfortunately, before the notification can be performed, the
thread must obtain the lock still held by the thread awaiting
notification, resulting is deadlock. We reported the problem
to Sun, it was confirmed to be a bug, fixed internally, and
the fix is scheduled to be part of a future beta release.
In prerelese versions of Sun’s JDK 1.4.2, we found serious
inconsistent synchronization bugs in the append(boolean)
metho d of StringBuffer and the removeRange(int,int)
metho d of Vector. Both classes are meant to be thread-
safe, and these methods were left unsynchronized, result-
ing in exploitable race conditions. Even though these were
acknowledged to be genuine bugs by sources at Sun, the
removeRange error will only be fix ed in the 1.5 branch, not
in the 1.4 branch of Java. This illustrates an interesting
asymmetry about the software engineering issues surround-
ing concurrency bugs in commercial software:
It is easy to introduce concurrency errors in new code
It is difficult to fix these bugs once they are introduced
The reason for the asymmetry is that maintenance engi-
neers are understandably reluctant to fix bugs which cannot
be easily reproduced with a simple test case.
4
Also, mainte-
tional call to wait() does not correctly implement an infinite
wait.
4
An interesting problem here is that many concurrency er-
rors cannot be reliably reproduced by a test case.
rt-1.5-b42 classpath-0.08
warnings serious harmless false pos warnings serious harmless false p os
Double check 78 92% 0% 7% 0
Lazy static initialization 146 100% 0% 0% 10 100% 0 % 0 %
Double check
78 92% 0% 7% 0
Inconsistent sync 204 56% 31% 11% 80 48% 30% 21%
Mutable lock 1 100% 0% 0% 0
Running, not starting a thread 1 0% 0% 100% 1 0% 0% 100%
Unconditional wait 4 0% 25% 75% 2 0% 0% 100%
Wait not in loop 6 0% 16% 83% 3 0% 0% 100%
jboss-4.0.0DR3 jacorb-2.1
warnings serious harmless false pos warnings serious harmless false p os
Double check 5 80% 0% 20% 1 100% 0% 0%
Lazy static initialization 643 100% 0% 0% 579 100% 0 % 0 %
Inconsistent sync 54 37% 24% 38% 35 57% 17% 25%
Unconditional wait 3 66% 0% 33% 5 60% 0% 40%
Wait not in loop 4 0% 0% 100% 5 20% 0% 80%
icdl
warnings serious harmless false pos
Lazy static initialization 10 100% 0% 0%
Inconsistent sync 3 100% 0% 0%
Spin Wait 4 100% 0% 0%
Unconditional wait 3 66% 0% 33%
Wait not in loop 3 100% 0% 0%
Table 1: False positive rates for concurrent bug pattern detectors.
nance engineers must be concerned as to whether introduc-
ing missing synchronization could possibly introduce dead-
lo ck. Because concurrency errors are almost always difficult
to reproduce, they can linger for a long time without being
fixed. This highlights the usefulness of running static tools
to catch bugs before they are introduced into a deployed
co de base.
4.1.2 Finding Bugs in Student Projects
Programmers with different skill levels tend to make dif-
ferent kinds of mistakes. We found that bug detectors such
as Wait Not In Loop and Naked Notify did not find many
serious bugs in production quality software. However, these
detectors were very effective at finding bugs in projects writ-
ten by students in an undergraduate advanced Java pro-
gramming course; for many of the students, the course is
their first significant exp osure to concurrent programming.
Table 2 shows, for a programming project (assigned when
we taught the course in Spring 2001) where students used
threads for the first time, the number of student projects
for which FindBugs reported various kinds of concurrency
warnings. Many of these bugs are ones that we would not
expect to see in production code. For example, Mismatched
Wait/Notify bugs manifest by throwing a runtime excep-
tion (specifically, IllegalMonitorStateException), which
should be diagnosed and fixed during testing. However,
since students do not always understand the meaning of
these exceptions, they may be tempted to ignore them, or
even write handlers for them. Because students do not un-
derstand threads well, these warnings typically indicate se-
rious errors in their code.
In recent semesters when we have taught the same course,
Number of
Bug Type Students
Inconsistent synchronization 7
Mutable Lock 1
Mismatched Wait/Notify 4
Naked Notify 4
Notify instead of NotifyAll 4
Running, not starting a thread 1
Unconditional Wait 5
Any of the above 19
Table 2: Number of student projects in an un-
dergraduate advanced Java programming course for
which FindBugs generated various kinds of concur-
rency warnings.
we have given students access to both FindBugs and a dy-
namic data race detection tool, both of which have been suc-
cessful at helping students find and fix concurrency problems
in their projects. However, giving students access to these
to ols has also made it more difficult for us evaluate the effec-
tiveness of FindBugs by applying it to submitted program-
ming assignments. We are working to develop infrastructure
that will allow us to record the effectiveness of FindBugs
as students develop code. Ensuring that FindBugs reports
problems in a way that enhances students’ understanding of
concurrency issues is something we are actively pursuing.
5. RELATED WORK
Static bug checkers have a long history. The original pro-
gram checker is Lint [16], which uses simple analyses to find
common errors in C programs. LCL int [12] is similar in
spirit to the original Lint, with the addition of checking code
for consistency with specifications supplied by the program-
mer. PREfix [7] symbolically executes C and C++ programs
to find a variety of dynamic errors, such as memory corrup-
tion and out of bound array accesses. MC (for “metacom-
pilation”) [10] uses a sophisticated interprocedural analysis
to check code over large numbers of paths through an entire
system; state machines driven by program statements are
used to check correctness properties on those paths. MC
uses a novel language, called Metal, to encode the state ma-
chines, allowing checks for new properties to be added easily.
The SABER project at IBM[14] uses an approach very simi-
lar to FindBugs in order to find errors in J2EE applications.
Many static bug checkers have focused on finding con-
currency errors in software. Warlock [21] checks variables
in multithreaded C programs to determine if they are pro-
tected by a consistent set of locks; accesses to variables with
an inconsistent lockset are flagged as potential race condi-
tions. J Lint [3] performs an interprocedural analysis on Java
programs to find potential deadlocks and race conditions. In
[13], Flanagan and Qadeer describe a static analysis to find
metho ds that are not atomic; as noted earlier, programs free
of race conditions can still have atomicity bugs. RacerX [11]
is system for finding race conditions and deadlocks in C pro-
grams. Its analysis is very s imilar to that performed by MC;
however, some new analysis techniques are introduced, in-
cluding “unlockset” analysis, which can be thought of as
lo ckset analysis backward in time. Using unlocksets can
increase the confidence of reports for unsynchronized field
accesses over using locksets alone.
A variety of dynamic techniques and tools have been de-
veloped to help find and diagnose concurrency errors. Eraser
[20] dynamically computes the set of locks held during ac-
cesses to shared data. Accesses to the same location with
inconsistent lock sets are potential bugs. JProbe [17] dy-
namically monitors a running Java program to detect race
conditions and deadlocks.
6. CONCLUSIONS
From our studies of concurrency bugs, we conclude that
many programmers have fundamental misconceptions about
how to write correct programs using threads. The intuition
many programmers have about how multithreaded programs
ought to work is flawed. Some of this can be attributed to
inaccurate information (such as the books and articles advo-
cating double checked locking). Some can be attributed to
inadequate educations—threads and concurrency are gener-
ally considered only briefly in the undergraduate Computer
Science curriculum, with more in-depth treatment coming
only in electives. Finally, modern multiprocessor architec-
ture and aggressive optimizing compilers can lead to sur-
prising and subtle b ehaviors in multithreaded programs.
We believe that static checking tools can aid programmers
in two important ways. First, they can help find bugs in
software. Second, and perhaps more importantly, they can
help educate programmers ab out error-prone idioms arising
from misconceptions about threads and concurrency.
In future work, we would like to develop detectors for
other kinds of concurrency errors, and continue to improve
the accuracy of the existing detectors. We would also like
to evaluate the extent to which static tools can help inexpe-
rience programmers learn to use threads correctly.
7. ACKNOWLEDGMENTS
Some of the bug detectors implemented in FindBugs were
suggested by Doug Lea and Josh Bloch. We would like to
thank Jeremy Manson and Jaime Spacco for helpful feed-
back on this paper. Finally, we would like to thank the
anonymous reviewers for their insights and suggestions.
8. REFERENCES
[1] S. V. Adve and K. Gharachorloo. Shared memory
consistency models: A tutorial. IEEE Computer,
29(12):66–76, 1996.
[2] Phillip G. Armour. The five orders of ignorance.
Commun. ACM, 43(10):17–20, 2000.
[3] Cyrille Artho and Armin Biere. Applying static
analysis to large-scale, multi-threaded Java. In
Proceedings of the 13th Australian Software
Engineering Conference, pages 68–75, August 2001.
[4] David F. Bacon, Ravi Konuru, Chet Murthy, and
Mauricio Serrano. Thin locks: featherweight
synchronization for Java. In Proceedings of the ACM
SIGPLAN 1998 conference on Programming language
design and implementation, pages 258–268. ACM
Press, 1998.
[5] David F. Bacon, Robert E. Strom, and Ashis
Tarafdar. Guava: a dialect of Java without data races.
In Proceedings of the 15th ACM SIGPLAN conference
on Object-oriented programming, systems, languages,
and applications, pages 382–400. ACM Press, 2000.
[6] Chandrasekhar Boyapati, Robert Lee, and Martin
Rinard. Ownership types for safe programming:
preventing data races and deadlocks. In Proceedings of
the 17th ACM SIGPLAN conference on
Object-oriented programming, systems, languages, and
applications, pages 211–230. ACM Press, 2002.
[7] William R. Bush, Jonathan D. Pincus, and David J.
Sielaff. A Static Analyzer for Finding Dynamic
Programming Errors. Software—Practice and
Experience, 30:775–802, 2000.
[8] The “double-checked locking is broken” declaration.
http://www.cs.umd.edu/users/pugh/java/
memoryMo del/DoubleCheckedLocking.html.
[9] A. Druin, Ben Bederson, A. Weeks, A. Farb er,
J. Grosjean, M.L. Guha, J.P. Hourcade, J. Lee,
S. Liao, K. Reuter, A. Rose, Y. Takayama, L., and
L Zhang. The international children’s digital library:
Description and analysis of first use. Technical Report
HCIL-2003-02, Human-Computer Interaction Lab,
Univ. of Maryland, January 2003.
[10] D. Engler, B. Chelf, A. Chou, and S. Hallem.
Checking System Rules Using System-Specific,
Programmer-Written Compiler Extensions. In
Proceedings of the Fourth Symposium on Operating
Systems Design and Implementation, San Diego, CA,
Octob e r 2000.
[11] Dawson Engler and Ken Ashcraft. RacerX: effective,
static detection of race conditions and deadlocks. In
Proceedings of the nineteenth ACM symposium on
Operating systems principles, pages 237–252, 2003.
[12] David Evans, John Guttag, James Horning, and
Yang Meng Tan. LCLint: A tool for using
specifications to check code. In Proceedings of the
ACM SIGSOFT ’94 Symposium on the Foundations of
Software Engineering, pages 87–96, 1994.
[13] Cormac Flanagan and Shaz Qadeer. A type and effect
system for atomicity. In Proceedings of the ACM
SIGPLAN 2003 conference on Programming language
design and implementation, pages 338–349, 2003.
[14] J2EE Code Validation Preview for WebSphere Studio.
http://www-106.ibm.com/developerworks/websphere/
downloads/j2ee code validation.html.
[15] Java Specification Request (JSR) 133. Java memory
mo del and thread specification revision, 2004.
http://jcp.org/jsr/detail/133.jsp.
[16] S. Johnson. Lint, a C Program Checker, Unix
Programmer’s Manual, AT&T Bell Laboratories, 1978.
[17] Quest Software JProbe Threadalyzer.
http://www.quest.com/threadalyzer.jsp.
[18] Kiyokuni Kawachiya, Akira Koseki, and Tamiya
Onodera. Lock reservation: Java locks can mostly do
without atomic operations. In Proceedings of the 17th
ACM SIGPLAN conference on Object-oriented
programming, systems, languages, and applications,
pages 130–141. ACM Press, 2002.
[19] John K. Ousterhout. Why threads are a bad idea (for
most purposes). Invited talk, USENIX 1996 Technical
Conference.
[20] Stefan Savage, Michael Burrows, Greg Nelson, Patrick
Sobalvarro, and Thomas Anderson. Eraser: A
dynamic data race d etector for multithreaded
programs. ACM Transactions on Computer Systems,
15(4):391–411, 1997.
[21] Nicholas Sterling. WARLOCK: A Static Data Race
Analysis Tool. In Proceedings of the USENIX Winter
Technical Conference, pages 97–106, January 1993.
... If so, an empty SLT tree is created (line 9) and checkNesting 3 is called to check whether there exist nested locks (line 10). If there are nested locks, they are all added into tree (lines [11][12]. Only the current checking object i is added otherwise (line 14). ...
... If targetChild and curChild are same (nodes node1 v1 and node2 v2 are same when v1 = v2), their subtrees are merged recursively (lines 7-10). If no same node can be found, targetChild is added as a new child (lines [11][12][13][14][15][16][17][18]. ...
... Indeed, many programmers believe that it is a bad practice to acquire locks on variables of primitive types. For example, Findbugs [12] determines such locks as bugs 4 . Therefore, we assume developers follow the good practise, but spend more resources on checking the locks on non-primitive variables for efficiency. ...
Conference Paper
Full-text available
Detecting lock-related defects has long been a hot research topic in software engineering. Many efforts have been spent on detecting such deadlocks in concurrent software systems. However, latent locks may be hidden in application programming interface (API) methods whose source code may not be accessible to developers. Many APIs have latent locks. For example, our study has shown that J2SE alone can have 2,000+ latent locks. As latent locks are less known by developers, they can cause deadlocks that are hard to perceive or diagnose. Meanwhile, the state-of-the-art tools mostly handle API methods as black boxes, and cannot detect deadlocks that involve such latent locks. In this paper, we propose a novel black-box testing approach, called LockPeeker, that reveals latent locks in Java APIs. The essential idea of LockPeeker is that latent locks of a given API method can be revealed by testing the method and summarizing the locking effects during testing execution. We have evaluated LockPeeker on ten real-world Java projects. Our evaluation results show that (1) LockPeeker detects 74.9% of latent locks in API methods, and (2) it enables state-of-the-art tools to detect deadlocks that otherwise cannot be detected.
... A major application of static analysis in the domain of concurrency has been in guaranteeing safety or finding bugs. There are techniques for ensuring safe programming using advanced typing [5], analyses for static race detection [44], approaches to concurrency bug fixing [32], tools that flag suspicious concurrency patterns [30,50], and much more. Additionally, dynamic techniques for concurrency bug detection [1,13,34,41,46] often benefit from symbolic reasoning, especially in approaches inspired by model checking or symbolic execution techniques [33,37,43]. ...
Article
Explicit signaling between threads is a perennial cause of bugs in concurrent programs. While there are several run-time techniques to automatically notify threads upon the availability of some shared resource, such techniques are not widely-adopted due to their run-time overhead. This paper proposes a new solution based on static analysis for automatically generating a performant explicit-signal program from its corresponding implicit-signal implementation. The key idea is to generate verification conditions that allow us to minimize the number of required signals and unnecessary context switches, while guaranteeing semantic equivalence between the source and target programs. We have implemented our method in a tool called Expresso and evaluate it on challenging benchmarks from prior papers and open-source software. Expresso-generated code significantly outperforms past automatic signaling mechanisms (avg. 1.56x speedup) and closely matches the performance of hand-optimized explicit-signal code.
... A major application of static analysis in the domain of concurrency has been in guaranteeing safety or finding bugs. There are techniques for ensuring safe programming using advanced typing [5], analyses for static race detection [44], approaches to concurrency bug fixing [32], tools that flag suspicious concurrency patterns [30,50], and much more. Additionally, dynamic techniques for concurrency bug detection [1,13,34,41,46] often benefit from symbolic reasoning, especially in approaches inspired by model checking or symbolic execution techniques [33,37,43]. ...
Conference Paper
Explicit signaling between threads is a perennial cause of bugs in concurrent programs. While there are several run-time techniques to automatically notify threads upon the availability of some shared resource, such techniques are not widely-adopted due to their run-time overhead. This paper proposes a new solution based on static analysis for automatically generating a performant explicit-signal program from its corresponding implicit-signal implementation. The key idea is to generate verification conditions that allow us to minimize the number of required signals and unnecessary context switches, while guaranteeing semantic equivalence between the source and target programs. We have implemented our method in a tool called Expresso and evaluate it on challenging benchmarks from prior papers and open-source software. Expresso-generated code significantly outperforms past automatic signaling mechanisms (avg. 1.56x speedup) and closely matches the performance of hand-optimized explicit-signal code.
... A major application of static analysis in the domain of concurrency has been in guaranteeing safety or finding bugs. There are techniques for ensuring safe programming using advanced typing [5], analyses for static race detection [43], approaches to concurrency bug fixing [31], tools that flag suspicious concurrency patterns [29,49], and much more. Additionally, dynamic techniques for concurrency bug detection [1,13,33,40,45] often benefit from symbolic reasoning, especially in approaches inspired by model checking or symbolic execution techniques [32,36,42]. ...
Article
Explicit signaling between threads is a perennial cause of bugs in concurrent programs. While there are several run-time techniques to automatically notify threads upon the availability of some shared resource, such techniques are not widely-adopted due to their run-time overhead. This paper proposes a new solution based on static analysis for automatically generating a performant explicit-signal program from its corresponding implicit-signal implementation. The key idea is to generate verification conditions that allow us to minimize the number of required signals and unnecessary context switches, while guaranteeing semantic equivalence between the source and target programs. We have implemented our method in a tool called Expresso and evaluate it on challenging benchmarks from prior papers and open-source software. Expresso-generated code significantly outperforms past automatic signaling mechanisms (avg. 1.56x speedup) and closely matches the performance of hand-optimized explicit-signal code.
Chapter
Lots of concurrent software is being developed for the now ubiquitous multicore processors. And concurrent programming is difficult because it is quite easy to introduce errors that are really hard to diagnose and fix. One of the main obstacles to concurrent programming is that threads are scheduled nondeterministically and their interactions may become hard to predict and to devise. This chapter addresses the nature of concurrent programming and some classes of concurrency errors. It discusses the application of dynamic program analysis techniques to detect, locate and diagnose some common concurrency errors like data races, atomicity violations and deadlocks. This chapter also mentions some techniques that can help with quality assurance of concurrent programs, regardless of any particular class of concurrency errors, like noise injection and systematic testing, and it is closed by some prospects of concurrent software development.
Conference Paper
Full-text available
Because of the built-in support for multi-threaded programming, Java programs perform many lock operations. Although the overhead has been significantly reduced in the recent virtual machines, One or more atomic operations are required for acquiring and releasing an object's lock even in the fastest cases.This paper presents a novel algorithm called lock reservation. It exploits thread locality of Java locks, which claims that the locking sequence of a Java lock contains a very long repetition of a specific thread. The algorithm allows locks to be reserved for threads. When a thread attempts to acquire a lock, it can do without any atomic operation if the lock is reserved for the thread. Otherwise, it cancels the reservation and falls back to a conventional locking algorithm.We have evaluated an implementation of lock reservation in IBM's production virtual machine and compiler. The results show that it achieved performance improvements up to 53% in real Java programs.
Conference Paper
Full-text available
Static analysis is a tremendous help when trying to find faults in complex software. Writing multi-threaded pro- grams is difficult, because the thread scheduling increases the program state space exponentially, and an incorrect thread synchronization produces faults that are hard to find. Program checkers have become sophisticated enough to find faults in real, large-scale software. In particular,Jlint, a very fast Java program checker, can check packages in a highly automated manner. The original version,Jlint1, still lacked full support for synchronization statements in Java. We extended Jlint1's model to include synchronizations on arbitrary objects, and named our versionJlint2. Our sta- tistical analysis proves that these extensions are relevant and useful. Applying Jlint2 to various large software pack- ages, including commercial packages from Trilogy, found 12 faults, two of which related to multi-threading.
Conference Paper
This paper describes RacerX, a static tool that uses flow-sensitive, interprocedural analysis to detect both race conditions and deadlocks. It is explicitly designed to find errors in large, complex multithreaded systems. It aggressively infers checking information such as which locks protect which operations, which code contexts are multithreaded, and which shared accesses are dangerous. It tracks a set of code features which it uses to sort errors both from most to least severe. It uses novel techniques to counter the impact of analysis mistakes. The tool is fast, requiring between 2-14 minutes to analyze a 1.8 million line system. We have applied it to Linux, FreeBSD, and a large commercial code base, finding serious errors in all of them. RacerX is a static tool that uses flow-sensitive, interprocedural analysis to detect both race conditions and deadlocks. It uses novel strategies to infer checking information such as which locks protect which operations, which code contexts are multithreaded, and which shared accesses are dangerous. We applied it to FreeBSD, Linux and a large commercial code base and found serious errors in all of them.
Conference Paper
Language-supported synchronization is a source of serious performance problems in many Java programs. Even single-threaded applications may spend up to half their time performing useless synchronization due to the thread-safe nature of the Java libraries. We solve this performance problem with a new algorithm that allows lock and unlock operations to be performed with only a few machine instructions in the most common cases. Our locks only require a partial word per object, and were implemented without increasing object size. We present measurements from our implementation in the JDK 1.1.2 for AIX, demonstrating speedups of up to a factor of 5 in micro-benchmarks and up to a factor of 1.7 in real programs.
Article
The memory consistency model of a system affects performance, programmability, and portability. We aim to describe memory consistency models in a way that most computer professionals would understand. This is important if the performance-enhancing features being incorporated by system designers are to be correctly and widely used by programmers. Our focus is consistency models proposed for hardware-based shared memory systems. Most of these models emphasize the system optimizations they support, and we retain this system-centric emphasis. We also describe an alternative, programmer-centric view of relaxed consistency models that describes them in terms of program behavior, not system optimizations