TrInc: Small Trusted Hardware for Large Distributed Systems.

Full text

Page 1

USENIX Association NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation 1
TrInc: Small Trusted Hardware for Large Distributed Systems
Dave LevinJohn R. Douceur
Microsoft Research
Jacob R. Lorch
Microsoft Research
Thomas Moscibroda
Microsoft ResearchUniversity of Maryland
Abstract
A simple yet remarkably powerful tool of selfish and
malicious participants in a distributed system is “equiv-
ocation”: making conflicting statements to others. We
present TrInc, a small, trusted component that combats
equivocation in large, distributed systems. Consisting
fundamentally of only a non-decreasing counter and a
key, TrInc provides a new primitive: unique, once-in-a-
lifetime attestations.
We show that TrInc is practical, versatile, and easily
applicable to a wide range of distributed systems. Its
deployment is viable because it is simple and because
its fundamental components—a trusted counter and a
key—are already deployed in many new personal com-
puters today. We demonstrate TrInc’s versatility with
three detailed case studies: attested append-only mem-
ory (A2M), PeerReview, and BitTorrent.
We have implemented TrInc and our three case stud-
ies using real, currently available trusted hardware.
Our evaluation shows that TrInc eliminates most of
the trusted storage needed to implement A2M, signifi-
cantly reduces communication overhead in PeerReview,
and solves an open incentives issue in BitTorrent. Mi-
crobenchmarks of our TrInc implementation indicate di-
rections for the design of future trusted hardware.
1Introduction
As wide-area systems grow in scale, so do their ex-
posure to threats. Much of the interesting distributed-
systems research of the past decade has focused on the
issues of security and adversarial incentive that are inher-
ent to large-scale systems. This research has addressed a
wide range of applications, including storage [2, 16, 19,
22, 28], communication [4, 45, 30], databases [40], con-
tent distribution [15, 24, 32, 36], grid computation [12],
and games [3, 10], in addition to generic infrastruc-
ture [1, 5, 9, 18, 23, 43]. Virtually all of this work shares
a common supposition, namely that the individual com-
ponents in the system are completely untrusted.
Recently, the necessity of this supposition has been
called into question. The Attested Append-only Mem-
ory (A2M) system by Chun et al. [7] showed that a small
trusted module in each distributed component can signif-
icantly improve system security. In addition to found-
ing this important new research direction, A2M made
two key contributions: First, they proposed a particu-
lar abstraction for such a module, namely a trusted log.
Second, they showed specifically that their proposed ab-
straction could improve the degree of fault tolerance
to Byzantine faults in the server components of client-
server systems.
Despite our appreciation for this work, we are con-
cerned that distributed-protocol designers may be reluc-
tant to start assuming the availability of such trusted
modules. We have two reasons for this concern: First,
the abstraction of a trusted log may require more stor-
age space and complexity than researchers are comfort-
able assuming, particularly for an embedded module in-
side a potentially hostile component. Second, designers
may have difficulty appreciating how broadly applicable
a trusted module can be to distributed protocols.
In this paper, we continue the research direction begun
by A2M, with an eye toward addressing these two issues.
First, we have developed a significantly smaller abstrac-
tion: Instead of a trusted log, we propose a trusted in-
crementer (TrInc), which is little more than a monotonic
counter and a key. Second, we demonstrate a more inclu-
sive set of architectures, running a broader range of pro-
tocols, yielding a wider set of benefits: Our architectures
include not only client-server systems but also peer-to-
peer systems. Our protocols include not only Byzantine-
fault-tolerantprotocolsbutalsoPeerReview[13]andBit-
Torrent [8]. Our demonstrated benefits include not only
improving fault tolerance but also reducing communica-
tion overhead and solving an open incentive problem.
We show that TrInc has several benefits over A2M.
First, its smaller size and simpler semantics make it
easier to deploy, as we demonstrate by implementing
it on real, currently available trusted hardware.
ond, we observe that TrInc’s core functional elements
are included in the Trusted Platform Module (TPM) [38]
found on many modern PCs, lending credence to the
idea that such a component could become widespread.
Third, TrInc makes use of a shared symmetric session
key among all participants in a protocol instance, which
significantly decreases the cryptographic overhead.
The rest of this paper is structured as follows. §2 pro-
vides background on the underlying problem addressed
by TrInc (and by A2M), as well as a primer on trusted
hardware. §3 then presents the design of TrInc, and §4
analyzes its security. §§5, 6, and 7 respectively describe
several protocols we modified to use TrInc, our trusted
hardware implementation, and our evaluation thereof.
Sec-

Page 2

2 NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation USENIX Association
Accountability layer
PeerReview [13]
∗


Trusted module
A2M [7]
Property
No centralized trust
Easy to deploy
Easy to apply to existing protocols
Immediate consistency
No assumptions about protocol’s determinism
No additional online assumptions
Additional communication overhead per protocol
message, with witness sets of size W
Table 1: Summary of the properties of various equivocation-fighting systems.∗While PeerReview and Nysiad do not
require centralized trust, they do make use of a PKI.†Nysiad deals with nondeterminism by treating nondeterministic
events as inputs; this requires protocol changes for nondeterministic state machines.‡We found that, although TrInc
requires a protocol redesign, the modifications are often localized, and vastly simplify security procedures.
Nysiad [14]
∗

†

†
TrInc

‡



O(1)



O(W2)
O(W2)
O(1)
2
2.1
Background and Related Work
Equivocation in distributed systems
Since 1982, it has been known that tolerating f Byzan-
tine faults requires at least 3f +1 participants [20]. This
stands in marked contrast to the case for f stopping
faults, which more intuitively requires 2f + 1 partici-
pants. A key insight behind A2M [7] was the observation
that a single property of Byzantine faults is responsible
for the difference between these two bounds. That prop-
erty is equivocation, meaning the ability to make con-
flicting statements to different participants. A2M pro-
vides a mechanism that prevents participants from equiv-
ocating, thereby improving the fault tolerance of Byzan-
tine protocols to f out of 2f + 1.
We make the further observation that equivocation is a
necessarypropertyformanyformsofcheatingandfraud,
not merely for classical Byzantine faults. For instance,
in BitTorrent, recent work [21] has shown an exploit in
which a peer can obtain an unfairly high download rate
by lying about which chunks of a file it has received.
This is equivocation, insofar as the peer acknowledges
receiving a chunk from the peer that provided it, but then
tells another peer that it does not have the chunk.
The following are three more brief examples:
• In a simultaneous-turn game, one can cheat by ob-
serving an opponent’s move before making one’s
own move; this is equivocating about whether one
has yet moved.
• In a distributed electronic currency system, one can
counterfeit money by equivocating to different pay-
ees about whether one has spent a particular bill.
• In an election, the tallier can disrupt the vote by
equivocatingtoavoterandanofficialaboutwhether
the voter’s vote was recorded.
In §5.5, we will consider many other cases of mali-
cious behavior that can be interpreted as equivocation.
2.2Prior solutions to equivocation
Several recent efforts have addressed the problem of
Byzantine faults in distributed systems. Although their
approaches to the problem are very different, they have
all effectively focused on the issue of equivocation. Ta-
ble 1 summarizes our analysis of their properties.
PeerReview [13] is a system that employs witnesses to
collect a tamper-evident record of all messages in a dis-
tributed system for subsequent checking against a refer-
ence implementation. Unlike the remaining approaches
we will discuss, PeerReview does not provide fault toler-
ance. Instead, it provides eventual fault detection and
localization, which the system’s designers argue leads
to fault deterrence. The tamper-evident record is a dis-
tributed collection of logs that are authenticated using
hash chains. The purpose of the tamper-evidence is to
detect equivocation about the messages recorded in a
log. As shown in Table 1, the communication required
to collectively manage the tamper-evident message log
is quadratic in the size of the witness set.
Nysiad [14] is a mechanism that transforms crash-
tolerant distributed systems into Byzantine-fault-tolerant
ones. It does this by assigning a set of guards (compara-
ble to witnesses) to each host in the system. The guards
validate the messages sent by their associated hosts, us-
ing replicas of the hosts’ execution engines. The po-
tential for equivocation in Nysiad is that the host might
send different messages to different guards or order its
messages differently for different guards. To deal with
this equivocation, the guards gossip among each other to
agree on the order and content of messages sent by the
host. As shown in Table 1, this gossip requires a count
of messages that is quadratic in the number of guards.
Relative to PeerReview, Nysiad has the benefit of imme-
diate consistency, rather than eventual detection. Nysiad
is also able to handle nondeterministic state machines,
but doing so requires protocol changes to treat nondeter-
ministic events as inputs.
Attested Append-only Memory, or A2M [7], is a

Page 3

USENIX Association NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation 3
trusted module that is embedded in an untrusted ma-
chine, for the purpose of improving the fault tolerance of
a distributed protocol. The A2M module provides the ab-
straction of a trusted log, which the machine can append
to but not otherwise modify. This limitation prevents the
machine from equivocating about whether it performed
a particular action at a particular step, because once the
action is recorded in the log, it cannot be overwritten.
A2M uses cryptography to enforce its properties and to
attest the log’s contents to other machines. Relative to
Nysiad and PeerReview, A2M does not require any addi-
tional online communication between machines beyond
what is required in the base protocol. Consequently, the
communication overhead is merely a constant factor due
to the cryptographic attestations that accompany the pro-
tocol’s messages.
As we will show in §3, TrInc is significantly smaller
than A2M, making it easier to deploy. TrInc also has
another advantage, namely that its use is less tightly
coupled to the distributed protocol than use of A2M is.
Specifically, because A2M’s trusted log has finite stor-
age, it provides a log-truncation operation, but opportu-
nities to truncate the log may be limited by the protocol.
Conversely, message sequencing in the protocol may be
constrained by the available space in A2M’s log. Perhaps
in part to address this concern, A2M considered various
implementations in addition to hardware, some of which
would likely have plentiful storage for the log. These in-
clude a remote service, a software-isolated process, and
a memory-isolated virtual machine. By contrast, the pro-
tocol modifications required to use TrInc tend to be quite
localized. Furthermore, TrInc’s use of a shared session
key often simplifies the protocol.
2.3 Trusted hardware
There have been many trusted hardware designs that
predate both TrInc and A2M. Perhaps most similar
to TrInc is the abstraction of virtual monotonic coun-
ters [34]. These are similar to the four increment-
only counters included in the current specification of
the TPM [38]. Van Dijk et al. propose an algorithm
by which to emulate multiple counters with a single
trusted counter [39].We believe a similar approach
could ease TrInc’s deployment by requiring fewer physi-
cal counters. Further, other systems have been proposed
that make use of trusted hardware, such as for securing
database systems [26] and auctions [31]. To the best of
our knowledge, TrInc is the first trusted component de-
signed to be used in large-scale, distributed systems.
3 TrInc Design
3.1Design Goals
The fundamental security goal of TrInc is to remove
participants’ ability to equivocate. Consider the situation
in which Mallory wishes to send conflicting messages
to Alice and Bob. Common approaches to combating
such equivocation require Alice and Bob to communi-
cate with one another [13, 14, 20] or with a third party,
so they can learn of the distinct messages sent to each.
Unfortunately, this additional communication overhead
can become a bottleneck for the overlying system, and
constitutes the super-linear number of messages in Peer-
Review [13].
One goal of TrInc is to therefore minimize both com-
munication overhead and the number of non-faulty par-
ticipantsrequired. Withtrustedhardware, itispossibleto
remove Mallory’s ability to equivocate without any com-
munication between Alice and Bob [7].
The other broad goal of TrInc is to be practical for dis-
tributed systems today. To be practical, a trusted com-
ponent must be small so that it is feasible to manufacture
and deploy. Arbitrary computation and a large amount of
storage are difficult and costly to make tamper-resistant.
Further, to be a practical primitive in distributed systems,
the trusted component must have an API with which it is
easy to build distributed systems.
3.2Overview
To gain the benefits of TrInc, a user must attach a
trusted piece of hardware we call a trinket to his com-
puter. Unlike a typical TPM, which must attest to states
of the associated computer, the trinket’s API depends
only on its internal state, so the trinket does not need
access to the state of the computer. All it needs is an un-
trusted channel over which it can receive input and pro-
duce output, so even USB is quite sufficient.
When Mallory wishes to send a message m to Al-
ice, she must include an attestation from her trinket that
(1) binds m to a certain value of a counter, and (2) en-
sures Alice that no other message will ever be bound to
that value of that counter, even messages sent to other
users.A trinket enables such attestation by using a
counter that monotonically increases with each new at-
testation. In this way, once Mallory has bound a message
m to a certain counter value c, she will never be able to
bind a different message mto that value.
As we show in our case studies in §5, some protocols
benefit from using multiple counters. In theory, any-
thing done with multiple counters can be done with a
single counter, but multiple counters allow certain per-
formance optimizations and simplifications, such as as-
signing semantic meaning to a particular counter value.
Furthermore, the user of a trinket may participate in mul-
tiple protocols, each requiring its own counter or coun-
ters. Therefore, a trinket provides the ability to allo-
cate new counters. However, we must identify each of
them uniquely so that a malicious user cannot create a
new counter with the same identity as an old counter
and thereby attest to a different message with the same
counter identity and value.
As a performance optimization, TrInc allows its attes-
tations to be signed with shared symmetric keys, which

Page 4

4 NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation USENIX Association
vastly improves its performance over using asymmetric
cryptography or even secure hashes. To ensure that par-
ticipants cannot generate arbitrary attestations, the sym-
metric key is stored in trusted memory, so that users can-
not read it directly. Symmetric keys are shared among
trinkets using a mechanism that ensures they will not be
exposed to untrusted parties.
3.3Notation
We use the notation xKto mean an attestation of x
that could only be produced by an entity knowing K. If
K is a symmetric key, then this attestation can be verified
only by entities that know K; if K is a private key, then
this attestation can be verified by anyone, or more accu-
rately anyone who knows the corresponding public key.
We use the notation {x}Kto mean the value x encrypted
with public key K, so that it can only be decrypted by
entities knowing the corresponding private key.
3.4TrInc state
Figure 1 describes the full internal state of a trinket,
which we describe in more detail here. Each trinket is
endowedbyitsmanufacturerwithauniqueidentityI and
a public/private key pair (Kpub,Kpriv). Typically, I will
be the hash of Kpub. The manufacturer also includes in
the trinket an attestation A that proves the values I and
Kpubbelong to a valid trusted trinket, and therefore that
the corresponding private key is unknown to untrusted
parties.
We leave open the question of what form A will take.
This attestation is meant to be evaluated by users, not by
trinkets, and so can be of various forms. For instance,
it might be a certificate chain leading to a well-known
authority trusted to oversee trinket production and ensure
their secrets are well kept.
Another element of the trinket’s state is the meta-
counter M. Whenever the trinket creates a new counter,
it increments M and gives the new counter identity M.
This allows users to create new counters at will, with-
out sacrificing the non-monotonicity of any particular
counter. Because M only goes up, once a counter has
been created it can never be recreated by a malicious user
attempting to reset it.
Yet another element is Q, a limited-size FIFO queue
containing the most recent few counter attestations gen-
erated by the trinket. It is useful for allowing users to
recover from power failures, as we will describe later.
Thefinalpartofatrinket’sstateisanarrayofcounters,
not all of which have to be in use at a time. For each in-
use counter, the state includes the counter’s identity i, its
current value c, and its associated key K. The identity
i is, as described before, the value of the meta-counter
when the counter was created. The value c is initialized
to 0 at creation time and cannot go down. The key K
contains a symmetric key to use for attestations of this
counter; if K = 0, attestations will use the private key
Kprivinstead.
Global state:
Notation
Kpriv
Kpub
I
A
M
Meaning
Unique private key of this trinket
Public key corresponding to Kpriv
ID of this trinket, the hash of Kpub
Attestation of this trinket’s validity
Meta-counter: the number of counters
this trinket has created so far
Limited-size FIFO queue containing the
most recent few counter attestations gen-
erated by this trinket
Q
Per-counter state:
Meaning
Identity of this counter, i.e., the value of
M when it was created
Current value of the counter (starts at 0,
monotonically non-decreasing)
Key to use for attestations, or 0 if Kpriv
should be used instead
Notation
i
c
K
Figure 1: State of a trinket
3.5TrInc API
Figure 2 shows the full API of a trinket, described in
more detail in this subsection.
3.5.1Generating attestations
The core of TrInc’s API is Attest. Attest takes
three parameters: i, c, and h. Here, i is the identity of
a counter to use, cis the requested new value for that
counter, and h is a hash of the message m to which the
user wishes to bind the counter value. Attest works as
follows:
Algorithm 1 Attest(i, c, h, n)
1. Assert that i is the identity of a valid counter.
2. Let c be the value of that counter, and K be the key.
3. Assert no roll-over: c ≤ c.
4. If K = 0, then let a ← I,i,c,c,hK; otherwise
let a ← I,i,c,c,hKpriv.
5. Insert a into Q, kicking out oldest value.
6. Update c ← c.
7. Return a.
Note that Attest allows calls with c= c. This is
crucial to allowing peers to attest to what their current
counter value is without incrementing it. To allow for
this while still keeping peers from equivocating, TrInc
includes both the prior counter value and the new one.
One can easily differentiate attestations intended to learn
a trinket’s current counter value (c = c) from attesta-
tions that bind new messages (c < c).
3.5.2 Verifying attestations
Suppose a user Alice with trinket A wants to send a
message to user Bob with trinket B. She first invokes

Page 5

USENIX Association NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation 5
Function
Attest(i,c,h)
Operation
Verifies that i is a valid counter with some value c and key K. Verifies
that c ≤ c. Creates an attestation a = COUNTER,I,i,c,c,hK; if
K = 0, uses Kprivinstead of K. Adds a to Q. Sets c = c. Returns a.
Returns a certificate of this trinket’s validity: (I,Kpub,A).
Returns a boolean indicating whether a is the output of invoking
Attest on a trinket using the same symmetric key as the one associated
with counter i.
Increments M. Creates a new counter with i = M, c = 0, and K = 0.
Returns i.
If i is the identity of a valid counter, deletes that counter.
Verifies that S is an encrypted symmetric key decryptable with Kpriv.
Decrypts it and installs the included key as K for counter i.
Returns Q.
Figure 2: API of a trinket
GetCertificate()
CheckAttestation(a, i)
CreateCounter()
FreeCounter(i)
ImportSymmetricKey(S,i)
GetRecentAttestations()
Attest on her trinket using the message’s hash, and
thereby obtains an attestation a. Next, she sends the mes-
sage to Bob along with this attestation. However, for Bob
to accept this message, he needs to be convinced that the
attestation was created by a valid trinket. There are two
cases to consider: first, that the attestation used A’s pri-
vate key KA
shared symmetric key K.
In the first case, the API call GetCertificate will
be useful. This call returns a certificate C of the form
(I,Kpub,A), where I is the trinket’s identity, Kpubis
its public key, and A is an attestation that I and Kpub
belong to a valid trinket. Alice can call this API routine
and send the resulting certificate CAto Bob. Bob can
then (a) learn Alice’s public key KA
that this is a valid trinket’s public key. After this, he can
verify the attestation Alice attached to her message, and
any future attestations she attaches to messages.
Inthesecondcase,
CheckAttestation
is
CheckAttestation(a, i) is invoked on a trin-
ket, the trinket checks whether a is the output of
invoking Attest on a trinket using the same symmetric
key as the one associated with the local counter i. It
returns a boolean indicating whether this is so. So, if
Alice sends Bob an attestation signed with a shared
symmetric key, Bob can invoke CheckAttestation
on his trinket to learn whether the attestation is valid.
3.5.3Allocating counters
Since a trinket may contain many counters, another
important component of TrInc’s API is the creation of
these counters. TrInc creates new logical counters, and
allows counters to be deleted, but never resets an ex-
isting counter.Logical counters are identified by a
unique ID, generated using a non-deletable, monotonic
meta-counter M. Every trinket has precisely one meta-
counter, and when it expires, the trinket can no longer be
used; we compensate for this by making M 64 bits, only
incrementing M, and assigning no semantic meaning to
priv, and second, that the attestation used a
pub, and (b) verify
theAPIcall
useful.When
M’s value. TrInc exports a CreateCounter function
that increments M; allocates a new counter with identity
i = M, initial value 0, and initial key K = 0; and re-
turns this new identity i. When the user no longer needs
the counter, she may call FreeCounter to free it and
thereby provide space in the trinket for a new counter.
3.5.4Using symmetric keys
TrInc allows its attestations to be signed with shared
symmetric keys, which vastly improves its performance
over using asymmetric cryptography or even secure
hashes. When a set of users are willing to use a single
symmetric key for a certain purpose, we call this a ses-
sion. Creating a session requires a session administrator,
a user trusted by all participants to create a session key
and keep it safe, i.e., to not reveal it to any untrusted par-
ties.
To create a session, the session administrator simply
generates a random, fresh symmetric key as the session
key K. To allow a certain user to join the session, he
asks that user for his trinket’s certificate C. If the session
administrator is satisfied that the certificate represents a
valid trinket, he encrypts the key in a way that ensures
it can only be decrypted by that trinket. Specifically, he
creates {KEY,K}Kpub, where Kpubis the public key in
C. He then sends this encrypted session key to the user
who wants to join the session.
Upon receipt of an encrypted session key, the user can
join one of his counters to the session by using the API
call ImportSymmetricKey(S,i). This call checks
that S is a valid encrypted symmetric key, meant to be
decrypted by the local private key. If so, it decrypts the
session key and installs it as K for local counter i. From
this point forward, attestations for this counter will use
the symmetric key. Also, the user will be able to verify
any trinket’s attestation a using this symmetric key by
invoking CheckAttestation(a,i).
3.5.5Handling power failures
One practical concern is that of power failure. Unlike
A2M, TrInc users need not query the trusted hardware to