Content uploaded by Nguyen Hoaison
Author content
All content in this area was uploaded by Nguyen Hoaison
Content may be subject to copyright.
Building a low-latency, proximity-aware DHT-based P2P network
Ngoc Ben DANG, Son Tung VU, Hoai Son NGUYEN
Department of Computer network
College of Technology, Vietnam National University, Hanoi
144 Xuan Thuy, Cau Giay, Hanoi, Vietnam
Abstract—DHT-based P2P networks have a problem of topol-
ogy mismatch which causes high latency for message routing.
This paper focuses on improving the latency of routing process
for Chord, which is a typical DHT-based P2P network protocol.
We propose a simple method to build a Chord network based
on the proximity of nodes in the underlying physical network.
The idea of our method is the combination of two techniques:
identifier selection in node joining phase and neighbor selection
in network stabilization phase, both are performed based on
physical network latency. We have evaluated our proposed
solution by simulations. In a simulation network with 4096
nodes, the average latency of routing process in our method can
reduces 30% comparing with the conventional Chord routing
method.
Keywords-DHT;Chord;low latency;proximity-aware
I. INTRODUCTION
Peer-to-peer network applications are becoming more and
more popular in the Internet today. Comparing to traditional
server-client network model, peer-to-peer networks have
many advantages like its scalability, self-organization and
fault tolerance.
Recently, Distributed Hashing Table (DHT) algorithms
such as Chord [1], CAN [2], Pastry [3] attract lots of
attention since they can offer a promising solution for
scalable and efficient message routing on P2P networks. The
algorithm constructs a structure of a virtual key space on a
P2P network where each node is responsible for a portion
of key space and keys are used as destinations to route
messages. Each node also maintains information about a
small set of neighbor nodes, which are selected based on the
positions of nodes in the virtual key space. The neighbor sets
are used to route messages to destination nodes efficiently.
However, since the construction of a DHT-based P2P net-
work is performed regardless of physical network topology,
there are differences between overlay network topology and
physical network topology (called ”topology mismatch”).
For example, two nodes which are near in the underlying
physical network (i.e. the latency of message routing be-
tween them is small), may be far away (i.e. several hops
away) from each other in the DHT-based network. In this
case, the latency of message routing between these nodes
in the DHT-based network is significantly higher compared
with the latency between them in the underlying physical
network.
This paper focuses on the problem of improving the
latency of message routing in Chord, a typical DHT-based
P2P network protocol. There are two main approaches to
solve this problem: proximity neighbor selection (PNS) [5],
[7], [9] and proximity identifier selection (PIS) [4], [8]. In
the PNS approach, a node will choose its neighbor nodes
based on their proximity. In the PIS approach, each node
chooses its node identifier based on its geographic location.
To realize the PIS approach, conventional researches [4],
[8] map participating nodes into a proximity-based two-
dimensional space and assigns node identifier in Chord
based on the node position in the two-dimensional space
such that nodes that are close in identifier are close in the
two-dimensional space. This solution has a drawback of
requiring a set of landmarks and a pre-processing phase for
nodes to compute their own coordinates in the proximity-
based two-dimensional space.
This paper proposes a simple solution to solve the topol-
ogy mismatch problem for Chord protocol. The main idea of
the solution consists of two modifications of Chord protocol.
First, in the node joining phase, a new node can select a
position in the Chord ring from a set of randomly self-
generated positions such that the latency to its successor
node and predecessor node is the smallest. Second, in the
stabilization phase, each node selects a neighbor node for
each finger table entry from a set of candidate nodes such
that the latency to each neighbor node is small. Our method
is easy to implement and has small overhead comparing
with conventional methods. With minor modifications of
Chord protocol, however, our method can reduce the latency
of message routing compared to the conventional Chord
protocol.
In order to evaluate the effectiveness of our method,
we have built a simulator to simulate the execution of
our algorithm and measure the latency of message routing
between participating nodes in the network. The simulation
results show that our method can reduce the average latency
of message routing by approximately 30%, compared to
the conventional Chord protocol. Our method also does not
affect much on the random mapping of identifier in the
Chord ring.
The rest of this paper is structured as follows. Section 2
explains Chord protocol and related works with their pros
and cons. Section 3 is dedicated to describe our proposal
method. Section 4 describes our simulator and simulation
results. Conclusion is written in section 5.
II. BACKGROUND
A. Chord protocol
Chord is a typical structured peer-to-peer network pro-
tocol using an m-bit address space which includes 2
m
identifiers arranged in a circle topology called a Chord ring
(Fig. 1). Using a consistent hash function, each participating
node randomly chooses an identifier and then joins to a
position in the ring by a joining phase. A resource is
distributed to a Chord network by the use of key-value
pairs: the key is the resourcefs identifier and the value is
the resource itself. Each key-value pair is stored in the node
which is nearest the key in Chord ring clockwise (i.e. the
successor of the key).
In this paper, we focus on the construction of Chord ring,
which includes a joining phase and a finger table building
phase. The joining phase of a node to a Chord network is
rather simple. When a node wants to join a Chord network,
it will generate its identifier by making use of a consistent
hash function on its own information such as its IP/Port
pair. The joining node then sends a joining message to a
well-known node in the Chord network (so-called ”bootstrap
node”), in order to query for its identifier’s successor. Once
the successor information is returned, the joining node will
contact to the successor to join the position next to the
successor counter-clockwise in Chord ring. After that, the
node performs a finger table building phase to build its own
finger table.
In an m-bit address space, each participating node builds a
finger table which has m entries. Assuming that the identifier
of the current node is k, the i
th
entry in the table holds the
information about the successor node of identifier (k+2
i−1
).
Finger tables are used to route messages between nodes in
the network. When a node receives a message destined to
an identifier in the Chord ring, it will look up its finger table
to find the nearest node to the identifier counter-clockwise
and forward the message to that node. The forwarded node
will recursively repeats the same operation. The process
continues until the message reaches the successor of the
destination identifier.
In Chord protocol, the construction of a finger table is
performed regardless of physical network topology (i.e. the
latency between nodes). Therefore, the latency of message
routing between two nodes in Chord network may be signifi-
cantly higher compared with the latency between these nodes
in the underlying physical network. Chord also defines some
other operations such as node leaving, network recovery
from a node failure... The interested reader is referred to
[1] for these details.
1) Related works: There are several conventional solu-
tions for Chord protocol to improve the latency of message
Figure 1. An example of a Chord ring
routing due to the differences between overlay network
topology and underlying physical network topology.
In the approach of Proximity Neighbor Selection (PNS),
each finger table entry holds the information of a node
that is selected from a number of nodes, which belong
to a predetermined interval of identifiers, based on their
proximity. The effectiveness of this approach is showed in
[5], [7], [9]. However, actual realization of this approach is
different between conventional researches. In [7], the authors
propose a method that makes use of latency information
collected from lookup process on the overlay network. After
a node joins a Chord network, it will gradually obtain low-
latency finger table by updating its finger table entries based
on collected latency information. In [9], the authors propose
the use of Vivaldi algorithm [11] to calculate the position of
a node on a virtual two-dimensional space so that distances
in the space approximate distances in the underlying physical
network. Each node then selects neighbor nodes that are
physically nearby by looking up for nodes that are close to
it on the virtual 2-dimension space.
In the approach of Proximity Identifier Selection (PIS), a
node will choose its node identifier based on its geographic
location. Like [9], the work [4] and Quasi-Chord [8] uses
a set of landmarks to map nodes into points in a two-
dimensional space. The identifier of nodes are selected such
that nodes are close in a two-dimensional space are close
in the identifier space. In common, both of approaches
try to decrease the latency of message routing on a DHT
network. However, they require a set of landmarks and a pre-
processing phase for nodes to compute their own coordinates
in the proximity-based two-dimensional space.
III. OUR APPROACH
A. Overview
In this chapter, we propose a new solution for the topology
mismatch problem of Chord protocol. The solution achieves
the same goals as related works, but with another approach
which requires less cost and can be deployed in a large-scale
network easily.
The idea of our approach is that a node can select a
position that it can join in a Chord ring such that nodes
that are close to it in the Chord ring are physically nearby
to it. As the result, nodes are close in a Chord ring will be
close regarding to physical network positions.
The idea comes from a consideration that the proximity
neighbor selection approach is not effective in upper entries
of finger table. It is because the number of nodes that can
be selected as an upper entry of a finger table (i.e. belong to
a predetermined interval of identifiers) is often small. Since
the very last hops of message routing in Chord usually use
the upper entries of finger tables, we need to improve the
latency due to the last hops by keeping physically nearby
nodes as close together in a Chord ring as possible.
Unlike conventional researches which force a joining node
to calculate its physical position, in our method, we allow
a joining node to select its identifier in a set of randomly
generated identifiers, each of which corresponds to a position
in the Chord ring. The node selects an identifier as its joining
position such that the latency to the successor node and the
predecessor node of the selected identifier is the smallest in
the set of candidate identifiers.
Our method also includes a proximity neighbor selection
phase. The i
th
entry of the finger table of a node k will be
selected from a set of candidate nodes which are next to the
successor node of the identifier (k+2
i−1
). The selected node
must satisfy the condition that the latency from the selected
node to node k is the smallest in the set of candidate nodes.
The details of our method are described as follows.
B. Our method
Our method includes two phases.
• Proximity-based identifier selection phase
When a new node joins in a Chord network, instead of
having the node to generate only one identifier to join in the
network, we let the node to generate a set C of identifiers.
The set C is generated randomly by making use of a hash
function on the combination of the nodefs own information
and a random value.
The node then contacts to a bootstrap node in the Chord
network to query the successor node and the predecessor
node of each identifier in the set C. We call CHOICE the size
of set C of identifiers. The node then measures the network
latency from it to the successor node and the predecessor
node of each identifier. The latency to a node can be simply
measured by a ping protocol. It then selects the identifier,
Figure 2. Pseudo-code of proximity-based identifier selection phase.
CHOICE: the size of set C of identifiers
Figure 3. Pseudo-code of proximity-based neighbor selection phase
which successor node or predecessor node has the smallest
latency to it, as its identifier to join in the Chord network.
The pseudo-code for the algorithm is shown in Fig. 2.
• Proximity-based neighbor selection phase
To update an entry in the finger table, first the node, which
identifier is k for instance, queries the successor node s of
the identifier (k + 2
i−1
) based on the Chord protocol. Then,
node k queries the successor node s and nodes next to the
successor node to get a set of candidate nodes which are
next to the successor clockwise and counter-clockwise in the
Chord ring. The set of candidate nodes has 2 * EXPANSION
+ 1 nodes including the successor node s, EXPANSION
nodes next to node s counter-clockwise and EXPANSION
nodes next to s clockwise. Here, we call EXPANSION the
spread degree. To avoid duplication of nodes in multiple
entries of the finger table, nodes in the set which are not
between k + 2
i−1
and k + 2
i+1
will be not considered. The
pseudo-code for the algorithm is shown in Fig. 3.
By probing for the latency from node k to each node in
the set of candidate nodes, we can select the desired node
for the entry of the finger table which latency from node k
to the selected node is the smallest in the set.
The update of finger table entries is performed whenever
the network stabilization phase of Chord protocol is run.
Hence, it can adapt to the join/leave of nodes.
With high probability, the selection of node identifier
based on network latency can keep the latency among nearby
nodes in a Chord network relative small. Hence, it takes
a short time to route messages of which destinations are
nearby in a Chord ring. Furthermore, the update of finger
tables based on network latency can reduce routing time
to forward query messages among nodes. Therefore, our
solution can achieve the goal of reducing latency of message
routing among participating nodes in a Chord network.
The solution has a drawback of delaying the joining
process of new nodes. Comparing to the original Chord
protocol, the solution requires a time interval to probe for
the latency from a joining node to a number of randomly
generated set of node IDs. Updating routing table process
also needs some time interval to select one node from a set
of candidate nodes. The more nodes or identifiers there are
in the sets, the more optimal degree of latency we get and
the longer time it takes for nodes to join in the network or
update routing tables. It is a trade-off and we need to run
simulations to get reasonable parameter values.
IV. EVALUATION
To evaluate the efficiency of the proposed solution, we
have performed experiments with a simple network simu-
lator. Our simulation focuses on measuring the difference
of message routing latency between the conventional Chord
and our proposed method.
A. Simulator
We build a simulator which generates a physical network
topology of participating nodes and maps it to conventional
and improved Chord overlay network. Generated queries
are routed among participating nodes based on routing
Figure 4. Average query latency comparison
information and the returned results are collected to build
statistics.
The simulation network topology is divided into separated
domains. Each domain has a number of participating nodes.
Nodes in the same domain are connected to each other to
create a star topology with a center switch. All the domains’
switches are connected directly to each other. Each of
connections between nodes and their domainfs switch, and
between switches has its own latency. The latency between
node A and node B is calculated by summing up the latency
between node A and its domain’s switch and the latency
between the switch to node B if A and B are in the same
domain. Otherwise, the latency is calculated as the sum of
the latency between node A and its domain’s switch, the
latency between the switch of A’s domain and the switch
of B’s domain and the latency between the switch of B’s
domain and node B.
With the simulator, we can specify following parameters:
number of participating nodes, number of domains, and
number of nodes in a domain and latency of connections. We
divide experiment time into intervals of time called ”round”.
To simulate the join/leave of nodes, the simulator makes
nodes join in and leave the network randomly after each
round. The join/leave timing of nodes is generated based on
the Pareto distribution. Then, all the nodes’ routing tables
are updated and the simulator generates queries. The average
latency in each round after updating routing tables and the
overall average latency of the experiment are stored to build
the statistics for reviewing later.
B. Simulation results
The first experiment aims to compare our solution’s effi-
ciency to conventional Chord’s. The simulator creates 4096
nodes which are divided into 32 domains, with EXPANSION
value of 3 and CHOICE value of 8 and 16 respectively.
The latency of connections between nodes in a domain is
assigned randomly from 1 to 30 ms, and the latency between
Figure 5. Average query latency comparison
Figure 6. Average latency with CHOICE adjustment
two switches is assigned randomly from 50 to 250ms. The
experiment is carried out in 360 rounds and there are about
1000 queries generated in each round. The average latency
of the network in the first 100 rounds is presented in Fig. 4.
The result indicates that applying our proposed method can
make average query latency in each round be steady and
reduce significantly comparing to the conventional Chord
protocol. After 360 rounds, the average latency in our
method with CHOICE=8 is about 759 ms, which equals to
67% of conventional Chordfs result (about 1123 ms). With
CHOICE=16, the average latency is improved slightly to 731
ms.
We also perform the same experiment but increase the
number of nodes in a Chord network. Figure 5 shows that
our solution can reduce the latency of message routing
about 33% comparing with conventional Chord protocol
even when the number of nodes in a network increases.
The second experiment focuses on examining the latency
optimization degree when we adjust CHOICE value - the
number of randomly generated node IDs in the joining
phase. The simulator parameters are the same as the first
Figure 7. Average latency with EXPANSION adjustment
Figure 8. The distribution of ID intervals between adjoining nodes in a
Chord ring
experiment: 4096 nodes are divided into 32 domains, EX-
PANSION value is 3, but there are 3600 rounds and about
1000 queries generated in each round. The result is presented
in Fig. 6.
The case in which CHOICE value is 1 can be viewed as
conventional Chordfs joining phase and only the selection
of routing table entries based on network latency is applied.
In this case, the average latency is 814ms. The more choices
of ID for a node in the joining phase are, the better latency
optimization degree we can achieve. When the CHOICE
value is 32, the average latency is 660 ms.
The third experiment focuses on examining optimization
degree of the solution when we adjust EXPANSION value
- the number of candidate nodes in the neighbor selection
phase. As the EXPANSION value increases, each node has
more choices to select a node for a routing table entry.
Hence, we get better latency optimization degree (Fig. 7).
The random mapping of identifier in Chord rings is one
of Chord advantages that we needs to keep. Conventional
Chord protocol uses a consistent hash algorithm to assign
one ID to each newly joining node. In our solution, an ID is
assigned to a newly joining node based on node proximity.
Because of the difference between two joining algorithms,
we need to examine the affects of our solution on the random
mapping of identifier in the Chord ring. The simulation of
our method is performed with CHOICE=8, EXPANSION
=3. The simulator results show the statistics of ID intervals
between every pair of adjoining nodes in a Chord ring. As
shown in Fig. 8, with about 2000 available nodes existed
in the network, the distribution of ID intervals between
adjoining nodes in a Chord ring in the case of our method is
similar to the one in the case of conventional Chord protocol.
It means that our method does not affect much on the random
mapping of identifier in the Chord ring.
V. CONCLUSION AND FUTURE WORKS
In the paper, we have proposed a method to optimize
Chord topology based on underlying network latency. The
solution focuses on two modifications of the Chord protocol:
selecting the ID in a set of randomly generated IDs from
which latency to its next and previous node is smallest and
updating routing table by selecting neighbor nodes based
on network latency. By doing experiments with simulator,
we have proved that our method can reduce average query
latency significantly but does not affect much on the random
mapping of identifier in the Chord ring. Moreover, it is rather
simple to deploy the solution in Chord-based applications.
In the near future, we are going to continue to evaluate
the efficiency of our method in a real network topology and
deployed our method in Chord-based applications.
ACKNOWLEDGMENT
This work is partly supported by the research project No.
QC.09.12 granted by Vietnam National University, Hanoi
REFERENCES
[1] I. Stoica, R. Morris, D. Karger, M.F. Kaashoek, H. Balakrisnan,
”Chord: A Scalable peer-to-peer lookup service for Internet
applications”, In Proceedings of ACM SIGCOMMf01, August
2001
[2] S. Ratnasamy, P. Francis, M. Handley and R. Karp, ”A Scal-
able Content-Addressable Network”, In Proceedings of ACM
SIGCOMMf01, Aug. 2001
[3] A. Rowstron and P. Druschel, ”Pastry: Scalable, distributed
object location and routing for large-scale peer-to-peer sys-
tems”, In Proceedings of IFIP/ACM International Conference
on Distributed Systems Platforms, Nov. 2001
[4] S. Ratnasamy, M. Handley, R. Karp and S. Shenker,
”Topologically-Aware Overlay Construction and Server Selec-
tion”, Proceedings of the INFOCOM 2002.
[5] K. Gummadi, R. Gummadi, S. Gribble, S. Ratnasamy and S.
Shenker, I. Stoica, ”The Impact of DHT Routing Geometry on
Resilience and Proximity”, Proceedings of SIGCOMM 2003
[6] F. Dabek, J. Li, E. Sit, J. Robertson, M. F. Kaashoek and
R. Morris, ”Designing a DHT for low latency and high
throughput”, Proceedings of 1st Symposium on Networked
Systems Design and Implementation, March 2004
[7] A. Goel, R. Govindan, H. Zhang, ”Improving Lookup Latency
in Distributed Hash Table Systems using Random Sampling”,
IEEE/ACM Transactions on Networking, Volume 13, February
2005
[8] M. Sun, Z. Zhang, ”Quasi-Chord: physical topology aware
structured P2P network”, Proceedings of the 11th Joint Con-
ference on Information Sciences 2008. December 2008.
[9] H. Duan, X. Lu, H. Tang, X. Zhou, Z. Zhao, ”Proximity
Neighbor Selection in Structured P2P Network”, Proceedings
of The Sixth IEEE International Conference on Computer and
Information Technology, September 2006.
[10] T. S. Eugene Ng and H. Zhang, ”Predicting Internet Network
Distance with Coordinates-Based Approaches”, Proceedings of
INFOCOM 2002
[11] R. Cox, F. Dabek, F. Kaashoek, J. Li, and R. Morris,
”Practical, distributed network coordinates”, Proceedings of
the Second workshop on Hot Topics in Networks (HotNets-
II) (Nov. 2003).
