TopDNS: Experiences Building Internet-Based Topologies with GNP.

Full text

Page 1

TopDNS: Experiences Building Internet-Based Topologies with GNP
Gert Pfeifer and Christof Fetzer
TU Dresden
Faculty of Computer Science
01062 Dresden, Germany
{firstname.lastname}@inf.tu-dresden.de
Abstract
Building realistic Internet topologies is an important
step towards predicting the behavior of new distributed ap-
plications and protocols. We are creating topologies that
predict the distances between identifiable nodes in the In-
ternet as it is today. These nodes are not anonymous aver-
age Internet nodes, but are known by name and network
address. We are using a landmark and network coordi-
nate based approach and evaluate the characteristics of the
topologies found. We provide hints, how to select important
parameters for such topology generators.
1. Introduction
Designing overlay solutions which can compete with
IP-based routing is difficult [2, 5]. Our motivation is to
show that our overlay network DNSPastry [10] can exploit
the DNS name structure of hostnames to connect nodes
in an overlay in a way that improves the efficiency of
overlay routing. So far we have been using transit-stub-
topologies [16] and random ping benchmarks. However,
we would like to evaluate if our overlay can provide a DNS
service to increase the dependability based on the fault tol-
erance and flexibility of peer-to-peer networks as proposed
in [2] and [9].
As a workload for such benchmarks we use traces of
DNS requests recorded at the servers of our department.
The main problem is that the realistic characteristics of the
traces, i.e., whichhostnameisusedhowfrequently, mustbe
mapped to the abstract instances of nodes and networks in
the topology that we are using for the simulation. The easi-
est way to do that is to have real node names in the topology.
A request in the DNS trace would then be routed to a node
representing this name. This might be the DNS server for
this zone, or even the node itself, if the name is used for
routing instead of resolving it to an IP in the first place.
To evaluate the performance of DNSPastry, we used the
stretch, i.e., overlay routing delay over IP routing delay
[5, 10], as a representation of the penalty that comes with
overlay routing. Prerequisite for the computation of stretch
is knowledge about the IP-routing delay between any pair of
nodes in the topology. Hence, we do not require our topol-
ogy to represent the actual route a packet takes between two
nodes. Instead, the distance between real identifiable hosts
in the Internet is sufficient. A host is identifiable, iff it is
known by name and IP address.
TopDNS is a topology generator that fullfills these re-
quirements. In this paper, we describe the experiences we
made while building and using it. It is able to provide very
precise topologies, where precision is defined as similarity
to the Internet at the point in time when the landmarks made
their measurements.
The rest of the paper is organized as follows: Section
2 discusses related work. Our system architecture is pre-
sented in Section 3. In Section 4 we discuss our results and
experiences and Section 5 concludes.
2. Related Work
Many topologies are created using characteristics of the
Internet that are statistically analyzed and used to represent
its intrinsic properties. Examples of such characteristics are
the AS structure and routes represented in the BGP routing
tables, in-degree and out-degree of routers, churn in rout-
ing tables, etc. A very common way to represent the AS
based hierarchical structure of the Internet are transit-stub-
topologies [1, 16]. We have used such topologies in our pre-
vious performance analysis of DNSPastry [10]. The main
problem with such topologies is, that it is nearly impossi-
ble to run benchmarks which use real Internet addresses on
them, since there is no way to map existing Internet nodes
to nodes in such a topology. A workload for such topologies
would also have to be abstract and represent the statistical
properties of a typical workload.
Vivaldi [3] is a very efficient distributed network coordi-
nate system. Each participating node can compute it’s coor-

Page 2

dinates using just a small number of latency measurements.
Each node in Vivaldi computes its coordinates by simulat-
ing its position in a network of physical springs. Such a
network behaves in a way that all springs are stretched to a
point that reduces the overall energy in the network. There-
fore, springs with a very high energy, i.e. a high distance
between neighbors in an overlay, might exist contributing
to the overall low energy of springs, i.e. low distance be-
tween neighbors. This system is used, e.g., in Bittorrent
clients. The disadvantage of such an algorithm is, that each
node must participate to do measurements, which is in our
case not possible. Vivaldi is usable with different metrics.
The authors of [3] reached best results using 2d coordinates
together with a height.
Several schemes to estimate Internet latencies use land-
marks. Landmarks are often nodes to which a remote login
is possible, hence they actively participate in the distance
computations by contributing measurements of round trip
times.
Internet Distance Maps (IDMaps) [4] places landmark
nodes at well distributed locations in the Internet. It esti-
mates the latency between two nodes A and B as the latency
from A to its closest landmark added to the latency from
B to its closest landmark plus the latency between the two
landmarks. With an increasing number of landmarks, the
error of distance estimation can be reduced. One problem
with IDMaps is that a client node has to measure distances
to all landmark to identify the closest among them and a
rather high number is needed to achieve good precision.
Dynamic Distance Maps (DDM) [14] is comparable to
IDMaps but it is utilizing a more sophisticated way to find
appropriate landmarks.DDM organizes them hierarchi-
cally, and a client node traverses the hierarchy top-down to
locate a near-by candidate.
M-coop [13] utilizes a network of nodes linked in a way
that mimics the autonomous system (AS) graph extracted
from BGP routing information. In contrast to IDMaps, each
node measures distances only to a small number of other
nodes. When an distance between two nodes is to be esti-
mated, a path containing several measurements is created to
provide it. The performance and quality is comparable to
IDMaps.
Global Network Positioning (GNP) [7] represents the
topology as a Euclidian space. Each node is positioned
using a set of coordinates. The authors measured that us-
ing a 7-dimensional Euclidean space, in 90% of the cases,
GNP can predict the Internet distances among a globally
distributed set of hosts with less than 50% error. GNP also
uses a set of landmarks. The number of available landmarks
limits the number of dimensions in the Euclidian space.
Measurements of [7] show that GNP reaches a better pre-
cision than IDMaps.
King [6] issimilar to IDMaps andM-coop, but uses DNS
servers as landmarks. For each pair of nodes that is mea-
sured, the distance between the nameservers that are au-
thoritative for their names is measured. Say we want to
estimate the distance between node A and B from our own
node C. Therefor, we measure the distance from C to A’s
name server and send a request for a non-existing name
to this server. The name must be chosen in a way that it
does not produce a cache hit and that it is served by B’s
name server. The answer time is measured. It contains a
round trip between C and A’s name server, which we can
measure directly and a round trip between C and B’s name
server. Hence, we can calculate the distance between A’s
and B’s name servers. The assumption is that this distance
is very similar to the distance between A and B. However,
the mushrooming of content distribution networks and the
world-wide replication of name servers reduces the prob-
ability that this assumption holds. The second problem is
that we cannot expect name servers to be recursion enabled
to everyone as we experienced in you studies for [11].
3. System Design
TopDNS is using DNS traces to collect node names. To
assign coordinates to these nodes, it uses GNP. We are us-
ing Planet-Lab [12] nodes as landmarks. A python script
is exercised on each of them which first determines the dis-
tances to all other landmarks and then measures the distance
to each of the hosts found in the DNS trace. To collect the
data we use a MySQL database. It provides the names of
the landmarks, as well as all names found in the DNS trace.
All landmark nodes ping all nodes in the trace and store the
distances, i.e., round trip times, in the data base. The com-
plexity of the topology generation is O(N), for N unique
nodes in the traces. A significant part of the work, i.e. mea-
suring the round trip times, is done in parallel.
Using DNS traces allows us to find hosts that are actually
in use. This gives us the advantage that our topology has a
strong emphasis on machines that are important for Internet
users.
To select landmarks we used CoMon [8] to find alive
lightly loaded Planet-Lab nodes. However, it turned out
that some of them nevertheless had more or less perma-
nent problems. The overall poor dependability of Planet-
Lab nodes as observed by Warns et al. in [15] is a general
problem that limits the number of available landmarks sig-
nificantly. Some of the problems we encountered are:
• a firewalls blocked MySQL traffic
• a gateway blocked traffic between the commercial In-
ternet and Internet2
• YUM problems caused iptables not to be started
(Planet-Lab nodes are running Fedora Linux)

Page 3

• slow or overloaded nodes produced measurements
with a high deviation
• nodes are not up and running during the complete mea-
surement process
To improve the quality of our topologies, we took several
measures to get rid of poor measurements:
• Landmarkswithtoofewmeasurementswereexcluded.
• Landmarks with a too high standard deviation in the
round trip times to nodes in our own network were ex-
cluded.
• We used a genetic algorithm to find a good selection
of landmark nodes. Starting with a random selection
we mutated 1/8 of the selection of landmarks a dozen
times. Then we evaluated the fitness of the generation.
At the end we used the best generation found so far. Of
course, this algorithm does not guarantee an optimal
solution, but it is easy to run in parallel and it can be
seen, that after a rather small number of generations
(5-10) hardly any further improvement can be reached.
From the remaining landmarks, we took the measure-
ments and used GNP to calculate node coordinates. We
tried to find the best number of dimensions and landmarks
to get a precise, i.e., realistic, topology (Section 4). How-
ever, this optimization is limited by the number of land-
marks available.
4. Results
WhenrunningTopDNS,therearetwothingstooptimize:
(1) to maximize the number of nodes in the topology, and
(2)tominimizetheerroronthedelayordistanceestimation.
The number of nodes is in our case limited by the num-
ber of names we can extract from the DNS traces and the
reachability of these names. In Figure 1 it can be seen
that DNS only resolved 85.54% of the names. Of course,
without having an IP address, we cannot use them in the
topology since we cannot estimate their position. Such non-
resolvable names stem from users providing non-existing
names in address fields of browsers or outdated links in
the Internet. Another source is the standard behavior of
many browsersto append the localdomain to relative names
that could not be resolved. Hence, if a name cannot be re-
solved, it often apprears twice in the DNS trace. Sometimes
browsers also try to resolve IPv6 addresses if the local sys-
tem supports this protocol stack. However, as we explored
in [11], this has hardly any chance to succeed.
Another thing that can be seen in Figure 1 is that DNS
problems cannot be the only reason of nodes been excluded
from the topology, since there are altogether only 65.89%
of the nodes are resolved and responding to ICMP pings.

success
connect failure
no IP
65.89%
19.65%
34.11%
Figure 1. success rate of node name resolu-
tion and measurement

0 1020 30 4050 60
0
50000
100000
150000
200000
250000
300000
350000
400000
success
total
Landmarks sorted by number of successful pings
Figure 2. Number of pings send by each land-
mark vs. successful pings
pings sent
However, even out of this fraction not all nodes are usable,
since the nodes would have to respond to all landmarks,
which is often not the case.
Please note that we exclude any names of e-mail black-
lists and search engines that are provided in the case of DNS
failures. We have seen many samples in our DNS traces,
where spam filters query DNS to search a black list for an
e-mail sender’s host name. Therefore, we exclude all names
that resolved to local IP addresses, which are usually an-
swersfromspamblacklistsorotherwisenotusableanyway.
Furthermore, we had to exclude the OpenDNS search page,
which was often returned when otherwise an NXDOMAIN
error would have been returned by a standard DNS server.
These cases are included in the failures in Figure 1.
In Figure 2 one can see, how many pings were send by
each landmark and how many of them returned a result.
We have used a trace with about 368,000 names. Even
the best Planet-Lab nodes were only able to resolve about
250,000. Some hosts were so overloaded or unavailable that
they were not even able to try all names. Note that we used

Page 4

13579
11 1315
Landmarks sorted by std dev
1719212325 272931333537 39414345 47
0
10
20
30
40
50
60
70
80
Standard deviation of round trip time
Figure 3. standard deviation of landmark
measurements
CoMon to select Planet-Lab nodes without any problems
for our selection at the beginning. An interesting observa-
tion is that some of the nodes had hardly any success even
when trying all names. We mentioned the reasons for this
phenomenon in Section 3.
Besides the number of successful pings, we also used
the standard deviation of ping measurements as a fitness
function to exclude certain landmarks, even if the measured
many distances successfully. In Figure 3 one can see the
measured distribution of the standard deviation. One can
see that the deviation grows continuously until about 15
and then grows significantly. We used this threshold and
excluded all nodes with a standard deviation of mode than
15, which is a very defensive value given the fluctuation
of round trip times in the Internet and an extraordinary de-
fensive value given the load variations on Planet-Lab nodes
[15].
Trying to minimize the error of the pairwise latency esti-
mation also influences the number of nodes available for the
topology. Note that we use a notation of landmarks and di-
mensions to name our experiments, e.g., 3d4l means that an
Euclidian space with 3 dimensions is used and 4 landmarks
to estimate the coordinates.
In Figure 4 one can see how many nodes were usable
for a topology with different numbers of dimensions and
landmarks. Some nodes, that we called outliers have shown
a deviation between calculated and measured distance to the
landmarks of more then 10%. These nodes are not used
later on. We are convinced that this threshold is selected
very conservatively, given the deviations we have seen in
our distance measurements.
The overall trend is that the number of usable nodes de-
creases with growing number of landmarks. The reason is,
that the probability of a node being measured from all se-
lected landmarks decreases with each additional landmark

2d3l2d6l3d4l3d7l 7d8l7d11l 11d12l 11d15l 15d16l 15d20l 17d18l 17d21l 19d20l
0
50000
100000
150000
200000
250000
300000
usable nodes
outliers
Figure 4. Nodes measured with sufficient pre-
cision vs. outliers in total numbers

2d3l2d6l3d4l3d7l7d8l 7d11l 11d12l 11d15l 15d16l 15d20l 17d18l 17d21l 19d20l
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
usable nodes
outliers
Figure 5. Nodes measured with sufficient pre-
cision vs. outliers
(see Figure 2). This has an even stronger influence when
the number of landmarks is very small.
Nevertheless, sometimes it can be found, that with a
higher number of landmarks even more nodes are usable,
e.g. with 11 and 15 dimensions.
with more landmarks the coordinate calculation can be im-
proved. Hence, the number of outliers decreases. The rel-
ative amount of outliers among the nodes can be seen in
Figure 5.
To achieve a good quality of the topology, as mentioned
before, we have to keep the error in the calculated distances
low. We compared the pairwise error in the distances be-
tween landmarks for different configurations of dimensions
and landmark numbers to find out, what is a reasonable
choice. The trade-off here is that the higher number of di-
mensions reduces the performance of the distance calcula-
The reason is, that

Page 5

0
20
40
60
80
100
0 0.2 0.4 0.6 0.8
pairwise relative error
1 1.2 1.4 1.6 1.8 2
CDF
2d3l
3d4l
7d8l
11d12l
15d16l
17d18l
19d20
Figure 6. Pairwise relative error for d+1 land-
marks
0
20
40
60
80
100
0 0.2 0.4 0.6 0.8
pairwise relative error
1 1.2 1.4 1.6 1.8 2
CDF
3d4l
3d7l
17d18l
17d21l
Figure 7. Pairwise relative error for different
numbers of landmarks
tions while it offers more precision. Even if the additional
cost for more dimensions was not important for us, it turns
out more then 15 dimensions are not contributing any im-
provement anymore as can be seen in Figure 6. It shows
the CDF of the pair-wise link delay estimation error. It can
clearly be seen that 2 or 3 dimensions are not providing a
good latency estimation.
We also tried to decide, whether it would be good to have
more landmarks than d + 1 for d dimensions. In Figure 7 it
can be seen that a better precision can be reached, if more
landmarks are used. However, this benefit is not as sig-
nificant as could be expected since the availability of high
quality landmarks is limited. We sorted the landmarks to be
used by number of measurements and standard deviation.
Hence, with every additional landmark, the overall quality
of landmarks decreases and so the overall increase of the
topology’s precision is rather limited.
95
96
97
98
99
100
0 0.2 0.4 0.6 0.8
pairwise relative error
1 1.2 1.4 1.6 1.8 2
CDF
11d12l
11d15l
15d16l
15d20l
19d20
Figure 8. Pairwise relative error of the most
precise solutions
In Figure 6 one can see a comparison between different
numbers of dimensions d where we always selected d + 1
landmarks. It is obvious that for two and three dimensions
there is a significant error. Approximately 10% of the dis-
tanceshaveanerrorofatleast100%, whileformorethan15
dimensions the amount of computed distances with such a
deviation is negligible. With seven dimensions, we reached
much better results. 72.91% of the computed distances had
less then 10% error and only 4.1% had more than 100%
error.
One can also see that we reached very similar results
with 15, 17, and 19 dimensions.
84.42% of the distances had an error of less than 10%. With
15 dimensions and 20 landmarks we even had no computed
distances which had more than 92.5% error. Note that we
never created any topology with more than 21 landmarks
due to the restricted availability of good landmarks.
In Figure 8 one can see that for an equal number of
dimensions, more landmarks are an advantage. However,
having more dimensions is not an advantage in general.
With 15 dimensions and 20 landmarks, we see the best re-
sult, while if we would only have 16 landmarks, it would
be better to reduce the number of dimensions. One can see
that with 11 dimensions we reached better results than with
16 landmarks for 15 dimensions.
We also tried to find hints to be able to create better
DNS name space-based transit stub topologies as proposed
in [10]. Therefore, we selected three times 500,000 links
and classified them based on the names of the two peers
to be intra-stub (IS) for names within the same 2nd-level
domain, stub-transit (ST) for names in the same top level
domain, or intra-transit (IT) otherwise.
As one can see in Table 1, there is always a significant
standard deviation. However, the median suggests that sim-
ilar names have most of the time a shorter distance to each
Between 71.86% and