Conference PaperPDF Available

Route Flap Damping Made Usable

Authors:

Abstract

The Border Gateway Protocol (BGP), the de facto inter-domain routing protocol of the Internet, is known to be noisy. The protocol has two main mechanisms to ameliorate this, MinRouteAdvertisementInterval (MRAI), and Route Flap Damping (RFD). MRAI deals with very short bursts on the order of a few to 30 seconds. RFD deals with longer bursts, minutes to hours. Unfortunately, RFD was found to severely penalize sites for being well-connected because topological richness amplifies the number of update messages exchanged. So most operators have disabled it. Through measurement, this paper explores the avenue of absolutely minimal change to code, and shows that a few RFD algorithmic constants and limits can be trivially modified, with the result being damping a non-trivial amount of long term churn without penalizing well-behaved prefixes’ normal convergence process.
Route Flap Damping Made Usable
Cristel Pelsser1,OlafMaennel
2, Pradosh Mohapatra
3,RandyBush
1,andKeyurPatel
3
1Internet Initiative Japan
Tok yo , Japan
{
cristel,randy
}
@iij.ad.jp
2Loughborough University
United Kingdom
O.M.Maennel@lboro.ac.uk
3Cisco Systems
San Jose, CA, USA
{
pmohapat,keyupate
}
@cisco.com
Abstract. The Border Gateway Protocol (BGP), the de facto inter-domain rout-
ing protocol of the Internet, is known to be noisy. The protocol has two main
mechanisms to ameliorate this, MinRouteAdvertisementInterval (MRAI), and
Route Flap Damping (RFD). MRAI deals with very short bursts on the order of a
few to 30 seconds. RFD deals with longer bursts, minutes to hours. Unfortunately,
RFD was found to severely penalize sites for being well-connected because topo-
logical richness amplifies the number of update messages exchanged. So most
operators have disabled it. Through measurement, this paper explores the avenue
of absolutely minimal change to code, and shows that a few RFD algorithmic
constants and limits can be trivially modified, with the result being damping a
non-trivial amount of long term churn without penalizing well-behaved prefixes’
normal convergence process.
1Introduction
Despite the huge success of the Internet, the dynamics of the critically important inter-
domain routing protocol, the Border Gateway Protocol (BGP), remain a subject of re-
search. In particular, despite a large numberofresearchefforts,theconvergenceof
BGP[6, 11], and lately, the chattiness of BGP, also called BGP churn [3], are still not
well understood. Further observations have been made of duplicated and/or ‘unneces-
sary’ updates [15]. These all ultimately lead to slow protocol convergence.
Understanding the BGP mystery is critical. In the case of convergence, vendors may
improve code based on insights into propagation patterns, which in turn could lead to
less churn, and thus lower load, a more robust network, and faster response to failure
events. Researchers suggesting replacement protocols could design them with an in-
depth understanding of what works today, what does not work well, and why.
This paper aims at one facet in this spectrum: how, with absolutely minimal code
change, to better differentiate the normal path-vector protocol convergence process
from abnormal activity, such as heavily flapping prefixes. It has been shown that a
single triggering event can cause multiple BGP updates elsewhere in the Internet [5, 6].
We say a BGP rou t e is flapp i ng or uns t abl e i f a r o uter originates multiple BGP update
N. Spring and G. Riley (Eds.): PAM 2011, LNCS 6579, pp. 143–152, 2011.
c
!Springer-Verlag Berlin Heidelberg 2011
144 C. Pelsser et al.
messages (reachable or unreachable) for the prefix in a ‘short’ time interval and prop-
agates those changes to its neighbors. However, BGP, being a path-vector protocol is
also subject to topological amplification, sometimes called path exploration.Onetrig-
gering event can cause multiple BGP updates at a topologically distant router. Studies
using BGP beacons [12] have illustrated this effect. It is important to understand that
this is a property (or artifact) of the BGP protocol itself and does not correspond to con-
stantly changing topology. In fact, studies of BGP update behavior and traffic flow have
found little correlation [20]. The traffic may continue to reach its destination despite the
constant noise of BGP update messages.
While this is conceptually very simple, it is not easy to distinguish real topological
changes from path exploration in the BGP signal. Ideally, we would like to maximize
the speed topological information is propagated, while minimizing exchanged messages
required to converge to a stable path. However, the root cause of a BGP update typically
cannot be known. Therefore mechanisms to reduce BGP’s chattiness face the dilemma
of finding appropriate algorithms and parameters.
Huston [8] has observed that a small portion of the prefixes generate a high number
of BGP update messages. In Figure 1 we show a similar observation. Most prefixes
receive very few updates. Only 3% of the prefixes are responsible for 36% percent of
the BGP messages. The plot shows the number of update messages that are received at
a router in our measurement setup (Fig. 3) for each prefix during the week from Sept.
29th to Oct. 6th, 2010.
1
10
100
1000
10000
100000
1e+06
0 50k 100k 150k 200k 250k 300k
Number of messages
Prefix index
Number of update messages per prefix on all sessions
All sessions at r0
Fig. 1. Update count per prefix at r0duringthe
week of Sept. 29th to Oct. 6th 2010
0.001
0.01
0.1
1
10
100
1000
10000
10000 100000
Number of messages in 1 hour intervals
Number of prefixes
CDF for the NTT session
min avg max
Fig. 2. Update count per prefix from a single
BGP session in one hour bins
Figure 2 illustrates the churn, i.e. update messages per hour that are received on
a session with a tier-1 ISP. The y-axis depicts the number of updates received for a
particular prefix per one hour bin, while the x-axis shows the prefixes sorted by the
number of update messages received. The majority of prefixes account for few updates,
while a small number of prefixes account for a very high number of updates within
a short time period. The figure shows three curves, the minimum (vertical line), the
average (lower curve) and maximum (top curve) number of updates in one hour bins.
Router r0 receives a full routing table, 326,575 routes, from NTT. One might expect
that most of those routes would be stable and not receive any updates at all. However,
Route Flap Damping Made Usable 145
we observe updates for 153,773 prefixes during one week of observation. And the router
receives up to 1,647 updates in one hour for the prefix with the highest churn (see right
most point on the top curve in Figure 2), there are less than ten updates for more than
100,000 of the prefixes for which there were any updates. Most prefixes for which we
observe BGP update messages are quiet most of the time. Only 0.01% of the prefixes
are always present in the trace, with one prefix having a minimum of 913 BGP updates
per hour over the whole trace (which explains the vertical line in Figure 2). These
observations confirm that most prefixes are very quiet, and only a very small number of
the prefixes are responsible for the majority of the BGP churn.
For some prefixes the router received hundreds and thousands of update messages,
over arbitrarily long time-periods. We hypothesize those updates are being caused by
some periodic events and/or flapping. This cannot be ‘normal’ protocol convergence.
This is causing an unnecessary load on the global routing system.
2Background
There are many causes for route flapping. One common cause is a router or a link going
up and down due to a faulty circuit or hardware. Another cause is a BGP session being
reset. BGP policy changes can also lead to the readvertisement of routes and can thus be
interpreted as a route flap, this also includes policy changes for traffic engineering. Fur-
thermore, IGP cost changes may cause BGP updates which then propagate across the
Internet [17]. Duplicate advertisements [15] are probably the best example of ‘unnec-
essary’ updates that do not contain any new topological information. Lastly, the BGP
protocol is known to be inherently unstable [1, 4, 7].
Toda y, t wo a ppr o ach es att emp t t o m ake t he trade-off between convergence time and
message count [6]. First, the MinRouteAdvertisementInterval timer (MRAI) [16] speci-
fies the minimum time between BGP advertisements to a peer. While it is recommended
to be a per prefix timer, existing implementations typically use a per-peer timer for
all prefixes sent via that peering. By default, it is 30 seconds (jittered) for an eBGP
peer, and five seconds for iBGP. The idea is that the router waits for the ‘path explo-
ration’ downstream to finish, before sending any updates. However, as mentioned ear-
lier, no technique can reliably discriminate between flapping routes and routes that are
‘converging’.
The second technique is Route Flap Damping (RFD) [19]. It is more complex and
fine-grained, as routers maintain a penalty value per prefix and per session. Routes with
a penalty above a given threshold are damped, e.g., newly received announcements are
suppressed and not considered as suitable alternatives to reach a destination. The idea is
that heavily flapping paths are putting a large burden on the routing system as a whole
and to protect the Internet from such routes, it is better to disregard the path and drop
its traffic than to let such prefixes potentially cause cascading failures due to system
overload. Of course, despite observations, stable routes are not supposed to be affected
by this mechanism. Thus, there is still room for research in this area. For instance, the
work of Huston [10] is promising in that it aims to categorize updates and determine
the types that are potential indicators of path hunting. However, live detection of such
updates is much more CPU and memory intensive than the brutally simple approach
explored in this paper.
146 C. Pelsser et al.
Using RFD [19], each prefix accumulates a penalty which is incremented on receipt
of an announce or withdraw message for that prefix. This penalty is a simple counter
and the values added to the penalty are listed in Table 4. When the penalty reaches
agiventhreshold,the‘suppresspenalty,therouteisdamped,i.e.quarantined.Itis
not advertised by the router until the penalty gets below another threshold, the ‘reuse
penalty’. The penalty value of a damped route is decremented using a ‘half-life’, i.e.
it is divided by two after ‘half-life’ seconds. Upon the receipt of further updates the
penalty continues to grow. However, there is a ‘max suppress time’, which constitutes
a maximum time the route can be damped. E.g., provided that the route is not receiving
any further updates, a damped prefix is typically released after one hour. This translates
into a ‘maximum suppress penalty’, which is computed using the suppress threshold,
the reuse threshold and the half-life time. For example, with Cisco default parameters a
penalty of 12,000 will result in a suppression of one hour if no further updates for that
prefix arrive. We refer to the work of Mao et al. [13] for a detailed study of the RFD
algorithm.
RFD has been reported to be harmful [2] in that, with current default settings and
recommendations [14], it penalizes routes which are not flapping, but receiving multi-
ple updates due to path exploration. This severely impacts convergence. Reachability
problems for over an hour have been observed where there was no physical outage,
network problem, or congestion that would justify any packet drops. In fact, it has been
shown that perfectly valid and fine paths can be withdrawn due to RFD [2]. As a con-
sequence most operators have disabled RFD. On the other hand, we see serious BGP
noise affecting router load and burdening the whole system [9].
Can research on BGP dynamics lead to an appropriate recommendation of RFD pa-
rameters? What would happen if we adopted a strategy to select only the ‘heavy hitters’,
the heavily flapping routes, or ‘elephants’ as we call them – but leave the converging
routes, or ‘mice’, in peace? BGP churn should decrease significantly compared to the
current situation where RFD is turned off, yet the BGP convergence for prefixes with
‘normal’ BGP activity would not be affected. In this paper, we try to find and propose
such appropriate parameters.
3MeasurementSetup
In this section, we present our experimental design. We describe a change to Cisco’s
IOS XR BGP implementation to enable the collection of damping statistics, the location
of the router in the Internet and the BGP feeds that it receives. Then we explain how we
collected and analyzed the RFD data.
Router r0 in Figure 3 is a Cisco 12406 running a minimally modified version of
Cisco’s IOS XR software to enable us to perform a detailed analysis of what the router
‘thinks’. The router applies the RFD algorithm using the normal penalty values. The
modified code does not actually damp the routes, instead it records the calculated
penalty values of each route and its supposed status, active or damped. The other modi-
fication was that no ‘Maximum Suppress Penalty’ was imposed, e.g., the penalty values
could increase above 12,000.
Route Flap Damping Made Usable 147













Fig. 3. Measurement Topology Setup
Parameter Va lue
1Half-life time 15 min
2Max suppress penalty 12,000
3Max suppress time 60 min
4Suppress penalty 2,000
5Reuse penalty 750
6Withdrawal penalty 1,000
7Re-advertisement penalty 0
8Attribute change penalty 500
Fig. 4. Cisco’s default RFD values
Figure 3 shows our measurement infrastructure. Router r0 is directly connected to
a large public Internet Exchange over which it receives both full and partial feeds. In
addition, the router connects to a global tier-1 provider for another full BGP feed.
We p ulle d s tat i sti c s f r om th e r o uter a t r egul a r int e rva l s for one w e ek, f r o m Sep t e m-
ber 29 through October 6, 2010, using the clogin command from the rancid tool. Data
included details of all route flap damping counters, although the router code did not
actually damp any route. The time to pull the data from the router depended on how
quickly the router responded to our queries, but was typically in the order of 4–5 min-
utes. Missing counter values due to slow router response time did not significantly af-
fect our observations in subsequent sections, as there were very few of them. The 95%
quantile was under ten minutes. However, in some circumstances it was longer, up to
45 minutes in one instance! We believe this was due to CPU utilization peaks.
4Results
We inv est i gate t h e pen a l ty va lues a s sig n e d to th e prefi xes re ceiv ed by o u r mod i ed r o uter,
r0(Figure3).WethenproviderecommendationsfornewRFDparametersettings.
Figure 5 shows the Cumulative Distribution Function (CDF) of the penalties as-
signed to prefixes by the router during the one week experiment. Let us assume there
are n snapshots during the week’s experiment. We define an ‘instance’ ip,tas the RFD
penalty of prefix pin snapshot t.Figure5showstheproportionofinstanceswithpenal-
ties smaller than or equal to xover the whole set of instances. Intuitively, this is the
proportion of prefixes which would have been damped in the time-prefix-space.
We observe that 14% percent of the instances reached a penalty greater or equal to
2,000 in the measurement period. 2,000 is a critical threshold as this is the default value
for RFD suppression on Cisco routers. This gives a feeling for how ‘bad’ it is, if one
turns on default RFD those instances would have been damped. Further, we observe in
Figure 5 that a suppress threshold of 4,000, 5,000 and 6,000 leads to the damping of
4.2%, 2.8% and 2.1% of the instances respectively. The number of damped instances
decreases very quickly. Finally, we note that very few of the prefixes are assigned a
very high penalty. Only 0.63%, 0.44% and 0.32% have a penalty value above 12,000,
148 C. Pelsser et al.
0 5000 10000 15000 20000 25000
0.0 0.2 0.4 0.6 0.8 1.0
Proportion of prefixes with penalty value below x (CDF)
Penalty
Proportion of prefixes
livefeed
Fig. 5. Distribution of penalty values. Vertical
lines are 2,000, 4,000, 6,000, and 12,000.
CDF of prefixes with penalty above threshold
Damping duration
Proportion of prefixes
0.0 0.2 0.4 0.6 0.8 1.0
10m 1h 2h 4h 1d 2d 4d 8d
2K4K
4K6K
6K12K
12K+
Fig. 6. CDF of the proportion of prefixes with
penalty values above a threshold
15,000 and 18,000, respectively. Thus, very few prefixes flap heavily for long in the
time-prefix-space. However, we observed earlier that those few prefixes are responsible
for a disproportionate part of the BGP churn. The maximum penalty value assigned to
arouteduringtheexperimentwas48,000. This value is huge compared to the median
penalty of 818 (Fn(818)=0.5).
We recommend conservative operators set the ‘suppress threshold’ to 12,000,
15,000 or 18,000, as these values likely penalize only the very heavy hitters. We show
later that, while values in the range [12,000 18,000]enable a non negligible BGP up-
date rate reduction, a suppress threshold in the range [4,000 6,000]damps far fewer
prefixes compared to current defaults and the BGP update rate is significantly reduced.
How long do prefixes typically stay at high penalty values? Figure 6 shows the CDF
of the durations a prefix is above a certain penalty value, and thus would be damped if
this was the threshold. The red solid curve shows the damping duration for the current
threshold of 2,000. Many prefixes have a penalty above 2,000 for a very short time.
For example, 68% of prefixes stay above 2,000 for up to one hour during the one week
of the experiment. This means the current default suppresses a lot of prefixes that are
unstable for a relatively short time. We suspect that many of those prefixes are inappro-
priately damped following a single event. They are given a penalty value above 2,000
during BGP convergence simply because of path exploration. We should not damp those
prefixes!
The other curves show suppression times for penalty values between 2K and 4K,
between 4K and 6K, 6K–12K, and above 12K relative to those prefixes in the 2K class.
If a prefix is not suppressed at all, then the duration is zero and thus the curve starts at
this point on the y-axis. Not surprisingly, the number of prefixes in each category varies
quite a lot (721 prefixes above 12K, top most curve; 4,429 prefixes between 6K–12K,
2nd from top; and 11,546 prefixes between 4K–6K, 3rd from top; and 44,846 prefixes
between 2K–4K, lowest curve). Furthermore, there are very few prefixes that have a
high penalty for a long time (e.g. rightmost points). There are 57 prefixes in the 2K–4K
band that stay in this band for more than two days, but only 12 prefixes in the above
12K-band that stay for more than two days. We noticed some prefixes change bands,
e.g., stay for a few hours/days in the 2K–4K band and then also stay a few hours/days
in a higher band. Overall, it is possible that high churn prefixes stay for quite some
Route Flap Damping Made Usable 149
time in lower bands; but we have also shown that ‘normal converging’ prefixes stay in
those bands. Therefore, we need to find a trade-off in the parameter space, that does not
penalize prefixes that only experience path exploration.
Figure 7 shows the number of prefixes which would be damped given the different
candidate thresholds. Clearly, (32,089) mice would be spared using a suppress thresh-
old of 4,000 or above. Moreover, we see that the number of prefixes damped with
higher suppress thresholds does not vary much. High thresholds are much more suit-
able to prevent damping of prefixes affected by normal BGP path exploration than the
current default threshold. Our intuition here is that a ‘badly behaving prefix’ will flap
for a long time and thus hit high penalty values; while ‘normal converging prefixes’,
which just receive multiple updates due to path exploration, will not be penalized.
0
5
10
15
20
25
30
35
40
45
2 4 6 8 10 12 14 16 18 20
Number of damped instances (K)
Threshold (K)
Number of damped instances as a function
of the suppress threshold
live feed
Fig. 7. Asuppressthresholdof4,000(and
above) already damps many fewer prefixes
50
55
60
65
70
75
80
85
90
95
2 4 6 8 10 12 14 16 18 20
Percentange of update rate
(one hour bins)
Threshold (K)
Percentage of update rate compared to
situation without RFD
avg
Fig. 8. Athresholdof6,000enablesanaverage
churn reduction of 19%
Increasing the value of the suppress threshold above today’s default will increase
the BGP churn rate, but it will save many mice and still be less churn than a router
with RFD turned off. Figure 8 illustrates this. The x-axis is the candidate value of the
suppress threshold. On the y-axis we show the update rate on a per minute average in
60 minute bins. 100% is the churn when RFD is disabled.
Here, we try to estimate how churn could change if we activate RFD at various
thresholds. Unfortunately, we face two problems: (1) the accuracy of timestamps and
(2) we are looking at only one router and not studying interactions in complex topolo-
gies. With regard to (1) we record incoming updates via tcpdump with sub-second accu-
racy. However, the router provides us with less frequent snapshots of the penalty values.
We the r e fore h ave o nly an e s tim a te of th e p enal t y. Espe c iall y, w e d o not kn ow th e exac t
time of onset, e.g., when an update would have been damped. Updates often come in
bursts with short inter-arrival times but the arrival process of bursts is rather uniformly
distributed in time, and thus within the snapshots. If a prefix has a penalty above the
considered threshold in the current snapshot, all its updates in the coming interval are
marked as being potentially removed. This provides an estimation of the update rate.
By averaging over the whole trace, the error smooths out. We tag all updates that cross
our 2K, 3K,.. . thresholds within a certain time-window. With respect to (2) we can-
not predict how MRAI and best path selection processes will or will not delay updates.
150 C. Pelsser et al.
While we believe that the overall properties of update behavior are comparable, we
leave it for future work to study the impact in complex topologies.
We o bse r ve a 47% r e d uct i on of th e a ver a ge upd a te rat e w ith a p e nal t y of 2,000, com-
pared to a situation without RFD, in Figure 8. 4,000, 5,000 and 6,000 correspond to an
average update rate reduction between 26% and 19%. Thus, it is worthwhile changing
the default suppress threshold value. Our proposal is a very simple modification which
is rather effective compared to more complex solutions such as [10].
We fur t h er no t e that t h e chur n r e duc t ion is s imil a r for al l t hre s hol d s abov e 12K . D amp-
ing thresholds of 12K, 15K and 18K suppress an average of 11.26%, 9.51% and 8.12%
of the updates, which is still non negligible for such a trivial change as we propose.
To com p are t he re all y h eav y hit ter s i n the i nter val s 12K -15K , 15K -18K , an d a bov e
18K, we concentrate on 64 prefixes which have a damped duration of six hours or
longer. We notice that 53 of those 64 prefixes (83%) at some point pass the high point of
18K. Only nine prefixes (14%) stay in the 12K-15K range, and only two prefixes (3%)
go over 15K, but not up to 18K. This strengthens our confidence that the ‘evil’ guys,
the really heavy hitters which constantly flap, will be caught by almost any threshold
setting, be it 12K, 15K, or 18K.
Thus, for more conservative operators that desire to spare most of the mice and still
see around 10% churn reduction, we recommend values 12K and above. It does not
matter much which of these three values are chosen. If a prefix is flapping so hard that
it reaches 12K, then it is also likely to go higher at some time.
5OtherFeeds
Acriticalquestioniswhethertheobservationsandrecommendationsintheprevious
section hold for other locations in the Internet topology? Can we make a generic rec-
ommendation for the ‘suppress penalty’ value? To understand this, we replayed addi-
tional varied BGP traces from Route Views into r0(see‘RV/RISUpdates’inFigure3)
in pseudo real-time. Again, r0loggedtheRFDpenaltiesofthereceivedroutes.
We p erfo r m ed tw o a d dit i onal e x peri m ents . Figu r e 1 0 d esc r i bes t h e a ddi t iona l w ork -
load traces that were replayed to the router. These were in addition to the in vivo feeds
from the tier-1 ISP and the Exchange Point.
These experiments were designed to determine if different update patterns recorded
at different places in the Internet topology would affect our conclusions.
Figure 9 shows the penalty values for prefixes with the new feeds replayed from
Route Views. It shows that the distributions are exceedingly similar to those from the
live feeds. Similar to Figure 5, this plot shows a CDF of penalties assigned by r0tothe
different instances in the time-prefix-space. The green curve is the workload from the
live feed plus a BGP feed from an African peering point. The blue curve is the 1.5day
live workload with the 10 additional Route Views feeds. The red curve is the one week
workload (live feed), previously shown (see Figure 5) for comparison. We observe that
all three curves have a similar shape. Adding more feeds just leads to more prefixes that
flap but the number of ‘elephants’ is very similar.
Therefore, our damping suppression threshold recommendation does not change for
BGP feeds from varying points in the Internet topology.
Route Flap Damping Made Usable 151
0 5000 10000 15000 20000 25000
0.0 0.2 0.4 0.6 0.8 1.0
Proportion of prefixes with penalty value below x (CDF)
Penalty
Proportion of prefixes
livefeed
10 large feeds
1 small feed
Fig. 9. Distribution of penalty values
10 large feeds: Te n Rout e Views [ 18 ] fe ed s
which received the maximum number of
updates from August 5 to mid-day August
6, 2010 (1.5days).
1smallfeed: AsmallselectedAfricanEx-
change, route-views.kixp. Data were col-
lected from August 27 to September 1,
2010 (5 days).
Fig. 10. Additional feeds
6Conclusion
We stu d ied th e i m pac t o f RFD on the I n tern e t. As p r evi o usly o bse r ved by m a ny ot h er
researchers, a small fraction of prefixes is responsible for a significant portion of the
update churn. RFD was developed to reduce this noise, but the current default parame-
ters do not properly take into account properties of the BGP protocol. Any path-vector
protocol is by it’s very design noisy due to path exploration.
We t here f ore lo oked a t t he eff e ct of a n a bsol u tely m i nim a l chan g e, onl y a djus t ing
RFD parameters, to get moderate churn reduction without adversely impacting nor-
mally converging prefixes. Our recommendation derived from this study would be to
damp a route when it reaches a penalty above 12,000. The suppress threshold can be
set to any value between 12,000 and 18,000. Such a setting will suppress the BGP
churn of routes that flap heavily while keeping paths for prefixes which only slightly
contribute to BGP churn. For operators extremely concerned about churn, a suppress
threshold of 4,000 to 6,000 is a far better compromise than today’s default parameters.
It may still damp some normally converging prefixes, but will also significantly reduce
the BGP update rate.
We do not recommend changing the maximum suppress time, but we strongly rec-
ommend the limit of the maximum suppress threshold value be raised. A maximum
suppress time of one hour is very reasonable to achieve recovery once the flapping
stops (and heavy hitters will anyway broadcast continuously), but the maximum sup-
press threshold needs to be able to allow higher values than 12,000.
Acknowledgments
We are very grateful to Cisco for the code modification that made those measure-
ments possible, and for engineering support, equipment, and funding. Google, NTT,
and Equinix contributed significant support. We are thankful to Matthew Roughan for
many comments on earlier versions of this idea. We would also like to thank Nate Kush-
man for the inspiring discussions.
152 C. Pelsser et al.
References
1. Basu, A., Ong, C.H.L., Rasala, A., Shepherd, F.B., Wilfong, G.: Route Oscillations in I-BGP
with Route Reflection. In: Proc. ACM SIGCOMM (2002)
2. Bush, R., Griffin, T., Mao, Z.M.: Route Flap Damping: Harmful? RIPE 43 (2002),
http://www.ripe.net/ripe/meetings/archive/ripe- 43/presentations/
ripe43-routing- flap.pdf
3. Elmokashfi, A., Kvalbein, A., Dovrolis, C.: BGP Churn Evolution: A Perspective from the
Core. In. In: Proceedings of INFOCOM (2010)
4. Griffin, T.G., Wilfong, G.: Analysis of the MED Oscillation Problem in BGP. In: Proceedings
of the International Conference on Network Protocols (2002)
5. Griffin, T.G.: What is the Sound of One Route Flapping? IPAM (2002)
6. Griffin, T.G., Premore, B.J.: An Experimental Analysis of BGP Convergence Time. In: Proc.
ICNP (2001)
7. Griffin, T.G., Wilfong, G.: On the Correctness of iBGP Configuration. SIGCOMM Comput.
Commun. Rev. 32(4), 17–29 (2002)
8. Huston, G.: The BGP Instability Report (2006),
http://bgpupdates.potaroo.net/ instability/bgpupd.html
9. Huston, G.: BGP Extreme Routing Noise. RIPE 52 (2006),
http://www.ripe.net/ripe/meetings/ripe- 52/presentations/
ripe52-plenary- bgp-review. pdf
10. Huston, G.: Update damping in BGP (2007),
http://www.potaroo.net/presentations/2007- 10-25- dampbgp.pdf
11. Labovitz, C., Ahuja, A., Bose, A.: Delayed Internet Routing Convergence. In: Proceedings
of SIGCOMM, pp. 175–177 (August 2000)
12. Mao, Z.M., Bush, R., Griffin, T.G., Roughan, M.: BGP Beacons. In: Proc. ACM IMC (2003)
13. Mao, Z.M., Govidan, R., Varghese, G., Katz, R.H.: Route Flap Damping Excacerbates Inter-
net Routing Convergence. In: Proceedings of SIGCOMM (August 2002)
14. Panigl, C., Schmitz, J., Smith, P., Vistoli, C.: RIPE Routing-WG Recommendation for Coor-
dinated Route-flap Damping Parameters (2001),
http://www.ripe.net/ripe/docs/ripe- 229.html
15. Park, J.H., Jen, D., Lad, M., Amante, S., McPherson, D., Zhang, L.: Investigating Occurrence
of Duplicate Updates in BGP Announcements. In: Krishnamurthy, A., Plattner, B. (eds.)
PAM 2010 . LNC S, vol. 6 03 2 , pp . 11– 20. S pri nge r, H eid elb erg (20 10)
16. Rekhter, Y., Li, T.: A Border Gateway Protocol 4 (BGP-4) (2006), RFC 4271
17. Teixeira, R., Shaikh, A., Griffin, T.G., Voelker, G.M.: Network Sensitivity to Hot-Potato
Disruptions. In: Proc. ACM SIGCOMM (2004)
18. University of Oregon RouteViews project,
http://www.routeviews.org/
19. Villamiyar, C., Chandra, R., Govindan, R.: BGP Route Flap Damping (1998), RFC 2439
20. Wang, F., Mao, Z.M., Wang, J., Gao, L., Bush, R.: A Measurement Study on the Impact
of Routing Events on End-to-End Internet Path Performance. In: Proc. ACM SIGCOMM
(2006)
... These adaptations may demand significant changes in SP's network infrastructure. By contrast, reliability is essential to achieve high throughput with a reasonable 10 Quality of Service (QoS). Several aspects are relevant to do it for communication networks. ...
... As a consequence, one needs n · (n − 1) neurons to represent a communication network with n communication nodes. 10 The cost of the link from node x to node i is denoted by C xi , that assuming positive real and normalized values between (0, 1]. The cost will be zero for non-existent links. ...
... To simplify and speed up the HNN convergence, Bastos-Filho et al. [24] proposed a finite and discrete difference equation to replace the differential equation proposed by Ali and Kamoun [23]. 10 Park and Choi [26] found and described another problem in the Ali and Kamouns model. They stated that the model of Ali and Kamoun do not converge for networks with more than 20 nodes. ...
Article
This paper presents the Hierarchical Hopfield Neural Networks (HHNN). HHNN is a novel Hopfield Neural Network (HNN) approach. HHNN is composed of a hierarchy of self-sufficient HNNs, aiming to reduce the neural network structure and mitigate convergence problems. The HNNN composition depends on the applied problem. In this paper, the problem approached is the inter-domain routing for communication networks. Thus, the hierarchy of HNNs mimics the structure of communication networks (domains, nodes, and links). The proof of concept and the comparison between HNNN with the state-of-art HNN occurs using an implementation of them in the Java programming language. Besides, the performance analysis of the HHNN runs on a parallel hardware platform, using VHDL to develop it. The results have demonstrated a reduction of 93.75% and 99.98% in the number of neurons and connections to build the neural network, respectively. Furthermore, the mean time to achieve convergence of HHNN is rough 1.52% of the total time needed by the current state-of-art HNN approach. It is also less susceptible to early convergence problems when used in communications networks with a large number of nodes. Last, but not least, the VHDL implementation shows that convergence time of HHNN is comparable to routing algorithms used in practical applications.
... Finally, our current proposal assumes human-in-the-loop decisions. If they were integrated into an automatic defense, one may need to moderate the rate of routing changes to avoid interactions with route-ap damping [44]. ...
Preprint
IP Anycast is used for services such as DNS and Content Delivery Networks to provide the capacity to handle Distributed Denial-of-Service (DDoS) attacks. During a DDoS attack service operators may wish to redistribute traffic between anycast sites to take advantage of sites with unused or greater capacity. Depending on site traffic and attack size, operators may instead choose to concentrate attackers in a few sites to preserve operation in others. Previously service operators have taken these actions during attacks, but how to do so has not been described publicly. This paper meets that need, describing methods to use BGP to shift traffic when under DDoS that can build a "response playbook". Operators can use this playbook, with our new method to estimate attack size, to respond to attacks. We also explore constraints on responses seen in an anycast deployment.
... Currently, more and more unstable BGP traffic is observed by some previous researchers [11,35,36] . Unstable BGP traffic is defined as the BGP traffic that does not disseminate the business relationship between Internet Service Providers (ISPs) or threatens BGP operations. ...
Article
Full-text available
As autonomous systems tend to forward packets along the path with minimal routing cost, Internet routes are unevenly distributed on physical links. Links which a large number of routes go through are called routing bottlenecks. Flooding such routing bottlenecks can degrade or even cut off the network connectivity in a large area, making them a serious vulnerability of the Internet. In this paper, we study the characteristics of inter-domain routing bottlenecks and point out that they can be further aggravated by manipulating BGP updates to launch prefix hijackings. We first simulate large quantities of AS-level routing paths to illustrate the pervasiveness of inter-domain routing bottlenecks, as well as their direction, topological location, distance and concentration. Then, we propose a method for measuring and aggravating inter-domain bottlenecks of some AS, such that link flooding on them can be effectively amplified in a stealthy way. Moreover, adversary can adjust the specific method according to its purpose of malicious behaviour. At last, we discuss how inter-domain routing bottlenecks may be affected as the Internet evolves, where we witness the new, or wider, deployment of some routing related mechanisms.
... Despite the BGP qualities, there are some well known problems regarding the use of BGP algorithm. Among them, we can cite: the routing tables are large; different routes can be obtained for the same source-destination pair, thus generating instability; sub-optimal routes can arise during the network operation; and the time needed to achieve convergence can be extremely high when the number of nodes increases [3], [4]. ...
Conference Paper
Full-text available
We presented for the first time an Hierarchical Hopfield Neural Networks (HNN) for routing in communication networks that mimics the Border Gateway Protocol (BGP) We successfully showed that our approach can be applied for inter-domain routing. Besides, with this approach, the required number of neurons and connections to build the neural network is much lower than the state-of-art HNN routing algorithm and the time required for the establishment of routes was reduced drastically. Preliminary results indicate a reduction of 75% and 98.75% in the number of neurons and the number of connections for routing in a 16 nodes network and 4 different domains. Besides, the time to achieve convergence of our proposal is 2.66% of the time need by the current state-of-art HNN approach.
... Additionally, the BGP RFC states that 30 seconds is the recommended time to hold routes in the update phase [26], and similar to LIFEGUARD, we found our routes converged even faster in nearly all 1,460 instances of BGP poisoning path-enumeration experiments. Furthermore, recent work has found that penalties to many updates sent at once, otherwise known as route-flap dampen-ing, is no longer widely deployed in practice [24], and current BGP RFCs recommend changing the algorithm for dampening to be less restrictive. ...
Preprint
The security of the routing infrastructure as a set of protocols and routing process has underpinned much of the past two decades of distributed systems security research. However, the converse is becoming increasingly true. Routing and path decisions are now important for the security properties of systems built on top of the Internet. In particular, BGP poisoning leverages the de facto routing protocol between Autonomous Systems (ASes) to maneuver the return paths of upstream networks onto previously unusable, new paths. These new paths can be used to avoid congestion, censors, geo-political boundaries, or any feature of the topology which can be expressed at an AS-level. Given the increase in use of BGP poisoning as a security primitive for security systems, we set out to evaluate the feasibility of poisoning in practice, going beyond simulation. To that end, using a novel multi-country and multi-router Internet-scale measurement infrastructure, we capture and analyze over 1,400 instances of BGP poisoning across thousands of ASes as a mechanism to maneuver return paths of traffic. We additionally analyze filtering of BGP poisoning, connectivity concerns when poisoning, the presence of ASes that completely ignore poisoned providers, and finally an exhaustive measurement of a first-of-its-kind upper bound on the maximum path length of the Internet.
... Changing BGP policies can lead to route flapping [48], the situation in which BGP speaker sends an excessive number of BGP updates related to a single prefix or a set of prefixes [49]. To improve Internet stability at Internet edges, Minimum Route Advertisement Interval (MRAI) and Route Flap Damping (RFD) mechanisms have been developed to limit propagation of unstable routes. ...
Thesis
Full-text available
The Internet is subject to many types of attacks such as Denial of Service, hijacking of hosts and servers, and threats to routing protocols. The Border Gateway Protocol (BGP) is the Internet’s default inter-domain routing protocol responsible for exchanging Network Reachability Information between Autonomous Systems. Unfortunately, BGP was developed at a time when information exchanged between Autonomous Systems could be assumed to be accurate and therefore, is now vulnerable. This thesis introduces a novel scheme to rapidly detect BGP anomalies, enabling network operators to protect their network from the worst consequence of anomalous behaviour and to improve Internet stability.
... The slow convergence causes practical difficulties for network operators, and degradation or loss of service for their customers. The technology of 'route flap damping', which modifies the timing behavior of BGP in order to avoid propagating temporary oscillations, has been developed [108], criticized [87], and readjusted [94], but in a purely empirical fashion that does not address the root cause of delayed convergence. ...
Article
The age of computing with massive data sets is highlighting new computational challenges. Nowadays, a typical server may not be able to store an entire data set, and thus data is often partitioned and stored on multiple servers in a distributed manner. A natural way of computing with such distributed data is to use distributed algorithms: these are algorithms where the participating parties (i.e., the servers holding portions of the data) collaboratively compute a function over the entire data set by sending (preferably small-size) messages to each other, where the computation performed at each participating party only relies on the data possessed by it and the messages received by it. ^ We study distributed algorithms focused on two key themes: convergence time and data summarization. Convergence time measures how quickly a distributed algorithm settles on a globally stable solution, and data summarization is the approach of creating a compact summary of the input data while retaining key information. The latter often leads to more efficient computation and communication. The main focus of this dissertation is on design and analysis of distributed algorithms for important problems in diverse application domains centering on the themes of convergence time and data summarization. Some of the problems we study include convergence time of double oral auction and interdomain routing, summarizing graphs for large-scale matching problems, and summarizing data for query processing.
Preprint
Amplification DDoS attacks inherently rely on IP spoofing to steer attack traffic to the victim. At the same time, IP spoofing undermines prosecution, as the originating attack infrastructure remains hidden. Researchers have therefore proposed various mechanisms to trace back amplification attacks (or IP-spoofed attacks in general). However, existing traceback techniques require either the cooperation of external parties or a priori knowledge about the attacker. We propose BGPeek-a-Boo, a BGP-based approach to trace back amplification attacks to their origin network. BGPeek-a-Boo monitors amplification attacks with honeypots and uses BGP poisoning to temporarily shut down ingress traffic from selected Autonomous Systems. By systematically probing the entire AS space, we detect systems forwarding and originating spoofed traffic. We then show how a graph-based model of BGP route propagation can reduce the search space, resulting in a 5x median speed-up and over 20x for 1/4 of all cases. BGPeek-a-Boo achieves a unique traceback result 60% of the time in a simulation-based evaluation supported by real-world experiments.
Article
The Border Gateway Protocol propagates routing information accross the Internet in an incremental manner. It only advertises to its peers changes in routing. However, as early as 1998, observations have been made of BGP announcing the same route multiple times, causing router CPU load, memory usage and convergence time higher than expected. In this paper, by performing controlled experiments, we pinpoint multiple causes of duplicates, ranging from the lack of full RIB-Outs to the discrete processing of update messages. To mitigate these duplicates, we insert a cache at the output of the routers. We test it on public BGP traces and discuss the relation of the cache performance with the existence of bursts of updates in the trace.
Conference Paper
Full-text available
The scalability limitations of BGP have been a major concern in the networking community lately. An important issue in this respect is the rate of routing updates (churn) that BGP routers must process. This paper presents an analysis of the evolution of churn in four networks in the backbone of the Internet over the last six years, using update traces from the Routeviews project. The churn rate varies widely over time and between networks, and cannot be understood through "black-box'' statistical analysis. Instead we take a different approach with a focus on investigating the underlying reasons for BGP churn evolution. Through our analysis we are able to identify and isolate the main reasons behind many of the anomalies in the churn time series. We find that duplicate announcements is a major churn contributor, and responsible for most large spikes in the churn time series. Other intense periods of churn are caused by misconfigurations or other special events in or close to the monitored AS, and hence limiting these is an important mean to limit churn. We then analyze the remaining "baseline'' churn, and find that it is increasing with a rate much slower than the increase in the routing table size.
Conference Paper
Full-text available
The desire to better understand global BGP dynamics has motivated several studies using active measurement techniques, which inject announcements and withdrawals of prefixes from the global routing domain. From these one can measure quantities such as the BGP convergence time. Previously, the route injection infrastructure of such experiments has either been temporary in nature, or its use has been restricted to the experimenters. The routing research community would benefit from a permanent and public infrastructure for such active probes. We use the term BGP Beacon to refer to a publicly documented prefix having global visibility and a published schedule for announcements and withdrawals. A BGP Beacon is to be used for the ongoing study of BGP dynamics, and so should be supported with a long-term commitment. We describe several BGP Beacons that have been set up at various points in the Internet. We then describe techniques for processing BGP updates when a BGP Beacon is observed from a BGP monitoring point such as Oregon's Route Views. Finally, we illustrate the use of BGP Beacons in the analysis of convergence delays, route flap damping, and update inter-arrival times.
Conference Paper
Full-text available
. Recent work has shown that hot-potato disruptions can have a substantial impact on large ISP backbones and thereby jeopardize the network robustness. As a result, there is a need for guidelines and tools to assist in the design of networks that minimize hot-potato disruptions. However, developing these tools is challenging due to the complex and subtle nature of the interactions between exterior and interior routing. In this paper, we address these challenges using an analytic model of hot-potato routing that incorporates metrics to evaluate network sensitivity to hot-potato disruptions. We then present a methodology for computing these metrics using measurements of real ISP networks. We demonstrate the utility of our model by analyzing the sensitivity of a large AS in a tier~1 ISP network.
Article
Hot-potato routing is a mechanism employed when there are multiple (equally good) interdomain routes available for a given destination. In this scenario, the Border Gateway Protocol (BGP) selects the interdomain route associated with the closest egress point based upon intradomain path costs. Consequently, intradomain routing changes can impact interdomain routing and cause abrupt swings of external routes, which we call hot-potato disruptions . Recent work has shown that hot-potato disruptions can have a substantial impact on large ISP backbones and thereby jeopardize the network robustness. As a result, there is a need for guidelines and tools to assist in the design of networks that minimize hot-potato disruptions. However, developing these tools is challenging due to the complex and subtle nature of the interactions between exterior and interior routing. In this paper, we address these challenges using an analytic model of hot-potato routing that incorporates metrics to evaluate network sensitivity to hot-potato disruptions. We then present a methodology for computing these metrics using measurements of real ISP networks. We demonstrate the utility of our model by analyzing the sensitivity of a large AS in a tier~1 ISP network.
Conference Paper
We study the route oscillation problem [16, 19] in the Internal Border Gateway Protocol (I-BGP) [18] when route reflection is used. We propose a formal model of I-BGP and use it to show that even deciding whether an I-BGP configuration with route reflection can converge is an NP-Complete problem. We then propose a modification to I-BGP and show that route reflection cannot cause the modified protocol to diverge. Moreover, we show that the modified protocol converges to the same stable routing configuration regardless of the order in which messages are sent or received.
Conference Paper
Route flap damping is considered to be a widely deployed mechanism in core routers that limits the widespread propagation of unstable BGP routing information. Originally designed to suppress route changes caused by link flaps, flap damping attempts to distinguish persistently unstable routes from routes that occasionally fail. It is considered to be a major contributor to the stability of the Internet routing system. We show in this paper that, surprisingly, route flap damping can significantly exacerbate the convergence times of relatively stable routes. For example, a route to a prefix that is withdrawn exactly once and re-announced can be suppressed for up to an hour (using the current RIPE recommended damping parameters). We show that such abnormal behavior fundamentally arises from the interaction of flap damping with BGP path exploration during route withdrawal. We study this interaction using a simple analytical model and understand the impact of various BGP parameters on its occurrence using simulations. Finally, we outline a preliminary proposal to modify route flap damping scheme that removes the undesired interaction in all the topologies we studied.
Article
An abstract is not available.