The Impact of Path Selection Algorithm of Adaptive Routing for Implementing Deterministic Routing.

Full text

Page 1

The Impact of Path Selection Algorithm of Adaptive Routing
for Implementing Deterministic Routing
Michihiro KoibuchiAkiya Jouraku Hideharu Amano
Dept. of Information and Computer Science
Keio University
3-14-1 Hiyoshi, Kohoku-ku, Yokohama, 223-8522 Japan
?koibuchi,jouraku,hunga
? @am.ics.keio.ac.jp
Abstract
In PC clusters or high performance I/O networks in-
cluding InfiniBand, network topologies often become irreg-
ular. Although various adaptive routings for irregular net-
works have been proposed, most of such commercial or ex-
perimental networks use a deterministic routing which en-
ables a simple switch structure and in-order packet deliv-
ery. A strategy of path selection algorithm which fixes a
single path among alternative paths between each pair of
switches(hosts) is required but only a few studies have been
asserted. In this paper, we propose three path selection al-
gorithms which have different concepts using a static anal-
ysis of routing path to distribute the traffic, and investigate
the influences of path selection algorithms on the through-
put. Result of simulations shows that the throughput of each
path selection algorithm depends on routing algorithm and
topology, and the path selection algorithms using a static
analysis of routing path achieves higher throughput com-
pared with one without using it.
Keywords
adaptive routing, deadlock avoidance, networks of worksta-
tions
Irregular networks, deterministic routing,
1Introduction
Switch-based irregular networks are commonly used in
high performance distributed computing systems with com-
modity personal computers[12],[14],[15] and also in high
performance I/O networks including InfiniBand[2]. Adap-
tive routing techniques for irregular networks have been
widely studied[1],[9],[6], and their superior performance
compared with deterministic routings have been demon-
strated. Nevertheless, a lot of real networks only support
deterministic routings because of the following reasons: (1)
in-order packet transfer property is important in PC (Per-
sonal Computer) networks, (2) once a system trouble oc-
curs, it is hard to trace adaptive routed packets in com-
plicated irregular network, and (3) switch structures with
a simple control mechanism are preferred in irregular net-
works. Also in InfiniBand, although multiple paths can be
selected between CA (Channel Adapter)s in a sub-net, a de-
terministicroutingwithtablesineachswitchismostlyused.
This comes from that the source CA indicates a path via its
selection of them[2],[13].
In order to apply an existing adaptive routing, such
as, up*/down* routing[8] or L-turn routing[9], to such
real irregular networks, a policy of path selection which
chooses a path among alternative paths between each pair
of switches(hosts) is essential to performance. In this pa-
per, such a policy is called “path selection algorithm”. Path
selection algorithm considers the traffic distribution regard-
less of guarantee of deadlock-free because adaptive routing
guarantees deadlock-free, and it would be a key to imple-
ment a deterministic routing using techniques for adaptive
routings in irregular networks
Unfortunately, only a few researches into path selection
algorithm have been done[4], and the impact to the perfor-
mance has not been well analyzed. So, when designing a
deterministic routing for real irregular networks based on
techniques used in an adaptive routing, it is difficult to se-
lect a suitable path selection algorithm.
Here, we present three path selection algorithms, which
use static analysis results of routing path by different man-
ners. The performance evaluation results with computer
simulation are shown to demonstrate their efficiency.
? .
?
Notingthatpath selectionalgorithmcannotapplytoaspecialadaptive
routing called Silla’s minimal routing because it guarantees deadlock-free
through selecting a path between original channel(deadlock-free path) and
new channel(fully adaptive path) dynamically.

Page 2

2 Existing path selection algorithms
An adaptive routing is a technique to select a route of
packetdynamically,and soit can dynamicallyavoidthe net-
work congestion. However, in order to implement a deter-
ministic routing, a path selection algorithm must be applied
to an adaptive routing for fixing a single path from alter-
native paths, and it can not dynamically avoid the network
congestion. Nevertheless, path selection algorithms are es-
sential to performance since it can mitigate the congestion
aroundthe hot spot in most case if well-distributed paths are
set.
The simplest path selection algorithm is random selec-
tion. Another simple one selects a path for the port with
smaller port-ID when more than two channels are available
in a switch. In this paper, this is called “low port first”.
However, above two path selection algorithms possibly se-
lect a path to congestion points even if there exist some can-
didates which can avoid it.
To address this problem, traffic balancing algorithm
using a static analysis of routing path is proposed by
Sancho[4] as follows.
1. All possible routing paths between every pair of
switches are calculated. Then, this algorithm asso-
ciates a counter to every channel, and each counter is
initialized to the number of routing paths crossing the
channel.
2. A routing path crossing the channel with the highest
value of counter is selected to be removed if there is
more than one routing paths between the source and
the destination switches of it. If there is more than
one routing path which can be removed in a chan-
nel, the routing path whose source and destination
hosts have the highest number of routing paths be-
tween them is selected.
3. When a routing path is removed, the counters associ-
ated with every channel crossed by the path are up-
dated.
4. Repeat the procedure 2 until the number of routing
paths between every pair of hosts is reduced down to
the unit.
The time complexity to compute this traffic balancing
algorithm is
switches.
??????????????????????????? , where
?
is the number of
3 Path selection algorithms based on a static
analysis of routing path
Although Sancho’s traffic balancing algorithm[4] is an
efficient method based on a static analysis of routing path
to distribute the traffic, there are other concepts worth to
try.
In this section, we present three novel path selection al-
gorithms: “high physical channel first”, “low virtual chan-
nel first”, and “low physical channel first”. These have
the same procedure flow as Sancho’s traffic balancing al-
gorithm, but the step 2 is different from Sancho’s one as
follows.
?
High physical channel first selects the virtual chan-
nel with the highest value of counter on the physi-
cal channel with the highest value of sum of its vir-
tual channel’s counters, and removes the routing path
on it if there is more than one routing paths between
the source and the destination switches of this rout-
ing path. If there is more than one routing path which
can be removed in a channel, the routing path whose
source and destination hosts have the highest num-
ber of routing paths between them is selected to be
removed.
?
Low virtual channel first selects the virtual channel
with the lowest value of counter, and fixes a routing
path on it. That is, the other routing paths between
the same source and the same destination switches
are removed. If there is more than one routing path
which is still not fixed in a channel, a routing path
crossing the channel with the lowest value of counter
is selected.
?
Low physical channel first selects the virtual channel
with the lowestvalue of counter on the physical chan-
nel with the lowest value of sum of its virtual chan-
nel’s counters, and fixes a routing path on it. If there
is more than one routing path which is still not fixed
in a channel, a routing path crossing the channel with
the lowest value of counter is selected.
Sancho’s traffic balancing algorithm is designed for a
networkwith a fewvirtual channels. However, virtual chan-
nels which can use the physical channel in time-sharing
manner are plentifully equipped in recent switches, and the
congestion of physical channels will tend to be a problem in
such networks. So, high physical channel first is designed
so as to avoid the physical channel bottleneck as well as
virtual channels. On the other hand, low virtual channel
first tries to use all virtual channels efficiently by avoiding
a virtual channels with extremely small utilization. Note
that Sancho’strafficbalancing algorithm means high virtual
channel first in the point of the selection policy.
Figure 1 shows an example of the irregular network
with five switches using one bidirectional channel between
switches, and up*/down* routing[8] is applied on it. which
is a typical partially adaptive routing. In up*/down* rout-
ing, a packet must be transferred by using the channels

Page 3

which face to the root (if needed) followed by the channels
which go away from the root(if needed) in order to avoid
deadlocks. This restriction prevents a packet from turning
from down direction to up direction.
In Figure 1, the value of counter to each channel is cal-
culated according to the number of routing paths crossing
it. For example, the value of the counter on the chan-
nel fromto
are crossing it, where
?? is four because
??? ?!?"???#???$???!???!??%????#?&?'???(?!?!?
??)*?&+,? is the routing path from
)
to
+ . When implementing a deterministic routing,
??? ?!?"? and
???#?-?.? have two candidates
rithms, random and low port first may select the former one
which goes through the congestion channel from
or
algorithms using a static analysis of routing paths select the
latter one.
???0/1%?/1?"???#???0/1??/2?3? and
???4/5%6/7?.???#???8/7?4/7?.? respectively. Simple algo-
? or
% to
%
? respectively. On the other hand, the four path selection
up direction
2
2
4
4
6
4
4
3
3
6
e
c
d
a
b
Figure 1. The example of counters to routing
paths on up*/down* routing
Figure 2 shows the next example of the counter on
up*/down* routing.Then,
candidates
???$???3? and
???#?!?.? have two
???9/:%6/;?"???'?&?</5?4/;?"? and
fic balancing algorithm and high physical channel first se-
lect the former one, while low physical channel first and
low virtual channel first select the latter one. This comes
from that Sancho’s one and high physical channel first try
to remove the bottleneck channels, while the low physical
channel first and the low virtual channel first are designed
to avoid the channels with extremely small utilization.
???=/:%6/
?>????????/@?A/@?>? respectively. In this case, Sancho’s traf-
4Performance evaluation
In this section, performance of path selection algorithms
on up*/down* routing or the UDWM[11] is evaluated by
the computer simulation,
up direction
x < y < z
y
x
zy
d
f
g
a
e
bc
Figure 2. The example of counters to routing
paths on up*/down* routing
4.1Network model
A flit-level simulator written in C++ was developed for
analysis. Topology, network size, and packet length are
selected just by changing parameters. A switching fabric
which provides eight bidirectional ports, using four ports to
connect with hosts and remaining four ports for connecting
other switches. Here, a simple model consisting of channel
buffers, crossbar, link controller and control circuits is used
for the switching fabric. 10 different topologies are ran-
domly generated on condition that every different link must
be connected with a different neighbor switches. A desti-
nation of a packet is determined by a traffic pattern used in
the simulation. Here, the uniform traffic in which all desti-
nations are selected randomly is used.
Simulation parameters are set as shown in Table 1.
Table 1. Simulation parameters
Simulation time1,000,000 clocks
(ignore the first 50,000 clocks)
Topologyirregular or torus
Network size16 switches or 64 switches
The number of1 or 5
virtual channels
Packet length128 flits
Adaptive routingup*/down* or the UDWM
Switching tech.virtual cut-through
Traffic patternuniform
4.2 Routing algorithms
Up*/down* routing
ular deadlock-free adaptive routing for irregular networks,
Up*/down* routing is the most pop-

Page 4

and has been used in Autonet[8].
tee connectivity and deadlock-free for irregular networks,
up*/down* routing needs a spanning tree based directed
graph in which up or down direction is assigned to each
network channel. Several spanning trees for an irregular
topology can be structured depending on the tree building
and root selection policies. In this simulation, simple poli-
cies used in Autonet are applied, that is, the BFS (Breadth
First Search) is used for building spanning trees, and the
switch with identifier
is selected as the root. As men-
tioned in Section 3, a packet must be transferred by using
the channels which face to the root (if needed) followed by
the channels which go away from the root(if needed) in or-
der to avoid deadlocks.
In order to guaran-
B
0
1
78
11
2
6
9
3
10
1213
45
up direction
up*/down* routing
The UDWM using two virtual channels
The UDWM using three virtual channels
Figure 3. 14 switches irregular network
TheUDWM
channels) is an improved routing algorithm of up*/down*
routing so as to best use of virtual channels[11].
UDWM has the same restrictions except the following con-
dition as up*/down* routing: The turn from down channel
to up channel is only used with descending virtual channel
number.
In the example shown by Figure 3, when a packet
is transferred from 7 to 13 with up*/down* routing, the
pathtakes 7 hops(7
410
of the number of virtual channels.
when the UDWM is used and each physical link splits
into two virtual channels called “
Figure 3, the path takes only 5 hops(7
96
by decreasing a number of virtual channel.
when each physical link splits into three virtual chan-
TheUDWM(up*/down*routing withmulti-
The
////
3
/
6
/
10
/
13) regardless
On the other hand,
??CEDB ” and“
??C?DGF ” in
/????C?DGFH??/
11
/
????C?DBE??//I?&??CED BE?J//;????CEDBE?J/
10
/:????C?D BE?J/
13)
Moreover,
nels called “
UDWM takes only 4 hops(7
12
tual channel twice.
??C?DB ”, “
??CED?F ”, and “
13) by decreasing a number of vir-
??CEDK ”, the path of the
11
/L????C?DKE??//M????CED?FH?N/
9
/
????C?DGFH?N//L????C?DBE?O/
4.3Simulation Results
4.3.1 16 switches irregular networks with 1 virtual
channel
0
0.04
0.08
0.12
0.16
Throughput[flits/clk/host]
RandomLow port
first
Traffic
balancing
Low pch first
Figure 4. Throughput on 16 switches irreg-
ular networks with 1 virtual channel under
up*/down* routing
Figure 4 shows average throughput of 10 irregular
topologies with 16 switches. Here, throughput is defined as
the maximum amount of accepted traffic. Accepted traffic
is the flit reception rate in host in each clock cycle.
Table 2 shows standard deviation(SD) of channel cross-
ing paths. Here, channel crossing paths are defined as the
averagenumber of routing paths crossing through any phys-
ical channel after selecting only one path between each pair
of switches. It shows how uniformly the paths are dis-
tributed, that is, the small channel crossing paths mean that
the paths are distributed uniformly.
In all figures and tables on this section, high physical
channel first, low virtual channel first, and low physical
channel first are shown as “high pch first”, ”low vch first”,
and “low pch first” respectively.
As shown in Figure 4, the throughput of Sancho’s traf-
fic balancing algorithm achieves better than low vch first a
little. This comes from that Sancho’s one distributes the
routing paths more uniformly as shown in Table 2.

Page 5

0.04
0.08
0.12
0.16
0.2
Throughput[flits/clk/host]
0
Random Low port
first
(a) Up*/down* routing
Traffic
balancing
Low vch
first
High pch
first
Low pch
first
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Throughput[flits/clk/host]
Random Low port
first
(b) The UDWM
Traffic
balancing
Low vch
first
High pch
first
Low pch
first
Figure 5. Throughput on 16 switches irregular networks with 5 virtual channels
Table 2. Routing metric on 16 switches irreg-
ular networks with 1 virtual channel under
up*/down* routing
Path selection
algorithm
Random
Low port first
Traffic balancing
Low vch first
SD of channel
crossing paths
5.63
5.61
5.06
5.39
4.3.216 switches irregular networks with 5 virtual
channels
Figure 5 shows average throughput of 10 irregular topolo-
gies with 16 switches. The condition of simulation is the
same in the before section except the number of virtual
channels.
Figure 5 and Table 3 demonstrate that the throughput
of each path selection algorithm is depending on the rout-
ing algorithm, but Sancho’s one and high physical channel
first outperform compared with low physical channel first
and lowest virtual channel first. Consequently, in order to
distribute the traffic, the methods to remove the bottleneck
channels are more efficient than the methods to avoid the
channels with extremely small utilization.
4.3.3 64 switches 2D torus
Figure 6 shows simulation results of
condition of simulation is the same in the above section ex-
ceptthe networksizeand topology. Table4also showsrout-
PRQ6P 2D torus. The
Table 3. Routing metric on 16 switches irreg-
ular networks with 5 virtual channels
Path selection
algorithm
SD of channel
crossing paths
ud UDWM
5.54
5.61
5.09
5.07
5.38
5.52
Random
Low port first
Traffic balancing
High pch first
Low vch first
Low pch first
2.87
3.04
2.27
2.23
2.70
2.75
ing metric in the case. As shown in Figure 6 and Table 4,
the path selection algorithms using a static analysis of rout-
ing paths achieves higher throughput compared with ones
without using it. From Figure 4, Figure 5, and Figure 6,
the throughput of each path selection algorithm depends on
routing algorithm and topology.
Figure 7 shows the distribution of channel utilization on
8
obtained. Here, a switch whose switch number is
the 2D torus is selected as the root.
Note that the uniform traffic is used in this simulation.
Although 88 torus is a uniform topology, all path selec-
tion algorithms tendto gather manypacketsaround the root.
This comes from that up*/down* routing essentially makes
concentrated traffic around the root. Nevertheless, the path
selection algorithms using a static analysis of routing path
mitigate to this problem, and achieve high channel utiliza-
tion.
Q 8 2D torus when the throughput shown in Figure 6 is
?SBT??B?? on
Q