A remotely accessible network processor-based router for network experimentation.

Full text

Page 1

A Remotely Accessible Network Processor-Based Router
for Network Experimentation
Charlie Wiseman, Jonathan Turner, Michela Becchi, Patrick Crowley, John DeHart,
Mart Haitjema, Shakir James, Fred Kuhns, Jing Lu, Jyoti Parwatikar, Ritun Patney,
Michael Wilson, Ken Wong, David Zar
Department of Computer Science and Engineering
Washington University in St. Louis
{cgw1,jst,mbecchi,crowley,jdd,mah5,scj1,fredk,jl1,jp,ritun,mlw2,kenw,dzar}@arl.wustl.edu
ABSTRACT
Over the last decade, programmable Network Processors
(NPs) have become widely used in Internet routers and other
network components. NPs enable rapid development of com-
plex packet processing functions as well as rapid response to
changing requirements. In the network research community,
the use of NPs has been limited by the challenges associ-
ated with learning to program these devices and with using
them for substantial research projects. This paper reports
on an extension to the Open Network Laboratory testbed
that seeks to reduce these “barriers to entry” by providing
a complete and highly configurable NP-based router that
users can access remotely and use for network experiments.
The base router includes support for IP route lookup and
general packet filtering, as well as a flexible queueing sub-
system and extensive support for performance monitoring.
In addition, it provides a plugin environment that can be
used to extend the router’s functionality, enabling users to
carry out significant network experiments with a relatively
modest investment of time and effort. This paper describes
our NP router and explains how it can be used. We provide
several examples of network experiments that have been im-
plemented using the plugin environment, and provide some
baseline performance data to characterize the overall system
performance. We also report that these routers have already
been used for ten non-trivial projects in an advanced ar-
chitecture course where most of the students had no prior
experience using NPs.
Categories and Subject Descriptors
C.2.1 [Computer-Communication Networks]: Network
Architecture and Design—network communications
General Terms
Design, Experimentation
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
ANCS’08, November 6–7, 2008, San Jose, CA, USA.
Copyright 2008 ACM 978-1-60558-346-4/08/0011 ...$5.00.
Keywords
Programmable routers, network testbeds, network proces-
sors
1.INTRODUCTION
Multi-core Network Processors (NPs) have emerged as a
core technology for modern network components. This has
been driven primarily by the industry’s need for more flexi-
ble implementation technologies that are capable of support-
ing complex packet processing functions such as packet clas-
sification, deep packet inspection, virtual networking and
traffic engineering. Equally important is the need for sys-
tems that can be extended during their deployment cycle to
meet new service requirements. NPs are also becoming an
important tool for networking researchers interested in cre-
ating innovative network architectures and services. They
also make it possible for researchers to create experimen-
tal systems that can be evaluated with Internet-scale traf-
fic volumes. It is likely that NP-based processing elements
will play a significant role in NSF’s planned GENI initiative
[6][15], providing researchers with expanded opportunities
to make use of NPs in their experimental networks.
Unfortunately, NPs pose significant challenges to research
users. While network equipment vendors can afford to in-
vest significant time and effort into developing NP software,
it is more difficult for academic researchers to develop and
maintain the necessary expertise.
sons that using NPs is challenging. First, it can be diffi-
cult to obtain NP-based products, since manufacturers of
those products sell primarily to equipment vendors and not
to end users. Second, developing software for NPs is chal-
lenging because it requires programming for parallel execu-
tion, something most networking researchers have limited
experience with, and because it requires taking advantage
of hardware features that are unfamiliar and somewhat id-
iosyncratic. Third, there is no established base of existing
software on which to build, forcing researchers to largely
start from scratch.
To address these challenges we have designed, implem-
ented, and deployed a gigabit programmable router based on
Intel’s IXP 2800 network processor that is easy to use and
can be easily extended through the addition of software plu-
gins. Our Network Processor-based Router (NPR) takes care
of standard router tasks such as route lookup, packet clas-
sification, queue management and traffic monitoring which
enables users to focus on new network architectures and ser-
There are several rea-
20

Page 2

Figure 1: Basic block diagram representation of the
IXP 2800 [16].
vices without the need to develop an entire router from the
ground up. In addition, we provide a plugin environment
with significant processing and memory resources, and an
API that implements common functions needed by most
users. This makes it possible for users to develop plugins
for non-trivial network experiments without having to write
a lot of new code. Fourteen of these routers are currently
deployed in the Open Network Laboratory (ONL) testbed
[5]. The ONL is an Internet-accessible testbed which makes
it possible for anyone to use these resources without hav-
ing to acquire and manage the systems themselves. All of
the source code for the plugin API and example plugins, as
well as source code for most of the main router pipeline1is
available through the ONL website [14].
The rest of the paper is organized as follows. Section 2
gives a detailed description of the NPR, including how it
fits into our larger network testbed. Section 3 discusses the
programmable components in the router. Section 4 includes
some examples of plugins that have already been developed
for our routers. Section 5 provides a brief performance eval-
uation of the NPR. Section 6 contains related work, and
Section 7 concludes the paper with a few closing remarks
about our overall experience with the system.
2.NPR OVERVIEW
To understand the design of the NPR, it is important to
know a few things about the architecture of the Intel IXP
2800 [8]. The IXP 2800, like most network processors, is
designed specifically for rapid development of high perfor-
mance networking applications. As such it has some unusual
architectural features relative to general purpose processors
which heavily influences how it is used.
Figure 1 is a block diagram showing the major compo-
nents of the IXP. To enable high performance, while still
providing application flexibility, there are 16 multi-threaded
MicroEngine (ME) cores which are responsible for the ma-
jority of the packet processing in the system. Each ME has
eight hardware thread contexts (i.e., eight distinct sets of
registers). Only a single thread is active at any given time on
one ME, but context switching takes a mere 2-3 clock cycles
(about 1.5-2 ns). These threads are the mechanism by which
1There is a very small amount of proprietary code from our
NP vendor.
IXP applications deal with the memory latency gap. Indeed,
there are no caches in the IXP because caches are not par-
ticularly effective for networking applications that exhibit
poor locality-of-reference. There is no thread preemption,
so a cooperative programming model must be used. Typi-
cally, threads pass control from one to the next in a simple
round-robin fashion. This is accomplished using hardware
signals, with threads commonly yielding control of the pro-
cessor whenever they need to access memory that is not local
to the ME. This round-robin style of packet processing also
provides a simple way to ensure that packets are forwarded
in the same order they are received. Finally, each ME has a
small hardware FIFO connecting it to one other ME, which
enables a fast pipelined application structure. These FIFOs
are known as next neighbor rings.
The IXP 2800 also comes equipped with 3 DRAM chan-
nels and 4 SRAM channels. There is an additional small
segment of data memory local to each ME along with a ded-
icated program store with a capacity of 8K instructions. A
small, shared on-chip scratchpad memory is also available.
Generally, DRAM is used only for packet buffers. SRAM
contains packet meta-data, ring buffers for inter-block com-
munication, and large system tables. In our systems, one of
the SRAM channels also supports a TCAM which we use for
IP route lookup and packet classification. The scratchpad
memory is used for smaller ring buffers and tables.
Finally, there is an (ARM-based) XScale Management
Processor, labeled MP in Figure 1, that is used for overall
system control and any tasks which are not handled by the
rest of the data path. The XScale can run a general-purpose
operating system like Linux or a real-time operating system
like VxWorks. Libraries exist which provide applications
on the XScale direct access to the entire system, including
all memory and the MEs. With this background, we now
proceed to a detailed description of the NPR.
2.1 Data Plane
The software organization and data flow for the NPR are
shown in Figure 2. Note the allocation of MEs to different
software components. The main data flow proceeds in a
pipelined fashion starting with the Receive block (Rx) and
ending with the Transmit block (Tx). Packets received from
the external links are transferred into DRAM packet buffers
by Rx. Rx allocates a new buffer for each incoming packet
and passes a pointer to that packet to the next block in the
pipeline, along with some packet meta-data. In order to best
overlap computation with high latency DRAM operations,
Rx processing is broken up into two stages with each stage
placed on a separate ME. Packets flow from the first stage
to the second over the next neighbor ring between the MEs.
Note that, in general, the information passed between
blocks in the diagram consists of packet references and se-
lected pieces of header information, not packet data. Thus,
packets are not copied as they move (logically) from block
to block. Each block can access the packet in DRAM using
its buffer pointer, but since DRAM accesses are relatively
expensive, the system attempts to minimize such accesses.
Also note that most blocks operate with all eight threads
running in the standard round-robin fashion.
The Multiplexer block (Mux) serves two purposes. Each
packet buffer in DRAM has an associated 32B entry in
SRAM which stores some commonly needed information
about the packet, such as the packet length. Mux initial-
21

Page 3

Figure 2: Data plane of the NPR.
izes this meta-data for packets coming from Rx. Its second
function is to multiplex packets coming from blocks other
than Rx back into the main pipeline. This includes packets
coming from the XScale and packets coming from plugins.
A simple user-configurable priority is used to determine how
to process packets from the different sources.
The Parse, Lookup, and Copy block (PLC) is the heart
of the router. Here, packet headers are inspected to form a
lookup key which is used to find matching routes or filters
in the TCAM. Routes and filters are added by the user, and
will be discussed further in Section 3.2. Based on the result
of the TCAM lookup, PLC takes one of five actions. First,
the packet can be sent out towards the external links via
the Queue Manager block (QM). Second, the packet can be
sent to the XScale if it has some special processing needs
not handled by the ME pipeline. Third, the packet can be
sent to a plugin ME (hosting user code). Plugins will be
discussed fully in Section 3.1, but, as can be seen in Fig-
ure 2, plugins will be able to forward packets to many other
blocks including the QM, Mux, and other plugins. Fourth,
the packet can be dropped. Finally, multiple references to
the same packet can be generated and sent to distinct desti-
nations. For example, one reference may be sent to a plugin
and another directly to the QM. In fact, this mechanism al-
lows the base router to support IP multicast (among other
things). Reference counts are kept with the packet meta-
data in SRAM to ensure that the packet resources are not
reclaimed prematurely. Note that this does not allow differ-
ent “copies” to have different packet data because there is
never more than one actual copy of the packet in the system.
Before moving on, it is worth discussing the design choices
made for PLC. Three MEs all run the entire PLC code block
with all 24 threads operating in a round-robin fashion. One
alternative would be to break up the processing such that
Parse is implemented on one ME, Lookup on a second, and
Copy on a third. Our experience shows that the integrated
approach chosen here yields higher performance. This is pri-
marily due to the nature of the operations in PLC. To form
the lookup key, Parse alternates between computation and
high latency DRAM reads of packet headers, and Lookup
spends most of its time waiting on TCAM responses. On the
other hand, Copy is computation-bound due to the poten-
tially complex route and filter results which must be inter-
preted. Combining all three blocks together provides enough
computation for each thread to adequately cover the many
memory operations.
Continuing down the main router pipeline, the QM places
incoming packets into one of 8K per-interface queues. A
weighted deficit round robin scheduler (WDRR) is used to
pull packets back out of these queues to send down the
pipeline. In fact, there is one scheduler for each external
interface, servicing all the queues associated with that in-
terface. Each queue has a configurable WDRR quantum
and a configurable discard threshold.
of bytes in the queue exceeds the discard threshold, newly
arriving packets for that queue are dropped. The QM has
been carefully designed to get the best possible performance
using a single ME. The design uses six threads. One handles
all enqueue operations, and each of the remaining five im-
plements the dequeue operations for one outgoing interface.
This decouples the two basic tasks and enables the QM to
achieve high throughput.
Following the QM is the Header Format block (HF) which
prepares the outgoing Ethernet header information for each
packet. It is responsible for ensuring that multiple copies
of a packet (that have potentially different Ethernet ad-
dresses) are handled correctly. Finally, the Transmit block
(Tx) transfers each packet from the DRAM buffer, sends it
to the proper external link, and deallocates the buffer.
There are two additional blocks which are used by all
the other blocks in the router. First is the Freelist Man-
ager block (FM). Whenever a packet anywhere in the sys-
tem is dropped, or when Tx has transmitted a packet, the
packet reference is sent to the FM. The FM then reclaims
the resources associated with that packet (the DRAM buffer
and the SRAM meta-data) and makes them available for re-
allocation. The Statistics block (Stats) keeps track of various
counters through-out the system. There are a total of 64K
counters that can be updated as packets progress through
the router. Other blocks in the system issue counter up-
dates by writing a single word of data to the Stats ring
buffer, which includes the counter to be updated and incre-
ment to be added. For example, there are per-port receive
and transmit counters which are updated whenever packets
are successfully received or transmitted, respectively. There
are also counters for each route or filter entry that are up-
dated both before and after matching packets are queued
in the QM. This provides a fine-grained view of packet flow
When the number
22

Page 4

Figure 3: Control plane of the NPR.
in the router. The counters all ultimately reside in SRAM,
but there are 192 counters which are also cached locally in
the Stats ME. One thread periodically updates the SRAM
counterparts to these counters while the other threads all
process update requests from the ring buffer.
The final block in the diagram is the XScale. The primary
purpose of the XScale is to control the operation of the data
plane, as discussed in detail in the next section. However,
the XScale also plays a small role in the data plane. Specifi-
cally, it handles any IP control packets and other exceptional
packets. This includes all ICMP and ARP handling as de-
fined by the standard router RFCs. All messages generated
by the XScale are sent back through Mux to PLC, allowing
users to add filters to redirect these packets to plugins for
special processing, should they desire to do so.
2.2Control Plane
The XScale’s main function is to control the entire router
and act as the intermediary between the user and the data
plane. This is accomplished by a user-space control daemon
running under Linux on the XScale. Libraries provided by
Intel are used to manage the MEs and the memory in the
router, and a library provided by the TCAM vendor contains
an API for interacting with the TCAM. Figure 3 summarizes
the most important roles of the XScale daemon. Messages
come from the user to request certain changes to the system
configuration. Typically the first such request is to start
the router which involves loading all the base router code
(i.e., everything except plugins) onto the MEs and enabling
the threads to run. Once the data path has been loaded
successfully, there are several types of control operations
that can be invoked.
The first of these involves configuration of routes and fil-
ters, which are discussed in detail in Section 3.2. The XS-
cale also supports run-time configuration of some data plane
blocks, by writing to pre-defined control segments in SRAM.
For example, queue thresholds and quanta used by the QM
can be dynamically modified by the user in this way.
The XScale also provides mechanisms to monitor the state
of the router. All of the counters in the system are kept in
SRAM, which allows the XScale to simply read the memory
where a certain counter is stored to obtain the current value.
These values can be sampled periodically in order to drive
real-time displays of packet rates, queue lengths, drop coun-
ters, etc. The system can support dozens of these ongoing
requests concurrently.
All plugin operations are also handled by the XScale. In
particular, this involves adding and removing user-developed
plugin code on the plugin MEs and passing control mes-
sages from the user to the plugins after they are loaded.
The control messages are forwarded by the XScale through
per-plugin rings, and replies are returned in a similar way.
The XScale is oblivious to the content of these control mes-
sages, allowing users to define whatever format and content
is most appropriate for their applications. Plugins can also
effect changes to the router by sending requests through
these rings to the XScale. Once a plugin is loaded, users
can add filters to direct specific packet flows to the plugin.
2.3 Testbed
The NPR has been deployed in the ONL testbed. Re-
searchers can apply for accounts to gain Internet access to
the testbed so that they can use the testbed resources to
run their experiments (at no cost, of course). The general
layout of the testbed is shown in Figure 4.
The testbed itself consists of standard Linux PCs along
with the NPRs. All of the data interfaces on these com-
ponents are connected directly to a gigabit configuration
switch, and thus indirectly to all other components. The
hosts all have separate control interfaces, and the XScales
serve as the control interface for the NPRs as they have their
own network interface which is separate from the data in-
terfaces. Everything is managed by our testbed controller
which runs on another Linux PC. Users configure network
topologies with the Remote Laboratory Interface (RLI) on
their local computer. The RLI also provides an intuitive
way to control and monitor the individual routers and hosts
in the topology. An example session is shown in the lower
left of the figure.
Once connected to the testbed controller the RLI sends
the topology to the controller so that hardware resources
can be allocated to the user, and the chosen components
can be connected together with VLANs in the configuration
switch. Hardware resources are reserved ahead of time either
through the RLI or on the ONL website so that when a
user is ready to run their experiment enough resources are
guaranteed to be available. All assigned resources are given
entirely to the user so that there can be no interference from
other concurrent experiments. Users are also granted SSH
access to hosts in their topology so that they can log in to
start any traffic sources and sinks.
The ONL currently has 14 NPRs and over 100 hosts as
well as some additional routers and other networking com-
ponents. The routers themselves are built on ATCA tech-
23

Page 5

Figure 4: A high-level view of our Internet-accessible testbed.
nology, which is an industry standard for networking com-
ponents that has broad industry support. ATCA technol-
ogy is also proving to be a boon to networking research, as
it enables the assembly of powerful, yet highly flexible ex-
perimental networking platforms. In particular, the NPR
is built on Radisys ATCA-7010 boards [11], shown in the
bottom right of Figure 4. Each board has two IXP 2800s,
a shared TCAM, and ten 1 gigabit data interfaces. In our
context, we use the two IXPs as separate five port routers,
assigning five of the data interfaces to each IXP.
3. PROGRAMMABILITY
Now that the basics of the NPR have been covered, we
turn our attention to the progammable components of the
router.There are two primary facets of the overall pro-
grammability of the NPR: plugins and filters. Together they
provide users a rich selection of options with which to cus-
tomize processing and packet flow in the NPR.
3.1Plugin Framework
Recall from Figure 2 that five MEs are used to host plu-
gins. NPR users are free to load any combination of code
blocks onto these MEs. In addition to the five MEs there
are five ring buffers leading from PLC to the plugins which
can be used in any combination with the MEs. For example,
each plugin may pull packets from a separate ring (this is
the default behavior) or any subset of plugins may pull from
the same ring. The ring buffers, then, are simply a level of
indirection that adds versatility to potential plugin archi-
tectures. The NPR also sets aside 4KB of the scratchpad
memory and 5MB of SRAM exclusively for plugin use.
The actual processing done by any particular plugin is en-
tirely up to the plugin developer. Plugins are written in Mi-
croEngine C which is the standard C-like language provided
by Intel for use on MEs. The most important differences be-
tween MicroEngine C and ANSI C are dictated by the IXP
architecture. First, there is no dynamic memory allocation
or use because there is no OS or other entity to manage the
memory. Second, all program variables and tables must be
explicitly declared to reside in a particular type of memory
(registers, ME local memory, scratchpad, SRAM, DRAM)
as there is no caching. Finally, there is no stack and hence
no recursion. Also recall from the IXP review that the eight
hardware contexts share control explicitly (no preemption).
To help users who are unfamiliar with this programming
environment we have developed a framework that lowers the
entry barrier for writing simple to moderately complex plu-
gins. Our framework consists of a basic plugin structure
that handles tasks common to most plugins and a plugin
API that provides many functions that are useful for packet
processing.
In the basic plugin structure there are three different types
of tasks to which the eight threads are statically assigned at
plugin compile time. The first of these tasks deals with
packet handling. The framework takes care of pulling pack-
ets from the incoming ring buffer and then calls a user sup-
plied function to do the actual plugin processing. When
the function returns, the packet is pushed into the outgoing
ring. As can be seen in Figure 2, the packet can be sent
back to MUX which results in the packet being matched
against routes and filters in the TCAM a second time. This
is useful if something in the packet, such as the destination
IP address, has changed and the packet might need to be
re-routed. Alternatively, the plugin can send the packet di-
rectly to the QM so that it will be sent out to the external
links. Packets can also be redirected to the next plugin ME
via the next neighbor rings. In fact, although it is not shown
in the figure to avoid confusion, plugins even have the ability
24