TCP onloading for data center servers

Abstract

To meet the increasing networking needs of server workloads, servers are starting to offload packet processing to peripheral devices to achieve TCP/IP acceleration. Researchers at Intel Labs have experimented with alternative solutions that improve the server's ability to process TCP/IP packets efficiently and at very high rates.

Full text

Page 1

0018-9162/04/$20.00 © 2004 IEEE
46
Computer
C O V E R F E A T U R E
Published by the IEEE Computer Society
TCP Onloading
for Data Center
Servers
T
client’s request. As a result, overall application per-
formance has become more dependent on the effi-
ciency of data center communication.
The Transmission Control Protocol is the most
widely used protocol for reliable data communica-
tions both on the Internet and in data center net-
works. TCP, the transport layer in the four-layer
Internet protocol suite, always runs over the Internet
Protocol. IP, the network layer, provides routing ser-
vices between source and destination machines while
TCP provides guaranteed and orderly delivery of
data to applications running at each end of the
communication pipe. These two protocols are col-
lectively referred to as TCP/IP (TCP over IP). First
used in the mid-1980s, TCP/IP over Ethernet has
increased in popularity with the evolution of the
Internet and related applications such as Web brows-
ing, e-commerce, e-trading, and messaging.
Inside data centers and corporate local area net-
works, TCP/IP runs on top of Ethernet, which pro-
vides, among other things, framing services (mark-
ing packet boundaries) to upper layer protocols and
sends and receives bits on the wire. The de facto
protocol on LANs, Ethernet is also being consid-
ered for metro and wide area networks.
he Internet and the applications it enables
have transformed data center architectures
from a simple single-server model to a
multitier model in which Web servers,
application servers, databases, and stor-
age servers work together to respond to each
Until a few years ago, Ethernet speeds inside data
centers averaged 100 Mbps. However, increased
Internet usage created data center networking
demands that spurred the IEEE Ethernet Standards
Committee’s 802.3 subgroup to develop faster
Ethernet network technologies. This work led to
the development of 1 Gbps and 10 Gbps Ethernet
networks. Today, 1-Gbps Ethernet networks are
being widely deployed, and 10 Gbps will be widely
deployed as it becomes affordable. The sudden
jump in Ethernet speeds from 100 Mbps to 1 and
10 Gbps requires TCP/IP processing on the data
center servers to scale proportionately so that net-
work-intensive applications can ultimately benefit
from the increased network bandwidth levels.
TCP/IP stacks traditionally have been imple-
mented in software as part of the operating system
kernel. TCP/IP interacts with the network interface
controller (NIC) through a device driver and is
exposed to applications through the traditional
sockets interface. TCP/IP processing starts at the
NIC hardware and extends all the way to the
TCP/IP stack interactions with the application layer.
Our recent measurements on state-of-the-art plat-
forms show that TCP/IP processing of application
data—a full frame payload size of 1,460 bytes—
consumes one entire current generation CPU to
achieve roughly 750 Mbps of throughput while
receiving data and around 1 Gbps of throughput
while transmitting data. Clearly, it currently is not
possible to achieve a tenfold increase in TCP/IP
To meet the increasing networking needs of server workloads, servers
are starting to offload packet processing to peripheral devices to achieve
TCP/IP acceleration. Researchers at Intel Labs are experimenting with
alternative solutions that improve the server’s ability to process TCP/IP
packets efficiently and at very high rates.
Greg Regnier
Srihari
Makineni
Ramesh
Illikkal
Ravi Iyer
Dave
Minturn
Ram
Huggahalli
Don Newell
Linda Cline
Annie Foong
Intel

Page 2

November 2004
47
throughput. The major impediment to achieving
10-Gbps throughput is that CPU and memory
speeds are not expected to improve as dramatically
as Ethernet technologies.
While TCP/IP acceleration efforts focus on
enabling 10-Gbps communication rates, improv-
ing the efficiency at which this rate is achieved is
equally important. In a recent study of data center
servers,1we analyzed the current and future net-
working requirements of Web servers, front-end
servers, and database servers. This study showed
that TCP/IP receive-and-transmit overhead con-
sumed about one-third of the compute resources.
Another recent study2showed that the processing
overhead can be as high as 60 to 70 percent for Web
servers after adding other TCP/IP overheads like
connection setup.
To avoid the seriously limiting scalability issues
of these server workloads in the future, TCP/IP pro-
cessing efficiency must improve dramatically.
Further, as different types of applications—some
throughput-intensive, others latency-sensitive—
employ TCP/IP, accelerating and scaling TCP/IP
processing requires a balanced approach.
TCP/IP PROCESSING OVERVIEW
TCP/IP processing can be classified into three
major categories:
• connection processing,
• transmit-side processing, and
• receive-side processing.
Connection processing refers to the establish-
ment and teardown of TCP connections between
communicating end systems. Once the systems
establish a connection, data transfer occurs over
the connection through receive and transmit oper-
ations. Typically, the frequent processing paths in
data center servers are receive and transmit opera-
tions—also referred to collectively as data-path
processing.
Receive-side processing
Receive-side processing begins when the NIC
hardware receives an Ethernet frame from the net-
work. To extract the packet embedded inside the
frame, the NIC removes the frame delineation bits
and updates the descriptor data structure with the
packet information. The NIC driver software sup-
plies these descriptors, which are typically orga-
nized in circular rings, to the NIC. Through these
descriptors, the NIC driver informs the NIC of
information such as the memory buffer address in
which to store the incoming packet data. The NIC
uses a DMA operation to copy the incoming data
into this memory buffer. Once it places the packet
in memory, the NIC updates a status field inside the
descriptor to indicate to the driver that this descrip-
tor holds a valid packet. It then generates an inter-
rupt to initiate processing of the received packet.
Figure 1 shows the overall receive-side process-
ing flow. The NIC device driver reads the descrip-
tor and passes the header and payload memory
locations to the TCP/IP stack.
The next step is to identify the connection that
this packet belongs to. The TCP/IP software stores
each connection’s state information in the TCP/IP
control block (TCB) data structure. Since there can
be many open connections and thus many TCBs,
the TCP/IP stack uses a hashing mechanism to per-
form a fast lookup of the right TCB. The hash value
is calculated from the IP address and port number
of both the source and destination machines.
When the TCP/IP stack receives a new packet, it
updates several fields in the TCB—such as the
sequence numbers for received and acknowledged
bytes—and checks whether the application has pre-
posted buffers. If the application has already posted
a buffer to receive the incoming data, the TCP/IP
stack copies the incoming data directly from the
NIC buffer to the application buffer. Otherwise the
stack stores the data in a temporary buffer for later
delivery to the application. Performing a memory
Application
Socket
interface
TCP/IP
stack
TCB
header
buffer
Driver
Application
buffer
Socket buffer/
Application buffer
Payload
NIC
descriptors
Ethernet packet =
header + payload
Network
interface
controller
User
Kernel
Copy and
signal
Copy
DMA
NIC
NIC
Figure 1. Data flow in receive-side processing. Receive-side processing starts
with the NIC device driver reading the descriptor and ends with the TCP/IP stack
notifying the application that the data has been placed in its buffer.

Page 3

48
Computer
copy to move data from the NIC buffer to
the application buffer is one of the most time-
consuming operations in receive-side pro-
cessing.
Transmit-side processing
Transmit-side processing starts when an
application passes a data buffer to the TCP/IP
stack. The application passes a socket ID
along with the data buffer, and the TCP/IP
stack uses this socket ID to locate the con-
nection’s TCB. The TCP/IP stack can then
copy the application’s data into an internal
buffer. Many TCP/IP stacks employ optimizations
to avoid this copy. When the receiver’s window size
indicates it is time to transmit data, the TCP/IP
stack divides the accumulated data into maximum
transfer unit segments. An MTU is typically 1,460
bytes on Ethernet LANs. The stack then computes
the header: 20 bytes for TCP, assuming no options,
and 20 bytes for IPv4. The stack then appends these
segments with the IP headers and passes them down
to the NIC driver. The driver sets up the DMA to
transfer the headers and application data to the
NIC.
TCP/IP PROCESSING EVOLUTION
Over the years, applications with differing
requirements for latency sensitivity and high
throughput have adopted TCP/IP over varying link
speeds. To make TCP work best across these vari-
ous scenarios, researchers have investigated several
enhancements to the base protocol.3
Initially, the primary source of overhead in
TCP/IP processing was unclear, and developers
assumed that the base protocol processing was the
culprit. However, landmark research analyzing the
associated overheads showed that TCP/IP proto-
col-specific processing by itself is not the main over-
head.4The source for most of the overhead was the
environment—interrupts, OS scheduling, buffer-
ing and data movement—in which the TCP/IP pro-
tocol operates. While this dissuaded researchers
from replacing the TCP/IP protocol, overall pro-
cessing continued to be expensive.
To ease some of these overheads, researchers
developed several mechanisms that have become
common in today’s platforms and TCP/IP stacks:
• Interrupt coalescing. Also referred to as inter-
rupt moderation, this mechanism helps reduce
the overhead of using the interrupt mechanism
to signal the arrival of incoming packets from
the network. To amortize interrupt cost, NICs
can accumulate multiple packets and notify
the processor only once for several packets.
• Checksum offload. TCP and IP headers use
checksums to ensure that the packet data is not
corrupted during transmission. Checksum cal-
culation on a general-purpose processor is an
expensive operation, but TCP/IP stacks can
offload this feature to the NIC if it supports
checksum calculation. Since implementing a
checksum in hardware is relatively simple,
offloading it to the NIC hardware does not
add much complexity or cost.
• Large segment offload (LSO). Segmenting
large chunks of data into smaller segments and
computing TCP and IP header information for
each segment is expensive if done in software.
Most current-generation NICs support this
feature because doing this in hardware does
not add much complexity. This feature bene-
fits only transmit-side processing and is only
beneficial when the application wants to send
data larger than the maximum segment size
(MSS), which the connection’s two end points
negotiated during the TCP establishment phase.
When transmitting a 64-Kbyte application
payload, LSO can improve performance by up
to 50 percent.
Combined with constantly increasing CPU
and memory speeds, these enhancements have
enabled server platforms to achieve TCP/IP trans-
mit throughput that meets or exceeds current
Ethernet speeds (10-1000 Mbps) with reasonable
efficiency. However, the imminent need to achieve
10 Gbps for both receive and transmit processing.
SERVER PROCESSING CHALLENGES
Today’s server platforms face major challenges
while performing TCP/IP processing.
Figure 2 shows an example of the processing
breakdown for a TCP/IP transmit of 1-Kbyte pack-
ets. This measurement was taken on an Intel Xeon
processor-based server system running Linux OS
v2.4.18. In this example, OS integration and system
overhead—interrupt, syscall, driver, and bufmgt
components—constitute 50 percent of the overall
TCP/IP processing. This creates significant over-
head because the server’s TCP/IP stack must share
the CPU, memory, and I/O resources with the OS.
To do this, the TCP/IP stack must follow the OS’s
rules in terms of scheduling, resource management,
and interfaces.
OS mechanisms such as system calls, preemptive
scheduling, layered drivers, and interrupt process-
The imminent need
to achieve 10 Gbps
is forcing
researchers to
revisit TCP/IP
overheads and
challenges.

Page 4

November 2004
49
ing are all sources of overhead that limit TCP/IP
efficiency and performance on servers. Researchers
are investigating hybrid interrupt and polling mech-
anisms such as NAPI5to avoid the overheads and
harmful effects of interrupts.6
The second major challenge for TCP/IP process-
ing on servers is memory latency and poor cache
locality. Server memory and cache architectures are
optimized for efficient execution of a broad set of
applications. The TCP/IP software stack generally
exhibits poor cache locality largely because it deals
with large amounts of control information and data
that is entering or exiting the system through the
network interface. For example, once the NIC
receives packets and places them in memory, the
TCP/IP software must examine the control infor-
mation that includes the descriptor and packet
headers for each packet. Since the packet was just
placed in memory, access to this data will always
cause a cache miss and thus a CPU stall.
Incoming application data also exhibits no tem-
poral locality because the network interface places
the incoming data into memory and forces the CPU
to suffer several cache misses—that is, memory
accesses—to bring the data into the cache for pro-
cessing. I/O-intensive workloads cause numerous
CPU stalls due to cache misses.7Given the rapid
advances in network link technologies combined
with the growing disparity between processor
speeds and memory latency,8this is becoming a
greater problem over time.
Buffering and copying application data offers
the third major challenge in TCP/IP processing on
servers. The TCP/IP stack copies incoming data
into internal buffer space if the application has not
posted any buffers to receive the data. The stack
will copy the data again into the application buffer
when it is available. Thus, depending on the sce-
nario, there may be one or two copies of the
incoming data.
When transmitting data, there is an opportunity
to avoid a copy by transferring the data directly
from the application buffer to the NIC using DMA.
Zero copy on transmit has been achieved under
some implementations—for example, in-kernel
Web servers and fast-path overlapped I/O in the
Microsoft Windows OS. Implementations based
on traditional BSD sockets, however, still require a
copy between application and kernel buffers in the
general case. These data copies are expensive, espe-
cially when any of the source or destination buffers
are not in the processor’s cache.
While efforts have been made to eliminate copies,
the proposed techniques are not applicable in many
scenarios. Even if we assume that it somehow alle-
viates system overheads, the significant amount of
time spent on memory copies and other memory
stalls during TCP/IP receive-side processing can
limit the achievable throughput to much less than
10 Gbps.
TCP/IP ACCELERATION SOLUTION
To achieve efficient TCP/IP throughput of 10
Gbps and beyond on server systems, researchers
and industry projects have adopted an approach
that offloads the bulk of TCP/IP processing from
server processors and executes it on a peripheral
device. However, there is an ongoing debate about
whether offloading TCP/IP processing is the right
long-term solution.10
Offload or not?
TCP/IP offload engines (TOEs) generally offload
the TCP/IP processing onto a device that attaches
to the server’s I/O system and uses separate, spe-
cialized processing and memory resources (www.
alacritech.com/html/tech_review.html).
The main argument for TCP/IP offloading is that
it increases the server’s network throughput while
reducing CPU utilization. For some application sce-
narios, especially for bulk transfers involving few
connections, using a TOE can improve throughput
and utilization for applications such as IP-based
data storage.10
The arguments against offloading TCP/IP include
factors such as scalability, flexibility, extensibility,
and cost. In terms of performance, TOE devices
generally have limited processing power because
App
3%
TCP/IP
26%
Bufmgt
14%
Driver
20%
Syscall
7%
Interrupt
10%
Copy
12%
Other
8%
Idle
Figure 2. Profile of TCP/IP processing on servers. OS
integration and system overhead constitute 50 percent
of the overall TCP/IP processing.

Page 5

50
Computer
Moore’s law applies to high-volume, general-pur-
pose processors, and the processors in TOE devices
tend to lag behind. Because most TOE devices must
store and forward the payload data, the additional
latency tends to hamper performance for real-life
applications other than bulk data transfers.
Since TOE devices attach to the server platform’s
I/O subsystem, the latency to main memory is rel-
atively high compared to the main CPU. Because
of this additional latency, TOEs require local mem-
ory to store state and to buffer data. This local
memory adds latency and cost, and the amount of
memory constrains scalability in terms of how
many concurrent connections a TOE can support.
The alternative to offloading is to improve the
server’s inherent ability to process TCP/IP packets
efficiently and at very high rates, referred to as
onloading. General-purpose processors, with their
economies of scale, have the advantage of being
flexible, extensible, and scalable. Their program-
mability and the availability of a rich set of pro-
gramming tools make them extensible and flexible.
Extensibility makes it possible to add value to the
solution over time in terms of new features, proto-
cols, and applications, while flexibility lets the net-
working software adapt to changing standards and
modify its behavior in subtle but important ways.
Stack and NIC enhancements
Stack and NIC enhancements that are either
already available or likely to become available in
the near future to help server platforms achieve
more efficient network throughput include the fol-
lowing:
• Asynchronous I/O. This technique lets appli-
cations use a completion port for receiving
socket notifications. To reduce the number of
copies needed during processing, applications
can pre-post one or more buffers to send and
receive data from the network and wait for
completion messages to arrive in the comple-
tion port.
• Header splitting. Instead of placing the entire
packet in one location in memory, this tech-
nique lets the NIC place headers independently
from the data. In addition to allowing locality
of incoming headers, this soon to be available
technique facilitates better prefetching schemes.
• Receive-side scaling. This approach allows
multiprocessor systems to better process net-
work traffic on multiple CPUs (www.microsoft.
com/whdc/device/network/NDIS_RSS.mspx).
It accomplishes receiver-side scaling by pro-
viding multiple receive queues in the NIC and
letting them be mapped to different processors
in the system. This allows for scalability
through connection-level parallelism and affin-
ity. We anticipate that NIC vendors will em-
ploy this feature in future products.
Cache/Memory
Core 0
Application
Core 1
Application
Core 2
Application
Core 3
TCP/IP
onload
processing
NIC device
CMP cores
TCP offload
TCP
onload to
CPU core
Figure 3. Onloading TCP/IP in a CMP server architecture. Using one or more of
the cores efficiently for TCP/IP processing is preferable to offloading the
processing to yet another compute element in a peripheral device.
Host
Shared
memory
PPE
DTI
Storage
stack
DTI
Kernel
apps
User
apps
DTI
Networking Storage
Interprocess
communication
Server platform
Ethernet fabric
Figure 4. ETA architecture. The architecture partitions the system between the
host and the packet-processing engine. The direct transport interface supports
direct commands for establishing socket connections—connect, listen, accept.
The interface also supports anonymous buffer pools for out-of-order packets and
for incoming data without a corresponding posted receive buffer.