Transparent Optimization of Grid Server Selection With Real-Time Passive Network Measurements

Abstract

Grid services have tremendously simplified the programming challenges in leveraging large-scale distributed computing. At the same time, the increased level of abstraction reduces the opportunities available to the application for optimizing its performance by monitoring the system. In this paper we introduce a monitoring grid services proxy, which transparently monitors network performance and selects between several replica service providers. This approach provides optimized server selection without any modification to or even awareness of the client application or service providers. We describe how we implement the proxy and monitor the available bandwidth to the service providers using the Wren monitoring toolkit. We present analysis indicating that our monitoring has negligible overhead. Finally, we demonstrate the practicality of our approach by optimizing the server selection for INCOGEN's VIBE, a bioinformatics workflow application that uploads gene sequences for analysis by remote service providers.

Full text

Page 1

College of William & Mary
Computer Science Department
Technical Report
WM-CS-2006-04
June 9, 2006
Transparent Optimization of Grid Server Selection With Real-Time Passive
Network Measurements
Marcia ZangrilliBruce B. Lowekamp
Abstract
Grid services have tremendously simplified the programming challenges in leveraging large-scale dis-
tributed computing. At the same time, the increased level of abstraction reduces the opportunities available
to the application for optimizing its performance by monitoring the system. In this paper we introduce
a monitoring grid services proxy, which transparently monitors network performance and selects between
several replica service providers. This approach provides optimized server selection without any modifica-
tion to or even awareness of the client application or service providers. We describe how we implement the
proxy and monitor the available bandwidth to the service providers using the Wren monitoring toolkit. We
present analysis indicating that our monitoring has negligible overhead. Finally, we demonstrate the practi-
cality of our approach by optimizing the server selection for INCOGEN’s VIBE, a bioinformatics workflow
application that uploads gene sequences for analysis by remote service providers.
We would like to thank INCOGEN for allowing us to use their VIBE software toolkit. In particular, we
would like to acknowledge the support provided by Jennifer Lowekamp and Dawn Cannan at INCOGEN.
This work was performed, in part, using computational facilities at the College of William and Mary which
were enabled by grants from Sun Microsystems, the National Science Foundation, and Virginia’s Common-
wealth Technology Research Fund.

Page 2

Transparent Optimization of Grid Server Selection With Real-Time Passive
Network Measurements
Marcia Zangrilli and Bruce B. Lowekamp
College of William and Mary
Williamsburg VA, USA
{mazang,lowekamp}@cs.wm.edu
Abstract
Grid services have tremendouslysimplified the program-
ming challenges in leveraging large-scale distributed com-
puting. At the same time, the increased level of abstraction
reduces the opportunities available to the application for
optimizingitsperformancebymonitoringthesystem. Inthis
paper we introduce a monitoring grid services proxy, which
transparently monitors network performance and selects
between several replica service providers. This approach
provides optimized server selection without any modifica-
tiontoor evenawarenessoftheclientapplicationorservice
providers. We describe how we implement the proxy and
monitor the available bandwidth to the service providers
usingtheWrenmonitoringtoolkit. We presentanalysisindi-
catingthatourmonitoringhasnegligibleoverhead. Finally,
we demonstratethe practicalityof ourapproachby optimiz-
ing the server selection for INCOGEN’s VIBE, a bioinfor-
matics workflow application that uploads gene sequences
for analysis by remote service providers.
1. Introduction
One of the challenges in developing grid applications
is the difficulty in creating applications that can function
across both high latency networks and tightly coupled clus-
ters. Grid services have emerged as a means to help the
coordination of grid applications by providing a standard
interface between clients and services. Grid services help
reduce the programmingcomplexities for clients and create
opportunities to help steer the decision of which service is
best for the client to invoke.
In this paper, we explore the use of a grid service aware
proxy server to providetransparent optimization services to
grid services applications. The proxy service offers a new
level of abstraction that hides the exact data or service re-
sourcefromthe client. Insuch a system, the client is config-
uredwith the locationof the proxyas the address for all ser-
vices, but it is totally unaware of the behavior of the proxy
and all optimization decisions are transparent to both the
service provider and the client.
While adding proxies to grid services architectures may
remove some control from the client, our goal is to provide
the middleware with sufficient power to make the appropri-
ate performance optimizations without the need to involve
either the user or the client application unless qualitative
decisions–such as choosing between databases managed by
different groups–need to be made.
Network performance monitoring is critical for running
applications in complex grid environments, but obtaining
this information often requires the proxy to actively probe
the network for available bandwidth. The active approach
often produces accurate measurements, but it may cause
competition between application traffic and the measure-
menttraffic,reducingtheperformanceofotherapplications.
Most of these active algorithms rely on UDP traffic to probe
the path for available bandwidth, whereas most grid appli-
cations use TCP traffic. If UDP traffic is packet-shaped dif-
ferently than TCP traffic, measurements made using UDP
trafficmaynotreflect theactualbandwidthavailabletoTCP
applications. For available bandwidth measurements to be
useful to grid services proxies, they must be both accurate
and non invasive.
Our solution is to incorporate passive Wren measure-
mentsintoagridservicesproxytohelpfacilitatetheproxy’s
decision of which service to invoke. Wren is less invasive
thanactiveprobingbecauseit uses passive traces ofexisting
application traffic to provide available bandwidth measure-
ments. Because Wren measures available bandwidth using
the application’s own traffic, the measurements providedby
Wren will reflect the bandwidth available to that applica-
tion.
Wren will allow the proxy server to choose between dif-
ferent options for executing the same service, e.g., whether
to process data locally on a single server or to spend the
time uploading the dataset to a remote high-performance
cluster that offers the same service. These features will be
used to hide the complex details of service and data loca-
tion from the client application, while still exposingoptions
of different services and datasets to the client and the user.

Page 3

Furthermore, Wren measurements enable the proxy to pro-
videfeedbacktotheclient onhowlongeachservicemodule
will take to transfer data or client requests.
The principle contribution of our work is the implemen-
tation and analysis of a system that uses real-time mea-
surements to make runtime server selection decisions with-
out modification of either the client application or service
provider’s code or infrastructure.
In this paper we describe the online Wren system and
the integration of Wren with a simple grid services proxy.
We have previously described the preliminary design and
implementation of Wren [14,15]. This paper significantly
extends our previous work by providing:
• Introduction and analysis of the online Wren system,
completing the circle by providing real-time measure-
ments back to the application for performance opti-
mization,
• Application of passive monitoring to a server selec-
tionproxythatfunctionsinagridservicesenvironment
without modification of any other component,
• Experimental results that demonstrate the accuracy of
Wren measurements using grid service requests,
• Quantification of lag associated with providing real-
time Wren measurements, and
• Demonstration that Wren measurements can aid the
proxy in server selection.
In Section 2 we review related work on grid services and
define available bandwidth. Section 3 describes the imple-
mentationof the Wren monitoringsystem and shows the ef-
ficiency of Wren tracing. Section 4 analyzes the overheads
of tracing proxy traffic, the accuracyof measuringavailable
bandwidth, and the measurement lag associated with Wren
measurements. Finally, Section 5 describes howWren mea-
surements can be used to by a grid service aware proxy to
improve client performance.
2. Background
2.1. Grid Services
As an alternative to the complex challenge of program-
mingdistributedgridresources,the area ofgridservices has
gainedgroundasameansofprovidingresearcherswithpro-
grammaticaccess to distributedcomputationalresources[1,
13]. In many fields the same applications are used repeat-
edly (e.g. BLAST in genomics or charm++ in biochem-
istry). Grid services that execute a known application on
user-submitted data provide users with powerful resources
andhelp reducethe programmingcomplexityof developing
grid applications.
Grid services are at the core of the effort to provide
seamless interoperation among resources in distributed sys-
tems. Grid services are fundamentally Web Services that
provide specific functionality. The OGSI defines Grid Ser-
vices with extensions to the standard Web Service Defini-
tion Language (WSDL) to provide: sophisticated security
infrastructure; standard standard service invocation mech-
anism for service lifetime management; state management
and modeling of time; and standard service inquiry mecha-
nism [13].
The OGSA Execution Management Services (OGSA-
EMS) [1] are a set of components that are collectively re-
sponsible for identifying service locations, selecting the
most appropriate location, and running the service. EMS
itself comprises multiple services that achieve these goals
and are designed in a composablefashion such that services
can be connected as required. They leverage information
provided through a Grid Monitoring Architecture to pro-
vide their services. In our case, the services provided by
our proxy are most similar to an OGSA Job Manager. By
combiningthe Wren monitoringtechniques with a Message
Broker that is used to deliver messages between compo-
nents, the same levelof transparent,passive networkperfor-
mance monitoring could be achieved, and that information
could be used to make resource mapping decisions in com-
bination with other information available. However, as our
current applications are not based on an OGSA-compliant
library, we have not yet pursued this option.
2.2. Available Bandwidth
Available bandwidth describes what portion of the path
is currently unused by other competing traffic. More pre-
cisely, available bandwidthis determinedby subtractingthe
utilization from the capacity of the network path [6,10].
Available bandwidth is useful for applications that are con-
sidering increasing their utilization, for a new application
approximating what bandwidth might be available, and for
network engineers monitoring the performanceof their net-
work.
One technique for measuring available bandwidth is the
self-induced congestion(SIC) technique. The basic princi-
ple of SIC is that if packets are sent at a rate larger than
the available bandwidth,the queuingdelays will have an in-
creasing trend, and the rate the packets arrive at the receiver
will be less than the sending rate. If the one-way delays are
not increasing and the rate the packets arrive is the same
as the sending rate of the packets, then the available band-
width is at least equal to the sending rate. Tools that utilize
this concept actively probe the network path for the largest
sending rate that does not result in queuing delays with an
2

Page 4

analysis
collector
provider
Network
Wren Packet Trace Facility
Wren user−level
Kernel level
User level
IP
Buffer
collect() init() recv()send()
UDPTCP
SOAP Interface
Proxy
Application
Figure 1. Integration of Wren packet trace
facility and Wren analysis to a grid ser-
vices proxy provides transparent monitoring
of network resources.
increasing trend because this sending rate reflects the avail-
able bandwidth of the path.
Tools that use principles similar to self-induced con-
gestion include Netest [5], PathChirp [11], Pathload [4],
PTR [2], and TOPP [9]. Unlike these tools, all of which ac-
tively inject traffic into the network, Wren passively applies
the self-inducedcongestionprincipleto existingapplication
traffic to measure available bandwidth.
A similar project to Wren is Inline measurement TCP
(ImTCP). ImTCP integrates an active self-induced conges-
tion algorithm into a TCP stack, waiting until the conges-
tion window has opened large enough to send an appropri-
ate length train and then delayingpacket transmissions until
enough packets are queued to generate a precisely spaced
train [7]. Wren avoids modifying the TCP sending algo-
rithm and, in particular, delaying packet transmission.
3. Online Wren Monitoring System
The Wren bandwidth monitoring system provides avail-
able bandwidth measurements by passively observing ex-
isting application traffic. Figure 1 shows an overview of
the Wren architecture. Wren is separated into a kernel-level
packet trace component and a user-level processing com-
ponent. The kernel-level packet trace facility is responsi-
ble for gathering information associated with incoming and
outgoing packets and the user-level is used to collect the
traces from the kernel. Run-time analysis determines avail-
able bandwidth and the measurements are reported to other
applications through a SOAP interface. Alternatively, the
packet traces can be transmitted to a remote repository.
In the remainder of this section we will describe the
kernel-level Wren packet trace facility, the efficiency of
Wren tracing, and the processing done by the Wren user-
level component, with an emphasis on the Wren available
bandwidth algorithm.
3.1. Wren Packet Trace Facility
We use the Wren packet trace facility to collect traces
of TCP traffic sent between the grid services proxy and the
servers accessed by the clients. Wren timestamps packets
in the kernel as they arrive or leave, collects other protocol
specific information, and passes the trace to the user-level
for analysis. Wren can capture full traces of small bursts of
trafficorperiodicallycapturesmallerportionsofcontinuous
traffic and use those to measure available bandwidth. For
high bandwidth-delay product networks, the tool can also
ensure that it collects data transmission and ACK receipt
logs of the same range of sequence numbers.
The key benefit of the Wren packet trace facility is its
use of kernel-level timestamps. Wren timestamps outgoing
packets in the kernel immediately before they are handed to
the layer-2 device driver and similarly timestamps incom-
ing packets as soon as they are passed to the TCP code.
The precision of the timestamps is crucial because our pas-
sive available bandwidth algorithm relies on observing the
behavior of small groups of packets on the network. This
additional accuracy, combined with the efficiency of col-
lecting only the information needed while reading from the
buffer at regular intervals, provides the traces required for
accurate monitoring without disturbing the running appli-
cation.
3.1.1Wren Tracing Efficiency
The principle alternative to a system like Wren is using
the Berkeley packet filter [8] to capture TCP traffic. The
most straightforward approach is to use tcpdump to capture
packet header traces and collect the data from the times-
tamps and packet headers. We have implemented this ap-
proach, as well as a direct libpcap implementation that ex-
tracts only the needed fields for later analysis.
We presentthe results ofexperimentingwith packet trac-
ing techniques between two 2.8GHz Hyperthreaded P4s
running Linux 2.4.29 with 1GB of RAM and an Intel CSA
(non-PCI) gigabit Ethernet NIC. The nodes were connected
using a Cisco 3750 gigabit switch. Here we analyze only
the cost of capturing packets and saving them to the hard-
drive for offline analysis.
At 100Mbps speeds, all three approaches can collect
100% of traffic at full speed with minimal affect on ap-
plication performance. Therefore, we focus our efficiency
3

Page 5

900
905
910
915
920
925
930
935
940
945
950
No Trace Wrentcpdumplibpcap
Throughput (Mb/s)
Trace Collection Technique
Throughput of iperf while collecting traces
Figure 2. Throughput achieved by iperf with
no packet trace collection, using Wren, using
tcpdump, and using libpcap
analysis on Gbps traffic. Figure 2 presents the throughput
achievedbyiperfwhile usingeach ofthe three capturetech-
niques, as well as without any packet capture. The Wren
approachachievesthe same throughputas the untracedcon-
nection; however the two packet-filter approaches both re-
duce throughput.
Most applications other than bulk data transfer perform
both computation and communication. To simulate this be-
havior, we repeated the same experiment while also run-
ning a simple 1000× 1000 dense matrix multiply. Figure 3
presents the execution time of the matrix multiply and Fig-
ure 4 presents the throughput achieved by iperf during the
matrix multiplication. Again, the Wren packet trace facility
clearly is the lowest overhead of the trace collection tools.
0
20
40
60
80
100
No TraceWrentcpdumplibpcap
Average Execution Time(s)
Trace Collection Technique
Execution Time of Matmult
Figure 3. Execution time of a 1000 × 1000 ma-
trix multiply while sending data using iperf
with no packet trace collection, using Wren,
using tcpdump, and using libpcap
0
100
200
300
400
500
600
700
800
900
No TraceWrentcpdumplibpcap
Throughput (Mb/s)
Trace Collection Technique
Throughput of iperf during Matmult
Figure 4. Throughput of iperf while running
a 1000 × 1000 matrix multiply with no packet
trace collection, using Wren, using tcpdump,
and using libpcap
3.2. Wren User-Level Analysis
Figure 1 shows that the Wren user-level is comprised
of the collector, analysis, and provider threads. The Wren
Analysis thread processes traces acquired from the collec-
tor thread by parsing the traces into separate connections
andthenapplyingWren’spassiveavailablebandwidthalgo-
rithm to packets obtained for each connection. The parsing
step also merges information about incoming and outgoing
packets to form a complete packet picture for the Wren al-
gorithm. The analysis thread passes Wren measurements to
the provider thread that makes the measurements available
to all interested applications through a SOAP interface.
The onlineWren algorithmgroupsoutgoingpackets into
trainsbyidentifyingsequencesofpacketswithsimilarinter-
departuretimes between successivepairs. The toolsearches
for maximal-length trains with consistently spaced packets
and calculates the initial sending rate (ISR) for those trains.
After identifying a train, we calculate the ACK return rate
for the matching ACKs. The available bandwidth is deter-
mined by observing the ISR at which the ACKs show an
increasing trend in the RTTs, indicating congestion on the
path.
Thefirststepinourone-sidedalgorithmistogrouppack-
ets intotrains. We lookatthe relationshipbetweentheinter-
departure times of sequential data packets. If interdeparture
timesofsuccessivepairsaresimilar,thenthepacketsarede-
parting the machine at approximatelythe same rate. Let ∆0
be the interdeparture time between data packets 0 and 1. To
form a train, the interdeparture times ∆ibetween each suc-
cessivepairofpacketsiandi+1inthetrainmustsatisfythe
requirementthat mini(log(∆i)) > maxj(log(∆j))−α, es-
sentially requiring consistent spacing between the packets.
4