A modeling approach for estimating execution time of long-running scientific applications

Abstract

In a grid computing environment, resources are shared among a large number of applications. Brokers and schedulers find matching resources and schedule the execution of the applications by monitoring dynamic resource availability and employing policies such as first- come-first-served and back-filling. To support applications with timeliness requirements in such an environment, brokering and scheduling algorithms must address an additional problem - they must be able to estimate the execution time of the application on the currently available resources. In this paper, we present a modeling approach to estimating the execution time of long-running scientific applications. The modeling approach we propose is generic; models can be constructed by merely observing the application execution "externally" without using intrusive techniques such as code inspection or instrumentation. The model is cross-platform; it enables prediction without the need for the application to be profiled first on the target hardware. To show the feasibility and effectiveness of this approach, we developed a resource usage model that estimates the execution time of a weather forecasting application in a multi-cluster grid computing environment. We validated the model through extensive benchmarking and profiling experiments and observed prediction errors that were within 10% of the measured values. Based on our initial experience, we believe that our approach can be used to model the execution time of other time-sensitive scientific applications; thereby, enabling the development of more intelligent brokering and scheduling algorithms.

Full text

Page 1

A Modeling Approach for Estimating Execution Time of
Long-Running Scientific Applications

Seyed Masoud Sadjadi1, Shu Shimizu2, Javier Figueroa1,3, Raju Rangaswami1, Javier Delgado1,
Hector Duran4, Xabriel J. Collazo-Mojica 5
1: Florida International University (FIU), Miami, Florida, USA; 2: IBM Tokyo Research Laboratory,
Tokyo, Japan; 3: University of Miami, Coral Gables, Florida, USA; 4: University of Guadalajara,
CUCEA, Mexico; 5: University of Puerto Rico, Mayaguez Campus, Puerto Rico;

Abstract

In a Grid computing environment, resources are
shared among a large number of applications. Brokers
and schedulers find matching resources and schedule the
execution of the applications by monitoring dynamic
resource availability and employing policies such as first-
come-first-served and back-filling.
applications with timeliness requirements in such an
environment, brokering and scheduling algorithms must
address an additional problem - they must be able to
estimate the execution time of the application on the
currently available resources. In this paper, we present a
modeling approach to estimating the execution time of
long-running scientific applications. The modeling
approach we propose is generic; models can be
constructed by merely observing the application
execution “externally” without using intrusive techniques
such as code inspection or instrumentation. The model is
cross-platform; it enables prediction without the need for
the application to be profiled first on the target hardware.
To show the feasibility and effectiveness of this approach,
we developed a resource usage model that estimates the
execution time of a weather forecasting application in a
multi-cluster Grid computing environment. We validated
the model through extensive benchmarking and profiling
experiments and observed prediction errors that were
within 10% of the measured values. Based on our initial
experience, we believe that our approach can be used to
model the execution time of other time-sensitive scientific
applications; thereby, enabling the development of more
intelligent brokering and scheduling algorithms.

1. Introduction

A Grid computing environment provides a shared
resource infrastructure for a large number of applications.
Entities such as brokers and schedulers decide how
resources get partitioned among the set of applications.
Typical policies employed by these entities include back-
filling, first-come-first-served, etc. and are based on
dynamically monitoring application resource usage
behavior. While such policies work reasonably well to
balance application resource requirements across physical
To support
resources and ensure high resource utilization, they fall
short when the scheduling task involves meeting task-
specific execution deadlines. To address timeliness
requirements, brokers and schedulers must be able to
estimate the execution time of the application on the
currently available resources in order to ensure that
scheduling decisions do not lead to deadline violations.
Extensive research exists in the area of resource usage
and execution time prediction. A large section of this
work focuses on platform-specific approaches [1,2,3,4,5],
which does not support resource usage prediction on
previously "unseen" target execution environments and
very few of these approaches address performance
prediction across different hardware configurations. This
is important since the set of available resources on the
Grid could have an arbitrary configuration. Some
approaches that do provide support for cross-platform
prediction of resource usage such as [6] and [7] are either
application specific or restricted to predicting for single-
node application executions alone. Other approaches are
intrusive in that they call for application source or binary
instrumentation.
Our approach differs from the other works in several
respects. First, our approach is application agnostic and
does not require application source or binary code
inspection or instrumentation. Second, unlike some
previous approaches, our approach does not require a
sample execution on the target platform before prediction.
Third, our approach is able to model execution scale,
thereby also addressing
especially, it allows prediction in a multi-cluster Grid
computing environment.
A key contribution of our approach is the generality of
our application resource usage model, which is a direct
result of constructing the model by merely observing the
application execution "externally" (i.e., observing its
resource usage rather than inspecting or instrumenting its
code). This allows a system design and implementation
that is completely oblivious of the semantics of the target
application. The advantage of such an application-
agnostic approach is more appreciated when application
semantics are complex, the source-code is not available,
or there is no documentation. The WRF (Weather
Research and Forecasting) application is an example of a
distributed applications;

Page 2

time-sensitive, resource-intensive, distributed application
with a complex codebase and little to no documentation.1
We use it to demonstrate the effectiveness of our approach
through the rest of the paper. Once a cross-platform
application model is built, predicting the execution time
on a different hardware configuration simply involves
populating the correct values for individual resources of
the new platform.
The proposed model estimates the application
execution time based on application resource usage
behavior. We note that the application execution time may
be dependent on the specific configuration of several
resource types. We started with a simple first step
assumption that the influence of a single resource on the
execution time of the application is independent from the
influences of the other resources. We concede that this
assumption may not be valid and some resource
correlations may influence application execution time and
intend to relax this assumption in future extensions of this
work. We shall demonstrate, even with the simplistic
assumption, that the model we developed is fairly
extensive, capable of incorporating multiple resources and
multi-node application executions.
The effectiveness of our approach was tested by
developing a model for predicting the execution time of
WRF in a multi-cluster environment. Several experiments
were done to validate the model. An error rate of less than
10% was observed, which leads us to believe that the
model is a viable option for developing timeliness
enhancements to brokering and scheduling applications
that take into account the dynamic resource availability of
target execution environments. It is worth pointing out
that we have applied the model proposed in this paper to
other applications with substantially different resource
consumption characteristics and have found it to be
effective [8].
The rest of this paper is organized as follows. In
Section 2, we describe the general approach taken by our
model, including the parameters we are modeling. In
Section 3, we introduce the mathematical model being
evaluated. Section 4 describes the software artifacts that
we developed for implementing our modeling and
prediction mechanisms. This section also describes how
this software was used to carry out our experiments to
evaluate the model. Section 5 shows the results obtained
from the experiments. Section 6 describes how the model
was validated, based on the accuracy of the results
obtained. Section 7 provides a more in-depth look at
related research (compared to that which has already been
covered above). Finally, in Section 8, we summarize the
paper and provide some future research directions.

1 Several of such applications exist in the domain of scientific
computing and elsewhere. Ironically, these applications are also
often mission-critical.
2. Approach to Modeling Resource Usage

Modeling the resource usage of a distributed
application must take into account several aspects of the
computational environment. In this section we overview
our approach to modeling the static resource properties of
the target execution platform and execution-specific
factors affecting resource usage such as the degree of
parallelism. Model construction is formally addressed in
Section 3.
To construct our model, we build upon our initial work
on modeling single-node application execution [ 8 ].
Following the philosophy of the original approach, we
construct the model to be application-agnostic (as
opposed to application-specific). While profiling using
source-code instrumentation (an application-specific
approach) can provide valuable insight into application
behavior that is typically unavailable with external
observation, our application-agnostic approach provides
generality in the modeling, profiling, and prediction
mechanisms. Consequently, this enables a system design
and implementation that is completely oblivious of the
semantics of the target application. The latter is especially
important when application semantics are either complex,
or are unknown, or if the source-code is unavailable.

2.1 Modeling the Resources

Resource properties of the execution platform fall into
the three basic categories of computation, communication,
and storage. The key computational resources that affect
execution time include the CPU clock speed, L2 cache
size, and the front-side-bus
Communication parameters include the maximum
bandwidth and latency of the network interconnection,
while storage parameters include the main memory size
and memory access bandwidth, as well as sequential and
random disk I/O bandwidths.
While the above set of parameters may seem excessive,
we point out that the significant parameters of the model
that affect the performance of a specific application is
typically a subset. Our modeling technique automatically
identifies these parameters; the remaining parameters are
typically eliminated from the model with sufficient
evidence. In the rest of this paper, however, we restrict
our WRF model to a subset of computation resources by
modeling the CPU clock speed and the number of nodes,
and ignore the influence of other resource properties for
the sake of simplicity of exposition.

2.2 Modeling Execution Parallelism

When modeling parallel and distributed applications
(as is typical for scientific applications), the two critical
parameters to address are platform heterogeneity and
execution scale.
(FSB) bandwidth.

Page 3

In our current model, we assume that all the nodes in
the target execution environment are identical, i.e. that
they have identical individual resource characteristics of
computation, communication, and storage. While we
realize that this may not apply to all distributed
environments, it is effective in addressing typical cluster
environments. More importantly, it allows us to construct
a practically usable model of the system. Relaxing this
assumption to accommodate heterogeneous clusters is an
important direction for future work.
We address execution scale by including in our model
a parameter associated with the number of processors
utilized during execution. For simplicity, our model
currently makes no distinction between shared and
distributed memory processors (e.g., SMP and Cluster).

2.3 Modeling Input Parameters

Accurate input parameter modeling requires knowledge
of application semantics. In the spirit of our application-
agnostic approach, we simplify the modeling of input
parameters by reducing input parameter modeling to a
load specification task for the developer. The load is
interpreted as linear, with higher values representing a
greater input load on the application.
In case load specification is infeasible (due to the
complexity of the application input data semantics), we
consider the execution of an application with a different
input data set as a “new application”, which must then
undergo independent profiling and modeling.

3. Application Resource Usage Model

In this Section, we present a model of application
execution correlating with application resource usage
characteristics. In constructing our model, we note that
applications may utilize different types of resources (as
elaborated in Section 2) and also demonstrate varied
patterns of resource usage during their execution.
Consequently, the execution time of each application is
typically dependent on different sets of resources. To take
this into account, we create application profiles to capture
and predict execution time for a specific application. As a
first step assumption, we suppose that the influence of the
resource properties of the target execution environment to
the application execution time takes a product form, in
which each term represents the influence of one or more
resource properties and is independent of other terms. It is
therefore represented, as follows:
−
=
i exec
CT
,
∏
=
i
1
0
m
(1)
where
exec
T
is the execution time,
i C is the i-th
contribution by one or more resources, and m is the
number of contribution terms. The term
i C may contain
resource parameters such as CPU clock frequency, L2
cache size, FSB bandwidth and disk I/O bandwidth, as
mentioned in Section 2.
In this paper, we consider and focus on two types of
independent contributions: the parallelism of task
execution and CPU’s performance factor, which is
currently just it’s clock frequency.2 In considering parallel
task execution on more than one node, we further assume
that all nodes are homogeneous in terms of their resource
properties.3 Further, the model more naturally addresses
applications whose resource consumptions are more or
less consistent across time. If this is not the case, the
model would still identify all the dominant resource usage
factors, but would however be unable to capture the
dynamics of such usage over the duration of the
application execution. We use the following simple form
of the parallelism term, which reasonably assumes that the
execution time is in inverse proportion to the degree of
parallelism, or
PC
100
αα +=
para
, (2)
where
para
P
is the degree of parallelism such as the
number of processors available to the application,
0
α and
1
α indicate the application’s characteristics and will
differ for each application. The first term
0
α indicates the
constant contribution such as execution overhead.
To include the CPU performance contribution, we use
the following simple form:
PC
101
ββ +=
P
is the CPU clock frequency,
clock
, (3)
where
clock
0
β and
1 β
indicate the application’s characteristics related to the
CPU performance as well as Equation (2).
After expanding the product form of Equation (1), we
have a simple summation form, as follows:


(
11
βα
+
(4)

which is a linear form of explanatory variables (basic
terms for regression analysis) multiplied by application

2 Please note that while a more exhaustive model would
consider additional resource properties (e.g., memory size) or
more complex dependencies (e.g., quadratic instead of linear),
our primary goal in this paper is to introduce our general
technique for resource usage modeling and prediction.
Therefore, we chose a minimal model to simplify presentation.
We minimize the effect of these parameters by making them
invariable in all experiments.
3 While some cluster environments would consist of a set of
machines of almost similar configuration, there may be cases
where this assumption may not hold. Relaxing this assumption,
however, substantially complicates the modeling mechanism.
)
clock
P
para
clock
P
para
P
exec
PT
100100
βαβαβα
++=

Page 4

profile parameters that define application characteristics.
We can denote it by its following general representation:
∑
=
i
1
+=
n
iiexec
yT
0
θθ
(5)
iθ is the i-th application profile parameter which are
values reflecting constant contributions related to a
particular application and type of resource and
iy is the i-
th explanatory variable which is a function of static
resource properties (e.g. clock speed) and the resource
competition status.
Equation (5) thus reduces the prediction task to a
problem of estimating profile parameters. We apply
regression analysis to solve the problem. While the
general technique we use, including the error analysis, is
detailed in [8], we summarize key steps of the estimation
process here. The observation at time t=k includes the
[k
x



After obtaining N sets of observations, Equation (5) is
now represented with an error term, as follows:


where





and ε indicates errors with zero mean. The error term
may include observation errors and model inaccuracy. We
then apply regression analysis to minimize the mean
square errors of ε to estimate the application profile
parameters, as follows:



Note that the matrixes H and x grow in size for more
monitored execution time
]
on the static resource set:
]
n
y
L
(6)
(7)
(8)
observations, however, the matrixes
are of the fixed size of
HHT
) 1
+
and
(
n
x
1
HT
) 1
+

() 1
+
(
×
nn
and
×
,
respectively. As a result, we do not need additional
storage to maintain the observations in general. It should
be also noted that the matrix
inverse matrix to obtain a solution to the above equation.
Thus, at least n+1 sets of independent observation data
are required for realizing a model of the application. For
reliable estimation, more than n+1 observation must be
maintained so as to decrease errors of observation.


HHT
must have a valid
4. Monitoring and Prediction

For profiling applications, we have developed two
software programs: (1) a monitoring program (called
amon) that runs on each compute node and reports the
execution time observed for any application with
appropriate values of explanatory variables every time it
detects completion of the application’s execution; and (2)
a prediction program (called aprof) that runs on a server
node and receives the reports from the monitoring
programs distributed over the compute nodes. It then
maintains the fixed-size matrixes
each application in an incremental manner. For example,
HHT
is updated, as follows:
HHT
and
xHT
for
each element
ij a of


[k

a

ij

0

≤
(9)
where
]
ij a
is the element value (
nji
≤ ,
) at time
t=k,
0
] 0 [
=
ij a
at t=0, and
1
][
0
=
k
y
. Elements of
xHT

are also calculated in an incremental manner without
increasing the data size in the profiling program.
After the profiling program receives sufficient number
of independent observation results, that is, at least n+1
sets of data, it can estimate and calculate the profile
parameters by using Equation (8).
Once the profile parameters for a certain application
are estimated, then we can apply them to predict an
execution time for a certain resource, or



(10)
where
iy is the value of the i-th explanatory variable in
the given resource set and
iθˆ is the i-th estimated profile
parameter for the application. Thus, the prediction
program not only estimates application profile parameters
but also applies them to predict an execution time based
on an explanatory variable set (
],,[
1
n yy K
).
The amon and aprof were designed to run in a
networked environment using a client/server architecture.
We run one instance of aprof on the head node of the
cluster, which is being profiled, to act as the server and
run one instance of amon on each compute node of the
cluster to act as clients. Due to our need to execute tests
for a number of different nodes and for a range of CPU
speeds (as will be explained in Section 5), it was
necessary to automate the experiment executions. Various
perl, bourne shell, and python based scripts were created
to execute these simulations automatically. The basic
functionality of these scripts is to gather the results in the
form of amon output and turn them into aprof compatible
prediction input to later compute the predictions that will
prove the accuracy of our model.
][
j
][
i
] 1
−
[][
ij
:
kkkk
yya
+=
[][]
T
nnexec
T
ˆ
yy
θ
ˆ
θ
ˆ
θ
ˆ
1
101
LL
=
ε
Hθ
x
+=
()
xHHH
θ
ˆ
TT
1
−
=
[
1
][][
1
][
kkk
y
=
y
[
[
[
θ
]
]
]
T
n
T
N
T
N
xx
θ
L
L
L
0
] 1
−
[] 0 [
y
] 1
−
[] 0 [
=
=
=
θ
yH
x

Page 5

5. Experiments and Results

Theoretically, by adding more processors to a
simulation one could assume that a scientific application
could run faster. This assumption is challenged by the
existence of a predicted point on the performance curve
where the running time of the application will worsen as
more processors are added to the system (i.e. the
saturation point). Several computational variables (e.g.
clock speed of the processor, processor load, number of
compute nodes, memory, network latency, network
bandwidth, communication overhead) determine the
execution time. In this section, we present a number of
experiments that help us analyze the behavior of WRF by
modifying two of the resource parameters, namely,
processor load and number of compute nodes.
To limit the number of concurrent changing variables,
we assume that the execution time of a forecast simulation
is based only on changes in clock speed and number of
compute nodes. For this, we have kept the effect of other
parameters such as memory minimized by making sure
that the value of the parameter either does not change or is
above an upper bound on the observed and target
platforms. The experiments were conducted using two
compute clusters located at Florida International
University. The first cluster is called GCB. This cluster is
based on the NPACI Rocks Linux distribution for
compute clusters, version 4.0. The cluster contains 8
nodes where each node contains two 32-bit x86 Intel
based CPUs, 1GB of main memory and uses a gigabit
network connection. The second cluster is called Mind.
Mind is also operating on Rocks version 4.0. The cluster
consists of 15 nodes, each containing dual Xeon 3.6GHz
processors and 2GB of main memory. They are also
connected through a gigabit network connection.
The WRF application running in a cluster environment
is capable of distributing the domain data points of a
forecast simulation into optimal size; this is called optimal
domain decomposition and is executed by WRF
communication RSL [9] layer. The code uses MPI [10]
communication subroutines for inter node communication,
and OpenMP [ 11 ] for intra node, inter processor
communication. These subroutines enable the program to
distribute the data and computational load among the
nodes of the cluster and the processors inside each node,
respectively. For the purpose of these experiments we
have used a small domain configuration to minimize the
size of our cases to match the size of our clusters’
computational power, while the effect of other resources
(e.g., memory size) will be minimal. As suggested by
meteorologists, we used a 75 by 75 domain decomposition
with 4km resolution, which contains 5625 grid points.
The approach to the experiments was to benchmark the
clusters by running experiments on different numbers of
available compute nodes as well as different effective
processing power. We employed three tools to assist us in
our experiments. To limit the effective clock speed of the
compute nodes, the open source CPUlimit [12] tool was
used. The other two tools, mentioned in Section 4, are,
amon and aprof. Amon was used to output run-related
resource-consumption statistics of WRF simulation
processes. Aprof was used for both receiving resource-
usage characteristics of the WRF simulations and
predicting the execution time for each of the simulations,
based on amon output. The percentage values used for the
CPU bindings were 100 (full utilization), 80, 60, 40, 30,
20, and 10 percent. All possible number of compute node
utilized (i.e. from one to seven in GCB and from one to 15
in Mind). For example, if the experiment being conducted
was 80-percent CPU bound then CPUlimit was run in
every node that was part of the experiment using 80 as the
“limit” parameter and wrf_arw_DM.exe as targeted
process. The following figures show the results.


Fig. 1. The execution times of WRF on GCB.

Fig. 2. The execution times of WRF on Mind.
Fig. 1 and Fig. 2 illustrate the curve obtained from
limiting the processor power for the same WRF forecast
simulation for different combinations of compute nodes
and clock speeds on GCB and Mind, respectively. The
performance decreases with less CPU power, but not
linearly. Fig. 3 illustrates the relationship between the