Flexible Skeletal Programming with eSkel.

Full text

Page 1

Flexible Skeletal Programming with eSkel
Anne Benoit, Murray Cole, Stephen Gilmore, and Jane Hillston
School of Informatics, The University of Edinburgh, James Clerk Maxwell Building
The King’s Buildings, Mayfield Road, Edinburgh EH9 3JZ, UK
enhancers@inf.ed.ac.uk
http://homepages.inf.ed.ac.uk/mic/eSkel
Abstract. We present an overview of eSkel, a library for skeletal parallel
programming. eSkel aims to maximise the conceptual flexibility afforded
by its component skeletons and to facilitate dynamic selection of skeleton
compositions. We present simple examples which illustrate these proper-
ties, and discuss the implementation challenges which the model poses.
1Introduction
The skeletal approach to parallel programming is well documented in the re-
search literature (see [1–4] for surveys). It observes that many parallel algorithms
can be characterised and classified by their adherence to one or more of a num-
ber of generic patterns of computation and interaction. For instance, a variety
of applications in image and signal processing are naturally expressed as process
pipelines, with parallelism both between pipeline stages, and within each stage
by replication and/or more traditional data parallelism [5].
Skeletal programming proposes that such patterns be abstracted and pro-
vided as a programmer’s toolkit, with specifications which transcend architec-
tural variations but implementations which recognise these to enhance perfor-
mance. This level of abstraction makes it easier for the disciplined programmer
to experiment with a variety of parallel structurings for a given application,
enabling a clean separation between structural aspects and the application spe-
cific details. Meanwhile, the explicit structural information provided by skeletons
enables static and dynamic optimisations of the underlying implementation.
In [2] we presented a number of observations and proposals designed to facili-
tate the popularisation of the skeletal approach, and outlined the first prototype
of eSkel, a library of skeleton operations for C and MPI. In the light of subse-
quent experiences we isolated and described two key concepts, nesting mode and
interaction mode, which capture important choices in the design of any skeletal
programming framework [6]. This caused us to substantially refine and partly
redesign eSkel. While these and other concepts embedded in eSkel may be found
in various combinations in previous skeletal systems, we believe that this paper
discusses the first attempt to enumerate and compose them systematically and
explicitly within a single framework.
This paper describes current work on the latest instantiation of eSkel. We
describe the ways in which key concepts have been incorporated into the pro-
gramming model and API in order to maximise the flexibility with which the
J.C. Cunha and P.D. Medeiros (Eds.): Euro-Par 2005, LNCS 3648, pp. 761–770, 2005.
c ? Springer-Verlag Berlin Heidelberg 2005

Page 2

762Anne Benoit et al.
programmer can shape and reshape skeletal programs, and on the substantial
implementation challenges which this flexibility creates.
2Dynamic Selection of Skeletons and Modes
eSkel is designed to provide the skeletal parallel programmer with flexibility in a
number of ways. As in the initial prototype [2], a skeleton abstracts the pattern
of interactions between a number of component activities, each of which may be
internally parallel, and which may choose to invoke further skeletons. The skele-
tons provided each encompass the ability to specify the use of either localised
or distributed data (the data spread), and the choice of either explicit or im-
plicit interaction mode [6] to trigger interactions between the various activities.
The aspects of the interface which capture these choices have been rationalised.
Guided by the concept of nesting mode [6], the programmer can choose to in-
corporate both transient and persistent skeleton calls within the same skeleton
nest. Crucially, all of these decisions, as well as the choice of overall skeleton
structure for each phase of a program, are made dynamically, and can there-
fore be based, where appropriate, on information gathered during execution (for
example, data values or sizes). In summary the significant improvements over
eSkel’s first version [2] are as follows.
– The definition of ‘families’ of related skeletons, capturing distinctions in in-
teraction style, have been replaced by the explicit selection of interaction
mode. For example, there is now only one pipeline skeleton, which can be
parameterized to behave as either of its predecessors.
– All skeletons in the original eSkel were (implicitly) in transient mode. There
were no persistent nestings. Both transient and persistent modes are now
available (by setting the corresponding parameter).
– The data model has been revised and extended to incorporate the concept of
molecules, enabling the expression of skeletons (such as Haloswap) in which
several distinct pieces of data are exchanged at each interaction.
2.1
The current specification of eSkel defines five skeletons, each with flexibility in
the dimensions discussed above. We will focus on the two of these, Pipeline
and Deal, which have so far been implemented in the current prototype, since
in combination these can illustrate all the points we wish to make. The reader
is referred to the eSkel homepage [7] for discussion of the Farm, HaloSwap and
Butterfly skeletons.
The Pipeline skeleton abstracts classical pipeline parallelism. It allows an
indexed set of activities (called “stages”) to be chained together and applied
to a sequence of inputs. Data is passed from a stage to its successor following
the natural ordering on stage indices. The user can choose between an implicit
pipeline, in which the transfer of the data is done automatically, constraining
the stages to produce one result for each input, or an explicit mode, in which a
stage can produce arbitrarily many results without necessarily consuming data
eSkel Interface

Page 3

Flexible Skeletal Programming with eSkel763
(for example, a generator or filter stage). In the latter case, the data transfer is
controlled by the programmer, by calls to the generic interaction functions Give
and Take.
The Deal skeleton is similar to a traditional farm, but with tasks distributed
strictly cyclically to workers. Thus it is most appropriately used when the work-
load associated with individual tasks is expected to be homogeneous. This skele-
ton is useful nested in a pipeline, in order to internally replicate a stage. Deal
semantics require the ordering of outputs from the skeleton to match that of the
corresponding inputs, irrespective of the internal speed of the workers.
The price for this flexibility is paid in the complexity of the API, where
each skeleton call must specify its choices. In addition to the issues already
discussed, the pragmatic decision to borrow and build upon MPI’s data model
with its already substantial parameter set, leads to rather long parameter lists.
However, when understood as a set of groups, each addressing orthogonal issues,
the parameters become more conceptually manageable. We now present these
groupings informally. Their formal use will be illustrated in Section 2.2.
One set of parameters deals with the called skeleton’s data buffers. As with
any MPI collective operation, there are distributed input and output buffers,
each requiring the standard MPI pointer, type, length definition. These, together
with the enclosing MPI communicator, capture the call’s interface to the rest of
the program. A second set of parameters deals with the call’s internal structure
and interfaces. For example, in a pipeline, we must describe the number of stages
and the types and data modes on their interfaces, and the allocation of processes
from the calling group to the distinct stages (captured by borrowing MPI’s
colouring parameter mechanism to describe the required sub-groups). Finally,
a group of parameters describe the details of the skeleton’s various activities
(stages in a Pipeline, workers in a Deal) with a choice of interaction mode and
a pointer to a function for each.
In the interests of regularity, we have chosen a single generic interface for all
functions which contain code for skeleton activities. The essence of activities is
that they interact in predefined patterns, via the skeleton infrastructure, with
each other and with skeleton calling programs. Each such interaction may involve
different numbers of data atoms, according to the semantics of the skeleton. For
example, “getting the next task” in a Deal skeleton involves a single atom,
whereas “updating the halo” in a HaloSwap skeleton will involve two atoms for
each neighbour (one arriving, one leaving). The eSkel_molecule_ttype collects
atoms involved in an interaction into a single sequence. Activity functions both
accept and return a single item, of type eSkel_molecule_t *. The specification
of each skeleton defines its detailed molecule usage.
2.2Examples
We present two variants of a toy program in order to concisely illustrate the ease
with which the eSkel programmer can describe and revise the parallel structure
of an application. The first version illustrates the use of persistent nesting and
implicit interactions. The second uses transient nesting and explicit interactions.

Page 4

764Anne Benoit et al.
1
2
3
4
5
6
7
8
9
#define STAGES
#define DEALWORKERS 2
#define INPUTNB 4
#define INPUTSZ 5
4// Number of pipeline stages
// Number of workers in the deal
// Input multiplicity
// Size of each input datum, per process
eSkel_molecule_t * SimpleStage (eSkel_molecule_t *thingy) {
int i;
for (i=0;i<thingy->len[0];i++) {
((int *) thingy->data[0])[i] += 1000;
}
return thingy;
}
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
eSkel_molecule_t * DealWorker (eSkel_molecule_t *thingy) {
int i;
for (i=0;i<thingy->len[0];i++) {
((int *) thingy->data[0])[i] += 1000;
}
return thingy;
}
void DealStage (void) {
int workercol, outmul;
if ((myrank() < 1))workercol = 0; else workercol = 1;
Deal (DEALWORKERS, IMPL, DealWorker, workercol, STRM,
NULL, 0, 0, SPGLOBAL, MPI_INT,
NULL, 0, &outmul, SPGLOBAL, MPI_INT, 0, mycomm());
}
int main (int argc, char *argv[])
{
int i, j, p, next, outmul, mystagenum, mymult, *inputs, *results;
spread_t spreads[STAGES+1]= {SPGLOBAL, SPGLOBAL, SPGLOBAL, SPGLOBAL, SPGLOBAL};
MPI_Datatype types[STAGES+1] = {MPI_INT, MPI_INT, MPI_INT, MPI_INT, MPI_INT};
Imode_t imodes[STAGES]= {IMPL, DEV, IMPL, IMPL};
eSkel_molecule_t *(*stages[STAGES])(eSkel_molecule_t *) =
{SimpleStage, (eSkel_molecule_t * (*)(eSkel_molecule_t *))DealStage, SimpleStage, SimpleStage};
MPI_Init(&argc, &argv); SkelLibInit();
inputs = (int *) malloc (sizeof(int)*INPUTNB*INPUTSZ);
for (i=0; i<INPUTNB*INPUTSZ; i++) inputs[i] = myrank()*100 + i + 1;
if
else if (myrank()<5)
else if (myrank()==6) mystagenum = 2;
else
(myrank()<2) mystagenum = 0;
mystagenum = 1;
mystagenum = 3;
results = (int *) malloc (sizeof(int)*INPUTNB*INPUTSZ); // Create output buffer
Pipeline (STAGES,imodes, stages, mystagenum, BUF, spreads, types,
(void *) inputs, INPUTSZ, INPUTNB, (void *) results, INPUTSZ, &outmul,
INPUTNB*INPUTSZ, mycomm());
MPI_Finalize(); return 0;
}
Fig.1. Example 1
Example 1: Persistent Nesting and Implicit Interactions
This program calls a Pipeline skeleton, in which the second stage contains a
persistently nested Deal. Some of the activities (pipeline stages and deal workers)
are assigned several processes, and process data with “global spread” (i.e. the
data is distributed across such processes and accessed in SPMD style).
The constant declarations define the number of pipeline stages (STAGES),
the number of workers (DEALWORKERS) in the deal, and the number and size of

Page 5

Flexible Skeletal Programming with eSkel765
input items (INPUTNB, INPUTSZ). When run with 8 processes, the input to this
pipeline consists of a total of (8*4*5) integers, which are treated as 4 (INPUTNB)
inputs to the pipeline, each of which is constructed from a 5 (INPUTSZ) integer
contribution per process. The first stage in the pipeline has been assigned two
processes. Thus, since we are in “global spread” mode, each pipeline input will
be fed to the two first stage processes as a block of (8*5)/2 integers each.
The main function (lines 31-57) makes the pipeline call. It defines all the
pipeline parameters and assigns processes to the stages. Lines 34-39 describe
the structure of the pipeline: all inter-stage data transfers use global spread
(SPGLOBAL) and involve sequence of integers. All activities except the second
stage use “implicit” mode (IMPL) transfers (in other words, handled implicitly
by the skeleton). The second stage has “devolved” mode (DEV), indicating that
this will be inherited from a persistently nested skeleton (the deal in this case).
All stages will execute the SimpleStage activity, except the second, which exe-
cutes DealStage. Lines 42-43 create some (distributed) input data. Lines 45-48
compute an assignment of processes to stages. Line 50 creates the (distributed)
output buffer. The pipeline call itself is on lines 52-54, and is passed the various
parameters as created, together with details of the input and output buffers in
the style of a conventional MPI collective operation.
Function SimpleStage, on lines 6-12, describes the actions of the first, third
and fourth stages. Since implicit interaction mode is used, the function takes
an eSkel molecule as a parameter and returns a molecule. Here we simply add
1000 to each integer in the block and return the modified molecule. In con-
trast, the second stage is described by DealStage. It computes an assignment
of processes (from the stage) to workers in the internal deal, then calls the Deal
skeleton on lines 26-28. The data mode is stream (STRM) indicating that the
data are streamed into the deal from the parent skeleton. The input and output
buffer related parameters are therefore redundant (all the NULLs and 0s). Deal
workers run the function DealWorker. In our simple example this is identical to
SimpleStage. We leave the duplication to emphasise that these functions can
be programmed independently, if appropriate.
Figure 2 illustrates the requested assignment of processes to stages and work-
ers, and the different communicators involved. This will help to clarify the dis-
cussion of eSkel implementation in Section 3. We have a 4-stage pipeline where
the second stage is a deal with two workers, of which one has two processes. Px
represents process x (0 ≤ x ≤ 7), as indexed in the original MPI_COMM_WORLD. Si
represents stage i (0 ≤ i ≤ 3), and Woj represents worker j (0 ≤ j ≤ 1).
Example 2: Transient Nesting and Explicit Interactions
Our second example demonstrates the use transient nesting, explicit interactions
and local data spread, and also illustrates the ease with which the programmer
can experiment with the high level structure of an application. Working from
the program of example 1, the programmer decides to
– turn the second stage (formerly the Deal) into a simple single process stage;
– reallocate the two spare processes from the second stage to the first stage;