# Algorithm 856: APPSPACK 4.0: asynchronous parallel pattern search for derivative-free optimization.

**ABSTRACT** APPSPACK is software for solving unconstrained and bound-constrained optimization problems. It implements an asynchronous parallel pattern search method that has been specifically designed for problems characterized by expensive function evaluations. Using APPSPACK to solve optimization problems has several advantages: No derivative information is needed; the procedure for evaluating the objective function can be executed via a separate program or script; the code can be run serially or in parallel, regardless of whether the function evaluation itself is parallel; and the software is freely available. We describe the underlying algorithm, data structures, and features of APPSPACK version 4.0, as well as how to use and customize the software.

**0**Bookmarks

**·**

**114**Views

- [Show abstract] [Hide abstract]

**ABSTRACT:**In the context of algorithmic parameter optimization, there is much room for efficient usage of computational resources. We consider the Opal framework in which a nonsmooth optimization problem models the parameter identification task, and is solved by a mesh adaptive direct search solver. Each evaluation of trial parameters requires the processing of a potentially large number of independent tasks. We describe and evaluate several strategies for using parallelism in this setting. Our test scenario consists in optimizing five parameters of a trust-region method for smooth unconstrained minimization.Optimization Letters 01/2013; 7(3). · 1.65 Impact Factor - [Show abstract] [Hide abstract]

**ABSTRACT:**Optimization is often used to perform model calibration, the process of inferring the values of model parameters so that the results of the simulations best match observed behavior. It can both improve the predictive capability of the model and curtail the loss of information caused by using a numerical model instead of the actual system. At its heart is the comparison of experimental data and simulation results. Complicating this comparison is the fact that both data sets contain uncertainties which must be quantified in order to make reasonable comparisons. Therefore, uncertainty quantification (UQ) techniques can be applied to identify, characterize, reduce, and, if possible, eliminate uncertainties. Incorporation of UQ into the calibration process can drastically improve the usefulness of computational models. Current approaches are serial approaches in that first, the calibration parameters are identified and then, a series of runs dedicated to UQ analysis is completed. Although this approach can be effective, it can be computationally expensive or produce incomplete results. Model analysis that takes advantage of intermediate optimization iterates can reduce the expense, but the sampling done by the optimization algorithms is not ideal. In this paper, we will review serial approaches and propose a joint calibration and UQ approach that combines Bayesian statistical models and derivative-free optimization in order to monitor sensitivity information throughout the calibration process.Journal of Computational Methods in Sciences and Engineering 01/2012; 12(1-2):99-110. - SourceAvailable from: Carsten HoffmannZhuo-Dong Zhang, Ralf Wieland, Matthias Reiche, Roger Funk, Carsten Hoffmann, Yong Li, Michael Sommer[Show abstract] [Hide abstract]

**ABSTRACT:**To provide physically based wind modelling for wind erosion research at regional scale, a 3D computational fluid dynamics (CFD) wind model was developed. The model was programmed in C language based on the Navier-Stokes equations, and it is freely available as open source. Integrated with the spatial analysis and modelling tool (SAMT), the wind model has convenient input preparation and powerful output visualization. To validate the wind model, a series of experiments was con-ducted in a wind tunnel. A blocking inflow experiment was designed to test the performance of the model on simulation of basic fluid processes. A round obstacle experiment was designed to check if the model could simulate the influences of the obstacle on wind field. Results show that measured and simulated wind fields have high correlations, and the wind model can simulate both the basic processes of the wind and the influences of the obstacle on the wind field. These results show the high reliability of the wind model. A digital elevation model (DEM) of an area (3800 m long and 1700 m wide) in the Xilingele grassland in Inner Mongolia (autonomous region, China) was applied to the model, and a 3D wind field has been successfully generated. The clear imple-mentation of the model and the adequate validation by wind tunnel experiments laid a solid foundation for the prediction and assessment of wind erosion at regional scale.Journal of Zhejiang University SCIENCE A 01/2012; 13(13):274-283.

Page 1

Algorithm 856: APPSPACK 4.0:

Asynchronous Parallel Pattern Search

for Derivative-Free Optimization

GENETHA A. GRAY and TAMARA G. KOLDA

Sandia National Laboratories

APPSPACK is software for solving unconstrained and bound-constrained optimization problems. It

implementsanasynchronousparallelpatternsearchmethodthathasbeenspecificallydesignedfor

problems characterized by expensive function evaluations. Using APPSPACK to solve optimization

problemshasseveraladvantages:Noderivativeinformationisneeded;theprocedureforevaluating

the objective function can be executed via a separate program or script; the code can be run serially

or in parallel, regardless of whether the function evaluation itself is parallel; and the software is

freelyavailable.Wedescribetheunderlyingalgorithm,datastructures,andfeaturesofAPPSPACK

version 4.0, as well as how to use and customize the software.

Categories and Subject Descriptors: J.2 [Computer Applications]: Physical Sciences and Engi-

neering—Mathematics and statistics

General Terms: Algorithms, Design, Documentation

Additional Key Words and Phrases: Parallel derivative-free optimization, pattern search

1. INTRODUCTION

APPSPACK is software for solving unconstrained and bound-constrained opti-

mization problems, that is, problems of the form

min

subject to l ≤ x ≤ u.

f (x)

(1)

This research was sponsored by the Mathematical, Information, and Computational Sciences Di-

vision at the United States Department of Energy and by Sandia National Laboratory, a multi-

program laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United

States Department of Energy under Contract DE-AC04-94AL85000.

Authors’ address: G. A. Gray, T. G. Kolda, Sandia National Laboratories, Livermore, CA 94551;

email: {gagray,tgkolda}@sandia.gov.

c ? 2006 Association for Computing Machinery. ACM acknowledges that this contribution was au-

thored or co-authored by a contractor or an affiliate of the [U.S.] Government. As such, the Gov-

ernment retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow

others to do so, for government purposes only.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is

granted without fee provided that copies are not made or distributed for profit or direct commercial

advantage, and that copies show this notice on the first page or initial screen of a display along

with the full citation. Copyrights for components of this work owned by others than ACM must be

honored. Abstracting with credit is permitted. To copy otherwise, to republish, or to post on servers,

to redistribute to lists, or to use any component of this work in other works requires prior specific

permission and/or a fee. Permission may be requested from Publications Dept., ACM, Inc., 2 Penn

Plaza, Suite 701, New York, NY 10121-0701 USA, fax: +1(212) 869-0481, or permission@acm.org.

C ?2006 ACM 0098-3500/06/0900-0485 $5.00

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006, Pages 485–507.

Page 2

486

•

G. A. Gray and T. G. Kolda

Here, f

are optional on an element-by-element basis; specifically, l is an n-vector

with entries in R ∪ {−∞} and u is an n-vector with entries in R ∪ {+∞}.

To find a solution of Equation (1), APPSPACK implements asynchronous

parallel pattern search (APPS) [Hough et al. 2001; Kolda 2004], a method

in the class of direct search methods [Wright 1996; Lewis et al. 2000].

More specifically, APPSPACK version 4.0 is a software implementation of

the APPS algorithm presented in Kolda [2004], while previous versions of

APPSPACK are based on the algorithm presented in Hough et al. [2001]. It

should also be noted that APPSPACK 4.0 is the first version with a related

publication.

The APPS algorithm is a variant on generating set search, as described in

Koldaetal.[2003].Itisprovablyconvergentiftheunderlyingobjectivefunction

issuitablysmooth[KoldaandTorczon2003,2004;Kolda2004].Moreover,APPS

is a direct search method, thus gradient information is not required. Therefore,

it is applicable to a variety of contexts.

APPSPACK is targeted to simulation-based optimization. These problems

are characterized by a relatively small number of variables (i.e., n < 100),

and an objective function whose evaluation requires the results of a complex

simulation. One standard application of this kind of optimization is parameter

estimation. The goal in this case is to identify the set of simulator input param-

eters that produces output most closely matching some given observed data.

For this problem, the objective function might be of the form

: Rn→ R ∪ {+∞} and x ∈ Rn. The upper and lower bounds

f (x) =

N

?

i=1

?si(x) − oi

?2.

Here, N denotes the number of data points to be compared; for example, the

points could correspond to times or spatial locations. The values oi for i =

1, ..., N are the given observed data values at these points, and the values

si(x) for i = 1, ..., N are simulator outputs at the same points, depending on

input x. Note that in order to discover the x that yields the best fit for the

observed data, multiple simulations are required.

Using APPSPACK to solve optimization problems has the following advan-

tages:

—No derivative information in needed.

—The procedure for evaluating the objective function does not need to be en-

capsulated in a subroutine and can, in fact, be an entirely separate program.

—The code can be run serially or in parallel, regardless of whether the objective

function itself runs in parallel.

—The software is freely available under the terms of L-GPL.

These advantages have prompted users to employ APPSPACK for a wide

variety of applications (see, e.g., Hough et al. [2001], Mathew et al. [2002],

Chiesa et al. [2004], Kupinksi et al. [2003], Croue [2003], Gray et al. [2004],

and Flower et al. [2004]).

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 3

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

487

APPSPACK4.0iswritteninC++andusesMPI[Groppetal.1996;Groppand

Lusk 1996] for parallelism. The underlying APPS algorithm and its algorithm-

specificclassstructuresaredescribedinSection2.Section3describesthemeth-

ods and class structures used by APPSPACK to manage the function evaluation

process; these tools are not unique to the APPS algorithm and could in fact be

used in other computer software programs. Potential users of APPSPACK will

be most interested in Section 4, where we explain how to use APPSPACK, and

Section 6, where we summarize and reference some successful applications.

In addition, customizations of APPSPACK are discussed in Section 5. Sections

5.1 and 5.2 describe how the software can be directly linked with the objec-

tive function and how different means of communication can be used for the

function evaluation. An adhoc means of handling general linear and nonlin-

ear constraints is discussed in Section 5.3. The implementation of general data

structures for handling these types of constraints is underway and will be com-

municated in a future publication and software release.

Thenotationinthisarticleisasfollows.Aboldfacecapitalletter,forexample,

T, denotes a set of vectors. A script capital letter, for example, I, denotes a set

of indices. A boldface lowercase letter, for example, x, denotes a vector, and its

ith entry is denoted by xi. Note that di represents the ith vector in a set of

vectors and not the ith component in d, which would instead be denoted di(no

boldface). Greek letters, for example, α, represent scalars.

2. APPS ALGORITHM AND IMPLEMENTATION

The APPS algorithm includes numerous details that are essential for efficient

and correct implementation, but not for a basic understanding of the method.

Omitting these details, APPS can be simply described as follows.

(1) Generate a set of trial points to be evaluated,

T = {x + ?idi: i ∈ I}.

(2)

Here, x is the best point known so far, diis the ith search direction, ?iis

the corresponding step length, and I is the subset of search directions for

which new trial points should be generated.

(2) Send the set T to the conveyor for evaluation, and collect a set of evaluated

points, E, from the conveyor (the conveyor is a mechanism for shuttling

trial points through the process of being evaluated).

(3) Process the set E and see if there exists a point y such that

y ≺ x.

The notation “≺” (described in detail in Figure 2) essentially means f (y) <

f (x).IfEcontainssuchapointy,thentheiterationissuccessful;otherwise,

it is unsuccessful.

(4) If the iteration is successful, replace x with the new best point (from E).

Optionally, regenerate the set of search directions and delete any pending

trial points in the conveyor.

(5) If the iteration is unsuccessful, reduce certain step lengths as appropriate.

In addition, check for convergence based on the step lengths.

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 4

488

•

G. A. Gray and T. G. Kolda

Fig. 1.APPS algorithm.

A detailed procedural version of APPS is given in Figure 1; for a complete

mathematical description and analysis, see Kolda [2004]. The iteration proce-

dure described in Figure 1 can be divided into two parts: steps (1) and (3)–(5),

and step (2). The implementation details of the first part are specifically tai-

lored to the APPS algorithm and described in the remainder of this section. The

second part includes more general features of the function evaluation manage-

ment used by APPSPACK and its implementation details are described in the

next section.

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 5

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

489

2.1 Points

Points (i.e., the best point and trial points) are stored as APPSPACK::Point ob-

jects. As previously described, new points are generated according to Equa-

tion (2), and each new point y is of the form

y = x + ?idi,(3)

where x is the parent, ?iis the step length, and diis the direction. Besides

the vector y ∈ Rnitself, each Point object stores some additional relevant in-

formation. Every Point includes a unique tag, that is, a positive integer that

is used as an identifier. In addition, each Point contains information about

the parent (i.e., the vector x), search direction, and step length used to gen-

erate y according to Equation (3). Once a trial point has been evaluated, its

function value f (y) ∈ R ∪ {+∞} is also stored in Point. All function val-

ues are stored as APPSPACK::Value objects which handle the possibility that

f (y) = +∞. APPSPACK uses the special case f (y) = +∞ to signify trial points

that could not be evaluated (e.g., the simulator failed) and certain types of

infeasible points.

In order for a trial point y to become the new best point (i.e. replace the cur-

rent best point), it must satisfy two conditions. First, it must satisfy a decrease

condition with respect to its parent, and second, it must have a function value

that improves upon that of the current best point. We describe both of these

conditions next.

A trial point y satisfies a decrease condition with respect to its parent x and

step length ? if the following holds:

f (y) < f (x) − α?2,(4)

where α ≥ 0 is a user-defined constant. If α = 0, this is called simple de-

crease [Torczon 1995]; otherwise, if α > 0, this is called sufficient decrease [Yu

1979; Lucidi and Sciandrone 2002]. To indicate whether a trial point satisfies

Equation (4) with respect to its parent, state information is stored in object

Point. The state also specifies whether a trial point has been evaluated.

Satisfying decrease condition Equation (4) is only part of the comparison

APPSPACK uses to determine whether a point is eligible to replace the current

best point. The comparison also considers the corresponding function values as

comparedtothatofthecurrentbestpointanddefinesaschemefortie-breaking,

in the case that these function values are equal. The complete procedure for

determining if a point y is better than a point x (i.e., whether y ≺ x) is detailed

in Figure 2.

In summary, an APPSPACK Point object stores the following information:

—The vector y ∈ Rn;

—its unique tag, denoted TAG(y);

—its parent’s tag, denoted PARENT TAG(y);

—its parent’s function value;

—the step used to produce it, denoted STEP(y);

—the index of the direction used to produce it, denoted DIRECTION INDEX(y);

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 6

490

•

G. A. Gray and T. G. Kolda

Fig. 2. Determining if y ≺ x.

—the state information; and

—its function value f (y), if it has been evaluated.

The state information indicates whether the point has been evaluated, and

if it has, whether it satisfies the sufficient decrease condition. Note that the

parent vector is not explicitly stored; instead, only its corresponding tag and

function value are stored. Likewise, the actual direction vector diis not stored;

instead, only its index i is stored.

2.2 Constraints and Scaling

Bound constraints are supported via the APPSPACK::Constraints::Bounds

class.TheconstraintsobjectispassedasanargumenttotheAPPSPACK::Solver,

which executes the APPS algorithm so that we leave open the possibility of al-

lowing user-developed constraint classes in future versions.

The bounds on the variables are specified by the user in the APPSPACK

input file (see Section 4.2). They are used both to determine a conforming set of

search directions (see Section 2.3) and to generate trial points (see Section 2.4).

Related to bounds and stored in the same object is the variable scaling. Be-

cause derivative-free methods do not use any gradient information, proper scal-

ing of the variables is critical. Although scaling is not explicitly mentioned in

the description of APPS provided in Figure 1, it plays various important roles

such as in convergence (see Section 2.5), determining a conforming set of search

directions(seeSection2.3),calculatingtrialpoints(seeSection2.4),andlooking

up points in the cache (see Section 3.4).

Inthecasethatallboundsarefinite,theusermaychooseeithertoemploythe

default scaling vector or to provide one in the APPSPACK input file (“Scaling”

in the “Bounds” sublist). The default scaling vector is defined using (finite)

bounds. Specifically, let l, u ∈ Rndenote the vectors of the lower and upper

bounds, respectively. Then, the components of the scaling vector s ∈ Rnare

defined as

si= ui− li

If finite bounds are not provided or do not exist, s must be provided by the user.

Note that this approach to scaling was motivated by another derivative-free

optimization software package, IFFCO [Choi et al. 1999], a Fortran implemen-

tation of the implicit filtering method [Gilmore and Kelley 1995; Kelley 1999].

for i = 1, ..., n.

(5)

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 7

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

491

2.3 Search Directions and Step Lengths

ThesearchdirectionsarehandledbytheAPPSPACK::Directionsclass.Anewset

of search directions, or a search pattern, is computed every time an iteration is

successful(seestep(4)inFigure1).Togeneratethissearchpattern,APPSPACK

considers the scaled coordinate directions and excludes directions outside the

tangent cone. Specifically, given the current best point x, the set of search

directions is defined as

D = {d1, ..., dp} = {siei: xi< ui} ∪ {−siei: xi> ?i},

where diis the ith search direction, eiis the ith unit vector, and ui∈ R ∪ {∞}

and ?i∈ R ∪ {−∞} are the bounds on xi(as defined in Equation (1)).

Each direction di has a tag τi and step length ?i associated with it. The

tag is an integer that indicates whether there are any points in the evaluation

conveyor associated with a given search direction. If τi = −1, then there are

currently no unevaluated trial points with parent x that were generated using

direction di. In this case, the step length ?iis the value that will be used in

step (1) of Figure 1 to compute a new trial point in direction i. Otherwise, τi

is the tag number of the point associated with direction di, and ?iis the step

length that was used to generate this point.

2.4 Generation of Trial Points

Trial points are generated in the APPSPACK::Solver class. As indicated in step

(1) of Figure 1, a trial point is computed for each direction i ∈ I, where I =

{i : ?i≥ ?toland τi= −1}. In other words, the set I contains search directions

that have not yet converged (see Section 2.5) and do not currently have a trial

point in the evaluation conveyor.

For each i ∈ I, a feasible trial point is calculated. If y = x + ?idi is not

feasible, then an appropriate pseudostep must be determined. The pseudostep

˜?iis the longest possible step that is feasible, formally defined as

˜?i= max{? ∈ [0, ?i] : l ≤ x + ?di≤ u}.

2.5 Stopping Conditions

The primary stopping condition in the APPS algorithm is based on step length.

This criterion was chosen because it can be shown that if the objective function

is continuously differentiable, then the norm of the gradient (or an analogous

measure of the constrained measure of stationarity in the bound-constrained

case) can be bounded as a multiple of the step size, for example, ?∇ f (x)? =

O(maxi{?i}) [Kolda et al. 2003]. In other words, the steps only get smaller if

the norm of the gradient is decreasing. Hence, the step length can be used to

define a stopping requirement.

The stopping condition based on step length is used in step (5) of Figure 1.

Specifically, APPS converges if all the step lengths are less than the specified

tolerance; that is, if

for i = 1, ..., p.

Here, we say that the ith direction is converged if ?i< ?tol. The tolerance ?tol

?i< ?tol

(6)

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 8

492

•

G. A. Gray and T. G. Kolda

can be specified by the user in the APPSPACK input file (“Step Tolerance” in

the “Solver” sublist). The default value is ?tol= 0.01, which corresponds to a

1% change in the variables when the default scaling defined in Equation (5) is

used.

An alternative stopping condition is based on whether the function has

reached a specified threshold. This criterion may be useful when the desired

minimum value is known. For example, in the parameter estimation problem

described in Section 1, it may be reasonable to stop when f (x) < 0.03 or when

the fit is within 3% of being exact. Step (4) of Figure 1 shows the implementa-

tion of the function-tolerance stopping criteria. Specifically, the iterations are

terminated if

f (x) ≤ ftol,(7)

where ftol is defined by the user in the APPSPACK input file (“Function

Tolerance” in the “Solver” sublist). By default, this stopping condition is not

employed.

Stopping can also be defined in terms of the number of function evaluations.

In other words, the algorithm can be discontinued after a specified number of

function evaluations has been completed. This sort of stopping criterion might

be useful when the function evaluations are based on a particularly expensive

simulation, and the user wants to adhere to a specified budget for evaluations.

By default this stopping criterion is not used, but it can be activated by the

user specifying a maximum number of function evaluations in the APPSPACK

input file (“Maximum Evaluations” in the “Solver” sublist).

3. APPSPACK FUNCTION EVALUATION MANAGEMENT

This section describes the implementation details and specific C++ objects re-

lated to management of the function evaluation process used in APPSPACK

version 4.0. Although we describe the evalution conveyor, executor, evaluators,

and cache, as they pertain to APPSPACK, it should be noted that they are

merely a general way to handle the process of parallelizing multiple indepen-

dent function evaluations and efficiently balance the computational load. Thus,

these classes may be applicable to other software packages.

3.1 The Evaluation Conveyor

From the point of view of the APPS algorithm, the evaluation conveyor simply

works as follows: A set of unevaluated points T is exchanged for a set of evalu-

ated points E (step (2) of Figure 1). The set T may be empty. However, because

returning an empty set of evaluated points means that the current iteration

cannot proceed, the set E must always be nonempty.

Within the conveyor, a trial point moves through three stages. The first

stage is for the trial point to wait in a holding pen (the “Wait” queue) until

resources become available for evaluating its function value. The second stage

occurs while the function value is in the process of being computed, in which

it sits in the “Pending” queue. The third stage takes place after the evaluation

has been completed, while the trial point waits to be returned as output from

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 9

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

493

the conveyor (in the “Return” queue). Each of these stages is described in more

detail to follow.

One key point in this process is that it may take more than one iteration for a

point to move through the conveyor. Thus, the set of points T that is input is not

necessarily the same as the set of points E that is output. Furthermore, because

it may take multiple iterations for a point to move through the conveyor, it is

often desirable to remove some or all of the points that are in the first stage of

the conveyor, waiting for evaluation. This removal of points is called pruning.

Every point submitted to the evaluation conveyor is eventually either re-

turned or pruned. Pruning occurs in step (4) of Figure 1, while returning takes

place in step (2). The evaluation conveyor facilitates the movement of points

through the following three queues:

—W. The “Wait” queue where trial points wait to be evaluated. This is the only

queue from which points can be pruned.

—P. The “Pending” queue for points with on-going evaluations. Its size is re-

stricted by the resources available for function evaluations.

—R. The “Return” queue where evaluated points are collected. Its size can be

controlled by the user.

This conveyor process is handled by the APPSPACK::Conveyor object. Each

time a set of trial points is received (step (2) of Figure 1), the conveyor follows

the procedure diagrammed in Figure 3.

Points that are waiting to be submitted to the executor for evaluation are

stored in W, and remain there until either there is space in P or they are

pruned. The W queue is pruned whenever an iteration is successful (step (4) of

Figure 1). By default, pruning is defined as the emptying of W. However, the

results of a pruning can be modified by setting the “Max Queue Size” parameter

in the “Solver” sublist of the APPSPACK input file. Here, the user can specify

the number of points that should remain in W after it is pruned. In this case,

the oldest points are deleted and the newest remain in the queue.

Before a point moves from W to P, the cache is checked to see if the function

value has already been calculated for this point (see Section 3.4). If so, the

cached function value is obtained, and the point is moved directly to R. If not,

the point moves to P and is evaluated. Once a point has been pushed from W

onto P or R, it can no longer be pruned.

Points which have been submitted to the executor for evaluation are stored

in P. The executor handles the distribution of evaluation tasks to workers and

is described in Section 3.2. The size of P is solely determined by the executor

and depends on available resources. Recall that APPSPACK was designed to

accommodate problems that may have expensive function evaluations. Hence,

points may remain in P for several iterations. Once the executor returns the

results of the function evaluation, the point is moved to R.

Essentially,theconveyorprocesscontinuesuntilenoughevaluatedpointsare

collected in the R. Enough is defined by the “Minimum Exchange Return” value

set in the “Solver” sublist of the APPSPACK input file. The default value is one,

but larger values can be used to force the conveyor to collect more evaluated

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 10

494

•

G. A. Gray and T. G. Kolda

Fig. 3. Conveyor actions for trial point exchange.

trial points before returning. In the extreme, the conveyor process can continue

until every trial point has been evaluated and collected. This behavior defines a

synchronous pattern search [Lewis and Torczon 1996] and can be activated by

setting the parameter “Synchronous” to true in the “Solver” sublist (the default

is false). Finally, note that the size of R can also be controlled by defining a

“Maximum Exchange Return” size in the “Solver” sublist, which defaults to

1000.

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 11

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

495

3.2 Executors: Function Evaluation Coordinators

When a point enters the second stage of the conveyor (the P queue) it must

be assigned to a worker (if running in parallel) and evaluated. The executor

coordinates the assignment of points to workers for function evaluation. Its

pure virtual abstract interface is defined in APPSPACK::Executor::Interface

and includes the following:

—A Boolean function that returns true if the executor is waiting to spawn more

function evaluations. The result of this function is used in the conveyor test

|P| < MaxPending, shown in Figure 3.

—A spawn function that initiates a function evaluation. The input is a vector

y and its corresponding tag, TAG(y). The tag is used as an input argument in

the function evaluation executable, and it is needed to match the resulting

function value with the appropriate point in queue P.

—A receive function that checks whether any function evaluations have fin-

ished. If an evaluation has been completed, the output is the objective func-

tion value and any related information.

The executor is passed as an argument to the APPSPACK::Solver, the

class that runs the APPS algorithm. APPSPACK 4.0 contains two con-

crete implementations of the executor, namely, APPSPACK::Executor::MPI

and APPSPACK::Executor::Serial, as well as the corresponding executables,

appspack mpiandappspack serial.Inaddition,theexecutorcanbecustomized

by the user, as explained in Section 5.1.

The MPI executor that comes with APPSPACK spawns function evaluations

toworkerprocesses.Allprocessorssaveoneareworkers;thatis,ifappspack mpi

is run with five processes, then there will be four workers and the size of the

pending queue P will be four. Each time APPSPACK calls the spawn func-

tion, the executor sends an MPI message to an idle worker, containing both a

point to be evaluated and its corresponding tag. The executor marks the worker

as busy. The worker performs the evaluation by calling the evaluator (see

Section 3.3) and returns an MPI message with the tag, a Boolean indicating if

there is a function value, the function value itself, and a message string (e.g.,

"Success"). When APPSPACK calls the receive function, the executor checks

for pending messages from a worker (and there may not be any), collects the

result, marks the worker that returned the message as idle, and returns the

result to APPSPACK.

The serial executor does exactly as its name implies—executes function eval-

uations one-at-a-time. In other words, the size of the pending queue P is exactly

oneandallevaluationsareperformedimmediatelyonthecurrentprocessor(be-

causetherearenoworkers).Thespawnoperationcallstheevaluator(described

in Section 3.3) directly. While there may be some situations where the serial

version of APPSPACK proves useful, this mode is provided primarily for testing

purposes.

The executor may be customized, the reasons and methods for which are

discussed in Section 5.2.

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 12

496

•

G. A. Gray and T. G. Kolda

Fig. 4.The system call evaluator.

3.3 Evaluators

The actual objective function evaluation of the trial points is han-

dled by the evaluator,whoseabstract

APPSPACK::Evaluator::Interface class. This structure allows the user to ei-

ther employ the provided default evaluator or create a customized one. The

default evaluator is the APPSPACK::Evaluator::SystemCall class, which works

as follows: A function input file containing the point to be evaluated is cre-

ated. Then, an external system call is made to the user-provided executable

that calculates the function value. After the function value has been com-

puted, the evaluator reads the result from the function output file. Finally, both

the function input and output files are deleted. This process is illustrated in

Figure 4. Information regarding the user-provided executable and the formats

of function input and output files is given in Section 4.1.

The evaluator is its own entity and not part of the executor, although it is re-

lated.InMPImode,eachworkerpossessesitsownevaluatorobjectandreceives

messages from the executor on the master processor with the information (e.g.,

the point to be evaluated) it needs to pass on to the evaluator. We also note

that any information needed to construct and intialize the evaluator has to

be passed from the master to the worker processes via an MPI message before

the APPSPACK::Solver::solve function is called. In our case, for example, the

master sends the workers the name of the executable to be used for the function

evaluation. Note that, in serial mode, the executor owns the evaluator directly

and calls it in the spawn function.

By default, APPSPACK runs function evaluations as separate executables

and communication with the evaluation executable is done via file input and

output. In other words, each worker makes an external system call, as illus-

trated in Figure 5. This default design ensures applicability of APPSPACK to

simulation-based optimization because simulations are often too complex to

be easily or reasonably encapsulated into a subroutine. Moreover, allowing the

user to supply a separate executable extends the usability of APPSPACK, since,

for example, the user can write the program for the function evaluation in

interface is definedin the

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 13

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

497

Fig. 5.Parallel APPSPACK using the (default) system call evaluator.

any language or simply provide a script that executes individual steps of a

complicated function evaluation. Finally, although system calls and file I/O

do add to the overall run time, the simulation time often makes the external

function calls and file I/O negligible. For applications in which the user may

prefer to eliminate this overhead (e.g., the function evaluation may be rela-

tively inexpensive), the user can provide a customized evaluator, as detailed in

Section 5.1.

3.4 Cache

Because the APPS algorithm is based on searching a pattern of points that lies

on a regular grid, the same point may be revisited several times. Thus, to avoid

evaluatingtheobjectivefunctionatanygivenpointmorethanonce,APPSPACK

employs a function value cache. Each time a function evaluation is completed

and a trial point is placed in the return queue R, the conveyor stores this point

and its corresponding function value in the cache. Then, before sending any

point to the pending queue P, the conveyor first checks the cache to see if its

value has already been calculated. If it has, the cached function value is used

instead of repeating the function evaluation.

Cache operations are controlled by the APPSPACK::Cache::Manager class. Its

functions include inserting new points into the cache, performing lookups, and

returning previously calculated function values. Optionally, the cache manager

can also create an output file with the contents of the cache or read an input file

generated by a previous run. These features can be activated using the “Cache

Output File” and “Cache Input File” parameters of the “Solver” sublist in the

APPSPACK input file.

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 14

498

•

G. A. Gray and T. G. Kolda

Likeitspredecessors,APPSPACKversion4.0usesasplaytreetostorepoints

in the cache [Hough et al. 2000]. A splay tree is a binary search tree that uses a

series of rotations to move any accessed node to the root (see Sleator and Tajan

[1985]). Because the most recently accessed nodes are kept near the root of the

tree, searching these nodes is fast. The APPS algorithm can take advantage of

this characteristic of splay trees because it normally only revisits points that

were evaluated recently.

CachepointsarestoredasAPPSPACK::Cache::Pointobjectsthatincludeonly

the vector itself and its corresponding function value. This structure eases stor-

age and comparison, including special rules for point comparison, as explained

next.

APPSPACK does not require exact matched in-cache lookups. Rather, it only

requires the points to be ε-equal. Specifically, trial points x, y ∈ Rnare ε-equal

if

|xi− yi| <= εsi

for i = 1, ..., n,

where siis the ith component of the scaling vector s and ε is some tolerance.

The value of ε can be set by the user using the “Cache Comparison Tolerance”

parameter in the “Solver” sublist of the APPSPACK input file, and defaults to

half the stopping tolerance ?tol.

It is also necessary to impose some order on the points so that they can be

placed in the tree. Correspondingly, this cache comparison test is also based on

the tolerance ε and the scaling vector s. Here, trial point x is ε-less than y if

there exists an index j such that

|xi− yi| < εsi

for i = 1, ..., j − 1and

yj− xj> εsi.

In other words, for ordering purposes x comes before y if the first j − 1 coordi-

nates of x and y are ε-equal, and the jth coordinate of y is sufficiently greater

than the jth coordinate of x.

4. USING APPSPACK

APPSPACK 4.0 is available as a free download under the terms of the GNU

Lesser General Public License.1Instructions for downloading, compiling, and

installing APPSPACK 4.0 can be found in the README file included with the

software.SeveralresourcesareavailableforAPPSPACKusers,includingonline

documentation, mailing lists, and a bug-reporting system. The access and use

of these tools is described in detail in the HTML documentation.

4.1 Creating a Function Evaluation for APPSPACK

An executable for evaluating the objective function must be provided by the

user. It can be a single program or a script that, in turn, calls other programs.

TheAPPSPACKevaluatorcallsthisexecutablerepeatedlytoevaluatedifferent

trial points via the C system() command.

1http://www.gnu.org/copyleft/lesser.html

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

Page 15

Algorithm 856: APPSPACK 4.0: Asynchronous Parallel Pattern Search

•

499

The executable command line should accept three input arguments: an input

file name, output file name, and a tag. In other words, the the calling sequence

is

<Executable Name> <Input File> <Output File> <Tag>

The input file is created by APPSPACK to be read by the executable and is

simply formatted. The first line is an integer that indicates the length of the

vector, and each subsequent line contains one component of the vector. For

example, the APPSPACK input file containing the vector [1.24e-1, 17.4e-3]

∈ R2is:

2

1.24e-1

17.4e-3

The output file is created by the executable and read by APPSPACK. It contains

either the function value as a single numeric entry or an error string. For

example, an output file containing a function value may contain the single line

7.58e-3, whereas an output file with an error string may contain the single line

Meshing Error.

APPSPACK has the ability to tabulate different error strings and can track

an unlimited number of messages. Strings are defined by the user in the

executable. They can be more than one word, but must be the length of

only one line, and the string “Success” is disallowed as an error string. This

feature may be useful for identifying the reasons for a function evaluation

failure.

The MPI version of APPSPACK executes multiple function evaluations in

parallel. To prevent these parallel processes from overwriting each other’s files,

each call to the executable includes uniquely named input and output files.

This is accomplished using the tag number, stored in the APPSPACK Point

structure, to name related files. The tag is also provided as the third input

argument of the executable so as to allow the user to uniquely name any addi-

tional files that may be created during an evaluation of the objective function.

Note that while the tags and subsequent file names are unique for a single run

of APPSPACK, they are not unique across repeated runs.

After a function evaluation has been completed, APPSPACK will automati-

cally delete the input and output files. These files should not be deleted by the

executable. However, any additional files that were created in the process of

evaluating the function should be deleted by the executable. So-called leftover

files, that is, temporary files that are not deleted, are frequent sources of error

and may cause system disk-space problems.

4.2 Creating the APPSPACK Input File

The user must provide an input file that specifies the parameters for run-

ning APPSPACK. There are three categories of input parameters: Evaluator,

Bounds, and Solver. Each is described in the subsections that follow. To clarify

our discussion of the general formatting of an APPSPACK input file, consider

ACM Transactions on Mathematical Software, Vol. 32, No. 3, September 2006.

#### View other sources

#### Hide other sources

- Available from Genetha A Gray · May 31, 2014
- Available from psu.edu