Page 1

DESIGNING SIMULATION EXPERIMENTS WITH

CONTROLLABLE AND UNCONTROLLABLE FACTORS

Christian Dehlendorff

Murat Kulahci

Klaus Kaae Andersen

Department of Informatics and Mathematical Modelling

Technical University of Denmark

Bygning 321, Richard Petersens Plads

Lyngby, DK-2800, DENMARK

ABSTRACT

In this study we propose a new method for designing com-

puter experiments inspired by the split plot designs used

in physical experimentation. The basic layout is that each

set of controllable factor settings corresponds to a whole

plot for which a number of subplots, each corresponding

to one combination of settings of the uncontrollable fac-

tors, is employed. The caveat is a desire that the subplots

within each whole plot cover the design space uniformly.

A further desire is that in the combined design, where all

experimental runs are considered at once, the uniformity

of the design space coverage should be guaranteed. Our

proposed method allows for a large number of uncontrol-

lable and controllable settings to be run in a limited number

of runs while uniformly covering the design space for the

uncontrollable factors.

1 INTRODUCTION

With the current advances in computing technology, com-

puter and simulation experiments are increasingly being

used to study complex systems for which physical experi-

mentation is usually not feasible. Our case study involves

a discrete event simulation model of an orthopedic surgical

unit. The discrete event simulation (DES) model describes

the individual patient’s progress through the system and

has been developed in collaboration with medical staff at

Gentofte University Hospital in Copenhagen. The unit un-

dertakes both acute and elective surgery and performs more

than 4,600 operative procedures a year. While the patients

come from various wards throughout the hospital, the main

sources of incoming patients are the four orthopedic wards

or the emergency care unit.

Thesimulationmodelis implementedinExtendversion

6 (Krahl 2002) on a Windows XP platform and controlled

from a Microsoft Excel spreadsheet with a Visual Basic for

application script. The model consists of 3 main modules:

The wards and arrival, the operating facilities, and the

recovery and discharge. Interaction with the surrounding

hospital is for example modeled with simplified processes

usingthe same resources as the processesin the surgicalunit

(occupyingthe resources) and with the patients entering and

exiting the model. Operating rooms, recovery beds, wards

andstaff are includedinthe model. Theaverageruntime for

simulating6 months(with oneweek ofwarm-up)operations

is around 7 minutes. Typical outcomes are waiting times,

patient throughput and the amount of overtime.

The simulation model has two sources of noise coming

from variations in the uncontrollable factors (a.k.a. envi-

ronmental factors in physical experimentation) and from

changes in the seed controlling the random number gen-

eration process embedded in the simulation model. The

controllable factors are for example the number of op-

erating rooms and the number of surgeons, whereas the

uncontrollable factors may include for example the arrival

rate of acute patients and the time required to clean the

operating rooms.

Inthis typeofapplication, severalissues needtobecon-

sidered. First, the controllable factors tend to be numerous

and often discrete. Moreover a single experiment usually

takes several minutes to run. Therefore a simple exhaus-

tive method, where all possible combinations of the factor

settings are considered, is often computationally infeasible

due to the exponentially increasing number of factor com-

binations. Furthermore, the settings of the uncontrollable

factors, e.g. the acute patient arrival rate or the duration of

surgical procedures, are also of interest and must be deter-

mined as they may influence the outcome of the simulations

and hence the robustness of the simulation analysis.

The paper is organized in the following manner: Sec-

tion 2 introduces design of computer experiments and de-

2909 978-1-4244-2708-6/08/$25.00 ©2008 IEEE

Proceedings of the 2008 Winter Simulation Conference

S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. W. Fowler eds.

Page 2

Dehlendorff, Kulahci, and Andersen

fines the performance measure for the designs. Section 3

describes the proposed design method and contrasts it with

other methods. In section 4 opportunitiesfor future research

are presented. Finally the main conclusions are summarized

in section 5.

2 DESIGN OF COMPUTER EXPERIMENTS

2.1 Literature Review

A general discussion on the issues regarding the de-

sign and analysis of computer experiments can be found

in Sacks et al. (1989), Santner, Williams, and Notz (2003)

and Fang, Li, and Sudjianto (2006).

the computer experiments are often considered to come

from a deterministic computer code.

ments, the classical design of experiment methods such

as replication is deemed to be redundant as replica-

tion of an experiment, for example, yields exactly the

same result (see Santner, Williams, and Notz (2003) and

Fang, Li, and Sudjianto (2006)).

Experiments based on a simulation model often involve

somestochastic component; makingthe outputalsostochas-

tic. Kleijnen (2008) discusses the design and analysis of

simulation experiments which typically have some sort of

noise in the output. Therefore these experiments differ

from the deterministic computer experiments. Furthermore,

a typical simulation application will have both controllable

and uncontrollable (environmental) factors, which should

be handled differently. In these applications the aim is to

manipulate the controllable factors so that the system is

insensitive (robust) to changes in the uncontrollable factors.

As described by Kleijnen (2008) and Sanchez (2000) the

solution’s robustness needs to be considered in order to

obtain applicable solutions in systems with uncontrollable

factors. That is, a good solution needs to perform well over

the entire range of uncontrollable factors.

The original concept of robustness in physical systems

is often attributed to Taguchi (1987). Taguchi’s methods

involve an inner array for the controllable factors and an

outer array for the uncontrollable factors. In simulation

studies, Kleijnen (2008) suggests using a crossed design,

e.g. combining a central composite design (CCD) for the

controllable factors and a Latin Hypercube Design (LHD)

for the uncontrollable factors. In a crossed design the same

set of subplots is used for each whole plot. However, as we

will show in this study, this may not be the most efficient

way of running such experiments.

The outputs from

In such experi-

2.2 Simulation Model

Our basis is a discrete event simulation model generating

output, y = f(xc,xe), for the settings for the sccontrollable

factors, xc, and the settings for the seuncontrollable factors,

xe. The objective is not only to select the settings, x∗

that the solution is robust to changes in the uncontrollable

factorsettingsasdescribedinp. 130-134in Kleijnen (2008),

butalsotounderstandthevariationcomingfromthechanges

in the uncontrollable factor settings.

Since little prior knowledge of both controllable and

uncontrollable factors is available, we require that a good

design is simultaneously uniform over the design space of

thecontrollableanduncontrollablefactors. Inthefollowing,

wewillassumethattheuniformcoverageofthedesignspace

of the controllable factors is already achieved and that we

are only concerned with the uncontrollable factors.

Robustness studies in physical experimentation often

involve split-plot designs (Montgomery 2005).

therefore use similar terminology when robustness studies

are performed using computer experiments. In classic split-

plot designs, a set of experiments called whole-plots is de-

signedsothatforeachwhole-plotanotherset ofexperiments

called subplots are run. In robustness studies, the settings

of the controllable factors often constitute the whole-plots,

whereas the settings of the uncontrollable factors constitute

the subplots. In Table 1, a whole-plot corresponds to a row

in which randomly selected combinations of settings for the

uncontrollable factors are run. It should be noted that the

randomization issue is irrelevant for computer experiments.

In the proposed method, each whole-plot corresponds

to one combination of settings of the controllable factors

(a row in Table 1), i.e. a total of ncwhole-plots are needed

(nc= 5 in Table 1). Each subplot (a column entry in any

row in Table 1) corresponds to a combination of settings for

the uncontrollable factors with a total of k subplots for each

whole-plot. ThustheoveralldesignconsistsofN =nckruns.

In a crossed design as proposed by Kleijnen (2008) these

k subplots would be the same from one whole-plot to the

next. Therefore there will only be a total of k combinations

of settings for the uncontrollable factors. In our proposed

methodology, different k combinations of settings for the

uncontrollable factors will be used for each whole-plot.

This is expected to give better overall coverage of the

uncontrollable factor space compared to the crossed design.

The challenge with the proposed method is to make the

uncontrollable factor settings comparable from one whole-

plot to the next.

c, such

We will

2.3 Measure of Uniformity

In order to evaluate the designs presented in the fol-

lowing sections a measure of uniformity is needed.

Fang, Li, and Sudjianto (2006) summarize a set of perfor-

mance measures frequently used for measuring the uni-

formity of a design: the star discrepancy, centered dis-

crepancy and the wrap-around discrepancy.

tered and the wrap-around discrepancy were proposed

by Hickernell (1998b) and Hickernell (1998a), respectively.

The cen-

2910

Page 3

Dehlendorff, Kulahci, and Andersen

Table 1: Uncontrollable factor design for five controllable

settings and five environmental settings within each con-

trollable setting

Controllable

setting

1

2

3

4

5

Environmental setting

23

xe2

xe3

xe7

xe8

xe12

xe13

xe17

xe18

xe22

xe23

145

xe1

xe6

xe11

xe16

xe21

xe4

xe9

xe14

xe19

xe24

xe5

xe10

xe15

xe20

xe25

Both have desirable properties. They are easy to compute,

invariant to permutations of factors or runs and rotation of

coordinates,andreliablemeasurementsfortheuniformityof

projections. However the wrap-around discrepancy is said

to be unanchored (i.e. it only involves the design points),

while the centered discrepancy is not, since it involves the

corners of the unit cube.

In this study only the wrap-around discrepancy is

considered as the measure of uniformity with a low

value corresponding to a high degree of uniformity. The

measure is chosen since the literature generally sug-

gests it as a good measure of uniformity (see for ex-

ample Fang and Ma (2001);

Fang, Li, and Sudjianto (2006)). The idea behindthis mea-

sure is that for any two points from a uniform design, x1

and x2, spanning a hyper cube (potentially wrapping around

the bounds of the unit cube); the hypercube should contain

a fraction of the total number of points equal to the fraction

of total volume covered by the cube. An analytic expres-

sion for the wrap-around discrepancy (WD(D)) is given

by Fang and Ma (2001) as

Fang, Lin, and Liu (2003);

(WD(D))2=−?4

with di(j,k) =3

number of points, s the number of factors (the dimension),

and xkithe i’th coordinate of the k’th point.

There are various ways of constructing uniform de-

signs. In this study the good lattice point method based

on the power generator is used with the modification

described in Fang, Li, and Sudjianto (2006). The design

construction is based on a lattice {1,...,n} and a gen-

erator h(k) = (1,k,k2,...,ks−1)(mod n), with k fulfilling

that k,k2,...,ks−1(mod n) are distinct.

such that the resulting design consisting of the elements

uij= ih(k)j(mod n) scaled down to [0,1]shas the lowest

WD-value.

3

?s+1

n

?3

2

?s+2

n2

n−1

∑

k=1

n

∑

j=k+1

s

∏

i=1

di(j,k) (1)

2−|xki−xji|(1−|xki−xji|), n being the

h(k) is chosen

3DESIGN ALGORITHM

Amethodforgeneratinggooddesignsforsimulationmodels

withbothcontrollableanduncontrollablefactorsispresented

inthefollowingsection. Hereweassumethatallfactorshave

been scaled to [0,1] and that the wrap-around discrepancy

is the measure of uniformity. It is furthermore assumed

that a design for the controllable factors is available. That

is, we are primarily concerned with designing experiments

for the uncontrollable factors. Two and three dimensional

examples are used since they can be illustrated graphically.

However, the method is general and results for 4 and 10

factors are also presented.

3.1 Bottom-up Approach

In section 2.2 the limitations of crossing a design for the

controllable factors with a design for the uncontrollable

factorsweredescribed. Abettermethodintermsofcovering

the uncontrollable factor space compared to the crossed

design is to generate different designs for the whole-plots,

each with k different combinations of uncontrollable factor

settings. This implies that nc designs of size k should be

constructed. For this method to succeed in the combined

design, not only sets of k subplots for different whole-

plots should be comparable, but also nck subplots need

to cover the design space for the uncontrollable factors

uniformly. This can be achieved by dividing the design

hyperspace for the uncontrollable factors into k sub-regions

and sample ncsettings in each. As shown in Figure 1, this

can be achieved fairly easily in two dimensions. However,

in higher dimensions an efficient way of generating the

sub-regions is required since the curse of dimensionality

dictates that exponentially increasing numbers of runs have

to be used in higher dimensions to obtain the same density

of runs as in the lower dimensions.

If regular partitioning of the hypercube is possible, a

design can be generated by randomly taking a run from

each sub-region for each whole-plot. Figure 1 illustrates

the approach in two dimensions with 16 subplots in each of

the 10 whole plots. The design in Figure 1 has poor overall

uniformity, which can also be seen from WD-values being

12 to 51 times higher compared to a uniform design of the

same size.

A general method for generating the sub-regions is to

generate a uniform design of size k and use these points as

center points of k hypercubes or spheres that will constitute

the sub-regions. The subplots are then generated within

these sub-regions by either uniform designs or maxi-min

distance designs for which the minimum distance of two

runs in a sub-region is maximized. Figure 2 illustrates the

performance of these methods for five controllable and 40

environmental settings for two environmental factors. The

performance parameter in the figure is the WD-value for

2911

Page 4

Dehlendorff, Kulahci, and Andersen

0.00.2 0.4 0.60.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

X1

X2

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

Figure 1: A total design of size 160 settings in 16 regions

with 10 settings in each. Circles correspond to centers and

crosses to sample settings.

0.00.5 1.0 1.52.0

1

2

5

10

20

50

100

Ratio of minimum distance

Normalized WD

Uniform subdesign

Maximin subdesign

Maximin full design

Figure 2: Average WD-value normalized using the WD-

value obtained for a uniform design with 200 runs. Black

curve with marks is for the maximum design and the red

for the uniform design with dashed curves corresponding

to approximate 95 % confidence intervals, the bottom black

solid curve indicates a ratio of 1, i.e. no difference. The

black dotted curve corresponds to a maxi-min distance. The

overall design consists of 200 settings with the number of

environmental settings being 40.

the combined environmental factor design, normalized by

the WD-value of a uniform design of size 200. It can be

seen that, compared to a uniform design generated directly

for the same number of runs, both bottom-up methods are

significantlyworse. A maxi-mindesigngenerateddirectlyis

also seen to be better than the bottom-up generated designs.

Figure 2 illustrates that using a bottom-up approach does

not ensure an overall uniform design for the uncontrollable

factors.

3.2 Top-down Approach

The second method we propose has more of a ”top-down”

structure. First, we generate a uniform design of size N

which is equal to knc. This assures that the combineddesign

is indeed uniform. But this does not solve the problem of

assigning k settings to each of the ncwhole-plots such that

in each whole-plot the subplots are uniformly spaced.

Oneapproachtogeneratethedesignsis firsttoconstruct

ksub-regionsaroundk centers, whereeachregionconsistsof

ncpoints. A method to obtain such a structure is to generate

another uniform design of size k and use these points as

starting center points, c, in an optimization algorithm that

finds the optimal center points by minimizing

∑

j

min

i

||xj−ci||+k∑

i

(ni−nc)2

(2)

In the above expression, niis the number of points having

center i as the closest center. That is, the objective is to

choose the centers, c∗such that they minimize the sum of

the smallest differences between points and the centers, and

the deviations from the required size of the region. This

should ensure reasonably good separation of the points.

Based on the optimal centers, c∗, the N points need

to be assigned to a center such that all points are assigned

and all centers have exactly ncpoints. This can be done in

various ways, for example by assigning the point with the

smallest distance to its nearest center, or by assigning the

point with the largest second-shortest distance to its nearest

center, or by simply considering the points’ membership to

each center based on euclidean distances.

A result of assigning 400 points to 10 groups of 40

points each is shown on the left of Figure 3, where it can be

seen that the resulting groups are not well defined. Apply-

ing an exchange-algorithm on the assignment significantly

improves the assignment as seen on the right of Figure 3.

The total distances of the points to their center are reduced

by 5 % by swapping less than 20 points and the points

are grouped in well-defined clusters. An example in three

dimensions is shown in Figure 4. The grouping in Figure 4

is generated by applying the exchange algorithm to a com-

pletely random assignment leading to a 49 % improvement

2912

Page 5

Dehlendorff, Kulahci, and Andersen

0.00.2 0.4 0.60.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

X1

X2

1

2

3

4

5

6

7

8

9

10

0.00.2 0.40.60.81.0

0.0

0.2

0.4

0.6

0.8

1.0

X1

X2

Figure 3: Left: The optimal assignment corresponding to a membership assignment. Right: The assignment after swapping

in the optimal design.

in the distance of the points to the centers by more than

200 swaps.

3.2.1 Generating Whole Plots

After grouping the subplots in k groups, we generate the

whole-plots. Each whole-plot is assigned to one setting

from each of the k groups so that all settings are assigned.

One method is to assign the settings such that the maximum

WD-value of the sub-designs is minimized, which can be

obtainedbyrepeatedlyassigningthesettingsrandomlytothe

whole-plots until a certain degree of uniformity is obtained.

Another method is to move the small uniform design

of size k so that the point closest to the origin in the small

designisplacedatthepointsinthegroupclosesttotheorigin

and then assign points based on the smallest distance. The

advantage of this approach compared to random assignment

is that the whole-plot approximately mimics the uniform

design structure.

For the designs considered in Figure 3 and 4 the perfor-

mance of each whole-plot is compared to a uniform design

generated directly in Table 2. The table shows that the over-

all uniformity of the combined design cannot be fulfilled

withoutgettingsub-designsthat arenot completelyuniform.

The designs with lowest maximum relative WD-value all

have WD-values below 3.7 times and the highest minimum

WD-values are less than twice the reference designs.

Itcanbe seenfromTable 2that theresultsareconsistent

for up to 10 factors. The mean and the smallest maximum

WD-value are all decreasing, whereas the remaining values

are inconclusive with respect to the number of factors. It

can also be seen from Table 2 that a design, which ensures

Table 2: Summary for relative WD-values for 2, 3 and 4 di-

mensional examples with 40 controllable factors, each with

10 environmental settings (400) or 20 controllable factors,

each with 10 environmental settings (200). The perfor-

mance is summarized by minimum (Min), mean (Mean)

and maximum (Max) relative WD-value and by the highest

minimum (Max min) and lowest maximum (Min max). The

values are relative to the WD-value for a uniform design

of the same size as the whole-plots

Factors

2 (400)

3 (400)

4 (400)

10 (400)

2 (200)

3 (200)

4 (200)

10 (200)

Min

1.15

1.19

1.25

1.32

1.14

1.17

1.22

1.29

Max minMean

2.78

2.70

2.56

1.76

2.69

2.68

2.50

1.73

Min maxMax

8.39

7.21

7.28

2.38

7.20

6.98

5.65

2.45

1.99

1.93

1.94

1.60

2.17

2.21

2.22

1.63

3.67

3.47

3.20

2.00

2.94

2.94

2.54

1.78

relativeWD-valuesforall whole-plotsbetween2 (Maxmin)

and 3.7 (Min max) can be achieved for up to 10 factors. The

results seem to be independent of the number of settings but

with 10 factors generally giving significantly lower values.

This may be caused by the sparsity of the settings in the

10 dimensional design space.

4 DISCUSSION

This study was originated from application of discrete event

simulation and computer experimentation at a hospital unit.

2913