Designing simulation experiments with controllable and uncontrollable factors.

Full text

Page 1

DESIGNING SIMULATION EXPERIMENTS WITH
CONTROLLABLE AND UNCONTROLLABLE FACTORS
Christian Dehlendorff
Murat Kulahci
Klaus Kaae Andersen
Department of Informatics and Mathematical Modelling
Technical University of Denmark
Bygning 321, Richard Petersens Plads
Lyngby, DK-2800, DENMARK
ABSTRACT
In this study we propose a new method for designing com-
puter experiments inspired by the split plot designs used
in physical experimentation. The basic layout is that each
set of controllable factor settings corresponds to a whole
plot for which a number of subplots, each corresponding
to one combination of settings of the uncontrollable fac-
tors, is employed. The caveat is a desire that the subplots
within each whole plot cover the design space uniformly.
A further desire is that in the combined design, where all
experimental runs are considered at once, the uniformity
of the design space coverage should be guaranteed. Our
proposed method allows for a large number of uncontrol-
lable and controllable settings to be run in a limited number
of runs while uniformly covering the design space for the
uncontrollable factors.
1INTRODUCTION
With the current advances in computing technology, com-
puter and simulation experiments are increasingly being
used to study complex systems for which physical experi-
mentation is usually not feasible. Our case study involves
a discrete event simulation model of an orthopedic surgical
unit. The discrete event simulation (DES) model describes
the individual patient’s progress through the system and
has been developed in collaboration with medical staff at
Gentofte University Hospital in Copenhagen. The unit un-
dertakes both acute and elective surgery and performs more
than 4,600 operative procedures a year. While the patients
come from various wards throughout the hospital, the main
sources of incoming patients are the four orthopedic wards
or the emergency care unit.
Thesimulationmodelis implementedinExtendversion
6 (Krahl 2002) on a Windows XP platform and controlled
from a Microsoft Excel spreadsheet with a Visual Basic for
application script. The model consists of 3 main modules:
The wards and arrival, the operating facilities, and the
recovery and discharge. Interaction with the surrounding
hospital is for example modeled with simplified processes
usingthe same resources as the processesin the surgicalunit
(occupyingthe resources) and with the patients entering and
exiting the model. Operating rooms, recovery beds, wards
andstaff are includedinthe model. Theaverageruntime for
simulating6 months(with oneweek ofwarm-up)operations
is around 7 minutes. Typical outcomes are waiting times,
patient throughput and the amount of overtime.
The simulation model has two sources of noise coming
from variations in the uncontrollable factors (a.k.a. envi-
ronmental factors in physical experimentation) and from
changes in the seed controlling the random number gen-
eration process embedded in the simulation model. The
controllable factors are for example the number of op-
erating rooms and the number of surgeons, whereas the
uncontrollable factors may include for example the arrival
rate of acute patients and the time required to clean the
operating rooms.
Inthis typeofapplication, severalissues needtobecon-
sidered. First, the controllable factors tend to be numerous
and often discrete. Moreover a single experiment usually
takes several minutes to run. Therefore a simple exhaus-
tive method, where all possible combinations of the factor
settings are considered, is often computationally infeasible
due to the exponentially increasing number of factor com-
binations. Furthermore, the settings of the uncontrollable
factors, e.g. the acute patient arrival rate or the duration of
surgical procedures, are also of interest and must be deter-
mined as they may influence the outcome of the simulations
and hence the robustness of the simulation analysis.
The paper is organized in the following manner: Sec-
tion 2 introduces design of computer experiments and de-
2909 978-1-4244-2708-6/08/$25.00 ©2008 IEEE
Proceedings of the 2008 Winter Simulation Conference
S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. W. Fowler eds.

Page 2

Dehlendorff, Kulahci, and Andersen
fines the performance measure for the designs. Section 3
describes the proposed design method and contrasts it with
other methods. In section 4 opportunitiesfor future research
are presented. Finally the main conclusions are summarized
in section 5.
2 DESIGN OF COMPUTER EXPERIMENTS
2.1 Literature Review
A general discussion on the issues regarding the de-
sign and analysis of computer experiments can be found
in Sacks et al. (1989), Santner, Williams, and Notz (2003)
and Fang, Li, and Sudjianto (2006).
the computer experiments are often considered to come
from a deterministic computer code.
ments, the classical design of experiment methods such
as replication is deemed to be redundant as replica-
tion of an experiment, for example, yields exactly the
same result (see Santner, Williams, and Notz (2003) and
Fang, Li, and Sudjianto (2006)).
Experiments based on a simulation model often involve
somestochastic component; makingthe outputalsostochas-
tic. Kleijnen (2008) discusses the design and analysis of
simulation experiments which typically have some sort of
noise in the output. Therefore these experiments differ
from the deterministic computer experiments. Furthermore,
a typical simulation application will have both controllable
and uncontrollable (environmental) factors, which should
be handled differently. In these applications the aim is to
manipulate the controllable factors so that the system is
insensitive (robust) to changes in the uncontrollable factors.
As described by Kleijnen (2008) and Sanchez (2000) the
solution’s robustness needs to be considered in order to
obtain applicable solutions in systems with uncontrollable
factors. That is, a good solution needs to perform well over
the entire range of uncontrollable factors.
The original concept of robustness in physical systems
is often attributed to Taguchi (1987). Taguchi’s methods
involve an inner array for the controllable factors and an
outer array for the uncontrollable factors. In simulation
studies, Kleijnen (2008) suggests using a crossed design,
e.g. combining a central composite design (CCD) for the
controllable factors and a Latin Hypercube Design (LHD)
for the uncontrollable factors. In a crossed design the same
set of subplots is used for each whole plot. However, as we
will show in this study, this may not be the most efficient
way of running such experiments.
The outputs from
In such experi-
2.2 Simulation Model
Our basis is a discrete event simulation model generating
output, y = f(xc,xe), for the settings for the sccontrollable
factors, xc, and the settings for the seuncontrollable factors,
xe. The objective is not only to select the settings, x∗
that the solution is robust to changes in the uncontrollable
factorsettingsasdescribedinp. 130-134in Kleijnen (2008),
butalsotounderstandthevariationcomingfromthechanges
in the uncontrollable factor settings.
Since little prior knowledge of both controllable and
uncontrollable factors is available, we require that a good
design is simultaneously uniform over the design space of
thecontrollableanduncontrollablefactors. Inthefollowing,
wewillassumethattheuniformcoverageofthedesignspace
of the controllable factors is already achieved and that we
are only concerned with the uncontrollable factors.
Robustness studies in physical experimentation often
involve split-plot designs (Montgomery 2005).
therefore use similar terminology when robustness studies
are performed using computer experiments. In classic split-
plot designs, a set of experiments called whole-plots is de-
signedsothatforeachwhole-plotanotherset ofexperiments
called subplots are run. In robustness studies, the settings
of the controllable factors often constitute the whole-plots,
whereas the settings of the uncontrollable factors constitute
the subplots. In Table 1, a whole-plot corresponds to a row
in which randomly selected combinations of settings for the
uncontrollable factors are run. It should be noted that the
randomization issue is irrelevant for computer experiments.
In the proposed method, each whole-plot corresponds
to one combination of settings of the controllable factors
(a row in Table 1), i.e. a total of ncwhole-plots are needed
(nc= 5 in Table 1). Each subplot (a column entry in any
row in Table 1) corresponds to a combination of settings for
the uncontrollable factors with a total of k subplots for each
whole-plot. ThustheoveralldesignconsistsofN =nckruns.
In a crossed design as proposed by Kleijnen (2008) these
k subplots would be the same from one whole-plot to the
next. Therefore there will only be a total of k combinations
of settings for the uncontrollable factors. In our proposed
methodology, different k combinations of settings for the
uncontrollable factors will be used for each whole-plot.
This is expected to give better overall coverage of the
uncontrollable factor space compared to the crossed design.
The challenge with the proposed method is to make the
uncontrollable factor settings comparable from one whole-
plot to the next.
c, such
We will
2.3 Measure of Uniformity
In order to evaluate the designs presented in the fol-
lowing sections a measure of uniformity is needed.
Fang, Li, and Sudjianto (2006) summarize a set of perfor-
mance measures frequently used for measuring the uni-
formity of a design: the star discrepancy, centered dis-
crepancy and the wrap-around discrepancy.
tered and the wrap-around discrepancy were proposed
by Hickernell (1998b) and Hickernell (1998a), respectively.
The cen-
2910

Page 3

Dehlendorff, Kulahci, and Andersen
Table 1: Uncontrollable factor design for five controllable
settings and five environmental settings within each con-
trollable setting
Controllable
setting
1
2
3
4
5
Environmental setting
23
xe2
xe3
xe7
xe8
xe12
xe13
xe17
xe18
xe22
xe23
145
xe1
xe6
xe11
xe16
xe21
xe4
xe9
xe14
xe19
xe24
xe5
xe10
xe15
xe20
xe25
Both have desirable properties. They are easy to compute,
invariant to permutations of factors or runs and rotation of
coordinates,andreliablemeasurementsfortheuniformityof
projections. However the wrap-around discrepancy is said
to be unanchored (i.e. it only involves the design points),
while the centered discrepancy is not, since it involves the
corners of the unit cube.
In this study only the wrap-around discrepancy is
considered as the measure of uniformity with a low
value corresponding to a high degree of uniformity. The
measure is chosen since the literature generally sug-
gests it as a good measure of uniformity (see for ex-
ample Fang and Ma (2001);
Fang, Li, and Sudjianto (2006)). The idea behindthis mea-
sure is that for any two points from a uniform design, x1
and x2, spanning a hyper cube (potentially wrapping around
the bounds of the unit cube); the hypercube should contain
a fraction of the total number of points equal to the fraction
of total volume covered by the cube. An analytic expres-
sion for the wrap-around discrepancy (WD(D)) is given
by Fang and Ma (2001) as
Fang, Lin, and Liu (2003);
(WD(D))2=−?4
with di(j,k) =3
number of points, s the number of factors (the dimension),
and xkithe i’th coordinate of the k’th point.
There are various ways of constructing uniform de-
signs. In this study the good lattice point method based
on the power generator is used with the modification
described in Fang, Li, and Sudjianto (2006). The design
construction is based on a lattice {1,...,n} and a gen-
erator h(k) = (1,k,k2,...,ks−1)(mod n), with k fulfilling
that k,k2,...,ks−1(mod n) are distinct.
such that the resulting design consisting of the elements
uij= ih(k)j(mod n) scaled down to [0,1]shas the lowest
WD-value.
3
?s+1
n
?3
2
?s+2
n2
n−1
∑
k=1
n
∑
j=k+1
s
∏
i=1
di(j,k) (1)
2−|xki−xji|(1−|xki−xji|), n being the
h(k) is chosen
3DESIGN ALGORITHM
Amethodforgeneratinggooddesignsforsimulationmodels
withbothcontrollableanduncontrollablefactorsispresented
inthefollowingsection. Hereweassumethatallfactorshave
been scaled to [0,1] and that the wrap-around discrepancy
is the measure of uniformity. It is furthermore assumed
that a design for the controllable factors is available. That
is, we are primarily concerned with designing experiments
for the uncontrollable factors. Two and three dimensional
examples are used since they can be illustrated graphically.
However, the method is general and results for 4 and 10
factors are also presented.
3.1 Bottom-up Approach
In section 2.2 the limitations of crossing a design for the
controllable factors with a design for the uncontrollable
factorsweredescribed. Abettermethodintermsofcovering
the uncontrollable factor space compared to the crossed
design is to generate different designs for the whole-plots,
each with k different combinations of uncontrollable factor
settings. This implies that nc designs of size k should be
constructed. For this method to succeed in the combined
design, not only sets of k subplots for different whole-
plots should be comparable, but also nck subplots need
to cover the design space for the uncontrollable factors
uniformly. This can be achieved by dividing the design
hyperspace for the uncontrollable factors into k sub-regions
and sample ncsettings in each. As shown in Figure 1, this
can be achieved fairly easily in two dimensions. However,
in higher dimensions an efficient way of generating the
sub-regions is required since the curse of dimensionality
dictates that exponentially increasing numbers of runs have
to be used in higher dimensions to obtain the same density
of runs as in the lower dimensions.
If regular partitioning of the hypercube is possible, a
design can be generated by randomly taking a run from
each sub-region for each whole-plot. Figure 1 illustrates
the approach in two dimensions with 16 subplots in each of
the 10 whole plots. The design in Figure 1 has poor overall
uniformity, which can also be seen from WD-values being
12 to 51 times higher compared to a uniform design of the
same size.
A general method for generating the sub-regions is to
generate a uniform design of size k and use these points as
center points of k hypercubes or spheres that will constitute
the sub-regions. The subplots are then generated within
these sub-regions by either uniform designs or maxi-min
distance designs for which the minimum distance of two
runs in a sub-region is maximized. Figure 2 illustrates the
performance of these methods for five controllable and 40
environmental settings for two environmental factors. The
performance parameter in the figure is the WD-value for
2911

Page 4

Dehlendorff, Kulahci, and Andersen
0.0 0.20.40.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
X1
X2
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Figure 1: A total design of size 160 settings in 16 regions
with 10 settings in each. Circles correspond to centers and
crosses to sample settings.
0.00.51.01.52.0
1
2
5
10
20
50
100
Ratio of minimum distance
Normalized WD
Uniform subdesign
Maximin subdesign
Maximin full design
Figure 2: Average WD-value normalized using the WD-
value obtained for a uniform design with 200 runs. Black
curve with marks is for the maximum design and the red
for the uniform design with dashed curves corresponding
to approximate 95 % confidence intervals, the bottom black
solid curve indicates a ratio of 1, i.e. no difference. The
black dotted curve corresponds to a maxi-min distance. The
overall design consists of 200 settings with the number of
environmental settings being 40.
the combined environmental factor design, normalized by
the WD-value of a uniform design of size 200. It can be
seen that, compared to a uniform design generated directly
for the same number of runs, both bottom-up methods are
significantlyworse. A maxi-mindesigngenerateddirectlyis
also seen to be better than the bottom-up generated designs.
Figure 2 illustrates that using a bottom-up approach does
not ensure an overall uniform design for the uncontrollable
factors.
3.2 Top-down Approach
The second method we propose has more of a ”top-down”
structure. First, we generate a uniform design of size N
which is equal to knc. This assures that the combineddesign
is indeed uniform. But this does not solve the problem of
assigning k settings to each of the ncwhole-plots such that
in each whole-plot the subplots are uniformly spaced.
Oneapproachtogeneratethedesignsis firsttoconstruct
ksub-regionsaroundk centers, whereeachregionconsistsof
ncpoints. A method to obtain such a structure is to generate
another uniform design of size k and use these points as
starting center points, c, in an optimization algorithm that
finds the optimal center points by minimizing
∑
j
min
i
||xj−ci||+k∑
i
(ni−nc)2
(2)
In the above expression, niis the number of points having
center i as the closest center. That is, the objective is to
choose the centers, c∗such that they minimize the sum of
the smallest differences between points and the centers, and
the deviations from the required size of the region. This
should ensure reasonably good separation of the points.
Based on the optimal centers, c∗, the N points need
to be assigned to a center such that all points are assigned
and all centers have exactly ncpoints. This can be done in
various ways, for example by assigning the point with the
smallest distance to its nearest center, or by assigning the
point with the largest second-shortest distance to its nearest
center, or by simply considering the points’ membership to
each center based on euclidean distances.
A result of assigning 400 points to 10 groups of 40
points each is shown on the left of Figure 3, where it can be
seen that the resulting groups are not well defined. Apply-
ing an exchange-algorithm on the assignment significantly
improves the assignment as seen on the right of Figure 3.
The total distances of the points to their center are reduced
by 5 % by swapping less than 20 points and the points
are grouped in well-defined clusters. An example in three
dimensions is shown in Figure 4. The grouping in Figure 4
is generated by applying the exchange algorithm to a com-
pletely random assignment leading to a 49 % improvement
2912

Page 5

Dehlendorff, Kulahci, and Andersen
0.00.2 0.40.60.81.0
0.0
0.2
0.4
0.6
0.8
1.0
X1
X2
1
2
3
4
5
6
7
8
9
10
0.00.20.40.60.81.0
0.0
0.2
0.4
0.6
0.8
1.0
X1
X2
Figure 3: Left: The optimal assignment corresponding to a membership assignment. Right: The assignment after swapping
in the optimal design.
in the distance of the points to the centers by more than
200 swaps.
3.2.1 Generating Whole Plots
After grouping the subplots in k groups, we generate the
whole-plots. Each whole-plot is assigned to one setting
from each of the k groups so that all settings are assigned.
One method is to assign the settings such that the maximum
WD-value of the sub-designs is minimized, which can be
obtainedbyrepeatedlyassigningthesettingsrandomlytothe
whole-plots until a certain degree of uniformity is obtained.
Another method is to move the small uniform design
of size k so that the point closest to the origin in the small
designisplacedatthepointsinthegroupclosesttotheorigin
and then assign points based on the smallest distance. The
advantage of this approach compared to random assignment
is that the whole-plot approximately mimics the uniform
design structure.
For the designs considered in Figure 3 and 4 the perfor-
mance of each whole-plot is compared to a uniform design
generated directly in Table 2. The table shows that the over-
all uniformity of the combined design cannot be fulfilled
withoutgettingsub-designsthat arenot completelyuniform.
The designs with lowest maximum relative WD-value all
have WD-values below 3.7 times and the highest minimum
WD-values are less than twice the reference designs.
Itcanbe seenfromTable 2that theresultsareconsistent
for up to 10 factors. The mean and the smallest maximum
WD-value are all decreasing, whereas the remaining values
are inconclusive with respect to the number of factors. It
can also be seen from Table 2 that a design, which ensures
Table 2: Summary for relative WD-values for 2, 3 and 4 di-
mensional examples with 40 controllable factors, each with
10 environmental settings (400) or 20 controllable factors,
each with 10 environmental settings (200). The perfor-
mance is summarized by minimum (Min), mean (Mean)
and maximum (Max) relative WD-value and by the highest
minimum (Max min) and lowest maximum (Min max). The
values are relative to the WD-value for a uniform design
of the same size as the whole-plots
Factors
2 (400)
3 (400)
4 (400)
10 (400)
2 (200)
3 (200)
4 (200)
10 (200)
Min
1.15
1.19
1.25
1.32
1.14
1.17
1.22
1.29
Max minMean
2.78
2.70
2.56
1.76
2.69
2.68
2.50
1.73
Min max Max
8.39
7.21
7.28
2.38
7.20
6.98
5.65
2.45
1.99
1.93
1.94
1.60
2.17
2.21
2.22
1.63
3.67
3.47
3.20
2.00
2.94
2.94
2.54
1.78
relativeWD-valuesforall whole-plotsbetween2 (Maxmin)
and 3.7 (Min max) can be achieved for up to 10 factors. The
results seem to be independent of the number of settings but
with 10 factors generally giving significantly lower values.
This may be caused by the sparsity of the settings in the
10 dimensional design space.
4 DISCUSSION
This study was originated from application of discrete event
simulation and computer experimentation at a hospital unit.
2913