Designing simulation experiments with controllable and uncontrollable factors.
ABSTRACT In this study we propose a new method for designing com puter experiments inspired by the split plot designs used in physical experimentation. The basic layout is that each set of controllable factor settings corresponds to a whole plot for which a number of subplots, each corresponding to one combination of settings of the uncontrollable fac tors, is employed. The caveat is a desire that the subplots within each whole plot cover the design space uniformly. A further desire is that in the combined design, where all experimental runs are considered at once, the uniformity of the design space coverage should be guaranteed. Our proposed method allows for a large number of uncontrol lable and controllable settings to be run in a limited number of runs while uniformly covering the design space for the uncontrollable factors.

Conference Paper: Robust Simulationoptimization using Metamodels.
[Show abstract] [Hide abstract]
ABSTRACT: Optimization of simulated systems is the goal of many methods, but most methods assume known environments. In this paper we present a methodology that does account for uncertain environments. Our methodology uses Taguchi's view of the uncertain world, but replaces his statistical techniques by either Response Surface Methodology or Kriging metamodeling. We illustrate the resulting methodology through the wellknown Economic Order Quantity (EOQ) model.Proceedings of the 2009 Winter Simulation Conference, WSC 2009, Hilton Austin Hotel, Austin, TX, USA, December 1316, 2009; 01/2009
Page 1
DESIGNING SIMULATION EXPERIMENTS WITH
CONTROLLABLE AND UNCONTROLLABLE FACTORS
Christian Dehlendorff
Murat Kulahci
Klaus Kaae Andersen
Department of Informatics and Mathematical Modelling
Technical University of Denmark
Bygning 321, Richard Petersens Plads
Lyngby, DK2800, DENMARK
ABSTRACT
In this study we propose a new method for designing com
puter experiments inspired by the split plot designs used
in physical experimentation. The basic layout is that each
set of controllable factor settings corresponds to a whole
plot for which a number of subplots, each corresponding
to one combination of settings of the uncontrollable fac
tors, is employed. The caveat is a desire that the subplots
within each whole plot cover the design space uniformly.
A further desire is that in the combined design, where all
experimental runs are considered at once, the uniformity
of the design space coverage should be guaranteed. Our
proposed method allows for a large number of uncontrol
lable and controllable settings to be run in a limited number
of runs while uniformly covering the design space for the
uncontrollable factors.
1INTRODUCTION
With the current advances in computing technology, com
puter and simulation experiments are increasingly being
used to study complex systems for which physical experi
mentation is usually not feasible. Our case study involves
a discrete event simulation model of an orthopedic surgical
unit. The discrete event simulation (DES) model describes
the individual patient’s progress through the system and
has been developed in collaboration with medical staff at
Gentofte University Hospital in Copenhagen. The unit un
dertakes both acute and elective surgery and performs more
than 4,600 operative procedures a year. While the patients
come from various wards throughout the hospital, the main
sources of incoming patients are the four orthopedic wards
or the emergency care unit.
Thesimulationmodelis implementedinExtendversion
6 (Krahl 2002) on a Windows XP platform and controlled
from a Microsoft Excel spreadsheet with a Visual Basic for
application script. The model consists of 3 main modules:
The wards and arrival, the operating facilities, and the
recovery and discharge. Interaction with the surrounding
hospital is for example modeled with simplified processes
usingthe same resources as the processesin the surgicalunit
(occupyingthe resources) and with the patients entering and
exiting the model. Operating rooms, recovery beds, wards
andstaff are includedinthe model. Theaverageruntime for
simulating6 months(with oneweek ofwarmup)operations
is around 7 minutes. Typical outcomes are waiting times,
patient throughput and the amount of overtime.
The simulation model has two sources of noise coming
from variations in the uncontrollable factors (a.k.a. envi
ronmental factors in physical experimentation) and from
changes in the seed controlling the random number gen
eration process embedded in the simulation model. The
controllable factors are for example the number of op
erating rooms and the number of surgeons, whereas the
uncontrollable factors may include for example the arrival
rate of acute patients and the time required to clean the
operating rooms.
Inthis typeofapplication, severalissues needtobecon
sidered. First, the controllable factors tend to be numerous
and often discrete. Moreover a single experiment usually
takes several minutes to run. Therefore a simple exhaus
tive method, where all possible combinations of the factor
settings are considered, is often computationally infeasible
due to the exponentially increasing number of factor com
binations. Furthermore, the settings of the uncontrollable
factors, e.g. the acute patient arrival rate or the duration of
surgical procedures, are also of interest and must be deter
mined as they may influence the outcome of the simulations
and hence the robustness of the simulation analysis.
The paper is organized in the following manner: Sec
tion 2 introduces design of computer experiments and de
2909 9781424427086/08/$25.00 ©2008 IEEE
Proceedings of the 2008 Winter Simulation Conference
S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. W. Fowler eds.
Page 2
Dehlendorff, Kulahci, and Andersen
fines the performance measure for the designs. Section 3
describes the proposed design method and contrasts it with
other methods. In section 4 opportunitiesfor future research
are presented. Finally the main conclusions are summarized
in section 5.
2 DESIGN OF COMPUTER EXPERIMENTS
2.1 Literature Review
A general discussion on the issues regarding the de
sign and analysis of computer experiments can be found
in Sacks et al. (1989), Santner, Williams, and Notz (2003)
and Fang, Li, and Sudjianto (2006).
the computer experiments are often considered to come
from a deterministic computer code.
ments, the classical design of experiment methods such
as replication is deemed to be redundant as replica
tion of an experiment, for example, yields exactly the
same result (see Santner, Williams, and Notz (2003) and
Fang, Li, and Sudjianto (2006)).
Experiments based on a simulation model often involve
somestochastic component; makingthe outputalsostochas
tic. Kleijnen (2008) discusses the design and analysis of
simulation experiments which typically have some sort of
noise in the output. Therefore these experiments differ
from the deterministic computer experiments. Furthermore,
a typical simulation application will have both controllable
and uncontrollable (environmental) factors, which should
be handled differently. In these applications the aim is to
manipulate the controllable factors so that the system is
insensitive (robust) to changes in the uncontrollable factors.
As described by Kleijnen (2008) and Sanchez (2000) the
solution’s robustness needs to be considered in order to
obtain applicable solutions in systems with uncontrollable
factors. That is, a good solution needs to perform well over
the entire range of uncontrollable factors.
The original concept of robustness in physical systems
is often attributed to Taguchi (1987). Taguchi’s methods
involve an inner array for the controllable factors and an
outer array for the uncontrollable factors. In simulation
studies, Kleijnen (2008) suggests using a crossed design,
e.g. combining a central composite design (CCD) for the
controllable factors and a Latin Hypercube Design (LHD)
for the uncontrollable factors. In a crossed design the same
set of subplots is used for each whole plot. However, as we
will show in this study, this may not be the most efficient
way of running such experiments.
The outputs from
In such experi
2.2 Simulation Model
Our basis is a discrete event simulation model generating
output, y = f(xc,xe), for the settings for the sccontrollable
factors, xc, and the settings for the seuncontrollable factors,
xe. The objective is not only to select the settings, x∗
that the solution is robust to changes in the uncontrollable
factorsettingsasdescribedinp. 130134in Kleijnen (2008),
butalsotounderstandthevariationcomingfromthechanges
in the uncontrollable factor settings.
Since little prior knowledge of both controllable and
uncontrollable factors is available, we require that a good
design is simultaneously uniform over the design space of
thecontrollableanduncontrollablefactors. Inthefollowing,
wewillassumethattheuniformcoverageofthedesignspace
of the controllable factors is already achieved and that we
are only concerned with the uncontrollable factors.
Robustness studies in physical experimentation often
involve splitplot designs (Montgomery 2005).
therefore use similar terminology when robustness studies
are performed using computer experiments. In classic split
plot designs, a set of experiments called wholeplots is de
signedsothatforeachwholeplotanotherset ofexperiments
called subplots are run. In robustness studies, the settings
of the controllable factors often constitute the wholeplots,
whereas the settings of the uncontrollable factors constitute
the subplots. In Table 1, a wholeplot corresponds to a row
in which randomly selected combinations of settings for the
uncontrollable factors are run. It should be noted that the
randomization issue is irrelevant for computer experiments.
In the proposed method, each wholeplot corresponds
to one combination of settings of the controllable factors
(a row in Table 1), i.e. a total of ncwholeplots are needed
(nc= 5 in Table 1). Each subplot (a column entry in any
row in Table 1) corresponds to a combination of settings for
the uncontrollable factors with a total of k subplots for each
wholeplot. ThustheoveralldesignconsistsofN =nckruns.
In a crossed design as proposed by Kleijnen (2008) these
k subplots would be the same from one wholeplot to the
next. Therefore there will only be a total of k combinations
of settings for the uncontrollable factors. In our proposed
methodology, different k combinations of settings for the
uncontrollable factors will be used for each wholeplot.
This is expected to give better overall coverage of the
uncontrollable factor space compared to the crossed design.
The challenge with the proposed method is to make the
uncontrollable factor settings comparable from one whole
plot to the next.
c, such
We will
2.3 Measure of Uniformity
In order to evaluate the designs presented in the fol
lowing sections a measure of uniformity is needed.
Fang, Li, and Sudjianto (2006) summarize a set of perfor
mance measures frequently used for measuring the uni
formity of a design: the star discrepancy, centered dis
crepancy and the wraparound discrepancy.
tered and the wraparound discrepancy were proposed
by Hickernell (1998b) and Hickernell (1998a), respectively.
The cen
2910
Page 3
Dehlendorff, Kulahci, and Andersen
Table 1: Uncontrollable factor design for five controllable
settings and five environmental settings within each con
trollable setting
Controllable
setting
1
2
3
4
5
Environmental setting
23
xe2
xe3
xe7
xe8
xe12
xe13
xe17
xe18
xe22
xe23
145
xe1
xe6
xe11
xe16
xe21
xe4
xe9
xe14
xe19
xe24
xe5
xe10
xe15
xe20
xe25
Both have desirable properties. They are easy to compute,
invariant to permutations of factors or runs and rotation of
coordinates,andreliablemeasurementsfortheuniformityof
projections. However the wraparound discrepancy is said
to be unanchored (i.e. it only involves the design points),
while the centered discrepancy is not, since it involves the
corners of the unit cube.
In this study only the wraparound discrepancy is
considered as the measure of uniformity with a low
value corresponding to a high degree of uniformity. The
measure is chosen since the literature generally sug
gests it as a good measure of uniformity (see for ex
ample Fang and Ma (2001);
Fang, Li, and Sudjianto (2006)). The idea behindthis mea
sure is that for any two points from a uniform design, x1
and x2, spanning a hyper cube (potentially wrapping around
the bounds of the unit cube); the hypercube should contain
a fraction of the total number of points equal to the fraction
of total volume covered by the cube. An analytic expres
sion for the wraparound discrepancy (WD(D)) is given
by Fang and Ma (2001) as
Fang, Lin, and Liu (2003);
(WD(D))2=−?4
with di(j,k) =3
number of points, s the number of factors (the dimension),
and xkithe i’th coordinate of the k’th point.
There are various ways of constructing uniform de
signs. In this study the good lattice point method based
on the power generator is used with the modification
described in Fang, Li, and Sudjianto (2006). The design
construction is based on a lattice {1,...,n} and a gen
erator h(k) = (1,k,k2,...,ks−1)(mod n), with k fulfilling
that k,k2,...,ks−1(mod n) are distinct.
such that the resulting design consisting of the elements
uij= ih(k)j(mod n) scaled down to [0,1]shas the lowest
WDvalue.
3
?s+1
n
?3
2
?s+2
n2
n−1
∑
k=1
n
∑
j=k+1
s
∏
i=1
di(j,k) (1)
2−xki−xji(1−xki−xji), n being the
h(k) is chosen
3DESIGN ALGORITHM
Amethodforgeneratinggooddesignsforsimulationmodels
withbothcontrollableanduncontrollablefactorsispresented
inthefollowingsection. Hereweassumethatallfactorshave
been scaled to [0,1] and that the wraparound discrepancy
is the measure of uniformity. It is furthermore assumed
that a design for the controllable factors is available. That
is, we are primarily concerned with designing experiments
for the uncontrollable factors. Two and three dimensional
examples are used since they can be illustrated graphically.
However, the method is general and results for 4 and 10
factors are also presented.
3.1 Bottomup Approach
In section 2.2 the limitations of crossing a design for the
controllable factors with a design for the uncontrollable
factorsweredescribed. Abettermethodintermsofcovering
the uncontrollable factor space compared to the crossed
design is to generate different designs for the wholeplots,
each with k different combinations of uncontrollable factor
settings. This implies that nc designs of size k should be
constructed. For this method to succeed in the combined
design, not only sets of k subplots for different whole
plots should be comparable, but also nck subplots need
to cover the design space for the uncontrollable factors
uniformly. This can be achieved by dividing the design
hyperspace for the uncontrollable factors into k subregions
and sample ncsettings in each. As shown in Figure 1, this
can be achieved fairly easily in two dimensions. However,
in higher dimensions an efficient way of generating the
subregions is required since the curse of dimensionality
dictates that exponentially increasing numbers of runs have
to be used in higher dimensions to obtain the same density
of runs as in the lower dimensions.
If regular partitioning of the hypercube is possible, a
design can be generated by randomly taking a run from
each subregion for each wholeplot. Figure 1 illustrates
the approach in two dimensions with 16 subplots in each of
the 10 whole plots. The design in Figure 1 has poor overall
uniformity, which can also be seen from WDvalues being
12 to 51 times higher compared to a uniform design of the
same size.
A general method for generating the subregions is to
generate a uniform design of size k and use these points as
center points of k hypercubes or spheres that will constitute
the subregions. The subplots are then generated within
these subregions by either uniform designs or maximin
distance designs for which the minimum distance of two
runs in a subregion is maximized. Figure 2 illustrates the
performance of these methods for five controllable and 40
environmental settings for two environmental factors. The
performance parameter in the figure is the WDvalue for
2911
Page 4
Dehlendorff, Kulahci, and Andersen
0.0 0.20.40.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
X1
X2
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Figure 1: A total design of size 160 settings in 16 regions
with 10 settings in each. Circles correspond to centers and
crosses to sample settings.
0.00.51.01.52.0
1
2
5
10
20
50
100
Ratio of minimum distance
Normalized WD
Uniform subdesign
Maximin subdesign
Maximin full design
Figure 2: Average WDvalue normalized using the WD
value obtained for a uniform design with 200 runs. Black
curve with marks is for the maximum design and the red
for the uniform design with dashed curves corresponding
to approximate 95 % confidence intervals, the bottom black
solid curve indicates a ratio of 1, i.e. no difference. The
black dotted curve corresponds to a maximin distance. The
overall design consists of 200 settings with the number of
environmental settings being 40.
the combined environmental factor design, normalized by
the WDvalue of a uniform design of size 200. It can be
seen that, compared to a uniform design generated directly
for the same number of runs, both bottomup methods are
significantlyworse. A maximindesigngenerateddirectlyis
also seen to be better than the bottomup generated designs.
Figure 2 illustrates that using a bottomup approach does
not ensure an overall uniform design for the uncontrollable
factors.
3.2 Topdown Approach
The second method we propose has more of a ”topdown”
structure. First, we generate a uniform design of size N
which is equal to knc. This assures that the combineddesign
is indeed uniform. But this does not solve the problem of
assigning k settings to each of the ncwholeplots such that
in each wholeplot the subplots are uniformly spaced.
Oneapproachtogeneratethedesignsis firsttoconstruct
ksubregionsaroundk centers, whereeachregionconsistsof
ncpoints. A method to obtain such a structure is to generate
another uniform design of size k and use these points as
starting center points, c, in an optimization algorithm that
finds the optimal center points by minimizing
∑
j
min
i
xj−ci+k∑
i
(ni−nc)2
(2)
In the above expression, niis the number of points having
center i as the closest center. That is, the objective is to
choose the centers, c∗such that they minimize the sum of
the smallest differences between points and the centers, and
the deviations from the required size of the region. This
should ensure reasonably good separation of the points.
Based on the optimal centers, c∗, the N points need
to be assigned to a center such that all points are assigned
and all centers have exactly ncpoints. This can be done in
various ways, for example by assigning the point with the
smallest distance to its nearest center, or by assigning the
point with the largest secondshortest distance to its nearest
center, or by simply considering the points’ membership to
each center based on euclidean distances.
A result of assigning 400 points to 10 groups of 40
points each is shown on the left of Figure 3, where it can be
seen that the resulting groups are not well defined. Apply
ing an exchangealgorithm on the assignment significantly
improves the assignment as seen on the right of Figure 3.
The total distances of the points to their center are reduced
by 5 % by swapping less than 20 points and the points
are grouped in welldefined clusters. An example in three
dimensions is shown in Figure 4. The grouping in Figure 4
is generated by applying the exchange algorithm to a com
pletely random assignment leading to a 49 % improvement
2912
Page 5
Dehlendorff, Kulahci, and Andersen
0.00.2 0.40.60.81.0
0.0
0.2
0.4
0.6
0.8
1.0
X1
X2
1
2
3
4
5
6
7
8
9
10
0.00.20.40.60.81.0
0.0
0.2
0.4
0.6
0.8
1.0
X1
X2
Figure 3: Left: The optimal assignment corresponding to a membership assignment. Right: The assignment after swapping
in the optimal design.
in the distance of the points to the centers by more than
200 swaps.
3.2.1 Generating Whole Plots
After grouping the subplots in k groups, we generate the
wholeplots. Each wholeplot is assigned to one setting
from each of the k groups so that all settings are assigned.
One method is to assign the settings such that the maximum
WDvalue of the subdesigns is minimized, which can be
obtainedbyrepeatedlyassigningthesettingsrandomlytothe
wholeplots until a certain degree of uniformity is obtained.
Another method is to move the small uniform design
of size k so that the point closest to the origin in the small
designisplacedatthepointsinthegroupclosesttotheorigin
and then assign points based on the smallest distance. The
advantage of this approach compared to random assignment
is that the wholeplot approximately mimics the uniform
design structure.
For the designs considered in Figure 3 and 4 the perfor
mance of each wholeplot is compared to a uniform design
generated directly in Table 2. The table shows that the over
all uniformity of the combined design cannot be fulfilled
withoutgettingsubdesignsthat arenot completelyuniform.
The designs with lowest maximum relative WDvalue all
have WDvalues below 3.7 times and the highest minimum
WDvalues are less than twice the reference designs.
Itcanbe seenfromTable 2that theresultsareconsistent
for up to 10 factors. The mean and the smallest maximum
WDvalue are all decreasing, whereas the remaining values
are inconclusive with respect to the number of factors. It
can also be seen from Table 2 that a design, which ensures
Table 2: Summary for relative WDvalues for 2, 3 and 4 di
mensional examples with 40 controllable factors, each with
10 environmental settings (400) or 20 controllable factors,
each with 10 environmental settings (200). The perfor
mance is summarized by minimum (Min), mean (Mean)
and maximum (Max) relative WDvalue and by the highest
minimum (Max min) and lowest maximum (Min max). The
values are relative to the WDvalue for a uniform design
of the same size as the wholeplots
Factors
2 (400)
3 (400)
4 (400)
10 (400)
2 (200)
3 (200)
4 (200)
10 (200)
Min
1.15
1.19
1.25
1.32
1.14
1.17
1.22
1.29
Max minMean
2.78
2.70
2.56
1.76
2.69
2.68
2.50
1.73
Min max Max
8.39
7.21
7.28
2.38
7.20
6.98
5.65
2.45
1.99
1.93
1.94
1.60
2.17
2.21
2.22
1.63
3.67
3.47
3.20
2.00
2.94
2.94
2.54
1.78
relativeWDvaluesforall wholeplotsbetween2 (Maxmin)
and 3.7 (Min max) can be achieved for up to 10 factors. The
results seem to be independent of the number of settings but
with 10 factors generally giving significantly lower values.
This may be caused by the sparsity of the settings in the
10 dimensional design space.
4 DISCUSSION
This study was originated from application of discrete event
simulation and computer experimentation at a hospital unit.
2913
Similar Publications
Christian Dehlendorff 