Advancing Simulation Experimentation Capabilities with Runtime Interventions

Abstract

Experimentation is a critical capability of simulations that allows one to test different scenarios safely and cost-effectively. In particular, agent-based simulations have been used in experimenting with different policy options to aid decision makers. Highly utilized experimentation methods such as parameter sweeping aim to explore the relationship between the initial parameter values (i.e., input) and simulation results (i.e., outputs). Experimentation, which involves changes of simulation states on-the-fly, is often conducted ad-hoc and entails manual code adjustments which are time consuming and error-prone. In this paper, we present a framework that facilitates intervening in a running simulation to change simulation states in a semi-automated manner so that a simulation user can explore alternative worlds. In our framework, such an intervention is implemented using an injection mechanism. The framework allows the user to weigh different policy options rapidly with minimal effort. We illustrate its use in an urban agent-based model.

Full text

ADVANCING SIMULATION EXPERIMENTATION CAPABILITIES
WITH RUNTIME INTERVENTIONS
Joon-Seok Kim
Hamdi Kavak
Dept. of Geography & Geoinformation Science
George Mason University
4400 University Drive
Fairfax, VA, USA
{jkim258, hkavak}@gmu.edu
Umar Manzoor
Dept. of Computer Science
Tulane University
6823 St. Charles Ave.
New Orleans, LA, USA
umanzoor@tulane.edu
Andreas Züfle
Dept. of Geography & Geoinformation Science
George Mason University
4400 University Drive
Fairfax, VA, USA
azufle@gmu.edu
SpringSim-ANSS, 2019 April 29-May 2, Tucson, AZ, USA; c
⃝2019 Society for Modeling & Simulation International (SCS)
ABSTRACT
Experimentation is a critical capability of simulations that allows one to test different scenarios safely and
cost-effectively. In particular, agent-based simulations have been used in experimenting with different policy
options to aid decision makers. Highly utilized experimentation methods such as parameter sweeping aim
to explore the relationship between the initial parameter values (i.e., input) and simulation results (i.e.,
outputs). Experimentation, which involves changes of simulation states on-the-fly, is often conducted ad-
hoc and entails manual code adjustments which are time consuming and error-prone. In this paper, we
present a framework that facilitates intervening in a running simulation to change simulation states in a
semi-automated manner so that a simulation user can explore alternative worlds. In our framework, such
an intervention is implemented using an injection mechanism. The framework allows the user to weigh
different policy options rapidly with minimal effort. We illustrate its use in an urban agent-based model.
Keywords: injection, intervention, checkpoint, experimentation, what-if analysis
1 INTRODUCTION
One of the grand challenges of social scientists is the lack of objective knowledge of the actual causes
of observed behaviors in the real world. Conducting the necessary experimental work for understanding
causality in social behaviors and systems is often impractical or unethical, while observational modeling,
data mining and machine learning approaches struggle to infer causality from correlation patterns. Planners
and decision-makers often rely upon agent-based simulation to help them understand and forecast a variety
of scenarios that involve complex human social systems and behaviors. In particular, decision-makers often

Kim, Kavak, Manzoor and Züfle
seek to identify, characterize, and model causal processes at different scales and for different social systems
to help explain or predict certain patterns of behavior for a wide range of applications, including disease
spread (Carley et al. 2006, Perez and Dragicevic 2009, Crooks and Hailegiorgis 2014), crime and riots
(Malleson, Heppenstall, and See 2010, Pires and Crooks 2017), and economic systems (Farmer and Foley
2009).
An agent-based model (ABM) is a computerized simulation of individual entities called agents, which in-
teract through prescribed rules in a well-defined environment. The goal in designing an ABM is to abstract
complex real-world human behavior into a finite set of realistic rules leading to a system of socially plau-
sible behavior. This goal can be achieved through careful validation and verification of the model, using
sensitivity analysis (Kleijnen 2005) and model calibration (Lamperti, Roventini, and Sani 2018).
Once an ABM has been implemented, validated, verified and calibrated for a particular application, it can be
used to experimentally evaluate new policies, unexpected events, and disruptive changes to the simulation
environment. For instance, in an economic system ABM new economic policies can be employed. Such
“what-if” scenario explorations allow us to observe outcomes of policy changes in settings that are too com-
plex to describe by analytic models. What-if analysis is a crucial process used to diagnose a model, analyze
its features, and pinpoint a circumstance for better decision making. In general terms, such scenario explo-
rations are realized by (1) changing the initial conditions of the simulation and (2) making changes during
runtime. While both allow realizing what-if analyses, their implementation and exploration capabilities are
vastly different.
Changing the initial conditions of a simulation, also known as initialization, is a more common approach
to scenario explorations. Grow (2017) proposed the use of a regression metamodel to facilitate the under-
standing of the complex behaviors in agent-based simulations. This type of metamodel is quite effective for
computational demography as it is highly accessible and easy to communicate. Thiele, Kurth, and Grimm
(2014) proposed the RNetLogo package by linking NetLogo (widely used toolkit for agent-based modeling)
and R (statistical analysis tool) to use the established methods to develop tools to facilitate parameter estima-
tion and sensitivity analysis in agent-based models. Concurrently, Ligmann-Zielinska et al. (2014) proposed
a simulation framework based on quantitative uncertainty and sensitivity analyses to develop parsimonious
socio-ecological agent-based models to understand model behavior and explore the outcome space.
Further, many popular agent-based modeling platforms, such as NetLogo and AnyLogic, provide standard-
ized ways for users to specify parameter value combinations and replications. This allows users to run
multiple scenario alternatives with minimal effort. On the other hand, such runs can only support narrow
explorations due to limiting the changes at the initialization level.
Researchers developed alternative mechanisms to increase experimentation capabilities of simulations. One
popular technique for runtime explorations is called “cloning” (Li, Cai, and Turner 2017). Simply put,
cloning makes a copy of a simulation while it’s running and allows the cloned instance to run semi-
independently (Hybinette and Fujimoto 1997). Majority of contributions in this line of research come from
parallel simulation community that aims to create massive-scale models and explore runtime changes (Hybi-
nette and Fujimoto 1997, Hybinette and Fujimoto 2001, Yoginath and Perumalla 2018). While these studies
provide efficient ways to share computational resources with cloned instances, they heavily rely on running
simulations on high-performance computing infrastructures (Yoginath and Perumalla 2018). Many ABM
platforms and open source tools do not support high-performance computing infrastructures making them
inaccessible to a wider community of simulation modelers. That is where our study comes into play.
In this work, we propose a framework to prescribe changes into a running ABM to support semi-automated
“what-if” analyses. Our framework provides an interface for existing ABMs, which allows a user to define
checkpoints in a running simulation. At these checkpoints, changes to model parameters can be prescribed,
and the checkpoint can be loaded at a later time to compare outcomes of different strategies and policies

Kim, Kavak, Manzoor and Züfle
in alternative simulation time-lines originating from the same checkpoint differing merely in the prescribed
changes. Our framework allows to define conditions for simulation checkpoints (such as the occurrence
of a disaster or the observed outbreak of a disease), and allows to define multiple policies to intervene the
running simulation (such as defining different solutions to prevent the spread of a disease), which may be
defined by a human expert based on information available at the time of the simulation checkpoint. Once
defined, the prescribed changes are automatically injected into the simulation. In addition to our injection
mechanism, we describe our injection builder for user-friendly analysis of the current simulation state and
for allowing the user to register prescriptions to the model and their conditions. We further demonstrate
the effectiveness of our framework using a practical use-case based on an ABM that simulates patterns of
life in an urban environment to study the spread of influenza-like disease. We evaluate various intervention
scenarios based on vaccinations, closures of public places, and employment of a stay-home-order that we
inject into our simulation.
2 FRAMEWORK FOR SIMULATION EXPERIMENTATION
Experimentation is a significant process to collect scientific evidence to support a hypothesis. It requires,
however, tremendous time and effort to not only 1) conduct experiments themselves but also 2) prepare
them. This section introduces a framework for simulation experimentation, focusing on how to expedite
the process of experimentation. Fig. 1 shows our proposed process of experimentation. It is an iterative
process of simulation of a model, intervention, and examination. Note that the model in Fig. 1 is a tested
and certified computational model that went through the validation and verification process; thus no change
of model is needed. In this paper, intervention denotes an intended action to intervene between simulation
and an analyst to achieve the altered outcome in hopes of resolving scientific questions. To decide whether
an analyst will proceed more experimentation is required after examination of collected results. In what
follows, the main loop is explained in detail.
Main instance
SimulatorModel
Inspector
Injection
builder
Predicates
Intervened
instancen
Result
ExaminationInterventions
instructions
Resultn
Reporting
Figure 1: Process of experimentation.
2.1 Instruction Injection
Dependency injection is a software design pattern that allows modules to be loosely coupled (Prasanna
2009). It enables a client to configure behaviors without explicit code changes, providing the flexibility
of configuration. Similarly, we adopt an injection mechanism as the implementation of interventions. We
interpret the content of injection as an instruction, a procedure that may affect simulation and produce
different results of the simulation. Instruction injection is an act to inject instructions into the simulation.

Kim, Kavak, Manzoor and Züfle
It allows an analyst to do experimentation without altering models and simulators at the code level. This
mechanism facilitates what-if analysis with less effort so that one can focus on their experimentation and
analysis. From an analyst’s perspective, to temporarily close public places to see how the decision impacts
simulation is an example of interventions (see Section 3).
2.2 Injection Builder
From an implementation perspective, an instruction manipulates values and invokes a series of functions
dynamically. To inject instructions into a running simulation, it is necessary to define an interface or injection
point. Namely, instruction injection requires a pair of an instruction and a point when the instruction is
triggered. This point can be a specific simulation time or a transition between states depending on paradigms
in simulation modeling. Fig. 2 illustrates an injection point between stand st+1and the differentiation of
simulation occurs from st+1with intervention δ.
St
δ
⋅
St+1
St+1
δ
⋅
St+n
St+n
St- 1 …
…
δ
Figure 2: Influence of intervention.
It is paramount to minimize the effort of an analyst. For this purpose, we have designed an injection builder
which is a tool that facilitates the creation of instruction injection in a semi-automated or automated way.
In the initial phase of simulation, an analyst might not recognize phenomena interesting to them unless they
see tangible results. Once they identify their targets to do experimentation, a task to define predicates of
interest is given. A predicate is a description of the condition to produce injection points. For instance, to
check if the number of infected agents in a disease spread ABM is greater than five percent of the population
can be a predicate. Our inspector module enables users to interact with simulation and carry out this task.
We note that for the task no change in code level is involved. If an analyst is aware of targets and injection
points beforehand, the inspection process is not required. We formally define an intervention as follows.
Definition 1 (Intervention).Intervention δis a pair of (t,λ), where t is an injection time and λis a set of
instructions.
2.3 Implementation
This subsection addresses the implementation of our framework including the inspector, injection builder,
and intervention agent (see Fig. 3). We implemented the framework based on MASON (Luke, Cioffi-Revilla,
Panait, Sullivan, and Balan 2005), a multiagent simulation library core in Java. Although MASON provides
inspectors that inspect properties, they are not designed to evaluate predicates during runtime. Thus, we
developed modules to dynamically evaluate predicates given by users. There is a trade-off between flexibility
and performance of dynamic evaluation. That is, dynamic evaluation allows to check undefined predicates,
but it entails subsidiary operations such as parsing predicates and searching methods. Therefore, handling
processed data such as statistical data at the code level is preferable to specifying detailed predicates. The
augmented inspectors are helpful when an analyst wants to pause simulation at the time when user-defined
conditions are met. In this setting, inspectors are used to pinpointing an injection time. If an analyst is aware
of injection timing, they can create intervention using the injection builder without explicit inspection. It

Kim, Kavak, Manzoor and Züfle
also allows creating a checkpoint of simulation on the fly. A checkpoint is used to restore the simulation
to the checkpoint for analysis of alternative timelines. Moreover, it expedites the experimentation process
because it is not required to simulate from the beginning.
Agent
Scheduler Intervention
Agent
Checkpoint
Inspector
Processed
Data inspect
push
Intervention
build
load
inject
Simulator load
run
step
Injection
Builder
push
create
Figure 3: Conceptual architecture of framework.
The injection builder is a library that facilitates performing instructions, namely invoking public methods
defined by users. It enables to generate various combinatorial test cases. As a result of building, the injec-
tion builder produces an intervention file that can be loaded from a simulator. The file contains intervention
information such as time, access methods, and parameters corresponding to each method. To implement the
injection mechanism, we introduced an intervention agent which is an agent to maintain schedules of inter-
ventions without disturbing user-defined agents. Restoring simulation from a checkpoint, the intervention
agent loads intervention files and schedules instructions.
3 USE CASE
To illustrate the usability of our framework, we created an agent-based urban model that simulates the
patterns of life with the goal to investigate the spread of influenza-like diseases. In this model, agents are
situated on a synthetic geographic area created based on procedural city generation techniques (Kim, Kavak,
and Crooks 2018). When it comes to behaviors, agents follow simple routines such as eating, working,
commuting, and visiting recreational facilities. In the real-world as well as in our simulated world, such
facilities serve as the hub for spreading contact/proximity-based diseases. Thus, disease spread is a useful
example to illustrate the important capabilities of our framework.
Disease models have been heavily investigated using compartmental mathematical models. In these models,
a population is divided into compartments based on disease stages, each compartment is then represented
as a state, and changes in these states are captured using simple rate-based differential equations. However,
such an approach assumes that all members of a compartment are homogeneous in their characteristics and
their spatial locations are not captured. This makes it challenging to represent populations realistically and
to develop reliable intervention scenarios. Agent-based disease models have the potential to capture the
heterogeneity of populations thanks to its bottom-up modeling approach. Such models have been used for
many critical purposes including public health policy development (Apolloni et al. 2009, Kumar et al. 2013),
investigation of biological virus evolution (Roche, Drake, and Rohani 2011), and forecasting (Nsoesie et al.
2013), etc. Here our purpose is not to create a novel disease model, but to show the applicability of our
framework in a critical domain.

Kim, Kavak, Manzoor and Züfle
S E I R
𝛽𝜎 𝛾
Between-host transmission
Within-host progression
Figure 4: A depiction of a disease transmission process with transition rate parameters.
3.1 Disease Model
We based our disease model on a well-known Susceptible-Exposed-Infectious-Recovered (SEIR) approach
where each of these letters represents different stages of an epidemic disease (Riley 2007). Here, Sindicates
that the person is Susceptible to infection but not infected yet, Erepresents a latent state for disease where
the person is Exposed to the disease but not infectious yet, Irepresents the disease state where the person
is Infectious and can also infect other people, and finally Rrepresents Recovered, the disease stage when
the person is immune to the disease either through vaccine or recovery. Following Barrett et al. 2008, we
constructed two stages of disease model: within-host progression and between-host transmission. A simple
depiction of this model is illustrated in Fig. 4.
The within-host progression part of the model deals with the life-cycle of the disease within an individual. In
other words, disease progression is represented as a finite state machine with specific transition probabilities.
All individuals are assumed to be starting with Susceptible(S, i.e., prone to getting infected). Unless an
individual gets the virus, Sdisease stage stays the same. Once the virus is in the body, the disease stage
immediately becomes Exposed (E) and stays like that for a certain time which is captured using the transition
rate σ. Then, the disease progresses to become Infectious (I). Similarly, this stage continues for a certain
time and the person becomes Recovered (R) which is captured using the transition rate γ. We assume that
people who are Recovered will be immune to the disease regardless of their future contacts with infectious
individuals.
The between-host transmission part of the disease model deals with the infection of a susceptible individual
through meeting with at least one Infectious (I) individual. Such meetings have the probability of trans-
mitting the virus from infectious individuals to susceptible individuals through contact or being at close
proximity. Following Barrett et al. 2008, we use the between-host transmission probability formulated in
Equation (1) for any individual iwho are susceptible.
β=1−exp(τ∑
k∈K
Nkln(1−ksiρ)) .(1)
Here, τis the duration of contact with infectious individuals, Nkis the number of infectivities with kfrom the
set of all infectivities K,siis the susceptibility of the individual, and ρis the probability that one susceptible
individual gets infected by one infectious individual during one minute of contact. We compute ρ, a disease-
specific property, using the formula R0=ρ/γ, where R0is the reproduction number for the disease. We
assume that all non-infected individuals have the same susceptibility s=1 and all infectious individuals
have the same infectivity k=1. Thus the probability βsimply becomes β=1−exp(τNln(1−ρ)), where
Nis the number of infectious individuals. To use these parameters in a realistic way, we follow Balcan et al.
2009 who estimated reproduction number R0as well as transition rates σand γ. All these parameters and
their values are summarized in Table 1.

Kim, Kavak, Manzoor and Züfle
Table 1: Disease parameter values used in the model.
R0ρ σ γ
1.75 34.2−1hour 26.4−1hour 60−1hour
25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
Day
0
10
20
30
40
50
60
70
80
90
100
Population percentage
Disease progression (baseline)
Susceptible
Exposed
Infectous
Recovered
Epidemic peak
Figure 5: Baseline disease spread progression over ten replications. Each color represents a different per-
centage of disease stages within the population. The solid lines indicate the average values while shaded
areas show 95% confidence band.
3.2 Simulation Setup, Intervention Scenarios, and Results
We populated a small-scale synthetic city with 6,000 inhabitants and added various places including work-
places, homes, restaurants, and other venues (e.g., recreational places). Agents follow their daily activities
such as working when it is a work day, eating when feeling hungry, and visiting venues. Each simulation
step represents 5 minutes in the real world time. We start the simulation at day 0 and run it for over 60
days while the first 25 days are considered warm-up period for agents to establish their networks. At the
beginning of day 30, we introduce 0.5% of the population a hypothetical influenza-like virus. Fig. 5 shows
our results for the baseline runs with the introduction of the virus. The most critical result is that, around
day 36, the population reaches its highest percentage of Infectious people (≈34%) while Susceptible,Ex-
posed, and Recovered population compartments seem to be plausible. We used our framework to create
three intervention scenarios and test the effects of different mitigation options for the epidemic.
Scenario 1 - Vaccination
In this intervention scenario, a certain percentage of the population is vaccinated before the start of the
first infection time (day 26). The vaccinated population is randomly sampled from the entire population, is
assumed to be in the recovered stage once vaccinated (not susceptible to disease). We tested three different
values (10%, 30%, and 50%) for the percentages of vaccinated people with ten replications. Fig. 6 shows
the comparative results among these interventions against the baseline. It is clear that vaccination even at
a low level reduces the peak of the epidemic. That is, 10% vaccination reduces the peak of the epidemic
by ≈13%; 30% vaccination reduces the peak of the epidemic by ≈37%, and 50% vaccination reduces the
peak of the epidemic by ≈62%. Overall, vaccination appears to be a plausible strategy to cope with such
epidemics.

Kim, Kavak, Manzoor and Züfle
29 31 33 35 37 39 41 43 45 47 49
Day
0
10
20
30
Population percentage
Impact of vaccination on infectious population
Baseline
10% vaccinated
30% vaccinated
50% vaccinated
Figure 6: The impact of different percentages of vaccinated people on the infectiousness of the disease.
The shaded areas show 95% confidence band coverage while the lines within the shaded areas indicate the
average values.
29 31 33 35 37 39 41 43 45 47 49
Day
0
10
20
30
Population percentage
Closure period
Impact of place closure on infectious population
Baseline
10% closed
30% closed
50% closed
Figure 7: The impact of closing different percentage of public places on the infectiousness of the disease.
Lines indicate the average values. Confidence bands are hidden because they overlap on each other making
it challenging to read.
Scenario 2 - Place Closure
In this intervention scenario, the city ordered a certain percentage of meeting venues to be closed for a
certain number of days. As a result, people who aim to visit those places spend their time in their current
location for the length of the planned visit. Similar to the vaccination case, the choice of places to be
closed are randomly sampled. With ten replications, we tested 10%, 30%, and 50% of places closed for
six days as indicated in Fig. 7. Unlike the vaccination case, place closures do not help to mitigate the
peak of the infectious population as much. Place closures only reduced the epidemic peak by ≈1%, ≈5%,
and ≈10%, respectively. Further, there were more infectious cases, compared to the baseline, right after the
closure period ended. As a result, considering the cost to businesses against very limited gain, place closures
doesn’t appear to be a feasible solution. We tried changing the closure period to three days, but our results
were almost identical to the 6-day closure intervention.
Scenario 3 - Stay Home Order
The third and final intervention scenario is the order from the city that asks people to stay home when they
know that they are sick. In this scenario, we randomly sampled 10%, 30%, and 50% of the population to

Kim, Kavak, Manzoor and Züfle
29 31 33 35 37 39 41 43 45 47 49
Day
0
10
20
30
Population percentage
Order active period
Impact of 'stay home' order on infectious population
Baseline
10% followed
30% followed
50% followed
Figure 8: The impact of stay home orders on the infectious population. Lines indicate the average values.
Confidence bands are hidden because they overlap on each other making it challenging to read.
follow such orders starting with day 33 as shown in Fig. 8. Again, we conducted ten replications to capture
uncertainties caused by stochastic processes in the model. Results indicate that these orders are even more
ineffective than place closures. When 10%, 30%, and 50% of people follow such orders, the peak of the
epidemic only reduced by ≈0.6%, ≈1.5%, and ≈7.5%, respectively. These results show that even worse
than place closure, asking people to stay home is not an effective strategy.
To summarize, vaccination appears to be the most effective method, reducing the peak of the epidemic
compared to the other two. Stay home and place closure measures don’t seem to work based on our inter-
pretation. We want to note that our model does not preclude people to work or does not close workplaces.
4 CONCLUSION
In spite of the importance of experimentation, it has been conducted ad-hoc and involves manual code
adjustments which are time-consuming and error-prone. In this paper, we introduced a framework that
facilitates changing agent-based model states while running the simulation to explore alternative worlds in
a semi-automated manner. Such a framework allows the simulation user to explore different policy options
rapidly with minimal effort. To demonstrate the effectiveness of our framework, we implemented an agent-
based urban model that simulates the spread of influenza-like diseases based on the framework. No code
for experimentation is included in the model. We successfully dispensed prescriptions for a given problem
through experimentation.
To reach a persuasive conclusion from the outcome of experimentation, numerous runs and substantial
resources such as computing power are required. Our future work is to extend our framework to leveraging
cloud computing. We further aim to extend our framework to facilities the automated comparison between
different “what-if” instances to support users in exploring relationship and causality between prescribed
changes and observable simulation results.
ACKNOWLEDGMENTS
This project is sponsored by the Defense Advanced Research Projects Agency (DARPA) under cooperative
agreement No.HR00111820005. The content of the information does not necessarily reflect the position or
the policy of the Government, and no official endorsement should be inferred.

Kim, Kavak, Manzoor and Züfle
REFERENCES
Apolloni, A., V. S. A. Kumar, M. V. Marathe, and S. Swarup. 2009, Dec. “Computational Epidemiology in
a Connected World”. Computer vol. 42 (12), pp. 83–86.
Balcan, D., H. Hu, B. Goncalves, P. Bajardi, C. Poletto, J. J. Ramasco, D. Paolotti, N. Perra, M. Tizzoni,
W. Van den Broeck, V. Colizza, and A. Vespignani. 2009, Sep. “Seasonal transmission potential and
activity peaks of the new influenza A(H1N1): a Monte Carlo likelihood analysis based on human mo-
bility”. BMC Medicine vol. 7 (1), pp. 45.
Barrett, C. L., K. R. Bisset, S. G. Eubank, X. Feng, and M. V. Marathe. 2008, Nov. “EpiSimdemics: An
efficient algorithm for simulating the spread of infectious disease over large realistic social networks”.
In SC ’08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, pp. 1–12.
Carley, K. M., D. B. Fridsma, E. Casman, A. Yahja, N. Altman, L.-C. Chen, B. Kaminsky, and D. Nave.
2006. “BioWar: scalable agent-based model of bioattacks”. IEEE Transactions on Systems, Man, and
Cybernetics-Part A: Systems and Humans vol. 36 (2), pp. 252–265.
Crooks, A. T., and A. B. Hailegiorgis. 2014. “An agent-based modeling approach applied to the spread of
cholera”. Environmental Modelling & Software vol. 62, pp. 164–177.
Farmer, J. D., and D. Foley. 2009. “The economy needs agent-based modelling”. Nature vol. 460 (7256),
pp. 685.
Grow, A. 2017. Regression Metamodels for Sensitivity Analysis in Agent-Based Computational Demogra-
phy, pp. 185–210. Springer International Publishing.
Hybinette, M., and R. Fujimoto. 1997. “Cloning: a novel method for interactive parallel simulation”. In wsc,
pp. 444–451. IEEE.
Hybinette, M., and R. M. Fujimoto. 2001. “Cloning parallel simulations”. ACM Transactions on Modeling
and Computer Simulation (TOMACS) vol. 11 (4), pp. 378–407.
Kim, J.-S., H. Kavak, and A. Crooks. 2018. “Procedural city generation beyond game development”.
SIGSPATIAL Special vol. 10 (2), pp. 34–41.
Kleijnen, J. P. 2005. “An overview of the design and analysis of simulation experiments for sensitivity
analysis”. European Journal of Operational Research vol. 164 (2), pp. 287–300.
Kumar, S., J. J. Grefenstette, D. Galloway, S. M. Albert, and D. S. Burke. 2013. “Policies to reduce in-
fluenza in the workplace: impact assessments using an agent-based model”. American journal of public
health vol. 103 (8), pp. 1406–1411.
Lamperti, F., A. Roventini, and A. Sani. 2018. “Agent-based model calibration using machine learning
surrogates”. Journal of Economic Dynamics and Control vol. 90, pp. 366–389.
Li, X., W. Cai, and S. J. Turner. 2017, May. “Cloning Agent-Based Simulation”. ACM Trans. Model. Com-
put. Simul. vol. 27 (2), pp. 15:1–15:24.
Ligmann-Zielinska, A., D. B. Kramer, K. Spence Cheruvelil, and P. A. Soranno. 2014, 10. “Using Uncer-
tainty and Sensitivity Analyses in Socioecological Agent-Based Models to Improve Their Analytical
Performance and Policy Relevance”. PLOS ONE vol. 9 (10), pp. 1–13.
Luke, S., C. Cioffi-Revilla, L. Panait, K. Sullivan, and G. Balan. 2005. “Mason: A multiagent simulation
environment”. Simulation vol. 81 (7), pp. 517–527.
Malleson, N., A. Heppenstall, and L. See. 2010. “Crime reduction through simulation: An agent-based
model of burglary”. Computers, environment and urban systems vol. 34 (3), pp. 236–250.

Kim, Kavak, Manzoor and Züfle
Nsoesie, E. O., J. S. Brownstein, N. Ramakrishnan, and M. V. Marathe. 2013. “A systematic review of stud-
ies on forecasting the dynamics of influenza outbreaks”. Influenza and Other Respiratory Viruses vol. 8
(3), pp. 309–316.
Perez, L., and S. Dragicevic. 2009. “An agent-based approach for modeling dynamics of contagious disease
spread”. International journal of health geographics vol. 8 (1), pp. 50.
Pires, B., and A. T. Crooks. 2017. “Modeling the emergence of riots: A geosimulation approach”. Comput-
ers, Environment and Urban Systems vol. 61, pp. 66–80.
Prasanna, D. R. 2009. Dependency Injection. 1st ed. Greenwich, CT, USA, Manning Publications Co.
Riley, S. 2007. “Large-Scale Spatial-Transmission Models of Infectious Disease”. Science vol. 316 (5829),
pp. 1298–1301.
Roche, B., J. M. Drake, and P. Rohani. 2011, Mar. “An Agent-Based Model to study the epidemiological
and evolutionary dynamics of Influenza viruses”. BMC Bioinformatics vol. 12 (1), pp. 87.
Thiele, J. C., W. Kurth, and V. Grimm. 2014. “Facilitating Parameter Estimation and Sensitivity Analysis of
Agent-Based Models: A Cookbook Using NetLogo and ’R”’. Journal of Artificial Societies and Social
Simulation vol. 17 (3), pp. 11.
Yoginath, S. B., and K. S. Perumalla. 2018, January. “Scalable Cloning on Large-Scale GPU Platforms with
Application to Time-Stepped Simulations on Grids”. ACM Trans. Model. Comput. Simul. vol. 28 (1),
pp. 5:11–5:26.
AUTHOR BIOGRAPHIES
JOON-SEOK KIM is a Postdoctoral Research Fellow in Department of Geography and Geoinformation
Science at George Mason University. His research interests lie in spatial and spatiotemporal databases,
geospatial simulation, and privacy protection. His email address is jkim258@gmu.edu.
HAMDI KAVAK is a Research Associate in the Department of Geography and Geoinformation Science
at George Mason University. His research focuses on data-driven methods to study human behavior in the
areas of cybersecurity and urban systems. His email address is hkavak@gmu.edu.
UMAR MANZOOR is a Postdoctoral Research Fellow in the Department of Computer Science at Tulane
University. His research focuses on multi-agent modelling and simulation. His email address is uman-
zoor@tulane.edu.
ANDREAS ZÜFLE is an assistant professor at the Department of Geography and Geoinformation Science
at George Mason University (GMU). Dr. Züfle’s research expertise includes big spatial data, spatial data
mining, social network mining, and uncertain database management. His email address is azufle@gmu.edu.