A Validation Framework for Bulk Distribution Logistics Simulation Models

Abstract

Background: Simulation of business processes allows decision-makers to explore the implications and trade-offs of alternative approaches, policies and configurations. Trust in the simulation as a stand-in proxy of the real system depends on the validation of the computer model as well as on that of the data used to run it and judge its behaviour. Though validation frameworks exist, they provide little guidance for validation in the context of data-poor endeavours, such as those where observations as sourced from historical records were acquired for purposes other than the simulation itself. As simulation of complex business systems as logistic distribution networks can only rely on this type of data, there is a need to address this void and provide guidance for practitioners and fostering the conversation among academics. This paper presents a high-level development and validation framework applicable to simulation in data-poor environments for modelling the process of bulk distribution of commodities. Method: Traditionally accepted approaches were synthesised so as to develop an into a flexible three-stage modelling and validation approach to guide the process and improve the transparency of adapting available data sources for the simulation itself. The framework suggests the development of parallel paths for the development of computer and data models which, in the last stage, are merged into a phenomenological model resulting from the combination of both. The framework was applied to a case study involving the distribution of bulk commodities over a country-wide network to show its feasibility. Results: The method was flexible, inclusive of other frameworks, and suggested considerations to be made during the acquisition and preparation of data to be used for the modelling and exploration of uncharted scenarios. Conclusions: This work provides an integrative, transparent, and straightforward method for validating exploratory-type simulation models for endeavours in which observations cannot be acquired through direct experimentation on the target system.

Full text

Academic Editors: Robert Handfield
and Rameshwar Dubey
Received: 17 August 2024
Revised: 16 December 2024
Accepted: 19 December 2024
Published: 25 December 2024
Citation: Guiguet, A.; Pons, D. A
Validation Framework for Bulk
Distribution Logistics Simulation
Models. Logistics 2025,9, 3. https://
doi.org/10.3390/logistics9010003
Copyright: © 2024 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license
(https://creativecommons.org/
licenses/by/4.0/).
Article
A Validation Framework for Bulk Distribution Logistics
Simulation Models
Andres Guiguet * and Dirk Pons
Department of Mechanical Engineering, University of Canterbury, Christchurch 8041, New Zealand;
dirk.pons@canterbury.ac.nz
*Correspondence: andres.guiguet@pg.canterbury.ac.nz
Abstract: Background: Simulation of business processes allows decision-makers to explore
the implications and trade-offs of alternative approaches, policies and configurations. Trust
in the simulation as a stand-in proxy of the real system depends on the validation of the
computer model as well as on that of the data used to run it and judge its behaviour. Though
validation frameworks exist, they provide little guidance for validation in the context of
data-poor endeavours, such as those where observations as sourced from historical records
were acquired for purposes other than the simulation itself. As simulation of complex
business systems as logistic distribution networks can only rely on this type of data,
there is a need to address this void and provide guidance for practitioners and fostering
the conversation among academics. This paper presents a high-level development and
validation framework applicable to simulation in data-poor environments for modelling
the process of bulk distribution of commodities. Method: Traditionally accepted approaches
were synthesised so as to develop an into a flexible three-stage modelling and validation
approach to guide the process and improve the transparency of adapting available data
sources for the simulation itself. The framework suggests the development of parallel
paths for the development of computer and data models which, in the last stage, are
merged into a phenomenological model resulting from the combination of both. The
framework was applied to a case study involving the distribution of bulk commodities over
a country-wide network to show its feasibility. Results: The method was flexible, inclusive
of other frameworks, and suggested considerations to be made during the acquisition and
preparation of data to be used for the modelling and exploration of uncharted scenarios.
Conclusions: This work provides an integrative, transparent, and straightforward method
for validating exploratory-type simulation models for endeavours in which observations
cannot be acquired through direct experimentation on the target system.
Keywords: operations research; logistics; discrete event simulation; validation; data-poor
modelling environments
1. Introduction
Simulation modelling of logistic networks is used to aid decision-making regarding
routing operations and provisioning processes by allowing the user to improve their
understanding of the network dynamics and explore the impacts of strategic choices. In
our area of study, discrete event simulation (DES) models have a long-standing history
of use due to their innate ability to represent systems interconnected by restricted linking
mechanisms, which give way to queues. The development of DES models is an elaborate
process by which the modeller helps users translate their current knowledge of the target
Logistics 2025,9, 3 https://doi.org/10.3390/logistics9010003

Logistics 2025,9, 3 2 of 22
system into executable computer code [
1
]. Once a functional representation of the current
state of understanding of the target-system is obtained, a process loosely referred to as
validation begins [
2
–
4
]. Validation in its engineering form [
5
] is seen as the process of
building a “legal-case” [6] in favour of the simulation’s fit-for-purposeness [7–11].
While modelling frameworks have existed since the early 1960s [
12
] the specific aspect
of validation seems to have not received enough attention [
13
,
14
]. This situation is particu-
larly obvious in practice when validation is ignored [
12
,
15
], is carried out superficially [
16
],
or is missing important aspects [
17
,
18
]. While some authors believe that this is tied to
differing approaches [
19
] and under-developed methodologies [
20
], many point to the
problem of developing and validating the data needed for the process [5,15,18,21–23].
These difficulties are specifically problematic in environmental [
24
] and logistic net-
work modelling [
23
], where data are gathered for operational and accounting purposes
with no consideration to the specific needs of simulation and modelling endeavours. The
problem in logistic network modelling is typical of all operations research (OR) modelling,
in which data acquisition is a recognized difficulty [
25
], as to render pre-existing data use-
able, some form pre-processing is needed [
26
]. Thus, gathering, preparing, and validating
data typically requires at least a third of the project’s duration [
27
] with no guarantee of
success [
28
], resulting in data-poor modelling environments characterized by large amounts
of historical data that cannot be directly used for simulation.
As guidance from the literature is scarce [
29
], modellers operating in these data-poor
environments must deal with data uncertainties and representativeness with constrained
resources and little chance of experimenting on the target-system [
30
]. Thus, modellers may
have to work with low-quality data characterized by inaccurate, incomplete, inconsistent,
or time-muddled records [
31
]. These data are likely with the wrong format, as they might
have been collected for a purpose unrelated to the simulation [
32
] or applied to misguided
levels of analysis, [
33
] thus requiring an additional layer of cleaning [
23
], preparation [
34
],
or interpretation [35], which is not taken into consideration in current validation schemes.
In this paper, we present a framework meant to guide the development and validation
of simulation models in endeavours in which data are not acquired through experimenta-
tion on the target system but by re-purposing historical observations recorded for account-
ing or business operations. Because the requirements will likely not be met by the data in
their original form, varying degrees of handling and interpretation will be needed, resulting
in a situation not covered by traditional frameworks which cover the data acquisition and
validation as something that happens in the background. In contrast, we suggest placing
the data and system model in parallel paths so as to allow concessions and interpretations
in the development of each to be cross-referenced, improving the conveyance stage of the
modelling process [36].
The paper is divided as follows: a brief literature review on validation-related issues
(Section 2), a description of objectives and approaches (Section 3), an explanation of the
framework itself (Section 4), a case study illustrating the application of the framework
(Section 5), and closing remarks and discussions (Section 6).
2. Review of Validation Methods
2.1. Validation Overview
Different definitions of validation have been developed along different fields of
use [
37
], with all aiming to ensure certain outcome insights from using the model. Se-
mantics range along reliability [
18
,
38
], credibility [
39
], adequacy [
12
,
21
,
40
–
42
], representa-
tiveness [
43
], relevancy [
39
], actionability [
44
], fidelity [
45
], usefulness [
46
], and faithful-
ness [
47
], among others [
48
]. Irrespective of differences, validation is not a pass or no-pass

Logistics 2025,9, 3 3 of 22
condition [
49
] but the result of an iterative process [
50
] aimed at assessing the simulation’s
capacity of fulfilling its purpose [51].
Thus, an operational definition of validity hinges on the purpose of the modelling
endeavour itself, which traditionally involves predicting and explaining [
52
,
53
]. However,
in OR and other “softer” sciences, models may also involve guiding “Action” [
48
,
54
]
by way of speculation [
55
], sense-making [
56
], decision-making [
57
], exploration [
58
],
and development of insights [
3
]. Because in all these cases the simulation is to serve
as a mediator for our understanding of reality [
59
], a central requirement is for it to be
credible [
60
] or trustworthy [
44
]. This is especially true in intervention-oriented fields [
61
],
like OR, where a post-positivist view requires stakeholder’s participation to ensure the
simulation impacts practice [62].
While the exact condition for credibility is unclear, some authors associate it with
the epistemological problem of representation [
3
,
34
,
63
], which in turn depends on the
condition of achieving a degree of fidelity or reliability [
13
,
64
]. From this perspective, a
model’s fidelity is subject to two conditions: representational fidelity, or the extent to which
the user can relate to the causal mechanisms used, and dynamical fidelity, or the degree to
which the modelled behaviour is consistent with the target system [43,65].
However, credibility in soft sciences, such as OR, also depends on the adequate
interpretation of the phenomenon and its context, and while the fidelity conditions as a
whole should cover proper interpretation, we suggest including an additional condition
dubbed grounded interpretation to cover assumptions [
13
] specifically regarding data
preparation as a guard against mapping the wrong phenomenon [63,66,67].
2.2. Validation Frameworks
Validation strategies in the OR field have developed around a cyclical model involving
users, modellers, and their shared interpretation of the system under study. In early models,
such as that of the Simulation Council Society Technical Committee (SCS) [
68
] shown in
Figure 1, the system under study is presented as the “Reality” in imitation of the traditional
positivistic stance adopted by other branches of science and engineering. The outcome of
the specific “interpretation” of this reality is assumed to form a conceptual model from
which a computer model can then be prepared. In terms of validation, this paradigm
assumes the process can be broken down into three validation types, named model qual-
ification, model verification, and model validation, contributing independently to the
process as a whole. Though this paradigm was developed for the field of OR, it has also
influenced other fields of engineering, triggering the debate of whether verification should
be separated from validation [13,63,69,70] but otherwise being left mostly untouched.
Expanding the ontology of the SCS model, Robert Sargent developed the Simplified
Model Development Process (SMDP) [
21
,
41
,
42
,
71
], also known as the Sargent Model [
72
]
and Sargent Circle [
70
,
73
]. In this upgraded version shown in Figure 2, a fourth outcome
phase is super imposed on the SGS model to account for the recognition that data needs to
be assessed before it can be used. Though the data validation stage is can be performed
both independently and as part of the model development process, it makes the point of
recognizing a difference between raw and structured data [12,21,40,42,71,74,75].

Logistics 2025,9, 3 4 of 22
Logistics 2025, 9, x FOR PEER REVIEW 4 of 24
Figure 1. Early guiding framework for development and validation of simulation models. Published
by the Simulation Council Society Technical Commiee, it defined the basic structure still used to-
day.
Expanding the ontology of the SCS model, Robert Sargent developed the Simplified
Model Development Process (SMDP) [21,41,42,71], also known as the Sargent Model [72]
and Sargent Circle [70,73]. In this upgraded version shown in Figure 2, a fourth outcome
phase is super imposed on the SGS model to account for the recognition that data needs
to be assessed before it can be used. Though the data validation stage is can be performed
both independently and as part of the model development process, it makes the point of
recognizing a difference between raw and structured data [12,21,40,42,71,74,75].
Figure 2. Sargent’s Simplified Model Development Process. This framework expands on the SCS
version (see Figure 1—reference only) by incorporating a data validation stage applicable to all data
used for the process.
The SMDP model recognizes intermediate forms of validation, three directly referred
to as validation (conceptual, operational, and data) and one as verification (computer ver-
ification). Just like its predecessor, conceptual model validation is directed at assessing the
match between the client’s and the modeller’s perception of the underlying “Reality” [76]
Figure 1. Early guiding framework for development and validation of simulation models. Published
by the Simulation Council Society Technical Committee, it defined the basic structure still used today.
Logistics 2025, 9, x FOR PEER REVIEW 4 of 24
Figure 1. Early guiding framework for development and validation of simulation models. Published
by the Simulation Council Society Technical Commiee, it defined the basic structure still used to-
day.
Expanding the ontology of the SCS model, Robert Sargent developed the Simplified
Model Development Process (SMDP) [21,41,42,71], also known as the Sargent Model [72]
and Sargent Circle [70,73]. In this upgraded version shown in Figure 2, a fourth outcome
phase is super imposed on the SGS model to account for the recognition that data needs
to be assessed before it can be used. Though the data validation stage is can be performed
both independently and as part of the model development process, it makes the point of
recognizing a difference between raw and structured data [12,21,40,42,71,74,75].
Figure 2. Sargent’s Simplified Model Development Process. This framework expands on the SCS
version (see Figure 1—reference only) by incorporating a data validation stage applicable to all data
used for the process.
The SMDP model recognizes intermediate forms of validation, three directly referred
to as validation (conceptual, operational, and data) and one as verification (computer ver-
ification). Just like its predecessor, conceptual model validation is directed at assessing the
match between the client’s and the modeller’s perception of the underlying “Reality” [76]
Figure 2. Sargent’s Simplified Model Development Process. This framework expands on the SCS
version (see Figure 1—reference only) by incorporating a data validation stage applicable to all data
used for the process.
The SMDP model recognizes intermediate forms of validation, three directly referred
to as validation (conceptual, operational, and data) and one as verification (computer veri-
fication). Just like its predecessor, conceptual model validation is directed at assessing the
match between the client’s and the modeller’s perception of the underlying “Reality” [
76
]
and modelling objectives [
40
,
71
,
77
]. Verification aims to ensure the translation of the con-
ceptual model to executable computer code preserves the desired characteristics. [
40
,
41
].
Operational validation targets ensuring the computer model serves its purpose of acting as
a proxy for the target-system by presenting similar enough patterns of behaviour [
4
,
21
,
40
].
Though the stages are presented in an ordered fashion, they are carried out in an iterative
fashion during or after the development process [
40
,
41
]. The model differs from the SGS
take on that it includes a data validation stage to assess the quality of data used in the
endeavour [21].
2.3. Validation Techniques
Though validity types are discussed in other parts of the literature, the above models
are the only ones integrating them in the simulation development process. In doing so,

Logistics 2025,9, 3 5 of 22
they focus on high-level types (conceptual, operational, etc.) without imposing specific
methods or techniques with which they are to be achieved. In a sense, they follow the belief
that the modeller is free to choose in a way that best suits the endeavour [
64
,
78
]. Scholars
surveying available techniques reported the options being too extensive to cover [
12
,
35
],
though despite apparent discrepancies, many only vary in name while following similar
rationales [
19
,
41
]. As validation techniques tend to be context- and application-dependent,
we only mention a few examples of the three categorical groups (structural, behavioural,
and miscellaneous) and suggest following the citations for greater depth of detail.
Techniques falling in the structure-focused type are those aimed at scrutinizing as-
sumptions regarding the internal dynamics and their implementation in the computer
code. Examples are extreme condition testing [
21
,
41
], parameter variability–sensibility
analysis [
41
], trace analysis [
41
], structured walkthrough [
41
], operational graphics [
41
].
Behaviour-focused techniques, on the other hand, are those looking at the simulation
from a black-box perspective and judging the resulting behaviour rather than how it is
reached. Usually associated with external validation [
79
], these techniques are contrasted
against simulated, real-life, or expected behaviour, and while they lend themselves to being
statistical by nature, they tend to rely on subjective judgment in practice [
21
,
40
,
41
]. In this
category, we commonly find animations; event validity; and, where possible, statistics-
based comparisons [
21
,
40
,
41
]. Data-based validation techniques asses the quality of data
used for the simulation and can be carried out with or without the participation of the
simulation model itself. These tools aim at ensuring that both data reflect the underlying
assumptions used to build the SM and do not introduce bias or unreasonable amounts of
noise into the analysis. Common examples are historical data validation [
41
], predictive
validation [41], Turing test [41].
As the categorization is a guide only, we group any remaining under a miscellaneous
type, the most prominent being the ubiquitous face validation [
10
]. This method being the
most commonly used in business-based simulations allows expert users to judge fit-for-
purposeness on the basis of their subjective perception of the justification that specific needs
differ widely and may not be represented as a value threshold or statistical method [
63
,
80
].
3. Methodology
3.1. Objectives
The research objective was to develop a framework for guiding model development
and validation in data-poor environments, as is the case in logistics networks from an
OR perspective, where direct experimentation on the target-system is unfeasible, making
reliance on historically acquired data the only option.
3.2. Approach
In this work, we aim to close the gap by developing a validation framework which
allows the data preparation stage to be carried out in parallel to the software code, breaking
the validation process into three stages: independent instances of data, code pre-validation,
and a final more holistic validation for the data and code combination.
The process is based on the Simplified Model Development Process (SMDP) [
12
,
21
,
40
,
42
,
71
,
74
,
75
], with the incorporation of a data-preparation branch inspired on Ziegler’s
notion of experimental frames [
30
,
45
,
81
] and a final fit-for-purpose assessment reliant
on a mix of validation techniques (see Section 2.3). An instrumentalist perspective was
adopted by which simulations are understood as tools for expanding thinking [
82
], thought
experiments [
83
,
84
], and modelled experiments [
84
]. This line of reasoning highlights
the representational function of a model, accepting that it need not be accurate, true, or
complete in a realist sense [85].

Logistics 2025,9, 3 6 of 22
The approach was to carry out a literature review to understand the nature of the
problem and survey proposed solutions, after which a synthesised high-level development
and validation process could be developed. Once completed, the framework was applied
to a case study requiring the validation of a multi-level simulation model developed for
the distribution of bulk commodities to illustrate its use.
4. Conceptual Framework for Validation
The three-stage framework shown in Figure 3is proposed as a means of facilitating
the validation of simulations developed in data-poor environments where pre-existing data
need to be conditioned and whose validation may be problematic using existing techniques.
Stage I follows the conventional process of converting user needs into a computer model.
Stage II involves the parallel preparation of available data into a usable form dubbed a
data model. Stage III consists in the use of the data model to run and assess the predictive
ability of the computer model, under the assumption that the combined characteristics of
computer and data models (dubbed the Phenomenological Model) will together is what
needs to be assessed.
Logistics 2025, 9, x FOR PEER REVIEW 7 of 24
Figure 3. Framework overview. The intermediate steps of each stage are colour-coded in green, blue,
and red. Validation stages are depicted as curved arrows marked as I, II, II, and IV. Dashed line
encloses the elements of the phenomenological model.
4.1. Stage I: Building the Computer Model
As stated, the first stage of the process (see Figure 4—partial of Figure 3) follows the
traditional sequence of the developing a conceptual and computer model from what other
frameworks call the “Reality” (Society for Modelling and Simulation International 1979)
[68] or “Problem Entity” [21,41,42,71]. We prefer the term “Users’ Expectations” (a) to
highlight the role of the users as judges of what is to be considered part of the system and
what is not, something hard to objectively define when the system under study is defined
by physical laws, local policies, and socially defined conventions. The second step in this
stage involves the development (b) and approval of a conceptual model (I). As commonly
defined [77,86–88] a conceptual model acts as an informal agreement between modellers,
experts, and users as to the project scope, goals, and assumptions regarding the system’s
structure and behaviour. The third step involves translating the conceptual model into
computer code known as the computer model (c). The exact nature of the process depends
on the possibilities and allowances of the programming language, but the final outcome
should be validated (II) in a process referred to as computer model “Verification”, which
involves ensuring that the computer code is executable and is consistent with expecta-
tions. Suggested techniques are the input–output type discussed (Section 2.3) and extreme
condition testing method [21,41].
Figure 3. Framework overview. The intermediate steps of each stage are colour-coded in green, blue,
and red. Validation stages are depicted as curved arrows marked as I, II, III, and IV. Dashed line
encloses the elements of the phenomenological model.
Stage I reflects the traditional approach suggested by the SCS (Section 2.1 Figure 1),
through which the expectations (a) are converted into a conceptual model (b) and finally into
a computer model (c). In Stage II, we replace Sargent’s notion of “Structured Data” with the
idea of a data model (f) constructed by two components, namely a set of driver events (f
1
)
and expected responses (f
2
). The data model is to be interpreted as a sanitized expression
of available observations or “Raw Data”, obtained after applying an interpretation criterion
embedded in the operation frame (e) (to be defined in Section 4.2) on available historical
data or records (d). Stage III is characterized by the emergence of a “Phenomenological

Logistics 2025,9, 3 7 of 22
Model” (g) representing what authors refer to as the “Simulation” which produces a
simulated response (h) to be contrasted with the expected responses (f
2
). Also depicted are
feedback loops (I, II, III, and IV) representing the validation of each of the four models in
the framework (conceptual, computer, data, and phenomenological.
4.1. Stage I: Building the Computer Model
As stated, the first stage of the process (see Figure 4—partial of Figure 3) follows
the traditional sequence of the developing a conceptual and computer model from what
other frameworks call the “Reality” (Society for Modelling and Simulation International
1979) [
68
] or “Problem Entity” [
21
,
41
,
42
,
71
]. We prefer the term “Users’ Expectations” (a) to
highlight the role of the users as judges of what is to be considered part of the system
and what is not, something hard to objectively define when the system under study is
defined by physical laws, local policies, and socially defined conventions. The second
step in this stage involves the development (b) and approval of a conceptual model (I). As
commonly defined [
77
,
86
–
88
] a conceptual model acts as an informal agreement between
modellers, experts, and users as to the project scope, goals, and assumptions regarding the
system’s structure and behaviour. The third step involves translating the conceptual model
into computer code known as the computer model (c). The exact nature of the process
depends on the possibilities and allowances of the programming language, but the final
outcome should be validated (II) in a process referred to as computer model “Verification”,
which involves ensuring that the computer code is executable and is consistent with
expectations. Suggested techniques are the input–output type discussed (Section 2.3) and
extreme condition testing method [21,41].
Logistics 2025, 9, x FOR PEER REVIEW 8 of 24
Figure 4. Stage I of the framework represents the traditional simulation council society technical
commiee framework, whose outcome is the development of the executable code known as the
computer model.
4.2. Stage II: Constructing the Data Model
While Stage I guides and assesses the representational fidelity of the computer
model, this is only one of many validity criteria, a second being the grounded interpreta-
tion of the target system (Section 2.1.). As in business-related simulation projects, experi-
mentation is an unlikely means of obtaining data; the modeller cannot assume historical
records have been acquired in accordance with the project’s requirements. In data-poor
environments, where the modeller needs to reconstruct the phenomena from pre-existing
records, we must guard against the problem of confusing the fossil for the dinosaur [66]
and guard against inadequate interpretations of existing records [3].
In Stage II, we specifically address this task with the aim of ensuring transparency
and epistemic access [67] to the process of interpretating of available records [5,89]. It does
so by relying on an explicit data-cleaning criterion termed the operations frame (OF). The
OF following the steps of Zeigler’s experimental frame [30,45,81] allows the modeler to
restrict the range of inputs and outputs values allowed by operational characteristics of
the target-system. The OF will stand as the criteria to distinguish records considered out-
liers, “exceptional cases”, “not-of-interest”, and others that the user may deem not part of
the “business-as-usual” behaviour. Through the OF, the user’s biases, assumptions, and
interpretation of events are made explicit, allowing for improved transparency in the data
preparation phase. While there is no systematic method of creating an OF, grounded the-
ory practices such as memoing and exploration of negative cases [90] could guide the
modeller in the process of creating the OF.
In Figure 5 (partial of Figure 3), we show how raw data or “Available Records” (d)
can be interpreted through the OF (e) to create a data model (f) as a “smoothed-out” [47]
interpreted version of available records. The data model itself consists of two distinct da-
tasets [51], the first containing “driving events” (f
1
) used to drive the computer model (c)
and a second called “Expected Response” (f
2
), which will serve as a baseline behaviour
from which simulated responses will be judged. Implicit in the process is the iterative
Figure 4. Stage I of the framework represents the traditional simulation council society technical
committee framework, whose outcome is the development of the executable code known as the
computer model.
4.2. Stage II: Constructing the Data Model
While Stage I guides and assesses the representational fidelity of the computer model,
this is only one of many validity criteria, a second being the grounded interpretation of the
target system (Section 2.1.). As in business-related simulation projects, experimentation is
an unlikely means of obtaining data; the modeller cannot assume historical records have

Logistics 2025,9, 3 8 of 22
been acquired in accordance with the project’s requirements. In data-poor environments,
where the modeller needs to reconstruct the phenomena from pre-existing records, we
must guard against the problem of confusing the fossil for the dinosaur [
66
] and guard
against inadequate interpretations of existing records [3].
In Stage II, we specifically address this task with the aim of ensuring transparency
and epistemic access [
67
] to the process of interpretating of available records [
5
,
89
]. It
does so by relying on an explicit data-cleaning criterion termed the operations frame (OF).
The OF following the steps of Zeigler’s experimental frame [
30
,
45
,
81
] allows the modeler
to restrict the range of inputs and outputs values allowed by operational characteristics
of the target-system. The OF will stand as the criteria to distinguish records considered
outliers, “exceptional cases”, “not-of-interest”, and others that the user may deem not part
of the “business-as-usual” behaviour. Through the OF, the user’s biases, assumptions, and
interpretation of events are made explicit, allowing for improved transparency in the data
preparation phase. While there is no systematic method of creating an OF, grounded theory
practices such as memoing and exploration of negative cases [
90
] could guide the modeller
in the process of creating the OF.
In Figure 5(partial of Figure 3), we show how raw data or “Available Records”
(d) can be interpreted through the OF (e) to create a data model (f) as a “smoothed-
out” [
47
] interpreted version of available records. The data model itself consists of two
distinct datasets [51], the first containing “driving events” (f1) used to drive the computer
model (c) and a second called “Expected Response” (f
2
), which will serve as a baseline
behaviour from which simulated responses will be judged. Implicit in the process is
the iterative nature of the development by which the user’s expectations (a) change as
understanding improves [
87
]. By using a combination of techniques like face, event, and
graphical validation (Section 2.2), the coherency of the resulting data model can be assessed,
providing an alternative to an external data validation stage [12,21,71,75].
Logistics 2025, 9, x FOR PEER REVIEW 9 of 24
nature of the development by which the user’s expectations (a) change as understanding
improves [87]. By using a combination of techniques like face, event, and graphical vali-
dation (Section 2.2), the coherency of the resulting data model can be assessed, providing
an alternative to an external data validation stage [12,21,71,75].
Figure 5. Stage II of the framework mirrors the previous stage with the operations frame acting
much as a conceptual model in its informal role as an embodiment of expectations and assumptions
guiding the development of the data model.
4.3. Stage III: Merging Models
Previous stages have provided us with the Conceptual, Computer and Data Model
types and a way of assessing two of the three big validation criteria, namely representa-
tional fidelity and grounded interpretations, allowing us to now focus on the problem of
assessing dynamical fidelity. Our first step will be to identify the recipient of the endeav-
our as none of the previous models can in themselves be considered the simulation.
Though the term simulation model is often used in the literature, and at times seems
to be referring to the computer model, this constitutes a lack of semantic clarity, which
can lead to conceptual confusion termed the “Model Muddle” [91]. In order to circumvent
this problem while keeping true to the idea of the “simulation” model as a holistic entity
referring to all and parts of the constituent models, we will rely on the notion of model
hierarchies [63,64]. In this layered scheme, models with different purposes increase in
complexity as they increase in sophistication, allowing the emergence of a final expres-
sion, termed the phenomenological model, representing a computationally reproducible
form of a specific way of interpretating a phenomenon.
As shown in Figure 6 (partial of Figure 3), we use the term phenomenological model
(g) as the outcome of combining previously obtained computer (c) and data (f) models to
run a simulation which is represented in the figure as a dashed line encloses the two. By
adopting this terminology, we expect to make it clear that the simulated response (h)
Figure 5. Stage II of the framework mirrors the previous stage with the operations frame acting
much as a conceptual model in its informal role as an embodiment of expectations and assumptions
guiding the development of the data model.

Logistics 2025,9, 3 9 of 22
4.3. Stage III: Merging Models
Previous stages have provided us with the Conceptual, Computer and Data Model
types and a way of assessing two of the three big validation criteria, namely representational
fidelity and grounded interpretations, allowing us to now focus on the problem of assessing
dynamical fidelity. Our first step will be to identify the recipient of the endeavour as none
of the previous models can in themselves be considered the simulation.
Though the term simulation model is often used in the literature, and at times seems
to be referring to the computer model, this constitutes a lack of semantic clarity, which can
lead to conceptual confusion termed the “Model Muddle” [
91
]. In order to circumvent
this problem while keeping true to the idea of the “simulation” model as a holistic entity
referring to all and parts of the constituent models, we will rely on the notion of model
hierarchies [
63
,
64
]. In this layered scheme, models with different purposes increase in
complexity as they increase in sophistication, allowing the emergence of a final expression,
termed the phenomenological model, representing a computationally reproducible form of
a specific way of interpretating a phenomenon.
As shown in Figure 6(partial of Figure 3), we use the term phenomenological model
(g) as the outcome of combining previously obtained computer (c) and data (f) models
to run a simulation which is represented in the figure as a dashed line encloses the two.
By adopting this terminology, we expect to make it clear that the simulated response (h)
comes from the phenomenological model, meaning that any of the component models
(computer and data models) as well as their interaction can affect the resulting simulated
behaviour. From this perspective, the problem of assessing representational fidelity is now
to be targeted on the phenomenological model.
Logistics 2025, 9, x FOR PEER REVIEW 10 of 24
comes from the phenomenological model, meaning that any of the component models
(computer and data models) as well as their interaction can affect the resulting simulated
behaviour. From this perspective, the problem of assessing representational fidelity is
now to be targeted on the phenomenological model.
Figure 6. Stage III involves the merge of computer and data models. In practice, this results from
using the driver events dataset (f
1
) to run the executable code of the computer model (c) to obtain a
simulated response (h). During the validation phase (IV) these results will be contrasted to the ex-
pected response dataset (f
2
) of the data model.
While important as a construct to describe the framework, in practice, the phenome-
nological model does not have a physical embodiment or independent existence as the
computer or data models. Instead, it distinguishes different instantiations of a “simula-
tion” based on differences on either the computer or data models themselves. From a prac-
tical point of view, the modeller does not have to worry about its construction, as it comes
into existence once a combination of data and computer models are chosen. From a con-
ceptual point of view, on the other hand, it emphasizes the fact that the alignment between
simulated and expected responses, used traditionally as a criterion of validity, is affected
by choices made during the development of both the computer and data models.
The specific interpretation that we impose on existing observations will impact the
outcomes of the modelling process in three ways. First, by the characteristics of the forcing
dataset used to drive the computer model. Second, the generated reaction (simulated re-
sponse) of the computer model (subject to a specific forcing dataset) will be an important
element for judging the validity of the model, as will later be shown. Third, the role as-
sumed by the evaluation dataset as the baseline reference behaviour, representing busi-
ness as usual, against which all comparison will be made and decisions based.
In Figure 3, we presented the process overview, which—while similar to the idea of
operational validation (see Section 2.1)—differs from the fact that we evaluate the similar-
ity of responses under the consideration that both the computer model and the assumed
stimuli represented by forcing data may need to be put to the test, even if both responses
are comparable. Thus, in a way, we are evaluating both the computer and the data models
at this stage. Thus, at this stage, the following questions should guide us: Is the simulated
response in agreement with the evaluation dataset? Could the way the forcing dataset was
Figure 6. Stage III involves the merge of computer and data models. In practice, this results from
using the driver events dataset (f
1
) to run the executable code of the computer model (c) to obtain
a simulated response (h). During the validation phase (IV) these results will be contrasted to the
expected response dataset (f2) of the data model.
While important as a construct to describe the framework, in practice, the phenomeno-
logical model does not have a physical embodiment or independent existence as the
computer or data models. Instead, it distinguishes different instantiations of a “simulation”
based on differences on either the computer or data models themselves. From a practical
point of view, the modeller does not have to worry about its construction, as it comes into
existence once a combination of data and computer models are chosen. From a conceptual
point of view, on the other hand, it emphasizes the fact that the alignment between sim-
ulated and expected responses, used traditionally as a criterion of validity, is affected by
choices made during the development of both the computer and data models.

Logistics 2025,9, 3 10 of 22
The specific interpretation that we impose on existing observations will impact the
outcomes of the modelling process in three ways. First, by the characteristics of the forcing
dataset used to drive the computer model. Second, the generated reaction (simulated
response) of the computer model (subject to a specific forcing dataset) will be an important
element for judging the validity of the model, as will later be shown. Third, the role assumed
by the evaluation dataset as the baseline reference behaviour, representing business as
usual, against which all comparison will be made and decisions based.
In Figure 3, we presented the process overview, which—while similar to the idea of
operational validation (see Section 2.1)—differs from the fact that we evaluate the similarity
of responses under the consideration that both the computer model and the assumed
stimuli represented by forcing data may need to be put to the test, even if both responses
are comparable. Thus, in a way, we are evaluating both the computer and the data models
at this stage. Thus, at this stage, the following questions should guide us: Is the simulated
response in agreement with the evaluation dataset? Could the way the forcing dataset was
developed contributed to the current results? Could the way the computer model been
developed contribute to the current results?
Applicable techniques include those categorized under the input–output type (see
Section 2.3). As for the specific techniques to be used, we again leave it to the choice of the
team, though some authors highlight the importance of not underestimating quantitative
methods like graphical comparisons, as they allow for insights not easily obtained through
other methods [
89
]. As for the actual comparison of the behavioural datasets, quantitative
methods, usually involving some types of statistical analysis are indisputably stronger
but are not always possible [
40
]. Qualitative methods are not as accurate as have less
mathematical rigor but can be more intuitive and are preferred to nothing at all [
38
]. In
these methods, the general characteristics of the outcome are usually either compared to
known values or are judged by expert users who generally possess an understanding of
what is reasonable and expected [40].
5. Application to a Case Study
The case study involves the application of the proposed framework to a case study
involving the simulation of a large-scale fertilizer distribution network in New Zealand. The
purposes of the study included exploration of the feasibility of changing a key operational
parameter—the size of the ship. The system as whole was composed of eight stores,
with port access distributed around the country, each feeding a second layer of stores,
customers, and retailers. Due to the complexity and possible emergent behaviours, a multi-
level resolution modelling method [
92
] was used to prepare the computer model using
non-agent-based discrete event simulation software (Arena
®
—version 16.10). Available
historical operational records were by the client and data were processed using open-source
R (version 4.3.1) software.
5.1. Stage I: Computer Model
As mentioned, Phase I follows the traditional method described in Section 2.1, the
details of which have been described in another paper [
92
], for which we present only brief
a summary here. The process involved the creation of conceptual and computer models,
which were validated using some of the techniques described in Section 2.2.
The model was developed as a multi-level system allowing the simulation to capture
behavioural perspectives. The topmost tier, dubbed the “network level” is composed by the
three boxes of Figure 7, representing the target-system, or distribution network consisting
of suppliers (teal-coloured box), interconnected storage and sales points (green-coloured
box), and the consumers (grey-coloured box). Physical movements are represented by solid

Logistics 2025,9, 3 11 of 22
lines, while information exchange is shown as dashed lines. Embedded in the network
are “Stores” (Cylinders A and B), representing a combination of storage and sales points,
and an MRP/DRP module representing the policies and transactions criteria that guide
operations. At this level, unitary sales operations (SO) are the driving forces, and the size of
incoming shipments (SH ships) are the outcomes. A second layer termed the “store level”
represents the change in states of individual stores as they interact with other stores and
customers. Stores represented as cylinders (only A and B shown here) serve customers via
sales orders (SO.A and SO.B), draining the stock levels which can be replenished either by
requests to neighbouring stores (RO.A and ST.A) or by offloading of a partial cargo of an
incoming ship in an operations dubbed a “Port-call” (PC.A and PC.B), determined by the
MRP/DRP module according to current and projected scenarios. At this level, the driving
force is again the individual sales orders, and outcomes are the size of port-calls and the
number and size of replenishment requests made among stores.
Logistics 2025, 9, x FOR PEER REVIEW 12 of 24
Figure 7. High-level description of the computer model developed for the simulation. Sales orders
(SO) are the driving force for the system’s reaction captured at two levels: network via incoming
shipments (SH) and store via port calls (PC).
5.2. Stage II: Data Model
A brief look into the structure and characteristics of the historical operational records
available for the modelling endeavour confirmed they were not directly usable for the
endeavour and needed some form of cleaning and preparation. To do so, we developed
an operation frame in collaboration with the expert users and later used this as an explicit
criterion to interpret, clean, and convert raw observations into our data model. During the
process, we relied on face validation from the expert user to validate the model as it was
developed. We only show snippets as to illustrate the process and, in cases, obfuscate ac-
tual values so as to protect the commercial sensibility of the information.
5.2.1. Operation Frame
We present examples of constraints, limitations and expectations that form part of
the OF and will later refer to these as they are used to process the historical records to
prepare the data model. As an illustration of use, we only present three though the list
was fourfold the number.
[OF-1] Standard Sales Condition (Sales Orders are limited to a maximum of 40T).
This stems from the customary billing of sales in terms of the actual weight of laden
road-based physical carriers that have a carry limit of 40 metric tonnes (T). This in and of
itself is an approximation, as occasionally, other forms of sales may occur that violate this
assumption.
[OF-2] Shipment Size condition (Ship cargo capacity is of two nominal sizes—32 kT
and 55 kT).
The cargo size of inbound shipments varies substantially with a common payload
size of 32 kilotonnes (kT). Though very occasionally, a larger ship of around 55 kT will be
used, part of the strategic configuration changes is to use these ships more often. Thus,
these rare cases were included as an operational commonality. Related is the fill rate
Figure 7. High-level description of the computer model developed for the simulation. Sales orders
(SO) are the driving force for the system’s reaction captured at two levels: network via incoming
shipments (SH) and store via port calls (PC).
5.2. Stage II: Data Model
A brief look into the structure and characteristics of the historical operational records
available for the modelling endeavour confirmed they were not directly usable for the
endeavour and needed some form of cleaning and preparation. To do so, we developed
an operation frame in collaboration with the expert users and later used this as an explicit
criterion to interpret, clean, and convert raw observations into our data model. During
the process, we relied on face validation from the expert user to validate the model as it
was developed. We only show snippets as to illustrate the process and, in cases, obfuscate
actual values so as to protect the commercial sensibility of the information.
5.2.1. Operation Frame
We present examples of constraints, limitations and expectations that form part of the
OF and will later refer to these as they are used to process the historical records to prepare
the data model. As an illustration of use, we only present three though the list was fourfold
the number.
[OF-1] Standard Sales Condition (Sales Orders are limited to a maximum of 40T).

Logistics 2025,9, 3 12 of 22
This stems from the customary billing of sales in terms of the actual weight of laden
road-based physical carriers that have a carry limit of 40 metric tonnes (T). This in and
of itself is an approximation, as occasionally, other forms of sales may occur that violate
this assumption.
[OF-2] Shipment Size condition (Ship cargo capacity is of two nominal sizes—32 kT
and 55 kT).
The cargo size of inbound shipments varies substantially with a common payload size
of 32 kilotonnes (kT). Though very occasionally, a larger ship of around 55 kT will be used,
part of the strategic configuration changes is to use these ships more often. Thus, these
rare cases were included as an operational commonality. Related is the fill rate constraint,
which states the occupancy level according to preset levels, which vary according to
preset conditions.
[OF-3] Virtual Shipments (Multiple shipments arriving together).
The procurement stage has traditionally relied on 32 kT ships; therefore, references to
the use of larger vessels we are exploring in the simulation are not enough in a statistical
sense. Records do show the arrival of multiple 32 kT shipments within several days of
each other; from a simulation perspective, this is equivalent to a larger “virtual” shipment
contributing a sum-size cargo.
5.2.2. System Drivers
Inventory depletion at store levels is the driving force in our simulation and has
not been recorded historically as such but rather as inventory deductions. In Figure 8
(bottom plot), we plotted the records of one of the stores to show the range of values
(0 to 4000 tons)
. Now we know that these figures mostly represent completed sales as well
as other sporadic operations, such as changes in ownership, stock balancing, and other
non-descript operations. As we cannot easily distinguish one from the other, we apply
OF-1, which dictates that business-as-usual sales cannot exceed 40 tons and simply filtered
any outgoing movements above that value with the resulting top plot of Figure 8. While we
know we might be introducing an element of uncertainty through this selective filtering, the
users are content with using these values as the best possible references of actual historical
sales. Thus, we have constructed and validated the first part of the data model.
Logistics 2025, 9, x FOR PEER REVIEW 13 of 24
constraint, which states the occupancy level according to preset levels, which vary accord-
ing to preset conditions.
[OF-3] Virtual Shipments (Multiple shipments arriving together).
The procurement stage has traditionally relied on 32 kT ships; therefore, references
to the use of larger vessels we are exploring in the simulation are not enough in a statistical
sense. Records do show the arrival of multiple 32 kT shipments within several days of
each other; from a simulation perspective, this is equivalent to a larger “virtual” shipment
contributing a sum-size cargo.
5.2.2. System Drivers
Inventory depletion at store levels is the driving force in our simulation and has not
been recorded historically as such but rather as inventory deductions. In Figure 8 (boom
plot), we ploed the records of one of the stores to show the range of values (0 to 4000
tons). Now we know that these figures mostly represent completed sales as well as other
sporadic operations, such as changes in ownership, stock balancing, and other non-de-
script operations. As we cannot easily distinguish one from the other, we apply OF-1,
which dictates that business-as-usual sales cannot exceed 40 tons and simply filtered any
outgoing movements above that value with the resulting top plot of Figure 8. While we
know we might be introducing an element of uncertainty through this selective filtering,
the users are content with using these values as the best possible references of actual his-
torical sales. Thus, we have constructed and validated the first part of the data model.
Figure 8. Raw historic data of sales orders moved by road vehicles. Boom (labelled His.Raw): orig-
inal records. Top (labelled His.P25): post-processed data removing unrealistic truck movements us-
ing above criteria OF-1. This is compliant with client wishes for a “Business-as-Usual” sales scenario.
5.2.3. Expected Response
To illustrate the preparation of the second part of the recorded response of the D-
model we rely on items OF-2 and OF-3 of the operations frame (Section 5.2.1). We illus-
trate two applications (OF-2 and OF-3) as they refer to different criterion types, the first—
aimed at assuring a specific understanding of “business-as-usual”—is adopted, and the
second is used as a way of using historical events to allow a specific behaviour the model
is probing to be validated.
Figure 8. Raw historic data of sales orders moved by road vehicles. Bottom (labelled His.Raw): origi-
nal records. Top (labelled His.P25): post-processed data removing unrealistic truck movements using
above criteria OF-1. This is compliant with client wishes for a “Business-as-Usual” sales scenario.

Logistics 2025,9, 3 13 of 22
5.2.3. Expected Response
To illustrate the preparation of the second part of the recorded response of the D-model
we rely on items OF-2 and OF-3 of the operations frame (Section 5.2.1). We illustrate two
applications (OF-2 and OF-3) as they refer to different criterion types, the first—aimed at
assuring a specific understanding of “business-as-usual”—is adopted, and the second is
used as a way of using historical events to allow a specific behaviour the model is probing
to be validated.
At the network level of the model the system response is the procurement request
materialized as incoming shipments (SH) shown in its original distribution in the bottom
plot of Figure 9. Here, we see most shipments are in the 32 kT size range, with a mix of
exceptions such as 24 kT and 55 kT imports. On application of OF-2, we obtain the top part
of the same figure which basically ignores any shipments with a declared cargo under the
minimum occupancy, which is around 29 kT. As a side note, before developing OF-2, an
investigation of the nature of the recorded exceptions was carried out, and it was decided
that it is not part of the business-as-usual behaviour embedded in the model.
Logistics 2025, 9, x FOR PEER REVIEW 14 of 24
At the network level of the model the system response is the procurement request
materialized as incoming shipments (SH) shown in its original distribution in the boom
plot of Figure 9. Here, we see most shipments are in the 32 kT size range, with a mix of
exceptions such as 24 kT and 55 kT imports. On application of OF-2, we obtain the top
part of the same figure which basically ignores any shipments with a declared cargo under
the minimum occupancy, which is around 29 kT. As a side note, before developing OF-2,
an investigation of the nature of the recorded exceptions was carried out, and it was de-
cided that it is not part of the business-as-usual behaviour embedded in the model.
Figure 9. Shipment data represented as frequency density plots (to protect sensitive commercial
data). Boom (labelled S1V.P00): Original records. Top (labelled S1V.P30): post-processed data us-
ing OF-2 criteria based on client interpretation of standard business practices.
In the second stage, start with the results of the previous application shown now in
the boom plot of Figure 10 (teal-colour, titled S1V.P30), where we now apply criteria OF-
3, by which we reframe the historical records to coalesce shipments occurring in close
temporal proximity as a single “virtual shipment”. As a consequence of this reframing,
we obtain the distribution shown on the top plot of the same figure (salmon-colour, titled
S1V.P40), which now reflects the existence of two ship-size groups, one reflecting the tra-
ditional 32 kT size 4 and another which would account for larger ships (54–67 kT range).
Figure 9. Shipment data represented as frequency density plots (to protect sensitive commercial data).
Bottom (labelled S1V.P00): Original records. Top (labelled S1V.P30): post-processed data using OF-2
criteria based on client interpretation of standard business practices.
In the second stage, start with the results of the previous application shown now in the
bottom plot of Figure 10 (teal-colour, titled S1V.P30), where we now apply criteria OF-3, by
which we reframe the historical records to coalesce shipments occurring in close temporal
proximity as a single “virtual shipment”. As a consequence of this reframing, we obtain the
distribution shown on the top plot of the same figure (salmon-colour, titled S1V.P40), which
now reflects the existence of two ship-size groups, one reflecting the traditional 32 kT size
4 and another which would account for larger ships (54–67 kT range).

Logistics 2025,9, 3 14 of 22
Logistics 2025, 9, x FOR PEER REVIEW 15 of 24
Figure 10. Shipment data further processed. Boom (labelled S1V.P30): results from Figure 9 (for
reference). Top (labelled S1V.P40): replacement of simultaneous shipments as one “‘virtual” ship-
ment of total size as per OF-3 to represent client expectations of not calling two ships when one
larger would suffice.
5.3. Stage III: Phenomenological Model
The combination of sales data from Section 5.2.2 and the Arena® code from Section
5.1 locks us into a phenomenological whose predictive ability needs to be assessed. The
multi-level structure that we opted to represent our target-system means multiple assess-
ments will need to be considered, only two of which we show to illustrate the case. The
first corresponds to the simulated behaviour of Level 1, where shipments (SH) are called
into the network in an aempt to keep stores stocked, while the second illustrates a Level
2 response in which port calls (PC) are allocated to distribute incoming product according
to a pre-set allocation criterion.
5.3.1. Shipments
Figure 11 shows a graphical comparison of the expected response (top plot, titled
S1V.P40) against the simulated response (boom plot, titled S1V.P30). When presented to
the expert users, it was agreed that the figures have acceptable resemble in grouping of
the standard (32 kT range) shipment sizes. In contrast, larger shipments (60 kT range) are
tightly grouped in simulation due to constraints placed in the computer model to repre-
sent real-life physical size and minimum load restrictions. While the Expected Response
does not have that same distribution, we need to remember the dispersion stems from the
fact the “virtual shipment” (Section 5.2.3) sizes accumulate from physical shipments
whose total cargo needs not limit itself to the 60 kT range.
Figure 10. Shipment data further processed. Bottom (labelled S1V.P30): results from Figure 9(for
reference). Top (labelled S1V.P40): replacement of simultaneous shipments as one “‘virtual” shipment
of total size as per OF-3 to represent client expectations of not calling two ships when one larger
would suffice.
5.3. Stage III: Phenomenological Model
The combination of sales data from Section 5.2.2 and the Arena
®
code from Section 5.1
locks us into a phenomenological whose predictive ability needs to be assessed. The multi-
level structure that we opted to represent our target-system means multiple assessments
will need to be considered, only two of which we show to illustrate the case. The first
corresponds to the simulated behaviour of Level 1, where shipments (SH) are called into
the network in an attempt to keep stores stocked, while the second illustrates a Level 2
response in which port calls (PC) are allocated to distribute incoming product according to
a pre-set allocation criterion.
5.3.1. Shipments
Figure 11 shows a graphical comparison of the expected response (top plot, titled
S1V.P40) against the simulated response (bottom plot, titled S1V.P30). When presented to
the expert users, it was agreed that the figures have acceptable resemble in grouping of
the standard (32 kT range) shipment sizes. In contrast, larger shipments (60 kT range) are
tightly grouped in simulation due to constraints placed in the computer model to represent
real-life physical size and minimum load restrictions. While the Expected Response does
not have that same distribution, we need to remember the dispersion stems from the fact
the “virtual shipment” (Section 5.2.3) sizes accumulate from physical shipments whose
total cargo needs not limit itself to the 60 kT range.

Logistics 2025,9, 3 15 of 22
Logistics 2025, 9, x FOR PEER REVIEW 16 of 24
Figure 11. Simulated vs. expected response at Level 1 (Shipments). Boom (labelled Sim.Log): sim-
ulated shipments. Top (labelled His.Val): post-processed historical records. Note the expected re-
sponse is compliant with clients understanding of ideal or targeted operational behaviour.
5.3.2. Port Calls
Figure 12 we compare port call sizes for two anonymous stores, the top (salmon col-
oured) plots representing expected responses and the boom teal coloured) plot the sim-
ulated outcomes. Present in all plots if a dashed vertical line dubbed MFD or “Minimum
Feasible Discharge”, which is a target lower threshold for PC to guarantee that the costs
of the operation are justified. At this level, we see that the simulated data tend to hold to
the MFD criteria with more zeal than what was historically observed, which makes for
sharper PC size distribution. This was discussed with the expert users, who admied that
the MFD criteria were a nice-to-have condition that was often set aside in favour of unique
commercial circumstances. Despite this, they preferred to see a more idealistic behaviour
in the model, as exceptions are always managed on a case-by-case basis.
Figure 11. Simulated vs. expected response at Level 1 (Shipments). Bottom (labelled Sim.Log):
simulated shipments. Top (labelled His.Val): post-processed historical records. Note the expected
response is compliant with clients understanding of ideal or targeted operational behaviour.
5.3.2. Port Calls
Figure 12 we compare port call sizes for two anonymous stores, the top (salmon
coloured) plots representing expected responses and the bottom teal coloured) plot the
simulated outcomes. Present in all plots if a dashed vertical line dubbed MFD or “Minimum
Feasible Discharge”, which is a target lower threshold for PC to guarantee that the costs
of the operation are justified. At this level, we see that the simulated data tend to hold to
the MFD criteria with more zeal than what was historically observed, which makes for
sharper PC size distribution. This was discussed with the expert users, who admitted that
the MFD criteria were a nice-to-have condition that was often set aside in favour of unique
commercial circumstances. Despite this, they preferred to see a more idealistic behaviour
in the model, as exceptions are always managed on a case-by-case basis.
Logistics 2025, 9, x FOR PEER REVIEW 16 of 24
Figure 11. Simulated vs. expected response at Level 1 (Shipments). Boom (labelled Sim.Log): sim-
ulated shipments. Top (labelled His.Val): post-processed historical records. Note the expected re-
sponse is compliant with clients understanding of ideal or targeted operational behaviour.
5.3.2. Port Calls
Figure 12 we compare port call sizes for two anonymous stores, the top (salmon col-
oured) plots representing expected responses and the boom teal coloured) plot the sim-
ulated outcomes. Present in all plots if a dashed vertical line dubbed MFD or “Minimum
Feasible Discharge”, which is a target lower threshold for PC to guarantee that the costs
of the operation are justified. At this level, we see that the simulated data tend to hold to
the MFD criteria with more zeal than what was historically observed, which makes for
sharper PC size distribution. This was discussed with the expert users, who admied that
the MFD criteria were a nice-to-have condition that was often set aside in favour of unique
commercial circumstances. Despite this, they preferred to see a more idealistic behaviour
in the model, as exceptions are always managed on a case-by-case basis.
Figure 12. Simulated vs. expected response at Level 2 for Stores A and C (port calls). Bottom (labelled
Sim.Log): simulated port calls sizes. Top (labelled His.Val): historical records. vertical dashed line
represents a “nice-to-avoid” minimum threshold called the minimum feasible discharge size.

Logistics 2025,9, 3 16 of 22
6. Discussion
6.1. Finding from the Literature Review
The literature review identifies the following. First, validation as an assessment can
be narrowed down to three distinct criteria here referred to as representational fidelity,
grounded interpretation, and dynamical fidelity. Second, simulation models in logistics
and other operations research fields fall short in their validation efforts. Third, the default
assumption of validating data prior to its incorporation into the model may not be possible
in all operations research projects. Fourth, in many cases, data are only available as histori-
cal records, which does not allow for their use in simulation modelling in a straightforward
way. These findings suggest that business-based simulations with no attempts to consider
dynamical fidelity and grounded interpretation may stem from a lack of an alternative
approach tailored to situations in which data cannot be improved through experimentation
or successive iterations of practice.
6.2. Finding from the Framework
The framework proved to be applicable to a complex, multi-level logistics distribu-
tion case study, which retained the staged development process while keeping a holistic
perspective. The framework allowed the seamless integration of pre-existent frameworks
and techniques in all stages of the process. Moreover, the first stage in which the computer
model was developed relied solely on a pre-existent framework developed for non-object-
orientated programming languages like Arena
®
, which are commonly used for this purpose
(see Section 5.1), all while relying on the same validation techniques modellers and users
are likely to have used in the past and minimizing additional cognitive burden.
The structure of the framework allowed results stemming from the model as a whole
(phenomenological model) to be traced back to the contributions made by the earlier stages
(computer and data models). This allowed a partial decoupling between assumptions
hard-coded into the computer model from those implicit in the data-preparation phase,
which in turn allowed the probable sources of predictive discrepancies to be traced into the
computer or data models separately (see, for example, Section 5.3.1). This in turn allowed
fit-for-purposeness to be judged in light of more practical possibilities.
A final, yet subtle, finding stems from upfront level of thought placed on the structur-
ing of the observations in the data-preparation phase (Section 4.2). As the framework calls
for a pre-empting action in terms of recognizing inconsistencies, noise, and unforeseen un-
certainties in the raw data at an early stage, it is less likely these confounding characteristics
will creep into the use stage of the model.
6.3. Findings from the Case Study
All the theoretical elements were represented and successfully applied in the case study.
The process allowed for a validation that was explicitly coupled to the operational realities
and expectations of the client for business as usual. Consequently, a transparent validation
was achieved. The method provided a systematic process for discussion interpretation of
raw data with the client. This also enables post-validation reviews.
The case study showed that historical data were primarily recorded for accountability
purposes. There was little consideration given to mass balance and flow, which simulation
requires. We suspect that this is a common problem in organisations. Ideal records would
have a data structure comprising product transformations and handling operations. The
current method was able to adequately convert existing data into something that could be
used for simulation purposes.
Observations showed that within a firm, different roles had different interpretations
of what was business as usual. It appears that operational and management roles base

Logistics 2025,9, 3 17 of 22
their understanding on either “usual” practice or “targeted” outcomes. In any case, the
framework allowed these differences to be accommodated. For example, while typical ship-
ments occurred in various cargo sizes, management intention of exploring the possibility
of using large ships for future operations suggested both ignoring shared shipments (see
Figure 9) as well as relying on the virtual shipment concept (see Figure 10). Likewise, the
“nice-to-have” feature of a minimum feasible discharge size for port calls (see Figure 12)
appeared to have been applied differently at the operational level.
The comparison shows the model is capable of approximately reproducing historical
behaviours at an accuracy that the client deemed “fit for purpose” under the current context
(more accuracy would imply better quality data as well as more model development, neither
of which is worth the cost at the moment). Possible future applications for such a model
include local storage capacity, impact of shipping delays, changes in sales volumes, and
loss of distribution centres.
6.4. Implications for Practitioners
The framework offers an approach for simulation of validation models tailored for
application cases in which clean data are not available and allows several benefits:
•
The staged structure allows for an easy integration of other methodologies, especially
those available for the development of the computer model itself.
•
The parallel nature of the computer and data model stages allows flexibility in the
development stages to accommodate for specific project constraints as waiting for
records or agreements to be made in meetings.
•
The need for an explicit criterion to interpretation historical records callas for early
and open reflection on the consistency of assumptions, quality of available data, and
alignment with final expectations.
•
The framework allows use of interpretations from users at any level of the organization
independent of consensus, allowing the simulation to be used with different degrees
of grounding allowing the freedom to choose a position between the speculative–
predictive spectrum.
6.5. Limitations of the Work
This work has several limitations, the first being that its high-level approach does not
provide specific techniques and relies on prior understanding and experience with model
validation. In practice, this means that no guarantee can be made that the application by
different modellers will achieve the same outcome. This also suggests that the framework
is not a standalone method and that novice modellers will need guidance from more
experienced practitioners. Second, the approach is tailored to the type of modelling used
in OR, mainly exploratory, explanatory or speculative, and is thus not well-suited for
predictive modelling. Third, an experience-based assumption was made that the historical
purpose of recording observations was made for legal, accounting or operational motives
which tend to not meet the requirements form simulation-based use. Fourth, while the
validation users are free to choose the validation techniques, the method as a whole relies
on subjective measures of fit, preventing the framework from acquiring an objective or
hard-metric status needed for non-OR-based fields.
The resulting framework is a proposed methodology for validating simulation in
situation where the business data lack the coherence necessary for the simulation to give
operationally valid outcomes. The framework is illustrated with a case study, but its
universal applicability has not been tested.

Logistics 2025,9, 3 18 of 22
7. Conclusions
The purpose of this paper was to develop a framework to guide the simulation
model development and validation in the context of data-poor environments. In these
situations, target system experimentation is unfeasible. Hence, data to run and validate
the simulation need to be sourced from potentially unreliable business records, which will
need to be transformed to varying degrees to be useable in the simulation. However, the
interpretation and transformation process of data is mostly ignored by other modelling
frameworks. The contribution of the present paper is the presentation of a conceptual
framework wherein a data model is developed simultaneously with the computer model
itself. Key steps in the application of the framework are: (1) convert client requirements
into a computer simulation per conventional methods; (2) take available business data and
work with the client to develop a baseline reference data model that represents common
operational behaviours—importantly, this data model comprises historical data that is
not simply cleaned but processed via assumptions and simplifications established with
the client; (3) use the data model in the simulation; (4) compare the simulation results
with the baseline reference behaviour; (5) reflect on the difference to detect if significant
variations may be caused by choices and simplifications made during the development of
the software model or data model; (6) if needed, either the software or data model may
need to be revised.
The framework was applied to a case study involving a simulation of a large-scale
fertilizer distribution network in New Zealand developed for the purpose of exploring the
feasibility of changes key operational parameter, mainly the size of restock shipments.
In the case study section, it was shown how validation was achieved using a three-
staged approach. Stage I mostly centred on the degree to which the computer model
covered user expectations. Stage II focused on the selection of data representative of what
expert users considered the main drivers of the system as well as the system’s common
response to those events. Stage III involved a comparative analysis between simulated and
historical behaviour with specific attention to explaining possible mismatches.
By using the idea of a phenomenological model as a combined computer and data
model, we allowed the implications of simplifications and assumptions made during the
development of both models to be addressed as a whole. This allowed the consequences
of using specific instances of data and computer models to be judged in terms of their
combined outcomes instead of their individual worth to the user. In this particular case,
while improvements were possible, their cost in terms of extra resources (more accuracy
required better quality data and model development time) were deemed worthy, and the
“Simulation” (phenomenological model) was deemed “fit for purpose” (validated).
The approach showed its potential as well as its ability to integrate with pre-existing
frameworks guiding both the development and validation of individual models at the
different stages. We also pointed out that the framework can also serve as a mapping tool
to direct attention to the where and how possible mismatches the support offered by the
data model or its interpretation and the expectations placed on the computer model or the
phenomenological model as the end-result of the endeavour itself.
Author Contributions: Conceptualization, A.G. and D.P.; methodology, A.G. and D.P.; software, A.G.;
validation, A.G.; formal analysis, A.G.; investigation, A.G.; data curation, A.G.; writing—original
draft, A.G.; writing—review and editing, A.G. and D.P.; visualization, A.G.; supervision, D.P. All
authors have read and agreed to the published version of the manuscript.
Funding: The work was funded by Callaghan Innovation R&D grant ARLAB2002.

Logistics 2025,9, 3 19 of 22
Data Availability Statement: The datasets presented in this article are not readily available as
they belong to the organization in which the study was carried out and are deemed to possess
commercial sensitivity.
Conflicts of Interest: The authors declare no financial conflicts of interest with the industrial partner
other than the above funding.
References
1.
Lyu, Z.; Pons, D.; Zhang, Y.; Ji, Z. Minimum Viable Model (MVM) Methodology for Integration of Agile Methods into Operational
Simulation of Logistics. Logistics 2022,6, 37. [CrossRef]
2.
Fagiolo, G.; Guerini, M.; Lamperti, F.; Moneta, A.; Roventini, A. Validation of Agent-Based Models in Economics and Finance.
In Computer Simulation Validation: Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C.,
Saam, N.J., Eds.; Simulation Foundations, Methods and Applications; Springer International Publishing: Cham, Switzerland,
2019; pp. 763–787. ISBN 978-3-319-70765-5.
3. Oberkampf, W.L.; Roy, C.J. Verification and Validation in Scientific Computing; Cambridge University Press: Cambridge, UK, 2010;
ISBN 978-0-521-11360-1.
4.
Tsioptsias, N.; Tako, A.; Robinson, S. Model Validation and Testing in Simulation: A Literature Review. In Proceedings of the 5th
Student Conference on Operational Research (SCOR 2016), Nottingham, UK, 8–10 April 2016. [CrossRef]
5. Rykiel, E.J. Testing ecological models: The meaning of validation. Ecol. Model. 1996,90, 229–244. [CrossRef]
6.
Hemez, F.M. Verifying and Validating Simulation Models; LA-UR-15-21344; Los Alamos National Lab. (LANL): Los Alamos, NM,
USA, 2015; p. 1170703. [CrossRef]
7.
Schwaninger, M.; Groesser, S. System Dynamics Modeling: Validation for Quality Assurance. In Encyclopedia of Complexity and
Systems Science; Meyers, R.A., Ed.; Springer: Berlin/Heidelberg, Germany, 2009; pp. 1–20, ISBN 978-3-642-27737-5.
8. Banks, J. Discrete-Event System Simulation; Always Learning, 5th ed.; Pearson: Harlow, UK, 2014; ISBN 978-1-292-02437-0.
9.
Kopec, J.A.; Finès, P.; Manuel, D.G.; Buckeridge, D.L.; Flanagan, W.M.; Oderkirk, J.; Abrahamowicz, M.; Harper, S.; Sharif, B.;
Okhmatovskaia, A.; et al. Validation of population-based disease simulation models: A review of concepts and methods. BMC
Public Health 2010,10, 710. [CrossRef] [PubMed]
10.
Murray-Smith, D. Verification and Validation Principles from a Systems Perspective. In Computer Simulation Validation: Funda-
mental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations,
Methods and Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 99–118, ISBN 978-3-319-
70765-5.
11.
Beisbart, C. What Is Validation of Computer Simualtion? Towards a Clarification of the Concept of Validation and of Related
Notions. In Computer Simulation Validation: Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart,
C., Saam, N.J., Eds.; Simulation Foundations, Methods and Applications; Springer International Publishing: Berlin/Heidelberg,
Germany, 2019; pp. 35–67, ISBN 978-3-319-70765-5.
12.
Sargent, R.; Balci, O. History of verification and validation of simulation models. In Proceedings of the 2017 Winter Simulation
Conference (WSC), Las Vegas, NV, USA, 3–6 December 2017; pp. 292–307.
13. Frigg, R.; Reiss, J. The philosophy of simulation: Hot new issues or same old stew? Synthese 2009,169, 593–613. [CrossRef]
14.
Yin, C.; McKay, A. Model Verification and Validation Strategies and Methods: An Application Case Study. 2018. Available online:
https://eprints.whiterose.ac.uk/135648/ (accessed on 10 March 2023).
15.
Heath, B.; Hill, R.; Ciarallo, F. A Survey of Agent-Based Modeling Practices (January 1998 to July 2008). J. Artif. Soc. Soc. Simul.
2009,12, 9.
16.
Beisbart, C.; Saam, N.J. Computer Simulation Validation. In Computer Simulation Validation: Fundamental Concepts, Methodological
Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods and Applications;
Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 1–31, ISBN 978-3-319-70765-5.
17.
O’Sullivan, D.; Evans, T.; Manson, S.; Metcalf, S.; Ligmann-Zielinska, A.; Bone, C. Strategic directions for agent-based modeling:
Avoiding the YAAWN syndrome. J. Land Use Sci. 2016,11, 177–187. [CrossRef]
18.
Eker, S.; Rovenskaya, E.; Langan, S.; Obersteiner, M. Model validation: A bibliometric analysis of the literature. Environ. Model.
Softw. 2019,117, 43–54. [CrossRef]
19.
Groesser, S.N.; Schwaninger, M. Contributions to model validation: Hierarchy, process, and cessation: S. N. Groesser and M.
Schwaninger: Contributions to Model Validation. Syst. Dyn. Rev. 2012,28, 157–181. [CrossRef]
20.
Olsen, M.; Raunak, M. Increasing Validity of Simulation Models Through Metamorphic Testing. IEEE Trans. Rel. 2019,68, 91–108.
[CrossRef]
21.
Sargent, R. Verification and validation of simulation models. In Proceedings of the 2010 Winter Simulation Conference, Baltimore,
MD, USA, 5–8 December 2010; pp. 166–183.

Logistics 2025,9, 3 20 of 22
22. Roache, P.J. Perspective: Validation—What Does It Mean? J. Fluids Eng. 2009,131, 034503. [CrossRef]
23.
Simard, V.; Rönnqvist, M.; Lebel, L.; Lehoux, N. A General Framework for Data Uncertainty and Quality Classification. IFAC-
PapersOnLine 2019,52, 277–282. [CrossRef]
24.
Schwanitz, V.J. Evaluating Integrated Assessment Models of Global Climate Change—From Philosophical Aspects to Practical
Examples. 2021. Available online: https://osf.io/preprints/socarxiv/63yd8/ (accessed on 10 March 2023).
25.
Ungureanu, D.; Sisak, F.; Kristaly, D.M.; Moraru, S. Simulation Modeling. Input Data Collection and Analysis. 2005, p. 8.
Available online: https://ecad.tu-sofia.bg/et/2005/pdf/Paper051-D_Ungureanu2.pdf (accessed on 10 March 2023).
26.
Onggo, S.; Pidd, M.; Soopramanien, D.; Worthington, D. Simulation of Career Development in the European Commission.
Interfaces 2010,40, 184–195. [CrossRef]
27.
Skoogh, A.; Johansson, B. A methodology for input data management in discrete event simulation projects. In Proceedings of the
2008 Winter Simulation Conference, Miami, FL, USA, 7–10 December 2008; pp. 1727–1735.
28.
Perera, T.; Liyanage, K. Methodology for rapid identi
®
cation and collection of input data in the simulation of manufacturing
systems. Simul. Pract. Theory 2000,7, 645–656. [CrossRef]
29.
Onggo, S.; Hill, J. Data identification and data collection methods in simulation: A case study at ORH Ltd. J. Simul. 2014,8,
195–205. [CrossRef]
30.
Zeigler, B.P.; Muzy, A.; Kofman, E. Theory of Modelling and Simulation: Discrete Event and Iterative System Computational Foundations,
3rd ed.; Academic Press: San Diego, CA, USA, 2019; ISBN 978-0-12-813370-5.
31.
Blake, R.; Mangiameli, P. The Effects and Interactions of Data Quality and Problem Complexity on Classification. J. Data Inf. Qual.
2011,2, 1–28. [CrossRef]
32. Banks, J.; Chwif, L. Warnings about simulation. J. Simul. 2011,5, 279–291. [CrossRef]
33.
Kim, Y.; Chen, Y.; Linderman, K. Supply network disruption and resilience: A network structural perspective. J Ops. Manag. 2015,
33–34, 43–59. [CrossRef]
34. Frigg, R. Models and Theories: A Philosophical Inquiry, 1st ed.; Routledge: London, UK, 2022; ISBN 978-1-00-328510-6.
35.
Balci, O. Verification, Validation and Testing. In Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice;
Banks, J., Ed.; Wiley: Norcross, GA, USA; Co-Published by Engineering & Management Press: New York, UK, 1998; ISBN
978-0-471-13403-9.
36.
Basole, R.; Bendoly, E.; Chandrasekaran, A.; Linderman, K. Visualization in Operations Management Research. Inf. J. Data Sci.
2022,1, 172–187. [CrossRef]
37.
Shaw, S.; Luckring, J.M.; Oberkampf, W.; Graves, R.E. Exploitation of a Validation Hierarchy for Modeling and Simulation. In
Proceedings of the AIAA SCITECH 2023 Forum, National Harbor, MD, USA, 23–27 January 2023.
38.
Petty, M.D. Verification, Validation, and Accreditation. In Modeling and Simulation Fundamentals; Sokolowski, J.A., Banks, C.M.,
Eds.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2010; pp. 325–372, ISBN 978-0-470-59062-1.
39.
Oberkampf, W.L. Simulation Accracy, Uncertainty, and Predictive Capability: A Physical Sciences Perspective. In Computer
Simulation Validation: Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.;
Simulation Foundations, Methods and Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp.
69–97, ISBN 978-3-319-70765-5.
40.
Sargent, R. Verification And Validation Of Simulation Models: An Advanced Tutorial. In Proceedings of the 2020 Winter
Simulation Conference (WSC), Orlando, FL, USA, 14–18 December 2020; pp. 16–29.
41. Sargent, R. Verification and validation of simulation models. J. Simul. 2013,7, 12–24. [CrossRef]
42.
Sargent, R. Some approaches and paradigms for verifying and validating simulation models. In Proceedings of the Proceeding of
the 2001 Winter Simulation Conference (Cat. No.01CH37304), Arlington, VA, USA, 9–12 December 2001; pp. 106–114.
43. Godfrey-Smith, P. The strategy of model-based science. Biol. Philos. 2007,21, 725–740. [CrossRef]
44.
Saam, N.J. The Users’ Judgements—The Stakeholder Approach to Simulation Validation. In Computer Simulation Validation: Fun-
damental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations,
Methods and Applications; Springer International Publishing: Cham, Switzerland, 2019; pp. 405–431, ISBN 978-3-319-70765-5.
45.
Van Mierlo, S.; Oakes, B.J.; Van Acker, B.; Eslampanah, R.; Denil, J.; Vangheluwe, H. Exploring Validity Frames in Practice. In
Systems Modelling and Management; Babur, Ö., Denil, J., Vogel-Heuser, B., Eds.; Communications in Computer and Information
Science; Springer International Publishing: Cham, Switzerland, 2020; Volume 1262, pp. 131–148, ISBN 978-3-030-58166-4.
46.
Gass, S.I. Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis. Oper. Res. 1983,31, 603–631.
[CrossRef]
47.
Contessa, G. Scientific Models and Representation. In The Continuum Companion to the Philosophy of Science; French, S., Saatsi, J.,
Eds.; Continuum Companions; Continuum: London, UK, 2011; pp. 120–137, ISBN 978-1-4411-8761-1.
48.
Lane, D.C. Validity is a Matter of Confidence-But Not Just in System Dynamics: Validity is a Matter of Confidence. Syst. Res.
2015,32, 450–458. [CrossRef]

Logistics 2025,9, 3 21 of 22
49.
Lamperti, F. Empirical Validation of Simulated Models through the GSL-div: An Illustrative Application. J. Econ. Interact. Coord.
2018,13, 143–171. [CrossRef]
50. Naylor, T.H.; Finger, J.M. Verification of Computer Simulation Models. Manag. Sci. 1967,14, B-92–B-101. [CrossRef]
51.
Beven, K.; Lane, S. Invalidation of Models and Fitness-for-Purpose: A Rejectionist Approach. In Computer Simulation Validation:
Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foun-
dations, Methods and Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 145–171, ISBN
978-3-319-70765-5.
52.
Epstein, J. Why Model? 2008, Volume 11. Available online: https://EconPapers.repec.org/RePEc:jas:jasssj:2008-57-1 (accessed on
23 November 2022).
53.
Marks, R.E. Validating Simulation Models: A General Framework and Four Applied Examples. Comput. Econ. 2007,30, 265–290.
[CrossRef]
54.
Gergen, K.; Gergen, M. Social Construction and Research as Action. In The Sage Handbook of Action Research: Participative Inquiry
and Practice; Reason, P., Bradbury, H., Eds.; SAGE Publications: London, UK; Thousand Oaks, CA, USA, 2008; pp. 159–171, ISBN
978-1-4129-2029-2.
55. Ross, J. Speculative method in digital education research. Learn. Media Technol. 2017,42, 214–229. [CrossRef]
56.
Sands, D. Modeling as sensemaking: Towards a theory of modelling in physics education. Eur. J. Phys. 2021,42, 064001. [CrossRef]
57. Pidd, M. Why modelling and model use matter. J. Oper. Res. Soc. 2010,61, 14–24. [CrossRef]
58.
Hughes, J.; Kornberger, M.; MacKay, B.; O’Brien, P.; Reddy, S. Organizational strategy and its implications for strategic studies: A
review essay. J. Strateg. Stud. 2023,46, 427–450. [CrossRef]
59.
Morgan, M.S.; Morrison, M. (Eds.) Models as Mediators: Perspectives on Natural and Social Sciences; Ideas in Context; Cambridge
University Press: Cambridge, UK; New York, NY, USA, 1999; ISBN 978-0-521-65097-7.
60.
Gelfert, A. Assessing the Credibility of Conceptual Models. In Computer Simulation Validation: Fundamental Concepts, Methodological
Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods and Applications;
Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 249–269, ISBN 978-3-319-70765-5.
61.
Johannesson, P.; Perjons, E. An Introduction to Design Science; Springer International Publishing: Cham, Switzerland, 2014; ISBN
978-3-319-10631-1.
62.
Kabak, K.E.; Hinckeldeyn, J.; Dekkers, R. A systematic literature review into simulation for building operations management
theory: Reaching beyond positivism? J. Simul. 2024,18, 687–715. [CrossRef]
63.
Winsberg, E. Science in the Age of Computer Simulation; University of Chicago Press: Chicago, IL, USA, 2010; ISBN 978-0-226-90202-9.
64. Winsberg, E. Sanctioning Models: The Epistemology of Simulation. Sci. Context 1999,12, 275–292. [CrossRef]
65. Weisberg, M. Who is a Modeler? Br. J. Philos. Sci. 2007,58, 207–233. [CrossRef]
66.
Hoffman, R.N.; Privé, N.; Bourassa, M. Comments on “Reanalyses and Observations: What’s the Difference?“. Bull. Am. Meteorol.
Soc. 2017,98, 2455–2459. [CrossRef]
67.
Imbert, C. The Multidimensional Epistemology of Computer Simulations: Novel Issues and the Need to Avoid the Drunkard’s
Search Fallacy. In Computer Simulation Validation: Fundamental Concepts, Methodological Frameworks, and Philosophical Perspec-
tives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods and Applications; Springer International Publishing:
Berlin/Heidelberg, Germany, 2019; pp. 1029–1055, ISBN 978-3-319-70765-5.
68. Society for Modeling and Simulation International Terminology for model credibility. Simulation 1979,32, 103–104. [CrossRef]
69.
Roache, P. Validation in Fluid Dynamics and Related Fields. In Computer Simulation Validation: Fundamental Concepts, Methodological
Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods and Applications;
Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 661–683, ISBN 978-3-319-70765-5.
70.
Beisbart, C. Should Validation and Verification be Separated Strictly? In Computer Simulation Validation: Fundamental Concepts,
Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods and
Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 1005–1028, ISBN 978-3-319-70765-5.
71.
Sargent, R.; Nance, R.; Overstreet, C.; Robinson, S.; Talbot, J. The Simulation Project Life-Cycle: Models and Realities. In
Proceedings of the 2006 Winter Simulation Conference, Monterey, CA, USA, 3–6 December 2006; pp. 863–871.
72.
Arthur, J.D.; Nance, R.E. Investigating the use of software requirements engineering techniques in simulation modelling. J. Simul.
2007,1, 159–174. [CrossRef]
73.
Jiang, X.; Cheng, X.; Yuan, Y. Validation Using Bayesian Methods. In Computer Simulation Validation: Fundamental Concepts,
Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods and
Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 497–524, ISBN 978-3-319-70765-5.
74.
Cota, B.; Sargent, R. A modification of the process interaction world view. ACM Trans. Model. Comput. Simul. 1992,2, 109–129.
[CrossRef]
75. Sargent, R. Validating Simulation Models; IEEE: Piscataway, NJ, USA, 1983.
76. Muller-Merbach, H. Five notions of OR/MS problems. Omega 2011,39, 1–2. [CrossRef]

Logistics 2025,9, 3 22 of 22
77.
Robinson, S. A tutorial on simulation conceptual modelling. In Proceedings of the 2017 Winter Simulation Conference (WSC), Las
Vegas, NV, USA, 3–6 December 2017; pp. 565–579.
78.
Kalua, A.; Jones, J. Epistemological Framework for Computer Simulations in Building Science Research: Insights from Theory
and Practice. Philosophies 2020,5, 30. [CrossRef]
79. Murray-Smith, D. Some Issues in the Testing of Computer Simulation Models. IJBTE 2016,5, 1–10. [CrossRef]
80. Winsberg, E. Simulated Experiments: Methodology for a Virtual World. Philos. Sci. 2003,70, 105–125. [CrossRef]
81.
Zeigler, B.; Muzy, A.; Kofman, E. Framework for Modeling and Simulation. In Theory of Modeling and Simulation; Elsevier:
Amsterdam, The Netherlands, 2019; pp. 27–41, ISBN 978-0-12-813370-5.
82.
Humphreys, P. Extending Ourselves: Computational Science, Empiricism, and Scientific Method., 1st ed.; Oxford University Press: New
York, NY, USA, 2004; ISBN 978-0-19-515870-0.
83.
Arnold, E. Validation of Computer Simulations from a Kuhnian Perspective. In Computer Simulation Validation: Fundamental
Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods
and Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 203–224, ISBN 978-3-319-70765-5.
84.
Beisbart, C. What is a Computer Simulation and What does this Mean for Simulation Validation? In Computer Simulation
Validation: Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation
Foundations, Methods and Applications; Springer International Publishing: Cham, Switzerland, 2019; pp. 901–923, ISBN
978-3-319-70765-5.
85. Suárez, M. An Inferential Conception of Scientific Representation. Philos. Sci. 2004,71, 767–779. [CrossRef]
86.
Robinson, S.; Arbez, G.; Birta, L.; Tolk, A.; Wagner, G. Conceptual modelling: Definition, purpose and benefits. In Proceedings of
the 2015 Winter Simulation Conference (WSC), Huntington Beach, CA, USA, 6–9 December 2015; pp. 2812–2826.
87.
Robinson, S. Simulation: The Practice of Model Development and Use, 2nd ed.; Palgrave Macmillan: Hampshire, UK; New York, NY,
USA, 2014; ISBN 978-1-137-32802-1.
88.
Robinson, S. Conceptual modelling for simulation Part I: Definition and requirements. J. Oper. Res. Soc. 2008,59, 278–290.
[CrossRef]
89.
Murray-Smith, D. The Use of Experimental Data in Simulation Model Validation. In Computer Simulation Validation: Fundamental
Concepts, Methodological Frameworks, and Philosophical Perspectives; Beisbart, C., Saam, N.J., Eds.; Simulation Foundations, Methods
and Applications; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 357–382, ISBN 978-3-319-70765-5.
90.
Morse, J.M.; Bowers, B.J.; Charmaz, K.; Clarke, A.E.; Corbin, J.; Porr, C.J.; Stern, P.N. Developing Grounded Theory: The Second
Generation Revisited, 2nd ed.; Routledge: New York, NY, USA, 2021; ISBN 978-1-315-16917-0.
91.
Wartofsky, M. The Model Muddle: Proposals for an Immodest Realism. In Models. Boston Studies in the Philosophy of Science;
Springer: Dordrecht, The Netherlands, 1966; Volume 48, ISBN 978-90-277-0947-9.
92.
Guiguet, A.; Pons, D. A Framework for Interactive Development of Simulation Models with Strategical–Tactical–Operational
Layering Applied to the Logistics of Bulk Commodities. Modelling 2022,3, 272–299. [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.