PreprintPDF Available

Continuous calibration of a digital twin: comparison of particle filter and Bayesian calibration approaches

Authors:
Preprints and early-stage research may not have been peer reviewed yet.

Abstract

Assimilation of continuously streamed monitored data is an essential component of a digital twin; the assimilated data are used to ensure the digital twin is a true representation of the monitored system. One way this is achieved is by calibration of simulation models, whether data-derived or physics-based, or a combination of both. Traditional manual calibration is not possible in this context hence new methods are required for continuous calibration. In this paper, a particle filter methodology for continuous calibration of the physics-based model element of a digital twin is presented and applied to an example of an underground farm. The methodology is applied to a synthetic problem with known calibration parameter values prior to being used in conjunction with monitored data. The proposed methodology is compared against static and sequential Bayesian calibration approaches and compares favourably in terms of determination of the distribution of parameter values and analysis run-times, both essential requirements. The methodology is shown to be potentially useful as a means to ensure continuing model fidelity.
Data Centric Engineering (2020), xx:xx 1–23
doi:10.1017/xxxx
RESEARCH ARTICLE
Continuous calibration of a digital twin: comparison of par-
ticle filter and Bayesian calibration approaches
Rebecca Ward1,2* , Ruchi Choudhary1,2, Alastair Gregory1and Mark Girolami1,2
1Data-centric Engineering, The Alan Turing Institute, The British Library, 96 Euston Road, NW1 2DB, London, UK.
2Engineering Department, University of Cambridge, Trumpington Street, CB2 1PZ, Cambridge, UK.
*Corresponding author. E-mail: rward@turing.ac.uk
(Received 10 October 2011)
Keywords: Digital twin, Bayesian calibration, Particle filter
Abstract
Assimilation of continuously streamed monitored data is an essential component of a digital twin; the assimilated
data are used to ensure the digital twin is a true representation of the monitored system. One way this is achieved is
by calibration of simulation models, whether data-derived or physics-based, or a combination of both. Traditional
manual calibration is not possible in this context hence new methods are required for continuous calibration. In
this paper, a particle filter methodology for continuous calibration of the physics-based model element of a digital
twin is presented and applied to an example of an underground farm. The methodology is applied to a synthetic
problem with known calibration parameter values prior to being used in conjunction with monitored data. The
proposed methodology is compared against static and sequential Bayesian calibration approaches and compares
favourably in terms of determination of the distribution of parameter values and analysis run-times, both essential
requirements. The methodology is shown to be potentially useful as a means to ensure continuing model fidelity.
Impact Statement
This study addresses the problem of ensuring that a simulation model is able to continuously assimilate
data and identify variation in the underlying model parameters. As the availability of monitored data rapidly
increases across diverse industries, the requirement for tools that are able to make timely use of the data is
also on the increase. While the technology for developing the simulation models already exists, in many fields
there is a dearth of tools that can facilitate rapid interpretation of streamed data to ensure continuing model
fidelity. The methodology presented here potentially offers one way to fill this gap by proposing a sequential
calibration process using a particle filter approach. The proposed approach is illustrated by application to a
simulation model of an underground farm.
1. Introduction
The technological advancement and drop in price of monitoring equipment has led to a boom in avail-
ability of monitored data across industrial fields as diverse as aviation, manufacturing and the built
environment. This facilitates the development of digital twin technology. The specification of what
constitutes a digital twin is still evolving, but the essence comprises a monitored system together with a
computational model of the system which demonstrably replicates the system behaviour in the context
of interest (Worden et al. (2020)). The computational model may be either data-derived or physics-
based; for engineering systems it is more common for a physics-based model to be used as this offers
the advantage of representing the salient physical laws that govern the system. The intention is that the
© Cambridge University Press 2019. This is an Open Access article, distributed under the terms of the CreativeCommons Attribution licence (http://
creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work
is properly cited.
arXiv:2011.09810v1 [cs.CE] 19 Nov 2020
2 Ward, R et al
computational model can give information that may not be easily accessible from the system directly,
and can be used to explore performance when it is impractical to run physical tests. The greatest poten-
tial for digital twinning perhaps lies in systems which are continuously operational generating live
streamed data that inform the model, in which case the computational model can be simulated in (close
to) real-time to advise changes to operational parameters for improved efficiency (Madni et al. (2019)).
Recent advancements in machine learning techniques are leading to a new generation of compu-
tational models that combine the physics and the data. On the one hand, physics enhanced machine
learning can improve a data-derived model by constraining it with the governing scientific laws (Choud-
hary et al. (2020)). A more common practice is to start with a physics-based model and incorporate the
data to calibrate the model and ensure it matches reality as closely as possible. This then means that
when exercising the model beyond the realm of the data we can have confidence in the model pre-
dictions. Calibration can be performed manually but it is time consuming and is not always practical;
indeed in the situation where model parameters are not static but dynamic it may be impossible owing to
the dimensions of the parameter space and the conflation between different reasons for the differences
between model outputs and the data. In the case of a digital twin, an automated calibration process that
forms an integral part of the system model is favourable, and thus a computationally expensive and
complex calibration process is not well suited.
Bayesian calibration (BC) offers a formal way to combine measured data with model predictions
to improve the model. In the field of building energy simulation considered here, the Bayesian frame-
work proposed by Kennedy and O’Hagan (Kennedy and O’Hagan (2001)) (KOH) has been explored in
some depth (Higdon et al. (2004); Heo et al. (2012); Li et al. (2016); Chong et al. (2017)), specifically
for the identification of static model parameters and the quantification of uncertainty. Bayesian calibra-
tion has been shown to be the optimal approach as it incorporates prior knowledge into the calibration
process which can significantly improve parameter identifiability. In addition, the KOH methodology
enables not only the inference of uncertain (and important) model parameters but also of the model
deficiency (model bias) and errors in observations. As such, the KOH methodology is considered the
’gold standard’ for calibration of computer models. However, the KOH methodology can be computa-
tionally expensive, increasingly so as the numbers of calibration parameters and data points increase,
owing to the typical Markov chain Monte Carlo (MCMC) implementation which generates a random
walk through the target distribution and is inherently inefficient (Chong et al. (2017)). This has impli-
cations for use in continuous calibration over short timescales - the run-time of the calibration must
be shorter than the time interval between acquisition of new data points, and yet there must be suffi-
cient data points to characterise the parameter space. Nonetheless there has been some exploration of
its extension to dynamically varying parameters. For example, Chong et al. (2019) used the KOH for-
mulation in conjunction with data from building energy management systems to continuously calibrate
a building energy model, updating the model every month, demonstrating that prediction of future per-
formance is improved with continuous calibration. The study required selection of a reduced sample
size and a more efficient MCMC algorithm to overcome the computational challenge presented by the
large dataset.
An alternative approach that potentially offers a time-efficient solution to the problem of continuous
calibration is Particle Filtering (PF). This is a sequential Bayesian technique in which a large sample of
possible parameter estimates are filtered according to their likelihood given some data. The use of par-
ticle filters for static parameter estimation is described in some detail by Andrieu et al. (2004), and the
approach has been used for sequential data assimilation in other fields for which the memory effect has
a significant impact on parameter values, e.g. hydrology (Moradkhani and Hsu (2005)), or for estima-
tion of dynamic state and parameter values in nonlinear state-space models e.g. Cheng and Tourneret
(2017). The novelty of the study presented here is its potential impact on digital twin technology. The
parameters are not memory-dependent but are time-varying and we require up to date estimates of the
parameter values to ensure that forecasting using the physics-based model is as accurate as possible.
This is particularly important if the physics-based model can only be a relatively simple representation
of the real system - which is often the case. To this end, we explore whether the PF approach can offer
Data Centric Engineering 3
a suitable calibration mechanism. We use monitored data in conjunction with the particle filter to cal-
ibrate uncertain model parameters - we then use the estimates of the model parameters in conjunction
with the model to predict model performance with a quantification of the uncertainty. The PF is also
compared against a sequential BC model using the KOH formulation. While the KOH approach is typ-
ically used to get the best estimates of static parameter values, in this study we also explore whether it
is feasible to use it with sequentially changing datasets. We demonstrate that while both methods are
suitable, the PF method is quicker without losing accuracy.
The digital twin considered here is a farm constructed in a previously disused underground tunnel,
in which salad crops are grown hydroponically. A physics-based simulation model of the farm has been
developed that calculates temperature and relative humidity as a function of external weather conditions
and farm operational strategies. This is a relatively simple model based on a single thermal zone that
simulates heat and mass transfer between the different system components. An extensive programme of
monitoring has also been carried out, so data are available for calibration of the model. The aim of the
digital twin (or the physics-based simulation model) is that at any time it can be used in conjunction with
forecast weather data and operational scenario of the farm to predict environmental conditions within
the tunnel with a view to alerting the farm operators to future potentially unsatisfactory conditions.
Calibration is a critical component of the digital twin system in this instance, as it is the only mechanism
by which the model can be adapted to represent the farm environment to an adequate degree of realism.
The paper is laid out as follows; in the following section the calibration approaches are compared
in more detail. We then describe the physics-based model of the underground farm. The feasibility of
the PF and sequential KOH approaches are first tested with synthetic data and known parameters, and
thereafter implemented with monitored data from the farm. In the discussion we explore the extent to
which the parameter values identified a) are the values that give the best fit of the model to the data, and
b) are indicative of ’real’ values, and compare the approaches used in terms of their applicability in the
context of a digital twin.
2. Calibration approach
In this study we consider a static Bayesian calibration approach as formulated by Kennedy and O’Hagan
(2001) (KOH) to be the basis against which we compare the viability of the proposed particle filter
approach. We thus also use the static KOH approach in a sequential manner to explore the comparison
as detailed below. Both approaches make use of Bayes’ formula, i.e. for a parameter 𝑌and an observed
data point 𝑦(𝑥),
𝑃(𝑌|𝑦(𝑥)) =𝑃(𝑦(𝑥) |𝑌).𝑃 (𝑌)
𝑃(𝑦(𝑥)) (1)
or, the posterior probability of the parameter value 𝑌, given the observed data point 𝑦(𝑥)is equal to
the likelihood of the observed data point, 𝑃(𝑦(𝑥)|𝑌), multiplied by the probability of 𝑌before making
the observation - the prior probability, 𝑃(𝑌)- all normalised by a normalisation factor 𝑃(𝑦(𝑥)).
2.1. The KOH Calibration Framework
In the Bayesian calibration formulation proposed by Kennedy and O’Hagan (Kennedy and O’Hagan
(2001)), a numerical simulator 𝜂(𝑥, 𝜃)can be related to field observations 𝑦(𝑥)by the following
equation:
𝑦(𝑥)=𝜂(𝑥, 𝜃) + 𝛿(𝑥) + 𝜖+𝜖𝑛(2)
where 𝑥are observable inputs into the model or scenarios, for example location of sensor or time
of sensing, and 𝜃represents the true but unknown values of the parameters 𝜃which characterise
the model. This formulation inherently expects a discrepancy, or bias, between the model and reality,
4 Ward, R et al
Model hyperparameter Prior distribution
𝜆𝜂Gamma(10,10)
𝜆𝑏Gamma(10,0.3)
𝜆𝑒Gamma(10,0.03)
𝜆𝑒𝑛 Gamma(10,0.001)
𝛽𝜂Beta(1,0.5)
𝛽𝑏Beta(1,0.4)
Table 1: KOH approach hyperparameter prior distributions
accounted for by 𝛿(𝑥).𝜖represents the observation error and 𝜖𝑛the numerical error. Calibration of the
model aims to identify the parameters 𝜃; whereas traditional calibration approaches require multiple
runs of the computer simulation with systematic variation of the input parameters and subsequent iden-
tification of the combination of parameters that gives the closest match to reality, the KOH framework
offers a more efficient way to identify the uncertainty associated with the calibration parameters. Rather
than using an exhaustive iterative approach, it is common practice to use an emulator to map the model
inputs to the outputs. Gaussian process (GP) models are commonly used as the basis for the emula-
tor, with separate models used for the simulator 𝜂(𝑥, 𝜃)and the discrepancy term 𝛿(𝑥)in the above
equation. These GP models have their own hyperparameters, specifically precision hyperparameters 𝜆𝜂
and 𝜆𝑏that relate to the magnitude of the emulator and model discrepancy respectively, and 𝛽𝜂and
𝛽𝑏that determine the correlation strength in each along the dimensions of 𝑥and 𝜃and determine the
smoothness of the emulator and discrepancy function. The random error terms for measurement and
numerical error are included as unstructured error terms and are incorporated into the GP covariance
matrix with precision hyperparameters 𝜆𝑒and 𝜆𝑒𝑛. All of these parameters are uncertain and require
a prior probability distribution to be specified in the KOH approach. We have used prior distributions
suggested in previous papers for the hyperparameters as detailed in Table 1(Menberg et al.,2019).
The practical application of the KOH approach to calibration of building energy simulation models
is described in detail by Chong and Menberg (2018), and the implementation of the framework is as
described by Guillas et al. (2009). The procedure is to compare measured data (observations) against
outputs from computer simulations generated using plausible ranges of uncertain input parameters
under known scenarios. By exploring the likelihood of the data given the simulation output, the poste-
rior distribution of each calibration parameter is derived. The process, in sum, requires the following
steps:
Sensitivity analysis to determine the model parameters that have the greatest impact on the
simulation output.
Perform calibration runs - run simulations over the plausible range of the parameters identified in
the sensitivity analysis, varying one parameter at a time.
Acquire field data - monitored observations corresponding to simulation output,
Fit GP emulator to the field data and simulation output by calculating the mean and covariance
function as described by Higdon et al. (2004),
Explore the parameter space using MCMC and extract the posterior distributions of the calibration
parameters.
In this process, all the monitored data and simulation model outputs are included in the model at
once, in the GP emulator, hence the run time of calibration is very dependent on the number of data
points, increasing in proportion to the square of the size of the dataset. This means that it is impractical
to perform Bayesian calibration with a large number of observations.
The KOH framework also has at its heart an assumption of stationarity i.e. the calibration parameters,
𝜃, are assumed to be constant over the range of the data. We are interested in parameters that are
not necessarily constant over the timescales considered. However, they may be essentially constant
Data Centric Engineering 5
over shorter time periods, so it is possible to use the KOH framework to estimate the parameters over
these shorter time periods. Given that assumption, it is feasible to implement the KOH calibration over
successive time periods: at each timestep the oldest data point is removed and a new data point added.
In that instance the posterior distribution for the previous timestep may be used as the prior distribution
for the new dataset. In essence this is a similar approach to the particle filter with the exception that data
points are not considered singly but in groups over which the calibration parameter values are assumed
to be constant. The potential benefit of this is that it facilitates exploration of the error terms in detail.
2.2. Particle filtering
A particle filter is a sequential Bayesian inference technique in which a large sample of possible param-
eter estimates are filtered according to their likelihood given some data. As an example, consider a
virus spreading through a community of people. With no additional knowledge we can only guess
which members of the population have the virus. If then we get some information - say measured tem-
peratures for each member, where an increased temperature is one of the symptoms of the virus - we
can update the likelihood of each of our guesses according to their measured temperature. We then
re-sample from our population taking the increased likelihood into account, generating a new sample
where each member again has an equal probability of having the virus and predict what the new tem-
peratures will be based on our updated estimates of the location of the virus hosts. When we get new
temperature information we update the likelihood again and repeat the re-sampling and prediction.
In the context of this study, the particles consist of possible values of the uncertain model
inputs/parameters and the information is the monitored data. Based on the data, we update the like-
lihood of the particles i.e. we look to see how close the monitored data is to the outputs of the model
with the parameter values assigned to each particle. We then re-sample the particles taking the like-
lihood into account and generate a new set of particles with equal weight. We repeat the process by
updating the likelihoods of the new particle set using the next value of monitored data and the values
predicted by the model using the parameter values from the new particles - and we then re-sample. This
repeating process results in a set of values for the model parameters that continuously update in line
with the monitored data. The re-sampling is an essential part of the filter to avoid degeneracy i.e. to
avoid the situation where a few particles dominate the posterior distribution, as discussed in detail by
Doucet and Johansen (2008).
Mathematically, the framework for the particle filtering approach is as follows: consider the model
𝜂(𝜃), where 𝜃are parameters of the model. We conjecture that observed data are given by 𝑦𝜌𝜂 +𝛿+𝜖,
where 𝜌is a scaling parameter, 𝛿is a mean-zero Gaussian process representing the mismatch between
the model and the data and 𝜖is the measurement error with variance 𝜎2. We assume that the smoothness
of the emulator and model mismatch are determined by a length scale, 𝑙.
1. Start by sampling 𝑁different particles of the hyperparameters 𝑙,𝜌and model input parameters 𝜃
from the prior distributions 𝑝(𝑙),𝑝(𝜌)and 𝑝(𝜃). Denote these particles {𝑙𝑗}𝑗=1:𝑁,{𝜌𝑗}𝑗=1: 𝑁and
{𝜃𝑗}𝑗=1:𝑁respectively.
2. Obtain the model outputs 𝑑𝑖taken at the coordinates 𝑋𝑑
𝑖and with the model input parameters 𝑡𝑖.
Also consider the monitored data 𝑌𝑖taken at the coordinates 𝑋𝑌
𝑖.
3. Define the covariance matrices (for each of the particles),
𝐾𝑗=𝑘𝜂(𝑋𝑑
𝑖, 𝑋𝑑
𝑖, 𝑡𝑖, 𝑡𝑖|𝑙𝑗)𝜌𝑗𝑘𝜂(𝑋𝑑
𝑖, 𝑋𝑌
𝑖, 𝑡𝑖, 𝜃 𝑗|𝑙𝑗)
𝜌𝑗𝑘𝜂(𝑋𝑌
𝑖, 𝑋𝑑
𝑖, 𝜃 𝑗, 𝑡𝑖|𝑙𝑗)𝜌2
𝑗𝑘𝜂(𝑋𝑌
𝑖, 𝑋𝑌
𝑖, 𝜃 𝑗, 𝜃 𝑗|𝑙𝑗) + 𝑘𝑌(𝑋𝑌
𝑖, 𝑋𝑌
𝑖, 𝜃 𝑗, 𝜃 𝑗|𝑙𝑗) + 𝜎2I
using the covariance functions 𝑘𝜂(𝑥, 𝑥 0, 𝑡, 𝑡0|𝑙)associated with the computer model emulator, 𝜂
and 𝑘𝑌(𝑥, 𝑥 0, 𝑡, 𝑡0|𝑙)associated with the model-data mismatch 𝛿. Note that these are conditioned
on the hyperparameter, 𝑙.
4. For each of the particles, compute the mean-zero Gaussian process marginal likelihoods
𝜉𝑗=𝑝([𝑑𝑖, 𝑌𝑖]𝑇|0, 𝐾 𝑗). These will be used to update the posterior distribution of 𝑙,𝜌and 𝜃in the
next step.
6 Ward, R et al
5. Compute the normalized weights 𝑤𝑗=𝜉𝑗/Σ𝑁
𝑗=1𝜉𝑗. Re-sample using these weights, such that the
new sample of particles all have even weight.
6. Increase 𝑖by one, and repeat steps (2) to (5) with this new sample of particles.
Throughout this process, the set of particles associated with the model input parameters provide
an empirical approximation to their posterior distribution. The implementation of this approach has
been written in Python and has used the Gaussian Process tools included in the Scikit-learn package
(Pedregosa et al.,2011). In this implementation, we assume a value for the scaling parameter, 𝜌=1,
to match the context in Equation 2. For the benefit of brevity in explaining the approach, we also set
the variance of the measurement error, denoted by 𝜎2, to the mean of the prior distribution in Table 1,
although this isn’t essential.
3. Digital Twin of the Underground Farm
Growing Underground is an unheated hydroponic farm developed in disused tunnels in London, UK.
The farm grows salad crops for sale to restaurants and supermarkets across London using artificial
lighting while relying on the relatively stable thermal ground conditions. The tunnels were built in
the 1940s to act as air raid shelters but with the intention of ultimately being incorporated into the
London Underground rail network. This never happened and the tunnels have been disused since acting
as temporary housing for migrants in the 1960s. They are 33m below ground, a location where the
deep soil temperature is approximately constant, and enjoy some heat transferred from the nearby rail
tunnels. The heat source however is predominantly the LED growing lights; these are typically switched
on overnight and turned off during the day time so as to compensate for external daily temperature
changes which tend to be colder at night and warmer in the day. The farm is ventilated to the outside
air using the original ventilation system designed in the 1940s. There are two ventilation shafts which
operate in tandem, one at Carpenter’s Place (CP) at the main tunnel entrance and one at Clapham
Common (CC) at the far end of the tunnel. These are manually operated and both affect the ventilation
rate experienced in the farm.
It is important to maintain environmental conditions at optimum levels for plant growth and to this
end the space is monitored for temperature, relative humidity and CO2levels (Jans-Singh et al. (2019)).
The purpose of the monitoring is both to facilitate the maintenance of adequate conditions for optimal
crop growth and also to help optimise the energy use. The tunnel environment is complex to understand,
primarily because of the age and lack of information relating to the design of the tunnels and change in
system properties over time. Traditional control systems developed for the horticultural industry cannot
be used here as there is limited scope for automatic control of the tunnel environment. The aim of the
digital twin is to combine the monitored data with a physics-based model which simulates the heat and
mass exchanges within the tunnel and with the external environment. By combining both monitoring
and simulation we aim to develop a system which can be used to analyse performance, to predict future
environmental conditions in conjunction with forecast weather data, and to help optimise the designs
for expansion of the farm.
3.1. The Simulation Model
A simple physics-based numerical model has been developed representing a 1D slice through the central
section of the farm. The model calculates temperature and relative humidity of the tunnel air as a
function of time by solving heat and mass exchange equations pertinent to the tunnel geometry, subject
to the temperature, moisture content and CO2concentration of the incoming ventilation air and the
deep soil temperature. The primary source of heat input is the LED lights which are typically operated
Data Centric Engineering 7
overnight and switched off during daytime. For example, the time-dependent temperature variation can
be calculated through a 1D slice of the tunnel using the heat balance equation:
𝑑𝑇𝑗
𝑑𝑡 =𝐴𝑔
𝑚𝑗𝑐𝑗
𝑖
𝑞𝑖, 𝑗 (3)
which gives the temperature, 𝑇in each layer, 𝑗- such as the growing medium, vegetation, air and
tunnel lining - as a function of the heat flows 𝑞between the different layers. Here, 𝑖represents the
different heat transfer processes such as radiation, convection, etc., 𝐴𝑔is the surface area, 𝑚is the mass
of each layer and 𝑐is the heat capacity.
In a similar manner, the moisture content of the air can be calculated from the mass balance equation:
𝑑𝐶𝑎
𝑑𝑡 =𝐴𝑔
𝑓 𝑔𝑉
𝑘
𝑞𝑘(4)
where 𝐶𝑎is the moisture content of the air, 𝑓 𝑔 is the latent heat of condensation of water, 𝑉is the
volume of the unit and 𝑞𝑘are the heat flows from the 𝑘different latent heat transfer processes. The
moisture content and temperature are inextricably linked as changes to the moisture content of the air
are associated with latent heat transfer. Hence the equations require solving in parallel to determine the
temperatures and relative humidities within the tunnel.
The model is also dependent on the operational conditions in the farm such as ventilation rate and
internal air speed. These parameters are difficult to measure accurately and only partial information is
available concerning the design of the tunnel ventilation system. In addition, the simplicity of the model
means that parameters may represent a mean effect of different physical components e.g. the combined
effect of infiltration and controlled ventilation are represented in the model as a single ventilation rate.
Calibration of the model and estimation of the parameters therefore becomes an important step in
improving the model accuracy. But the parameters are not constant and therefore it becomes necessary
to re-calibrate the model to infer the change in parameter values. The problem is that it is not always
obvious when parameters are changing - some changes, such as manual alterations to the ventilation
settings are known, but the impact of the alteration in terms of the change to the ventilation rate is not
a linear function of the dial settings.
This need to re-calibrate, together with the availability of continuously monitored data, makes
development of a sequential calibration process particularly desirable.
3.2. Sensitivity study
For calibration of the model, the first step is to ascertain which model output is the most important with
respect to farm environment. From the viewpoint of the growers the air temperature and the relative
humidity are the main quantities of interest. Plants grow best in optimal temperature conditions and
if the the relative humidity gets too high, growth of mould can occur affecting crop yield. So the
calibration of the model focuses on these model outputs.
A sensitivity study has been carried out using the Morris method (Menberg et al. (2016)) to identify
the relative significance of each input parameter with respect to the quantities of interest. The 8 input
parameters of primary importance for the heat and mass balance and their plausible ranges are given
in Table 2. The parameter ranges given in the table have been derived either from measurements, are
anecdotal, or have been estimated based on an understanding of the farm processes. For example, the
fraction of planted area approaching harvest has been estimated based on the records of the number
of newly planted trays and the number of trays being harvested at any time. Equally, the mean mat
saturation is estimated based on the watering strategy employed in the farm in which the trays are
flooded periodically and then drained over a period of time. The internal air speed has been measured
(Jans-Singh et al. (2019)). The ventilation rate range is based on the tunnel ventilation system having
been designed for an air change rate of 4𝐴𝐶𝐻 but with the knowledge that additional uncontrolled
8 Ward, R et al
Model parameter Symbol Min Max
Fraction of planted area approaching harvest 𝐴𝐹𝑔0.25 0.375
Internal air speed IAS 0.1 0.85
Ventilation rate ACH 1 10
Fraction of lighting power output as heat 𝑓ℎ𝑒𝑎𝑡 0.7 0.9
Surface temperature of lights 𝑇𝑎𝑙 22.5 27.5
Characteristic leaf dimension 𝑑𝑣0.01 1
Characteristic mat dimension 𝑑𝑚0.1 1
Mat saturation 𝑑𝑠 𝑎𝑡 0.4 0.6
Table 2: Uncertain parameters assessed in model sensitivity analysis
ventilation can occur through the access shafts. The fraction of the lighting power that is emitted as
heat has been estimated according to values found in the literature (e.g. LED Magazine (2005)), and the
surface temperature of the lights is based on estimates provided by farm personnel. The characteristic
dimensions of the mat and leaf control whether the heat convection is primarily characterised by bulk
heat transfer from the entire tray area - in which case the characteristic dimension is similar to that of the
tray i.e. 1, or whether it is more characterised by the dimension of the leaf or the gaps between plants.
The Morris method works by using a factorial sampling strategy to discretise the parameter space in
order to ensure that the entire space is covered efficiently. In this way for 10 trajectories for each of the
8 parameters, 80 different combinations of the parameters have been run with each combination only
one parameter value different from the previous one. A statistical evaluation of the elementary effects
(EEs), i.e. the impact of a change in one parameter by a pre-defined value on the output of the model,
has been performed using the absolute median and standard deviation of the EEs (Menberg et al.,2019).
The results of the sensitivity study are summarised in Figure 1. This shows the standard deviation as
a function of the absolute median of the EEs when considering the summed absolute difference between
the predicted and monitored a) temperature, and b) relative humidity, over a 30 day period. A higher
value for the standard deviation for a specific parameter implies that changes to that parameter have a
more significant impact on the output of the model. Figure 1a indicates that the most influential param-
eters for temperature are the ventilation rate, 𝐴𝐶 𝐻, and the internal air speed, 𝐼 𝐴𝑆, closely followed
by the fraction of energy emitted as heat by the lights, 𝑓ℎ 𝑒𝑎𝑡 . For relative humidity, Figure 1b indicates
that the internal air speed is very significant, followed again by 𝑓ℎ𝑒 𝑎𝑡 . The ventilation rate, 𝐴𝐶𝐻 is the
next most significant parameter. The heat fraction of the lighting has a significant impact on the sim-
ulation output and we could calibrate the model for this parameter to get a better estimate of the true
value; however, we cannot influence the value in any way and so we choose to focus on the two param-
eters over which we have some control i.e. the ventilation rate and internal air speed. This approach is
directed by the digital twin ethos in that we wish to be able to forecast the impact of changes to the
operational conditions on the tunnel internal environment.
Relative humidity is itself a function of temperature, so rather than calibrating the model separately
for both parameters, we have chosen to calibrate for relative humidity alone.
4. Calibration results
4.1. Toy problem
As a test of the approach we start by fixing values for 𝐴𝐶 𝐻 and 𝐼 𝐴𝑆 and run the simulation for 20
days to generate synthetic test data. These data are illustrated in Figure 2. The simulation gives an
hourly output, plotted in blue, but for calibration of the model we select values every 12 hours (shown
as circles) so as to retain a manageable amount of data for the KOH approach. As a result, we have two
data points per day: one when the LED lights are switched on and the second when the LED lights are
switched off. The internal air speed is fixed at a constant value of 0.3𝑚/𝑠. The ventilation rate as model
Data Centric Engineering 9
0 1000 2000 3000 4000
eff median
0
1000
2000
3000
4000
eff std
AFg
IAS
ACH
fheat
Tal
dv
dm
dsat
(a) Sensitivity analysis results for temperature, T
0 20 40 60
eff median
0
10
20
30
40
50
60
eff std
AFg
IAS
ACH fheat
Tal
dv
dm
dsat
(b) Sensitivity analysis results for relative humidity, RH
Figure 1: Sensitivity analysis results showing the impact of each parameter on the simulation output -
higher values imply a more significant effect
input is initially fixed, then changed halfway through the simulation period, so that for the first 20 data
points the ventilation rate is 4𝐴𝐶𝐻 , dropping to 2𝐴𝐶𝐻 for the remaining period. The values chosen
are for illustration but we believe they are typical of the magnitude of the ventilation rate in the tunnel.
The purpose of changing the ventilation rate halfway through the simulation is to investigate whether
the sequential KOH and the PF approach are capable of inferring the change in parameter value.
As aforementioned, the synthetic test data have been generated from the simulation with fixed
parameters. We also require outputs from the simulation over a range of parameter values for input into
the KOH and PF calibration frameworks. For this we run the simulation with a range of values of the
two parameters of interest: for ventilation rate we use a range from 1 to 10 ACH in steps of 1.5 ACH,
and for internal air speed we use a range from 0.1 to 0.85 m/s in steps of 0.15 m/s: taking all combi-
nations gives a total of 42 calibration runs. Again, this toy problem is an illustration, but the range of
values selected are in line with the values we expect to be realistic in the tunnel situation.
We use the light state - on/off - in conjunction with the external air moisture content as scenarios
under which the model is executed. Within the KOH formulation these are defined as 𝑥and we seek
posterior estimates of the parameters 𝜃, in this case 𝐴𝐶 𝐻 and 𝐼 𝐴𝑆 . The rationale for selecting light
state as one of the 𝑥values is that the lights are the main providers of heat in the system and as we
expect to observe a diurnal variation in temperature and hence relative humidity according to whether
the lights are on or off, it is essential to know the light state. The external moisture content of the air
makes a less significant contribution, but it does directly impact on the relative humidity dependent on
the ventilation rate.
The test data are plotted as a function of these two parameters in Figure 3, together with the output
of the simulations (shown in blue), showing that the synthetic test data fall within the range of outputs
from the simulation runs, as expected.
The KOH approach uses a MCMC technique for exploration of the parameter space: for all of the
simulations performed here we have used 3 chains of 5000 iterations as this number has typically given
good levels of convergence.
Considering the first 20 data points which form the first half of the test data (shown as red open
circles on Figure 2), using the KOH approach with a uniform prior distribution for each of the two
calibration parameters gives the posterior distributions for the calibration parameters shown in Figure
4. These plots show the uniform prior distributions in red, with the posterior distributions shown in
blue. Also indicated on the figure are the true values, ACH = 4 and IAS = 0.3. As expected, the KOH
calibration results in estimates for the calibration parameters which are centred about a mean value close
to the true value. Extending the KOH approach to all 40 data points gives the posterior distributions
illustrated in Figure 5. This shows nicely the issue with the KOH approach for time-varying parameters.
10 Ward, R et al
0 5 10 15 20
Day
0.4
0.5
0.6
0.7
0.8
Relative Humidity
Simulation
Data 1 - 20
Data 21 - 40
Figure 2: Synthetic data: the solid line is the output of the simulation model with fixed parameters
ACH=4, IAS=0.3. The data points used for the calibration are indicated as red circles (time period 1)
and green dots (time period 2)
(a) Light State On/Off
0.006 0.007 0.008 0.009 0.01 0.011 0.012 0.013 0.014 0.015
External Cw
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
RH
Simulation
Data 1-20
Data 21-40
(b) External Moisture Content, 𝐶𝑤
Figure 3: Toy Problem: Synthetic data and model outputs plotted along the scenarios 𝑥for KOH
approach
The simulation results in a value for ACH midway between the two true values - ACH = 4 for the first
half of the simulation and ACH = 2 for the second half - whereas the mean value of IAS is close to the
true value which remains static throughout the simulation.
One of the benefits of the KOH approach is that it gives posterior distributions not only for the cal-
ibration parameters but also for the model hyperparameters and hence the error terms (see description
following Equation 2). Figure 6shows a comparison of the prior and posterior distributions for the four
precision hyperparameters for simulation of the first 20 data points. While the posterior distribution of
the emulator precision hyperparameter is very similar to the prior, the model discrepancy or bias param-
eter has a median value lower than our prior assumption, whereas the random error and numerical error
are higher. This suggests that the calibration is not identifying a systematic error from the information
provided, but instead is absorbing all differences between the model and the data into the random and
numerical errors.
The particle filter methodology has also been run with the synthetic test data. In this approach 1000
particles are initiated comprising values for the two calibration parameters, 𝜃and the lengthscale, 𝑙. The
𝜃s are sampled from an initial uniform prior distribution over the possible range of values, whereas the
values for 𝑙are sampled from a lognormal distribution to ensure they are positive. At the first timestep
the particle filter compares the value of relative humidity that would be predicted by the simulation
using the values from each particle against the true data. Here we do not run the simulation explicitly
Data Centric Engineering 11
Figure 4: Toy problem: prior and posterior parameter probability distributions for KOH approach,
period 1 only
Figure 5: Toy problem: prior and posterior parameter probability distributions for KOH approach,
period 1-2
for each particle at each timestep but instead emulate the simulation output using a Gaussian process
fitted to the simulation results over the range of calibration parameters run prior to the particle filtering.
The likelihood of each particle is then calculated based on the comparison, and weights are assigned
to each particle according to the likelihood. Particles are re-sampled from the prior distribution taking
the particle weights into account and this new distribution forms the prior for the next step. The key
difference between this approach and the KOH approach is that here each data point is considered
sequentially whereas in the KOH approach all data points are considered together.
The evolution of the posterior probability distribution for each of the calibration parameters is illus-
trated in Figure 7. Each plot shows the evolution in the parameter value distribution, starting from the
uniform prior distribution at the top, progressing in steps of 4 data points to the end of the simulation
at the bottom. Note that all 40 data points were used in the simulation, but the output for the interme-
diate points is suppressed for brevity (as might be inferred from the figures, the suppressed values are
very similar to those shown on either side). Here, the change in ventilation rate at the mid-point of the
simulation is clearly visible (Figure 7(a)), and although there is a small perturbation to the values of the
internal air speed (Figure 7(b)), the distribution recovers quickly and the mean stays close to the true
value throughout the simulation. Figure 7(c) shows the evolution of the length scale parameter for this
test case. The distribution of values shows a wider spread at the start of the analysis, but this narrows
as the analysis progresses and the mean value remains close to a value of 0.2 throughout. This is a true
unknown parameter as it is a hyperparameter of the emulator, and approximately reflects the distance
in the input space that you have to travel to have an impact on the output - the small value derived here
reflects the high degree of variability from one data point to the next.
The synthetic test data are noise-free i.e. the data are generated directly from the simulation model.
But real data are typically noisy with errors arising from measurement - how do the approaches compare
for noisy data? To test this, the calibration exercise has been re-run with noise added to the data in the
form of random values generated from a normal distribution 𝑁(0, 𝜎2)where for illustrative purposes
𝜎has been calculated such that 2𝜎is equal to relative humidity values of 0.01, 0.03, 0.05 and 0.10 i.e.
±1% to 10%, where 10% is the difference in relative humidity observed for a change in ventilation rate
12 Ward, R et al
Figure 6: Toy problem: precision hyperparameters for KOH approach, period 1
(a) Θ1, ACH (b) Θ2, IAS (c) Length scale
Figure 7: Particle Filter approach: evolution of posteriors, showing a) ACH, b) IAS and c) Length scale
from 4𝐴𝐶𝐻 to 2𝐴𝐶𝐻. Figure 8illustrates the comparison for the KOH approach. In this example the
prior distributions for the two calibration parameters have been chosen as normal distributions as giving
better information in the prior distribution should improve the ability of the methodology to identify the
posterior. However, as the noise level increases, the mean of the posterior moves increasingly further
from the true value, more significantly for the ventilation rate, 𝐴𝐶 𝐻, on the left. To understand the
impact of introducing noise in the data on the particle filter approach, consider Figure 9, which plots
the mean of the particle filter posterior distribution at each time step. Again the initial prior distribution
for each 𝜃was specified as a normal distribution. Looking first at the ventilation rate, Figure 9(a), the
zero noise results, shown as a blue dashed line, clearly show the step change from a value of 𝐴𝐶𝐻 =4
to 𝐴𝐶𝐻 =2at the mid-point of the analysis (step = 20). The results for 1 and 3% noise are not too
dissimilar (green and cyan), and even at a noise level of 5% the posterior finishes close to the true value
after 40 steps, albeit without such a clear step-change. At a noise level of 10% however, the PF approach
cannot identify the change in parameter value. This is because the change in relative humidity due to a
change in ventilation rate from 4 to 2 is close to 10% and so the model cannot distinguish what is noise
and what is a genuine parameter change. It is clear that if there is too much noise in the data then both
sequential approaches fail to correctly infer the parameter values.
Figure 9(b) shows the evolution of the posterior of IAS as a function of the step for the increasing
noise levels. The results are reasonably consistent up until time step 20, after which the mean IAS
Data Centric Engineering 13
Figure 8: Toy problem: results of the KOH calibration with period 1 data showing effect of increased
noise in the data
0 10 20 30 40
Step
0
1
2
3
4
5
6
7
8
ACH
Zero noise
1%
3%
5%
10%
(a) ACH
0 10 20 30 40
Step
0.1
0.2
0.3
0.4
0.5
0.6
0.7
IAS
Zero noise
1%
3%
5%
10%
(b) IAS
0 10 20 30 40
Step
0.1
0.15
0.2
0.25
0.3
Length scale
Zero noise
1%
3%
5%
10%
(c) Lengthscale
Figure 9: Toy problem: Mean of PF evolution demonstrating effect of noise in the data
posterior increases, dependent on the noise level, before dropping back to the true value; for 10%
noise, the true value is not achieved by step 40. Interestingly, although there are clear differences in
the evolution of the mean length scale for the different noise levels, the absolute value varies most for
zero noise, the mean value increasing from 0.21 to 0.26 (Figure 9(c)). For higher noise levels the mean
length scale value meanders close to 0.2. This suggests that despite the noise the model is still detecting
similar relationships between the data points.
The hyper-parameters of the particle filter approach have been set to be a similar magnitude to the
KOH approach to ensure comparability. One parameter that has no equivalence, however, is the number
of particles. This is important as it directly affects the run time of the model. Figure 10 shows the
impact of changing the number of particles on the output, with particle number equal to 100 (green),
1000 (blue) and 10000 (magenta). The results demonstrate that the number of particles has little impact
on the outputs of this test case.
In addition to the static Bayesian calibration, we have run the KOH approach sequentially over the
whole time period; we have already seen in Figure 5that using all the data gives posterior parameter
estimates in line with the mean values over the whole time period, but we wish to know whether we can
use fewer data points and run the model sequentially and identify a sudden change in parameter value.
Figure 11 shows how the results of the sequential KOH approach compare against the particle filter
model. The sequential approach has been used with 2, 4 and 8 data points, as the methodology requires
at least 2 data points in order to fit the emulator. In this approach, at each timestep the oldest data point
is discarded and a new data point added - this means that for 2 data points there are 39 runs of the
model, 4 data points there are 37 runs and 8 data points 33 runs. In addition the mean of the posterior
distribution from the previous timestep is used as the mean of the prior for the new timestep. Note we
have not used the full posterior from the previous timestep as prior for the next as the standard deviation
14 Ward, R et al
0 10 20 30 40
Step
0
1
2
3
4
5
6
7
8
ACH
100
1000
10000
(a) ACH
0 10 20 30 40
Step
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
IAS
100
1000
10000
(b) IAS
0 10 20 30 40
Step
0.1
0.15
0.2
0.25
0.3
Length scale
100
1000
10000
(c) Length scale
Figure 10: Mean of PF evolution demonstrating effect of the number of particles
(a) ACH (b) IAS
Figure 11: Sequential KOH: evolution of the mean posterior for a) ventilation rate, 𝐴𝐶𝐻 and b) internal
air speed, 𝐼 𝐴𝑆 , showing the impact of increasing the number of data points included in each step from
𝑁=2to 𝑁=8
of the posterior becomes too narrow, making the prior too specific - instead we have maintained the
standard deviation of the first prior.
We are interested to see whether this approach can identify the parameter values and specifically
the change in ACH value at the half-way point in a similar manner to the PF approach. Using 2 data
points (N=2) the results show some fluctuations from the true value, particularly towards the end of the
simulation for the internal air speed (Figure 11b), but pick up the timing of the change in ventilation
rate well (Figure 11a). As the number of data points increases, this timing is harder to identify as the
approach assumes a constant value over the data, so when for example 8 data points are used, the
timing of the change in ACH value is smoothed and appears not as a sudden change but as a smoother
transition. Using 4 data points appears to match the transition time well with a sharper transition than 8
data points, yet the mean of the posterior shows less fluctuation than 2 data points, hence this appears to
be the best approach. Figure 12 shows the mean and 90% confidence limits of the model bias function as
a function of the step for the sequential KOH approach with 𝑁=4, separated according to whether the
lights are switched on or off. The model bias is small, less than 0.5%, hence will not make a significant
difference to the results.
The more data points are used, however, the greater the time taken to run the analysis. Table 3
shows the run times for analysis of this toy problem using the different models. The models have been
processed on a PC with an i7-6700 CPU processor and 32 MB RAM, and of course a more powerful
PC would improve these times, but the relative difference between the run times for the models will
still hold. It is clear that the PF approach is substantially quicker than the static KOH approach as
Data Centric Engineering 15
0 10 20 30 40
Step
-0.01
-0.005
0
0.005
0.01
0.015
Relative Humidity
Lights off
Lights on
Figure 12: Sequential KOH: mean and 90% confidence limits of the model bias function
Model Number of Data Points Run Time (s)
KOH
20 5,443
40 22,572
Particle Filter (step / total)
1000 particles 40 20 / 786
10,000 particles 40 197 / 7,915
KOH-Sequential (step / total)
N=2 40 60 / 2,535
N=4 40 168 / 6,216
N=8 40 626 / 20,658
Table 3: Comparison of run times: the run times for the KOH approach are for the entire simulation,
whereas for the PF and sequential KOH run times are given both for a single step and for the entire
simulation.
even with 10,000 particles the step time is only just over 3 minutes, giving a total time of just over 2
hours, whereas the KOH approach required a run time in excess of 6 hours for the same problem. The
sequential KOH is substantially quicker, however, but using 4 data points the run time is still much
slower than the PF approach with 1,000 particles.
4.2. Calibration with monitored data from the farm
The toy problem has facilitated comparison between the three approaches, but the main aim of this
study is to explore the applicability of using the particle filter approach for model calibration in the
context of a digital twin. To this end, real monitored data from August to October 2018 have been
selected for further study. This time period has been chosen as the data are complete and there were
no known significant changes to operation except known changes to the ventilation settings. This is
important, as we would like to understand to what extent the two calibration approaches are capable of
identifying the change in parameter. While we know what the changes are to the ventilation settings,
we do not know exactly what that means in terms of changes to the parameter value (the ventilation
rate). We also do not know to what extent other operational conditions were in place that might have
impacted on ventilation rate or air speed.
Implementation of the models with the monitored data has followed the same approach as described
for the toy problem above. The monitored data and the corresponding simulation outputs across the
input parameter ranges are illustrated in Figures 13 and 14. The dots in the first figure illustrates how
16 Ward, R et al
Aug 15 Aug 22 Aug 29 Sep 05 Sep 12 Sep 19 Sep 26 Oct 03
2018
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Relative Humidity
Simulation
Monitored Data
Data Period 1
Data Period 2
Figure 13: Monitored data and the corresponding simulation outputs over 3 months
0 1
Li
ght State 0 = Off, 1 = On
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Relative Humidity
Simulation
Monitored Data
Data Period 1
Data Period 2
(a) Light State On/Off
0.005 0.006 0.007 0.008 0.009 0.01 0.011 0.012 0.013 0.014 0.015
External Cw
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Relative Humidity
Simulation
Monitored Data
Data Period 1
Data Period 2
(b) External Moisture Content, 𝐶𝑤
Figure 14: Monitored data and corresponding simulation outputs for KOH approach, showing the
scenarios a) light state, and b) external moisture content
the data evolve in time, whereas in the second figure they show how the data vary as a function of the
light state and external air moisture content, the two scenarios used in the KOH approach. As for the toy
problem, data points have been extracted at 4pm and 4am, corresponding to the lights being off and on.
Each black dot in Figure 13 is a data point, and we have selected two separate periods of 20 data points
highlighted in red and green for analysis using the standard BC approach. Period 1 (red) is chosen as
it lies before the change in ventilation settings, and Period 2 (green) after the change. It is not feasible
to use all 118 data points for the KOH approach as the run-time would be too long. Also shown in the
figure are the outputs from the simulation runs executed for the calibration (blue crosses). Unlike the
toy problem, here there are several observations which lie outside the range covered by the simulation
outputs, specifically there are points of low and high relative humidity that are not predicted by the
model. This is likely due to simplifications in the model that do not adequately represent the sources of
humidity, for example the irrigation of the plant trays is represented by a single mean saturation level
that does not account for daily fluctuations in moisture level.
As a first step, the KOH approach has been used to estimate the model input parameters for periods
1 and 2. Figure 15 shows the posterior distributions for both parameters for both time periods, with the
first time period shown in red and the second in green. Here a normal prior distribution has been used
(shown in black) to give a more informative prior as we expect the observations to have some degree
of measurement error. We have selected prior mean and standard deviation values to position the prior
at the centre of the possible range of parameter values, but with sufficient variance to generate samples
across the range, as we anticipate the parameter values being towards the centre of the possible range of
values. The posterior parameter distributions suggest that the ventilation rate drops from a mean value
Data Centric Engineering 17
Figure 15: Prior and posterior distribution of ACH and IAS using the KOH approach with monitored
data from the farm
0.008 0.009 0.01 0.011 0.012 0.013 0.014
External Air Moisture Content, Cw (kg/m3)
-8
-6
-4
-2
0
2
4
Relative Humidity
x 10
-3
Lights off
Lights on
(a) Model bias, period 1
5 6 7 8 9 10 11
External Air Moisture Content, Cw
(kg/m3)x10-3
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
Relative Humidity
Lights off
Lights on
(b) Model bias, period 2
Figure 16: Model bias function
of 6.5 in August to a mean value of 5.6 ACH in October, whereas the mean internal air speed increases
from 0.22 to 0.43 m/s. These results are plausible given the increase in relative humidity observed
towards the end of period 2; both a reduction in ventilation rate and an increase in internal air speed
tend to result in an increase in relative humidity.
It is particularly useful when using monitored data to explore the model bias function as it can
give some idea as to where the physics-based model might be improved. Figure 16 shows the mean
and 90% confidence limits of the model bias for periods 1 and 2 plotted separately for the two light
states - off and on - as a function of external moisture content on the x axis. As for the toy problem
the magnitude of the bias is small, suggesting that differences between the model and the data are
primarily accounted for by measurement error. However there is a clear difference between the bias
function for lights on and lights off; when the lights are on, the absolute magnitude of the bias function
is greater implying that the agreement between the model and the data is better when the lights are off,
particularly in period 1. The fact that the bias function is negative when the lights are on suggests that
the model is overpredicting the relative humidity when compared with the data. Relative humidity is a
combined effect of temperature and air moisture content, so a value that is too high can be due to either
temperatures that are too low or too much moisture in the air. This information gives us valuable insight
into how to improve the physics-based model.
As for the test case, we run the particle filter approach for the entire period from August to October,
a total of 118 data points.
18 Ward, R et al
(a) ACH
(b) IAS
Figure 17: Posterior predictions from PF approach using the 118 monitored data points
The evolution of the PF posterior distributions for the ventilation rate and internal air speed are
indicated in Figure 17; the length scale remained fairly constant at a value of just under 0.2. The mean
of the posterior parameter values are shown in Figure 17 as a black solid line. We have also run a
sequential Bayesian calibration updating after 4 data points, similar to the test case, shown in the figure
as the blue dashed line. The mean of the two static Bayesian calibrations are indicated on the figures
as red dashed and green dashed lines over the periods of interest. Also indicated on Figure 17a are
the settings of the two ventilation systems over this period. The CP ventilation setting is constant until
September 22nd, at which point it is reduced from 2 to 0.5 ACH, whereas the CC setting is higher, at
4.5 ACH, during the initial period, reduced in early September to a value of 2 ACH.
The first point to note is that the results for the two sequential calibrations are consistent. The sequen-
tial KOH results are more extreme, particularly for the internal air speed (Figure 17b) where the value
peaks on several occasions, reaching a high of 0.8m/s on August 30th. The PF approach has peaks at
similar times, but they do not reach the same magnitude, only approaching 0.6m/s on August 30th.
Secondly, the sequential KOH results are very different from the static KOH results, particularly for
period 2 where the static values of 5.6ACH and 0.43m/s for the ventilation rate and internal air speed
respectively are substantially higher than the sequential values which are closer to 3ACH and 0.2 m/s
for most of period 2.
The third point is that neither of the sequential models strongly reflect the change in settings of
the ventilation system. This could mean that the settings of the ventilation system are less influential
on overall ventilation rates than non-controlled ventilation arising from lift shafts/door infiltration etc.
Equally it may reflect the fact that the data are quite noisy, so the true values are masked by the noise
in the data.
Even if the models are not able to identify ’true’ values for these parameters, are we able to infer
parameter values that enable the model to accurately simulate the farm environment? To assess this,
Data Centric Engineering 19
we have run the model using parameter values for ACH and IAS equal to the outputs of the sequential
models, and also equal to the static values for period 1.
The mean and 90% confidence limits of the relative humidity values simulated using the three differ-
ent approaches are shown in Figure 18. Figure 18a shows the simulation results across the entire time
period compared against the monitored data, with the complete data set shown as red circles and the
data used for the calibration picked out as black dots. Specifics are difficult to pick out, but the overall
trend for the sequential models to give a much closer approximation to the data than the static model
is clear, as the PF approach results in green and the sequential KOH approach results in blue show a
much greater degree of variation than the static KOH results in red, in line with the data. This is partic-
ularly clear over the central section of the simulation around August 30th, and this section of the plot is
enlarged for clarity in Figure 18b. Here we can clearly see that using the parameters derived for period
1 using the static KOH approach gives a poor approximation to the data in the days before August 30th,
whereas the two sequential approaches give an increase in relative humidity over this period in line
with the data. That they do not get even closer is due to the fact that these data lie outside the range of
the simulation runs illustrated in Figure 13. Another point of interest from Figure 18a is that there is a
gradual increase in variance of the PF results as time progresses i.e. the 90% confidence limit shown at
the beginning of August is much narrower than that shown in October. It is a known feature of the par-
ticle filter that the variance of the particle weights increases over time and for longer term simulations
some form of correction will need to be implemented. This is not an issue for the sequential KOH as
we have not allowed the variance of the parameter values to change as the simulation progresses.
The KOH approach yields a model bias function for periods 1 and 2 which suggests that the model
may overestimate the relative humidity when the lights are switched on (Figure 16). Figure 19 shows
the mean and 90% confidence limit of the bias function for the sequential KOH approach plotted as
a function of time. It is clear that the bias function when the lights are on almost mirrors that when
the lights are off, and again the sign of the bias suggests the model overestimates the relative humidity
when the lights are on and underestimates when the lights are off. The absolute magnitude is small
however, only peaking over 0.02 when the lights are off just before August 30th, and when the lights
are on just after September 6th. The mean estimated measurement error is a similar magnitude with a
mean standard deviation of 0.02 but a maximum standard deviation of 0.07 in the region where the data
points fall higher than the simulation range i.e. around August 29th.
5. Discussion
In the context of a digital twin, calibrating the simulation element of the twin with current data is essen-
tial for ensuring that the simulation is representing reality as closely as possible. Manual calibration
is time consuming and not suitable when parameter values are changing. This study explores the suit-
ability of a particle filter approach for calibration and compares it against static Bayesian calibration
following the Kennedy and O’Hagan (2001) approach, and also uses the KOH approach sequentially
as a further comparison.
The static KOH approach assumes that the calibration parameters do not change over the timespan
of the data and so is only able to give good estimates if there is confidence that the parameter values
are unchanging; we have seen using a toy problem that if the parameter values do change the KOH
approach can only give an average of the true value (Figure 5). If the parameters are unchanging how-
ever, it is still the ’gold standard’ for calibration against which we compare other approaches. It is
particularly useful as the KOH approach is formulated in such a way as to give estimates not only of
the calibration parameters but also of the discrepancy between the model and the data (model bias), and
the measurement and numerical errors. We have seen in the toy problem (Figure 6) that the posterior
distributions of the error terms have low median values as expected when the data are derived directly
from the model; but for monitored data the approach gives a more useful insight, particularly into the
model bias (Figure 16). The figure illustrates that while the bias only shows a small variation with the
20 Ward, R et al
(a) Entire period
Aug 27 Aug 28 Aug 29 Aug 30 Aug 31 Sep 01 Sep 02
2018
0.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Relative Humidity
AllData
Data used for calibration
Particle Filter
Sequential KOH
KOH Period 1
(b) Central period
Figure 18: Relative Humidity results for the PF, sequential KOH and static KOH approachs showing a)
the entire simulation, and b) a central period
external air moisture content, there is a more significant difference according to whether the lights are
switched on or off. With the lights on - a direct source of heat - the bias is negative, suggesting that
the model is predicting relative humidities that are higher than observed in the data. Relative humidity
increases as temperature decreases for the same air moisture content, so if it is too high that suggests
either the predicted temperature is too low, or the predicted air moisture content is too high. This gives
direct insight into how to improve the physics-based model going forwards.
The sequential approaches offer the potential to track parameter values, which is what we desire for
the digital twin. We have applied the different approaches to both a toy problem, with data derived from
the simulation with known parameters and a controlled change in the value of ventilation rate, and to
real monitored data. The difference is very clear; whereas with the toy problem the sequential methods
were able to track a clearly defined change in ventilation rate accurately (Figure 7), the monitored data
gives a much more variable output (Figure 18). We have also seen that in the monitored data there are
data points that fall outside the range of the simulation runs. The very high relative humidity values
occurring around the 29th August and September 5th (Figure 13) seem to be not following the diurnal
pattern typically seen on other days so could be due to something happening within the farm which we
are not aware of - a change to the operation of the dehumidifiers for example. Equally, there are values
which are lower than the simulation runs, particularly in early August. More work is needed to explore
the model inadequacies, particularly for extreme values of relative humidity.
Since we know the settings of the ventilation system, we might expect a clearer link between the
ventilation rate parameter values and the system settings, but this is not observed. One possible reason
Data Centric Engineering 21
Aug 09 Aug 16 Aug 23 Aug 30 Sep 06 Sep 13 Sep 20 Sep 27 Oct 04
2018
-0.03
-0.02
-0.01
0
0.01
0.02
0.03
Relative Humidity
Lights off
Lights on
Figure 19: Sequential KOH: mean and 90% confidence limits of the model bias function
for this is that the data are subject to a high degree of measurement error - we saw in the toy problem
how noisy data can prohibit the ability of the sequential approaches to track the parameter change.
But we should consider what we are tracking: we are tracking the parameter values that give the best
agreement of the model with the data. The relationship between the tracked parameters and reality
depends on the extent to which the model represents reality. For example, here the ventilation rate
assumed in the model must encompass all controlled and uncontrolled components of the ventilation
and is not simply equal to the setting on the dial.
This highlights the balance to be considered in the complexity of the simulation model. Here we
use a simple physics-based model calibrated using monitored data to give the best agreement between
the model and the data. But there is limited insight into why the ventilation rate and internal air speeds
appear to fluctuate so dramatically - this could be a real effect, or as is more likely it could be that other
events are happening that impact on the relative humidity but are not incorporated in the model. In
this instance the calibration approaches are seeking to match the observed data with a simulation based
on an incomplete description of events and will push the parameter values to give as close a match as
possible. The calibration parameters then become a proxy for all events affecting their values i.e. as
outlined above the ventilation rate has to encompass both controlled and uncontrolled ventilation, and
the internal air speed will be affected by factors such as blockages in the tunnel which affect air flow. A
more complex model could include a more detailed representation of the system processes explicitly;
but a more complex model would take longer to run and longer to calibrate. This balance must be
assessed in the design and development of a digital twin.
In this study we have compared the PF approach against a sequential version of the KOH approach,
and it is clear that they give similar results. The main differences between them are time and information
- on the one hand, the PF is substantially quicker depending on the approach details - for the PF, a
higher number of particles causes the run time to increase linearly, whereas in the KOH approach the
run time increases approximately as the square of the number of data points for the same number
of MCMC iterations. On the other hand the sequential KOH approach estimates the model bias and
the measurement and numerical error terms in addition to the calibration parameters. The model bias
extracted for the sequential KOH study of the monitored data (Figure 19) reinforces the trend observed
for the toy problem i.e. that the model is overpredicting relative humidity when the lights are on and
underpredicting when the lights are off. This study was based purely on two data points each day, one
from when the lights are switched on and one from when they are off; a more targeted choice of data
or scenario could give more insight into the model discrepancy and the error terms (Menberg et al.
(2019)). Further work will assess the sensitivity of the approaches to the choice of data with reference
to maximising the impact on the accuracy of the digital twin.
22 Ward, R et al
The particle filter approach works well for the purposes of providing quick estimates of parameter
probability distribution in real time. The sequential KOH approach also works well, but in this study
has been shown to be slower and give more extreme results. In the implementation of the sequential
KOH approach we have prescribed the variance of the prior for all time steps i.e. we have not allowed
changes in variance to propagate through time. This was a necessary assumption to ensure the prior
remained sufficiently diverse. By comparison, in implementing the PF approach we have allowed the
variance of the prior to propagate and it is clear from Figure 18a that the variance of the PF posterior
distribution is steadily increasing as time progresses. This is a natural feature of the PF approach, but
for implementation in the digital twin it will be necessary to address this issue, perhaps by periodically
re-setting the prior.
Both approaches make use of an emulator fitted to the simulation output in order to map the inputs
of the model to the output of interest, and for both approaches Gaussian Process models have been
used. This requires the physics-based model to be run a considerable number of times with parameter
values covering the ranges of interest prior to running the calibration. Within the digital twin framework
this approach will require modification such that the model predictions are generated as the calibration
progresses, rather than in advance. The PF approach is more suited to this framework than the KOH
approach as the PF considers each data point separately rather than mapping the entire parameter space.
6. Conclusions
This study makes an essential contribution to the development of digital twins of the built environment.
Combination of data and a model - in this case the physics-based model - is at the heart of what is meant
by a digital twin. To maximise the benefits it is essential to ensure that the data are used to continuously
update the model. In this study the continuous calibration of the model has been explored using a
particle filter approach. This has been compared against Bayesian calibration using the framework put
forward by Kennedy and O’Hagan (2001), and against a sequential form of the KOH approach.
The development of the digital twin for an underground farm provides an exemplar for the built
environment more broadly. It is necessary to maintain favourable environmental conditions in the farm;
to that end an extensive programme of monitoring has been undertaken and is still ongoing. A simple
physics-based model of the farm has also been developed which can aid prediction of environmental
conditions and provide useful information for future planned farm expansion.
The PF approach has been shown to give a good estimate of the mean and variance of the model
input parameters ventilation rate and internal air speed which were selected owing to their impact on
the relative humidity in the farm. When using the calibrated time varying parameters in the model a
much closer agreement with the monitored data is observed than when using parameters inferred from
the static BC approach. It is clear, however, that the physics-based model is unable to predict the full
range of relative humidities observed in reality; there are data points that fall both above and below the
predicted range of relative humidity. This model inadequacy cannot be explored using the PF approach
alone - the KOH approach is required in order to quantify the model bias function and error terms.
In terms of suitability for incorporation in a digital twin, the PF approach is more suitable than the
sequential KOH approach used here as it is quicker and is more suited to combination with execution
of the physics-based model at each data point. It also quantifies the variance in the posterior param-
eter probability distributions, although the inevitable increase in variance over time will need to be
addressed. A combined approach, using the PF approach in real time and the KOH approach for peri-
odic assessment of the model bias function would yield the best result for ensuring continuity of model
accuracy. Further work will address the practical implementation of the particle filter within the digital
twin framework.
Acknowledgments. We are grateful for the continued support of Growing Underground without whose input this study would
not have been possible.
Data Centric Engineering 23
Funding statement. This research was supported by AI for Science and Government (ASG), UKRI’s Strategic Priorities Fund
awarded to the Alan Turing Institute, UK (EP/T001569/1) and the Lloyd’s Register Foundation programme on Data-centric
Engineering.
Competing interests. None
Data availability statement. The data used were made available by Growing Underground for the purpose of this study.
Ethical standards. The research meets all ethical guidelines, including adherence to the legal requirements of the UK.
Author contributions. Conceptualization: M.G.; R.C.; A.G. Methodology: R.W.; A.G; Data curation: R.W. Data visualisation:
R.W. Writing original draft: R.W; R.C. All authors approved the final submitted draft.
Supplementary material. No supplementary information has been provided.
References
Andrieu, C., Doucet, A., Singh, S. S., and Tadi´
c, V. B. (2004). Particle methods for change detection, system identification and
control. Proceedings of the IEEE, 92(3):423–438.
Cheng, C. and Tourneret, J.-Y. (2017). Kernel density-based particle filter for state and time-varying parameter estimation in
nonlinear state-space models.
Chong, A., Lam, K. P., Pozzi, M., and Yang, J. (2017). Bayesian calibration of building energy models with large datasets.
Energy and Buildings, 154:343–355.
Chong, A. and Menberg, K. (2018). Guidelines for the Bayesian calibration of building energy models. Energy and Buildings,
174:527–547.
Chong, A., Xu, W., Chao, S., and Ngo, N.-T. (2019). Continuous-time Bayesian calibration of energy models using BIM and
energy data. Energy and Buildings, 194:177–190.
Choudhary, A., Lindner, J. F., Holliday, E. G., Miller, S. T., Sinha, S., and Ditto, W. L. (2020). Physics-enhanced neural networks
learn order and chaos. Physical review E, 101(062207).
Doucet, A. and Johansen, A. M. (2008). Particle filtering and smoothing: Fifteen years later, Handbook of nonlinear filtering.
Guillas, S., Rougier, J., A.Maute, A.D.Richmond, and C.D.Linkletter (2009). Bayesian calibration of the Thermosphere-
Ionosphere Electrodynamics General Circulation Model (TIE-GCM). Geoscience Model Development, 2:137–144.
Heo, Y., Choudhary, R., and Godfried.A.Augenbroe (2012). Calibration of building energy models for retrofit analysis under
uncertainty. Energy and Buildings, 47:550–560.
Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., and Ryne, R. D. (2004). Combining field data and computer simulations
for calibration and prediction. SIAM Journal on Scientific Computing, 26(2):448–466.
Jans-Singh, M., Fidler, P., Ward, R. M., and Choudhary, R. (2019). Monitoring the performance of an underground hydroponic
farm. pages 133–141. ICE Publishing.
Kennedy, M. C. and O’Hagan, A. (2001). Bayesian calibration of computer models. Royal Statistical Society, 63(3):425–464.
LED Magazine (2005). Fact or fiction - LEDs don’t produce heat. https://www.ledsmagazine.com/leds-ssl-design/thermal/
article/16696536/fact-or-fiction-leds-dont-produce-heat.
Li, Q., Augenbroe, G., and Brown, J. (2016). Assessment of linear emulators in lightweight Bayesian calibration of dynamic
building energy models for parameter estimation and performance prediction. Energy and Buildings, 124:194–202.
Madni, A. M., Madni, C. C., and Lucero, S. D. (2019). Leveraging digital twin technology in model-based systems engineering.
Systems, 7(7):328–351.
Menberg, K., Heo, Y., and Choudhary, R. (2016). Sensitivity analysis methods for building energy models: comparing
computational costs and extractable information. Energy and Buildings, 133:433–445.
Menberg, K., Heo, Y., and Choudhary, R. (2019). Influence of error terms in Bayesian calibration of energy system models.
Journal of Building Performance Simulation, 12(1):82–96.
Moradkhani, H. and Hsu, K.-L. (2005). Uncertainty assessment of hydrologic model states and parameters: sequential data
assimilation using the particle filter. Water Resources Research, 41(W05012).
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg,
V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine
learning in Python. Journal of Machine Learning Research, 12:2825–2830.
Worden, K., Cross, E., Gardner, P., Barthorpe, R., and D.J.Wagg (2020). On digital twins, mirrors and virtualisations. In
Barthorpe, R., editor, Model Validation and Uncertainty Quantification, Volume 3. Springer.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Artificial neural networks are universal function approximators. They can forecast dynamics, but they may need impractically many neurons to do so, especially if the dynamics is chaotic. We use neural networks that incorporate Hamiltonian dynamics to efficiently learn phase space orbits even as nonlinear systems transition from order to chaos. We demonstrate Hamiltonian neural networks on a widely used dynamics benchmark, the Hénon-Heiles potential, and on nonperturbative dynamical billiards. We introspect to elucidate the Hamiltonian neural network forecasting.
Article
Full-text available
Digital twin, a concept introduced in 2002, is becoming increasingly relevant to systems engineering and, more specifically, to model-based system engineering (MBSE). A digital twin, like a virtual prototype, is a dynamic digital representation of a physical system. However, unlike a virtual prototype, a digital twin is a virtual instance of a physical system (twin) that is continually updated with the latter’s performance, maintenance, and health status data throughout the physical system’s life cycle. This paper presents an overall vision and rationale for incorporating digital twin technology into MBSE. The paper discusses the benefits of integrating digital twins with system simulation and Internet of Things (IoT) in support of MBSE and provides specific examples of the use and benefits of digital twin technology in different industries. It concludes with a recommendation to make digital twin technology an integral part of MBSE methodology and experimentation testbeds.
Article
Full-text available
Calibration represents a crucial step in the modelling process to obtain accurate simulation results and quantify uncertainties. We scrutinize the statistical Kennedy & O’Hagan framework, which quantifies different sources of uncertainty in the calibration process, including both model inputs and errors in the model. In specific, we evaluate the influence of error terms on the posterior predictions of calibrated model inputs. We do so by using a simulation model of a heat pump in cooling mode. While posterior values of many parameters concur with the expectations, some parameters appear not to be inferable. This is particularly true for parameters associated with model discrepancy, for which prior knowledge is typically scarce. We reveal the importance of assessing the identifiability of parameters by exploring the dependency of posteriors on the assigned prior knowledge. Analyses with random datasets show that results are overall consistent, which confirms the applicability and reliability of the framework.
Article
The present study proposes a framework for the continuous Bayesian calibration of whole building energy simulation (BES) models utilizing data from building information models (BIM) and building energy management systems (BEMS). The ability to import data from BIM and BEMS provides the potential to significantly reduce the time and effort needed for the continuous calibration of BES models. First, five gbXML geometric test cases were used to check the BIM to BES model translation. Translation of the test cases indicates good geometric agreement between the native BIM and the gbXML-based BES model. An actual building calibration case study (with BIM and three years of monthly electrical energy consumption data) was then used to evaluate the proposed continuous calibration method. The results suggest that compared to a non-continuous approach, the continuous Bayesian calibration method showed reduced prediction uncertainty and improved prediction accuracy on a test dataset. The paper also presents information and comparison of the coefficient of variance of the root mean square error (CVRMSE) and the normalized mean biased error (NMBE), recommending looking at their distributions when working with probabilistic BES predictions.
Article
This paper provides practical guidelines to the Bayesian calibration of building energy models using the probabilistic programming language Stan. While previous studies showed the applicability of the calibration method to building simulation, its practicality is still impeded by its complexity and the need to specify a whole range of information due to its Bayesian nature. We ease the reader into the practical application of Bayesian calibration to building energy models by providing the corresponding code and user guidelines with this paper. Using a case study, we demonstrate the application of Kennedy and O’Hagan’s (KOH) [1] Bayesian calibration framework to an EnergyPlus whole building energy model. The case study is used to analyze the sensitivity of the posterior distributions to the number of calibration parameters. The study also looks into the influence of prior specification on the resulting (1) posterior distributions; (2) calibrated predictions; and (3) model inadequacy that is revealed by a discrepancy between the observed data and the model predictions. Results from the case study suggest that over-parameterization can result in a significant loss of posterior precision. Additionally, using strong prior information for the calibration parameters may dominate any influence from the data leading to poor posterior inference of the calibration parameters. Lastly, this study shows that it may be misleading to assume that the posteriors of the calibration parameters are representative of their true values and their associated uncertainty simply because the calibrated predictions matches the measured output well.
Technical Report
Optimal estimation problems for non-linear non-Gaussian state-space models do not typically admit analytic solutions. Since their introduction in 1993, particle filtering methods have become a very popular class of algorithms to solve these estimation problems numerically in an online manner, i.e. recursively as observations become available, and are now routinely used in fields as diverse as computer vision, econometrics, robotics and navigation. The objective of this tutorial is to provide a complete, up-to-date survey of this field as of 2008. Basic and advanced particle methods for filtering as well as smoothing are presented.