Content uploaded by Simone Baldi
Author content
All content in this area was uploaded by Simone Baldi on Jan 18, 2020
Content may be subject to copyright.
Passive versus active learning in operation and adaptive
maintenance of Heating, Ventilation, and Air Conditioning
Simone Baldib,a,∗, Fan Zhangc,b, Thuan Le Quangd,a, Petr Endele, Ondrej Holube
aDelft Center for Systems and Control, Delft University of Technology (TU Delft), Delft 2628CD, The Netherlands
bSchool of Mathematics, Southeast University, Jiangsu Province, Nanjing 210096, China
cSchool of Aeronautics and Astronautics (Shenzhen Campus), Sun Yatsen University, Guangzhou 510275, China
dDepartment of Mathematics, Quy Nhon University, Binh Dinh province, Quy Nhon city, Viet Nam
eHoneywell Prague Laboratory, V Parku 2326/18, 148 00 Prague 4, Czech Republic
Abstract
In smart buildings, the models used for energy management and those used for maintenance scheduling differ in scope and struc
ture: while the models for energy management describe continuous states (energy, temperature), the models used for maintenance
scheduling describe only a few discrete states (healthy/faulty equipment, and fault typology). In addition, models for energy man
agement typically assume the Heating, Ventilation, and Air Conditioning (HVAC) equipment to be healthy, whereas the models for
maintenance scheduling are rarely humancentric, i.e. they do not take possible human factors (e.g. discomfort) into account. As a
result, it is very difﬁcult to integrate energy management and maintenance scheduling strategies in an efﬁcient way. In this work, a
holistic framework for energyaware and comfortdriven maintenance is proposed: energy management and maintenance schedul
ing are integrated in the same optimization framework. Continuous and discrete states are embedded as hybrid dynamics of the
system, while considering both continuous controls (for energy management) and discrete controls (for maintenance scheduling).
To account for the need to estimate the equipment efﬁciency online, the solution to the problem is addressed via an adaptive dual
control formulation. We show, via a zoneboilerradiator simulator, that the best economic cost of the system is achieved by active
learning strategies, in which control interacts with estimation (dual control design).
Keywords: Energy management, maintenance scheduling, adaptive learningbased control, smart buildings.
1. Introduction
The economic cost of buildings is largely dependent on con
trol and maintenance of Heating, Ventilating and Air Condition
ing (HVAC) equipment [1]. For example, neglecting any per
formance degradation or even faults in HVAC will irredeemably
lead to increased costs for facility managers and building own
ers. While control decisions have a direct impact on energy
consumption [2], literature has shown that the effect of perfor
mance degradation is more complex and essentially twofold:
ﬁrstly, increased consumption of resources in order to compen
sate for the system inefﬁciency [3]; secondly, failure to meet
the given set points leading to decreased comfort within the
building (with loss of productivity, complaints, etc.) [4]. This
humancentered impact adds to the cost of maintenance, which
should be scheduled optimally in such a way to minimize the
overall adverse economic effects. In a nutshell, joint control
and predictive maintenance is a complex and largely unsolved
optimization problem involving the joint design of estimation
∗Corresponding author. Tel.: +31 15 2781823
Email addresses: s.baldi@tudelft.nl (Simone Baldi),
zhangfan6@mail.sysu.edu.cn (Fan Zhang),
lequangthuan@qnu.edu.vn (Thuan Le Quang),
petr.endel@Honeywell.com (Petr Endel),
ondrej.holub@Honeywell.com (Ondrej Holub)
and control. In this work, we formulate such problem as a dual
control problem: the term ‘dual’ refers to the twofold action
of the control action, which is in charge of both running the
HVAC system toward optimal performance and of reducing the
uncertainty when estimating HVAC degradation. We address
two types of actions: a continuous action involving the selec
tion of the HVAC set point (e.g. water temperature set point for
boilers), and a discrete action determining maintenance (e.g.
repairing or not the boiler). Due to the thermostatic mechanism
of HVAC operation, we embed such mechanism in a hybrid dy
namical system with continuous and discrete dynamics, thus
requiring the solution of a hybrid control problem.
Because it involves the monetization of HVAC performance
and discomfort, the formulation and the solution of such control
problem is relevant and, to the best of the authors’ knowledge,
novel. Previous researches have analyzed smaller aspects of
the global problem, namely: (a) optimization of HVAC energy
consumption and thermal comfort (with no focus on mainte
nance); (b) performance monitoring, i.e. fault detection and
identiﬁcation of HVAC equipment (with no focus on schedul
ing maintenance); (c) scheduling HVAC maintenance (with no
focus on the humancentered impact of decreased comfort). In
the following, we review these three research directions.
Preprint submitted to Applied Energy June 24, 2019
1.1. Related works in optimization of energy/comfort
With respect to optimization of energy consumption and
thermal comfort, several strategies can be found in literature
to predict the effect of changing the control strategy on indoor
comfort [5], or energy consumption [6], or both [7]: typical
criteria driving the optimization include maximizing economy
while satisfying power demand [8], optimizing components siz
ing [9], maximizing selfconsumption [10], balancing natural
ventilation and air conditioning [11], and many more techno
economic criteria [12] (see also references therein). The terms
‘humanintheloop optimization’ [13], or ‘comfortdriven op
timization’ [14], or ‘ occupancybased optimization’ [15] are
sometimes adopted, referring to the fact that the energy demand
is ultimately driven by human needs [16]. See also the recent
review [17]. In [18], a datadriven approach for minimization
of HVAC energy consumption and room temperature ramp rate
is presented. Intelligent glazed facades is the subject of [19],
with emphasis on the inﬂuence of different control policies on
energy and comfort performance. The authors in [20] apply
particle swarm optimization to optimize the set points based on
some comfort zones. In [21] the operation of variable air vol
ume HVAC is optimized with respect to comfort and indoor air
quality. The inﬂuence of thermostat operation on energy con
sumption and thermal comfort is studied in [22]; [23] focuses
on integration of multiple HVAC systems, [24] studies how
to optimize simultaneously several HVAC set points, and [25]
studies cooperation among intelligent HVAC systems. Coop
erative HVAC control has lead to studying the effect of HVAC
operation at the grid level, such as demand response [26] or
other ancillary services [27]. All these approaches show, some
times also via reallife experiments, that relevant energy savings
can be achieved without compromising thermal comfort. How
ever, in these and other related works the degradation of HVAC
components is neglected to a large extent: the HVAC system is
assumed to work as good as new, thus neglecting the possible
waste of energy and loss of comfort due to HVAC degradation.
1.2. Related works in performance monitoring
On the other hand, much literature has been focusing on
HVAC performance monitoring, both at a systemlevel or at
a componentlevel [28]. Systemlevel approaches describe the
HVAC system as a network of interconnected subsystems [29]:
for every subsystem, a monitoring agent is designed that com
bines local and transmitted information from its neighboring
agents in order to provide a decision on the type and location of
the faults [30]. In the presence of uncertainty, decisions can be
based on stochastically robust thresholds [31], adaptive thresh
olds [32], or on state estimation techniques [33]. Centralized
(in place of distributed) strategies are also possible, like the
datadriven automated building HVAC fault detection methods
in [34] and the system identiﬁcationbased method in [35]. At
a componentlevel, mainly boilers and air handling units have
been studied. For boilers, in [36] a model was developed to pre
dict the seasonal efﬁciency based on the efﬁciency at full load
evaluated at return water mean temperature. In [37] heat and
mass transfer analytical models of a condensing heat exchanger
system were developed to predict the boiler efﬁciency accord
ing to design parameters choices: the model in [38] includes
ﬂue gas outlet temperature, supply water temperature, water
vapor mole fraction, and condensation rate of water vapor. A
dynamic relation between boiler efﬁciency and state of the heat
exchange can be derived from the model in [39]. In [40] algo
rithms for realtime monitoring of condensing boilers have been
developed. For air handling units, the work in [41] focuses on
monitoring techniques as part of the ongoing commissioning
process. The set of expert rules derived from mass and energy
balances in [42] is able to detect faults in air handling units,
whereas [43] adopts Kalman ﬁltering techniques instead of ex
pert rules. A detailed overview of fault detection and diagnosis
methodologies on airhandling units is given in [44]. What is
missing in current fault detection and diagnosis methodologies
is a complete monetization analysis taking into account the bal
ance between costs due to loss of performance and costs due
to maintenance actions. A work partly going in this direction
is [45], which adopts a hybrid approach utilizing expert rules,
performance indexes and statistical process control models: in
this way it is possible to include increased energy consumption
due to HVAC degradation. Summarizing, most works on fault
detection and diagnosis do not investigate the whole economic
aspects of degraded HVAC operation.
1.3. Related works in scheduling maintenance
In the category of maintenance, the authors in [4] develop
commissioning strategies to identify costeffective operational
and maintenance measures in buildings to bring them up to the
optimum operation. The aim of [46] is to early plan mainte
nance interventions for a multicomponents system based on
stoppages characteristics, system remaining useful life and com
ponents criticalities. Retroﬁtting is the focus of [47]. The ap
proach in [48] focuses on operational and cleaning costs of a
biomass boiler. In [49] the energy and economic performance
of energy recovery ventilators is studied as a function of pa
rameters such as climate, building design and HVAC system
parameters. An overview of procedures about continuous com
missioning in ofﬁce buildings is given in [50]: interestingly, this
work discusses how to select good models not only for mainte
nance, but also for modelbased control. However, rarely these
two aspects are connected into a humancentric (e.g. comfort
driven) maintenance strategy: notable exceptions are [51], where
it is recognized that discomfort plays an important role in de
termining when the maintenance is performed, and [52], that
investigates the maintenance characteristics of HVAC system
that affect occupants’ satisfaction. However, what is missing in
these works is recognizing the role of control in reducing un
certainty (e.g. uncertainty around efﬁciency parameters). To
clarify this point let us observe this: the use of identiﬁcation
techniques as in [52] to establish relationships among quanti
ties, e.g. regression models, is a passive learning method; on
the other hand, the use of the control action to improve the ﬁ
delity of the regression models while minimizing HVAC oper
ational costs (dual control action) is an active learning frame
work, whose formulation and solution is still missing.
2
1.4. Main contribution and originality
In this work we address the gaps in the state of the art
by considering an active learning framework which is relevant
to the maintenance optimization problem. The monetization
model we propose will incorporate in a comprehensive cost
function the operational costs of HVAC equipment subject to
degradation, the humancentered costs of the fault occurring in
the system, and the costs of maintenance actions. The following
points are covered in this work that, to the best of the authors’
knowledge, have not been covered in the state of the art:
•The control, the monitoring, and the maintenance prob
lems are recast in the same optimization framework via
a dual control formulation (joint design of control and
estimation);
•The thermostat hysteretic behavior is embedded as hy
brid dynamics of the system (with continuous and dis
crete states). In addition, both continuous and discrete
control actions are considered;
•Comparisons between passive learning strategies and ac
tive learning strategies are provided.
In order to keep the optimization problem tractable, assump
tions and simpliﬁcations have been made when describing the
joint control/estimation problem: such assumptions and sim
pliﬁcations, have been studied by the authors in such a way to
retain the main features of the HVAC problem. It is worth men
tioning that the proposed framework is validated (cf. Sect. 7) on
a zoneboilerradiator simulation environment studied within
the European Union project ‘Advanced Methods in Building
Diagnostics and Maintenance (AMBI)’ (FP7PEOPLE2012
IAPP  IndustryAcademia Partnerships and Pathways).
The rest of the paper is organized as follows: in Section
2 the HVAC and room models are given; in Section 3 the ef
ﬁciency model is given, whereas in Section 4 all the continu
ous/discrete dynamics are recast as a hybrid system. The role
of uncertainty is covered in Section 5, and the proposed adap
tive approach is in Section 6. Section 7 gives the simulation
results, and Section 8 concludes the work.
Notation: The notation is quite standard as explained in Ta
ble 1. The subscripts B,R,Zrefer to boiler, radiator, and zone,
respectively. The subscripts rw,sw stand to return and supply
water. The numerical values of such parameters used for simu
lation purposes are reported in the Appendix.
2. HVAC and room models
In the following we provide the details of the model used
for synthesis of the maintenance strategy. We consider a sin
gle zone whose HVAC consists of a radiator driven by a boiler.
The model has been selected as a tradeoff between depth of
description of the thermal/energy dynamics and computational
feasibility of the maintenance strategy. Dynamics of boiler, ra
diator and zone are presented in order.
Explanation Symbol Unit
Boiler supply water temperature Tsw [oC]
Boiler return water temperature Trw [oC]
Radiator return water temperature TrwR[oC]
Dew point temperature Tdew [oC]
Room (zone) temperature TZ[oC]
Neighbor room temperature Tn[oC]
Outside temperature To[oC]
Desired temperature Td[oC]
Boiler mass ﬂow rate wwB[kg/s]
Radiator mass ﬂow rate wwR[kg/s]
Shunt mass ﬂow rate wwS[kg/s]
Percentage shunt mass ﬂow rate cS[%]
Input power of boiler (gas side) pin [kW]
Output power of boiler (water side) pout [kW]
Boiler efﬁciency curve
η
(Trw)
Speciﬁc heat of water cw[kJ/kgoC]
Density of water
ρ
w[kg/m3]
Speciﬁc heat of air ca[kJ/kgoC]
Density of air
ρ
a[kg/m3]
Volume of the radiator VR[m3]
Area of the radiator section AR[m2]
Radiator heat transfer coeff. KR[kW/oC]
Convection heat transfer coeff. hR[kW/m2oC]
Room area with outside Ao[m2]
Room area with neighbors An[m2]
Roomtoout. heat transfer coeff. ho[kW/m2oC]
Roomtoroom heat transfer coeff. hn[kW/m2oC]
Thermostat hysteretic threshold h[oC]
Valve quantization
δ

Boiler degradation rate
α
[s−1]
Forgetting factor
κ
[s−1]
Efﬁciency parameter
θ
1[oC−1]
Efﬁciency parameter
θ
2
Efﬁciency parameter
θ
3[oC−1]
Variance radiator noise
σ
2
R[kW2]
Variance zone noise
σ
2
Z[kW2]
Variance efﬁciency noise
σ
2
B
Variance
θ
1noise
σ
2
1[oC−2]
Variance
θ
2noise
σ
2
2
Variance
θ
3noise
σ
2
3[oC−2]
Table 1: Table of symbols
2.1. Boiler
We will focus on a condensing boiler which, whenever the
return temperatures from the heating system is below the dew
temperature of the ﬂue gas, can recover the latent heat of wa
ter vapor in the ﬂue gas so as to achieve higher efﬁciency than
traditional boilers. Above the dew temperature, no latent heat
is recovered and the boiler will operate in a noncondensing
mode [53]. We assume that the boiler has no dynamics, which
amounts to assuming being in steadystate operation. This is
a reasonable assumption, since most boiler models available
in literature are static models [40]. The input provided to the
3
Figure 1: Condensing boiler efﬁciency curve. As a condensing boiler can re
cover the latent heat of water vapor in the ﬂue gas, its efﬁciency is higher low
water temperature.
boiler is the supply temperature set point Tsw, which determines
the power (on the water side) necessary to reach the set point,
according to
pout =cwwwB(Tsw −Trw)(1)
where cwis the speciﬁc heat of water in [kJ/kgoC], wwBis the
boiler mass ﬂow rate in [kg/s], Tsw and Trw are the temperatures
in [oC] of the supply and return water exiting and entering the
boiler, respectively. Let us distinguish between the power on
the water side and the power on the gas side, by calling them
pout and pin, respectively. The output power of the boiler in
[kW] (on the water side) is pout =
η
(Trw)pin , where pin is the
input power to the boiler in [kW] (on the gas side) and
η
(Trw)
is the (dimensionless) efﬁciency curve depending on Trw (an
example of this curve is shown in Figure 1)1. The boiler mass
ﬂow rate wwBand the ﬁring rate will be assumed to be constant.
2.2. Radiator
The radiator is modeled as a ﬁrstorder system as follows
cw
ρ
wVR˙
TrwR=
cwwwR(Tsw −TrwR)
+KRTZ−TrwR+Tsw
2+
ξ
Rif VALV E =1
ξ
Rif VALV E =0
(2)
where
ρ
wis the density of the water in [kg/m3], VRis the vol
ume of the radiator in [m3], wwRis the water mass ﬂow rate
into the radiator in [kg/s], KR=hRARis the heat transfer coef
ﬁcient in [kW/oC], where ARis the surface area of the radiator
and hRis the convection heat transfer coefﬁcient of the radiator
in [kW/m2oC]. In addition, TrwRrepresents the temperature in
[oC] of the return water out of the radiator, and TZis the temper
ature in [oC] of the air in the zone. The quantity
ξ
Rrepresents
a stochastic process noise. According to (2), the heat exchange
with the room occurs via the difference between the zone tem
perature and the radiator mean temperature.
1The reason which the xaxis in Figure 1 is given in Fahrenheits, is that most
boiler manufactures provide the efﬁciency curve with this unit. Nevertheless,
all the calculations in this work are carried out in Celsius.
Figure 2: Radiator shunt. The radiator operation depends on its valve which in
turn is open/closed via the thermostatic control.
Assuming a heating setting (late fall or winter), we model
the thermostatic control in the radiator as follows:
VALV E =
1 if ˙
TZ>0 and TZ≤Td+hor
if ˙
TZ≤0 and TZ≤Td−h
0 otherwise
(3)
where CLOSED =0, OPEN =1. In other words, the valve can
be fully open or fully closed, leading to raising or decreasing
temperature (thermostat hysteretic behavior).
Remark 1. It is also possible to model multiple valve positions
(not only open or closed), at the expense of complicating the
model, i.e. VALVE =
1 if ˙
TZ>0 and Td+(
δ
−1)h/
δ
≤TZ≤Td+hor
if ˙
TZ≤0 and Td−h≤TZ≤Td−(
δ
−1)h/
δ
.
.
.
2/
δ
if ˙
T>0 and Td+h/
δ
≤TZ≤Td+2h/
δ
or
if ˙
T≤0 and Td−2h/
δ
≤TZ≤Td−h/
δ
1/
δ
if ˙
T>0 and Td≤TZ≤Td+h/
δ
or
if ˙
T≤0 and Td−h/
δ
≤TZ≤Td
0 otherwise
(4)
where
δ
is the discretization step of the valve. This model is not
considered to keep the control setting as simple as possible.
The radiator receives water from the boiler: the presence of
a radiator shunt splits the mass ﬂow rate as
wwS=wwB−wwR
wwS= (1−VALV E(t) + cSV ALV E (t))wwB
wwR= (1−cS)VALV E (t)wwB(5)
where wwSis the water mass ﬂow rate of into the shunt in [kg/s]
and cSis the minimum percentage of ﬂow circulating in the
shunt. In other words, even if the radiator valve is fully open,
a mass ﬂow rate cSwwBwill circulate in the shunt. As shown in
Figure 2, the return water to the boiler is given by the mixing
between the return water from the radiator and the supply water,
Trw =wwRTrwR+wwSTsw
wwB
(6)
4
2.3. Zone
The zone is modeled as a ﬁrstorder system interacting with
the outside air, with the neighbor zones, and with the radiator
ca
ρ
aVZ˙
TZ=
hoAo(To−TZ)+ hnAn(Tn−TZ)
+KRTrwR+Tsw
2−TZ+
ξ
Zif VALV E =1
hoAo(To−TZ)
+hnAn(Tn−TZ)+
ξ
Zif VALV E =0
(7)
where
ρ
ais the density of the air in [kg/m3], VZis the volume
of the zone in [m3], cais the speciﬁc heat of air in [kJ/kg oC],
hoand hnare the convection heat transfer coefﬁcient in [kW/m2
oC] of the indoor air with outside and neighbor zones, respec
tively, Aoand Anare the surface in [m2] of the zone with out
side and neighbor zones, respectively. In addition, Toand Tnare
the temperatures [oC] of the outside and neighbor zone air, re
spectively, while the quantity
ξ
Zrepresents a stochastic process
noise.
3. Boiler efﬁciency
The efﬁciency of the boiler is approximated as a piecewise
afﬁne function of Trw , similarly to what shown in Figure 3. The
approximation range is 3071 oC (corresponding to 90160 oF):
η
(Trw) = ¯
θ
1Trw +¯
θ
2if Trw ≤Tdew
¯
θ
3Trw +¯
θ
4if Trw >Tdew
(8)
with ¯
θ
1Tdew +¯
θ
2=¯
θ
3Tdew +¯
θ
4for continuity of the efﬁciency
curve. In order to reduce the number of parameters from four
to three, we explicitly make use of the continuity condition, so
that the previous expression can be written as
η
(Trw) =
θ
1Trw +
θ
2if Trw ≤Tdew
θ
1Trw +
θ
2+
θ
3(Trw −Tdew)if Trw >Td ew
(9)
The curve (9) has not only less parameters, but it is also con
tinuous by construction. The dew point Tdew is the temperature
at which the condensing process will occur: as commonly done
by all boiler manufacturers, this temperature is given in terms
of the return water temperature even if, from a physical point of
view, it should be calculated in terms of the ﬂue gas temperature
[39]. The dew point is commonly in the range Trw ≈5458oC
(slightly depending on the ﬂue gas composition). The tempera
ture of the return water Tr w is calculated as in (6). In the follow
ing, it is described how the parameters of the curve (9) change
with time as a consequence of performance degradation.
3.1. Boiler degradation and maintenance
In order to include performance degradation we consider
the following multiplicative degradation
η
(Trw)(t) = (1−d(t))
η
(Trw)(0)(10)
where d(t)∈[0,1]indicates the level of degradation from ‘new’
(d(t) = 0)to ‘completely faulty’(d(t) = 1)condensing boiler.
Figure 3: Piecewise afﬁne approximation of the efﬁciency curve (in the range
90160 oF)
The multiplicative degradation (10) is used to model the delete
rious effects of processes like deposition, erosion and corrosion.
The formulation (10) leads to the degradation model
η
(Trw)(t) =
θ
1(1−d(t))Trw +
θ
2(1−d(t)) if Trw ≤Tdew
θ
1(1−d(t))Trw +
θ
2(1−d(t))
+
θ
3(1−d(t))(Trw −Tdew )if Tr w >Td ew
(11)
Remark 2. In other words, the linearintheparameter efﬁciency
model (9) leads to a degradation model (11) whose parameters
θ
1(1−d(t)),
θ
2(1−d(t)),
θ
3(1−d(t)) evolve linearly with the
degradation d(t). So, these parameters (which are typically un
known) can be estimated using standard state estimation tech
niques (cf. Section 5).
We assume an exponential incipient degradation
d(t) = 1−e−
α
t(12)
where
α
>0 is the degradation rate, i.e. the rate of deposi
tion, erosion and corrosion deteriorating the efﬁciency of the
boiler. The relation (12) implies that the halflife of the boiler,
i.e. the time necessary for the efﬁciency to fall to one half of its
initial value is
α
ln(2). Most condensing boilers have halflife
time constant of several months or a few years. The exponential
incipient degradation (12) results in the following degradation
model for the parameters
˙
θ
1(t) = −
αθ
1(t) +
ζ
1,
θ
1(0) =
θ
1new
˙
θ
2(t) = −
αθ
2(t) +
ζ
2,
θ
2(0) =
θ
2new
˙
θ
3(t) = −
αθ
3(t) +
ζ
3,
θ
3(0) =
θ
3new (13)
which is a set of ﬁrstorder ﬁlters driven by stochastic noises.
The parameters
θ
1new ,
θ
2new and
θ
3new describe the efﬁciency for
a new boiler.
Remark 3. The stochastic noises
ζ
1,
ζ
2,
ζ
3account for model
inaccuracies, for example if the degradation is not exactly ex
ponential. Similarly to
ζ
Rand
ζ
Z, by setting appropriate co
variances for such disturbances, the designer will be able to
set to which extent the model is an approximation of the ac
tual system (the larger the covariance, the larger the modelling
inaccuracies).
5
3.2. Actions
Two possible actions, namely control and maintenance ac
tions, can be taken on the system:
•The ﬁrst type of action is the local continuous control,
i.e. setting the set point for building equipment: in the
case at hand, this amounts to properly setting the supply
hot water temperature set point Tsw for the boiler.
•The second type of action is the maintenance discrete
action, i.e. the repair at a certain time ¯
tof the build
ing equipment. In this work we consider an ideal repair
restoring its performance to the initial performance:
θ
1(¯
t) =
θ
1new
θ
2(¯
t) =
θ
2new
θ
3(¯
t) =
θ
3new
if x(¯
t)∈MAINT (14)
where xand MAINT are the variables and the set to be
used by the maintenance strategy, as they will be deﬁned
later. In other words, whenever it occurs, the mainte
nance action is supposed to restore the state of the boiler
to its initial value
θ
new = [
θ
1new
θ
2new
θ
3new ]′.
4. Automaton formulation
The boilerradiatorzone system can be described by a par
ticular class of stochastic hybrid system. A hybrid dynamical
system is an indexed collection of dynamical systems along
with some map for jumping among them (switching dynamical
system and/or resetting the state). This jumping occurs when
ever the state satisﬁes certain conditions, given by its member
ship in a speciﬁed subset of the state space. The hybrid dynam
ical system can be described as
Hc= [Q,Σ,A,G,V,C,F](15)
with constituent parts as follows
•Qis the set of index states or discrete states.
•Σ=Σqq∈Qis the collection of controlled dynamical
systems, where each Σq= [Xq,fq,Uq]is a controlled dy
namical system. Here, Xqare the continuous state spaces,
and fqare the continuous dynamics; Uqis the set of con
tinuous controls.
•A=Aqq∈Q,Aq⊂Xqfor each q∈Q, is the collection
of autonomous jump sets.
•G=Gqq∈Q, where Gq:Aq×Vq→Sis the autonomous
jump transition map, parameterized by the transition con
trol set Vq, a subset of the collection V=Vqq∈Q; they
are said to represent the discrete dynamics and controls.
•C=Cqq∈Q,Cq⊂Xqis the collection of controlled
jump sets.
•F=Fqq∈Q, where Fq:Cq→2Swhere is the collec
tion of controlled jump destination maps.
Figure 4: Automaton associated to the joint energy/maintenance problem: Tit
comprises two regimes (on/off) driven by the thermostat. Each regime con
tains continuous dynamics for evolution of temperature and degradation. The
maintenance action is a discrete control that restores the efﬁciency of the boiler.
•Finally, S=∪q∈QXq×{q}is the hybrid state space of
the dynamical system.
For the problem at hand we have:
•The continuous state space Xqarises from the room tem
perature (7), radiator return water temperature (2) (which
can be observed) and the boiler performance parameters
(13) (which have to be estimated);
•The discrete state space Qarises from the states of the
valve (closed/open state {0,1}or, in case of valve with
quantized openings, states {0,1/
δ
,...,1});
•The autonomous transitions Aare driven by the ther
mostat behavior (3) or (4). This deﬁnes the map G: the
vector ﬁeld changes regime, when the state (room tem
perature) hits the hysteresis boundaries;
•The controlled transitions Care driven by the repair of
the boiler (11). This deﬁnes the map F: the continu
ous state (boiler performance parameters) changes im
pulsively on hitting prescribed regions of the state space
(maintenance action region).
The hybrid dynamical system can be represented as an au
tomaton as in Figure 4: note that the room temperature and
radiator return water temperature evolve continuously, but the
performance parameter evolve discontinuously after repair. The
different regimes of the automaton are formally deﬁned as:
Regime valve open
cw
ρ
wVR˙
TrwR=cwwwR(Tsw −TrwR)
+KRTZ−TrwR+Tsw
2+
ξ
R
ca
ρ
aVZ˙
TZ=hoAo(To−TZ) + hnAn(Tn−TZ)
+KRTrwR+Tsw
2−TZ+
ξ
Z
˙
θ
1(t) = −
αθ
1(t) +
ζ
1
˙
θ
2(t) = −
αθ
2(t) +
ζ
2
˙
θ
3(t) = −
αθ
3(t) +
ζ
3(16)
6
Regime valve closed
cw
ρ
wVR˙
TrwR=
ξ
R
ca
ρ
aVZ˙
TZ=hoAo(To−TZ) + hnAn(Tn−TZ) +
ξ
Z
˙
θ
1(t) = −
αθ
1(t) +
ζ
1
˙
θ
2(t) = −
αθ
2(t) +
ζ
2
˙
θ
3(t) = −
αθ
3(t) +
ζ
3(17)
In addition, some limits are selected taking into account typ
ical operating conditions:
Operating conditions
θ
1≤0,
θ
2≥0,
θ
3≤0
30 ≤Trw ≤70
40 ≤Tsw ≤85
Tz,Tn>To,Tsw >Trw (18)
where the constraints have been selected taking into account
typical operating conditions. It is clear that the control actions
inﬂuence the transitions and thus the behavior of the hybrid dy
namical system. In the following we will introduce a cost to
quantify the performance associated to a certain behavior.
4.1. Cost
The operation of the automaton presented in Figure 4 must
be optimized taking into account the following cost:
Ctot (t) = Coper (t) +Cf ail (t) + Cmaint(t) + Ccycl (t).(19)
The four terms are all monetized in AC, as explained in the fol
lowing.
•Operational costs Coper : the ﬁrst term in (19) is related
to the costs of system operation, which in most cases is
simply the cost of energy consumed by the HVAC system
for given time step as function of observations and inputs.
In the boiler case, the energy consumption is given by pin,
i.e. the energy (at the gas side) necessary to reach the set
point temperature, based on the efﬁciency of the boiler.
As the boiler degrades its performance, more and more
energy will be necessary to achieve the same set point
(for the same return water temperature). The economic
value of this term is derived from the natural gas price
statistics in EU28 [54], which is 0.07 AC/kWh
Coper (t) = 0.07pin(t)(20)
•Failure costs Cf ail : the second term in (19) is related to
the costs due to improper behavior of the system, which
in our case is not following the desired temperature Td.
We focus on zone set points, since in building domain
these are crucial constraints to occupant comfort and thus
productivity. It has been estimated by the Federation
of European Heating, Ventilation and Air Conditioning
(REHVA) that improving indoor environment in ofﬁce
buildings would result in a direct increase in productivity
of 0.5% to 5%: reduction in performance is around 4% at
cooler temperatures and 6% at warmer temperatures [55].
To the purpose of this study, the failure cost is taken as
the squared distance from the desired temperature
Cf ail (t) = 0.2(Td(t)−TZ(t))2/dt (21)
where dt is the sample time. The following estimate is
made: losses of 0.2 AC per sample time for one degree far
from the desired one and 0.8 AC per sample time for two
degrees far from the desired one, and so on.
•Maintenance costs Cmaint : the third term in (19) is related
to the costs of maintenance actions. We will focus on the
maintenance action of reparation.
Cmaint(t) = 500AC at controlled jumps (22)
as the cost of condensing boiler reparation vary in the
range 500  2000 AC depending on the brand or output
[56].
•Cycling costs Ccycl : ﬁnally, one last term should be con
sidered in (19) mainly due to wellposedness reasons
Ccycl(t) = 0.15AC at autonomous jumps (23)
In fact, in order to have a wellposed formulation with no
chattering phenomena (high frequency control actions),
it is necessary to penalize any transition caused by au
tonomous switches (i.e. changes in the valve). Roughly
speaking, assigning a cost to such switches has an inter
pretation in terms of avoiding fast cycling, which could
potentially wear out the equipment.
5. Uncertainty
Let us specify which coefﬁcients can be measured and which
ones must be estimated. The known coefﬁcients are:
•The thermostat hysteretic threshold h;
•Properties of ﬂuids (density and heat capacitance of air
and water);
•Volumes of boiler, radiator and zone;
•The heat transfer coefﬁcients;
•The exponential decay of the boiler efﬁciency
α
;
Uncertainty arises from the unknown coefﬁcients in the boiler
efﬁciency curve (the efﬁciency of the boiler cannot be known
perfectly and it is thus subject to uncertainty):
•The coefﬁcients
θ
1,
θ
2and
θ
3are unknown and must be
estimated.
Let us consider a leastsquares estimator for the following
linearintheparameter model
pin
η
(Trw) = cwwwB(Tsw −Trw)(27)
7
where pin is assumed to be measured (from gas side measure
ments). The leastsquares estimator takes the form
˙
¯
θ
=P
θεφ
˙
P
θ
=
κ
P
θ
−P
θφφ
′P
θ
(28)
where ¯
θ
= [
θ
1
θ
2
θ
3]′,
ε
=pin ˆ
η
(Trw)−cwwwB(Tsw −Trw)is the
estimation error, ˆ
η
(Trw)is the estimate of
η
(Trw),
κ
>0 is a
forgetting factor,
φ
= [Trw 1Trw −Tdew]′is the regressor vector,
and P
θ
is the covariance matrix of the uncertainty. In order to
take into account the error arising from neglecting the boiler
transient, the following stochastic model is taken
η
(Trw) = cwwwB(Tsw −Tr w)
pin
+
ζ
B(29)
where
ζ
Bis a stochastic noise. Because the management/ main
tenance algorithm is ultimately implemented on a digital con
troller, we consider the estimator in discrete time in place of
(28). After discretization of (13) using backward Euler and
sample time dt, the parameters in
η
(Trw)are estimated using
the stochastic Kalman ﬁlter at the end of the page.
Remark 4. It has to be noted that, despite the fact that the con
densing boiler operates in two modes, a unique estimate ¯
θ
and
a unique covariance matrix P
θ
are updated. Because
θ
1and
θ
2are shared in both modes, but
θ
3can be observed only in
the noncondensing mode, the following estimation strategy is
adopted:
•in the noncondensing mode, fullorder update (for the
three components from measurements) and fullorder pre
diction (based on dynamics for the three components) is
performed;
•in the noncondensing mode, reducedorder update (for
the ﬁrst two components from measurements) and full
order prediction (based on dynamics for the three com
ponents) is performed.
This amounts to assuming that when
θ
3is not observed its esti
mate is only based on the evolution of its dynamics. In fact,
θ
3will decay exponentially also when it is not observed (in
the condensing mode). Also, every time a maintenance oc
curs, we reset the covariance matrix P
θ
to some initial value,
i.e. P
θ
(¯
t) = P
θ
new , where ¯
t is the same instant as in (14). This
is done in order to reset the a priori knowledge of the Kalman
estimator.
6. Optimization approach
The idea is to adopt the uniﬁed framework for optimal con
trol in hybrid systems [57, 58]. The following total discounted
cost is considered
Z∞
0
e−
λτ
Coper (
τ
) + Cfail (
τ
)d
τ
+∑
i
e−
λ
tiCcycl(ti) + ∑
i
e−
λ
¯
tiCmaint(¯
ti)(30)
where
λ
is the discounting factor for the future cost. The deci
sion variables over which this cost has to be minimized are: the
continuous control Tsw (boiler set point), and the maintenance
strategy, which will both be deﬁned in the following.
According to the way the automaton in Fig. 4 is operated,
the cost (30) can be closer to or more distant from the optimum.
It is clear that the continuous control Tsw (supply water temper
ature) is a crucial parameter in operating the automaton. Let
us deﬁne a simple base case strategy for operating the supply
water temperature:
Tsw(t) =
Trw if V ALV E =0 or
if VALV E =1 and TZ>Td
Trw +kp(Td−TZ)if V ALV E =1 and TZ≤Td
(31)
with kp>0 a proportional gain to be designed. The strategy
(31) amounts to setting the boiler off when the radiator valve is
Backward Euler discretization:
Degradation
θ
1(k+1) =
θ
1(k) + dt
ζ
1
1+dt
α
,
θ
2(k+1) =
θ
2(k) + dt
ζ
2
1+dt
α
,
θ
3(k+1) =
θ
3(k) + dt
ζ
3
1+dt
α
(24)
Update (from measurements):
Innovation or measurement residual
ε
=z−
φ
′
θ
kk−1
Innovation (or residual)covariance S=
φ
′P
θ
kk−1
φ
+
σ
B
Optimal Kalman gain K=P
θ
kk−1
φ
S−1
Updated (a posteriori)state estimate
θ
kk=
θ
kk−1+K
ε
Updated (a posteriori)estimate covariance P
θ
kk= (I−K
φ
′)P
θ
kk−1(25)
Predict (based on dynamics):
Predicted (a priori)state estimate
θ
k+1k=A
θ
kk
Predicted (a priori)estimate covariance P
θ
k+1k=AP
θ
kkA′+E′
σ
2
DE(26)
where
σ
2
Bis the covariance of
ζ
B,
σ
2
Dis the covariance of
ζ
D= [
ζ
1
ζ
2
ζ
3],A=1
1+dt
α
I3and E=dt
1+dt
α
I3.
8
Table 2: Summary of the tasks to be accomplished (energy management and maintenance schedule) and of the policies to accomplish them (certainty equivance,
cautious and dual policy).
Task Type of control Required measurements Variable Optimization parameter
Energy
management
Continuous (water
temperature)
Room temperature
Supply water temperature
Hot water
supply temperature
Proportional
gain kp
Maintenance
scheduling
Discrete (maintain
 not maintain)
Boiler input power (gas)
Return water temperature
Minimum allowed
efﬁciency
Degradation
threshold rth
Policy Energy management Maintenance scheduling Type of learning
Certainty
equival.
Strategy (32): nothing is done
to reduce estimation error
Strategy (33): deals with estimated
efﬁciency as the true one Passive
Cautious Strategy (32): nothing is done
to reduce estimation error
Strategy (35): uncertainty bound is
added to estimated efﬁciency Passive
Dual Strategy (37): tries to reduce
estimation error via probing
Strategy (35): uncertainty bound is
added to estimated efﬁciency Active
closed or when the valve is open but the temperature is above
the desired temperature (note that when Tsw =Trw then pin =0).
When the valve is open and the temperature of the zone is below
the desired temperature, the supply water set point is increased
proportionally to the difference Td−TZ.
In order to reformulate (31) as a function of the continuous
state (TrwR,TZ), we equivalently rewrite it as
Tsw(t) =
TrwRif V ALV E =0 or
if VALV E =1 and TZ>Td
wwRTrwR+wwBkp(Td−TZ)
wwR
if VALV E =1 and TZ≤Td
(32)
For the maintenance strategy, a simple idea is to perform main
tenance once the efﬁciency falls below a certain percentage r%
of the efﬁciency of a new boiler. However, in view of uncer
tainty in the parameters
θ
1,
θ
2and
θ
3, the efﬁciency must be
estimated. Therefore, we propose two different strategies de
pending on how the estimated efﬁciency is used.
6.1. Certainty equivalence strategy
The simplest idea for maintenance strategy is to perform
maintenance once the estimated efﬁciency falls below a cer
tain percentage rth% of the efﬁciency of a new boiler. In other
words, in this certainty equivalence framework the control ac
tion is calculated as if the estimate ˆ
θ
(k)were exact. This amounts
to neglecting any uncertainty in the estimate
MAI NT (t) = (1 if ˆ
η
(Trw)(t)
η
new(Trw)≤rth
0 otherwise (33)
where
η
new(Trw)is the efﬁciency curve of a new boiler (with
parameters
θ
1new ,
θ
2new and
θ
3new ), and it has to be noted that we
take into account that the efﬁciency curve depends on Trw .
Remark 5. Two things must be noticed. The ﬁrst one is that
it is not difﬁcult to include the ﬁring rate, call it f r in the ef
ﬁciency model, i.e.
η
(Trw,f r ): this can be simply achieved by
a linear in the parameter model not only with respect to Trw ,
but with respect to Trw and f r. In fact, most efﬁciency curves
for boilers, heat pumps etc are given as linear in the param
eter models with respect to two or more parameters (cf. [23]
for more details). The second one is that, according to (14), we
have x = [Trw
θ
1
θ
2
θ
3]′and that the strategy MAINT is deﬁned
by the parameters Td ew, rth ,
θ
1new ,
θ
2new and
θ
3new .
6.2. Cautious strategy
The maintenance strategy (33) does not take into account
any uncertainty in the estimation of the efﬁciency. A simple
design to take into account the uncertainty is a ‘cautious’ con
trol action that adds a measure of caution depending on the un
certainty: to this purpose we ﬁrst deﬁne the covariance of the
efﬁciency
σ
2
η
(t) =
φ
′(t)P
θ
(t)
φ
(t)(34)
and then we deﬁne the ‘cautious’ threshold
MAI NT (t) = (1 if ˆ
η
(Trw)(t)−3
ση
(t)
η
new(Trw)≤rth
0 otherwise (35)
In other words, for the same threshold rth , the cautions con
troller will tend to do maintenance more often if
ση
is large.
Basically, we notice that the numerator decreases if the un
certainty associated to ˆ
η
(Trw)increases. The uncertainty on
ˆ
η
(Trw)is measured as the square root of the covariance
σ
2
η
(t).
If
ση
reduces to zero, then the cautious strategy converges to
the certainty equivalence strategy. Therefore, a crucial question
arises: is it possible to actively reduce uncertainty by means of
the control action? The next strategy tries to address this ques
tion.
6.3. Dual strategy
Both the certainty equivalence and the cautious strategy are
adaptive because they depend on an efﬁciency curve that is
estimated, and thus adapted, online. However, both the cer
tainty equivalence and the cautious strategies are passive learn
ing policies because they do not involve any active probing sig
nal generated to improve the estimation of the efﬁciency curve.
9
Figure 5: Features of the zoneboilerradiator test case used to test the proposed framework. The building test case (of around 150 m2) contains a boiler driving a
zone with radiators. The boiler uses a proportional controller to set the water supply temperature, while the thermostatic controller determines on and off regimes.
The efﬁciency of the boiler degrades with time, so that maintenance is needed. The ﬂow diagrams of energy/maintenance controls are in the middle. The overall
energy/maintenance scenario can be managed according to three policies (certainty equivalence/cautious/dual).
In the following, we want to create a more active learning mech
anism. Let us deﬁne the proportional gain:
¯
kp=kp+3k
σση
,kp,k
σ
>0 (36)
In other words, the control gain increases if the uncertainty in
ˆ
η
(Trw)increases. This is because when the uncertainty is large,
larger control actions might help in reducing the estimation er
ror (and thus in reducing
σ
2
η
(t)). It has to be remarked that pin
enters the linearintheparameters model, thus the selection of
Tsw has an effect on reducing the uncertainty. Note that the op
posite mechanism (the gain ¯
kpis decreased as the uncertainty is
increased) is not desirable: since the effect of a control action
on estimation is not taken into account, this can lead to turning
off the controller if the uncertainty becomes too large.
In view of these considerations, the dual strategy becomes
Tsw(t) =
TrwRif V ALV E =0 or
if VALV E =1 and TZ>Td
wwRTrwR+wwB¯
kp(Td−TZ)
wwR
if VALV E =1 and TZ≤Td
(37)
Summarizing, the proposed strategies are illustrated in Table 2:
their performance will be compared via numerical simulations.
7. Simulation experiments
Simulation experiments are performed on a ‘smart building’
simulator developed by the authors. The simulator environ
ment is based on the zoneboilerradiator dynamics described
in the previous sections, implemented in Matlab in a similar
way as previously done by some of the authors in [23]. A vi
sualization of the features of the simulator can be seen in Fig.
5, with the available measurements, the ﬂow diagrams of en
ergy/maintenance controls and the different policies. The sim
ulator also comprises a few additional modules, such as ther
mostat features and testing criteria which are proprietary and
0 100 200 300 400 500 600 700 800 900
TO [o C]
20
10
0
10
20
time [h]
0 100 200 300 400 500 600 700 800 900
SO [kW/m2]
0
0.2
0.4
0.6
0.8
1
Figure 6: Weather data (outside temperature and solar radiation) during 36 win
ter days. The data are used to drive the ‘smart building’ simulator.
cannot be disclosed due to intellectual property agreement. The
weather data used for the simulations represent outside temper
ature and solar radiation for 36 days; as it can be seen from the
trend of the weather, initially we have a quite rigid winter that
evolves into a milder one. In order to consider a longer sim
ulation horizon, we repeated these 36 days for 60 times, with
random perturbations on the values. In this way we are able to
simulate around 2200 days of winter season. The simulations
are run to optimize the following parameters:
•For the certainty equivalence strategy: kpand rth ;
•For the cautious strategy: kpand rth ;
•For the dual strategy: kp,k
σ
and rth.
10
50
40
30
kp
20
10
0
0
0.2
rth
0.4
1.5
0
1
2
0.5
0.6
Cost/h
Figure 7: Certainty equivalence strategy: Average hourly cost as a function of
kpand rth .
50
40
30
kp
20
10
0
0
0.2
rth
0.4
0
0.5
1
1.5
2
0.6
Cost/h
Figure 8: Cautious strategy: Average hourly cost as a function of kpand rth.
50
40
30
kp
20
10
0
0
0.2
rth
0.4
1.2
1
0.8
0.6
0.4
1.6
1.4
0.6
Cost/h
Figure 9: Dual strategy: Average hourly cost as a function of kpand rth (with
k
σ
=0.5).
0.8
0.6
Oper
0.4
0.2
0
0
0.5
Fail
1
0.35
0.05
0.1
0.15
0.2
0.25
0.3
1.5
Maint
Figure 10: Certainty equivalence strategy: Pareto front for Coper,Cf ail and
Cmaint.
0.5
0.4
0.3
Oper
0.2
0.1
0
0
0.5
Fail
1
0.3
0.2
0.1
0
0.4
0.6
0.5
1.5
Maint
Figure 11: Cautious strategy: Pareto front for Coper,Cf ail and Cmaint .
0.5
0.4
0.3
Oper
0.2
0.1
0
0
0.5
Fail
1
0
0.4
0.3
0.6
0.2
0.1
0.5
1.5
Maint
Figure 12: Dual strategy: Pareto front for Coper ,Cfail and Cmaint.
11
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Tsw, Trw [ o C]
40
60
80
time [days]
0 200 400 600 800 1000 1200 1400 1600 1800 2000
pin [kW]
0
10
20
30
Average pin =4.86 kW/h
Figure 13: Certainty equivalence strategy: water temperatures and boiler power.
The average power input is 4.86 kW/h.
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Tsw, Trw [ o C]
40
60
80
time [days]
0 200 400 600 800 1000 1200 1400 1600 1800 2000
pin [kW]
0
10
20
30
Average pin =4.75 kW/h
Figure 14: Cautious strategy: water temperatures and boiler power. The aver
age power input is 4.75 kW/h.
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Tsw, Trw [ o C]
40
60
80
time [days]
0 200 400 600 800 1000 1200 1400 1600 1800 2000
pin [kW]
0
10
20
30
Average pin =4.10 kW/h
Figure 15: Dual strategy: water temperatures and boiler power. The average
power input is 4.10 kW/h.
0 200 400 600 800 1000 1200 1400 1600 1800 2000
θ1, θ2, θ3
0.5
0
0.5
time [days]
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Trace Pθ
×103
0
0.5
1
1.5
Average error = 5.91× 104
Figure 16: Certainty equivalence strategy: estimation of efﬁciency parameters.
The average estimation error is 5.91·10−4(adimensional).
0 200 400 600 800 1000 1200 1400 1600 1800 2000
θ1, θ2, θ3
0.5
0
0.5
time [days]
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Trace Pθ
×103
0
0.5
1
1.5
Average error = 3.98× 104
Figure 17: Cautious strategy: estimation of efﬁciency parameters. The average
estimation error is 3.98·10−4(adimensional).
0 200 400 600 800 1000 1200 1400 1600 1800 2000
θ1, θ2, θ3
0.5
0
0.5
time [days]
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Trace Pθ
×103
0
0.5
1
1.5
Average error = 1.97× 104
Figure 18: Dual strategy: estimation of efﬁciency parameters. The average
estimation error is 1.97·10−4(adimensional).
12
Because of the low number of parameters, we can optimize
the parameters using a brute force approach over a grid. The
following initial grid has been chosen for optimization
kp: 0.1,0.25,0.5,1,2,5,10,20,50
k
σ
: 0.1,0.25,0.5,1,2.5
rth : 0.1,0.15,0.2,0.25,0.3,0.35,0.4,0.45,0.5,0.55
(38)
whose meaning is the following: the proportional gain kpis
selected from low gain (shallow control) to high gain (aggres
sive control); the threshold rth goes from 10% degradation to
55% degradation with steps of 5%; ﬁnally, the probing gain k
σ
goes from low probing to high probing. All experiments run
on a Dell OptiPlex 7060 MT, Intel Core i58500, 8GB RAM
DDR4, Windows 10. The simulation platform is developed
in Matlab R2016b. Spanning the complete grid (450 possible
policies) takes around 4 hours (around half a minute to simu
late one policy over a 2200 day horizon). Then, when a rough
estimate of the optimal point has been found, the grid can be
reﬁned and reduced in size to further improve the performance.
Spanning this smaller grid takes around 30 minutes. Therefore,
we veriﬁed that the computational complexity of the proposed
approach is relatively low, and it is due to the fact that one pol
icy can be simulated over a 2200 day horizon in less than half
a minute. Key to such a fast simulation is the relatively sim
ple nature of the proposed hybrid modelling, which can signiﬁ
cantly simplify the model while still retaining the main features
of the HVAC maintenance problem. We believe that the pro
posed hybrid modelling reaches quite a good tradeoff in terms
of realistic maintenance scenario and complexity of the formu
lation. It is inevitable that, when increasing the number of states
and actions, the proposed optimization would become more and
more complex and cumbersome: such an issue can be studied
in future work.
The results of the optimization are shown in Table 3, where
it is shown that the dual strategy can improve the cost of around
0.02 AC/h with respect to the certainty equivalence strategy, and
of around 0.01 AC/h with respect to the cautious strategy (note
that this amounts to around 175 AC/year and 87.5 AC/year, which
are non negligible savings for a zone of 150 m2). The ﬁrst thing
to be noticed is that apparently the certainty equivalence strat
egy is not the best strategy to be adopted: this is because the
controller is designed independently from the estimator, there
fore learning is passive and uncertainty leads to non optimal
maintenance decisions. By selecting a lower threshold, the cau
tious strategy adds some caution in the maintenance which helps
in improving the cost: however, even in this case the controller
Table 3: Parameters and costs for each strategy
kprth k
σ
Hourly cost
Certainty 12 0.32  0.494 AC/h
Cautious 5 0.17  0.486 AC/h
Dual 6 0.24 0.5 0.475 AC/h
is designed independently from the estimator, therefore learn
ing is passive. The dual strategy is the one for which the con
troller is codesigned with the estimator (active learning): this
dual role of control is apparently really necessary to improve
the performance even more. This is clearly an interesting re
sult, as it shows that passive learning (certainty equivalence or
cautious strategies) on the long run does not pay. Reducing un
certainty (active learning) can lead to economic beneﬁts on the
long run. This might stimulate future study and implementation
of active learning policies.
In Figs. 7, 8 and 9 we plot how average hourly cost depends
on the parameters kpand rth. Figs. 10, 11 and 12 show how
the different components in the cost (Coper ,Cf ail and Cmaint)
change by changing the parameters. It can be seen that a Pareto
front exists in all cases: there are tradeoffs regarding optimiza
tion of the different costs. In order to better investigate such
Pareto from, let us compare Figs. 17 and 14 (cautious strat
egy), with Figs. 18 and 15 (dual strategy). As indicated in
Figs. 13, 14 and 15, it can be seen that the dual strategy is the
one that saves more on energy (followed by the cautious and
the certainty equivalence): therefore, the dual strategy seems
to be the one with the best tradeoff between energy costs and
maintenance costs. Figs. 16, 17 and 18 show the estimation of
the efﬁciency parameters and the covariance of the efﬁciency
curve: the peaks that can be noticed in the ﬁgures correspond
to a maintenance action restoring the efﬁciency to their initial
values (this also requires to reset the covariance matrix of the
estimator). The boiler has to be maintained several times be
cause the degradation rate of the boiler has been set a bit higher
than usual on purpose, in such a way to have a richer scenario
in which maintenance is required quite often. As indicated in
the ﬁgures, the dual strategy is the one with the smallest es
timation error (this can be seen also by noticing that the dual
strategy is the one whose covariance matrix decreases faster),
followed by the cautious strategy and by the certainty equiv
alence strategy. Therefore, the estimation of the dual strategy
is more accurate, which is due to the presence of the term k
σ
.
Note that the certainty equivalence strategy is the one with the
largest gain kp, therefore one would expect somehow a better
estimation performance due to high gain: however, this does
not happen and it is really the term k
σ
in the dual strategy that,
by making the controller interact with the estimator, contributes
in a sensible reduction of the estimation error. Overall, the sim
ulations demonstrate that the best economic cost of the system
is achieved by active learning, i.e. the dual strategy, in which
control interacts with estimation.
8. Conclusions
In smart buildings, the models used for Heating, Ventila
tion, and Air Conditioning energy management and for mainte
nance scheduling differ in scope and structure: while the mod
els for energy management describe continuous quantities (en
ergy, temperature), the models used for maintenance scheduling
describe only a few discrete states (healthy/faulty equipment,
and fault typology). In addition, models for energy manage
ment typically assume the Heating, Ventilation, and Air Condi
13
tioning equipment to be healthy, whereas the models for main
tenance scheduling do not take possible human factors (e.g. dis
comfort) into account. In this work, a framework for human
centric optimal maintenance is proposed: energy management
and maintenance scheduling for Heating, Ventilation, and Air
Conditioning are recast in the same optimization framework.
Both continuous and discrete states are embedded as hybrid dy
namics of the system: in addition, both continuous controls (for
energy management) and discrete controls (for maintenance
scheduling) are considered. Because of the presence of uncer
tainty (the status of the equipment) the solution to the problem
is addressed via an adaptive dual control formulation, where
control occurs jointly with estimation. Numerical examples ob
tained via a zoneboilerradiator test case demonstrate the ef
fectiveness of the approach.
This work can further proceed in many directions: (1) con
sidering more complex models can give a more realistic mea
sure of human comfort, such as the Predicted Mean Vote and the
Predicted Percentage of Dissatisfaction; (2) studying tradeoffs
between complexity of human comfort models (requiring mea
surements of metabolic rate, ratio of clothed/nude surface area,
surface temperature of clothing, air velocity relative to human
body, etc.) and most commonly available sensors (temperature
and humidity); (3) studying the feasibility and the effectiveness
of considering simpliﬁed models, i.e. studying how the sim
pliﬁcations in the human comfort model introduce additional
uncertainties to be estimated on line and to be embedded in the
optimization; (4) extending the maintenance actions by consid
ering nonideal reparation (that would decrease the degradation
without restoring the initial state) or inspection (whose effect
would be to improve the estimate of the efﬁciency, possibly re
setting the estimate and the covariance matrix).
Acknowledgment
The research leading to these results has been partially funded
by the European Commission FP7PEOPLE2012IAPP  Marie
Curie Action: IndustryAcademia Partnerships and Pathways,
under contract #324432 (Advanced Methods in Building Di
agnostics and Maintenance, AMBI). The ﬁrst author acknowl
edges the support by ”the Fundamental Research Funds for The
Central Universities” under the project RECONSTRUCT. The
second author appreciates the ﬁnancial supports by the State
Key Laboratory of Intelligent Control and Decision of Com
plex Systems, the Young Scientist Fund of the National Natural
Science Foundation of China (Grant No. 61703099), and the
China Postdoctoral Science Foundation Funded Project (Grant
No. 2017M621589).
References
[1] Luis PerezLombard, Jose Ortiz, and Ismael R. Maestre. The map of
energy ﬂow in hvac systems. Applied Energy, 88(12):5020 – 5031, 2011.
ISSN 03062619.
[2] D.H. Blum, K. Arendt, L. Rivalin, M.A. Piette, M. Wetter, and C.T. Veje.
Practical factors of envelope model setup and their effects on the perfor
mance of model predictive control for building heating, ventilating, and
air conditioning systems. Applied Energy, 236:410 – 425, 2019.
[3] Rongpeng Zhang and Tianzhen Hong. Modeling of hvac operational
faults in building performance simulation. Applied Energy, 202:178 –
188, 2017.
[4] L. Wang, S. Greenberg, J. Fiegel, A. Rubalcava, S. Earni, X. Pang, R. Yin,
S. Woodworth, and J. HernandezMaldonado. Monitoringbased HVAC
commissioning of an existing ofﬁce building for energy efﬁciency. Ap
plied Energy, 102:1382 – 1390, 2013. Special Issue on Advances in sus
tainable biofuel production and use  XIX International Symposium on
Alcohol Fuels  ISAF.
[5] S. Baldi, A. Karagevrekis, I. Michailidis, and E. B. Kosmatopoulos.
Joint energy demand and thermal comfort optimization in photovoltaic
equipped interconnected microgrids. Energy Conversion and Manage
ment, 101:352–363, 2015.
[6] E. H. Mathews, D. C. Arndt, C. B. Piani, and E. Van Heerden. Developing
cost efﬁcient control strategies to ensure optimal energy use and sufﬁcient
indoor comfort. Applied Energy, 66(2):135–159, 2000.
[7] S. Baldi, I. Michailidis, C. Ravanis, and E. B. Kosmatopoulos. Model
based and modelfree plugandplay building energy efﬁcient control. Ap
plied Energy, 154:829–841, 2016.
[8] Xiaohua Wu, Xiaosong Hu, Xiaofeng Yin, Caiping Zhang, and Shide
Qian. Optimal battery sizing of smart home via convex programming.
Energy, 140:444 – 453, 2017.
[9] Xiaohua Wu, Xiaosong Hu, Yanqiong Teng, Shide Qian, and Rui Cheng.
Optimal integration of a hybrid solarbattery power source into smart
home nanogrid with plugin electric vehicle. Journal of Power Sources,
363:277 – 283, 2017.
[10] J.C. Solano, L. Olivieri, and E. Caama˜
noMart´
ın. Assessing the potential
of pv hybrid systems to cover hvac loads in a gridconnected residen
tial building through intelligent control. Applied Energy, 206:249 – 266,
2017.
[11] Shuangping Duan, Zhiwen Luo, Xinyan Yang, and Yuguo Li. The im
pact of building operations on urban heat/cool islands under urban densi
ﬁcation: A comparison between naturallyventilated and airconditioned
buildings. Applied Energy, 235:129 – 138, 2019.
[12] Dashamir Marini. Optimization of hvac systems for distributed generation
as a function of different types of heat sources and climatic conditions.
Applied Energy, 102:813 – 826, 2013.
[13] C. D. Korkas, S. Baldi, and E. B. Kosmatopoulos. 9  gridconnected
microgrids: Demand management via distributed control and humanin
theloop optimization. In I. Yahyaoui, editor, Advances in Renewable
Energies and Power Technologies, pages 315 – 344. Elsevier, 2018.
[14] A. Ghahramani, F. Jazizadeh, and B. BecerikGerber. A knowledge based
approach for selecting energyaware and comfortdriven HVAC tempera
ture set points. Energy and Buildings, 85:536 – 548, 2014.
[15] Siddharth Goyal, Prabir Barooah, and Timothy Middelkoop. Experimen
tal study of occupancybased control of hvac zones. Applied Energy, 140:
75 – 84, 2015.
[16] P. Carreira, A. Aguiar Costa, V. Mansur, and A. Ars´
enio. Can HVAC
really learn from users? a simulationbased study on the effectiveness of
voting for comfort and energy use optimization. Sustainable Cities and
Society, 41:275 – 285, 2018.
[17] Wooyoung Jung and Farrokh Jazizadeh. Humanintheloop hvac opera
tions: A quantitative review on occupancy, comfort, and energyefﬁciency
dimensions. Applied Energy, 239:1471 – 1508, 2019.
[18] X. He, Z. Zhang, and A. Kusiak. Performance optimization of HVAC sys
tems with computational intelligence algorithms. Energy and Buildings,
81:371 – 380, 2014.
[19] M. Liu, K. B. Wittchen, and P. K. Heiselberg. Control strategies for in
telligent glazed fac¸ade and their inﬂuence on energy and comfort perfor
mance of ofﬁce buildings in Denmark. Applied Energy, 145:43–51, 2015.
[20] Z. Wang, L. Wang, A. I. Dounis, and R. Yang. Multiagent control system
with information fusion based comfort model for smart buildings. Applied
Energy, 99:247–254, 2012.
[21] M. Mossolly, K. Ghali, and N. Ghaddar. Optimal control strategy for a
multizone air conditioning system using a genetic algorithm. Energy, 34
(1):58–66, 2009.
[22] C. Tzivanidis, K. A. Antonopoulos, and F. Gioti. Numerical simulation
of cooling energy consumption in connection with thermostat operation
mode and comfort requirements for the Athens buildings. Applied Energy,
88(8):2871–2884, 2011.
[23] H. Satyavada and S. Baldi. An integrated controloriented modelling for
14
HVAC performance benchmarking. Journal of Building Engineering, 6:
262 – 273, 2016.
[24] Hussain Syed Asad, Richard Kwok Kit Yuen, and Gongsheng Huang.
Multiplexed realtime optimization of hvac systems with enhanced con
trol stability. Applied Energy, 187:640 – 651, 2017.
[25] I. T. Michailidis, T. Schild, R. Sangi, P. Michailidis, C. Korkas, J. Futterer,
D. Muller, and E. B. Kosmatopoulos. Energyefﬁcient HVAC manage
ment using cooperative, selftrained, control agents: A reallife german
building case study. Applied Energy, 211:113 – 125, 2018.
[26] T. Samad and A. M. Annaswamy. Controls for smart grids: Architectures
and applications. Proceedings of the IEEE, 105(11):2244–2261, 2017.
[27] Huilong Wang, Shengwei Wang, and Rui Tang. Development of grid
responsive buildings: Opportunities, challenges, capabilities and applica
tions of hvac systems in nonresidential buildings in providing ancillary
services by fast demand responses to smart grids. Applied Energy, 250:
697 – 712, 2019.
[28] S. Wang, Q. Zhou, and F. Xiao. A systemlevel fault detection and di
agnosis strategy for HVAC systems involving sensor faults. Energy and
Buildings, 42:477–490, 2010.
[29] S. Wu and J.Q. Sun. Crosslevel fault detection and diagnosis of building
HVAC systems. Energy and Environment, 46:1558–1566, 2011.
[30] P. M. Papadopoulos, V. Reppa, M. M. Polycarpou, and C. G. Panayiotou.
Distributed diagnosis of actuator and sensor faults in HVAC systems.
IFACPapersOnLine, 50(1):4209 – 4215, 2017. 20th IFAC World
Congress.
[31] R. Ferrari, H. Dibowski, and S. Baldi. A message passing algorithm for
automatic synthesis of probabilistic fault detectors from building automa
tion ontologies. IFACPapersOnLine, 50(1):4184 – 4190, 2017. 20th
IFAC World Congress.
[32] Dian ce Gao, Shengwei Wang, Kui Shan, and Chengchu Yan. A system
level fault detection and diagnosis method for low deltat syndrome in the
complex hvac systems. Applied Energy, 164:1028 – 1038, 2016.
[33] Marco Bonvini, Michael D. Sohn, Jessica Granderson, Michael Wetter,
and Mary Ann Piette. Robust online fault detection diagnosis for hvac
components based on nonlinear state estimation techniques. Applied En
ergy, 124:156 – 166, 2014.
[34] W.J.N. Turner, A. Staino, and B. Basu. Residential HVAC fault detection
using a system identiﬁcation approach. Energy and Buildings, 151:1 –
17, 2017.
[35] C. Yang, W. Shen, Q. Chen, and B. Gunay. A practical solution for HVAC
prognostics: Failure mode and effects analysis in building maintenance.
Journal of Building Engineering, 15:26 – 32, 2018.
[36] L. Rosa and R. Tosato. Experimental evaluation of seasonal efﬁciency of
condensing boilers. Energy and Buildings, 14:237–241, 1990.
[37] K. Jeong, M. J. Kessen, H. Bilirgen, and E. K. Levy. Analytical mod
eling of water condensation in condensing heat exchanger. International
Journal of Heat and Mass Transfer, 53:2361–2368, 2010.
[38] C. Di Perna, G. Magri, G. Giuliani, and G. Serenelli. Experimental as
sessment and dynamic analysis of a hybrid generator composed of an air
source heat pump coupled with a condensing gas boiler in a residential
building. Applied Themal Engineering, 76:86–97, 2015.
[39] H. Satyavada and S. Baldi. Monitoring energy efﬁciency of condensing
boilers via hybrid ﬁrstprinciple modelling and estimation. Energy, 142:
121 – 129, 2018.
[40] S. Baldi, T. Le Quang, O. Holub, and P. Endel. Realtime monitoring en
ergy efﬁciency and performance degradation of condensing boilers. En
ergy Conversion and Management, 136:329 – 339, 2017.
[41] K. Bruton, P. Raftery, B. Kennedy, M. M. Keane, and D. T. J. O’Sullivan.
Review of automated fault detection and diagnostic tools in air handling
units. Energy efﬁciency, 7:335–351, 2014.
[42] J. Schein, S. T. Bushby, N. S. Castro, and J. M. House. A rulebased
fault detection method for air handling units. Energy and Buildings, 38:
1485–1492, 2006.
[43] S. Baldi, S. Yuan, P. Endel, and O. Holub. Dual estimation: Constructing
building energy models from data sampled at low rate. Applied Energy,
169:81 – 92, 2016.
[44] Y. Yu, D. Woradechjumroen, and D. Yu. A review of fault detection and
diagnosis methodologies on airhandling units. Energy and Buildings, 82:
550–562, 2014.
[45] J. Qin and S. Wang. A fault detection and diagnosis strategy of vav air
conditioning systems for improved energy and control performances. En
ergy and Buildings, 37:1035–1048, 2005.
[46] C. Nzukam, A. Voisin, E. Levrat, D. Sauter, and B. Iung. A dynamic
maintenance decision approach based on maintenance action grouping
for HVAC maintenance costs savings in nonresidential buildings. IFAC
PapersOnLine, 50(1):13722 – 13727, 2017. 20th IFAC World Congress.
[47] Syed Aftab Rashid, Zeeshan Haider, S.M. Chapal Hossain, Kashan
Memon, Fazil Panhwar, Momoh Karmah Mbogba, Peng Hu, and Gang
Zhao. Retroﬁtting lowcost heating ventilation and airconditioning sys
tems for energy management in buildings. Applied Energy, 236:648 –
661, 2019.
[48] S. Haesaert, N. Cauchi, and A. Abate. Certiﬁed policy synthesis for gen
eral markov decision processes: An application in building automation
systems. Performance Evaluation, 117:75 – 103, 2017.
[49] M. Rasouli, G. Ge, C. J. Simonson, and R. W. Besant. Uncertainties
in energy and economic performance of HVAC systems and energy re
covery ventilators due to uncertainties in building and HVAC parameters.
Applied Thermal Engineering, 50(1):732 – 742, 2013. ISSN 13594311.
[50] J. Verhelst, G. Van Ham, D. Saelens, and L. Helsen. Model selection
for continuous commissioning of HVACsystems in ofﬁce buildings: A
review. Renewable and Sustainable Energy Reviews, 76:673 – 686, 2017.
[51] N. Cauchi, K. Macek, and A. Abate. Modelbased predictive maintenance
in building automation systems with user discomfort. Energy, 138:306 –
315, 2017.
[52] A.Y. Cheong Peng, A. Azlan Shah, and A. Faizah. Improving occupants’
satisfaction with effective maintenance management of HVAC system in
ofﬁce buildings. Automation in Construction, 43:31 – 37, 2014.
[53] R. M. Lazzarin. Condensing boilers in buildings and plants refurbish
ment. Energy and Buildings, 47:61–67, 2012.
[54] Natural gas price statistics. URL \url{http://ec.europa.eu/
eurostat/statisticsexplained/index.php/Natural_
gas_price_statistics}.
[55] Optimal thermal environment improves performance
of ofﬁce work. URL \url{https://www.
rehva.eu/fileadmin/hvacdictio/01 2012/
optimalthermal environmentimprovesperformance\
of officework_rj1201.pdf}.
[56] Condensing boilers prices. URL \url{https://www.
boilersprices.co.uk/condensingboilers prices/}.
[57] M. S. Branicky, V. S. Borkar, and S. K. Mitter. A uniﬁed framework for
hybrid control: model and optimal control theory. IEEE Transactions on
Automatic Control, 43(1):31–45, 1998.
[58] M. Rungger and O. Stursberg. Optimal control for deterministic hybrid
systems using dynamic programming. IFAC Proceedings Volumes, 42
(17):316 – 321, 2009. 3rd IFAC Conference on Analysis and Design of
Hybrid Systems.
15
Appendix A. Parameters used for simulation
Symbol Value Unit
Tdew 57.2 [oC]
Td21 [oC]
wwB0.15 [kg/s]
cS0.5 [%]
cw4.179 [kJ/kgoC]
ρ
w992.3 [kg/m3]
ca1.005 [kJ/kgoC]
ρ
a1.205 [kg/m3]
VR0.971 [m3]
AR14.63 [m2]
KR0.015 [kW/oC]
hR0.005 [kW/m2oC]
Ao45 [m2]
An405 [m2]
ho0.002 [kW/m2oC]
hn0.001 [kW/m2oC]
h1 [oC]
α
1/10368000 [s−1]
κ
1/10368000 [s−1]
θ
10.360 [oC−1]
θ
20.359 
θ
30.276 [oC−1]
σ
2
R200 [kW2]
σ
2
Z10 [kW2]
σ
2
B0.001 
σ
2
10.001
α
[oC−2]
σ
2
20.001
α

σ
2
30.001
α
[oC−2]
Table A.4: Numerical parameters
16