PreprintPDF Available

SimQuality A novel test suite for dynamic building energy simulation tools

September 2021

DOI:10.13140/RG.2.2.27833.90725

Authors:

Andreas Nicolai

Umwelt- und Ingenieurtechnik GmbH Dresden

Stephan Hirth

TUD Dresden University of Technology

Madjid Madjidi

Munich University of Applied Sciences

Preprints and early-stage research may not have been peer reviewed yet.

Based on existing test suites like the ASHRAE 140 BESTEST and EN ISO 13791, a novel test suite for inter-model comparison of building energy simulation tools has been developed. The test cases investigate physical models and implementations individually and in combined tests, hereby identifying sources of deviations between tested tools. The methodology to build test cases incrementally and evaluate changes resulting from new features added to the tests, allows a much better isolation of error sources. The test suite is particularly focused on providing reference solutions for simulation tool developers. Another novel aspect of the test suite is a set of rules that ensures publication of models and simulation input data, such that results can be reproduced and verified easily by other users. Any software certified with the test suite can be considered to fulfill minimum requirements for the anticipated use cases. The article describes the test cases and variants formulated so far, and gives exemplary result for the tested tools NANDRAD, IDA-ICE, TRNSYS, TAS, Model-ica/AixLib, THERAKLES and ETU-Simulation. Key Innovations • Novel inter-model comparison test suite for modern building energy simulation tools • Test and evaluation criteria, including interactive quality tool compliance webpage Practical Implications The availability of independently verified and documented software tests may help increase confidence in numerical simulation tools. The standardization on physical problem description and model complexity increases comparability of results from different simulation tools.

participating Editors in SimQuality

…

Figures - uploaded by Stephan Hirth

Content may be subject to copyright.

Content uploaded by Stephan Hirth

Content may be subject to copyright.

SimQuality

A novel test suite for dynamic building energy simulation tools

Andreas Nicolai1, Stephan Hirth1, Madjid Madjidi2

1TU Dresden, Dresden, Germany

2University of Applied Sciences, Munich, Germany

Abstract

Based on existing test suites like the ASHRAE 140

BESTEST and EN ISO 13791, a novel test suite for

inter-model comparison of building energy simulation

tools has been developed. The test cases investi-

gate physical models and implementations individu-

ally and in combined tests, hereby identifying sources

of deviations between tested tools. The methodology

to build test cases incrementally and evaluate changes

resulting from new features added to the tests, al-

lows a much better isolation of error sources. The

test suite is particularly focused on providing refer-

ence solutions for simulation tool developers. An-

other novel aspect of the test suite is a set of rules

that ensures publication of models and simulation in-

put data, such that results can be reproduced and

veriﬁed easily by other users. Any software certiﬁed

with the test suite can be considered to fulﬁll mini-

mum requirements for the anticipated use cases. The

article describes the test cases and variants formu-

lated so far, and gives exemplary result for the tested

tools NANDRAD, IDA-ICE, TRNSYS, TAS, Model-

ica/AixLib, THERAKLES and ETU-Simulation.

Key Innovations

•Novel inter-model comparison test suite for mod-

ern building energy simulation tools

•Test and evaluation criteria, including interac-

tive quality tool compliance webpage

Practical Implications

The availability of independently veriﬁed and docu-

mented software tests may help increase conﬁdence

in numerical simulation tools. The standardization

on physical problem description and model complex-

ity increases comparability of results from diﬀerent

simulation tools.

Introduction to simulation tool valida-

tion

Since the oil crisis in the 1970ties numerous build-

ing energy performance simulation (BES) tools have

been developed. These tools have since increased in

complexity and number of incorporated physical sub-

models. As with any sophisticated software, with in-

creasing complexity of the tools it becomes generally

more diﬃcult for engineering users to verify calcu-

lation results, or even do plausibility checks. Eﬀec-

tively, users have to trust that the model predictions

are accurate, detailed enough and computed without

errors. Tool validation can establish trust in soft-

ware models and tools and help ensure high quality

of planning tools. However, the term validation is

not trivially deﬁned. In fact, validation of simulation

models is a fairly complicated and time-consuming

task.

To make things even more diﬃcult, a number of terms

such as veriﬁcation, benchmarking, regression testing,

and others are used synonymously, and depending on

the author, describe sometimes the same or diﬀerent

things. Hence, before we look into relevant literature

and existing validation methods, we should attempt

a deﬁnition of validation, or at least describe our de-

scription of its intended outcome.

A theoretical view to tool validation

When we (engineering users as well as scientists) use

a simulation model, we generally do that to get an-

swers to questions. For the authors a very simple and

overall deﬁnition of validation might be:

“A validated simulation tool should correctly provide

the requested answers within the desired accuracy.”

The interesting terms in this statement are: requested

answers and desired accuracy and correctly.

Now, with respect to multi-zone building energy per-

formance simulations, there will be questions like:

•Do we get a summer-time overheating problem?

This might be re-phrased into: How many over-

heating hours will be in critical zones, and where

are those zones?

•What will be the net energy demand of the build-

ing? How much cooling/heating energy will be

needed?

•What will be the primary energy demand?

•What is the peak load which has to be provided

by the HVAC equipment?

When drilling down into the details of HVAC equip-

ment sizing there will be many more. And in future,

with the introduction of more and more renewable

energy sources, there might be questions like:

•How much generated renewable energy is used by

the building itself directly? How much is pushed

into the grid and how much taken out again?

•Can we shift energy demand such that it better

coincides with renewable energy supply?

•How much external carbon-based energy is uti-

lized during nominal building operation?

Depending on the actual task at hand, the require-

ment on the simulation output detail can be quite

diﬀerent. For some purposes, monthly balances may

be suﬃcient. Clearly, to answer the load-balancing

questions related to renewable energy use, this will

not be detailed enough. For many tasks, hourly bal-

ances/outputs are suﬃcient. And then there might

be some questions requiring sub-hourly results, for

example when dealing with HVAC control system de-

sign and parametrization. To summarize, the pur-

poses to run a simulation model and the expectations

towards the outcome are diverse, and so are the re-

quested answers.

Similarly, the accuracy requirements might be dif-

ferent, and depend very much on the accuracy of

available input data. For example, in early plan-

ning phases not much may be known about build-

ing fa¸cade materials or room usage/occupants pat-

terns, so it may be in vain to expect accurate hourly

temperature results. With progressing state of build-

ing planning and design, accuracy of input data in-

creases, and so should the simulation results. Again,

the level of achievable and required accuracy will be

task-speciﬁc.

And ﬁnally, the term correct is very tricky to deﬁne.

Models are always more or less detailed approxima-

tions to the reality. Clearly, diﬀerent approximations

(equation sets) will produce diﬀerent results, even

if the same input parametrization is used. If fur-

thermore input parameters diﬀer, for example when

models require diﬀerent sets of input data that need

to be approximated/estimated/converted, the devi-

ations of model outputs may further increase. To

complicate things even further, the same set of math-

ematical equations may be solved using diﬀerent al-

gorithms, especially when numerical methods are in-

volved. Hereby, additional numerical parameters are

introduced that further increase diﬀerences in calcu-

lation results. These diﬀerent (independent) sources

of deviations may, if combined in diﬀerent combina-

tions, yield a substantial number of varying results

with quite a large standard deviation. Assuming er-

rors related to numerical techniques or number preci-

sions are small, each result will be a correct solution

for each combination of model equations, input data,

calculation/numerical parameters and result interpre-

tation.

Surprisingly, the interpretation of results is rather of-

ten a problem, especially when the tools documen-

tation is not very clear about how results shall be

interpreted. A very common problem when compar-

ing simulation results is the handling of momentary

values (i.e. values at a given time instant) vs. mean

values over a given interval. Simulation models using

hourly balances often produce mean values over each

hour, whereas other simulation tools generate outputs

for speciﬁc time points. Plotting the data will yield

a characteristic shift (see Fig. 2 below)

Also, interpretation of input data can be troublesome.

In the SimQuality test procedure we ﬁrst used epw-

ﬁles to deﬁne weather data. However, it turned out

that due to ambiguous/unclear speciﬁcation of the ﬁle

standard there are diﬀerent interpretations of time

stamps, described by Crawley et al. (1999) and U.S.

Department of Energy (2021).

Currently, the epw variants 2001,1,1,1,60,... and

2001,1,1,1,0,... are both being used, for example in

the EnergyPlus weather directory (US Department of

Energy, 2021). In simulation tools using hourly bal-

ances this is not a problem, since the 5th column is

ignored and the provided hourly value is interpreted

as mean value of the hour. For modern simulation

tools, however, that use arbitrarily spaced input data,

the usual procedure is to convert time stamps to spe-

ciﬁc points in time and use linear interpolation to get

climate data at interim points. In the tools used in

the SimQuality project we found 3 diﬀerent interpre-

tations for a time stamp in format 2001,1,1,1,60,...:

(1) 01.01.2001 0:00 (begin of first hour)

(2) 01.01.2001 1:00 (end of first hour)

(3) 01.01.2001 2:00 (end of second hour)

Variant (3) was used in the Modelica library by using the

calculation formula:

seconds_of_day = <hour column>*3600 +

<minute column>*60

In that light, care should be taken when using arbi-

trary epw ﬁles with Modelica building simulation mod-

els (AixLib/Buildings/... etc.). As a consequence of such

problems, we ceased to use epw ﬁles in the description of

the test cases. They are still being provided with the test

cases for convenience, but with a very detailed description

on how they should be interpreted, and with the obliga-

tion to the tool user to verify that the tool processes the

data as intended (in a dedicated test case).

Given the large number of possible reasons for result de-

viations, it appears that the only way to deﬁne correct-

ness properties for several tools would be to deﬁne a very

narrow band of possible model equation variations and

deﬁne the input parameters and solution procedures very

well. This approach is followed to some extend by the

VDI 2078/6007 (see literature review below). However,

we have to consider the substantial diﬀerences in numer-

ical solution methods (from simple daily/hourly balances

to time-accurate integration with adaptive time steps),

present in current simulation tools. Also, model formula-

tion and model complexity varies greatly in modern simu-

lation tools. If the model formulation and required input

parameters as well as solution procedure would be ﬁxed in

a test suite as it is done in VDI 2078/6007, this eﬀectively

disqualiﬁes many equally suited simulation tools. For ex-

ample, suppose you require hourly balances as solution

procedure. Then, obtained results will be average tem-

peratures for each hour. If you are to run the same test

case with numerical methods of higher time resolution and

higher integration accuracy, you may not meet the refer-

ence results deﬁned for hourly balances. Thus, the much

more detailed and accurate simulation tool could not be

validated and test suites deﬁned as in VDI 2078/6007 with

strict speciﬁcation of solution methods and model com-

plexity are not meaningful for comparison of broad range

of diﬀerent simulation tools.

Indeed, it appears that none of the three properties out-

puts/requested answers,accuracy/level of detail or cor-

rectness can be easily deﬁned. Instead, all appears to be

problem-speciﬁc.

Hence, we need a consensus on the speciﬁc capabilities

of individual model components with their level of de-

tail being deﬁned based on typical application scenarios.

We could then classify tools as “suitable and validated

for purpose XXX”. To further reduce deviations due to

input/output data interpretation, the description of the

test cases should be very detailed. These considerations

led to the SimQuality test suite described below.

Existing test suites and standards

The topic of tool validation is by far not a new one. Nu-

merous papers/scientiﬁc works have been written on the

subject, and most tool developments started with some-

thing similar in place. We will restrict the literature dis-

cussion only to standards/building codes or otherwise oﬃ-

cial documents dealing with validation of simulation tools.

EN ISO 13791:2012

The European standard EN ISO 13791 speciﬁes the as-

sumptions, boundary conditions, equations and validation

tests for a calculation procedure, under transient hourly

conditions, of the internal temperatures (air and opera-

tive) during the warm period, of a single room without

any cooling/heating equipment in operation. In section

7 a validation procedure is given for transient BES tools

consisting of 4 sections with 4 diﬀerent test cases. In the

ﬁrst test case the focus lies on heat conduction through

opaque elements, in the second on internal long-wave radi-

ation heat exchanges, in the third the evaluation of short-

wave radiation heat transfer (calculation of shaded area

of a window due to external obstructions). Finally there

is a test case concerning the validation procedure for the

whole calculation method, where all physical eﬀects are

taken into account. This leads to numerous eﬀects in the

combined test case. It is generally not possible to deter-

mine speciﬁcally what causes a deviation of results if it is

not detectable by the ﬁrst three test cases. Furthermore

the test suite only checks single zone models. Coupled

zones and interzonal heat ﬂows/exchange is not tested or

validated at all.

EN ISO 52017-1:2018

In 2018, the EN ISO 13791 has been replaced by European

standard ISO 52017-1. In that standard only a calcula-

tion procedure for the internal temperatures, heating and

cooling in buildings is described. It contains a short sec-

tion about validation, yet with no test cases, only refering

the standard EN ISO 52016-1.

EN ISO 52016-1:2018

In this standard the most authoritative BESTEST cases

have been included.

ANSI/ASHRAE 140 - BESTEST

The aim of the Building Energy Simulation Test

(BESTEST) and ANSI/ASHRAE Standard 140 is to in-

crease conﬁdence in the use of building energy simula-

tion programs. This is achieved by creating standard-

ized and citable test procedures for validating, diagnosing,

and improving the current generation of software. The

BESTEST was originally published in the year 1995 and

has been updated periodically (Judkoﬀ (2013)). There are

in total 39 test cases organized into a basic series and an

in-depth series, with many diﬀerent physical parameters

and models to be tested incrementally. A series of diﬀer-

ent buildings are speciﬁed, from the thermally simple to

the more realistic approximation. The speciﬁed test cases

are deﬁned such that the thermal properties, geometric

proportions, and thermal responses are reasonable with

respect to actual building envelope loads. Various BES

tools already took part in the validation procedure and

provided result data (ANSI/ASHRAE, 2017).

While the BESTEST is in our opinion a great test suite

for quality checking and tool comparison. However, it suf-

fers from some of the issues that apply to the other men-

tioned suites and standards. The in-depth (195 - 320)

test cases do not test each physical model individually,

which can result in deviations in detailed combined test

cases (600 - 900). Also, only a rather small amount of dif-

ferent tools ran some of the in-depth test cases 195-220,

which makes the cross-checking for them rather diﬃcult

(Fig. 1). Further, the use of only one climatic location

(Denver), may not allow generalization of a tool’s cor-

rect calculation when used. For example, in the southern

hemisphere or in areas with very wide time zones, models

may calculate the the suns position incorrectly and thus

result in faulty radiation loads. Lastly, the reference for

compliance testing (e.g. range of annual heating loads)

is deﬁned through aggregated results (mean, min, max)

from a variety of tools. Result data is only for some test

cases compared at diﬀerent timesteps, which means that

too large loads during some part of the simulation can be

compensated by too small loads at other times.

Acceptance is deﬁned through results from tools that have

completed the test suite. While this is certainly a prac-

tical approach, this leads to the problem that if all pro-

grams have implemented a faulty/very simple model, a

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

200-195

Heating

Surface

Convection

200-195

Cooling

Surface

Convection

210-200

Cooling

Ext IR

(Int IR "off")

220-215

Cooling

Ext IR

(Int IR on)

215-200

Heating

Int IR

(Ext IR "off")

215-200

Cooling

Int IR

(Ext IR "off")

220-210

Heating

Int IR

(Ext IR "on")

220-210

Cooling

Int IR

(Ext IR "on")

Load Difference (MWh)

Figure B8-42. BESTEST IN-DEPTH

Cases 195 to 220 (Delta)

Peak Heating and Sensible Cooling

ESP/DMU BLAST/US-IT TSYS/BEL-BRE TASE/FINLAND E+/NREL

ASHRAE Standard 140-2014, Informative Annex B8, Section B8.1

Example Results for Section 5.2 -Building Thermal Envelope and Fabric Load Cases 195-960 & 600FF-950FF

Figure 1: Examplary Delta for Test Cases 195 to 220:

Is Energy Plus 9.4 (red) validated?

novel tool with a more detailed closer-to-reality model

approach may yield results outside the previously estab-

lished band of acceptance and thus, may not be validated.

To summarize, the BESTEST is currently the most so-

phisticated, cited and used suite for validating BES tools

and deﬁnitely a good way to validate most simulation

models.

VDI 6020/6007/2078

The VDI 6020 forms a validation procedure with various

test cases for BES programs. The validation is carried

out according to two diﬀerent scenarios: In scenario 1

only thermal energy simulation without technical building

equipment is validated. This part is completely described

inside the VDI 6007. In scenario 2, BES Tools are vali-

dated with regard to their technical building equipment.

These test cases are mainly described inside VDI 2078. It

contains 16 diﬀerent test cases with diﬀerent geometries.

There are several problems with the described validation

procedure. Firstly, there are three diﬀerent documents

(main document, expansions and excel documents) with

redundant and sometimes conﬂicting input data. When

setting up a test, a lot of eﬀort is needed to determine

the correct input data for each test case (Schpfer et al.

(2010)). In some cases, the documented input data is

plain wrong or at least mismatching. For example, in

Part 1 (VDI (2016b) p.49 Tab.A3) a window with an area

of 10.5 m2is described, whereas a window area of 7 m2

is also mentioned in Part 1 (VDI (2016b) p.47 Fig. 41),

in the guideline itself (VDI (2016c) p.55, Tab.C1) and on

the CD (VDI (2016a)) concerning the same test cases 01

- 07. In the same way, it is diﬃcult to identify the correct

reference results from the Excel sheets from CD1 (VDI

(2016a)) needed to verify results.

Further, the validation suite actually tests a very speciﬁc

model description and solution procedure. While this is

normally a good approach to verify diﬀerent implemen-

tations of the exact same model and parameter set, this

also represents the strongest weakness of this test norm.

The model to be tested is the speciﬁc VDI calculation

model,based on the Beuken model. This model, however,

simpliﬁes the actual building physics strongly, hereby in-

troducing in some situations rather large approximation

errors. When using a very detailed physical model to

calculate the prescribed VDI test cases, these approxi-

mation errors become apparent. Since the VDI validation

results are rather tightly set around those obtainable with

the VDI Beuken model, it is hence impossible to obtain a

match with a more detailed model. There is also an option

documented to validate a diﬀerent calculation procedure,

hereby using statistical methods. Yet the reference results

are still being computed with the simpliﬁed VDI model,

and hence more physically correct simulation results will

still not always be accepted as correct. Consequently, it

would be meaningless to require a modern, physi-

cally detailed building energy simulation model to

comply to the VDI validation norm.

We conclude from our analysis of the VDI 6020 test norm,

that this VDI guideline should indeed only be used to vali-

date implementations of the VDI speciﬁc (Beuken) model

(VDI (2015b, 2016c, 2015a)).

Empirical validation

There is, of course, also the option to validate tools

against monitoring data. However, a large number of un-

certainties and error sources may be introduced. Also,

imprecise input data may give room for ”ﬁtting” results

towards measurements, hereby masking potential mod-

elling/software errors.

The SimQuality test suite

Test suite requirements/target audience

The SimQuality test suite targets two distinct user

groups: simulation software developers and engineer-

ing users. Furthermore, the tests can be used as well-

structured exercises within lectures in building simula-

tion.

The ﬁrst group should expect very detailed test descrip-

tions (models and parameters) and results to check their

own implementation against implementation errors. Since

designing suitable test cases with a broad enough param-

eter variation can be time-consuming, this availability of

a test series will eﬀectively reduce development time and

eﬀort. For such a testing to be successful, the individ-

ual tests of the series should test individual models such,

that occurring errors can directly be attributed to a spe-

ciﬁc model/parameter combination.

The second target group, the engineering users, will be

primarily interested in the tool ratings and tool capabil-

ity summary tables. With this information they can make

informed decisions on which tool to use for which applica-

tion case. Further, the test suite can speed up the learning

curve for new simulation users, since test cases with in-

dividual modeling tasks can be downloaded, investigated

(how are the model parameters set?), and used as means

for self-checking. For this user group the webpage content

will be most useful, as well as detailed descriptions of how

tool results came together.

Scope

The scope of the SimQuality test suite (as far as the cur-

rent research project is concerned) is the building envelope

and general building physics. With increasing use of re-

newable energy sources we must ensure that the building’s

response to climate loads is represented well in a simula-

tion program. This is particularly true for the rather mas-

sive building constructions used throughout Germany and

Europe, where much of the indoor thermal comfort can

(and should) be already maintained by suitable damp-

ening of environmental conditions. The dynamic loads

obtained from such a simulation are a critical input to

any equipment model, so getting these right is our ﬁrst

priority.

Test methodology

The test suite is centered around typical application use

cases. In such an application case, for example summer

thermal comfort analysis, heating and cooling load analy-

sis, a number of diﬀerent model components play a critical

role. Depending on the actual building/application sce-

nario, the impact of the model components diﬀers. Some-

times window models are more critical, sometimes it will

be the shading, or the thermal construction models.

In order to generalise the validation status, we follow a

similar approach as used in modern numerical time inte-

gration solvers. In order to control the global integration

error, the local errors in each step are limited, thus limit-

ing also the global error occurring over a number of steps.

With respect to the SimQuality test suite we strive to

limit the error occurring in each individual model compo-

nent, thus also limiting the error occurring in any combi-

nation of said components. Hence, the test suite is com-

posed of a series of tests for individual model components,

alone.

The current test suite consists of the following tests:

TF00 weather data interpretation

TF01 solar position calculation

TF02 solar load calculation

TF03 heat transfer and storage - single-zone

TF04 heat transfer and storage - multi-zone

TF05 ventilation/inﬁltration

TF06 solar gains onto and heat transfer through

opaque constructions

TF07 solar gains through windows and distribution

in rooms

TF08 internal loads

TF09 outer shading (building geometry)

TF10 passive cooling (component activation)

TF11 complex case: summer overheating protection

calculation

TF12 complex geometry case: real building

geometry

Table 1: participating Editors in SimQuality

Tools Editors

NANDRAD Stephan Hirth, IBK TU Dresden

IDA ICE Stefan Lehr, INNIUS D

TRNSYS Julian Agudelo, HM Munich

THERAKLES Andreas Nicolai, IBK TU Dresden

ETU

Simulation Rainer Rolﬀs, ETU Hottgenroth

Modelica/AixLib Amin Nouri, RWTH Aachen

TAS Rainer Strobel, PGMM

There is also a basic test case (TF0) that checks the cor-

rect import of provided climate data ﬁles, in case that the

provided epw-ﬁles are being used/imported by a software.

This was necessary due to the aforementioned speciﬁca-

Begin of the hour

End of the hour

At midpoint of the hour

Linearly interpolated midpoints

Temperature [C]

Time [h]

166 167 168 169 170 171

Figure 2: Il lustration of the output interpretation/-

time shift problem

0.1m2.8m

0.1m 0.1m

1.8m

0.1m

Figure 3: Geometry for the test case validating dif-

ferent window models

tion problems with the epw format. To avoid such errors

we speciﬁcally describe our interpretation rule for the pro-

vided epw ﬁles in the test case.

Test procedure

Hereafter, the principle procedure is illustrated using test

case 07. In this case, simple to detailed window models

are incrementally tested and validated with increasingly

more physical model components taken into account.

(07.1) Opaque window with only heat conduction

(07.2) Simple window model with constant

SHGC-Value

(07.3) Simple window model with angle dependent

SHGC-Value

(07.4) Detailed window model with two panes

(07.5) Detailed window model with two panes and

coating (lowE)

(07.6) Detailed window model with two panes and

coating (lowE) and longwave radiation

(07.7) Detailed window model with two panes,

longwave radiation and internal distribution

Result interpretation

When analysing results it is important to use a consistent

output value interpretation. For time series data there are

two basic output types: values at a given instant in time,

and mean values over a certain time interval. The latter

is frequently used in simulation models working with bal-

ances in a ﬁxed grid and usually hourly steps. Modern

time integration engines rather use variable adaptive step

integration methods that adjust time steps based on error

estimates. While it is easy to generate hourly mean val-

ues from these detailed integration engines, hourly values

generated by balance methods needs some interpretation.

Figure 2 illustrates diﬀerent options. If the balance model

uses classic time integration formulation (e.g. explicit/im-

plicit Euler), then the states are deﬁned at begin and end

of the hourly interval. Which of the curves ”Begin of the

hour” (blue) or ”End of the hour” (green) is meant by

the tool depends on the hour indexing method (which is

hopefully documented).

Var 03.3 Air Temperature

Temperatur [C]

Time [h]

150 200 250 300

Var 04.3 Air Temperature Room A

Temperature [C]

23,5

24,5