Available via license: CC BY 4.0

Content may be subject to copyright.

A Nested Multi-Scale System Implemented in the Large-Eddy

Simulation Model PALM model system 6.0

Antti Hellsten1, Klaus Ketelsen2, Matthias Sühring3, Mikko Auvinen1, Björn Maronga3,6,

Christoph Knigge4, Fotios Barmpas5, Georgios Tsegas5, Nicolas Moussiopoulos5, and Siegfried Raasch3

1Finnish Meteorological Institute, P.O.Box 503, FI-00101, Helsinki, Finland

2Software Consultant, Beethovenstr. 29A, 12247 Berlin, Germany

3Leibniz University Hannover, Institute of Meteorology and Climatology, Herrenhäuser Strasse 2, 30419 Hannover, Germany

4Deutscher Wetterdienst, Frankfurter Straße 135, D-63067 Offenbach, Germany

5Aristotle University Thessaloniki, P.O.Box 483, GR-54124, Thessaloniki, Greece

6University of Bergen, Geophysical Institute, Postboks 7803, 5020 Bergen, Norway

Correspondence to: A. Hellsten (antti.hellsten@fmi.ﬁ)

Abstract.

Large-eddy simulation provides a physically sound approach to study complex turbulent processes within the atmospheric

boundary layer including urban boundary layer ﬂows. However, such ﬂow problems often involve a large separation of turbulent

scales, requiring a large computational domain and very high grid resolution near the surface features, leading to prohibitive

computational costs. To overcome this problem, an online LES-LES nesting scheme is implemented into the PALM model5

system 6.0. The hereby documented and evaluated nesting method is capable of supporting multiple child domains which

can be nested within their parent domain either in a parallel or recursively cascading conﬁguration. The nesting system is

evaluated by simulating ﬁrst a purely convective boundary layer ﬂow system and then three different neutrally-stratiﬁed ﬂow

scenarios with increasing order of topographic complexity. The results of the nested runs are compared with corresponding non-

nested high- and low-resolution results. The results reveal that the solution accuracy within the high-resolution nest domain10

is clearly improved as the solutions approach the non-nested high-resolution reference results. In obstacle-resolving LES,

the two-way coupling becomes problematic as anterpolation introduces a regional discrepancy within the obstacle canopy of

the parent domain. This is remedied by introducing canopy-restricted anterpolation where the operation is only performed

above the obstacle canopy. The test simulations make evident that this approach is the most suitable coupling strategy for

obstacle-resolving LES. The performed simulations testify that nesting can reduce the CPU time up to 80% compared to15

the ﬁne-resolution reference runs while the computational overhead from the nesting operations remained below 16% for the

two-way coupling approach and signiﬁcantly less for the one-way alternative.

1 Introduction

Large-Eddy Simulation (LES) has been used for basic research of atmospheric boundary layer (ABL) phenomena using ide-

alized model setups for decades. Now, it is becoming an important method in applied research on realistic, very detailed and20

complicated ﬂow systems such as urban ABL problems, e.g. (Britter and Hanna, 2003; Tseng et al., 2006; Bou-Zeid et al.,

1

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

2009; Tominaga and Stathopoulos, 2013; Giometto et al., 2016; Buccolieri and Hang, 2019). Until the recent years, there were

no ABL LES models capable of modelling detailed surface structures, such as buildings or steep complex terrain shapes in

ABL. Nowadays, it is possible to carry out LES for complex built areas (e.g., Letzel et al., 2008), but this is still limited to

relatively small areas because of the high spatial resolution requirement. Concerning urban LES, Xie and Castro (2006) have

shown that at least from 15 to 20 grid nodes are needed across street canyons to satisfactorily resolve the most important turbu-5

lent structures within the canyons. This requirement typically leads to grid spacings on the order of 1 m. However, the vertical

extent of the LES domain should scale with the ABL height, and the horizontal size should span over several ABL heights in

order to capture the ABL-scale turbulent structures (de Roode et al., 2004; Fishpool et al., 2009; Chung and McKeon, 2010;

Auvinen et al., 2020a). To adequately capture processes on the street-scale and to simultaneously capture large ABL-scale tur-

bulence, sufﬁciently large model domains at small grid sizes are required, posing high demands on the computational resources10

in terms of CPU time and memory. Moreover, the uncertainty related to the lateral boundary conditions usually decreases as

the domain is made larger.

Many numerical solution methods (e.g. ﬁnite-element and ﬁnite-volume methods) allow variable resolution so that the

resolution can be concentrated to the area of principal interest and relaxed elsewhere. However, only unstructured grid systems

allow to take full advantage of spatially variable resolution. Many general-purpose computational ﬂuid dynamics packages offer15

unstructured grid systems, but according to our experience such solvers are usually computationally decidedly less efﬁcient than

ABL-tailored LES models, such as PALM (Raasch and Schröter, 2001; Maronga et al., 2015, 2020), the Weather Research and

Forecasting Model (WRF) (Skamarock et al., 2008) with its LES option and the Dutch Atmospheric Large-Eddy Simulation

(DALES) (Heus et al., 2010) that are based on structured orthogonal grid system with constant horizontal resolution. The model

nesting approach can be exploited to further speed up ABL LES models or to allow larger domain sizes without compromizing20

the resolution in the area of primary interest.

The idea of grid nesting is to simultaneously run a series of two or more LES model domains with different spatial extents and

grid resolutions. In this implementation the outermost and coarsest-resolution LES domain (termed root domain henceforth),

which acts as a parent to its child domains, obtains its boundary conditions in a conventional manner, whereas the nested

LES domain (child) always obtains its boundary condition from its respective parent domain through interpolation. In one-25

way coupled nesting only the children obtain information from their parents. In such a coupling strategy, the instantaneous

child and parent solutions can deviate within the volume of the nest. If a stronger binding between the solutions is desired,

the child solution needs to be incorporated into the parent solution. This is achieved in two-way coupled nesting, where the

parent solutions are inﬂuenced by their children through so-called anterpolation (Clark and Farley, 1984; Clark and Hall, 1991;

Sullivan et al., 1996).30

The child-to-parent anterpolation can be implemented using, for instance, the post insertion (PI) approach (Clark and Hall,

1991) where the parent solution is replaced by the child solution within the volume occupied by both domains. In practice,

some buffer zones where anterpolation is omitted are necessary near the child boundaries to avoid growth of unphysical

perturbations near the child boundaries (Moeng et al., 2007). An example of a two-way coupled nesting implemented in the

WRF-LES model is given by Moeng et al. (2007), and later successfully applied to a stratocumulus study by Zhu et al. (2010).35

2

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

However, this WRF-LES nesting system is limited to horizontal directions, i.e. all the domains have equal height which may

lead to computational inefﬁciency. The WRF-LES nesting system can also be used in one-way coupled mode (Mirocha et al.,

2013), and this way it has been applied, e.g. to a complex-terrain study (e.g. Nunalee et al., 2014; Muñoz-Esparza et al.,

2017) and to an offshore convective boundary layer study (Muñoz-Esparza et al., 2014). More recently, Daniels et al. (2016)

introduced a vertical interpolation into the WRF model, but this method is restricted to one-way coupled nesting. Moreover,5

according to our knowledge, WRF-LES is not applicable to blunt-obstacle resolving LES required for urban turbulence studies.

In addition to WRF-LES, the numerical weather prediction model ICON features an LES mode and includes an online nesting

capability (Heinze et al., 2017). However, due to their terrain following coordinate system, neither WRF-LES nor ICON-LES

can resolve sharp obstacle structures, hence these models cannot be applied to urban studies in obstacle-resolving fashion.

Recently, Huq et al. (2019) implemented a purely vertical nesting system into PALM in which the child and parent domains10

are required to have the same horizontal extent. Although this approach is useful, e.g. when the grid resolution near the surface

needs to be reﬁned to better capture the atmosphere-surface exchange, the requirement of equal horizontal domain extensions

poses high demands on the computational resources, limiting this approach to only academic studies. This implementation is

also limited to have a single child domain only. For these reasons, we decided to develop the present, more general and fully

three-dimensional nesting system in PALM. It can also be run in a pure vertical nesting mode.15

One-way coupled obstacle-resolving LES has been applied to a built environment by Nakayama et al. (2016) and by Von-

lanthen et al. (2016, 2017). Also the present PALM implementation has already been demonstrated by MAronga et al. (2019);

Maronga et al. (2020) and applied to obstacle-resolving urban studies (Kurppa et al., 2019; Auvinen et al., 2020a; Karttunen

et al., 2020; Kurppa et al., 2020) using the one-way coupling. At current stage, we are not aware of any research on obstacle-

resolving LES employing two-way coupled nesting approach. Through our studies, we have observed that the application of20

two-way coupling in obstacle-resolving LES can become problematic. Therefore, in addition to documenting and evaluating

the newly implemented nesting method in the PALM model, this paper addresses the applicability of the two-way coupled

nesting approach in obstacle-resolving LES.

The paper is organized as follows: Sect. 2 gives a brief description of the LES mode of the PALM model system 6.0. Section 3

presents the technical, algorithmic and numerical aspects of the implemented nesting. In Sect. 4 the implemented nesting is25

evaluated for a series of test cases featuring different kinds of boundary-layer ﬂow. Finally, Sect. 5 summarizes the results and

gives and outlook of future developments.

2 The PALM model system 6.0 (LES mode)

The PALM model system (Raasch and Schröter, 2001; Maronga et al., 2015, 2020) is based on the non-hydrostatic, ﬁltered,

Navier-Stokes-equations in the Boussinesq-approximated or anelastic form. It solves the prognostic equations for the con-30

servation of momentum, mass, energy, and moisture on a staggered Cartesian Arakawa-C grid. Subgrid-scale turbulence is

parameterized using a 1.5-order closure after Deardorff (1980) in the formulation of Saiki et al. (2000). In its standard conﬁg-

uration PALM has thus seven prognostic quantities: the velocity components ui(where u1=u,u2=v,u3=w), the potential

3

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

temperature θ, speciﬁc humidity qv, a passive scalar s, and the subgrid-scale (SGS) turbulent kinetic energy e. By default,

discretization in time and space is achieved using a 3rd-order Runge-Kutta scheme after Williamson (1980) and a 5th-order

advection scheme after Wicker and Skamarock (2002). The horizontal grid spacing is always equidistant, whereas it is possible

to use variable grid spacing in the vertical direction. Often, the vertical grid spacing is set equidistant within the boundary

layer, and stretching is applied above the boundary layer to save computational time in the non-turbulent free atmosphere. At5

the lateral boundaries cyclic conditions or more advanced in- and outﬂow conditions can be employed.

Both the Boussinesq and the anelastic approximation require incompressibility of the ﬂow. To provide this feature a predictor-

corrector method is used where an equation is solved for the modiﬁed perturbation pressure after every Runge-Kutta sub-time

step (e.g. Patrinos and Kistler, 1977). The method involves the calculation of a preliminary prognostic velocity. Divergences in

the ﬂow ﬁeld are then attributed solely to the pressure term, leading to a Poisson equation for the perturbation pressure. In case10

of cyclic lateral boundaries, the Poisson equation is solved by using a direct fast Fourier transform (FFT) method. However, in

case of non-cyclic boundary conditions, an iterative multigrid scheme is used (e.g. Hackbusch, 1985).

Parallelization of PALM is achieved by using the Message Passing Interface (MPI, e.g. Gropp et al., 1999) and a two-

dimensional (horizontal) domain decomposition.

3 Nesting system15

3.1 General concept

The nesting system we have developed is based on the concept of parent and child domains. Each parent domain can enfold

multiple child domains but a child domain can, naturally, only have one parent domain. The top-level domain, also called

root domain, acts as a parent domain to child domains at the ﬁrst nesting level. The child domains at ﬁrst nesting level might

have subsequent child domains for which they then act as parent domains (cascading arrangement), see Fig. 1. Our nesting20

system allows for up to 63 nested domains plus the root domain. The implementation requires that all child domains are always

completely located inside their respective parent domain. Also, the grid spacings of a child domain naturally have to be smaller

than the grid spacings of its parent domain. The grid-spacing ratios ∆Xi/∆ximust always be integer valued although different

ratios may be used in different directions. There may be multiple child domains at the same nesting levels, but overlapping

child domains at the same nesting level are not permitted. Finally, all the nest domains have to be surface-bound so that elevated25

child domains are not allowed.

In general, the system is designed as two-way coupled nesting, in which a child domain can affect its parent domain and

vice versa. It is possible, however, to run the system in a one-way coupled mode where no feedback from the child domain is

incorporated in its parent domain. Moreover, it is possible to use the system as a pure vertical one-dimensional nesting, where

the lowest part of the model (e.g. the atmospheric surface layer where the dominant turbulent eddies are usually very small)30

can be run as a child domain with ﬁner grid spacing than its parent domain that compasses the entire boundary layer. In the

case of pure vertical nesting, cyclic boundary conditions must be set on all the lateral boundaries. Unlike the method proposed

by Huq et al. (2019), the present method allows a cascade of more than one child domain also in the pure vertical nesting cases.

4

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

parent domain

x,i

y,j

z,k

Figure 1. A schematic example of a nested conﬁguration involving both cascading and parallel child domains is shown on x, y-plane on the

left-hand side. On the right-hand side, a three-dimensional view of a nested child domain inside its parent domain is shown.

The present nesting approach is a variant of the PI method, in which the communication between each parent-child couple

is realized via interpolations (from parent to child) and anterpolations (from child to parent) after each Runge-Kutta sub-step

and just before the pressure solver. The latter then ensures that mass conservation is enforced in the anterpolated solution in

the parent domain.

3.2 Restrictions5

The current implementation poses a few restrictions for the nested setups. Moreover, the interpolation and anterpolation meth-

ods, which are discussed in the following sections, are based on certain assumptions, e.g. on the grid-line matching between

parent and child domains leading to a few more restrictions. Altogether these restrictions are:

–the child domain must always be completely inside its parent domain and there must be a margin of four parent-grid

cells between the boundaries of child and parent domains10

–parallel child domains must not overlap each other

–the domain decomposition of all child domains must be such that the sub-domain size is never smaller than the parent

grid-spacing in the respective direction

–buildings or any other topography must not reach the child domain top

–all the grid-spacing ratios must be integer valued15

5

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

–the outer boundaries of child domains must match with grid planes in its parent domain

–no grid stretching is allowed in the child domains and in root domain it is allowed only above the top boundary of the

highest child domain

3.3 Structure of the nesting algorithm

Ideally, the coupling actions, i.e data transfers between the domains, anterpolation and interpolation, would be performed after5

the pressure-correction step using the ﬁnal divergence free velocity ﬁeld on both parent and child. To achieve this in the context

of the pressure-correction method employed in PALM requires a staged arrangement of the coupling actions such that a child

ﬁrst sends data to its parent and after receiving the data the parent anterpolates and performs the pressure correction step. After

the pressure-correction step the parent sends data to the child which interpolates new boundary conditions from the received

data and performs the pressure-correction step. The purely vertical nesting method implemented in PALM by Huq et al. (2019)10

features this kind of staged structure. However, Huq et al.’s method may lead to excessive waiting times as the child has to

wait until the parent performs the pressure-correction step and vice versa. Moreover, the staged coupling approach becomes

more complicated and more inefﬁcient when a cascade of several nested domains is used. Therefore, Huq’s implementation

allows for only one child domain. The possibility to employ cascades of child domains was an initial requirement for the

present system design and therefore the staged coupling arrangement had to be abandoned. In principle, it would be possible15

to perform the pressure-correction step twice, ﬁrst time before the coupling actions for all domains and second time for all

parent domains after the coupling to make the anterpolated ﬁelds divergence free. However, this would be computationally

very expensive severely compromising the beneﬁts from the nesting. This is because the pressure solution is typically the most

time consuming part of the solution process. To avoid this extra penalty, the coupling is based on the preliminary prognostic

velocity ﬁelds upre in the present implementation. The sequence of the coupling actions is illustrated in Fig. 2. This choice20

has the consequence that the interpolated velocity boundary conditions for a child domain may violate the global mass balance

over the child domain such that

Z

S

ρe

upre ·ndS6= 0,(1)

where the tilde symbol is the interpolation operator and nis the unit inner surface normal vector of the child domain boundary

Sexcluding the bottom boundary. This mass-conservation error, though typically small, is eliminated in an integral sense by25

adding a constant velocity correction ∆upre

∆ul,pre =−nlRSρe

upre ·ndS

RSρdS(2)

to the interpolated child boundary values to exactly eliminate the global mass-balance error in Eq. (1). Here l∈ {1,1,2,2,3}

corresponding to the left (1), right (1), south (2), north (2), and top (3) boundaries. In case of purely vertical nesting mode, the

correction is applied only on the top boundary and Sspans only over it. This correction is made for all child domains right30

6

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

before the pressure-correction step. According to our tests, ∆upre is typically three or four orders of magnitude smaller than

the dominant velocity scales of the ﬂow.

Huq et al. (2019) showed results for a zero mean-wind CBL case. In this case, especially if the nest-top boundary is set

on a relatively low level, unphysical overestimation of horizontal velocity component variances easily develop if the coupling

is based on upre. Huq et al. (2019) showed that using the staged sequence of coupling actions, allowing the coupling based5

on the ﬁnal velocity ﬁeld u, mostly removes the overestimation of the horizontal-velocity variances. We have conﬁrmed this

by temporarily modifying the current implementation to adhere to Huq et al.’s staged arrangement and simulating a vertically

nested zero mean wind CBL case similar to Huq et al.’s test case.

In the present method, the overestimation of the horizontal velocity component variances can be mostly avoided by using

the integral mass-balance forcing (Eq. 2) and further by setting a narrow buffer zone below the top boundary in which the10

anterpolation is not performed. This is described in more detail in Sec. 3.5.

In addition to the velocity ﬁeld, also all other prognostic variables are coupled except the SGS turbulent kinetic energy e, as

it depends on the resolution by deﬁnition and therefore it is not straightforward to couple between parent and child domains

having different resolutions. The anterpolated values should be increased by some unknown resolution dependent factor and

the interpolated values should be reduced accordingly. estrongly follows the velocity-gradient ﬁeld and therefore it tends to15

adapt to the anterpolated velocity ﬁeld on the parent side during the next Runge-Kutta step without being anterpolated itself.

Relying on this reasoning, we omit the anterpolation of e. Moreover, we assume that the local generation of eoften dominates

its advection implying that replacing the interpolation of its child-boundary values by simple zero-gradient conditions may be

acceptable. In our numerical tests we compared the zero-gradient conditions with interpolated boundary values reduced by an

estimated resolution-difference dependent factor. The comparisons revealed no signiﬁcant differences in the results.20

Further technical implementation issues are discussed in Appendix A.

3.4 Interpolation (parent to child)

Boundary conditions for the child domain lateral and top boundaries are given using data from its parent domain. This data

is interpolated and set to the boundary grid points right behind the outer boundary of the respective child sub-domain. As

mentioned in Sec. 3.3, all prognostic variables are interpolated except the SGS turbulent kinetic energy e. It is not interpolated25

since it depends on the grid resolution by deﬁnition and therefore the parent einterpolated to the child-domain boundaries

would be inconsistent with the child grid. Instead, a simple Neumann condition (zero-gradient condition) is used on child-

domain boundaries.

It is very important that the interpolation method conserves the mass (volume) ﬂow rate through the boundaries. If the mass

conservation is violated in a two-way coupled run, a non-physical secondary circulation usually develops. Clark and Farley30

(1984) introduced a quadratic interpolation scheme that forms a reversible pair with their anterpolation scheme which we also

employ. This reversibility guarantees the mass conservation. However, as recently noted by Zhou et al. (2018), the conservation

of ﬂuxes other than the mass ﬂux is violated if both advective velocity component and advected variable are interpolated using

the Clark and Farley (1984) scheme or in fact any interpolation scheme of higher than zeroth order. They selected to use the

7

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 2. Flowchart illustrating the nesting actions in case of three domains in cascading order. In case of more than three levels of domains,

more branches similar to the current middle branch would be added. Blue boxes represent baseline PALM actions while the other colors

indicate nesting-speciﬁc actions. In one-way coupling only the actions indicated by pink color are invoked.

Clark and Farley (1984) method only for the advective velocity component and the simple zeroth-order method, which in one

dimension is

φi= ΦIfor iwithin the parent-grid cell I , (3)

for all advected variables. The child variables and indices are denoted by lowercase letters and those of parent by uppercase.

This choice satisﬁes the ﬂux conservation requirement for all variables if the grid-spacing ratio is odd valued. Conservation5

of ﬂuxes through the nest boundaries is an important condition for a nesting algorithm. According to our tests the lack of ﬂux

conservation for example for the velocity component von the left nest boundary may lead to a wrong mean-ﬂow angle in the

whole system of domains, and such an error can be remarkably large. Therefore, we design the interpolation method such a

way that the ﬂux conservation errors on the nested boundaries are minimized. In PALM, the interpolation algorithm has to

8

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

cope with complex topography. Therefore we do not select the quadratic scheme of Clark and Farley (1984) at all. Instead, we

use a zeroth-order interpolation for all variables. Our approach is to use Eq. (3) for the boundary normal velocity component

uNand all scalar variables, and another zeroth-order interpolation for the staggered velocity components uS

ias follows (in one

dimension)

uS

i=

US

Ifor grid points ico-located with a parent-grid point Iin the direction of the interpolation

1

2(US

I−1+US

I)for grid points ibetween parent-grid points I−1and I . (4)5

The reason behind this choice is explained and discussed in what follows.

In principle, the most straightforward way to satisfy the ﬂux conservation would be to directly use the ﬂux on the parent-grid

cell face on the boundary and to distribute it onto the underlying child-grid cell faces in the proportion of the cell-face areas

in the fashion of the ﬁnite volume method in which the ﬂuxes are typically stored. However, PALM is formulated as a ﬁnite

difference method and thence its architecture does not support this method. Therefore the interpolation procedure should be at10

least approximately ﬂux conservative.

Zhou et al. (2018) require separately conservation of a prognostic variable φand its resolved-scale turbulent ﬂux hun0φ0i

through the boundary, where uNis the boundary-normal velocity component, as

hφib=hΦib,(5)

huN0φ0ib=hUN0Φ0ib,(6)15

where φand uNare variables of the child grid and Φand UNare variables on the parent grid, and h·ibdenote spatial averaging

over the child domain boundary. The ﬁrst condition Eq. (5) has originally been stated by Kurihara et al. (1979). Later Clark

and Farley (1984) stated a stronger local form of this condition called reversibility condition. Reversibility of interpolation and

anterpolation means that the following holds locally, i.e. not only in the sense of spatial averaging over the whole boundary

[

e

φ(Φ) = Φ,(7)20

where the tilde is the interpolation operator and the hat is the anterpolation operator. Naturally, the reversibility guarantees

that the conservation condition Eq. (5) is fulﬁlled. Obviously the zeroth-order interpolation satisﬁes the reversibility condition

Eq. (7) in addition to Eq. (5). It is straightforward to show that if Eq. (5) holds, the ﬂux conservation condition Eq. (6) can be

also be written for the total ﬂux as

huNφib=hUNΦib.(8)25

We shall study the ﬂux conservation using this condition instead of Eq. (6).

Zhou et al. (2018) use the quadratic interpolation scheme by Clark and Farley (1984) for the boundary-normal velocity

component, i.e. the advective component. For all other variables they selected to use the simple zeroth-order interpolation

Eq. (3). This selection was made in order to satisfy the ﬂux conservation condition Eq. (6) which is equivalent to the condition

Eq. (8). This choice readily satisﬁes the ﬂux conservation for all variables which are non-staggered relative to uNand UN. It also30

9

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

satisﬁes the ﬂux conservation condition for the velocity components uSand USstaggered relative to uNand UNin the boundary

plane, but only if the grid-spacing ratio is odd-valued. The difference of the odd- and even-valued grid-spacing ratio cases is

illustrated in Fig. 3. The method by Zhou et al. (2018) is, indeed, strictly limited to odd-valued grid-spacing ratios. This is a

strong restriction and therefore we decided to allow also even valued grid-spacing ratios. In such cases, the ﬂux conservation

condition is still readily satisﬁed for the non-staggered variables, but for staggered velocity components the situation becomes5

more complicated and it is not necessary possible to satisfy it exactly. By using Eq. (4) for the staggered velocity components

the ﬂux conservation condition can be satisﬁed approximately for both odd and even grid-spacing ratios. As an example and

for the sake of clarity but without any loss of generality, we compute the spatially averaged ﬂuxes of v- and V-components

for a nested domain face and assume that the boundary-normal direction is x. The velocity components in the x-direction are

uand U. The advective uvelocity for the ﬂux is interpolated linearly to the ﬂux point of vas (uj−1+uj)/2. The chosen10

interpolation technique Eq. (4) leads to the following averaged resolved-scale advection ﬂux of vthrough the child boundary

(the SGS-ﬂuxes are assumed small and omitted here)

huvib=1

[Ry(Ny+ 1) −1]Nz"Ry

Jn

X

J=Js

Kt

X

K=Kb

UJ−1,K +UJ,K

2VJ,K + (Ry−1)

Kt

X

K=KbU0,K

2V0,K +UNy,K

2VNy+1,K #,

(9)

while on the parent grid, the averaged advection ﬂux of Vis (as expanded by Ryfor easier comparison)

hUV ib=1

RyNyNz

Jn

X

J=Js

Kt

X

K=Kb

Ry

UJ−1,K +UJ,K

2VJ,K .(10)15

Here, Jand Kare the parent-grid indices, and the child-domain boundary covers the parent-grid node range where Js≤

J≤Jnand Kb≤K≤Ktwith Ny=Jn−Js+ 1 and Nz=Kt−Kb+ 1.Ryis the integer-valued grid-spacing ratio in the

y-direction. Clearly Eq. (9) and Eq. (10) do not exactly equal each other because of the edge terms depending on e.g. V0and

VNy+1, and because the denominator of Eq. (9) slightly deviates from RyNyNz. It should be noted that these two differences

usually have opposite effects, and that huvi − hUV itends towards zero as Nybecomes large. In typical applications, the order20

of magnitude of Nyis hundreds making the ﬂux conservation error negligibly small. Nzis usually smaller than Ny, maybe even

one order of magnitude smaller, making the ﬂux conservation error of the vertical velocity component wpossibly somewhat

larger than that of the horizontal components. If Nz= 32 and the grid-spacing ratio Rz= 3 for example, huvi − hU V ican

be expected to be of the order of 1%. Moreover, a small ﬂux conservation error does not distort was easily as the horizontal

components, because wunlike the horizontal components is relatively strongly controlled by its surface boundary condition.25

In the above considerations, it is assumed that the advected parent-grid variable values on the child boundary are equal

to the values used for the parent-grid ﬂux computation. This is not the case in reality since on the parent side the advected

variable values used for calculating the advection ﬂuxes are interpolated using the 5th-order interpolation scheme by Wicker

and Skamarock (2002) while on the child boundary this is not the case. Here, it is important to understand, that the above

mentioned interpolation onto the ﬂux point in the advection scheme is a separate procedure from the interpolation from parent30

to child, and it is performed in a different phase of the time-step cycle.

10

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 3. Staggered velocity-component nodes in cases of odd (3) a) and even (4) b) grid-spacing ratios. The staggered velocity-component

nodes are shown as arrows, thick arrows are for the parent grid and thin ones for the child grid. The parent scalar grid-cell faces are drawn

with solid lines and the corresponding child-grid cells with dotted lines. Locations of the corresponding parent-grid scalar nodes are shown

as black dots. The blue color indicates the left-hand parent-grid node and red the right-hand node. The violet colored child grid nodes receive

the averaged values according to Eq. (4).

The default advection scheme in PALM is the ﬁfth-order Wicker and Skamarock (2002) scheme which employs a stencil

of seven nodes, i.e. three boundary ghost-point values would be needed behind the nest boundaries. However, in PALM, all

boundaries except cyclic boundaries are treated in a special way. Next to a boundary, the advection scheme is degraded such

that only one layer of boundary ghost nodes is actually needed. This is achieved by using the third-order Wicker and Skamarock

(2002) scheme for the second layer inside the domain and the ﬁrst-order upwind scheme for the ﬁrst layer of nodes next to the5

boundary. This has to be taken into account in designing of the interpolation procedure since the use of the ﬁrst-order upwind

scheme may increase the ﬂux-conservation error and compromise the accuracy around the nest boundaries. However, we can

reduce this additional ﬂux-conservation error by using the following ’trick’. Instead of substituting the original parent-grid

values in the interpolation formulae (3) and (4), we replace them by values interpolated onto the ﬂux points using a scheme

of higher than ﬁrst order and substitute these values instead. As a result, the formally ﬁrst-order upwind advection scheme10

becomes the selected higher order scheme if the local ﬂow direction is into the domain. The main purpose of this ’trick’ is to

reduce the ﬂux-conservation error. Ideally the ﬁfth order scheme would be used for the ﬂux-point interpolation. However, in

practice it is not possible to use any interpolation scheme using more than one grid point behind child-domain boundaries. The

reason for this is that the child has no information about the parent domain topography outside the ﬁrst parent-grid layer behind

a child-domain boundary. An interpolation stencil reaching further away than this could penetrate a vertical wall leading to15

11

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

erroneous interpolation. Therefore, the only alternative to the 1st-order upwind interpolation is to use the simple average of the

parent-grid values both sides of the child-domain boundary. This leads to the 2nd-order central advection scheme. Obviously

it is different from the 5th-order scheme, but we argue that the difference between the ﬂuxes computed using the 5th-order

and 2nd-order schemes is usually smaller than the difference between the ﬂuxes computed using the 5th-order and 1st-order

schemes. Our numerical tests support this argument. On the top boundary, there is no topography and hence we can use wider5

interpolation stencil there. We ended up using the 3rd-order Wicker and Skamarock (2002) scheme because in our numerical

tests it yielded similar results as the more complicated more communication-intensive 5th-order scheme. It should also be

noted, that this trick improves the accuracy only on those boundary regions where the ﬂux is into the child domain.

According to our experience, the conservation properties of an interpolation method are more important than its local ac-

curacy. Increasing the interpolation accuracy on the child boundary planes does not yield any evident beneﬁt as the solution10

requires a development distance (i.e. a border zone) as it adapts to the changed resolution within the child domain. Therefore

the zeroth-order method has turned out to be fully sufﬁcient and remains the only interpolation method implemented in PALM.

However, we have considered also an alternative interpolation approach for the advected variable based on tri-linear interpola-

tion with a speciﬁc reversibility correction. Although it is not implemented a short discussion is provided in Appendix B.

3.5 Anterpolation (child to parent)15

Anterpolation is used to feed the child domain solution back to its parent domain. Generally, anterpolation consists of ﬁltering

the ﬁne-grid child solution φi,j,k and mapping it to the parent domain grid. We select to employ the anterpolation scheme

proposed by Clark and Farley (1984), which consists of simple averaging over one parent-domain grid volume around the

parent grid node of the variable in question corresponding to top-hat ﬁltering, viz.

b

φI,J,K =1

NI,J,K

i2(I)

X

i1(I)

j2(J)

X

j1(J)

k2(K)

X

k1(K)

φi,j,k .(11)20

The original parent solution ΦI,J,K is replaced by the anterpolated solution in the domain of overlap. Here, i,j,k and I, J, K

are the child- and parent-grid indices, respectively, and the hat is the anterpolation operator. The summation index limits, i.e.

the span of the anterpolation cell i1(I),i2(I),j1(J),j2(J),k1(K)and k2(K)are pre-computed during the initialization and

they depend on the grid conﬁguration and the variable in question, i.e. the staggered velocity components have different index

limits than the grid-cell centered scalars. Note that for the velocity components, the anterpolation volume is reduced to the grid25

cell face on which the velocity component is deﬁned. This means that the upper index limit in the direction of the velocity

component is reduced to the lower one, for instance i2=i1for u, because the coordinates of the velocity component node in

the respective direction in the parent and the child readily coincide, thus there is no need for anterpolation in this direction.

NI,J,K is the number of child domain values used for anterpolation at a given parent-grid location, and is pre-computed during

the initialization as30

NI,J,K = [i2(I)−i1(I) + 1][j2(J)−j1(J) + 1][k2(K)−k1(K) + 1] .(12)

12

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Note that due to the staggered grid, four sets of the index limits and NI,J,K are pre-computed and stored: one for each velocity

component and one for all scalars. Generally, the anterpolation cells can be spanned in more than one way. We deﬁne the

anterpolation cells similarly to Clark and Farley (1984). For scalar variables (non-staggered variables) the anterpolation cell

spans Xi±∆Xi/2where Xi(i= 1,2,3) are the coordinates of the scalar node in the parent grid. For the velocity compo-

nents (staggered variables), for example for u, the anterpolation cell spans X1, X2±∆X2/2, X3±∆X3/2, where Xiare the5

coordinates of the staggered u-node in the parent grid.

Buffer zones where the anterpolation is omitted are applied next to the child-domain boundaries except the bottom boundary.

The main purpose of the buffer zones is to avoid an unstable feedback loop between the anterpolation and interpolation. The

default width of these buffer zones is two prognostic grid nodes. The user may choose a different value for the buffer width,

but the minimum allowed width is one parent-grid spacing. This is because the layer of nodes nearest to the child boundary10

is directly used in the interpolation, and using an anterpolated value for interpolation leads to a strongly unstable behaviour.

The buffer zones are comparable to the relaxation zones applied in the nesting system of the WRF-LES model (Moeng et al.,

2007). In the WRF-LES nesting system the anterpolation is under-relaxed within these zones such that the under-relaxation

coefﬁcient varies linearly across the relaxation zones which are ﬁve grid spacings wide. As mentioned in Sec. 3.3 the buffer

zone below the top boundary also reduces the overestimation of the horizontal velocity variances observed in zero mean-wind15

CBL tests in purely vertical nesting mode. According to these tests in purely vertical nesting mode, simulation results are not

particularly sensitive to the extent of the vertical downward shift of the upper edge of the anterpolation domain.

Canopy-restricted anterpolation

The anterpolation algorithm is implemented in the PALM model with a feature that enables its application in a spatially

selective manner such that the operation is only performed within the computational domain that is above a user-deﬁned20

vertical threshold. This practice is discovered to resolve complications that arise when two-way coupled nesting is applied in

obstacle-resolved LES simulations where the anterpolated solution within the obstacle canopy introduces discrepancies in the

coarser parent solution. Thus we label this approach canopy-restricted (CR) anterpolation and the coupling is referred to as

two-way CR, for short. The necessity of this anterpolation strategy is motivated and its effectiveness demonstrated in Sec. 4.2.3

where nesting is applied to obstacle-resolved LES test case.25

4 Numerical experiments

In order to evaluate the nesting strategy, to show its beneﬁts and point out its limits, we performed a series of nested model sim-

ulations for different grid-spacing ratios and respective non-nested reference simulations for different atmospheric situations.

For this purpose, we simulated a homogeneously-heated ﬂat-terrain convective boundary layer as well as a purely shear-driven

ﬂat-terrain boundary layer. Further, to investigate the performance of the grid nesting in more complex situations where non-30

ﬂat topography is present, we performed simulations for a neutrally-stratiﬁed ﬂow over a smooth three-dimensional hill and

will compare the results against wind-tunnel data. Second, we simulated a neutrally-stratiﬁed urban boundary-layer ﬂow over

13

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

a regular staggered arrangement of building cubes, and will compare the nested simulation results to corresponding non-nested

ﬁne- and coarse-grid simulation results. Finally, we will demonstrate the applicability of a two-stage nesting for the same ﬂow

over the cube array. Details concerning the different simulation setups are given in their respective sections. Please note, for

the sake of simplicity velocity components will hereafter be addressed by lower case variable names only, no matter if it refers

to the ﬂow in the parent or the child domain.5

4.1 Convective boundary layer

The nesting method is ﬁrst evaluated for a pure convective boundary layer (CBL) with zero mean wind. We set up one child

domain that is centered within the parent domain. For the root domain, cyclic lateral boundary conditions were set. A ho-

mogeneous and time-constant surface sensible heat ﬂux of 0.1Kms−1was prescribed. The simulation was initialized with a

potential temperature proﬁle that increases linearly with height at a lapse-rate of 0.3 K/100 m. The root-model domain size is10

10.2 km ×10.2 km ×3.0 km in the x-, y- and z-directions, respectively, with an isotropic grid spacing of 20 m. The top of the

child domain is set to be within the middle part of the CBL, and the domain size is 2.5 km×2.5 km×0.48 km in the x-, y- and

z-directions, respectively, with an isotropic grid spacing of 10 m, resulting in a grid-spacing ratio of 2. In order to examine how

turbulence statistics behave for different grid-spacing ratios between parent and child in the CBL, we additionally run nested

simulations with grid-spacing ratio of 3 and 4 by increasing the isotropic grid spacing in the parent domain to 30 m and 40 m,15

respectively. Non-nested coarse- and ﬁne-grid reference simulations were carried out corresponding to the nested simulations

with different grid-spacing ratios. The simulated time was 4 hours for all convective cases. Data analysis started after 2 hours

of simulated time when model spin-up effects are not present any more and the simulations reached steady-state conditions. In

order to perform a spectral analysis of time-series data, the time step was held constant at 1.0 s in all convective simulations

during the data analysis period.20

Figure 4a) shows an instantaneous horizontal cross-section of the w-component at a height of 40 m for the parent and

child (overlaid) domains for the grid-spacing ratio 2. A hexagonal pattern of convective cells with strong updrafts and weaker

downdrafts is visible, as it can be typically observed in LES. The transition between parent and child appears smooth and the

ﬂow structures are continuous in terms of shape and amplitude, while within the inner part of the child domain more ﬁne-scale

structures can be observed with slightly stronger up- and downdrafts, as also reported by Moeng et al. (2007). Furthermore,25

Fig. 4b), showing an instantaneous vertical cross-section for the w-component, also depicts how the up- and downdrafts are

consistently maintained across the child boundary without any obvious impact on the turbulent structures.

Figure 5 shows horizontally- and time-averaged vertical proﬁles of potential temperature θ, vertical turbulent heat ﬂux

hw0θ0i, variances of horizontal and vertical velocity components, as well as the skewness of the vertical velocity component

w, being one of the most grid sensitive quantities (Sullivan and Patton, 2011). The proﬁles of hθiindicate a well mixed CBL.30

With increasing grid-spacing ratio the corresponding parent and coarse grid simulations deviate from the ﬁne grid reference,

particularly near the surface and within the inversion layer, while the child results adhere well with the non-nested ﬁne-reference

simulation, indicating that the proﬁles of hθiin the child domains are rather independent of the parent grid for the employed

grid spacings.

14

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 4. Instantaneous horizontal a) and vertical b) cross section of wafter 4 h of simulated time for the grid-spacing ratio case 2. The

horizontal cross-section is give at a height of 40 m. The black box indicates the lateral and top boundaries of the child domain. The white

line indicates the y-position of the vertical cross section of wshown in (b). The vertical axis in (b) is normalized with the horizontal mean

boundary-layer depth zi. Note that only part of the parent domain is shown for the sake of visibility.

15

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 5. 30-min time and horizontally averaged proﬁles of a) hθi, b) hw0θ0i, c) a close-up view of hθiand hw0θ0i, d) variance of the

horizontal velocity components, e) variance of the vertical velocity component, and f) skewness of the vertical velocity component after

4 hours of simulated time. Please note the second upper abscissa in c). Proﬁles are shown for the grid-spacing ratio of 2, 3, and 4 for the

respective child domains, indicated by the respective numbers. The corresponding proﬁles from the coarse-grid reference simulations for the

20 m, 30 m, and 40 m grid spacing are indicated the same. For the sake of clarity, the resulting proﬁles for the parent domain are only shown

for the grid-spacing ratio of 2. Squared brackets indicate a horizontal domain average.

The heat ﬂux proﬁles in the child and parent simulations decrease linearly with height within the CBL and are in good

agreement with the ﬁne reference simulation. For the parent simulation we note the near-surface kink in the heat ﬂux (see

Fig. 5c for a close-up view). Moeng et al. (2007) observed a similar kink in the heat ﬂux and attributed it to inaccuracies

in the statistical evaluation of the heat ﬂux, more precisely, to errors that arise from interpolation from a mass- to a height-

coordinate system. However, to evaluate ﬂuxes PALM does not apply any interpolations but uses directly the resolved- and5

subgrid-scale ﬂuxes as calculated in the advection scheme and the subgrid model, respectively, so that interpolation errors

cannot explain the kink in this case. Instead, we attribute the kink in the parent domain to the anterpolation from the ﬁne child

16

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

solution. In simulations with different vertical grid spacing, the vertical gradients of hθiwithin the unstable near-surface layer

are differently resolved, resulting in slightly different near-surface temperatures, as it can, e.g., be observed between the ﬁne

and coarse reference simulations in Fig. 5a. This indicates that the parent simulation will yield slightly different hθi-proﬁles

than the child simulation. After the anterpolation is performed, the parent solution is replaced by the underlying child solution,

where the near-surface vertical gradients of hθiin the parent domain partly deviate from the ones the model would create5

without feedback from the child domain, i.e. the near-surface hθi-proﬁle in the parent is not in equilibrium with the applied

surface boundary condition any more. In the following time step the parent model tries to re-adjust the post-inserted hθito the

vertical gradients as being present without feedback from the child, altering the heating rates and thus the near surface vertical

gradients of the heat ﬂux, which in turn becomes visible as near-surface kink. In fact, we veriﬁed this hypothesis in a test case

by using identical vertical grid spacing in parent and child. In this case, no kink in the vertical heat ﬂux was visible any more10

(not shown).

The variances of the horizontal and vertical velocity components, as well the skewness of the vertical component, depend

strongly on the grid spacing as the coarse- and ﬁne-grid reference simulations show, where the variances (skewness) become

smaller (larger) for increasing grid spacing. The parent simulation agrees well with the coarse-resolution simulation, indicating

that the anterpolation changes the parent ﬂow ﬁeld only marginally. The variances and skewness in the child simulations agree15

with the ﬁne-reference proﬁles, except for the upper regions of the child domain where the variances are slightly overestimated.

The child proﬁles are almost independent of grid-spacing ratio, and are close to the reference simulation proﬁle. This indicates

that the child solutions are almost independent on the chosen grid-spacing ratio in the studied cases.

Although there can be no mean horizontal advection in the zero-mean wind CBLs, spatially and temporally local horizontal

advection always takes place and therefore ﬂow structures are advected locally from parent to child (and vice versa). Therefore,20

advected ﬂow structures may need a certain fetch to adjust to the changed grid spacing. In order to get an idea of how much

distance apart from the lateral child boundaries is required to observe similar turbulence properties as in a non-nested ﬁne

resolution reference simulation, we performed a spectral analysis. Therefore, we sampled time series of TKE and θat different

locations within the child domain and calculated frequency spectra from the sampled time series. Subsequently, we averaged

spectra over all sampling locations with the same distance from the lateral child boundaries. It should be noted that transforming25

frequency spectra into wave number spectra using Taylor’s hypothesis in order to directly link spectral information and grid

spacing is not strictly correct in this case where we have no background wind; nevertheless we will assume that frequency

and wave number space are connected, i.e. large frequencies belong to small spatial scales and vice versa. Fig. 6 shows the

resulting frequency spectra, as well as corresponding spectra from ﬁne- and coarse-grid reference simulation. As expected, the

coarse-resolution spectra exhibit less spectral energy at larger frequencies, compared to the child- and ﬁne-resolution spectra.30

This is due to the larger ﬁlter length assumed for the subgrid model, removing more energy at larger spatial scales and thus also

affect smaller frequencies. The child-spectra agree well with the ﬁne-reference spectra, especially for the grid-spacing ratio

case 2 where even locations close to the lateral boundaries show good agreement with the reference. We attribute this to the

nature of the CBL, where turbulence is mostly produced locally by buoyancy and horizontal advection is almost negligible,

so that turbulence is almost not affected by any transport from the boundaries. Also for the grid-spacing ratios of 3 and 435

17

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

the differences are generally small, but locations close to the lateral boundaries show slightly smaller spectral densities at

larger frequencies, indicating that for larger grid-spacing ratios the ﬂow needs a fetch of a few tens of meters in a purely

buoyancy-driven boundary layer to adjust to the ﬁner grid spacing.

Even though the child simulations yield turbulence proﬁles, spectra and instantaneous ﬂow patterns similar to the ﬁne-grid

reference simulation for a pure-buoyancy driven ﬂow, the nested simulation nevertheless creates side effects on the ﬂow which5

appear as a secondary circulation (SC). This SC is not caused by a violation of mass conservation that has been discussed

in Section 3.4. Figure 7 shows the 5-hour time-averaged w-component at the middle part of the CBL in a homogeneously-

heated nested simulation. In order to compute the 5-hour time average, we continued the simulation with grid-spacing ratio

2 for further 3 hours. Within the region of the nested child domain a mean updraft can be observed, which is in the range of

0.4−0.9ms−1and extends throughout the entire depth of the CBL (not shown). At the child domain boundaries and outside10

the child-domain region the ﬂow subsides in average, and horizontally directed branches at the upper and lower parts of the

CBL occur, giving the overall picture of a SC. The strength of this SC, indicated by the amplitude of the mean updraft, is in the

order of the strength of SCs observed in previous simulations over idealized stripe-like surface heterogeneities (Sühring et al.,

2014) and even exceeds the strength of SCs observed in simulations over realistic surface forms (Maronga and Raasch, 2013).

SCs develop above surface heterogeneities mainly due to differential surface heating of the air, resulting in mean updrafts15

and downdrafts over the stronger- and less-heated patches, respectively. However, since we prescribe the same surface sensible

heat ﬂux in the parent as well as in the child simulation, differential surface heating cannot be the reason of the SC in the

nested simulation. Moeng et al. (2007) observed a temperature bias in their child domain that led to mean vertical motion to

compensate the temperature bias. They observed temperature biases that go either way, i.e. a too cold or a too warm child

domain, which they attributed to a nested child domain of too small horizontal extent. If only a few up- or downdrafts are20

resolved in the child domain, the vertical transport is dominated by these up- or downdrafts and thus a warmer or cooler

CBL can be quickly produced in the child domain, respectively. They showed that for larger horizontal child domain size the

temperature bias and thus the associated vertical motion vanished. However, they only considered instantaneous differences

between parent and child, meaning that the temperature bias is a result of insufﬁcient sampling of the large up- and downdrafts

rather than an inherent feature of the nesting which can only been observed after time-averaging. In our case the SC becomes25

visible only after considerable time-averaging. The updraft branch of the secondary circulations is always located within the

child domain also for larger child domain extensions. We hypothesize that this SC is triggered by a slightly different divergence

of the vertical heat ﬂux between the region occupied by the child domain and the remaining parent domain due to different grid

spacing. It might be impossible to eliminate, because higher resolution better represents the turbulent mixing, so differences

between the parent and the child solutions are to be expected in general.30

Even though this inherent artiﬁcially-induced SC only appears when the ﬂow is averaged over a longer time under quasi-

stationary conditions (no daily cycle, no change in the mean wind, etc.), nested simulation results should be interpreted carefully

in terms of SCs. In particular, since the strength of the artiﬁcial SC is in the order of ’real-world’ circulations over heterogeneous

terrain, these may superimpose each other, altering the pattern of the vertical transport of sensible and latent heat. Although

18

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 6. Frequency spectra of the TKE (left column) and θvariances (right column) at different distances from the lateral child boundaries

for the grid-spacing ratio of 2 (a,b), 3 (c,d), and 4 (e,f). Furthermore, spectra for ﬁne- and corresponding coarse-grid references simulations

are displayed. TKE and θwere sampled at z= 120m.

19

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

we did not succeed to proof our hypothesis, we encourage other researchers to look for the existence of such SCs in any nested

models by analyzing the time averaged results.

Figure 7. Horizontal cross-section of 5-h time-averaged vertical velocity at z= 400m in a nested simulation with grid ratio of 2. The black

box indicates the location of the child domain.

20

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

4.2 Neutrally stratiﬁed boundary layer tests

Initialization and inﬂow conditions

As further test cases, we set up boundary layer ﬂow simulations with increasing order of complexity. First, to evaluate the

performance of grid nesting in shear-driven boundary-layer ﬂows, we simulated a ﬂow over a homogeneous ﬂat surface in

order to compare ﬁrst and second order moments from a nested simulation against reference simulations. In a second step, we5

simulated a ﬂow over a smooth three-dimensional hill for comparison of nested simulation results against wind-tunnel data.

Finally, in order to illustrate the advantages of the grid nesting in more complex setups, we simulated a ﬂow over a staggered

arrangement of cubes mounted on a ﬂat surface.

The parent domain size for all neutrally-stratiﬁed simulations was Lx×Ly×Lz= 5.1×1.5×0.32 km3in the x−,y−,

and z-directions, respectively. In all neutral simulations we prescribed a homogeneous roughness length of z0= 0.01 m. At the10

top boundary we applied a free-slip condition for the horizontal wind components and zero vertical motion. At the spanwise

lateral boundaries (north and south boundary) we applied cyclic conditions. At the western lateral boundary (hereafter referred

to as inﬂow boundary) we prescribed mean inﬂow proﬁles for the uand vcomponent, obtained from a cyclic precursor run.

Two different precursor simulations were employed for the subsequent test cases. The one used for the ﬂat surface and for the

smooth hill featured a geostrophic wind of ug= 4.8 ms−1and vg=−1.3 ms−1at a latitude of 55 degrees, adjusted such that15

the surface-layer mean ﬂow became parallel with the x-axis. This precursor simulation ran for 36 hours to reach a stationary

state. The second precursor simulation, used for the cuboid case, was driven by a ﬁxed pressure gradient angled to result in a

mean ﬂow of u= 10 ms−1at z=Lzwith a 3 degree angle from the x-axis.

In order to obtain a turbulent inﬂow, we applied a turbulence recycling method according to Kataoka and Mizuno (2002),

where the inﬂow mean vertical proﬁles of uand vare superimposed by turbulent ﬂuctuations sampled at a recycling plane,20

which is placed at xrc = 1.5 km downstream the inﬂow boundary. The recycling plane is placed sufﬁciently far apart from the

inﬂow boundary to allow for statistically-independent turbulence, but also sufﬁciently far apart from the location of the child

domain to avoid any feedback between the grid nesting and the inﬂow conditions. For further details on the implementation of

the turbulence recycling method see Maronga et al. (2015).

Further, in order to avoid persistent streaks in the u-component, which may develop in neutrally-stratiﬁed ﬂows and will25

be recurrently recycled in case of vanishing v-component, we shifted the recycled turbulent signals along the y-direction at

the inﬂow boundary, following Munters et al. (2016). At the eastern outﬂow boundary we set a radiation boundary condition

(Miller and Thorpe, 1981). The root domain was initialized with three-dimensional data recursively copied from the precursor

run, while the child domain was initialized with data obtained from the parent. We used an isotropic grid spacing of 4 m and

2 m within the root and the nested child domain, respectively. The cuboid case also encompasses a third domain with 1 m30

resolution (two-stage nesting).

In order to evaluate the effect of the nesting, we performed additional non-nested reference simulations with 4 m and 2 m

grid spacing. The simulated time of the neutrally stratiﬁed simulations ranged from 4 to 7 hours. Data analysis started after 2

hours of simulated time. When spectral analysis was performed, the time step was held constant at 1.0 s for that simulation.

21

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

4.2.1 Neutrally stratiﬁed boundary-layer ﬂow over ﬂat terrain

Figure 8 shows an instantaneous horizontal cross-section of the u-component for the nested simulation. As typical for a neu-

trally stratiﬁed boundary layer, elongated streak-like structures can be observed (Hutchins and Marusic, 2007; Hutchins et al.,

2012). These elongated structures preserve their size and amplitude when entering the child domain from the left and exiting

to the right.5

Figure 8. Instantaneous horizontal cross-section of the u-component in ms−1at z= 40m. The black box indicates the location of the child

domain. The white solid lines indicate the x-locations where the proﬁles shown in Fig. 10 are averaged over the y-direction.

Figure 9 shows horizontal proﬁles of the time- and y-averaged friction velocity u∗within the child domain and the corre-

sponding coarse- and ﬁne-grid reference cases. In the coarse and ﬁne reference cases u∗is constant along the x-axis indicating

that the ﬂow is in equilibrium with the surface friction. In the coarse-grid simulations u∗shows slightly higher values compared

to the ﬁne-grid reference simulation, even though the prescribed surface roughness is identical in all simulations. This suggests

that the ﬂow in the coarse-grid simulations sees a slightly rougher surface, which we attribute to the less accurate representa-10

tion of the vertical near-surface gradients of the wind proﬁle compared to the ﬁne-grid simulation. When the ﬂow enters the

child domain, the coarse-grid inﬂow wind proﬁle is not in equilibrium with the surface friction any more and the near-surface

ﬂow decelerates, indicated by the higher values of u∗near the child inﬂow boundaries. With increasing distance to the inﬂow

boundary, u∗rapidly decreases and reaches a minimum with lower values compared to the reference cases, until it increases

again reaching a secondary maximum and then asymptotically approaches a constant value, which is similar to the value of the15

ﬁne-grid reference case in the grid-ratio case 2 and 3. However, at least for the given model domain size, u∗does not approach

the ﬁne-grid solution in grid-ratio case 4 but still exhibits higher values. This kind of spatial oscillation of u∗, which indicates

an alternating deceleration and acceleration of the near-surface ﬂow along the x-direction, shows that the surface-momentum

exchange in the child domain needs a sufﬁciently large development length. For grid-ratio case 2 the required fetch length is

22

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

of at least 1km to adjust to the ﬁne-grid resolution. With increasing grid ratio the amplitude of the spatial oscillation increases

and the fetch length becomes longer, and, as in grid-ratio case 4 even exceeds the model domain size.

Figure 9 shows that u∗gradually adjusts to the ﬁne-reference value, at least for grid ratio case 2 and 3. This is in contrast

to Moeng et al. (2007), who revealed a friction velocity bias between parent and child in their neutrally-stratiﬁed simulation

when employing grid-size dependent SGS model as it is also used in this study.5

Figure 10 shows time- and y-averaged proﬁles of the horizontal wind speed within the child domain for different grid ratios,

taken at different distances downstream of the inﬂow boundary, indicated by the white solid lines in Fig. 8. At a distance

of 100 m the proﬁles in the child domains agree well with the ﬁne-reference proﬁle within the lowest 10 m. Even though

the surface-momentum exchange is still not in equilibrium at that position (see Fig. 9), one could already conclude from the

near-surface wind proﬁles that the ﬂow has already been adapted to the ﬁner grid resolution. However, further above, the wind10

proﬁles of the child model still deviate from the ﬁne-reference solution and are closer to the coarse reference proﬁles. This

is especially obvious for the grid-spacing ratio case 4, where the wind proﬁle shows a discontinuity at a height of z= 20 m.

With increasing distance from the child inﬂow boundary, the child-proﬁles gradually adjust to the ﬁne-reference simulation,

while at a distance of 2000 m the child proﬁles agree with the ﬁne-reference solution, except for the grid-ratio case 4 which

still deviates from the ﬁne-reference solution.15

In order to further analyze the ﬂow adjustment within the child domain, we computed resolved-scale turbulent kinetic energy

(TKE) spectra at different distances from the child inﬂow boundary. The spectra were calculated from time-series of the three

velocity components that were sampled at different locations within the domain. The ﬁnal spectra were then obtain by averaging

individual spectra over all locations with identical distance to the inﬂow boundary, assuming that the ﬂow is parallel to the x-

axis. Figure 11 shows TKE spectra obtained from the child domain and for the corresponding reference simulations. At low20

frequencies (large wave numbers), the spectra look quite similar and no obvious differences to the ﬁne- and coarse-reference

spectra can be observed, indicating the grid nesting does not induce any larger-scale oscillations which propagate through the

model domain. At higher frequencies, however, especially the near-inﬂow boundary child spectra differ to the ﬁne-reference

spectra and resemble more the corresponding coarse reference spectra. With increasing distance from the inﬂow boundary, the

spectral properties gradually adjusts to those of the ﬁne-reference case, while at a fetch of 500–1000 m almost no differences25

can be observed any more at that height level.

In contrast to a buoyancy-driven boundary layer, the ﬂow in a purely shear-driven boundary layer requires a sufﬁciently

large development distance to adjust to the ﬁner grid resolution in terms of spectrally similar conclusion. However, a purely

shear-driven ﬂow over a ﬂat homogeneous surface can certainly be considered as an extreme case in terms of ﬂow adjustment,

as the vertical turbulent exchange, which is primarily driven by surface-roughness induced shear, is rather low compared to less30

idealized ﬂows over non-ﬂat terrain or with obstacles included. Hence, we expect that the required fetch length may decrease

for rougher surfaces and more complex surface geometries.

23

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 9. Two-hour time- and y-averaged horizontal proﬁles of the friction velocity for the grid-spacing ratio of 2, 3, and 4, as well as the

corresponding coarse- and ﬁne-grid reference simulations.

Figure 10. Two-hour time- and y-averaged proﬁles of the horizontal wind speed for the grid-spacing ratio of 2, 3, and 4, taken at different

distances downstream of the inﬂow boundary, indicated by the white solid lines in Fig. 8. Also, corresponding time- and y-averaged proﬁles

from the ﬁne and coarse reference simulations taken at the same locations are shown.

24

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 11. Frequency spectra of the resolved-scale TKE taken at different sampling locations downstream of the inﬂow boundary for the

neutrally-stratiﬁed boundary layer at z= 40m, for grid-spacing ratio a) 2, b) 3, and c) 4. TKE-spectra for the ﬁne- and the corresponding

coarse-grid reference simulations are also shown.

25

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

4.2.2 Neutrally stratiﬁed boundary-layer ﬂow over a smooth three-dimensional hill

The hill case was setup to compare ﬂow statistics against wind-tunnel observations conducted by Ishihara et al. (1999), who

sampled data at different up- and downstream locations along the centre hill axis. The terrain height of the smooth three-

dimensional hill is given by

z(x,y) = Hcos2 πp(x−x0)2+ (y−y0)2

2l!,(13)5

with the hill height H= 40m and a hill radius l= 100 m, while xand yindicated the location on the discrete grid and x0and

y0the location of the hill top. Please note, with respect to the wind-tunnel experiment, we up-scaled the hill dimension by a

factor of 1000.

Figure 12 shows the mean ﬂow ﬁeld along the centerline of the three-dimensional hill for the nested as well as ﬁne and coarse

reference simulation. Upwind of the hill the mean ﬂow in the nested simulation agrees with the one in the ﬁne and coarse10

reference simulation. Also in the lee of the hill the ﬂow ﬁeld in the nested and ﬁne reference simulation agree, both showing

a re-circulation that extends up to about 3.75Hdownstream of the hill top, while in the coarse reference simulation the re-

circulation extends farther downstream up to about 4.1H. Figure 13 and 14 show the corresponding standard deviations of the

u- and w-component sampled at different locations along the centerline of the hill. Upwind of the hill the standard deviations

of the u- and w-component agree with the observations. However, leeward of the hill at 1.25H, the LES underestimates15

the standard deviation of the u- and w-component, which is most pronounced in the coarse reference simulation. Further

downstream, the coarse reference run still slightly underestimates the observed standard deviations, while the nested and ﬁne-

reference simulation slightly overestimate the standard deviations, which is in agreement with results from the EPFL-LES

model presented in Diebold et al. (2013) who employed a similar grid resolution of the hill. The standard deviations from

the nested and ﬁne reference simulation agree, showing only marginal differences among each other. Considering that the20

hilltop is placed only about 300 m apart of the child domain inﬂow boundary, this indicates that in more complex setups where

topography is present the adjustment fetch can become signiﬁcantly smaller compared to purely ﬂat terrain as discussed in

Sect. 4.2.1.

26

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

a

b

c

Figure 12. 2-hour time-averaged vertical cross-section of the u-component (colored contours) as well as the mean ﬂow ﬁeld (vector arrows)

displayed along the centerline of the three-dimensional hill for a) the nested child domain, b) the 2m reference simulation, and c) the 4m

reference simulation. Vector arrows as well as the u-component are normalized with the reference wind speed taken at z=Hupwind of the

hill. The ordinate and the abscissa are scaled with the hill height H. Note, the abscissa is centered at the hill top. The black vertical lines

indicate the positions of the proﬁles displayed in Fig. 13 and 14.

27

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

a

b

c

Figure 13. Two-hour time-averaged vertical proﬁles of the standard deviation of the u-component for the LES and well as the observed wind

tunnel ﬂow, for a) the nested child domain, b) the 2m reference simulation, and c) the 4m reference simulation. The ordinate is scaled with

the hill height H. The standard deviation is normalized with the reference wind speed taken at z=Hupwind of the hill. The black dashed

horizontal lines indicate the discrete height of the surface at the sampling location.

28

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

a

b

c

Figure 14. Two-hour time-averaged vertical proﬁles of the standard deviation of the w-component for the LES and well as the observed wind

tunnel ﬂow, for a) the nested child domain, b) the 2m reference simulation, and c) the 4m reference simulation. The ordinate is scaled with

the hill height H. The standard deviation is normalized with the reference wind speed taken at z=Hupwind of the hill. The black dashed

horizontal lines indicate the discrete height of the surface at the sampling location.

29

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

4.2.3 Neutrally stratiﬁed boundary layer over a regular array of cubes

The ﬁnal test case features a neutral atmospheric boundary layer ﬂow over ﬂat terrain which becomes incident with a staggered

pattern of cubical obstacles. The resulting ﬂow scheme resembles urban canopy turbulence where the interaction between

roughness elements and ABL turbulence is primarily resolved. Here, the cubical obstacle height is H= 40 m. The distance

between the obstacles is 3Hin the x-directions and 1Hin the y-direction.5

To demonstrate the ﬂexibility of the nesting implementation, we carried out simulations with two different nested conﬁgu-

rations illustrated in Fig. 15. The ﬁrst (v1) case features a single child domain while the second (v2) case contains a two-stage

nesting system where a second child domain is nested within the ﬁrst. In the latter conﬁguration, the ﬁrst child acts as a parent

for the second child domain. The isotropic grid spacing is 4 m in the root domain, 2 m in the second level nest (ﬁrst child) and

1 m in the third level nest (second child). (Note, that the implementation does allow child locations to be selected such that10

their domain boundaries intersect with the obstacles.) The two example conﬁgurations represent nesting applications designed

to meet different levels of accuracy demands. The v2 conﬁguration is set to resolve the transition effect at the leading edge

of the cube canopy and to capture the blunt-body wake interactions in sufﬁcient detail within the center region of the cuboid

canopy.

First, in the context of obstacle-resolved LES, we motivate the employment of an optional canopy-restricted (CR) anterpo-15

lation strategy introduced in Sec. 3.5. For this purpose, consider Fig. 16 showing an instantaneous horizontal cross-section of

vorticity vector magnitude at z= 0.9Hheight for conﬁguration v2. The image is focusing on a region where all domains with

different resolutions are visible.

The visualization indicates the strength and spatial structure of the resolved turbulent eddies and how they are affected by

grid resolution. The differences are signiﬁcant. In such obstacle-resolving LES, the increased grid resolution has the ability20

alter the ﬂow solution to such a degree that the anterpolation introduces details to the coarser parent which are inconsistent with

the rest of the parent’s ﬂow solution. Particularly with blunt-body obstacle canopy ﬂows, this discrepancy is clearly manifested

as a locally changing resultant pressure drag (caused by the obstacles) within the anterpolated domain. To inspect this, we

compute the resultant pressure drag coefﬁcient

CF p =2

ρU2

ref Sref F2

p,x +F2

p,y1/2(14)25

for the differently coupled simulations. In Eq. 14 uref =hui|z=1.25His the reference wind speed, Fpis the resultant pressure

force exerted on the cubes obtained by integrating the pressure over vertical walls, ρis the density of air and Sref is the

accumulated frontal area of the cubes. The results are listed in Table 1, which makes evident the drastic difference between the

values for the coarse reference and the two-way coupled parent (CFp [Coarse] vs. CF p[Root]: two-way). This large difference

arises as the anterpolated solution within the obstacle canopy introduces a large-scale disturbance to the parent solution giving30

rise to unphysical secondary effects. These effects, in turn, lead to complicated feedback systems in the two-way coupled

solutions whose realizations become depended on the chosen nesting conﬁguration.

30

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 15. An overview of the cubical-obstacle case layout. The obstacles are cubes with 40 m sides. The ﬁgure displays two nested

arrangements: version 1 (v1) featuring a root domain and a secondary nest domain whereas version 2 (v2) also includes a tertiary nest

domain embedded within a larger secondary nest. The root and nested domains are indicated with (?),(??), and (???)respectively in the

upper left-hand-corner of each domain. The ﬁrst child domain is displayed with white background for better visualization.

This problematic behavior is signiﬁcantly abated by adopting the CR anterpolation strategy setting here the vertical threshold

at 1.25Hvia experimenting. This CR anterpolation allows the parent and child ﬂow ﬁelds to become strongly coupled while

minimizing global inconsistencies in the parent solution. While all the child domain solutions over-predict the pressure drag,

the two-way CR solution yield CF p[Child 2] values that are closes to the ﬁne reference.

To further evaluate the nesting performance, we exploit root (normalized) mean square difference (RNMSD or RMSD)5

and fractional bias (F B) as comparison metrics (see, Britter and Hanna, 2003) evaluated over successive xy-planes to assess

the effectiveness of the nesting approach in obstacle-resolving LES cases. RNM SD and RMSD provide a measure of mean

difference that is composed of random scatter and systematic bias whereas the fractional bias (F B) yields a speciﬁc measure

31

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 16. Instantaneous close-up view of vorticity magnitude (s−1) on xy-plane at elevation z= 0.9Hfor the v2 case with two nested

domains. Black lines indicate the bounds of the ﬁrst and the second nest. Note, only parts of the domain extents are displayed.

Table 1. Resultant pressure force coefﬁcients CF p evaluated over Child 1 (??)domain shown in Fig. 15 (v1). Results for parent and child

solutions are reported for one-way, two-way and two-way canopy-restricted (CR) methods, where the latter is a modiﬁed two-way coupling

approach where the anterpolation is restricted (i.e. not allowed) within the obstacle canopy.

Version 1 Version 2

one-way two-way two-way CR one-way two-way two-way CR

CF p[Root] 0.592 0.735 0.549 0.790 1.017 0.785

CF p[Child 1] 0.602 0.599 0.594 0.828 0.859 0.826

CF p[Fine] 0.583 0.808

CF p[Coarse] 0.592 0.790

for the systematic bias between the two solutions. The metrics are deﬁned as

RN M SD (ψ) = v

u

u

tDψ−ψRef2E

ψψRef or RMS D (ψ) = rDψ−ψRef2E(15)

F B (ψ) = ψ−ψRef

0.5ψ+ψRef.(16)

32

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

where ψis a generic prognostic variable from the considered case, while ψRef refers to the value from the reference simulation

with 2 m resolution. RMSD is used instead of RN M SD in cases where the product of double-averaged quantities used

for normalization approaches zero. Similarly, FB is only evaluated for the streamwise velocity component because other

components yield a near-zero denominator which contaminates the metric. The evaluations are performed for 15 xy-planes

within the child domain (zone (??)in Fig. 15) which are equally spaced over the range 0≤z/H ≤1.5. We have excluded 1285

m and 64 m wide development zones at the boundaries in the xand y-directions respectively. When the coarse (4 m resolution)

reference solution is compared to the ﬁne (2 m resolution) reference, the coarse solution is interpolated onto the ﬁne grid before

the comparison metrics are evaluated.

Both model variants v1 and v2 are included in the analysis to demonstrate how the size and placement of the child domains

effect the metrics and also to illustrate the possibility to employ a cascade of nested domains. Although no comparison metrics10

are presented for the second child solution featuring 1 m resolution, its inﬂuence is embedded in the solution of the ﬁrst child.

The RNMSD and RM SD proﬁles for the velocity components and their variances depicted in Figs. 17 and 18 lay bare

the effectiveness of the presented nesting system and reveal the added beneﬁt of the CR anterpolation. While all the coupling

approaches succeed in signiﬁcantly reducing the discrepancy compared to the ﬁne reference, the conventional two-way cou-

pling exhibits the most pronounced level of deviation. The F B results in Fig. 19 indicate that the two-way coupled solution15

also contains the most systematic deviation, which is conform with the pressure drag results.

The one-way coupling approach performs consistently better than the unmodiﬁed two-way coupled in all metrics, but it is

also associated with a systematic bias that is larger than the value by coarse reference. However, if the modest systematic shift

in streamwise velocity can be accepted, the one-way coupling offers a cost-effective nesting coupling approach (see Sec. 4.3

for performance measures). Nonetheless, the results conclude that the introduced CR anterpolation approach presents the most20

recommended coupling strategy for obstacle-resolving LES as it provided the best metrics in every category.

33

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 17. Vertical distributions of root (normalized) mean square difference (RNMSD or RMSD) of velocity components for conﬁgu-

ration v1 a-c) and conﬁguration v2 d-f). The metrics are evaluated for 15 horizontal planes between the range 0≤z/H ≤1.5.

34

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Figure 18. Vertical distribution of RNMS D of the horizontal velocity variances for conﬁguration v1 a-c) and conﬁguration v2 d-f). The

metrics are evaluated for 15 horizontal planes between the range 0≤z/H ≤1.5.

Figure 19. Fractional bias (F B) values evaluated over 15 xy-planes between the range 0≤z/H ≤1.5, for a) v1 and b) v2.

35

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

4.3 Performance issues

Table 2 gives an overview of the consumed CPU time in the nested as well as ﬁne- and coarse-grid reference simulations

for the hill and the cube case simulations. As a rule of thumb, doubling the resolution leads to an increase in CPU time

by approximately a factor of 16 (when the numerical time step is determined according to the CFL criterion). This can be

observed comparing the coarse- and ﬁne-grid reference simulations. Compared to the ﬁne-grid reference simulations, the5

nested simulations consumed signiﬁcantly less CPU time (up to 80% reduction) while increasing the computational cost by

factors of 3.8 and 3.4 in hill and cube canopy cases compared to the coarse reference. Although these factors depend on the

child domain size, these test make evident that the nesting technique can signiﬁcantly reduce the computational cost, while

yielding results that closely adhere to the non-nested ﬁne-resolution simulation.

Due to the inter/anterpolation and the accompanied inter-model data transfer, the nesting itself consumes CPU time. In our10

tests the workload with respect to the number of grid points treated by a processor element was equal among the parent and the

child simulation. With this optimal conﬁguration, the two-way nesting consumed about 10-16% of the CPU time in our tests,

while it consumes only about 2% in the one-way nesting. This suggests that most of the CPU time taken by two-way nesting

is consumed in the anterpolation and the associated child to parent data transfer.

Please note, if the workload between child and parent processes is not well balanced, the faster processes need to wait before15

the data-transfer can start until the slower processes reach that point, reducing the computational efﬁciency of the nesting.

Table 2. Required CPU time for the neutrally stratiﬁed and the convective test case.

Case CPU time (×102h) Overhead nesting

Hill coarse 2.2 –

Hill ﬁne 39.1 –

Hill nest 8.31 16%

Cubes coarse 8.55 –

Cubes ﬁne 129.0 –

Cubes v1 nest one-way 25.8 2.1%

Cubes v1 nest two-way 29.1 12%

5 Conclusions and future outlook

This article documents and evaluates an online LES-LES nesting scheme implemented into the PALM model system 6.0. The

nesting system relies on the post-insertion approach and features both one-way and two-way coupling approaches. We give a

detailed description of the model’s relevant technical, algorithmic and numerical aspects and provide evidence for the accuracy20

gains the method introduces with a dramatically reduced computational cost compared to globally reﬁned grid resolution.

Particularly in urban boundary layer studies requiring obstacle-resolving LES, the nesting approach has proven essential.

36

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

The implementation of this three-dimensional nesting system is based on two-level parallelism involving inter-model and

intra-model parallelization using MPI. This enables our nesting implementation to ﬂexibly support multiple child domains

which can be nested within their parent domain either in a parallel or recursively cascading conﬁguration. All solutions in-

volved within the nested simulation are advanced using a globally synchronized time step whereas the coupling between each

parent/child pair is performed with interpolation (parent to child) and anterpolation (child to parent) operations.5

The nesting method is evaluated by performing a series of numerical experiments with an objective to demonstrate that the

reﬁned child solution (nested within a coarser parent) approaches the non-nested reference solution obtained by employing ﬁne

resolution globally.

The ﬁrst test case features horizontally homogeneous convective boundary layer (CBL) with no mean mean wind. In this

case, ﬁrst and second order boundary-layer statistics are well captured in the child domain and are closely comparable to non-10

nested high-resolution reference statistics. Further, due to the local nature of turbulence production and the weak advection

from parent into the child, the ﬂow statistics show almost no dependence on the distance to the child boundaries. However, in

case of several hours long averaging times we found that a nonphysical secondary circulation develops although the surface

heating is homogeneous. We hypothesize that this secondary circulation is an inherent consequence of the spatially changing

description of ﬂow physics in the parent and child solutions. This should be kept in mind when applying the nesting system to15

CBL-problems.

The second test case simulated neutrally stratiﬁed boundary layer ﬂow over ﬂat terrain. The nested simulations reveal that the

ﬂow solution within the child domain must undergo a development phase, as the ﬂow solution adjusts to the higher resolution,

before reaching equilibrium state again. The required development length depends on the grid-spacing ratio between parent

and child. However, a purely shear-driven ﬂow over a homogeneous ﬂat terrain can be considered as an extreme scenario with20

respect to the development length of turbulence, while in cases with more complex surface geometry the ﬂow adapts within

shorter development distances. Beyond the development distance, the child solution for grid-spacing ratios of two and three

agree well with the non-nested ﬁne-reference solution, but in case of grid-spacing ratio of four the results clearly deviate from

the ﬁne-reference solution.

The third numerical experiment featured boundary layer ﬂow (similar to second test case) over a smooth three-dimensional25

hill. This test case also exploits wind tunnel measurements to strengthen the nesting model evaluation. In this case, the ﬂow

statistics in the windward and the leeward part of the hill are almost the same as in a ﬁne-reference simulation and agree well

with wind-tunnel observations presented in Ishihara et al. (1999).

The ﬁnal test case examines a ﬂow system where a fully developed boundary layer ﬂow becomes incident with a staggered

arrangement of cube-shaped obstacles. This ﬂow scenario closely resembles an obstacle-resolving urban boundary layer ﬂow30

situation. The case revealed that in two-way coupled simulations, the anterpolated child solution introduces discrepancies

within the parent domain which is manifested as elevated pressure drag within the anterpolated zone. This complication is

remedied by introducing canopy-restricted anterpolation approach, where anterpolation is omitted within the obstacle canopy.

By computing comparison metrics, root-normalized mean square difference and fractional bias, to quantify the difference

37

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

between the ﬁne reference and the nested solutions, the canopy-restricted two-way coupling is shown to be the best coupling

strategy for obstacle-resolving LES studies.

Future outlook

Future development is planned to include the following tasks. Incorporation of PALM’s Lagrangian particle model in the

nesting system in order to enable Lagrangian dispersion studies in urban environments in such a way that particles can be5

transferred between parent and child domains depending on their position. Thus, the long-distance transport of e.g. pollutants,

can be simulated in a coarse-resolution parent grid, while dispersion on the street-scale for speciﬁc locations can be simulated

in a ﬁne-resolution child domain. We note that this has been already implemented into PALM and is available to users, but

further sensitivity tests with respect to the treatment of stochastic subgrid-scale particle speeds (Weil et al., 2004) are still

pending. A thorough description and veriﬁcation of the particle nesting will be published in a follow-up article.10

Further, we note that the PALM model system 6.0 includes also a RANS (Reynolds-averaged Navier-Stokes) mode offering

two different turbulence closures to calculate the eddy diffusivity, that are a TKE-land a TKE-closure according to Mellor and

Yamada (1974, 1982). Besides the LES-LES nesting the nesting system is being extended to handle RANS-LES and RANS-

RANS nesting, which require coupling of additional RANS-variables. Moreover, in a companion paper in this special issue

we present a pure one-way off-line mesoscale nesting method in which the PALM model system 6.0 is nested into meso-scale15

models such as COSMO or WRF. This will allow modelling of meso-scale processes on a much larger coarse-grid domain as

e.g. shown by Muñoz-Esparza et al. (2017), while concurrently focusing on ﬁne-scale processes within certain areas using the

present LES nesting approach.

Furthermore, to date, the timestep in all parent and child models is synchronized and restricted to the minimum of the

time steps determined by each model independently using the CFL criterion. To our experience, the global timestep is often20

restricted by the ﬂow around building edges where high wind speeds occur within the ﬁne-grid child domains. Hence, we plan

to implement a time-splitting into PALM where the parent and child models will be coupled only at the end of the parent

timesteps. This would allow to run coarser-scale parent domains with larger time steps. Thus, computational time could be

saved in the time-integration of the parent simulation as well as in the inter-model communication between parent and child.

Appendix A: Technical realization25

A1 General

The nested model system is implemented using two levels of MPI communicators. The inter-model communication (commu-

nication between model domains) is handled by a global communicator using the one-sided communication pattern (Remote

Memory Access, RMA). The intra-model communication (communication between subdomains within each model domain) is

two-sided and it is handled using a 2-D communicator that has different color for each model. The intra-model communication30

system is the baseline parallelization of PALM (Maronga et al., 2015).

38

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Data transferred from parent to child and from child to parent is always stored in the coarser parent-model grid in order to

minimize the amount of data transfer. This means that the interpolations and anterpolations are always performed by the child.

For these purposes, children contain auxiliary arrays which follow the parent-grid spacings and indexing for each prognostic

variable to be coupled covering the overlap domain plus necessary number of ghost-node layers.

A2 Initialization5

Mapping between each parent and child model domain decompositions as well as all the necessary index mappings are de-

termined in the initialization phase and stored so that the coupling actions during the time-stepping are straightforward and

efﬁcient.

Initial conditions for the root are set similarly to non-nested runs. The root then sends initial ﬁeld data to its children which

interpolate their own initial conditions from the data received from the root. Next the ﬁrst-level children send their data to10

their children, if any, and so on. The basic interpolation subroutines for child boundary conditions operate only on the ghost

nodes behind the child-model boundaries. Therefore a separate three-dimensional interpolation subroutine is implemented to

generate initial ﬁelds for all the nest domains from their parent-model ﬁelds. The same interpolation algorithm is used here as

in the interpolations for child boundary conditions.

A3 Time synchronization15

Time synchronization is taken care by simply selecting the minimum of the time steps determined by each model independently

and broadcasting this time-step value for all models. Each model inputs and outputs in the same way.

A4 Modularization

The data transfer between parents and children is conducted by code contained by ﬁve speciﬁc fortran modules forming a

module set called PALM Model Coupler (PMC). Calls to the PMC-subroutines are mostly made in PMC-interface module20

(pmc_interface_mod.f90) such that only a small number of calls to the PMC-interface subroutines are needed within the

baseline PALM code. This way, the changes to the baseline code were kept minimal. The PMC-interface module also contains

subroutines for the nesting-related initialization actions, interpolation, anterpolation, child mass-balance forcing, etc.

A5 MPI implementation

While reading the input namelists, the PALM root process checks if a namelist called ”&nesting_parameters” is given in the25

parameter input ﬁle PARIN. If not, subroutine called pmc_init_model resets all nesting-related parameters (coupling_layout

etc.) and sets MPI_COMM_WORLD as the base global MPI-communicator comm_palm. The run then continues in standard

way without nesting. If the namelist ”&nesting_parameters” is found and correctly input, the root process of the root model

distributes this information to all other processes via MPI_COMM_WORLD. Then, all the necessary nesting related parameters

39

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

are determined and the base communicator is split into different colors 1for each model based on the model id number. This is

done by calling MPI_COMM_SPLIT. Now each model has its own process group and associated individual base communicator

color such that each model’s internal communication is not visible to other models. After this, the mappings between models

are determined. Each model, except the root model, identiﬁes its parent model and creates an inter-communicator between

the process groups of itself and its parent model. This is realized by calling MPI_INTERCOMM_CREATE. In the same way,5

each model identiﬁes its all children if any, and creates inter communicators between the process groups of itself and all of

its children. These inter communicators are only used to transfer setup data between the root processes of the parent and

child models. For 3D model data transfers between parent and child, speciﬁc intra-communicators are created by merging

inter-communicators. This is made after pmc_init_model separately for child and parent models (note that a model may be

both parent and child) in subroutines pmc_childinit and pmc_parentinit by calling MPI_INTERCOMM_MERGE. After the10

pmc-initialization, the run of each model goes as usual. Cartesian topology-based communicator comm_2d is created by each

model from its color of the base communicator comm_palm using MPI_CART_CREATE.

The model internal communication is done in the usual way, e.g. by calling the boundary exchange routines. All data transfer

between parent and child models is done within the PMC interface. For this communication MPI one sided communication

(RMA) is used. An RMA window is opened on the parent side. To transfer data from parent to child, the parent ﬁlls the RMA15

window via local copy. After synchronization via MPI_WIN_FENCE, the child processes can fetch the data across the network

with MPI_GET. While transfering data in the opposite direction, the child ﬁrst transfers the data via MPI_PUT. After another

MPI_WIN_FENCE call, the parent copies the data out of the RMA window into the local model data area.

Appendix B: Thoughts on an alternative interpolation method

Should higher interpolation accuracy across the boundaries be sought, the following considerations are relevant. As stated by20

Zhou et al. (2018), to satisfy the global ﬂux-conservation requirement, one of the ﬂux factors, either the advective velocity

component or the advected variable, must be constant within the anterpolation cell. This implies zeroth-order interpolation.

The other factor must be interpolated using any reversible interpolation scheme.

As stated above, the quadratic Clark and Farley (1984) scheme should not be used because it employs a stencil wider than the

parent-grid cell which leads to problems with complex geometries. On the other hand, tri-linear interpolation has a favorable25

stencil width, but it is not suitable for the scheme as such is not reversible. However, linearly interpolated values e

φi,j,k can

be made reversible by introducing an additional correction φi,j,k =e

φi,j,k + ∆φi,j,k which guarantees the reversibility. The

reversibility correction ∆φi,j,k depends on the difference ∆ΦI,J,K between the original parent-grid value ΦI,J,K and the

value obtained by anterpolating the linearly interpolated values to the parent-grid node I, J,K as

∆Φ = Φ −b

e

φ. (B1)30

1The term color means here that the communicator has the same name for all models (process groups), but they are, however, individual communicators

guaranteeing that communication of one model is not interfered by the others.

40

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

∆ΦI,J,K is a constant value within the parent-grid cell, hence a question arises: how to distribute the correction to the child-grid

nodes i,j,k such that c

∆φI,J,K = ∆ΦI ,J,K? The simplest choice is ∆φi,j,k = ∆ΦI ,J,K , but this choice is not recommendable

in the cases of positive deﬁnite scalar variables as it could lead to negative values when Φis close to zero. In principle this

problem could be avoided by weighting the local corrections in proportion to the local differences ΦI,J,K −e

φi,j,k but this simply

reduces the method back to the zeroth-order baseline method. To make this approach useful, a more advanced technique to5

distribute the correction ought to be developed. However, this is beyond the scope of the present work as stated in Sec. 3.4.

Acknowledgements. Test runs with PALM have been performed at the supercomputers of the North-German Super-computing Alliance

(HLRN), Germany, and CSC – IT Center for Science, Finland.

This research was funded by Academy of Finland grant number 277664. BM and MS were supported by the Federal German Ministry

of Education and Research (BMBF) under grant 01LP1601 within the framework of Research for Sustainable Development (FONA; https:10

//www.fona.de). KK was supported by the Federal German Ministry of Economy and Energy (BMU) under grant 0325719C.

Author contributions. Coordination of the study: AH, SR. Design and implementation of the inter-model communication: KK. Theoretical

considerations and implementation of the nesting interface: mainly AH with contributions from SR, MA, MS and BM. Simulations, post-

processing and analysis of model results: AH, MS, MA, BM, CK, FB, GT, NM. Drafting of the manuscript: AH, MA, MS, BM. Revision of

the manuscript: all authors.15

Appendix C: Code availability

The PALM model system is freely available at http://palm-model.org and distributed under the GNU General Public Licence v3 (http://www.

gnu.org/copyleft/gpl.html). However, the simulations presented in this document were performed using a slightly modiﬁed code based on

revision 4295. This modiﬁed source code (4295M) as well as the input ﬁles for the test runs are available at https://doi.org/10.25835/0090593

(Hellsten et al., 2020). Numerous pre- and post-processing scripts are available at http://doi.org/10.5281/zenodo.4005687 (Auvinen et al.,20

2020b).

Competing interests. The authors declare that they have no conﬂict of interest.

41

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

References

Auvinen, M., Boi, S., Hellsten, A., Tanhuanpää, T., and Järvi, L.: Study of Realistic Urban Boundary Layer Turbulence with High-Resolution

Large-Eddy Simulation, Atmosphere, 11, 201, doi:10.3390/atmos11020201, http://dx.doi.org/10.3390/atmos11020201, 2020a.

Auvinen, M., Karttunen, S., and Kurppa, M.: P4UL: Pre- and Post-Processing Python Library for Urban LES Simulations,

doi:10.5281/zenodo.4005687, https://doi.org/10.5281/zenodo.804282, 2020b.5

Bou-Zeid, E., Overney, J., Rogers, B. D., and Parlange, M. B.: The Effects of Building Representation and Clustering in Large-Eddy Sim-

ulations of Flows in Urban Canopies, Boundary-Layer Meteorology, 132, 415–436, doi:10.1007/s10546-009-9410-6, https://doi.org/10.

1007/s10546-009-9410-6, 2009.

Britter, R. E. and Hanna, S. R.: FLOW AND DISPERSION IN URBAN AREAS, Annual Review of Fluid Mechanics, 35, 469–496,

doi:10.1146/annurev.ﬂuid.35.101101.161147, https://doi.org/10.1146/annurev.ﬂuid.35.101101.161147, 2003.10

Buccolieri, R. and Hang, J.: Recent Advances in Urban Ventilation Assessment and Flow Modelling, Atmosphere, 10, 144,

doi:10.3390/atmos10030144, http://dx.doi.org/10.3390/atmos10030144, 2019.

Chung, D. and McKeon, B. J.: Large-eddy simulation of large-scale structures in long channel ﬂow, Journal of Fluid Mechanics, 661, 341–

364, doi:10.1017/S0022112010002995, 2010.

Clark, T. and Farley, R.: Severe downslope windstorm calculations in two and three spatial dimensions using anelastic interactive grid nesting:15

A possible mechanism for gustiness, Journal of Atmospheric Sciences, 41, 329–350, 1984.

Clark, T. and Hall, W.: Multi-domain simulations of the time dependent Navier-Stokes equations: benchmark error analysis of some nesting

procedures, Journal of Computational Physics, 92, 456–481, 1991.

Daniels, M., Lundquist, K., Mirocha, J., Wiersema, D., and Chow, F.: A new vertical grid nesting capability in the Weather Research and

Forecasting WRF Model, Monthly Weather Review, 144, 3725–3747, 2016.20

de Roode, S. R., Duynkerke, P. G., and Jonker, H. J.: Large-eddy simulation: How large is large enough?, Journal of the atmospheric sciences,

61, 403–421, 2004.

Deardorff, J.: Stratoculumus-capped mixed layers derived from a three-dimensional model, Boundary-Layer Meteorology, 18, 495–527,

1980.

Diebold, M., Higgins, C., Fang, J., Bechmann, A., and Parlange, M. B.: Flow over Hills: A Large-Eddy Simulation of the Bolund Case,25

Boundary-Layer Meteorology, 148, 177–194, doi:10.1007/s10546-013-9807-0, 2013.

Fishpool, G. M., Lardeau, S., and Leschziner, M. A.: Persistent Non-Homogeneous Features in Periodic Channel-Flow Simulations, Flow,

Turbulence and Combustion, 83, 323–342, doi:10.1007/s10494-009-9209-z, https://doi.org/10.1007/s10494-009-9209-z, 2009.

Giometto, M., Christen, A., Meneveau, C., Fang, J., Krafczyk, M., and Parlange, M.: Spatial Characteristics of Roughness Sublayer Mean

Flow and Turbulence Over a Realistic Urban Surface, Boundary-Layer Meteorology, pp. 1–28, 2016.30

Gropp, W., Lusk, E., and Skjellum, A.: Using MPI: Portable parallel programming with the Message Passing Interface, 2nd edition, MIT

Press, Cambridge, MA, 1999.

Hackbusch, W.: Multigrid methods and applications, Springer, Berlin, Heidelberg, New York, 378 pp., 1985.

Heinze, R., Dipankar, A., Henken, C. C., Moseley, C., Sourdeval, O., Trömel, S., Xie, X., Adamidis, P., Ament, F., Baars, H., Barthlott,

C., Behrendt, A., Blahak, U., Bley, S., Brdar, S., Brueck, M., Crewell, S., Deneke, H., Di Girolamo, P., Evaristo, R., Fischer, J., Frank,35

C., Friederichs, P., Göcke, T., Gorges, K., Hande, L., Hanke, M., Hansen, A., Hege, H.-C., Hoose, C., Jahns, T., Kalthoff, N., Klocke,

D., Kneifel, S., Knippertz, P., Kuhn, A., van Laar, T., Macke, A., Maurer, V., Mayer, B., Meyer, C. I., Muppa, S. K., Neggers, R. A. J.,

42

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Orlandi, E., Pantillon, F., Pospichal, B., Röber, N., Scheck, L., Seifert, A., Seifert, P., Senf, F., Siligam, P., Simmer, C., Steinke, S.,

Stevens, B., Wapler, K., Weniger, M., Wulfmeyer, V., Zängl, G., Zhang, D., and Quaas, J.: Large-eddy simulations over Germany using

ICON: a comprehensive evaluation, Quarterly Journal of the Royal Meteorological Society, 143, 69–100, doi:10.1002/qj.2947, https:

//rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.2947, 2017.

Hellsten, A., Ketelsen, K., Sühring, M., Auvinen, M., Maronga, B., Knigge, C., Barmpas, F., Tsegas, G., Moussiopoulos, N., and5

Raasch, S.: Dataset: A Nested Multi-Scale System Implemented in the Large-Eddy Simulation Model PALM model system 6.0,

doi:10.25835/0090593, https://doi.org/10.25835/0090593, 2020.

Heus, T., van Heerwaarden, C., Jonker, H., Siebesma, A. P., Axelsen, S., van den Dries, K., Geoffroy, O., Moene, A., Pino, D., de Roode,

S., et al.: Formulation of and numerical studies with the Dutch Atmospheric Large-Eddy Simulation (DALES), Geosci. Model Dev, 3,

415–444, 2010.10

Huq, S., De Roo, F., Raasch, S., and Mauder, M.: Vertically nested LES for high-resolution simulation of the surface layer in PALM (version

5.0), Geoscientiﬁc Model Development, 12, 2523–2538, 2019.

Hutchins, N. and Marusic, I.: Evidence of very long meandering features in the logarithmic region of turbulent boundary layers, Journal of

Fluid Mechanics, 579, 1–28, 2007.

Hutchins, N., Chauhan, K., Marusic, I., Monty, J., and Klewicki, J.: Towards reconciling the large-scale structure of turbulent boundary layers15

in the atmosphere and laboratory, Boundary-layer meteorology, 145, 273–306, 2012.

Ishihara, T., Hibi, K., and Oikawa, S.: A wind tunnel study of turbulent ﬂow over a three-dimensional steep hill, Journal of Wind Engineering

and Industrial Aerodynamics, 83, 95 – 107, doi:https://doi.org/10.1016/S0167-6105(99)00064-1, 1999.

Karttunen, S., Kurppa, M., Auvinen, M., Hellsten, A., and Järvi, L.: Large-eddy simulation of the optimal street-tree layout for

pedestrian-level aerosol particle concentrations – A case study from a city-boulevard, Atmospheric Environment: X, 6, 100 073,20

doi:https://doi.org/10.1016/j.aeaoa.2020.100073, http://www.sciencedirect.com/science/article/pii/S2590162120300125, 2020.

Kataoka, H. and Mizuno, M.: Numerical ﬂow computation around aeroelastic 3D square cylinder using inﬂow turbulence, 2002.

Kurihara, Y., Tripoli, G., and Bender, M.: Design of a Movable Nested-Mesh Primitive Equation Model, Monthly Weather Review, 107,

239–249, 1979.

Kurppa, M., Hellsten, A., Roldin, P., Kokkola, H., Tonttila, J., Auvinen, M., Kent, C., Kumar, P., Maronga, B., and Järvi, L.: Implementation25

of the sectional aerosol module SALSA2.0 into the PALM model system 6.0: model development and ﬁrst evaluation, Geoscientiﬁc Model

Development, 12, 1403–1422, doi:10.5194/gmd-12-1403-2019, https://www.geosci-model-dev.net/12/1403/2019/, 2019.

Kurppa, M., Roldin, P., Strömberg, J., Balling, A., Karttunen, S., Kuuluvainen, H., Niemi, J. V., Pirjola, L., Rönkkö, T., Timonen, H.,

Hellsten, A., and Järvi, L.: Sensitivity of spatial aerosol particle distributions to the boundary conditions in the PALM model system

6.0, Geoscientiﬁc Model Development Discussions, 2020, 1–33, doi:10.5194/gmd-2020-163, https://www.geosci-model-dev-discuss.net/30

gmd-2020-163/, 2020.

Letzel, M., Krane, M., and Raasch, S.: High resolution urban large-eddy simulation studies from street canyon to neighbourhood scale,

Atmospheric Environment, 42, 8770–8784, 2008.

Maronga, B. and Raasch, S.: Large-Eddy Simulations of Surface Heterogeneity Effects on the Convective Boundary Layer During

the LITFASS-2003 Experiment, Boundary-Layer Meteorology, 146, 17–44, doi:10.1007/s10546-012-9748-z, https://doi.org/10.1007/35

s10546-012-9748-z, 2013.

43

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Maronga, B., Gryschka, M., Heinze, R., Hoffmann, F., Kanani-Sühring, F., Keck, M., Ketelsen, K., Letzel, M., Sühring, M., and Raasch,

S.: The Parallelized Large-Eddy Simulation Model (PALM) version 4.0 for atmospheric and oceanic ﬂows: model formulation, recent

developments, and future perspectives, Geoscientiﬁc Model Development, 8, 1539–1637, 2015.

MAronga, B., Gross, G., Raasch, S., Banzhaf, S., Forkel, R., Heldens, W., Kanani-Sühring, F., Matzarakis, A., Mauder, M., Pavlik, D.,

Pfafferott, J., Schubert, S., Seckmeyer, G., Sieker, H., and Winderlich, K.: Development of a new urban climate model based on the model5

PALM - Project overview, planned work, and ﬁrst achievements, Meteorol. Z., 28, 105–119, doi:10.1127/metz/2019/0909, 2019.

Maronga, B., Banzhaf, S., Burmeister, C., Esch, T., Forkel, R., Fröhlich, D., Fuka, V., Gehrke, K. F., Geletiˇ

c, J., Giersch, S., Gronemeier, T.,

Groß, G., Heldens, W., Hellsten, A., Hoffmann, F., Inagaki, A., Kadasch, E., Kanani-Sühring, F., Ketelsen, K., Khan, B. A., Knigge, C.,

Knoop, H., Krˇ

c, P., Kurppa, M., Maamari, H., Matzarakis, A., Mauder, M., Pallasch, M., Pavlik, D., Pfafferott, J., Resler, J., Rissmann, S.,

Russo, E., Salim, M., Schrempf, M., Schwenkel, J., Seckmeyer, G., Schubert, S., Sühring, M., von Tils, R., Vollmer, L., Ward, S., Witha,10

B., Wurps, H., Zeidler, J., and Raasch, S.: Overview of the PALM model system 6.0, Geoscientiﬁc Model Development, 13, 1335–1372,

doi:10.5194/gmd-13-1335-2020, https://www.geosci-model-dev.net/13/1335/2020/, 2020.

Mellor, G. L. and Yamada, T.: A Hierarchy of Turbulence Closure Models for Planetary Boundary Layers, Journal of the Atmospheric

Sciences, 31, 1791–1806, doi:10.1175/1520-0469(1974)031<1791:AHOTCM>2.0.CO;2, 1974.

Mellor, G. L. and Yamada, T.: Development of a turbulence closure model for geophysical ﬂuid problems, Reviews of Geophysics, 20,15

851–875, doi:10.1029/RG020i004p00851, 1982.

Miller, M. J. and Thorpe, A. J.: Radiation conditions for the lateral boundaries of limited-area numerical models, Quarterly Journal of

the Royal Meteorological Society, 107, 615–628, doi:10.1002/qj.49710745310, https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.

49710745310, 1981.

Mirocha, J., Kirkil, G., Bou-Zeid, E., Chow, F. K., and Kosovi´

c, B.: Transition and equilibration of neutral atmospheric boundary layer ﬂow20

in one-way nested large-eddy simulations using the Weather Research and Forecasting model, Monthly Weather Review, 141, 918–940,

doi:10.1175/MWR-D-11-00263.1, https://doi.org/10.1175/MWR-D-11-00263.1, 2013.

Moeng, C., Dudhia, J., Klemp, J., and Sullivan, P.: Examinning two-way grid nesting for large-eddy simulation of the PBL using the WRF

model, Monthly Weather Review, 135, 2295–2311, 2007.

Muñoz-Esparza, D., Kosovi´

c, B., García-Sánchez, C., and van Beeck, J.: Nesting turbulence in an offshore convective boundary layer using25

large-eddy simulations, Boundary-layer meteorology, 151, 453–478, 2014.

Muñoz-Esparza, D., Lundquist, J. K., Sauer, J. A., Kosovi´

c, B., and Linn, R. R.: Coupled mesoscale-LES modeling of a diurnal cycle during

the CWEX-13 ﬁeld campaign: From weather to boundary-layer eddies, 2017.

Munters, W., Meneveau, C., and Meyers, J.: Shifted periodic boundary conditions for simulations of wall-bounded turbulent ﬂows, Physics

of Fluids, 28, 025 112, doi:10.1063/1.4941912, https://doi.org/10.1063/1.4941912, 2016.30

Nakayama, H., Takemi, T., and Nagai, H.: Development of LOcal-scale High-resolution atmospheric DIspersion Model using Large-Eddy

Simulation. Part 5: detailed simulation of turbulent ﬂows and plume dispersion in an actual urban area under real meteorological condi-

tions., Journal Nuclear Science and Technology, 53, 887–908, doi:10.1080/00223131.2015.1078262, https://doi.org/10.1080/00223131.

2015.1078262, 2016.

Nunalee, C. G., Kosovi´

c, B., and Bieringer, P. E.: Development of LOcal-scale High-resolution atmospheric DIspersion Model using Large-35

Eddy Simulation. Part 5: detailed simulation of turbulent ﬂows and plume dispersion in an actual urban area under real meteorological

conditions., Atmospheric Environment, 99, 571–581, doi:10.1016/j.atmosenv.2014.09.070, https://doi.org/10.1016/j.atmosenv.2014.09.

070, 2014.

44

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.

Patrinos, A. N. A. and Kistler, A. L.: A numerical study of the Chicago lake breeze, Boundary-Layer Meteorol., 12, 93–123, 1977.

Raasch, S. and Schröter, M.: PALM – A large-eddy simulation model performing on massively parallel computers, Meteorologische

Zeitschrift, 10, 363–372, 2001.

Saiki, E. M., Moeng, C.-H., and Sullivan, P. P.: Large-eddy simulation of the stably stratiﬁed planetary boundary layer, Boundary Layer

Meteorol., 95, 1–30, 2000.5

Skamarock, W. C., Klemp, J. B., Dudhia, J., Gill, D. O., Barker, D. M., Duda, M. G., Huang, X.-Y., Wang, W., and Powers, J. G.: G.: A

description of the Advanced Research WRF version 3, Tech. Rep. NCAR/TN-475+ STR, NCAR, 2008.

Sühring, M., Maronga, B., Herbort, F., and Raasch, S.: On the Effect of Surface Heat-Flux Heterogeneities on the Mixed-Layer-Top Entrain-

ment, Boundary-Layer Meteorology, 151, 531–556, doi:10.1007/s10546-014-9913-7, https://doi.org/10.1007/s10546-014-9913-7, 2014.

Sullivan, P. and Patton, E.: The effect of mesh resolution on convective boundary layer statistics and structures generated by large-eddy10

simulation, Journal of the Atmospheric Sciences, 68, 2395–2415, 2011.

Sullivan, P., McWilliams, J., and Moeng, C.-H.: A grid nesting method for large-eddy simulation of planetary boundary-layer ﬂow, Boundary-

Layer Meteorology, 80, 167–202, 1996.

Tominaga, Y. and Stathopoulos, T.: CFD simulation of near-ﬁeld pollutant dispersion in the urban environment: A review of current modeling

techniques, Atmospheric Environment, 79, 716 – 730, doi:https://doi.org/10.1016/j.atmosenv.2013.07.028, http://www.sciencedirect.com/15

science/article/pii/S1352231013005499, 2013.

Tseng, Y.-H., Meneveau, C., and Parlange, M. B.: Modeling ﬂow around bluff bodies and predicting urban dispersion using large eddy

simulation, Environmental science & technology, 40, 2653–2662, 2006.

Vonlanthen, M., Allegrini, J., and Carmeliet, J.: Assessment of a one-way nesting procedure for obstacle resolved large eddy simulation of

the ABL, Computers and Fluids, 140, 136–147, doi:10.1016/j.compﬂuid.2016.09.016, https://doi.org/10.1016/j.compﬂuid.2016.09.016,20

2016.

Vonlanthen, M., Allegrini, J., and Carmeliet, J.: Multiscale interaction between a cluster of buildings and the abl developing over a real

terrain, Urban Climate, 20, 1–19, doi:10.1016/j.uclim.2017.02.009, https://doi.org/10.1016/j.uclim.2017.02.009, 2017.

Weil, J., Sullivan, P., and Moeng, C.: The use of large-eddy simulations in Lagrangian particle dispersion models, Journal of the Atmospheric

Sciences, 61, 2877–2887, 2004.25

Wicker, L. J. and Skamarock, W. C.: Time-splitting methods for elastic models using forward time schemes, Mon. Wea. Rev., 130, 2088–

2097, 2002.

Williamson, J. H.: Low-storage Runge-Kutta schemes, J Comput. Phys., 35, 48–56, 1980.

Xie, Z. and Castro, I.: LES and RANS for turbulent ﬂows over arrays of wall-mounted obstacles, Flow, Turbulence and Combustion, 76,

291–312, 2006.30

Zhou, B., Xue, M., and Zhu, K.: A grid-reﬁnement-based approach for modelling the convective boundary layer in the gray zone: Algorithm

implementation and testing, Journal of the Atmospheric Sciences, 75, 1143–1161, 2018.

Zhu, P., Albrecht, B. A., Ghate, V. P., and Zhu, Z.: Multiple-scale simulations of stratocumulus clouds, Journal of Geophysical Research:

Atmospheres, 115, 2010.

45

https://doi.org/10.5194/gmd-2020-222

Preprint. Discussion started: 7 September 2020

c

Author(s) 2020. CC BY 4.0 License.