Available via license: CC BY 4.0
Content may be subject to copyright.
A Nested Multi-Scale System Implemented in the Large-Eddy
Simulation Model PALM model system 6.0
Antti Hellsten1, Klaus Ketelsen2, Matthias Sühring3, Mikko Auvinen1, Björn Maronga3,6,
Christoph Knigge4, Fotios Barmpas5, Georgios Tsegas5, Nicolas Moussiopoulos5, and Siegfried Raasch3
1Finnish Meteorological Institute, P.O.Box 503, FI-00101, Helsinki, Finland
2Software Consultant, Beethovenstr. 29A, 12247 Berlin, Germany
3Leibniz University Hannover, Institute of Meteorology and Climatology, Herrenhäuser Strasse 2, 30419 Hannover, Germany
4Deutscher Wetterdienst, Frankfurter Straße 135, D-63067 Offenbach, Germany
5Aristotle University Thessaloniki, P.O.Box 483, GR-54124, Thessaloniki, Greece
6University of Bergen, Geophysical Institute, Postboks 7803, 5020 Bergen, Norway
Correspondence to: A. Hellsten (antti.hellsten@fmi.fi)
Abstract.
Large-eddy simulation provides a physically sound approach to study complex turbulent processes within the atmospheric
boundary layer including urban boundary layer flows. However, such flow problems often involve a large separation of turbulent
scales, requiring a large computational domain and very high grid resolution near the surface features, leading to prohibitive
computational costs. To overcome this problem, an online LES-LES nesting scheme is implemented into the PALM model5
system 6.0. The hereby documented and evaluated nesting method is capable of supporting multiple child domains which
can be nested within their parent domain either in a parallel or recursively cascading configuration. The nesting system is
evaluated by simulating first a purely convective boundary layer flow system and then three different neutrally-stratified flow
scenarios with increasing order of topographic complexity. The results of the nested runs are compared with corresponding non-
nested high- and low-resolution results. The results reveal that the solution accuracy within the high-resolution nest domain10
is clearly improved as the solutions approach the non-nested high-resolution reference results. In obstacle-resolving LES,
the two-way coupling becomes problematic as anterpolation introduces a regional discrepancy within the obstacle canopy of
the parent domain. This is remedied by introducing canopy-restricted anterpolation where the operation is only performed
above the obstacle canopy. The test simulations make evident that this approach is the most suitable coupling strategy for
obstacle-resolving LES. The performed simulations testify that nesting can reduce the CPU time up to 80% compared to15
the fine-resolution reference runs while the computational overhead from the nesting operations remained below 16% for the
two-way coupling approach and significantly less for the one-way alternative.
1 Introduction
Large-Eddy Simulation (LES) has been used for basic research of atmospheric boundary layer (ABL) phenomena using ide-
alized model setups for decades. Now, it is becoming an important method in applied research on realistic, very detailed and20
complicated flow systems such as urban ABL problems, e.g. (Britter and Hanna, 2003; Tseng et al., 2006; Bou-Zeid et al.,
1
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
2009; Tominaga and Stathopoulos, 2013; Giometto et al., 2016; Buccolieri and Hang, 2019). Until the recent years, there were
no ABL LES models capable of modelling detailed surface structures, such as buildings or steep complex terrain shapes in
ABL. Nowadays, it is possible to carry out LES for complex built areas (e.g., Letzel et al., 2008), but this is still limited to
relatively small areas because of the high spatial resolution requirement. Concerning urban LES, Xie and Castro (2006) have
shown that at least from 15 to 20 grid nodes are needed across street canyons to satisfactorily resolve the most important turbu-5
lent structures within the canyons. This requirement typically leads to grid spacings on the order of 1 m. However, the vertical
extent of the LES domain should scale with the ABL height, and the horizontal size should span over several ABL heights in
order to capture the ABL-scale turbulent structures (de Roode et al., 2004; Fishpool et al., 2009; Chung and McKeon, 2010;
Auvinen et al., 2020a). To adequately capture processes on the street-scale and to simultaneously capture large ABL-scale tur-
bulence, sufficiently large model domains at small grid sizes are required, posing high demands on the computational resources10
in terms of CPU time and memory. Moreover, the uncertainty related to the lateral boundary conditions usually decreases as
the domain is made larger.
Many numerical solution methods (e.g. finite-element and finite-volume methods) allow variable resolution so that the
resolution can be concentrated to the area of principal interest and relaxed elsewhere. However, only unstructured grid systems
allow to take full advantage of spatially variable resolution. Many general-purpose computational fluid dynamics packages offer15
unstructured grid systems, but according to our experience such solvers are usually computationally decidedly less efficient than
ABL-tailored LES models, such as PALM (Raasch and Schröter, 2001; Maronga et al., 2015, 2020), the Weather Research and
Forecasting Model (WRF) (Skamarock et al., 2008) with its LES option and the Dutch Atmospheric Large-Eddy Simulation
(DALES) (Heus et al., 2010) that are based on structured orthogonal grid system with constant horizontal resolution. The model
nesting approach can be exploited to further speed up ABL LES models or to allow larger domain sizes without compromizing20
the resolution in the area of primary interest.
The idea of grid nesting is to simultaneously run a series of two or more LES model domains with different spatial extents and
grid resolutions. In this implementation the outermost and coarsest-resolution LES domain (termed root domain henceforth),
which acts as a parent to its child domains, obtains its boundary conditions in a conventional manner, whereas the nested
LES domain (child) always obtains its boundary condition from its respective parent domain through interpolation. In one-25
way coupled nesting only the children obtain information from their parents. In such a coupling strategy, the instantaneous
child and parent solutions can deviate within the volume of the nest. If a stronger binding between the solutions is desired,
the child solution needs to be incorporated into the parent solution. This is achieved in two-way coupled nesting, where the
parent solutions are influenced by their children through so-called anterpolation (Clark and Farley, 1984; Clark and Hall, 1991;
Sullivan et al., 1996).30
The child-to-parent anterpolation can be implemented using, for instance, the post insertion (PI) approach (Clark and Hall,
1991) where the parent solution is replaced by the child solution within the volume occupied by both domains. In practice,
some buffer zones where anterpolation is omitted are necessary near the child boundaries to avoid growth of unphysical
perturbations near the child boundaries (Moeng et al., 2007). An example of a two-way coupled nesting implemented in the
WRF-LES model is given by Moeng et al. (2007), and later successfully applied to a stratocumulus study by Zhu et al. (2010).35
2
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
However, this WRF-LES nesting system is limited to horizontal directions, i.e. all the domains have equal height which may
lead to computational inefficiency. The WRF-LES nesting system can also be used in one-way coupled mode (Mirocha et al.,
2013), and this way it has been applied, e.g. to a complex-terrain study (e.g. Nunalee et al., 2014; Muñoz-Esparza et al.,
2017) and to an offshore convective boundary layer study (Muñoz-Esparza et al., 2014). More recently, Daniels et al. (2016)
introduced a vertical interpolation into the WRF model, but this method is restricted to one-way coupled nesting. Moreover,5
according to our knowledge, WRF-LES is not applicable to blunt-obstacle resolving LES required for urban turbulence studies.
In addition to WRF-LES, the numerical weather prediction model ICON features an LES mode and includes an online nesting
capability (Heinze et al., 2017). However, due to their terrain following coordinate system, neither WRF-LES nor ICON-LES
can resolve sharp obstacle structures, hence these models cannot be applied to urban studies in obstacle-resolving fashion.
Recently, Huq et al. (2019) implemented a purely vertical nesting system into PALM in which the child and parent domains10
are required to have the same horizontal extent. Although this approach is useful, e.g. when the grid resolution near the surface
needs to be refined to better capture the atmosphere-surface exchange, the requirement of equal horizontal domain extensions
poses high demands on the computational resources, limiting this approach to only academic studies. This implementation is
also limited to have a single child domain only. For these reasons, we decided to develop the present, more general and fully
three-dimensional nesting system in PALM. It can also be run in a pure vertical nesting mode.15
One-way coupled obstacle-resolving LES has been applied to a built environment by Nakayama et al. (2016) and by Von-
lanthen et al. (2016, 2017). Also the present PALM implementation has already been demonstrated by MAronga et al. (2019);
Maronga et al. (2020) and applied to obstacle-resolving urban studies (Kurppa et al., 2019; Auvinen et al., 2020a; Karttunen
et al., 2020; Kurppa et al., 2020) using the one-way coupling. At current stage, we are not aware of any research on obstacle-
resolving LES employing two-way coupled nesting approach. Through our studies, we have observed that the application of20
two-way coupling in obstacle-resolving LES can become problematic. Therefore, in addition to documenting and evaluating
the newly implemented nesting method in the PALM model, this paper addresses the applicability of the two-way coupled
nesting approach in obstacle-resolving LES.
The paper is organized as follows: Sect. 2 gives a brief description of the LES mode of the PALM model system 6.0. Section 3
presents the technical, algorithmic and numerical aspects of the implemented nesting. In Sect. 4 the implemented nesting is25
evaluated for a series of test cases featuring different kinds of boundary-layer flow. Finally, Sect. 5 summarizes the results and
gives and outlook of future developments.
2 The PALM model system 6.0 (LES mode)
The PALM model system (Raasch and Schröter, 2001; Maronga et al., 2015, 2020) is based on the non-hydrostatic, filtered,
Navier-Stokes-equations in the Boussinesq-approximated or anelastic form. It solves the prognostic equations for the con-30
servation of momentum, mass, energy, and moisture on a staggered Cartesian Arakawa-C grid. Subgrid-scale turbulence is
parameterized using a 1.5-order closure after Deardorff (1980) in the formulation of Saiki et al. (2000). In its standard config-
uration PALM has thus seven prognostic quantities: the velocity components ui(where u1=u,u2=v,u3=w), the potential
3
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
temperature θ, specific humidity qv, a passive scalar s, and the subgrid-scale (SGS) turbulent kinetic energy e. By default,
discretization in time and space is achieved using a 3rd-order Runge-Kutta scheme after Williamson (1980) and a 5th-order
advection scheme after Wicker and Skamarock (2002). The horizontal grid spacing is always equidistant, whereas it is possible
to use variable grid spacing in the vertical direction. Often, the vertical grid spacing is set equidistant within the boundary
layer, and stretching is applied above the boundary layer to save computational time in the non-turbulent free atmosphere. At5
the lateral boundaries cyclic conditions or more advanced in- and outflow conditions can be employed.
Both the Boussinesq and the anelastic approximation require incompressibility of the flow. To provide this feature a predictor-
corrector method is used where an equation is solved for the modified perturbation pressure after every Runge-Kutta sub-time
step (e.g. Patrinos and Kistler, 1977). The method involves the calculation of a preliminary prognostic velocity. Divergences in
the flow field are then attributed solely to the pressure term, leading to a Poisson equation for the perturbation pressure. In case10
of cyclic lateral boundaries, the Poisson equation is solved by using a direct fast Fourier transform (FFT) method. However, in
case of non-cyclic boundary conditions, an iterative multigrid scheme is used (e.g. Hackbusch, 1985).
Parallelization of PALM is achieved by using the Message Passing Interface (MPI, e.g. Gropp et al., 1999) and a two-
dimensional (horizontal) domain decomposition.
3 Nesting system15
3.1 General concept
The nesting system we have developed is based on the concept of parent and child domains. Each parent domain can enfold
multiple child domains but a child domain can, naturally, only have one parent domain. The top-level domain, also called
root domain, acts as a parent domain to child domains at the first nesting level. The child domains at first nesting level might
have subsequent child domains for which they then act as parent domains (cascading arrangement), see Fig. 1. Our nesting20
system allows for up to 63 nested domains plus the root domain. The implementation requires that all child domains are always
completely located inside their respective parent domain. Also, the grid spacings of a child domain naturally have to be smaller
than the grid spacings of its parent domain. The grid-spacing ratios ∆Xi/∆ximust always be integer valued although different
ratios may be used in different directions. There may be multiple child domains at the same nesting levels, but overlapping
child domains at the same nesting level are not permitted. Finally, all the nest domains have to be surface-bound so that elevated25
child domains are not allowed.
In general, the system is designed as two-way coupled nesting, in which a child domain can affect its parent domain and
vice versa. It is possible, however, to run the system in a one-way coupled mode where no feedback from the child domain is
incorporated in its parent domain. Moreover, it is possible to use the system as a pure vertical one-dimensional nesting, where
the lowest part of the model (e.g. the atmospheric surface layer where the dominant turbulent eddies are usually very small)30
can be run as a child domain with finer grid spacing than its parent domain that compasses the entire boundary layer. In the
case of pure vertical nesting, cyclic boundary conditions must be set on all the lateral boundaries. Unlike the method proposed
by Huq et al. (2019), the present method allows a cascade of more than one child domain also in the pure vertical nesting cases.
4
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
parent domain
x,i
y,j
z,k
Figure 1. A schematic example of a nested configuration involving both cascading and parallel child domains is shown on x, y-plane on the
left-hand side. On the right-hand side, a three-dimensional view of a nested child domain inside its parent domain is shown.
The present nesting approach is a variant of the PI method, in which the communication between each parent-child couple
is realized via interpolations (from parent to child) and anterpolations (from child to parent) after each Runge-Kutta sub-step
and just before the pressure solver. The latter then ensures that mass conservation is enforced in the anterpolated solution in
the parent domain.
3.2 Restrictions5
The current implementation poses a few restrictions for the nested setups. Moreover, the interpolation and anterpolation meth-
ods, which are discussed in the following sections, are based on certain assumptions, e.g. on the grid-line matching between
parent and child domains leading to a few more restrictions. Altogether these restrictions are:
–the child domain must always be completely inside its parent domain and there must be a margin of four parent-grid
cells between the boundaries of child and parent domains10
–parallel child domains must not overlap each other
–the domain decomposition of all child domains must be such that the sub-domain size is never smaller than the parent
grid-spacing in the respective direction
–buildings or any other topography must not reach the child domain top
–all the grid-spacing ratios must be integer valued15
5
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
–the outer boundaries of child domains must match with grid planes in its parent domain
–no grid stretching is allowed in the child domains and in root domain it is allowed only above the top boundary of the
highest child domain
3.3 Structure of the nesting algorithm
Ideally, the coupling actions, i.e data transfers between the domains, anterpolation and interpolation, would be performed after5
the pressure-correction step using the final divergence free velocity field on both parent and child. To achieve this in the context
of the pressure-correction method employed in PALM requires a staged arrangement of the coupling actions such that a child
first sends data to its parent and after receiving the data the parent anterpolates and performs the pressure correction step. After
the pressure-correction step the parent sends data to the child which interpolates new boundary conditions from the received
data and performs the pressure-correction step. The purely vertical nesting method implemented in PALM by Huq et al. (2019)10
features this kind of staged structure. However, Huq et al.’s method may lead to excessive waiting times as the child has to
wait until the parent performs the pressure-correction step and vice versa. Moreover, the staged coupling approach becomes
more complicated and more inefficient when a cascade of several nested domains is used. Therefore, Huq’s implementation
allows for only one child domain. The possibility to employ cascades of child domains was an initial requirement for the
present system design and therefore the staged coupling arrangement had to be abandoned. In principle, it would be possible15
to perform the pressure-correction step twice, first time before the coupling actions for all domains and second time for all
parent domains after the coupling to make the anterpolated fields divergence free. However, this would be computationally
very expensive severely compromising the benefits from the nesting. This is because the pressure solution is typically the most
time consuming part of the solution process. To avoid this extra penalty, the coupling is based on the preliminary prognostic
velocity fields upre in the present implementation. The sequence of the coupling actions is illustrated in Fig. 2. This choice20
has the consequence that the interpolated velocity boundary conditions for a child domain may violate the global mass balance
over the child domain such that
Z
S
ρe
upre ·ndS6= 0,(1)
where the tilde symbol is the interpolation operator and nis the unit inner surface normal vector of the child domain boundary
Sexcluding the bottom boundary. This mass-conservation error, though typically small, is eliminated in an integral sense by25
adding a constant velocity correction ∆upre
∆ul,pre =−nlRSρe
upre ·ndS
RSρdS(2)
to the interpolated child boundary values to exactly eliminate the global mass-balance error in Eq. (1). Here l∈ {1,1,2,2,3}
corresponding to the left (1), right (1), south (2), north (2), and top (3) boundaries. In case of purely vertical nesting mode, the
correction is applied only on the top boundary and Sspans only over it. This correction is made for all child domains right30
6
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
before the pressure-correction step. According to our tests, ∆upre is typically three or four orders of magnitude smaller than
the dominant velocity scales of the flow.
Huq et al. (2019) showed results for a zero mean-wind CBL case. In this case, especially if the nest-top boundary is set
on a relatively low level, unphysical overestimation of horizontal velocity component variances easily develop if the coupling
is based on upre. Huq et al. (2019) showed that using the staged sequence of coupling actions, allowing the coupling based5
on the final velocity field u, mostly removes the overestimation of the horizontal-velocity variances. We have confirmed this
by temporarily modifying the current implementation to adhere to Huq et al.’s staged arrangement and simulating a vertically
nested zero mean wind CBL case similar to Huq et al.’s test case.
In the present method, the overestimation of the horizontal velocity component variances can be mostly avoided by using
the integral mass-balance forcing (Eq. 2) and further by setting a narrow buffer zone below the top boundary in which the10
anterpolation is not performed. This is described in more detail in Sec. 3.5.
In addition to the velocity field, also all other prognostic variables are coupled except the SGS turbulent kinetic energy e, as
it depends on the resolution by definition and therefore it is not straightforward to couple between parent and child domains
having different resolutions. The anterpolated values should be increased by some unknown resolution dependent factor and
the interpolated values should be reduced accordingly. estrongly follows the velocity-gradient field and therefore it tends to15
adapt to the anterpolated velocity field on the parent side during the next Runge-Kutta step without being anterpolated itself.
Relying on this reasoning, we omit the anterpolation of e. Moreover, we assume that the local generation of eoften dominates
its advection implying that replacing the interpolation of its child-boundary values by simple zero-gradient conditions may be
acceptable. In our numerical tests we compared the zero-gradient conditions with interpolated boundary values reduced by an
estimated resolution-difference dependent factor. The comparisons revealed no significant differences in the results.20
Further technical implementation issues are discussed in Appendix A.
3.4 Interpolation (parent to child)
Boundary conditions for the child domain lateral and top boundaries are given using data from its parent domain. This data
is interpolated and set to the boundary grid points right behind the outer boundary of the respective child sub-domain. As
mentioned in Sec. 3.3, all prognostic variables are interpolated except the SGS turbulent kinetic energy e. It is not interpolated25
since it depends on the grid resolution by definition and therefore the parent einterpolated to the child-domain boundaries
would be inconsistent with the child grid. Instead, a simple Neumann condition (zero-gradient condition) is used on child-
domain boundaries.
It is very important that the interpolation method conserves the mass (volume) flow rate through the boundaries. If the mass
conservation is violated in a two-way coupled run, a non-physical secondary circulation usually develops. Clark and Farley30
(1984) introduced a quadratic interpolation scheme that forms a reversible pair with their anterpolation scheme which we also
employ. This reversibility guarantees the mass conservation. However, as recently noted by Zhou et al. (2018), the conservation
of fluxes other than the mass flux is violated if both advective velocity component and advected variable are interpolated using
the Clark and Farley (1984) scheme or in fact any interpolation scheme of higher than zeroth order. They selected to use the
7
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 2. Flowchart illustrating the nesting actions in case of three domains in cascading order. In case of more than three levels of domains,
more branches similar to the current middle branch would be added. Blue boxes represent baseline PALM actions while the other colors
indicate nesting-specific actions. In one-way coupling only the actions indicated by pink color are invoked.
Clark and Farley (1984) method only for the advective velocity component and the simple zeroth-order method, which in one
dimension is
φi= ΦIfor iwithin the parent-grid cell I , (3)
for all advected variables. The child variables and indices are denoted by lowercase letters and those of parent by uppercase.
This choice satisfies the flux conservation requirement for all variables if the grid-spacing ratio is odd valued. Conservation5
of fluxes through the nest boundaries is an important condition for a nesting algorithm. According to our tests the lack of flux
conservation for example for the velocity component von the left nest boundary may lead to a wrong mean-flow angle in the
whole system of domains, and such an error can be remarkably large. Therefore, we design the interpolation method such a
way that the flux conservation errors on the nested boundaries are minimized. In PALM, the interpolation algorithm has to
8
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
cope with complex topography. Therefore we do not select the quadratic scheme of Clark and Farley (1984) at all. Instead, we
use a zeroth-order interpolation for all variables. Our approach is to use Eq. (3) for the boundary normal velocity component
uNand all scalar variables, and another zeroth-order interpolation for the staggered velocity components uS
ias follows (in one
dimension)
uS
i=
US
Ifor grid points ico-located with a parent-grid point Iin the direction of the interpolation
1
2(US
I−1+US
I)for grid points ibetween parent-grid points I−1and I . (4)5
The reason behind this choice is explained and discussed in what follows.
In principle, the most straightforward way to satisfy the flux conservation would be to directly use the flux on the parent-grid
cell face on the boundary and to distribute it onto the underlying child-grid cell faces in the proportion of the cell-face areas
in the fashion of the finite volume method in which the fluxes are typically stored. However, PALM is formulated as a finite
difference method and thence its architecture does not support this method. Therefore the interpolation procedure should be at10
least approximately flux conservative.
Zhou et al. (2018) require separately conservation of a prognostic variable φand its resolved-scale turbulent flux hun0φ0i
through the boundary, where uNis the boundary-normal velocity component, as
hφib=hΦib,(5)
huN0φ0ib=hUN0Φ0ib,(6)15
where φand uNare variables of the child grid and Φand UNare variables on the parent grid, and h·ibdenote spatial averaging
over the child domain boundary. The first condition Eq. (5) has originally been stated by Kurihara et al. (1979). Later Clark
and Farley (1984) stated a stronger local form of this condition called reversibility condition. Reversibility of interpolation and
anterpolation means that the following holds locally, i.e. not only in the sense of spatial averaging over the whole boundary
[
e
φ(Φ) = Φ,(7)20
where the tilde is the interpolation operator and the hat is the anterpolation operator. Naturally, the reversibility guarantees
that the conservation condition Eq. (5) is fulfilled. Obviously the zeroth-order interpolation satisfies the reversibility condition
Eq. (7) in addition to Eq. (5). It is straightforward to show that if Eq. (5) holds, the flux conservation condition Eq. (6) can be
also be written for the total flux as
huNφib=hUNΦib.(8)25
We shall study the flux conservation using this condition instead of Eq. (6).
Zhou et al. (2018) use the quadratic interpolation scheme by Clark and Farley (1984) for the boundary-normal velocity
component, i.e. the advective component. For all other variables they selected to use the simple zeroth-order interpolation
Eq. (3). This selection was made in order to satisfy the flux conservation condition Eq. (6) which is equivalent to the condition
Eq. (8). This choice readily satisfies the flux conservation for all variables which are non-staggered relative to uNand UN. It also30
9
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
satisfies the flux conservation condition for the velocity components uSand USstaggered relative to uNand UNin the boundary
plane, but only if the grid-spacing ratio is odd-valued. The difference of the odd- and even-valued grid-spacing ratio cases is
illustrated in Fig. 3. The method by Zhou et al. (2018) is, indeed, strictly limited to odd-valued grid-spacing ratios. This is a
strong restriction and therefore we decided to allow also even valued grid-spacing ratios. In such cases, the flux conservation
condition is still readily satisfied for the non-staggered variables, but for staggered velocity components the situation becomes5
more complicated and it is not necessary possible to satisfy it exactly. By using Eq. (4) for the staggered velocity components
the flux conservation condition can be satisfied approximately for both odd and even grid-spacing ratios. As an example and
for the sake of clarity but without any loss of generality, we compute the spatially averaged fluxes of v- and V-components
for a nested domain face and assume that the boundary-normal direction is x. The velocity components in the x-direction are
uand U. The advective uvelocity for the flux is interpolated linearly to the flux point of vas (uj−1+uj)/2. The chosen10
interpolation technique Eq. (4) leads to the following averaged resolved-scale advection flux of vthrough the child boundary
(the SGS-fluxes are assumed small and omitted here)
huvib=1
[Ry(Ny+ 1) −1]Nz"Ry
Jn
X
J=Js
Kt
X
K=Kb
UJ−1,K +UJ,K
2VJ,K + (Ry−1)
Kt
X
K=KbU0,K
2V0,K +UNy,K
2VNy+1,K #,
(9)
while on the parent grid, the averaged advection flux of Vis (as expanded by Ryfor easier comparison)
hUV ib=1
RyNyNz
Jn
X
J=Js
Kt
X
K=Kb
Ry
UJ−1,K +UJ,K
2VJ,K .(10)15
Here, Jand Kare the parent-grid indices, and the child-domain boundary covers the parent-grid node range where Js≤
J≤Jnand Kb≤K≤Ktwith Ny=Jn−Js+ 1 and Nz=Kt−Kb+ 1.Ryis the integer-valued grid-spacing ratio in the
y-direction. Clearly Eq. (9) and Eq. (10) do not exactly equal each other because of the edge terms depending on e.g. V0and
VNy+1, and because the denominator of Eq. (9) slightly deviates from RyNyNz. It should be noted that these two differences
usually have opposite effects, and that huvi − hUV itends towards zero as Nybecomes large. In typical applications, the order20
of magnitude of Nyis hundreds making the flux conservation error negligibly small. Nzis usually smaller than Ny, maybe even
one order of magnitude smaller, making the flux conservation error of the vertical velocity component wpossibly somewhat
larger than that of the horizontal components. If Nz= 32 and the grid-spacing ratio Rz= 3 for example, huvi − hU V ican
be expected to be of the order of 1%. Moreover, a small flux conservation error does not distort was easily as the horizontal
components, because wunlike the horizontal components is relatively strongly controlled by its surface boundary condition.25
In the above considerations, it is assumed that the advected parent-grid variable values on the child boundary are equal
to the values used for the parent-grid flux computation. This is not the case in reality since on the parent side the advected
variable values used for calculating the advection fluxes are interpolated using the 5th-order interpolation scheme by Wicker
and Skamarock (2002) while on the child boundary this is not the case. Here, it is important to understand, that the above
mentioned interpolation onto the flux point in the advection scheme is a separate procedure from the interpolation from parent30
to child, and it is performed in a different phase of the time-step cycle.
10
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 3. Staggered velocity-component nodes in cases of odd (3) a) and even (4) b) grid-spacing ratios. The staggered velocity-component
nodes are shown as arrows, thick arrows are for the parent grid and thin ones for the child grid. The parent scalar grid-cell faces are drawn
with solid lines and the corresponding child-grid cells with dotted lines. Locations of the corresponding parent-grid scalar nodes are shown
as black dots. The blue color indicates the left-hand parent-grid node and red the right-hand node. The violet colored child grid nodes receive
the averaged values according to Eq. (4).
The default advection scheme in PALM is the fifth-order Wicker and Skamarock (2002) scheme which employs a stencil
of seven nodes, i.e. three boundary ghost-point values would be needed behind the nest boundaries. However, in PALM, all
boundaries except cyclic boundaries are treated in a special way. Next to a boundary, the advection scheme is degraded such
that only one layer of boundary ghost nodes is actually needed. This is achieved by using the third-order Wicker and Skamarock
(2002) scheme for the second layer inside the domain and the first-order upwind scheme for the first layer of nodes next to the5
boundary. This has to be taken into account in designing of the interpolation procedure since the use of the first-order upwind
scheme may increase the flux-conservation error and compromise the accuracy around the nest boundaries. However, we can
reduce this additional flux-conservation error by using the following ’trick’. Instead of substituting the original parent-grid
values in the interpolation formulae (3) and (4), we replace them by values interpolated onto the flux points using a scheme
of higher than first order and substitute these values instead. As a result, the formally first-order upwind advection scheme10
becomes the selected higher order scheme if the local flow direction is into the domain. The main purpose of this ’trick’ is to
reduce the flux-conservation error. Ideally the fifth order scheme would be used for the flux-point interpolation. However, in
practice it is not possible to use any interpolation scheme using more than one grid point behind child-domain boundaries. The
reason for this is that the child has no information about the parent domain topography outside the first parent-grid layer behind
a child-domain boundary. An interpolation stencil reaching further away than this could penetrate a vertical wall leading to15
11
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
erroneous interpolation. Therefore, the only alternative to the 1st-order upwind interpolation is to use the simple average of the
parent-grid values both sides of the child-domain boundary. This leads to the 2nd-order central advection scheme. Obviously
it is different from the 5th-order scheme, but we argue that the difference between the fluxes computed using the 5th-order
and 2nd-order schemes is usually smaller than the difference between the fluxes computed using the 5th-order and 1st-order
schemes. Our numerical tests support this argument. On the top boundary, there is no topography and hence we can use wider5
interpolation stencil there. We ended up using the 3rd-order Wicker and Skamarock (2002) scheme because in our numerical
tests it yielded similar results as the more complicated more communication-intensive 5th-order scheme. It should also be
noted, that this trick improves the accuracy only on those boundary regions where the flux is into the child domain.
According to our experience, the conservation properties of an interpolation method are more important than its local ac-
curacy. Increasing the interpolation accuracy on the child boundary planes does not yield any evident benefit as the solution10
requires a development distance (i.e. a border zone) as it adapts to the changed resolution within the child domain. Therefore
the zeroth-order method has turned out to be fully sufficient and remains the only interpolation method implemented in PALM.
However, we have considered also an alternative interpolation approach for the advected variable based on tri-linear interpola-
tion with a specific reversibility correction. Although it is not implemented a short discussion is provided in Appendix B.
3.5 Anterpolation (child to parent)15
Anterpolation is used to feed the child domain solution back to its parent domain. Generally, anterpolation consists of filtering
the fine-grid child solution φi,j,k and mapping it to the parent domain grid. We select to employ the anterpolation scheme
proposed by Clark and Farley (1984), which consists of simple averaging over one parent-domain grid volume around the
parent grid node of the variable in question corresponding to top-hat filtering, viz.
b
φI,J,K =1
NI,J,K
i2(I)
X
i1(I)
j2(J)
X
j1(J)
k2(K)
X
k1(K)
φi,j,k .(11)20
The original parent solution ΦI,J,K is replaced by the anterpolated solution in the domain of overlap. Here, i,j,k and I, J, K
are the child- and parent-grid indices, respectively, and the hat is the anterpolation operator. The summation index limits, i.e.
the span of the anterpolation cell i1(I),i2(I),j1(J),j2(J),k1(K)and k2(K)are pre-computed during the initialization and
they depend on the grid configuration and the variable in question, i.e. the staggered velocity components have different index
limits than the grid-cell centered scalars. Note that for the velocity components, the anterpolation volume is reduced to the grid25
cell face on which the velocity component is defined. This means that the upper index limit in the direction of the velocity
component is reduced to the lower one, for instance i2=i1for u, because the coordinates of the velocity component node in
the respective direction in the parent and the child readily coincide, thus there is no need for anterpolation in this direction.
NI,J,K is the number of child domain values used for anterpolation at a given parent-grid location, and is pre-computed during
the initialization as30
NI,J,K = [i2(I)−i1(I) + 1][j2(J)−j1(J) + 1][k2(K)−k1(K) + 1] .(12)
12
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Note that due to the staggered grid, four sets of the index limits and NI,J,K are pre-computed and stored: one for each velocity
component and one for all scalars. Generally, the anterpolation cells can be spanned in more than one way. We define the
anterpolation cells similarly to Clark and Farley (1984). For scalar variables (non-staggered variables) the anterpolation cell
spans Xi±∆Xi/2where Xi(i= 1,2,3) are the coordinates of the scalar node in the parent grid. For the velocity compo-
nents (staggered variables), for example for u, the anterpolation cell spans X1, X2±∆X2/2, X3±∆X3/2, where Xiare the5
coordinates of the staggered u-node in the parent grid.
Buffer zones where the anterpolation is omitted are applied next to the child-domain boundaries except the bottom boundary.
The main purpose of the buffer zones is to avoid an unstable feedback loop between the anterpolation and interpolation. The
default width of these buffer zones is two prognostic grid nodes. The user may choose a different value for the buffer width,
but the minimum allowed width is one parent-grid spacing. This is because the layer of nodes nearest to the child boundary10
is directly used in the interpolation, and using an anterpolated value for interpolation leads to a strongly unstable behaviour.
The buffer zones are comparable to the relaxation zones applied in the nesting system of the WRF-LES model (Moeng et al.,
2007). In the WRF-LES nesting system the anterpolation is under-relaxed within these zones such that the under-relaxation
coefficient varies linearly across the relaxation zones which are five grid spacings wide. As mentioned in Sec. 3.3 the buffer
zone below the top boundary also reduces the overestimation of the horizontal velocity variances observed in zero mean-wind15
CBL tests in purely vertical nesting mode. According to these tests in purely vertical nesting mode, simulation results are not
particularly sensitive to the extent of the vertical downward shift of the upper edge of the anterpolation domain.
Canopy-restricted anterpolation
The anterpolation algorithm is implemented in the PALM model with a feature that enables its application in a spatially
selective manner such that the operation is only performed within the computational domain that is above a user-defined20
vertical threshold. This practice is discovered to resolve complications that arise when two-way coupled nesting is applied in
obstacle-resolved LES simulations where the anterpolated solution within the obstacle canopy introduces discrepancies in the
coarser parent solution. Thus we label this approach canopy-restricted (CR) anterpolation and the coupling is referred to as
two-way CR, for short. The necessity of this anterpolation strategy is motivated and its effectiveness demonstrated in Sec. 4.2.3
where nesting is applied to obstacle-resolved LES test case.25
4 Numerical experiments
In order to evaluate the nesting strategy, to show its benefits and point out its limits, we performed a series of nested model sim-
ulations for different grid-spacing ratios and respective non-nested reference simulations for different atmospheric situations.
For this purpose, we simulated a homogeneously-heated flat-terrain convective boundary layer as well as a purely shear-driven
flat-terrain boundary layer. Further, to investigate the performance of the grid nesting in more complex situations where non-30
flat topography is present, we performed simulations for a neutrally-stratified flow over a smooth three-dimensional hill and
will compare the results against wind-tunnel data. Second, we simulated a neutrally-stratified urban boundary-layer flow over
13
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
a regular staggered arrangement of building cubes, and will compare the nested simulation results to corresponding non-nested
fine- and coarse-grid simulation results. Finally, we will demonstrate the applicability of a two-stage nesting for the same flow
over the cube array. Details concerning the different simulation setups are given in their respective sections. Please note, for
the sake of simplicity velocity components will hereafter be addressed by lower case variable names only, no matter if it refers
to the flow in the parent or the child domain.5
4.1 Convective boundary layer
The nesting method is first evaluated for a pure convective boundary layer (CBL) with zero mean wind. We set up one child
domain that is centered within the parent domain. For the root domain, cyclic lateral boundary conditions were set. A ho-
mogeneous and time-constant surface sensible heat flux of 0.1Kms−1was prescribed. The simulation was initialized with a
potential temperature profile that increases linearly with height at a lapse-rate of 0.3 K/100 m. The root-model domain size is10
10.2 km ×10.2 km ×3.0 km in the x-, y- and z-directions, respectively, with an isotropic grid spacing of 20 m. The top of the
child domain is set to be within the middle part of the CBL, and the domain size is 2.5 km×2.5 km×0.48 km in the x-, y- and
z-directions, respectively, with an isotropic grid spacing of 10 m, resulting in a grid-spacing ratio of 2. In order to examine how
turbulence statistics behave for different grid-spacing ratios between parent and child in the CBL, we additionally run nested
simulations with grid-spacing ratio of 3 and 4 by increasing the isotropic grid spacing in the parent domain to 30 m and 40 m,15
respectively. Non-nested coarse- and fine-grid reference simulations were carried out corresponding to the nested simulations
with different grid-spacing ratios. The simulated time was 4 hours for all convective cases. Data analysis started after 2 hours
of simulated time when model spin-up effects are not present any more and the simulations reached steady-state conditions. In
order to perform a spectral analysis of time-series data, the time step was held constant at 1.0 s in all convective simulations
during the data analysis period.20
Figure 4a) shows an instantaneous horizontal cross-section of the w-component at a height of 40 m for the parent and
child (overlaid) domains for the grid-spacing ratio 2. A hexagonal pattern of convective cells with strong updrafts and weaker
downdrafts is visible, as it can be typically observed in LES. The transition between parent and child appears smooth and the
flow structures are continuous in terms of shape and amplitude, while within the inner part of the child domain more fine-scale
structures can be observed with slightly stronger up- and downdrafts, as also reported by Moeng et al. (2007). Furthermore,25
Fig. 4b), showing an instantaneous vertical cross-section for the w-component, also depicts how the up- and downdrafts are
consistently maintained across the child boundary without any obvious impact on the turbulent structures.
Figure 5 shows horizontally- and time-averaged vertical profiles of potential temperature θ, vertical turbulent heat flux
hw0θ0i, variances of horizontal and vertical velocity components, as well as the skewness of the vertical velocity component
w, being one of the most grid sensitive quantities (Sullivan and Patton, 2011). The profiles of hθiindicate a well mixed CBL.30
With increasing grid-spacing ratio the corresponding parent and coarse grid simulations deviate from the fine grid reference,
particularly near the surface and within the inversion layer, while the child results adhere well with the non-nested fine-reference
simulation, indicating that the profiles of hθiin the child domains are rather independent of the parent grid for the employed
grid spacings.
14
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 4. Instantaneous horizontal a) and vertical b) cross section of wafter 4 h of simulated time for the grid-spacing ratio case 2. The
horizontal cross-section is give at a height of 40 m. The black box indicates the lateral and top boundaries of the child domain. The white
line indicates the y-position of the vertical cross section of wshown in (b). The vertical axis in (b) is normalized with the horizontal mean
boundary-layer depth zi. Note that only part of the parent domain is shown for the sake of visibility.
15
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 5. 30-min time and horizontally averaged profiles of a) hθi, b) hw0θ0i, c) a close-up view of hθiand hw0θ0i, d) variance of the
horizontal velocity components, e) variance of the vertical velocity component, and f) skewness of the vertical velocity component after
4 hours of simulated time. Please note the second upper abscissa in c). Profiles are shown for the grid-spacing ratio of 2, 3, and 4 for the
respective child domains, indicated by the respective numbers. The corresponding profiles from the coarse-grid reference simulations for the
20 m, 30 m, and 40 m grid spacing are indicated the same. For the sake of clarity, the resulting profiles for the parent domain are only shown
for the grid-spacing ratio of 2. Squared brackets indicate a horizontal domain average.
The heat flux profiles in the child and parent simulations decrease linearly with height within the CBL and are in good
agreement with the fine reference simulation. For the parent simulation we note the near-surface kink in the heat flux (see
Fig. 5c for a close-up view). Moeng et al. (2007) observed a similar kink in the heat flux and attributed it to inaccuracies
in the statistical evaluation of the heat flux, more precisely, to errors that arise from interpolation from a mass- to a height-
coordinate system. However, to evaluate fluxes PALM does not apply any interpolations but uses directly the resolved- and5
subgrid-scale fluxes as calculated in the advection scheme and the subgrid model, respectively, so that interpolation errors
cannot explain the kink in this case. Instead, we attribute the kink in the parent domain to the anterpolation from the fine child
16
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
solution. In simulations with different vertical grid spacing, the vertical gradients of hθiwithin the unstable near-surface layer
are differently resolved, resulting in slightly different near-surface temperatures, as it can, e.g., be observed between the fine
and coarse reference simulations in Fig. 5a. This indicates that the parent simulation will yield slightly different hθi-profiles
than the child simulation. After the anterpolation is performed, the parent solution is replaced by the underlying child solution,
where the near-surface vertical gradients of hθiin the parent domain partly deviate from the ones the model would create5
without feedback from the child domain, i.e. the near-surface hθi-profile in the parent is not in equilibrium with the applied
surface boundary condition any more. In the following time step the parent model tries to re-adjust the post-inserted hθito the
vertical gradients as being present without feedback from the child, altering the heating rates and thus the near surface vertical
gradients of the heat flux, which in turn becomes visible as near-surface kink. In fact, we verified this hypothesis in a test case
by using identical vertical grid spacing in parent and child. In this case, no kink in the vertical heat flux was visible any more10
(not shown).
The variances of the horizontal and vertical velocity components, as well the skewness of the vertical component, depend
strongly on the grid spacing as the coarse- and fine-grid reference simulations show, where the variances (skewness) become
smaller (larger) for increasing grid spacing. The parent simulation agrees well with the coarse-resolution simulation, indicating
that the anterpolation changes the parent flow field only marginally. The variances and skewness in the child simulations agree15
with the fine-reference profiles, except for the upper regions of the child domain where the variances are slightly overestimated.
The child profiles are almost independent of grid-spacing ratio, and are close to the reference simulation profile. This indicates
that the child solutions are almost independent on the chosen grid-spacing ratio in the studied cases.
Although there can be no mean horizontal advection in the zero-mean wind CBLs, spatially and temporally local horizontal
advection always takes place and therefore flow structures are advected locally from parent to child (and vice versa). Therefore,20
advected flow structures may need a certain fetch to adjust to the changed grid spacing. In order to get an idea of how much
distance apart from the lateral child boundaries is required to observe similar turbulence properties as in a non-nested fine
resolution reference simulation, we performed a spectral analysis. Therefore, we sampled time series of TKE and θat different
locations within the child domain and calculated frequency spectra from the sampled time series. Subsequently, we averaged
spectra over all sampling locations with the same distance from the lateral child boundaries. It should be noted that transforming25
frequency spectra into wave number spectra using Taylor’s hypothesis in order to directly link spectral information and grid
spacing is not strictly correct in this case where we have no background wind; nevertheless we will assume that frequency
and wave number space are connected, i.e. large frequencies belong to small spatial scales and vice versa. Fig. 6 shows the
resulting frequency spectra, as well as corresponding spectra from fine- and coarse-grid reference simulation. As expected, the
coarse-resolution spectra exhibit less spectral energy at larger frequencies, compared to the child- and fine-resolution spectra.30
This is due to the larger filter length assumed for the subgrid model, removing more energy at larger spatial scales and thus also
affect smaller frequencies. The child-spectra agree well with the fine-reference spectra, especially for the grid-spacing ratio
case 2 where even locations close to the lateral boundaries show good agreement with the reference. We attribute this to the
nature of the CBL, where turbulence is mostly produced locally by buoyancy and horizontal advection is almost negligible,
so that turbulence is almost not affected by any transport from the boundaries. Also for the grid-spacing ratios of 3 and 435
17
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
the differences are generally small, but locations close to the lateral boundaries show slightly smaller spectral densities at
larger frequencies, indicating that for larger grid-spacing ratios the flow needs a fetch of a few tens of meters in a purely
buoyancy-driven boundary layer to adjust to the finer grid spacing.
Even though the child simulations yield turbulence profiles, spectra and instantaneous flow patterns similar to the fine-grid
reference simulation for a pure-buoyancy driven flow, the nested simulation nevertheless creates side effects on the flow which5
appear as a secondary circulation (SC). This SC is not caused by a violation of mass conservation that has been discussed
in Section 3.4. Figure 7 shows the 5-hour time-averaged w-component at the middle part of the CBL in a homogeneously-
heated nested simulation. In order to compute the 5-hour time average, we continued the simulation with grid-spacing ratio
2 for further 3 hours. Within the region of the nested child domain a mean updraft can be observed, which is in the range of
0.4−0.9ms−1and extends throughout the entire depth of the CBL (not shown). At the child domain boundaries and outside10
the child-domain region the flow subsides in average, and horizontally directed branches at the upper and lower parts of the
CBL occur, giving the overall picture of a SC. The strength of this SC, indicated by the amplitude of the mean updraft, is in the
order of the strength of SCs observed in previous simulations over idealized stripe-like surface heterogeneities (Sühring et al.,
2014) and even exceeds the strength of SCs observed in simulations over realistic surface forms (Maronga and Raasch, 2013).
SCs develop above surface heterogeneities mainly due to differential surface heating of the air, resulting in mean updrafts15
and downdrafts over the stronger- and less-heated patches, respectively. However, since we prescribe the same surface sensible
heat flux in the parent as well as in the child simulation, differential surface heating cannot be the reason of the SC in the
nested simulation. Moeng et al. (2007) observed a temperature bias in their child domain that led to mean vertical motion to
compensate the temperature bias. They observed temperature biases that go either way, i.e. a too cold or a too warm child
domain, which they attributed to a nested child domain of too small horizontal extent. If only a few up- or downdrafts are20
resolved in the child domain, the vertical transport is dominated by these up- or downdrafts and thus a warmer or cooler
CBL can be quickly produced in the child domain, respectively. They showed that for larger horizontal child domain size the
temperature bias and thus the associated vertical motion vanished. However, they only considered instantaneous differences
between parent and child, meaning that the temperature bias is a result of insufficient sampling of the large up- and downdrafts
rather than an inherent feature of the nesting which can only been observed after time-averaging. In our case the SC becomes25
visible only after considerable time-averaging. The updraft branch of the secondary circulations is always located within the
child domain also for larger child domain extensions. We hypothesize that this SC is triggered by a slightly different divergence
of the vertical heat flux between the region occupied by the child domain and the remaining parent domain due to different grid
spacing. It might be impossible to eliminate, because higher resolution better represents the turbulent mixing, so differences
between the parent and the child solutions are to be expected in general.30
Even though this inherent artificially-induced SC only appears when the flow is averaged over a longer time under quasi-
stationary conditions (no daily cycle, no change in the mean wind, etc.), nested simulation results should be interpreted carefully
in terms of SCs. In particular, since the strength of the artificial SC is in the order of ’real-world’ circulations over heterogeneous
terrain, these may superimpose each other, altering the pattern of the vertical transport of sensible and latent heat. Although
18
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 6. Frequency spectra of the TKE (left column) and θvariances (right column) at different distances from the lateral child boundaries
for the grid-spacing ratio of 2 (a,b), 3 (c,d), and 4 (e,f). Furthermore, spectra for fine- and corresponding coarse-grid references simulations
are displayed. TKE and θwere sampled at z= 120m.
19
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
we did not succeed to proof our hypothesis, we encourage other researchers to look for the existence of such SCs in any nested
models by analyzing the time averaged results.
Figure 7. Horizontal cross-section of 5-h time-averaged vertical velocity at z= 400m in a nested simulation with grid ratio of 2. The black
box indicates the location of the child domain.
20
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
4.2 Neutrally stratified boundary layer tests
Initialization and inflow conditions
As further test cases, we set up boundary layer flow simulations with increasing order of complexity. First, to evaluate the
performance of grid nesting in shear-driven boundary-layer flows, we simulated a flow over a homogeneous flat surface in
order to compare first and second order moments from a nested simulation against reference simulations. In a second step, we5
simulated a flow over a smooth three-dimensional hill for comparison of nested simulation results against wind-tunnel data.
Finally, in order to illustrate the advantages of the grid nesting in more complex setups, we simulated a flow over a staggered
arrangement of cubes mounted on a flat surface.
The parent domain size for all neutrally-stratified simulations was Lx×Ly×Lz= 5.1×1.5×0.32 km3in the x−,y−,
and z-directions, respectively. In all neutral simulations we prescribed a homogeneous roughness length of z0= 0.01 m. At the10
top boundary we applied a free-slip condition for the horizontal wind components and zero vertical motion. At the spanwise
lateral boundaries (north and south boundary) we applied cyclic conditions. At the western lateral boundary (hereafter referred
to as inflow boundary) we prescribed mean inflow profiles for the uand vcomponent, obtained from a cyclic precursor run.
Two different precursor simulations were employed for the subsequent test cases. The one used for the flat surface and for the
smooth hill featured a geostrophic wind of ug= 4.8 ms−1and vg=−1.3 ms−1at a latitude of 55 degrees, adjusted such that15
the surface-layer mean flow became parallel with the x-axis. This precursor simulation ran for 36 hours to reach a stationary
state. The second precursor simulation, used for the cuboid case, was driven by a fixed pressure gradient angled to result in a
mean flow of u= 10 ms−1at z=Lzwith a 3 degree angle from the x-axis.
In order to obtain a turbulent inflow, we applied a turbulence recycling method according to Kataoka and Mizuno (2002),
where the inflow mean vertical profiles of uand vare superimposed by turbulent fluctuations sampled at a recycling plane,20
which is placed at xrc = 1.5 km downstream the inflow boundary. The recycling plane is placed sufficiently far apart from the
inflow boundary to allow for statistically-independent turbulence, but also sufficiently far apart from the location of the child
domain to avoid any feedback between the grid nesting and the inflow conditions. For further details on the implementation of
the turbulence recycling method see Maronga et al. (2015).
Further, in order to avoid persistent streaks in the u-component, which may develop in neutrally-stratified flows and will25
be recurrently recycled in case of vanishing v-component, we shifted the recycled turbulent signals along the y-direction at
the inflow boundary, following Munters et al. (2016). At the eastern outflow boundary we set a radiation boundary condition
(Miller and Thorpe, 1981). The root domain was initialized with three-dimensional data recursively copied from the precursor
run, while the child domain was initialized with data obtained from the parent. We used an isotropic grid spacing of 4 m and
2 m within the root and the nested child domain, respectively. The cuboid case also encompasses a third domain with 1 m30
resolution (two-stage nesting).
In order to evaluate the effect of the nesting, we performed additional non-nested reference simulations with 4 m and 2 m
grid spacing. The simulated time of the neutrally stratified simulations ranged from 4 to 7 hours. Data analysis started after 2
hours of simulated time. When spectral analysis was performed, the time step was held constant at 1.0 s for that simulation.
21
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
4.2.1 Neutrally stratified boundary-layer flow over flat terrain
Figure 8 shows an instantaneous horizontal cross-section of the u-component for the nested simulation. As typical for a neu-
trally stratified boundary layer, elongated streak-like structures can be observed (Hutchins and Marusic, 2007; Hutchins et al.,
2012). These elongated structures preserve their size and amplitude when entering the child domain from the left and exiting
to the right.5
Figure 8. Instantaneous horizontal cross-section of the u-component in ms−1at z= 40m. The black box indicates the location of the child
domain. The white solid lines indicate the x-locations where the profiles shown in Fig. 10 are averaged over the y-direction.
Figure 9 shows horizontal profiles of the time- and y-averaged friction velocity u∗within the child domain and the corre-
sponding coarse- and fine-grid reference cases. In the coarse and fine reference cases u∗is constant along the x-axis indicating
that the flow is in equilibrium with the surface friction. In the coarse-grid simulations u∗shows slightly higher values compared
to the fine-grid reference simulation, even though the prescribed surface roughness is identical in all simulations. This suggests
that the flow in the coarse-grid simulations sees a slightly rougher surface, which we attribute to the less accurate representa-10
tion of the vertical near-surface gradients of the wind profile compared to the fine-grid simulation. When the flow enters the
child domain, the coarse-grid inflow wind profile is not in equilibrium with the surface friction any more and the near-surface
flow decelerates, indicated by the higher values of u∗near the child inflow boundaries. With increasing distance to the inflow
boundary, u∗rapidly decreases and reaches a minimum with lower values compared to the reference cases, until it increases
again reaching a secondary maximum and then asymptotically approaches a constant value, which is similar to the value of the15
fine-grid reference case in the grid-ratio case 2 and 3. However, at least for the given model domain size, u∗does not approach
the fine-grid solution in grid-ratio case 4 but still exhibits higher values. This kind of spatial oscillation of u∗, which indicates
an alternating deceleration and acceleration of the near-surface flow along the x-direction, shows that the surface-momentum
exchange in the child domain needs a sufficiently large development length. For grid-ratio case 2 the required fetch length is
22
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
of at least 1km to adjust to the fine-grid resolution. With increasing grid ratio the amplitude of the spatial oscillation increases
and the fetch length becomes longer, and, as in grid-ratio case 4 even exceeds the model domain size.
Figure 9 shows that u∗gradually adjusts to the fine-reference value, at least for grid ratio case 2 and 3. This is in contrast
to Moeng et al. (2007), who revealed a friction velocity bias between parent and child in their neutrally-stratified simulation
when employing grid-size dependent SGS model as it is also used in this study.5
Figure 10 shows time- and y-averaged profiles of the horizontal wind speed within the child domain for different grid ratios,
taken at different distances downstream of the inflow boundary, indicated by the white solid lines in Fig. 8. At a distance
of 100 m the profiles in the child domains agree well with the fine-reference profile within the lowest 10 m. Even though
the surface-momentum exchange is still not in equilibrium at that position (see Fig. 9), one could already conclude from the
near-surface wind profiles that the flow has already been adapted to the finer grid resolution. However, further above, the wind10
profiles of the child model still deviate from the fine-reference solution and are closer to the coarse reference profiles. This
is especially obvious for the grid-spacing ratio case 4, where the wind profile shows a discontinuity at a height of z= 20 m.
With increasing distance from the child inflow boundary, the child-profiles gradually adjust to the fine-reference simulation,
while at a distance of 2000 m the child profiles agree with the fine-reference solution, except for the grid-ratio case 4 which
still deviates from the fine-reference solution.15
In order to further analyze the flow adjustment within the child domain, we computed resolved-scale turbulent kinetic energy
(TKE) spectra at different distances from the child inflow boundary. The spectra were calculated from time-series of the three
velocity components that were sampled at different locations within the domain. The final spectra were then obtain by averaging
individual spectra over all locations with identical distance to the inflow boundary, assuming that the flow is parallel to the x-
axis. Figure 11 shows TKE spectra obtained from the child domain and for the corresponding reference simulations. At low20
frequencies (large wave numbers), the spectra look quite similar and no obvious differences to the fine- and coarse-reference
spectra can be observed, indicating the grid nesting does not induce any larger-scale oscillations which propagate through the
model domain. At higher frequencies, however, especially the near-inflow boundary child spectra differ to the fine-reference
spectra and resemble more the corresponding coarse reference spectra. With increasing distance from the inflow boundary, the
spectral properties gradually adjusts to those of the fine-reference case, while at a fetch of 500–1000 m almost no differences25
can be observed any more at that height level.
In contrast to a buoyancy-driven boundary layer, the flow in a purely shear-driven boundary layer requires a sufficiently
large development distance to adjust to the finer grid resolution in terms of spectrally similar conclusion. However, a purely
shear-driven flow over a flat homogeneous surface can certainly be considered as an extreme case in terms of flow adjustment,
as the vertical turbulent exchange, which is primarily driven by surface-roughness induced shear, is rather low compared to less30
idealized flows over non-flat terrain or with obstacles included. Hence, we expect that the required fetch length may decrease
for rougher surfaces and more complex surface geometries.
23
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 9. Two-hour time- and y-averaged horizontal profiles of the friction velocity for the grid-spacing ratio of 2, 3, and 4, as well as the
corresponding coarse- and fine-grid reference simulations.
Figure 10. Two-hour time- and y-averaged profiles of the horizontal wind speed for the grid-spacing ratio of 2, 3, and 4, taken at different
distances downstream of the inflow boundary, indicated by the white solid lines in Fig. 8. Also, corresponding time- and y-averaged profiles
from the fine and coarse reference simulations taken at the same locations are shown.
24
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 11. Frequency spectra of the resolved-scale TKE taken at different sampling locations downstream of the inflow boundary for the
neutrally-stratified boundary layer at z= 40m, for grid-spacing ratio a) 2, b) 3, and c) 4. TKE-spectra for the fine- and the corresponding
coarse-grid reference simulations are also shown.
25
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
4.2.2 Neutrally stratified boundary-layer flow over a smooth three-dimensional hill
The hill case was setup to compare flow statistics against wind-tunnel observations conducted by Ishihara et al. (1999), who
sampled data at different up- and downstream locations along the centre hill axis. The terrain height of the smooth three-
dimensional hill is given by
z(x,y) = Hcos2 πp(x−x0)2+ (y−y0)2
2l!,(13)5
with the hill height H= 40m and a hill radius l= 100 m, while xand yindicated the location on the discrete grid and x0and
y0the location of the hill top. Please note, with respect to the wind-tunnel experiment, we up-scaled the hill dimension by a
factor of 1000.
Figure 12 shows the mean flow field along the centerline of the three-dimensional hill for the nested as well as fine and coarse
reference simulation. Upwind of the hill the mean flow in the nested simulation agrees with the one in the fine and coarse10
reference simulation. Also in the lee of the hill the flow field in the nested and fine reference simulation agree, both showing
a re-circulation that extends up to about 3.75Hdownstream of the hill top, while in the coarse reference simulation the re-
circulation extends farther downstream up to about 4.1H. Figure 13 and 14 show the corresponding standard deviations of the
u- and w-component sampled at different locations along the centerline of the hill. Upwind of the hill the standard deviations
of the u- and w-component agree with the observations. However, leeward of the hill at 1.25H, the LES underestimates15
the standard deviation of the u- and w-component, which is most pronounced in the coarse reference simulation. Further
downstream, the coarse reference run still slightly underestimates the observed standard deviations, while the nested and fine-
reference simulation slightly overestimate the standard deviations, which is in agreement with results from the EPFL-LES
model presented in Diebold et al. (2013) who employed a similar grid resolution of the hill. The standard deviations from
the nested and fine reference simulation agree, showing only marginal differences among each other. Considering that the20
hilltop is placed only about 300 m apart of the child domain inflow boundary, this indicates that in more complex setups where
topography is present the adjustment fetch can become significantly smaller compared to purely flat terrain as discussed in
Sect. 4.2.1.
26
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
a
b
c
Figure 12. 2-hour time-averaged vertical cross-section of the u-component (colored contours) as well as the mean flow field (vector arrows)
displayed along the centerline of the three-dimensional hill for a) the nested child domain, b) the 2m reference simulation, and c) the 4m
reference simulation. Vector arrows as well as the u-component are normalized with the reference wind speed taken at z=Hupwind of the
hill. The ordinate and the abscissa are scaled with the hill height H. Note, the abscissa is centered at the hill top. The black vertical lines
indicate the positions of the profiles displayed in Fig. 13 and 14.
27
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
a
b
c
Figure 13. Two-hour time-averaged vertical profiles of the standard deviation of the u-component for the LES and well as the observed wind
tunnel flow, for a) the nested child domain, b) the 2m reference simulation, and c) the 4m reference simulation. The ordinate is scaled with
the hill height H. The standard deviation is normalized with the reference wind speed taken at z=Hupwind of the hill. The black dashed
horizontal lines indicate the discrete height of the surface at the sampling location.
28
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
a
b
c
Figure 14. Two-hour time-averaged vertical profiles of the standard deviation of the w-component for the LES and well as the observed wind
tunnel flow, for a) the nested child domain, b) the 2m reference simulation, and c) the 4m reference simulation. The ordinate is scaled with
the hill height H. The standard deviation is normalized with the reference wind speed taken at z=Hupwind of the hill. The black dashed
horizontal lines indicate the discrete height of the surface at the sampling location.
29
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
4.2.3 Neutrally stratified boundary layer over a regular array of cubes
The final test case features a neutral atmospheric boundary layer flow over flat terrain which becomes incident with a staggered
pattern of cubical obstacles. The resulting flow scheme resembles urban canopy turbulence where the interaction between
roughness elements and ABL turbulence is primarily resolved. Here, the cubical obstacle height is H= 40 m. The distance
between the obstacles is 3Hin the x-directions and 1Hin the y-direction.5
To demonstrate the flexibility of the nesting implementation, we carried out simulations with two different nested configu-
rations illustrated in Fig. 15. The first (v1) case features a single child domain while the second (v2) case contains a two-stage
nesting system where a second child domain is nested within the first. In the latter configuration, the first child acts as a parent
for the second child domain. The isotropic grid spacing is 4 m in the root domain, 2 m in the second level nest (first child) and
1 m in the third level nest (second child). (Note, that the implementation does allow child locations to be selected such that10
their domain boundaries intersect with the obstacles.) The two example configurations represent nesting applications designed
to meet different levels of accuracy demands. The v2 configuration is set to resolve the transition effect at the leading edge
of the cube canopy and to capture the blunt-body wake interactions in sufficient detail within the center region of the cuboid
canopy.
First, in the context of obstacle-resolved LES, we motivate the employment of an optional canopy-restricted (CR) anterpo-15
lation strategy introduced in Sec. 3.5. For this purpose, consider Fig. 16 showing an instantaneous horizontal cross-section of
vorticity vector magnitude at z= 0.9Hheight for configuration v2. The image is focusing on a region where all domains with
different resolutions are visible.
The visualization indicates the strength and spatial structure of the resolved turbulent eddies and how they are affected by
grid resolution. The differences are significant. In such obstacle-resolving LES, the increased grid resolution has the ability20
alter the flow solution to such a degree that the anterpolation introduces details to the coarser parent which are inconsistent with
the rest of the parent’s flow solution. Particularly with blunt-body obstacle canopy flows, this discrepancy is clearly manifested
as a locally changing resultant pressure drag (caused by the obstacles) within the anterpolated domain. To inspect this, we
compute the resultant pressure drag coefficient
CF p =2
ρU2
ref Sref F2
p,x +F2
p,y1/2(14)25
for the differently coupled simulations. In Eq. 14 uref =hui|z=1.25His the reference wind speed, Fpis the resultant pressure
force exerted on the cubes obtained by integrating the pressure over vertical walls, ρis the density of air and Sref is the
accumulated frontal area of the cubes. The results are listed in Table 1, which makes evident the drastic difference between the
values for the coarse reference and the two-way coupled parent (CFp [Coarse] vs. CF p[Root]: two-way). This large difference
arises as the anterpolated solution within the obstacle canopy introduces a large-scale disturbance to the parent solution giving30
rise to unphysical secondary effects. These effects, in turn, lead to complicated feedback systems in the two-way coupled
solutions whose realizations become depended on the chosen nesting configuration.
30
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 15. An overview of the cubical-obstacle case layout. The obstacles are cubes with 40 m sides. The figure displays two nested
arrangements: version 1 (v1) featuring a root domain and a secondary nest domain whereas version 2 (v2) also includes a tertiary nest
domain embedded within a larger secondary nest. The root and nested domains are indicated with (?),(??), and (???)respectively in the
upper left-hand-corner of each domain. The first child domain is displayed with white background for better visualization.
This problematic behavior is significantly abated by adopting the CR anterpolation strategy setting here the vertical threshold
at 1.25Hvia experimenting. This CR anterpolation allows the parent and child flow fields to become strongly coupled while
minimizing global inconsistencies in the parent solution. While all the child domain solutions over-predict the pressure drag,
the two-way CR solution yield CF p[Child 2] values that are closes to the fine reference.
To further evaluate the nesting performance, we exploit root (normalized) mean square difference (RNMSD or RMSD)5
and fractional bias (F B) as comparison metrics (see, Britter and Hanna, 2003) evaluated over successive xy-planes to assess
the effectiveness of the nesting approach in obstacle-resolving LES cases. RNM SD and RMSD provide a measure of mean
difference that is composed of random scatter and systematic bias whereas the fractional bias (F B) yields a specific measure
31
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 16. Instantaneous close-up view of vorticity magnitude (s−1) on xy-plane at elevation z= 0.9Hfor the v2 case with two nested
domains. Black lines indicate the bounds of the first and the second nest. Note, only parts of the domain extents are displayed.
Table 1. Resultant pressure force coefficients CF p evaluated over Child 1 (??)domain shown in Fig. 15 (v1). Results for parent and child
solutions are reported for one-way, two-way and two-way canopy-restricted (CR) methods, where the latter is a modified two-way coupling
approach where the anterpolation is restricted (i.e. not allowed) within the obstacle canopy.
Version 1 Version 2
one-way two-way two-way CR one-way two-way two-way CR
CF p[Root] 0.592 0.735 0.549 0.790 1.017 0.785
CF p[Child 1] 0.602 0.599 0.594 0.828 0.859 0.826
CF p[Fine] 0.583 0.808
CF p[Coarse] 0.592 0.790
for the systematic bias between the two solutions. The metrics are defined as
RN M SD (ψ) = v
u
u
tDψ−ψRef2E
ψψRef or RMS D (ψ) = rDψ−ψRef2E(15)
F B (ψ) = ψ−ψRef
0.5ψ+ψRef.(16)
32
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
where ψis a generic prognostic variable from the considered case, while ψRef refers to the value from the reference simulation
with 2 m resolution. RMSD is used instead of RN M SD in cases where the product of double-averaged quantities used
for normalization approaches zero. Similarly, FB is only evaluated for the streamwise velocity component because other
components yield a near-zero denominator which contaminates the metric. The evaluations are performed for 15 xy-planes
within the child domain (zone (??)in Fig. 15) which are equally spaced over the range 0≤z/H ≤1.5. We have excluded 1285
m and 64 m wide development zones at the boundaries in the xand y-directions respectively. When the coarse (4 m resolution)
reference solution is compared to the fine (2 m resolution) reference, the coarse solution is interpolated onto the fine grid before
the comparison metrics are evaluated.
Both model variants v1 and v2 are included in the analysis to demonstrate how the size and placement of the child domains
effect the metrics and also to illustrate the possibility to employ a cascade of nested domains. Although no comparison metrics10
are presented for the second child solution featuring 1 m resolution, its influence is embedded in the solution of the first child.
The RNMSD and RM SD profiles for the velocity components and their variances depicted in Figs. 17 and 18 lay bare
the effectiveness of the presented nesting system and reveal the added benefit of the CR anterpolation. While all the coupling
approaches succeed in significantly reducing the discrepancy compared to the fine reference, the conventional two-way cou-
pling exhibits the most pronounced level of deviation. The F B results in Fig. 19 indicate that the two-way coupled solution15
also contains the most systematic deviation, which is conform with the pressure drag results.
The one-way coupling approach performs consistently better than the unmodified two-way coupled in all metrics, but it is
also associated with a systematic bias that is larger than the value by coarse reference. However, if the modest systematic shift
in streamwise velocity can be accepted, the one-way coupling offers a cost-effective nesting coupling approach (see Sec. 4.3
for performance measures). Nonetheless, the results conclude that the introduced CR anterpolation approach presents the most20
recommended coupling strategy for obstacle-resolving LES as it provided the best metrics in every category.
33
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 17. Vertical distributions of root (normalized) mean square difference (RNMSD or RMSD) of velocity components for configu-
ration v1 a-c) and configuration v2 d-f). The metrics are evaluated for 15 horizontal planes between the range 0≤z/H ≤1.5.
34
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Figure 18. Vertical distribution of RNMS D of the horizontal velocity variances for configuration v1 a-c) and configuration v2 d-f). The
metrics are evaluated for 15 horizontal planes between the range 0≤z/H ≤1.5.
Figure 19. Fractional bias (F B) values evaluated over 15 xy-planes between the range 0≤z/H ≤1.5, for a) v1 and b) v2.
35
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
4.3 Performance issues
Table 2 gives an overview of the consumed CPU time in the nested as well as fine- and coarse-grid reference simulations
for the hill and the cube case simulations. As a rule of thumb, doubling the resolution leads to an increase in CPU time
by approximately a factor of 16 (when the numerical time step is determined according to the CFL criterion). This can be
observed comparing the coarse- and fine-grid reference simulations. Compared to the fine-grid reference simulations, the5
nested simulations consumed significantly less CPU time (up to 80% reduction) while increasing the computational cost by
factors of 3.8 and 3.4 in hill and cube canopy cases compared to the coarse reference. Although these factors depend on the
child domain size, these test make evident that the nesting technique can significantly reduce the computational cost, while
yielding results that closely adhere to the non-nested fine-resolution simulation.
Due to the inter/anterpolation and the accompanied inter-model data transfer, the nesting itself consumes CPU time. In our10
tests the workload with respect to the number of grid points treated by a processor element was equal among the parent and the
child simulation. With this optimal configuration, the two-way nesting consumed about 10-16% of the CPU time in our tests,
while it consumes only about 2% in the one-way nesting. This suggests that most of the CPU time taken by two-way nesting
is consumed in the anterpolation and the associated child to parent data transfer.
Please note, if the workload between child and parent processes is not well balanced, the faster processes need to wait before15
the data-transfer can start until the slower processes reach that point, reducing the computational efficiency of the nesting.
Table 2. Required CPU time for the neutrally stratified and the convective test case.
Case CPU time (×102h) Overhead nesting
Hill coarse 2.2 –
Hill fine 39.1 –
Hill nest 8.31 16%
Cubes coarse 8.55 –
Cubes fine 129.0 –
Cubes v1 nest one-way 25.8 2.1%
Cubes v1 nest two-way 29.1 12%
5 Conclusions and future outlook
This article documents and evaluates an online LES-LES nesting scheme implemented into the PALM model system 6.0. The
nesting system relies on the post-insertion approach and features both one-way and two-way coupling approaches. We give a
detailed description of the model’s relevant technical, algorithmic and numerical aspects and provide evidence for the accuracy20
gains the method introduces with a dramatically reduced computational cost compared to globally refined grid resolution.
Particularly in urban boundary layer studies requiring obstacle-resolving LES, the nesting approach has proven essential.
36
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
The implementation of this three-dimensional nesting system is based on two-level parallelism involving inter-model and
intra-model parallelization using MPI. This enables our nesting implementation to flexibly support multiple child domains
which can be nested within their parent domain either in a parallel or recursively cascading configuration. All solutions in-
volved within the nested simulation are advanced using a globally synchronized time step whereas the coupling between each
parent/child pair is performed with interpolation (parent to child) and anterpolation (child to parent) operations.5
The nesting method is evaluated by performing a series of numerical experiments with an objective to demonstrate that the
refined child solution (nested within a coarser parent) approaches the non-nested reference solution obtained by employing fine
resolution globally.
The first test case features horizontally homogeneous convective boundary layer (CBL) with no mean mean wind. In this
case, first and second order boundary-layer statistics are well captured in the child domain and are closely comparable to non-10
nested high-resolution reference statistics. Further, due to the local nature of turbulence production and the weak advection
from parent into the child, the flow statistics show almost no dependence on the distance to the child boundaries. However, in
case of several hours long averaging times we found that a nonphysical secondary circulation develops although the surface
heating is homogeneous. We hypothesize that this secondary circulation is an inherent consequence of the spatially changing
description of flow physics in the parent and child solutions. This should be kept in mind when applying the nesting system to15
CBL-problems.
The second test case simulated neutrally stratified boundary layer flow over flat terrain. The nested simulations reveal that the
flow solution within the child domain must undergo a development phase, as the flow solution adjusts to the higher resolution,
before reaching equilibrium state again. The required development length depends on the grid-spacing ratio between parent
and child. However, a purely shear-driven flow over a homogeneous flat terrain can be considered as an extreme scenario with20
respect to the development length of turbulence, while in cases with more complex surface geometry the flow adapts within
shorter development distances. Beyond the development distance, the child solution for grid-spacing ratios of two and three
agree well with the non-nested fine-reference solution, but in case of grid-spacing ratio of four the results clearly deviate from
the fine-reference solution.
The third numerical experiment featured boundary layer flow (similar to second test case) over a smooth three-dimensional25
hill. This test case also exploits wind tunnel measurements to strengthen the nesting model evaluation. In this case, the flow
statistics in the windward and the leeward part of the hill are almost the same as in a fine-reference simulation and agree well
with wind-tunnel observations presented in Ishihara et al. (1999).
The final test case examines a flow system where a fully developed boundary layer flow becomes incident with a staggered
arrangement of cube-shaped obstacles. This flow scenario closely resembles an obstacle-resolving urban boundary layer flow30
situation. The case revealed that in two-way coupled simulations, the anterpolated child solution introduces discrepancies
within the parent domain which is manifested as elevated pressure drag within the anterpolated zone. This complication is
remedied by introducing canopy-restricted anterpolation approach, where anterpolation is omitted within the obstacle canopy.
By computing comparison metrics, root-normalized mean square difference and fractional bias, to quantify the difference
37
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
between the fine reference and the nested solutions, the canopy-restricted two-way coupling is shown to be the best coupling
strategy for obstacle-resolving LES studies.
Future outlook
Future development is planned to include the following tasks. Incorporation of PALM’s Lagrangian particle model in the
nesting system in order to enable Lagrangian dispersion studies in urban environments in such a way that particles can be5
transferred between parent and child domains depending on their position. Thus, the long-distance transport of e.g. pollutants,
can be simulated in a coarse-resolution parent grid, while dispersion on the street-scale for specific locations can be simulated
in a fine-resolution child domain. We note that this has been already implemented into PALM and is available to users, but
further sensitivity tests with respect to the treatment of stochastic subgrid-scale particle speeds (Weil et al., 2004) are still
pending. A thorough description and verification of the particle nesting will be published in a follow-up article.10
Further, we note that the PALM model system 6.0 includes also a RANS (Reynolds-averaged Navier-Stokes) mode offering
two different turbulence closures to calculate the eddy diffusivity, that are a TKE-land a TKE-closure according to Mellor and
Yamada (1974, 1982). Besides the LES-LES nesting the nesting system is being extended to handle RANS-LES and RANS-
RANS nesting, which require coupling of additional RANS-variables. Moreover, in a companion paper in this special issue
we present a pure one-way off-line mesoscale nesting method in which the PALM model system 6.0 is nested into meso-scale15
models such as COSMO or WRF. This will allow modelling of meso-scale processes on a much larger coarse-grid domain as
e.g. shown by Muñoz-Esparza et al. (2017), while concurrently focusing on fine-scale processes within certain areas using the
present LES nesting approach.
Furthermore, to date, the timestep in all parent and child models is synchronized and restricted to the minimum of the
time steps determined by each model independently using the CFL criterion. To our experience, the global timestep is often20
restricted by the flow around building edges where high wind speeds occur within the fine-grid child domains. Hence, we plan
to implement a time-splitting into PALM where the parent and child models will be coupled only at the end of the parent
timesteps. This would allow to run coarser-scale parent domains with larger time steps. Thus, computational time could be
saved in the time-integration of the parent simulation as well as in the inter-model communication between parent and child.
Appendix A: Technical realization25
A1 General
The nested model system is implemented using two levels of MPI communicators. The inter-model communication (commu-
nication between model domains) is handled by a global communicator using the one-sided communication pattern (Remote
Memory Access, RMA). The intra-model communication (communication between subdomains within each model domain) is
two-sided and it is handled using a 2-D communicator that has different color for each model. The intra-model communication30
system is the baseline parallelization of PALM (Maronga et al., 2015).
38
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Data transferred from parent to child and from child to parent is always stored in the coarser parent-model grid in order to
minimize the amount of data transfer. This means that the interpolations and anterpolations are always performed by the child.
For these purposes, children contain auxiliary arrays which follow the parent-grid spacings and indexing for each prognostic
variable to be coupled covering the overlap domain plus necessary number of ghost-node layers.
A2 Initialization5
Mapping between each parent and child model domain decompositions as well as all the necessary index mappings are de-
termined in the initialization phase and stored so that the coupling actions during the time-stepping are straightforward and
efficient.
Initial conditions for the root are set similarly to non-nested runs. The root then sends initial field data to its children which
interpolate their own initial conditions from the data received from the root. Next the first-level children send their data to10
their children, if any, and so on. The basic interpolation subroutines for child boundary conditions operate only on the ghost
nodes behind the child-model boundaries. Therefore a separate three-dimensional interpolation subroutine is implemented to
generate initial fields for all the nest domains from their parent-model fields. The same interpolation algorithm is used here as
in the interpolations for child boundary conditions.
A3 Time synchronization15
Time synchronization is taken care by simply selecting the minimum of the time steps determined by each model independently
and broadcasting this time-step value for all models. Each model inputs and outputs in the same way.
A4 Modularization
The data transfer between parents and children is conducted by code contained by five specific fortran modules forming a
module set called PALM Model Coupler (PMC). Calls to the PMC-subroutines are mostly made in PMC-interface module20
(pmc_interface_mod.f90) such that only a small number of calls to the PMC-interface subroutines are needed within the
baseline PALM code. This way, the changes to the baseline code were kept minimal. The PMC-interface module also contains
subroutines for the nesting-related initialization actions, interpolation, anterpolation, child mass-balance forcing, etc.
A5 MPI implementation
While reading the input namelists, the PALM root process checks if a namelist called ”&nesting_parameters” is given in the25
parameter input file PARIN. If not, subroutine called pmc_init_model resets all nesting-related parameters (coupling_layout
etc.) and sets MPI_COMM_WORLD as the base global MPI-communicator comm_palm. The run then continues in standard
way without nesting. If the namelist ”&nesting_parameters” is found and correctly input, the root process of the root model
distributes this information to all other processes via MPI_COMM_WORLD. Then, all the necessary nesting related parameters
39
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
are determined and the base communicator is split into different colors 1for each model based on the model id number. This is
done by calling MPI_COMM_SPLIT. Now each model has its own process group and associated individual base communicator
color such that each model’s internal communication is not visible to other models. After this, the mappings between models
are determined. Each model, except the root model, identifies its parent model and creates an inter-communicator between
the process groups of itself and its parent model. This is realized by calling MPI_INTERCOMM_CREATE. In the same way,5
each model identifies its all children if any, and creates inter communicators between the process groups of itself and all of
its children. These inter communicators are only used to transfer setup data between the root processes of the parent and
child models. For 3D model data transfers between parent and child, specific intra-communicators are created by merging
inter-communicators. This is made after pmc_init_model separately for child and parent models (note that a model may be
both parent and child) in subroutines pmc_childinit and pmc_parentinit by calling MPI_INTERCOMM_MERGE. After the10
pmc-initialization, the run of each model goes as usual. Cartesian topology-based communicator comm_2d is created by each
model from its color of the base communicator comm_palm using MPI_CART_CREATE.
The model internal communication is done in the usual way, e.g. by calling the boundary exchange routines. All data transfer
between parent and child models is done within the PMC interface. For this communication MPI one sided communication
(RMA) is used. An RMA window is opened on the parent side. To transfer data from parent to child, the parent fills the RMA15
window via local copy. After synchronization via MPI_WIN_FENCE, the child processes can fetch the data across the network
with MPI_GET. While transfering data in the opposite direction, the child first transfers the data via MPI_PUT. After another
MPI_WIN_FENCE call, the parent copies the data out of the RMA window into the local model data area.
Appendix B: Thoughts on an alternative interpolation method
Should higher interpolation accuracy across the boundaries be sought, the following considerations are relevant. As stated by20
Zhou et al. (2018), to satisfy the global flux-conservation requirement, one of the flux factors, either the advective velocity
component or the advected variable, must be constant within the anterpolation cell. This implies zeroth-order interpolation.
The other factor must be interpolated using any reversible interpolation scheme.
As stated above, the quadratic Clark and Farley (1984) scheme should not be used because it employs a stencil wider than the
parent-grid cell which leads to problems with complex geometries. On the other hand, tri-linear interpolation has a favorable25
stencil width, but it is not suitable for the scheme as such is not reversible. However, linearly interpolated values e
φi,j,k can
be made reversible by introducing an additional correction φi,j,k =e
φi,j,k + ∆φi,j,k which guarantees the reversibility. The
reversibility correction ∆φi,j,k depends on the difference ∆ΦI,J,K between the original parent-grid value ΦI,J,K and the
value obtained by anterpolating the linearly interpolated values to the parent-grid node I, J,K as
∆Φ = Φ −b
e
φ. (B1)30
1The term color means here that the communicator has the same name for all models (process groups), but they are, however, individual communicators
guaranteeing that communication of one model is not interfered by the others.
40
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
∆ΦI,J,K is a constant value within the parent-grid cell, hence a question arises: how to distribute the correction to the child-grid
nodes i,j,k such that c
∆φI,J,K = ∆ΦI ,J,K? The simplest choice is ∆φi,j,k = ∆ΦI ,J,K , but this choice is not recommendable
in the cases of positive definite scalar variables as it could lead to negative values when Φis close to zero. In principle this
problem could be avoided by weighting the local corrections in proportion to the local differences ΦI,J,K −e
φi,j,k but this simply
reduces the method back to the zeroth-order baseline method. To make this approach useful, a more advanced technique to5
distribute the correction ought to be developed. However, this is beyond the scope of the present work as stated in Sec. 3.4.
Acknowledgements. Test runs with PALM have been performed at the supercomputers of the North-German Super-computing Alliance
(HLRN), Germany, and CSC – IT Center for Science, Finland.
This research was funded by Academy of Finland grant number 277664. BM and MS were supported by the Federal German Ministry
of Education and Research (BMBF) under grant 01LP1601 within the framework of Research for Sustainable Development (FONA; https:10
//www.fona.de). KK was supported by the Federal German Ministry of Economy and Energy (BMU) under grant 0325719C.
Author contributions. Coordination of the study: AH, SR. Design and implementation of the inter-model communication: KK. Theoretical
considerations and implementation of the nesting interface: mainly AH with contributions from SR, MA, MS and BM. Simulations, post-
processing and analysis of model results: AH, MS, MA, BM, CK, FB, GT, NM. Drafting of the manuscript: AH, MA, MS, BM. Revision of
the manuscript: all authors.15
Appendix C: Code availability
The PALM model system is freely available at http://palm-model.org and distributed under the GNU General Public Licence v3 (http://www.
gnu.org/copyleft/gpl.html). However, the simulations presented in this document were performed using a slightly modified code based on
revision 4295. This modified source code (4295M) as well as the input files for the test runs are available at https://doi.org/10.25835/0090593
(Hellsten et al., 2020). Numerous pre- and post-processing scripts are available at http://doi.org/10.5281/zenodo.4005687 (Auvinen et al.,20
2020b).
Competing interests. The authors declare that they have no conflict of interest.
41
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
References
Auvinen, M., Boi, S., Hellsten, A., Tanhuanpää, T., and Järvi, L.: Study of Realistic Urban Boundary Layer Turbulence with High-Resolution
Large-Eddy Simulation, Atmosphere, 11, 201, doi:10.3390/atmos11020201, http://dx.doi.org/10.3390/atmos11020201, 2020a.
Auvinen, M., Karttunen, S., and Kurppa, M.: P4UL: Pre- and Post-Processing Python Library for Urban LES Simulations,
doi:10.5281/zenodo.4005687, https://doi.org/10.5281/zenodo.804282, 2020b.5
Bou-Zeid, E., Overney, J., Rogers, B. D., and Parlange, M. B.: The Effects of Building Representation and Clustering in Large-Eddy Sim-
ulations of Flows in Urban Canopies, Boundary-Layer Meteorology, 132, 415–436, doi:10.1007/s10546-009-9410-6, https://doi.org/10.
1007/s10546-009-9410-6, 2009.
Britter, R. E. and Hanna, S. R.: FLOW AND DISPERSION IN URBAN AREAS, Annual Review of Fluid Mechanics, 35, 469–496,
doi:10.1146/annurev.fluid.35.101101.161147, https://doi.org/10.1146/annurev.fluid.35.101101.161147, 2003.10
Buccolieri, R. and Hang, J.: Recent Advances in Urban Ventilation Assessment and Flow Modelling, Atmosphere, 10, 144,
doi:10.3390/atmos10030144, http://dx.doi.org/10.3390/atmos10030144, 2019.
Chung, D. and McKeon, B. J.: Large-eddy simulation of large-scale structures in long channel flow, Journal of Fluid Mechanics, 661, 341–
364, doi:10.1017/S0022112010002995, 2010.
Clark, T. and Farley, R.: Severe downslope windstorm calculations in two and three spatial dimensions using anelastic interactive grid nesting:15
A possible mechanism for gustiness, Journal of Atmospheric Sciences, 41, 329–350, 1984.
Clark, T. and Hall, W.: Multi-domain simulations of the time dependent Navier-Stokes equations: benchmark error analysis of some nesting
procedures, Journal of Computational Physics, 92, 456–481, 1991.
Daniels, M., Lundquist, K., Mirocha, J., Wiersema, D., and Chow, F.: A new vertical grid nesting capability in the Weather Research and
Forecasting WRF Model, Monthly Weather Review, 144, 3725–3747, 2016.20
de Roode, S. R., Duynkerke, P. G., and Jonker, H. J.: Large-eddy simulation: How large is large enough?, Journal of the atmospheric sciences,
61, 403–421, 2004.
Deardorff, J.: Stratoculumus-capped mixed layers derived from a three-dimensional model, Boundary-Layer Meteorology, 18, 495–527,
1980.
Diebold, M., Higgins, C., Fang, J., Bechmann, A., and Parlange, M. B.: Flow over Hills: A Large-Eddy Simulation of the Bolund Case,25
Boundary-Layer Meteorology, 148, 177–194, doi:10.1007/s10546-013-9807-0, 2013.
Fishpool, G. M., Lardeau, S., and Leschziner, M. A.: Persistent Non-Homogeneous Features in Periodic Channel-Flow Simulations, Flow,
Turbulence and Combustion, 83, 323–342, doi:10.1007/s10494-009-9209-z, https://doi.org/10.1007/s10494-009-9209-z, 2009.
Giometto, M., Christen, A., Meneveau, C., Fang, J., Krafczyk, M., and Parlange, M.: Spatial Characteristics of Roughness Sublayer Mean
Flow and Turbulence Over a Realistic Urban Surface, Boundary-Layer Meteorology, pp. 1–28, 2016.30
Gropp, W., Lusk, E., and Skjellum, A.: Using MPI: Portable parallel programming with the Message Passing Interface, 2nd edition, MIT
Press, Cambridge, MA, 1999.
Hackbusch, W.: Multigrid methods and applications, Springer, Berlin, Heidelberg, New York, 378 pp., 1985.
Heinze, R., Dipankar, A., Henken, C. C., Moseley, C., Sourdeval, O., Trömel, S., Xie, X., Adamidis, P., Ament, F., Baars, H., Barthlott,
C., Behrendt, A., Blahak, U., Bley, S., Brdar, S., Brueck, M., Crewell, S., Deneke, H., Di Girolamo, P., Evaristo, R., Fischer, J., Frank,35
C., Friederichs, P., Göcke, T., Gorges, K., Hande, L., Hanke, M., Hansen, A., Hege, H.-C., Hoose, C., Jahns, T., Kalthoff, N., Klocke,
D., Kneifel, S., Knippertz, P., Kuhn, A., van Laar, T., Macke, A., Maurer, V., Mayer, B., Meyer, C. I., Muppa, S. K., Neggers, R. A. J.,
42
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Orlandi, E., Pantillon, F., Pospichal, B., Röber, N., Scheck, L., Seifert, A., Seifert, P., Senf, F., Siligam, P., Simmer, C., Steinke, S.,
Stevens, B., Wapler, K., Weniger, M., Wulfmeyer, V., Zängl, G., Zhang, D., and Quaas, J.: Large-eddy simulations over Germany using
ICON: a comprehensive evaluation, Quarterly Journal of the Royal Meteorological Society, 143, 69–100, doi:10.1002/qj.2947, https:
//rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.2947, 2017.
Hellsten, A., Ketelsen, K., Sühring, M., Auvinen, M., Maronga, B., Knigge, C., Barmpas, F., Tsegas, G., Moussiopoulos, N., and5
Raasch, S.: Dataset: A Nested Multi-Scale System Implemented in the Large-Eddy Simulation Model PALM model system 6.0,
doi:10.25835/0090593, https://doi.org/10.25835/0090593, 2020.
Heus, T., van Heerwaarden, C., Jonker, H., Siebesma, A. P., Axelsen, S., van den Dries, K., Geoffroy, O., Moene, A., Pino, D., de Roode,
S., et al.: Formulation of and numerical studies with the Dutch Atmospheric Large-Eddy Simulation (DALES), Geosci. Model Dev, 3,
415–444, 2010.10
Huq, S., De Roo, F., Raasch, S., and Mauder, M.: Vertically nested LES for high-resolution simulation of the surface layer in PALM (version
5.0), Geoscientific Model Development, 12, 2523–2538, 2019.
Hutchins, N. and Marusic, I.: Evidence of very long meandering features in the logarithmic region of turbulent boundary layers, Journal of
Fluid Mechanics, 579, 1–28, 2007.
Hutchins, N., Chauhan, K., Marusic, I., Monty, J., and Klewicki, J.: Towards reconciling the large-scale structure of turbulent boundary layers15
in the atmosphere and laboratory, Boundary-layer meteorology, 145, 273–306, 2012.
Ishihara, T., Hibi, K., and Oikawa, S.: A wind tunnel study of turbulent flow over a three-dimensional steep hill, Journal of Wind Engineering
and Industrial Aerodynamics, 83, 95 – 107, doi:https://doi.org/10.1016/S0167-6105(99)00064-1, 1999.
Karttunen, S., Kurppa, M., Auvinen, M., Hellsten, A., and Järvi, L.: Large-eddy simulation of the optimal street-tree layout for
pedestrian-level aerosol particle concentrations – A case study from a city-boulevard, Atmospheric Environment: X, 6, 100 073,20
doi:https://doi.org/10.1016/j.aeaoa.2020.100073, http://www.sciencedirect.com/science/article/pii/S2590162120300125, 2020.
Kataoka, H. and Mizuno, M.: Numerical flow computation around aeroelastic 3D square cylinder using inflow turbulence, 2002.
Kurihara, Y., Tripoli, G., and Bender, M.: Design of a Movable Nested-Mesh Primitive Equation Model, Monthly Weather Review, 107,
239–249, 1979.
Kurppa, M., Hellsten, A., Roldin, P., Kokkola, H., Tonttila, J., Auvinen, M., Kent, C., Kumar, P., Maronga, B., and Järvi, L.: Implementation25
of the sectional aerosol module SALSA2.0 into the PALM model system 6.0: model development and first evaluation, Geoscientific Model
Development, 12, 1403–1422, doi:10.5194/gmd-12-1403-2019, https://www.geosci-model-dev.net/12/1403/2019/, 2019.
Kurppa, M., Roldin, P., Strömberg, J., Balling, A., Karttunen, S., Kuuluvainen, H., Niemi, J. V., Pirjola, L., Rönkkö, T., Timonen, H.,
Hellsten, A., and Järvi, L.: Sensitivity of spatial aerosol particle distributions to the boundary conditions in the PALM model system
6.0, Geoscientific Model Development Discussions, 2020, 1–33, doi:10.5194/gmd-2020-163, https://www.geosci-model-dev-discuss.net/30
gmd-2020-163/, 2020.
Letzel, M., Krane, M., and Raasch, S.: High resolution urban large-eddy simulation studies from street canyon to neighbourhood scale,
Atmospheric Environment, 42, 8770–8784, 2008.
Maronga, B. and Raasch, S.: Large-Eddy Simulations of Surface Heterogeneity Effects on the Convective Boundary Layer During
the LITFASS-2003 Experiment, Boundary-Layer Meteorology, 146, 17–44, doi:10.1007/s10546-012-9748-z, https://doi.org/10.1007/35
s10546-012-9748-z, 2013.
43
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Maronga, B., Gryschka, M., Heinze, R., Hoffmann, F., Kanani-Sühring, F., Keck, M., Ketelsen, K., Letzel, M., Sühring, M., and Raasch,
S.: The Parallelized Large-Eddy Simulation Model (PALM) version 4.0 for atmospheric and oceanic flows: model formulation, recent
developments, and future perspectives, Geoscientific Model Development, 8, 1539–1637, 2015.
MAronga, B., Gross, G., Raasch, S., Banzhaf, S., Forkel, R., Heldens, W., Kanani-Sühring, F., Matzarakis, A., Mauder, M., Pavlik, D.,
Pfafferott, J., Schubert, S., Seckmeyer, G., Sieker, H., and Winderlich, K.: Development of a new urban climate model based on the model5
PALM - Project overview, planned work, and first achievements, Meteorol. Z., 28, 105–119, doi:10.1127/metz/2019/0909, 2019.
Maronga, B., Banzhaf, S., Burmeister, C., Esch, T., Forkel, R., Fröhlich, D., Fuka, V., Gehrke, K. F., Geletiˇ
c, J., Giersch, S., Gronemeier, T.,
Groß, G., Heldens, W., Hellsten, A., Hoffmann, F., Inagaki, A., Kadasch, E., Kanani-Sühring, F., Ketelsen, K., Khan, B. A., Knigge, C.,
Knoop, H., Krˇ
c, P., Kurppa, M., Maamari, H., Matzarakis, A., Mauder, M., Pallasch, M., Pavlik, D., Pfafferott, J., Resler, J., Rissmann, S.,
Russo, E., Salim, M., Schrempf, M., Schwenkel, J., Seckmeyer, G., Schubert, S., Sühring, M., von Tils, R., Vollmer, L., Ward, S., Witha,10
B., Wurps, H., Zeidler, J., and Raasch, S.: Overview of the PALM model system 6.0, Geoscientific Model Development, 13, 1335–1372,
doi:10.5194/gmd-13-1335-2020, https://www.geosci-model-dev.net/13/1335/2020/, 2020.
Mellor, G. L. and Yamada, T.: A Hierarchy of Turbulence Closure Models for Planetary Boundary Layers, Journal of the Atmospheric
Sciences, 31, 1791–1806, doi:10.1175/1520-0469(1974)031<1791:AHOTCM>2.0.CO;2, 1974.
Mellor, G. L. and Yamada, T.: Development of a turbulence closure model for geophysical fluid problems, Reviews of Geophysics, 20,15
851–875, doi:10.1029/RG020i004p00851, 1982.
Miller, M. J. and Thorpe, A. J.: Radiation conditions for the lateral boundaries of limited-area numerical models, Quarterly Journal of
the Royal Meteorological Society, 107, 615–628, doi:10.1002/qj.49710745310, https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.
49710745310, 1981.
Mirocha, J., Kirkil, G., Bou-Zeid, E., Chow, F. K., and Kosovi´
c, B.: Transition and equilibration of neutral atmospheric boundary layer flow20
in one-way nested large-eddy simulations using the Weather Research and Forecasting model, Monthly Weather Review, 141, 918–940,
doi:10.1175/MWR-D-11-00263.1, https://doi.org/10.1175/MWR-D-11-00263.1, 2013.
Moeng, C., Dudhia, J., Klemp, J., and Sullivan, P.: Examinning two-way grid nesting for large-eddy simulation of the PBL using the WRF
model, Monthly Weather Review, 135, 2295–2311, 2007.
Muñoz-Esparza, D., Kosovi´
c, B., García-Sánchez, C., and van Beeck, J.: Nesting turbulence in an offshore convective boundary layer using25
large-eddy simulations, Boundary-layer meteorology, 151, 453–478, 2014.
Muñoz-Esparza, D., Lundquist, J. K., Sauer, J. A., Kosovi´
c, B., and Linn, R. R.: Coupled mesoscale-LES modeling of a diurnal cycle during
the CWEX-13 field campaign: From weather to boundary-layer eddies, 2017.
Munters, W., Meneveau, C., and Meyers, J.: Shifted periodic boundary conditions for simulations of wall-bounded turbulent flows, Physics
of Fluids, 28, 025 112, doi:10.1063/1.4941912, https://doi.org/10.1063/1.4941912, 2016.30
Nakayama, H., Takemi, T., and Nagai, H.: Development of LOcal-scale High-resolution atmospheric DIspersion Model using Large-Eddy
Simulation. Part 5: detailed simulation of turbulent flows and plume dispersion in an actual urban area under real meteorological condi-
tions., Journal Nuclear Science and Technology, 53, 887–908, doi:10.1080/00223131.2015.1078262, https://doi.org/10.1080/00223131.
2015.1078262, 2016.
Nunalee, C. G., Kosovi´
c, B., and Bieringer, P. E.: Development of LOcal-scale High-resolution atmospheric DIspersion Model using Large-35
Eddy Simulation. Part 5: detailed simulation of turbulent flows and plume dispersion in an actual urban area under real meteorological
conditions., Atmospheric Environment, 99, 571–581, doi:10.1016/j.atmosenv.2014.09.070, https://doi.org/10.1016/j.atmosenv.2014.09.
070, 2014.
44
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
Patrinos, A. N. A. and Kistler, A. L.: A numerical study of the Chicago lake breeze, Boundary-Layer Meteorol., 12, 93–123, 1977.
Raasch, S. and Schröter, M.: PALM – A large-eddy simulation model performing on massively parallel computers, Meteorologische
Zeitschrift, 10, 363–372, 2001.
Saiki, E. M., Moeng, C.-H., and Sullivan, P. P.: Large-eddy simulation of the stably stratified planetary boundary layer, Boundary Layer
Meteorol., 95, 1–30, 2000.5
Skamarock, W. C., Klemp, J. B., Dudhia, J., Gill, D. O., Barker, D. M., Duda, M. G., Huang, X.-Y., Wang, W., and Powers, J. G.: G.: A
description of the Advanced Research WRF version 3, Tech. Rep. NCAR/TN-475+ STR, NCAR, 2008.
Sühring, M., Maronga, B., Herbort, F., and Raasch, S.: On the Effect of Surface Heat-Flux Heterogeneities on the Mixed-Layer-Top Entrain-
ment, Boundary-Layer Meteorology, 151, 531–556, doi:10.1007/s10546-014-9913-7, https://doi.org/10.1007/s10546-014-9913-7, 2014.
Sullivan, P. and Patton, E.: The effect of mesh resolution on convective boundary layer statistics and structures generated by large-eddy10
simulation, Journal of the Atmospheric Sciences, 68, 2395–2415, 2011.
Sullivan, P., McWilliams, J., and Moeng, C.-H.: A grid nesting method for large-eddy simulation of planetary boundary-layer flow, Boundary-
Layer Meteorology, 80, 167–202, 1996.
Tominaga, Y. and Stathopoulos, T.: CFD simulation of near-field pollutant dispersion in the urban environment: A review of current modeling
techniques, Atmospheric Environment, 79, 716 – 730, doi:https://doi.org/10.1016/j.atmosenv.2013.07.028, http://www.sciencedirect.com/15
science/article/pii/S1352231013005499, 2013.
Tseng, Y.-H., Meneveau, C., and Parlange, M. B.: Modeling flow around bluff bodies and predicting urban dispersion using large eddy
simulation, Environmental science & technology, 40, 2653–2662, 2006.
Vonlanthen, M., Allegrini, J., and Carmeliet, J.: Assessment of a one-way nesting procedure for obstacle resolved large eddy simulation of
the ABL, Computers and Fluids, 140, 136–147, doi:10.1016/j.compfluid.2016.09.016, https://doi.org/10.1016/j.compfluid.2016.09.016,20
2016.
Vonlanthen, M., Allegrini, J., and Carmeliet, J.: Multiscale interaction between a cluster of buildings and the abl developing over a real
terrain, Urban Climate, 20, 1–19, doi:10.1016/j.uclim.2017.02.009, https://doi.org/10.1016/j.uclim.2017.02.009, 2017.
Weil, J., Sullivan, P., and Moeng, C.: The use of large-eddy simulations in Lagrangian particle dispersion models, Journal of the Atmospheric
Sciences, 61, 2877–2887, 2004.25
Wicker, L. J. and Skamarock, W. C.: Time-splitting methods for elastic models using forward time schemes, Mon. Wea. Rev., 130, 2088–
2097, 2002.
Williamson, J. H.: Low-storage Runge-Kutta schemes, J Comput. Phys., 35, 48–56, 1980.
Xie, Z. and Castro, I.: LES and RANS for turbulent flows over arrays of wall-mounted obstacles, Flow, Turbulence and Combustion, 76,
291–312, 2006.30
Zhou, B., Xue, M., and Zhu, K.: A grid-refinement-based approach for modelling the convective boundary layer in the gray zone: Algorithm
implementation and testing, Journal of the Atmospheric Sciences, 75, 1143–1161, 2018.
Zhu, P., Albrecht, B. A., Ghate, V. P., and Zhu, Z.: Multiple-scale simulations of stratocumulus clouds, Journal of Geophysical Research:
Atmospheres, 115, 2010.
45
https://doi.org/10.5194/gmd-2020-222
Preprint. Discussion started: 7 September 2020
c
Author(s) 2020. CC BY 4.0 License.
