Content uploaded by Simone Marras

Author content

All content in this area was uploaded by Simone Marras on Apr 20, 2015

Content may be subject to copyright.

Noname manuscript No.

(will be inserted by the editor)

A Review of Element-Based Galerkin Methods for Numerical

Weather Prediction

Finite Elements, Spectral Elements, and Discontinuous Galerkin

Simone Marras1·James F. Kelly2·Margarida

Moragues3·Andreas Müller1·Michal A. Kopera1·

Mariano Vázquez3,4·Francis X. Giraldo1·Guillaume

Houzeaux3·Oriol Jorba5

Received: date / Accepted: date

Abstract Numerical Weather Prediction (NWP) is in a period of transition. As resolutions increase,

global models are moving towards fully nonhydrostatic dynamical cores, with the local and global

models using the same governing equations; therefore we have reached a point where it will be nec-

essary to use a single model for both applications. The new dynamical cores at the heart of these

uniﬁed models are designed to scale eﬃciently on clusters with hundreds of thousands or even millions

of CPU cores and GPUs. Operational and research NWP codes currently use a wide range of numer-

ical methods: ﬁnite diﬀerences, spectral transform, ﬁnite volumes and, increasingly, ﬁnite/spectral

elements and discontinuous Galerkin, which constitute element-based Galerkin (EBG) methods. Due

to their important role in this transition, will EBGs be the dominant power behind NWP in the next

10 years, or will they just be one of many methods to choose from? One decade after the review of

numerical methods for atmospheric modeling by Steppeler et al. (2003) [Review of numerical methods

for nonhydrostatic weather prediction models Meteorol. Atmos. Phys. 82, 2003], this review discusses

EBG methods as a viable numerical approach for the next-generation NWP models. One well-known

weakness of EBG methods is the generation of unphysical oscillations in advection-dominated ﬂows;

special attention is hence devoted to dissipation-based stabilization methods. Since EBGs are geo-

metrically ﬂexible and allow both conforming and non-conforming meshes, as well as grid adaptivity,

this review is concluded with a short overview of how mesh generation and dynamic mesh reﬁnement

are becoming as important for atmospheric modeling as they have been for engineering applications

for many years.

Keywords Galerkin Methods ·Finite Elements ·Spectral Elements ·Discontinuous Galerkin ·

HPC ·Stabilization ·Dynamic Diﬀusion ·Large Eddy Simulation ·Numerical Weather Prediction

Contents

1 Introduction.................................................... 3

Tel.: +1 831 656 3885

E-mail: smarras1@nps.edu

1Naval Postgraduate School, Dept. of Applied Mathematics

833 Dyer Rd., SP249A

93943 Monterey (CA) U.S.A. ·

2Exa Corporation

Burlington (MA), U.S.A. ·

3Barcelona Supercomputing Center BSC-CNS, CASE. Barcelona, Spain ·

4IIIA - CSIC, Bellaterra, Spain ·

5Barcelona Supercomputing Center BSC-CNS, Earth Sciences. Barcelona, Spain

2 Simone Marras1et al.

2 Equation sets for atmospheric modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Galerkin methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4 Stabilization of EBG for advection-dominated problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5 Vertical discretization, computational grids, and adaptive mesh reﬁnement in NWP . . . . . . . . . . . 51

6 Summary ..................................................... 63

EBG in atmospheric modeling 3

1 Introduction

Numerical Weather Prediction (NWP), which began with the work of Richardson during World War

I [250], remains one of the most challenging problems in the computational sciences. The two main

challenges to producing an accurate forecast are 1) mathematically modeling atmospheric phenom-

ena over a wide range of physical and temporal scales (e.g., turbulence, radiation, cloud formation),

and 2) harnessing the available computational resources to evaluate these models in an accurate

and eﬃcient manner. While the goal of the ﬁrst challenge is probably static (that is, a comprehen-

sive mathematical description of the atmosphere at a given time), the second challenge represents

a moving target. Computational resources not only expand; they change in character. Richardson’s

original idea of a "forecasting factory" consisting of thousands of human computers assembled in an

amphitheater was never realized; the ﬁrst NWP codes were implemented on mainframe computers.

Mainframes gave way to minicomputers and later vector supercomputers such as the Cray 1, 2, X-MP,

and Y-MP. By the mid-90s, vector supercomputers were replaced by massively parallel distributed

systems. Now, in 2015, we are seeing the proliferation of many-core architectures (e.g. GPUs) and

hybrid distributed/shared memory architectures (e.g. clusters of many-core processors, heterogeneous

clusters). Moreover, as models increase their accuracy by resolving more phenomena (e.g. resolving

non-hydrostatic eﬀects, incorporating more complex moisture parameterizations), their appetite for

High Performance Computing (HPC) resources grow.

The modeling challenge and computational challenge meet in the choice of the numerical method

used to discretize the underlying continuum model(s), which are generally expressed as systems of

both partial and ordinary diﬀerential equations. The numerical model, as this ﬁgurative middle-man,

must both 1) accurately represent the continuum model, and 2) eﬃciently utilize the hardware used to

implement the numerical method. Hence, the numerical method mediates these two grand challenges

by adapting to the hardware; moreover, since NWP models may take on the order of 100 man-years to

develop, test, and deploy, the designers of the numerical method should target their model to future

HPC resources. Just as biological organisms must constantly adapt to their physical environment,

numerical methods must adapt to their computational environment, competing for available resources.

A natural question arises: which numerical methods will survive and ﬂourish, and which will stagnate,

decline, and perhaps go extinct?

This question was partially addressed in the review of the numerical methods for non-hydrostatic

atmospheric modeling reported by Steppeler et al. [282]. Based on some of the questions posed in

[282], we concentrate on a class of numerical methods that may emerge victorious in next generation

atmospheric (and climate) models: element-based Galerkin methods (EBGs). Among other questions,

Steppeler and co-workers asked whether the numerical error caused by terrain-following coordinates

could be avoided by means of z-coordinate based methods [281; 282]; element-based Galerkin methods

are a natural choice to fulﬁll this recommendation. Furthermore, they questioned the ability of low

order methods to resolve certain phenomena at high resolution without aﬀecting accuracy: "Experience

from current models suggests that approximations of overall third order will be adequate." It is shown

in this review how things have indeed evolved towards the high order approach that Steppeler et al.

were discussing 10 years ago and how those schemes that in 2003 had not been used in operational

mode (because considered "advanced" [281]), are currently the driving force behind the next generation

NWP models.

As discussed above, element-based Galerkin schemes today are tied to their relationship with the

evolution of computer hardware. We will see this in the sections that follow, after giving a short

overview of the current trends in HPC and how atmospheric models are developing around this

paradigm.

4 Simone Marras1et al.

1.1 Trends in High Performance Computing

Twenty-ﬁve years ago (1990), state-of-the-art HPC were the Cray supercomputers (e.g. Cray Y-MP).

These machines had a small number (2 to 8) of expensive custom vector processors, which perform

a single instruction on multiple data (SIMD); all the processors fetched data from a bank of shared

memory. This trend changed in the 1990s as commodity processors and memory became relatively

inexpensive; suddenly, large clusters of commodity processors that utilized distributed memory ar-

chitectures became available. Unlike the vector machines, distributed memory systems require com-

munication between independent processes. At the present time (2015) another shift is occurring as

many-core architectures, with a relatively small amount of shared memory, are being coupled with

massively parallel systems. These distributed memory systems eclipsed the older vectorized machines

by the late 1990s, and vectorized machines are no longer used in HPC.

Today, HPC is in the Petascale era, with core counts exceeding O(106)[226] while exascale tech-

nologies are rapidly approaching. For instance, the largest cluster as of November 2014 (Top5001) is

Tianhe-2 with 3.12 million cores and a maximum LINPACK [80] performance of 33.8 PetaFLOPS.

The next largest machine is Titan, a Cray XK7 with 560640 cores and a maximum LINPACK per-

formance of 17.59 PetaFLOPS. To take full advantage of the performance of these architectures, the

need for speciﬁc characteristics in new models drove scientists from diﬀerent ﬁelds to go back to the

design board and start from scratch in the construction of their numerical algorithms [118]. This is

required by the need for very speciﬁc features that the numerical method must have to reach very high

levels of scalability on the new machines. The next section reports on most operational and research

atmospheric models developed until today with special emphasis on how atmospheric modelers are

moving towards numerical methods that have proved more scalable on current and future computers.

1.2 Existing atmospheric models and NWP systems

Table 1 shows a non-exhaustive list of atmospheric models developed until today. Most of the listed

codes are based on the ﬁnite diﬀerence method. Except for ENDGame (UK Met Oﬃce), the Nonhydro-

static Multiscale Model core of the NCEP NAM, and EULerian LAGrangian (EULAG), all FD-based

codes are limited area models (LAM). Spectral transform and ﬁnite volumes represent the second ma-

jor trend. Codes based on the spectral transform are common for General Circulation Models (GCM)

only. High-order element-based methods (spectral element method, SEM, and discontinuous Galerkin,

DG) follow, while the ﬁnite element method (FEM), only used by a handful of models, is the least

common of all. For reasons that will become clearer in later sections, the temporal integration schemes

that are mostly used are the split-explicit and the semi-implicit methods.

1www.top500.org

EBG in atmospheric modeling 5

Table 1: Non-exhaustive compilation of NWP systems. The acronyms used in this table, some of which have not been deﬁned before, are the

following. FD for ﬁnite diﬀerences; FV for Finite Volumes; FE for Finite Elements; SE for Spectral Elements; DG for Discontinuous Galerkin;

ST for Spectral Transform; NH for Non-Hydrostatic; HS for Hydrostatic; HPE for Primitive Equations; CEE for Compressible Euler Equations;

SISL for Semi-Implicit + Semi-Lagrangian; EX for Explicit; IMEX for Implicit-Explicit or Semi-Implicit; SpEx for Split-Explicit; FB-EX for

Forward-Backward explicit; LF for Leap-Frog; NFT for Non-oscillatory Forward in Time; FDGC for FD on generalized coordinates; LES for

Large Eddy Simulation; HEVI for Horizontally Explicit-Vertically Implicit

Model Country Institution NH/HS Type Equations Space Time

-ARPEGE [68] France Meteo France NH/HS LAM/GCM HPE ST+FD (z) SI

-ALADIN-NH [192] France Meteo France NH/HS LAM CEE ST+FD (z) SISL

-ETA [165] USA NCEP NH/HS LAM HPE FD FB-EX

-MC2 [21] Canada Res. Ctr. NWP NH LAM CEE FD SISL

-COAMPS [135] USA NRL NH LAM CEE FD SpEx

-GEM [64] Canada CMC &MRB HS+NH LAM/GCM HPE FEM SISL

-HIRLAM [256] France Meteo France NH LAM HPE FD SISL

-GFS USA NOAA HS GCM HPE ST+FD (z) SI

-GME [206] Germany DWD HS GCM HPE FV SI

-COSMO/LM [77; 281] Germany et al. DWD NH LAM CEE FD SpEx

-IFS [305] UK ECMWF HS GCM CEE ST+FEM (z) SISL

-ICON [107; 311] Germany MPIfM/DWD HS/NH GCM CEE/HPE FV SI

-CAM EUL [230] USA NCAR HS GCM HPE ST+FD (z) SI

-CAM FV [230] USA NCAR HS GCM HPE FV Explicit

-CAM SE [74] USA NCAR/SNL HS GCM HPE SE+FD (z) EX

-NAVGEM [136] USA NRL HS GCM HPE ST+FD (z) SISL

-ENDGame [323] UK Met Oﬃce NH/HS LAM/GCM CEE FD SISL

-KIAPS-GM 2Korea KIAPS HS GCM HPE SEM+FD (z) EX

-NEPTUNE [176; 117] USA NRL NH LAM/GCM CEE SE IMEX/SpEx

-HIRAM [331] USA GFDL NH GCM CEE FV SISL

-ECHAM6 [283] Germany MPIfM HS GCM HPE ST+FD SI

-SLAV 3Russia RAS HS/(NH) GCM HPE FD SISL

-JMA [261] Japan JMA NH LAM/GCM CEE FD HEVI

-VCAM/CCAM [218] Australia CSIRO HS GCM HPE FV/FD SISL

-GRAPES [326] China CMA NH LAM/GCM CEE FD SLSI

Continued on next page

2www.kiaps.org

3www.meteoinfo.ru

6 Simone Marras1et al.

Table 1 – Continued from previous page

Model Country Institution NH/HS Type Equations Space Time

-NAM [166] USA NCEP NH/HS LAM/GCM HPE FD/FV SI

EBG in atmospheric modeling 7

Table 2: Non-exhaustive compilation of atmospheric research models. The acronyms, some of which were not previously introduced, are

the following. NH for Non-Hydrostatic; HS for Hydrostatic; HPE for Primitive Equations; CEE for Compressible Euler Equations; SISL for

Semi-Implicit + Semi-Lagrangian; EX for Explicit; IMEX for Implicit-Explicit or Semi-Implicit; SpEx for Split-Explicit; FB-EX for Forward-

Backward explicit; LF for Leap-Frog; NFT for Non-oscillatory Forward in Time; FDGC for FD on generalized coordinates; LES for Large Eddy

Simulation; HEVI for Horizontally Explicit-Vertically Implicit

Model Country Institution NH/HS Type Equations Space Time

-TASS [243] USA NASA NH LAM, LES CEE FD/FV SI

-RAMS [238] USA Col. State U. NH/HS LAM HPE FD LF

-MM5 [83] USA NCAR NH LAM HPE FD LF

-ARPS [324] USA U. Oklah. NH LAM CEE FD SI

-OMEGA [8] USA Centr. Atmo. Phys. NH GCM CEE FV SI

-OLAM [310] USA U. of Miami NH GCM CEE FV SpEx

-NSEAM [119] USA NRL HS GCM HPE SE SISL

-PUMA [99] Germany U. of Hamburg HS GCM HPE ST SI

-HOMME [296] USA NCAR HS GCM HPE SE EX+other

-WRF-ARW [271] USA NCAR NH LAM CEE FD SpEx

-AROME 4Europe consortium NH LAM ALADIN ST SI

-EULAG [244] USA NCAR NH LAM/GCM CEE/Incompr. FDGC NFT

-NICAM [262] Japan JAMSTEC NH LAM/GCM CEE FV SpEx

-FIM [195] USA NOAA HS GCM HPE FV Expl

-NIM [196] USA NOAA NH GCM CEE FV SpEx

-UZIM [3] USA Co. State U. NH GCM Anel. FD SI

-DALES [134] Netherlands R. Nether. Meteor. I. NH LAM CEE LES FD IMEX

-CM15USA NCAR NH LAM CEE FD IMEX

-ExnerFOAM [313] UK Reading U. NH GCM CEE FV IMEX

-DUNE [34] Germany Freiburg U. NH LAM CEE DG EX

-MPAS [273] USA NCAR/LANL NH LAM/GCM CEE FV SpEx

-MCore [304] USA U. Mich. NH GCM CEE FV IMEX

-NUMA [176; 117] USA NPS NH LAM/GCM CEE SE/DG IMEX

-Alya [212; 213] Spain BSC-CNS NH LAM CEE/Incomp./Bouss. FE EX/Impl.

-DYNAMICO 6France IPSL HS GCM HPE FD

Continued on next page

4www.cnrm.meteo.fr/arome/

5www2.mmm.ucar.edu/people/bryan/cm1/

6www.lmd.polytechnique.fr/ dubos/DYNAMICO/

8 Simone Marras1et al.

Table 2 – Continued from previous page

Model Country Institution NH/HS Type Equations Space Time

-Gung-Ho [301] UK Met Oﬃce NH/HS LAM/GCM CEE FE SI/HEVI

-ASAM [162] Germany TROPOS NH LAM CEE FD/FV Impl.

-GEOS [255] USA NASA HS GCM HPE FD SISL

EBG in atmospheric modeling 9

Fig. 1: Large Eddy Simulation of the evolution of a single cloud with the Nonhydrostatic Uniﬁed Model of

the Atmosphere (NUMA). From [214]. The MayaR

computer graphics software was used for the photo-realistic

rendering of the simulation (for more details see http ://anmr.de/cloudwithmaya).

1.3 Traditional approaches: Finite Diﬀerence (FD) and Spectral Transform (ST) methods

As noticeable from the tables above, most operational NWP codes in use are based on either the

ﬁnite diﬀerence (FD) method, or, in the case of global models, the spectral transform (ST) method.

It is diﬃcult to ﬁnd models using these methods that scale optimally on massively parallel computers

(ST methods due to their all-to-all communication requirements and FD due to non-compact stencils

especially at high-order). This is also true of non-compact (high-order) ﬁnite volume methods. In order

to understand the strengths and weaknesses of these traditional approaches and how EBGs address

some of their shortcomings, we brieﬂy review the FD and ST methods in this subsection.

Limited area models (LAMs) consider atmospheric ﬂows over a subsection of the earth’s surface.

Examples include mesoscale models, which typically span hundreds of kilometers in the horizontal, and

cloud resolving models (CRMs), which span approximately up to tens of kilometers in the horizontal.

See an example of a simulated single cloud in Fig. 1.

The ﬁnite diﬀerence method (FD) is the method of choice for LAMs for several reasons. First,

it is simple to implement on a Cartesian grid, especially if the curvature of the earth is neglected.

Unlike EBG methods, or the ﬁnite volume method, grid generation is trivial and very few ancillary

data structures are needed. Second, it is very eﬃcient on a single processor, or on a small number

of processors within a shared memory architecture (e.g. vector machines). Third, constructing both

upwinded and higher order discretizations is relatively straightforward, although increasing the order

of accuracy may hurt its scalability due to the larger halo required.

Global models (or General Circulation Models, GCMs) solve the governing equations on the whole

planet, which is usually approximated as a sphere. The reader is referred to the 2007 paper by

Williamson [321] for a review of GCMs. Many operational GCMs utilize ST, where spherical harmonics

are used to represent both diagnostic and prognostic variables on the sphere. Spherical harmonics are

the natural basis functions to solve PDEs on a sphere since they are the eigenfunctions of the negative

Laplacian. Hence, great accuracy is achieved with a minimal number of grid points on the sphere.

In order to advance the dynamical equations in time using ST, it is necessary to transform between

physical and spectral space; this spectral transform is evaluated using a combination of Fourier and

10 Simone Marras1et al.

Legendre transforms. We perform an elementary complexity analysis of the ST method to illustrate

a fundamental bottleneck as the resolution of NWP models increases.

Letting nbe the number of grid points, Fourier transforms are evaluated along the longitudinal

(zonal) direction with an FFT with a cost O(nlogn); along the latitudinal (meridional) direction,

a Legendre transform is required with a cost of O(n3). Although fast Legendre methods exist, they

are not widely used in NWP since they have high cross-over points. Therefore the cost of the ST

method is O(nlog n+n3), which scales adversely as nincreases (e.g. horizontal resolution is increased).

For a grid spacing greater than 10 km, the hydrostatic, rather than non-hydrostatic equations are

the governing equations in GCMs (we will touch more on the equation sets in Section 2). These

equations are solved via a vertical mode decomposition [119] which results in a constant-coeﬃcient

Helmholtz operator. Since spherical harmonics are exact solutions to this Helmholtz operator, no

matrix inversion is required. Furthermore, ST have a very small dispersion error. ST models were

developed during the era of smaller, shared-memory machines which did not require communicating

data across processors. As the architectures transitioned from shared to distributed memory, the

communication overhead became more important; the all-to-all communication required by both the

FFT and Legendre transform poses a barrier to scalability (not all distributed-memory hardware can

do this operation eﬀectively). For instance, the ST-based model NOGAPS, used by the U. S. Navy,

could not scale beyond 150 processes at typical resolutions [119]. Hence, ST methods, while both

highly accurate and eﬃcient at small processor counts, cannot compete in the era of hundreds of

thousands (or millions) of processors.

To overcome the limitations of FD and ST in the current era of massively parallel computers, EBG

methods are becoming the new trend in atmospheric modeling for the same reason they have always

been popular in other ﬁelds of computational mechanics. This alternative is justiﬁed by the proven

high parallel eﬃciency of local methods [226; 320; 176; 73]. The eﬃciency of EBGs on large to very

large machines is facilitated by their small parallel communication footprint. To understand this small

footprint, consider Fig. 2, where the grids needed by a a) ﬁnite element and by a b) ﬁnite diﬀerence

method are compared. In Fig. 2-a, the grid consists of nine ﬁnite elements Ωel

h. With EBG the solution

is sought on an element-wise basis and each element communicates information to the others only

through its shared boundaries (nodes in the case of CG; faces in the case of DG). When the ﬁnite

element grid is partitioned into smaller portions of the global domain, the only information that needs

to be exchanged among the subdomains of the partition is that on the boundary that each subdomain

shares with its neighbors. In contrast, in Fig. 2-b the grid is a classical structured, rectangular ﬁnite

diﬀerence grid that here is plotted to be a direct analogue (in terms of node count) of the ﬁnite

element mesh. Because a ﬁnite diﬀerence stencil is such that diﬀerentiation on each node in the

domain requires information from a set of adjacent nodes that varies with the order of diﬀerentiation,

when the domain is partitioned, some nodes will belong to two overlapping subdomains. Because of

this, additional communication is necessary. In the case of element-based schemes communication is

naturally low by construction. The details of EBG and which models are based on them are reviewed

in Section 3.

1.4 A Roadmap for Element-Based Galerkin Methods and this Review

Historically, ﬁnite element, spectral transform, and discontinuous Galerkin methods have been devel-

oped in relative isolation. In the past several decades, especially with the advent of spectral elements,

common threads were identiﬁed. The two most important ideas are: 1) decomposing a continuous do-

main Ωinto a ﬁnite number Neof non-overlapping elements Ωeand 2) expanding the state variables

in Nbasis functions ψiwithin each sub-domain (or element) Ne. In the ﬁrst operation, we express

the geometry in an element-wise fashion; in the second operation, we perform a Galerkin expansion of

the state variables. Hence, the moniker Element-based Galerkin method. As discussed, EBG methods

are classiﬁed as either continuous (CG) or discontinuous (DG). Each of these methods may be char-

EBG in atmospheric modeling 11

(a) (b)

Fig. 2: Examples of the adjacency pattern for a ﬁnite element Ωel

h(a), and for a node that belongs to a ﬁnite

diﬀerence grid (b). In (a) and (b) information is exchanged, respectively, element- and node-wise. In (a), the only

nodes that allow information to be shared between elements are the shared nodes on the boundary of neighboring

elements (blue dots on the boundary of Ωel

h.). In (b), the cross made of blue circular nodes and a central red node

is the stencil of a 4th-order diﬀerentiation performed on the central node. How these plots relate to parallelization

is described in the text.

acterized by the number of elements Ne(or equivalently, the element diameter h) and the order of the

basis function p. Resolution may be increased by increasing either Neor pindependently, allowing

a wide range of combinations. In the limit of Ne= 1, the spectral transform (ST) can be seen as an

EBG method; however, being ST a degenerate EBG, in the rest of the paper it won’t be considered

among the EBG methods. This h−pparameter space is mapped in Figure 3. In the left panel (CG),

three numerical methods are displayed: ﬁnite elements, spectral elements, and the spectral transform

method. Since continuity is required between elements, the lowest possible order pis one. The ﬁnite

element method (FEM) is the special case when p= 1,2,3basis functions are employed, while the

spectral transform method is recovered if a single element is used with a very large order p1. In the

right panel, we see three numerical methods: ﬁnite volumes, DG, and the spectral transform method.

Since continuity is not required between elements, we may use constant-valued basis functions, which

is equivalent to cell-averaging; hence, we recover the classical ﬁnite volume (FV) method if p= 0. For

1≤p < ∞and Ne>1we have DG, while for large pand Ne= 1, we again recover the ST method.

Gabersek et. al. [103] systematically mapped out the h-pspace for SEM. They concluded that poly-

nomial order pbetween 5 and 10 with an eﬀective resolution of ¯

∆x =h/p between 0.5 and 2.0 km is

optimal for mesoscale simulations in terms of both accuracy and eﬃciency. To our knowledge, the h-p

space for global non-hydrostatic simulations has not been explored yet.

1.5 Scalability of EBG methods

In the following we report on some recent scalability results of EBG on diﬀerent systems and for

diﬀerent numerical conﬁgurations. For a more theoretical discussion on Galerkin scalability, we refer

to [142; 176].

12 Simone Marras1et al.

Fig. 3: EBGs are divided into two classes: continuous Galerkin (CG) methods, whose solutions are continuous with

bounded weak derivative (H1), and discontinuous Galerkin (DG) methods, whose solutions are square integrable

(L2), but not necessarily continuous. The resolution of both CG and DG methods may be characterized by

the polynomial order pof their basis functions and the number of elements Neutilized, or, equivalently, by

the diameter h∝1/Neof each element. CG: If low order basis functions (p= 1,2,3) are utilized with a large

number of elements, we recover the classical Finite Element Method. For p≥3and a smaller number of elements

used, we have the spectral element method (SEM). As pis increased, Nemay be decreased. In the extreme case

of a single element (on the sphere) and p1, the ST method is recovered. If we are considering problems in

Cartesian geometry, this extreme case is generally termed "spectral" or "pseudo-spectral". DG: Since DG admits

discontinuous solutions, a constant basis function p= 0 is admissible, yielding the classical ﬁnite volume (FV)

method. As pis increased and Nedecreases, we enter the arena of DG methods. As with CG, if a single element

is utilized, the ST method is recovered. In this extreme case, the solution becomes continuous.

1.5.1 Scalability for (horizontally) explicit time integration

In global atmospheric simulations the vertical resolution is usually much ﬁner than the horizontal.

This leads to a much smaller time scale for vertical processes than for horizontal motion. For this

reason, it is often more eﬃcient to solve the fast processes in the vertical direction implicitly while

using explicit time integration in the horizontal direction, or horizontally-explicit, vertically-implicit

(HEVI). If a 2D domain decomposition strategy is adopted where all the elements in a vertical column

are maintained on a single processor, HEVI does not incur any additional communication. A recent

result for this strategy with the Nonhydrostatic Uniﬁed Model of the Atmosphere (NUMA) [176; 117]

is shown in Fig. 4. This ﬁgure shows that NUMA achieves weak scaling up to 777,600 cores and strong

scaling to about 40,000 cores; moreover, the last blue data point in this ﬁgure indicates that NUMA

scales in this fashion to the limit of one horizontal element per core.

EBG in atmospheric modeling 13

Fig. 4: Scalability study with the atmospheric model NUMA for the baroclinic instability test case [160] for three

diﬀerent horizontal resolutions of 25.0km, 12.5km and 2.78km (given in the legend). This scalability study was

performed on the Blue Gene Mira of the Argonne National Lab. The number next to each data point shows the

average number of elements per core. These simulations use a cubed sphere mesh generated by the function library

p4est [42]. All simulations use 6 elements in the vertical direction with HEVI time integration and a ﬁfth-order

CG method.

One important factor that contributed to the excellent speedup shown in Fig. 4 is that the amount

of work on each core needs signiﬁcantly more runtime than the time spent in communicating the data

among neighboring cores. This becomes more diﬃcult when fully explicit time integration is used and

when the spatial discretization order is reduced (Fig. 5).

1.5.2 Scalability for fully implicit time integration

Scalability studies with the model Alya [142; 308] and fully implicit time integration on diﬀerent

machines are shown in Fig. 6.

Alya is an unstructured ﬁnite element code. The mesh partitioning therefore relies on the ele-

ment graph, whose complexity depends on the geometry considered. Libraries such as ZOLTAN [28],

SCOTCH [49], or METIS [174], which are based on graph partitioning algorithms, may be used to

decompose an EBG mesh. Just like NUMA, Alya does not require halos and the information exchange

between neighbors is carried out on the interface nodes, that is, the nodes shared by diﬀerent subdo-

mains. From the parallelization point of view, the load balance and the communication scheduling for

these two codes depend on the quality of the partition.

14 Simone Marras1et al.

Fig. 5: Scalability study with NUMA using a 1D semi-implicit (HEVI) simulation of a 3D rising thermal bubble

in a 1 km3cubed domain for polynomial degrees 4 and 8 (see legend), using 323elements. The average number

of elements per core is given by the numbers next to each data point. This scalability study was performed on

the Blue Gene Vesta of the Argonne National Lab.

Several iterative solvers are available, and the selection depends on the physical problem con-

sidered. The incompressible Navier-Stokes equations require the solution of the momentum equation

and the pressure equation [140]. For the ﬁrst algebraic system, the GMRES method with a simple

diagonal preconditioning is eﬃcient in most of the cases, and few iterations are required to obtain

convergence. For the pressure equation, a deﬂated conjugate gradient method [203] is used together

with linelet preconditioning [277], which is very eﬃcient in the presence of boundary layers. The four

scalabilities presented in Fig. 6 were obtained for the Navier-Stokes equations. The last one represents

the combustion in a kiln, which consists in solving the low Mach Navier-Stokes equations together

with a temperature equation and chemical reactions.

1.6 Plan of the paper

The rest of the review is organized as follows: in Section 2 we give an overview of the diﬀerent equation

sets used in the dynamical cores of atmospheric models. Element-based Galerkin methods within the

context of NWP are introduced in Section 3. Since EBG methods may produces unphysical extrema

(especially high-order EBGs), stabilization/ﬁltering is often required: a review of some stabilization

methods follows in Section 4. Section 5 explores accurate grid generation within high resolution

EBG in atmospheric modeling 15

Fig. 6: Scalability study for a fully implicit simulation using Alya [142; 308] on four diﬀerent HPC architectures.

simulations (e.g. well resolved topography), along with static and dynamic grid adaptivity. A summary

is reported in Section 6.

16 Simone Marras1et al.

2 Equation sets for atmospheric modeling

For typical atmospheric scales (1 m to 1000+ km), the earth’s atmosphere can be treated as a contin-

uum governed by the compressible Navier-Stokes equations with body forces to model the eﬀects of

gravity and the Earth’s rotation (i.e. Coriolis force). Although the gravitational force varies with both

altitude and lattitude, these minor perturbations are generally neglected. In this section, we neglect

the eﬀects of moisture, solar radiation, and heat ﬂux from the ground and consider the dry dynamics

of the atmosphere. Let Ωbe a three-dimensional domain in a rotating reference system xand let

t≥0be time. The state of dry, stratiﬁed air can be described by density, ρ, pressure, p, absolute

temperature, T, and velocity ﬁeld, u,

∂ρu

∂t +∇·(ρu⊗u)+ ∇p−∇·σ=−2ρ(ω×u)−ρg,(1a)

∂ρ

∂t +∇·(ρu)=0,(1b)

∂E

∂t +∇·((E+p)u)−∇·µcp

Pr ∇T+u·σ= 0,(1c)

where ωis the rotational velocity of the Earth, σis the viscous stress tensor, gis the sum of true

gravity and the centrifugal force, and the total energy, E, is given by

E=ρcvT+1

2ρu·u+ρgr, (2)

where ris the radial distance from a ﬁxed reference point at the center of the earth. Eq. (2) consists of

three components: internal energy, kinetic energy, and gravitational potential energy. For a Newtonian

ﬂuid with dynamic viscosity µ, the viscous stress tensor is given by

σ=µ∇u+(∇u)T−2

3(∇·u)I,(3)

where 2/3is a constant derived from the Stokes hypothesis and Tis the vector transpose [232]. The

system (1) of ﬁve conservation laws in six unknowns is closed by the equation of state (ideal gas law)

for pressure:

p=R

cvE−1

2ρu·u−ρgr.(4)

We note that Eq. (3) does not incorporate any eﬀects of turbulent dissipation. Since the Kolmogorov

length scale of a typical atmospheric problem is on the order of 0.1 mm, direct numerical simulation

(DNS) of atmospheric motion is not possible with the current computational resources. To properly

account for unresolved turbulent motion (e.g. turbulent dissipation), a sub grid scale model or turbu-

lence closure scheme should be included. To simplify the treatment of the most commonly used sets

of equations and of the numerical methods discussed below, we will neglect viscosity and restrict our

analysis to the Euler equations (µ= 0) and various approximations utilized in atmospheric modeling;

however, we will revisit viscous eﬀects in the context of stabilization in Section 4.

Atmospheric models can be broadly classiﬁed into three groups: 1) non-hydrostatic models based

on the compressible Euler equations, 2) hydrostatic models, which assume a vertical momentum

balance between gravity and the vertical pressure gradient but include the vertical stratiﬁcation of

the atmosphere, and 3) sound-proof models. We also mention the shallow water model, which neglects

all vertical motion by assuming each column of air moves as a rigid body, as shallow water models

are often developed to test the horizontal propagation of features by numerical methods before they

are applied to the solution of the equations for a full atmosphere.

EBG in atmospheric modeling 17

The set of governing equations constitutes the dynamical core of the model. In the following

sections, we broadly survey the equation sets commonly used in existing operational and research

atmospheric models, beginning with non-hydrostatic models and ending with shallow water mod-

els. Please, also consult [322]. For a discussion of the interplay between the choice of equation set

and numerical challenges encountered, consult [299]. For an analysis of the diﬀerences between non-

hydrostatic, hydrostatic, shallow atmosphere (note, not to be confused with the shallow water model)

and deep atmosphere approximations, see [317].

2.1 Non-hydrostatic Models

The fully compressible Euler and Navier-Stokes equations model all the scales and motions of the

atmosphere. In NWP the equations expressed in the form of (1) are very often algebraically manip-

ulated via the introduction of derived physical variables to help the physical interpretation of the

atmosphere. For example, let us introduce potential temperature, θ, which is the temperature that

an air parcel would have if it were expanded or compressed adiabatically to a standard pressure p0=

1000 hPa [138]. Potential temperature is related to pand Tvia the expression θ=T /π, where

π= (p/p0)R/cp(5)

is a normalized pressure (known as Exner pressure) with respect to a reference pressure p0. Given θ,

the following conservation laws for (ρ, u, θ)Tare obtained by transforming Eq. (1c):

∂ρu

∂t +∇·(ρu⊗u)+ ∇p=−ρg−2ρ(ω×u),(6a)

∂ρ

∂t +∇·(ρu)=0,(6b)

∂ρθ

∂t +∇·(ρθu)=0.(6c)

The equation of state for pressure (4)

p=p0ρRθ

p0cp/cv

(7)

completes the system. Numerical methods for the solution of this system can be easily constructed to

conserve mass and momentum. It is, however, much more diﬃcult to formulate numerical schemes that

also conserve energy. However, for an adiabatic and reversible system, entropy is conserved. Entropy

smay be related to potential temperature θvia the relation

s=cplnθ+ constant,

thereby justifying the use of θrather than E.

The ARW-WRF model [271] is based on this set, and so are the ﬁnite volume model described

in [2], the Met Oﬃce ENDGame [323], and the German LM model [62]. The Nonhydrostatic Uniﬁed

Model of the Atmosphere (NUMA) [176; 117] developed at the Naval Postgraduate School is designed

around two diﬀerent sets, including (6). NUMA is the underlying dynamical core of the next generation

NWP model of the U.S. Navy, NEPTUNE.

Constructing the divergence of ﬂux in Eq. (6) requires some additional computational overhead;

this overhead may be reduced by converting Eqs. (6) to their advection form:

∂u

∂t +u·∇u+1

ρ∇p=−g−2ω×u,(8a)

18 Simone Marras1et al.

∂ρ

∂t +∇·ρu= 0,(8b)

∂θ

∂t +u·∇θ= 0,(8c)

again, completed by an equation of state given by Eq. (7). Numerical approximations to this set of

equations can be constructed to conserve mass, although conservation of momentum and energy is

more diﬃcult to obtain. NUMA is designed to be able to handle this set as well, although the ﬂux

form (6) is the required formulation when NUMA is executed in the discontinuous Galerkin mode.

By combining the deﬁnition of the Exner pressure (Eq. (5)) with the continuity equation in Eqs.

(8), we obtain:

∂u

∂t +u·∇u+cpθ∇π=−g−2ω×u,(9a)

∂π

∂t +u·∇π−R

cv

π∇·u= 0,(9b)

∂θ

∂t +u·∇θ= 0,(9c)

where (π, u, θ)Tis the vector of the solution variables [85; 71]. The practitioners who use this set

justify it by saying that it is self-contained because there is no need for a state equation. For as much

as it is evident that no equation of state is directly necessary, we still need to point out that the

algebraic computation of pfrom an equation of state similar to (4) is here simply substituted by the

diagnosis of pand ρfrom πor θ; an operation that is still necessary when it comes to the analysis

of the forecast. This still contributes to the net operation count to Eq. (4). Eqs. (9) do not conserve

mass, momentum, and energy; yet, they are widely used in operational NWP models such as MM5

developed at Penn State and NCAR [83], NMM based on the work by Janjic [166] at NCEP, COAMPS

[135] from the U.S. Naval Research Laboratory (NRL), and HIRLAM [256; 257] by a consortium of

European numerical weather services.

2.1.1 Sound waves: anelastic models and implicit time integration

All of the non-hydrostatic equation sets described in the previous section are compressible; therefore,

they all contain sound waves which propagate at a very high speed (approximately 300 m/s) relative

to meteorologically relevant phenomena. If these equations are discretized explicitly, a small time-step

must be utilized in order to satisfy the stability criterion based on the Courant-Friedrichs-Lewy (CFL)

condition [67], thereby increasing the computational cost of the model. Since the vertical grid-spacing

is typically much smaller than the horizontal grid spacing, the vertically propagating sound waves

are the most problematic aspect in these equation sets. To bypass the small time-step requirement

of the models that support sound-waves, yet preserve the remaining dynamics, the anelastic model

was introduced in 1953 by Batchelor [15] and later analyzed in [233; 202; 10], where the continuity

equation in Eqs. (6) and (8) is replaced by

∇·(ρ(z)u)=0.(10)

In (10), density ρis only a function of height. An improved soundproof approximation is the pseudo-

incompressible model proposed by Durran [84; 86], where the time dependence of density is ac-

counted for, although density is a function of a time-invariant reference state pressure and time-

dependent potential temperature. All these models are able to ﬁlter sound from the original com-

pressible Euler/Navier-Stokes equations, but still account for the most important waves (e.g. Rossby)

in the solution of the atmospheric motion. The interested reader may consult the review [179] for more

EBG in atmospheric modeling 19

on the validity of these approximations. A step towards the blending of soundproof and compressible

Euler equations was recently investigated in [20].

The soundproof approximation of the governing equations is one option to the necessary ﬁltering

of sound waves. The fully compressible, non-hydrostatic equations can, on the other hand, be approx-

imated in time via a semi-implicit scheme as done in [292; 71; 291]. Because the fast waves are treated

implicitly in a semi-implicit approximation, the time step is only limited by the non-linear advective

part of the equation; hence, the time-step is limited by the advective CFL condition ∆t ≤C∆x/||u||,

which is far less stringent than the CFL condition ∆t ≤C∆x/(||u||+cs), where Cis a constant of

order one and csis the speed of sound.

Semi-implicit methods are closely related to implicit-explicit (IMEX) methods [189]. Semi-implicit

is, for the most cases, tied to the combination explicit leap-frog + implicit Crank-Nicholson, whereas

IMEX can be viewed as a generalization that allows for diﬀerent time-diﬀerencing schemes, as ﬁrst

proposed in 2009 by Restelli and Giraldo [249] to solve the fully compressible Navier-Stokes of non-

hydrostatic stratiﬁed ﬂows approximated in space by DG. Moreover, the IMEX+DG by Restelli and

Giraldo is a general method applicable to diﬀerent Mach regimes for viscous and inviscid ﬂows. In

2004, Dolejši and Feistauer [76] coupled DG with an implicit-explicit time discretization scheme to

solve the Euler equations of fully compressible ﬂows. In that paper, the numerical ﬂux term was ﬁrst

discretized in a fully implicit manner; then, the implicit numerical ﬂux was linearized via a Taylor

expansion, resulting in a linear system of equations which is solved via a sparse iterative solver, as

opposed to a more expensive non-linear solver (e.g. Newton-Krylov) as required by a fully implicit

discretization. More recent work on IMEX methods includes [87], which utilizes Adams and backward

diﬀerence methods, and [314], which takes a horizontally-explicit vertically-implicit (HEVI) approach.

In 2013, an IMEX version of the Nonhydrostatic Uniﬁed Model of the Atmosphere NUMA was

introduced in [117]. Both a 3D IMEX scheme which discretized the horizontal and vertical (linear)

operators implicitly, and a 1D IMEX which only discretized the vertical operators implicitly (HEVI),

were derived and compared using both second-order backward diﬀerence formulas and higher-order

(up to order 4) implicit Runge-Kutta methods.

As mentioned earlier, 3D-IMEX methods require the solution of a linear system of equations.

This linear solve may be poorly conditioned (especially for large Courant numbers) and hence com-

putationally expensive. An alternative method that does not require a linear solve is the split-explicit

method [293]. The split-explicit approach relies on sub-time stepping to treat the terms that represent

sound and gravity waves within one larger explicit time-step for the remaining terms. This method is

common in atmospheric simulations, in spite of its low accuracy [318; 319] and potential instabilities.

2.1.2 Nearly-hydrostatic ﬂows

Dynamics in the atmosphere are characterized by small variations of thermodynamic quantities with

respect to some background state [207; 178]:

ρ(x,t) = ρ0(x,t) + ¯ρ(z)(11a)

p(x,t) = p0(x,t) + ¯p(z)(11b)

Θ(x,t) = Θ0(x,t) + ¯

Θ(z)(11c)

where the primed and barred quantities represent, respectively, the perturbation and the background

state of ρ,p, and Θ. In Eq. (11c), Θ=ρθ. In typical atmospheric simulations, ρ0¯ρ,p0¯pand

Θ0¯

Θ. If the vertical acceleration is zero, the vertical component of the momentum equation reduces

to the hydrostatic balance given by the following equation:

∂p

∂z =−gρ. (12)

20 Simone Marras1et al.

Given these considerations and the analysis of nearly-hydrostatic ﬂows for well-balanced methods

[30], the system (6) is transformed in terms of perturbation variables where the Coriolis term is

neglected. Substituting Eq. (11) into Eq. (6) and applying Eq. (12) to the z-component yields

∂ρu

∂t +∇·(ρu⊗u)+ ∇p0=−ρ0g,(13a)

∂ρ0

∂t +∇·(ρu)=0,(13b)

∂(ρθ)0

∂t +∇·(ρθu)=0.(13c)

Throughout this review, the primes will be mostly omitted to simplify notation.

2.2 Hydrostatic vs non-hydrostatic models

Atmospheric models can be distinguished as hydrostatic and non-hydrostatic. If we assume the vertical

acceleration to be negligible, the vertical momentum equation of the hydrostatic system reduces to

the diagnostic equilibrium equation (12). At every time-step, this time-independent equation is solved

instead of the full equation for vertical momentum. Sound waves are eliminated in the vertical direction

[85] but not in the horizontal direction. Because the size of the domain in the horizontal direction is

typically much larger than the vertical depth of the atmosphere and the grid size along xand ymay

be orders of magnitude larger than the grid spacing along z, a much larger time-step may be utilized.

The hydrostatic approximation has been a central approximation of NWP for the past four decades

and is used in the Hydrostatic Primitive Equations (HPE) discussed in the next section. This approxi-

mation is valid for horizontal grid spacing larger than 10 km [165; 299]. The hydrostatic approximation

is still appropriate to simulate synoptic scale phenomena where the vertical acceleration can be ne-

glected, but is no longer considered in any mesoscale simulation. With the availability of more powerful

computers, the non-hydrostatic formulations described above are standard for mesoscale NWP. The

reader should refer to [167; 21; 29; 106; 124; 135; 166; 271; 324; 118] for more on the evolution of

non-hydrostatic models.

2.2.1 Hydrostatic Primitive Equations

The hydrostatic primitive equations (HPE) govern the dynamics in synoptic scale (e.g. global-scale)

meteorology and are valid for horizontal resolution coarser than 10 km. The HPE are expressed in

so-called σcoordinates which allow the boundary condition on the ground to be easily applied, even

in the presence of complex orography. The HPE are derived by ﬁrst expressing the compressible Euler

equations in terms of pressure, velocity, and potential temperature. A hydrostatic balance is applied

in the vertical direction, which removes vertical acceleration from the momentum equation. Since the

HPE rarely appear outside of atmospheric and climate studies, we present a brief derivation from the

compressible Euler equations. A more comprehensive treatment is found in [138].

We ﬁrst apply a Coriolis term to the right hand side of the momentum equation (Eq. (8a)).

Decomposing the velocity uinto a horizontal uHand vertical wcomponents, the horizontal momentum

balance is given by

DuH

Dt =−1

ρ∇Hp−fk×uH(14)

where f= 2ωsinαis the Coriolis constant at the latitude αfor angular rotation ω. Next, we transform

Eq. (14) into isobaric coordinates (x,y,p), where pressure is the vertical component; this is a useful

EBG in atmospheric modeling 21

intermediate step on the path to σcoordinates. Introducing a velocity potential Φ(x,y, p, t), it can be

shown that ∇PΦ=∇p/ρ, where the gradient is taken with respect to isobaric coordinates, yielding

DPuH

DPt=−∇PΦ−fk×uH,(15)

where the total derivative in Eq. (15) is deﬁned as

DP

DPt=∂

∂t +uH·∇H+ ˆω∂

∂p (16)

and ˆω=Dp/Dt. Next, we transform Eq. (15) into σcoordinates via σ=p/ps, where ps=ps(x,y, t)

is the surface pressure. Note that in this coordinate system, the boundary condition on the ground is

always σ= 1. Application of the chain rule to the gradient of the velocity potential yields

∇PΦ=∇σΦ−σ∇lnps

∂Φ

∂σ (17)

while the total derivative is given by

Dσ

Dσt=∂

∂t +uH·∇H+ ˙σ∂

∂σ (18)

Combining Eqs. (17) and (18) in Eq. (15) yields

∂uH

∂t +uH·∇HuH+ ˙σ∂uH

∂σ =∇σΦ−σ∇ln ps

∂Φ

∂σ −fk×uH(19)

In a similar manner, an equation of continuity for the surface pressure psis derived from (6a)

∂ps

∂t +∇H·(psuH) +ps

∂σ

∂σ = 0 (20)

along with a transport equation for potential temperature θfrom (6c)

∂θ

∂t +∇·uH·∇Hθ+ ˙σ∂θ

∂σ = 0 (21)

In each σlevel, we solve for the prognostic variables q= (ps,uH, θ)T, while the diagnostic variables are

the vertical velocity ˙σ, pressure p, and geopotential φ. Because these equations are in exact hydrostatic

balance, there are no vertically propagating acoustic or gravity waves. By computing the eigenvalues

of the HPE, it is shown that the fastest moving waves are horizontally propagating gravity waves

[114]. Hence, even with an explicit time integrator, a much larger time-step may be used with the

HPE than with the compressible Euler equations. For this reason, the HPE form the basis of most

global atmospheric and climate models.

2.3 Shallow Water Equations (SWE)

The hydrostatic primitive equations require a solution at Nmodel levels (independent σor pressure

levels). This requires signiﬁcant computational eﬀort. The HPE may be simpliﬁed even further to

remove all vertical dependence. One approach is to expand each prognostic variable in Eq. (20) in

a 1D Fourier series with height σas the argument and only retaining the zeroth-term in this series,

commonly called the barotropic mode. Another approach is to start with the full compressible Euler

equations and apply both the hydrostatic approximation given by Eq. (12) and the shallow water

approximation where the deviation of the geopotential height Φfrom a given reference geopotential

Φ0is small. From an ocean modeling point of view, this assumption is equivalent to assuming the

22 Simone Marras1et al.

water depth is small compared to the wavelength of the waves of interest (gravity waves and Coriolis

induced Rossby waves). In ﬂux form, the SWE of a viscous atmosphere of depth hon a rotating sphere

of radius rare: ∂Φu

∂t +∇·(Φu⊗u) = −Φ∇Φ−f(x×Φu)+ µx+ν∇2(Φu),(22a)

∂Φ

∂t +∇·(Φu)=0.(22b)

Eq. (22) may be expressed in Cartesian coordinates instead of spherical coordinates by applying a

ﬁctitious force µx, where µis the Lagrange multiplier; this approach, which facilitates an arbitrary

spherical grid, was ﬁrst proposed by Coté [63] for the semi-Lagrangian solution of the problem and

later used in [110; 112] for the solution of the full nonlinear equations. The numerical solution of SWE

on spherical geometries is reported by many authors such as [242] (FEM), [204; 158; 294; 112] (SEM),

[116; 227; 303] (DG), [211] (uniﬁed CG/DG on diﬀerent unstructured grids with static and dynamic

adaptivity), [191; 309; 316; 196] (FV), [278] (comparison between SEM and FV), [245] (comparison

using diﬀerent numerical methods).

2.4 Transport in the atmosphere

In a typical atmospheric model, there are multiple forms of water (e.g. vapor, rain, ice); in a climate

model, there are also hundreds of chemical species. These quantities are transported and diﬀused

by atmospheric dynamics and are classiﬁed as tracers. In turn, these tracers actively feedback to

dynamics (e.g. latent heat release). To model these tracers, the governing equations of a dry ﬂow

must be coupled to a set of transport-diﬀusion equations for such tracers. For simplicity, we describe

how tracers are treated in atmospheric models by looking at the transport of three water quantities

only; however, this approach applies to an arbitrary number of tracers.

Let us deﬁne the mixing ratios of water vapor, cloud water, and rain as qv=ρv/ρ,qc=ρc/ρ and

qr=ρr/ρ, where ρv,c,r are the densities of water vapor, cloud, and rain. Let us also choose one of

the nonhydrostatic equation sets described previously and write the coupled system of equations that

model a moist atmosphere; we consider system (8) and write the following:

∂u

∂t +u·∇u+1

ρ∇p=−g(1+ qv−qc−qr)−2ω×u+Sturb,(23a)

∂ρ

∂t +∇·(ρu)=0,(23b)

∂θ

∂t +u·∇θ=∇·(κθ∇θ) + Sθ(ρ,θ,qv,qc, qr),(23c)

∂qi

∂t +u·∇qi=∇·(kqi∇qi) + Sqi(ρ,θ,qv,qc, qr),for i=v, c, r, (23d)

where =R/Rvis the ratio of the gas constants of dry air, R, and of water vapor, Rv. Moist air

contributes to the buoyancy of the ﬂow, so that the right hand side of the momentum equation must

be corrected by the total buoyancy B=−gg(1 + qv−qc−qr). The diﬀusion coeﬃcients kθiand kqi

are typically modeled via an algebraic turbulence closure via

kθ=ν/Pr0+νt/Prt(24a)

kqi=ν/Sc0+νt/Sct(24b)

while the closure term Sturb depends on the turbulence model employed. In Eq. (24), νis molecular

viscosity, νtis eddy viscosity, Sc0is the molecular Schmidt number and Sctis the turbulent Schmidt

number. Typical values are Sc0=S ct= 0.7. The microphysical processes that involve phase change

EBG in atmospheric modeling 23

in the water content are modeled by the source/sink terms, Sθ,qi, in the equations. For example, in

the case of water vapor, Siis driven by evaporation and condensation. These terms can be modeled

and computed by some properly designed microphysics scheme, such as the Kessler [177] scheme for

warm clouds (no ice involved).

The appropriate numerical discretization of Eqs. (23d) is still an active topic of research, espe-

cially since moisture possesses large gradients that can cause instabilities. In addition, since the mass

fractions qiare a priori non-negative, the numerical discretization should be monotonic or, at the

very least, positivity-preserving. If, for example, our system produced negative moisture, the physical

parameterization would have to resolve this issue in some way (e.g. clipping the negative values); in

addition, the resulting incorrect feedback may pollute the overall solution and cause artiﬁcial rain to

be produced by the model. The words of John P. Boyd are an amusing conclusion to this paragraph:

"[...] Clever adaptive algorithms that work for smooth, straight shocks disintegrate into computational

anarchy when ﬂayed by gravity waves, assaulted by moist convective instability, battered by highly

temperature-sensitive photochemistry, and coupled to the vastly diﬀerent time and space scales of

the ocean[...]" (SIAM News, Multiscale Numerical Algorithms for Weather Forecasting and Climate

Modeling: Challenges and Controversies. Nov 2008, Vol.41 issue 9). Monotonic solutions are certainly

more diﬃcult to achieve with high order numerical methods. The problem is particularly challenging

when the transport equation is solved by high order methods such as spectral elements or DG. High

order methods produce Gibbs oscillations near sharp gradients; these oscillations are unphysical and

are exacerbated by increasing the order. Hence, limiters [187] or adaptive ﬁltering is necessary. We will

address this problem in Section 4, along with some issues involved with unstable Galerkin solutions.

2.4.1 Cloud microphysics: Kessler parameterization

Cloud microphysics include all thermo-physical processes at the scales of the particles that form the

cloud. Examples are the phase change of water quantities or the agglomeration of particles into larger

ones. Most physical processes typical of storm dynamics (e.g., precipitation, freezing, deposition,

or sublimation) have physics across a large range of spatial and temporal scales that makes direct

numerical simulation unfeasible (see [88], Ch. 10). For this reason, parameterization is commonly used

within numerical models. Microphysical parameterization relies on the physical knowledge of certain

processes without the need for fully resolving all the microscale processes that are involved. The clear

limitation is that certain phenomena cannot be represented with high accuracy if they lie outside

of the conditions required by the parameterization. diﬀerentiation A simple representation of cloud

microphysics was designed by Kessler and reported in his monograph [177].

Kessler’s is a bulk model, meaning that water species are categorized only with respect to the

particles’ type. In other words, if we speak about cloud water, we would model it through one equation

that represents the transport of cloud water concentration with water droplets of one single size. Bulk

models are contrasted by explicit models, where, within each category (e.g., cloud, rain) the size of

the water particles is considered as well. Explicit models are certainly more physically accurate, but

they are more costly due to the greater number of quantities that must be accounted for. For more

information on the topic refer to Houze’s book [139] and to more recent literature (e.g. [39]).

Kessler’s is a simple scheme based on the main assumption that ice is not contemplated (warm

rain). The main limitation of the warm condition is that only moist convection at the tropics or at

mid-latitudes in the warm season can be represented. The three forms of water that are considered

are: (i) water vapor; (ii) cloud water (liquid water whose size is so small that its terminal fall speed

is negligible); and (iii) precipitating water that only includes rain (namely, drops whose diameter is

>0.5mm). Drizzle is excluded (rain of drop diameter between 0.2and 0.5mm).

The main processes resolved by a warm cloud microphysics scheme are brieﬂy described below.

These processes dictate how the source terms of the previous equations are deﬁned and how they

aﬀect the dynamics of the simulation. The reader is referred to, e.g., [139] and references therein for

a more thorough analysis.

24 Simone Marras1et al.

Given the approximated Teten’s formula [27] for the saturation vapor pressure,

e∗= 611.2exp17.67T

T+243.5,

the saturation mixing ratio is given by

qvs =e∗

p−e∗.(25)

From [181], the source terms in (23) are

Sθ=−Lv

cpT( ˙qv s +Er),(26a)

Sqv= ˙qv s +Er,(26b)

Sqc=−˙qv s −Ar−Cr,(26c)

Sqr=1

ρ

∂

∂z (ρVrqr)−Er+Ar+Cr,(26d)

where cpl and cpv are the heat coeﬃcients at constant pressure of liquid water and water vapor,

respectively, Lv=Lv0−(cpl −cpv )(T−T0)is the latent heat of vaporization with reference value

Lv0= 2.5e+6 J kg−1,T0is a reference temperature, Vris the terminal fall speed of raindrops (taken

positive in the downward direction), and ˙qvs is the rate of condensation or evaporation (the dot

symbol indicates diﬀerentiation with respect to time). Ar,Cr, and Erare the rates of autoconversion,

collection, and evaporation of rain. They are computed using the formulas:

Ar=MAX (0,k1(qc−aT)),(27a)

Cr=k2ρ0.375 qcq0.875

r,(27b)

Er=−1

ρ

(qv/qvs −1)k(ρ qr)0.525

5.4×105+2.55 ×106(pqvs ),(27c)

where k1= 0.001s−1,k2= 2.2s−1,aT= 0.001kg k g−1are Kessler’s parameters and kis the ventilation

factor that is a function of the terminal fall speed. Eq. (27a) was derived by Kessler considering that

a cloud is converted into rainwater whenever qcexceeds a threshold aT.Autoconversion is the rate at

which the rain water content increases at the expense of cloud water due to the coalescence of smaller

particles. Yet, this process is not fully understood. Nor is it fully understood how collection occurs. As

the name suggests, collection can be explained as cloud water particles being collected by the falling

larger rainwater droplets that go through the cloud layers during their fall. Evaporation occurs when

the sensible heat ﬂux from the environment into the water droplet is balanced by the latent heat of

evaporation of the water particle. As in [276], the cloud droplets move at the same speed of the ﬂow

because they are considered having negligible terminal velocity.

The values of the constants in (27) are, to a certain extent, arbitrary [139]; however, by the

observations, it is of common agreement that k1, k2and aTare non-linear terms with respect to qc

itself. They are also a function of temperature and of the distribution of the condensation nuclei.

As it is pointed out in Emanuel’s book [88], the lack of understanding of the underlying physics is

such that diﬀerent results are being obtained by diﬀerent and more sophisticated schemes. However,

this topic is beyond the scope of the present article. Nevertheless, it is important to emphasize that

microphysical parameterization has a major role in forecasting clouds and precipitation, but is still

an active ﬁeld of investigation (see the 2008 paper by Morrison and Grabowski [223]).

EBG in atmospheric modeling 25

2.4.2 Method of solution via saturation adjustment

Regardless of the type of space approximation, phase changes are classically treated via the saturation

adjustment technique explained in detail in the appendix of [276]. Saturation adjustment –or fractional

step method– is not the only option; however, due to its simplicity, it is convenient to describe it here

to give a sense of how phase change is accounted for in these models.

The saturation adjustment technique consists of solving the problem in two steps. First, the

prognostic equations are solved by neglecting all the terms that involve phase changes (all the S-

terms are set to zero). This means that the dynamics and transport equations are advanced for-

ward to an intermediate time-step n∗so that the intermediate values of the prognostic variables,

(ρ,p,θρ, qvs , qv, qc, qr)∗, are obtained. These values are plugged into the Kessler module to compute

the S-quantities deﬁned above. Once the computation of Shas completed, thermodynamic variables

are updated and returned to the Euler/transport solver as the initial values for the next time step

n+1.

26 Simone Marras1et al.

3 Element-based Galerkin methods: ﬁnite elements, spectral elements, and nodal

discontinuous Galerkin

As discussed in the EBG roadmap, the ﬁnite element (FEM), spectral element (SEM), and discontin-

uous Galerkin (DG) methods, are speciﬁc types of Galerkin approximation techniques. In this section,

we introduce the ideas behind Galerkin schemes in general and then distinguish between FEM, SEM,

and DG in particular. We then trace the history of EBG methods in NWP and climate modeling over

the past twenty years.

3.1 Element-based continuous Galerkin methods

The birth of Galerkin methods dates back to Boris Grigoryevic Galerkin and his work on the numerical

solution of the equations of the elastic equilibrium of rods and plates [105], and to the original

ideas of Walter Ritz [253] six years earlier. Popularized by Courant in the early 40s for the study of

vibration and equilibrium [66] and extensively developed only in the late 1950s and 1960s by structural

dynamicists in the aircraft industry [4], ﬁnite element methods in particular are among the most

common numerical methods based on a Galerkin approach and that are used today in a wide range of

applications. They are used in industry and for research purposes in, e.g., structural analysis [325], ﬂuid

dynamics [334], and electromagnetism [14]. Galerkin based methods are a robust tool for the solution

of any diﬀerential problem [79] and are accepted by scientists and engineers in theoretical studies

and applications for a series of reasons such as the ease in modeling complex geometries, the ﬂexible

and general purpose programming format that they imply, and the intrinsic treatment of diﬀerential-

type boundary conditions. In the following, we will describe the idea behind the method of weighted

residuals, of which the Galerkin ﬁnite element, spectral element, and discontinuous Galerkin methods

represent special cases. For a simple but quasi-rigorous analysis of the method we use a problem

of real engineering interest and that is a fundamental problem in numerical weather prediction: the

advection-diﬀusion equation. The reader is referred to the books by Fletcher [95] or by Karniadakis

and Sherwin [173] as a reference for the more mathematical aspects of Galerkin methods, and to

the lecture notes by Giraldo [115] for a uniﬁed treatment of high-order continuous and discontinuous

Galerkin methods.

Let us take a general diﬀerential problem

L(q) = S, (28)

where Lis the combination of both linear and non-linear diﬀerential operators in space xand time

t, and Sis a source function. Let dindicate the space dimension and let Ω⊂Rdbe the domain with

the boundary ∂Ω where (28) is deﬁned within the time interval (0, tf), and tf∈R+. For the problem

to be well-posed, suitable boundary and initial conditions must be added to (28). Unless otherwise

stated, given a known function g, Dirichlet boundary conditions q(x) = gfor x∈∂ Ω will be applied

to the problems described throughout this section.

As previously stated, Galerkin methods are a particular case of the method of weighted residuals.

The idea behind this method is the numerical representation of the solution variable qby a ﬁnite

dimensional approximation qhobtained by the expansion

qh(x) =

N

X

k=0

ψk(x) ˆqk,(29)

where Nis the number of nodes pkof a possible partition of the domain Ω. On its discrete counterpart,

Ωh, a set of k= 0,...,N known analytic test functions ψkare deﬁned (The two terms test and basis

will be used interchangeably. The unknown coeﬃcients ˆqkcorrespond to the physical values of qat

node pk. The ﬁnite diﬀerence method is conceptually diﬀerent in that what is approximated in the

EBG in atmospheric modeling 27

diﬀerential problem are the diﬀerential operators and not the solution variable. Substitution of (29)

into (28) is such that L(qh)−S6= 0. The method is called method of weighted residuals because a

linear system of algebraic equations in the unknowns ˆqis built by imposing that

ZΩ

w R dΩ = 0,(30)

where R=L−Sis the (non-zero) residual of (28) and wis the weight function that has certain

properties. Diﬀerent methods arise from the selection of diﬀerent w. The Bubnov-Galerkin method is

found when w=ψk. We can then write the following:

ZΩ

ψ[L−S]dΩ = 0.(31)

This is the weak form of the original equation to be solved.

Remark 3.1. So far, no distinction between the ﬁnite and spectral element methods has been made.

The diﬀerence stems from the deﬁnition of the interpolation points used to construct ψk.

3.1.1 Suitable function spaces

The choice of basis and test functions depend on the operator Lunder consideration. In the speciﬁc

cases of the advection-diﬀusion equation and the Navier-Stokes equations of compressible ﬂows, the

highest order of the derivatives is 2, and the choice of the basis functions and the space to which they

belong must depend on this regularity condition.

We show that the weak solutions to a linear elliptic operator must resides in the Sobolev space

H1(Ω). Consider an operator Ldeﬁned on a global domain Ωwith boundary Γacting on a state

vector q; speciﬁcally, consider the elliptic operator

L(q) = ∇·(ν∇q)(32)

where ν > 0is the kinematic viscosity. We consider the boundary-value problem

L(q)=0 (33)

with a Dirichlet boundary condition q(x) = q0(x)for all x∈Γand q0(x)∈C1(Γ). Consider a test

function ψ∈L2(Ω)and also assume q∈L2(Ω). The following calculation demonstrates that q,ψ ∈

H1⊂L2.

Integrating Eq. (32) by parts yields

ZΩ

ν∇ψ·∇q dΩ =ZΓ

ψ∇·(ν∇q)dΓ. (34)

Since q0∈C1and ψ∈L2, the right hand side of Eq. (34) is bounded, implying that the left hand side

is bounded as well. We then write that:

ZΩ

ν∇ψ·∇q dΩ ≤ZΩ|ν∇ψ·∇q|dΩ (35)

≤νk∇ψkL2k∇qkL2

where the second line follows from the Cauchy-Schwartz inequality. Hence, both the norms k∇ψkL2

and k∇qkL2are bounded, implying that ∇ψand ∇qare square-integrable over the global domain Ω.

In other words, ψ,q ∈H1(Ω).

With regards to CG methods, this elementary calculation illustrates two key points:

28 Simone Marras1et al.

Fig. 7: Mapping from reference, (ξ, η), to physical space, (x, z).∀Ki∈ P h:Ki=Hi(I).

1. The space of test functions must be a subset of H1.

2. Since H1⊂C0, the state vector qis necessarily continuous.

We hence deﬁne the space Wof test functions ψand trial solutions qas a subset of H1such that

W.

={ψ, q ∈H1(Ω) s.t. ψ = 0 and q=gon ∂Ω}.(36)

3.2 Finite and Spectral Elements: discretization and basis functions

To discretize the problem in a ﬁnite and spectral element sense, the domain tΩis ﬁrst decomposed

into a ﬁnite element partition Ph={Ki}i=1,...,nel of nel conforming elements Kisuch that

Ω=

nel

[

i=1

Ki,and

nel

\

i=1

Ki= 0,(37)

where every element Kiis the image of the reference element Iby a non-singular bijective mapping

x=Hi(ξ)from physical space xto computational space ξ.J= dx/dξis the transformation Jacobian

matrix. A two-dimensional example of the map is represented in Fig. 7.

The need for mapping is purely practical and forms the foundations of the ﬁnite element compu-

tation. For details see [157].

Basis functions: Finite Elements. Lagrange basis functions are a common choice in ﬁnite elements

since they interpolate a continuous function exactly at the nodes xl. These functions, deﬁned by hk

from now on, have the property of being piecewise continuous and are such that

hk(xl) = δkl k, l = 0,...,N,

where δkl is the Kronecker delta.

For linear, quadratic, and cubic ﬁnite elements, the roots of the basis function along the reference

element Iare the N+1 equi-spaced nodes within the element. Using the deﬁnition of the Lagrange

polynomials

hk(ξ) =

N

Y

l=0,l6=k

ξ−ξl

ξk−ξl

,(38)

EBG in atmospheric modeling 29

in Fig. 8 we plot hkalong a reference element up to 2nd-order. A 4th-order ﬁnite element and corre-

sponding basis function are plotted in Fig. 9 (left).

Basis functions: Spectral Elements. Unlike the case of high-order ﬁnite elements, the polynomials

used with spectral elements are associated with zeros that are not equi-spaced. A convenient set is

represented by the Legendre-Gauss-Lobatto (LGL) points. LGL nodes ξiare the roots of

(1−ξ2)P0

N(ξ)=0,(39)

where PN(ξ)are the Nth-order Legendre polynomial whose construction by recursive formulas can be

found in [173]. The polynomials that are used have the same δ-property of the Lagrange polynomials

deﬁned above. Their analytic expression is given by

hk(ξ) = (ξ2−1)P0

N(ξ)

N(N+1)(ξ−ξk)PN(ξ), k = 0,...,N, (40)

where P0indicates diﬀerentiation with respect to x7. The 4th-order k-polynomials along I= [−1,1]

are plotted on the right panel of Fig. 9. The comparative plot (ﬁnite element on the left and spectral

element on the right) is used to show that, if high-order is required, equi-spaced nodes produce

unsatisfactory types of basis functions in the proximity of the edge points of the element. In other

words, we lose control on the maximum and minimum values of hkat the extrema of the element.

When this happens, interpolation of any function is likely to suﬀer from such a condition. To show

how this feature translates into the interpolation of a known analytic function, we use the following

example from [115]. We deﬁne the Witch of Agnesi of unitary height as

z(x) = 1

1.0+ 50x2,

where z(x)is smooth and continuous, and interpolate it using the basis functions ψ(x) = hk(x)deﬁned

above. The test is performed by 4th-order interpolation. Equi-spaced and non equi-spaced points are

used along the unitary domain. Fig. 10 shows that the more the polynomial order is increased, the

better the result is when LGL nodes are employed. This is tied to the deﬁnition of the Lagrange

polynomials and their interpolation strength given by the Lebesgue constant. The reader is referred

to [115] for more details on this issue. Roughly speaking, this analysis serves as a practical way of

showing one reason for the use of LGL points in high-order simulations rather than high-order elements

with evenly distributed nodes. Fig. 11 is a schematic representation of two 4th-order elements in two

dimensions.

3.3 Discontinuous Galerkin

The discontinuous Galerkin method allows the numerical solution and, therefore, the basis functions

to be discontinuous at the interface between neighboring elements. For this reason, the basis function

is no longer required to live in H1but, rather, in L2. Assume niis the number of elements that share

grid point i. Then we will have nidiﬀerent values of the solution at that grid point; one coming from

the computation on the left element and one on the right element, where left and right are deﬁned

with respect to the shared edge (or face in 3D). The basis functions for element Ωevanish everywhere

outside the element. Hence Eq. (31) becomes a set of nel independent equations for each element Ωe:

ZΩe

ψ[L(q)−S(q)] dΩe= 0.(41)

7Alternatively, the basis functions can be constructed using Eq. (38).

30 Simone Marras1et al.

-1.0

-0.5

0.0

0.5

1.0

-1 -0.5 0 0.5 1

h(x)

x

1st order Lagrange Basis Functions

-1.0

-0.5

0.0

0.5

1.0

-1 -0.5 0 0.5 1

h(x)

x

2nd order Lagrange Basis Functions

Fig. 8: Lagrange polynomials of order 1 (left) and 2 (right) along the 1D reference element I= [−1,1]. Clearly,

they are equivalent for FE and SE.

-1.0

-0.5

0.0

0.5

1.0

-1 -0.5 0 0.5 1

h(x)

x

4th order Lagrange Basis Functions

-1.0

-0.5

0.0

0.5

1.0

-1 -0.5 0 0.5 1

h(x)

x

4th order Lagrange Basis Functions

Fig. 9: Basis functions of order 4 along the 1D reference element I= [−1,1]. Left: the nodes within the element

are equi-spaced as for classical high-order FE. Right: Lagrange-Legendre polynomials of order 4 whose roots are

the non-equi-spaced Legendre-Gauss-Lobatto (LGL) quadrature points. Nodal SE and DG may employ LGL or

LG quadrature. However, to obtain a diagonal mass matrix then LGL is the only choice for SE, while LG can

still be used for DG.

The equations for the diﬀerent elements are coupled by means of the ﬂuxes between neighboring

elements. For this purpose we write our equations in ﬂux form:

L(q) = ∂q

∂t +∇·F(q)(42)

EBG in atmospheric modeling 31

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

-1 -0.5 0 0.5 1

qh, qe

x

4th-order Interpolated vs Exact Solution

qe

Equisp. qh

LGL qh

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

-1 -0.5 0 0.5 1

qh, qe

x

10th-order Interpolated vs Exact Solution

qe

Equisp. qh

LGL qh

Fig. 10: Interpolation of a known function (Witch of Agnesi) using high-order interpolating functions with equi-

spaced and LGL points. Left: 4th-order interpolation. Right: 10th -order interpolation

Fig. 11: Nodes disposition for a two-dimensional 4th-order ﬁnite element (left), and spectral element (right).

where Fis the ﬂux tensor. Eq. (41) becomes

ZΩe∂q

∂t +∇·F−Sψ(x)dΩe= 0.(43)

with F=F(q)and S=S(q). Integration by parts leads to

ZΩe∂q

∂t −F·∇−Sψ(x)dΩe=−ZΓe

ψ(x)n·FdΓe,(44)

where Γeis the boundary of Ωeand nis the outward pointing unit normal vector on Γe. To compute

the numerical solution, qh, we replace the ﬂux in the boundary integral by a so called numerical ﬂux

Fh∗:

ZΩe∂qh

∂t −Fh·∇−Shψ(x)dΩe=−ZΓe

ψ(x)n·Fh∗dΓe,(45)

32 Simone Marras1et al.

with Fh=Fqhand Sh=Sqh. The numerical ﬂux Fh∗describes the ﬂux through the discon-

tinuous interface between neighboring elements in the same way of the ﬁnite volume (FV) method;

therefore, we can choose any of the ﬂuxes that are used with FV. For an introduction to diﬀerent

choices of ﬂuxes we refer to [302]. Unlike FV, DG is relatively insensitive to the choice of the numer-

ical ﬂux due to the high order (p≥3) basis functions that are used within the element. Therefore a

common choice for the numerical ﬂux is the simple Rusanov ﬂux:

Fh∗=1

2Fqh

L+Fqh

R−λnqh

R−qh

L,(46)

where λ=||u||2+cis the maximum wave speed, ||u||2=√u2+v2+w2, and cis the speed of sound.

The subscript Ldenotes the index of the element Ωewhereas the subscript Rdenotes the index of the

neighboring element. There are recent approaches to incorporate ﬂuxes that are not perpendicular to

the interface between elements [328].

The integration by parts of Eq. (45) leads to

ZΩe∂qh

∂t +∇·Fh−Shψ(x)dΩe=ZΓe

ψ(x)n·Fh−Fh∗dΓe.(47)

Using the expansion (29) of the numerical solution gives us

∂ˆqk

∂t =−ZΩe

ˆ

ψk(x)∇·Fh−ShdΩe+ZΓe

ˆ

ψk(x)n·Fh−Fh∗dΓe,(48)

with the deﬁnition ˆ

ψi(x) = PMh

k=1 M−1

ik ψk(x)where Mik =RΩeψi(x)ψk(x)dΩeare the components

of the mass matrix M.

Second order derivatives in the diﬀerential operator Lcan be discretized with DG by transforming

the problem into a coupled set of equations containing only ﬁrst order derivatives as done in [54]. This

approach is called the local DG (LDG) method. Other choices for discretizing second order operators

are given in [266].

3.4 EBG methods in atmospheric and climate modeling

3.4.1 Continuous Galerkin

The use of continuous Galerkin methods in atmospheric simulations began ﬁve decades ago with

the work on ﬁnite elements by Holmstrom [137] and Simons [269] in the 60s. This continued in the

70s (e.g. [101; 69; 70]) and was followed by an extensive production of articles in the 80s and 90s

with, e.g., Staniforth, [279], Beland et al. [19], or Burridge et al. [41], who set the foundations of the

operational Global Environmental Multiscale (GEM) model [65; 327] of the Canadian Meteorological

Center &Meteorological Research Branch (CMC-MRB). In the UK, Untch and Hortal [305] used

ﬁnite elements for the vertical discretization of a semi-Lagrangian transport scheme and introduced it

in the operational version of the European Centre for Medium-Range Weather Forecasts (ECMWF)

global spectral model (IFS), with great improvement with respect to the FD version of the code. In the

domain of Geophysical Fluid Dynamics, more Galerkin-type models have appeared since the beginning

of the new millennium. In, e.g., [34; 193; 228] or [119], diﬀerent variational formulations mostly based

on spectral elements are employed to solve the shallow water equation or the Navier-Stokes and Euler

equations for non-hydrostatic atmospheres. More examples of element-based models are the SE-Core

[296], CAM-SEM by [74], the SE/DG Nonhydrsotatic Uniﬁed Model for the Atmosphere (NUMA)

[176; 117], the SEM Community Earth System Model (CESM) [73], the ﬁnite element multi physics

model ALYA in atmospheric-mode [212; 213].

EBG in atmospheric modeling 33

Possibly, the spectral element method is the most common EBG method used today to develop

the next generation research NWP models. Spectral elements were ﬁrst introduced in geophysical ﬂuid

dynamics by Ma in 1993 [204]. Ma built on the pioneering work of Patera [235], who developed the

spectral element method for incompressible CFD and developed an ocean model based on the shallow

water equations. In particular Ma stressed the ability of SEM to 1) accurately simulate ﬂows with high

Rossby numbers and 2) simulate phenomena with long time durations due to SEM’s low dissipation

and dispersion error. Although Ma’s primary application was coastal ocean modeling, he was explicitly

aware of the intimate connection between ocean models and atmospheric/climate models and predicted

that his work would serve as a basis for atmosphere and climate studies. Iskandarani [158] built on

Ma’s ocean model, showing that the accuracy of spectral elements successfully suppressed spurious

pressure modes in ocean ﬂows. Both Ma’s and Iskandarani’s work utilized Cartesian grids suitable for

oceanographic problems. Two years later, Taylor, Tribbia, and Iskandarani [294] developed the ﬁrst

SWE spectral element model using spherical geometry. In particular, this work used a cubed sphere

with quadrilateral elements which built on the geometrical ﬂexibility of spectral elements. The cubed

sphere grid circumvented the well-known pole problem that is present for traditional latitude-longitude

grids (we will get back to this point shortly). At the same time, a spectral element shallow water code

was developed by Haidvogel et al.[131]. As a result, this methodology was extended to solving the

hydrostatic primitive equations on the sphere in the Spectral Element Atmospheric Model (SEAM)

[98]. By this time, massively distributed memory clusters had become available, thus motivating the

development of highly scalable numerical methods; Fournier and coworkers noted the high parallel

eﬃciency of spectral elements, thus making SEM a suitable numerical method for the dynamical core

of climate models, which are computationally expensive. Taylor’s SEM solver later became the basis

for the NCAR’s high-order method modeling environment (HOMME), which facilitated the rapid

development of next generation atmospheric global circulation models (AGCM).

Taylor’s SEM model utilized spherical coordinates to solve atmospheric problems on the sphere;

however, since the sphere is a sub-manifold of three-dimensional space, Cartesian coordinates can

also be used to solve problems on spheres provided that the ﬂuid is constrained to lie on the sphere

using a Lagrange multiplier [63]. Although computationally more expensive because there is an extra

degree of freedom, this approach has two advantages over spherical coordinates: 1) analytical Jacobian

transformations for the grid do not need to be derived and 2) any spherical grid (including the cubed

sphere grid) may be utilized, thereby liberating the solver from the grid. Giraldo utilized this Cartesian

SEM approach to solve the SWE using an Icosahedral grid in [113]. Collaborating with the Naval

Research Lab, he applied this framework to solve the hydrostatic primitive equations in [119] and

develop the U.S. Navy’s spectral element atmospheric model (NSEAM) including a semi-implicit solver

[114]. As we shall see in the next section, Giraldo and coworkers developed DG concurrently with SEM

solvers, thereby exposing the common themes and machinery shared by the two methods. To prove

how arbitrary grids can be used to solve the SWE on the sphere, a uniﬁed continuous/discontinuous

Galerkin model has been recently presented in [211]. Using this model, the equations were solved in

Cartesian coordinates using static and dynamic adaptivity using both, continuous and discontinuous

Galerkin approximations on the grids illustrated in Fig. 12.

3.4.2 Discontinuous Galerkin

This subsection presents a short overview of the important steps in the historical development of

DG towards atmospheric applications. A more general overview of the history of DG until the year

2000 can be found in [52]. Some information can also be found in the textbook by Hesthaven and

Warburton [133].

The discontinuous Galerkin method was ﬁrst introduced by Reed and Hill in 1973 [248]. Reed

and Hill were working on the solution of the stationary linear transport equation of neutrons with a

34 Simone Marras1et al.

Fig. 12: Examples of spherical grids for the solution of the SWE. From left to right, classical cubed-sphere, a

reduced longitude-latitude, and icosahedral grid. These are high order grids with curved elements on the spherical

shell. Figures adapted from [211] with permission of John Wiley & Sons.

constant velocity vin two dimensions:

µ∂ψ

∂x +η∂ψ

∂y +σ ψ (x, y,µ, η) = S(x, y,µ, η),(49)

where ψis the angular neutron ﬂux in the direction (µ,η)T=v/||v||2, the total macroscopic cross-

section for neutron-nucleus interaction σand the source term S. The source term describes scattering,

ﬁssion and inhomogeneous sources. This transport equation was solved by Reed and Hill on a triangular

mesh. They compared a method allowing discontinuities at the interfaces between diﬀerent triangles

with a continuous method and found that DG was computationally more expensive but slightly more

accurate and much more robust. One of the main advantages of the discontinuous method was that it

showed fewer oscillations at the boundary between areas with two diﬀerent values of the cross-section

σ. This allowed Reed and Hill to reduce the resolution of the method while still obtaining a less

oscillatory result than the continuous method allowed.

The discontinuous method introduced in [248] was analyzed theoretically by Lesaint and Raviart

[198]. In this early work the discontinuous Galerkin method was applied to linear equations. The

ﬁrst application of DG to nonlinear conservation equations is attributed to Chavent and Salzano

[48]. Chavent and Salzano used ﬁrst order polynomials for spatial discretization and a simple explicit

Euler method for time discretization. A von Neumann analysis showed that this approach is unstable

if the time step ∆t is proportional to the grid spacing ∆x. This approach becomes stable only for

∆t ∝∆x3/2. This severe restriction for the time step with explicit time integration was solved by the

development of Runge-Kutta discontinuous Galerkin methods (RKDG) by Cockburn and Shu [53].

Discontinuous Galerkin methods were applied to parabolic equations by Jamet in 1978 [163],

displacement of oil by water in a porous slab by Chavent and Salzano in 1982 [48], viscoelastic ﬂows

by Fortin and Fortin in 1989 [97] and to the solution of the Maxwell equations by Warburton and

Karniadakis in 1999 [312].

The ﬁrst numerical experiment using DG for the Euler equations of gas dynamics is reported in

the 1991 work by Bey and Oden [26] and by Bassi and Rebay in 1997 [12; 13]. The ﬁrst application

of DG to geophysical problems started with the work on shallow water equations by Schwanenberg et

al. [264], followed by Giraldo et al. [116] who introduced inexact integration for DG and extended this

to the sphere. In 2008, DG was ﬁnally used to solve the Navier-Stokes equations of non-hydrostatic

atmospheric ﬂows by Giraldo and Restelli [118]. Discontinuous Galerkin methods have not been used

in operational global circulation models (GCM) yet. However, a hydrostatic GCM was presented by

Nair et al. in 2009 [225], followed in 2011 by the German DG model DUNE [34], and, in 2012, by the

scalable non-hydrostatic model by Kelly and Giraldo [176]. The linear scalability properties of DG

was shown by Wilcox et al. in 2010 [320] and Kelly and Giraldo in 2012 [176].

EBG in atmospheric modeling 35

As mentioned earlier, the geometrical ﬂexibility of EBG, including the potential for adaptivity, is a

major strength of EBG methods–in particular, DG. An adaptive DG model based on non-conforming

quads was introduced in [184]. This paper introduced a tree-based adaptive mesh reﬁnement (AMR)

strategy, demonstrated the potential for DG to achieve order-of-magnitude eﬃciency gains, which are

diﬃcult, if not impossible, with traditional ﬁnite diﬀerence or spectral transform methods. Dynamic

adaptivity, using conforming triangular elements, was also explored in [224]. These two papers are not

the ﬁrst ones that report on grid adaptivity using EBG; however, they seem to be the ﬁrst publications

on this topic in the framework of non-hydrostatic atmospheric simulations using DG.

4 Stabilization of EBG for advection-dominated problems

The straight numerical approximation of problems with dominating advection may result in unphysical

oscillations in the solution. Finite and spectral element methods are no exceptions [170] and an error

estimate of the standard Galerkin approximation of the problem proves it (see, e.g., [247]). Here, we

show it by deriving the 1D ﬁnite element solution of the advection-diﬀusion problem with Dirichlet

boundary conditions. The problem consists in solving

∂q

∂t +L(q) = S , (50)

where

L(q) = u·∇q−∇·(ν∇q),(51)

by linear (p= 1) ﬁnite elements. In (65), νis a positive, uniform, constant diﬀusivity coeﬃcient,

u= (u,0,0) is the velocity vector, and Sis a source function. The domain of interest is the unit interval

Ω= [0,1]. A uniform partition Phof Ωwith N+ 1 nodes pk,k= 0,...,N, and nel =Nelements K

of length h=kpk−pk−1k2is assumed. For uniqueness of the solution, q(0) = 0 and q(1) = 1 are the

assigned boundary conditions. Let Wh⊆H1be the space of piece-wise linear Lagrange polynomials

of class C0(Fig. 8, left.) The projection of Eq. (50) onto Whby the L2scalar product and integration

by parts of the diﬀusion term yields the equation

ZΩh

ψhu·∇qhdΩh+ZΩh

ν∇ψh·∇qhdΩh=ZΩh

ψhS dΩh∀ψh∈Wh,(52)

qhis expanded by (29). When S= 0, the 1D ﬁnite element discretization of (52) yields the discrete

equation

u

2−ν

hˆqk+1 +2ν

hˆqk−u

2+ν

hˆqk−1= 0, k = 1...,N −1.(53)

Eq. (53) is equivalent to the 1D discretization of the same problem by second-order ﬁnite diﬀerences.

After algebraic manipulation and given the deﬁnition of the element Péclet number

P e =||u||h

2ν,(54)

Eq. (53) is written as a function of (54):

(P e −1) ˆqk+1 +2 ˆqk−(P e +1) ˆqk−1= 0, k = 1,...,N −1.(55)

It represents a tridiagonal linear system in the unknowns qk, k = 1,N −1, whose solution is the

function (see [246])

36 Simone Marras1et al.

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.75 0.8 0.85 0.9 0.95 1

qh

x

1D AD problem

Pel = 2.5

Pel = 2.5 stabilized

Pel = 0.625

Exact

Fig. 13: Finite element solution of the advection diﬀusion problem (50) using uniform, linear elements. u= 10,

ν= 0.1, in a domain of unitary total length. With these values, the global Péclet is Peg= 50. The plot shows the

approximate solutions obtained for diﬀerent grid spacing (Pe = 2.5and P e = 0.625) with and without stabilization.

It is shown how the computed solution can approach the exact solution by either increasing the number of grid

points (P e = 0.625), or by maintaining the grid suﬃciently coarse but with the addition of a stabilizing term

(How this term is built has not been shown yet, but the result gives a hint on what to expect from it).

ˆqk=1+P e

1−P e k−1

1+P e

1−P e N−1, k = 1, N −1.(56)

The power of (1+Pe)/(1-Pe) at the numerator produces an oscillatory behavior of the solution when-

ever P e > 1, as it is shown in Fig. 13. P e is a linear function of hso that the grid, in principle, could

be always constructed in such a way that, for a given value of uand ν,Pe ≤1. However, this is

not viable for most real problems because of the extremely high number of grid points that may be

necessary to achieve such a condition. The only way to solve the problem of numerical instabilities

in the Galerkin solution of transport problems with dominant advection remains that of stabiliza-

tion by proper means. A certain category of stabilization methods applied to the multi-dimensional

advection-diﬀusion equation will be described in the next section.

4.1 Viscosity-based stabilization techniques

Regardless of the numerical method that an atmospheric model is built on, dissipation of some sort is

added for various reasons; stabilization is one of them. As is pointed out in [161], the most common

mean of dissipation that is found in current research and operational weather forecast models is

artiﬁcial diﬀusion (or hyper-diﬀusion, HV from now on) in the form of

HV =ZΩh

(−1)α+1ψh∇α·(ν2α∇αqh)dΩh(57)

where αis a positive integer and ν2αis the matrix of the diﬀusivity coeﬃcients that may vary along

diﬀerent grid directions [125]. When α= 1, HV reduces to second-order Artiﬁcial Viscosity [169].

EBG in atmospheric modeling 37

Fig. 14: Pure transport of a square wave in a 2D doubly-periodic channel. The velocity is directed along the x-axis

(the bottom-right edge of the squared domain in the three plots). From left to right: stabilization achieved by ∇2,

∇4, and using a variational multiscale scheme [143]. Adapted from Fig. 21 of [210], with permission by Elsevier.

HV is easy to implement and is robust. These are features that make it attractive for models that

are not allowed to break during a forecast simulation. HV is found in other ﬁelds of computational

ﬂuid dynamics as well; the work by [11] is an example where HV is used to stabilize the simulation

of high speed ﬂows. One justiﬁcation that practitioners in NWP give to HV for α > 1is its scale

selectiveness; it damps higher order frequencies that are usually the result of numerical error and

dispersion, but tends to leave the important and physical modes untouched. However, even if HV is

indeed scale-selective, it is not physical. Since the artiﬁcial term given by Eq. (57) is a perturbation

to the original equation, if this perturbation does not go to zero as h→0, the exact solutions of the

original and of the perturbed problems are not equivalent. As is evident in Fig. 14, these methods add

an uncontrolled and non-localized diﬀusion that yields a certain deterioration of the solution. For a

stabilizing scheme to preserve the shape of the tracer, dissipation should be avoided in the direction

normal to the ﬂow and only act in the direction parallel to the ﬂow [146; 152; 55].

To preserve the correct physical dimensions of the hyper-viscous term, the value of ν2αmust scale

with respect to αand the grid spacing. Its selection not only is non-trivial, but has a great impact

on the solution of the problem. Jablonowski and Williamson [161] clearly state that "[...] the choice

of the ∇2,∇4coeﬃcient is most often motivated by empirical arguments and chosen in a somewhat

arbitrary manner [...]." More advanced, and by now classical, stabilizing schemes for ﬁnite elements,

spectral elements, and discontinuous Galerkin are described in the following subsections.

4.2 Filtering of (high-order) EBG Methods

Both CG and DG, like all higher-order methods, are limited by Godunov’s Theorem: all linear numer-

ical methods for solving PDEs that do not generate additional extrema (so-called monotone schemes),

are all ﬁrst-order accurate. As an immediate consequence, high-order CG and DG are not monotonic-

ity preserving, especially so near sharp gradients. In NWP, this problem is especially problematic for

tracer transport, where mass fractions may become negative due to these artiﬁcial extrema. Finally,

spectral elements and discontinuous Galerkin methods on quadrilateral and hexahedral elements typ-

ically use inexact integration to diagonalize the mass matrix; this approximate integration introduces

errors which must be stabilized by a ﬁlter or a more sophisticated scheme such as the VMS method

discussed later.

To circumvent these problems, ﬁlters were introduced in the development of both spectral methods

[96] and, later, spectral elements in [32; 94]. They were also applied to discontinuous Galerkin methods

in [116; 118]. Filters reduce the aliasing that occurs in the higher-order modes of the solution that

are largely responsible for Gibbs oscillations; hence, ﬁlters act upon the modal representation of the

solution. Once the oﬀending high modes are eliminated, the modal solution is inverse transformed to

physical, or nodal space. Hence, the spectral ﬁltering operation consists of a three-step process: 1)

transform the nodal solution to a modal solution, 2) apply a low-pass ﬁlter to eliminate the largest

spatial frequencies, and 3) inverse transform the ﬁltered modal solution to nodal space. For SE and

38 Simone Marras1et al.

DG that utilize LGL basis functions, a modiﬁed Legendre transform may be utilized for steps 1 and

3; in addition, it is possible to perform these operations on an element by element basis, thereby

eliminating the need for a global assembly operation.

Ideally, the spectral ﬁlter applied during step 2 should eliminate all non-physical oscillations while

faithfully preserving the physics of the solution. In practice, satisfying both of these requirements is

not possible. An eﬀective ﬁlter was developed by Boyd [31; 32], based on previous theoretical work

by Vandeven [307], resulting in an erfc-log ﬁlter for polynomials of order p. Letting ¯

θ=k/p −1/2, the

ﬁlter σ(k;p)is equal to unity if ¯

θ≤0and

σ(k;p) = 1

2erfc

2√p¯

θs−log1−4¯

θ2

4¯

θ2

.(58)

Eq. (58) rapidly eliminates all higher-order modes, and completely eliminates the highest order node.

Most DG methods utilize exact integration and therefore may not require ﬁlter-based stabilization.

It should be kept in mind that ﬁltering may not be suﬃcient to stabilize the solution; for this reason,

the dissipation schemes described so far and later in this section are often considered as a possible

option for high order methods as well [44; 45]. Finally, it is diﬃcult to derive idempotent ﬁlters; when

the ﬁlter is not idempotent [172], the solution may vary based on how many times the ﬁlter is applied

along the simulation.

4.3 Towards consistent stabilization methods

Streamline Upwinding (SU) [146], Streamline Upwind Petrov-Galerkin (SUPG) [38], Galerkin/Least-

Squares (GLS) [150], Galerkin methods with bubble functions [35; 9; 37], or sub-grid projection meth-

ods [126] are some of the most used stabilization techniques for ﬁnite elements. To bypass some

drawbacks of streamline-type schemes such as these, much work was done in the same years on shock

capturing, as found in [170; 171; 132]. The Taylor-Galerkin method [78], the Characteristic-Galerkin

formulation [241], and the Characteristic-Based Split (CBS) method [333; 332] are more ways for

FE stabilization that, however, rely on a reasoning that has no relationship with the methods we

are interested in reviewing in this article. We mention them here but we will not delve into their

description.

Streamline-upwind (SU). The problem of isotropic smearing of the solution mentioned above was

partially solved by [146] with the construction of the Streamline-upwind method, although the idea

of ﬁnite element upwinding can already be found one year earlier with the work by [287] and then

continued in [288; 289]. With SU, stabilization is projected in the direction of the ﬂow only, as visible

from

bSU =ZΩh

τu·∇ψhu·∇qhdΩh.(59)

However, like HV, SU is not numerically consistent either in the sense that no residual information is

used to construct this perturbation term. The Streamline-upwind/Petrov-Galerkin (SUPG) method

described below is the consistent evolution of SU and will be among the most common methods of

stabilization of ﬁnite elements used since its introduction.

Streamline-upwind/Petrov-Galerkin (SUPG). The SUPG method was designed by [38] and was later

generalized for multidimensional problems by [151]. It is a consistent alternative to the HV approach

or to the overly diﬀusive SU. Its use has been ubiquitous in the solution of transport problems by

the ﬁnite element method (e.g., [156; 100; 35; 295]). The application of this strategy to higher-order

EBG in atmospheric modeling 39

schemes was ﬁrst tested for spectral methods by Canuto and colleagues in [43], [44], [46], [45], and

later by Hughes and co-workers in [148] using non-uniform rational B-splines (NURBS). SUPG is a

Petrov-Galerkin method in that it does not assume that the basis and test functions live in the same

space. We introduce the additional space Ψhof test functions whdeﬁned by

Ψh.

=wh:wh=ψh+τu·∇ψh:ψh∈Wh.

We have the problem of ﬁnding the function qh∈Whsuch that

ZΩh

whu·∇qhdΩh−ZΩh

wh∇·(ν∇qh)dΩh=ZΩh

whShdΩh∀wh∈Ψh.(60)

Some algebra and rearrangement of (60) yields the problem of ﬁnding qh∈Whsuch that

ZΩh

ψhu·∇qhdΩh+ZΩh

ν∇ψh·∇qhdΩh

| {z }

Galerkin

+bSU P G =ZΩh

ψhShdΩh

| {z }

Galerkin

∀ψh∈Wh,(61)

where

bSU P G =ZΩu·∇ψhτu·∇qh−∇·ν∇qh−S

| {z }

L(qh)−S

dΩh(62)

is the consistent SUPG stabilizing term. In (62), u·∇qh−∇·ν∇qh−Sis the residual of (50) and

τis the stabilization parameter. The deﬁnition of τthat yields a nodally exact SUPG solution with

continuous piecewise linear ﬁnite elements is derived in [50] from 1D analysis. Its generalization to

multi-dimensional problems is given by the simple substitution of uwith ||u||, although, in multi-

dimensions this does not necessarily imply nodal exactness. With respect to higher order elements,

a thorough analysis of τfor quadratic elements is given by [59]. In the context of VMS, diﬀerent

deﬁnitions exist for parameter τ, some of them are discussed in 4.4. For a brief review on SUPG, the

reader is also referred to [145] and the report [102].

Galerkin/Least-square (GLS). A generalization of SUPG was obtained by [150] as

bGLS =ZΩu·∇ψh−∇·ν∇ψh

| {z }

L(ψh)

τu·∇qh−∇·ν∇qh−S

| {z }

L(qh)−S

dΩh.(63)

In analogy with the ﬁndings of [82] to stabilize the Stokes equation, a sign change in the Laplace term

of the stabilizing term in the perturbed equation proved to yield better stabilization characteristics

(more accurate results) than the original generalized SUPG (or GLS) method [100]. In (63), for better

properties, instead of using the diﬀerential operator L, the method should use the negative part of

the adjoint L∗of the original operator L. We have that the last perturbation term of the original AD

equation should be

b=−ZΩL∗(ψh)τu·∇qh−∇·ν∇qh−S

| {z }

L(qh)−S

dΩh,(64)

where

L∗=−u·∇−∇·(ν∇)(65)

40 Simone Marras1et al.

is the adjoint of L.

Based on what was learned on stabilization of the scalar advection-diﬀusion equation, researchers

in ﬂuid dynamics applied these methods and their evolution to the stabilization of ﬂuid problems.

Thanks to the work of [143; 149], the methods that we have just described have been recognized to

belong to the same family known as the family of Variational Multiscale Stabilization, or VMS. The

VMS approach is summarized below for scalar advection problems, and will be derived and discussed

for the Euler equations in Section 4.7.

4.4 Variational Multiscale Stabilization (VMS)

In 1995 and 1996, groups of researchers lead by Hughes [143] and Brezzi [36] proposed a theory to

explain the reasons of instabilities and a new way to attack the problem. They concluded that the un-

resolved scales (the scales that cannot be captured by the computational grid) are responsible for the

numerical instabilities of the Galerkin solution of the diﬀerential problem. The analysis, that continues

with [155] and [149], forms the unifying theory of all stabilized ﬁnite element methods. According to

this theory, stabilized methods are subgrid scale models where the unresolved scales are intimately

related to the instabilities at the level of the resolved scales, and thus should be used in the construc-

tion of the stabilization term. More speciﬁcally, in the formulation of the discrete problem, the eﬀects

of the unresolved scales must be introduced by modeling them on the grid. These schemes are known

as Variational Multiscale Stabilization (VMS) method. Generally speaking, the stabilization term of

VMS corresponds to bdeﬁned in (64).

VMS has been extensively applied to the solution of the advection-diﬀusion/advection-diﬀusion-

reaction equations (e.g. [143; 149; 58; 61; 141]), and to the solution of the Navier-Stokes equations

for incompressible ﬂows (e.g. [153; 56; 57; 123; 17; 6]). Recently, it was applied to spectral elements

in the context of atmospheric ﬂows in [210; 209]. In Section 4.7 a review of VMS for the compressible

Euler equations is found.

The multiscale description of the stabilization scheme relies on the splitting of the solution into a

resolved, qh, and a sub-grid, unresolved component, ˜q, to give q=qh+ ˜q. Let W=Wh⊕˜

Wbe the

space decomposition such that ˜

Wcompletes Whin W. This translates into the decomposition of the

solution variables q=qh+ ˜q, and of the basis functions ψ=ψh+˜

ψ. Substituting the decomposition

into the general weak form of Eq. (50),

ψ, ∂q

∂t +a(ψ,q)=(ψ,S)∀ψ∈W, (66)

where (·,·)is the L2inner product and a(·,·)is a bilinear form that satisﬁes

a(ψ, q)=(ψ, u∇q) + ν(∇ψ,∇q),

and anticipating that we will consider the quasi-static approximation ∂t˜q= 0 [57], we obtain:

ψh+˜

ψ, ∂qh

∂t +a(ψh+˜

ψ, qh+ ˜q)=(ψh+˜

ψ, S)∀ψh∈Wh,˜

ψ∈˜

W . (67)

By virtue of the linear independence of ψhand ˜

ψwe can ﬁrst take ˜

ψ= 0 and then ψh= 0 and ﬁnd

the split problem:

ψh,∂qh

∂t +a(ψh, qh)+ a(ψh,˜q)=(ψh, S)∀ψh∈Wh(68a)

˜

ψ, ∂qh

∂t +a(˜

ψ, qh)+ a(˜

ψ, ˜q)=(˜

ψ, S)∀˜

ψ∈˜

W . (68b)

EBG in atmospheric modeling 41

In the subgrid Eq. (68b) we come back to the original diﬀerential operator Lfrom Eq. (50). We

assume that ˜

ψ(∂K )=0and ˜q(∂K )=0, for each element Kof the grid and, following [143], in (68a)

we integrate by parts the bilinear forms that depend on the subgrid scale and ﬁnd:

ψh,∂qh

∂t +a(ψh, qh)+(L∗ψh,˜q)=(ψh,S)∀ψh∈Wh(69a)

˜

ψ, ∂qh

∂t + ( ˜

ψ, Lqh)+(˜

ψ, L˜q)=(˜

ψ, S)∀˜

ψ∈˜

W , (69b)

where L∗(in Eq. (65)) is the adjoint operator of L.

4.5 Approximation of the sub-grid scales

The unresolved quantity ˜qhas not been deﬁned yet. Eq. (69b) is used as the starting point to approx-

imate ˜q. By re-arranging the terms in (69b), the equation for the subgrid scales is found,

(˜

ψ, L(˜q)) = ( ˜

ψ, R(qh)) ∀˜

ψ∈˜

W , (70)

where

R(qh) = S−∂qh

∂t −L(qh)(71)

is the residual of the original equation. The strong form of (70) is considered on each element K

L(˜q) = R(qh),(72)

and τ≈L−1, an algebraic approximation of the inverse of the diﬀerential operator Lis deﬁned. Then

the sub-scale has the form

˜q=τ R(qh).(73)

Expression (73) is plugged into equation (69a), to ﬁnd the expression for the VMS stabilized Galerkin

method as follows: Find qh∈Whsuch that

ψh,∂qh

∂t +a(ψh, qh)+(L∗(ψh),τ R(qh)) = (ψh, S)∀ψh∈Wh.(74)

Eq. (74) diﬀers from Eq. (66) by the additional term that models the subgrid scales. The extra term

is the viscous-like contribution that stabilizes the equation.

Diﬀerent formulations for ˜qare found in the literature, some of them are reviewed in 4.5.1-4.5.3. All

of them depend on the deﬁnition of the stabilization parameter τ. The parameter τis a topic of

active research still today, since a general deﬁnition is not known [168]. This statement is true for

all the residual-based stabilization methods described so far. The quantity τis an intrinsic time that

is built as a function of the local Pèclet number of the ﬂow which, for stability, should respect the

condition P e < 1. Many problems in atmospheric CFD are advection dominated, implying Pe 1,

so that stabilization is indeed necessary for all the problems that are of any interest for atmospheric

modelers.

42 Simone Marras1et al.

4.5.1 Approximation via Green’s functions

This approach is used by Hughes and collaborators [143; 149; 144] to derive τ. In brief, they consider

Eq. (72) with ˜q= 0 on ∂K, and the associated Green’s function problem for the adjoint operator

(L∗g(x,y) = δ(x, y)∀x∈K

g= 0 on ∂K . (75)

Then a uniform element-wise deﬁnition of τis obtained as the average value of the exact element

Green’s function

τ=1

|K|2ZKZK

g(x,y)dKxdKy,(76)

where |K|is the measure (volume/area/length) of the domain. For the one-dimensional linear scalar

advection-diﬀusion equation there is an analytical expression for the Green’s function and the stabi-

lization parameter is computed from (76) to give

τ=1

2

h

||u|| coth(P e)−1

P e ,(77)

for the Péclet number (54). In [147; 38] the same expression (77) is obtained in the context of SUPG

stabilization by following an error minimization criterion. For the purely advective case (ν= 0 and

P e → ∞), we ﬁnd

τ=1

2

h

||u||.(78)

Instead of (76), Corsini et al. [61] propose a non-uniform τon each element:

τ(x) = 1

|K|ZK

g(x,y)dKy.(79)

4.5.2 Approximation via Fourier analysis

The strategy of Codina et al. [58] is explained here for the multidimensional advection-diﬀusion Eq.

(50). The starting point is to transform Eq. (72) into the Fourier space. Which is interesting because

the diﬀerential operator transform, b

L, is easy to invert. Let’s call Tits inverse, T= ( b

L)−1, thus the

Fourier transform of the sub-scale is approximated on each element Kas

ˆ

˜q(ω) = T(ω)ˆ

R(ω),(80)

where ωis the wave number and

T(ω)≈iu·ω

h+ν||ω||2

h2−1

.

Considering expressions (80) and (73), the Plancherel’s formula and the mean value theorem are

applied to obtain an approximated value for the stabilization parameter on each element Kas

τ=1

2

h

||u||

P e

P e +1 =2||u||

h+4ν

h2−1

.(81)

For pure advection problems (ν= 0 and P e →∞), the stabilization parameter becomes as in Eq. (78).

EBG in atmospheric modeling 43

4.5.3 Approximation via bubble functions

More options to build the stabilization parameter τare found in [141] for linear, quadratic, and cubic

elements. The space ˜

Wis made of bubble functions (see [9; 37]), that are vanishing functions on the

boundaries of each element. The unresolved scales, ˜q, are deﬁned as a function of the bubbles b(x)

that are derived as described in the referenced literature. Omitting the details, τis a function of the

bubble as:

τ=1

hZh

0

b(x)dx, (82)

that, once evaluated, yields the expression (77) for the parameter τ; the same expression encountered

with the Green’s function approach.

On the steps of [141], τfor spectral elements of arbitrary order and with unequally spaced element

nodes was derived in [210]. The stabilization parameter τis built inside the element as a function of

the bubbles on every segment delimited by two consecutive LGL points. The uneven spacing of the

element nodes is the major diﬀerence with respect to the deﬁnitions derived in previous studies. In

this case, the intrinsic time is non-uniform along the element.

4.6 Preserving positivity

SUPG, GLS, and VMS are not monotonicity preserving. This issue is particularly important for the

simulation of tracer dynamics in the atmosphere. If over- and undershoots aﬀect the solution in the

proximity of strong gradients, the net mass balance of the advected tracers, will be negatively aﬀected.

To overcome this issue, a controlled crosswind discontinuity capturing can be added to the principal

stabilization scheme. For example, the method introduced in [55] was successfully adapted to high-

order spectral elements in [210] for standard 2D test cases and, more recently, by [209] to support

positivity in the solution of fully 3D cloud simulations. For comparison, we reproduce Fig. 29 of [210] in

Fig. 15. In the case of high-order SEM, in [210] the First Order Subcell (FOS) method was introduced.

FOS consists in lowering the high-order method to ﬁrst order in the spectral elements that contain the

discontinuity only. The FOS results are encouraging, although there is some overhead coming from

looping over the linear sub-elements within the high-order elements that contain the localized over-

and under-shoots.

Other positivity preserving schemes such as high-order limiters for both CG and DG are often

used as well, as is shown in, e.g., the report by [329].

4.7 VMS stabilization for the Euler equations

VMS for compressible ﬂows appears in [251; 252; 222]. A review of residual-based stabilization meth-

ods for compressible ﬂows can be found in [145]. Recently, VMS was used to stabilize the FEM

solution of the Euler equations of atmospheric non-hydrostatic ﬂows in [212; 213]. VMS was derived

for discontinuous Galerkin as well [154] although it has not been applied to atmospheric modeling.

In the following, we then limit the analysis to continuous Galerkin (without distinguishing between

FEM and SEM in its derivation.) For the treatment that follows, it is convenient to express the Euler

equations (i.e., the inviscid counterpart of system (1)) in compact form as

∂q

∂t +∂Fi(q)

∂xi

= 0,i= 1,2,3,(83)

where the Einstein summation on the repeated indices is assumed, where qis the vector of the

unknowns, and Fis the vector of the ﬂux quantities. Without compromising the generality of the

stabilization method, gravity and Coriolis are here omitted. As usual, the problem consists in ﬁnding

44 Simone Marras1et al.

0 0.5 1

0

0.2

0.4

0.6

0.8

1

x (km)

z (km)

tracer [g/gk]

0

0.2

0.4

0.6

0 0.5 1

0

0.2

0.4

0.6

0.8

1

x (km)

z (km)

tracer [g/gk]

0

0.1

0.2

0.3

0.4

0.5

0 0.5 1

0

0.2

0.4

0.6

0.8

1

x (km)

z (km)

tracer [g/gk]

0

0.1

0.2

0.3

0.4

0.5

0 0.5 1

0

0.2

0.4

0.6

0.8

1

x (km)

z (km)

tracer [g/gk]

0

0.1

0.2

0.3

0.4

0.5

Fig. 15: Stable SEM solutions of the transport equations for a sharp cylinder that is transported by a rising

thermal ﬂow (i.e. the tracer is driven by the thermal ﬂow that is modeled by the Euler equations of stratiﬁed

ﬂows.) Top left: ﬁltered solution using the ﬁlter of Section 4.2. Top right: 2nd-order artiﬁcial diﬀusion without

discontinuity capturing. Bottom left: VMS. Bottom right: VMS with discontinuity capturing. Adapted from [210],

with permission by Elsevier.

q(x,t)that veriﬁes Eq. (83) for all (x,t)∈Ω×R+. To proceed and derive VMS applied to this set, we

write the three-dimensional Euler equations in ﬂux form for the conservative variables qand deﬁne

the advective system:

∂q

∂t +Ai(q)∂q

∂xi

= 0,(84)

where

Ai(q) = ∂Fi

∂q(85)

are the Jacobian matrices. As already done in Sec. 3 for scalar problems, the variational form of Eq.

(84) can then be written as

ZΩh

ψh·∂qh

∂t dΩh+ZΩh

ψh·Ai(qh)∂qh

∂xi

dΩh= 0 ∀ψh∈Wh.(86)

EBG in atmospheric modeling 45

As it is done in page 40, the decompositions q=qh+˜

qand ψ=ψh+˜

ψare plugged into the variational

problem (86), which hence can be split into the two equations

ZΩh

ψh·∂qh

∂t dΩh+ZΩh

ψh·Ai(q)∂qh

∂xi

dΩh

+

nel

X

m=1 ZKm

ψh·∂˜

q

∂t dK m+ZKm

ψh·Ai(q)∂˜

q

∂xi

dKm= 0 ∀ψh∈Wh(87a)

nel

X

m=1 ZKm

˜

ψ·∂˜

q

∂t +Ai(q)∂˜

q

∂xidK m=

nel

X

m=1 ZKm

˜

ψ·R(qh)dKm∀˜

ψ∈f

W(87b)

where

R=−∂

∂t +Ai(q)∂

∂xi(88)

is the residual operator of the governing Eq. (84). Equation (87a) is solved numerically on the compu-

tational grid, whereas (87b) is the subgrid scale equation from which an expression for ˜

qis obtained

and hence plugged back into (87a). Concerning Eq. (87a) for the large scales, some assumptions should

be made. For details, see the referenced literature. In the case of non-viscous problems (i.e. Euler equa-

tions), SUPG [194], GLS [267] and VMS end up having the same structure, unless the approximation

of the subgrid scales, ˜

q, is such that VMS diﬀerentiates itself from the other two schemes.

VMS as implicit Large Eddy Simulation (LES): Without entering much into this discussion, it is

important to underline the fact that VMS is also used as an Implicit Large Eddy Simulation (ILES)

scheme that relies on the variational projection of the original equations rather than the traditional

ﬁltering. This was ﬁrst applied to incompressible turbulent ﬂow in [153; 17]. In [183; 91; 234], a

turbulent compressible ﬂow is modeled using the VMS framework although the ﬁne scales are modeled

by a Smagorinsky model. Similarly, this is done in [60; 199]. In [306], a VMS formulation obtained by

extension of the Favre averaging to general projection operators is proposed, where no explicit subgrid

modeling is presented. Using SEM, VMS was used in [120] to solve turbulent incompressible ﬂows.

4.7.1 Approximation of the sub-grid scales

For the Navier-Stokes equations, analogously to the advection-diﬀusion equation, the subgrid scale ˜

q

is computed from the subscale Eq. (87b) and has the general form of

˜

q=τR(qh),(89)

where τis a diagonal matrix. Shakib et al. [267] and Hughes and Mallet [151] compute the parameter τ

for GLS to solve the compressible Euler and Navier-Stokes equations. For the same equations, Hughes

and Tezduyar [156] and Le Beau and Tezduyar [194] compute τfor SUPG. Just like for the scalar case,

the parameter τhas been derived in diﬀerent ways by diﬀerent authors, although the ﬁnal expressions

seldom diﬀer greatly. In, e.g., [222], τis derived from a Fourier analysis. Another approach involves

the use of Green’s functions as done in [61]. Regardless of the deﬁnition of τ, let us notice the local

nature of the subscales that only exist where the residuals of the large scales are important. This, with

non-constant values, marks the major diﬀerence with respect to artiﬁcial diﬀusion. The structure of

˜

qfor the problem of a rising thermal is shown in the top two plots of Fig. 16. By comparison with

the pattern of potential temperature (bottom left plot) and horizontal velocity (bottom right plot),

the structure of the sub-grid scale is clearly tied to the residual.

An example of simulation where VMS was used to stabilized both the dynamics (Euler equations)

and the advection-diﬀusion equations of water tracers is shown in Fig. 17, and is compared against

the solution obtained in [103] with a ﬁltered high-order spectral element at an equivalent resolution.

46 Simone Marras1et al.

Fig. 16: 2D Rising thermal bubble. Top row: sub-grid scales: ˜

θ/θmax (top-left) and ˜

U/Umax (top-right). Bottom

row: potential temperature θ0(K) (left), and horizontal velocity, u(m/s). This vertically displacing ﬂow is triggered

by the thermal perturbation θ0of a neutrally stratiﬁed ﬂow (i.e. uniform and constant θ0). The characteristic

shape of the perturbation ﬁeld θ0is shown in the right panel. The plots are adapted from [212].

4.8 Alternative Consistent Schemes: Spectral Vanishing Viscosity and Entropy Viscosity method

Also based on a second order operator, the spectral vanishing viscosity (SVV) of [290] is a stabilizing

method used by practitioners of high order spectral element and spectral Fourier methods. The idea

of SVV comes from an entropy analysis of the problem at hand and is such that the added dissipation

satisﬁes the entropy condition. For more on this, please, see [290; 130; 173] and references therein.

Also tied to the entropy equation, the entropy viscosity method was ﬁrst introduced by [128]. The

fundamental diﬀerence between this method and SVV is in the way the entropy equation is used.

The entropy viscosity method builds the local and dynamic viscosity of the equations based on the

residual of the associated entropy equation [127; 129]. In [335] we ﬁnd how this regularization of the

governing equations is applied to the discontinuous Galerkin method as well.

EBG in atmospheric modeling 47

Fig. 17: 2D squall line simulation. The cloud content is delimited by the thick black contour line in both plots.

The color/grey shading in the left/right plots is the equivalent potential temperature. Both horizonthal domains

extend along 240 km. The left ﬁgure is adapted from [213] where a VMS stabilized FEM solution was computed

with linear elements (with permission by Elsevier.) The right ﬁgure is adapted from [103] (with permission of the

American Meteorological Society) and the solution was computed with 8th-order SEM stabilized with a constant

coeﬃcient diﬀusion (ν= 200m2/s) and a ﬁlter.

We ﬁnally comment on the use of adaptive viscosity methods for both CG [125] and DG [236].

These two methods are not consistent, but are element-based and dynamic. An adaptive artiﬁcial

viscosity for non-hydrostatic modeling using DG has been recently proposed in [329].

4.9 Physics-based stabilization

A computationally inexpensive and numerically stable sub-grid scale model for compressible large-

eddy simulation was introduced in [229] for adaptive ﬁnite elements. Due to its stabilizing properties,

this method was easily adapted to the solution of low Mach number atmospheric ﬂows via high order

spectral elements in [215] and [216]. Like VMS, this method is a residual-based alternative to the

more classical artiﬁcial diﬀusion in a way that not only is numerically consistent, but could also serve

as a turbulence model. Unlike VMS, however, stabilization is attacked starting from the governing

equations rather than from their numerical approximation. More speciﬁcally, the Euler equations are

ﬁrst ﬁltered to separate the resolved from the un-resolved scales [108; 260]. The ﬁltering operation

leads to a new set of equations containing additional terms that are dissipative in nature and that are

then modeled in some way. The steps described below (following the treatment of [216]) show how

stabilization is then achieved. In LES, given a quantity q(e.g., density, velocity, potential temperature),

its large scale (grid resolved) component qis obtained via the application of the ﬁlter

q(x) = ZΩ

G∆(x−χ)q(χ)dχ.(90)

Eq. (90) is a spatial convolution of the ﬁltering function G∆with q, where ∆is the ﬁlter width. The

ﬁlter functions can vary; the most commonly used in LES are the Gaussian, the top hat in real space,

and the sharp Fourier cutoﬀ functions [197; 240].

Remark 4.1 The barred quantities introduced in Section 2.1.2 have no relation with qdeﬁned in

Eq. (90) for Large Eddy Simulation (LES). LES implies a separation between the resolved and unre-

solved scales, whereas the splitting given in Section 2.1.2 was introduced for numerical convenience in

the simulation of atmospheric problems and does not aﬀect the way LES is constructed or derived.

For compressible ﬂows, the Favre ﬁlter eq=ρq/ρ [92], although not necessary, is classically intro-

duced. The application of these ﬁlters to equations (6) –excluding the Coriolis terms for simplicity–

48 Simone Marras1et al.

yield the ﬁltered system

∂ρeu

∂t +∇·(ρeu⊗eu) +∇p=∇·τ−ρg,(91a)

∂ρ

∂t +∇·(ρeu)=0,(91b)

∂ρe

θ

∂t +∇·ρe

θeu=∇·Q,(91c)

where the two derivatives on the right-hand sides of (91a) and (91c) represent the contributions of

the unresolved scales. If Favre ﬁltering were not applied, an additional ﬂux term would also appear

on the right-hand side of Eq. (91b). With Favre, the ﬁltered density ρis conserved and no modeling

is required for the continuity equation. In (91a), τis the turbulent stress tensor,

τ=ρ^

u⊗u−eu⊗eu,

approximated by

τ= 2D(eu),(92)

where

D(eu) = µn

2∇eu+∇euT

is the velocity deformation tensor multiplied by a dynamic coeﬃcient µnthat will be deﬁned shortly.

Similarly, in (91c) Qis the kinematic heat ﬂux deﬁned as

Q=ρf

θu−e

θeu,(93)

and is modeled via

Q=κn∇e

θ(94)

Like µn, the deﬁnition of κndetermines the method proposed in [229]. The coeﬃcients µnand κnare

calculated element-wise on every high order element Ωefor a Galerkin approximation of Equations

(91). More speciﬁcally, for the sensible temperature T=θ(p/p0)R/cpand one ﬁnite/spectral element

of characteristic length hΩe, we start by deﬁning the dynamic viscosities

µmax|Ωe= 0.5hΩek|eu|+qγe

Tk∞,Ωe,(95)

and

µres|Ωe=h2

Ωemax kR(ρ)k∞,Ωe

kρ−bρk∞,Ω

,kR(ρeu)k∞,Ωe

kρeu−ρbuk∞,Ω

,kR(e

ρθ)k∞,Ωe

kρe

θ−ρb

θk∞,Ω !.(96)

In (96) b·indicates the space average of the quantity at hand over Ωand the norms k·k∞,Ω at the

denominator are used for normalization to preserve the correct dimension of the resulting equation.

Having µmax and µres constructed, the dynamic coeﬃcients of the viscosity terms can be computed

as

µn|Ωe= minkρk∞,Ωe(µmax|Ωe, µres|Ωe)(97)

and

κn|Ωe=Pr

γ−1µn|Ωe,(98)

where Pr is an artiﬁcial Prandtl number. The residuals in (96) are simply:

R(eu) = ∂ρeu

∂t +∇·(ρeu⊗eu) +∇p+ρg,(99a)

EBG in atmospheric modeling 49

Fig. 18: Stabilized solution of the density current problem [284]. Reproduced from [216].

R(ρ) = ∂ρ

∂t +∇·(ρeu),(99b)

R(ρe

θ) = ∂ρe

θ

∂t +∇·ρe

θeu.(99c)

The time derivatives are to be included or the consistency of the method would be lost. An example of

the stabilized spectral element solutions reported in [216] is plotted in Fig. 18, where also the results

obtained using a constant coeﬃcient Lilly-Smagorinsky model [201; 274] are given for comparison.

Putting together the moist problem brieﬂy described in Section 2.4 and the current LES-based stabi-

lization, the simulation of a fully three-dimensional deep convection problem is reported in [215]; in

Fig. 19, we reproduce Fig. 3 contained therein.

The multi-scale properties of this scheme have been veriﬁed via the simulation of a turbulent ﬂow

on the sphere whose radius is that of the earth. As an example, a turbulent ﬂow in a geostrophically

balanced atmosphere is shown in Fig. 20, after [216].

50 Simone Marras1et al.

Fig. 19: Deep convection: 3D view of qc(grey surface), surface velocity (vectors), and the instantaneous distribution

of qron the ground (contours). Reproduced from Fig. 3 of [215].

120oW

60oW

0o

60oE

120oE

180oW

15oN

30oN

45oN

60oN

75oN

Time = 20

120oW

60oW

0o

60oE

120oE

180oW

15oN

30oN

45oN

60oN

75oN

Max=2.13e+01 1/s

Min=−9.79e+00 1/s −10

−8

−6

−4

−2

0

2

4

6

8

10

Fig. 20: Turbulent ﬂow on the sphere after 12, 20, and 25 days. Top-view, looking down onto the northern

hemisphere. The radial component of vorticity is plotted and colored by intensity. Plot adapted from [216].

EBG in atmospheric modeling 51

(a) (b) (c)

Fig. 21: Representation of a smooth mountain using: (a) height coordinate system with step topography, (b)

σ-terrain following coordinates, and (c) height coordinate system with shaved cells.

5 Vertical discretization, computational grids, and adaptive mesh reﬁnement in NWP

We brieﬂy discuss the issue of vertical discretization in atmospheric models since it is characterized

by some constraints that do not apply to more traditional and general CFD models. Because of

the classical use of ﬁnite diﬀerences with Cartesian rectangular grids, the accurate approximation

of topography has always been a major concern both in atmospheric and ocean models. The verti-

cal coordinate systems can be separated into two main branches: σterrain-following [237; 104] and

height-coordinates. Terrain-following coordinates have the advantage of the accurate representation

of topography and ease of application of boundary conditions as the grid cells follow the shape of

the varying bottom of the domain. However, the large truncation errors that increase with increasing

topography slope [286; 164] require vertical coordinates that are more suitable for steep topographies.

The height-coordinate system was ﬁrst proposed as the η-system by [219]. It consists of the use of a

rectangular grid that intersects the topography and deﬁnes the orographic height at the cells edges.

Modiﬁcation of both approaches have been later deﬁned. Examples are the hybrid terrain-following

coordinates [268] as an improvement of σ, or the shaved-cell method in z-coordinates introduced by

[1] for ocean models. Fig. 21 shows a schematic of these grids.

The σgrid mentioned above is simple, but on steep topography the regularity of the grid in the

inner domain is compromised. To overcome this drawback, [263] introduced the smooth level vertical

(SLEVE) mapping that helps maintain a suﬃcient degree of regularity of the node distribution away

from the bottom boundary. Given a mountain ridge, a SLEVE grid is obtained from the decomposition

of a large and small scale variation of topography (e.g. a Gaussian terrain perturbed by a wave-like

function). Through this solution the grid distortion is controlled from bottom to top by means of

two free parameters. Somewhere between σand SLEVE stands the hybrid grid of [268]. The hybrid

grid uses the same vertical coordinate σand combines the topography and the height of the domain

through two functions a(σ)and b(σ)whose values are properly tabulated.

Finite elements and Galerkin methods in general (ﬁnite volumes included) are free of all the

drawbacks of methods that are not ﬂexible with regard to the grid. Finite elements depend on compu-

tational grids of quadrilateral and triangular elements (in 2D) or hexahedra, tetrahedra, and prisms

(in 3D) that adjust to the physical geometry to be discretized without aﬀecting the formulation of the

governing equations. The grid shape is inherently deﬁned in the numerical formulation of the method.

Generally speaking, they are z-coordinate based methods with full control of the shape of the to-

pography. The grid itself looks like a σ-grid, but the fundamental diﬀerence is that ﬁnite diﬀerence

methods with σgrids require re-expressing the equations using a coordinate transformation.

Due to the geometrical ﬂexibility of Element-Based Galerkin (EBG) methods, no coordinate trans-

formation is needed to apply the ground boundary condition. Complex orography can be modeled with

ease using ﬁner or, perhaps, adaptive grids (see Section 5.2), as long as certain criteria on regularity

and smoothness of the element shape are respected. Furthermore, in a time when high resolution is the

rule, complex orography can be modeled with ease and better grids. High resolution terrain-following

coordinates induce grids to lose the property of orthogonality at the boundaries. The internal elements

as well would suﬀer from great stretching up to a point that the grid is no longer suﬃciently smooth

52 Simone Marras1et al.

for the numerical method to perform correctly. For example, if the Jacobian of the transformation

from physical to computational space is singular, large numerical errors and instability in the solution

would occur [300]. The application of CFD grid generation techniques for use in atmospheric problems

is being considered more and more. Simple and fast structured grid generation with boundary layer

grids or elliptic smoothing has been described in, e.g., [208]. Unstructured grids are also becoming of

interest, as shown a few years ago by [5] and, more recently, by [275]. However, the inertia from the

atmospheric community towards grids that do not have a characteristic column-wise structure is still

large. This is because all of the packages that involve the computation of atmospheric parameteriza-

tions (e.g., precipitation, radiation) are designed to work on such grids and would have to be adapted

(i.e. re-written) to work on diﬀerent grids. The reasoning behind this inertia is understandable, al-

though steps ahead in this direction must be made now that high-resolution atmospheric modeling is

approaching fast.

5.1 3D grid generation for domains with orography and bathymetry

Volume grid generation in atmospheric models is commonly performed by a one directional simpli-

ﬁcation of Transﬁnite interpolation (TFI) [122; 89]. TFI is robust, simple, and arguably the fastest

grid generation technique in use in many ﬁelds of computational mechanics, of which geophysical ﬂuid

dynamics represents a particular case. Nevertheless, generally the quality of TFI grids degenerates

when the geometric features of the domain boundaries present sharp corners, quasi-vertical boundary

walls, or similar characteristics. This has a direct eﬀect on the quality of the numerical solution of the

problem [217]. The problem exists regardless of the underlying numerical method of solution. In NWP,

sharp mountain ridges and canyons are an example. With the ever increasing trend towards high spa-

tial resolution that we are experiencing in numerical weather prediction today, sharp topographies are

certainly an issue. In the following sections, we describe the current way of generating structured grids

in topographical domains and present a few examples to underline the possible limitations. At that

point, we introduce the idea behind elliptic grid generation and how grids may be improved in terms

of smoothness and orthogonality properties by this simple technique. Most of the ideas presented in

this appendix are found in the books by [182] and [298], and in the recent paper by [175].

5.1.1 Algebraic grid generation

As we have mentioned above, transﬁnite interpolation has a major drawback that comes from the

constraint on the regularity of the boundaries. If the boundaries of the simply-connected domain are

not suﬃciently smooth, TFI fails to generate good grids. The sharpness of internal corners given by a

possible discontinuity in the space derivative of the boundary functions, reﬂects into folding grids with

unacceptable node overlapping. The problem of folding grids with diﬃcult geometries is usually solved

by subdividing the domain into smaller subdomains with more regular boundaries. This technique is

robust but diﬃcult to automate. In Fig. 23, although the edges do not cross, the vertical wall on the

left-hand side of the hill is a challenge for the grid generator, as it can be noted by the extremely

stretched elements in the region of the hill’s front.

Nonetheless, because topography is usually smooth in current operational models (at horizontal

resolution of 1 km or coarser, all mountain peaks are likely to be smoothed out), TFI is still the perfect

and quick solution that can be properly modiﬁed for diﬀerent types of vertical node distributions.

These improved methods are suﬃciently good as long as the boundaries are never vertical. This

is because the transformations are performed along zonly. For full control of the nodes’ distribution

in all directions, these schemes should be incorporated into a full TFI interpolation.

EBG in atmospheric modeling 53

5.1.2 Elliptic grid generation

One simple, yet eﬃcient solution to the generation of smooth grids with suﬃciently good properties

is given by the solution of the Thompson-Thames-Mastin (TTM) problem [297], an elliptic system

of partial diﬀerential equations. Two-dimensional elliptic grid generation was introduced for ocean

circulation modeling in [259]. The penalty of elliptic equation methods is the higher cost with respect

to algebraic methods. To control the point distribution with TTM some parameters must be selected by

the user. To overcome the need for parameter selection, in [175] an automatic elliptic grid generation

method is proposed. A similar approach is described in [180] for grids around topography. In this

recent paper, the author also uses an iterative method to smooth the grid on a layer-by-layer basis,

with a check on the grid spacing to avoid the overlapping of grid cells. This check is necessary because

the method of [180], unlike the elliptic scheme, is not designed to respect the maximum principle.

Orthogonality When it comes to high resolution simulations, with very ﬁne LES grids, the boundary

layer may be solved explicitly. In this case, boundary layer grids may be necessary for atmospheric

models like they are for, e.g., industrial ﬂows at much smaller scales. For how the atmospheric com-

munity is responding to the introduction of new grids, orthogonal boundary grids may still be seen

as futuristic. However, their use is already common in the simulation of atmospheric ﬂows in the

micro-scale (i.e. 20 to 500 m domains) (see, e.g., the Bolund experiment starting from [22]) so that it

seems appropriate to mention them here.

Orthogonality in three-dimensional structured grid generation systems is still an active ﬁeld of

work (see [298; 175]). The elliptic grid generation system herein described and implemented in [221]

is able to reach reasonable orthogonality properties at the lower boundary by either using Neumann

boundary conditions to move the nodes, or by a proper deﬁnition of the control functions as done

in [175]. Currently, a quasi-orthogonal system is the best that we can achieve with the available

algorithms from the literature. Fig. 22 shows how a non-orthogonal grid is transformed to a quasi-

orthogonal mesh in the proximity of the boundary. This grid was deliberately relaxed to the point

where the boundary layer is completely lost. This was done to clearly show orthogonality at the

boundary. However, maintaining a proper stretching ratio in the proximity of the boundary without

aﬀecting orthogonality remains an open problem. A compromise is needed in building the grid, and

experimentation on diﬀerent topographies may be necessary.

We report a few two- and three-dimensional examples of grids generated using both algebraic

and elliptic methods. Fig. 22 shows the computational grid around a cosine function obtained by

TFI interpolation, TFI with an orthogonal multi-surface method, and with an elliptic grid generator.

The geometry is straightforward to mesh. The three methods give similar results; however, using the

elliptic method together with a multi-surface technique to achieve orthogonality clearly produces a

better boundary layer grid. The problem is taken a little further with the fully three-dimensional mesh

of the Bolund hill in Denmark. The improvement in terms of regularity of the grid in the internal

volume and in terms of quasi-orthogonality, is evident from panel (b) in Fig. 23, where the elliptic

solver was applied with a few iterations to improve the algebraic grid of panel (a).

5.2 Adaptive mesh reﬁnement

The term adaptive mesh reﬁnement (AMR) describes mesh generation techniques in which the spatial

resolution is adjusted depending on certain properties of the speciﬁc application. Within AMR one

distinguishes between static and dynamic AMR. In static AMR the mesh is adjusted once at the

beginning of the simulation whereas dynamic AMR adapts the resolution for the whole duration of

the simulation as a function of the structure of the solution based on some pre-deﬁned criterion.

The idea to increase the resolution in part of the domain has a long history in scientiﬁc computing

and also in meteorology. Usually this adjustment of the resolution is done by coupling two numerical

54 Simone Marras1et al.

(a) (b) (c)

Fig. 22: (a) TFI, (b) orthogonal, (c) elliptic (not orthogonal).

(a) (b)

Fig. 23: (a) TFI and (b) elliptic volume grids. In this plot there is no grid control in the proximity of the boundary

surface. The elliptic grid is computed with 50 iterations.

models with diﬀerent resolutions (so called nesting). The easiest way to implement nesting is to ﬁrst

run a full coarse simulation and then use the result of this coarse simulation as boundary conditions for

a higher resolved simulation in a smaller domain. In this approach the result of the higher resolution

simulation cannot aﬀect the coarse simulation. For this reason this approach is called one-way nesting.

An example for one-way nesting can be found in the 1976 work by Davies [72] and the work of

Miyakoda and Rosati in 1977 [220]. More diﬃcult but also more accurate is two-way nesting in which

both numerical models are allowed to interact which each other. This means that the result on the

coarse mesh is not only used as an initial and boundary condition of the ﬁner mesh but the result

on the ﬁner mesh is also used to improve the accuracy of the simulation using the coarse mesh. An

example of two-way nesting is the work by Zhang et al. [330]. Nesting does not need to be static.

The domain of the higher resolution simulation can move within the domain of the coarse resolution

simulation like Ley and Elsberry did in 1976 [200]. Nesting is not restricted to combining two diﬀerent

simulations. More than two diﬀerent resolutions can be combined, as in Ginis et al.[109].

An alternative to increasing the spatial resolution via nesting is to use variable mesh spacing in

the diﬀerent directions like in Staniforth and Mitchell [280], or to adjust the mesh with the help of a

coordinate transformation, as in Dietachmeyer and Droegemeier [75].

Dynamic AMR in which the mesh is repeatedly adjusted according to the current intermediate

result of the simulation has been used in engineering applications for a long time [24; 23]. The ﬁrst

application of this kind of dynamic AMR in atmospheric sciences was done by Skamarock et al. in

1989 [272] and Skamarock and Klemp in 1993 [270]. A ﬁrst approach to use dynamic adaptive mesh

reﬁnement operationally was given by the OMEGA model (OMEGA stands for Operational Multiscale

EBG in atmospheric modeling 55

Environment Model with Grid Adaptivity). The OMEGA model was presented in the work of Bacon

et al. in 2000 [8]. Simulations of hurricane tracks by Gopalakrishnan et al. in 2002 [121] demonstrate

that the accuracy of the hurricane simulation can be improved signiﬁcantly by using dynamic AMR

while at the same time reducing the runtime of the simulation. There are however still many open

questions that need to be resolved for a broader application of dynamic AMR in atmospheric sciences

[315]. More details about the historical evolution of AMR can be found in [18] and [159].

Within dynamic mesh reﬁnement there are three possible approaches to adjust the accuracy of

the simulation according to the current ﬂow:

–h-adaptive mesh reﬁnement: the spatial resolution is adjusted by adding or removing grid points in

the mesh. In an element-based method this is done by subdividing elements into smaller elements

or merging elements into larger elements. The total number of elements and grid points is allowed

to change in this approach. This makes it necessary to redistribute elements sometimes when

multiple computing nodes are used for the computation. We discuss this approach more in detail

below.

–r-adaptive mesh reﬁnement (or moving mesh): the grid points and therefore elements are moved

and deformed in such a way that the spatial resolution gets ﬁner in those parts of the domain

where the accuracy of the simulation needs to be increased. This reduces automatically the density

of grid points in other parts of the domain and therefore reduces accuracy in those parts. The total

number of grid points and elements is constant in this approach. An example of this approach can

be found in the work of Kühnlein et al. [188], Budd and Williams [40], and Bauer et al. [16].

–p-adaptive mesh reﬁnement: the accuracy of the simulation is adjusted by changing the polynomial

order of the spatial discretization. The size and location of the elements remains unchanged in

this approach. One of the ﬁrst descriptions of this approach can be found in the work of Babuska

et al. in 1981 [7]. Application of this approach to geophysical modeling can be found recently in

[303] and, earlier on in [90].

These three approaches for dynamic AMR can be combined with each other like in the work by

Lang et al. [190], Pigott et al. [239], and [111] and references therein.

We concentrate in the following on h-adaptive mesh reﬁnement more in detail. Within this ap-

proach we can distinguish between conforming AMR and non-conforming AMR. A rising warm air

bubble simulation using conforming and non-conforming AMR is shown in Fig. 24.

5.3 Non-conforming mesh reﬁnement

As explained in the section above, mesh reﬁnement techniques create either conforming or non-

conforming meshes. In conforming meshes each element has only one neighbor per element face

(h-conforming), so a situation where more than two elements share the same face is not allowed.

Also, the elements have to be p-conforming, that is the approximating polynomials in both neighbor-

ing elements are of the same order and the nodal points on both sides of the face coincide. An example

of a hp-conforming element interface is shown in Fig. 25, where elements A and B exclusively share

the same face (side) and have the same polynomial expansion order. In such a situation there are no

additional requirements for the numerical method. The burden of creating a conforming mesh after

the reﬁnement falls entirely on the AMR algorithm.

In non-conforming meshes, however, one needs to account for faces that are shared by more

than two elements (h-non-conforming, see interface between elements C, D, and E in Fig. 25) or

with diﬀerent polynomial approximation on both sides (p-non-conforming interface between elements

B and C in Fig. 25). Mind that Fig. 25 does not illustrate all the possibilities of non-conforming

conﬁgurations. One other possibility is a hp-non-conforming interface, where an edge (or face) is

shared by more than two elements of diﬀerent polynomial orders. To complete the discussion of

meshes and element-based Galerkin methods, in this section we provide an overview of methods used

56 Simone Marras1et al.

s21 s22

s23s24

v10

v11

v12

v13

v14

v15

s25

s26

s27

x14 x15 x16 x17 x18 x19

v04

v05

v06

v07

v08

v09

s28

s29

s30

x08 x09 x10 x11 x12 x13

s31

s32

s33

s34

s35

s36

s37

s38

s39

s40

s41

s42

s43

Fig. 24: Conforming (left) and non-conforming (right) AMR simulations of a rising thermal bubble. The left ﬁgure

is adapted from [224]. The right one is adapted from [184].

Fig. 25: Diﬀerent element interfaces, where A-B is conforming, B-C p-non-conforming, and C-D-E h-non-

conforming. Dashed lines symbolize the higher order mesh within the elements.

to reconcile non-conforming elements. Note that conforming AMR grids do not require any special

handling by the numerical method and so we will discuss conforming AMR no further, but rather,

shall concentrate on non-conforming AMR.

5.3.1 Mortar element method

The ﬁrst to introduce a non-conforming formulation for spectral element methods were Maday, Mar-

viplis and Patera [205], who presented the mortar element method (MEM), where the domain is split

into blocks of conforming elements, and a new trace space, namely mortars, is introduced to couple

the non-conforming blocks. The MEM was an extension of classical non-conforming methods in the

ﬁnite element community [285; 51; 81] with the diﬀerence that, besides being applied to the spectral

element method, it did not rely on Lagrange multipliers or master-slave relations of non-conforming

edges of the elements.

In MEM, the mortars are one-dimensional constructs (in 2D; they are two-dimensional in 3D) with

a polynomial space deﬁned on them. The task of the mortar is to reconcile the C0continuity condition

between the non-conforming elements that the mortar is connecting. In other words, the mortar is

an interface between the non-conforming element faces (see Fig. 26). The polynomial order on the

mortar is typically chosen to match the highest order expansion among the elements contributing

EBG in atmospheric modeling 57

Fig. 26: Schematic of mortar element method. The mortars are binding non-conforming elements that sum the

contribution from element edges and apply an L2projection of the mortar data back to the element edges. A

single arrow represents direct assignment of the vertex value, while double arrow represents L2projection.

to the mortar. The end-point values of the mortar solutions are constrained to match the values at

corresponding vertices of the original elements (represented by single arrows in Fig. 26). The integral

projection operation (L2projection) is deﬁned to project the solution from the mortar to the interior

points of the non-conforming element edges (double arrows in Fig. 26). If we write this operation in

matrix form as Qn×m, where nis the number of nodes on the element edge, and mis the number

of nodal points on the mortar, then the operation QTwill sum the contributions from the element

edges to the mortar. To reconcile the C0continuity condition we ﬁrst sum the contributions from

element edges on the mortar (QT), perform weighted averaging, and project the result back to the

element edges (Q). Mind that this method only minimizes the discontinuity but does not enforce a

strict C0continuity. Even though here we present only a limited spectrum of possible non-conforming

conﬁgurations, MEM is very general and can be applied in more complicated situations [205].

5.3.2 Pointwise-matching method

Another method, stemming from the ﬁnite element community, is the pointwise matching method

(PMM), or the interpolation-based method [93; 254; 47]. In this approach both hand p-non-conforming

elements are allowed, however it is assumed that for h-non-conforming elements there is one parent

edge on one side of the interface, and two children edges on the other side. In the case of the p-non-

conforming, the parent edge is the one with lower polynomial order.

Fig. 27: Schematic of the pointwise-matching method. Parent points are marked with ﬁlled circles. Values at

children points depend on the values of the parent points. Here operation Qmarks the interpolation from the

parent element to children elements.

58 Simone Marras1et al.

In Fig. 27 the parent edge belongs to element C and the parent points are marked with ﬁlled

circles. The values at the child points are interpolated from the parent points. Here Qdenotes this

interpolation. To ensure strict C0continuity, the solution from the child points is ﬁrst added to the

parent points via the operation QT. The solution is averaged at the parent edge and interpolated

onto the child edge via Q. We mark the corner points of children elements with a ﬁlled circle, as

the interpolation between those points and corresponding points at the parent edge is trivial. Unlike

MEM, the continuity here is strictly enforced.

It is possible to express PMM using mortars and hence bring those two methods into one frame-

work. This can be achieved by using the mortar infrastructure and replacing the L2projection by

interpolation. In such an approach, the choice of Q(projection or interpolation) will deﬁne the method.

Traditionally, in MEM we choose the size of mortars to correspond to short (children) edges, while

in PMM we use the parent edge as a mortar analogue. [265] investigates how diﬀerent choices of the

size of mortars aﬀects the performance of both PMM and MEM.

5.3.3 Mortar elements for DG and application to atmospheric simulations

The early work on MEM focused mainly on elliptic problems and spectral element methods [205; 25].

Kopriva [186] applied the MEM to compressible ﬂows and the DG method by imposing an additional

condition on the global conservation of the mortar approximation, as well as outﬂow conditions. The

outﬂow condition required that the solution from the "upwind side" of the mortar after projection

onto the mortar and back to the face remains unchanged. This work was used later for atmospheric

simulations in [33; 184]. Based on the two-dimensional work reported in [184], Marras et al. [211] used

MEM in a uniﬁed CG/DG shallow water model on the sphere (with static and dynamic adaptivity.)

An application of the pointwise-matching scheme to geophysical simulations can be found in [258].

In [185], Kopera and Giraldo presented a uniﬁed framwework including both CG and DG methods,

as well as integral projection (for DG) and pointwise-matching (for CG) schemes for non-conforming

interfaces and found that similar mass conservation properties can be obtained for both conﬁgurations.

5.3.4 Uniﬁed CG/DG non-conforming method

The idea of a uniﬁed CG/DG method stems from the similarity of both approaches. Much of the

mathematical operations, and therefore much of the code implementation is the same, with an excep-

tion of communication between elements. A time-step of a uniﬁed CG/DG method can be described

by the following algorithm.

1. Evaluate volume integrals for each element and store as right-hand-side RHS.

2. Perform inter-element communication:

For CG, perform Direct Stiﬀness Summation on RHS to ensure C0continuity,

For DG, evaluate element-boundary integrals (ﬂuxes) and update RHS.

3. Divide by the mass matrix. Notice that the mass matrix for CG corresponds to the assembled DG

mass matrix.

4. Perform the time-step ∂ q

∂t =RH S.

This recipe can be applied regardless of whether one constructs a conforming or non-conforming

method. The non-conforming element treatment will aﬀect only the inter-element communication

step. Even though this step is diﬀerent between the methods, it is desirable to construct both non-

conforming edge algorithms in a similar fashion. As discussed in previous sections, using the mortar

element method one can incorporate the integral projection method by [186] as well as the pointwise-

matching method into the same framework. The implementation of a uniﬁed CG/DG method with

non-conforming interfaces is described in detail in [184; 185]. Here we outline the general approach to

both CG (using pointwise-matching method) and DG (using integral projection method) treatment

of non-conforming interfaces.

EBG in atmospheric modeling 59

(a) (b)

Fig. 28: Schematic of the mortar based non-conforming methods for both CG (a) and DG (b). Both methods follow

the same algorithm of communicating data to the mortar, performing operations on the mortar and communicating

the data back. The diﬀerences lie in the choice of mortars (mortar conforming with the long edge plus an additional

point mortar to communicate between vertex neighbors for CG, mortars conforming with short edges for DG)

and matrices used to communicate data to and from the non-conforming mortar. In the case of CG, the data is

interpolated using matrices JT

1,2and J1,2; for DG we use projection matrices Ps

1,2and Pg

1,2.

Fig. 28 shows a schematic of the uniﬁed approach to inter-element communication of CG and

DG methods for conforming and non-conforming interfaces. Panel (a) shows the interpolation-based

mortar point wise matching method used for CG, while panel (b) presents the integral projection

method for DG. In both situations the solution from non-conforming edges is ﬁrst communicated

onto mortars, then an appropriate action is performed on the conforming mortars and the result is

communicated back to the element edges. The ﬁrst diﬀerence between the two approaches is the choice

of the mortar. For the CG method we choose the mortar to be conforming with the longer parent edge

(so called long rule [265]), while for the DG method we choose the shorter, children edges to deﬁne

the size of the mortar (short rule). Additionally, for the CG method we need to deﬁne an additional,

point mortar to ensure the communication between vertex neighboring elements.

The second diﬀerence between the two methods is the choice of non-conforming communication

matrices. For CG we perform an interpolation with the matrix Jk:

Jk,ij =hi(ξk(zj)), k = 1,2,

where hiare the basis functions deﬁned on the mortar, ξkis a map from the element edge coordinate

zto the mortar coordinate ξ, and zjis the coordinate of the j-th nodal point at the element edge.

The interpolation matrices Jkwill scatter the solution from the mortar to two children element edges.

The opposite action, gather, is performed by the transpose matrices JT

k. The matrices Jkcorrespond

to the operation Qshowed in Fig. 27.

For the DG method the communication from parent element (long edge) to mortars requires

projection matrices, deﬁned in [186] by imposing the integral condition on a non-conforming interface

Z1

−1

(qk

M(ξ)−q(ξk(z)))ψ(ξ)dξ = 0, k = 1,2

where qk

M(ξ)is the solution projected at the k-th mortar, q(ξk(z)) is the solution at the parent edge,

ξis the coordinate deﬁned on the mortar, zis the coordinate of the parent edge and ξk(z)is the map

60 Simone Marras1et al.

between the parent edge and k-th mortar coordinates. This integral condition can be expressed in

matrix form as:

Mqk

M−Skq= 0,

where Mij =R1

−1ψi(ξ)ψj(ξ)dξ and Sk

ij =R1

−1ψi