Mark Petersen

Los Alamos National Laboratory | LANL · Computer, Computational, and Statistical Sciences Division

The U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) version 2.1 builds on E3SMv2 with several changes, the most notable being the addition of the Fox-Kemper et al. (2011) mixed layer eddy parameterization. This parameterization captures the effect of finite-amplitude, mixed layer eddies as an overturning streamfunction and has...

The Courant-Friedrichs-Lewy (CFL) condition is a well known, necessary condition for the stability of explicit time-stepping schemes that effectively places a limit on the size of the largest admittable time-step for a given problem. We formulate and present a new local time-stepping (LTS) scheme optimized, in the CFL sense, for the shallow water e...

The development of any atmosphere or ocean model warrants a suite of test cases (TCs) to verify its spatial and temporal discretizations, order of accuracy, stability, reproducibility, portability, scalability, etc. In this paper, we present a suite of shallow water TCs designed to verify the barotropic solver of atmosphere and ocean models. These...

We present the formulation and optimization of a Runge–Kutta-type time-stepping scheme for solving the shallow-water equations, aimed at substantially increasing the effective allowable time step over that of comparable methods. This scheme, called FB-RK(3,2), uses weighted forward–backward averaging of thickness data to advance the momentum equati...

Some programming languages are easy to develop at the cost of slow execution, while others are fast at runtime but much more difficult to write. Julia is a programming language that aims to be the best of both worlds – a development and production language at the same time. To test Julia's utility in scientific high-performance computing (HPC), we...

We demonstrate a formulation of the Azouani-Olson-Titi (AOT) algorithm in the MPAS-Ocean implementation of the primitive equations of the ocean, presenting global ocean simulations with realistic coastlines and bathymetry. We observe an exponentially fast decay in the error before reaching a certain error level, which depends on the terms involved...

We present the formulation and optimization of a Runge-Kutta-type time-stepping scheme for solving the shallow water equations, aimed at substantially increasing the effective allowable time-step over that of comparable methods. This scheme, called FB-RK(3,2), uses weighted forward-backward averaging of thickness data to advance the momentum equati...

Self attraction and earth-loading effects are important for accurately modeling global tides. A common approach of handling this forcing is to expand mass anomalies into spherical harmonics, which are scaled by load Love numbers to account for elastic earth deformation. We investigate two different approaches to perform these calculations for ocean...

Oceanic tides are seldom represented in Earth system models (ESMs) owing to the need for high horizontal resolution to accurately represent the associated barotropic waves close to coasts. This paper presents results of tides implemented in the Model for Prediction Across Scales–Ocean or MPAS-Ocean, which is the ocean component within the U.S. Depa...

Some programming languages are easy to develop at the cost of slow execution, while others are fast at run time but much more difficult to write. Julia is a programming language that aims to be the best of both worlds – a development and production language at the same time. To test Julia’s utility in scientific high-performance computing (HPC), we...

This paper presents the first practical application of local time‐stepping (LTS) schemes in the Model for Prediction Across Scales‐Ocean (MPAS‐O). We use LTS schemes in a single‐layer, global ocean model that predicts the storm surge around the eastern coast of the United States during Hurricane Sandy. The variable‐resolution meshes used are of unp...

Some programming languages are easy to develop at the cost of slow execution, while others are lightning fast at run time but are much more difficult to write. Julia is a programming language that aims to be the best of both worlds—a development and production language at the same time. To test Julia’s utility in scientific high-performance computi...

The development of any atmosphere or ocean model warrants a suite of test cases to verify its spatial and temporal discretizations, order of accuracy, stability, reproducability, portability, scalability, etc. In this paper, we present a suite of shallow water test cases designed to verify the barotropic solver of atmosphere and ocean models. These...

This work documents version two of the Department of Energy's Energy Exascale Earth System Model (E3SM). E3SMv2 is a significant evolution from its predecessor E3SMv1, resulting in a model that is nearly twice as fast and with a simulated climate that is improved in many metrics. We describe the physical climate model in its lower horizontal resolu...

We examine ocean tides in the barotropic version of the Model for Prediction Across Scales (MPAS‐Ocean), the ocean component of the Department of Energy Earth system model. We focus on four factors that affect tidal accuracy: self‐attraction and loading (SAL), model resolution, details of the underlying bathymetry, and parameterized topographic wav...

We present a new unstructured CD-grid formulation of the elastic-viscous-plastic rheology, where the velocity unknowns are co-located at the edges. Our framework of choice is the Model for Prediction Across Scales (MPAS) within E3SM, the climate model of the U.S. Department of Energy, although our approach is general and could be applied to other m...

B-grid formulations of the sea ice dynamics have been dominant historically because they have matched the grid type used by ocean models. With a B-grid, the sea ice dynamics are solved at the vertices of the mesh cells. The divergence of the stress is the most challenging term to discretize in the sea-ice dynamics, and with a B-grid formulation, ea...

Oceanic tides are seldom represented in Earth System Models (ESMs) owing to the need for high horizontal resolution to accurately represent the associated barotropic waves close to coasts. This paper presents results of tides implemented in the Model for Prediction Across Scales–Ocean or MPAS-Ocean, which is the ocean component within the US Depart...

This paper presents the first scientific application of local time-stepping (LTS) schemes in the Model for Prediction Across Scales-Ocean (MPAS-O). We use LTS schemes in a single-layer, global ocean model that predicts the storm surge around the eastern coast of the United States during Hurricane Sandy. The variable-resolution meshes used are of un...

The Energy Exascale Earth System Model (E3SM) is a state-of-the-science Earth system model (ESM) with the ability to focus horizontal resolution of its multiple components in specific areas. Regionally refined global ESMs are motivated by the need to explicitly resolve, rather than parameterize, relevant physics within the regions of refined resolu...

The processes responsible for freshwater flux from the Antarctic Ice Sheet (AIS), ice‐shelf basal melting and iceberg calving, are generally poorly represented in current Earth System Models (ESMs). Here we document the cryosphere configuration of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) v1.2. This includes simulati...

Historically, B-grid formulations of sea ice dynamics have been dominant because they have matched the grid type used by ocean models. The reason for the grid match is simple - it facilitates penetration of the curl of ice-ocean stress into the deep ocean with minimal numerical diffusivity because sea ice and ocean velocity are co-located. In recen...

Ground-level ozone is a pollutant that is harmful to urban populations, particularly in developing countries where it is present in significant quantities. It greatly increases the risk of heart and lung diseases and harms agricultural crops. This study hypothesized that, as a secondary pollutant, ground-level ozone is amenable to 24 h forecasting...

Arsenic, a potent carcinogen and neurotoxin, affects over 200 million people globally. Current detection methods are laborious, expensive, and unscalable, being difficult to implement in developing regions and during crises such as COVID-19. This study attempts to determine if a relationship exists between soil’s hyperspectral data and arsenic conc...

Today’s supercomputing capabilities allow ocean scientists to generate simulation data at increasingly higher spatial and temporal resolutions. However, I/O bandwidth and data storage costs limit the amount of data saved to disk. In situ methods are one solution to generate reduced data extracts, with the potential to reduce disk storage requiremen...

We assess the performance of a set of local time-stepping (LTS) schemes for the shallow water equations implemented in the Model for Prediction Across Scales (MPAS). The goal of LTS is to speed up the simulation by allowing different time-steps on different regions of the computational grid. The LTS schemes considered here were originally introduce...

The Energy Exascale Earth System Model (E3SM) is a state-of-the-science Earth system model (ESM) with the ability to focus horizontal resolution of its multiple components in specific areas. Regionally refined global ESMs are motivated by the need to explicitly resolve, rather than parameterize, relevant physics within the regions of refined resolu...

This manuscript investigates the effectiveness of two possible horizontal discretizations for the global ocean model MPAS-Ocean, both applied to Spherical Centroidal Voronoi Tessellations (SCVTs). The first discretization is TRiSK, a C-grid, finite-volume method, that possesses many desirable mimetic properties, but has a low order accuracy. The se...

Vegetation drag is a fundamental quantity directly affecting results for both long- and short-term coastal marsh and geomorphological studies. The vegetation drag in coastal marshland has been modeled by various two-dimensional (2D) and three-dimensional (3D) numerical parameterizations. 2D parameterizations treat coastal marshes as bottom roughnes...

We assess the performance of a set of local time-stepping schemes for the shallow water equations implemented in the global ocean model MPAS-Ocean. The availability of local time-stepping tools is of major relevance for ocean codes such as MPAS-Ocean, which rely on a multi-resolution approach to perform regional grid refinement, for instance in pro...

Plain Language Summary The fastest gravity‐induced oceanic waves, known as the barotropic mode, travel with a vertically uniform motion. Simulating these fast two‐dimensional waves while also capturing the slower three‐dimensional waves is a major challenge in ocean modeling. In the Model for Prediction Across Scales‐Ocean, this is currently addres...

High-resolution simulation of global climate physics enables us to model how the climate may change under a variety of future scenarios. Such simulations produce vast amounts of information and dense datasets. If interrogated in tandem, these datasets can provide holistic, vital information on Earth’s many integrated systems by revealing the manifo...

High-resolution simulation of global climate physics enables us to model how theclimate may change under a variety of future scenarios. Such simulations produce vast amountsof information and dense datasets. If interrogated in tandem, these datasets can provide holistic,vital information on Earth’s many integrated systems by revealing the manifold...

The characteristics of the wintertime Arctic Oscillation (AO) and North Atlantic Oscillation (NAO) and their impacts on climate variability over the Northern Hemisphere are important metrics for evaluating a climate system model. Observational analyses reveal that the horizontal and vertical structures in the AO and NAO exhibit a meridional dipole...

The Southern Ocean overturning circulation is driven by winds, heat fluxes, and freshwater sources. Among these sources of freshwater, Antarctic sea-ice formation and melting play the dominant role. Even though ice-shelf melt is relatively small in magnitude, it is located close to regions of convection, where it may influence dense water formation...

Abstract Climate model components utilizing unstructured meshes enable variable resolution, regionally enhanced simulations within global domains. Here we investigate the relationship between mesh quality and simulation statistics using the JIGSAW unstructured meshing library and the Model for Prediction Across Scales‐Ocean (MPAS‐Ocean) with a focu...

Climate variability and change in the Southern Hemisphere (SH) are influenced by the Southern Annular Mode (SAM) and are closely related to changes in the kinematic properties of the SH surface zonal winds. The SAM and SH surface zonal winds have strong effects on the atmospheric and oceanic circulation system. In this study we investigate the vari...

Can we do better than Voronoi diagrams when building unstructured meshes for C-grid GCMs? By exploring the relationship between discretisation error and mesh layout, a new unstructured meshing kernel is proposed for the generation of 'weighted', 'Voronoi-like' configurations known as 'Power diagrams'. Seeking to minimise errors in the discrete nume...

Abstract This work documents the first version of the U.S. Department of Energy (DOE) new Energy Exascale Earth System Model (E3SMv1). We focus on the standard resolution of the fully coupled physical model designed to address DOE mission‐relevant water cycle questions. Its components include atmosphere and land (110‐km grid spacing), ocean and sea...

Abstract The Energy Exascale Earth System Model (E3SM) is a new coupled Earth system model sponsored by the U.S Department of Energy. Here we present E3SM global simulations using active ocean and sea ice that are driven by the Coordinated Ocean‐ice Reference Experiments II (CORE‐II) interannual atmospheric forcing data set. The E3SM ocean and sea...

We report the results of a computational investigation of two recently proved blow-up criteria for the 3D incompressible Euler equations. These criteria are based on an inviscid regularization of the Euler equations known as the 3D Euler-Voigt equations. The latter are known to be globally well-posed. Moreover, simulations of the 3D Euler-Voigt equ...

In a previous article a characteristic discontinuous Galerkin (CDG) advection scheme was presented for tracer transport \cite{Lee16}. The scheme is conservative, unconditionally stable with respect to time step and scales sub-linearly with the number of tracers being advected. Here we present the implementation of the CDG advection scheme for trace...

In a previous article a characteristic discontinuous Galerkin (CDG) advection scheme was presented for tracer transport. The scheme is conservative, unconditionally stable with respect to time step and scales sub-linearly with the number of tracers being advected. Here we present the implementation of the CDG advection scheme for tracer transport w...

Figure 1: Progression of video compression results for kinetic energy. From left to right: original quality, encoding only, completely usable (CRF=30), marginally acceptable (CRF=34), and maximal compression (CRF=51). Encoding alone and CRF=30 are nearly indistinguishable from the original, CRF=34 is more noticeable, and maximal compression washes...

A new characteristic discontinuous Galerkin (CDG) advection scheme is presented. In contrast to standard discontinuous Galerkin schemes, the test functions themselves follow characteristics in order to ensure conservation and the edges of each element are also traced backwards along characteristics in order to create a swept region, which is integr...

Ocean salinity is a critical component to understanding climate change. Salinity concentrations and temperature drive large ocean currents which in turn drive global weather patterns. Melting ice caps lower salinity at the poles while river deltas bring fresh water into the ocean worldwide. These processes slow ocean currents, changing weather patt...

Fig. 1. Left: The Agulhas current and retroflection, shown in kinetic energy, m 2 s −2. Right: A three-dimensional image of the Agulhas current and retroflection, shown in kinetic energy in the Southern Ocean just south of Africa, where aqua is 1 m 2 s −2. Abstract— Climate change research relies on models to better understand and predict the compl...

MPAS-Ocean is used to simulate an idealized, density-driven overflow using the dynamics of overflow mixing and entrainment (DOME) setup. Numerical simulations are carried out using three of the vertical coordinate types available in MPAS-Ocean, including z-star with partial bottom cells, z-star with full cells, and sigma coordinates. The results ar...

Due to power and I/O constraints associated with extreme scale scientific simulations, in situ analysis and visualization will become a critical component to scientific exploration and discovery. Current analysis and visualization options at extreme scale are presented in opposition: write files to disk for interactive, exploratory analysis, or per...

An eddy is a feature associated with a rotating body of fluid, surrounded by a ring of shearing fluid. In the ocean, eddies are 10 to 150 km in diameter, are spawned by boundary currents and baroclinic instabilities, may live for hundreds of days, and travel for hundreds of kilometers. Eddies are important in climate studies because they transport...

Isopycnal diffusivity due to stirring by mesoscale eddies in an idealized, wind-forced, eddying, midlatitude ocean basin is computed using Lagrangian, in Situ, Global, High-Performance Particle Tracking (LIGHT). Simulation is performed via LIGHT within the Model for Prediction across Scales Ocean (MPAS-O). Simulations are performed at 4-, 8-, 16-,...

Scientists from the Climate, Ocean and Sea Ice Modeling Team (COSIM) at the Los Alamos National Laboratory (LANL) are interested in gaining a deeper understanding of three primary ocean currents: the Gulf Stream, the Kuroshio Current, and the Agulhas Current & Retroflection. To address these needs, visual artist Francesca Samsel teamed up with expe...

Extreme scale scientific simulations are leading a charge to exascale computation, and data analytics runs the risk of being a bottleneck to scientific discovery. Due to power and I/O constraints, we expect in situ visualization and analysis will be a critical component of these workflows. Options for extreme scale data analysis are often presented...

The vertical coordinate of the Model for Prediction Across Scales-Ocean (MPAS-Ocean) uses the Arbitrary Lagrangian-Eulerian (ALE) method, which offers a variety of configurations. When fully Eulerian, the vertical coordinate is fixed like a z-level ocean model; when fully Lagrangian there is no vertical transport through the interfaces so that the...

A three-dimensional eddy census data set was obtained from a global ocean simulation with one-tenth degree resolution and a duration of 7 years. The census includes 6.7 million eddies in daily data, which comprise 152,000 eddies tracked over their lifetimes, using a minimum lifetime cutoff of 28 days. Variables of interest include eddy diameter, th...

The ocean contains many large-scale, long-lived vortices, called mesoscale eddies, that are believed to have a role in the transport and redistribution of salt, heat, and nutrients throughout the ocean. Determining this role, however, has proven to be a challenge, since the mechanics of eddies are only partly understood; a standard definition for t...

The development of the Rayleigh-Taylor mixing layer is studied using data from an extensive new set of Direct Numerical Simulations (DNS), performed on the 0.5 Petaflops, 150k compute cores BG/L Dawn supercomputer at Lawrence Livermore National Laboratory. This includes a suite of simulations with grid size of 10242 × 4608 and Atwood number ranging...

The Model for Prediction Across Scales (MPAS) is a new software framework for the rapid development of climate model components on unstructured grids. The grids may be quasi-uniform or variable density, on a sphere or rectangular domain, and may use quadrilateral cells, triangle cells, or Voronoi tessellations. MPAS variable density grids are parti...

Eddies at a scale of approximately one hundred kilometers have been shown to be surprisingly important to understanding large-scale transport of heat and nutrients in the ocean. Due to difficulties in observing the ocean directly, the behavior of eddies below the surface is not very well understood. To fill this gap, we employ a high-resolution sim...

This article presents a visualization-assisted process that verifies scientific-simulation codes. Code verification is necessary because scientists require accurate predictions to interpret data confidently. This verification process integrates iterative hypothesis verification with comparative, feature, and quantitative visualization. Following th...