Technical ReportPDF Available

ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code

ALE3D: An Arbitrary
Multi-Physics Code
Charles Noble, Andrew Anderson, Nathan Barton, Jamie
Bramwell, Arlie Capps, Michael Chang, Jin Chou, David
Dawson, Emily Diana, Timothy Dunn, Douglas Faux, Aaron
Fisher, Patrick Greene, Ines Heinz, Yuliya Kanarska, Saad
Khairallah, Benjamin Liu, Jon Margraf, Albert Nichols, Robert
Nourgaliev, Michael Puso, James Reus, Peter Robinson, Alek
Shestakov, Jerome Solberg, Daniel Taller, Paul Tsuji,
Christopher White, Jeremy White
May 23, 2017
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory ii
This document was prepared as an account of work sponsored by an agency of the United States
government. Neither the United States government nor Lawrence Livermore National Security,
LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any
legal liability or responsibility for the accuracy, completeness, or usefulness of any information,
apparatus, product, or process disclosed, or represents that its use would not infringe privately
owned rights. Reference herein to any specific commercial product, process, or service by trade
name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its
endorsement, recommendation, or favoring by the United States government or Lawrence
Livermore National Security, LLC. The views and opinions of authors expressed herein do not
necessarily state or reflect those of the United States government or Lawrence Livermore
National Security, LLC, and shall not be used for advertising or product endorsement purposes.
Lawrence Livermore National Laboratory is operated by Lawrence Livermore National Security,
LLC, for the U.S. Department of Energy, National Nuclear Security Administration under
Contract DE-AC52-07NA27344.
ALE3D is a multi-physics numerical simulation software tool utilizing arbitrary-Lagrangian-
Eulerian (ALE) techniques. The code is written to address both two-dimensional (2D plane and
axisymmetric) and three-dimensional (3D) physics and engineering problems using a hybrid
finite element and finite volume formulation to model fluid and elastic-plastic response of
materials on an unstructured grid. As shown in Figure 1, ALE3D is a single code that integrates
many physical phenomena.
Figure 1. ALE3D supports a wide range of applications.
ALE3D supports a wide range of simulation needs. The ALE and mesh relaxation capabilities
broaden the scope of application in comparison to tools restricted to Lagrangian-only or
Eulerian-only approaches, while maintaining accuracy and efficiency for large, multi-physics
and complex geometry simulations. For some applications ALE can deliver accuracy similar to
Eulerian techniques using as few as 1/10th the number of mesh elements and a reduction in
memory requirements.
Figure 2 provides a chart showing that ALE3D has an integrated flexible and extendable
architecture. Beyond its foundation as a hydrodynamics and structures code, ALE3D has multi-
physics capabilities that integrate various packages through an operator splitting approach.
Additional ALE3D features include heat conduction, chemical kinetics, species diffusion,
incompressible flow, a wide range of material models, chemistry models, multi-phase flow, and
magnetohydrodynamics, which can be used in numerous combinations for long (implicit) to
short (explicit) time-scale applications.
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 4
Figure 2. ALE3D has an integrated flexible and extendable architecture supporting a
variety of mesh generators, and material models can be pluggable components.
The discretized domain or mesh may consist of arbitrarily connected hexahedral, shell and beam
elements. In 2D simulations, the mesh can comprise of arbitrarily connected quadrilaterals. The
mesh can be constructed from disjointed blocks of elements that interact at the boundaries via
slide surfaces or other types of boundary conditions.
Large mesh distortions can be addressed via mesh relaxation and/or the use of ALE techniques
where material is advected through the mesh. All components of the code participate in
advection and all the mature capabilities operate with slide surfaces. Advection is the process
whereby the mesh is modified to alleviate tangling or to preserve an Eulerian grid. Advection is
implemented in a Lagrange plus remap fashion. For each cycle, after a Lagrangian motion, the
state variables flow from the original mesh to the modified mesh. The mesh is allowed to cross
material boundaries and create multi-material elements. The ALE and mesh relaxation capability
broadens the scope of applications in comparison to tools restricted to Lagrangian or Eulerian
(advection) only approaches, while maintaining accuracy and efficiency for large, multi-physics
and complex geometry simulations.
Slide surfaces are boundaries between disjoint sections of the grid that may or may not be in
contact. They represent either physical contact discontinuities or a discontinuity in the zoning.
The unstructured grid is composed of 3D hexagonal elements that can be arbitrarily connected.
Triangular or prismatic elements are not allowed, but shell and beam structural elements can
couple to 3D elements.
The 2D capability includes explicit and implicit hydrodynamics, thermal diffusion, chemistry,
deflagration, shape generation, advection, and most explicit slide surface features.
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 5
Thermal and chemistry packages are tightly coupled and have been successfully used for long
(implicit) to short (explicit) time-scale applications. The incompressible flow module has been
used in simulations of turbulent thermal convection. A multi-phase flow package (a continuum
representation of solid particulate transport in solids, liquids and gases and impulse load at
impact) is available for simulation of multi-phase fluid, structure interaction. Another package
that solves the magnetohydrodynamics equations (MHD in 3D only) can simulate a variety of
high magnetic field, high velocity, thermal applications.
ALE3D operates on a wide variety of platforms, ranging from laptops to the world’s largest
supercomputers. ALE3D has native implementations for Windows™ and Mac workstations for
smaller scale problem sets, and it is portable to virtually any Unix-based machine with C++/C
and Fortran compilers available. The code will also run in parallel on multi-processor
Windows™ and Mac machines. While most users will be interested in Linux-based versions of
the code, it has also been ported to several other lightweight kernel operating systems.
Operation on massively parallel machines has always been a core requirement for the code.
Parallelization is implemented by decomposing the mesh into computational subdomains.
Message passing is the primary mode for communication between domains. The emphasis on
message passing means that it is possible to run parallel computations on a network of
workstations, as well as utilizing multi-core processors available on most workstations.
Explicit Hydrodynamics
ALE3D’s hydrodynamics capability captures the behavior of solids and fluids and has been
successfully used for long (implicit) to short (explicit) time-scale applications. The explicit
hydrodynamics module was developed to model the behavior of objects undergoing deformation
due to the application of shocks in the kilobar regime. For such problems, the natural time step is
consistent with the Courant time scale which governs the stability of the equations.
The code explicitly conserves mass and momentum. Following DYNA3D (Hallquist, 1982), the
stress gradients and strain rates for the Lagrange step are evaluated by a lowest-order finite-
element method. A diagonal mass matrix is used. For second-order accuracy a staggered space
and time grid is also used. The stress gradient calculation has been modified so that it is
represented by an integral of the shape function over the surface of an element rather than an
integral of the gradient of the shape function over the volume. (The stress in general is
discontinuous at the surface so that the stress gradient is a delta function.) This modification
corrects a problem in which distorted elements generated forces even though the stress field was
constant. Likewise, a more accurate volume calculation is done following J. Dukowicz (JCP)
(1984). Hour-glass modes are damped using the method of Flanagan and Belytschko (1981).
The hydrodynamics is energy based rather than temperature based. Energy conservation is not
explicitly enforced but depends on the accuracy of the time integration. This averts the problem
of converting lost kinetic energy into internal energy and overheating materials. The pressure,
viscosity and strain work evaluations are fully time centered. To integrate –PdV for nonlinear
equations-of-state, a third-order Runge-Kutta method is used. Although the explicit
hydrodynamics module is isentropic except at shocks, a temperature variable exists for each
region, and the temperature Equation of State (EOS) can be evaluated either as part of a
constitutive model or from EOS tables.
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 6
Shocks are treated with a scalar artificial viscosity. The default version simply uses
v/v for
compressing elements. Both linear and quadratic terms are available. R. Christensen’s monotonic
artificial viscosity (Christensen, 1990) has also been implemented. This scalar viscosity is
constructed using velocity gradients from which the linear component of the gradient has been
subtracted. The form can be derived from the linearized Riemann problem.
The most commonly used equations-of-state and constitutive models are analytic models using
the Mie-Grüneisen and Steinberg-Guinan forms, respectively (Steinberg, 1980 and 1996).
Several engineering constitutive models are available to facilitate modeling of structures.
Tabular equation-of-state models can be accessed through the Livermore equation-of-state
(LEOS) tables. The majority of the available material models are for isotropic materials and a
von Mises yield condition is applied. The deviatoric part of the stress tensor is integrated through
time, and the Jaumann stress rate is used to satisfy objectivity. A number of micro-structurally
based models have also been added to provide a more accurate representation of damage
evolution mechanisms leading to fracture and to capture anisotropy at both the single crystal and
polycrystal size scales. A few of these latter models do not have time-centered energy
evaluations or use high-order integration for the EOS. For all material models, a Navier-Stokes
viscosity is available to augment the stress tensor.
For most problems, high explosive (HE) detonation is simulated by using a programmed burn
model with a beta burn override. Several options are available for computing HE lighting times.
Reactive flow models for HE detonation are also available. These models use pressure and
volume dependent rate laws to describe the detonation process. The reactive flow models are
zoning dependent, need high resolution, and may have limited usefulness in 3D.
Implicit Mechanics
The implicit mechanics module was developed to model problems that evolve at time scales that
are orders of magnitude greater than the Courant time scale that determines the stability of the
explicit equations. The implicit hydrodynamics is a finite element displacement formulation with
single- or eight-point integration. Single-point integration requires the addition of hourglass
stabilization forces. The formulation solves the non-linear equilibrium equations using a
Newton-Raphson iteration surrounding a linearization of the equations. This linearization is
based on estimates for material properties, etc., at the end of the time step. Convergence of the
non-linear iterations is achieved when corrections to the displacements and the nodal forces are
sufficiently small. Slide surface constraints are supported. The implicit time integration can be
run using a quasi-static approximation, or the inertial terms can be included via a Hilber-Hughes-
Taylor stabilization of the standard Newmark time integration. The code can convert
automatically from implicit to explicit hydrodynamics when the time step is sufficiently small
that the explicit integration is more computationally efficient. The reverse transfer from explicit
to implicit hydrodynamics is also possible.
Slide Surfaces
The interaction between multiple material surfaces is captured with a numerical technique called
slide surfaces. As mentioned above, slide surfaces model contact discontinuities or mesh
discontinuities. Slide surfaces may be either two-sided (master / slave) or single-sided. In two-
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 7
sided surfaces, the nodes of each side interact with the faces of the other side to conserve
momentum and enforce impenetrability. With single-sided surfaces, any node may interact with
any face, barring nearest neighbors. Single-sided slides are useful for surfaces that fold upon
themselves and surfaces consisting of many discrete segments.
There are several options for contact enforcement: the point-on-plane method for explicit or
implicit hydrodynamics, the common-plane method (available only for single-sided surfaces) for
explicit hydrodynamics, and a mortar method for implicit hydrodynamics. In addition, ALE3D
provides an automatic slide surface capability with either the point-on-plane or common-plane
For the point-on-plane method, slide surface velocities are computed with a momentum
conserving algorithm. Each side of a slide surface is independently accelerated using interpolated
physics quantities from the opposing side. First a node is accelerated as if it were on a free
surface or a surface with a pressure boundary condition. The boundary pressure is the average of
the normal stresses on each side of the surface. The acceleration of the node is decomposed into
normal and tangential components. The normal component for each side is combined with the
interpolated normal component from the opposite side to form a center-of-mass acceleration.
This acceleration is used to integrate the velocity. Unless friction is called for, the tangential
acceleration remains that of the free surface. A distinction is then made between master and slave
surfaces for the final application of continuity boundary conditions. This last step corrects for
any lack of perfect continuity in the normal direction due to truncation errors.
Techniques for ordering nodes on one side of the slide surface with respect to nodes on the other
side and for making projections normal to the slide surface are borrowed from DYNA3D. These
techniques have been modified, however, to make them more robust. The ordering algorithm has
been improved so that nodes are not prone to penetrating the slide surface and getting lost. The
normal projection algorithm uses normals that vary across a slide surface element and capture the
effects of curvature of the surface.
The opposing sides of a slide surface may be separated and come into contact during the course
of a calculation. This is called a void. When voids close, the momenta from the opposing sides
are combined to form the center-of-mass momentum. The collision is inelastic for the first layer
of nodes on each side of the slide surface. Void closing is always calculated from the perspective
of the master side. This is done to avoid miscounting momentum transferred from one side to the
other if nodes on each side of the surface close in different cycles.
By default, the tangential velocities on each side of a slide surface are decoupled. However,
coulomb friction can be applied. In this case, a tangential force is applied that opposes any
relative velocity. This force is proportional to the normal force that is inferred from the
acceleration required to change the normal velocity from its free surface value to its center-of-
mass value. For so-called tied sliding both the normal and tangential accelerations are combined
into center-of-mass accelerations. This enables one to use slide surfaces to affect a zoning
change in the middle of a region.
Special provision is made for intersecting or overlapping slide surfaces. Intersections are
assumed to be orthogonal and errors grow to the extent that this condition is violated.
The common-plane method of contact enforcement checks for contact between each pair of
faces, defined as penetration of a plane constructed midway between the two faces. If both faces
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 8
penetrate the plane, then the intersection of their penetrations projected onto the common plane
is the area of contact, and an opposing force (penalty method) is applied to resist and reverse
penetration. This force is distributed to the nodes of the faces in contact.
The mortar method of contact is a face-on-face type of method and is implemented for use with
implicit hydrodynamics. This method considers the faces on one side of a surface that are
connected to a node, and the faces on the other side that overlap these faces, and enforces that a
weighted sum of the gap volumes of the overlap be zero.
At most one slide surface may be designated as an autocontact surface. The faces of the
autocontact surface are not explicitly specified, but typically consist of all external faces (faces
with a zone on only one side) which are not included in any other slide surface.
FEusion Embedded Mesh Coupling
The FEusion library provides an embedded mesh method that allows the coupling of two
separate, spatially overlapping meshes. The embedded mesh approach can greatly simplify the
pre-processing requirements while potentially avoiding many run-time issues related to tangling
of an ALE mesh conforming around Lagrange objects.
The approach used here, which was developed by Puso et al., uses Lagrange multipliers from a
piecewise constant space on the cut background elements to constrain the jump between
background and foreground velocities. A stabilization scheme penalizes the difference in face-
adjacent Lagrange multipliers tractions. These multipliers are solved for implicitly using a
conjugate gradient (CG) iterative method on the subset of the mesh where they are active, i.e.,
only on the cut background cells.
ALE3D uses an arbitrary Lagrangian-Eulerian (ALE) algorithm. The algorithm consists of two
distinct steps: (i) a Lagrangian step that updates nodal positions, nodal velocities, and zonal
quantities, and (ii) a remap/advection step that remaps the results of the Lagrangian step onto a
mesh determined by the relaxation method specified by the user.
In the ALE3D implementation of the ALE algorithm, the Lagrangian step is executed, a new
representation of the mesh is created, and then material variables such as mass, momentum, and
energy are advected from the old mesh to the new mesh. The advection of the material variables
is done by computing fluxes of the state variables between the Lagrangian mesh and the new
mesh. The pressure is recalculated after the remap with a call to the EOS routines. If the
materials have strength, the stress deviators, plastic strain, and other constitutive model variables
are also advected.
The advection step comprises a relaxation phase and an advection phase. An “ideal” grid is first
created using an equipotential or condition number grid relaxation algorithm. (A pure Eulerian
option is also available.) The state variables are then remapped onto the new grid by constructing
fluxes between the old grid and the new grid. The fluxes for extensive variables (mass, internal
energy, and momentum) are conservative. For pure zones, a second-order, monotonic algorithm
is used to calculate the fluxes. This technique was pioneered by van Leer (1977). Velocities are
updated by applying the results of momentum conservation. This leads to a loss of kinetic energy
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 9
(which is quadratic rather than linear in the velocity), and high-order advection is required to
limit the loss. As a rule, lost kinetic energy is not added to internal energy to explicitly force total
energy conservation. However, kinetic energy lost in the vicinity of a shock can be put back into
the internal energy field. This is useful for the purpose of propagating shocks over long distances
without degradation.
For problems with several material regions, the process of advection can create mixed elements
in which more than one material region resides in an element. While material region interfaces
are currently not tracked explicitly during the advection step, they are inferred from the volume
fractions of each region in neighboring zones. This is done to determine the order in which
regions are moved from one element to another. The ordering process preserves the integrity of
region objects as they move through the mesh. When volume fractions provide no guidance,
regions are moved from one element to another in the order in which they are numbered, so it
can be very important to order regions in such a way that “more important” (i.e., non-
background) materials have lower region numbers. Separate values of the thermodynamic state
variables are maintained for each region in a mixed element, and advection is done using a first-
order upwind method. Regions in mixed elements are allowed to relax towards pressure
equilibrium. The relaxation algorithm is based on a linearized solution to the Riemann problem.
Free surfaces are not relaxed unless tangential relaxation is explicitly set up for a free surface
nodeset. Advection can take place up to, but not across, a slide surface. Ordinarily only the slave
nodes of a two-sided slide surface are relaxed, but master-side relaxation is available (similar to
tangential relaxation). If necessary, and if slide surface nodes are forced to line up across the
slide surface, the slide surface can be deleted during the course of the calculation and thereby
allow for advection after that time.
Thermal Diffusion
Thermal diffusion is the conduction of heat from a hot to a cold temperature location in a solid or
fluid. The thermal diffusion module was originally incorporated to model manufacturing
processes such as casting, forging, rolling, and extrusion. It is implemented in a manner that
allows for application to most engineering heat transfer problems. The thermal diffusion module
has also been coupled to a chemical kinetics module for the purpose of modeling thermal
ignition of high explosives. The existing capabilities include conduction with orthotropic and
temperature dependent properties, phase changes, enclosure thermal radiation, thermal contact
resistance across interfaces, and temperature, flux, thermal radiation, and convection boundary
Two modes of coupling between the hydrodynamics and the heat transfer are supported. The first
solves the hydrodynamics and heat transfer consecutively at each step by operator splitting. The
heat transfer step changes energy without changing volume and the hydrodynamics step changes
volume without transferring heat. Cell-centered energy is made consistent with nodal
temperatures by including an adiabatic expansion source term, g (dV/V), and strain heating.
Chemical reactions also contribute a source term. The second mode solves the hydrodynamics
and heat transfer iteratively using the intermediate results of the package until they both
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 10
The heat conduction equations are solved using a finite element approach. The solution includes
implicit time integration, direct or iterative matrix solution, hexahedral elements with second-
order Gaussian quadrature integration, nonlinear convergence by successive substitution or
Newton-Raphson methods, and variable time step control with sub-cycling of the thermal
diffusion and hydrodynamics.
Heat transfer can occur across a slide surface by perfect thermal contact or by heat conduction
using ‘virtual elements’. Virtual elements are conduction and thermal radiation resistive elements
that are inserted between the slide surfaces. The virtual elements can have zero thickness or their
resistance can be scaled with gap distance. The perfect thermal contact algorithm is formulated
as a penalty method. It maintains symmetry at boundary edges through geometrical and mass
weighting, and scales the penalty function with the magnitude of the diagonal term of the
coefficient matrix.
Chemistry and Chemical Diffusion
ALE3D’s chemistry model is capable of capturing the composition, structure and properties of
chemically reacting materials. The chemical kinetics module was developed to model
decomposition of high explosive materials in thermal environments. A chemical material is a set
of materials that can transform among each other. These materials (or species) define all the
properties of the chemical material. Several models for combining the properties of the
individual species are provided. In principle, any material model that supplies a temperature and
pressure can be used as a species in a chemical material. It is even possible to combine several
models by including a chemical material as a species in another chemical material.
The composition of a chemical material can be modified through a set of user defined chemical
reactions. ALE3D organizes the reactions into groups that act on the same set of species. Only
reactions that involve all of the species present in a chemical material are included in that
material’s full reaction scheme. The change in composition is calculated implicitly with a self-
correcting Newton-Raphson technique. The temperature derivative of the change is also
calculated for use by the thermal module.
The chemical kinetics package is normally run tightly coupled to the thermal module. However,
it can also be run during the hydrodynamics phase. When run during the hydrodynamics phase,
the chemical kinetics package can be used to emulate a variety of reactive flow detonation
propagation models. This emulation is accomplished by including the appropriate set of reaction
mechanisms with the appropriate material models for the reacting species. Both the standard
Lee-Tarver (Lee and Tarver, 1980) and PERMS (Propellant Energetic Response to Mechanical
Stimuli) (Maienschein et al., 1997) models can be implemented this way.
A chemical diffusion model is also available in ALE3D. It allows for diffusion within a chemical
material (not between chemical materials). The species diffusion equations are solved using a
finite-volume approach. The operator is explicit in time and computes fluxes of chemical species
across faces between elements based on the locally determined chemical potential gradient. Both
tracer, where the diffusing species is assumed to be only a small fraction of the total mass of the
system, and non-tracer models are available.
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 11
Incompressible Flow
The incompressible flow package solves the incompressible Navier-Stokes equations and is
useful for simulations involving a fluid that can be approximated with a constant density, such as
low-speed aerodynamics or internal fluid mechanics. As a rule of thumb, flows with a Mach
number less than 0.3 are candidates to be considered incompressible. An obvious exception to
this rule is any flow where the density variations drive the flow, no matter how fast it is going.
The package can be coupled to the thermal package for simulations of natural convection of a
fluid with temperature gradients. The incompressible flow model may also be run simultaneously
with the standard ALE hydro package for coupled fluid-structure problems. The fluid and solid
components are coupled through their boundary conditions. The incompressible flow module
includes explicit, semi-implicit (either the advection or viscous terms implicit) and fully implicit
time integration options. An advection-diffusion solver is also available to model species
concentrations within an incompressible material.
Multiphase Flow
The ALE3D multiphase flow model was developed for simulating mixed materials with separate
velocity fields. This model reuses the species framework developed for the chemical materials.
In principle, any ALE3D material model that supplies a temperature and pressure can be used as
a species in the chemistry and multiphase packages. Often, it is desirable to create a hierarchical
chemical material for use in the multiphase model. An example of this is a multiphase flow of
particles in a fluid flow, each composed of multiple constituents that can react amongst
themselves. This hierarchy of material models enables chemical reactions to be modeled in a
unified manner with multiphase flow.
There are several models for drag and compaction in the multiphase flow package. These
include semi-analytic models where the drag terms can be integrated analytically to those which
are evaluated using backward Euler. For cases where the compaction viscosity is unknown, the
pressure relaxation module defaults to an infinite relaxation rate or pressure equilibration.
ALE3D’s magnetohydrodynamics model is capable of capturing the dynamics of electrically
conducting solids and fluids. The magnetohydrodynamics (MHD) module was developed
primarily for the modeling of coupled electro-thermal-mechanical (ETM) systems that are
inherently 3D in nature. Example applications for this capability include explosively driven
magnetic flux compression generators, induction heating / metal forming and electromagnetic
rail gun systems. The ALE3D MHD module solves the resistive magnetic induction equation
given a collection of specified current and voltage sources. The equation is solved in the
Lagrangian frame using a mixed finite element method employing H(Curl) and H(Div) finite
element basis functions which preserves the solenoidal nature of the magnetic field to machine
precision. Electromagnetic force and resistive Joule heating terms are coupled to the equations of
motion and thermal diffusion in an operator split manner. For problems that require mesh
relaxation, magnetic advection is performed using the method of algebraic constrained transport
that is valid for unstructured hexahedral grids with arbitrary mesh velocities. The advection
method maintains the divergence free nature of the magnetic field and is second-order accurate
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 12
in regions where the solution is sufficiently smooth. For regions in which the magnetic field is
discontinuous (e.g., MHD shocks), the advection step is limited using the method of algebraic
flux correction, which is local extremum diminishing and divergence preserving.
Parallelization and Scaling
ALE3D is parallelized across the problem space using domain decomposition to associate
separate pieces of physical space to individual processors. The implementation uses MPI to
communicate across processor boundaries. Several of the key characteristics of the ALE3D
software infrastructure contribute to the code’s scalability. Field data are contained in simple C
arrays, giving the compiler canonical loop iteration patterns to optimize. Ghost boundary data is
always stored contiguously, removing the need for gather / scatter operations during MPI
communication. Nearly all MPI communication is point-to-point, with only two global
reductions per time step: the duration of the next time step, and a global error status check.
The amount of point-to-point communication required is dependent on the type of calculation
being run. Problems running explicit hydrodynamics require a single communication to collect
the sum of the forces at nodes along domain boundaries, although some optional algorithms (e.g.
monotonic Q) require more communication. Problems running advection require a 20x – 30x
greater amount of communication, both in terms of the number of communication points, and the
amount of data typically sent. A rough breakdown of the steps performed in the advection are:
nodal relaxation; calculation of volume fluxes; identification of mixed elements and interface
reconstruction; advection of element centered variables; and momentum (node centered)
advection. If there are slide surfaces, additional communications are required. The approach
taken in ALE3D is to use a separate decomposition for slide surfaces. Nodes on one side of a
surface (the “master” side) are assigned statically to the various processors in a load-balanced
manner, and then nodes on the other (“slave”) side of the surface that are currently “close” to the
masters on a processor are assigned to the same processor.
The daunting task of fully characterizing the performance of ALE3D’s many packages has never
been done in a systematic way, but we present an overview of the scaling of the Lagrangian
hydrodynamics package that forms the core of ALE3D. Figure 3 shows the weak scaling
behavior of a 5695 Element / process Sedov problem from 256 to 96,000 processes.
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 13
Figure 3. Weak scaling results for ALE3D’s Lagrangian mechanics package.
The ALE3D code continues be under active development at LLNL. In addition to new physics
and methodology improvements, the underlying computer science framework is being modified
to ensure excellent performance on the latest generation of High Performance Computing
machines. This Export Controlled and Official Use Only code is available to analysts in the
Department of Defense and associated contractors for work related to national defense. Contact
the authors for additional information.
Christensen, Randy B., “Godunov Methods on a Staggered Mesh – An Improved Artificial
Viscosity,” Lawrence Livermore National Laboratory, Livermore, CA, UCRL-JC-105269
Dukowicz, J. K., “Conservative Rezoning (Remapping) for General Quadrilateral Meshes,” J.
Comp. Phys. 54, 411-424 (1984).
Flanagan, D.P. and Belytschko, T., “A uniform strain hexahedron and quadrilateral with
orthogonal hourglass control,” Int. J. Numer. Methods Eng. 17, 679-706 (1981).
ALE3D: An Arbitrary Lagrangian-Eulerian Multi-Physics Code
Lawrence Livermore National Laboratory 14
Hallquist, J.O., “Theoretical Manual for DYNA3D,” Lawrence Livermore National Laboratory,
Livermore, CA, UCID-19401 (June 1982).
Steinberg, D.J., Cochran, S.G., and Guinan, M.W., J. Appl. Phys. 51, 1498 (1980).
Steinberg, D.J., “Equation of State and Strength Properties of Selected Materials,” Lawrence
Livermore National Laboratory, Report UCRL-MA-106439 (1996).
Lee, EL and Tarver, C.M., "Phenomenological Model of Shock Initiation in Heterogenious
Explosives," Physics of Fluids, Vol 23, Issue 12, pp. 2362-2372.
Maienschein, J.L., Lee, E.L., Reaugh, J.E., Merrill, C.I., and Lambert, R.R., “Modeling the
Impact Response of Booster Propellants,” Proc. JANNAF Propulsion Systems Hazards
Subcommittee Meeting, 27-30 October 1997, CPIA Publication 657 Volume 2, pp 163-177
Puso, M., Sanders, J., Settgast, R. and Liu, B., “An embedded mesh method in a multiple
material ALE,” Computer Methods in Applied Mechanics and Engineering, 245, 273-289
Van Leer, B, “Monotonicity and conservation combined in a second order scheme,” J. Comp.
Phys. 14, 361-70 (1977).
Wilkins, M., “Calculation of Elastic-Plastic Flow,” Methods in Computational Physics, Vol. 3,
Fundamental Methods in Hydrodynamics, Academic Press, pp. 211-263 (1964).
... The current research directly models instrumented dynamic spherical indentation experiments in a miniature Kolsky bar apparatus, where the experiments were recently designed and implemented by Casem and co-workers [11][12][13]. The first stage of the investigation assesses suitability of the explicit dynamic FE code ALE3D [24] to reproduce the experimental test configuration and the data acquired from three different complex applied velocity histories, rather than constant indenter velocities which are much more easily implemented numerically [6,7] but impossible to achieve in real experiments. To this end, accuracy of the Johnson-Cook model enters the procedure, where representative baseline properties from the literature [15,[25][26][27] on the substrate material are used, without attempting calibration to the current indentation data. ...
... Variations among Johnson-Cook parameters exist in the literature for the same Al 6061-T6 material, depending on which experimental data and testing regimes are emphasized in the calibration procedure. Representative values based on the collective literature [15,[24][25][26]36] are used here, to be refined in Section 4 on dynamic property extraction. All current simulations are performed at an initial temperature equal to the reference temperature: 0 = , in agreement with experimental protocols for tests 1, 2, and 3. Later in Section 3, 0 is increased among some simulations to better elucidate effects of thermal softening. ...
... Even though five Johnson-Cook parameters , , , , and elastic properties , affect the dynamic indentation response when different initial temperatures are considered, whether or not reasonably unique values of all parameters could be extracted from instrumented dynamic spherical indentation data at a single representative loading rate is unknown, since multiple constitutive parameters may affect the vs. ℎ response similarly. Some non-uniqueness is inherent in the Johnson-Cook parameters even when calibrated to traditional stress-strain data, as evidenced by property ranges reported elsewhere for Al 6061-T6, for example [15,[24][25][26]36]. A potentially more robust strategy, avoiding traditional uniaxial stress-strain testing, would obtain rate insensitive properties , , and from static indentation data using previously documented procedures [28,43], then strain rate sensitivity parameter from dynamic indentation on a specimen initially at room temperature, and finally temperature sensitivity parameter from indentation on a specimen at sufficiently elevated initial temperature. ...
... These mini-apps attempt to mimic a portion of the computational workflow of scientific applications. For example, AMG2013 [35] mimics a multi-grid linear system solver for unstructured mesh in physical applications; Quick-Silver [36] mimics a portion of the Mercury code [37]; and LULESH [38] that attempts to mimic a portion of ALE3D [39]. Nonetheless, most mini-apps still contain intrinsic differences compared to scientific applications. ...
... In addition, LULESH supports multiple parallelization schemes and programming paradigms: MPI, OpenMP, CUDA, OpenACC and OpenCL [38]. LULESH was developed as part of one of the five challenge problems defined by the DARPA UHPC program [38], [42] and attempts to mimic a small portion of various multi-physics applications (specifically, ALE3D [39]) that consume up to 30% of the computing resources of the DoD and DoE [43]- [45]. ...
Full-text available
Supercomputers worldwide provide the necessary infrastructure for groundbreaking research. However, most supercomputers are not designed equally due to different desired figure of merit, which is derived from the computational bounds of the targeted scientific applications' portfolio. In turn, the design of such computers becomes an optimization process that strives to achieve the best performances possible in a multi-parameters search space. Therefore, verifying and evaluating whether a supercomputer can achieve its desired goal becomes a tedious and complex task. For this purpose, many full, mini, proxy, and benchmark applications have been introduced in the attempt to represent scientific applications. Nevertheless, as these benchmarks are hard to expand, and most importantly, are over-simplified compared to scientific applications that tend to couple multiple scientific domains, they fail to represent the true scaling capabilities. We suggest a new physical scalable benchmark framework, namely ScalSALE, based on the well-known SALE scheme. ScalSALE's main goal is to provide a simple, flexible, scalable infrastructure that can be easily expanded to include multi-physical schemes while maintaining scalable and efficient execution times. By expanding ScalSALE, the gap between the over-simplified benchmarks and scientific applications can be bridged. To achieve this goal, ScalSALE is implemented in Modern Fortran with simple OOP design patterns and supported by transparent MPI-3 blocking and non-blocking communication that allows such a scalable framework. ScalSALE is compared to LULESH via simulating the Sedov-Taylor blast wave problem using strong and weak scaling tests. ScalSALE is executed and evaluated with both rezoning options - Lagrangian and Eulerian.
... Lulesh supports various parallelization schemes and programming paradigms such as MPI, OpenMP, CUDA, OpenACC, and OpenCL [39]. It was developed as part of one of the five challenge problems defined by the DARPA UHPC program [39,40] and aims to simulate a small portion of various multi-physics applications (specifically, ALE3D [41]) that consume up to 30% of the computing resources of the DoD and DoE [26,42,43]. Due to its popularity, Lulesh is widely used for testing new hardware, applying optimizations, parallelization schemes, APIs, and more [26,44]. ...
Full-text available
Over the last decade, most of the increase in computing power has been gained by advances in accelerated many-core architectures, mainly in the form of GPGPUs. While accelerators achieve phenomenal performances in various computing tasks, their utilization requires code adaptations and transformations. Thus, OpenMP, the most common standard for multi-threading in scientific computing applications, introduced offloading capabilities between host (CPUs) and accelerators since v4.0, with increasing support in the successive v4.5, v5.0, v5.1, and the latest v5.2 versions. Recently, two state-of-the-art GPUs – the Intel Ponte Vecchio Max 1100 and the NVIDIA A100 GPUs – were released to the market, with the oneAPI and NVHPC compilers for offloading, correspondingly. In this work, we present early performance results of OpenMP offloading capabilities to these devices while specifically analyzing the portability of advanced directives (using SOLLVE’s OMPVV test suite) and the scalability of the hardware in representative scientific mini-app (the LULESH benchmark). Our results show that the coverage for version 4.5 is nearly complete in both latest NVHPC and oneAPI tools. However, we observed a lack of support in versions 5.0, 5.1, and 5.2, which is particularly noticeable when using NVHPC. From the performance perspective, we found that the PVC1100 and A100 are relatively comparable on the LULESH benchmark. While the A100 is slightly better due to faster memory bandwidth, the PVC1100 reaches the next problem size (\(400^3\)) scalably due to the larger memory size.The results are available at:
... Lulesh supports various parallelization schemes and programming paradigms such as MPI, OpenMP, CUDA, OpenACC, and OpenCL [39]. It was developed as part of one of the five challenge problems defined by the DARPA UHPC program [39,40] and aims to simulate a small portion of various multi-physics applications (specifically, ALE3D [41]) that consume up to 30% of the computing resources of the DoD and DoE [26,42,43]. Due to its popularity, Lulesh is widely used for testing new hardware, applying optimizations, parallelization schemes, APIs, and more [26,44]. ...
In situ visualization and analysis is a valuable yet under utilized commodity for the simulation community. There is hesitance or even resistance to adopting new methodologies due to the uncertainties that in situ holds for new users. There is a perceived implementation cost, maintenance cost, risk to simulation fault tolerance, potential lack of scalability, a new resource cost for running in situ processes, and more. The list of reasons why in situ is overlooked is long. We are attempting to break down this barrier by introducing Inshimtu. Inshimtu is an in situ “shim” library that enables users to try in situ before they buy into a full implementation. It does this by working with existing simulation output files, requiring no changes to simulation code. The core visualization component of Inshimtu is ParaView Catalyst, allowing it to take advantage of both interactive and non-interactive visualization pipelines that scale. We envision Inshimtu as stepping stone to show users the value of in situ and motivate them to move to one of the many existing fully-featured in situ libraries available in the community. We demonstrate the functionality of Inshimtu with a scientific workflow on the Shaheen II supercomputer.Inshimtu is available for download at:
... (b) Pressure isotherms from the EOS, prior to a Maxwell construction, in the liquid-vapor transition regime. This nonconvex version of the EOS is required by many of the continuum, multiphysics codes [179][180][181] that are ubiquitous in high-energy-density science. In regions where the EOS returns a negative pressure, material damage/spall models in these codes come into play and override the EOS, thereby avoiding thermodynamic-stability issues that would otherwise be encountered. ...
Full-text available
We describe the construction of a wide-ranged multiphase equation of state for elemental iron, consisting of four solid phases and one fluid phase. The free-energy models for the phases are constrained by fitting to a broad swath of ambient, static, and dynamic high-pressure experimental data, including recent ramp-compression and shock–ramp recrystallization experiments. In order to better describe the conditions near Earth's inner-core boundary, ab initio electronic-structure calculations of various types have been carried out for the hcp and liquid phases and used as additional constraints. Given the variability of reported iron melting temperatures in the 1–3 Mbar range, we construct EOS model variations based on two separate Tmelt(P) curves: lower-Tmelt and higher-Tmelt. Our analysis shows that only the higher-Tmelt(P) data are consistent with reported shock-melting pressure measurements. Furthermore, our examination of the recrystallization data from a recent shock–ramp x-ray diffraction study motivates a family of EOS models that address experimental uncertainties and provide a range of possible Tmelt(P) curves that have a somewhat lower melt temperature above 5 Mbar than the one reported in that study. This family of multiphase EOS models allow us to suggest new experiments to further reduce the uncertainties on the high-pressure melt curve, and they should also enable more accurate predictions of complex high-P,T processes involving iron. We show their potential utility by applying these EOSs to the investigation of properties of interest to planetary science, such as the vaporization entropy, critical temperature and density, Earth inner-core boundary temperature plus inner- and outer-core density deficits.
... Lulesh supports various parallelization schemes and programming paradigms such as MPI, OpenMP, CUDA, OpenACC, and OpenCL [38]. It was developed as part of one of the five challenge problems defined by the DARPA UHPC program [39,38] and aims to simulate a small portion of various multi-physics applications (specifically, ALE3D [40]) that consume up to 30% of the computing resources of the DoD and DoE [25,41,42]. Due to its popularity, Lulesh is widely used for testing new hardware, applying optimizations, parallelization schemes, APIs, and more [43,25]. ...
Full-text available
Over the last decade, most of the increase in computing power has been gained by advances in accelerated many-core architectures, mainly in the form of GPGPUs. While accelerators achieve phenomenal performances in various computing tasks, their utilization requires code adaptations and transformations. Thus, OpenMP, the most common standard for multi-threading in scientific computing applications, introduced offloading capabilities between host (CPUs) and accelerators since v4.0, with increasing support in the successive v4.5, v5.0, v5.1, and the latest v5.2 versions. Recently, two state-of-the-art GPUs - the Intel Ponte Vecchio Max 1100 and the NVIDIA A100 GPUs - were released to the market, with the oneAPI and GNU LLVM-backed compilation for offloading, correspondingly. In this work, we present early performance results of OpenMP offloading capabilities to these devices while specifically analyzing the potability of advanced directives (using SOLLVE's OMPVV test suite) and the scalability of the hardware in representative scientific mini-app (the LULESH benchmark). Our results show that the vast majority of the offloading directives in v4.5 and 5.0 are supported in the latest oneAPI and GNU compilers; however, the support in v5.1 and v5.2 is still lacking. From the performance perspective, we found that PVC is up to 37% better than the A100 on the LULESH benchmark, presenting better performance in computing and data movements.
... To assist in the redesign and assessment of the hole closure experiments platform, finite element simulations were performed using an arbitrary-Lagrangian-Eulerian finite element code, ALE3D, developed at Lawrence Livermore National Laboratory. 19 Pre-shot simulations were performed to help design the updated experimental configuration, with simulations examining flyer material selection, flyer layer thicknesses, and shot velocities. These simulations utilized previously published parameter sets for the strength of tantalum. ...
The science and engineering communities have significant interest in experimental platforms to evaluate and improve models for dynamic material deformation. While well-developed platforms exist, there are still gaps to fill for strain and strain rate conditions accessed during impact and other high-rate loading scenarios. To fill one such gap for strength measurements, a platform was recently developed that accesses high strain rate ([Formula: see text]) and large strain ([Formula: see text]) conditions by measuring the transient closure of a cylindrical hole using in situ x-ray imaging. In the work reported here, further refinement of the platform is performed to reduce the potential effects of porosity and anelasticity on the measurement. This helps us to isolate the strength effects that are the focus of the experiment. The updated experimental configuration employs a two-layer flyer design and elongated target to reduce the magnitude of the tensile excursions associated with rarefaction wave interactions. This allows for a more direct assessment of strength models commonly used for dynamic simulations of metals. We apply the new technique to well-characterized tantalum material, allowing for a robust connection to other experimental techniques. Deformation localization can be a concern in large strain experiments, and to help inform future use of the experimental platform, we use simulations with a sub-zone treatment of shear banding to explore potential localization behavior. Overall, we develop and utilize an experimental configuration with improved isolation of strength effects that can be applied to an expanded range of materials.
In this paper, we present results of plate impact simulations of shock compressed woven glass fiber-reinforced plastic (GRP) performed using the Arbitrary Lagrangian–Eulerian three-dimensional finite element code. A hyperelastic large-strain-based empirical Continuum Damage Mechanics (CDM) formulation is employed to describe damage initiation and growth in the shock-compressed GRP. The model parameters calibration scheme utilizes the Velocity Interferometer System for Any Reflector normal particle velocity measurements at the free surface of the GRP target plates. The impact velocity in the experiments ranged from 8.5 to 418 m/s. The finite element model considered planar 0°/90° bidirectional plies with an individual ply thickness of 0.68 mm, stacked to reach a total laminate thickness of 6.8 mm. The anisotropic elastic strains were estimated from the experimentally determined tetragonal symmetry stiffness matrix for the GRP. The strain-based damage model captures several salient features observed in the measured free surface particle wave profiles, including the shock rise time, onset of Elastic—Elastic Cracking, and the shape of the nonlinear portion of the experimental particle velocity profiles. The CDM model predicts the dominant damage mode to be matrix microcracking due to shear and the associated bulk expansion (bulking) under the global compressive loading in the plate impact configuration.
There have been several numerical studies on the collapse of internal voids in energetic grains but fewer investigations have probed grain‐grain interface effects. In this study, we examine the effects of grain surface morphology and binder conditions on hot spot mechanisms during shock loading using the multi‐physics hydrocode, ALE3D, coupled with the thermochemical code, Cheetah. To improve the accuracy of our grain‐scale simulations, the HMX material models have been updated from previous studies to incorporate new property predictions from molecular dynamics simulations. In our simulations of the interface between two neighboring energetic grains, the upstream grain surface is described by a single sinusoid with amplitude (0.5‐2 μm) and wavelength (0.5‐2 μm). The effects of grain surface coating (i.e., no binder, partially coated, and fully coated with binder) is also explored. A range of shock loading pressures (10‐30 GPa) is considered. Binder coating reduces hot spots at grain‐grain interfaces by limiting plastic deformation and heating of upstream grain surfaces as well as preventing transmission of localized deformation modes (e.g., jetting) to the downstream grain. More plastic work and heating is observed with new HMX material models versus the previous models because the strength does not thermally soften as much as a result of the higher melt temperature; this also reduces vortical flow and jetting mechanisms with the new HMX material models. A surrogate model based on neural networks is developed for the early stage reaction rate. Physics‐based input features improve the performance of the neural networks over basic input features alone with a root mean squared error = 1.05 μs‐1 and R2 = 0.98. Accurate and fast‐running surrogate models can effectively serve as structure‐property‐sensitivity relationships.
Full-text available
A new approach for treating the mechanical interactions of overlapping finite element meshes is presented. Referred to as embedded mesh methods here, these overlapping mesh methods typically include a foreground solid mesh and a background Euler fluid grid or solid mesh. A number of different approaches have been used in previous work to characterize the interactions of the background and foreground meshes at the interface. Lagrange multipliers are well suited to enforce the continuity constraints but care must be taken such that the resulting formulation is stable. Several Lagrange multiplier techniques are examined in this work and applied to coupling solid meshes and fluid-structure interaction type problems. In addition, details regarding implementation in a two-step, multi-material, Arbitrary Lagrangian Eulerian (ALE) code are presented. Example problems demonstrate convergence and applicability to a range of problems. In particular, the fluid-structure interaction examples focus on blast applications.
Full-text available
Fromm's second-order scheme for integrating the linear convection equation is made monotonic through the inclusion of nonlinear feedback terms. Care is taken to keep the scheme in conservation form. When applied to a quadratic conservation law, the scheme notably yields a monotonic shock profile, with a width of only 112 mesh.
A classic problem in Lagrangian numerical hydrodynamics is the conservative transfer of quantities from an old,, distorted mesh to a new mesh. The same problem arises whenever the mesh is changed as, for example, in adaptive mesh techniques. This transfer of information is an interpolation process which is frequently called rezoning (or remapping). The general problem of conservative rezoning from one arbitrary mesh to another may be formulated as follows: m/sub k/ = ..integral integral integral../sub V//sub k/rho(r)dV. That is, we compute the mass m/sub k/ of each cell of the new mesh by integrating the known density distribution in the old mesh over the cell volume V/sub k/. A direct intregation is generally prohibitive. We show, however, that is is possible to convert this integral to a surface integral by the appropriate use of the divergence theorem, thus greatly reducing the complexity of the problem. For two-dimensional generall quadrilateral meshes the resulting method is exact and particularly simple.
An ignition and growth concept is used, within the framework of a one-dimensional Lagrangian hydrodynamic code, to model the shock initiation of heterogeneous solid explosives. The leading shock wave of an initiating pulse is assumed to ignite a small fraction of the explosive at localized heated regions. These ignited regions then grow as material is consumed at their boundaries. The growth rate for a particular material is assumed to have the characteristic pressure dependence of high-pressure laminar burning experiments. Results of the model calculations are in good quantitative agreement with recent manganin pressure gage and particle velocity gage measurements of the buildup of the initiating shock front to detonation for both sustained and short duration pulses in four solid explosives: PBX−9404, TATB, cast TNT, and PETN. The predicted run distances to detonation as functions of shock pressure at various initial densities and the predicted reaction zone lengths of the fully developed detonation waves also correlate well with experimental data for these four solid explosives.
This report provides a theoretical manual for DYNA3D, a vectorized explicit three-dimensional finite element code for analyzing the large deformation dynamic response of inelastic solids. A contact-impact algorithm that permits gaps and sliding along material interfaces is described. By a specialization of this algorithm, such interfaces can be rigidly tied to admit variable zoning without the need of transition regions. Spatial discretization is achieved by the use of 8-node solid elements, and the equations-of-motion are integrated by the central difference method. DYNA3D is operational on the CRAY-1 and CDC7600 computers.
The treatment of zero energy modes which arise due to one-point integration of first-order isoparametric finite elements is addressed. A method for precisely isolating these modes for arbitrary geometry is developed. Two hourglass control schemes, viscous and elastic, are presented and examined. In addition, a convenient one-point integration scheme which analytically integrates the element volume and uniform strain modes is presented.
A model, applicable at high‐strain rate, is presented for the shear modulus and yield strength as functions of equivalent plastic strain, pressure, and internal energy (temperature). The parameters needed to implement the model have been determined for 14 metals. Using this model, hydrodynamic computer simulations have been successful in reproducing measured stress and free‐surface‐velocity–vs–time data for a number of shock‐wave experiments.
Godunov Methods on a Staggered Mesh -An Improved Artificial Viscosity
  • Randy B Christensen
Christensen, Randy B., "Godunov Methods on a Staggered Mesh -An Improved Artificial Viscosity," Lawrence Livermore National Laboratory, Livermore, CA, UCRL-JC-105269 (1990).
Equation of State and Strength Properties of Selected Materials
  • D J Steinberg
Steinberg, D.J., "Equation of State and Strength Properties of Selected Materials," Lawrence Livermore National Laboratory, Report UCRL-MA-106439 (1996).