High Resolution Forward And Inverse Earthquake Modeling on Terascale Computers.
ABSTRACT For earthquake simulations to play an important role in the reduction of seismic risk, they must be capable of high resolution and high fidelity. We have developed algorithms and tools for earthquake simulation based on multiresolution hexahedral meshes. We have used this capability to carry out 1 Hz simulations of the 1994 Northridge earthquake in the LA Basin using 100 million grid points. Our wave propagation solver sustains 1.21 teraflop/s for 4 hours on 3000 AlphaServer processors at 80% parallel efficiency. Because of uncertainties in characterizing earthquake source and basin material properties, a critical remaining challenge is to invert for source and material parameter fields for complex 3D basins from records of past earthquakes. Towards this end, we present results for material and source inversion of high-resolution models of basins undergoing antiplane motion using parallel scalable inversion algorithms that overcome many of the difficulties particular to inverse heterogeneous wave propagation problems.
- SourceAvailable from: Tan Bui-Thanh[Show abstract] [Hide abstract]
ABSTRACT: Methodology is presented to derive reduced-order models for large-scale parametric applications in unsteady aerodynamics. The specific case considered in this paper is a computational fluid dynamic (CFD) model with parametric dependence that arises from geometric shape variations. The first key contribution of the methodology is the derivation of a linearized model that permits the effects of geometry variations to be represented with an explicit affine function. The second key contribution is an adaptive sampling method that utilizes an optimization formulation to derive a reduced basis that spans the space of geometric input parameters. The method is applied to derive efficient reduced-order models for probabilistic analysis of the effects of blade geometry variation for a two-dimensional model problem governed by the Euler equations. Reduced-order models that achieve three orders of magnitude reduction in the number of states are shown to accurately reproduce CFD Monte Carlo simulation results at a fraction of the computational cost.AIAA Journal 10/2008; 46(10):16. · 1.08 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: We provide an overview of some of the issues that need to be considered in the context of quantitative seismic hazard assessment. To begin with, one needs to inventory and characterize the major faults that could produce earthquakes that would impact the region of interest. Next, one needs a seismographic network that continually records ground motion throughout the region. Data from this network may be used to assess and locate seismicity, calibrate ground motion simulations, and to conduct seismic early-warning experiments. To assess the response of engineered structures to strong ground motion, seismographs should also be installed at various locations within such engineered structures, e.g., on bridges, overpasses, dams and in tall buildings. The ultimate goal would be to perform 'end-to-end' simulations, starting with the rupture on an earthquake fault, followed by the propagation of the resulting seismic waves from the fault to an engineered structure of interest, and concluding with an assessment of the response of this structure to the imposed ground motion. To facilitate accurate ground motion and end-to-end simulations, one needs to construct a detailed three-dimensional (3D) seismic model of the region of interest. In particular, one needs to assess the slowest shear-wave speeds within the sediments underlying the metropolitan area. Geological information, and, in particular, seismic reaction and refraction surveys are critical in this regard. In the context of end-to-end simulations, detailed numerical models of engineered structures of interest need to be constructed as well. Data recorded by the seismographic network and in engineered structures after small to moderate earthquakes may be used to assess and calibrate the seismic and engineering models based upon numerical simulations. Once the seismic and engineering models produce synthetic ground motion that match the observed ground motion reasonably well, one can perform simulations of hypothetical large earthquakes to assess anticipated strong ground motion and potential damage. Throughout this article we will use the Los Angeles and Taipei metropolitan areas as examples of how to approach quantitative seismic hazard assessment.Journal of Earthquake and Tsunami 01/2012; 01(02). · 0.31 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: Abstract Snowbird is a system that simplifies the development,and use of applications that alternate between
University of Pennsylvania
Departmental Papers (MEAM)
Department of Mechanical Engineering &
High resolution forward and inverse earthquake
modeling on terascale computers
Carnegie Mellon University
Carnegie Mellon University
University of Pennsylvania, email@example.com
Carnegie Mellon University
Carnegie Mellon University
See next page for additional authors
© ACM, 2003. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The
definitive version was published inProceedings of the ACM/IEEE SC2003 ConferenceNovember 15 - 21, 2003, Phoenix, Arizona.
NOTE: At the time of publication, author George Biros was affiliated with New York University. Currently (March 2005), he is a faculty member in
the Department of Mechanical Engineering and Applied Mechanics at the University of Pennsylvania.
This paper is posted at ScholarlyCommons.http://repository.upenn.edu/meam_papers/4
For more information, please firstname.lastname@example.org.
Volkan Akcelik, Jacobo Bielak, George Biros, Ioannis Epanomeritakis, Antonio Fernandez, Omar Ghattas,
Eui Joong Kim, Julio Lopez, David O'Hallaron, Tiankai Tu, and John Urbanic
This conference paper is available at ScholarlyCommons:http://repository.upenn.edu/meam_papers/4
HIGH RESOLUTION FORWARD AND INVERSE
EARTHQUAKE MODELING ON TERASCALE COMPUTERS∗
VOLKAN AKC ¸ELIK†, JACOBO BIELAK†, GEORGE BIROS‡, IOANNIS EPANOMERITAKIS†, ANTONIO
FERN´ANDEZ†, OMAR GHATTAS§, EUI JOONG KIM†, JULIO L´OPEZ¶, DAVID O’HALLARON?,
TIANKAI TU∗∗, AND JOHN URBANIC††
Abstract. For earthquake simulations to play an important role in the reduction of seismic risk, they must be
capable of high resolution and high fidelity. We have developed algorithms and tools for earthquake simulation
based on multiresolution hexahedral meshes. We have used this capability to carry out 1 Hz simulations of the
1994 Northridge earthquake in the LA Basin using 100 million grid points. Our wave propagation solver sustains
1.21 teraflop/s for 4 hours on 3000 AlphaServer processors at 80% parallel efficiency. Because of uncertainties in
characterizing earthquake source and basin material properties, a critical remaining challenge is to invert for source
and material parameter fields for complex 3D basins from records of past earthquakes. Towards this end, we present
results for material and source inversion of high-resolution models of basins undergoing antiplane motion using
parallel scalable inversion algorithms that overcome many of the difficulties particular to inverse heterogeneous
wave propagation problems.
1. Introduction. The main objective of our research is to develop the capability for gen-
erating realistic inversion-based models of complex basin geology and earthquake sources,
and to use this capability to model and forecast strong ground motion during earthquakes
in such large basins as Los Angeles. This problem is of great importance to hazard mitiga-
tion, because assessing the ground motion to which structures will be exposed during their
lifetimes is an essential fi rst step in designing earthquake-resistant facilities and retrofi tting
existing structures. Thus, ground motion modeling and forecasting are necessary precur-
sors of the design process. The Los Angeles region is a particularly critical and appropriate
basin to study, because it is the most highly populated seismic region in the U.S., it has well-
characterized geological structures (including a varied fault system), and extensive records of
past earthquakes are available.
Modeling and forecasting earthquake ground motion in large basins is a challenging and
complex task. The complexity arises from several sources. First, multiple spatial scales char-
acterize the earthquake source and basin response: the shortest wavelengths are measured in
∗This work was supported by the National Science Foundation’s Knowledge and Distributed Intelligence (KDI)
and Information Technology Research (ITR) programs (through grants CMS-9980063, ACI-0121667, and ITR-
0122464), the Department of Energy’s Scientific Discovery through Advanced Computation (SciDAC) program
through the Terascale Optimal PDE Simulations (TOPS) Center, the Computer Science Research Institute at Sandia
National Laboratories, and a grant from the Intel Corporation. Computing resources on the HP AlphaCluster system
at the Pittsburgh Supercomputing Center are NSF/AAB/PSC award BCS020001P.
†Mechanics, Algorithms, and Computing Laboratory, Department of Civil & Environmental Engineering,
Carnegie Mellon University, Pittsburgh, Pennsylvania, 15213, USA.
‡Courant Institute, for Mathematical Sciences, New York University, New York, NY, 10012, USA.
§Mechanics, Algorithms, and Computing Laboratory, Departments of Biomedical Engineering and Civil &
Environmental Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, 15213, USA.
¶Electrical and Computer Engineering Department, Carnegie Mellon University, Pittsburgh, Pennsylvania,
?Computer Science Department and Electrical and Computer Engineering Department, Carnegie Mellon Uni-
versity, Pittsburgh, Pennsylvania, 15213, USA.
∗∗Computer Science Department, Carnegie Mellon University, Pittsburgh, Pennsylvania, 15213, USA.
††Pittsburgh Supercomputing Center, Pittsburgh, Pennsylvania, 15213, USA.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted
without fee provided that copies are not made or distributed for profit or commercial advantage, and that copies bear
this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
SC’03, November 15-21, 2003, Phoenix, Arizona, USA Copyright 2003 ACM 1-58113-695-1/03/0011...$5.00
tens of meters, whereas the longest measure in kilometers; basin dimensions are on the order
of tens of kilometers, and earthquake sources up to hundreds of kilometers. Second, temporal
scales vary from the hundredths of a second necessary to resolve the highest frequencies of
the earthquake source up to as much as several minutes of shaking within the basin. Third,
many basins have highly irregular geometry. Fourth, the soils’ material properties are highly
heterogeneous. And fi fth, geology and source parameters are observable only indirectly, and
thus introduce uncertainty into the modeling process. Because of its modeling and compu-
tational complexity and its importance to hazard mitigation, earthquake simulation has been
recognized in several U.S. Federal agency reports as one of the computational Grand Chal-
In this paper we present recent work that extends our earlier capabilities for large-scale
forward modeling of earthquake ground motion  to larger basins, with softer soils, and
for higher resolved frequencies, all of which add signifi cant computational complexity, yet
are crucial for practical applications. The key components that have enabled these capa-
bilities include an octree-based out-of-core mesh generator that can generate unstructured
wavelength-tailored hexahedral meshes of sizes limited only by available disk space, and an
unstructured hexahedral mesh parallel elastic wave propagation solver that uses very little
memory and scales to thousands of processors with high parallel effi ciency and good node
performance, despite the highly irregular meshes needed to capture effi ciently the wide range
of spatial and temporal scales that characterize heterogeneous basin response.
We have used these tools to simulate the 1994 Northridge earthquake in the Greater LA
Basin at 1 Hz maximum frequency resolution and 100 m/s minimum shear wave velocity.
The resulting unstructured mesh contains over 100 million grid points and 80 million hexa-
hedral fi nite elements, which places it among the largest unstructured mesh simulations ever
conducted.1These are the most highly resolved simulations of the Northridge earthquake
carried out to date; they are made possible by the multiresolution octree-based meshes we
use (a uniform grid would have required over 1000 times more grid points to resolve the
same frequencies), as well as the low memory required by the hexahedral data structures.
The simulation code exhibits nearly 90% parallel effi ciency in scaling from 1 to 2048 pro-
cessors on LeMieux, the HP AlphaServer system at the Pittsburgh Supercomputing Center; it
sustains nearly a teraflop/s over 12 hours in solving the 300 million wave propagation ODEs
that result upon spatial discretization; and executes at 25% of the peak floating point rate on
the 2 Gflops/s Alpha processors—excellent fi gures considering the highly irregular, multires-
olution meshes that we use.2
These levels of performance are due in part to effective use of LeMieux’s fast Quadrics
network, and in part due to the design of our hexahedral code’s data structures and algorithms
so that they eschew sparse matrix-vector products in favor of more cache-friendly local dense
matrix-vector products. Most importantly, we now have in place a mesh generation and wave
propagation framework that will enable us to scale effi ciently up to the 2–4 Hz frequencies
that are of critical interest to design engineers.3Results from a 2 Hz simulation—which
involves 1.2 billion unstructured grid points—will be presented at SC2003. Because of the
low memory required by our hexahedral solver, there is suffi cient capacity on the current
AlphaServer system to accommodate the 2 Hz simulation. Because of the fi ner granularity,
1Model elasticity problem with up to 700 million grid points have been solved on the Earth Simulator, but
although the code employs an unstructured finite element data structure, the tests were conducted on regular cubic
2The 25% of peak figure compares favorably with single processor efficiencies of about 33% that are beginning
to be reported for the Earth Simulator supercomputer for unstructured meshes [28,29].
3Each doubling of frequency leads to a factor of 8 increase in grid size and factor of 16 increase in work, for a
given material model.
TERASCALE FORWARD AND INVERSE EARTHQUAKE MODELING
we expect that parallel performance will be even better than that reported here for the 1 Hz
Such high resolution simulations have been able to reproduce observed ground motion
from past earthquakes at some locations that could not be captured by lower resolution sim-
ulations. Observations at other sites, however, have not been reproduced well by high res-
olution ground motion simulations. This discrepancy is likely the result of uncertainties in
both the source and the geological model. Thus, we are led to an inverse problem: we wish
to estimate the soil property distribution that results in a predicted response that most closely
matches observed records of past earthquakes. This inverse problem requires knowledge of
the earthquake source, which means we must invert for the source model in the process of
inverting for the material model.
We have developed a capability for earthquake inversion for both material and source
properties. In this paper we describe the methodology and provide typical results for a 2D
earthquake model; results from 3D inversion will be presented at SC2003. The inverse prob-
lem is signifi cantly more diffi cult to solve than the associated forward wave propagation prob-
lem. Even when the forward problem is well-posed, possesses a unique and continuous solu-
tion, can be evolved in time to obtain a solution, and is characterized by sparse operators—the
inverse problem is ill-posed and characterized by multiple solutions that are discontinuous,
and has an operator that is dense and couples the entire time-history of response. Specialized
algorithms are therefore required, and these are described below.
The capability to invert for the source and for the crustal and basin structures permits us
to generate improved earthquake source models and improved basin material models. This,
in turn, permits us to model an ensemble of potential rupture scenarios, which improves our
ability to forecast strong ground motion during future earthquakes, an essential fi rst step in
assessing the earthquake hazard and reducing the seismic risk. The ingredients necessary to
achieve this goal are a forward earthquake simulation capability that scales to highly-resolved
geologic models and frequencies of engineering interest on terascale supercomputers; and an
inversion capability that addresses all of the fundamental challenges of inverse wave propa-
gation while also scaling to large problem sizes, high resolution, and large numbers of pro-
Below we describe our efforts to create these capabilities. Section 2 describes the earth-
quake elastic wave propagation model, fi nite element approximation, and explicit wave prop-
agation solution, and presents verifi cation, performance, and scalability data. The inverse
model and algorithm are presented in Section 3, and used to solve an inverse shear wave
propagation problem for unknown earthquake source and basin structure.
2. Forward earthquake modeling. The Quake group has been working on modeling
earthquakes in large basins on parallel supercomputers for over a decade. Over the years our
forward earthquake modeling codes have run on the TMC CM-2, Intel iWarp and Paragon,
SGI Origin, Cray T3D and T3E, and most recently the HP AlphaServer cluster. Each gen-
eration of architecture has prompted evolutionary changes to our algorithms and software
implementations. Our simulations are based on multiresolution mesh algorithms, which over-
come many of the obstacles related to the wide range of length and time scales characteriz-
ing basin earthquake response. We have pursued these methods despite the challenges they
pose for obtaining good node performance and parallel scalability on highly parallel cache-
based systems. In heterogeneous geological structures such as sedimentary basins, where
material properties vary signifi cantly throughout the domain, multiresolution meshes allow a
tremendous reduction in the number of grid points (compared to uniform meshes), because
element sizes can adapt locally to the highly-variable wavelengths of propagating seismic
waves. Furthermore, a mesh tailored to local wavelengths permits much longer time steps