Content uploaded by Yavor Kamer
Author content
All content in this area was uploaded by Yavor Kamer on Jan 22, 2021
Content may be subject to copyright.
Eur. Phys. J. Special Topics 230, 425–449 (2021)
c
EDP Sciences, Springer-Verlag GmbH Germany,
part of Springer Nature, 2021
https://doi.org/10.1140/epjst/e2020-000259-3
THE EUROPEAN
PHYSICAL JOURNAL
SPECIAL TOPICS
Regular Article
Global models for short-term earthquake
forecasting and predictive skill assessment?
Shyam Nandan1,2,a, Yavor Kamer2, Guy Ouillon3, Stefan Hiemer2, and
Didier Sornette4,5
1Windeggstrasse, 5, 8953 Dietikon, Zurich, Switzerland
2RichterX.com, Mittelweg 8, Langen 63225, Germany
3Lithophyse, 4 rue de l’Ancien S´enat, 06300 Nice, France
4ETH Zurich, Department of Management, Technology and Economics, Scheuchzerstrasse
7, 8092 Zurich, Switzerland
5Institute of Risk Analysis, Prediction and Management (Risks-X), Academy for Advanced
Interdisciplinary Studies, Southern University of Science and Technology (SUSTech),
Shenzhen, P.R. China
Received 5 October 2020 / Accepted 7 October 2020
Published online 19 January 2021
Abstract. We present rigorous tests of global short-term earthquake
forecasts using Epidemic Type Aftershock Sequence models with two
different time kernels (one with exponentially tapered Omori kernel
(ETOK) and another with linear magnitude dependent Omori kernel
(MDOK)). The tests are conducted with three different magnitude
cutoffs for the auxiliary catalog (M3, M4 or M5) and two different
magnitude cutoffs for the primary catalog (M5 or M6), in 30 day long
pseudo prospective experiments designed to forecast worldwide M≥5
and M≥6 earthquakes during the period from January 1981 to Octo-
ber 2019. MDOK ETAS models perform significantly better relative to
ETOK ETAS models. The superiority of MDOK ETAS models adds
further support to the multifractal stress activation model proposed
by Ouillon and Sornette [J. Geophys. Res.: Solid Earth 110, B04306
(2005)]. We find a significant improvement of forecasting skills by low-
ering the auxiliary catalog magnitude cutoff from 5 to 4. We unearth
evidence for a self-similarity of the triggering process as models trained
on lower magnitude events have the same forecasting skills as models
trained on higher magnitude earthquakes. Expressing our forecasts in
terms of the full distribution of earthquake rates at different spatial
resolutions, we present tests for the consistency of our model, which
is often found satisfactory but also points to a number of potential
improvements, such as incorporating anisotropic spatial kernels, and
accounting for spatial and depth dependant variations of the ETAS
parameters. The model has been implemented as a reference model on
the global earthquake prediction platform RichterX, facilitating pre-
dictive skill assessment and allowing anyone to review its prospective
performance.
?Supplementary material in the form of one pdf file available from the Journal web page
at https://doi.org/10.1140/epjst/e2020-000259-3.
ae-mail: shyam4iiser@gmail.com
426 The European Physical Journal Special Topics
1 Introduction
Over the last 40 years, the number of people living in earthquake prone regions has
almost doubled, from an estimated 1.4 to 2.7 billion [40], making earthquakes one of
the deadliest natural hazards. Currently there are no reliable methods to accurately
predict earthquakes in a short time-space window that would allow for evacuations.
Nevertheless, real-time earthquake forecasts can provide systematic assessment of
earthquake occurrence probabilities that are known to vary greatly with time. These
forecasts become especially important during seismic sequences where the public is
faced with important decisions, such as whether to return to their houses or stay out-
side. Short term earthquake probabilities vary greatly from place to place depending
on the local seismic history and their computation requires scientific expertise, com-
puter infrastructure and resources. While in most developed countries, such as Japan,
New Zealand, Italy, earthquake forecasts are publicly available, the vast majority of
the seismically vulnerable population is residing in developing countries that do not
have access to this vital product. In this sense there is a global need for a system
that can deliver worldwide, publicly accessible earthquake forecasts updated in real-
time. Such forecasts would not only inform the public and raise public risk awareness
but they would also provide local authorities with an independent and consistent
assessment of the short-term earthquake hazard.
In addition to its social utility, a real-time earthquake forecasting model with
global coverage would be an essential tool for exploring new horizons in earthquake
predictability research. In the last two decades, the Collaboratory for the Study of
Earthquake Predictability (CSEP) has facilitated internationally coordinated efforts
to develop numerous systematic tests of models forecasting future seismicity rates
using observed seismicity [21]. These efforts, commendable as they are, address only
a very specific type of models, namely seismicity based forecasts which express their
forecast as occurrence rates under the assumption of Poissonian distribution. Thus,
studies investigating the predictive potential of various dynamic and intermittent
non-seismic signals (such as thermal infrared, electromagnetic waves, electric poten-
tial differences, ground water chemistry etc.), are effectively left out since they can-
not be adequately tested in the provided CSEP framework. Recent studies have
also pointed out deficiencies that introduce biases against models that do not share
the assumptions of the CSEP testing methodology. Moreover, some researchers have
expressed concerns regarding the effective public communication of the numerous
test results associated with each model. Some have argued that test metrics should
be tailored not only to the small community of statistical seismologists, but to the
intuitive understanding of the general public and civil protection agencies.
We suggest that the drawbacks and limitations of the previous testing methodolo-
gies can be addressed by establishing a real-time global earthquake forecasting model
that serves as a benchmark for evaluating not only grid based seismicity rate models
but also a wide variety of alarm based methods. We thus introduced RichterX: a
global earthquake prediction platform where participants can query the probability
of an earthquake (or a number of earthquakes) above a given magnitude to occur in
a specific time-space window, and issue to-occur or not-to-occur predictions [25]. By
analyzing the prospective outcomes of the issued predictions we can establish if they
exhibit significant skill compared to our global reference model, and if they do, we
can successfully rank them accordingly.
This work documents the development of such a global earthquake forecasting
model derived from the Epidemic Type Aftershock Sequence (ETAS) family and
presents a novel set of rigorous tests tailored to the specific needs of short term earth-
quake forecasting and benchmarking of short term earthquake predictions. ETAS
based models have been shown to be the best contenders in the “horse race” organised
The Global Earthquake Forecasting System 427
within CSEP. Moreover, they contain generic and parsimonious assumptions that
provide consistent descriptions of the statistical properties of realised seismicity.
Specifically, the ETAS family of models is based on the following assumptions: (i)
the distinction between foreshocks, mainshocks, and aftershocks is artificial and all
earthquakes obey the same empirical laws describing their probability to occur and
their ability to trigger future earthquakes; (ii) earthquakes have their magnitude dis-
tributed according to the Gutenberg–Richter distribution; (iii) the rate of triggered
events by a given earthquake obeys the Omori-Utsu temporal law of aftershocks;
(iv) the number of earthquakes triggered by a given event obeys a productivity law
usually linking the average number of offsprings to the exponential of the triggering
earthquake magnitude; (v) triggered events are distributed in space according to a
spatially dependent power law function.
Here, we develop “horse races” between two ETAS models differing in their specifi-
cation of their time kernels, one with an exponentially tapered Omori kernel (ETOK)
and another with a magnitude dependent Omori kernel (MDOK). We define three
different training settings for the auxiliary catalog’s magnitude cutoff (3, 4 or 5)
and two different training settings for the primary catalog’s magnitude cutoff (5 or
6), in 362 pseudo prospective global experiments designed to forecast M≥5 and
M≥6 earthquakes at three different spatial resolutions, spanning scales from 45km
to 180 km. While previous works have shown the importance of accounting for spa-
tially varying ETAS parameters [30], here we assume the same ETAS parameters
hold for the whole Earth. This assumption is made for computational simplicity and
with the intention to have a uniform global reference model allowing for an easier
interpretation of the participants’ predictive performance. This also allows us to focus
on the key question we want to investigate, namely the role of a possibly magnitude
dependent Omori exponent on forecasting skills. This hypothesis has been derived
from a physics-based model of triggered seismicity based on the premises that (1)
there is an exponential dependence between seismic rupture and local stress and (2)
the stress relaxation has a long memory [37,55]. These physical ingredients predict
that the exponent of the Omori law for triggered events is an increasing function
of the magnitude of the triggering event. This prediction has been corroborated by
systematic empirical studies for California and worldwide catalogues [37], as well as
for Taiwanese [60] and Japanese [38] catalogues.
Therefore we consider the addition of magnitude dependant Omori law as a poten-
tial improvement to a global implementation of the ETAS model. We propose a
general pseudo-prospective testing experiment that can be applied to any future can-
didate model. The testing experiment is designed to address the specific needs of
a global, short-term earthquake forecasting application and correct for some of the
defects highlighted by our previous work [32,33]. In particular, we use equal sized
meshes compatible with the spatial scales available on the RichterX platform (radius
in the range of 30–300 km) and target duration of 30 days, which is the maximum
prediction time window in RichterX.
The organisation of the paper is as follows. Section 2presents the data used in
our tests and its main properties. Section 3starts with a description of the pseudo-
prospective forecasting experiments. Then, it defines the two ETAS models that are
compared. We explain how parameters inversion is performed and the details of the
simulations used to construct the forecasts. Section 4presents the results, starting
with a rate map and full distributions of earthquake numbers in the different cells cov-
ering the Earth at the eleven different resolution levels. The model comparisons are
performed in terms of pair-wise cumulative information gains, and we calculate the
statistical significance of right tailed paired t-tests. We study in details the sensitivity
of our results to the spatial resolution, number of simulations and inclusion of smaller
magnitudes in the auxiliary catalogs. We also describe how the best performing
428 The European Physical Journal Special Topics
model is adopted as a benchmark for the RichterX global earthquake prediction con-
test, which is presented in more details in a companion paper. Section 5concludes
by summarising and outlining further developments. A supplementary material doc-
ument provides additional figures and descriptions for the interested readers.
2 Data
We use the global earthquake catalog obtained from the Advanced National Seismic
System (ANSS) database. To maintain the same precision for all reported earthquakes
in the catalog, we first bin the reported magnitudes at 0.1 units. In this study, we
use all M≥5 earthquakes that occurred between January 1981 and October 2019
as our primary data source for target earthquakes.
Figure 1shows the different features of this dataset. In Figure 1a we show the
location, time and magnitudes of these earthquakes. Figure 1b shows the spatial
density of M≥5 earthquakes. This spatial density is obtained by first counting the
number of earthquakes in 1 ×1 deg2pixels, normalizing the counts by the area of
each pixel and then smoothing the resultant density. We also show the time series of
cumulative number of M≥5 and M≥6 earthquakes and the magnitudes vs. times
of M≥7 earthquakes in Figures 1c and 1d, respectively.
Finally, in Figure 1e, we show the empirical magnitude distribution of M≥5 and
M≥6 earthquakes. To each of them, we separately fit the Gutenberg–Richter(GR)
law. The maximum likelihood estimate of the parameters of the GR distribution for
M≥5 and M≥6 earthquakes are indicated in the inset in Figure 1e. In order of
obtain the exponent of the GR distribution, we use the analytical maximum likelihood
estimator for binned magnitude derived by Tinti and Mulargia [59]. Having obtained
the exponent, the prefactor of the GR distribution can be analytically estimated.
The GR law exponents obtained for both magnitude thresholds are 1.05 (M≥5)
and 1.02 (M≥6) and are thus nearly identical. Such consistency is often treated
as an indication of the completeness of the catalog [3,28]. With this reasoning, the
consistency of GR exponents indicates the completeness of the catalog for M≥5 in
our case.
For the appropriate calibration of the ETAS model, we also use the M≥3 earth-
quakes between January 1975 and October 2019 as auxiliary dataset. The use of
auxiliary dataset is often encouraged in ETAS literature [48,52,62], as it allows to
minimize the biases in the genealogy tree of earthquakes due to the missing sources
[57]. Earthquakes in the auxiliary catalogs can act only as sources during the cali-
bration of the ETAS model, thus, their completeness is not required.
For the sake of reproducibility, since catalogs are subject to updates, the catalog
used in this study is provided as a supplementary material.
3 Method
In this study, our aim is to compare the performance of different models (described in
Sect. 3.2) in forecasting future earthquakes. We do this by means of pseudo prospec-
tive experiments.
3.1 Pseudo prospective forecasting experiments
Prospective forecasting experiments are a powerful tool allowing scientists to check
if the improvements lead to better forecasts of future unseen observations. Truly
The Global Earthquake Forecasting System 429
Fig. 1. Features of primary data used in this study; (a) Location of M≥5 earthquakes
since 1981; Sizes of the earthquakes scale with their magnitude and colors show the year of
occurrence; (b) Spatial density of M≥5 earthquakes since 1981 obtained by smoothing the
counts of earthquakes in 1 ×1 deg2grid with a Gaussian filter; (c) Cumulative number of
M≥5 and M≥6 earthquakes, with their respective scaled axes on the left and right side
of the plot, respectively; (d) Magnitude vs. Time plot of M≥7 earthquakes; (e) Empirical
frequency magnitude distribution of M≥5 and M≥6 earthquakes; Lines show the best
fit Gutenberg Richter Distribution to the empirical distribution of M≥5 and M≥6
earthquakes.
prospective experiments are time consuming, as they require many years before
enough future observations accumulate to strengthen (or weaken) the evidence in
favor of a model [26]. A practical solution is to conduct pseudo prospective exper-
iments. In these experiments, one uses only the early part of the dataset for the
calibration of the models and leaves the rest as virtually unseen future data. Subse-
quently, the calibrated models are used to simulate a forecast of future observations
and the left out data is used to obtain a score for each of the forecasting models.
These scores can then be compared to identify the best model.
Although, (pseudo) prospective experiments have started to catch up in the field
of earthquake research [9,19,20,22,24,32,33,36,51,64,65], they still have not become
the norm. In this regard, the work done by the collaboratory for the study of earth-
quake predicatbility (CSEP) and others [26,51] has been very commendable, as they
have tried to bring the prospective model validation on the center stage of earth-
quake forecasting research. However, the prescribed prospective validation settings,
in particular by CSEP, remain too simple and sometimes may be biased in favour
of certain model types. For instance, most of the CSEP experiments have been con-
ducted with spatial grids which are defined as 0.1×0.1 deg2cells (see for instance [65])
or 0.05 ×0.05 deg2cells (e.g. [5]), mostly for computational convenience. However,
as the area of these cells vary with latitude, becoming smaller as one moves north
or south from the equator, a model gets evaluated at different spatial resolutions at
different latitudes, thus, by construction, yielding different performances as a func-
tion of latitude. This areal effect on the performance then gets convoluted with the
underlying spatial variation in the performance of the model. For instance, modelers
might find that their models yield better performance in California, but very poor
performance in Indonesia at a fixed spatial resolution. Should they then infer that
430 The European Physical Journal Special Topics
their models are better suited for a strike slip regime than for a subduction setting?
Due to the inherent nature of the varying spatial resolution of the grid prescribed by
CSEP, the answer to this question becomes obfuscated.
Another aspect of pseudo prospective experiments that is poorly handled by
CSEP is that it forces the modelers to assume Poissonian rates in a given space-time-
magnitude window irrespective of their best judgement. Nandan et al. [32] showed
that this choice puts the models that do not comply with the Poissonian assumption
on weaker footing than those models which agree with this assumption. As a result,
the reliability of the relative rankings obtained from the models evaluated by CSEP
remains questionable to some extent.
Last but not the least, in some model evaluation categories [50,63], CSEP eval-
uates the models using only “background” earthquakes, which are identified using
Reasenberg’s declustering algorithm as being independent [41]. However, as the earth-
quakes do not naturally come with labels such as “background”, “aftershocks” and
“foreshocks” that can be used for validation, this a posteriori identification remains
highly subjective. In regards to CSEP’s use of the Reasenberg’s declustering algo-
rithm, Nandan et al. [33] pointed out that the subjective nature of declustering
introduces a bias towards models that are consistent with the declustering tech-
nique, rather than the observed earthquakes as a whole. This puts into questions the
value of such experiments, as their results are subject to change as a function of the
declustering parameters.
Since the aim of our forecasting experiment is to assess which model is more
suitable to serve as a reference model for the global earthquake prediction platform
RichterX, it becomes important to address the drawbacks mentioned above by design-
ing a testing framework tailored to multi-resolution, short-term forecasts on a global
scale. Accordingly we have designed the pseudo prospective experiments in this study
with the following settings:
1. Many training and testing periods: we start testing models beginning on
January 1, 1990 and continue testing till October 31, 2019, spanning a duration
of nearly 30 years. The maximum duration of an earthquake prediction that can
be submitted on the RichterX platform is 30 days. Using this time window, our
pseudo prospective experiments are composed of 362 non overlapping, 30 days
long testing periods. To create the forecast for each of the testing periods, the
models are calibrated only on data prior to the beginning of each testing period
for calibration as well as simulation. The forecasts are specified on an equal area
mesh with predefined spatial resolution.
2. Equal area mesh: to create this equal area mesh, we tile the whole globe with
spherical triangles whose area is constant all over the globe. This mesh is designed
in a hierarchical fashion. To create a higher resolution mesh from a lower resolu-
tion one, the triangles in the lower resolution mesh are divided into four equal area
triangles. In this way, we create eleven levels of resolution: at the first level, the
globe is tiled with 20 equal area triangles (corresponding to an areal resolution of
≈25.5×106km2each); at the second level 80 equal area triangles tile the globe,
and so on. Finally, at level eleven ≈21 ×106triangles tile the globe with an areal
resolution of ≈24 km2. In this study, we evaluate the models at level six (unless
otherwise stated), which has an areal resolution equivalent to a circle with radius
≈90 km. To test the sensitivity of our results to the choice of areal resolution, we
also evaluate the models at level five and level seven, which correspond to an areal
resolution equivalent to circles with radii ≈180 km and ≈45 km, respectively. In
principle, the models can be evaluated at all spatial resolutions (from 1 to 11).
The resolutions in this study are chosen to be in accordance with the the spatial
extents used on the RichterX platform (radius of 30–300 km).
The Global Earthquake Forecasting System 431
3. Flexibility to use parametric or non-parametric probability distribu-
tions: the model forecasts can be specified on the equal area mesh during each
testing period either as the full distribution of earthquake numbers (as in [32])
empirically obtained from the simulations, or as a Poissonian rate or as any other
analytical probability distribution function that may be in line with the model
assumptions.
4. Performance evaluation using all earthquakes with M≥5 or M≥6:
we test the models against target sets consisting of M≥5 and M≥6 events
that occurred during each testing period. During a given testing period, competing
models forecast a distribution of earthquake numbers (M≥5 or M≥6 depending
on the choice of target magnitude threshold or Mt) in each triangular pixel. We
then count the actual number of observed earthquakes in each pixel. With these
two pieces of information, the log likelihood LLi
Aof a given model A during the
ith testing period is defined as:
LLi
A=
N
X
j=1
ln ( P ri
j(ni
j) ) (1)
where P ri
jis the probability density function (PDF) of earthquake numbers fore-
casted by model A and ni
jis the observed number of earthquakes (≥Mt) during
the ith testing period in the jth pixel. Nis the total number of pixels, which at
level six is equal to 20 480. Similarly, LLi
Bfor another competing model (B) can
be obtained. The information gain IGi
AB of model A over model B for the ith
testing period is equal to
IGi
AB = LLi
A−LLi
B.(2)
In order to ascertain if the information gain of model A over B is statistically
significant over all testing periods, we conduct a right tailed paired t-test, in which
we test the null hypothesis that the mean information gain (MIGAB =PiIGi
AB
362 )
over all testing periods is equal to 0 against the alternative that it is larger than
0. We then report the p-values obtained from the test. If the p-values obtained
from the tests are smaller than the “standard” statistical significance threshold
of 0.05, model A is considered to be statistically significantly more informative
than model B.
3.2 Competing models
3.2.1 Preliminaries on ETAS models
In this study, we conduct a contest between different variants of ETAS models only.
The reasons for this are multifold:
1. Our target use case for RichterX requires providing near-real time short time
earthquake forecasts on a global scale. ETAS models are suitable for such appli-
cations as they rely only on a timely stream of earthquake locations and mag-
nitudes, whereas models based on stress transfer require additional data, such
as fault plane orientations, rupture extent, slip distributions etc. which are often
available only after a few days.
2. Due to the abundance of target events (the world-wide average number of M≥5
and M≥6 earthquakes per month, since 1990, is ≈137 and ≈13, respectively),
global models providing short term (here, monthly) earthquake forecasts can be
432 The European Physical Journal Special Topics
tested with a greater statistical significance (even at high magnitude threshold,
such as M≥5 and M≥6, of the testing catalog), compared to their regional
counterparts.
3. On the global scale, there exist some “long term” models [2,22,24]. However, there
is no model (to the best of our knowledge) that provides short term forecasts. We
intend to fill this gap with the best performing ETAS model of this study.
4. On the regional scale, ETAS models [33] have been shown to be much more effec-
tive than standard smoothed seismicity models [18,63], which provide forecasts
of future earthquakes by smoothing the location of past background earthquakes.
However, their forecasting effectiveness on the global scale remains to be assessed.
In this study, our goal is not to provide a comprehensive test between various types
of state-of-the-art forecasting approaches [1,4,5,10,17,42,58], but rather to design
an experiment in which short term global earthquake forecasting models can be
developed, compared and enhanced. Furthermore, we only conduct the “horse race”
between the simplest ETAS models. This means that we exclude space and time
variation of its parameters, which have been actively reported by numerous authors,
at regional and global scales [6,30,35,39,66], to lead to enhanced performance. Fur-
thermore, the ETAS models considered in this study do not make use of any other
datasets such as fault networks, global strain rates, source models, focal mechanisms
and so on. Some authors [1,5,11] have shown in case studies that these additional
datasets enhance the forecasting potential of ETAS type models. We disregard these
complexities, not due to any underlying belief that they are not informative, but
because their limited availability would hinder a real-time implementation on the
RichterX platform. We also maintain that the initial model should be simple enough.
Then, adding complexities should follow sequentially, only if they can be justified
by their forecasting gains over simpler models. With these points in mind, in the
following, we describe the different variants of ETAS models that we have compared
in this study.
3.2.2 ETAS model with exponentially tapered Omori kernel (ETOK)
Model description. In this model, the seismicity rate λ(t, x, y|Ht) at any time tand
location (x, y) depends on the history of the seismicity Htup to tin the following
way:
λ(t, x, y|Ht) = µ+X
i:ti<t
g(t−ti, x −xi, y −yi, Mi)(3)
where µis the time-independent background intensity function, gis the triggering
function and (ti, xi, yi, Mi) represents the time, x-coordinate, y-coordinate and mag-
nitude of the ith earthquake in the catalog, respectively.
The memory function gin equation (3) is formulated as:
g(t−ti, x −xi, y −yi, Mi) = Kea(Mi−Mc)×e−
t−ti
τ
(t−ti+c)1+ω
×h(x−xi)2+ (y−yi)2+deγ(Mi−Mc)i−1−ρ(4)
which is the product of three kernels:
1. The productivity kernel K ea(Mi−Mc)quantifies the expected number of after-
shocks triggered by an earthquake with magnitude Miabove the magnitude of
completeness, Mc, where Kand aare the productivity constant and exponent
respectively.
The Global Earthquake Forecasting System 433
2. The exponentially tapered Omori kernel e−
t−ti
τ
(t−ti+c)1+ωquantifies the time distribu-
tion of the direct aftershocks of an earthquake that occurred at ti. The exponential
taper term e−
t−ti
τensures that the parameter ωcan attain even negative values
during the calibration of the model, which is not possible for a pure power law
distribution, as it becomes unnormalizable for exponents smaller than 1.
3. The isotropic power-law spatial kernel (x−xi)2+ (y−yi)2+deγ(Mi−Mc)−1−ρ
quantifies the spatial distribution of the aftershocks of an earthquake with mag-
nitude Mithat occurred at (xi, yi).
Note that the model defined in equations (3) and (4) implicitly assumes that
the magnitudes of both the background and the triggered earthquakes follow the
Gutenberg Richter (GR) distribution, which is described by the following probability
density function (PDF):
f(M) = βe−β(M−Mc).(5)
Note that the exponent βin expression (5) is related to the b-value reported above
in the inset of Figure 1e via β=bln 10 ≈2.3b. Thus a value of βin the range 2.3–2.4
as shown in Figure S1 in the supplementary materials corresponds to a b-value in the
range 1.0–1.04.
Due to its commonality in both the background and the triggering function, GR
law is usually factored out of the explicit formulation of the ETAS model. However,
one could imagine other formulations of ETAS models in which such factoring out
is not possible. For instance, simply allowing the background earthquakes and after-
shocks to follow GR distribution with different exponents (β1and β2) makes the
factoring impossible and the exponents β1and β2then have to be jointly inverted
with the other ETAS parameters. In this context, Nandan et al. [31] showed that,
using the Californian earthquake catalog, not only the exponents corresponding to
the background earthquakes are distinct from those of aftershocks, but also that
the magnitude distribution of direct aftershocks is scaled by the magnitude of their
mainshock. Despite these findings, we make the simplifying assumption that both
background earthquakes and aftershocks follow the same GR distribution, and factor
it out from the explicit formulation of the ETAS model. Nevertheless, the GR law
plays an explicit role when the ETAS model is used to forecast the magnitude of the
future earthquakes and its parameter βhas thus to be inverted from the training
period.
Simulation. We follow the standard algorithms for the simulation of synthetic earth-
quake catalogs for the ETAS model [32,67,68]. For completeness, a detailed descrip-
tion of the simulation is also provided in the Supplementary Text S1.
Parameter inversion and modelling choices. As described in Supplementary Text
S1, the set of parameters {µ, K, a, c, ω, τ , d, ρ, γ, β}are necessary for the simula-
tion of future earthquake catalogs. The values of these parameters are not known
in practice and they have to be inverted from the training data. The parameters
{µ, K, a, c, ω, τ , d, ρ, γ}can be inverted by calibrating the model (Eq. (3)) on the real
data by means of the Expectation Maximization (EM) algorithm proposed by Veen
and Schoenberg [61]. To obtain the parameter β, we first bin the magnitudes of the
earthquakes in the ANSS catalog in 0.1 units and then use the analytical maximum
likelihood estimator derived by Tinti and Mulargia [59] for binned magnitudes.
An important consideration before calibrating the ETAS model is the choice of
the primary and auxiliary catalogs. The main difference between these two catalogs
434 The European Physical Journal Special Topics
is that earthquakes in the primary catalog can act both as targets and sources during
the calibration of the ETAS model, while the earthquakes in the auxiliary catalogs
can act only as sources. In ETAS literature [48,52,62], the use of auxiliary catalogs
is encouraged during inversion of ETAS parameters, so as to minimize the biases in
the genealogy tree of earthquakes due to the missing sources [44,57]. In this study,
we calibrate the ETAS model (described in Eqs. (3) and (4)) using primary catalogs
with two different magnitude thresholds: Mpri = 5 and 6. Both these primary catalogs
start in year 1981 and include earthquakes from all over the globe. For the auxiliary
catalogs, which start in 1975 and are also composed of earthquakes from all over
the globe, we use three different magnitude thresholds: Maux = 3, 4 and 5 during
calibration. We use different magnitude thresholds for the primary catalogs to test
the hypothesis that better forecasting potential can be achieved for higher magnitude
thresholds if we specifically train our models for them. We use different magnitude
thresholds for the auxiliary catalog to test the hypothesis that smaller earthquakes
play an important role in triggering and can improve the forecasting potential of the
ETAS models.
Note that, even though the available ANSS catalog extends down to magnitude 0,
we do not use such a low magnitude threshold for the auxiliary and the primary cat-
alogs because: (1) in the formulation of the ETAS model, the primary catalog should
follow a GR law and be complete above the considered threshold magnitude. These
two criteria can not be fulfilled for the global ANSS catalog at magnitude thresholds
lower than 5, and extending back to year 1981; (2) lowering the magnitude threshold
of both the primary and auxiliary catalog increases enormously the computational
burden for both the inversion and simulations.
In Figure S2, we show the time evolution of the estimates of the parameters
for the ETAS model with exponentially tapered kernel for Maux = 3, 4 and 5 and
Mpri = 5. The time evolution for Mpri = 6, for the same model and the three auxiliary
magnitude settings, is shown in Figure S3. Beside the “usual” ETAS parameters,
Figure S2 shows the time series of the branching ratio. This parameter quantifies
the average number of triggered earthquakes of first generation per triggering event,
as well as the fraction of triggered earthquakes in the training catalog [16]. For a
branching ratio <1, the system is in the the sub-critical regime. For a branching
ratio >1, the system is in the super-critical regime [15]. In addition, in Figure S1,
we show the time evolution of the parameter βfor two Mpri settings. Since the
parameter βis only estimated from the primary catalog, only two time series are
obtained and not six (one for each of the two Mpri settings) as in the case of other
ETAS parameters. The time series of all the parameters is composed of 362 points,
each corresponding to one of the training catalogs preceding the 362 testing periods.
We notice that parameters show conspicuous variation with time, with a tendency to
stabilise after about 2011, perhaps reflecting a better global catalogue completeness.
We cannot exclude a genuine trend resulting from the shortness of the time series,
which are strongly impacted by the two great earthquakes of magnitude larger than
9 that occurred in 2004 (great Indian ocean earthquake) and 2011 (Tohoku, Japan).
Furthermore, some of the parameter pairs (µ,nor branching ratio), (c,ω) and so on
exhibit cross correlations. In addition, the parameters also seem to be systematically
dependent on the choices of Maux and Mpri. Investigating the sources of these time
variations, cross correlations and dependencies on auxiliary and primary magnitude
thresholds is beyond the scope of this paper. In this article, we focus on evaluating the
importance of these hyper-parameter choices (Maux and Mpri) in terms of forecasting
performance. Nevertheless, we report the time evolution of these parameter estimates
as it would aid the readers in reproducing the results presented in later sections.
The Global Earthquake Forecasting System 435
3.2.3 ETAS model with Magnitude Dependent Omori Kernel (MDOK)
Model description. While the primary equation of the seismicity rate for this ETAS
model remains the same (Eq. (3)), the triggering kernel is modified to account for a
possible magnitude dependence of Omori-Utsu parameters cand ω. The triggering
kernel for this model is redefined as:
g(t−ti, x −xi, y −yi, Mi) = Kea(Mi−Mc)×e−
t−ti
τ
[t−ti+c(Mi)]1+ω(Mi)
×h(x−xi)2+ (y−yi)2+deγ(Mi−Mc)i−1−ρ
(6)
where c(Mi) = 10c0+c1Miand ω=ω0+ω1Mi.
The functional form for c(M) is inspired from the works of Shcherbakov et al. [53],
Davidsen et al. [7] and Hainzl [12]. All three authors found the c-value to increase
exponentially with the mainshock magnitude. While the first two authors interpreted
the c-value dependence on the mainshock magnitude as a part of a self-similar earth-
quake generation process (i.e. a physical process), Hainzl [12] attributed this depen-
dence to the rate dependent aftershock incompleteness (i.e. a data sampling issue).
The latter would require to replace the missing events in some way, as they play a role
in triggering of future events. Yet no such procedure has ever been proposed. Note
that several other authors [8,34,49] have also argued for the magnitude-dependence of
the onset of the power-law decay based on ideas such as stress corrosion and rate and
state dependent friction. However, these authors suggest that the c-value would cor-
relate negatively with mainshock magnitude, as their model predicts that the larger
the stress perturbation, the shorter would be the duration between the mainshock
and the onset of the power-law decay. Regardless of the underlying mechanism for
the dependence of the c-value on mainshock magnitude, the evidence for such an
exponential dependence is rather clear, and thus warrants an explicit formulation
within the ETAS model.
The linear dependence of the Omori exponent ωon the mainshock magnitude is
based on the work of Ouillon and Sornette [37,55], who reported strong empirical
evidence together with a physics-based theory for such a dependence for mainshocks
in Californian and worldwide catalogs. Tsai et al. [60] confirmed this observation for
the Taiwanese catalog and Ouillon et al. [38] for the Japanese catalog. These authors
used a wealth of different techniques, such as various space-time windowing meth-
ods, binned aftershock time-series, wavelet analysis and time evolution of aftershocks
maximum magnitude, in order to ascertain the robustness of the results and that
the observed magnitude dependence of ωwould not be due to some bias induced
by a specific method. Ouillon and Sornette [37,55] proposed a theoretical statistical
physics framework in which the seismic rate results from an exponential Arrhenius
like activation with an energy barrier influenced by the total stress fields induced
by past earthquakes and far-field tectonic loading. These authors showed that the
combination of the exponential activation rate together with the long memory ker-
nel of stress relaxation leads to temporal multifractality expressed empirically as a
magnitude-dependent Omori exponent ω. They coined this model the multifractal
stress activation (MSA) model. More precisely, the MSA model can be rationalized
as follows:
1. the stress at any location is the sum of the far-field contribution due to tectonic
loading and the stress fluctuations due to past events;
2. each earthquake ruptures a complex set of patches whose number increases expo-
nentially with the magnitude of the event;
436 The European Physical Journal Special Topics
3. each failing patch redistributes stress in its surrounding according to the laws
of linear elasticity, so that positive or negative stress contributions add up as
patches fail and consecutive earthquakes occur. The stress transferred by a failed
patch at any target location can be treated as a random variable distributed
according to a Cauchy law, i.e. decaying as a power law with exponent (1 + ν)/2
[23]. The effect of the earthquake rupture at the target location is thus the sum
of the corresponding random variables. The exponent νthus encompasses all the
geometrical complexity of the problem: the (fractal) nature of the fault system,
the Gutenberg–Richter law (i.e. the size of the source events), the distribution of
focal mechanisms, the (possibly self-affine) morphology of slip along the rupture
plane, and the spatial decay of the stress Greens function;
4. the memory of local past stress fluctuations decays as a power-law of time, due
to rock (nonlinear) viscosity, with exponent 1 + θ. This function encapsulates all
brittle and ductile relaxation phenomena such as dislocations motion, pressure-
dissolution, slow earthquakes or even those too small to be detected. In that sense,
θcharacterizes the whole complexity of stress relaxation in the crust.
5. at any location, the seismicity rate depends exponentially on the local shear stress,
in agreement with many known underlying failure processes.
The model then predicts that the seismicity rate consists in a time invariant
base rate due to the tectonic loading, nonlinearly modulated by a time varying term
depending on past seismicity. This term can increase the rate if past (and/or most
recent) stress fluctuations are positive, but may also decrease if they are negative.
When solved self-consistently by considering all (statistical) mechanical interactions
between events, the model predicts that the Omori exponent of the triggered sequence
following an event of magnitude Mdecays with time with an exponent pincreasing
linearly with M. This peculiar feature is indeed predicted to hold exactly when the
condition ν(1 + θ) = 1 is fulfilled, which can be viewed as the consequence of the
space-time self-organization of fault networks in the brittle crust. Reviewing the
possible values of parameters νand θfor the Earth’s crust, Ouillon et al. [38] showed
that their estimations allowed them to bracket this criterion, thus evidencing another
analogy with second-order phase transitions where critical exponents are linked by
such relationships close to a critical point.
In this forecasting experiment, we aim to systematically test the idea that explic-
itly taking account of magnitude dependence in these two Omori parameters would
lead to an improvement in the forecasting ability of the modified ETAS models rel-
ative to the ones in which these dependencies are ignored.
Simulation. Given the set of parameters {µ, K, a, c0, c1, ω0, ω1, τ, d, ρ, γ , β}, the sim-
ulation of the time, location and magnitude of the future earthquakes proceed in the
same way as for a standard ETAS model (see Supplementary Text S1), except for
one difference. In this case, the times of the direct aftershocks of an earthquake with
magnitude Miare simulated using the time kernel whose parameters depend on Mi
in the way described in equation (6). This means that, despite the fact that the
MSA model is by construction nonlinear, we here consider a linear approximation for
the purpose of tractability. Indeed, in the MSA model, the exponential nonlinearity
occurs in the stress space, a variable that is not computed within the ETAS formu-
lation which focuses only on rates. A full MSA approach would require to compute
the stress transfer (and its time dependence) due to all past events, taking account
of their individual rupture complexity, and assessing all their uncertainties. As this
remains challenging in the present state of seismological research, we bypass this
obstacle and provide a simplified approach by introducing a magnitude-dependent
Omori kernel.
The Global Earthquake Forecasting System 437
Table 1. All twelve models resulting from different calibration choices; ET and MD stand
for ETAS models with exponentially tapered Omori kernel and magnitude dependent Omori
kernels, respectively.
Model numbers
1 2 3 4 5 6 7 8 9 10 11 12
Omori
kernel
ET ET ET ET ET ET MD MD MD MD MD MD
Mpri 5 5 5 6 6 6 5 5 5 6 6 6
Maux 3 4 5 3 4 5 3 4 5 3 4 5
Parameter inversion and modelling choices. Again in this case, we adapt the EM
algorithm proposed by Veen and Schoenberg [61] to invert the parameters of the
model (Eq. (6)). For the sake of completeness, we also calibrate these models with
six primary and auxiliary catalog settings as described in Section 3.2.2. Again, with-
out going into the possible underlying causes of the time variation of the estimated
parameters and their dependence on the choice of Maux and Mpri hyper-paramters,
we report the time evolution of the estimated parameters for ETAS model with mag-
nitude dependent Omori kernel in Figures S4 and S5.
3.3 Summary of competing models and experiment settings
In summary, we have twelve competing models: six models belong to the ETOK class
and six belong to the MDOK class. In each of these two classes, three models are
calibrated with a primary catalog magnitude threshold Mpri = 5 and three others are
calibrated with a threshold Mpri = 6. These three models can be distinguished based
on the different magnitude thresholds for the auxiliary catalog, Maux = 3, Maux = 4
and Maux = 5, used during calibration and simulations.
Each of these twelve models are shown in Table 1and are individually calibrated
on the 362 training period periods. We then compare their forecasting performance
using the M≥5 and M≥6 earthquakes under the validation settings prescribed
in Section 3.1. Only models that have been calibrated with Mpri = 5 are used to
forecast M≥5 earthquakes to avoid “extrapolated” forecasts of models trained
with only M≥6 earthquakes. All the models are used to forecast the M≥6
earthquakes as targets during each 30day long validation period. In summary, six
models are validated and scored using M≥5 earthquakes and all the twelve models
are validated and scored using M≥6 earthquakes.
4 Results and discussion
4.1 Forecasted rate map and full distribution of earthquake numbers
In this section, we illustrate how the forecasts of different models are constructed.
We do this only for a selected model and a particular testing period, as the procedure
for all other testing periods and models is the same.
Figure 2a shows the net forecasted rate of earthquakes (per km2per month) in
the time period immediately following the Tohoku earthquake (between March 12,
2011 and April 11, 2011) for the ETAS model with magnitude dependent Omori
kernel (MDOK) and auxiliary magnitude setting of Maux = 4 and primary magni-
tude setting of Mpri = 5. Figures 2b–2d show the contributions of the three type of
438 The European Physical Journal Special Topics
Fig. 2. (a) Total Rate of M≥5 earthquakes (km−2month−1) between March 12, 2011
and April 11, 2011 forecasted using ETAS model with MDOK, Maux = 4 and Mpri = 5;
The Mw 9.1 Tohuku earthquake occurred on March 11, 2011; (b)–(d) Rates of three types
of earthquakes (see Sect. S1) that are superposed to create the final forecast shown in
panel a; (b) background events; (c) aftershocks of type 1; (d) aftershocks of type 2; the
average number of each earthquake type in the final forecast is indicated in panels b–d;
(e)–(l) Probability density functions (PDF) of earthquake numbers that are forecasted by
the model in the circular geographic region of radius = 300 km around “earthquake prone”
cities of the world.
earthquakes to the net forecasted rate. The first contribution comes from the back-
ground earthquakes that are expected to occur during the testing period (Fig. 2b).
The second contribution is from the cascade of aftershocks (Aftershock Type 1) that
are expected to be triggered by the earthquakes in the training period (Fig. 2c). The
third and the final contribution comes from the cascade of aftershocks (Aftershock
Type 2) that are expected to be triggered by the background earthquakes occurring
during the testing period (Fig. 2d). In this particular testing period, the Type 1 after-
shocks have the highest contribution, with ≈264 earthquakes on average, while the
contributions of the background earthquakes and Type 2 aftershocks are relatively
minuscule. The occurrence of the Tohoku earthquake just before the testing period
is the main cause of this dominance. However, it is important to note that the rela-
tive importance of these three components depends on the time scale of the testing
period. For longer testing periods, such as on the order of a decade to a century, the
contribution of background earthquakes and especially of Type 2 aftershocks becomes
significant, if not dominating compared to the Type 1 aftershocks.
It is important to mention here that these average rate maps are just used for
the sake of illustration of the generated forecasts, as they provide a convenient rep-
resentation. In reality, each pixel on the globe is associated with a full distribution
of forecasted earthquake numbers. To illustrate this, we show in Figures 2e–2l the
probability density function (PDF) of earthquake numbers that is forecasted by the
model in circular geographic regions (with 300 km radii) around some of the earth-
quake prone cities of the world. These PDFs are obtained by counting the number
of simulations in which a certain number of earthquakes were observed and then by
dividing those by the total number of simulations that were performed. In this study,
we perform 100 000 simulations for all the models and for all testing periods. Notice
that the PDF of the forecasted number of earthquakes varies significantly from one
city to another, despite the fact that none of the competing models feature spatial
variation of the ETAS parameters. This variation can be attributed to variation in
the local history of seismicity from one place to another. Other factors that control
the shape of these distributions include the time duration of the testing period and
The Global Earthquake Forecasting System 439
Fig. 3. (a) Time evolution of cumulative information gain (CIG) of the six magnitude
dependent Omori kernel (MDOK) models when forecasting M≥5 earthquakes during
the 362 testing periods over the base model; base model is calibrated with exponentially
tapered Omori kernel (ETOK), Maux = 5 and Mpri = 5; (b) Same as panel (a) except that
the twelve competing models are used to forecast M≥6 earthquakes during the testing
periods; the solid (resp. dashed) lines track the CIG evolution for the models with Mpri = 5
(resp. Mpri = 6).
the size of the region of interest (see Fig. 1 in [32]). It is also evident that the fore-
casted distributions of earthquake numbers around these selected cities display thick
tails and cannot be approximated by a Poisson distribution. In fact, Nandan et al.
[32] showed that, if a Poissonian assumption is imposed, the ETAS model yields a
worse forecast relative to the case in which it was allowed to use the full distribution.
Therefore we use the full distribution approach proposed by Nandan et al. [32] to
evaluate the forecasting performance of the models in the following section.
4.2 Model comparison
Cumulative information gain (CIG). In Figure 3, we show the time series of cumu-
lative information gain of all competing models over the base ETAS model in the two
experiments designed to forecast M≥5 (Fig. 3a) and M≥6 (Fig. 3b) earthquakes
during the 362 testing periods. The base model has been calibrated with the expo-
nentially tapered Omori kernel (ETOK), Mpri = 5 and Maux = 5. The six models
shown in Figure 3a have been trained with either magnitude dependent Omori kernel
(MDOK) or ETOK, auxiliary magnitude threshold (Maux) of 3, 4 or 5 and primary
magnitude threshold (Mpri) of 5. In Figure 3b, we show the cumulative information
gain of these six models along with those variants that have been trained specifically
with Mpri = 6. The performance of these models has been tracked with dashed lines.
The configurations of all the twelve models are indicated in Figure 3b.
From both panels in Figure 3, we can make the following observations:
1. All other model settings being the same, the ETAS models with MDOK achieves
higher CIG over the base model than the ETAS models with ETOK. This obser-
vation is independent of the Maux and Mpri settings in both experiments, i.e. when
forecasting M≥5 earthquakes as well as when forecasting M≥6 earthquakes.
440 The European Physical Journal Special Topics
Fig. 4. (a) Pairwise mean information gain (MIG, per testing period) matrix of the six
models used to forecast M≥5 earthquakes; (i, j) element indicates the MIG of the ith
model over the jth model; (b) MIG matrix of twelve models in the experiments dealing
with forecasting M≥6 earthquakes; Black and grey labels correspond to models trained
with Mpri = 5 and Mpri = 6, respectively; (c) p-value matrix obtained from right tailed
paired t-test, testing the null hypothesis that the MIG of the ith model over the jth model,
when forecasting M≥5 earthquakes, is significantly larger than 0 against the alternative
that it is not; (d) same as panel (c) but when forecasting M≥6 earthquakes.
2. There is a slight deterioration (possibly a saturation) in the model performance
with the decreasing magnitude threshold of the auxiliary catalog. For instance,
when forecasting M≥5 earthquakes, the performance of ETAS model with
MDOK increases substantially when decreasing Maux from 5 to 4 but then slightly
diminishes when decreasing Maux further from 4 to 3. Similarly, using ETOK, the
model performance first substantially increases and then only shows a marginal
increase when decreasing Maux from 5 to 4 and then from 4 to 3 respectively.
Similar observations can be made in Figure 3b.
3. Except in one case (MDOK, Maux = 5), the models that have been specifically
trained with Mpri = 6 show either no improvement or only marginal improvement
over the models that have been trained with Mpri = 5.
4. Model performance increases either by changing Maux from 5 to 4 or by switching
the time kernel from ETOK to MDOK, or both, leading to the model with MDOK,
Maux = 4 and Mpri = 5 being the best performing model (albeit marginally) in
both experiments.
Mean information gain (MIG) and statistical significance. So far, we have com-
pared all models to a common null model and then compared their cumulative infor-
mation gain over this null model to each other. In order to assess whether one model
performs significantly better than others, we also compare the models pairwise. In
Figure 4a, we show the pairwise mean information gain (MIG) per testing period
corresponding to the six models that are used to forecast M≥5 earthquakes. In this
matrix, (i, j) element indicates the MIG of the ith model over the jth model. The
The Global Earthquake Forecasting System 441
MIGij terms are computed by averaging the information gain of the ith model over
the jth model in the 362 testing periods. Note that this matrix is antisymmetric. In
Figure 4b, we show the MIG matrix for the twelve models in the experiments deal-
ing with forecasting M≥6 earthquakes. The models that have been trained with
Mpri = 6 are labelled in grey while the ones trained with Mpri = 5 are labelled in
black.
In order to find if the MIG of one model over the other is statistically significant,
we perform right tailed paired t-test. In this test, we test the hypothesis that the MIG
of the ith model over the jth model is significantly larger than 0 against the alter-
native that it is not. Figures 4c and 4d shows the matrix of log10 (p-value) obtained
from the t-test corresponding to the MIGs shown in panel a and b, respectively. From
these MIG and p-value matrices, we can make the following observations:
1. MIG matrices echo the observations made from Figure 3.
2. All other configurations being the same, the models with MDOK almost always
perform statistically significantly (at a standard significance level of 0.05) better
than the models with ETOK when forecasting both M≥5 and M≥6 earth-
quakes.
3. We also find that, when decreasing Maux from 5 to 4, the models tend to always
perform statistically significantly better (all other settings being the same), not
only when forecasting M≥5 earthquakes but also nearly always when forecasting
M≥6 earthquakes. In the latter case, there is just one exception, i.e. when the
MDOK kernels are used with Mpri = 6 setting. However, the same trend does not
hold when decreasing the Maux from 4 to 3.
4. We also find that the models that have been trained specifically with Mpri = 6
almost never significantly outperform the models trained with Mpri = 5, with one
exception being the model with MDOK and Maux = 5.
Sensitivity to the spatial resolution. To investigate if the observations from
Figures 3and 4exhibit sensitivity to the spatial resolution, Figure S6 show the time
evolution of CIG for two different spatial resolutions (level 5 and level 7). For these
two resolutions, we also present the table of pairwise MIG and p-value in Figures S7
and S8, respectively. We find that the observations made earlier from Figures 3and 4
are robust with respect to the choice of spatial resolution.
Sensitivity to the number of simulations. An important point to consider when
evaluating and comparing the models is the number of simulations to perform. As the
models are evaluated based on the empirical distribution of earthquake numbers that
they provide in a given space-time-magnitude bin, performing too few simulations
would introduce random fluctuations in the log likelihood (Eq. (1)), thus making
the model comparisons unreliable. This is due to the fat-tailed distribution of seis-
mic rates [45–47], which implies strong sample to sample fluctuations and a slow
convergence of statistical properties [54].
On the other hand, more simulations come at higher computational costs. As
a result, it is important to optimize this trade-off. Figure S9 shows the net log-
likelihood (summed over all testing periods) that a model obtains as a function of
the number of simulations. The default number of simulations (100 000) considered
in this study to obtain all the results is indicated with a shaded vertical bar. At
100 000 simulations, all the models show a slow convergence towards their “true” log
likelihood score. Furthermore, the relative ranking of the models seem to be stable
for more than 100 000 simulations at all spatial resolutions, further justifying this
choice.
442 The European Physical Journal Special Topics
On the superiority of ETAS with the MDOK Kernel. In summary, the results point
to the significant superiority of the MDOK kernel over the ETOK when forecasting
the rate of future earthquakes using the ETAS models. However, as the ETAS model
with magnitude dependent Omori kernel (Eq. (6)) features a magnitude dependence
of both c(M) and ω(M), we cannot distinguish just from the model comparisons
presented thus far if this model’s superiority results from the magnitude dependence
of c(M) or of ω(M). To investigate this question, we define two variants of this
model: one that features an ω(M) dependence with a magnitude independent c,
and another one that features a c(M) dependence with a magnitude independent ω.
We then calibrate these models on all the 362 training periods. The time evolution
of estimated parameters is reported in Figures S10 and S11, respectively. We use
the estimated parameters to simulate 100 000 catalogs for the corresponding testing
periods. To limit the needed computational resources, we only calibrate these models
with the Maux = 5 and Mpri = 5 setting and then use these two models to create
and evaluate the forecasts for M≥5 earthquakes. We then compare (Fig. S12) the
performance of these two models to the one obtained from the ETAS model which
features both c(M) and ω(M) dependence and has been calibrated with the same
Maux and Mpri setting.
We find that, while the model with only ω(M) dependence outperforms the base
model, the model featuring only c(M) dependence systematically underperforms at
all spatial resolutions. These results indicate that the superiority of MDOK models
over ETOK models (Fig. 3) results from the ω(M) dependence rather than a c(M)
dependence. In fact, the latter dependence inhibits it from realizing its true poten-
tial in forecasting (Fig. S12). In other words, accounting for the ω(M) dependence
is the crucial improvement for forecasting, while including a c(M) dependence is
detrimental.
It is thus natural to ask why do our calibrations of the MDOK model yield a pos-
itive correlation between c(M) and mainshock magnitude (Fig. S4)? The answer to
this question potentially lies in the strong correlation between the two parameters ω
and c, as seen from Figures S2 and S3. Assuming that a positive correlation between
ωand mainshock magnitude exists, as proposed by Ouillon and Sornette [37,55] and
also apparent in Figure S10, then the strong positive correlation between ωand c
would artificially introduce a positive correlation between cand mainshock magni-
tude, masking the true underlying correlation of cand mainshock magnitude, which
may indeed be negative as revealed in the model featuring only a c(M) dependence
(Fig. S11).
In Figure S12c, we assess whether accounting for a negative correlation between
cand mainshock magnitude could lead to any information gain over the model with
the ETOK kernel. We find that the c(M) model does not provide any systematic
information gain. One possible reason for the poor performance of the c(M) models in
forecasting could lie in the short-term aftershock incompleteness [13], which is present
in both the training and testing catalogs. This rate-dependent incompleteness would
not only dampen the negative correlation between cand mainshock magnitude, but
also lead to very low information gain, as the events that would have led the c(M)
type model to be more informative are missing from the testing catalog in the first
place.
On the importance of small earthquakes in forecasting. Our results also indi-
cate that including smaller earthquakes (to an extent) in the auxiliary catalog leads
to a significant improvement in the forecast. This significant improvement can be
attributed to the improved coverage and resolution of the global seismogenic zones
as well as to the improved estimates of the parameters during calibration. However,
the improvement starts to saturate (and sometimes even deteriorates) when even
The Global Earthquake Forecasting System 443
smaller earthquakes are included in the auxiliary catalog. This could potentially be
due to the existence of a minimum triggering magnitude, M0, below which earth-
quakes do not trigger any aftershocks [56]. If we assume that the global average
value of M0is somewhere in between 3 and 4, it naturally follows that we would
also observe a saturation in model performance when reducing Maux from 4 to 3, as
the newly added earthquakes do not contribute to the triggering process. The inclu-
sion of earthquakes smaller than the actual M0may even lead to deterioration in
performance, as the calibration process implicitly assumes that all earthquakes have
the potential to trigger aftershocks, and thus leads to biased parameter estimates.
Moreover, if such a magnitude threshold exists, it could vary spatially, complicating
the analysis and interpretation.
Another possible way to explain the saturation in performance improvement is by
noting that, with a decrease of Maux from 5 to 4, there is a nearly 5.5 fold increase
in the number of earthquakes in the catalog (M≥Maux, between January 1975 and
October 2019), while when Maux is decreased from 4 to 3 the increase is only 1.5 fold,
indicating a significant number of missing events in the global earthquake catalogue
at these small magnitudes. This saturation in earthquakes numbers, i.e. the catalogue
incompleteness, could also explain the saturation in the performance of the models,
because the calibration of the ETAS models becomes intrinsically biased [57].
On the possible self-similarity of the triggering process. The insignificant differ-
ence in the performance of the models that have been trained with Mpri = 6 and
Mpri = 5 suggests the existence of self similarity in triggering processes. More con-
cretely, the models do not need to be trained specifically on Mpri = 6 to perform best
in forecasting M≥6 earthquakes, as even the models trained on Mpri = 5 can do an
equally good job. This observation could potentially be generalized to even higher
magnitude thresholds, although we have not tested it in this work.
On the exclusivity of the two model improvements. Finally, the cumulative
improvements obtained by changing the time kernel from ETOK to MDOK and
Maux from 5 to 4, indicate that these two modifications capture, to some extent,
mutually exclusive aspect of the triggering process. Furthermore, these two modifi-
cations seem to be equally important, as they separately lead to similar information
gains over the base model (see the solid orange and red curve in Fig. 3).
4.3 Consistency test
In an earthquake forecasting experiment, consistency tests play an important part, as
they allow for the direct comparison of model’s expectations with the observations,
thus serving as necessary sanity checks. One such important sanity check is the
“N-test” in which the overall number of earthquakes forecasted by a model is com-
pared against the actual number of earthquakes observed during the testing period.
Indeed, this test, along with other consistency tests such as L,Mand Stests (see [43]
for details), have been used by CSEP to measure the consistency of the models relative
to the data. It is important to note that these tests are not used to rank the models.
Not surprisingly, one of the hard-coded assumption in these tests, thus far in
CSEP, has been that the distribution of the overall number of earthquakes forecasted
by the models is Poissonian. Thus, when the numbers of earthquakes forecasted by
the models are compared against the observed numbers (especially when aftershocks
were deliberately not removed), most often the models are found to be inconsistent
(see for e.g. Fig. 9 in [63]). For instance, Werner et al. [63] have showed that, with a
444 The European Physical Journal Special Topics
Fig. 5. (a)–(f) Consistency between the PDF of forecasted numbers of earthquakes and
the actual number of earthquakes observed during the 362 testing periods, for all the
six competing models used to forecast M≥5 earthquakes; model specifications (type of
time kernel used and Maux values) are indicated as row and column headings; colors show
the log10(probability) of observing a certain number of earthquakes during a given testing
period; 95%iles of the PDF for each testing period are traced using the dashed red lines
in the figure; white crosses show the actual number of earthquakes (M≥5) during each
testing period; solid black lines show the mean number of earthquakes observed during all
the testing periods.
retrospective assumption of negative binomial distribution, the smoothed seismicity
models developed in their study “passed” the N-tests for all testing periods.
Indeed, it is prohibitively reductive to enforce the same assumption on all models
regardless of their formulation. Furthermore, the assumptions of the models should
not be modified retrospectively. Last but not least, the assumptions in a model should
be self consistent at all scales. For instance, if a model assumes that the rate of future
earthquakes is Poissonian, it cannot then be evaluated using a negative binomial
assumption for the N-test and a Poissonian assumption for estimating the informa-
tion gain. In summary, the consistency tests should be modified to allow for simul-
taneous testing of models with diverse assumptions. One possible way to do this for
the “N-tests” is to build an empirical PDF of earthquake numbers forecasted by the
models from the numerous simulations as per [32] and as done here.
Figure 5shows the consistency between the PDF of forecasted number of earth-
quakes and the actual number of earthquakes observed during the 362 testing periods,
for all the six competing models used to forecast M≥5 earthquakes. In these fig-
ures, the model type can be inferred by combining the row and the column names.
Colors used in these figures show the probability of observing a certain number of
earthquakes during a given testing period. The 95%ile of the PDF for each testing
period is traced using the dashed red lines in the figure and the white crosses show
the actual number of earthquakes (M≥5) during each testing period. Finally, the
solid black line shows the mean number of earthquakes observed during all the testing
periods. Figure S13 shows the same information as in Figure 5, but for the twelve
models used to forecast M≥6 earthquakes. Recall that the six extra models in this
case comes from the distinction introduced by the minimum threshold of the primary
catalog (Mpri = 5 or 6) used to train the models.
The Global Earthquake Forecasting System 445
We can observe from both Figures 5and S13 that the number of earthquakes
forecasted by all the models seem consistent with the average number of earthquakes
observed during all testing periods. However, when looking at individual testing peri-
ods, a lot of inconsistencies can be found. For instance, in testing periods immedi-
ately following very large earthquakes such as the Tohuku earthquake (March 11,
2011, Mw 9.1) or the Sumatra earthquake (December 26, 2004, Mw 9.3), the fore-
casted number is much lower than the observed number of earthquakes and not even
the best model (Figs. 5e and S13k) is able to account for this inconsistency. This
inconsistency can be primarily attributed to the isotropic assumption of the spatial
kernel leading to the underestimation of the productivity exponent a(see Eqs. (4)
and (6)). Note that this effect of underestimating the productivity exponent due to
the isotropic assumptions has been documented by several researchers [1,11,14,18],
who have also proposed solutions to account for anisotropy in specific case studies.
In the future, we aim to generalize those solutions for real-time applications on the
global scale. Moreover, other simplifications in the models, such as ignoring the spa-
tial variation and depth dependence of parameters, may also be at the origin of some
of these inconsistencies. The quantification of the extent to which each of these dif-
ferent factors contribute to inconsistencies will be undertaken in future studies. We
also observe from Figure 5, that there are extended periods (such as between 1997
and 2005) in which the observed numbers of earthquakes are systematically smaller
than the forecasted numbers, possibly pointing towards a time variation of the trig-
gering parameters and (or) background rate. Such inconsistencies are less evident for
M≥6 earthquakes (Fig. S13), possibly because of their sparse numbers during a
given testing period, making it easier for models to pass the N-tests.
4.4 Real time application for short-term forecasts and predictive skill
assessment
The design of the forecasting experiment has been tailored to a global application
for short term (up to 30 days) and regional (up to 300 km) earthquake forecasts.
Accordingly, we have operationalized the best performing ETAS model (with MDOK,
Maux = 4 and Mpri = 5) developed in this study via the RichterX platform available
at www.richterX.com [25]. On this website, the public can query the real-time model
probabilities for earthquakes with M≥5 anywhere on the globe. The forecasts are
provided in real-time in the sense that (1) global simulations are updated every hour
as new earthquakes (M≥4) are entered in the ANSS catalog and (2) the probabilities
depend on the actual time at which the user is requesting the forecast. A forecast
request is performed by centering a circle at any location on the globe. The user then
has the option to adjust the circle radius (30–300 km), time duration (1–30 days),
minimum magnitude (M5+ toM9+) and the minimum number of earthquakes. These
parameters are then used to query the database of real-time prospective simulations.
The number of simulations that feature events satisfying the forecast criteria are used
to construct an empirical PDF that defines the reported probability.
Michael et al. [27] showed that a statement regarding the probability of Nor
more earthquakes within a specific space-time-magnitude window helps the media to
accurately report probabilistic earthquake forecast. Therefore, we see the RichterX
platform as an important step in improving public earthquake awareness and pre-
paredness. It is important to note that the RichterX platform does not distinguish
between an aftershock or a mainshock to assess the future probability of an earth-
quake. Furthermore, it allows the users to interact with the probabilities, by adjusting
the forecast parameters online, facilitating an intuitive understanding of the under-
lying hazard. In these two regards, the RichterX platform differs from the efforts of
446 The European Physical Journal Special Topics
the USGS, which started to publicly release aftershock forecasts for all events M≥5
throughout the United States in September 2019 as a table of the probability of one
or more earthquakes for the next day, week, month, and year for M≥3, ≥5, ≥6,
and ≥7, respectively [27].
The availability of such a publicly accessible, real-time global earthquake fore-
casting model allows for new testing applications. Namely, it can be used as a refer-
ence benchmark to evaluate other short-term forecasts or deterministic predictions.
Since the model probabilities are based on synthetic event sets, the forecasts are
independent of prescribed grids and are not hindered by assumptions about distri-
butions. Building on this feature, we introduce the RichterX platform as a global
earthquake prediction contest, where participants can challenge the reference model
by issuing deterministic to-occur or not-to-occur predictions anywhere on the globe.
In the accompanying paper, we introduce the platform and demonstrate metrics
that allow for consistent ranking of competing models [25]. In this way, we aim to
address the deficiencies found in the current CSEP testing methodologies, allow-
ing for the inclusion of model types that were previously deemed incompatible and
encourage a broader participation. We do not intend to keep the platform limited to
“seismological” experts, but rather make it accessible to experts from other fields
as well as “amateur” scientists. In fact, anyone with an idea, intuition or a model
is invited to challenge the forecast developed in this study by submitting testable
predictions.
5 Conclusion and outlook
Upon rigorous testing of the two ETAS models with two different time kernels (one
with exponentially tapered Omori kernel and another with magnitude dependent
Omori kernel), with three different training settings for the auxiliary catalog’s mag-
nitude cutoff (3, 4 or 5) and two different training settings for the primary catalog’s
magnitude cutoff (5 or 6), in 362 pseudo prospective global experiments designed to
forecast M≥5 and M≥6 earthquakes, we can derive the following conclusions:
1. ETAS models with Omori kernels whose parameters explicitly depend on the
magnitude of the mainshock perform significantly better relative to the ETAS
models that ignores such dependencies. The superiority of ETAS models with
magnitude dependent Omori kernel only results from the incorporation of the
magnitude dependence of the Omori exponent, thus adding further support to
the multifractal stress activation model proposed by Ouillon and Sornette [37,55].
2. While inclusion of more data in the auxiliary catalog by lowering the minimum
magnitude cutoff from 5 to 4 leads to significant improvement in the forecast-
ing performance, the performance saturates (and even deteriorates) when even
smaller magnitudes (M≥3) are included in the auxiliary catalog. This counter-
intuitive observation could have its origin in biases resulting from the incom-
pleteness of the catalogue at these small magnitudes. Alternatively or together,
this may also provide an observational evidence for the theoretical concept of a
minimum magnitude of earthquakes that can trigger aftershocks [56].
3. ETAS models do not need to be trained specifically with M≥6 earthquakes
in the primary catalog to have outstanding forecasting performance above this
magnitude threshold. Models trained using a lower magnitude threshold (M≥5)
can do an equally good job. This observation could be generalized to even higher
magnitude thresholds possibly pointing to the self-similarity of the triggering
process.
4. The number of earthquakes forecasted by the models is not always consistent with
the observed number of earthquakes during the testing period. This is especially
The Global Earthquake Forecasting System 447
true in experiments designed for forecasting M≥5 earthquakes. These inconsis-
tencies possibly arise from the simplifications, such as using an isotropic spatial
kernel, as well as spatially homogeneous, depth independent and time invariant
ETAS parameters, hardwired in the models presented in this study.
In order to obtain a fair and reliable comparison of the model performance, we
have corrected some of the obvious defects of the past model testing experiments.
These corrections include:
1. using equal sized mesh to ensure homogeneity of testing scores over the globe.
2. allowing the models the flexibility to specify the forecasts in accordance with their
assumptions.
3. no declustering of the testing catalogs.
The models developed and tested in this work constitute a first imperfect attempt
at developing global models that are capable of making short-term operational fore-
casts. Several simplifications have been made, especially in terms of diversity of the
models developed and tested. Some of the obvious simplifications include (a) consid-
ering only ETAS type models, (b) assuming the parameters of the ETAS models to
be spatially homogeneous and time invariant, (c) ignoring the depth dependence of
parameters, (d) ignoring errors in the data, (e) assuming isotropic spatial kernels and
so on. Nevertheless, by introducing fair and reliable testing schemes, in which mod-
ellers have the flexibility to adhere to their best judgement consistently, this study
can serve as a framework for further model developments. Indeed, by operationaliz-
ing the best performing model as a benchmark for the RichterX prediction contest,
we enable fellow modellers to use our results as a stepping stone for improving their
models. This also constitutes a continuing process of peer-review, whereby anyone
who finds the forecast probabilities too low or high can issue a to-occur or a not-to-
occur prediction, providing us with important prospective feedback to improve our
model.
On more general grounds, forecasting models can be split into two broad cate-
gories, namely statistical models (such as ETAS) and physical ones (using quantities
such as static or dynamic stress transfer). The latter require the knowledge of many
additional parameters, including the spatial extent and orientation of each rupture, as
well as a detailed description of the slip over the failure planes. Nandan [29] showed
that our ability to forecast aftershock sequences using a stress-transfer approach
increased if one took into account the triggering probabilities provided by an inde-
pendent ETAS declustering process (the stress-based forecast being logically more
appropriate for direct aftershocks). This, in return, suggests that a better knowledge
of the space-time variations of the stress field may help to improve the forecasts of
ETAS-like models. Nevertheless, the difficulty of such a forecasting framework is that
the details of rupture must be known in real time for all past events, and forecasted
as well for all future events. As this is clearly out of scope given our very limited
knowledge of the deterministic structure of fault networks in the Earth crust, the
MSA model thus offers the best opportunity to encode some of the universal prop-
erties of the mechanics of brittle media within a purely statistical framework. That
certainly explains the superiority of this model for forecasting purposes, even in its
simplified, linearized form presented in this paper.
Publisher’s Note The EPJ Publishers remain neutral with regard to jurisdictional claims
in published maps and institutional affiliations.
448 The European Physical Journal Special Topics
References
1. C. Bach, S. Hainzl, J. Geophys. Res.: Solid Earth 117, B04312 (2012)
2. P. Bird, D.D. Jackson, Y.Y. Kagan, C. Kreemer, R.S. Stein, Bull. Seismol. Soc. Am.
105, 2538 (2015)
3. A. Cao, S.S. Gao, Geophys. Res. Lett. 29, 48 (2002)
4. C. Cattania, S. Hainzl, L. Wang, F. Roth, B. Enescu, J. Geophys. Res.: Solid Earth
119, 7846 (2014)
5. C. Cattania, M.J. Werner, W. Marzocchi, S. Hainzl, D. Rhoades, M. Gerstenberger,
M. Liukis, W. Savran, A. Christophersen, A. Helmstetter, et al. Seismol. Res. Lett. 89,
1238 (2018)
6. A. Chu, F.P. Schoenberg, P. Bird, D.D. Jackson, Y.Y. Kagan, Bull. Seismol. Soc. Am.
101, 2323 (2011)
7. J. Davidsen, C. Gu, M. Baiesi, Geophys. J. Int. 201, 965 (2015)
8. J. Dieterich, J. Geophys. Res.: Solid Earth 99, 2601 (1994)
9. D.A. Eberhard, J.D. Zechar, S. Wiemer, Geophys. J. Int. 190, 1579 (2012)
10. M.C. Gerstenberger, S. Wiemer, L.M. Jones, P.A. Reasenberg, Nature 435, 328 (2005)
11. Y. Guo, J. Zhuang, S. Zhou, J. Geophys. Res.: Solid Earth 120, 3309 (2015)
12. S. Hainzl, J. Geophys. Res.: Solid Earth 121, 6499 (2016).
13. S. Hainzl, Seismol. Res. Lett. 87, 337 (2016)
14. S. Hainzl, A. Christophersen, B. Enescu, Bull. Seismol. Soc. Am. 98, 2066 (2008)
15. A. Helmstetter, D. Sornette, J. Geophys. Res.: Solid Earth 107, ESE–10 (2002)
16. A. Helmstetter, D. Sornette, Geophys. Res. Lett. 30, 1576 (2003)
17. A. Helmstetter, M.J. Werner, Bull. Seismol. Soc. Am. 104, 809 (2014)
18. A. Helmstetter, Y.Y. Kagan, D.D. Jackson, Bull. Seismol. Soc. Am. 96, 90 (2006)
19. S. Hiemer, Y. Kamer, Seismol. Res. Lett. 87, 327 (2016)
20. S. Hiemer, J. Woessner, R. Basili, L. Danciu, D. Giardini, S. Wiemer, Geophys. J. Int.
198, 1159 (2014)
21. T.H. Jordan, Seismol. Res. Lett. 77, 3 (2006)
22. Y. Kagan and D. Jackson, Pure Appl. Geophys. 167, 709 (2010)
23. Y.Y. Kagan, Geophys. J. Int. 110, 305 (1992)
24. Y.Y. Kagan, D.D. Jackson, Geophys. J. Int. 184, 759 (2011)
25. Y. Kamer, S. Nandan, G. Ouillon, S. Hiemer, D. Sornette, Eur. Phys. J. Special Topics
230, 451 (2021)
26. V.G. Kossobokov, Nat. Hazards 69, 1155 (2013)
27. A.J. Michael, S.K. McBride, J.L. Hardebeck, M. Barall, E. Martinez, M.T. Page,
N. van der Elst, E.H. Field, K.R. Milner, A.M. Wein, Seismol. Res. Lett. 91, 153 (2020)
28. A. Mignan, J. Woessner, Estimating the magnitude of completeness for earthquake
catalogs. Community Online Resource for Statistical Seismicity Analysis (2012),
doi:10.5078/corssa-00180805
29. S. Nandan, Towards a physics based epidemic type aftershock sequence model, Ph.D.
thesis, ETH Zurich, 2017
30. S. Nandan, G. Ouillon, S. Wiemer, D. Sornette, J. Geophys. Res.: Solid Earth 122,
5118 (2017)
31. S. Nandan, G. Ouillon, D. Sornette, J. Geophys. Res.: Solid Earth 124, 2762 (2019)
32. S. Nandan, G. Ouillon, D. Sornette, S. Wiemer, Seismol. Res. Lett. 90, 1650 (2019)
33. S. Nandan, G. Ouillon, D. Sornette, S. Wiemer, J. Geophys. Res.: Solid Earth 124,
8404 (2019)
34. C. Narteau, P. Shebalin, M. Holschneider, Geophys. Res. Lett. 32, L22312 (2005)
35. Y. Ogata, Earth Planets Space 63, 6 (2011)
36. Y. Ogata, K. Katsura, G. Falcone, K. Nanjo, J. Zhuang, Bull. Seismol. Soc. Am. 103,
1692 (2013)
37. G. Ouillon, D. Sornette, J. Geophys. Res.: Solid Earth 110, B04306 (2005)
38. G. Ouillon, D. Sornette, E. Ribeiro, Geophys. J. Int. 178, 215 (2009)
39. M.T. Page, N. Van Der Elst, J. Hardebeck, K. Felzer, A.J. Michael, Bull. Seismol. Soc.
Am. 106, 2290 (2016)
The Global Earthquake Forecasting System 449
40. M. Pesaresi, D. Ehrlich, T. Kemper, A. Siragusa, A.J. Florczyk, S. Freire, C. Corbane,
Atlas of the Human Planet 2017: Global Exposure to Natural Hazards, EUR 28556 EN
(2017)
41. P. Reasenberg, J. Geophys. Res.: Solid Earth 90, 5479 (1985)
42. T. Reverso, S. Steacy, D. Marsan, J. Geophys. Res.: Solid Earth 123, 9750 (2018)
43. D.A. Rhoades, D. Schorlemmer, M.C. Gerstenberger, A. Christophersen, J.D. Zechar,
M. Imoto, Acta Geophys. 59, 728 (2011)
44. A. Saichev, D. Sornette, Eur. Phys. J. B 51, 443 (2006)
45. A. Saichev, D. Sornette, Eur. Phys. J. B 49, 377 (2006)
46. A. Saichev and D. Sornette, Tectonophysics 431, 7 (2007)
47. A. Saichev, A. Helmstetter, D. Sornette, Pure Appl. Geophys. 162, 1113 (2005)
48. F.P. Schoenberg, A. Chu, A. Veen, J. Geophys. Res.: Solid Earth 115, B04309 (2010)
49. C. Scholz, Bull. Seismol. Soc. Am. 58, 1117 (1968)
50. D. Schorlemmer, M. Gerstenberger, Seismol. Res. Lett. 78, 30 (2007)
51. D. Schorlemmer, M.J. Werner, W. Marzocchi, T.H. Jordan, Y. Ogata, D.D. Jackson,
S. Mak, D.A. Rhoades, M.C. Gerstenberger, N. Hirata, et al. Seismol. Res. Lett. 89,
1305 (2018)
52. S. Seif, A. Mignan, J.D. Zechar, M.J. Werner, S. Wiemer, J. Geophys. Res.: Solid Earth
122, 449 (2017)
53. R. Shcherbakov, D.L. Turcotte, J.B. Rundle, Geophys. Res. Lett. 31, L11613 (2004)
54. D. Sornette, Critical phenomena in natural sciences (chaos, fractals, self-organization
and disorder: Concepts and tools), Springer Series in Synergetics (Springer, Heidelberg,
2006)
55. D. Sornette, G. Ouillon, Phys. Rev. Lett. 94, 038501 (2005)
56. D. Sornette and M.J. Werner, J. Geophys. Res.: Solid Earth 110, B08304 (2005)
57. D. Sornette and M.J. Werner, J. Geophys. Res.: Solid Earth 110, B09303 (2005)
58. S. Steacy, M. Gerstenberger, C. Williams, D. Rhoades, A. Christophersen, Geophys. J.
Int. 196, 918 (2014)
59. S. Tinti, F. Mulargia, Bull. Seismol. Soc. Am. 77, 2125 (1987)
60. C.-Y. Tsai, G. Ouillon, D. Sornette, Bull. Seismol. Soc. Am., 102, 2128 (2012)
61. A. Veen, F.P. Schoenberg, J. Am. Stat. Assoc. 103, 614 (2008)
62. Q. Wang, D.D. Jackson, J. Zhuang, Geophys. Res. Lett. 37, L21307 (2010)
63. M.J. Werner, A. Helmstetter, D.D. Jackson, Y.Y. Kagan, Bull. Seismol. Soc. Am., 101,
1630 (2011)
64. J.D. Zechar, T.H. Jordan, Ann. Geophys. 53, 99 (2010)
65. J.D. Zechar, D. Schorlemmer, M.J. Werner, M.C. Gerstenberger, D.A. Rhoades, T.H.
Jordan, Bull. Seismol. Soc. Am. 103, 787 (2013)
66. L. Zhang, M.J. Werner, K. Goda, Bull. Seismol. Soc. Am. 110, 191 (2020)
67. J. Zhuang, Y. Ogata, D. Vere-Jones, J. Am. Stat. Assoc. 97, 369 (2002)
68. J. Zhuang, Y. Ogata, D. Vere-Jones, J. Geophys. Res.: Solid Earth 109, B05301 (2004)
