The Mock LISA Data Challenges: from Challenge 3 to Challenge 4

Abstract

The Mock LISA Data Challenges are a program to demonstrate LISA data-analysis capabilities and to encourage their development. Each round of challenges consists of one or more datasets containing simulated instrument noise and gravitational waves from sources of undisclosed parameters. Participants analyze the datasets and report best-fit solutions for the source parameters. Here we present the results of the third challenge, issued in Apr 2008, which demonstrated the positive recovery of signals from chirping Galactic binaries, from spinning supermassive--black-hole binaries (with optimal SNRs between ~ 10 and 2000), from simultaneous extreme-mass-ratio inspirals (SNRs of 10-50), from cosmic-string-cusp bursts (SNRs of 10-100), and from a relatively loud isotropic background with Omega_gw(f) ~ 10^-11, slightly below the LISA instrument noise. Comment: 12 pages, 2 figures, proceedings of the 8th Edoardo Amaldi Conference on Gravitational Waves, New York, June 21-26, 2009

Full text

Page 1

IOP PUBLISHING
CLASSICAL AND QUANTUM GRAVITY
Class. Quantum Grav. 25 (2008) 184026 (20pp)
doi:10.1088/0264-9381/25/18/184026
The Mock LISA Data Challenges: from Challenge 1B
to Challenge 3
Stanislav Babak1, John G Baker2, Matthew J Benacquista3,
Neil J Cornish4, Jeff Crowder5, Shane L Larson6, Eric Plagnol7,
Edward K Porter1, Michele Vallisneri5,8, Alberto Vecchio9(The Mock
LISA Data Challenge Task Force), Keith Arnaud2, Leor Barack10,
Arkadiusz Błaut11, Curt Cutler5,8, Stephen Fairhurst12,
Jonathan Gair1,13, Xuefei Gong14, Ian Harry12, Deepak Khurana15,
Andrzej Kr´ olak16, Ilya Mandel8,17, Reinhard Prix18, B S Sathyaprakash12,
Pavlin Savov8, Yu Shang15, Miquel Trias19, John Veitch9, Yan Wang20,
Linqing Wen1,8,21and John T Whelan1(the Challenge-1B participants)
1Max-Planck-Institut f¨ ur Gravitationsphysik (Albert-Einstein-Institut), Am M¨ uhlenberg 1,
D-14476 Golm bei Potsdam, Germany
2Gravitational Astrophysics Laboratory, NASA Goddard Space Flight Center, 8800 Greenbelt
Rd., Greenbelt, MD 20771, USA
3Center for Gravitational Wave Astronomy, University of Texas at Brownsville, Brownsville,
TX 78520, USA
4Department of Physics, Montana State University, Bozeman, MT 59717, USA
5Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109, USA
6Department of Physics, Weber State University, 2508 University Circle, Ogden, UT 84408, USA
7APC, UMR 7164, University Paris 7 Denis Diderot, 10, rue Alice Domon et Leonie Duquet,
75025 Paris Cedex 13, France
8Theoretical Astrophysics, California Institute of Technology, Pasadena, CA 91125, USA
9School of Physics and Astronomy, University of Birmingham, Edgbaston, Birmingham B15
2TT, UK
10School of Mathematics, University of Southampton, Southampton, SO17 1BJ, UK
11Institute of Theoretical Physics, University of Wrocław, Wrocław, Poland
12School of Physics and Astronomy, Cardiff University, 5, The Parade, Cardiff, CF24 3YB, UK
13Institute of Astronomy, University of Cambridge, Madingley Rd., Cambridge, CB3 0HA, UK
14Institute of Applied Mathematics, Academy of Mathematics and System Science, Chinese
Academy of Sciences, 55 Zhongguancun Donglu, Beijing 100080, People’s Republic of China
15Indian Institute of Technology, Kharagpur, India
16Institute of Mathematics, Polish Academy of Sciences, Warsaw, Poland
17Department of Physics and Astronomy, Northwestern University, Evanston, IL, USA
18Max-Planck-Institut f¨ ur Gravitationsphysik (Albert-Einstein-Institut), D-30167 Hannover,
Germany
19Department de F´ ısica, Universitat de les Illes Balears, Cra. Valldemossa km 7.5, E-07122
Palma de Mallorca, Spain
20Department of Astronomy, Nanjing University, 22 Hankou Road, Nanjing 210093, People’s
Republic of China
21School of Physics, M013, University of Western Australia, 35 Stirling Highway, Crawley,
WA 6009, Australia
E-mail: Michele.Vallisneri@jpl.nasa.gov
Received 21 April 2008, in final form 11 June 2008
Published 2 September 2008
Online at stacks.iop.org/CQG/25/184026
0264-9381/08/184026+20$30.00© 2008 IOP Publishing Ltd Printed in the UK1

Page 2

Class. Quantum Grav. 25 (2008) 184026S Babak et al
Abstract
The Mock LISA Data Challenges are a programme to demonstrate and
encourage the development of LISA data-analysis capabilities, tools and
techniques. At the time of this workshop, three rounds of challenges had
been completed, and the next was about to start. In this paper we provide
a critical analysis of the entries to the latest completed round, Challenge 1B.
The entries confirm the consolidation of a range of data-analysis techniques for
galactic and massive-black-hole binaries, and they include the first convincing
examples of detection and parameter estimation of extreme-mass-ratio inspiral
sources. In this paper we also introduce the next round, Challenge 3. Its data
sets feature more realistic waveform models (e.g., galactic binaries may now
chirp, and massive-black-hole binaries may precess due to spin interactions),
as well as new source classes (bursts from cosmic strings, isotropic stochastic
backgrounds) and more complicated nonsymmetric instrument noise.
PACS numbers: 04.80.Nn, 95.55.Ym
1. Introduction
The Laser Interferometer Space Antenna (LISA), an ESA–NASA mission to survey the
gravitational-wave (GW) sky at frequencies between 10−5and 10−1Hz, will record
gravitational radiation from millions of sources, most of them in our galaxy, but many
populating the low-to-high-redshift universe [1].
in both the time and frequency domains (to the point of creating confusion noise at some
frequencies) poses a number of interesting new challenges for GW data analysis, whose
solution is essential if we are to draw the greatest possible science payoff from such a bold
and innovative observatory.
At the end of 2005, the LISA International Science Team (LIST) initiated a programme of
Mock LISA Data Challenges (MLDCs) with the goal of understanding at the conceptual
and quantitative level the peculiarities of LISA data analysis, of demonstrating LISA’s
observational capabilities, and kickstarting the development of data-analysis algorithms,
pipelines and infrastructural elements. An MLDC Task Force, chartered by the LIST,
periodically issues challenge data sets containing GW signals from sources of undisclosed
parameters, embedded in synthetic LISA noise; challenge participants have a few months to
analyze the data and submit detection candidates, which are then compared with the sources
originally injected in the data sets. (Training data sets with public source parameters are also
provided to help participants tune and troubleshoot their codes.)
Three rounds of MLDCs had been completed at the time of this workshop, each spanning
approximately 6 months. Challenge 1 [2, 3] was focused on establishing basic techniques
to observe GWs from compact galactic binaries, intrinsically monochromatic, isolated or
moderately interfering; as well as from the inspiral phase of bright, isolated, nonspinning
massive-black-hole (MBH) binaries. Challenge 2 [4, 5] featured three considerably more
complex data-analysis problems: a data set containing GW signals from approximately
26 million galactic binaries (again monochromatic) drawn from a randomized population-
synthesis catalog; a data set (the ‘whole enchilada’) with a similar galactic-binary population,
plus GW signals from an unknown number (between 4 and 6) of nonspinning-MBH binary
and from five extreme-mass-ratio inspirals (EMRI); and five more data sets with single-EMRI
signals.
Such a variety of signals, overlapping
2

Page 3

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
The very steep increase in complexity introduced over a short time-scale with Challenge
2 and the need to consolidate analysis techniques (especially so for EMRIs) before moving to
evenmoretaxingchallengesmotivatedtheorganizationofChallenge1B,arepeatofChallenge
1 with the addition of single-EMRI data sets. Challenge-1B data sets were distributed in the
latesummer2007,withadeadlineofDecember2007forentries. Tencollaborationssubmitted
solutions. Highlights from this round include the range of techniques used, the participation
of a number of new groups that successfully recovered signals from galactic binaries and
MBH binaries, and the first convincing demonstration of EMRI detection and parameter
estimation. Section 2 provides a brief summary of the entries; additional details about the
work of individual collaborations are given elsewhere in this volume.
As we write (April 2008), Challenge 3 data sets have just been released, with entries due
at the beginning of December 2008. Challenge 3 represents a definite step in the direction of
more realistic source models (such as chirping galactic binaries and spinning-MBH binaries)
and of new source classes (such as short-lived bursts and stochastic backgrounds). Section 3
describes the Challenge 3 data sets and waveform models in detail.
2. Report on Challenge 1B
Challenge1BfocusedonthreeclassesofGWsources, eachtackledinaseparatesubchallenge:
monochromatic galactic binaries, MBH binaries and EMRIs. The galactic-binary data sets
(1B.1.1a–c and 1B.1.2–1.5) and the MBH-binary data sets (1B.2.1–2.2) had a duration of
approximately 1 year (31 457 280 s, sampled at intervals of 15 s), while the EMRI data sets
(1B.3.1–3.5) were twice as long (with the same sampling time). The challenge solutions
submitted by the participants were assessed with the simple criteria adopted in previous
rounds [3, 5]. Detector-response data were generated with the best-fit source parameters?λsub
submitted by the participants, using the same code previously employed to build the challenge
data sets. These data were then compared to the detector response to the true waveforms,
using as a figure of merit the recovered SNR
(Atrue|Asub) + (Etrue|Esub)
√(Asub|Asub) + (Esub|Esub),
whereAandEdenotetimeseriesforthenoise-orthogonalTDIobservables(2X−Y−Z)/3and
(Z − Y)/√3 [6] and (·|·) denotes the usual signal product weighted by instrument noise. We
also quote the correlation C = SNR/SNRopt, where SNRopt=√(Atrue|Atrue) + (Etrue|Etrue)
is the optimal SNR. For a perfect detection C = 1, but fluctuations ∼ 1/SNRoptare expected
because of instrument noise. When we examine parameter errors, these are defined simply
as ?λi= λi
errors ?λi/λi=?λi
at www.tapir.caltech.edu/˜mldc/results1B/results.html.
SNR(?λsub) =
(1)
sub− λi
true; in some cases it makes sense to consider the fractional parameter
sub− λi
entries submitted by participants; the technical notes accompanying the entries can be found
true
??λi
true. In the remainder of this section we briefly discuss the
2.1. Galactic binaries: Challenges 1B.1.X
Data sets 1B.1.1a–c and 1B.1.2–1.5 contained GW signals from monochromatic galactic
binaries, in a variety of parameter ranges and source combinations. Seven parameters are
required to fully characterize each such source: the amplitude A, the (constant) frequency f,
the ecliptic latitude and longitude β and λ, the inclination and polarization angles ι and ψ,
and the initial phase φ0. Entries were submitted by five groups: GSFC (scientists at Goddard
Space Flight Center), IMPAN (the Institute of Mathematics of the Polish Academy of Science
3

Page 4

Class. Quantum Grav. 25 (2008) 184026S Babak et al
Table 1. Correlations for single-galactic-binary challenges 1B.1.1a–c. Asterisks denote entries
corrected by maximizing the F-statistic and using the resulting extrinsic parameters; these
corrections are not reported where the frequency is well off, and the F-statistic is merely fitting
noise.
1B.1.1a
SNRopt= 13.819
0.108 → 0.984∗
0.992
0.988
0.952 → 0.996∗
0.992
1B.1.1b
SNRopt= 24.629
0.922 → 0.996∗
0.807 → 0.814∗
0.981 → 0.997∗
0.906 → 0.994∗
0.996
1B.1.1c
SNRopt= 15.237
−0.190 → 0.989∗
−0.138
0.924 → 0.946∗
0.033
Group
AEI
GSFC
IMPAN
MCMNJU
UIBBham
andtheInstituteofTheoreticalPhysicsattheUniversityofWrocław), AEI(theAlbertEinstein
Institute in Golm, Germany), MCMNJU (the Institute of Applied Mathematics of the Chinese
Academy of Sciences and the Department of Astronomy of Nanjing University), UIBBham
(the University of the Balearic Islands and the University of Birmingham). However, all
groups except AEI concentrated only on a subset of the challenges.
Participants employed a fair range of techniques, in the same broad class as adopted for
similar challenges in the past [3, 5]; however, new implementations and different technical
solutions were pursued. GSFC used the X-Ray Spectral Fitting Package (XSPEC) [7] to
fit templates to energy spectra. The package includes a Levenberg–Marquardt optimization
algorithm, which was used to obtain an initial guess for the source parameters. A Markov-
chain Monte Carlo (MCMC) routine, also available in XSPEC, was then used to converge to
the best-fit source parameters. IMPAN set up a grid-based matched-filtering search with an
optimized placement of templates on a hypercubic lattice. The F-statistic [8] was used to
reduce the search space from 7 to 3 parameters. A similar technique was adopted by AEI [9],
in conjunction with a rigid-adiabatic model of detector response. MCMNJU used a genetic
algorithm that optimized the F-statistic; UIBBham implemented a MCMC search described
in more detail in [10].
Each of data sets 1B.1.1a–c contained a GW signal from a single monochromatic binary
(differing by frequency), with SNR ≈ 13–25. Table 1 lists the correlations recovered by
each collaboration. Some of the entries included close matches for the intrinsic parameters
(f, θ and φ), but not for the remaining (extrinsic) parameters: this was due to the (relative)
inaccuracy of the LISA response models used by the participants, or by mismatch between
their definitions of the extrinsic parameters and the MLDC’s. In these cases we recomputed
C by maximizing the F-statistic [8] for the intrinsic parameters provided; the resulting Cs
are denoted by asterisks, and show that the intrinsic parameters were indeed recovered well
enough to ensure solid detections. Table 2 lists parameter errors, and figure 1 shows where in
the sky each collaboration placed the single binary of data set 1B.1.1a, compared to its true
position, and to the Fisher-matrix 1-σ error contour.
Data set 1B.1.2 contained GW signals from 25 ‘verification’ binaries of known (i.e.,
disclosed) frequency and sky location. Five of them were taken from the list of observed
binaries on Gijs Nelemans’ wiki [11], while the remaining 20 were placed randomly in
the galaxy, varying their frequencies over a representative range. Table 3 lists the global
correlations (computed for the combined signals of all reported and true binaries) recovered
by the three groups that participated in this challenge. Just as it happened for Challenges
1B.1.1a–c, problems in assigning extrinsic parameters reduced the correlations. (Since the
intrinsic parameters were provided to the participants, we did not perform F-statistic-based
4

Page 5

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Figure 1. Sky positions reported by participants in Challenge 1B.1.1a, compared to the true
location of the source. All the sky positions fall within the Fisher-matrix 1-σ contour, shown
as a dotted line. (The spread of the reported sky positions is due to both the difference in noise
realizations between the Synthetic LISA and LISA Simulator versions of data set 1B.1.1a, and the
systematic errors of the searches.)
Table 2. Parameter errors for Challenges 1B.1.1a–c. All angles are expressed in radians.
Group
?β ?λ ?f (nHz)
?ψ ?ι?ϕ?A (10−23)
Challenge 1B.1.1a (ftrue= 1.060 mHz)
−0.032 −0.120
−0.004 −0.071
−0.031
MCMNJU −0.017 −0.042
UIBBham0.005 −0.079
Challenge 1B.1.1b (ftrue= 2.904 mHz)
AEI
−0.056 −0.0090
GSFC
−0.462
IMPAN 0.020 0.0007
MCMNJU −0.067 −0.0063
UIBBham
−0.044 −0.0082
Challenge 1B.1.1c (ftrue= 9.943 mHz)
AEI
−0.026
GSFC
−0.452 −1.48
IMPAN
−0.016
MCMNJU −0.555 −0.3680
AEI
GSFC
IMPAN
−2.43
−1.81
2.13
−0.53
−1.51
0.217 −0.454
0.708
0.454
0.662
0.708
1.17
1.33
1.22
1.20
1.25
2.34
0.65
0.252
0.212 −1.06
0.426 −1.57
0.173 −1.32
0.018
0.95
−1.050
2.560
0.333
−0.732 −0.064
−0.636
0.283
0.182
0.339 −0.60
1.63 −0.066
0.52 −0.024 0.0606
−30.9
0.85
2.07
1.78
0.713
0.84 −0.223
1.13 −0.029 0.043
0.00531.84
−0.499 −1.120
1.820 −0.471 −0.66 −0.695
−1.510 −0.197
−1.590 −0.250 −0.94 −0.532
3.020.124
140
0.0248 3.722.68 0.478
359
adjustments, which would amount to solving the entire problem. As a result, the low C values
of table 3 may be symptomatic only of extrinsic-parameter systematics.)
Data set 1B.1.3 contained GW signals from 20 unknown binaries distributed across the
LISA band, well separated in frequencies. Unfortunately, a bug in the random generation of
sourceparameterscausedallSNRstobetoosmallfordetection(allwerebelow1). Happily,no
5

Page 6

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Table 3. Correlations, number of recovered sources, and number of false positives for Challenges
1B.1.2, 1B.1.4 and 1B.1.5. Correlations computed after correcting extrinsic parameters using the
F-statistic are denoted with asterisks.
Group
C
# recovered
1B.1.2 (SNRopt= 634.918,25 sources)
−0.822
0.006
MCMNJU 0.267
AEI
GSFC
25
25
25
1B.1.4 (SNRopt= 340.233,51 sources)
0.774 → 0.966∗
0.003 → 0.282∗
1B.1.5 (SNRopt= 273.206,44 sources)
0.453 → 0.929∗
AEI
GSFC
13 (2 false positives)
6 (1 false positives)
AEI
3
participating group reported apositivedetection, consistentwiththecorrectbehavior expected
of search algorithms.
Challenge 1B.1.4 was meant to test search algorithms in the presence of mild source
confusion. Fifty-one sources were spread across a band of 15 μHz beginning at 3 mHz, with
an average density of 0.108 sources per frequency bin. By contrast, Challenge 1B.1.5 tested
algorithms in the presence of a higher level of source confusion, comparable to that expected
from our galaxy. Forty-four sources were spread across a band of 3 μHz centered at 3 mHz,
withanaveragedensityof0.465sourcesperfrequencybin. Table3liststheglobalcorrelations
andthenumberofsourcesrecoveredbytheparticipatinggroups, aswellasthenumberoffalse
positives, defined here as reported sources farther than one frequency bin (1/year) from any
true source, or with F-statistic-adjusted correlation less than 0.7 with all true sources within a
frequency bin. The top panel of figure 2 shows the combined A and E spectral amplitude for
the GW signals in data set 1B.1.4, together with the residual after subtracting the signal model
submitted by AEI. The bottom panel shows the same subtraction after extrinsic parameters
have been recomputed for this entry by maximizing the F-statistic.
Altogether, it must be said that Challenge 2 provided a more forceful demonstration of
LISA’s science objectives for galactic binaries [5]; but Challenge 1B was still very useful
for new groups to start implementing search methods, and for established groups to continue
tuning them. The extrinsic-parameter reporting errors seen here are easy to commit, because
these parameters are very sensitive to the modeling of the LISA response, and because
their definitions are somewhat conventional; but these errors have little bearing on detection
confidence. To avoid such problems in the future, we plan to provide a web tool to check the
recovered SNR against the challenge data sets using the fiducial MLDC waveform-generation
code.
2.2. Massive black hole binary systems: Challenges 1B.2.X
Each of data sets 1B.2.1 and 1B.2.2 contained a loud GW signal from a single MBH binary
embeddedininstrumentnoise. Gravitationalradiationwasmodeledastherestrictedwaveform
forspinlesspointmassesmovingonanadabaticsequenceofcircularorbits,evolvingaccording
to 2PN energy-balance equations [4, section 4.4]. (See also section 3.2 for the new waveform
features being introduced for Challenge 3.) Nine parameters are needed to describe each
source: the two masses m1and m2, the time of coalescence tc, the sky-position angles β and
λ, the luminosity distance DL, the orbital-inclination and GW-polarization angles ι and ψ,
and the initial orbital phase ϕ0. The binaries in both data sets had masses drawn from the
6

Page 7

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Figure 2. Combined A and E spectral amplitude for the GW signals in data set 1B.1.4, before
and after subtracting the AEI signal model. In the bottom panel, the extrinsic parameters for all
binaries in the model were adjusted by maximizing the F-statistic.
(This figure is in colour only in the electronic version)
same ranges (m1= 1–5 × 106M?,m1/m2= 1–4), but were distinguished by the times of
coalescence tc= 6 ± 1 months for Challenge 1B.2.1 and 400 ± 40 days (past the end of the
data set) for Challenge 1B.2.2; the SNRoptfor the sources were chosen to be ?500 and ?80,
respectively.
Two groups submitted entries: JPL (a collaboration between researchers at Caltech and at
the Jet Propulsion Laboratory) employed a three-step hierarchical strategy combining a time–
frequency track-search analysis, a template-bank matched-filtering search, and a final MCMC
stage to evaluate the posterior probability densities of source parameters for data sets 1B.2.1
and 1B.2.2. Cardiff (a collaboration based at that university) used a stochastic-template-bank
matched-filtering search [12] to analyze data set 1B.2.1.
Table 4 summarizes these entries. While the recovered SNRs are very close to SNRopt,
which indicates detections of very high confidence, there are large discrepancies in the sky-
position angles for Challenge 1B.2.1, where both the JPL and Cardiff searches converged on
secondarylikelihoodmaxima, verycloseinheighttothetruemode(asshownbytherecovered
SNR), but quite distant in parameter space. In fact, the JPL result places the source almost at
the antipodal sky position, even if SNR ? SNRoptto better than one part in a thousand.
This is a true global degeneracy, which does not appear in local Fisher-matrix analyses
(another example of why mock-data endeavors are useful!), and which may indicate the need,
in EM searches of counterparts to LISA binary-MBH detections, to examine unconnected
regions of the sky.It may however be premature to make such an inference, since the
degeneracy could be broken by spin effects (now included in Challenge 3 waveforms) and
higher waveform harmonics. In addition, EM-counterpart searches would require a data-
analysissystemcapableofdeterminingtheskypositionofMBHbinariesafewdaysinadvance
of their merger (corresponding to the interval between data dumps from LISA to the ground),
whereas data set 1B.2.1 included the inspiral waveform all the way to the approximate merger
frequency. Therefore this challenge was not aimed directly at establishing the feasibility of
sky-position determination for EM-counterpart searches. In this context, it is however worth
7

Page 8

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Table4. Relative/absoluteerrorsfortheMBHbinariesinChallenges1B.2.1and1B.2.2. Allangles
are given in radians. Error estimates have been adjusted to account for two perfect symmetries of
thewaveformswithrespecttosourceparameters: ψ → ψ+π,andthesimultaneoustransformation
ψ → ψ + π/2,ϕ0→ ϕ0− π/2.
1B.2.1 (SNRopt= 531.84)
Group JPL Cardiff JPL
1B.2.2 (SNRopt= 80.67)
SNR
?m1/m1
?m2/m2
?tc[s]
?DL/DL
?β
?λ
?ι
?ψ
?ϕ0
531.57
5.991 × 10−3
−5.252 × 10−3
206.1
−0.139
2.429
3.133
0.713
−0.564
−2.846
511.78
0.108
−0.111
−541.8
−1.438
1.374
0.548
0.678
1.448
−2.389
79.86
0.122
−0.134
−2688.2
4.781 × 10−3
5.862 × 10−3
−1.461 × 10−2
−6.955 × 10−2
−4.878 × 10−2
1.293 × 10−2
pointingoutthattheentriesprovidedveryaccuratedeterminationsofthetimesofcoalescence,
corresponding to time windows of a few minutes (for Challenge 1B.2.1) and about 45 min (for
1B.2.2).
If we compare the errors of table 4 with the parameter-determination accuracies predicted
in the Fisher-matrix formalism, we see that the JPL estimates for m1and m2fall within the
2-σ contour for data set 1B.2.1 (notwithstanding the problem with sky position), and near the
1-σ contour for 1B.2.2. For data set 1B.2.2, JPL’s ?tc,?DL,?β, and ?ϕ0, are also close to
1-σ, and ?λ is ∼2.2σ; the errors in ι and ψ are all tens of σs (but the Fisher-matrix formalism
is not always reliable for extrinsic parameters). Altogether, we conclude that the JPL search
essentially achieves the theoretical limits of parameter extraction for data set 1B.2.2, while it
does so for an important subset of parameters for data set 1B.2.1.
2.3. Extreme mass ratio inspirals: Challenges 1B.3.X
Although Challenge 2 saw a few successful detections of EMRI signals [5], data sets 1B.3.1–
3.5 represented the first real testbed for the search algorithms developed for this critical source
class. EachdatasetcontainedaGWsignalfromasingleEMRIembeddedininstrumentnoise,
with SNRoptbetween ?55 and ?135, and source parameters chosen randomly as described in
[4]. Fourteen parameters are needed to describe each EMRI source [4]: the ecliptic latitude
and longitude β and λ and the luminosity distance DL; the central-BH and compact-object
masses M and μ; the magnitude a and orientation angles θK,φKof the central-BH spin; the
initial radial orbital frequency ν0and eccentricity e0; and three angles ˜ γ0,α0and λ describing
the initial orientation of the orbit.
Entries were received from three groups: BBGP (a collaboration of scientists at the AEI,
Cambridge, andtheUniversityofSouthampton); EtfAG(AEI,Northwestern, andCambridge);
and MT (Montana State University).
EtfAG employed a time–frequency technique [13],
whereas BBGP [14] and MT [15] developed coherent approaches based on Monte Carlo
techniques, differing in their implementation. The entries were assessed as discussed at
the beginning of this section, although the EtfAG time–frequency analysis cannot determine
the extrinsic parameters, so their recovered SNR and correlation could not be computed.
Tables 5 and 6 summarize all results.
Both the time–frequency and coherent approaches succeeded in detecting these relatively
strongEMRIsignalsandinconstrainingtheirparameters(withespeciallyremarkableaccuracy
8

Page 9

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Table 5. Overlaps and recovered SNRs for TDI observables A,E and combined recovered SNR
for data sets 1B.3.1–3.5.
Group
CA
SNRA
CE
SNRE
Total SNR
1B.3.1 (SNRopt= 123.7)
51.0
86.1
BBGP
MT
0.57
0.998
0.58
0.997
51.6
88.3
72.5
123.4
1B.3.2 (SNRopt= 133.5)
6.6
37.6
49.5
BBGP
BBGPa
MT
0.07
0.39
0.54
0.18
0.41
0.54
18.2
39.8
50.8
17.6
54.7
70.9
1B.3.3 (SNRopt= 81.0)
−4.2 −0.0003
−11.5 −0.32
22.0
BBGP
BBGPa,b−0.2
MT
−0.06
−0.05
−19.0
20.9
−3.0
−21.5
30.40.38 0.35
1B.3.4 (SNRopt= 104.5)
2.1
−0.0002 −0.8
13.9 0.04
BBGPb
BBGPc
0.0007
0.16
2.1
14.6 6.7
1B.3.5 (SNRopt= 57.6)
3.4 BBGP0.090.14.2 5.3
aC and SNR after correcting the sign of β, lost on input to the MLDC webform.
bThe BBGP SNRs can be negative because BBGP maximized likelihood analytically over
amplitude, which makes SNR sign-insensitive (a minus sign corresponds to a change of π in the
phase of the dominant harmonic). This degeneracy is broken when all the harmonics are found
correctly.
cC and SNR after correcting phases at t = 0, to account for a BBGP bug.
for data set 1B.3.1), although not all groups analyzed all data sets, and the performance of
the same search pipeline varied across them. Challenge participants report that in some cases
this was due to a lack of time for extended computations before the challenge deadline, so
some parameter set were submitted as ‘best fits’ although they were clearly understood to
be secondary likelihood maxima. Thus, this early development work indicates that the main
challenge for (isolated) EMRI analyses is the very complex structure of the likelihood surface
in source-parameter space, which features a number of secondary maxima of similar height,
even more so than for MBH binaries.
We caution the reader that it would be inappropriate at this time to draw general
conclusions about the relative merits of search methods and about the expected science payoff
of LISA EMRI astronomy: it is not known how these techniques scale as the SNR decreases
and in situations where EMRI signals overlap with each other and are affected by galactic
confusion noise. The first two complications will be addressed in Challenge 3.
3. Synopsis of Challenge 3
The third round of the MLDCs consists of five challenges (3.1–3.5).
consist of approximately 2 years of data (222samples at a cadence of 15 s) for time-delay
interferometry (TDI) observables X,Y and Z. These data sets are released both as time series
of equivalent strain generated by the LISA Simulator [18] and as time series of fractional
frequency fluctuations generated by Synthetic LISA [19]; see [4, p S556] for the conversion
Data sets 3.1–3.3
9

Page 10

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Table 6. Errors for a subset of EMRI parameters in Challenges 1B.3.1–3.5. ‘?x/[x]’ denote
fractional errors relative to the physical or prior range of the parameter. (This shows the amount
of information about the parameters that is gained with the search.) Large errors correspond to
searches converging to secondary maxima of the likelihood; the true global maximum was found
only by MT in Challenge 1B.3.1.
Group
?β
[β]
?λ
[λ]
?θK
[θK]
?φK
[φK]
?a
[a]
?μ
[μ]
?M
[M]
?ν0
ν0
?e0
0.15
?λSL
[λSL]
Challenge 1B.3.1
0.0530.31
0.33 0.16
0.0051 −0.0022
Challenge 1B.3.2
−0.33
−0.36
0.41
BBGP −0.03
EtfAG
MT
−0.0059 −0.14
−0.0045
0.0027
−0.20
−0.11
0.0065
−0.84
−0.27
0.014
0.026
−9.3 × 10−5
3.2 × 10−6−0.0085 −0.0020
0.37
0.17
−0.022
0.078
0.019
0.0058
0.56
4.4 × 10−4
BBGP −0.16
EtfAG −0.014
MT
−0.43
0.0042
0.46
0.97
0.79
−0.0088 −0.0040
0.0043 −0.046
0.093
0.016
−0.069 −6.5 × 10−5
0.35
−0.035
1.4 × 10−4−0.010
−0.0013
0.0041
0.092
0.041
0.068 0.0040 −0.0086
−0.064
Challenge 1B.3.3
0.045
−0.32
−0.34
0.077
−0.066
Challenge 1B.3.4
−0.31
−0.34
Challenge 1B.3.5
0.1
−0.094
−0.084
BBGP
EtfAG −0.01
MT
0.0910.50
−0.004
−0.019
−0.23
0.49
−0.1
−0.49
−0.029
−0.061 −7.8 × 10−5
0.593.6 × 10−4−0.33
6.1 × 10−4
0.019
0.038
0.054
0.0061
0.010
0.0073 −0.059
0.0450.13
BBGP −0.57
EtfAG −0.56
−0.37
0.49
0.37
0.56
−0.025
0.059
0.020
0.12
−0.88
0.04
0.066
2.8 × 10−4−0.039
0.065
−0.16
0.0040
BBGP −0.48
EtfAG −0.58
−0.14
0.46
−0.35
0.27
−0.094
−0.7
0.55
0.83
−0.0021
−0.066
−0.017
0.066
−0.060
0.270.20
between the two. Indeed (with a few exceptions, described below, for 3.4 and 3.5), the
Challenge-3 data sets are built using the ‘pseudo-LISA’ model of Challenges 1 and 2: the
orbits of the LISA spacecraft are e2-accurate Keplerian ellipses with conventional orientations
and time offsets; modified TDI (a.k.a. TDI 1.5) expressions are used for the observables;
and Gaussian, stationary instrument noise is included from six proof masses and six optical
benches with known noise levels that are identical across each set of six22. See [4] for details.
• Dataset3.1containsagalacticGWforegroundfrom∼60millioncompactbinarysystems.
This data set is a direct descendant of Challenge 2.1, but it improves on the realism of the
latter by including both detached and interacting binaries with intrinsic frequency drifts
(either positive or negative). Section 3.1 gives details about the binary waveform models,
22The six proof-mass noises are uncorrelated and white in acceleration, with one-sided power spectral density (PSD)
S1/2
acc(f) = 3 × 10−15[1 + (10−4Hz/f)2]1/2m s−2Hz−1/2;
the six optical-path noises are uncorrelated and white in phase with PSD
S1/2
opt(f) = 20 × 10−12m Hz−1/2;
the conversion to Synthetic LISA’s dimensionless fractional frequency fluctuations is described on [19, p 6]; the
values actually used in the MLDCs are
Sacc(f) = 2.5 × 10−48(f/Hz)−2[1 + (10−4Hz/f)2] Hz−1;
Sopt(f) = 1.8 × 10−37(f/Hz)2Hz−1.
10

Page 11

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
about their implementation in the LISAtools suite [16] and about the generation of the
galactic population.
• Data set 3.2 contains GW signals from 4–6 binaries of spinning MBHs, on top of
a confusion galactic-binary background.
Challenges 1.2.1–2 and 2.2 by modeling the orbital precession (and ensuing GW
modulations) due to spin–orbit and spin–spin interactions. Section 3.2 gives details
about the MBH-binary waveforms.
Because this challenge focuses on the effects of spins rather than on the joint search for
MBH signals and for the brightest galactic binaries, the background is already partially
subtracted—it is generated from the population of detached binaries used for Challenge
3.1, withholding all signals with SNR > 5.
• Data set 3.3 contains five GW signals from EMRIs. As in Challenges 1.3.1–5, EMRI
waveforms are modeled with Barack and Cutler’s ‘analytic kludge’ waveforms [17]; this
challenge introduces the complication of detecting five such signals with lower SNRs,
and in the same data set. By contrast, galactic confusion is not included. See section 3.3
for details.
This data set improves on the realism of
Challenges 3.4 and 3.5 address the detection of two GW sources that are new to the
MLDCs, and that have (respectively) bursty and stochastic characters: thus, these searches
require an accurate characterization of instrument noise, which in reality will not be available
apriori,butwillbeobtainedfromtheLISAmeasurementsthemselves. Tomodelthisproblem,
in data sets 3.4 and 3.5 the levels of the six + six secondary noises have been independently
randomized by ±20%; the noises are however still uncorrelated. In addition, these data
sets contain time series for all 12 ‘raw’ LISA phase measurements yijk and zijk [19], so
that challenge participants may now build additional TDI observables to help characterize
instrument noise. The phase measurements do include laser phase noise, because otherwise
theywouldconveyextrainformationunavailablefromtherealLISA;butlasernoiseisreduced
in level to ∼10 times the secondary noise at 1 mHz, so that it can be canceled relatively easily
with TDI 1.5 implemented with moderate timing precision. To wit:
• Data set 3.4 consists of 221samples at a cadence of 1 s (∼24 days altogether), and it
contains GW burst signals from cosmic string cusps, occurring as a Poissonian random
process throughout the dataset, withanexpectation value of five events. Detailsabout the
waveforms are given in section 3.4. The data set is provided only as fractional frequency
fluctuations generated by Synthetic LISA.
• Data set 3.5 consists of 220samples at a cadence of 2 s (again ∼24 days), and it contains
a stochastic GW background, which is isotropic, unpolarized, Gaussian and stationary;
its spectrum grows at low frequencies as 1/f3, and its magnitude is set to a few times
the secondary noise over a broad range of frequencies. Details about the synthesis of the
background and the simpler model of the LISA orbits used for this challenge are given
in section 3.5. The data set is provided as fractional frequency fluctuations generated by
Synthetic LISA and by the new simulator LISACode [20], recently integrated into the
LISAtools suite [16]; thus, cross checks are possible between the two simulators.
LISACode [20] was developed at APC-Paris with the purpose of accurately mapping the
impact of the different LISA subsystems on its science observations, and of bridging the gap
between the basic principles of the LISA measurement and a future, more sophisticated end-
to-endsimulator. Thus, LISACodeincludesrealisticrepresentationsofmostoftheingredients
that will influence LISA’s sensitivity (such as orbits, instrument noise, ultra-stable-oscillator
time stamps, phasemeter response functions), internal waveform generators for several kinds
of sources (monochromatic and chirping binaries, stochastic backgrounds, etc), as well as
11

Page 12

Class. Quantum Grav. 25 (2008) 184026S Babak et al
the possibility of building various TDI combinations. Many user-defined parameters make it
possible to study the impact of different LISA configurations on its sensitivity. LISACode’s
conventions follow closely those of the MLDCs and of Synthetic LISA.
All the Challenge-3 data sets can be downloaded at astrogravs.nasa.gov/docs/
mldc/round3/datasets.html, encoded in lisaXML [2], an XML-based format that can be
displayed directly in modern web browsers, and handled easily in C/C++, Python and
MATLAB with the LISAtools I/O libraries [16]. Each data set is released in the blind
challengeversionandinatrainingversionthatincludesthesourceparametersusedtogenerate
it. Additional training data sets can be generated easily with the LISAtools suite23.
The remainder of this section describes the GW signal models adopted for each data set.
See [4] for the conversion of the GW polarizations in source frame (given here) to the LISA
frame. Table 7 is a glossary of source parameters with their symbols and lisaXML descriptors,
while table 8 is a summary of the GW content of each data set along with the ranges used to
choose source parameters randomly.
3.1. Chirping galactic binaries
Dataset3.1containsGWsfromapopulationof∼26×106detachedand∼34×106interacting
galacticbinaries. Eachbinaryismodeledasasystemoftwopointmassesm1andm2incircular
orbit with linearly increasing or decreasing frequency (depending on whether gravitational
radiation or equilibrium mass transfer is dominant). The polarization amplitudes at the solar-
system barycenter, expressed in the source frame, are given by
+(t) = A(1 + cos2ι)cos[2π(ft +˙ft2/2) + φ0],
hS
hS
(2)
×(t) = −2A(cosι)sin[2π(ft +˙ft2/2) + φ0],
where the amplitude is derived from the physical parameters of the source as A =
(2μ/DL)(πMf)2/3, with M = m1+ m2the total mass, μ = m1m2/M the reduced mass,
and DLthe distance;˙f is the (constant) frequency derivative, and φ0is the phase at t = 0.
Since it would be unfeasible to process millions of barycentric binary waveforms
individually through the LISA simulators to compute the TDI-observable time series, we
adopt a fast frequency-domain method [21] that rewrites the LISA phase measurements as the
fast–slow decomposition
yij(t) = C(t)cos(2πf0t) + S(t)sin(2πf0t);
the functions C(t) and S(t) describe slowly varying effects such as the rotation of the LISA
arms, the Doppler shift induced by orbital motion, and the intrinsic frequency evolution of the
source. These‘slow’termscanbesampledverysparselyandFourier-transformednumerically,
while the ‘fast’ sine and cosine terms can be Fourier-transformed analytically. The results
are then convolved to produce the LISA phase measurements, and these are assembled into
the desired TDI variables. This algorithm is three-to-four orders of magnitude faster than
the time-domain LISA simulators, although it effectively approximates LISA as a rigidly
rotating triangle with equal and constant armlengths. See [21] for full details, and directory
MLDCwaveforms/galaxy3 in LISAtools for the source code.
The starting point for each realization of data set 3.1 are two large catalogs provided
by Gijs Nelemans (files MLDCwaveforms/galaxy3/Data/AMCVn GWR MLDC.dat and
(3)
23After installing LISAtools following the instructions at code.google.com/p/lisatools/wiki/Install, generating a
training set is as simple as running (say, for Challenge 3.1)
MLDCpipelines2/bin/challenge3.py -T -R 3.1.
12

Page 13

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Table 7. Source parameters in Challenge 3.
Standard parameter name
(lisaXML descriptor)
Standard unit
(lisaXML descr.) Parameter Symbol
Common parameters
EclipticLatitude
EclipticLongitude
Polarization
Inclination
DL
Distance
Ecliptic latitude
Ecliptic longitude
Polarization angle
Inclination
Luminosity distancea
β
λ
ψ
ι
Radian
Radian
Radian
Radian
Parsec
Galactic binaries
Amplitude
Frequency
FrequencyDerivative
InitialPhase
Amplitudeb
Frequency
Frequency derivative
Initial GW phase
A
f
˙f
φ0
1 (GW strain)
Hertz
Hertz/second
Radian
Spinning massive black-hole binaries
m1,m2
Mass1,Mass2
a1,a2
Spin1,Spin2
θS1,φS1
PolarAngleOfSpin1
AzimuthalAngleOfSpin1
θS2,φS2
...likewise
Tc
CoalescenceTime
?c
PhaseAtCoalescence
θL,φL
InitialPolarAngleL
InitialAzimuthalAngleL
Masses of component MBHs
Magnitude of spins S1,S2
Initial orientation of spin S1
SolarMass
MassSquared
Radian
Radian
Initial orientation of spin S2
Time to coalescence
Phase at coalescence
Initial orientation
of orbital momentum
Second
Radian
Radian
Radian
EMRIs: see table 5 of [4]
Cosmic string cusp bursts
A
tC
CentralTime
fmax
MaximumFrequency
Amplitudeb(Fourier)
Central time of arrival
Maximum frequencyc
Isotropic stochastic background
PSDb,dat 1 Hz
Amplitude Hertz^(1/3)
Second
Hertz
Sh
PowerSpectralDensity
(f/Hz)^ − 3/Hz
aWe do not deal explicitly with the redshifting of sources at cosmological distances, so DLis a
luminosity distance, and all masses and frequencies are measured at the solar-system barycenter
and red/blue-shifted by factors (1 + z)±1with respect to those measured locally near the sources.
bReplaces DL for galactic binaries, cosmic-string-cusp bursts and stochastic-background
pseudosources.
cEffectively replaces ι for cosmic-string-cusp bursts.
dNote also that Sh= Stot
h/384; ψ is set to 0, and ι not used.
dwd GWR MLDC.dat in the LISAtools installation), which contain the parameters of
26.1×106detached and 34.2×106interacting systems produced by the population synthesis
codes described in [22, 23]. Figure 3 shows the distribution of the binaries in the catalogs over
f and˙f. Recent work by Roelofs, Nelemans and Groot [24] suggests that the model in [23]
overpredicts the number of (AM CVn) interacting systems by a factor of 5–10, but we did not
implement this correction for Challenge 3.
The parameters of each binary in the catalogs are modified by randomly tweaking f
by ±1%,A by ±10%, β and λ by ±0.5◦, and by randomly assigning ψ, ι and φ0(˙f is
computed fromthecatalog’sbinary-periodderivative andfromthetweaked f). Theserandom
perturbations are large enough to render the original population files useless as answer keys,
13

Page 14

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Table 8. Summary of data-set content and source-parameter selection in Challenge 3. Parameters
are sampled randomly from uniform distributions across the ranges given below, and all angles
(including spin and orbital-angular-momentum directions for MBH binaries) across the entire
appropriate ranges. Source distances are set from individual-source SNRs, which are drawn
randomly from the ranges specified below (where ‘SNR’ refers to the multiple-TDI-observable
SNR approximated as√2 × max{SNRX,SNRY,SNRZ}). The MBH time of coalescence tcand
the cosmic-string-cusp burst central time tCare given relative to the beginning of the relevant data
sets.
Data setSources Parameters
3.1
Galactic-binary background
Randomized population (see section 3.1)
∼34 × 106interacting, ∼26 × 106detached
Known parameters (see section 3.1) plus 20 verification binaries
3.2
4–6 MBH binaries
for each: m1= 1–5 × 106M?,m1/m2= 1–4,
a1/m1= 0–1,a2/m2= 0–1
MBH1: tc= 90 ± 30 days, SNR ∼ 2000
MBH2: tc= 765 ± 15 days, SNR ∼ 20
MBH3: tc= 450 ± 270 days, SNR ∼ 1000
MBH4: tc= 450 ± 270 days, SNR ∼ 200
MBH5: tc= 540 ± 45 days, SNR ∼ 100
MBH6: tc= 825 ± 15 days, SNR ∼ 10
randomized population with approx. SNR < 5
∼26 × 106binaries; no verification
for each: μ = 9.5–10.5M?,S = 0.5–0.7M2,
time at plunge = 221–222× 15 s,
ecc. at plunge = 0.15–0.25, SNR = 10–50
emri1: M = 0.95–1.05 × 107M?
emri2and emri3: M = 4.75–5.25 × 106M?
emri4and emri5: M = 0.95–1.05 × 106M?
(with n Poisson-distributed with mean 5)
fmax= 10−3–1Hz,tC= 0–221s, SNR = 10–100
all instrument noise levels randomized ±20%
2 × 192 incoherent h+and h×sources over sky
Stot
all instrument noise levels randomized ±20%
... including
... and 2–4 chosen from
plus galactic confusion
3.3
5 EMRIs
... including
3.4
n Cosmic-string-cusp bursts
3.5
Isotropic stochastic background
h= 0.7–1.3 × 10−47(f/Hz)−3Hz−1
but small enough to preserve the overall parameter distributions. Binaries with approximate
single-Michelson SNR > 10 are regarded as ‘bright’ and listed in a separate table in the
challenge keys. Data set 3.1 also includes 20 verification binaries of known parameters
(specified in LISAtools file MLDCwaveforms/galaxy3/Data/Verification.dat as rows of f,˙f,
β, λ, A).
3.2. Spinning MBH binaries
The spinning-MBH-binary GW signals of data set 3.2 are modeled as restricted waveforms
(no higher harmonics) from 2PN circular adiabatic inspirals, with uncoupled orbital frequency
evolution and spin and orbital precession. Both the orbital phase and frequency are computed
as explicit functions of time, corresponding to T3 waveforms in the classification of [25]:
14

Page 15

Class. Quantum Grav. 25 (2008) 184026 S Babak et al
Figure3. HistogramofthedensityofgalacticbinariesintheNelemanscatalogs,binnedbylog10f
and log10|˙f|.
Mω =1
8τ−3/8
?
1 +
?743
?1855099
2688+1132η
?
τ−1/4−
56975
258048η +
3
10
?
371
2048η2−
π −β
4
?
τ−3/8
+
14450688+
3
64σ
?
τ−1/2
?
,
(4)
where M = m1+ m2is the total mass, η = m1m2/M2≡ μ/M is the symmetric mass ratio,
and
τ =
η
5M(Tc− t),
1
12
i=1,2
σ = −1
(5)
β =
?
48ηχ1χ2[247(ˆS1·ˆS2) − 721(ˆLN·ˆS1)(ˆLN·ˆS2)].
?
χi(ˆLN·ˆSi)
?
113m2
i
M2+ 75η
??
,
(6)
(7)
HereˆLN,ˆS1andˆS2are the unit vectors along the leading-order angular orbital momentum
and the MBH spins. The intrinsic orbital phase is
?orb= ?C−τ5/8
η
?9275495
however, because the spin–orbit coupling causes the orbital angular momentum to precess
aroundthetotalangularmomentum, thephasethatentersthegravitationalwaveformscontains
the additional correction [26]:
˙? = ω +(ˆLN· ˆ n)[ˆLN× ˆ n] ·˙ˆLN
1 − (ˆLN· ˆ n)2
?
1 +
?3715
14450688+284875
8064+55 96η
?
τ−1/4−
258048η +1855
3
16(4π − β)
2048η2−15
+
64σ
?
τ−1/2
?
;
(8)
≡ ω + δ˙?,
(9)
15