Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era

Abstract

The rate of image acquisition in modern synoptic imaging surveys has already
begun to outpace the feasibility of keeping astronomers in the real-time
discovery and classification loop. Here we present the inner workings of a
framework, based on machine-learning algorithms, that captures expert training
and ground-truth knowledge about the variable and transient sky to automate 1)
the process of discovery on image differences and, 2) the generation of
preliminary science-type classifications of discovered sources. Since follow-up
resources for extracting novel science from fast-changing transients are
precious, self-calibrating classification probabilities must be couched in
terms of efficiencies for discovery and purity of the samples generated. We
estimate the purity and efficiency in identifying real sources with a two-epoch
image-difference discovery algorithm for the Palomar Transient Factory (PTF)
survey. Once given a source discovery, using machine-learned classification
trained on PTF data, we distinguish between transients and variable stars with
a 3.8% overall error rate (with 1.7% errors for imaging within the Sloan
Digital Sky Survey footprint). At >96% classification efficiency, the samples
achieve 90% purity. Initial classifications are shown to rely primarily on
context-based features, determined from the data itself and external archival
databases. In the ~one year since autonomous operations, this discovery and
classification framework has led to several significant science results, from
outbursting young stars to subluminous Type IIP supernovae to candidate tidal
disruption events. We discuss future directions of this approach, including the
possible roles of crowdsourcing and the scalability of machine learning to
future surveys such a the Large Synoptical Survey Telescope (LSST).

Full text

Page 1

arXiv:1106.5491v1 [astro-ph.IM] 27 Jun 2011
Automating Discovery and Classification of Transients and Variable Stars in
the Synoptic Survey Era
J. S. Bloom1, J. W. Richards1,2, P. E. Nugent3,1, R. M. Quimby4, M. M. Kasliwal4, D. L. Starr1,
D. Poznanski1,3, E. O. Ofek4, S. B. Cenko1, N. R. Butler1, S. R. Kulkarni4, A. Gal-Yam5, N. Law6
1Department of Astronomy, University of California, Berkeley, CA 94720-3411.
2Department of Statistics, University of California, Berkeley, CA 94720-7450.
3Computational Cosmology Center, Lawrence Berkeley National Laboratory, 1 Cyclotron Road,
Berkeley, CA 94720, USA
4Cahill Center for Astrophysics, California Institute of Technology, Pasadena, CA, 91125, USA
5Department of Particle Physics and Astrophysics, Faculty of Physics, The Weizmann Institute
of Science, Rehovot 76100, Israel
6Dunlap Institute for Astronomy and Astrophysics, University of Toronto, 50 St. George Street,
Toronto M5S 3H4, Ontario, Canada
ABSTRACT
The rate of image acquisition in modern synoptic imaging surveys has already begun
to outpace the feasibility of keeping astronomers in the real-time discovery and classifica-
tion loop. Here we present the inner workings of a framework, based on machine-learning
algorithms, that captures expert training and ground-truth knowledge about the vari-
able and transient sky to automate 1) the process of discovery on image differences and,
2) the generation of preliminary science-type classifications of discovered sources. Since
follow-up resources for extracting novel science from fast-changing transients are pre-
cious, self-calibrating classification probabilities must be couched in terms of efficiencies
for discovery and purity of the samples generated. We estimate the purity and effi-
ciency in identifying real sources with a two-epoch image-difference discovery algorithm
for the Palomar Transient Factory (PTF) survey. Once given a source discovery, using
machine-learned classification trained on PTF data, we distinguish between transients
and variable stars with a 3.8% overall error rate (with 1.7% errors for imaging within
the Sloan Digital Sky Survey footprint). At >96% classification efficiency, the samples
achieve 90% purity. Initial classifications are shown to rely primarily on context-based
features, determined from the data itself and external archival databases. In the ∼one
year since autonomous operations, this discovery and classification framework has led
to several significant science results, from outbursting young stars to subluminous Type
IIP supernovae to candidate tidal disruption events. We discuss future directions of this
approach, including the possible roles of crowdsourcing and the scalability of machine
learning to future surveys such a the Large Synoptical Survey Telescope (LSST).

Page 2

– 2 –
Subject headings: Data Analysis and Techniques—Astronomical Techniques
1.Introduction
The arrival of the era of synoptic imaging surveys heralds the start of a new chapter of time-
domain astrophysics, where the real-time processing of images taxes the capacity to transport the
data from remote sites and pushes to the limit the computational capabilities at processing centers
(e.g., Juri´ c & Ivezi´ c 2011). More profoundly novel, however, is that the data volumes have begun
to surpass what is possible to visually inspect by even large teams of astronomers and volunteer
“citizen scientists.” This necessitates an increasingly more central role of software and hardware
frameworks to supplant the traditional roles of humans in the real-time loop.
This abstraction of people away from the logistics of the scientific process has been progressing
rapidly, starting with the acquisition process itself. Indeed, robotic telescopes1, capable of taking
data autonomously at remote sites, have become an increasingly common form of operation at
the sub-meter- and meter-class level (cf. Castro-Tirado 2010). Many robotic systems use queuing
algorithms that optimize nightly observing over several scientific programs and many are capable
of being interrupted by external alerts to observe high-priority transients (e.g., Filippenko et al.
2001; Vestrand et al. 2002; Akerlof et al. 2003; Cenko et al. 2006; Bloom et al. 2006; Saunders
et al. 2008; Kubanek 2010). Data from such facilities can be automatically transported, processed,
photometered, and ingested into databases without human intervention.
Since imaging data has spurious sources of noise and artifacts that can mimic real astrophysical
sources, in the absence of watchful trained eyes on the images themselves, autonomous discovery
of transients and variable stars on synoptic imaging surveys is a significant challenge. Threshold
cuts on photometric quality, changes in apparent magnitudes, etc., are effective in discovering bona
fide astrophysics sources (Drake et al. 2009; Soko? lowski et al. 2010). However, multi-parameter
thresholding tends to be suboptimal because it treats each parameter derived from a given candidate
as an independent variable when clearly there can be correlations between parameters. Matched
filtering—looking for light curve trends that fit the scientific expectation from a certain class of
variables (e.g., microlensing; Tomaney & Crotts 1996; Belokurov et al. 2003)—can be a very effective
tool to discover new events, but other sorts of variables and transients are not easily recovered from
that view of the dataset. Likewise, previous machine-learning based discovery (e.g., supernova
discovery with the Supernova Factory; Bailey et al. 2007) have been optimized on domain-specific
discovery, leaving aside the multitude of other variables not of direct interest to that particular
project.
Discovery that a varying source is truly astrophysical does not mean that the origin of that
1For a list of robotic telescopes currently operating, see
http://www.uni-sw.gwdg.de/~hessman/MONET/links.html.

Page 3

– 3 –
variability is understood. Indeed, while it is tempting to conflate the process of discovery with
classification, by making sequential the two decisions, different machineries can be brought to
bear on each. The literature on autonomous classification, by various computational techniques,
has been growing rapidly; indeed a wide range of machine-learning techniques have been applied
to classification of large astronomical datasets (see Mahabal et al. 2008 and Bloom & Richards
2011 for review). Aside from domain-specific classification (microlensing and supernovae), most
work concerns classification of variables stars on historical datasets in retrospect, where analysis is
performed after most of the data have been collected and cleaned (e.g., Sarro et al. 2006; Debosscher
et al. 2007; Richards & et al. 2011; Willemsen & Eyer 2007; Sarro et al. 2009; Butler & Bloom
2010).
We are interested in a related, but more urgent challenge: classification on streaming data,
where analysis is performed while the data are still being accumulated. At a logistical level, keeping
up with classification (and discovery) assures that the survey producing the data can be continually
informed of the progress, allowing the survey to change course midstream if scientifically warranted.
But at a more fundamental level, the reason for real-time classification is that the vast majority of
science conducted with time-variable objects, especially one-off transients, comes when more data
are accumulated about the objects of interest. Enabling intelligent follow-up, then, becomes a main
driver for rapid classification. Ultimately, one can view classification as a means to maximizing
scientific return in a resource-limited environment.
Given this view of real-time classification, the advantages of a computational (rather than
human-centric) approach become clear:
• machines, properly trained, are faster than humans at discovery and classification of individual
candidates/events, allowing for operations at arbitrarily high data rates (limited only by
computational resources);
• the turn-around for well-informed follow-up can be almost instantaneous for computationally
based discovery and classification. This allows for more efficient use of the suite of follow-
up facilities. For example, observations on a small-aperture telescope can obtain the same
signal-to-noise of a fading transient as obtainable on a large-aperture telescope observed after
a longer delay;
• Experimentation with new discovery and classification schema requires little more than re-
running new codes on existing data, whereas a change to human-based approaches requires
additional labor-intensive work with people on a massive scale;
• machine-learned classification is reproducible and deterministic, whereas human-based clas-
sification is not;
• the reproducibility allows for calibration of the uncertainties of classification probability state-
ments, based on “ground-truth” results from the survey itself, with assurances that those
classifications are robust as the survey proceeds.

Page 4

– 4 –
Robust statements about the demographics of variability of different types requires well-calibrated
discovery and classification. And this, in turn, suggests that a machine-based approach is also
preferred. Ultimately, there may still be a vital role for humans in the real-time loop, such as serving
as “tie-breakers” on ambiguous classifications or uncertain follow-up paths for a particular source
(Gal-Yam et al. 2011), but our long-term view is that if a body of human-produced classification
statements can be reproduced by machine-learned frameworks, those sorts of statements (during
the full-scale production mode of a real synoptic survey) should not come from humans.
In this paper, we describe a methodology and formalism for producing discoveries of astrophysi-
cal transients and variable stars using a machine-learned framework based on human expert-trained
input (§2). We show how false-negative and false-positive rates can be calibrated with data from
the survey itself. In §3, we discuss a machine-based approach to autonomous classification based on
feature sets derived from context and time-series data on individually discovered sources. In §3.4
we show how a machine-learned model on Palomar Transient Factory (PTF; Rau et al. 2009; Law
et al. 2010) data produces highly reliable initial classifications2. We end with a discussion about
the outstanding challenges and look to future incarnations that may be used on upcoming synoptic
surveys.
2. Discovery on Images
To identify new sources or brightness changes of known sources in synoptic imaging there
are primarily two computational paths: catalog-based searches and imaging-differencing analysis.
With the former, sources in each image are found and extracted into a database consisting of
flux and position (and associated uncertainty) as well as ancillary metrics on individual detections
(such as shape parameters and photometric quality flags). Time-variable sources are then found
by cross matching detections on the sky and computing changes in brightness with time. With
the latter, a deep reference image is constructed from several images of a portion of the sky, it is
astrometrically aligned with and flux-scaled to an individual image, and it is subtracted from each
individual image. The result is a difference image (e.g., Bond et al. 2001), in which objects are
then found and extracted into a database. Since image differencing usually involves the expensive
cross-convolution of two images, catalog-based searches are considered computationally faster than
image-differencing. Catalog-based searches do well in the regime of large brightness changes and do
not suffer from color-correlated misalignment effects due to differential chromatic refraction (Drake
et al. 2009). However, in crowded fields (where the typical separation between objects is of order a
few PSF distances) or in the presence of high-frequency spatial variations in the background (i.e.,
2To be sure, an active group of citizen scientists enabled by the “Supernova zoo” also offer an important discovery
channel of supernovae (Smith et al. 2011) within the PTF collaboration that is largely separate from the autonomous
discovery and classification framework described herein.

Page 5

– 5 –
near galaxy positions), image-difference searches for variable sources excels3. For well-constructed
reference images, photometric uncertainties of sources found in image differences can approach
the statistical photon limit of an individual image (Wozniak 2000). Given the particular interest
in finding variable stars in crowded fields and events in and around galaxies (supernovae, novae,
and circumnuclear sources), especially while the sources are still faint and on the rise, the PTF
collaboration chose to perform discovery on image differences. This is also the intended discovery
path for most of the new upcoming synoptic surveys: Skymapper (Keller et al. 2007), Dark Energy
Survey (Flaugher 2005), and the Large Synoptic Survey Telescope (LSST; Becker et al. 2005; Ivezic
et al. 2008). The Catalina Real-Time Sky Survey (Drake et al. 2009) and the 3π survey of Pan-
STARRS (Kaiser et al. 2002) conduct catalog-based searches for transients (cf. Gal-Yam & Mazzali
2011).
2.1. Identification
Frameworks for identifying and characterizing significantly detected objects (e.g., SExtractor;
Bertin & Arnouts 1996) in images can be be applied to image differences.
drawbacks of discovery on image differences, however, is the number of spurious “candidate” objects
that can arise from improperly reduced new images, edge effects on the reference or new image,
misalignment of the images, improper flux scalings, incorrect PSF convolution, CCD array defects,
and cosmic rays4. Even with signal-to-noise thresholds and some requirements on metrics related
to the candidate shape (e.g., candidate FWHM compared with the image seeing), we have found
that the vast majority of SExtracted objects on a given difference image are spurious: in PTF, only
about 1 in 1000 (Negahban et al. 2011) extracted candidate objects (considered to be at least as
significant as a 5-σ detection) in a typical field are what we would deem to be astrophysically “real”
(i.e., an origin owning to a change beyond the Earth’s atmosphere). Nugent et al. (2011) provides
details on the SExtractor extraction requirements and which candidate/subtraction parameters are
saved into a database.
One of the major
2.1.1. Real or Bogus?
Beyond the subtraction and source extraction steps, our first significant challenge is in de-
termining which of the candidates are worth pursuing as real astrophysical events and which are
“bogus.” With training, many astronomers can identify when subtractions are poor or if a candi-
date is dubious to reasonable accuracy. But given the rate of candidate extractions, about 1–1.5
3If all detected objects are to be saved in each epoch, databases derived from image-differencing can be made
vastly smaller, since only those sources which change are saved.
4To be sure, some of these effects are also present in catalog-based searches.