ArticlePDF Available

Deep Learning Technology for Predicting Solar Flares from (Geostationary Operational Environmental Satellite) Data

Authors:

Abstract and Figures

Solar activity, particularly solar flares can have significant detrimental effects on both space-borne and grounds based systems and industries leading to subsequent impacts on our lives. As a consequence, there is much current interest in creating systems which can make accurate solar flare predictions. This paper aims to develop a novel framework to predict solar flares by making use of the Geostationary Operational Environmental Satellite (GOES) X-ray flux 1- minute time series data. This data is fed to three integrated neural networks to deliver these predictions. The first neural network (NN) is used to convert GOES X-ray flux 1-minute data to Markov Transition Field (MTF) images. The second neural network uses an unsupervised feature learning algorithm to learn the MTF image features. The third neural network uses both the learned features and the MTF images, which are then processed using a Deep Convolutional Neural Network to generate the flares predictions. To the best of our knowledge, this work is the first flare prediction system that is based entirely on the analysis of pre-flare GOES X-ray flux data. The results are evaluated using several performance measurement criteria that are presented in this paper.
Content may be subject to copyright.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
492 | P a g e
www.ijacsa.thesai.org
Deep Learning Technology for Predicting Solar
Flares from (Geostationary Operational
Environmental Satellite) Data
Tarek A M Hamad Nagem, Rami
Qahwaji, Stan Ipson
School of Electrical Engineering and
Computer Science
University of Bradford
Bradford, United Kingdom
Zhiguang Wang
GE Global Research
San Ramon, CA, United States of
America
Alaa S. Al-Waisy
School of Electrical Engineering and
Computer Science
University of Bradford
Bradford, United Kingdom
AbstractSolar activity, particularly solar flares can have
significant detrimental effects on both space-borne and grounds
based systems and industries leading to subsequent impacts on
our lives. As a consequence, there is much current interest in
creating systems which can make accurate solar flare
predictions. This paper aims to develop a novel framework to
predict solar flares by making use of the Geostationary
Operational Environmental Satellite (GOES) X-ray flux 1-
minute time series data. This data is fed to three integrated
neural networks to deliver these predictions. The first neural
network (NN) is used to convert GOES X-ray flux 1-minute data
to Markov Transition Field (MTF) images. The second neural
network uses an unsupervised feature learning algorithm to learn
the MTF image features. The third neural network uses both the
learned features and the MTF images, which are then processed
using a Deep Convolutional Neural Network to generate the
flares predictions. To the best of our knowledge, this work is the
first flare prediction system that is based entirely on the analysis
of pre-flare GOES X-ray flux data. The results are evaluated
using several performance measurement criteria that are
presented in this paper.
KeywordsConvolutional; neural; network; deep; learning;
solar; flare; prediction; space; weather insert
I. INTRODUCTION
The concept of space weather has been defined by the US
National Space Weather Program as “Conditions on the Sun
and in the solar wind, magnetosphere, ionosphere and
thermosphere that can influence the performance and reliability
of space-borne and ground-based technological systems and
can endanger human life or health” [1]. There are several
influences, originating from space weather phenomena that
detrimentally affect important industries relying on avionics,
satellites, mobile communication networks, and electricity
distribution [2]. All these industries touch our daily lives and
this means that space weather can impact our lives
dramatically.
Painstaking efforts are currently being made in a number of
international centres to create accurate solar flare prediction
systems. This is because many infrastructures could be affected
by significant flares and the cost of building an accurate solar
flare prediction system would be much cheaper than the cost of
repairing damage caused by such a flare. In this work, the
proposed prediction system generates two probabilities for
Event and No-event. Event predictions cover significant X and
M class flares that might be harmful, while No-event
predictions cover no-flares and the non-harmful A, B and C
class flares.
Although scientific progress has increased enormously the
rate of generation of data monitoring solar activity, scientists
are not yet able to fully understand all the detailed causes of
solar flares. Consequently, efforts are being made to develop
methods to predict solar storms, making direct use of the data
using advances in data analysis.
Since 1987, there have been many approaches that
attempted to predict solar flares. The first solar flare prediction
system (called THEOPHRASTUS) was launched by the Space
Environment Services Centre at NOAA, and it predicts X-ray
flares with a time window of 24 hours [3]. More recently, three
solar flare prediction systems, ASSA (Automatic Solar
Synoptic Analyser), MAG4 (Magnetic Forecast system) [7]
and ASAP (Automated Solar Activity Prediction), have
become a part of the NASA Integrated Space Weather Analysis
(ISWA) system [5] and these three systems are briefly
described below.
The first system, ASSA, is based on an artificial neural
network technique and the ASSA coronal hole data archive,
from the period 1997 till 2013, including SDO solar images, to
predict solar flares, solar radiation storms and geomagnetic
storms. ASSA predicts C, M and X flares. ASSA predictions
are based on statistical analysis of the ASSA sunspot catalogue
[6]. The second system, MAG4 was developed at the
University of Alabama in Huntsville, to assist NASA Space
Radiation Analysis Group (SRAG) at the Johnson Space Flight
Centre. MAG4 is using Magnetogram data for the Sun. MAG4
forecasts X and M class flares, CMEs, and Solar Proton Events
(SPE) using McIntosh active-region (AR) classes as the basis
of their forecasts [7]. The University of Bradford developed a
forecasting model, the Automated Solar Activity Prediction
(ASAP) system in 2009. ASAP uses McIntosh classes and
other sunspots features which it generates from the solar data.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
493 | P a g e
www.ijacsa.thesai.org
ASAP uses SDO/HMI Continuum and Magnetogram images
as an input to the system, also it uses two neural networks to
predict solar flares [3].
Recently, the new field of deep learning neural network
research has achieved remarkable successes compared with
previous artificial intelligence methods [5]. These include
complex tasks like medical diagnoses, dealing with huge
amounts of data, pattern recognition and numerous others, such
as the virtualization frameworks for big data reported in [8].
Using the deep learning technology for space weather
prediction is still a novel area of research, which needs to be
investigated to help analyse the huge amount of solar activity
data that are publically available.
UFCORIN (Universal Forecast Constructor by Optimized
Regression of Inputs) is open-source software available online
which has been used to predict general time series and solar
flares. This system uses HMI image data and GOSE X-ray data
as input to predict X, M, and C solar flare class. In 2016,
UFCORIN was extended to use deep learning, and provides
24-hour-ahead predictions of solar flares, every 12 minutes by
using a deep learning approach.
In this paper, we introduce a solar flare prediction system,
summarised in the following subsection, working solely with
GOES X-ray flux data that integrates three neural networks to
deliver these predictions and provides an automated prediction
of solar flares by utilising deep learning techniques.
GOES data are available in real-time (available every
minute) and they provide a general indication of flaring across
the solar disk. These data come in soft and hard x-ray and are
available from 2002. However, GOES data provide an
indication of flaring without much info about the exact location
of flaring on the solar disk. This could be one of the reasons
why it is not used heavily for space weather prediction. The
format of GOES Data is also challenging as it is represented as
a time-series signal, which makes it challenging for machine-
learning based prediction (Deep learning in particular).
Fig. 1. The diagram showing the internal procedures of the system.
A. Overview of the System
Fig. 1 shows the system model which consists of three
units. Starting from the input (GOES X-ray flux time series
data) to the output (solar flare prediction) and including the
evaluation of the predictive performance.
Unit 1in Fig. 1 converts a sequence of GOES X-ray flux 1-
minute data time series data to a 64 × 64 MTF image in two
stages. Firstly, it converts the original text data to a Markov
Transition Matrix. Then it encodes the Markov Transition
Matrix as a 64 × 64 Markov Transition Field (MTF) image as
illustrated in Fig. 6. Unit 2 in Fig. 1 learns the features within
the MTF images. Unit 2 pre-processes and normalizes the
images and then divides the 64×64 images into 64 8×8 patches.
These patches are encoded using a Back-propagation Auto-
encoder to obtain learned feature mappings as indicated in Fig.
1. Unit 3 in Fig. 1 provides predictions for solar flares using a
CNN. This unit starts by utilising the historical knowledge and
linking the MFT images with the Flare or No-Flare labels.
Subsequently, datasets are created for training and testing the
neural networks. After training on the associated dataset is
carried out, the trained CNN is run on the test dataset to
generate prediction results, which are evaluated using space
weather verification metrics.
The rest of this paper is organized as follows. Section 2
describes the operation of Unit 1 which converts GOES X-ray
flux time series data to 64 ×64 MTF images. Section 3
describes Unit 2, which learns features within MTF images
using an unsupervised learning algorithm by applying back-
propagation. Section 4 describes Unit 3, which makes solar
flare predictions using a Deep Convolutional Neural Network.
Section 5 discusses the evaluation and performance of the
whole system and Section 6 presents concluding remarks and
suggestions for future work.
Fig. 2. A sample 6 hour plot of GOES X-ray flux 1-minute data.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
494 | P a g e
www.ijacsa.thesai.org
II. PREPARATION OF THE DATA
A. The Source X-Ray Data
In this work, 1-minute X-ray flux data from the American
Geostationary Operational Environmental Satellites (GOES)
are used. The data used are provided from four GOES
satellites, GOES-10, GOES-11, GOES-14, and GOES-15. All
the data produced are archived and available, and it can be
found online at [9]. Two X-ray channels are available as shown
in Fig. 2; a harder X-ray channel (0.05-0.4 nm), and a softer X-
ray channel (0.1-0.8 nm) [10]. For this work, the soft channel is
used because provides information about the intensity of solar
flares and is used in this work to investigate its suitability for
investigating the temporal evolution of flares [10].
B. Extraction of Relevant X-Ray Flux Data
The temporal evolution of solar flares generally occurs in
three phases [4].
Pre-flare phase: This is the region shown in Fig. 3
which consists of fluctuations and a slow increase of X-
ray flux before the start of the flare event.
Impulsive phase: Here the X-ray flux increase quickly
and the main flare energy release occurs during this
phase.
Gradual phase: In this phase, the X-ray flux gradually
decreases to the background level.
Fig. 4 shows the cropped AIA images of a flaring region
corresponding to the GOES X-ray data regions in Fig. 3. The
left image in Fig. 4, captured in the pre-flare phase, shows two
sets of nested loops. The middle image in Fig. 4, captured
during the main phase, shows inner loops becoming
significantly brighter. In the right-hand image, the flare
launches a CME. There are many relationships which have
been recognized between the pre-flare activities and flaring,
and these appear as loop brightening activities [15]. However,
the method introduced here bases its prediction solely on
changes in the overall X-ray flux during the pre-flare phase.
Fig. 3. The solar flare phases on C8.8 flare that occured on 5th May 2010
From NASA [11].
Fig. 4. Cropped AIA images showing three phases of the solar flare which
contributes to the GOES data shown in Fig. 3 From NASA [11].
C. Prediction Optimization for Different Time Windows
The Time windows of 20, 30, 60 and 120 minutes between
the end of a data sample and the start of a flare/no-flare are
investigated, using the Quadratic score QR, to determine the
time window with the best prediction performance. QR is
widely used as a verification measure to evaluate the accuracy
of prediction. The prediction accuracy is calculated by finding
the mean square error between the predictions and the
observations as given by [2].
QR =

 (1)
where ot are the binary observation outcomes where 1
means that flare occurred and 0 means that a flare did not
happen, N is the sample size, and ft is the prediction
probability. QR ranges from 0 (perfect prediction) to 1 (worst
possible prediction) [18].
The result for each time window is shown in Table I. It is
clearly seen that the best QR is when the time window equals
20 minutes. To find the prediction window duration that would
provide the best QR value, we followed the method presented
in [2] and applied QR to determine the best prediction window
duration.
TABLE I. THE QUADRATIC SCORE (QR) RESULTS FOR 20, 30, 60 AND 120
MINUTES
D. Data Presentation
Fig. 5 shows a sub-system that has been created to generate
datasets by selecting specific data from GOES X-ray flux 1-
minute data using three steps. The first step identifies a flare.
Then selects 120 minutes of data, starting 140 minutes before
the beginning of the flaring event. Finally, the selected data is
saved in a matrix as described in the next subsection.
Sample Size 20 minutes 30 minutes 60 minutes 120 minutes
2124 0.136 0.153 0.249 0.590
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
495 | P a g e
www.ijacsa.thesai.org
Fig. 5. Creating dataset of a time series of X-ray flux data with a 20-minute
data window before the flare occurs.
E. Conversion of Time Series Data to MTF Images
Temporal and frequency correlations are major
dependencies embedded in time series data. To build a
comprehensive but intuitive visualization, the extracted
features of the designed data transformation framework should
be able to represent the dynamics in both time and frequency
while there should exist a reverse operation to map the
information back to the raw GOES time series. The following
sub-sections describe how to encode the dynamical frequency
information in the temporal ordering, illustrated in Fig. 6, step
by step.
The main idea of this stage is to use GOES time series data
to generate Markov transition field while maintaining the time-
series properties. The method applied in this research is taken
from [14]. MTF images were generated by applying the code
used in [14] to GOES data.
Fig. 6. Conversion of GOES X-ray data time series data to MTF images.
Fig. 7. Learning the features within MTF images.
III. LEARNING THE FEATURES WITHIN MTF IMAGES
The Auto-encoder is an unsupervised back-propagation
neural network which tries to learn a function hW,b(x) x, and
is adjusted so that the input values correspond to the target y(i)
  (i) [12]. In this work, we assume x is the input
corresponding to the pixel intensity values for an 8×8 MTF
image patch with 64 pixels so x = 64, and there are s2 = 32
hidden units in layer L2. The network is required to learn a
compressed representation of the input, because there exist
only 32 hidden units. Therefore the auto-encoder should
attempt to reconstruct the input to 8 × 8 images (64 pixels) [16]
as illustrated in Fig. 7.
IV. PREDICTION OF SOLAR FLARES USING A DEEP
CONVOLUTIONAL NEURAL NETWORK
As you can see in Fig. 8 the Convolutional Neural Network
(CNN) consists of convolutional layers and sub-sampling
layers followed by fully connected layers.
A. The Convolutional Layer
The input to this layer is a d × d × ch MTF image where d
is the height and the width of the image (d =64 in this case) and
ch is the number of channels. Since the MTF images are RGB
images, ch = 3. As illustrated in Fig. 9 the convolutional layer
uses Kf filters (also called Kernels) of size n × n × ch where n
is the dimension of the filter and n =8 to produce feature maps.
The Kf filters are convolved over the MTF image to create Kf
feature maps of size dn+1 [16].
Fig. 8. Convolutional neural network designed to predict solar flares.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
496 | P a g e
www.ijacsa.thesai.org
Fig. 9. Convolving filter over an input image in convolutional layer.
Fig. 10. An example of Max pooling.
B. The Pooling Layer
After the generation of the feature maps by the
convolutional layer, the features are then used for
classification. Fig. 10 shows each feature map is down-
sampled by max-pooling to size p × p. Typically, p ranges from
2 to 5, for small to big images respectively, and in this work
p=4 [16].
C. The Fully Connected Layer
This layer takes the outputs from the previous layers which
were reduced to a one-dimensional feature vector. This layer is
fully connected and there is just one output for each class label.
The high-level inference in the CNN is performed by this fully
connected layer.
V. IMPLEMENTATION AND EVALUATION OF THE SYSTEM
Three neural networks are integrated into the system to
predict solar flares. Fig. 4 shows the integrated system starting
from the input (GOSE data) to the output of the system
(Flare/No-Flare prediction).
The first part of the system, which encodes the GOES data
to MTF images, is implemented in Python and the rest of the
system is implemented in Matlab [17]. The system makes
flares predictions based on embedded learning rules. The
system was trained using training sets covering data from 3rd
Dec 2002 till 30th Jan 2017, to ensure this covered a range of
activity including both solar Maximum and solar Minimum of
the solar cycle.
A. System Evaluation
The performance evaluation was done by comparing the
generated predictions with the actual flare occurrences as
reported by 1-minute GOES data. The data were taken from
four satellites, GOES-10 data covering (03 Dec 2002 -22 Jun
2006) and (11 Apr 2007-30 Dec 2009); GOSE-11 data
covering (23 Jun 2006-10 Apr 2007); GOSE-14 data covering
(01 Nov 2009 -26 Oct 2010); and finally GOSE-15 data
covering (27 Oct 2010 -30 Jan 2017). The number of flaring
and No-flaring events for each satellite is detailed in Table II.
All GOES X-ray data were taken from [9].
As noted earlier in this paper, the data is classified as
flaring if they produced at least one M or X class flare in the
following 20 min period and No-flare if they did not cause any
M or X class flares during that period. To determine the flare
prediction capability we carried out experiments with 1-minute
GOES data covering (Dec 2002-Dec 2005, Jun 2009- Dec
2012) to train the deep learning algorithm. The data covering
(Jun 2006 - Dec 2008, Jun 2013 - Jan 2017) are used to test the
system as shown in Table III. Table IV details the number of
flare and no-flare data that were used in these experiments. The
time coverage of the training set was chosen so that the
remaining testing set would contain flare activity from periods
around the maximum and minimum levels of solar activity.
TABLE II. THE NUMBER OF FLARING AND NO-FLARING FOR GOES-10
DATA COVERING (03 DEC 2002 -22 JUN 2006), (11 APR 2007-30 DEC 2009);
GOSE-11 DATA COVERING (23 JUN 2006-10 APR 2007); GOSE-14 DATA
COVERING (01 NOV 2009 -26 OCT 2010); GOSE-15 DATA COVERING (27 OCT
2010 -30 JAN 2017) USED IN THIS EXPERIMENT
TABLE III. NUMBER OF FLARE AND NO-FLARE DATA COVERING (03 DEC
2002-30 JAN 2017)
03 Dec 2002-30 Jan 2017
Flare
No-flare
Total
1327
3981
5308
GOES-10
From
03 Dec 2002
To 22 Jun 2006
and
From
11 Apr 2007
To 30 Dec 2009
GOES-11
from
23 Jun 2006
To
10 Apr 2007
GOES-14
From
01 Nov 2009
To
26 Oct 2010
GOES-15
From
27 Oct 2010
To
30 Jan 2017
Flare
events
Flare
events
Flare
events
No
Flare
events
Flare
events
No
Flare
events
518
22
24
213
763
2070
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
497 | P a g e
www.ijacsa.thesai.org
TABLE IV. NUMBER OF FLARE AND NO-FLARE DATA IN TIME
INDEPENDENT TRAINING AND TESTING SETS
Training set
(Dec 2002-Dec 2005)
(Jun 2009- Dec 2012)
Testing set
( Jun 2006- Des 2008)
(Jun 2013-30 Jan 2017)
Flare
No-Flare
Total
Flare
No-Flare
Total
793
2391
3184
534
1590
2124
B. Machine Learning using Cross-Validation
Cross-validation is a method that partitions the input data
into subsets so that the learning algorithm can be trained on a
subset and internally tested on a different subset. Cross-
validation is a useful approach for analysing the prediction
performance of machine learning, as it is could help avoid
over-fitting. Over-fitting occurs when the learning algorithm
performs very well on the training data, but not so well when
provided with new data. Different forms of cross-validation
exist and the repeated random sub-sampling validation is
applied here. This method is based on randomly dividing the
data into a number of subsets, which is repeated a number of
times so that the learning algorithm is trained and tested on
different data. For each repetition, one subset is used for
training and the rest are used to evaluate the prediction
performance by calculating a number of forecast verification
metrics. These measurements are then averaged in order to
provide an indication of the effectiveness of the machine
learning on the training data [19].
Two separate portions of data are created: a training portion
(60%) and a testing portion (40%). The MTF images and their
corresponding flare/no-flare classifications from the training
portion are fed into the learning algorithm for training
purposes. When the training process is completed, the learning
algorithm is fed with the MTF images from the testing portion.
The learning algorithm attempts to predict their Flare/No-Flare
classifications. These predicted outputs are compared with the
testing datasets actual classifications using standard forecast
verification measures to evaluate the prediction performance of
the learning algorithm. Among the prediction measures, HSS is
one of the best indicators of the overall performance of a
prediction method since it accounts for correct chance forecasts
[20]. The cross-validation process is repeated 9 times and the
means of the prediction measures are calculated.
C. Verification Results
This system generates a prediction in binary form so 0
means no flare and 1 means a flare. In practice, flares occur
rarely compared to no-flares events. Various measures are used
to evaluate the predictions of the system. These measures are
for categorical prediction (Yes or No) and take the binary
prediction as an input to evaluate the output of the system. As
shown in Table V, the following four criteria are used to
investigate the predictions generated by the system.
TABLE V. CONTINGENCY TABLE FOR PERFORMANCE MEASUREMENTS
CONTAINING THE FOLLOWING ABBREVIATIONS FOR THE NUMBERS OF
PREDICTED TRUE POSITIVES A, FALSE POSITIVES B, FALSE NEGATIVES C, AND
TRUE NEGATIVES D
Flare prediction
Flare observations
Flare
No- Flare
Flare
a
b
No- Flare
c
d
n= a+b+c+d
If an MTF image is associated with a flare, and the
system prediction is a flare then this successful
prediction is a true positive (TP).
If an MTF image is associated with a flare, but the
system prediction is no-flare then this failed prediction
is a false positive (FP).
If an MTF image is not associated with a flare and the
system prediction is no-flare then this successful
prediction is a true negative (TN).
If an MTF image is not associated with a flare and the
system prediction is flare then this failed prediction is a
false negative (FN).
To further evaluate the results we used various prediction
verification measures for the 20 minute time window, shown in
Table III. The measures used are the Heidke Skill Score (HSS),
the percentage corrects (PC), the false alarm rate (FAR), the
probability of detection (POD), and the Brier Score (BS). The
formulae for these measures are defined in terms of the
abbreviations given in Table IV.
The percentage correct measure, PC, is used to calculate the
rate of predictions that are correct [13], and is defined as:
   
 (2)
The PC rate for the 20 minute time window is shown in
Table VI for all the predictions (flare or no-flare) and is 78%.
The Heidke Skill Score (HSS) is a measure showing the
improvement of the prediction over random prediction. HSS
ranges from -1 (for no correct predictions) to +1 (for very
accurate predictions) and a value of zero indicates that the
predictions are randomly generated [13]. HSS is defined by:
   
 (3)
HSS is a really useful measure for verifying systems that
seek to predict rare events, as in the present case.
The False Alarm Ratio FAR is the fraction of flare
predictions that are wrong. The range of FAR is from 0 (best
outcome) to 1 (poorest outcome) [18]. FAR is defined as:
FAR


(4)
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 1, 2018
498 | P a g e
www.ijacsa.thesai.org
TABLE VI. PREDICTION MEASURES ACHIEVED BY APPLYING MACHINE
LEARNING AND CROSS-VALIDATION WITH DATASETS COVERING (03 DEC 2002-
30 JAN 2017)
SPEC
SENC
QR
FAR
POD
PC
HSS
0.851
0.574
0.136
0.492
0.574
0.787
0.365
The Probability of Detection (POD) Pd, also known as the
Hit Rate (H), measures the probability of a solar flare being
correctly predicted by the system [18]. POD is given by:
Pd =
 (5)
It ranges from 0 (poorest outcome) to 1 (best outcome).
The Pd result for this system with a 20 min time window is
0.574.
This process separately uses data covering the complete
time range (03 Dec 2002 - 30 Jan 2017). The prediction
measures achieved for datasets are shown in Table VI. It can
be seen that the good levels of prediction measures are
achieved.
VI. CONCLUSION
This paper has introduced a prediction system that uses a
new technology for predicting solar flares from GOES data
using deep learning. This is the major contribution of this
paper. The system predicts automatically whether a flaring
event is going to occur in the next 20 minutes. Different
prediction windows were investigated using the QR measure,
and the most promising performance was found to be for the 20
minutes prediction window.
The performance of the prediction system introduced here
depends on the ability of the deep learning neural network to
efficiently classify the MTF images that have been generated to
visualise the GOES data. As demonstrated in Table VI all the
metrics used to evaluate the prediction performance (POD,
FAR, HSS, KSS, and PC) provide fairly good performances. In
particular, HSS results prove that the generated predictions are
definitely not generated by chance.
The prediction rates for our systems can be improved by
exploiting the advanced classification capabilities of machine
learning systems. Hence, we believe that it is important to
monitor the performance of the system during its initial stages
which include comparing the prediction performance with the
actual flares reported by NOAA. Evolutionary algorithms may
be used to allow the learning algorithms to evolve and provide
better optimization.
This work is continuing but we believe the initial results, as
reported in this paper, are very encouraging. However, we note
that not all flares have pre-flare phases occurring before them,
and this could be one of the reasons affecting our predictions.
To tackle these causes, our system could be integrated with
another statistical or machine learning prediction model (e.g.
ASAP
1
).
1
http://spaceweather.inf.brad.ac.uk/
REFERENCES
[1] T. I. Gombosi, D. L. Dezeeuw, C. P. T. Groth, K. G. Powell, C. Robert
Clauer, and P. Song, “From Sun to Earth: Multiscale MHD Simulations
of Space Weather,” in Space Weather. (2001), Geophys. Monogr. Ser.,
vol. 125, edited by P. Song, H. J. Singer, and G. L. Siscoe, pp. 169-176,
AGU, Washington, D. C., vol. 125, 2013, pp. 169176.
[2] T. Colak and R. Qahwaji, “Automated Solar Activity Prediction: A
hybrid computer platform using machine learning and solar imaging for
automated prediction of solar flares,” Sp. Weather, vol. 7, no. 6, p. n/a-
n/a, Jun. 2009.
[3] T Colak, R Qahwaji. AUTOMATED PREDICTION OF SOLAR
FLARES: Integrating Image Processing and Machine Learning for the
Creation of a Hybrid Computer Platform that Provides Real-Time
Prediction of Solar Flares. s.l. : LAP, 2010. ISBN-13: 978-3838370309.
[4] A david. Falconer, Ronald L. Moore, Abdulnasser F. Barghouty, Igor
Khazanov. MAG4 versus alternative techniques for forecasting active
region flare productivity. 306317, s.l. : Space Weather AGU Journal,
2014, Vol. 12. 10.1002/2013SW001024. vol. 125, edited by P. Song, H.
J. Singer, and G. L. Siscoe, pp. 169-176, AGU, Washington, D. C., vol.
125, 2013, pp. 169176.
[5] O. W. A. Ahmd, “ENHANCED FLARE PREDICTION BY
ADVANCED FEATURE EXTRACTION FROM SOLAR IMAGES,”
University of Bradford, 2011.
[6] Sunhak Hong, Jaehun Kim, Jinwook Han, Yungkyu Kim l. An
Automated Solar Synoptic Analysis Software System. s.l. : American
Geophysical Union, Fall Meeting 2012
[7] C. Chifor, D. Tripathi, H. E. Mason, and B. R. Dennis, “X-ray
precursors to flares and filament eruptions,” Astron. Astrophys., vol.
472, no. 3, pp. 967979, Sep. 2007.
[8] Janki Bhimani, Zhengyu Yang, Miriam Leeser, and Ningfang Mi.
“Accelerating Big Data Applications Using Lightweight Virtualization
Framework on Enterprise Cloud. “ s.l. : IEEE, 2017. 978-1-5386-3472-
1/17.
[9] http://darts.isas.ac.jp/pub/solar/sswdb/goes/xray/
[10] Caspi, T. N. Woods, and J. Stone, “A New Observation of the Quiet Sun
Soft X-ray (0.5-5 keV) Spectrum.”
[11] NASA, “GMS: SDO EVE Late Phase Flares.” [Online]. Available:
https://svs.gsfc.nasa.gov/10817. [Accessed: 28-Mar-2017].
[12] Ng, “CS294A Lecture notes Sparse autoencoder.”
https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf
[13] J. A. Guerra, A. Pulkkinen, and V. M. Uritsky, “Ensemble forecasting of
major solar flares: First results,” Sp. Weather, vol. 13, no. 10, pp. 626
642, Oct. 2015.
[14] Liu, L., & Wang, Z. (2016). Encoding Temporal Markov Dynamics in
Graph. Arxiv, 2.
[15] S. I. Syrovatskii and S. I., Comments on astrophysics and Space
Physics., vol. 4. Gordon and Breach], 1972.
[16] Adam Coates, Andrew Ng, Honglak Lee. An Analysis of Single-Layer
Networks in Unsupervised Feature Learning. : Proceedings of the
Fourteenth International Conference on Artificial Intelligence and
Statistics. PMLR 15:215-223,.
[17] https://github.com/cauchyturing/Imaging-time-series-to-improve-
classification-and-imputation/wiki
[18] C. C. Balch, “Updated verification of the space weather prediction
center’s solar energetic particle prediction model,” Sp. Weather, vol. 6,
no. 1, pp. 113, 2008.
[19] Hall, M.A.: Correlation based feature selection for Machine Learning
.PhD Thesis, The University of Waikato, Hamilton, New Zealand.
[20] Barnes and Leka, 2008 Astrophys. J.Lett. 688, L107.
... To our knowledge, this is the first time that machine learning method has been used to predict the solar flare index. In recent years, many excellent works have used some new deep learning techniques (such as CNN) to predict the occurrence of flares based on X-ray flux or extreme ultraviolet data (Nagem et al., 2018;Nishizuka et al., 2018b). We consider using more magnetic field parameters and image data to improve the performance of our prediction model in the future work. ...
Article
Full-text available
Solar flares are one of the most important sources of disastrous space weather events, leading to negative effects on spacecrafts and living organisms. It is very important to predict solar flares to minimize the potential losses. In this paper, we use three different machine learning algorithms: K-Nearest Neighbors (KNN), Random Forest (RF), and XGBoost (XGB) to predict the total flare index \(\text{T}_{\mathrm{flare}}\) and the maximum flare index \(\text{M}_{\mathrm{flare}}\) of an active region (AR) within the subsequent of 24, 48, and 72 hrs. First, we selected 54514 vector magnetograms of 129 ARs on the visible solar hemisphere in solar cycle 24 whose maximum sunspot groups’ area was larger than 400 μh. Then the following four magnetic parameters of each magnetogram were calculated: 1) the total magnetic flux \(|\Phi _{\mathrm{tot}}|\), 2) the total photospheric free magnetic energy density \(\text{E}_{\mathrm{free}}\), 3) the gradient-weighted integral length of the neutral line with horizontal magnetic gradient of line-of-sight magnetic field larger than \(0.1~\text{G}\,\text{km}^{\mathrm{-1}}\) (\(\text{WL}_{\mathrm{SG}}\)), and 4) the area with magnetic shear angle larger than \(40^{\circ }\) (\(\text{A}_{\Psi }\)), as well as \(\text{T}_{\mathrm{flare}}\) and \(\text{M}_{\mathrm{flare}}\) corresponding to each magnetogram. Afterward, we split samples randomly into training (85% of the whole data) and testing (15%) data sets. After hyperparameter tuning and model construction we found that RF is an optimal algorithm for the prediction task and that the coefficients of determination (\(\text{R}^{\mathrm{2}}\)) of test data set via the majority of RF models are beyond 0.97. In addition, the feature importance of RF and XGB models indicates that \(|\Phi_{\mathrm{tot}}|\) and \(\text{E}_{\mathrm{free}}\) are two optimal parameters to predict both \(\text{T}_{\mathrm{flare}}\) and \(\text{M}_{\mathrm{flare}}\), and \(|\Phi _{\mathrm{tot}}|\) and \(\text{E}_{\mathrm{free}}\) are the best parameters for \(\text{M}_{\mathrm{flare}}\) and \(\text{T}_{\mathrm{flare}}\), respectively.
... However, we note that not all flares have sunspots regions such as flares occurring near the sun's limb and the far side, which could significantly reduce the amount of data we can include in our learning process. To tackle this problem, the suggested system could be linked with another prediction model, such as the flares prediction system based on GOES data, reported in Nagem et al. (2018). Second, this study used instantaneous measurements to produce a binary flare forecast (flare, non-flare) with the class-balance dataset. ...
Article
In the last few years, there has been growing interest in near-real-time solar data processing, especially for space weather applications. This is due to space weather impacts on both space-borne and ground-based systems, and industries, which subsequently impacts our lives. In the current study, the deep learning approach is used to establish an automated hybrid computer system for a short-term forecast; it is achieved by using the complexity level of the sunspot group on SDO/HMI Intensitygram images. Furthermore, this suggested system can generate the forecast for solar flare occurrences within the following 24 hours. The input data for the proposed system are SDO/HMI full-disk Intensitygram images and SDO/HMI full-disk magnetogram images. System outputs are the “Flare or Non-Flare” of daily flare occurrences (C, M, and X classes). This system integrates an image processing system to automatically detect sunspot groups on SDO/HMI Intensitygram images using active-region data extracted from SDO/HMI magnetogram images (presented by (Colak and Qahwaji, 2008)) and deep learning to generate these forecasts. Our deep learning-based system is designed to analyze sunspot groups on the solar disk to predict whether this sunspot group is capable of releasing a significant flare or not. Our system introduced in this work is called ASAP_Deep. The deep learning model used in our system is based on the integration of the Convolutional Neural Network (CNN) and Softmax classifier to extract special features from the sunspot group images detected from SDO/HMI (Intensitygram and magnetogram) images. Furthermore, a CNN training scheme based on the integration of a back-propagation algorithm and a mini-batch AdaGrad optimization method is suggested for weight updates and to modify learning rates, respectively. The images of the sunspot regions are cropped automatically by the imaging system and processed using deep learning rules to provide near real-time predictions. The major results of this study are as follows. Firstly, the ASAP_Deep system builds on the ASAP system introduced in (Colak and Qahwaji, 2009) but improves the system with an updated deep learning-based prediction capability. Secondly, we successfully apply CNN to the sunspot group image without any pre-processing or feature extraction. Thirdly, our system results are considerably better, especially for the false alarm ratio (FAR); this reduces the losses resulting from the protection measures applied by companies. Also, the proposed system achieves a relatively high scores for True Skill Statistics (TSS) and Heidke Skill Score (HSS).
... Basic applications of CNNs to solar physics data classification is shown in [e.g. 1,16], however CNNs have also started being applied to the prediction of solar events, in particular flares and CMEs, that can affect space weather as considered, for example, by Bobra and Couvidat [3], Nagem et al. [21], Fang et al. [6]. ...
Preprint
In order to efficiently analyse the vast amount of data generated by solar space missions and ground-base instruments, modern machine learning techniques such as decision trees, support vector machines (SVMs) and neural networks can be very useful. In this paper we present initial results from using a convolutional neural network (CNN) to analyse observations from the Atmospheric Imaging Assembly (AIA) in the 1600A wavelength. The data is pre-processed to locate flaring regions where flare ribbons are visible in the observations. The CNN is created and trained to automatically analyse the shape and position of the flare ribbons, by identifying whether each image belongs into one of four classes: two-ribbon flare, compact/circular ribbon flare, limb flare or quiet Sun, with the final class acting as a control for any data included in the training or test sets where flaring regions are not present. The network created can classify flare ribbon observations into any of the four classes with a final accuracy of 94%. Initial results show that most of the images are correctly classified with the compact flare class being the only class where accuracy drops below 90% ad some observations are wrongly classified as belonging to the limb class.
Article
Full-text available
In this “Methods” paper, we investigate how to compress SDO/AIA data by transforming the AIA source maps into the Fourier domain at a limited set of spatial frequency points. Specifically, we show that compression factors of one order of magnitude or more can be achieved without significant loss of information. The exploration of data compression techniques is motivated by our plan to train Neural Networks on AIA data to identify features that lead to a solar flare. Because the data is spatially resolved and polychromatic (as opposed to spatially-integrated, such as GOES, or monochromatic, such as magnetograms), the network can be trained to recognize features representing changes in plasma properties (e.g., temperature, density), in addition to temporal changes revealed by Sun-integrated data or physical restructuring revealed by monochromatic spatially-resolved data. However, given the immense size of a suitable training set of SDO/AIA data (more than 10 ¹¹ pixels, requiring more than one TB of memory), some form of data compression scheme is highly desirable and, in this paper, we propose a Fourier based one. Numerical experiments show that, not only Fourier maps retain more information on the original AIA images compared to straightforward binning of spatial pixels, but also that certain types of changes in source structure (e.g., thinning or thickening of an elongated filamentary structure) may be equally, if not more, recognizable in the spatial frequency domain. We conclude by describing a program of work designed to exploit the use of spatial Fourier transform maps to identify features in four-dimensional data hypercubes containing spatial, spectral, and temporal information of the state of the solar plasma prior to possible flaring activity.
Research
Full-text available
Along with the development of technology, space weather activity becomes a very important thing in science. It's caused by whether activity that occurs in space can affect the activities of life on the earth. Therefore, it's important to be able to detect whether events in space, including solar flares. We believe that there has not been a single solar flare prediction study that did a prediction using PROBA-2 SWAP data, because flares are difficult to catch at that frequency. Nearly all previous researches have been focused on SOHO / MDI and SDO / HMI satellite. If the two satellites can't capture the image for some reason, then PROBA2 SWAP satellite imagery can be an alternative. This research is aimed to implement image processing and machine learning methods on SWAP PROBA2 satellite imagery to predict event numbers of solar flare. The machine learning algorithm used is random forest, while the segmentation algorithm used is seeded region growing. This solar flare prediction research using SWAP PROBA2 satellite imagery produced the best f-measure value of 0.897
Conference Paper
Full-text available
Hypervisor-based virtualization technology has been successfully used to deploy high-performance and scal-able infrastructure for Hadoop, and now Spark applications. Container-based virtualization techniques are becoming an important option, which is increasingly used due to their lightweight operation and better scaling when compared to Virtual Machines (VM). With containerization techniques such as Docker becoming mature and promising better performance, we can use Docker to speed-up big data applications. However, as applications have different behaviors and resource requirements, before replacing traditional hypervisor-based virtual machines with Docker, it is important to analyze and compare performance of applications running in the cloud with VMs and Docker containers. VM provides distributed resource management for different virtual machines running with their own allocated resources, while Docker relies on shared pool of resources among all containers. Here, we investigate the performance of different Apache Spark applications using both Virtual Machines (VM) and Docker containers. While others have looked at Docker's performance, this is the first study that compares these different virtualization frameworks for a big data enterprise cloud environment using Apache Spark. In addition to makespan and execution time, we also analyze different resource utilization (CPU, disk, memory, etc.) by Spark applications. Our results show that Spark using Docker can obtain speed-up of over 10 times when compared to using VM. However, we observe that this may not apply to all applications due to different workload patterns and different resource management schemes performed by virtual machines and containers. Our work can guide application developers, system administrators and researchers to better design and deploy big data applications on their platforms to improve the overall performance.
Article
Full-text available
Time series is attracting more attention across statistics, machine learning and pattern recognition as it appears widely in both industry and academia, but few advances has been achieved in effective time series visualization due to its temporal dimensionality and complex dynamics. Inspired by recent effort on using network metrics to characterize time series for classification, we present an approach to visualize time series as complex networks based on first order Markov process and temporal ordering. Different to classical bar charts, line plots and other statistics based graph, our approach delivers more intuitive visualization that better preserves both the temporal dependency and frequency structures. It provides a natural inverse operation to map the graph back to time series, making it possible to use graph statistics to characterize time series for better visual exploration and statistical analysis. Our experimental results suggest the effectiveness on various tasks such as system identification, classification and anomaly detection on both synthetic and the real time series data.
Poster
Full-text available
The solar corona is the brightest source of X-rays in the solar system, and the X-ray emission is highly variable with solar activity. While this is particularly true during solar flares, when emission can be enhanced by many orders of magnitude up to gamma-ray energies, even the so-called "quiet Sun" is bright in soft X-rays (SXRs), as the ~1-2 MK ambient plasma of the corona emits significant thermal bremsstrahlung up to ~5 keV. However, the actual solar SXR (0.5-5 keV) spectrum is not well known, particularly during quiet periods, as, with few exceptions, this energy range has not been systematically studied in many years. Previous observations include ultra-high-resolution but very narrow-band spectra from crystral spectrometers (e.g. Yohkoh/BCS), or integrated broadband irradiances from photometers (e.g. GOES/XRS, TIMED/XPS, etc.) that lack detailed spectral information. In recent years, broadband measurements with fair energy resolution (~0.5-0.7 keV FWHM) were made by SphinX on CORONAS-Photon and XRS on MESSENGER, although they did not extend below ~1 keV. We present observations of the quiet Sun SXR emission obtained using a new SXR spectrometer flown on the third SDO/EVE underflight calibration rocket (NASA 36.286). The commercial off-the-shelf Amptek X123 silicon drift detector, with an 8-micron Be window and custom aperture, measured the solar SXR emission from ~0.5 to >10 keV with ~0.15 keV FWHM resolution (though, due to hardware limitations, with only ~0.12 keV binning) and 2-sec cadence over ~5 minutes on 23 June 2012. Despite the rising solar cycle, activity on 23 June 2012 was abnormally low, with no visible active regions and GOES XRS emission near 2010 levels; we measured no solar counts above ~4 keV during the observation period. We compare our X123 measurements with spectra and broadband irradiances from other instruments, including the SphinX observations during the deep solar minimum of 2009, and with upper limits of >3 keV quiet Sun emission determined from RHESSI. We discuss the possible implications for X-ray-producing physical processes in the quiescent, active-region-free corona. The X123 spectrum could potentially serve as a reference for ~0.5-4 keV quiet Sun emission, to help improve solar spectral models such as CHIANTI and XPS Level 4. Our comparisons indicate that XPS Level 4 likely requires significant revisions in the SXR range, which may have downstream implications for the Earth ionosphere models that have used XPS Level 4 as their solar input.
Article
Full-text available
MAG4 is a technique of forecasting an active region's rate of production of major flares in the coming few days from a free magnetic energy proxy. We present a statistical method of measuring the difference in performance between MAG4 and comparable alternative techniques that forecast an active region's major-flare productivity from alternative observed aspects of the active region. We demonstrate the method by measuring the difference in performance between the "Present MAG4" technique and each of three alternative techniques, called "McIntosh Active-Region Class," "Total Magnetic Flux," and "Next MAG4." We do this by using (1) the MAG4 database of magnetograms and major flare histories of sunspot active regions, (2) the NOAA table of the major-flare productivity of each of 60 McIntosh active-region classes of sunspot active regions, and (3) five technique performance metrics (Heidke Skill Score, True Skill Score, Percent Correct, Probability of Detection, and False Alarm Rate) evaluated from 2000 random two-by-two contingency tables obtained from the databases. We find that (1) Present MAG4 far outperforms both McIntosh Active-Region Class and Total Magnetic Flux, (2) Next MAG4 significantly outperforms Present MAG4, (3) the performance of Next MAG4 is insensitive to the forward and backward temporal windows used, in the range of one to a few days, and (4) forecasting from the free-energy proxy in combination with either any broad category of McIntosh active-region classes or any Mount Wilson active-region class gives no significant performance improvement over forecasting from the free-energy proxy alone (Present MAG4). Quantitative comparison of performance of pairs of forecasting techniques Next MAG4 forecasts major flares more accurately than Present MAG4 Present MAG4 forecast outperforms McIntosh AR Class and total magnetic flux.
Article
Full-text available
The importance of real-time processing of solar data especially for space weather applications is increasing continuously. In this paper, we present an automated hybrid computer platform for the short-term prediction of significant solar flares using SOHO/Michelson Doppler Imager images. This platform is called the Automated Solar Activity Prediction tool (ASAP). This system integrates image processing and machine learning to deliver these predictions. A machine learning-based system is designed to analyze years of sunspot and flare data to create associations that can be represented using computer-based learning rules. An imaging-based real-time system that provides automated detection, grouping, and then classification of recent sunspots based on the McIntosh classification is also created and integrated within this system. The properties of the sunspot regions are extracted automatically by the imaging system and processed using the machine learning rules to generate the real-time predictions. Several performance measurement criteria are used and the results are provided in this paper. Also, quadratic score is used to compare the prediction results of ASAP with NOAA Space Weather Prediction Center (SWPC) between 1999 and 2002, and it is shown that ASAP generates more accurate predictions compared to SWPC.
Article
We present the results from the first ensemble prediction model for major solar flares (M and X classes). Using the probabilistic forecasts from three models hosted at the Community Coordinated Modeling Center (NASA-GSFC) and the NOAA forecasts, we developed an ensemble forecast by linearly combining the flaring probabilities from all four methods. Performance-based combination weights were calculated using a Monte Carlo-type algorithm by applying a decision threshold $P_{th}$ to the combined probabilities and maximizing the Heidke Skill Score (HSS). Using the probabilities and events time series from 13 recent solar active regions (2012 - 2014), we found that a linear combination of probabilities can improve both probabilistic and categorical forecasts. Combination weights vary with the applied threshold and none of the tested individual forecasting models seem to provide more accurate predictions than the others for all values of $P_{th}$. According to the maximum values of HSS, a performance-based weights calculated by averaging over the sample, performed similarly to a equally weighted model. The values $P_{th}$ for which the ensemble forecast performs the best are 25 \% for M-class flares and 15 \% for X-class flares. When the human-adjusted probabilities from NOAA are excluded from the ensemble, the ensemble performance in terms of the Heidke score, is reduced.
Article
Aims:To study preflare X-ray brightenings as diagnostics of the destabilisation of flare-associated erupting filaments/prominences. Methods: We combine new observations from the Transition Region and Coronal Explorer (TRACE) and the Reuven Ramaty High Energy Solar Spectroscopic Imager (RHESSI), as well as revisit events reported in the literature to date, in order to scrutinise the preflare activity during eight flare-associated filament eruptions. Results: The preflare activity occurs in the form of discrete, localised X-ray brightenings observed between 2 and 50 min before the impulsive phase of the flare and filament acceleration. These transient preflare brightenings are situated on or near (within 10{ arcsec} of) the polarity inversion line (PIL), coincident with emerging and/or canceling magnetic flux. The filaments begin to rise from the location of the preflare brightenings. In five out of eight events, the preflare brightenings were observed beneath the filament channel, close to the filament footpoint first observed to rise. Both thermal and nonthermal hard X-ray emissions during the preflare enhancement were detected with RHESSI, suggesting that both plasma heating and electron acceleration occurred at this time. The main energy release during the impulsive phase of the flare is observed close to (within 50{ arcsec} of) the preflare brightenings. The fast-rise phase of the filament eruption starts at the same time as the onset of the main flare or up to 5 min later. Conclusions: The preflare brightenings are precursors to the flare and filament eruption. These precursors represent distinct, localised instances of energy release, rather than a gradual energy release prior to the main flare. The X-ray precursors represent clearly observable signatures in the early stages of the eruption. Together with the timing of the filament fast-rise at or after the main flare onset, the X-ray precursors provide evidence for a tether-cutting mechanism initially manifested as localised magnetic reconnection being a common trigger for both flare emission and filament eruption. Movies and Appendix A are only available in electronic form at http://www.aanda.org
Article
This paper evaluates the performance of an operational proton prediction model currently being used at NOAA's Space Weather Prediction Center. The evaluation is based on proton events that occurred between 1986 and 2004. Parameters for the associated solar events determine a set of necessary conditions, which are used to construct a set of control events. Model output is calculated for these events and performance of the model is evaluated using standard verification measures. For probability forecasts we evaluate the accuracy, reliability, and resolution and display these results using a standard attributes diagram. We identify conditions for which the model is systematically inaccurate. The probability forecasts are also evaluated for categorical forecast performance measures. We find an optimal probability and we calculate the false alarm rate and probability of detection at this probability. We also show results for peak flux and rise time predictions. These findings provide an objective basis for measuring future improvements.
Article
There is an increasing need to develop physics-based, high performance models of the Sun-Earth system - from the solar surface to the Earth's upper atmosphere - which can operate faster than real time and which can provide reliable predictions of the near Earth space environment based upon solar observations and upstream solar wind measurements. Taking advantage of the advent of massively parallel computers, sophisticated solution-adaptive techniques, and recent fundamental advances in basic numerical methods we have developed a high performance, multiscale MHD code capable of resolving many of the critical processes in the Sun-Earth system which range over more than 9 orders of magnitude. We report on the first comprehensive numerical simulation of a synthetic space weather event, starting with the generation of a CME and subsequently following this transient solar wind disturbance as it evolves into a magnetic cloud and travels through interplanetary space towards Earth where its interaction with the terrestrial magnetosphere-ionosphere system is also predicted as part of the simulation.