Access to this full-text is provided by Wiley.
Content available from Computational Intelligence and Neuroscience
This content is subject to copyright. Terms and conditions apply.
Research Article
Spatial and Time Domain Feature of ERP Speller System
Extracted via Convolutional Neural Network
Jaehong Yoon ,1Jungnyun Lee ,2and Mincheol Whang 2
1Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA
2Department of Digital Media Engineering, Sangmyung University, Seoul 03016, Republic of Korea
Correspondence should be addressed to Jaehong Yoon; jaehong.ryan.yoon@gmail.com
Received 6 January 2018; Revised 5 March 2018; Accepted 1 April 2018; Published 15 May 2018
Academic Editor: Victor H. C. de Albuquerque
Copyright © Jaehong Yoon et al. is is an open access article distributed under the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Feature of event-related potential (ERP) has not been completely understood and illiteracy problem remains unsolved. To this end,
P peak has been used as the feature of ERP in most brain–computer interface applications, but subjects who do not show such
peak are common. Recent development of convolutional neural network provides a way to analyze spatial and temporal features
of ERP. Here, we train the convolutional neural network with convolutional layers whose feature maps represented spatial and
temporal features of event-related potential. We have found that nonilliterate subjects’ ERP show high correlation between occipital
lobe and parietal lobe, whereas illiterate subjects only show correlation between neural activities from frontal lobe and central lobe.
e nonilliterates showed peaks in P, P, and P, whereas illiterates mostly showed peaks in around P. P was strong
in both subjects. We found that P peak may be the key feature of ERP as it appears in both illiterate and nonilliterate subjects.
1. Introduction
A brain–computer interface (BCI) is a system which provides
a communication method by utilizing biophysiological sig-
nals []. BCI system enables the users to communicate with
external world through measurements of biological signals
and mostly do not require voluntary muscle movement.
e system has been utilized to support severe locked-in
syndrome (LIS) patients who lack motor ability, such as
amyotrophic lateral sclerosis (ALS) and Guillain–Barre syn-
drome patients, as a means of communication [–]. Of many
biophysiological signals, electroencephalography (EEG) has
been most widely used in BCI eld for its easiness in and low
cost of measurement [, ].
Among dierent applications of BCI, event-related po-
tential (ERP) based speller system has been one of the most
widely used paradigms. e system was pioneered by Farwell
and Donchin [] in which utilized oddball paradigm
in order to induce visual evoked potential (VEP), especially
the P response. However, there are still illiteracy problems
associated with ERP speller system [, ]. ere has been
reports of ERP features other than P [, ] which may
be a key feature of distinguishing identifying illiterates.
One of the most prominent classication methods for
ERP system is support vector machine (SVM) [–]. SVM is
mathematically simple and, with sucient knowledge of fea-
ture matrix, the experimenter can modulate the kernel for the
target problem. Unfortunately, the kernel of SVM is sensitive
to overtting []. As EEG are measured from multiple elec-
trodes [–], feature matrix can have high dimension with
possible duplicates, which increase possibility of overtting.
As most of ERP system paradigms are dependent on P
peak, the information (peak magnitude and latency) from
eachelectrodeshouldbesimilar.Moreover,itishardto
extract temporal and spatial information of EEG of a single
kernel. Although multiple kernel learning (MKL) problem
has been suggested [], it is hard to extract intuition of the
given problem through the method.
Recent development of deep learning provides an alter-
native approach. e convolutional neural network (CNN)
can extract the feature from a given feature vector by using
convolution. When an optimal lter is applied, the convolu-
tion will magnify the feature of interest and reduce the others
[].CNNhasbeenusedinpatternrecognition,especially
in image recognition and speech recognition, as it provides
topological information within the extracted feature [–].
Hindawi
Computational Intelligence and Neuroscience
Volume 2018, Article ID 6058065, 11 pages
https://doi.org/10.1155/2018/6058065
Computational Intelligence and Neuroscience
erefore, data with sequence or topological information can
be recognized more eciently as CNN enables extracting
both temporal and spatial information within the raw data.
AstheERPshowssequenceofriseandfallasaresponseto
visual stimuli, pattern recognition technique as CNN can be
applied. Moreover, the convolution kernel of CNN can be
used as tool for interpreting the spatial correlation among
EEG electrodes.
In this paper, we explore the performance of CNN on ERP
data to identify the key features that distinguish illiterates of
ERP speller system. e convolution kernels of trained model
will be explored to analyze the spatial correlation between
cortices and pattern within ERP of each electrode. e
subjects were grouped as either strong (nonilliterate) or weak
(illiterate) depending on clarity of ERP signals. Results of two
groups were compared to analyze dierence in features.
2. Methods
2.1. ERP Speller Design. iconsshowninFigurewereused
as visual stimuli for the speller system of this paper. Rapid
serial visual presentation (RSVP) panel design was adopted
for the speller system to avoid gaze eect. During the experi-
ment, screen size icons appeared on the center of the monitor
in a random sequence []. e oddball paradigm was
implemented by presenting target icon with distractors in a
random sequence []. Each icon appeared times per trial.
e interstimulus interval (ISI) between icon appearances
was set to ms.
2.2. Data Acquisition. For this paper, subjects ( female,
male) participated in the experiment. e subjects’ age
ranged from to (mean = ., std = ±.). During
the experiment, subjects were asked to sit upright on a chair
and instructed to keep still. No straps or ties were attached.
Subjectswereaskedtoself-reportanyinconveniencethat
might bother the concentration.
Each trial was initiated with an acoustic cue instructing
the target of the given trial in subjects’ mother tongue
(Korean). seconds aer the acoustic cue was given, the
icons appeared on the monitor according to RSVP design in
random sequence. e subjects were instructed to mentally
count the target occurrence during each trial (Figure (b)).
Each session consisted of trials. Each icon was selected as
a target during the session twice in random sequence.
All subjects were naive; –-minute preexperiment ses-
sion was given to get subjects used to the procedure. e
subjects were asked to self-report if they felt condent of
the procedure. Aer the preexperiment session ended, the
measurements of EEG were made. During the experiment,
one training session and online session were conducted as a
pair. To minimize subject’s stress level and fatigue, -minute
break was given in between training and online session. Each
subject conducted minimum of pairs of training and online
session. No subjects had participated in more than pairs of
sessions.
EEGwascollectedbyB-AlertXheadsetfromAdvanced
Brain Monitoring (ABM) with sampling rate of Hz. e
EEG electrodes recorded followed international / system
[] as shown in Figure (a). All experiments were held in
accordance with the Declaration of Helsinki, and the protocol
was approved by the Ethics Committee of Sangmyung Uni-
versity.
2.3.ConvolutionalNeuralNetwork. e architecture of CNN
for this paper was as shown in Figure (c). e CNN con-
sisted of convolutional layers, max-pooling layers, and
fully connected layers. Rectied linear unit (ReLU) function
was applied as activation function for each convolutional
layer since its performance was proven by another []. A
somax function was applied to output the last layer to
regularize the nal output to be between and . e output
of CNN was vector of elements where each element
represented the score of target and nontarget.
e CNN was designed to perform both spatial and
temporalltering.efeaturemapsofeachlayerwereusedto
access correlation between adjacent electrodes and temporal
feature of target ERP. In the st convolutional layer (L1), a
lter of size × was applied to extract correlation of
EEG recorded in adjacent electrodes. e row number of
the lter was set to as electrodes were placed on each
lobe (except for occipital lobe where two electrodes were
placed). e size of lter enables analyzing the correlation of
all electrodes from adjacent lobes. For analysis of temporal
feature of feature map from L1among dierent lobes, a lter
with size of × was applied for nd convolutional layer (L2)
whose window size was approximately ms in time scale.
To reduce the receptive eld size for ease of calculation
and prevent overtting, max-pooling layers (M1and M2)
were inserted aer each convolutional layer [, ]. e
max-pooling layers downsample the feature map by applying
a sliding window without overlap. As the name implies, the
maximum value within the window is extracted. As the max-
pooling introduces downsampling eect, a generalization of
feature map was achieved which prevented overtting of the
model. Sliding window sizes of M1and M2were ×and
×, respectively.
To further reduce the possibility of overtting while train-
ing the model, drop-out technique was applied on the rst
fully connected layer (F1). e drop-out technique padded
zerostorandomlyselectedrowsinthegivenfeaturemap.By
intentionally losing the data within the feature map, general-
ization was achieved for the feature map which prevented the
model from being overtted by the training data [, ].
e size of input matrix fed into the CNN was ×
where each row corresponded to EEG collected from each
electrode in Figure .
e CNN architecture was implemented in Python via
TensorFlow on Python [, ]. e Adam optimizer was
used to train the CNN which controls the learning rate to use
larger step size. , iterations were conducted for training
the model for each subject’s data.
2.4. Tie Breaking. Ideally, if the model is perfect, only one
icon will be identied as the target for a given trial. However,
the system identied multiple icons as the targets in several
trials. On the other extreme, the system failed to identify any
target icon for some trials. For each case, the tie breaking rule
was applied as follows.
Computational Intelligence and Neuroscience
(i) Multiple icons cases: When multiple icons were
thought to be the target of a given trial by the CNN,
the tie breaking rule was applied to select the target
among these candidates. Since the rst element of
output vector represents the icons aliation to target
ERPproperty,theiconwiththegreatestvalueofthe
elementwasselectedasthetargetofthetrial.
(ii) No target case: When the system failed to nd the
association of the ERP from any icons to property
of target ERP, that is, no icons were identied as the
target,samerulesasthoseinmultipleiconscasewere
appliedtoselectthetargetforthegiventrial.Inthis
case, the rst elemenet of output vector from all icons
was compared. e icon whose rst element of output
vectorwasthegreatestwasselectedasthetargetofthe
trial.
2.5. Analysis. Both qualitative and quantitative analysis were
performed to analyze the characteristics of lters of each
convolutional layer. e subjects were divided into two
groups according to their relative strength of ERP as follows:
(i) ERP detection: if the target icon was detected as pos-
itive in a given trial, the ERP is considered detected.
e subjects were divided accordingly into either H
or L group (H and L for high and low) ERP detection
group. e threshold between H and L group was
%.
(ii) Feature map: feature maps from L1and L2were drawn
in color map. As higher weights of featuremap denote
high discriminant power, the colormap can qualita-
tively give insight of how each electrode is correlated
andatwhichtimethemainpeakisformed.
(iii) Statistical analysis: for quantitative analysis of per-
formance, accuracy, sensitivity, precision, F mea-
sure, and ROC were calculated for each subject and
ANOVA test was held to compare mean dierence.
e accuracy is dened as the ratio of number of
correctly identied trial to total trial numbers. e
classical statistic measurements for quantitative eval-
uation are as follows:
TP ≡true positive,
FP ≡false positive
TP ≡true negative,
FP ≡false negative
Sensitivity =TP
FN +TP
Precision =TP
TP +FP
F1measure =2×Sensitivity ×Precision
Sensitivity +Precision .
()
(iv) Receiver operating characteristic: receiver operat-
ing characteristic (ROC), which plots the sensitivity
against specicity, widely used statistical measure-
ment for its diagnostic ability of binary classier. As
the CNN of the paper is a binary classier, the ROC
information is provided to compare the performance
of CNN between H and L group.
(v) Peak signal to noise ratio: peak signal to noise ratio
(PSNR) is used as measurement of qualitative recon-
struction method of compression codes []. As the
performance of lter will depend on how many core
features are extracted from raw ERP, the PSNR of L1s
feature map was calculated as a mean of measure-
ment of performance. e greater PSNR shows the
presence of signicantly high weight inside feature
map whereas lower PSNR represents only low weights
that are present in the given feature map and the
discriminant power of the lter is low.
3. Results
3.1. ERP Detection. Of subjects, were identied as H
group. In Figure , time course of learning curve and other
statistical measurements over the training iteration from H
and L subject are presented. e learning curve of L subject
shown in Figure (a) indicates that although the false negative
rate (FN) drops according to the training iteration, reaching
eventually, the false positive rate (FP) becomes . Although
the learning curve shows sharp increase at st and th
iteration, mostly it remains around .. is indicates that the
CNN becomes overtrained to positives (target). Moreover, as
the CNN identies most of the ERP to be positive (high FP
and low FN), the result indicates that discriminant feature of
target ERP was not found. On the other hand, both FN and
FP of H subject drop to around and .. e learning curve
saturates around . indicating nonovertting of the CNN
(Figure (b)).
e errors shown in Figures (c) and (d) are dened as
follows for training and online data:
error =TP +TN
TP +TN +FP +FN.()
Although both H and L group show drop in both training and
validation error as training iteration continues, the validation
errorofLsubjectishigherthanthatofHsubject.
eROCsofHandLsubjectshowninFigure(e)
indicate the performance of CNN of H group to be greater
than that of L group subject.
3.2. Spatial and Temporal Features. e feature map of each
convolutional layer did not contain negative weights associ-
ated with negative peaks, such as N [] as the activation
function was set to ReLU [].
e target ERP and feature map of L1of sampler H and L
subjectareshowninFiguresand.etargetERPshownin
both gures is target ERP averaged over all trials. To analyze
the correlation of frontal and occipital lobe electrodes, the
rst electrodes (rst rows of averaged target ERP matrix)
werecopiedandpastedattheendofERPmatrix.Asshown
in Figure (a), the target ERP of L group subject shows
Computational Intelligence and Neuroscience
T : Results of the CNN classication. Data are sorted according to the ERP group. Accuracy (Acc.), sensitivity (Sens.), precision (Prec.),
F measure, ROC, PSNR, and peak time of nd layer (PeT.) are given for comparison.
Subject number Type Acc. Sens. Prec. F measure ROC PSNR PeT.
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
H . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
L . . . . . −. .
broadpeakaroundPrangeonFandCZ.ERPofother
lobes did not show any signicant positive weight indicating
nonsignicant features associated with target being observed
and being at. Feature maps shown in Figures (b) through
(i) have shown high correlation between ERP from central
and parietal lobe electrodes.
On the other hand, the correlation of ERP among adjacent
electrodes for H group subject shown in Figure indicates
the correlation is restricted to specic time range. Most of
the high weights of feature maps shown in Figures (b), (d),
(f), and (e) show signicant positive value around P
and P range for frontal and central lobe electrodes. e
correlation between central and parietal lobe is shown in
Figure (c) around P range. Some features around P
region were found to show high correlation among all elec-
trodes.UnlikethatofLgroupsubjects,featuremapofL
1for H
group subject showed high correlation among all electrodes,
where each case shows specic temporal characteristics.
etemporalfeaturesshowninfeaturemapinFigure
indicate that temporal features associated with P peak are
present for L group subjects as expected. In Figures (a), (b),
and (c), high positive weights were found around P range
(row and ). However, most of the feature maps did not
show signicant weights or were either at as in Figure (i).
e temporal features of H group subjects showed more
variety. Some feature maps showed high positive weights in
their feature maps around P and P range as shown in
Figures (a), (b), (c), and (d), whereas the others indicated
signicant positive weight around P range as in Figures
(a)–(i). However, the weight associated with P range
is more widely dened than those associated with P and
P.
3.3. Statistical Analysis. Comparison of classical statistical
measurements and other measurements is shown in Table .
e accuracy, sensitivity, and precision showed signicant
mean dierence between H and L group (𝑝values were
Computational Intelligence and Neuroscience
A1A2
NASION
INION
F7F3FzF4F8
Fp1Fp2
T3C3CzC4T4
T5P3PzP4T6
O1O2
(a) (b)
1
0
output: 2
feature map: 6510
feature map: 7 × 15 × 62
max-pooling
max-pooling
convolution
convolution
feature map: 7×150×62
[1×12]
[1×10]
feature map: 7×150×32
[2×2]
[6×20]
feature map: 14 × 300 × 32
input: 14 × 300
···
···
···
···
(c)
F : Experimental paradigm. (a) e position of EEG channels in / system. e EEG were collected from F, Fz,F,C,Cz,C,
P, Pz, P, O, and O positions as indicated by red circle. (b) Experimental setting schematics. Subjects were sat on a chair and were asked
to mentally count the occurrence of target icon. e ERP speller system for this paper was implemented with RSVP. e icon appeared on
the center of the monitor. (c) Schematics of CNN architecture. e architecture consisted of convolutional layers, max-pool layers, and
fully connected layers. e number on top of each layer indicates size of feature map.
., 0.8.88e−05, and ., resp.). A signicant mean
dierence in F measure did not exist between H and L
group. e accuracy of H and L group was . and .,
respectively. e sensitivity of H group was higher than that of
L group, but the precision of H group was signicantly lower
than that of L group. e area under ROC of H group was
signicantly higher than that of L group (𝑝value = .).
e PSNR for L1of H group was signicantly lower than
that of L group. As all PSNR measured were negative, the ab-
solutevalueofPSNRofHgroupwasgreaterthanthatofL
group. On the other hand, no mean dierence of the peak
time (PeT.) between H and L group was found (𝑝value =
.).
4. Discussion
In this study, CNN has been used to investigate the spatial
and temporal characteristics of ERP that distinguish the
performance dierence between illiterates and nonilliterates
(L and H group). As a comparison of performance, classical
statistic measurements as well as lter comparison mea-
surementhadbeencollectedtocomparethecorrelation
of ERP taken from dierent EEG electrodes and identify
characteristic temporal features associated with each group.
e statistical measurement shows that the mean per-
formance of CNN with H and L group data had signicant
dierence. e accuracy of H group data was higher than
that of L group data. Interestingly, although the sensitivity
of H group was higher than that of L group, the precision of
H group was signicantly lower than that of L group. is
reects the fact that the ERP of L group was not identied as
target in most of the cases, and the CNN identied ERP from
all icons to be nontarget in more than half of the trials.
e learning curve and errors in Figure demonstrate
how the statistical measurement aects the performance of
CNN. Although the false negative rate remains mostly near
Computational Intelligence and Neuroscience
(a) (b) (c)
(d) (e) (f)
F : Schematics of icons used for rapid serial visual presentation (RSVP) panel. e design of icons was taken from television remote
controller. (a) Turn on. (b) Volume up. (c) Channel up. (d) Turn o. (e) Volume down. (f) Channel down.
, as the false positive rate remains close to , the learning
curve remains stable around . for the L group subject. is
again reects the characteristics of L group ERP who were
mostly identied as nontarget. Some of the ERP that were
identied as target ERP were mostly from nontarget icons,
indicating lack of distinctive feature associated with target
ERP.However,bothfalsenegativeandfalsepositiveratedrop
as training iteration continues for H group subject’s data,
leading to increase of learning accordingly to the iteration.
As the ERP of L group does not have sucient distinctive
features, the model becomes slightly overtrained compared to
the model of H group subject as shown in validation error plot
in Figures (c) and (d). e comparison of ROC validates the
analysis as ROC of H group was signicantly higher than that
of L group (𝑝value = .).
As shown in Figure , most of the ERP collected from L
group were at in most of the channels. Most of the positive
weights in target ERP were observed in frontal and central
lobe electrodes (st and th row of Figure (a)) which was
contrary to the expectation as previous research indicated
positive peaks associated with target event were mostly
observed in parietal or occipital lobe [, ]. e correlation
of ERP collected from adjacent electrodes did not show
existence of signicant correlation between occipital and
parietal lobe data in L group subjects. On the other hand, ERP
of H group were more invigorated, showing stronger activity
in P area as shown in Figure (a). e ERP correlation
indicated in feature map also indicated stronger correlation
of ERP data collected from occipital and parietal lobe with
other lobes. e spatial correlation shown in feature map of
H group also indicated that the correlation was restricted in
specic time range corresponding to either P, P, or
P.
e feature map of nd convolutional layer demonstrated
the dierence in temporal features between H and L group
subjects. In most of L group subjects, the feature map did not
show strong positive weights and was at. Some indication of
positive weights was mostly restricted in P region. On the
otherhand,thepositiveweightsofHgroupweredistributed
around P, P, and P and the positive weights found
near P and P range was sharper compared to those
found around P range. Previous researches have indicated
the possibility of existence of dierent features other than
P [, , ] e result of the paper also supports the
idea that P may not be the only key feaure of ERP speller
system. Rather, the P, which were identied among both
L and H group subjects, may represent more universal ERP
feature. However, the ERP from central lobe area observed
in L group indicates the possibility of eect of stimulus
probability [] (Figure (a)).
e PSNR indicated that lack of activities in occipi-
tal/parietal lobe and broad peak found in P aect the
Computational Intelligence and Neuroscience
iteration number
0 50 100 150 200 250
Rate
0
0.2
0.4
0.6
0.8
1
Type I error (FN)
Type II error (FP)
Learning curve
(a)
iteration number
0 50 100 150 200 250
Rate
0
0.2
0.4
0.6
0.8
1
Type I error (FN)
Type II error (FP)
Learning curve
(b)
iteration number
0 50 100 150 200 250
error rate
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Training Error
Validation Error
(c)
iteration number
0 50 100 150 200 250
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Training Error
Validation Error
(d)
False Positive
0 0.2 0.4 0.6 0.8 1
True Positive
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ROC of L subject
ROC of H subject
(e)
F : Learning curve and receiver operating characteristic curve (ROC) of L and H subject. (a) False negative rate (FN) and learning
curve of L subject saturates near and ., respectively, whereas false positive rate (FP) increase to . (b) Both FP and FN drop over the time
course for H subject and learning curve saturates near .. (c) Training and validation error of drops over the time course for both L subject
and (d) H subject. Both validation and training error are lower for H subject. (e) ROC curve of H and L subjects.
Computational Intelligence and Neuroscience
50
100
150
200
250
300
2
4
6
8
10
12
14 −30
−20
−10
0
10
(a)
100 200 300
2
6
10
14 0
5
10
15
20
100 200 300
2
6
10
14 0
5
10
15
20
100 200 300
2
6
10
14 0
5
10
15
20
100 200 300
2
6
10
14 0
5
10
15
20
(f)
100 200 300
2
6
10
14 0
5
10
15
20
100 200 300
2
6
10
14 0
5
10
15
20
100 200 300
2
6
10
14 0
5
10
15
20
100 200 300
2
6
10
14 0
5
10
15
20
(b) (c) (d) (e)
(g) (h) (i)
F : ERP averaged over all trials and feature map of L1of L subject. e ERP from frontal lobe was copied and pasted on last three rows.
(a) Grand average ERP over all trials. Feature maps from L1shownin(b),(c),(d),(e),(f),(g),(h),and(i).Strongcorrelationbetweenfrontal
and central lobe and between central and parietal lobe was found. Spatial correlation among other electrodes is not well dened.
50 100 150
2
4
6
0
1
2
3
4
5
(a)
50 100 150
2
4
6
0
1
2
3
4
5
(b)
0
1
2
3
4
5
50 100 150
2
4
6
(c)
50 100 150
2
4
6
0
1
2
3
4
5
(d)
50 100 150
2
4
6
0
1
2
3
4
5
(e)
50 100 150
2
4
6
0
1
2
3
4
5
(f)
50 100 150
2
4
6
0
1
2
3
4
5
(g)
50 100 150
2
4
6
0
1
2
3
4
5
(h)
50 100 150
2
4
6
0
1
2
3
4
5
(i)
F : Feature map of L2of L subject data. Temporal feature associated with P peak is found as shown in (a), (b), and (c).
Computational Intelligence and Neuroscience
50
100
150
200
250
300
2
4
6
8
10
12
14
0
10
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
100 200 300
2
6
10
14 0
1
2
3
4
5
(a)
(f)
(b) (c) (d) (e)
(g) (h) (i)
−30
−20
−10
F : ERP averaged over all trials and feature map of L1of H group. e format is the same as shown in Figure . (a) e grand averaged
ERP of H group shows signicant peak around P and P (rows , , , and ). Correlation between ERP from adjacent electrodes shows
high correlation related to specic time rage (P and P) in (b), (d), (f), and (e).
50 100 150
2
4
6
0
1
2
3
4
5
(a)
50 100 150
2
4
6
0
1
2
3
4
5
(b)
0
1
2
3
4
5
50 100 150
2
4
6
(c)
50 100 150
2
4
6
0
1
2
3
4
5
(d)
50 100 150
2
4
6
0
1
2
3
4
5
(e)
50 100 150
2
4
6
0
1
2
3
4
5
(f)
50 100 150
2
4
6
0
1
2
3
4
5
(g)
50 100 150
2
4
6
0
1
2
3
4
5
(h)
50 100 150
2
4
6
0
1
2
3
4
5
(i)
F : Feature map of L2of H subject. Format is the same as Figure . High positive weight around P and P range were found in
(a), (b), (c), and (d). (e)–(i) Moderate positive weight around P were also found.
Computational Intelligence and Neuroscience
performance of spatial lter in L as well. As the PSNR
measures the maximum power of a signal and the power
of corrupting noise [], the result indicates that the lter
was not able to extract distinctive signal of target ERP from
background noise for L group subjects’ data. is may be
since peaks near P were broad and uctuating. On the
other hand, P and P peaks found in H group subjects
were sharper, which made the lter extract relevant features
more precisely without being aected by background noise.
Interestingly, the major peak of L2of H and L group subjects
did not dier signicantly (𝑝value = .). As the major
peak was found by averaging the feature maps from L2,the
dierence in each feature map may have been overshadowed.
Further statistical analysis to access temporal feature within
each feature map must be applied to validate the results found
in this study.
5. Conclusions
is study has investigated the dierence in spatial and
temporal features of ERP between high performance group
(H group) and low performance group (L group). e result
indicated that the major dierence arises from spatial correla-
tion of ERP among other lobes rather than temporal features.
Although the temporal feature dierence was not found to
be quantitative in this study, the qualitative analysis indicated
lack of P in low performance group. Interestingly, both
low and high performance group showed activity near P
which may be the key activity of ERP speller system instead of
traditional P peak. Further analysis of individual feature
map will be needed to investigate the key temporal feature of
ERP speller system.
Conflicts of Interest
e authors declare that they have no conicts of interest.
Acknowledgments
is work was partly supported by Institute for Information
& Communications Technology Promotion (IITP) grant
funded by the Korea government (MSIT) (no. --,
the development of technology for social life logging based
on analyzing social emotion and intelligence of convergence
contents) and National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (no. -
).
References
[] J.R.Wolpaw,N.Birbaumer,D.J.McFarland,G.Pfurtscheller,
and T. M. Vaughan, “Brain-computer interfaces for communi-
cation and control,” Clinical Neurophysiology, vol. , no. , pp.
–, .
[] T. Fomina, G. Lohmann, M. Erb, T. Ethofer, B. Sch¨
olkopf, and
M. Grosse-Wentrup, “Self-regulation of brain rhythms in the
precuneus: A novel BCI paradigm for patients with ALS,” Jour-
nalofNeuralEngineering,vol.,no.,ArticleID,.
[] W.Speier,N.Chandravadia,D.Roberts,S.Pendekanti,andN.
Pouratian, “Online BCI typing using language model classiers
by ALS patients in their homes,” Brain-Computer Interfaces,vol.
, no. -, pp. –, .
[] L. Botrel, E. M. Holz, and A. K¨
ubler, “Using brain painting at
home for years: Stability of the P during prolonged BCI
usage by two end-users with ALS,” Lecture Notes in Computer
Science (including subseries Lecture Notes in Articial Intelligence
and Lecture Notes in Bioinformatics): Preface,vol.,pp.
–, .
[] S. Saeedi, R. Chavarriaga, R. Leeb, and J. d. Millan, “Adaptive
Assistance for Brain-Computer Interfaces by Online Prediction
of Command Reliability,” IEEE Computational Intelligence Mag-
azine,vol.,no.,pp.–,.
[] S.Saeedi,R.Chavarriaga,andJ.D.R.Millan,“Long-TermStable
Control of Motor-Imagery BCI by a Locked-In User rough
Adaptive Assistance,” IEEE Transactions on Neural Systems and
Rehabilitation Engineering,vol.,no.,pp.–,.
[] R. Swaminathan and S. Prasad, “Brain computer interface used
in health care technologies,” SpringerBriefs in Applied Sciences
and Technology,vol.,pp.–,.
[] C. Reichert, S. D ¨
urschmid, H.-J. Heinze, and H. Hinrichs, “A
comparative study on the detection of covert attention in event-
related EEG and MEG signals to control a BCI,” Frontiers in
Neuroscience, vol. , article no. , .
[] D. McFarland and J. Wolpaw, “EEG-based brain–computer
interfaces,” Current Opinion in Biomedical Engineering,vol.,
pp.–,.
[] L. A. Farwell and E. Donchin, “Talking o the top of your head:
Toward a mental prosthesis utilizing event-related brain poten-
tials,” Electroencephalography and Clinical Neurophysiology,vol.
, no. , pp. –, .
[] J. Yoon, M. Whang, and J. Lee, “Methodology of improving
illiteracy in P speller system with ICA blind detection,”
Proceedings of HCI Korea,pp.–,.
[] R. Carabalona, “e role of the interplay between stimulus type
and timing in explaining BCI-illiteracy for visual P-based
Brain-Computer Interfaces,” Frontiers in Neuroscience, vol. ,
article no. , .
[] S. L. Shishkin, I. P. Ganin, I. A. Basyul, A. Y. Zhigalov, and A. Y.
Kaplan, “N wave in the P BCI is not sensitive to the physical
characteristics of stimuli,” Journal of integrative neuroscience,
vol. , no. , pp. –, .
[] L. Bianchi, S. Sami, A. Hillebrand, I. P. Fawcett, L. R. Quitadamo,
and S. Seri, “Which physiological components are more suitable
for visual ERP based brain-computer interface? A preliminary
MEG/EEG study,” Brain Topography,vol.,no.,pp.–,
.
[] K. Yoon and K. Kim, “Multiple kernel learning based on three
discriminant features for a P speller BCI,” Neurocomputing,
vol. , pp. –, .
[] D. B. Ryan, G. Townsend, N. A. Gates, K. Colwell, and E.
W. Sellers, “Evaluating brain-computer interface performance
using color in the P checkerboard speller,” Clinical Neuro-
physiology,vol.,no.,pp.–,.
[] V.Guy,M.-H.Soriani,M.Bruno,T.Papadopoulo,C.Desnuelle,
and M. Clerc, “Brain computer interface with the P speller:
Usability for disabled people with amyotrophic lateral sclerosis,”
Annals of Physical and Rehabilitation Medicine,.
[] Q. Li, K. Shi, S. Ma, and N. Gao, “Improving classication accu-
racy of SVM ensemble using random training set for BCI P-
speller,” in Proceedings of the 13th IEEE International Conference
on Mechatronics and Automation, IEEE ICMA 2016,pp.–
, China, August .
Computational Intelligence and Neuroscience
[] G. C. Cawley and N. L. Talbot, “On over-tting in model selec-
tion and subsequent selection bias in performance evaluation,”
Journal of Machine Learning Research, vol. , pp. –,
.
[] D.J.Krusienski,E.W.Sellers,F.Cabestaingetal.,“Acomparison
of classication techniques for the P Speller,” Journal of Neu-
ral Engineering, vol. , no. , article , pp. –, .
[] A. Rakotomamonjy and V. Guigue, “BCI competition III:
dataset II-ensemble of SVMs for BCI P speller,” IEEE Trans-
actions on Biomedical Engineering,vol.,no.,pp.–,
.
[]Y.Yu,Z.Zhou,J.Jiangetal.,“TowardaHybridBCI:Self-
Paced Operation of a P-based Speller by Merging a Motor
Imagery-Based “Brain Switch” into a P Spelling Approach,”
International Journal of Human-Computer Interaction,vol.,
no. , pp. –, .
[] Y.Yu,J.Jiang,Z.Zhouetal.,“Aself-pacedbrain-computerinter-
face speller by combining motor imagery and P potential,”
in Proceedings of the 8th International Conference on Intelligent
Human-Machine Systems and Cyber netics, IHMSC 2016,pp.
–, China, September .
[] S. Sonnenburg, G. R¨
atsch, C. Sch¨
afer, and B. Sch¨
olkopf, “Large
scale multiple kernel learning,” Journal of Machine Learning
Research,vol.,pp.–,.
[] Q.V.Le,J.Ngiam,A.Coates,A.Lahiri,B.Prochnow,andA.Y.
Ng, “On optimization methods for deep learning,” in Proceed-
ings of the 28th International Conference on Machine Learning
(ICML ’11), pp. –, Bellevue, Wash, USA, July .
[] S. Lawrence, C. L. Giles, A. C. Tsoi, and A. D. Back, “Face
recognition: a convolutional neural-network approach,” IEEE
Transactions on Neural Networks and Learning Systems,vol.,
no.,pp.–,.
[] A.Krizhevsky,I.Sutskever,andG.E.Hinton,“Imagenetclassi-
cation with deep convolutional neural networks,” in Proceedings
of the 26th Annual Conference on Neural Information Processing
Systems (NIPS ’12), pp. –, Lake Tahoe, Nev, USA,
December .
[] P.Y.Simard,D.Steinkraus,andJ.C.Platt,“Bestpracticesfor
convolutional neural networks applied to visual document
analysis,” in Proceedings of the 7th International Conference on
Document Analysis and Recognition,vol.,pp.–,IEEE
Computer Society, Edinburgh, UK, August .
[] O. Abdel-Hamid, A.-R. Mohamed, H. Jiang, L. Deng, G. Penn,
andD.Yu,“Convolutionalneuralnetworksforspeechrecog-
nition,” IEEE Transactions on Audio, Speech and Language
Processing,vol.,no.,pp.–,.
[] Y. LeCun and Y. Bengio, “Convolutional networks for images,
speech, and time series,” e Handbook of Brain eory and
Neural Networks,vol.,no.,p.,.
[] M. S. Treder, N. M. Schmidt, and B. Blankertz, “Gaze-independ-
ent brain-computer interfaces based on covert attention and
feature attention,” Journal of Neural Engineering,vol.,no.,
Article ID , .
[] R. W. Homan, J. Herman, and P. Purdy, “Cerebral location
of international – system electrode placement,” Electroen-
cephalography and Clinical Neurophysiology,vol.,no.,pp.
–, .
[] V. Nair and G. E. Hinton, “Rectied linear units improve
Restricted Boltzmann machines,” in Proceedings of the 27th
International Conference on Machine Learning (ICML ’10),pp.
–, Haifa, Israel, June .
[] G. Benjamin, “Fractional max-pooling, ,” https://arxiv.org/
abs/..
[] G.E.Dahl,T.N.Sainath,andG.E.Hinton,“Improvingdeep
neural networks for LVCSR using rectied linear units and
dropout,” in Proceedingsofthe38thIEEEInternationalConfer-
ence on Acoustics,Speech, and Signal Processing (ICASSP ’13),pp.
–, May .
[] N. Srivastava, “Improving neural networks with dropout,” Uni-
versity of Toronto,vol.,.
[] M. Abadi, P. Barham, C. Jianmin et al., “Tensorow: A system
for large-scale machine learning,” 12th USENIX Symposium on
Operating Systems Design and Implementation,vol.,pp.–
, .
[] G. Aur´
elien, “Hands-on machine learning with scikit-learn and
tensorow: concepts, tools, and techniques to build intelligent
systems, ”.
[] Q. Huynh-u and M. Ghanbari, “Scope of validity of PSNR in
image/video quality assessment,” IEEE Electronics Letters,vol.
,no.,pp.-,.
[] R. N¨
a¨
at¨
anen, Attention and Brain Function, Psychology Press,
.
[]K.Takano,H.Ora,K.Sekihara,S.Iwaki,andK.Kansaku,
“Coherent activity in bilateral parieto-occipital cortices during
P-BCI operation,” Frontiers in Neurology,vol.,ArticleID
Article , .
[]F.A.Capati,R.P.Bechelli,andM.C.F.Castro,“Hybrid
SSVEP/P BCI keyboard: Controlled by Visual Evoked
Potential,” in Proceedings of the 9th International Conference on
Bio-Inspired Systems and Signal Processing, BIOSIGNALS 2016
- Part of 9th International Joint Conference on Biomedical
Engineering Systems and Technologies, BIOSTEC 2016,pp.–
, ita, February .
[] S. Ikegami, K. Takano, M. Wada, N. Saeki, and K. Kansaku,
“Eect of the green/blue icker matrix for P-based brain-
computer interface: An EEG-fMRI study,” Frontiers in Neurol-
ogy,.
[] W. Speier, A. Deshpande, and N. Pouratian, “A method for
optimizing EEG electrode number and conguration for signal
acquisition in P speller systems,” Clinical Neurophysiology,
vol. , no. , pp. –, .
[] C.-Y. Chen, C.-H. Chen, C.-H. Chen, and K.-P. Lin, “An auto-
matic ltering convergence method for iterative impulse noise
lters based on PSNR checking and ltered pixels detection,”
Expert Systems with Applications,vol.,pp.–,.
Available via license: CC BY 4.0
Content may be subject to copyright.
