Localization of class-related mu-rhythm desynchronization in motor imagery based brain-computer interface sessions.

Localization of class-related mu-rhythm desynchronization in motor
imagery based Brain-Computer Interface sessions
Stefan Haufe*, Ryota Tomioka, Thorsten Dickhaus, Claudia Sannelli,
Benjamin Blankertz, Guido Nolte and Klaus-Robert Mu¨ller
Abstract— We localize the sources of class-dependent event-
related desynchronisation (ERD) of the mu-rhythm related to
different types of motor imagery in Brain-Computer Interfacing
(BCI) sessions. Our approach is based on localization of single-
trial Fourier coefficients using sparse basis field expansions (S-
FLEX). The analysis reveals focal sources in the sensorimotor
cortices, a finding which can be regarded as a proof for the
expected neurophysiological origin of the BCI control signal.
As a technical contribution, we extend S-FLEX to the multiple
measurement case in a way that the activity of different
frequency bins within the mu-band is coherently localized.
I. INTRODUCTION
Brain-computer interfacing aims at providing paralyzed
patients a communication device that “reads thoughts” and
thereby obviates the need of using the usual motor pathway.
A particularly successful approach to BCI is motor imagery,
i.e., a system is controlled by a user deliberately switching
between movement imaginations of certain limbs. It is known
that for most people the associated contralateral sensorimotor
cortex becomes active already during imagination of a move-
ment. This leads to attenuation of local sensorimotor rhythms
(predominantly in the mu-range) detectable by electroen-
cephalography (EEG); a phenomenon which is also known
as event-related desynchronization (ERD) [1].
Using state-of-the-art machine learning and signal process-
ing techniques, BCI performance has increased considerably
over the last years compared to the conventional approach
of extensive subject training [2]. A key ingredient for this
success has been the development of algorithms that auto-
matically adapt to the subject’s individual physiology and
even his/her mental strategy of performing the task (e.g. [3],
[4], [5], [6]). The downside of the increased flexibility is,
however, that the experimenter can no longer fully control on
which features the BCI is trained. A careful examination of
the extracted EEG signatures is thus required in order to draw
conclusions about the physiological basis underlying a good
BCI performance. Source reconstruction methods facilitate
this validation as they link predictive features to the actual
anatomy.
This work was supported in part by BMBF grant No. 01GQ0850, DFG
grant No. MU 987/3-1 and the FP7-ICT Programme of the European
Community, under the PASCAL2 Network of Excellence, ICT-216886.
S. Haufe, T. Dickhaus, C. Sannelli, B. Blankertz and K.-R. Mu¨ller are
with Berlin Institute of Technology, Franklinstr. 28/29, D-10587 Berlin,
Germany
R. Tomioka is with the University of Tokyo, 7-3-1 Hongo, Bunkyo-ku,
Tokyo 113-8656, Japan.
G. Nolte is with Fraunhofer Institute FIRST, Kekule´str. 7, D-12489
Berlin, Germany.
* haufe@cs.tu-berlin.de
In this paper we localize the discriminative part of the
ERD occuring during different types of motor imagery,
which is used as a control signal by the Berlin Brain-
Computer Interface. Our approach is based on localizing raw
Fourier coefficients in single trials. We deploy the recently
published source reconstruction method S-FLEX [7], which
is extended to deal with multiple interrelated EEG scalp
patterns. This is used to co-localize all frequency patterns
in the subjects’ individual mu ranges, and has a noise-
suppressing effect.
II. MATERIALS AND METHODS
A. Multiple Measurement Sparse Basis Field Expansions
Assume that the EEG activity is comprised in a single
vector z = (z1, . . . , zM )> ∈ CM containing the responses
of M EEG time-series to a temporal filter. Responses are
assumed to take complex values here. Let B ⊂ R3 be the
volume covered by the brain (i.e. white and gray matter).
The current density is a vector field y : B → C3 assigning
a (complex) vectorial current source to each location in the
brain. Considering a discrete sample of locations (voxels) and
source currents (xn,y(xn) =: yn), n = 1, . . . , N , we denote
by Y = (y>1 , . . . ,y>N )> the N × 3 matrix of sources and by
vec (Y ) a column vector containing the stacked transposed
rows of Y . Instead of estimating the currents yn directly,
it was proposed to model the current density as a linear
combination of (potentially many) spatial basis fields, the
coefficients of which are to be estimated. We here consider
an expansion into Gaussians
bn,s(x) =
(√2piσs
)−3 exp
(
−
1
2 ‖x− xn‖
2
2 σ
−2
s
)
, (1)
which are smooth, but also well localized due to exponen-
tially decaying tails. Using a redundant dictionary containing
Gaussians at different centers xn an of different widths s,
it has been shown that sources with arbitrary shape can
be reconstructed by selecting the appropriate basis element.
The respective method has been coined localization through
sparse basis field expansions (S-FLEX) [7].
Given a set (dictionary) of basis functions bl, l = 1, . . . , L
The forward mapping from the sources Y to the measure-
ments z can be written in matrix form as
z = F vec (BC) , (2)
using the known lead field matrix F ∈ RM×3N . Solving
Eq. (2) for C does not yield a unique solution if the
number of coefficients is larger than the number of electrodes

M , which is the common situation. The ambiguity can be
overcome by assuming that, for an appropriately chosen
dictionary, the current density can be well approximated by a
small number of basis fields, i.e., the coefficients are sparse.
A natural choice for vectorial quantities is the so-called `1,2-
norm penalty which minimizes the (sparsity inducing) `1-
norm of the current vector amplitudes. The coefficients are
sought which provide the best compromise between sparsity
and model error, i.e.
Cˆ = arg min
C
L(C) = ‖z−Γvec (C) ‖22 +λ
L∑
l=1
‖cl‖2 (3)
where the first and second terms are the quadratic loss
function and the sparsity-inducing regularizer, respectively,
Γ ≡ FW (B ⊗ I(3×3)) ∈ RM×3SN and λ is a positive
constant controlling the tradeoff between loss function and
regularization. Given the coefficients the estimated current
density at node xn is defined by
yˆn = Wn
SN∑
l=1
cˆlbl(xn) . (4)
While Eq. (3) considers only single field patterns, we
would now like to extend S-FLEX to the localization of
multiple measurements z(t). Let Z = (z(1), . . . , z(T )) ∈
CM×T and cl(t) ∈ C3 be the coefficient vector describing
the contribution of the l-th basis field to the t-th pat-
tern. Defining cl = (cl(1)>, . . . , cl(T )>)> ∈ R3T , C =
(c1, . . . , cSN )> ∈ RSN×3T and rearranging
C˜ =



c1(1) . . . cl(T )
...
. . .
...
cSN (1) . . . cSN (T )


 ∈ R3SN×T , (5)
we propose to estimate
Cˆ = arg min
C
L∑
l=1
‖cl‖2 + λ
∥
∥
∥vec
(
Z − ΓC˜
)∥
∥
∥
2
2
(6)
which is equivalent to Eq. (3) for T = 1. However, for
T > 1 it is not equivalent to solving T problems of
type Eq. (3) separately, as in our case the 3T coefficients
belonging to a certain basis function are tied under a common
`2-norm penalty and can only be pruned to zero at the
same time. Thus, the selection of basis functions which
contribute coherently to several patterns is facilitated, while
at the same time orientations, amplitudes and phases of the
corresponding fields are allowed to differ per pattern.
Minimizing the objective function Eq. (6) is a convex
problem. Similar problems have been solved using second-
order cone programming [8], [9], [10], [11] or iteratively
reweighted least squares [12]. We here deploy a novel
approach based on augmented Lagrangians, which has been
proven to work for very large-scale instances [13].
1) Effect of Joint Localization: To illustrate the effect
of co-localization, we performed the following simple ex-
periment. A single dipolar source was placed in a cortical
region of the brain and the resulting field pattern was
computed. Ten different phase-shifted versions of the pattern
were constructed by multiplication with a random unit-
length complex number exp(iφ). Each resulting pattern was
then superimposed by equal amounts of measurement and
biological noise (signal-to-noise ratio = 1.5). Note that in this
scenario, the SNR cannot be increased by averaging, since
both signal and noise are zero-mean complex quantities.
Source localization was carried out using both the single-
and multiple-measurement variants of S-FLEX, where the
regularization constant was set using knowledge of the exact
SNR. The source estimates of all patterns were combined to
yield the estimated mean dipole amplitude.
B. Localizing mu-rhythm ERD during Motor Imagery
We consider real data from experiments recently con-
ducted within the Berlin BCI project. These experiments
originally had the purpose of screening subjects with respect
to BCI aptitude. The experimental protocol is described
in detail in our preceding publication [14], but we here
recapitulate the parts that are important for the current
investigation.
Fourty healthy BCI-naive subjects (33.6 ± 13.1 years old,
22 female) participated in the study. During the experiment,
they sat in a comfortable chair with arms and legs resting
conveniently. EEG was acquired from 119 Ag/AgCl elec-
trodes. In a calibration session, arrows pointing left, right or
down were presented on a screen and had to be responded by
five seconds of left hand, right hand or foot motor imagery,
respectively. Each arrow was presented 75 times. Trials con-
taining artifacts were discarded, as well as trials containing
real limb movements (as detected by EMG). Using heuristics,
well-discriminating contiguous post-stimulus time-intervals
and frequency bands were identified [6].
The locations of mu-ERD generators in hand- and foot
areas of the sensorimotor cortex are well known from the lit-
erature (e.g., [15], [16]). These results obtained by functional
Magnetic Resonance Imaging (fMRI) serve as a hypothesis
here to be confirmed by source localization. Since reliable
source localization can only be expected from clean EEG
signals, we restrict ourselves to cases in which EEG traces
are clearly separable. That is, for each subject we are only
interested in the two motor imagery classes showing best
classifiability, and consider only those subjects who achieved
very good BCI performance (error rates below 10%) in both
the calibration and the online session. Ten subjects fell into
that category. For nine of them, discriminability was found
in the mu-band, as expected. A tenth subject, that achieved
best BCI control using a broad band covering β- and γ-range,
was excluded from the present investigation.
We apply S-FLEX to complex Fourier coefficients, which
are calculated for each trial in the preselected band and
pre-stimulus time interval by means of an FFT. Within the
selected frequency band, five equidistantly-sampled Fourier
coefficients are taken for each subject. Localization is con-
ducted in the standard head model [17] using a source grid of
N = 6 249 dipoles (7mm inter-dipole distance). We consider
basis functions with widths σ1 = 0.75 cm, σ2 = 1 cm, σ3 =

1.25 cm and σ4 = 1.5 cm. The regularization parameter λ
is selected by means of five-fold cross-validation, i.e. by
maximizing the estimated generalization performance with
respect to fitting the EEG. We use a multiple-measurement
variant of S-FLEX, where all Fourier-coefficient patterns of
a trial are co-localized.
III. RESULTS
A. Effect of Joint Localization
Results regarding the effect of joint- as compared to
single-pattern localization are shown in Figure 1. It is ap-
parent that, while both estimated source distributions peak
similarly close to the true source location (indicated by a
red cross-hair), the multiple-measurement approach has the
advantage of being less spread-out. This shows that joint
localization effectively removes the noise-induced spatial
instability seen in single-trial estimates.
SIN
GL
E
JO
IN
T
Fig. 3. Comparison of individual (SINGLE) and joint localization of ten simulatednoisy measurements. The location of the true simulated source is indicated by a redcross-hair.
37
Fig. 1. Comparison of individual (SINGLE) and joint localization of ten
simulated noisy measurements. The location of the true simulated source is
indicated by a red cross-hair.
B. Localizing mu-rhythm ERD during Motor Imagery
The preprocessing steps performed prior to source lo-
calization are subject-specific, i.e., include filtering in in-
dividual frequency bands, the selection of individual time
intervals and class combinations, and so on. Table I lists the
heuristically-chosen frequency-bands, time-intervals and the
optimal class combinations for all subjects.
SUBJECT BAND INTERVAL CLASSES
js 10.0 – 13.0 Hz 1000 – 4250 ms LEFT/RIGHT
kp 8.0 – 12.5 Hz 1000 – 4500 ms LEFT/RIGHT
ks 8.5 – 12.5 Hz 770 – 3950 ms LEFT/RIGHT
kg 8.5 – 13.5 Hz 830 – 3440 ms LEFT/FOOT
jj 8.5 – 13.5 Hz 1170 – 3970 ms LEFT/FOOT
jl 9.5 – 13.5 Hz 1150 – 4160 ms LEFT/FOOT
jy 7.5 – 11.5 Hz 1240 – 4070 ms LEFT/FOOT
kc 9.5 – 13.5 Hz 1270 – 4000 ms LEFT/FOOT
kd 9.5 – 13.5 Hz 1340 – 4110 ms LEFT/FOOT
TABLE I
MAXIMALLY-DISCRIMINATING FREQUENCY-BANDS, TIME-INTERVALS
AND OPTIMAL CLASS COMBINATIONS FOR NINE SUBJECTS.
Figure 2 allows in-depth examination of the data of one
prototypical subject (subject kp). In the upper left part of
the figure, sensor-space spectra and ERD (calculated as
envelopes of the Hilbert-transformed signal) time courses
at electrode C4 are shown. Gray areas mark the selected
frequency-band and time-interval. Additionally, scalp maps
showing classwise-averaged Band Amplitudes (BA), as well
as per-channel discriminability between classes, are provided
(PATTERN). Band Amplitude is defined as the `2-norm of
the vector of Fourier-coefficients in the heuristically selected
frequency band, while discriminability is measured in terms
of the signed r2-value, which is defined as the signed squared
biserial correlation coefficient between class label and Band
Amplitude.
Source maps obtained from localizing Fourier coefficient
patterns are presented in the lower part of Figure 2 (LO-
CALIZATION). These maps can be regarded as the source-
space equivalents of the scalp maps depicted in the panel
PATTERN. Voxel-wise Band Amplitude in source space is
defined as in sensor-space, but it is accounted for that in
source space three Fourier-coefficients exist per voxel and
frequency. Discriminability in source space is measured in
terms of a weighted signed r2-value, where the discrim-
inability index at each voxel is normalized by the average
Band Amplitude at this voxel (taken over trials of both
classes). This is done in order to suppress high r2-values
at locations with negligible estimated activity, which are
otherwise observed as a result of the sparsity of the S-FLEX
solution.
The r2 scalp map of subject kp shows strong (oppos-
ing) contralateral ERD effects for left and right hand con-
ditions, which are maximal around C3 and C4, respec-
tively. Classwise-averaged maps of source activity as esti-
mated from Fourier coefficients show distinguished blobs
in left/right hand areas within the sensorimotor cortex for
both subjects. More such blobs are observed in other cortical
regions, in particular in frontal and occipital lobes. However,
in accordance with our expectation and the sensor-space
plots, these sources cancel out as being nondiscriminative,
and only hand areas remain in the source r2-maps.
Figure 3 provides source-space discriminablity maps for
the rest of the nine subjects considered. These plots generally
resemble those of either subject kp. For subjects js and ks,
having the optimal class combination LEFT vs. RIGHT, the
same opposing contralateral activation of hand areas as in
subject kp is observed. The rest of the subjects utilize foot
imagery, although a clear desynchronization in the central
somatosensory cortex during that condition is only found in
subject jl. In the left hand condition, all these subjects exhibit
desynchronization of both hand areas simultaneously. This
supports the hypothesis that the foot condition often serves
as a pseudo class, and some subjects may effectively achieve
one-dimensional BCI control utilizing only the presence or
absence of (left and right) hand-related ERD.
IV. CONCLUSION
The localization results obtained here confirm that the
features that discriminate best between different types of
motor imagery are spectral differences in corresponding parts
of the sensorimotor cortex. This is in agreement with [18]

sbj. kp
SPECTRUM / ENVELOPE / PATTERN
10 15 20 25 30 [Hz]5
1015
2025
3035
[AU] C4
0
0.441 r2 0 2000 4000[ms]
−5−4
−3−2
−10
12
34
[AU]C4
0
0.298 r2 BA [AU] 50 60 70 80 90 100 110 120 130
L Rleft right
± r2
−0.2 0 0.2
± r2(L,R)
L
O
C
A
L
IZ
A
T
IO
N
L
E
F
T
R
IG
H
T
D
IS
C
R
Fig. 2. Results for subject kp: Event-related desynchronization (ERD)
occurring during LEFT and RIGHT hand motor imagery conditions. Upper
part: Average poststimulus amplitude spectrum (SPECTRUM) and time
course (ENVELOPE) at C4. PATTERN: Average topographical Band Am-
plitude plots for single conditions, as well as signed r2 measure of discrim-
inability between conditions. LOCALIZATION: S-FLEX estimated source
distributions (Band Amplitude) for LEFT and RIGHT hand conditions, as
well as discriminability (DISCR) between conditions.
sb
j.
js
sb
j.
ks
sb
j.
kg
sb
j.
jj
sb
j.
jl
sb
j.
jy
sb
j.
kc
sb
j.
kg
Fig. 3. S-FLEX source-space discriminability between best (discriminating)
conditions for eight subjects. Source estimates are obtained from single-
trial localization of Fourier-coefficients. For subjects js and ks the optimal
conditions are left and right hand imagery, while for all other subjects the
optimal combination is left hand vs. foot imagery.
and previous fMRI studies. The present study hence serves
as a validation of neurophysiological significance of the
Berlin BCI feature extraction. The obtained separability in
source space raises the question whether source reconstruc-
tion methods can also be used to improve BCI accuracy.
REFERENCES
[1] G. Pfurtscheller and F. H. L. da Silva, “Event-related EEG/MEG
synchronization and desynchronization: basic principles,” vol. 110,
no. 11, pp. 1842–1857, Nov 1999.
[2] G. Dornhege, J. d. R. Milla´n, T. Hinterberger, D. McFarland, and K.-
R. Mu¨ller, Eds., Toward Brain-Computer Interfacing. Cambridge,
MA: MIT Press, 2007.
[3] C. Vidaurre, A. Schlo¨gl, R. Cabeza, R. Scherer, and G. Pfurtscheller,
“A fully on-line adaptive BCI,” IEEE Trans Biomed Eng, vol. 53, pp.
1214–1219, Jun 2006.
[4] K.-R. Mu¨ller, M. Tangermann, G. Dornhege, M. Krauledat, G. Curio,
and B. Blankertz, “Machine learning for real-time single-trial EEG-
analysis: From brain-computer interfacing to mental state monitoring.”
J. Neurosci. Meth., vol. 176, pp. 82–90, 2008.
[5] B. Blankertz, G. Dornhege, M. Krauledat, K.-R. Mu¨ller, and G. Curio,
“The non-invasive Berlin Brain-Computer Interface: Fast acquisition
of effective performance in untrained subjects,” NeuroImage, vol. 37,
no. 2, pp. 539–550, 2007.
[6] B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Mu¨ller,
“Optimizing spatial filters for robust EEG single-trial analysis,” IEEE
Signal Proc. Magazine, vol. 25, pp. 41–56, 2008.
[7] S. Haufe, V. V. Nikulin, A. Ziehe, K.-R. Mu¨ller, and G. Nolte,
“Estimating vector fields using sparse basis field expansions,” in
Advances in Neural Information Processing Systems 21, D. Koller,
D. Schuurmans, Y. Bengio, and L. Bottou, Eds. Cambridge, MA:
MIT Press, 2009, pp. 617–624.
[8] S. Haufe, V. V. Nikulin, A. Ziehe, K.-R. Mu¨ller, and G. Nolte,
“Combining sparsity and rotational invariance in EEG/MEG source
reconstruction,” NeuroImage, vol. 42, no. 2, pp. 726–738, 2008.
[9] D. Malioutov, M. C¸etin, and A. S. Willsky, “A sparse signal recon-
struction perspective for source localization with sensor arrays,” IEEE
Trans. Signal Proces., vol. 55, no. 8, pp. 3010–3022, 2005.
[10] W. Ou, M. S. Ha¨ma¨la¨inen, and P. Golland, “A distributed spatio-
temporal EEG/MEG inverse solver,” NeuroImage, vol. 44, pp. 932–
946, 2008.
[11] L. Ding and B. He, “Sparse source imaging in EEG with accurate
field modeling,” Hum. Brain Mapp., vol. 29, no. 9, pp. 1053–1067,
2008.
[12] D. Wipf and B. Rao, “An empirical bayesian strategy for solving
the simultaneous sparse approximation problem,” IEEE Trans. Signal
Proces., vol. 55, no. 7, pp. 3704–3716, 2007.
[13] R. Tomioka and M. Sugiyama, “Dual augumented lagrangian method
for efficient sparse reconstruction,” IEEE Signal Proc. Let., vol. 16,
no. 2, pp. 1067–1070, 2009.
[14] B. Blankertz, C. Sannelli, S. Halder, E. M. Hammer, A. Ku¨bler, K. R.
Mu¨ller, G. Curio, and T. Dickhaus, “Neurophysiological predictor of
SMR-based BCI performance,” NeuroImage, vol. 51, pp. 1303–1309,
Jul 2010.
[15] M. Leonardo, J. Fieldman, N. Sadato, G. Campbell, V. Ibanez,
L. Cohen, M. P. Deiber, P. Jezzard, T. Pons, R. Turner, D. L. Bihan,
and M. Hallett, “A functional magnetic-resonance-imaging study of
cortical regions associated with motor task execution and motor
ideation in humans,” Hum. Brain Mapp., vol. 3, pp. 83–92, 1995.
[16] C. A. Porro, M. P. Francescato, V. Cettolo, M. E. Diamond, P. Baraldi,
C. Zuiani, M. Bazzochi, and P. di Prampero, “Primary motor and sen-
sory cortex activation during motor performance and motor imagery:
A functional magnetic resonance imaging study,” J. Neurosci., vol. 16,
pp. 7688–7698, 1996.
[17] C. J. Holmes, R. Hoge, L. Collins, R. Woods, A. W. Toga, and
A. C. Evans, “Enhancement of MR images using registration for signal
averaging,” J. Comput. Assist. Tomogr., vol. 22, no. 2, pp. 324–333,
1998.
[18] H. Yuan, A. Doud, A. Gururajan, and B. He, “Cortical imaging
of event-related (de)synchronization during online control of brain-
computer interface using minimum-norm estimates in the frequency
domain,” IEEE Trans. Neural. Syst. Rehabil. Eng., vol. 16, pp. 425–
431, 2008.