Conference PaperPDF Available

Brain-Computer Interface adaptation for an end user to compete in the Cybathlon

October 2016

DOI:10.1109/SMC.2016.7844499

Conference: 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC 2016)
At: Budapest, Hungary

Authors:

Andreas Schwarz

FFG Austrian Research Promotion Agency

David Steyrl

University of Vienna

Gernot R Müller-Putz

Graz University of Technology

Non-invasive brain-computer interfaces (BCI) aim to assist severely motor impaired persons in their daily life routine, however only a few BCIs have made it out of the laboratory. To foster further development, the Cybathlon, an international multi-discipline tournament, has been founded. One of the disciplines is the BCI-Race, where end users control avatars in a virtual race game by their thoughts. The game supports 4 different commands which accelerate the avatar and increase the chance to win. So far, no gold standard procedure has been established on how to enable, train and individualize multi-class BCI control for users. In this work, we present a 4-stage procedure to closely tailor a multi-class BCI to an end user who will participate in the Cybathlon. In stage I, we test for basic BCI-capability, in stage II we evaluate the most suitable mental tasks for the user and in stage III, we test user compliance while perceiving feedback. Finally in stage IV, the user is playing the competition game. Our procedure provides a promising way to guide users from first contact with BCI technology to actually play a videogame by thoughts. We demonstrate the feasibility of our procedure at the pilot of the GRAZ-BCI racing team MIRAGE91. We believe that an evidence based procedure, maybe similar to the one presented in this work, is a necessity to introduce BCI technology in the daily life of potential end users.

Brainrunners, the game of the BCI competition. The game is played in a competitive manner: Up to four people compete against each other. The racetrack consists of different action fields (colored blocks) and no action fields (grey). When triggering the right command on an action block, a speed boost is enabled. False positive commands are penalized.

…

Pre-screening results: ERD/ERS maps calculated for each session for right hand and boot feet ME. Top right each, shows the cross-validation accuracy. Bottom left each, shows the confusion matrix for second 1 to 5.

…

Figures - uploaded by David Steyrl

Content may be subject to copyright.

Content uploaded by David Steyrl

Content may be subject to copyright.

Brain-Computer Interface adaptation for an end user

to compete in the Cybathlon

Andreas Schwarz, David Steyrl and Gernot R. Müller-Putz

Institute of Neural Engineering, Graz, University of Technology

Stremayrgasse 16 IV, 8010 Graz, Austria

Email:gernot.mueller@tugraz.at

Abstract—Non-invasive brain-computer interfaces (BCI) aim

to assist severely motor impaired persons in their daily life

routine, however only a few BCIs have made it out of the

laboratory. To foster further development, the Cybathlon, an

international multi-discipline tournament, has been founded. One

of the disciplines is the BCI-Race, where end users control avatars

in a virtual race game by their thoughts. The game supports 4

different commands which accelerate the avatar and increase

the chance to win. So far, no gold standard procedure has been

established on how to enable, train and individualize multi-

class BCI control for users. In this work, we present a 4-stage

procedure to closely tailor a multi-class BCI to an end user who

will participate in the Cybathlon. In stage I, we test for basic

BCI-capability, in stage II we evaluate the most suitable mental

tasks for the user and in stage III, we test user compliance while

perceiving feedback. Finally in stage IV, the user is playing the

competition game. Our procedure provides a promising way to

guide users from ﬁrst contact with BCI technology to actually

play a videogame by thoughts. We demonstrate the feasibility

of our procedure at the pilot of the GRAZ-BCI racing team

MIRAGE91. We believe that an evidence based procedure, maybe

similar to the one presented in this work, is a necessity to

introduce BCI technology in the daily life of potential end users.

I. INTRO DUC TIO N

Non-invasive brain-computer interfaces (BCI) enable its

users to interact with their environment by means of changes

in brain activity, captured for example by the electroencephalo-

gram (EEG) [1]. BCI control strategies rely either on focused

attention to external stimuli [2] or on speciﬁc mental tasks [3],

[4]. One application of BCIs aim to enable severely motor

impaired persons to control a computer and consequently

assistive devices [5], [6]. So far, only a few BCIs [7] have

made it out of the lab for use on daily life basis. Reasons are:

(i) handling, (ii) robustness (iii) and reliability of BCI system.

Case studies indicate that reliable BCI use can be trained

[3], [6]. However, according to these studies, training may

take weeks or months. To foster future (competitive) develop-

ment, the Eidgenössische Technische Hochschule Zürich (ETH

Zürich) organizes a novel competition - the Cybathlon [8]. It is

a tournament for people with severe motor impairments using

assistive prototype devices to compete against each other in

various disciplines. The competition is designed for teams.

Participating teams consist of an end user, termed the pilot,

and a number of undergraduate/graduate students, named the

tech-team. One of the disciplines is the BCI-Race, where users

have to control avatars in a virtual race game against up to 3

other pilots (see ﬁgure 1) by means of mental task based BCIs.

Controlling the game by means of a BCI is a challenging task.

The avatar in the game is constantly moving along the race

track. Three different action ﬁelds are placed on the race track

which can be triggered by a correct control command. Once

correctly triggered, the avatar of the pilot receives a speed

boost. If the wrong control command is triggered, the pilot is

penalized by slowing his avatar down for a certain time. The

game supports four different commands, which increase the

speed of the avatar. To be competitive, all 4 commands need

to be reliably controlled by the pilot. In the competition, the

user has to complete several qualiﬁcation runs and a ﬁnal race.

Fig. 1. Brainrunners, the game of the BCI competition. The game is

played in a competitive manner: Up to four people compete against each

other. The racetrack consists of different action ﬁelds (colored blocks) and no

action ﬁelds (grey). When triggering the right command on an action block,

a speed boost is enabled. False positive commands are penalized.

There are studies which already attempted robust BCI-

multiclass control for games [5], [9]. In particular, Scherer

et al. [10] showed the feasibility of a multi-class BCI system

where the user was able to navigate through a virtual envi-

ronment. In a follow up study [11], they applied the same

principle for controlling the popular multiplayer game World

of Warcraft. Both studies present results which indicate that

the Cybathlon challenge lies within possible realms. However,

no gold standard procedure to enable users control of multi-

class BCI has been established. In the following, we describe

our multi-stage procedure for individualizing and adapting

BCI technology to a severely motor impaired user. We share

our experience and demonstrate the feasibility of each stage

by means of the designated pilot of the GRAZ-BCI racing

team MIRAGE 91. The team will compete in the Cybathlon978-1-5090-1897-0/16/$31.00 c

2016 European Union

preprint

Abstract

—Non-invasive brain-computer interfaces (BCI) aim—Non-invasive brain-computer interfaces (BCI) aim

to assist severely motor impaired persons in their daily lifeto assist severely motor impaired persons in their daily life

routine, however only a few BCIs have made it out of the

laboratory. To foster further development, the Cybathlon, an

international multi-discipline tournament, has been founded. Oneinternational multi-discipline tournament, has been founded. One

of the disciplines is the BCI-Race, where end users control avatars

in a virtual race game by their thoughts. The game supports 4

different commands which accelerate the avatar and increase

final

of the disciplines is the BCI-Race, where end users control avatars

in a virtual race game by their thoughts. The game supports 4

different commands which accelerate the avatar and increasedifferent commands which accelerate the avatar and increase

the chance to win. So far, no gold standard procedure has been

established on how to enable, train and individualize multi-established on how to enable, train and individualize multi-

class BCI control for users. In this work, we present a 4-stageclass BCI control for users. In this work, we present a 4-stage

procedure to closely tailor a multi-class BCI to an end user who

will participate in the Cybathlon. In stage I, we test for basic

BCI-capability, in stage II we evaluate the most suitable mentalBCI-capability, in stage II we evaluate the most suitable mental

tasks for the user and in stage III, we test user compliance while

BCI-capability, in stage II we evaluate the most suitable mental

tasks for the user and in stage III, we test user compliance while

perceiving feedback. Finally in stage IV, the user is playing the

competition game. Our procedure provides a promising way tocompetition game. Our procedure provides a promising way to

guide users from ﬁrst contact with BCI technology to actuallyguide users from ﬁrst contact with BCI technology to actually

play a videogame by thoughts. We demonstrate the feasibility

IEEE

play a videogame by thoughts. We demonstrate the feasibilityplay a videogame by thoughts. We demonstrate the feasibility

of our procedure at the pilot of the GRAZ-BCI racing teamof our procedure at the pilot of the GRAZ-BCI racing team

MIRAGE91. We believe that an evidence based procedure, maybe

SMC

rainrunners, the game of the BCI competition.rainrunners, the game of the BCI competition.

played in a competitive manner: Up to four people compete against eachplayed in a competitive manner: Up to four people compete against each

other. The racetrack consists of different action ﬁelds (colored blocks) and noother. The racetrack consists of different action ﬁelds (colored blocks) and no

action ﬁelds (grey). When triggering the right command on an action block,

a speed boost is enabled. False positive commands are penalized.

There are studies which already attempted robust BCI-

2016

There are studies which already attempted robust BCI-There are studies which already attempted robust BCI-

multiclass control for games [5], [9]. In particular, Scherer

et al. [10] showed the feasibility of a multi-class BCI systemet al. [10] showed the feasibility of a multi-class BCI system

where the user was able to navigate through a virtual envi-where the user was able to navigate through a virtual envi-

ronment. In a follow up study [11], they applied the sameronment. In a follow up study [11], they applied the same

principle for controlling the popular multiplayer game World

of Warcraft. Both studies present results which indicate thatof Warcraft. Both studies present results which indicate that

the Cybathlon challenge lies within possible realms. However,

tournament in October 2016.

II. MET HOD S

To achieve reliable BCI control, it is essential to closely

tailor the BCI to the user. In this work, we report on our

experience with a procedure based on ﬁndings of Müller-Putz

et al. [12], [13] and Friedrich et al. [4], but also incorporate

personal experiences and ideas. The procedure consists of 4

consecutive stages (see ﬁg 2):

Pre-Screening

basic Motor Imageries

& rest Condition

BCI capability &

user compliance

small EEG setup 16 channels

Screening

Proceed?

7 Mental strategies

& rest Condition

EEG setup 32 channels

Find best combination

performance & compliance

BCI +Feedback

Best 4-class

combination

EEG setup 32 channels

Game based training

Best 4-class

combination

EEG setup 32 channels

Evaluate Tune

Test feedback &

feedback compliance REPEAT

III III IV

Fig. 2. 4 Stage training procedure: In pre-screening (stage I) the BCI

aptitude of the user is evaluated and a Go/NoGo for further training is decided.

In stage II, screening, the best 4-class combination out of a pool of mental

strategies is evaluated. Stage III tests user compliance for receiving feedback.

Based on all collected data, a closely tailored BCI is implemented. In stage

IV the user starts training with the competition game.

In stage I, we perform a pre-screening to test whether the

user is able to produce different brain patterns. This also

implies that the user is able to concentrate and is compliant

with the designated tasks. Results of the ﬁrst stage indicate

whether continuing with this user is reasonable or not. Stage

II incorporates a general screening of several mental tasks.

Friedrich et al. [4] tested several different mental tasks for

inducing changes in the EEG and found that the optimal task

is dependent on users. The main goal of this stage is not only

to ﬁnd the most effective 4 mental tasks combination, but also

a combination the user is willing to train. In stage III, the user

receives feedback based on performed mental tasks for the

ﬁrst time. This stage is implemented primarily to evaluate the

ﬁndings from stage II and to determine compliance of the user

to feedback. Based on the results of stage III, close tailoring

of the BCI is done. Modern machine-learning methods [14],

[15], which are adapted to the individual brain patterns found

and evaluated in stages I to III, are implemented. In stage IV,

we already incorporate BCI training using the actual virtual

race game which is eventually used in the competition. From

this point, the user only trains with the game in short intervals.

A. Pilot

The designated pilot of the GRAZ BCI racing team MI-

RAGE91 is a 36 year old male. In 2014, he was diagnosed with

an incomplete locked-in syndrome resulting from thrombosis

of the basilar vein which resulted in an extended stroke

of the brainstem and cerebellum (right side). At hospital

admission, the patient was almost completely paralyzed with

little motor residua of the upper extremity. During treatment,

the motor abilities increased to a point where he is able to

operate an electric wheelchair using a joystick as assistive

device. Currently, the user is vigilant and fully aware of his

environment.

B. Stage I: Pre-screening

At the beginning, we performed a pre-screening. The goal

was to evaluate whether the user is able to understand and

to perform the requested tasks, and if the tasks trigger un-

wanted side effects, like spasms or discomfort for the user.

Furthermore, we wanted to identify whether the user is able

to produce distinguishable brain patterns. Pre-screening was

done in two sessions on two different days.

1) Methods: For both sessions EEG was recorded using a

biosignal ampliﬁer equipped with 16 active electrodes (g.tec

OG, Austria). Sample rate was 512 Hz and bandpass ﬁlter

was set to 0.1 - 100 Hz (8th order butterworth ﬁlter). A notch

ﬁlter was applied (50 Hz). We covered C3, Cz and C4 with

ﬁve electrodes each in an equidistant manner (2.5cm) so that

for each position a single orthogonal (Laplacian) derivation

was possible. The remaining electrode was placed at position

AFz (see ﬁg 3). All signals were referenced to a clip on the

right ear lobe and the ground electrode was placed frontal.

C3 Cz C4

Fig. 3. Electrode layout: For pre-screening, only the 16 black-outlined

electrodes were used. The consecutive stages used all plotted electrodes)

We used the cue-guided 3 class GRAZ-BCI paradigm which

is further described here [16](see ﬁg 4). In the ﬁrst session,

we focussed on standard MI tasks, namely the imagination of

movement (motor imagery, MI) of the left hand, right hand and

both feet. We recorded 40 trials per class in 4 consecutive runs.

In the second session we changed our paradigm to two MI

classes, namely MI of right hand and both feet but incorporated

a separate rest class where the user was asked to perform no

action at all. We recorded 50 trials per class in 5 consecutive

runs.

time (s)

Fig. 4. GRAZ-BCI Paradigm: At second 0 a cross appears on the screen

followed by an auditory cue at second 2 to get the attention of the user.

At second 3 the cue is presented followed by a ﬁve second imagery period.

Depending on the cue, the user performed the designated task over the whole

imagery period.

Each analysis step was preceded by statistical outlier re-

jection (amplitude threshold, kurtosis, probability) in order to

exclude artefact contaminated trials as described in [17]. Time-

frequency maps were calculated using Laplacian derivations

preprint

experience with a procedure based on ﬁndings of Müller-Putz

et al. [12], [13] and Friedrich et al. [4], but also incorporateet al. [12], [13] and Friedrich et al. [4], but also incorporate

personal experiences and ideas. The procedure consists of 4personal experiences and ideas. The procedure consists of 4

consecutive stages (see ﬁg 2):consecutive stages (see ﬁg 2):

preprint

Pre-Screening

preprint

Screening

preprint

Proceed?

7 Mental strategies

& rest C

ondition

E E

preprint

3 3

preprint

c c

preprint

Find best combination

perform

ance & compliance

preprint

Stage training procedure:

aptitude of the user is evaluated and a Go/NoGo for further training is decided.

final

In pre-screening (stage I) the BCI

aptitude of the user is evaluated and a Go/NoGo for further training is decided.aptitude of the user is evaluated and a Go/NoGo for further training is decided.

In stage II, screening, the best 4-class combination out of a pool of mental

strategies is evaluated. Stage III tests user compliance for receiving feedback.strategies is evaluated. Stage III tests user compliance for receiving feedback.

Based on all collected data, a closely tailored BCI is implemented. In stageBased on all collected data, a closely tailored BCI is implemented. In stage

IV the user starts training with the competition game.IV the user starts training with the competition game.

In stage I, we perform a pre-screening to test whether theIn stage I, we perform a pre-screening to test whether the

user is able to produce different brain patterns. This also

In stage I, we perform a pre-screening to test whether the

user is able to produce different brain patterns. This also

implies that the user is able to concentrate and is compliant

with the designated tasks. Results of the ﬁrst stage indicatewith the designated tasks. Results of the ﬁrst stage indicate

whether continuing with this user is reasonable or not. Stagewhether continuing with this user is reasonable or not. Stage

IEEE

II incorporates a general screening of several mental tasks.II incorporates a general screening of several mental tasks.

Friedrich et al. [4] tested several different mental tasks forFriedrich et al. [4] tested several different mental tasks for

Fig. 3.Fig. 3.

electrodes were used. The consecutive stages used all plotted electrodes)electrodes were used. The consecutive stages used all plotted electrodes)

We used the cue-guided 3 class GRAZ-BCI paradigm whichWe used the cue-guided 3 class GRAZ-BCI paradigm which

is further described here [16](see ﬁg 4). In the ﬁrst session,is further described here [16](see ﬁg 4). In the ﬁrst session,

we focussed on standard MI tasks, namely the imagination of

movement (motor imagery, MI) of the left hand, right hand andmovement (motor imagery, MI) of the left hand, right hand and

both feet. We recorded 40 trials per class in 4 consecutive runs.

SMC

movement (motor imagery, MI) of the left hand, right hand and

both feet. We recorded 40 trials per class in 4 consecutive runs.both feet. We recorded 40 trials per class in 4 consecutive runs.

In the second session we changed our paradigm to two MI

classes, namely MI of right hand and both feet but incorporatedclasses, namely MI of right hand and both feet but incorporated

a separate rest class where the user was asked to perform noa separate rest class where the user was asked to perform no

action at all. We recorded 50 trials per class in 5 consecutiveaction at all. We recorded 50 trials per class in 5 consecutive

2016

time (s)

2016

At second 0 a cross appears on the screenAt second 0 a cross appears on the screen

followed by an auditory cue at second 2 to get the attention of the user.

Fig. 5. Pre-screening results: ERD/ERS maps calculated for each session for right hand and boot feet ME. Top right each, shows the cross-validation

accuracy. Bottom left each, shows the confusion matrix for second 1 to 5.

from channels C3, Cz and C4. Data was bandpass ﬁltered

between 2 and 40 Hz and segmented 3 seconds before to

5 seconds after presentation of the visual cue. Event-related

(de)synchronization (ERD/ERS) analysis [18] was performed

with respect to a speciﬁc reference interval (-2 to -1 second

before the visual cue). Statistical signiﬁcance of the the

ERD/ERS data was determined by applying a t-percentile

bootstrap algorithm with a signiﬁcance level of alpha = 0.05.

We also performed a cross-validation analysis of the recorded

data to determine class discriminability. We bandpass ﬁltered

the data from 6 to 35 Hz using a 4th order zero-phase

butterworth ﬁlter.

We applied 10 times 5 fold cross validation to the following

steps:

1) Three separate common spatial patterns ﬁlters [19] were

trained using (training) trial data from one second to four

seconds after the visual cue. Filters were calculated so

that every possible class combination (class 1 vs. 2, 1 vs.

3, 2. vs. 3) was reﬂected. We took the ﬁrst two and the

last two CSP projections and calculated 12 bandpower

features. Subsequently we applied the logarithm to the

band power features.

2) An analytic shrinkage regularized linear discriminant

analysis (sLDA) classiﬁer [20] was trained on band-

power features located 2.5 s after the visual cue.

3) CSP ﬁlter and sLDA model were applied to the data for

performance evaluation.

We calculated the confusion matrix over the feedback period

from second 4 to second 8: If the majority of the class

prediction was correct, the trial was valued as correct. In this

manner, all trials were evaluated.

2) Results: Figure 5 summarizes the results from pre-

screening session 1 and session 2.

3) Discussion: Results show distinguishable brain patterns

as can be seen in ﬁgure 5 (left each). Patterns lead to

classiﬁcation performances which both lay over chance level,

which is also conﬁrmed by the confusion matrix (ﬁgure 4, right

each). Setting and experiment did not lead to any unwanted

effects like spasms or discomfort to the user, moreover, he

was vigilant and concentrated in both sessions. However, in

the ﬁrst session, the user was quite nervous due to the novelty

of the situation. As can be seen in the ERD/ERS maps of

that session (ﬁgure 5, left), EOG artefacts are present right

after the presentation of the cue. When comparing to the

second session, the user was far more focussed on the task and

managed to suppress eye movements. Based on these results

we decided to proceed further in the training process and the

user became the ofﬁcial pilot for the team.

C. Stage II: Screening

The goal for screening was to ﬁnd a suitable combination

of 4 different classes which on the one hand promised high

classiﬁcation performance and on the other hand was in

compliance with the user.

1) Methods: Since screening incorporated not only motor

imagery tasks, we extended our EEG-setup to 32 active

electrodes positioned in an equidistant manner as displayed in

ﬁgure 3. For the actual screening we chose 7 different mental

strategies and a rest condition to be performed which were

also used in [4]:

•MI of right hand movement (HAND): repetitive squeezing a

training ball

•MI of both feet movement (FEET): planar ﬂexion/extension

of both feet

preprint

final

ERD/ERS maps calculated for each session for right hand and boot feet ME. Top right each, shows the cross-validationERD/ERS maps calculated for each session for right hand and boot feet ME. Top right each, shows the cross-validation

accuracy. Bottom left each, shows the confusion matrix for second 1 to 5.

from channels C3, Cz and C4. Data was bandpass ﬁltered

between 2 and 40 Hz and segmented 3 seconds before to

5 seconds after presentation of the visual cue. Event-related5 seconds after presentation of the visual cue. Event-related

(de)synchronization (ERD/ERS) analysis [18] was performed

IEEE

(de)synchronization (ERD/ERS) analysis [18] was performed(de)synchronization (ERD/ERS) analysis [18] was performed

with respect to a speciﬁc reference interval (-2 to -1 secondwith respect to a speciﬁc reference interval (-2 to -1 second

3) Discussion:

as can be seen in ﬁgure 5 (left each). Patterns lead toas can be seen in ﬁgure 5 (left each). Patterns lead to

classiﬁcation performances which both lay over chance level,classiﬁcation performances which both lay over chance level,

which is also conﬁrmed by the confusion matrix (ﬁgure 4, rightwhich is also conﬁrmed by the confusion matrix (ﬁgure 4, right

each). Setting and experiment did not lead to any unwantedeach). Setting and experiment did not lead to any unwanted

effects like spasms or discomfort to the user, moreover, heeffects like spasms or discomfort to the user, moreover, he

was vigilant and concentrated in both sessions. However, in

the ﬁrst session, the user was quite nervous due to the noveltythe ﬁrst session, the user was quite nervous due to the novelty

of the situation. As can be seen in the ERD/ERS maps of

SMC

the ﬁrst session, the user was quite nervous due to the novelty

of the situation. As can be seen in the ERD/ERS maps ofof the situation. As can be seen in the ERD/ERS maps of

that session (ﬁgure 5, left), EOG artefacts are present right

after the presentation of the cue. When comparing to theafter the presentation of the cue. When comparing to the

second session, the user was far more focussed on the task andsecond session, the user was far more focussed on the task and

managed to suppress eye movements. Based on these resultsmanaged to suppress eye movements. Based on these results

we decided to proceed further in the training process and the

user became the ofﬁcial pilot for the team.

2016

user became the ofﬁcial pilot for the team.

The goal for screening was to ﬁnd a suitable combinationThe goal for screening was to ﬁnd a suitable combination

of 4 different classes which on the one hand promised highof 4 different classes which on the one hand promised high

classiﬁcation performance and on the other hand was inclassiﬁcation performance and on the other hand was in

Since screening incorporated not only motor

imagery tasks, we extended our EEG-setup to 32 active

•Word association (WORD): generating as many words as

possible with a presented letter

•Mental subtraction (SUB): repeated subtraction from a given

number

•Auditory Imagery (AUDI): Imagination of singing a

song/jingle

•Spatial Navigation (SPATNAV): Imagine navigation through

your own apartment

•Mental rotation (ROT): Imagine rotating a 3 dimensional

object

•Rest(REST): Relax, but be focussed on the cross on the screen.

We displayed white icons as cue onsets in the center of the

screen. In this manner we recorded 45 trials per class (TPC)

over 9 consecutive runs.

We calculated ERD/ERS maps for each class in the same way

as already described in pre-screening. Since we wanted to

ﬁnd the 4-class combination with the highest performance, we

performed for each possible 4-class (70 in total) an analysis to

determine class discriminability. Again we bandpass ﬁltered

the data from 6 to 35 Hz using a 4th order zero-phase

butterworth ﬁlter and used a 10 times 5 fold cross validation

technique to avoid overﬁtting. We trained CSP ﬁlters on

(training) trial data from one second to three seconds after

the visual cue on every possible class combination (class 1

vs. 2, 1 vs. 3, 1 vs. 4, 2 vs. 3, 2 vs. 4, 3 vs. 4). We took

the ﬁrst two and the last two CSP projections and calculated

24 bandpower features. Additionally we took the logarithm

of the band power features. Thereafter, a sLDA classiﬁer was

trained on bandpower features located 2.5 seconds after the

visual cue.

2) Results: Table I shows peak and median accuracies of

the best ﬁve combinations. Figure 6 gives a detailed overview

of performance calculated ofﬂine over all trials.

Fig. 6. Overall performance of the mental strategy combination. The

left plot shows the overall classiﬁcation performance over all trials. Peak

performance is at 72.2.% (left, red dot). The confusion matrix was calculated

over the feedback period from second 1 to second 5: If the majority of the

class prediction was correct, the trial was valued as correct. In this manner

all trials were evaluated.

3) Discussion: We found distinguishable mental tasks. Of-

ﬂine testing of all mental task combinations showed that

HAND-FEET-SUB-REST results in highest classiﬁcation per-

TABLE I

SCR EE NI N G RE SU LTS ( BES T 5OU T O F 70 ): TH E C OM BINATI O N OF THE

TAS KS HA ND -F EET-SUB TR AC TIO N -RES T WO RK ED BES T FOR T HE US ER ,

NOT O NLY I N P EA K AC CU RA CY,BUT AL SO I N MED I AN AC C UR AC Y OVE R

TH E F EE D BAC K PE RI O D FR OM S E CO ND 4TO S EC ON D 8.

Combination Peak accuracy (%) Median (4-8s) (%)

Hand-Feet-Subtraction-Rest 75.6 66.1

Hand-Feet-Word-Rest 72.2 63.3

Hand-Feet-Spatial-Rest 68.4 56.1

Hand-Feet-Rotation-Rest 68.9 56.1

Hand-Feet-Subtraction-SpatNav 67.8 60.0

formance. In the best 5 combinations tested (out of 70) both

MI classes are always present. This is in line with the ﬁndings

of Friedrich et al [4], where motor tasks were present at

least once in each best-class combination of all their study

participants (n=8). The confusion matrix shows high accuracy

and low false positive/negative activations. Apart from the

technical performance measures, the user is also comfortable

with the combination and agreed on it for further training.

D. Stage III: First Feedback

BCI use commonly incorporates feedback. We created a

4 class BCI, using the class combination identiﬁed in the

previous stage to test user compliance for perceiving feedback.

1) Methods: We applied conﬁgurations for ampliﬁer and

electrode setup from the previous screening session. We chose

the most suitable combination (HAND-FEET-SUB-REST)

from the screening and used these classes for the feedback

session. The representative icons which are shown in the

paradigm are already color-coded with respect to the action

ﬁelds of the virtual race (grey (REST), yellow (SUB), blue

(HAND) and pink (FEET)). We provided a bar for presenting

feedback to the user. The length of the bar was controlled by

the amount of correct classiﬁcations during the last second.

We recorded 50 trials per class using the described paradigm.

Thereafter, we performed statistical outlier rejection (ampli-

tude threshold, kurtosis, probability) on the data to exclude

artefact contaminated trials [17]. Six separate CSP ﬁlters were

trained using (training) trial data from one second to four

seconds after the visual cue. Filters were calculated so that

every possible class combination was reﬂected. We took the

ﬁrst two and the last two CSP projections and calculated 24

bandpower features. Additionally we took the logarithm of

the band power features. A (sLDA) classiﬁer was trained on

bandpower features located at 2.5, 3.5 and 4.5 seconds after

the visual cue. CSP ﬁlters and sLDA model were now used

for online classiﬁcation and providing feedback to the user. In

this manner, we recorded an additional 40 trials per class.

2) Results: Figure 7 shows the overall performance of the

stage 3 experiment.

3) Discussion: In this experiment, the user received feed-

back for the ﬁrst time. Performance was above chance level

and peaked at 63.1% for online feedback and 68.4% for trial

based evaluation. In comparison to ofﬂine performance, online

preprint

•

Spatial Navigation (SPATNAV):

your own apartment

•

Mental rotation (ROT):Mental rotation (ROT):

object

•

Rest(REST):Rest(REST):

We displayed white icons as cue onsets in the center of theWe displayed white icons as cue onsets in the center of the

screen. In this manner we recorded 45 trials per class (TPC)screen. In this manner we recorded 45 trials per class (TPC)

over 9 consecutive runs.over 9 consecutive runs.

We calculated ERD/ERS maps for each class in the same way

as already described in pre-screening. Since we wanted to

ﬁnd the 4-class combination with the highest performance, weﬁnd the 4-class combination with the highest performance, we

performed for each possible 4-class (70 in total) an analysis to

determine class discriminability. Again we bandpass ﬁltered

final

performed for each possible 4-class (70 in total) an analysis to

determine class discriminability. Again we bandpass ﬁltereddetermine class discriminability. Again we bandpass ﬁltered

the data from 6 to 35 Hz using a 4th order zero-phase

butterworth ﬁlter and used a 10 times 5 fold cross validationbutterworth ﬁlter and used a 10 times 5 fold cross validation

technique to avoid overﬁtting. We trained CSP ﬁlters ontechnique to avoid overﬁtting. We trained CSP ﬁlters on

(training) trial data from one second to three seconds after

the visual cue on every possible class combination (class 1the visual cue on every possible class combination (class 1

vs. 2, 1 vs. 3, 1 vs. 4, 2 vs. 3, 2 vs. 4, 3 vs. 4). We took

the visual cue on every possible class combination (class 1

vs. 2, 1 vs. 3, 1 vs. 4, 2 vs. 3, 2 vs. 4, 3 vs. 4). We took

the ﬁrst two and the last two CSP projections and calculated

24 bandpower features. Additionally we took the logarithm24 bandpower features. Additionally we took the logarithm

of the band power features. Thereafter, a sLDA classiﬁer wasof the band power features. Thereafter, a sLDA classiﬁer was

IEEE

trained on bandpower features located 2.5 seconds after thetrained on bandpower features located 2.5 seconds after the

1) Methods:

electrode setup from the previous screening session. We choseelectrode setup from the previous screening session. We chose

the most suitable combination (HAND-FEET-SUB-REST)the most suitable combination (HAND-FEET-SUB-REST)

from the screening and used these classes for the feedbackfrom the screening and used these classes for the feedback

session. The representative icons which are shown in thesession. The representative icons which are shown in the

paradigm are already color-coded with respect to the actionparadigm are already color-coded with respect to the action

ﬁelds of the virtual race (grey (REST), yellow (SUB), blue

(HAND) and pink (FEET)). We provided a bar for presenting(HAND) and pink (FEET)). We provided a bar for presenting

feedback to the user. The length of the bar was controlled by

SMC

(HAND) and pink (FEET)). We provided a bar for presenting

feedback to the user. The length of the bar was controlled byfeedback to the user. The length of the bar was controlled by

the amount of correct classiﬁcations during the last second.

We recorded 50 trials per class using the described paradigm.We recorded 50 trials per class using the described paradigm.

Thereafter, we performed statistical outlier rejection (ampli-Thereafter, we performed statistical outlier rejection (ampli-

tude threshold, kurtosis, probability) on the data to excludetude threshold, kurtosis, probability) on the data to exclude

artefact contaminated trials [17]. Six separate CSP ﬁlters were

trained using (training) trial data from one second to fourtrained using (training) trial data from one second to four

seconds after the visual cue. Filters were calculated so that

2016

trained using (training) trial data from one second to four

seconds after the visual cue. Filters were calculated so thatseconds after the visual cue. Filters were calculated so that

every possible class combination was reﬂected. We took the

ﬁrst two and the last two CSP projections and calculated 24ﬁrst two and the last two CSP projections and calculated 24

bandpower features. Additionally we took the logarithm ofbandpower features. Additionally we took the logarithm of

the band power features. A (sLDA) classiﬁer was trained onthe band power features. A (sLDA) classiﬁer was trained on

bandpower features located at 2.5, 3.5 and 4.5 seconds after

the visual cue. CSP ﬁlters and sLDA model were now usedthe visual cue. CSP ﬁlters and sLDA model were now used

for online classiﬁcation and providing feedback to the user. In

Fig. 7. Overall performance of the mental strategy combination. The left

plot shows the classiﬁcation performance of the trials with feedback. Peak

performance is at 63.1.% (left, red dot). The confusion matrix was calculated

over the feedback period from second 1 to second 5.

results were lower, however the user was fond of receiving

feedback and was eager to continue BCI use. Overall user

compliance was high. He was focussed on the tasks, avoided

eye movements (blinks) and swallowing as much as possible.

E. Stage IV: BCI Game

The feedback experiment was conducted in a cue-based

way, with abstract and simpliﬁed feedback. Games however,

cause distractions which can inﬂuence BCI performance nega-

tively. Therefore, we switched over to the actual game to allow

our pilot to accustom to the game.

1) Methods: The experiment consisted of two parts. First,

collecting training trials and second, playing the game. In

training mode, the game sends triggers over UDP to mark

game events, like entering new action ﬁelds. The pilot was

instructed to start with his mental imagery as soon as the

avatar enters a new action ﬁeld. Due to these markers, we

were able to assign periods of mental imagery to EEG.

These EEG periods were used to train the algorithms of the

BCI. In detail, EEG of second 1 to second 3 after a new

action ﬁeld marker was cut out and used to train CSP ﬁlters.

Classiﬁer training was performed with data at second 3. The

BCI itself contained the following signal processing steps: (i)

band pass ﬁltering, 2 bands: 8-16Hz and 16-30Hz to separate

alpha and beta band activity. (ii) CSP ﬁltering with one

separate CSP ﬁlter per band and per binary task combination,

resulting in 12 CSP ﬁlters (6 possible binary combinations

of the four mental imageries and 2 frequency bands). Four

CSP channels per CSP ﬁlter were selected: 2 related to the

highest eigenvalues and 2 related to the lowest eigenvalues,

resulting in 48 CSP channels in total. (iii) Squaring, logarithm

calculation, and subsequent averaging over a sliding window

of one second was performed in the next stage. (iv) 48

logarithmic bandpower values were fed to a multi class sLDA

classiﬁer which calculated class probabilities. (v) Smoothed

class probabilities were compared to thresholds. If a class

probability exceeded the threshold, a command was sent

to the game. (vi) The game reaction was the feedback to

the pilot. For training data collection, we recorded 50 TPC

in 5 consecutive training runs (10 TPC each) where the

user only performed the mental tasks, but no actions were

triggered. Detailed explanation of this procedure is depicted

in ﬁgure 9. Thereafter, BCI models were calculated and the

user performed 4 regular runs including the full spectrum of

control options.

2) Results: Figure 8 shows the performance results of 4

runs where the user played the game.

Fig. 8. First results of the game based training: (A) User-decision based

confusion matrix. In 4 runs, the user triggered an action 101 times. (B)

Confusion matrix of all possible decisions.

Fig. 9. Brainrunners training paradigm: For data collection the user was

instructed to perform the mental task on the ﬁrst half of the action ﬁeld and

relax on the second half. In this manner the user performed 5 seconds of the

designated mental task and had 5 seconds until the next mental task.

We performed analysis of the runs where the user was

actually playing the game. (A) At ﬁrst we investigated all the

user triggered events in a confusion matrix. Whenever the user

triggered the corrected event on the respective action ﬁeld, we

valued it as a true positive. If the user did not trigger an event

on a non-action ﬁeld (grey ﬁelds), we also counted it as a true

positive. In this analysis we did not include action ﬁelds where

the user did not trigger an action but merely ran over. (B) In a

second analysis we also included ﬁelds where the user should

have triggered action ﬁelds but did not trigger them.

3) Discussion: In this experiment, the user played the

Brainrunners game for the ﬁrst time. 82.2% of all triggered

commands were correct but there is room for improvement:

preprint

verall performance of the mental strategy combination.

plot shows the classiﬁcation performance of the trials with feedback. Peakplot shows the classiﬁcation performance of the trials with feedback. Peak

performance is at 63.1.% (left, red dot). The confusion matrix was calculated

over the feedback period from second 1 to second 5.

performance is at 63.1.% (left, red dot). The confusion matrix was calculatedperformance is at 63.1.% (left, red dot). The confusion matrix was calculated

over the feedback period from second 1 to second 5.

final

over the feedback period from second 1 to second 5.

results were lower, however the user was fond of receivingresults were lower, however the user was fond of receiving

feedback and was eager to continue BCI use. Overall userfeedback and was eager to continue BCI use. Overall user

compliance was high. He was focussed on the tasks, avoided

eye movements (blinks) and swallowing as much as possible.

The feedback experiment was conducted in a cue-based

way, with abstract and simpliﬁed feedback. Games however,way, with abstract and simpliﬁed feedback. Games however,

cause distractions which can inﬂuence BCI performance nega-

IEEE

cause distractions which can inﬂuence BCI performance nega-cause distractions which can inﬂuence BCI performance nega-

tively. Therefore, we switched over to the actual game to allowtively. Therefore, we switched over to the actual game to allow

Fig. 8.Fig. 8.

confusion matrix. In 4 runs, the user triggered an action 101 times.confusion matrix. In 4 runs, the user triggered an action 101 times.

Confusion matrix of all possible decisions.Confusion matrix of all possible decisions.

SMC

rainrunners training paradigm:rainrunners training paradigm:

For data collection the user was

instructed to perform the mental task on the ﬁrst half of the action ﬁeld and

relax on the second half. In this manner the user performed 5 seconds of the

designated mental task and had 5 seconds until the next mental task.

2016

relax on the second half. In this manner the user performed 5 seconds of the

designated mental task and had 5 seconds until the next mental task.designated mental task and had 5 seconds until the next mental task.

We performed analysis of the runs where the user wasWe performed analysis of the runs where the user was

At ﬁrst we investigated all theAt ﬁrst we investigated all the

user triggered events in a confusion matrix. Whenever the useruser triggered events in a confusion matrix. Whenever the user

triggered the corrected event on the respective action ﬁeld, we

valued it as a true positive. If the user did not trigger an eventvalued it as a true positive. If the user did not trigger an event

on a non-action ﬁeld (grey ﬁelds), we also counted it as a true

the user only triggered around 60% of all possible action

ﬁelds. It was hard for the user to trigger HAND and FEET

action ﬁelds, however he robustly triggered SUB ﬁelds and

was able to generate a stable REST condition. Figure 8 shows

the corresponding confusion matrices. User compliance was

high. He was focussed on the task and tried to avoid producing

artefacts of any kind.

III. CO NC LUS ION & OUTL OOK

In this work, we presented our multi-stage procedure for

individualizing and adapting BCI technology to a severely

motor impaired user. Single stages of our procedure have

resemblance to or are based on already published procedures

[12], [13], [6], [4]. Nevertheless our procedure provides a

promising way to guide users from ﬁrst contact with BCI

technology to actually play a videogame by thoughts. When

aiming for BCI control, one has to be certain that the user

understands instructions. The results of our ﬁrst stage indicate

that the pilot is not only able to understand the instructions

but was also able to create distinct ERD/ERS patterns for

different mental tasks. Patterns were more pronounced and

performance was also higher in the second session of pre-

screening. Therefore we favour to perform more than one

pre-screening session before deciding to continue training.

Past works show [4] that mental task combinations perform

differently for each user. We started screening eight different

mental tasks and found that classiﬁcation performance varied

widely for different task combinations. From our experience,

users can be distracted by received feedback. Results from

stage III suggest that the user was able to maintain the mental

task while receiving feedback, although his performance was

lower than in ofﬂine analysis. The transition from an abstract

BCI-paradigm to a game-based paradigm is complex. Active

game environments provide multimodal visual and auditory

distractions, which change brain patterns and therefore in-

ﬂuence BCI performance negatively. Nevertheless, we are

conﬁdent that once the user gets more familiar with the game

and establishes a training routine, these distractions will take

less inﬂuence. In the upcoming weeks and months we will

repeat game-based training in short intervals. We hope that

our procedure enables the pilot of the GRAZ-BCI racing

team MIRAGE91 to successfully participate in the Cybathlon.

In conclusion, we believe that an evidence based procedure,

maybe similiar to the one presented in this work, is a necessity

to introduce BCI technology in the daily life of potential end

users.

ACKNOWL E DG MEN T

This project was partly supported by: VAMED, City of

Graz, Graz University of Technology, Institue of Neural Engi-

neering, Swietelsky Tunnelbau GmbH, and the Horizon 2020

Project MoreGrasp (No. 643955). The GRAZ-BCI racing team

MIRAGE91: Maria Höller, Karina Statthaler, Marvin Bigga,

Johannes Steininger, Dominik Narnhofer, Lydia Lindner, Lea

Hehenberger, Julia Brandstetter, Markus ’Pikachu’ Adamek,

Bernhard Frohner and Reinmar Kobler.

REF ERE NCE S

[1] J. R. Wolpaw, N. Birbaumer, and D. J. M. et al., “Brain-computer

interfaces for communication and control,” Clinical Neurophysiology,

vol. 113, pp. 767–791, 2002.

[2] S. Halder, A. Pinegger, I. Käthner, S. Wriessnegger, and e. a. J. Faller,

J.B.P., “Brain-controlled applications using dynamic p300 speller matri-

ces,” Artiﬁcial Intelligence in Medicine, vol. 63, no. 1, pp. 7–17, 2015.

[3] G. Pfurtscheller, C. Guger, and G. M. et al., “Brain oscillations control

hand orthosis in a tetraplegic,” Neuroscience Letters, vol. 292, pp. 211–

214, 2000.

[4] E. Friedrich, C. Neuper, and R. Scherer, “Whatever works: A systematic

user-centered training protocol to optimize brain-computer interfacing

individually,” PLOS ONE, vol. 8, no. 9, 2013.

[5] JR.Millan, R. Rupp, G. Müller-Putz, R. Murray-Smith, G. Claudio,

M. Tangermann, C. Vidaurre, F. Cincotti, A. Kübler, R. Leeb, C. Neuper,

K. Müller, and D. Mattia, “Combining brain-computer interfaces and

assistive technologies: State-of-the-art and challenges,” Frontiers in

Neuroscience, vol. 4, no. 161, 2010.

[6] M. Rohm, M. Schneiders, C. Müller, A., V. Kaiser, G. R. Müller-

Putz, and R. Rupp, “Hybrid brain-computer interfaces and hybrid

neuroprostheses for restoration of upper limb functions in individuals

with high-level spinal cord injury,” Artiﬁcial Intelligence in Medicine,

vol. 59, no. 2, pp. 133 – 142, 2013.

[7] E. Holz, L. Brotel, T. Kaufmann, and A. Kübler, “Long-term inde-

pendent brain-computer interface home use improves quality of life of

a patient in the locked-in state: A case study,” Archives of physical

medicine and rehabilitation, vol. 96, no. 3, pp. 16–26, 2015.

[8] R. Riener and R. Kundert. (2016, April) Cybathlon championship

for athletes with disabilities. Website. ETH Zürich, Professur

f. Sensomotorische Syst. Tannenstrasse 1,8092 Zürich,Switzerland.

[Online]. Available: http://www.cybathlon.ethz.ch/

[9] R. Scherer, M. Billinger, and J. W. et al., “Thought-based row-column

scanning communication board for individuals with cerebral palsy,”

Annals of Physical and Rehabilitation Medicine, 2014.

[10] R. Scherer, F. Lee, and A. S. et al., “Toward self-paced brain-computer

communication: navigation through virtual worlds,” IEEE Transactions

on Biomedical Engineering, vol. 55, pp. 675–682, 2008.

[11] R. Scherer, J. Faller, D. Balderas, E. Friedrich, M. Pröll, B. Allison, and

G. Müller-Putz, Brain-Computer interfacing: more than the sum of its

parts, ser. Soft Computing - A Fusion of Foundations, Methodologies

and Applications. Springer, 2013, vol. 17, no. 2.

[12] G. Müller-Putz, R. Scherer, G. Pfurtscheller, and R. Rupp, “Eeg-based

neuroprosthesis control: A step towards clinical practice„” Neuroscience

Letters, vol. 382, no. 1-2, 1-8, pp. 169–174, 2005.

[13] G. Müller-Putz, R. Scherer, and G. P. et al., “Temporal coding of brain

patterns for direct limb control in humans,” Frontiers in Neuroscience,

vol. 4, no. 34, 2010.

[14] A. Schwarz, R. Scherer, D. Steyrl, J. Faller, and G. Müller-Putz,

“A co-adaptive sensory motor rhythms brain-computer interface based

on common spatial patterns and random forest,” in Engineering in

Medicine and Biology Society (EMBC), 2015 37th Annual International

Conference of the IEEE, Aug 2015, pp. 1049–1052.

[15] D. Steyrl, R. Scherer, and J. F. et al., “Random forests in non-

invasive sensorimotor rhythm brain-computer interface: A practical and

convenient non-linear classiﬁer,” Biomedical engineering, 2015.

[16] G. Pfurtscheller and C. Neuper, “Motor imagery and direct brain-

computer communication,” Proceedings of the IEEE, vol. 89, pp. 1123–

1134, 2001.

[17] J. Faller, C. Vidaurre, and T. S.-E. et al., “Autocalibration and recurrent

adaptation: Towards a plug and play online ERD-BCI,” IEEE Transac-

tions on Neural Systems Rehabilitation Engineering, vol. 20, no. 3, pp.

313–319, May 2012.

[18] B. Graimann, J. Huggins, S. Levine, and G. Pfurtscheller, “Visualization

of signiﬁcant erd/ers patterns in multichannel eeg and ecog data,”

Clinical Neurophysiology, vol. 113, no. 1, pp. 43–47, 2002.

[19] H. Ramoser, J. Müller-Gerking, and G. Pfurtscheller, “Optimal spatial

ﬁltering of single trial EEG during imagined hand movement,” IEEE

Transactions on Rehabilitation Engineering, vol. 8, pp. 441–446, 2000.

[20] B. Blankertz, S. Lemm, M. Treder, S. Haufe, and K. Müller, “Single-trial

analysis and classiﬁcation of erp components - a tutorial,” Neuroimage,

vol. 56, no. 11, pp. 814–825, 2011.

preprint

high. He was focussed on the task and tried to avoid producing

artefacts of any kind.artefacts of any kind.

In this work, we presented our multi-stage procedure forIn this work, we presented our multi-stage procedure for

individualizing and adapting BCI technology to a severelyindividualizing and adapting BCI technology to a severely

motor impaired user. Single stages of our procedure havemotor impaired user. Single stages of our procedure have

resemblance to or are based on already published proceduresresemblance to or are based on already published procedures

[12], [13], [6], [4]. Nevertheless our procedure provides a[12], [13], [6], [4]. Nevertheless our procedure provides a

promising way to guide users from ﬁrst contact with BCI

technology to actually play a videogame by thoughts. Whentechnology to actually play a videogame by thoughts. When

aiming for BCI control, one has to be certain that the user

understands instructions. The results of our ﬁrst stage indicate

that the pilot is not only able to understand the instructions

final

understands instructions. The results of our ﬁrst stage indicate

that the pilot is not only able to understand the instructions

but was also able to create distinct ERD/ERS patterns for

different mental tasks. Patterns were more pronounced anddifferent mental tasks. Patterns were more pronounced and

performance was also higher in the second session of pre-

screening. Therefore we favour to perform more than one

pre-screening session before deciding to continue training.pre-screening session before deciding to continue training.

Past works show [4] that mental task combinations perform

pre-screening session before deciding to continue training.

Past works show [4] that mental task combinations perform

differently for each user. We started screening eight different

mental tasks and found that classiﬁcation performance variedmental tasks and found that classiﬁcation performance varied

widely for different task combinations. From our experience,

IEEE

widely for different task combinations. From our experience,widely for different task combinations. From our experience,

users can be distracted by received feedback. Results fromusers can be distracted by received feedback. Results from

[10] R. Scherer, F. Lee, and A. S. et al., “Toward self-paced brain-computer[10] R. Scherer, F. Lee, and A. S. et al., “Toward self-paced brain-computer

communication: navigation through virtual worlds,”communication: navigation through virtual worlds,”

on Biomedical Engineeringon Biomedical Engineering

[11] R. Scherer, J. Faller, D. Balderas, E. Friedrich, M. Pröll, B. Allison, and[11] R. Scherer, J. Faller, D. Balderas, E. Friedrich, M. Pröll, B. Allison, and

G. Müller-Putz,G. Müller-Putz,

parts

, ser. Soft Computing - A Fusion of Foundations, Methodologies, ser. Soft Computing - A Fusion of Foundations, Methodologies

and Applications. Springer, 2013, vol. 17, no. 2.and Applications. Springer, 2013, vol. 17, no. 2.

[12] G. Müller-Putz, R. Scherer, G. Pfurtscheller, and R. Rupp, “Eeg-based[12] G. Müller-Putz, R. Scherer, G. Pfurtscheller, and R. Rupp, “Eeg-based

neuroprosthesis control: A step towards clinical practice„”

LettersLetters

, vol. 382, no. 1-2, 1-8, pp. 169–174, 2005., vol. 382, no. 1-2, 1-8, pp. 169–174, 2005.

[13] G. Müller-Putz, R. Scherer, and G. P. et al., “Temporal coding of brain[13] G. Müller-Putz, R. Scherer, and G. P. et al., “Temporal coding of brain

patterns for direct limb control in humans,”

SMC

, vol. 382, no. 1-2, 1-8, pp. 169–174, 2005.

[13] G. Müller-Putz, R. Scherer, and G. P. et al., “Temporal coding of brain

patterns for direct limb control in humans,”patterns for direct limb control in humans,”

vol. 4, no. 34, 2010.

[14] A. Schwarz, R. Scherer, D. Steyrl, J. Faller, and G. Müller-Putz,[14] A. Schwarz, R. Scherer, D. Steyrl, J. Faller, and G. Müller-Putz,

“A co-adaptive sensory motor rhythms brain-computer interface based“A co-adaptive sensory motor rhythms brain-computer interface based

on common spatial patterns and random forest,” inon common spatial patterns and random forest,” in

Medicine and Biology Society (EMBC), 2015 37th Annual InternationalMedicine and Biology Society (EMBC), 2015 37th Annual International

, Aug 2015, pp. 1049–1052.

[15] D. Steyrl, R. Scherer, and J. F. et al., “Random forests in non-

invasive sensorimotor rhythm brain-computer interface: A practical andinvasive sensorimotor rhythm brain-computer interface: A practical and

Biomedical engineering

2016

invasive sensorimotor rhythm brain-computer interface: A practical and

Biomedical engineeringBiomedical engineering

[16] G. Pfurtscheller and C. Neuper, “Motor imagery and direct brain-

Proceedings of the IEEEProceedings of the IEEE

, vol. 89, pp. 1123–, vol. 89, pp. 1123–

[17] J. Faller, C. Vidaurre, and T. S.-E. et al., “Autocalibration and recurrent[17] J. Faller, C. Vidaurre, and T. S.-E. et al., “Autocalibration and recurrent

adaptation: Towards a plug and play online ERD-BCI,”adaptation: Towards a plug and play online ERD-BCI,”

IEEE Transac-IEEE Transac-

, vol. 20, no. 3, pp.

[18] B. Graimann, J. Huggins, S. Levine, and G. Pfurtscheller, “Visualization[18] B. Graimann, J. Huggins, S. Levine, and G. Pfurtscheller, “Visualization

of signiﬁcant erd/ers patterns in multichannel eeg and ecog data,”

Long Multi-Stage Training for a Motor-Impaired User in a BCI Competition

Article

Full-text available

Mar 2021
FRONT HUM NEUROSCI

In a Mental Imagery Brain-Computer Interface the user has to perform a specific mental task that generates electroencephalography (EEG) components, which can be translated in commands to control a BCI system. The development of a high-performance MI-BCI requires a long training, lasting several weeks or months, in order to improve the ability of the user to manage his/her mental tasks. This works aims to present the design of a MI-BCI combining mental imaginary and cognitive tasks for a severely motor impaired user, involved in the BCI race of the Cybathlon event, a competition of people with severe motor disability. In the BCI-race, the user becomes a pilot in a virtual race game against up to three other pilots, in which each pilot has to control his/her virtual car by his/her mental tasks. We present all the procedures followed to realize an effective MI-BCI, from the user's first contact with a BCI technology to actually controlling a video-game through her EEG. We defined a multi-stage user-centered training protocol in order to successfully control a BCI, even in a stressful situation, such as that of a competition. We put a specific focus on the human aspects that influenced the long training phase of the system and the participation to the competition.

Direct comparison of supervised and semi-supervised retraining approaches for co-adaptive BCIs

Article

Full-text available

Sep 2019

For Brain-Computer interfaces (BCIs), system calibration is a lengthy but necessary process for successful operation. Co-adaptive BCIs aim to shorten training and imply positive motivation to users by presenting feedback already at early stages: After just 5 min of gathering calibration data, the systems are able to provide feedback and engage users in a mutual learning process. In this work, we investigate whether the retraining stage of co-adaptive BCIs can be adapted to a semi-supervised concept, where only a small amount of labeled data is available and all additional data needs to be labeled by the BCI itself. The aim of the current work was to evaluate whether a semi-supervised co-adaptive BCI could successfully compete with a supervised co-adaptive BCI model. In a supporting two-class (190 trials per condition) BCI study based on motor imagery tasks, we evaluated both approaches in two separate groups of 10 participants online, while we simulated the other approach in each group offline. Our results indicate that despite the lack of true labeled data, the semi-supervised driven BCI did not perform significantly worse (p > 0.05) than the supervised counterpart. We believe that these findings contribute to developing BCIs for long-term use, where continuous adaptation becomes imperative for maintaining meaningful BCI performance.

Data-efficient Motor Imagery Decoding in Real-time for the Cybathlon Brain-Computer Interface Race

Conference Paper

Full-text available

Jan 2018

Compact Convolutional Neural Networks for Multi-Class, Personalised, Closed-Loop EEG-BCI

Preprint

Jul 2018

For many people suffering from motor disabilities, assistive devices controlled with only brain activity are the only way to interact with their environment. Natural tasks often require different kinds of interactions, involving different controllers the user should be able to select in a self-paced way. We developed a Brain-Computer Interface (BCI) allowing users to switch between four control modes in a self-paced way in real-time. Since the system is devised to be used in domestic environments in a user-friendly way, we selected non-invasive electroencephalographic (EEG) signals and convolutional neural networks (CNNs), known for their ability to find the optimal features in classification tasks. We tested our system using the Cybathlon BCI computer game, which embodies all the challenges inherent to real-time control. Our preliminary results show that an efficient architecture (SmallNet), with only one convolutional layer, can classify 4 mental activities chosen by the user. The BCI system is run and validated online. It is kept up-to-date through the use of newly collected signals along playing, reaching an online accuracy of 47.6% where most approaches only report results obtained offline. We found that models trained with data collected online better predicted the behaviour of the system in real-time. This suggests that similar (CNN based) offline classifying methods found in the literature might experience a drop in performance when applied online. Compared to our previous decoder of physiological signals relying on blinks, we increased by a factor 2 the amount of states among which the user can transit, bringing the opportunity for finer control of specific subtasks composing natural grasping in a self-paced way. Our results are comparable to those shown at the Cybathlon's BCI Race but further improvements on accuracy are required.

Highlights and Interviews with Winners

Chapter

Oct 2020

The preceding nine chapters in this book presented an introduction and summaries of eight projects that were nominated for a BCI Research Award in 2018. In this chapter, we summarize the 2018 Awards Ceremony where we announced the three winning projects. We interviewed authors of these winning projects – Drs. Ajiboye, Tangermann, and Herff – and then wrote a conclusion with future directions, including BCI Hackathons and Cybathlons. We hope these chapters have been informative and helpful, and may have even helped to spark some new ideas.

B2CI 2019: The IEEE Brain to Computer Interface Competition’s Gaming Event

Conference Paper

Aug 2019

Unimanual and Bimanual Reach-and-Grasp Actions Can Be Decoded From Human EEG

Article

Full-text available

Sep 2019

While most tasks of daily life can be handled through a small number of different grasps, many tasks require the action of both hands. In these bimanual tasks, the second hand has either a supporting role (e.g. for fixating a jar) or a more active role (e.g. grasping a pot on both handles). In this study we attempt to discriminate the neural correlates of unimanual (performed with left and right hand) from bimanual reach-and grasp actions using the low-frequency time-domain electroencephalogram (EEG). In a self-initiated movement task, 15 healthy participants were asked to perform unimanual (palmar and lateral grasps with left and right hand) and bimanual (double lateral, mixed palmar/lateral) reach-and-grasps on objects of daily life. Using EEG time-domain features in the frequency range of 0.3-3 Hz, we achieved multiclass-classification accuracies of 38.6 ± 6.6% (7 classes, 17.1% chance level) for a combination of 6 movements and 1 rest condition. The grand average confusion matrix shows highest true positive rates (TPR) for the rest (63%) condition while TPR for the movement classes varied between 33 to 41%. The underlying movement-related cortical potentials (MRCPs) show significant differences between unimanual (e.g. left hand vs. right hand grasps) as well unimanual vs. bimanual conditions which both can be attributed to lateralization effects. We believe that these findings can be exploited and further used for attempts in providing persons with spinal cord injury a form of natural control for bimanual neuroprostheses.

Improving the quality of the electroencephalogram simultaneously recorded with functional magnetic resonance imaging

Thesis

Full-text available

Nov 2018

David Steyrl

The concurrent recording of the electroencephalogram (EEG) with functional magnetic resonance imaging (fMRI) allows the simultaneous study of the electrophysiology, the blood oxygen level dependent signal, and particularly also their interplay. However, the EEG is affected by a large number of fMRI-related, partly repetitive, artifacts. Average artifact subtraction (AAS) – the most frequently used artifact reduction technique – computes artifact templates from artifact repetitions and subtracts them from the EEG. This effectively reduces repetitive, invariant artifacts, but serious artifact residuals remain. Therefore, this thesis pursued two objectives: analysis of the artifact residuals and development of a new technique for the reduction of the residuals. The inherent variability of artifacts is known to cause residuals after the AAS method, because the subtraction template does not fit the actual artifact. In this thesis, an additional cause of artifact residuals was identified. An intrinsic vulnerability of the AAS technique to correlated artifacts leads to artifact contaminated subtraction templates and consequently to artifact residuals in the EEG. The new artifact reduction technique uses recordings of artifact residuals from a reference-layer EEG cap combined with adaptive filtering to remove the residuals from the EEG and is referred to as reference-layer adaptive filtering (RLAF). The RLAF method is highly effective in offline and online application scenarios. It improves the signal-to-noise ratio as well as the classification accuracy of physiological EEG components substantially. The RLAF technique’s ability to reduce all kinds of artifact residuals – including non-stationary and varying components – in combination with its easy handling, makes it a candidate for a future gold standard method.

Compact Convolutional Neural Networks for Multi-Class, Personalised, Closed-Loop EEG-BCI

Conference Paper

Aug 2018

BCI and Games: Playful, Experience-Oriented Learning by Vivid Feedback?

Chapter

Jan 2018

Play, that is, self-motivated activities for enjoyment, is a significant aspect for human devel- opment and essential to learning and skill acquisition. Games, the structured form of play, are increasingly being used in brain–computer interface (BCI) and neurofeedback (NF) applications. In BCI and NF applications, patterns of the users’ brain activation are assessed in real time and fed back to the users. When users become successful in modulating their own brain activation, improvements in behavior, cognition, or motor function follow or they are able to control external devices such as a computer, wheelchair, or neuroprosthe- sis. In electroencephalogram-based applications, however, a large number of users cannot attain control over their own brain signals. Current approaches to attaining control require lengthy repetitive trainings. The use of games and game-like feedback aims at keeping user motivation and engagement high over time. This chapter provides an overview of existing game-like feedback modalities and critically discusses their potential value and also possible drawbacks in BCI and NF applications.

Brain-computer interfaces - Communication with the P300

Article

Full-text available

Jan 2005

A Co-adaptive Sensory Motor Rhythms Brain-Computer Interface Based on Common Spatial Patterns and Random Forest

Conference Paper

Full-text available

Aug 2015

Sensorimotor rhythm (SMR) based Brain- Computer Interfaces (BCI) typically require lengthy user training. This can be exhausting and fatiguing for the user as data collection may be monotonous and typically without any feedback for user motivation. Hence new ways to reduce user training and improve performance are needed. We recently introduced a two class motor imagery BCI system which continuously adapted with increasing run-time to the brain patterns of the user. The system was designed to provide visual feedback to the user after just five minutes. The aim of the current work was to improve user-specific online adaptation, which was expected to lead to higher performances. To maximize SMR discrimination, the method of filter-bank common spatial patterns (fbCSP) and Random Forest (RF) classifier were combined. In a supporting online study, all volunteers performed significantly better than chance. Overall peak accuracy of 88.6 +- 6.1 (SD) % was reached, which significantly exceeded the performance of our previous system by 13%. Therefore, we consider this system the next step towards fully auto-calibrating motor imagery BCIs.

Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: A practical and convenient non-linear classifier

Article

Full-text available

Apr 2015
BIOMED TECH

There is general agreement in the brain-computer interface (BCI) community that although non-linear classifiers can provide better results in some cases, linear classifiers are preferable. Particularly, as non-linear classifiers often involve a number of parameters that must be carefully chosen. However, new non-linear classifiers were developed over the last decade. One of them is the random forest (RF) classifier. Although popular in other fields of science, RFs are not common in BCI research. In this work, we address three open questions regarding RFs in sensorimotor rhythm (SMR) BCIs: parametrization, online applicability, and performance compared to regularized linear discriminant analysis (LDA). We found that the performance of RF is constant over a large range of parameter values. We demonstrate - for the first time - that RFs are applicable online in SMR-BCIs. Further, we show in an offline BCI simulation that RFs statistically significantly outperform regularized LDA by about 3%. These results confirm that RFs are practical and convenient non-linear classifiers for SMR-BCIs. Taking into account further properties of RFs, such as independence from feature distributions, maximum margin behavior, multiclass and advanced data mining capabilities, we argue that RFs should be taken into consideration for future BCIs.

Whatever Works: A Systematic User-Centered Training Protocol to Optimize Brain-Computer Interfacing Individually

Article

Full-text available

Sep 2013
PLOS ONE

This study implemented a systematic user-centered training protocol for a 4-class brain-computer interface (BCI). The goal was to optimize the BCI individually in order to achieve high performance within few sessions for all users. Eight able-bodied volunteers, who were initially naïve to the use of a BCI, participated in 10 sessions over a period of about 5 weeks. In an initial screening session, users were asked to perform the following seven mental tasks while multi-channel EEG was recorded: mental rotation, word association, auditory imagery, mental subtraction, spatial navigation, motor imagery of the left hand and motor imagery of both feet. Out of these seven mental tasks, the best 4-class combination as well as most reactive frequency band (between 8-30 Hz) was selected individually for online control. Classification was based on common spatial patterns and Fisher's linear discriminant analysis. The number and time of classifier updates varied individually. Selection speed was increased by reducing trial length. To minimize differences in brain activity between sessions with and without feedback, sham feedback was provided in the screening and calibration runs in which usually no real-time feedback is shown. Selected task combinations and frequency ranges differed between users. The tasks that were included in the 4-class combination most often were (1) motor imagery of the left hand (2), one brain-teaser task (word association or mental subtraction) (3), mental rotation task and (4) one more dynamic imagery task (auditory imagery, spatial navigation, imagery of the feet). Participants achieved mean performances over sessions of 44-84% and peak performances in single-sessions of 58-93% in this user-centered 4-class BCI protocol. This protocol is highly adjustable to individual users and thus could increase the percentage of users who can gain and maintain BCI control. A high priority for future work is to examine this protocol with severely disabled users.

Autocalibration and Recurrent Adaptation: Towards a Plug and Play Online ERD-BCI

Article

Full-text available

Apr 2012
IEEE T NEUR SYS REH

System calibration and user training are essential for operating motor imagery based brain-computer interface (BCI) systems. These steps are often unintuitive and tedious for the user, and do not necessarily lead to a satisfactory level of control. We present an Adaptive BCI framework that provides feedback after only minutes of autocalibration in a two-class BCI setup. During operation, the system recurrently reselects only one out of six predefined logarithmic bandpower features (10-13 and 16-24 Hz from Laplacian derivations over C3, Cz, and C4), specifically, the feature that exhibits maximum discriminability. The system then retrains a linear discriminant analysis classifier on all available data and updates the online paradigm with the new model. Every retraining step is preceded by an online outlier rejection. Operating the system requires no engineering knowledge other than connecting the user and starting the system. In a supporting study, ten out of twelve novice users reached a criterion level of above 70% accuracy in one to three sessions (10-80 min online time) of training, with a median accuracy of 80.2 ± 11.3% in the last session. We consider the presented system a positive first step towards fully autocalibrating motor imagery BCIs.

Long-Term Independent Brain-Computer Interface Home Use Improves Quality of Life of a Patient in the Locked-In State: A Case Study

Article

Mar 2015
ARCH PHYS MED REHAB

Despite intense brain-computer interface (BCI) research for >2 decades, BCIs have hardly been established at patients' homes. The current study aimed at demonstrating expert independent BCI home use by a patient in the locked-in state and the effect it has on quality of life. In this case study, the P300 BCI-controlled application Brain Painting was facilitated and installed at the patient's home. Family and caregivers were trained in setting up the BCI system. After every BCI session, the end user indicated subjective level of control, loss of control, level of exhaustion, satisfaction, frustration, and enjoyment. To monitor BCI home use, evaluation data of every session were automatically sent and stored on a remote server. Satisfaction with the BCI as an assistive device and subjective workload was indicated by the patient. In accordance with the user-centered design, usability of the BCI was evaluated in terms of its effectiveness, efficiency, and satisfaction. The influence of the BCI on quality of life of the end user was assessed. At the patient's home. A 73-year-old patient with amyotrophic lateral sclerosis in the locked-in state. Not applicable. The BCI has been used by the patient independent of experts for >14 months. The patient painted in about 200 BCI sessions (1-3 times per week) with a mean painting duration of 81.86 minutes (SD=52.15, maximum: 230.41). BCI improved quality of life of the patient. In most of the BCI sessions the end user's satisfaction was high (mean=7.4, SD=3.24; range, 0-10). Dissatisfaction occurred mostly because of technical problems at the beginning of the study or varying BCI control. The subjective workload was moderate (mean=40.61; range, 0-100). The end user was highy satisfied with all components of the BCI (mean 4.42-5.0; range, 1-5). A perfect match between the user and the BCI technology was achieved (mean: 4.8; range, 1-5). Brain Painting had a positive impact on the patient's life on all three dimensions: competence (1.5), adaptability (2.17) and self-esteem (1.5); (range: -3 = maximum negative impact; 3 maximum positive impact). The patient had her first public art exhibition in July 2013; future exhibitions are in preparation. Independent BCI home use is possible with high satisfaction for the end user. The BCI indeed positively influenced quality of life of the patient and supports social inclusion. Results demonstrate that visual P300 BCIs can be valuable for patients in the locked-in state even if other means of communication are still available (eye tracker). Copyright © 2015 American Congress of Rehabilitation Medicine. Published by Elsevier Inc. All rights reserved.

Thought-based row-column scanning communication board for individuals with cerebral palsy

Article

Jan 2015

Impairment of an individual's ability to communicate is a major hurdle for active participation in education and social life. A lot of individuals with cerebral palsy (CP) have normal intelligence, however, due to their inability to communicate, they fall behind. Non-invasive electroencephalogram (EEG) based brain-computer interfaces (BCIs) have been proposed as potential assistive devices for individuals with CP. BCIs translate brain signals directly into action. Motor activity is no longer required. However, translation of EEG signals may be unreliable and requires months of training. Moreover, individuals with CP may exhibit high levels of spontaneous and uncontrolled movement, which has a large impact on EEG signal quality and results in incorrect translations. We introduce a novel thought-based row-column scanning communication board that was developed following user-centered design principles. Key features include an automatic online artifact reduction method and an evidence accumulation procedure for decision making. The latter allows robust decision making with unreliable BCI input. Fourteen users with CP participated in a supporting online study and helped to evaluate the performance of the developed system. Users were asked to select target items with the row-column scanning communication board. The results suggest that seven among eleven remaining users performed better than chance and were consequently able to communicate by using the developed system. Three users were excluded because of insufficient EEG signal quality. These results are very encouraging and represent a good foundation for the development of real-world BCI-based communication devices for users with CP.

Brain-controlled applications using dynamic P300 speller matrices

Article

Dec 2014
ARTIF INTELL MED

Brain-computer interfacing: More than the sum of its parts

Article

Feb 2012

The performance of non-invasive electroencephalogram-based (EEG) brain–computer interfaces (BCIs) has improved significantly in recent years. However, remaining challenges include the non-stationarity and the low signal-to-noise ratio of the EEG, which limit the bandwidth and hence the available applications. Optimization of both individual components of BCIs and the interrelationship between them is crucial to enhance bandwidth. In other words, neuroscientific knowledge and machine learning need to be optimized by considering concepts from human–computer interaction research and usability. In this paper, we present results of ongoing relevant research in our lab that addresses several important issues for BCIs based on the detection of transient changes in oscillatory EEG activity. First, we report on the long-term stability and robustness of detection of oscillatory EEG components modulated by distinct mental tasks, and show that the use of mental task pairs “mental subtraction versus motor imagery” achieves robust and reliable performance (Cohen’s κ > 0.6) in seven out of nine subjects over a period of 4 days. Second, we report on restricted Boltzmann machines (RBMs) as promising tools for the recognition of oscillatory EEG patterns. In an off-line BCI simulation we computed average peak accuracies, averaged over ten subjects, of 80.8 ± 7.2 %. Third, we present the basic framework of the context-aware hybrid Graz-BCI that allows interacting with the massive multiplayer online role playing game World of Warcraft. We show how a more integrated design approach that considers all components of BCIs, their interrelationships, other input signals and contextual information can increase interaction efficacy.

Hybrid brain–computer interfaces and hybrid neuroprostheses for restoration of upper limb functions in individuals with high-level spinal cord injury

Article

Sep 2013

The bilateral loss of the grasp function associated with a lesion of the cervical spinal cord severely limits the affected individuals' ability to live independently and return to gainful employment after sustaining a spinal cord injury (SCI). Any improvement in lost or limited grasp function is highly desirable. With current neuroprostheses, relevant improvements can be achieved in end users with preserved shoulder and elbow, but missing hand function. The aim of this single case study is to show that (1) with the support of hybrid neuroprostheses combining functional electrical stimulation (FES) with orthoses, restoration of hand, finger and elbow function is possible in users with high-level SCI and (2) shared control principles can be effectively used to allow for a brain-computer interface (BCI) control, even if only moderate BCI performance is achieved after extensive training. The individual in this study is a right-handed 41-year-old man who sustained a traumatic SCI in 2009 and has a complete motor and sensory lesion at the level of C4. He is unable to generate functionally relevant movements of the elbow, hand and fingers on either side. He underwent extensive FES training (30-45min, 2-3 times per week for 6 months) and motor imagery (MI) BCI training (415 runs in 43 sessions over 12 months). To meet individual needs, the system was designed in a modular fashion including an intelligent control approach encompassing two input modalities, namely an MI-BCI and shoulder movements. After one year of training, the end user's MI-BCI performance ranged from 50% to 93% (average: 70.5%). The performance of the hybrid system was evaluated with different functional assessments. The user was able to transfer objects of the grasp-and-release-test and he succeeded in eating a pretzel stick, signing a document and eating an ice cream cone, which he was unable to do without the system. This proof-of-concept study has demonstrated that with the support of hybrid FES systems consisting of FES and a semiactive orthosis, restoring hand, finger and elbow function is possible in a tetraplegic end-user. Remarkably, even after one year of training and 415 MI-BCI runs, the end user's average BCI performance remained at about 70%. This supports the view that in high-level tetraplegic subjects, an initially moderate BCI performance cannot be improved by extensive training. However, this aspect has to be validated in future studies with a larger population.

Brain-Computer Interface adaptation for an end user to compete in the Cybathlon

Abstract and Figures

Recommended publications

BCI adaptation for end users The Graz-BCI approach

Analyzing and Decoding Natural Reach-and-Grasp Actions Using Gel, Water and Dry EEG Systems

Lets play Tic-Tac-Toe: A Brain-Computer Interface case study in cerebral palsy

Cybathlon experiences of the Graz BCI racing team Mirage91 in the brain-computer interface disciplin...

From classic motor imagery to complex movement intention decoding: The noninvasive Graz-BCI approach