Usability and Workload of Access Technology for People With Severe Motor Impairment: A Comparison of Brain-Computer Interfacing and Eye Tracking

Abstract

Background. Eye trackers are widely used among people with amyotrophic lateral sclerosis, and their benefits to quality of life have been previously shown. On the contrary, Brain-computer interfaces (BCIs) are still quite a novel technology, which also serves as an access technology for people with severe motor impairment. Objective. To compare a visual P300-based BCI and an eye tracker in terms of information transfer rate (ITR), usability, and cognitive workload in users with motor impairments. Methods. Each participant performed 3 spelling tasks, over 4 total sessions, using an Internet browser, which was controlled by a spelling interface that was suitable for use with either the BCI or the eye tracker. At the end of each session, participants evaluated usability and cognitive workload of the system. Results. ITR and System Usability Scale (SUS) score were higher for the eye tracker (Wilcoxon signed-rank test: ITR T = 9, P = .016; SUS T = 12.50, P = .035). Cognitive workload was higher for the BCI (T = 4; P = .003). Conclusions. Although BCIs could be potentially useful for people with severe physical disabilities, we showed that the usability of BCIs based on the visual P300 remains inferior to eye tracking. We suggest that future research on visual BCIs should use eye tracking-based control as a comparison to evaluate performance or focus on nonvisual paradigms for persons who have lost gaze control.
© The Author(s) 2015.

Full text

Neurorehabilitation and
Neural Repair
1 –8
© The Author(s) 2015
Reprints and permissions:
sagepub.com/journalsPermissions.nav
DOI: 10.1177/1545968315575611
nnr.sagepub.com
Original Research Articles
Introduction
Because of degenerative neuromuscular diseases or neuro-
logical disorders, such as amyotrophic lateral sclerosis
(ALS), persons with severe physical disabilities can gradu-
ally lose control of speech muscles and limbs and, conse-
quently, the ability to communicate with their voice or with
conventional assistive devices.
1
Self-expression is funda-
mental for quality of life,
2
and lack of communication can
result in restrictions of participation, as defined by the
International Classification of Functioning, Disability and
Health.
By using gaze, or pupil size, eye trackers enable users to
communicate or control devices.
3
These systems require
only a short training period
4
and result in a low workload.
5
In a comparison between an eye tracker and a single switch
scanning system,
6
participants with ALS reported less
fatigue and a faster access when using the eye tracker. The
main difficulty in using eye trackers as an access technol-
ogy (AT), is the so-called Midas touch problem.
3
This refers
to the fact that gaze direction is not always related to the
focus of the attention, causing users to select a command
against their will.
People with ALS were involved only in a small number
of studies regarding the assessment of eye trackers.
2,5,7
Calvo et al
5
found significant improvement in the quality of
life of people with ALS after being provided with an eye
575611NNR
XXX10.1177/1545968315575611Neurorehabilitation and Neural RepairPasqualotto et al
research-article2015
1
Université Catholique de Louvain, Louvain-la-Neuve, Belgium
2
Eberhard Karls Universität, Tübingen, Germany
3
University of Perugia, Perugia, Italy
4
Sapienza Università di Roma, Rome, Italy
5
Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS), Venezia
Lido, Italy
6
Universität Würzburg, Würzburg, Germany
7
National Rehabilitation Center for Persons with Disabilities,
Tokorozawa, Japan
Corresponding Author:
Emanuele Pasqualotto, PhD, Université Catholique de Louvain, Avenue
Hippocrate 54, bte B1.54.09, 1200 Brussels, Belgium.
Email: emanuele.pasqualotto@uclouvain.be
Usability and Workload of Access
Technology for People With Severe Motor
Impairment: A Comparison of Brain-
Computer Interfacing and Eye Tracking
Emanuele Pasqualotto, PhD
1
, Tamara Matuz, PhD
2
, Stefano Federici, PhD
3,4
,
Carolin A. Ruf, PhD
2
, Mathias Bartl
2
, Marta Olivetti Belardinelli, MA
4
,
Niels Birbaumer, PhD
2,5
, and Sebastian Halder, PhD
6,7
Abstract
Background. Eye trackers are widely used among people with amyotrophic lateral sclerosis, and their benefits to quality
of life have been previously shown. On the contrary, Brain-computer interfaces (BCIs) are still quite a novel technology,
which also serves as an access technology for people with severe motor impairment. Objective. To compare a visual P300-
based BCI and an eye tracker in terms of information transfer rate (ITR), usability, and cognitive workload in users with
motor impairments. Methods. Each participant performed 3 spelling tasks, over 4 total sessions, using an Internet browser,
which was controlled by a spelling interface that was suitable for use with either the BCI or the eye tracker. At the end of
each session, participants evaluated usability and cognitive workload of the system. Results. ITR and System Usability Scale
(SUS) score were higher for the eye tracker (Wilcoxon signed-rank test: ITR T = 9, P = .016; SUS T = 12.50, P = .035).
Cognitive workload was higher for the BCI (T = 4; P = .003). Conclusions. Although BCIs could be potentially useful for
people with severe physical disabilities, we showed that the usability of BCIs based on the visual P300 remains inferior to
eye tracking. We suggest that future research on visual BCIs should use eye tracking–based control as a comparison to
evaluate performance or focus on nonvisual paradigms for persons who have lost gaze control.
Keywords
BCI, brain-computer interface, eye tracking, usability, cognitive workload, assistive technology, ALS
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

2 Neurorehabilitation and Neural Repair
tracker. The participants in the study were able to communi-
cate independently and rated the communication as easier,
faster, and less effortful than before. Ball et al
7
investigated
the acceptance, training, and extended use patterns of an
eye tracker in a group of 15 people with ALS. The capacity
of the users for social interactions increased considerably.
Indeed, the system was not only used for face-to-face com-
munication, but also for many other functions, such as
group communication (43%), phone calls (71%), e-mail
(79%), and Internet access (86%).
Independent of motor inputs, a brain-computer interface
(BCI) provides a direct connection between the brain and
any device capable of receiving brain signals. Most BCI
systems use electroencephalogram (EEG) signals and
require users to intentionally control specific features of
their own brain activity, such as slow cortical potentials or
the sensorimotor rhythm.
8,9
The P300 Speller
10
has been
successfully used by people with severe motor impair-
ment.
11-14
The P300 is an event-related potential, recorded
using EEG, evoked by a rare, task-relevant stimulus,
15
with
a latency between 200 and 700 ms, and is often related to
attention. P300 BCIs do not require users to learn to modu-
late their brain response.
9
In a typical P300 paradigm, a participant is presented
with a 6 × 6 alphanumeric matrix, where each row and each
column flashes randomly in a fixed interval, and the user
selects the desired character by focusing the attention on the
corresponding cell. This combined mechanism of focused
attention and random flashing makes the single matrix cell
a rare task-relevant stimulus eliciting the P300 peak
response. The P300 Speller has been adapted to control
wheelchairs,
16
to control real and virtual environments,
17
to
browse the Internet,
18
and to paint.
19
In a recent telephone survey of people with ALS,
Huggins et al
20
investigated the users’ opinions and priori-
ties on BCI design. Their study reports that the desired BCI
accuracy for people with ALS is at least 90%, but accura-
cies reported in the literature are usually lower. Moreover,
the expected spelling speed would be 15 to 19 letters per
minute, whereas in the published studies, it is only about 5
letters per minute.
Through a recent literature review, we have shown that
although there is a large amount of studies on BCIs, most of
them focus on methodological approaches, neglecting
usability aspects.
9,21
Most of the BCI studies, in fact, do not
consider that users often discard assistive technology after
only a few attempts and that personal factors (eg, mood,
motivation, belief, and predispositions) should be taken into
account because they can serve as both barriers and facilita-
tors in sustaining efficient use.
22,23
Although the effectiveness of BCIs in terms of character
selection is assessed in most studies, the efficiency and the
satisfaction from the users’ perspective are not always
addressed; however, they are part of a recent area of
growth.
12,21,24-27
Kleih et al
24
and Nijboer et al
12
explored the
effects of motivational factors on BCI performance.
Pasqualotto and colleagues
21,25
compared 2 prototypes of
BCIs in terms of usability and cognitive workload. Riccio et
al
26
investigated the influence of workload on the perfor-
mance of 2 P300-based BCI applications in healthy partici-
pants. Among these studies, only the study by Zickler et al
27
examined usability and cognitive workload in people with
severe disabilities, even though on a very limited sample.
Anyway, none of these studies compared BCIs with any
other existing AT.
It is our aim to provide a full usability assessment of a
P300-based BCI for controlling an Internet browser. To
accomplish this aim, we compared users’ performance and
usability scores using a P300 BCI and an eye tracking sys-
tem. More specifically, in a group of people with severe dis-
ability, we assessed and compared (1) the accuracy of
correct selections during 3 Internet tasks and (2) usability
indicators, such as control and cognitive workload, and
users’ satisfaction with both communication systems.
This study addresses the issue of whether and under
which conditions a BCI (specifically, the P300-BCI-based
Internet browser) could be a method of choice for users
with severe physical disabilities who still have residual con-
trol over some specific muscle groups and who could,
therefore, use conventional ATs. Furthermore, the study will
provide insights regarding which direction future BCI
research should follow in order to provide a viable alterna-
tive to conventional ATs.
Methods
We performed a comparison between the user’s experience
of controlling a BCI and an eye tracker, in a within-subject
design. Through 4 sessions of AT use (2 for the BCI and 2
for the eye tracker), the participants carried out 3 character
selection tasks, which represent common tasks that AT
users may perform when browsing the Internet. In each ses-
sion, participants used one of the ATs and were asked to
complete 2 questionnaires. The study was approved by the
Ethics Committee of the Medical Faculty of the University
of Tübingen and was performed in compliance with the
Code of Ethics of the World Medical Association
(Declaration of Helsinki).
Participants and Procedure
A total of 12 native German-speaking participants (4
women; mean age = 56.5 years; standard deviation =
±10.07) with severe motor impairment (11 affected by ALS,
and 1 affected by Duchenne muscular dystrophy; see Table
1), all naive to the AT assessed, were involved in the study.
All participants had home care assistance. Measurements
were performed in the participants’ homes.
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

Pasqualotto et al 3
After obtaining informed consent, we made 4 weekly
appointments, depending on the participants’ availability.
The order of presentation of the technology was balanced
between participants. On average, including calibration,
BCI sessions lasted 4 hours, and eye tracker sessions 2
hours. This difference was because of the longer time
needed to prepare the EEG cap and BCI calibration and the
fixed amount of time required by the BCI for the selection
of characters. Letters selection was, in fact, constrained by
the time required by the Speller to run one, or more, full
cycles of random flashing of columns and rows, in contrast
to the eye tracker where the user could select at his/her own
pace.
In the first BCI session and in the first eye tracker ses-
sion, we administered the Amyotrophic Lateral Sclerosis
Functional Rating Scale–Revised (ALSFRS-R).
28
In each
session, participants were asked to complete 3 copy-spell-
ing tasks (as described below in the Task section) by using
an Internet browser, which was controlled by the BCI or by
the eye tracker. We used the information transfer rate (ITR)
to measure performance. At the end of each session, we
administered the System Usability Scale (SUS)
29
and the
National Aeronautics and Space Administration–Task Load
Index (NASA-TLX).
30
Equipment
Brain-Computer Interface. We used an IBM Thinkpad laptop
to collect data, using the BCI2000 software.
31
All sessions
were recorded using an electrode cap (Easycap GmbH, Ger-
many) with 16 Ag/AgCl ring electrodes (F3, Fz, F4, T7, C3,
Cz, C4, T8, CP3, CP4, P3, Pz, P4, PO7, PO8, Oz) and
impedance held under 5 kΩ. The electrodes were connected
to a g.USBamp amplifier (g.tec OG, Austria) with a sam-
pling frequency of 256 Hz (bandpass: 0.1-30 Hz; notch:
48-52 Hz). Reference and ground were, respectively, on the
right and left mastoid. The intensifying of rows/columns of
the P300 Speller (see Figure 1) lasted for 62.5 ms, with an
interstimulus interval of 125 ms. The number of intensifica-
tion sequences varied per participant, according to their
calibration, performed at each of the 2 sessions. The cali-
bration consisted in the spelling of the same 2 words for
everyone (Apfelkuchen and Goldfisch in German—respec-
tively, apple pie and goldfish), consisting of 20 letters in
total. We set the number of intensification sequences to the
minimum number needed to reach 70% accuracy offline.
Table 1. Summary of Demographic and Clinical Status of the Participants.
a
Sex Age Diagnosis
Year of
Diagnosis ALSFRS-R
Degree of
Impairment Communication
Artificial
Nutrition
(PEG) Ventilation
01 F 53 Spinal 2008 23 Moderate Verbal No No
02 M 55 Spinal 2003 43 Minor Verbal No No
03 F 50 Bulbar 2003 17 Moderate Keyboard text-
to-speech
No Noninvasive
04 M 55 Spinal 1992 0 LIS Eye blink Yes Invasive
05 M 66 Spinal 1999 2 LIS Eye blink Yes No
06 M 71 Spinal 2005 11 Major Verbal No Noninvasive
07 M 55 Spinal 2006 23 Moderate Verbal No No
08 F 48 Spinal 2007 10 LIS Verbal Yes Noninvasive
09 M 70 Spinal 2008 18 Moderate Verbal No Noninvasive
10 F 54 Spinal 2003 0 LIS Chin joystick Yes Invasive
11 M 36 Duchenne 1976 7 LIS Verbal No Noninvasive
12 M 65 Bulbar 2009 32 Minor Verbal No No
Abbreviations: ALSFRS-R, Amyotrophic Lateral Sclerosis Functional Rating Scale–Revised; LIS, locked-in syndrome; PEG, percutaneous endoscopic
gastrostomy.
a
Degree of impairment categories were drawn according to Kübler and Birbaumer (2008).
40
Figure 1. The P300 matrix contains the complete alphabet
and part of the asterisk commands needed to select a double
code. Every row and column flashes randomly in a fixed interval.
By focusing the attention on a cell, the user can perform the
selection. In this frame, the third row is intensified.
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

4 Neurorehabilitation and Neural Repair
The lowest number of sequences reached was 5 and the
highest 8.
Eye Tracker. We used a SeeTech Pro (HumanElektronik
GmbH, Germany) set with a 7 × 7 grid. The SeeTech Pro is
a binocular infrared system with a 32-sample/s camera. The
grid locks the gaze in a cell to obviate the lack of precision.
For character selection, we used a static matrix of letters
modeled on the graphical appearance of the P300 Speller.
Participants could select characters by gazing at the intended
target and closing their eyes for 1.5 s. The visual appearance
of the eye tracker interface was identical to the one used for
the BCI, except for the input modality. The calibration
phase, performed at both sessions, consisted in fixating 9
positions on the screen located in the corners, the center,
and the extremities on the horizontal and vertical middle
lines. The screen, for both the eye tracker and the BCI, was
located at around 50 cm from the participant.
The P300 Browser. For Internet navigation, we used a newer
version (see Figure 1) of the P300 Speller-based browser
described by Mugler et al.
18
When loading a page, the
browser automatically assigns an alphabetical code to rep-
resent the hyperlinks (eg, the character “A” or a combina-
tion of 2 characters such as “AB”). By means of the Speller,
the user can select the codes corresponding to a particular
hyperlink and thus explore the web pages.
Tasks
The copy-spelling task is widely used in the BCI field.
12,21,32
The general idea behind this task is to provide the user with
a set of words or sentences to write with the communication
interface. In each session with the BCI or the eye tracker,
participants were asked to select a sequence of predefined
characters in the matrix, representing common Internet
tasks. Each participant performed the same task, with the
same instructions and words to spell. However, the codes
assigned by the browser to the links in the instructions were
not always the same. Participants were allowed to correct
errors, resulting in trials that could differ in length. The first
task consisted in checking the weather forecast by using a
search engine. The participants spelled the name of a
German magazine in the search bar of a search engine Web
site. After selecting the first result of the search, participants
were asked to select the weather section. Then, they had
time to observe the image of the forecast by selecting pause.
Finally, after unpausing they were asked to select the legend
at the bottom of the page, after scrolling the page. A mini-
mum of 15 character selections was required to accomplish
the task. The second task was a search in an online encyclo-
pedia. Starting from the encyclopedia’s main page, partici-
pants performed a search for the term brain (in German,
Gehirn). After checking the resulting page, participants
were asked to select the section about the human brain and
to scroll the page to the bottom. This task required a mini-
mum of 11 selections. The third task involved playing 2
songs on a Web site. Starting from the main page of a music
Web site, the participants performed a search for the term
Jazz. The result was a list of playable songs 30 s in length.
After selecting and then listening to the sixth song on the
list, they were asked to select the “Country” section and
then to listen to the first song. This task needed a minimum
of 14 total selections.
Measures
Performance. We used bits per minute (or ITR) to compare
the performance of the BCI and the eye tracker. Bit rate is a
standard measure for communication systems and repre-
sents the amount of information communicated per unit
time, depending on both speed and accuracy.
33
Based on
Pierce’s formula,
33
the bit rate is defined as follows:
BNPP P
P
N
=+ −−
()
−
−






loglog log
22 2
1
1
1
(1)
Here, N represents the number of possible selections in the
matrix and P the accuracy of character selection of the user.
Usability. We used the SUS to assess the usability of the sys-
tems. The SUS is a 10-item scale, with a global subjective
assessment of usability. It consists of 10 sentences with a
5-point Likert scale that ranges from 1 (strongly disagree)
to 5 (strongly agree). The SUS scale provides a score, rang-
ing from 0 to 100, that can be used to compare the usability.
Although, as also stated by the original author of the ques-
tionnaire, usability does not exist in an absolute sense, a
score of 70 has been suggested as the acceptable
minimum.
34
Cognitive Workload. The NASA-TLX is a multiscale tool to
evaluate the subjective cognitive workload considering its
possible different sources.
30
The NASA-TLX consists of 6
scales (mental demands, physical demands, temporal
demands, performance, effort, and frustration level), rated
in 2 stages. In the first stage, the user assigns a value to each
scale. In the second stage, the user is provided with the 15
pairs obtained by combining the 6 scales, with the goal to
choose the scale more relevant to workload from each pair.
This procedure is used to assess the weight of each scale.
The ratings and weights are combined to obtain the final
score, ranging from 0 to 100, with 100 representing the
highest workload experienced by the participant.
Functional Status. To investigate the relationship between
the functional status and the performance and to monitor
potential changes in participants’ functionality, we assessed
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

Pasqualotto et al 5
the functionality using the ALSFRS-R.
28
This scale implies
rating 12 items referring to different functions, such as
speech, swallowing, handwriting, and walking, on a 5-point
scale. The obtained score may range from 0 (most severe
impairment) to 48 (no impairment).
Analysis
We performed the comparison between the BCI and the eye
tracker using a Wilcoxon signed-rank test, on the averaged
data of the 2 sessions. We used bits per minute and the
scores of SUS and NASA-TLX, as independent variables.
For each technology, we correlated the scores of the ques-
tionnaires and the demographic data with performance data
to investigate how person-related factors affect perfor-
mance. We used stepwise linear discriminant analysis to
classify the EEG signal.
10
Calibration was performed with
the P300 Classifier tool provided with BCI2000.
Results
All participants were able to use both interfaces to accom-
plish the tasks (see Table 2). The mean ITR with the BCI
was 8.67 bits/min and with the eye tracker, 12.87 bits/min.
Thus, the Wilcoxon signed-rank test showed that the eye
tracker (median = 12.72) had a significantly higher ITR
than the BCI (median = 9.04; T = 9; P = .016, with an effect
size r = −0.68). The outcome of the SUS and the NASA-
TLX yielded similar results. The mean SUS score for the
BCI was 71.15, which according to Bangor et al
34
is right
above the acceptability threshold (in the third quartile),
whereas the score for the eye tracker was 78.54, which is
just beneath the lower boundary of the fourth quartile. The
direct comparison showed that the difference was signifi-
cant (eye tracker median = 80; BCI median = 71.25; T =
12.50; P = .035; r = −0.60). When using the NASA-TLX,
the absolute values are usually not considered a viable way
of describing the workload, and a comparison (eg, pre/post
measures, 2 different devices) is normally preferred. The
results of the NASA-TLX showed that the cognitive work-
load is higher for the BCI (median = 49.75) than for the eye
tracker (median = 33.53; T = 4; P = .003; r = −0.79).
The Spearman ρ analysis showed correlations between
performance, usability, and workload data with functional
status and disease duration. Age was not significant in our
comparison. The correlations revealed that the lower the
functional status of the participants, the lower their com-
munication rate, both using the BCI (r
s
= 0.640; P = .001)
and the eye tracker (r
s
= 0.430; P = .046; see Table 3).
Finally, results show that the longer the disease duration,
the lower the usability of a BCI (r
s
= −0.547; P = .008) and
the higher the cognitive workload of the BCI (r
s
= 0.544;
P = .009). We did not find this relation when considering
the eye tracker.
Discussion
We performed an exploratory study with the aim of compar-
ing 2 access technologies designed for people with severe
motor impairment. Our comparison between these inter-
faces has shown that the eye tracker is a faster and more
accurate technology that allows users to communicate with
a higher ITR than the BCI. The performance measures used
in this study showed the advantages of using the eye tracker
as a communication device. This advantage in performance
matches the findings on the usability and cognitive work-
load of the 2 interfaces. Participants rated the eye tracker as
a more satisfying device and considered the BCI as a tech-
nology requiring more effort and that was more time-con-
suming than the eye tracker. Differences in usability may be
partially a result of the longer time required for the use of
the BCI, and it is known that time can affect the perceived
fatigue, as also pointed out in more recent findings.
35
Because this time is intrinsic to the way this technology
works, future work on BCIs should address the issue of the
effort required by the end-users. Refined technology, such
as dry and wireless EEG,
36
and shifts to a classical condi-
tioning paradigm
37-39
could enhance not only the perfor-
mance, but also help in reducing the perceived effort of
BCIs. It is interesting to note that with the BCI, disease
duration plays a role with usability and workload, whereas
age does not. This finding, not confirmed with the eye
Table 2. Summary of the BCI and ET Interface Mean
Scores and Standard Deviations for the SUS, the NASA-TLX
Questionnaire for the Cognitive Workload, and the Bits Per
Minute.
Bits Per Minute SUS NASA-TLX
BCI 8.67 (3.43) 71.15 (11.31) 47.64 (14.87)
ET 12.87 (4.41) 78.54 (13.25) 32.72 (8.83)
Abbreviations: BCI, brain-computer interface; ET, eye tracking; SUS,
System Usability Scale; NASA-TLX, National Aeronautics and Space
Administration–Task Load Index.
Table 3. Summary of the Spearman ρ Correlations.
a
BCI Eye Tracker
Bits Per
Minute SUS
NASA-
TLX
Bits Per
Minute SUS
NASA-
TLX
ALSFRS-R 0.640** 0.280 −0.202 0.430* 0.241 −0.207
Disease
duration
−0.332 −0.547* 0.544** −0.118 −0.328 0.291
Age 0.239 0.153 0.140 −0.217 −0.269 0.002
Abbreviations: BCI, brain-computer interface; SUS, System Usability Scale; NASA-
TLX, National Aeronautics and Space Administration–Task Load Index; ALSFRS-R,
Amyotrophic Lateral Sclerosis Functional Rating Scale–Revised.
a
*P < .05; **P < .01.
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

6 Neurorehabilitation and Neural Repair
tracker, may be also explained by the longer time required
by BCIs. The relation between the functional status and per-
formance is a surprising finding. This result, found both
with the BCI and the eye tracker, seems to be contrary to
that of other studies,
11,40-42
although differences in the per-
formance measures and in the statistical tests should be
taken into account. Consequently, we cautiously abstain
from drawing strong conclusions based on this specific
finding.
Because the concept of usability is context related, our
findings should also be confirmed using the same method-
ology in contexts other than Internet use. Using different
control paradigms or classification methods may optimize
the BCI implementation we used, but we decided to evalu-
ate a well-established version of the paradigm for better
compatibility. Moreover, the eye tracker may profit from an
interface tailored specifically for eye tracking, whereas in
our study, it was used to control an interface made for BCIs.
It could also be interesting to investigate the usability of
invasive BCIs. As Hochberg et al
43
showed, people with
severe motor impairment can use intracortical neuronal
ensemble activity to achieve control, although rudimental,
of neuromotor prostheses. Despite the potential benefits of
a better signal, either because of restrictions on candidacy
or because of personal preferences, even when informed
about the advantages, a large number of patients do not
undergo the procedure.
8
Nevertheless, it is worth to mention
that the use of the P300 Speller in our implementation is
similar to the one used in previous noninvasive studies.
11-14
Although our study is based on a limited number of par-
ticipants in a specific context, we showed in a direct com-
parison that the use of the visual P300 might not be the first
choice as a communication channel for people with severe
physical disabilities. As suggested by our findings, when
users can rely on eye movements, they tend to consider the
eye tracker as a superior technology. As the literature sug-
gests, people with ALS can lose control of their eye move-
ments.
44
Moreover, Brunner et al
45
have recently shown that
in healthy individuals, the use of the visual P300 BCI may
be dependent on gaze direction. To overcome this issue,
new gaze-independent paradigms have been devel-
oped.
42,46,47
However, other studies on neuro-ophthalmic
abnormalities in people with ALS report retinal damage
48
and loss of visual acuity
49
associated with the course of the
disease. Considering the implications of our finding together
with the literature, we may conclude that when gaze control
is retained, people with ALS will choose to use the eye
tracker instead of the BCI, but when gaze and acuity are
lost, neither the eye tracker nor the visual BCI will work.
A possible solution to overcome this issue could be the
so-called hybrid BCI, which combines different brain fea-
tures with different data acquisition techniques, or even dif-
ferent non-BCI systems (such as electro-oculography or
electromyography). Hybrid BCIs can be used sequentially
or simultaneously, and in several studies, it has been proved
that they improve accuracy by focusing on the advantages
offered by the combined communication devices. For
example, if the user of an eye tracker has difficulties per-
forming eye blinks for selections and using dwell time gen-
erates too many errors, a BCI can be used to control
selections (brain switch).
50
Exploring different P300 BCI modalities, such as tac-
tile
51,52
and auditory devices,
32,53-58
may be another viable
solution to overcome the issues of the visual P300. Only 1
study on the auditory P300 included clinically relevant end
users.
53
Moreover, whereas in a single case study on a per-
son in locked-in syndrome (LIS) the authors reported prom-
ising results in the tactile modality,
52
in another case study
about the transition from LIS to complete LIS (CLIS), the
authors found no vibrotactile brain-evoked response.
59
Because of the inconclusive results, further research is
needed to determine the value of the tactile-haptic modality
for BCIs. Even though a delayed response in the auditory
areas has been reported in the literature,
60
we suggest that in
addition, the auditory modality should be further explored.
Conclusions
The present study highlighted that in certain conditions,
people with severe physical disabilities may prefer eye
trackers to visual BCIs, for their performance, usability, and
required cognitive effort. We suggest that future research in
BCIs should take into account these preferences and explore
modalities other than visual.
Acknowledgments
We would like to thank all the participants involved in this study.
We are very grateful to Humanelektronik GmbH for providing us
with one of their eye trackers and to Sven Körber (SirValUse
Consulting GmbH) for the German version of the System Usability
Scale. This study was carried out with the precious support of
Lasse Wiesinger, Anna-Antonia Pape, and Slavica Von Hartlieb
during the measurements. We are also grateful to the anonymous
reviewers for their suggestions as well as those of Dr Giulia
Liberati, which helped in improving the quality of the article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect
to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support
for the research, authorship, and/or publication of this article: This
study was partially funded by the Inter-University Centre for
Research on Cognitive Processing in Natural and Artificial
Systems (ECONA), the Werner Reichardt Centre for Integrative
Neuroscience (CIN) pool project 2009-10, the Deutsche
Forschungsgemeinschaft (DFG), and the European ICT Program
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

Pasqualotto et al 7
Project FP7-288566. SH received funding as international research
fellow from the Japan Society for the Promotion of Science (JSPS)
and the Alexander von Humboldt Foundation.
References
1. Ball LJ, Beukelman DR, Pattee GL. Acceptance of augmen-
tative and alternative communication technology by persons
with amyotrophic lateral sclerosis. Augment Altern Commun.
2004;20:113-122.
2. Pannasch S, Helmert JR, Malischke S, Storch A, Velichkovsky
BM. Eye typing in application: a comparison of two systems
with ALS patients. J Eye Mov Res. 2008;2:1-8.
3. Majaranta P, Räihä KJ. Twenty years of eye typing: sys-
tems and design issues. In: Duchowski AT, Vertegaal R,
Senders JW, eds. Proceedings of the Eye Tracking Research
and Application Symposium, ETRA 2002, New Orleans,
Louisiana, USA, March 25-27, 2002. New York, NY: ACM;
2002:15-22.
4. Stampe DM, Reingold EM. Selection by looking: a novel
computer interface and its application to psychological
research. In: Findlay JM, Walker R, Kentridge RW, eds.
Eye Movement Research: Mechanisms, Processes and
Applications. Amsterdam, Netherlands: Elsevier Science
Publishers; 1995:467-478.
5. Calvo A, Chiò A, Castellina E, et al. Eye tracking impact on
quality-of-life of ALS patients. In: Miesenberger K, Klaus J,
Zagler W, Karshmer A, eds. Computers Helping People With
Special Needs: Vol 5105. Lecture Notes in Computer Science.
Heidelberg, Germany: Springer; 2008:70-77.
6. Gibbons C, Beneteau E. Functional performance using eye
control and single switch scanning by people with ALS.
Perspect Augment Altern Commun. 2010;19:64.
7. Ball LJ, Nordness A, Fager S, et al. Eye-gaze access of AAC
technology for persons with amyotrophic lateral sclerosis. J
Med Speech Lang Pathol. 2010;18:11-23.
8. Birbaumer N. Breaking the silence: brain-computer
interfaces (BCI) for communication and motor control.
Psychophysiology. 2006;43:517-532.
9. Pasqualotto E, Federici S, Olivetti Belardinelli M. Toward
functioning and usable brain computer interfaces (BCIs): a lit-
erature review. Disabil Rehabil Assist Technol. 2012;7:89-103.
10. Farwell LA, Donchin E. Talking off the top of your head:
toward a mental prosthesis utilizing event-related brain poten-
tials. Electroencephalogr Clin Neurophysiol. 1988;70:510-523.
11. Nijboer F, Sellers EW, Mellinger J, et al. A P300-based brain-
computer interface for people with amyotrophic lateral scle-
rosis. Clin Neurophysiol. 2008;119:1909-1916.
12. Nijboer F, Birbaumer N, Kübler A. The influence of psy-
chological state and motivation on brain-computer interface
performance in patients with amyotrophic lateral sclerosis: a
longitudinal study. Front Neurosci. 2010;4:pii: 55.
13. Sellers EW, Donchin E. A P300-based brain-computer
interface: initial tests by ALS patients. Clin Neurophysiol.
2006;117:538-548.
14. Piccione F, Giorgi F, Tonin P, et al. P300-based brain com-
puter interface: reliability and performance in healthy and
paralysed participants. Clin Neurophysiol. 2006;117:531-537.
15. Duncan CC, Barry RJ, Connolly JF, et al. Event-related
potentials in clinical research: guidelines for eliciting, record-
ing, and quantifying mismatch negativity, P300, and N400.
Clin Neurophysiol. 2009;120:1883-1908.
16. Iturrate I, Antelis J, Kübler A, Minguez J. Non-invasive
brain-actuated wheelchair based on a P300 neurophysiologi-
cal protocol and automated navigation. IEEE Trans Robot.
2009;25:614-627.
17. Bayliss JD. Use of the evoked potential P3 component for
control in a virtual apartment. IEEE Trans Neural Syst
Rehabil Eng. 2003;11:113-116.
18. Mugler EM, Ruf CA, Halder S, Bensch M, Kler A. Design
and Implementation of a P300-based brain-computer inter-
face for controlling an internet browser. IEEE Trans Neural
Syst Rehabil Eng. 2010;18:599-609.
19. Münßinger JI, Halder S, Kleih SC, et al. Brain painting: evalu-
ation of a new brain-computer interface application with ALS
patients and healthy volunteers. Front Neurosci. 2010;4:182.
20. Huggins JE, Wren PA, Gruis KL. What would brain-computer
interface users want? Opinions and priorities of potential
users with amyotrophic lateral sclerosis. Amyotroph Lateral
Scler. 2011;12:318-324.
21. Pasqualotto E, Simonetta A, Gnisci V, Federici S, Olivetti
Belardinelli M. Toward a usability evaluation of BCIs. Int J
Bioelectromagn. 2011;13:121-122.
22. Federici S, Borsci S. The use and non-use of assistive tech-
nology in Italy: a pilot study. In: Gelderblom GJ, Soede M,
Adriaens L, Miesenberger K, eds. Everyday Technology
for Independence and Care: Vol 29. Assistive Technology
Research Series. Amsterdam, Netherlands: IOS Press;
2011:979-986.
23. Scherer MJ, Craddock G, Mackeogh T. The relationship of
personal factors and subjective well-being to the use of assis-
tive technology devices. Disabil Rehabil. 2011;33:811-817.
24. Kleih SC, Nijboer F, Halder S, Kübler A. Motivation modu-
lates the P300 amplitude during brain-computer interface use.
Clin Neurophysiol. 2010;121:1023-1031.
25. Pasqualotto E, Federici S, Simonetta A, Olivetti Belardinelli
M. Usability of brain computer interfaces. In: Gelderblom
GJ, Soede M, Adriaens L, Miesenberger K, eds. Everyday
Technology for Independence and Care: Vol 29. Assistive
Technology Research Series. Amsterdam, Netherlands: IOS
Press; 2011:481-488.
26. Riccio A, Leotta F, Bianchi L, et al. Workload measurement in
a communication application operated through a P300-based
brain-computer interface. J Neural Eng. 2011;8:025028.
27. Zickler C, Halder S, Kleih SC, Herbert C, Kübler A. Brain
painting: usability testing according to the user-centered
design in end users with severe motor paralysis. Artif Intell
Med. 2013;59:99-110.
28. Cedarbaum JM, Stambler N, Malta E, et al. The ALSFRS-R:
a revised ALS functional rating scale that incorporates assess-
ments of respiratory function. BDNF ALS Study Group
(Phase III). J Neurol Sci. 1999;169:13-21.
29. Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan
PW, Thomas B, Weerdmeester BA, McClelland IL, eds.
Usability Evaluation in Industry. London, UK: Taylor &
Francis; 1996:189-194.
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from

8 Neurorehabilitation and Neural Repair
30. Hart SG, Staveland LE. Development of NASA-TLX (task
load index): results of empirical and theoretical research. In:
Hancock PA, Meshkati N, eds. Human Mental Workload.
Amsterdam, Netherlands: Elsevier Science; 1988:139-183.
31. Schalk G, McFarland DJ, Hinterberger T, Birbaumer N,
Wolpaw JR. BCI2000: a general-purpose brain-com-
puter interface (BCI) system. IEEE Trans Biomed Eng.
2004;51:1034-1043.
32. Käthner I, Ruf CA, Pasqualotto E, Braun C, Birbaumer N,
Halder S. A portable auditory P300 brain-computer interface
with directional cues. Clin Neurophysiol. 2013;124:327-338.
33. Pierce JR. An Introduction to Information Theory: Symbols,
Signals and Noise. New York, NY: Dover Publications; 1980.
34. Bangor A, Kortum PT, Miller JT. An empirical evaluation
of the system usability scale. Int J Hum Comput Interact.
2008;24:574-594.
35. Käthner I, Wriessnegger SC, Müller-Putz GR, Kübler A,
Halder S. Effects of mental workload and fatigue on the P300,
alpha and theta band power during operation of an ERP (P300)
brain-computer interface. Biol Psychol. 2014;102:118-129.
36. Zander TO, Lehne M, Ihme K, et al. A dry EEG-system
for scientific research and brain-computer interfaces. Front
Neurosci. 2011;5:53.
37. Furdea A, Ruf CA, Halder S, et al. A new (semantic) reflexive
brain-computer interface: in search for a suitable classifier. J
Neurosci Methods. 2012;203:233-240.
38. Liberati G, Dalboni da Rocha JL, van der Heiden L, et al.
Toward a brain-computer interface for Alzheimer’s disease
patients by combining classical conditioning and brain state
classification. J Alzheimers Dis. 2012;31:S211-S220.
39. van der Heiden L, Liberati G, Sitaram R, et al. Insula and
inferior frontal triangularis activations distinguish between
conditioned brain responses using emotional sounds for basic
BCI communication. Front Behav Neurosci. 2014;8:247.
40. Kübler A, Birbaumer N. Brain-computer interfaces and com-
munication in paralysis: extinction of goal directed think-
ing in completely paralysed patients? Clin Neurophysiol.
2008;119:2658-2666.
41. Silvoni S, Volpato C, Cavinato M, et al. P300-based brain-
computer interface communication: evaluation and follow-up
in amyotrophic lateral sclerosis. Front Neurosci. 2009;3:60.
42. Marchetti M, Piccione F, Silvoni S, Gamberini L, Priftis
K. Covert visuospatial attention orienting in a brain-com-
puter interface for amyotrophic lateral sclerosis patients.
Neurorehabil Neural Repair. 2013;27:430-438.
43. Hochberg LR, Serruya MD, Friehs GM, et al. Neuronal
ensemble control of prosthetic devices by a human with tet-
raplegia. Nature. 2006;442:164-171.
44. Sharma R, Hicks S, Berna CM, Kennard C, Talbot K, Turner
MR. Oculomotor dysfunction in amyotrophic lateral sclero-
sis: a comprehensive review. Arch Neurol. 2011;68:857-861.
45. Brunner P, Joshi S, Briskin S, Wolpaw JR, Bischof H, Schalk
G. Does the “P300” speller depend on eye gaze? J Neural
Eng. 2010;7:056013-056013.
46. Lim JH, Hwang HJ, Han CH, Jung KY, Im CH. Classification
of binary intentions for individuals with impaired oculomotor
function: “eyes-closed” SSVEP-based brain-computer inter-
face (BCI). J Neural Eng. 2013;10:026021.
47. Ahani A, Wiegand K, Orhan U, et al. RSVP IconMessenger:
icon-based brain-interfaced alternative and augmentative
communication. Brain Comput Interface. 2014;1:192-
203.
48. Ringelstein M, Albrecht P, Südmeyer M, et al. Subtle retinal
pathology in amyotrophic lateral sclerosis. Ann Clin Transl
Neurol. 2014;1:290-297.
49. Moss HE, McCluskey L, Elman L, et al. Cross-sectional
evaluation of clinical neuro-ophthalmic abnormalities in
an amyotrophic lateral sclerosis population. J Neurol Sci.
2012;314:97-101.
50. Amiri S, Fazel-Rezai R, Asadpour V. A review of hybrid
brain-computer interface systems. Adv Hum Comput Interact.
2013;2013:1.
51. Brouwer A-M, van Erp JBF. A tactile p300 brain-computer
interface. Front Neurosci. 2010;4:19.
52. Kaufmann T, Holz EM, Kübler A. Comparison of tactile,
auditory and visual modality for brain-computer interface
use: a case study with a patient in the locked-in state. Front
Neurosci. 2013;7:129.
53. Kübler A, Furdea A, Halder S, Hammer EM, Nijboer F,
Kotchoubey B. A brain-computer interface controlled audi-
tory event-related potential (p300) spelling system for locked-
in patients. Ann N Y Acad Sci. 2009;1157:90-100.
54. Halder S, Rea M, Andreoni R, et al. An auditory oddball brain-
computer interface for binary choices. Clin Neurophysiol.
2010;121:516-523.
55. Schreuder M, Blankertz B, Tangermann M. A new auditory
multi-class brain-computer interface paradigm: spatial hear-
ing as an informative cue. PLoS One. 2010;5:e9813.
56. Furdea A, Halder S, Krusienski DJ, et al. An auditory odd-
ball (P300) spelling system for brain-computer interfaces.
Psychophysiology. 2009;46:617-25.
57. Hill NJ, Moinuddin A, Häuser A-K, Kienzle S, Schalk G.
Communication and control by listening: toward optimal
design of a two-class auditory streaming brain-computer
interface. Front Neurosci. 2012;6:181.
58. Simon N, Käthner I, Ruf CA, Pasqualotto E, Kübler A,
Halder S. An auditory multiclass brain-computer interface
with natural stimuli: Usability evaluation with healthy partic-
ipants and a motor impaired end user. Front Hum Neurosci.
2015;8:1039.
59. Ramos Murguialday A, Hill J, Bensch M, et al. Transition
from the locked in to the completely locked-in state: a
physiological analysis. Clin Neurophysiol. 2011;122:925-
933.
60. Lulé D, Diekmann V, Müller HP, Kassubek J, Ludolph AC,
Birbaumer N. Neuroimaging of multimodal sensory stimu-
lation in amyotrophic lateral sclerosis. J Neurol Neurosurg
Psychiatry. 2010;81:899-906.
at Univ Catholique Louvain Bib on March 16, 2015nnr.sagepub.comDownloaded from