Content uploaded by Anton Nijholt
Author content
All content in this area was uploaded by Anton Nijholt
Content may be subject to copyright.
BrainGain: BCI for HCI and Games
Anton Nijholt1, Jan B.F. van Erp2, Dirk Heylen1
1Abstract. In this position paper we describe part of the Dutch
BrainGain research project on Brain Computer Interfacing (BCI)
and our planned research in this project. We focus on BCI
research for healthy users. In the BrainGain project our task is to
look at Human Factors aspects of BCI applications, to look at
multimodal interactions that include BCI interactions, and to
design games, game environments and game interfaces that
allow BCI interactions. Recently we see game companies taking
an interest in BCI, among others leading to some games where
movements of the ‘healthy’ user help to intensify brain patterns
that control a virtual environment. One line of research we hope
to exploit is the use of BCI in exertion interfaces.
1 INTRODUCTION
BCI (Brain-Computer Interfacing) has become a research topic
in computer science and, in particular, human-computer
interaction. In 2007 a large scale BCI project was approved in
the Netherlands. This BrainGain project
(http://www.nici.ru.nl/braingain/) started in September 2007 and
is funded by the Dutch government with 14 million Euro. Part of
this funding is assigned to BCI research for the ‘healthy user’.
That is, research that does not necessarily aim at results and
applications for users with special needs. In the project
description it is mentioned that:
• The psychiatric and neurological professionals in the
consortium also want to investigate the use of modern
methods of analysis of brain signals for specifically
developed therapies. These developments could also be
applied to the needs of healthy users, in terms of health,
performance, or quality of life. For example, the costs of
stress to the society are high, and learning to relax,
concentrate or meditate could provide a useful application
of BCI for healthy users.
And there is an economical perspective too:
• In order to also create an economical impulse, the
consortium will develop a broad range of applications,
which will allow healthy users to also benefit from the
newly developed technologies. Possible applications
include entertainment, such as computer games driven by
brain signals. Or, in more professional surroundings, to
present information on a computer screen only when visual
attention is detected, such as might be useful for air traffic
controllers or customs officials checking scanned luggage.
From [1]: “Also, the elderly in general and the 100 million
baby boomers in specific –in control of the largest concentration
1 University of Twente, Human Media Interaction, PO Box 217, 7500
AE Enschede, the Netherlands, {anijholt,heylen}@cs.utwente.nl
2 TNO Human Factors, Soesterberg, The Netherlands,
Jan.vanerp@tno.nl
of funds than any other demographic group– will demand longer
life, personalised health care, intelligence and memory support,
and improvement of their senses and mobility.” Future interfaces
will allow us to communicate at an emotional and intentional
level. Sensors and actuators will be integrated everywhere in our
environment. They will capture verbal, nonverbal, physiological,
and brain information and this information will be processed and
interpreted in order to support the users in their daily activities.
Obviously, also in professional environments captured
information can help the environment to support a user in
performing his tasks. BCI can play a role in solving the threat of
sensory and cognitive overload for, for example, pilots and crisis
team members, but also for everyday life activities such as
driving, controlling devices and gaming. Especially in the latter
applications the hardware must be designed for use in everyday
life, i.e. unobtrusive, lightweight and wearable, preferably
wireless, and with low power consumption. Often there is not a
single task to be performed, as is mostly the case for severely
disabled persons. Moreover there is information to be captured
and fused from various input modalities and brain activity
displayed in various brain regions with not always
distinguishable functions.
2 BCI FOR HEALTHY USERS: TOPICS
In the part of the BrainGain project that is devoted to BCI for
healthy users we have chosen the following topics to research
[1]:
• Attention Monitoring and Adaptation: To stay highly
alert for extended periods of time is critical for flight
controllers, truck drivers and security personnel scanning
luggage or checking many video monitors. To detect visual
alertness is an important prerequisite to warrant user
performance. Experiments have shown that ongoing brain
activity (in particular posterior alpha activity) is a better
detector of visual alertness than behavioural measures.
These new findings could be used to create a BCI that
determines the user’s visual alertness and for example
adjust the visual load in the interface or even advise the user
to take a break. Such systems can be installed at airport
traffic controllers, security inspectors etc. The combination
with other physiological measures used in HCI is an
important multi-dimensional challenge.
• Classifying Images: The brain outperforms software tools
when it comes to classifying images or the semantic
understanding of images. In many areas, enormous amounts
of images are available but very hard to access because they
are not labelled. Automatic analysis of image contents is
very difficult and despite the huge efforts put into machine
algorithms, limited progress is made, while the brain does
these kinds of tasks very easily. Using a BCI may give us
access to these very powerful brain mechanisms to interpret
32
images. E.g., specific event related potentials may occur
when a primed object is present in an image, even when
many images are shown in rapid serial presentation. By
using this effect, observers may be able to reliably classify
images at very high speeds.
• Motion Control for Virtual or Remote Worlds: The
general question here is “to what extent can brain signals be
used for navigation in (relatively) fast in-the-loop
applications for gaming, simulation, and remote control
applications”. In these areas, using our locomotion system
as input device (e.g., walking on a treadmill) is
cumbersome, complicated and expensive. Usually, motion
control is accomplished by keyboard or joystick, sometimes
in combination with a head tracker to allow for a natural
way of looking around. The drawback of these motion
control devices is that they are unnatural, possibly
disturbing the user’s feeling of presence, and that they
occupy the hands. The latter is undesirable when the hands
are required to interact with the remote or virtual world. In
this research hands-free (self-) motion control interfaces
based on brain signals will be investigated.
• Multimodal Measures of the User Experience: In this
research we investigate the following topics. (1)
Brainsignals and user experience: In the case of intelligent,
adaptive interfaces the system tries to adapt itself to the way
the user experiences the interaction. The brain signals
contain information about this experience. In a series of
controlled experiments it will be determined what
brainsignals can tell us about the user experience. (2)
Correlations between brainsignals and other information
from the body: Measures of biosignals such as heart rate,
respiration, perspiration, body temperature and muscle
tension can point to factors of the user experience as well.
In experiments brainsignals will be traced together with
other physiological measures and information from voice,
face and head. (3) Expressivity and reliability: For each
modality and each combination of modalities it has to be
determined what they can express and how reliably they
express this. Combination of modalities reduces noise and
can dissolve ambiguities. It is important to have a good
indication of the reliability of the various measures. (4)
Interface: The inferences about the cognitive and affective
state of the user that can be made on the basis of the
information from the various measures will be used in the
development of adaptive interfaces for games.
• Employing BCI in game environments: Currently there is
a development from traditional videogames using keyboard,
mouse or joystick, to games that use all kinds of sensors
and algorithms that know about speech characteristics,
about facial expressions, gestures, location and identity of
the gamer and even physiological processes that can be
used to adapt or control the game [2]. The next step in game
development is input obtained from the measurement of
brain activity [3,4]. User-controlled brain activity has been
used in games that involve moving a cursor on the screen or
guiding the movements of an avatar in a virtual
environment by imagining these movements. Relaxation
games have been designed and also games that adapt to the
affective state of the user. For the design of game and
training environments we need the integration of theoretical
research on multimodal interaction, intention detection,
affective state and visual attention monitoring, and on-line
motion control. It also requires the design of several
prototypes of games. Some of these games will be
elaborated into events for the general audience (as
dissemination projects). Next to games for amusement we
will explore (serious) games for educational, training and
simulation purposes. Selection and design of BCI methods
feasible for commercial computer games is still difficult.
Here price, ease of fitting, required data rate, etc. put strict
constraints on the technology. However, the computer game
industry is ready to embrace these applications and can
even drive some of the developments.
It should be mentioned that the development of (serious)
brain games fits in many initiatives in the Netherlands to develop
company-based game technology, such as the Benelux Game
Initiative (BGIn) in which Dutch game development companies
are the founding fathers and the GATE research project (started
in 2006) in which many Dutch game development companies are
involved. The entertainment games market in the Netherlands
was estimated (AGS) 200MEuro in 2005 and is growing, with
impact on education, care, sports and digital lifestyle. For serious
gaming the market was estimated to be over 350MEuro, with
areas of interest that include care, safety and defense. Initiatives
to stimulate economic activity in these areas are taken by
governmental organizations (Ministry of Economic Affairs, and
others).
3 BCI FOR GAMES
3.1 BCI for Games: Commercial Explorations
Presently, the majority of BCI users are patients that do not have
control, or full control, of their muscles and that have to learn to
control a prosthetic device, a communication device, or a
mobility device (e.g., a wheelchair) by thought. Nevertheless,
there are various reasons to look at the use of BCI technology in
the context of exertion interfaces [5] for healthy users. Exertion
interfaces are interfaces that deliberately require physical effort.
These interfaces can play a role in sports, health (fitness), and
entertainment. Often they are accompanied with a large screen
where opponents are displayed and where computer vision and
other sensors are used to capture the bodily activity of players.
There are good reasons to investigate the role of BCI for such
interfaces. BCI allows:
• Finding out about the user’s mental state and trying to adapt
the interface and the interaction modalities to this mental
state. Obviously, there are other modalities that can be
considered too, for example, physiological information or
information obtained from nonverbal cues (pose, facial
expression, prosody). In exertion interfaces monitoring this
information can help to adapt the required or desired
exertion efforts to the user’s physical and mental state [2].
Existing exertion interfaces only have limited knowledge
about the user. For example, in a mediated football game
[5] the interface knows about who kicked the ball that hits a
wall, where the wall is hit, and how hard the wall is hit.
More direct information about the player is, however, not
available. Adding knowledge about brain activity to
33
knowledge obtained from other measured input modalities
helps to adapt the interface to the user.
• Making exertion interfaces more interesting and engaging
by adding a new modality to the already available and more
‘traditional’ input modalities for exertion interfaces. Again,
existing exertion interfaces have not only poor knowledge
about their users; they also make poor use of modalities that
are available for the user to control the exertion interface.
Obviously, it should be mentioned, that depending on the
interaction that is required, there is not always a need to
take into account all possible input modalities for an
exertion interface. Nevertheless, BCI provides an extra
input modality. That is, BCI allows the adding of an extra
input modality to the ones that have already been made
available for a long time. This input modality consists of
voluntarily and consciously produced or externally evoked
brain activity that can be recognized and translated into
commands to the interface.
Measuring brain activity for gamers can be used so that the
game environment (1) knows what a subject experiences and can
adapt game and interface in order to keep the gamer ‘in the flow’
of the game, and (2) allows the gamer to add brain control
commands to the already available control commands for the
game. The general assumption is that the added value of BCI
commands or the adaptation of the game to a mental state of a
gamer that can be measured from brain activity, may lead to a
commercial ‘killer application’. For example, a game that can be
played by enormous numbers of gamers, but a top level in the
game can only be reached when the gamer is able to master a
certain BCI command that adds to the already available
multimodal commands of the game or that can be used to modify
a more traditional game control command. The willingness of
gamers to spend large amounts of time to games they are
interested in makes it possible to integrate BCI learning
requirements in a natural way in game situations. Another issue
that need to be dealt with when we want to move forward in
attracting the game audience to BCI is the hardware that has to
be used, in particular the use of EEG caps. This ‘hardware’ is
improving. Some game companies provide rather fancy caps
(Figure 1) that rather than being considered obtrusive provide
more status to the gamer. It is expected that wireless technology
will allow a gamer to move around freely in an environment,
rather than being connected through cables to a computer.
3.2 BCI for Games: Motor Imagery Applications
As is well-known, an interesting class of brain activity for game
playing is related to motor imagery. That is, the user imagines a
certain movement. For example imagining a left foot movement
can be distinguished from imaging a right foot movement. This
kind of mental simulation of movement can be measured and
distinguished. Not only for feet, but also for arms or hand, the tip
of the tongue, et cetera. Intending to move, imagine a movement,
planning a movement, they all activate similar cortical areas.
This explains the succes that BCI has for patients who are not
able to use hands or feet, or patients who suffer the locked in
syndrome (ALS) and are not able to move or to speak. In various
applications it has been shown that they can learn to move a
cursor on a screen, to navigate in a virtual world and to control a
wheelchair. Much of the current BCI research concentrates on
improving such medical applications and also at looking at other
ways to improve the quality of life of those patients.
However, although not really of interest for ALS patients and
other disabled patients, these imagined movements activate the
brain areas that are also activated by the execution of the same
movements. Hence, for healthy users it becomes possible to
activate brain patterns by consciously produced movements and
have these brain patterns measured and translated into
commands for a computer, in order to navigate in a virtual
world, to move or lift (heavy) virtual objects (Figure 2 [6]) or to
control a robotic device. Moreover, it allows us to design games,
game environments, and exertion interfaces that are also
controlled by body movements but where the capturing of the
body movement is not done by sensors or cameras, but by
measuring associated brain activity.
3.3 BCI for Games: The Braingain Project
Investigating the possibilities of BCI for HCI and game
applications, including exertion interfaces, is one of our tasks in
the Dutch national BrainGain project. Apart from fundamental
research on distinguishing various types of brain activity when
the user (or gamer) is involved in various tasks, using different
modalities to perform this task, we will also introduce BCI
versions of games and exertion interfaces we have introduced
previously [7]. One example is the ‘Jump and Run’ exertion
interface where the gamer controls the movements of an avatar,
Figure 1. Left: A traditional EEG cap. Right: A helmet used in
commercial applications.
Figure 2. Lifting a heavy stone in a Stonehenge game
desi
g
ned b
y
Emotive S
y
stems.
34
who moves at high speed in a virtual world and has to avoid
obstacles (Figure 3). A camera observes the movements of the
human player and our aim is to play a similar game (similar, i.e.,
not necessarily requiring imagined movements resembling the
physical movements in the original game) by measuring brain
activity associated with imaginary and/or real movements in
such a way that no cameras are needed anymore: “Look Ma, No
Cameras!”
Figure 3. The ‘Jump and Run’ exertion interface
Acknowledgements
This work has been supported by funding from the Dutch
National SmartMix project Braingain on BCI, funded by the
Ministry of Economic Affairs and the GATE project, funded by
the Netherlands Organization for Scientific Research (NWO)
and the Netherlands ICT Research and Innovation Authority
(ICT Regie).
REFERENCES
[1] J. van Erp, A. Nijholt, D. Heylen, and O. Jensen. BCI for Healthy
Users. Project proposal description for the BrainGain project.
December 2006.
[2] K. Gilleade, A. Dix & J. Allanson. Affective Videogames and Modes
of Affective Gaming: Assist Me, Challenge Me, Emote Me. In: Proc.
DIGRA'2005, 2005.
[3] A. Nijholt and D. Tan (Eds.). Brain-Computer Interfaces and Games.
Proceedings Workshop at ACE (Advances in Computer
Entertainment), 12 June 2007.
[4] A. Nijholt, D. Tan, G. Pfurtscheller, C. Brunner, J. del R. Millan, B.
Allison, B. Graimann, F. Popescu, B. Blankertz, and K.-R. Müller.
Brain-Computer Interfacing for Intelligent Systems. IEEE Intelligent
Systems, 2008, to appear.
[5] F. Mueller and S. Agamanolis. Exertion interfaces. In: Proceedings
CHI '07: CHI '07 Extended Abstracts on Human factors in computing
systems. ACM, New York, NY, USA, 2857-2860 (2007).
[6] Emotive Systems: www.emotiv.com
[7] A. Nijholt, D. Reidsma, Z. Ruttkay, H. van Welbergen, and P. Bos.
Non-verbal and Bodily Interaction in Ambient Entertainment. In:
Proc. Fundamentals of Verbal and Non-verbal Communication and
the Biometrical Issue, A. Esposito et al. (eds.), IOS Press,
Amsterdam, 343-348.
35
