Download full-text PDF

EarFieldSensing: A Novel In-Ear Electric Field Sensing to Enrich Wearable Gesture Input through Facial Expressions

Conference Paper (PDF Available)  · May 2017with771 Reads
DOI: 10.1145/3025453.3025692
Conference: Conference: CHI 2017, At Denver, USA
Denys J C Matthies
Denys J C Matthies
Bodo Urban at Fraunhofer Institute for Computer Graphics Research IGD
  • 12.28
  • Fraunhofer Institute for Computer Graphics Research IGD
Abstract
EarFieldSensing (EarFS) is a novel input method for mobile and wearable computing using facial expressions. Facial muscle movements induce both electric field changes and physical deformations, which are detectable with electrodes placed inside the ear canal. The chosen ear-plug form factor is rather unobtrusive and allows for facial gesture recognition while utilizing the close proximity to the face. We collected 25 facial-related gestures and used them to compare the performance levels of several electric sensing technologies (EMG, CS, EFS, EarFS) with varying electrode setups. Our developed wearable fine-tuned electric field sensing employs differential amplification to effectively cancel out environmental noise while still being sensitive towards small facial-movement-related electric field changes and artifacts from ear canal deformations. By comparing a mobile with a stationary scenario, we found that EarFS continues to perform better in a mobile scenario. Quantitative results show EarFS to be capable of detecting a set of 5 facial gestures with a precision of 90% while sitting and 85.2% while walking. We provide detailed instructions to enable replication of our low-cost sensing device. Applying it to different positions of our body will also allow to sense a variety of other gestures and activities.
a
b
Figure 1. EarFS is a wearable electric field sensing device
which enables to sense mobile facial-related gestures. It
consists of a) an ear plug plus a reference electrode (a clothes
peg that has to be attached to the ear lobe), and b) four
sensing shields that are connected to an Arduino which runs
on a 9V battery supply and transmits data via Bluetooth.
EarFieldSensing: A Novel In-Ear Electric Field Sensing to
Enrich Wearable Gesture Input through Facial Expressions
Denys J.C. Matthies1, Bernhard A. Strecker1,2, Bodo Urban1
1 Fraunhofer IGD Rostock, Germany, {denys.matthies, bodo.urban}@igd-r.fraunhofer.de
2
University of Cologne, Germany, bernhard.strecker@uni-koeln.de
ABSTRACT
EarFieldSensing (EarFS) is a novel input method for
mobile and wearable computing using facial expressions.
Facial muscle movements induce both electric field changes
and physical deformations, which are detectable with
electrodes placed inside the ear canal. The chosen ear-plug
form factor is rather unobtrusive and allows for facial
gesture recognition while utilizing the close proximity to
the face. We collected 25 facial-related gestures and used
them to compare the performance levels of several electric
sensing technologies (EMG, CS, EFS, EarFS) with varying
electrode setups. Our developed wearable fine-tuned
electric field sensing employs differential amplification to
effectively cancel out environmental noise while still being
sensitive towards small facial-movement-related electric
field changes and artifacts from ear canal deformations. By
comparing a mobile with a stationary scenario, we found
that EarFS continues to perform better in a mobile scenario.
Quantitative results show EarFS to be capable of detecting
a set of 5 facial gestures with a precision of 90% while
sitting and 85.2% while walking. We provide detailed
instructions to enable replication of our low-cost sensing
device. Applying it to different positions of our body will
also allow to sense a variety of other gestures and activities.
Author Keywords
Electric field sensing, body potential sensing, facial
expression control, wearable computing, hands-/eyes-free.
ACM Classification Keywords
H.5.2. [User interfaces] Input devices and strategies.
INTRODUCTION
In Human-Computer Interaction, wearables become
increasingly important, which is indicated by the prevalence
of smart devices such as glasses or watches. Their tendency
to engage the center of attention still hinders the interaction
to become truly mobile, though. Therefore, one should
reconsider how to access and to interact with technology.
In 1998 already, Steve Mann stated that wearables should
be: »Unmonopolizing of the user's attention: […] One can
attend to other matters while using the apparatus, [while it
should be] unrestrictive to the user [23]
Mann envisions wearable computers to provide situational
benefits while not obstructing the user and enabling him for
subtle multitasking. In contrast, most of the current
interaction concepts still do not provide these qualities.
Users are often distracted by current smart devices, such as
mobile phones, as they usually require the users full
attention while involving the users hands and eyes. For
instance, rejecting a phone call or switching between songs
on a music player forces the user to take out the device,
which unnecessarily demands visual attention and occupies
at least one hand. However, EarFS enables the user to have
these interaction channels available for a potential primary
task. This is especially relevant for critical tasks, such as
when being involved in traffic. Therefore, we make use of a
facial expression control in the manner of microinteractions
»…because they may minimize interruption; that is, they
allow for a tiny burst of interaction with a device so that the
user can quickly return to the task at handAshbrook [1]
While some facial gestures are also subtle, we potentially
enable a shifting of microinteractions to the periphery of
our attention [16], which matches the described affordances
sketched by Mann. In this paper, we demonstrate such
hands-free and eyes-free peripheral microinteractions with:
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than the author(s) must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
CHI 2017,
May 06 - 11, 2017, Denver, CO, USA
Copyright is held by the owner/author(s). Publication rights licensed to
ACM.
ACM 978-1-4503-4655-9/17/05…$15.00
DOI:
http://dx.doi.org/10.1145/3025453.3025692
a broad gesture set based on facial expressions, which has
been evaluated with various in-ear electrode setups using
different wearable technologies (EMG, CS, EFS, EarFS);
a differential amplification EFS (EarFS), applicable for
wearable computing, that is sensitive enough towards
very small changes in electric fields of the human body to
detect micro-gestures, such as facial expressions.
RELATED WORK
Facial-expressions, have been widely investigated in the
area of affective computing [29]. Affective computing can
be described as a system being able to recognize, interpret,
process, and simulate human affects. Human affects can be
expressed through our faces, which has been extensively
investigated starting in the 1970s by Paul Ekman. In one of
his fundamental works, he established a facial action coding
system (FACS) which is still the ground-truth database for
all facial movements and their associated emotional states
[8]. Nowadays, we are able to use facial-expressions to
determine the frustration level of a user. In terms of
technology, two different major setups exist: (1) contact
electrodes, which are attached to the face, such as
electromyography (EMG) [17] or piezoelectric sensing
[36], and (2) proximity sensing, which is often vision-based
[13]. Still, utilizing facial expressions for gesture input has
not been extensively investigated yet, as we will illustrate.
Facial Expression Control in Medical Context
A major field for application in a medical context is the
support of patients, such as those suffering from locked-in
syndrome [37]. A common solution is eye tracking (mostly
based on vision/camera [20] or electrooculography (EOG)
[11]), which can be considered as a facial expression
approach. These solutions often include a displayed
software keyboard on which the user focuses his vision on
in order to enter text [20]. Other use cases include steering a
wheelchair by gaze, as already demonstrated by Gips [12],
who distributed several EOG electrodes onto the face
around the eyes. Furthermore, eye blinks can be used as a
binary input in order to provide locked-in patients who are
unable to control their eye movements with the ability to
communicate. Eye blinks can be detected with several
technologies, such as electroencephalography (EEG) [39],
or in an optical way [2]. Those text-input systems usually
combine a typical P300 speller in scanning mode.
Sensing Technologies for Facial Activity
Technology-wise, there are various ways to detect facial
expressions. In the following, we provide a rough overview:
Optical Sensing
The most commonly used technology is a vision-based
camera tracking of facial expressions [10]. Obvious
expressions, such as frowning, mouth movements, head
movements, etc. are detectable with high precision [3,6].
Although visual processing presents one of the most
effective techniques, it yields drawbacks: cameras are
quickly affected by bad lighting conditions, camera-based
systems are usually bulky or stationary, and very small
movements, such as tongue gestures, cannot be detected
sufficiently.
Electromyography (EMG)
The most rudimentary action is a binary on/off-switch,
which can be achieved by measuring an emerging action
potential, such as caused by contracting muscles. This has
been demonstrated by San Agustin with an EMG headband
that detects a frowning or a tightening of the user’s jaw
[34]. In TongueSee [40], 8 EMG electrodes have been
attached to the cheeks and throat to detect tongue muscle
movements. This setup enables the user to perform 6
different tongue gestures with an average accuracy of 94%.
Electroencephalography (EEG)
With EEG we usually measure neuro-activity on the
cortical surface or within the brain by so called Brain-
Computer Interfaces (BCIs). We can use BCIs as a control
in two ways: by either utilizing a clean data stream or by
using “artifacts” which are created through muscle activity,
such as by nose wrinkling, eye blinks, and other facial
expressions [27]. Matthies et al. [25] utilize eye winking,
ear wiggling, and head gestures, such as nodding and
shaking to control a handheld with Emotiv’s mobile EEG
headset. Since an EEG headset is bulky and hardly
applicable in realistic scenarios, an in-ear headset consisting
of a hacked NeuroSky EEG system and two gyroscopes
was presented, which enables the same gesture set [24]. A
similar setup, a foam earplug with two electrodes, has
recently been used to classify sleep stages [28]. In our
opinion, an ear-plug form factor is the least obtrusive setup.
Electrooculography (EOG)
With EOG Glasses, eye gestures, which are basically
tracked eye-movements, could control smart environments
such as suggested by Bulling et al. [4]. Other researchers,
such as Ishimaru et al. [19], used EOG goggles to roughly
identify chewing, talking, eating, and reading with an
accuracy of 70%. Manabe et al. attached EOG sensors to a
pair of headband headphones [21] and to an in-ear headset
[22] in order to sense eye gestures. We believe that placing
electrodes into an in-ear headset is rather unobtrusive and
apparently offers great sensing potential.
Capacitive Sensing (CS)
Rantanen et al. [30] presented a capacitive sensing glass
which is capable of detecting a frowning and a lifting of
eyebrows to execute click-events with an average accuracy
of 82.5%. In 2013, Rantanen et al. [31] furthermore
introduced a face-hugging device which consists of 12
electrodes. They found the activation of four different
muscle groups to be detectable with a proximity sensing.
While these results are impressive, wearing a face-hugger is
rather obtrusive since it almost covers the whole face.
Electromagnetic Sensing
In 2006, Fagan et al. [9] placed seven magnets on the lips,
teeth and tongue that cause a significant change in the
magnetic field when performing mouth-movements. 6 Dual
axis magnetic sensors were mounted on a prepared pair of
glasses, which enabled a detection of 13 phonemes with an
accuracy of 94%, and 9 words with an accuracy of 97%.
Even though the physical setup is quite bulky and obtrusive,
the results are impressive. In 2014, Sahni et al. [32]
attached only one magnet onto the tongue and utilized the
built-in 3 axis magnetometer of Google Glass plus an in-ear
piece measuring the optical ear canal deformations in order
to detect tongue and jaw movements. They report to be
capable of distinguishing 11 sentences with 90.5%.
Prior research reveals that we face a trade-off between
having an obtrusive hardware setup providing quite
meaningful features versus unobtrusive hardware setups
that are limited in features and recognition precision.
However, we believe that it is possible to find a more
advantageous solution compared to those presented before
a device that is unobtrusive (such as an in-ear plug) and
that still provides a reasonable feature set.
EARFIELDSENSING
We present EarFieldSensing (EarFS), an improved electric
field sensing device capable of sensing electrical changes in
the ear canal by an in-ear electrode setup. We think, hiding
a sensing device in a subtle ear plug is less obtrusive than
other approaches demonstrated in literature. Also, using
facial expressions for input enables for hands-free and eyes-
free interaction, which is safe when operating devices, such
as a smartphone, while being involved in traffic.
Contribution
As an essential part of this work, we developed an
improved electric field sensing for which we provide
technical details to enable reproduction of our sensing
technology.
To gain insights into the performance level, we conducted a
lab study to compare previous technologies with a gesture
set of 25 facial-related gestures. Compared technologies:
Electromyography EMG (Shimmer31),
Capacitive Sensing CS (FDC2214 Texas Instruments2),
Electrical Field Sensing EFS (hacked OpenCapSense [14]),
Improved Electrical Field SensingEarFS.
A comparison of technologies in a stationary setup can
reveal theoretical performance differences, but does not
reflect reality, such as when the user freely moves around.
Therefore, we conducted a second study in which we
present more insights into performance differences in a
mobile context. As a result, we found EarFS (see Figure 1)
to outperform other evaluated electrical sensing
technologies when it comes to the recognition of facial-
related gestures in mobility while walking.
1 Shimmer3 EMG Unit:
http://www.shimmersensing.com/images/uploads/docs/Shimmer3
_ECG_EMG_Specification_Sheet_Revision_1.7.pdf
2 Texas Instruments FDC2214:
http://www.ti.com/lit/ds/symlink/fdc2214.pdf
Background
In this subsection, we describe the reason of being able to
sense various facial muscle movements and head gestures
by placing a sensor piece into the ear canal.
Ear Canal
Mandibular
Condyle
Ear Plug
Mastoid
Tympanic
Membrane
Skin
Connective
Tissue
Bone
Muscle
Lateral
Pterygoid
Temporalis
Figure 2. An ear plug enables the experience of deformations
and changes in an electrical field while resting in the ear canal.
When talking of the ear canal, we mean the tunnel between
Mastoid and Mandibular Condyle (see Figure 2). Facial
expressions, such as yawning, cause an opening of the
mouth which is triggered by a contraction of the Lateral
Pterygoid. This process causes the Mandibular Condyle to
slide forward and thus a tiny void is created, which is then
filled with the surrounding tissue. A change in volume and
deformed tissue creates a very different electrical field,
which is detectable. Even eye movements and head
movements are perceivable, although the electrical change
is comparably small. As we quickly figured out, movements
of the jaw are quite easily perceivable. Other muscle
activities, such as raising eye brows, are apparently
triggered by other muscle groups (e.g. Frontalis) located on
the forehead. Still, we can sense these activities in the ear,
because many facial muscles are connected with the
Temporalis, the biggest muscle of the head, which forwards
mechanical and electrical artifacts towards the ear canal.
Performing a manual self-test: putting the pinky inside our
ear, while executing facial expressions, lets us sense these
deformations.
Nature of Signals
In a spot so small as the ear canal, we measure compound
electrical activity (white sensor noise, environmental noise,
potential changes from muscle activity, characteristic signal
peaks from ear canal deformations, and very tiny signals
from neural activity such as from brainwaves). As
mentioned before, ear canal deformations inducing
changing electrode-skin contact play a major role. As a
matter of fact, increasing skin-contact gradually decreases
the electrode input impedance and leads to a transition in
signal contributions, e.g. the action potentialsshare of the
total signal increases.
Mobile Sensing of Facial Expressions
The application of facial expression recognition via an in-
ear-positioned electric field sensing is challenging and far
more delicate than just recognizing hand/arm gestures. This
Figure 3. Schematic of the EarFS prototype, supporting both (1) single electrode and (2) differential electrode mode. In single
electrode mode, to cover electric field changes of both polarities, a large pull-up/-down resistor is used to elevate the signal level of
the earplug-electrode to half the supply voltage. We use fifteen 10 M resistors (R7-21; 10 M resistors are more common than 150
M ones) in series between a simple voltage divider (|R2|=|R3|) and the signal path in order to pull slowly enough for detecting
electric field changes. In differential mode, the INA128P instrumentation amplifier filters out environmental noise by common-
mode rejection. The difference in voltage between the earplug- (SENSOR1) and earlobe-reference (SENSOR2) electrodes is
expected to be rather small, so it is amplified by a factor of 5001 (R1 = 10 ), which is well within the gain-range of the INA128P (10k
is max). Also the output signal of the INA128P is elevated to half the supply voltage. A band-pass filter (C1 = C2 = 4.7 nF, R4 = 1.8
M, R5 = 390 k) reduces power-hum (50 or 60 Hz) by negatively feeding it back into the signal. Based on application context, C3
& R6 can be used to implement a low-pass filter of choice. Please note: band-pass- & low-pass-filtering are not compulsory.
is due to the electric field changes that are brought upon by
facial muscle movements, which are much smaller in
magnitude. Especially in a mobile situation, artifacts caused
by walking are crucial. Nevertheless, we envision a facial
gesture recognition in mobile scenarios that works
independently from side-actions, such as walking, running,
biking, jumping and sitting. For the example activity of
walking, the user’s body experiences a periodically
changing capacitive coupling to ground, which substantially
impacts an electric field sensing on any part of the humans
body. Unfortunately, it is hard to anticipate the frequency of
the signal caused by walking or running since speed levels
are likely to change when the user, for example, hurries to
catch a bus. Therefore, it is hard to target specific
frequencies for filtering out. Moreover, these frequencies
are rather low and can typically range from anywhere in
between 1 to 5 Hz, which are the same frequencies that
carry information of facial gestures.
Technical Solution
The first step of EarFS is to isolate electric field changes
brought upon by facial gestures as effectively as possible
while simultaneously reducing environmental artifacts, such
as caused by walking. As mentioned before, an option
would be to filter out periodical signals which are
reappearing over a longer period of time. However, this
does not solve the problem since parts of the unwanted
artifacts may also overlap with signals stemming from
facial expressions. A simple filtering of artifacts would
possibly erase signals of facial gestures as well, because
they are too marginal in amplitude in comparison to the
artifacts’ signal strengths. In fact, as long as artifacts occur
on the signal we cannot amplify these comparably small
facial gestures. Otherwise, the operational amplifiers would
saturate and low magnitude facial gestures are prone to
disappear in the signal. Therefore, we eliminate these high
magnitude artifacts early on by isolating them beforehand
and subtracting them from the original signal with a dual
electrode approach as described next.
Differential Amplification using a second Electrode
Our solution uses a second referenceelectrode that needs
to be placed relatively far away from the face. We then feed
a difference / instrumentation amplifier with the two
signals, the one gathered from the reference electrode, and
the other from the in-ear electrode. This way, common-
mode signals stemming from walking artifacts, which are
similar on the whole body, are likely to be filtered out or at
least substantially reduced. It is important to note that the
placement of the reference electrode is crucial, because any
limb movements may affect signals. By attaching the
reference electrode to the waist, for example, the arm would
create a change in electrical field while nearing or passing
the reference electrode when the user walks. An ideal place
of the reference electrode would be a relatively stationary
position that is far away from the face to get a significantly
different electric potential sensing compared to the
electrode placed in close proximity to the face. As a matter
of fact, the electric field strength declines exponentially
with distance, so the reference electrode can also be placed
close to the face, such as at the backside of the neck, spine,
shoulders, or at the ear lobe. While both electrodes
accumulate artificats, the in-ear electrode yields a
sufficiently different signal containing facial gestures that
remain when subtracting both signals from each other and
become visible when amplifying the subtracted signal. To
our knowledge, previous work did not use differential
amplification in this context before, and we seldom
encounter it in HCI applications yet.
Implementation
An electric field sensing circuit was designed (see Figure
4), which can be used similarly to common EFS sensing
circuitry, but also offers signal acquisition by amplifying a
differential signal from two separate electrodes. In this
mode, the differential instrumentation amplifier reduces
and even cancels out most environmental noise.
Figure 4. Left: Eagle PCB layout. U1, U2: OPA2705PA; IC1:
INA128P. Right: Final EarFS PCB. Switches offer two modes
(1) ↑↓↓ single electrode / antenna and (2) ↓↑↑ differential
electrode / antenna setup.
In order to let other researchers replicate our hardware, we
additionally provide the schematics of our sensing circuit
(see Figure 3). Once the hardware is built, one can easily
connect the Signal-Out Pin to the Analogue Input Pin of
any microcontroller board, such as A0 on an Arduino
board. As most microcontroller boards, our sensing device
also runs with 5V DC.
Single Electrode Mode and Differential Mode
Three switches have been included in the circuit to allow
the user to choose between (1) single-electrode / antenna
setup and (2) differential electrode /antenna setup. (1) The
slider switch in Figure 3’s top left corner connects PAD2 to
PAD1, SW2 is off and SW1 is on. (2) All three switches are
being reversed the slider switch connects PAD2 to PAD3.
A pull-up/down resistor was included for single-electrode
(S1) usage, so that electric field signals will return to the
baseline of half VCC when no change in electric fields is
present. Thus, only movements that create field changes are
perceivable. Concerning the differential configuration, the
instrumentation amplifier was biased to half VCC, so that
electric potential changes of either polarity can be sensed.
STUDY 1: TECHNOLOGY PERFORMANCE
In this section, we evaluate the detection of facial-related
gestures by a variety of electric sensing technologies.
Research Questions
The goal of this study was to gain an insight into the
following research questions while trying to keep all
variables as constant as possible (e.g., testing all setups by
the same user, only testing one session per day):
Q1: How does our technology perform compared to other
electric sensing technologies?
Q2: What would be the best electrode setup providing the
highest accuracy rates for each technology?
Q3: Which gestures are the top 5 performing ones with the
given technology?
In study 1, we were not yet interested in finding out about
varying performance levels across users, nor the
applicability in mobile scenarios. Therefore, we forfeited on
testing all possible setups with multiple users in mobility.
Task and Procedure
To answer these research questions, we performed an
extensive study in which we recorded 14,000 gestures (= 7
ear plugs * 2 un/covered * 25 gestures * 10 repetitions * 4
sensing technologies) from a single user. To avoid fatigue
effects, we split the recordings into several sessions, which
included 1 technology with all earplugs in sequential order.
25 gestures * 10 reps were recorded with each setup before
insulating the earplug or taking the next one. Each gesture
was recorded in a time window of 1.25s. To prevent invalid
data distortion, the earplug was not rearranged during
sessions. When the user was not sure about the correct
execution, he was enabled to record an additional repetition.
The test subject trained steady gesture execution
beforehand and triggered the recording manually after
being randomly presented with a gesture left in the pool of
250. A complete session contained 250 * 7 = 1750 gestures.
Facial Gesture Set
We compiled a set of 25 facial- and head-related gestures
(see Figure 5) to compare all technologies based on their
performance level. The gesture set covers a broad spectrum
of which we are aware that not all of them are subtle or
socially acceptable. The set was chosen for straightforward
repeatability while it includes gestures involving various
muscle groups. The contraction of different muscle groups
presumably leads to a distinctive signal in order to identify
gestures. Apart from typical gestures, such as ‘eye-wink’,
‘smile’, and ‘protrude-tongue’, simple speech was included
as well, because speech is performed highly automated due
to it being easy to memorize.
linguaOculi
Cervice
Ostium
say-a say-sh say-tee say-e say-r say-g say-i say-o say-u
eye-wink eyes-down eyes-left eyes-right eyes-up lift-eyebrows pull-eyebrows
together head-back chin-on-chest head-right head-left default
smile open-mouth press-lips
together protude-tongue
Figure 5. With a set of 25 gestures (including a default gesture) we evaluated four different technologies (EMG, CS, EFS, EarFS) .
Apparatus: Electrode Ear Plug
We prepared 7 earplugs which are made out of
polyurethane foam and go by the name of OHROPAX
Color3 (see Figure 6). #1 is a single electrode wrapped
around the earplug. #2h - #4h are two to four increasingly
smaller electrodes wrapped horizontally around the earplug
in a similar fashion. #2v - #4v have electrodes in decreasing
sizes, which are vertically placed alongside the earplug.
Accordingly, #2v has two electrodes, #3v has three, and
#4v has four electrodes mounted on the earplug.
horizontal vertical
1 2h 3h 4h 2v 3v 4v
Figure 6. With 7 different electrode layouts we evaluated each
technology. For us, it seemed natural arranging the electrodes
lengthwise and widthwise alike with varying partitions while
we used them blank (as shown), and covered.
All electrodes were cut out from copper foil, soldered to the
connecting cables, and subsequently glued onto the
earplugs with Pattex superglue. All 7 setups have been
tested both with blank electrodes and while being covered
with the cut-off tip of a common condom. The lubricant
was thoroughly washed off beforehand, and remaining
moisture left on the latex was dried off before conducting
experiments.
Apparatus: Electromyography (EMG)
EMG is the most common technology to measure action
potentials stemming from muscle activity, which is usually
done invasively by needle electrodes. Nonetheless, the
superimposed voltage is also detectable on the surface of
the skin while it still shows ranges of up to -100mV [18]. In
an interaction scenario, surface electrodes on the skin are
typically used [33] for measuring electrical potentials
through a relatively thick layer of skin and fat. For
classifying gestures one can use not only a clean signal, but
also noise [26] and accumulated movement artifacts [24],
which occur in the ear canal when performing gestures.
3OHROPAX Color: http://www.ohropax.de/produkte/color.html
Figure 7. Shimmer3 ECG/EMG Bluetooth device, configured
in EMG mode.
In our study, two Shimmer3 EXG units1 were connected via
Bluetooth to a computer (see Figure 7). The Shimmer
Android/Java API was used to configure the EMG units and
to establish communication. A suggested digital filtering
(50 Hz noise cancellation and low-pass filtering for signal
smoothing) was also implemented. The earplug electrodes
were connected to a single channel each in the following
way: The earplug-electrode was connected to the positive
differential input of the Shimmer3 EMG channel and a
clothespin-mounted copper foil reference electrode was
clipped to the earlobe of the opposite ear that the earplug
was worn in. The reference electrode was connected to the
REF input of the up to three Shimmer3 units and connected
to all negative differential inputs of active channels.
Apparatus: Capacitive Sensing (CS)
Capacitance describes a body’s ability to store an electrical
charge when a voltage is applied. The higher the electrical
charge a body can store, the higher its’ capacitance. As a
matter of fact, the human body’s cells also have the ability
to store electrons and thus a negative electrical charge.
Depending on the body part, we can speak of an overall
capacitance varying between 50 and 150pF [35]. Excited
cells, which accumulate a certain amount of electrons,
create the change in capacitance. While this capacitance can
be measured invasively, we can also measure it on top of
the skin or in distance, such as with an isolated earplug
electrode. A typical CS measures the charging time of an
electrode. This is also referred to as loading mode [38].
Figure 8. The Capacitive Sensing shield FDC2214 EVM from
Texas Instruments was plugged to an Arduino board
transmitting the raw data via a Bluetooth 2.0 modem.
The FDC22142 also uses capacitive sensing in loading
mode. We connected it to a Genuino Micro streaming all
raw data via an HC05 Bluetooth modem (see Figure 8). It is
essential to use a battery plus a wireless transmission to
avoid irregularities, such as a varying capacitive ground
coupling triggered by other hardware components that may
also be connected to the computer. To measure each of the
four channels in turn, 512 oscillations were used to
determine the momentary frequency of the LC oscillator
circuit compared to the EVM board’s 40 MHz oscillator.
After each channel switch, the first 128 oscillations were
not considered to allow for the frequency to stabilize.
Apparatus: Electrical Field Sensing (EFS)
Figure 9. We “hacked” four Loading Mode capacitive sensors
from OpenCapSense [14] to act like an Electric Field Sensor.
Electric fields are ubiquitous and exist due to the static
electricity of our surroundings. Besides everyday objects,
also the human body carries several small electrical fields.
Fluctuations in electric fields quickly occur when moving
the human body or other charged objects. While we can
utilize electrical field changes for a gesture recognition [7],
it is also perfectly suitable for an intended facial expression
recognition. However, factors such as ambient noise and
baseline drift are the most cumbersome obstacles that
gesture recognition and classification endeavours face.
Anyhow, all that is needed to implement electric field
sensing is basically apassive” electrode (i.e. antenna) with
an operational amplifier connected to an analogue-to-digital
converter (ADC). To compensate for noise, such as power
hum, low-pass filters may apply between op-amp and ADC.
Our EFS setup consists of four „hacked“ OpenCapSense
loading mode sensors, which basically consist of an
operational amplifier and an astable Multivibrator that is
usually used for a capacitive measurement. However, we
only utilize the op-amp whose positive input is connected to
the electrode. The op-amp’s output is connected to the
analogue input of an Arduino Nano (see Figure 9), which
serves as an ADC and transmits the raw data. It should be
particularly noted that here, the op-amps are not connected
to an external power source. However, they still output
discriminable voltage based on the acquired earplug signal,
which also serves as a power supply in a way that the
electrode is wired to the op-amp pin right next to the
negative supply pin, facilitating a discriminable voltage
between the negative and positive op-amp supply pins.
Apparatus: EarFS
Figure 10. Four EFS shields are connected to an Arduino in
order to use a four-electrode ear plug. The data is being
streamed via a Bluetooth 2.0 modem to a computer, while the
prototype is powered by a 9V battery.
Fluctuations in ambient electric fields can originate both
from negative and positive charge balance and thus, a
standard single supply op-amp design like seen before
would be doomed to miss one of the polarities. Therefore,
we introduce a second DC-voltage, keeping the antenna
voltage at a proportionally steady and elevated level. It is
wise to choose a DC-voltage of half the op-amp’s supply
voltage, since in this way, incoming antenna signals can
deviate from the baseline voltage in the direction of both
electrical polarities. When no changing electric field is
present, a relatively large resistor pulls up/down the antenna
voltage to the baseline voltage eventually. It should be
noted that larger resistors cause longer latencies. The
addition of such a pull up/down resistor with its tendency to
pull the antenna voltage back to half the VCC voltage is the
reason that only movements and changes are measurable. In
addition, we added a reference electrode (see Figure 10) to
eliminate extrinsic changes in electrical fields with a
differential amp.
EMG
(Shimmer3)
CS
(FDC2214 Texas Instruments)
EFS
(hacked OpenCapSense)
EarFS
Electrodes
Average Accuracy (TP)
Average Accuracy (TP)
Average Accuracy (TP)
Average Accuracy (TP)
blank
all 25
n50%
top 5
all 25
n50%
top 5
all 25
n50%
top 5
all 25
n50%
top 5
1
19.2%
-
-
1.6%
-
-
1.6%
-
-
3.2%
-
-
horizontal
2
11.7%
-
-
16.8%
-
-
35.1%
-
-
5.2%
-
-
3
4.8%
-
-
30.8
-
-
40.8%
-
-
8.8%
-
-
4
12.8%
-
-
43.2%
-
-
39.5%
-
-
13.2%
-
-
vertical
2
30.8%
4
84%
12%
-
-
43.6%
-
-
13.6%
-
-
3
10.4%
-
-
34.4%
-
-
52%
11
94.5%
12.4%
-
-
4
19.2%
-
-
48.4%
13
90%
49%
-
-
32%
5
90%
covered
1
11.6%
0
64.4%
9.6%
-
-
4%
-
-
5.6%
0
20%
horizontal
2
5.6%
-
-
38%
11
84%
4.4%
-
-
2.8%
-
-
3
6.4%
-
-
35.2%
-
-
5.2%
-
-
4%
-
-
4
6.8%
-
-
21.6%
-
-
4.4%
-
-
4.8%
-
-
vertical
2
6.2%
-
-
22.4%
-
-
3.6%
-
-
4%
-
-
3
6%
-
-
29.6%
-
-
7.6%
1
20%
5.2%
-
-
4
3.2%
-
-
28%
-
-
5.6%
-
-
2.4%
-
-
Table 1. Performance levels using a J48 DT (C4.5 algorithm). For each technology we can find three columns: 1) true-positive (TP)
rates of the complete gesture set, 2) number of gestures yielding at least 50% TP, and 3) TP score of a reduced top 5 gesture set.
Signal Gathering and Data Processing
The aforementioned electrode-earplugs have been
combined with all four technologies while we recorded
each gesture with a sample rate of 200 Hz and a window-
size of 256. Then, we computed 46 state-of-the-art features
found in literature on all raw data recordings. Because we
are not aware of any library providing them, we
implemented them by hand in Java. For analysing the data,
we use the Weka data mining tool [11] in order to gain an
impression on the performance using five state-of-the-art
classifiers (Bayes Net - BN, K-nearest neighbours - Ibk,
J48 Decision Tree J48, Random Forest - RF, Sequential
Minimal Optimization - SMO) while performing a stratified
10-fold-crossvalidation. We have chosen this method,
because conducting a manual leave-kinstances-out method on
our huge dataset (14.000 instances) is extremely time
consuming and beyond practicality. Nevertheless, we had a
quick look (k=5) at a single session (EarFS, 4-vertical) and
could perceive a marginal accuracy drop of Δ= -1.6 %.
Results
Before presenting the result, it is important to note that we
are talking of a theoretical performance level. To make a
sophisticated statement on realistic recognition rates, one
should have tested users n>10 in ambiguous environments
(including critical environments with high level of electric
noise, e.g. a server room). In this paper, we decided to keep
experiments within reasonable boundaries and share early
results of the exact composition with the community.
Classifier & Feature Selection
In order be able to answer our research questions, we first
determined the best classifier. We compared all five
classifiers (BN, J48, Ibk, SMO, RF) by means of an
independent samples one-way ANOVA, but which showed
no significant differences for EMG (F4,30=1.14; p<.357);
CS (F4,30=0.58; p<.680); EarFS (F4,30=0.06; p<.993).
Nevertheless, the EFS showed strong significant differences
(F4,30=17.96; p<.0001). Conducting a Tukey HSD Test
revealed the J48 (M=43.35; SD=6.25), BN (M=43.47;
SD=5.50), and RF (M=37.57; SD=12.12) to perform better
than the Ibk (M=20.30; SD=7.53; p<.01). Moreover, the J48
and BN were deemed to significantly perform better than
the SMO (M=31.86; SD=8.29; p<.05). Beholding the mean
performance over all technologies, we can perceive the J48
and the RF to perform quite well. Because the J48 is most
computationally inexpensive and a rather simple classifier,
we selected it for further investigations.
Across all best setups, top 5 meaningful features, selected
by a Greedy Stepwise (forwards) algorithm [5], include:
spectralEnergy, spectralFlux, spectralSignalToNoiseRatio,
minMaxDifference, and pairDifference.
Answering Research Questions
Q1: As seen in Table 1, EarFS performs similar to other
electric sensing technologies, comparing their best setups.
A one-way ANOVA (F3,27=193.91; p<.001) showed EarFS
(M=32%) to perform equally to the EMG (M=30.8%) and
EFS (M=52%) equally to CS (M=48.4%). Still, a Tukey
HSD (p<.01) reveals both EFS and CS to perform
significantly better among EMG and EarFS.
Q2: The electrode setups providing best performance are
indicated in Table 1. Generally, we can say that non-
insulated, vertically arranged electrodes perform better,
because these are more sensitive towards ear-canal
deformations (changing skin / electrode contact). Since the
vertical electrodes are distributed in circular fashion, an
increase in their number leads to higher spatial resolution
inside the ear canal.
Q3: We determined a top 5 gesture set for the best setup of
each technology (see Table 2). In fact, the recognition rates
look quite reasonable and foster curiosity: EMG (M=84%),
CS (M=90%), EFS (M=94.5%), and EarFS (M=90%).
EMG
(Shimmer3)
CS
(FDC2214)
EFS
(OpenCS) EarFS
1
eyes-left head-back
chin-on-
chest
eye wink
2
head-back open-mouth eye wink
head-
right
3
head-left
protrude-
tongue
say-u
open-
mouth
4
say-e
eye-brows
together
eyes-down say-sh
5
smile say-a head-right smile
Table 2. Top 5 gestures for the best technology setup. We
chose to select the number of 5 gestures, because the ability to
remember shortcuts, such as gestures, dramatically decreases
with larger numbers than 7 in a real scenario. Following
cognitive engineering, 5 is also a suggested maximum.
Summary
The analysis revealed all technologies to be capable of a
facial-gesture recognition by measuring them inside the ear
canal. In our opinion, the classification accuracy is
astonishing considering the broad gesture set of 25 facial
expressions. Two characteristic ‘clusters’ of confusions
occurred among the gestures. One cluster can be found
around gestures of the Oculi, and the other around the
Lingua. Because these gestures are similar in type, the
confusion between them is most likely connected to their
actual similarity.
STUDY 2: WEARABLE PERFORMANCE
Since the first study was performed in a very controlled
environment, we thought it may be interesting to see
whether our evaluated technologies could be employed as a
wearable technology in a mobile context as well.
Study Setup
Therefore, we conducted an experiment with 3 participants,
aged 26, 29, and 30 years. While each technology was
tested with all users, the task was to perform all top 5
gestures of each technology (see Table 2) with itsbest
earplug setup for 10 times in a random order.
There was a marginal training phase in which the user had
the chance to perform each gesture once or twice. After the
study started, the study leader was shouting each gesture
out loud while he was triggering the recording. To test the
technologieslimits, we instructed each user to randomly
walk around within a spot of 10 x 10 meters in a medium-
sized lobby with stone-tiled floor.
In summary, we recorded 600 gestures (3 users * 4
technologies * 5 gestures * 10 repetitions). We again
calculated 46 state-of-the-art features from the raw data and
used a J48 Decision Tree while performing a stratified 10-
fold-crossvalidation.
Hypotheses
Since we already know about the theoretical performance in
a stationary context, we can establish these hypotheses:
H1: EarFS will perform equally or better than other
technologies, because it works with a differential
amplification. Hence, it should be more robust
towards influences from external noise in mobility.
H2: All other technologies will experience a substantial
drop in accuracy, because they are heavily affected by
environmental noise occurring while moving.
Results
The results confirm our assumption. EarFS performs well
in context of mobility. Table 3 shows the performance of
EarFS in a confusion metrics accumulated over all users:
a
b
c
d
e
<- classified as
96.7%
3.3%
-
-
-
a = eye wink
- 89.7%
3.4% - 6.9% b = head right
3.3%
-
80.0%
16.7%
-
c = open mouth
-
-
13.3%
80.0%
6.7%
d = say SH
-
3.3%
6.7%
10.0%
80.0%
e = smile
Table 3. Accumulated confusion matrix of all users showing
overall performance of the EarFS using a J48 decision tree.
Answering Hypotheses
H1: Looking at Table 4, we can see that over all users,
EarFS (M=85.2%) achieves a substantially higher mean
accuracy than EMG (M=76.7%) and CS (M=79.9) when the
user walks around randomly. A one-way ANOVA
(F3,8=6.27; p<.02) also found statistical differences in terms
of performance level. A Tukey HSD test confirmed our
technology to significantly outperform EFS (M=73.7%).
Therefore, we accept this hypothesis: EarFS is more robust
towards external noise in mobility and yields higher
accuracy while it even significantly outperforms EFS.
EMG
(Shimmer3)
CS
(FDC2214)
EFS
(OpenCS)
EarFS
84%
90%
94.5%
90%
sitting
76.7%
79.9%
52.8%
85.2%
walking
80.4% 85% 73.7% 87.6% Ø
Table 4. Overall performance (True-Positive rates) of study 1
(sitting) in comparison to study 2 (walking). The setup: top 5
gestures, preferred electrode setup, J48 classifier.
Incidentally, it is even more surprising to see that EFS
initially outperformed EarFS while sitting. One reason
would be because OpenCapSense is a more integrated PCB
and does not suffer from small distortions of loose wires
like EarFS. However, as shown before, it is bound to
underperform while walking, since it is not supporting
differential measurements.
H2: Running a simple t-Test confirms CSwalking (M=79.9%)
to be significantly worse than CSsitting (M=90%). Also,
EFSwalking (M=52.8%) is performing significantly worse
than EFSsitting (M=94.5%). We can also see a decrease from
EMGsitting (M=84%) to EMGwalking (M=76.7%). However,
while EMG is generally performing low, it is not yet
statistically different. EarFS experiences the lowest
accuracy drop (Δ= -4.8%) and does not perform
significantly worse. Although CS and EFS significantly
dropped in accuracy, we have to dismiss this hypothesis,
because EMG did not significantly decrease.
Summary
The second study shows EarFS to not experience a
substantial performance drop in mobility while the user is
walking. Moreover, the study reveals that EMG is also not
heavily affected by walking artifacts due to the nature of
its’ sensing method. Therefore, the study indicates that
electrical field sensing related technologies may not be the
perfect choice for a wearable gesture recognition, unless
one applies a differential amplification, such as proposed in
EarFS.
DISCUSSION
Considering the rather rudimentary electrode setup and the
low-cost sensing device, in our opinion, the achieved
classification accuracy above 90% with a gesture set of five
is astonishing. This is due to the heterogeneous signal,
which is a combination of facial-movement-induced ear
canal deformations and biopotential processes. Still, a
custom six channel monopolar EMG, using surface
electrodes similar to Zhang et al. [40] but distributed over
the entire face, tends to outperform any in-ear setups. We
confirmed this in a pilot study where we attached 7
silver/silver chloride gel electrodes to the face in places
right above facial muscles of interest (see Figure 11).
Figure 11. We utilized 3 Shimmer3 EXG sensor devices with 7
Ag/AgCl gel electrodes (6 channels + 1 common ground placed
behind the ear, an area that remains relatively unaffected by
muscular movement) to detect same gesture set.
We again recorded the complete facial gesture set of 25
with a sampling rate of 200Hz and a window size of 256. A
total of 346 features (based on 46 state-of-the-art features)
have been extracted from the raw data, whereby the most
meaningful features included: maxAmpFrequency,
spectralEntropy, and logLikelihood. With this setup, a
RandomForest classifier performed best while detecting 25
facial gestures with an accuracy of 62%. A reduced set of
only 5 facial gestures scored maximum accuracy of 100%.
This pilot clearly highlights the typical trade-off between
technology that is obtrusive on the one hand, but on the
other hand achieves high accuracy rates. Scoring
comparably low precision with an in-ear setup is not
surprising, since (1) the maximum number of channels
tested with the earplugs was four and (2) sensors cannot
directly sense evoking action potentials from the source
while resting inside the ear canal. Nevertheless, we expect
EarFS to technically mature with further iterations (testing
different building blocks, EM shielding). However, placing
more electrodes inside the ear is not expected to provide
significant performance boosts. Instead, a combination of
different technologies seems promising and is highly
encouraged for further research. While electrodes with
direct skin contact could be combined with electrically
insulated electrodes, it did not increase performance in our
study. In contrast, a future improvement would be to
additionally determine the deformations of the ear canal
with pressure-activated distance sensors. Another method
would be laser-based distance measurements by using
modulated laser beams and image-based phase-shift
analysis in order to get a distance-to-skin measurement
inside the ear canal. Particularly, laser modulation
frequencies would have to be very high to cover the sub-
millimetre distance range in this approach, and thus suitable
hardware would increase the costs of such a sensing device.
CONCLUSION
In this paper, we presented a novel variant of an electrical
field sensing (EarFS) which provides hands-free and partly
eyes-free interaction for mobile and wearable computing.
We introduced our developed sensing circuit in detail so
that it can be replicated by any HCI researcher or
practitioner. With EarFS, we closed an open gap in
research while we systematically investigated detecting
various facial-related gestures via an electric field sensing
inside the ear canal, which has not been done before in this
manner. We provided two studies that reveal how electric
sensing technologies could possibly perform when using an
electrode in-ear plug. On top of that, we were able to show
that EarFS tends to outperform other electrical sensing
approaches when it comes to facial-gesture recognition in
mobility while the user is on the go.
FUTURE WORK
Since facial gestures and expressions cannot typically be
‘switched off’ by users, the field of mobile facial expression
recognition still yields great potential as far as implicit
interaction is concerned. Based on facial expressions, a
future system would be able to know and anticipate the
user’s intentions before conscious interaction becomes
necessary. In terms of apparatus, we believe that in-ear
devices, such as earbuds, are much more unobtrusive and
socially acceptable than other known hands-free and eyes-
free technologies. Therefore, we envision similar sensing
approaches to be integrated into in-ear headsets in the near
future. Besides headsets, we also see great potential in
EarFS to be implemented into various other kinds of
wearables, since our sensing approach offers a much wider
range of recognition capabilities for gestures and activities
in mobility than discussed in this paper. Exploring these
capabilities in future research could be very beneficial.
ACKNOWLEDGMENTS
We would like to thank all reviewers, in particular Thijs
Roumen, for spending their valuable time on providing very
valuable feedback. This research is supported by the
Federal Republic of Germany under the following grant:
BMWi 16KN049121.
REFERENCES
1. Ashbrook, D. (2010). Enabling mobile
microinteractions. PhD Thesis, Georgia Institute of
Technology
2. Ashtiani, B., & MacKenzie, I. S. (2010). BlinkWrite2:
an improved text entry method using eye blinks. In
Proceedings of the 2010 Symposium on Eye-Tracking
Research & Applications, 339-345. ACM.
http://dx.doi.org/10.1145/1743666.1743742
3. Bartlett, M. S., Littlewort, G., Fasel, I., & Movellan, J.
R. (2003, June). Real Time Face Detection and Facial
Expression Recognition: Development and
Applications to Human Computer Interaction. In
Conference on Computer Vision and Pattern
Recognition. CVPRW'03. (Vol. 5), 53-53. IEEE.
http://dx.doi.org/10.1109/CVPRW.2003.10057
4. Bulling, A., Roggen, D., & Tröster, G. (2009).
Wearable EOG goggles: eye-based interaction in
everyday environments, In CHI'09 Extended Abstracts
on Human Factors in Computing System, 3259-3264.
ACM.
http://dx.doi.org/10.1145/1520340.1520468
5. Caruana, R., & Freitag, D. (1994). Greedy Attribute
Selection. In Proceedings of International Conference
on Machine Learning, 28-36.
6. Chai, J. X., Xiao, J., & Hodgins, J. (2003). Vision-
based control of 3d facial animation. In Proceedings of
the 2003 ACM SIGGRAPH/Eurographics symposium
on Computer animation, 193-206. ACM.
7. Cohn, G., Gupta, S., Lee, T. J., Morris, D., Smith, J. R.,
Reynolds, M. S., ... & Patel, S. N. (2012). An ultra-
low-power human body motion sensor using static
electric field sensing. In Proceedings of the 2012 ACM
Conference on Ubiquitous Computing, 99-102. ACM.
http://dx.doi.org/10.1145/2370216.2370233
8. Ekman, P., & Friesen, W. V. (1977). Facial action
coding system. Consulting Psychologists Press,
Stanford University, Palo Alto.
9. Fagan, M. J., Ell, S. R., Gilbert, J. M., Sarrazin, E., &
Chapman, P. M. (2008). Development of a (silent)
speech recognition system for patients following
laryngectomy. In Medical engineering & physics,
30(4), 419-425.
http://dx.doi.org/10.1016/j.medengphy.2007.05.
003
10. Fasel, B., & Luettin, J. (2003). Automatic facial
expression analysis: a survey. In Pattern recognition,
36(1), 259-275. Elsevier.
http://dx.doi.org/10.1016/S0031-3203(02)00052-
3
11. Gips, J., Olivieri, P., & Tecce, J. (1993). Direct Control
of the Computer Through Electrodes Placed Around
the Eyes. In Human-Computer Interaction:
Applications and Case Studies, 630-635. Elsevier.
12. Gips, J. (1998). On building intelligence into
EagleEyes. In Assistive technology and artificial
intelligence, 50-58. Springer Berlin Heidelberg.
http://dx.doi.org/10.1007/BFb0055969
13. Grafsgaard, J., Wiggins, J. B., Boyer, K. E., Wiebe, E.
N., & Lester, J. (2013). Automatically recognizing
facial expression: Predicting engagement and
frustration. In Educational Data Mining 2013.
14. Grosse-Puppendahl, T., Berghoefer, Y., Braun, A.,
Wimmer, R., & Kuijper, A. (2013). OpenCapSense: A
rapid prototyping toolkit for pervasive interaction using
capacitive sensing. In Proceedings of International
Conference on Pervasive Computing and
Communications, 152-159. IEEE.
http://dx.doi.org/10.1109/PerCom.2013.6526726
15. Hall, M., Frank, E., Holmes, G., Pfahringer, B.,
Reutemann, P., & Witten, I. H. (2009). The WEKA
data mining software: an update. In ACM SIGKDD
explorations newsletter, 11(1), 10-18.
http://dx.doi.org/10.1145/1656274.1656278
16. Hausen, S. (2014) Peripheral Interaction - Exploring
the Design Space, PhD Thesis, Faculty of
Mathematics, Computer Science and Statistics,
University of Munich.
17. Hazlett, R. (2003). Measurement of user frustration: a
biologic approach. In CHI'03 extended abstracts on
Human factors in computing systems, 734-735. ACM.
http://dx.doi.org/10.1145/765891.765958
18. Hodgkin, A. L. (1951). The ionic basis of electrical
activity in nerve and muscle. In Biological Reviews,
26(4), 339-409.
http://dx.doi.org/10.1111/j.1469-
185X.1951.tb01204.x
19. Ishimaru, S., Kunze, K., Uema, Y., Kise, K., Inami,
M., & Tanaka, K. (2014). Smarter eyewear: using
commercial EOG glasses for activity recognition. In
Proceedings of the 2014 ACM International Joint
Conference on Pervasive and Ubiquitous Computing:
Adjunct Publication, 239-242. ACM.
http://dx.doi.org/10.1145/2638728.2638795
20. Majaranta, P., & Räihä, K. J. (2007). Text entry by
gaze: Utilizing eye-tracking. In Text entry systems:
Mobility, accessibility, universality, 175-187.
21. Manabe, H., & Fukumoto, M. (2006). Full-time
wearable headphone-type gaze detector. In CHI'06
Extended Abstracts on Human Factors in Computing
Systems, 1073-1078. ACM.
http://dx.doi.org/10.1145/1125451.1125655
22. Manabe, H., Fukumoto, M., & Yagi, T. (2015).
Conductive rubber electrodes for earphone-based eye
gesture input interface. In Personal and Ubiquitous
Computing, 19(1), 143-154. Springer.
http://dx.doi.org/10.1007/s00779-014-0818-8
23. Mann, S. (1998). Humanistic computing: “WearComp”
as a new framework and application for intelligent
signal processing. In Proceedings of the IEEE, 86(11),
2123-215. IEEE.
http://dx.doi.org/10.1109/5.726784
24. Matthies, D. J. C. (2013). InEar BioFeedController: a
headset for hands-free and eyes-free interaction with
mobile devices. In CHI'13 Extended Abstracts on
Human Factors in Computing Systems, 1293-1298.
ACM.
http://dx.doi.org/10.1145/2468356.2468587
25. Matthies, D. J. C., Antons, J. N., Heidmann, F.,
Wettach, R., & Schleicher, R. (2012). NeuroPad: use
cases for a mobile physiological interface. In
Proceedings of the 7th Nordic Conference on Human-
Computer Interaction: Making Sense Through Design,
795-796. ACM.
http://dx.doi.org/10.1145/2399016.2399152
26. Matthies, D. J.C., Perrault, S. T., Urban, B., & Zhao, S.
(2015). Botential: Localizing On-Body Gestures by
Measuring Electrical Signatures on the Human Skin. In
Proceedings of the 17th International Conference on
Human-Computer Interaction with Mobile Devices and
Services, 207-216. ACM.
http://dx.doi.org/10.1145/2785830.2785859
27. McFarland, D. J., & Wolpaw, J. R. (2011). Brain-
computer interfaces for communication and control. In
Communications of the ACM, 54(5), 60-66. ACM.
http://dx.doi.org/10.1145/1941487.1941506
28. Nguyen, A., Alqurashi, R., Raghebi, Z., Banaei-
kashani, F., Halbower, A. C., & Vu, T. (2016). A
Lightweight And Inexpensive In-ear Sensing System
For Automatic Whole-night Sleep Stage Monitoring. In
Proceedings of the 14th ACM Conference on
Embedded Network Sensor Systems 2016, 230-244.
ACM.
http://dx.doi.org/10.1145/2994551.2994562
29. Picard, R. W. (1995). Affective computing. MIT Media
Laboratory Perceptual Computing Section Technical
Report No. 321. Cambridge: MIT press.
30. Rantanen, V., Niemenlehto, P. H., Verho, J., &
Lekkala, J. (2010). Capacitive facial movement
detection for human–computer interaction to click by
frowning and lifting eyebrows. In Medical &
biological engineering & computing,48(1), 39-47.
http://dx.doi.org/10.1007/s11517-009-0565-6
31. Rantanen, V., Venesvirta, H., Spakov, O., Verho, J.,
Vetek, A., Surakka, V., & Lekkala, J. (2013).
Capacitive measurement of facial activity intensity. In
IEEE Sensors Journal, 13(11), 4329-4338.
http://dx.doi.org/10.1109/JSEN.2013.2269864
32. Sahni, H., Bedri, A., Reyes, G., Thukral, P., Guo, Z.,
Starner, T., & Ghovanloo, M. (2014). The tongue and
ear interface: a wearable system for silent speech
recognition. In Proceedings of the 2014 ACM
International Symposium on Wearable Computers, 47-
54. ACM.
http://dx.doi.org/10.1145/2634317.2634322
33. Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan,
R., Turner, J., & Landay, J. A. (2009). Enabling
always-available input with muscle-computer
interfaces. In Proceedings of the 22nd annual ACM
symposium on User interface software and technology,
167-176. ACM.
http://dx.doi.org/10.1145/1622176.1622208
34. San Agustin, J., Hansen, J. P., Hansen, D. W., &
Skovsgaard, H. (2009). Low-cost gaze pointing and
EMG clicking. In CHI'09 Extended Abstracts on
Human Factors in Computing Systems, 3247-3252.
ACM.
http://dx.doi.org/10.1145/1520340.1520466
35. Scharfetter, H., Hartinger, P., Hinghofer-Szalkay, H.,
& Hutten, H. (1998). A model of artefacts produced by
stray capacitance during whole body or segmental
bioimpedance spectroscopy. In Physiological
measurement, 19(2), 247.
http://dx.doi.org/10.1088/0967-3334/19/2/012
36. Scheirer, J., Fernandez, R., & Picard, R. W. (1999).
Expression glasses: a wearable device for facial
expression recognition. In CHI'99 Extended Abstracts
on Human Factors in Computing Systems, 262-263.
ACM.
http://dx.doi.org/10.1145/632716.632878
37. Smith, E., & Delargy, M. (2005). Locked-in syndrome.
BMJ, 330(7488), 406-409.
38. Smith, J. R. (1999). Electric Field Imaging. PhD
Thesis, MIT - Massachusetts Institute of Technology.
39. Yin, E., Zhou, Z., Jiang, J., Chen, F., Liu, Y., & Hu, D.
(2013). A novel hybrid BCI speller based on the
incorporation of SSVEP into the P300 paradigm. In
Journal of neural engineering, 10(2), 026012.
http://dx.doi.org/10.1088/1741-
2560/10/2/026012
40. Zhang, Q., Gollakota, S., Taskar, B., & Rao, R. P.
(2014). Non-intrusive tongue machine interface. In
Proceedings of the 32nd annual ACM conference on
Human factors in computing systems, 2555-2558.
ACM.
http://dx.doi.org/10.1145/2556288.2556981
Project
Ubiquitous systems are becoming an integral part of our everyday lives. Functionality and user experience often depends on accurate, sensor-based activity recognition and interaction. Systems aimin…" [more]
Project
Improvement of current approaches in Ballisto- and Seismo-Cardiography, in order to achieve higher accuracy for derived vital parameters (e.g. heart rate, heart rate variability). The aim is to red…" [more]
Project
Wearable sensors potentially enable for a better and unobtrusive recognition of human activity and the state of rest, sleep, stress and drive the ongoing trend of the quantified self-movement. As a…" [more]
Conference Paper
August 2015
    Figure 1. Botential leverages the entire human body as an interactive surface, while enabling the identification of tap gesture locations on different body parts and a hovering through the clothes. It is conceivable to integrate this technology with various wearable objects, such as a) Necklace b) Bracelet c) Ring d) Belt and e) Legband. ABSTRACT We present Botential, an on-body interaction... [Show full abstract]
    Conference Paper
    July 2015
      In this research project, we present an alternative approach to recognize various walking-based activities based on the technology of capacitive sensing. While accelerometry-based walking detections suffer from reduced accuracy at low speeds, the technology of capacitive sensing uses physical distance parameters, which makes it invariant to the duration of step performance. Determining... [Show full abstract]
      Conference Paper
      April 2013
        Nowadays control of a mobile device usually requires the use of a touch screen. In everyday life, while engaged in real world tasks, it is sometimes impossible to control a device with one's hands. Speech control is the most common solution to tackle this problem, but it is still error prone, uncomfortable and works poorly when ambient noise is present. Alternative control concepts must work... [Show full abstract]
        Conference Paper
        September 2017
          Foot interfaces, such as pressure-sensitive insoles, still yield unused potential such as for implicit interaction. In this paper, we introduce CapSoles, enabling smart insoles to implicitly identify who is walking on what kind of floor. Our insole prototype relies on capacitive sensing and is able to sense plantar pressure distribution underneath the foot, plus a capacitive ground coupling... [Show full abstract]
          Discover more