ArticlePDF Available

# A Hybrid Brain-Computer Interface Based on Visual Evoked Potential and Pupillary Response

Authors:
• Institute of semiconductors, CAS

## Abstract

Brain-computer interface (BCI) based on steady-state visual evoked potential (SSVEP) has been widely studied due to the high information transfer rate (ITR), little user training, and wide subject applicability. However, there are also disadvantages such as visual discomfort and “BCI illiteracy.” To address these problems, this study proposes to use low-frequency stimulations (12 classes, 0.8–2.12 Hz with an interval of 0.12 Hz), which can simultaneously elicit visual evoked potential (VEP) and pupillary response (PR) to construct a hybrid BCI (h-BCI) system. Classification accuracy was calculated using supervised and unsupervised methods, respectively, and the hybrid accuracy was obtained using a decision fusion method to combine the information of VEP and PR. Online experimental results from 10 subjects showed that the averaged accuracy was 94.90 ± 2.34% (data length 1.5 s) for the supervised method and 91.88 ± 3.68% (data length 4 s) for the unsupervised method, which correspond to the ITR of 64.35 ± 3.07 bits/min (bpm) and 33.19 ± 2.38 bpm, respectively. Notably, the hybrid method achieved higher accuracy and ITR than that of VEP and PR for most subjects, especially for the short data length. Together with the subjects’ feedback on user experience, these results indicate that the proposed h-BCI with the low-frequency stimulation paradigm is more comfortable and favorable than the traditional SSVEP-BCI paradigm using the alpha frequency range.
fnhum-16-834959 January 28, 2022 Time: 21:15 # 1
ORIGINAL RESEARCH
published: 03 February 2022
doi: 10.3389/fnhum.2022.834959
Edited by:
Max Planck Institute of Neurobiology
(MPIN), Germany
Reviewed by:
Alexander M. Dreyer,
Carl von Ossietzky
University of Oldenburg, Germany
Chang-Hwan Im,
Hanyang University, South Korea
*Correspondence:
Yijun Wang
wangyj@semi.ac.cn
Specialty section:
Brain-Computer Interfaces,
a section of the journal
Frontiers in Human Neuroscience
Received: 14 December 2021
Accepted: 14 January 2022
Published: 03 February 2022
Citation:
Jiang L, Li X, Pei W, Gao X and
Wang Y (2022) A Hybrid
Brain-Computer Interface Based on
Visual Evoked Potential and Pupillary
Response.
Front. Hum. Neurosci. 16:834959.
doi: 10.3389/fnhum.2022.834959
A Hybrid Brain-Computer Interface
Based on Visual Evoked Potential
and Pupillary Response
Lu Jiang1,2 , Xiaoyang Li3, Weihua Pei1,4 , Xiaorong Gao3and Yijun Wang1,4,5*
1State Key Laboratory on Integrated Optoelectronics, Institute of Semiconductors, Chinese Academy of Sciences, Beijing,
China, 2College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing,
China, 3Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China, 4School of Future
Technology, University of Chinese Academy of Sciences, Beijing, China, 5Chinese Institute for Brain Research, Beijing, China
Brain-computer interface (BCI) based on steady-state visual evoked potential (SSVEP)
has been widely studied due to the high information transfer rate (ITR), little user training,
and wide subject applicability. However, there are also disadvantages such as visual
discomfort and “BCI illiteracy.” To address these problems, this study proposes to
use low-frequency stimulations (12 classes, 0.8–2.12 Hz with an interval of 0.12 Hz),
which can simultaneously elicit visual evoked potential (VEP) and pupillary response
(PR) to construct a hybrid BCI (h-BCI) system. Classiﬁcation accuracy was calculated
using supervised and unsupervised methods, respectively, and the hybrid accuracy was
obtained using a decision fusion method to combine the information of VEP and PR.
Online experimental results from 10 subjects showed that the averaged accuracy was
94.90 ±2.34% (data length 1.5 s) for the supervised method and 91.88 ±3.68% (data
length 4 s) for the unsupervised method, which correspond to the ITR of 64.35 ±3.07
bits/min (bpm) and 33.19 ±2.38 bpm, respectively. Notably, the hybrid method
achieved higher accuracy and ITR than that of VEP and PR for most subjects, especially
for the short data length. Together with the subjects’ feedback on user experience, these
results indicate that the proposed h-BCI with the low-frequency stimulation paradigm is
more comfortable and favorable than the traditional SSVEP-BCI paradigm using the
alpha frequency range.
Keywords: hybrid brain-computer interface, electroencephalogram, visual evoked potential, pupillary response,
BCI illiteracy, task-related component analysis, canonical correlation analysis
INTRODUCTION
Brain-computer interface (BCI) allows people to establish an alternative communication channel
between the user’s intention and output devices which is completely independent of the normal
motor output paths of the nervous system (Gandhi, 2007;Volosyak et al., 2017). BCI is especially
relevant for severely disabled users, such as amyotrophic lateral sclerosis, spinal cord injury, and
stroke victims (Hoﬀmann et al., 2008;Kuebier et al., 2009), while applications in entertainment,
safety, and security are also emerging (Zhu et al., 2010). Electroencephalography (EEG) is the most
favorable method in non-invasive BCIs (Gao et al., 2014) due to its essential attributes such as low
cost, high time resolution, and easy access to data (Mason et al., 2007;Volosyak, 2011).
Frontiers in Human Neuroscience | www.frontiersin.org 1February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 2
Jiang et al. h-BCI With VEP and PR
Many laboratories and clinical tests have demonstrated the
convincing robustness of visual evoked potential (VEP)-based
BCI systems (Wang et al., 2008). In the 1970s, Vidal developed
a BCI system that used VEPs to determine the visual ﬁxation
point (Vidal, 1977). Among diﬀerent types of VEPs, steady-
state visual evoked potential (SSVEP) is a continuous electrical
activity recorded at the occipital and parietal cortex areas, which
is elicited at the same frequency (and/or harmonics) when the
retina is excited by visual stimuli at a speciﬁc frequency (Luo
and Sullivan, 2010). The stimulation frequency can be divided
into low (below 12 Hz), medium (12–30 Hz), and high (above
30 Hz) frequency bands (Bouma, 1962;Floriano et al., 2019).
SSVEP has been recognized as a reliable, fast, and easy-to-use
communication paradigm (Allison et al., 2010) due to its high
information transfer rate (ITR), little training cost, and fewer
electrode requirements (Hoﬀmann et al., 2009;Brunner et al.,
2010).
Nevertheless, SSVEP-BCI also has some disadvantages that
need to be improved, such as fatigue, discomfort, safety, and
user limitation. First of all, durative visual stimulations can cause
dizziness and visual fatigue, even impair the user’s vision, and
the amplitude of SSVEP response will also be reduced (Wu
and Su, 2014). Secondly, the alpha frequency range (8–13 Hz)
used in the traditional SSVEP paradigm oﬀers low level of
comfort (Zhu et al., 2010). Our pre-experiment on the eﬀect
of stimulation frequency on user experience has found that
the ﬂickering stimulations in the low-frequency range (<3 Hz)
can provide a better comfort level to the alpha frequency
range. Stimulation frequency higher than the critical fusion
frequency (CFF, e.g., 60 Hz) makes people feel comfortable with
imperceptible ﬂickers (Hartmann et al., 1978;Williams et al.,
2004). Frequency-modulated (FM) and amplitude-modulated
(AM) methods (Chang et al., 2014;Dreyer and Herrmann,
2015;Dreyer et al., 2017) have been adopted to reduce visual
fatigue and enhance the user experience of the SSVEP-BCI in
the alpha frequency band. Thirdly, the ﬂickering stimulus has
some possibility of causing seizures in photosensitive individuals.
Photosensitivity is an abnormal brain electrical response to light
or pattern stimulation, which occurs in 0.3–3% of the population
(Fisher et al., 2005). Harding investigated the proportion
of photoparoxysmal responses in 170 photosensitive patients
(Harding and Harding, 2010). The results indicated that only 3%
of photosensitive patients were at risk at 3 Hz, compared with a
maximum of 90% at 16 Hz. Finally, “BCI illiteracy” is a common
problem in the BCI ﬁeld. A non-negligible portion of the subjects
(estimated from 15 to 30%) were unable to achieve control of the
interface because they did not show the expected brain activity
modulated by the mental task (Vidaurre and Blankertz, 2009). Up
to now, there is still no universal BCI applicable to all users. For
example, Brunner asked 14 healthy subjects to complete a visual
attention task to produce SSVEP and an imagined movement
task to produce event-related desynchronization (ERD) (Brunner
et al., 2010). The number of illiterates was 11 in the ERD
condition and was 3 in the SSVEP condition. In order to improve
the accuracy of users with poor performance, they completed an
oﬄine simulation of a hybrid BCI (h-BCI) in which the subjects
performed the two tasks simultaneously, and the number of
illiterates was reduced to one in the hybrid condition. To improve
user experience of VEP-BCIs, this study proposes to design and
implement a hybrid BCI system based on the low-frequency
(<3 Hz) stimuli with high comfort level and safety.
Hybrid BCI can improve the classiﬁcation accuracy, increase
the number of commands, and shorten the detection time of the
BCI system by combining two or more patterns (at least one
of which is a brain signal) (Hong and Jawad, 2017). Recently,
pupillary responses (PR), such as the pupillary light reﬂex, have
been used as the second pattern in addition to EEG due to the
low user burden, non-invasiveness, and no need for training
(Muto et al., 2020). Pupil diameter changes steadily with the
illuminance of the observed object to regulate the amount of light
entering the eye (Crawford, 1936;Woodhouse, 1975;Woodhouse
and Campbell, 1975), and the modulation frequency of PR is
synchronized with the luminance-modulation frequency of the
visual stimulus. The amplitude of PR decreases as the stimulation
frequency increases (Muto et al., 2020), and the consistent,
measurable PR can be induced at the ﬂickering frequency up
to 2.3 Hz (Naber et al., 2013). Compared with the detection of
gaze position (Ma et al., 2018;Yao et al., 2018), the measurement
of PR does not require system calibration. There were only
few studies on human-computer interaction (HCI) based on
PR. Sebastiaan presented a human-computer interface based on
decoding of attention through pupillometry (Sebastiaan et al.,
2016). Two sets of items with the same ﬂickering frequency
(0.8 Hz) and opposite phase were presented on the display,
and each participant covertly attended to one set, and the pupil
size reﬂected the illuminance of the selected items. The binary
classiﬁcation was realized based on PR, and the mean accuracy
of ten subjects was 88.9%, resulting in an ITR of 2.58 bits/min
(bpm) with a mean selection time of 14.9 s. Ponzio et al. (2019)
proved that the illuminance was the only factor that signiﬁcantly
aﬀects pupil constriction and found no diﬀerences between the
monocular and the binocular vision. They also established a
binary communication based on PR and achieved an accuracy of
100% at 10 bpm and 96% at 15 bpm. Muto et al. (2020) realized
an information input interface with 12 options (from 0.58 to
1.90 Hz, with an interval of 0.12 Hz) based on PR. The averaged
power spectral density (PSD) peak decreased with increasing
luminance-modulation frequency, and the averaged classiﬁcation
accuracy reached 85.4% with a data length of 7 s. PR and SSVEP
have also been combined to implement an h-BCI. De’Sperati
et al. (2020) reported a frequency tagging approach based on the
evoked oscillatory responses of the pupil and the visual cortex.
Each of the two ﬂickers contained a sum of two sinusoidally
modulated luminance (0.9 and 15 Hz for one stimulus, 1.4 and
20 Hz for another stimulus). By applying a binary linear classiﬁer
to PR and SSVEP signals for 18 subjects, hybrid classiﬁcation
accuracy was 83% with the data length of 7.5 s, which was
higher than that of PR (75%) and SSVEP (80%). These PR-based
HCI and h-BCI studies showed limitations, such as the small
number of targets and the longer detection time compared with
the existing SSVEP-BCIs (Nakanishi et al., 2018;Mao et al., 2020;
Ming et al., 2021).
This study intends to use low-frequency visual stimulations
that can simultaneously elicit VEP and PR (Jiang et al., 2020) to
Frontiers in Human Neuroscience | www.frontiersin.org 2February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 3
Jiang et al. h-BCI With VEP and PR
implement a 12-target h-BCI speller. Compared with the existing
HCI and BCI work related to PR, this system aims to achieve
a shorter detection time and higher classiﬁcation accuracy by
adopting eﬃcient coding and decoding methods. Compared to
other VEP-based BCIs, the proposed system has the advantage of
better comfort and is applicable to more subjects.
MATERIALS AND METHODS
Experimental Environment
Subjects
Ten healthy subjects (2 males and 8 females, ages 23–29 years,
with a mean age of 25.7) participated in the oﬄine and online BCI
experiments. Twelve subjects (6 males and 6 females, ages 23–
28 years, with a mean age of 25) participated in the behavioral test
of user experience without EEG or PR recording, and 5 of them
participated in the BCI experiments. All subjects had normal or
corrected-to-normal vision by wearing contact lenses. Before the
experiment, each subject was asked to read and sign an informed
consent form approved by the Research Ethics Committee of
Tsinghua University.
Data Recording
In the experiment, EEG data were recorded by a SynAmps2
ampliﬁer (Neuroscan Inc.) at a sampling rate of 1,000 Hz.
According to the international 10–20 system, nine electrodes
(Pz, PO5, PO3, POz, PO4, PO6, O1, Oz, and O2) were placed
at the occipital and parietal regions to record VEPs, and two
electrodes located on the forehead (AFz) and vertex (between
Cz and CPz) regions as ground and reference, respectively. The
contact impedance was kept below 10 k. The binocular PR data
were recorded by an infrared eye tracker (EyeLink 1000 Plus, SR
Research Inc.) with a sampling rate of 1,000 Hz, and the PR data
were preprocessed by an interpolation method (Kret and Sjak-
Shie, 2019). The focal length of the lens was adjusted manually to
observe a clear pupil image and stable corneal reﬂection point.
Visual Stimulus Design
This study designed a 12-target online h-BCI system to realize a
virtual keypad. A 24.5-inch LCD display (Dell AW2518H) with
a resolution of 1280 ×720 pixels and a refresh rate of 240 Hz
was used to present a 3 ×4 stimulus matrix (Figure 1A). The
size of each stimulus was 75 ×75 pixels (3×3), and the
horizontal and vertical distances between the centers of any two
adjacent stimuli were 225 pixels (9). Each target adopted the
style of a grid stimulus (Ming et al., 2021), which consisted of
8×8 small ﬂickering squares. The size of each square was
5×5 pixels. The 12 stimulus targets correspond to 12 characters
(1, 2, 3, 4, 5, 6, 7, 8, , 9, 0, and #). As shown in Figure 1B,
a red square with a size of 10 ×10 pixels in the center of
each stimulus was used to highlight the characters and serve
as the subject’s ﬁxation point. The stimulation was drawn and
stably presented by the Psychophysics Toolbox Ver. 3 (Brainard,
1997) in MATLAB (MathWorks, Inc), as evidenced by the high
consistency of visual stimulation waveforms in multiple trials
recorded by a photodiode.
All visual stimuli were coded according to the joint frequency-
phase modulation method (Chen et al., 2015b), as shown in
Figure 1C. Twelve frequencies (from 0.8 to 2.12 Hz, with
0.12 Hz interval) were used, and the phase diﬀerence between
two adjacent frequencies was set to 0.69πin order to minimize
the correlation of the stimulus signal. Each visual ﬂicker was
presented on the display in accordance with the square wave
signal encoded with the corresponding frequency fand phase ϕ:
sf, ϕ, i=sign sin 2πf(i/240)+ϕ+1
2
Where sin(·)generates a sine waveform, sign(·)is a sign
function, and fand ϕare the frequency and phase of the
target, respectively. irepresents the index of the frames.
s(·)represents the stimulus sequence, with only two
values, 0 and 1, corresponding to the lowest and highest
illuminance, respectively.
Experimental Platform
This study developed an online experiment platform for the
h-BCI based on EEG and PR, including user interface and
data analysis computer (C1), EEG computer (C2), and PR
computer (C3), as shown in Figure 2A. The visual stimulation
was presented to the subjects by C1. At the beginning of each
trial, the stimulus trigger was sent from the C1’s parallel port to
the EEG ampliﬁer and the eye tracker, which was synchronized
with EEG and PR signals. At the same time, the subject’s EEG and
PR signals were sent from C2 and C3 to C1 in real time. Data
were analyzed by C1, and auditory feedback (i.e., pronunciation
of the target character in Chinese) was provided to the subject.
The experimental scene diagram including electrode locations
and eye tracker settings was shown in Figure 2B. In a room with
normal lighting, the subjects sat comfortably on a chair 60 cm
away from the stimulation monitor to complete all experiments.
Experiment Design
In this study, a 12-target h-BCI system based on cue-guided target
selecting task was designed, which included oﬄine and online
experiments. Data collected in the oﬄine experiment were used
to optimize parameters in data analysis toward high classiﬁcation
accuracy. The online experiment further demonstrated and
evaluated the system performance using a close-loop paradigm
with online feedback. Figure 3A shows the timing procedure of
the oﬄine experiment. Each subject completed three blocks of
target selections. Each block contained four runs and there was a
rest time about 1 min between any two runs. Each run contained
12 trials, corresponding to 12 targets cued in a random order.
Therefore, there were total 12 trials (four trials per block, three
blocks) for each target in the oﬄine experiment. The subjects
were given several minutes (ranging from 5 to 10) to rest between
the blocks to avoid fatigue. Each trial started with a visual cue
(a red triangle), which appeared below the target stimulus for
6 s, as shown in Figure 3A. Subjects were asked to ﬁxate at
the red square in the middle of the target within 1 s and avoid
eye blinking for the next 5 s when all targets started ﬂickering
together. After that, subjects can blink to reduce visual fatigue
during the 4 s rest time at the end of the trial.
Frontiers in Human Neuroscience | www.frontiersin.org 3February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 4
Jiang et al. h-BCI With VEP and PR
FIGURE 1 | (A) Stimulation interface of the 12-target BCI. (B) Stimulation pattern (e.g., target 5) with ﬁxation point (a red square), character and visual cue (a red
triangle). (C) Frequency and phase values for all targets.
FIGURE 2 | (A) Online experiment platform for the h-BCI based on EEG and PR. (B) Experimental scene of the h-BCI.
FIGURE 3 | Experimental procedures of the (A) ofﬂine and (B) online experiments.
The online experiment included three parts: test with an
unsupervised method, train and test with a supervised method,
as shown in Figure 3B. These methods were mentioned in
section “Classiﬁcation Algorithm.” Each part contained two
blocks. Diﬀerent from the oﬄine experiment, the total duration
of each trial was 5.5 s (1 s sight shift, 4 s stimulus, 0.5 s rest, and
0.6 s waiting for feedback) for the unsupervised method and 3 s
(1 s sight shift, 1.5 s stimulus, 0.5 s rest, and 0.6 s waiting for
feedback) for the supervised method. The stimulation duration
was manually selected toward optimal performance by jointly
considering accuracy and ITR in oﬄine data analysis. In addition,
the visual cue of the next trial was presented after the rest time
of the current trial in order to leave enough time for subjects to
shift their eyesight.
In addition, a behavioral test was designed to compare the
subjective perception of the 12-target stimulation interface at
low (0.8–2.12 Hz) and medium (9–10.12 Hz) frequency bands.
Twelve subjects were asked to ﬁll out a questionnaire after
completing 1 block of cue-guided target selecting task for each
frequency band. The questionnaire (Bieger and Molina, 2010)
Frontiers in Human Neuroscience | www.frontiersin.org 4February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 5
Jiang et al. h-BCI With VEP and PR
included three parts: the comfort level (scores 1–5 correspond
to very uncomfortable, uncomfortable but tolerable, somewhat
uncomfortable, comfortable, very comfortable, respectively), the
perception of ﬂicker (scores 1–5 correspond to very strong ﬂicker,
strong ﬂicker, slight ﬂicker, perceptible ﬂicker, imperceptible
ﬂicker, respectively) and the preference level (score 1 indicates
the worst and 5 indicates the best).
Data Analysis
Latency Delay
The continuous EEG and PR data were divided into multiple
trials according to the event channel, which recorded the stimulus
onsets. The selection of time window used to extract data epochs
requires an estimation of latency, which represents the time delay
between stimulus onset and VEP or PR responses. The latency of
VEP was set to 140 ms (Chen et al., 2015a). The latency of PR
was estimated with experimental data, including the latency of
the pupil constriction lDelay1 and the duration of the response to
the stimulus onset lDelay2 (Muto et al., 2020). First, lDelay1 was
observed according to the clean temporal waveforms obtained by
ﬁltering and averaging the PR signals in the oﬄine experiment
(as shown in Figure 4B). PR began to decrease at 250 ms after
the stimulus onset, so lDelay1 was set to 250 ms. Second, lDelay2
was obtained by calculating the maximum classiﬁcation accuracy
according to the standard canonical correlation analysis (CCA)
method (Lin et al., 2006), which was often used to extract the
narrow-band frequency component of the signals. PR data in
the oﬄine experiment were extracted in [lDelay2, 5.25 s] (time 0
indicated stimulus onset) to exclude the common response after
the stimulus onset and pre-processed (down-sampled to 250 Hz,
band-pass ﬁltered within 0.7550 Hz to reduce DC drift and
power line interference). lDelay2 was set to diﬀerent values (from
0 to 1.5 s, with an interval of 0.02 s) to calculate the corresponding
CCA classiﬁcation accuracy. lDelay2 was set to 1.2 s in this study,
which corresponded to the highest accuracy, and the PR data
extracted in [1.2 s, 5.25 s] were modulated by the local target’s
brightness and showed steady-state periodic responses at the
stimulus frequency.
Characteristics of Visual Evoked Potential and
Pupillary Response
The temporal waveform, amplitude spectrum, and signal-to-
noise ratio (SNR) were used to evaluate whether there was a
frequency-locked relationship between stimulation signal and
VEP or PR (Nakanishi et al., 2014). First of all, the VEP and PR
signals were epoched from 0.1 s before the stimulus onset to 0.5 s
after the stimulus oﬀset, down-sampled to 250 Hz, and passed
through a Chebyshev Type I band-pass ﬁlter (0.5–10 Hz). In
particular, the baselines of all PR trials were corrected with respect
to the mean value over the 100 ms window preceding stimulus
onset. A total of 120 trials (12 trials ×10 subjects) extracted
in [0 s, 5 s] (time 0 indicated stimulus onset) were averaged
for each target in the oﬄine experiment to draw clean temporal
waveforms of the elicited VEP at the Oz electrode and PR of the
right eye. Secondly, the data epochs in [0 s, 0.5 s] (time 0 indicated
the stimulus signal changes from 0 to 1) were extracted to explore
how the EEG and PR were modulated by illuminance. To remove
the transient response of VEP and PR, the data in the ﬁrst second
after stimulus onset were excluded. Besides, the down-sampled
signals ﬁltered by a band-pass ﬁlter (0.75–50 Hz) were extracted
(PR: [0.25 s, 5.25 s], VEP: [0.14 s, 5.14 s], time 0 indicated stimulus
onset) for fast Fourier transform (FFT). Limited by the frequency
resolution of 0.2 Hz, three frequencies (0.8, 1.4, and 2 Hz) were
selected for amplitude spectrum analysis of VEP and PR. Finally,
SNRs of VEP and PR were calculated as the ratio of the signal
spectrum amplitude at the target to the mean of background
noises:
SNR f=
20 ×log10 n×Yf
Pn/2
m=1Yf0.2×m+Yf+0.2×m!
Where fdenotes the stimulus frequency, Y(·)denotes the
amplitude spectrum, nwas set to 4 (i.e., 0.4 Hz on each side of
the target frequency).
Classiﬁcation Algorithm
Classiﬁcation accuracy and ITR were often used to evaluate the
performance of the BCI system. Firstly, the classiﬁcation accuracy
in the oﬄine experiment was calculated using both supervised
and unsupervised methods. Speciﬁcally, the supervised method
required calibration data for training, while the unsupervised
method performed classiﬁcation without calibration data. For
the supervised method, classiﬁcation accuracy was calculated by
a leave-one-out cross-validation method, which meant that 11
of the 12 runs in the oﬄine experiment were used for training,
and the remaining 1 run was used for testing in each step of
validation. For the EEG, task related component analysis (TRCA)
(Nakanishi et al., 2018), which can extract eﬀective response
information from EEG by maximizing the reproducibility of
task-related components, has been proved highly eﬃcient in the
detection of SSVEP. EEG data extracted in [0.14 s 0.14 +ds] (time
0 indicated stimulus onset, ddenoted the data length) were used
for classiﬁcation. xk, which was denoted as the EEG signals after
down-sampling and band-pass ﬁltering corresponding to the kth
target, could be averaged among trials to obtain the template
Xk. The spatial ﬁlters W=[ω1,ω2,· · · ,ω12 ]were calculated
according to the TRCA method. Then, the unlabeled EEG signal
Yand the templates Xkpassed through the spatial ﬁlters Wto
calculate the two-dimensional correlation coeﬃcient rk,1, as:
rk,1=ρXT
kW,YTW
For the PR, a template matching method was used, and the
latency delay was 250 ms. Similar to EEG, the averaged two-
channel PR signals Pkwere used as the kth target’s template. Then,
the correlation coeﬃcient rk,2between the unlabeled PR data Q
and the template Pkwas calculated as:
rk,2=ρ(Pk,Q)
For the unsupervised method, CCA was used for both VEP
and PR, and the latency delay was diﬀerent (VEP: 140 ms, PR:
lDelay1 = 250 ms, lDelay2 = 1,200 ms). PR data were extracted
Frontiers in Human Neuroscience | www.frontiersin.org 5February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 6
Jiang et al. h-BCI With VEP and PR
FIGURE 4 | (A) Elicited PRs and VEPs corresponding to twelve frequencies. The gray dotted line represents the stimulus signals. (B) PR after stimulus onset,
(C) PR, and (D) VEP when stimulus changes from 0 to 1 averaged over twelve frequencies. Each dashed line corresponds to one subject, and the thick black line is
the average across all subjects. The gray areas in (D) represent the point-wise analysis of variance (ANOVA) results with values p<0.05, to highlight the time periods
with signiﬁcant differences between the VEP and zeros.
in [lDelay2, lDelay1+ds] (time 0 indicated stimulus onset, d
denoted the data length).
Secondly, the ﬁlter-bank method was used to improve
classiﬁcation accuracy by making better use of the information on
the fundamental and harmonic frequencies (Chen et al., 2015a).
The EEG and PR data in the oﬄine experiment were used to
optimize the parameters of both supervised and unsupervised
methods. For the supervised method, the optimization of ﬁlter
banks was to generate Nsub-bands covering multiple harmonic
frequency bands f1,n(n=1,2,· · · ,N)with a same high cutoﬀ
frequency f2. The low cutoﬀ frequency f1,nof each sub-band was
increased by a certain step size 1f, as f1,n=f1,1+1f×(n
1)where f1,1was set to 0.75 Hz and f1,nwas up to 9.75 Hz.
The weight of nth sub-band was deﬁned as w(n)=na+b.
Therefore, the frequency bands and weights of the sub-bands
were determined by a grid search method simultaneously, where
1f,f2,a, and bwere limited to [0.5:0.5:9], [30:2.5:50], [0:0.25:2]
and [0:0.25:1], and Nwas determined by 1f. The weighted sum
of correlation coeﬃcient values corresponding to all sub-bands
(i.e., r1
k,r2
k,···rn
k) was calculated as:
erk=
N
X
n=1
w(n)rn
k2
The correlation coeﬃcient erkwas calculated with the templates
of each target, and the one with the largest correlation coeﬃcient
was determined as the target label of the test data. Besides, for
the unsupervised method, the number of harmonics Nhof the
sinusoidal signal, which was used as the reference signal in the
standard CCA preprocesses (Lin et al., 2006) required to be
optimized and further used in the ﬁlter-bank CCA (FBCCA)
method (Chen et al., 2015a).
Thirdly, the oﬄine experiment data were also used to calculate
the weights of the hybrid data fusion method toward the
Frontiers in Human Neuroscience | www.frontiersin.org 6February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 7
Jiang et al. h-BCI With VEP and PR
highest accuracy across subjects. A decision fusion method was
developed to combine the information of VEP and PR, which
can further improve the accuracy of target recognition (Ma et al.,
2018). The hybrid method was calculated as:
Rk
hybrid =normalize frk,1×ACC2
EEG +normalize frk,2
×ACC2
PR
Where frk,1and frk,2were the correlation coeﬃcients
corresponding to EEG and PR, respectively. normalize (·)
denoted the values were normalized to [0,1] interval across the
12 targets. ACCPR and ACCEEG were the averaged classiﬁcation
accuracy across all subjects. The kth character corresponding to
the maximal Rk
hybrid was chosen as the target character for the
decision fusion method.
Finally, the optimized parameters of the ﬁlter-bank method
and the weights of the fusion method based on the oﬄine
experiment were transferred to the online experiment. For the
supervised method, the templates of EEG and PR and the
TRCA spatial ﬁlters were re-trained in the online experiment.
Besides, for both oﬄine and online experiments, ITR (bpm) was
calculated as follows:
ITR =log2M+Plog2P+(1P)log21P
M1×60
T
Where, Mis the total number of targets (12 in this study), Pis the
averaged classiﬁcation accuracy across all the targets, and Tis the
averaged time to complete the detection of the target, including
the stimulus duration and the interval time of 1.5 s.
Statistical Analysis
Statistical analyses were performed with SPSS software (IBM
SPSS Statistics, IBM Corporation) in this study. Repeated-
measures analysis of variance (RMANOVA) was used to check
the diﬀerence of performance (e.g., SNR, accuracy, ITR, and
so on) between diﬀerent conditions (Chen et al., 2015a, 2019;
Nakanishi et al., 2018;Xu et al., 2020). Greenhouse–Geisser
correction was used if the data violated the sphericity assumption
by Mauchly’s test of sphericity. Bonferroni correction was
applied to post hoc pairwise comparisons. The signiﬁcance level
was set to 0.05.
RESULTS
Stimulus and Characteristics of Pupillary
Response and Visual Evoked Potential
The waveforms of the right PRs and VEPs at the Oz electrode
averaged across all subjects are illustrated in Figure 4A. PR
decreased at 250 ms after stimulus onset (magniﬁed in
Figure 4B) for all frequencies, which was caused by the change
of the overall illuminance in the visual ﬁeld. Then, PR increased
slightly after falling to the valley value. The clear time-locked
characteristic appeared after 1.2 s with a latency delay of
300 ms (magniﬁed in Figure 4C), which was modulated by the
local target’s illuminance. For the VEP signals, the frequency and
phase information of the target stimulus were accurately coded,
and a pattern onset VEP (Odom et al., 2004) with negative peaks
(at 90 ms and 180 ms) and a positive peak (at 250 ms) could
be observed (in Figure 4D).
Figure 5 shows the amplitude spectra at three stimulus
frequencies (0.8, 1.4, and 2 Hz) averaged across 10 subjects,
marked with red asterisks at the fundamental and harmonic
frequencies. As shown in Figure 5A, the amplitude spectrum
of PR showed a clear response at the fundamental frequency
and the amplitude decreased with increased frequencies (0.8 Hz:
0.28 a.u., 1.4 Hz: 0.14 a.u., and 2 Hz: 0.09 a.u.). The responses
at the harmonic frequencies were less signiﬁcant for the higher
stimulus frequencies. Diﬀerent from the PR, the VEP response at
the Oz electrode showed signiﬁcant peaks at the fundamental and
harmonic frequencies (see Figure 5B). As the stimulus frequency
increased, the response amplitude increased at the second
harmonic (0.8 Hz: 0.92 µV, 1.4 Hz: 1.29 µV, 2 Hz: 1.77 µV).
Comparison of the Signal-to-Noise Ratio
of Pupillary Response and Visual Evoked
Potential
Figure 6 shows the averaged SNR of PR and VEP across
three stimulus frequencies (0.8 Hz, 1.4 Hz, 2 Hz) and 10
subjects. A two-way RMANOVA shows main eﬀects of harmonic
[F(9,81) = 10.88, p<0.05] and modality [F(1,9) = 38.35,
p<0.05], and signiﬁcant interaction of these two factors
[F(9,81) = 9.77, p<0.05]. Pairwise comparisons indicate that
PR had a signiﬁcantly higher SNR at the fundamental frequency
than harmonic frequencies (p<0.05, fundamental frequency:
10.31 ±0.79 dB, second harmonic: 5.01 ±0.67 dB, third
harmonic: 5.79 ±0.68 dB). Diﬀerently, the SNR of VEP had
no signiﬁcant diﬀerences in the fundamental and harmonics
frequencies (p>0.05) and reached the highest at the second
harmonic (9.37 ±1.34 dB). In contrast, the SNR of PR was higher
than VEP at the fundamental frequency and lower at harmonics.
Pairwise comparisons revealed that the diﬀerence between PR
and VEP was signiﬁcant at each harmonic (p<0.05), except for
the third and ﬁfth harmonics (p>0.05).
Ofﬂine Analysis
Supervised Method
The nine-channel EEG and binocular PR signals with the data
length of 1.5 s after the visual latency were used to optimize
the supervised algorithm. Firstly, Chebyshev Type I band-pass
ﬁlter with a low cutoﬀ frequency f1(from 0.75 to 9.75 Hz, with
an interval of 0.5 Hz) and a high cutoﬀ frequency f2(from 30
to 50 Hz, with an interval of 2.5 Hz) were adopted to calculate
the classiﬁcation accuracy using one band-pass ﬁlter, as shown in
Figure 7. The highest classiﬁcation accuracy was 74.03 ±4.10%
for PR at 1.2550 Hz and 90.76 ±3.64% for VEP at 1.7530 Hz
(the gray dots in Figure 7). The optimized results of the ﬁlter-
bank method indicated that the maximal classiﬁcation accuracy
of PR and VEP were 84.10 ±3.47% and 93.68 ±2.86%, both of
which were higher than the accuracy of using one band-pass ﬁlter
(the asterisk in Figure 8). One-way RMANOVA indicates that
the diﬀerence between the accuracy of one band-pass ﬁlter and
Frontiers in Human Neuroscience | www.frontiersin.org 7February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 8
Jiang et al. h-BCI With VEP and PR
A
B
FIGURE 5 | Averaged amplitude spectrum of (A) PR and (B) VEP for all subjects. The red asterisk represents the fundamental and harmonic frequencies of the
stimulus frequency.
FIGURE 6 | Averaged SNR of PR and VEP at Oz electrode for three stimulus
frequencies and 10 subjects. The error bars represent standard errors across
the subjects.
the ﬁlter-bank method was signiﬁcant [PR: F(1,9) = 43.75, VEP:
F(1,9) = 8.03, p<0.05]. The corresponding optimal parameters
were 1f= 0.5, f2= 50, N= 6, a= 0.5, b= 0.5 for PR and 1f= 0.5,
f2= 50, N= 12, a= 0.5, b= 0 for VEP.
Figure 8 compared the classiﬁcation performance for the
ﬁlter-bank supervised method at diﬀerent data lengths (from 0.5 s
to 5 s with a step of 0.5 s). The classiﬁcation accuracy was the
lowest at 0.5 s (PR: 28.96 ±2.95%, VEP: 77.57 ±5.09%, hybrid:
79.03 ±4.85%) and increased as the data length increased.
In particular, the hybrid accuracy was better than that of PR
or VEP at short data length. Two-way RMANOVA shows
main eﬀects of data length [F(9,81) = 90.68, p<0.05) and
modality (F(2,18) = 27.51, p<0.05], and there were signiﬁcant
interactions of data lengths and modalities [F(18,162) = 44.70,
p<0.05]. Pairwise comparisons indicate the accuracy of the
hybrid method was signiﬁcantly higher than that of PR with
data length 2.5 s and was signiﬁcantly higher than that
of VEP with data length 0.5 s (p<0.05). In addition,
as the data length increased, ITR ﬁrst increased and then
decreased. At the data length of 1.5 s, the hybrid accuracy was
96.81 ±1.49%, and the hybrid ITR was 66.77 ±2.17 bpm
(PR: 50.72 ±3.79 bpm, VEP: 62.91 ±3.69 bpm). Two-way
RMANOVA reveals main eﬀects of data length [F(9,81) = 27.68,
p<0.05] and modality [F(2,18) = 29.74, p<0.05], and signiﬁcant
interactions [F(18,162) = 39.09, p<0.05]. Pairwise comparisons
indicate that the diﬀerence between the hybrid method and PR
was signiﬁcant at data length 2.5 s (p<0.05).
Frontiers in Human Neuroscience | www.frontiersin.org 8February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 9
Jiang et al. h-BCI With VEP and PR
A B
FIGURE 7 | Classiﬁcation accuracy of (A) PR and (B) VEP data through one band-pass ﬁlter using the supervised method at the data length of 1.5 s. The gray dots
correspond to the ﬁlter settings with the highest accuracy.
FIGURE 8 | (A) Classiﬁcation accuracy and (B) ITR using ﬁlter-bank supervised method at different data lengths (from 0.5 s to 5 s with a step of 0.5 s). The error
bars represent standard errors across the subjects. The dot corresponds to the optimal classiﬁcation accuracy using one band-pass ﬁlter. The asterisk represents a
signiﬁcant difference between the single-modal method and the hybrid method (pairwise comparison, p<0.05).
Figure 9 shows the classiﬁcation accuracy using diﬀerent
parameters: the number of sub-bands (Nfrom 1 to 20) and the
number of training trials (Ntfrom 1 to 11) at the data length
of 1.5 s. Overall, the accuracy of PR was always lower than that
of VEP. As the Nvalue increased, the accuracy of PR increased
ﬁrst and reached the maximum at N= 6, then decreased and
reached the lowest at N= 20 (73.75 ±5.62%). It suggests that only
the ﬁrst several harmonics of PR contribute to the classiﬁcation.
For VEP, the accuracy was the lowest at N= 1 (80.83 ±5.12%),
then increased as the number of sub-bands increased and tended
to be saturated. In addition, one-way RMANOVA reveals that
the accuracy of PR or VEP has a signiﬁcant diﬀerence between
diﬀerent numbers of training data [PR: F(9,81) = 27.08, VEP:
F(9,81) = 14.40, p<0.05]. Pairwise comparisons show that the
accuracy will not increase signiﬁcantly when Ntis greater than 8
(p>0.05, PR: 82.08 ±3.97%, VEP: 91.18 ±3.61% at Nt= 8).
Therefore, this study set the number of training trials Nt= 8 to
reduce the training cost in the online experiment.
Unsupervised Method
For the unsupervised algorithm, the optimization was based on
the VEP and PR data with a length of 4 s starting from the visual
latency. First, the classiﬁcation accuracies of the standard CCA
method corresponding to diﬀerent band-pass ﬁlters at Nh= 12
were shown in Figures 10A,B. The highest classiﬁcation accuracy
was 63.61 ±3.84% for PR at 1.2550 Hz and 62.71 ±9.42%
for VEP at 3.2535 Hz (the gray dots in Figures 10A,B).
Then, the classiﬁcation accuracies using diﬀerent Nhvalues (from
1 to 12) in this optimal band-pass ﬁlter were compared, as
shown in Figures 10C,D. As the number of Nhincreased, the
classiﬁcation accuracy of PR increased ﬁrst and then decreased,
and the highest was 65.76 ±4.13% when Nh= 3. The accuracy
of VEP increased when Nhincreased and reached the highest
when Nh= 10 (63.33 ±8.64%). Therefore, this study used
Nh= 3 for PR and Nh= 10 for VEP in all standard CCA
preprocesses for the FBCCA method. The optimized ﬁlter-bank
results indicated that the maximal classiﬁcation accuracy of PR
and VEP were 86.74 ±4.28% and 65.83 ±9.07%, respectively.
The corresponding optimal parameters were 1f= 0.5, f2= 50,
N= 19, a= 0.25, b= 1 for PR and 1f= 0.5, f2= 50, N= 10, a= 0,
b= 0 for VEP. Compared with the standard CCA method (the
asterisk in Figure 11), the ﬁlter bank related improvement of PR
was greater than that of VEP. One-way RMANOVA indicates that
the improvement of PR was signiﬁcant [F(1,9) = 68.89, p<0.05],
while that of VEP was not signiﬁcant [F(1,9) = 4.38, p>0.05].
Figure 11 shows the classiﬁcation performance with the
optimized parameters for the FBCCA method at diﬀerent data
lengths (from 2 s to 5 s with a step of 0.5 s). The accuracy
Frontiers in Human Neuroscience | www.frontiersin.org 9February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 10
Jiang et al. h-BCI With VEP and PR
FIGURE 9 | Classiﬁcation accuracy with different (A) numbers of sub-bands and (B) numbers of training trials using the supervised method at the data length of
1.5 s. The error bars represent standard errors across the subjects.
FIGURE 10 | Classiﬁcation accuracy of (A) PR and (B) VEP data through one band-pass ﬁlter using the standard CCA method with a data length of 4 s. The gray
dots correspond to the ﬁlter settings with the highest accuracy. Classiﬁcation accuracy of (C) PR and (D) VEP with different numbers of harmonics in the standard
CCA method. The error bars represent standard errors across the subjects.
Frontiers in Human Neuroscience | www.frontiersin.org 10 February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 11
Jiang et al. h-BCI With VEP and PR
of PR was lower than VEP at the data length of 2 s (hybrid:
42.36 ±7.54%, PR: 15.56 ±1.77%, VEP: 41.53 ±7.54%). The
accuracy increased with the increase of data length, and the
accuracy of PR was higher than that of VEP with data lengths
3 s. The classiﬁcation accuracy of the hybrid method was
higher than that of PR and VEP and reached 91.04 ±4.76%
at 4 s. Two-way RMANOVA shows main eﬀects of data length
[F(6,54) = 164.77, p<0.05] and modality [F(2,18) = 5.72,
p<0.05], and signiﬁcant interactions of the two factors
[F(12,108) = 19.39, p<0.05]. Moreover, ITR increased as
the data length increased and reached 32.85 ±2.74 bpm at
4 s for the hybrid method (PR: 29.52 ±2.38 bpm, VEP:
19.49 ±4.25 bpm). Two-way RMANOVA shows main eﬀects
of data length [F(6,54) = 49.07, p<0.05] and modality
[F(2,18) = 5.92, p<0.05], and there were signiﬁcant interactions
[F(12,108) = 16.35, p<0.05]. Pairwise comparisons indicate that
the accuracy and ITR of the hybrid method were signiﬁcantly
higher than that of PR with data length 4.5 s and were
signiﬁcantly higher than that of VEP with data length 2.5 s
(p<0.05).
Figure 12 shows the classiﬁcation accuracy of the FBCCA
method with diﬀerent numbers of sub-bands (Nfrom 1 to 15)
at the data length of 4 s. For PR, the classiﬁcation accuracy was
the lowest at N= 1 (47.71 ±3.33%), then increased slightly as the
number of sub-bands increased. The accuracy of VEP increased
ﬁrst and then decreased after reaching the maximum at N= 10.
Online Hybrid BCI Performance
Table 1 lists the results of the online cue-guided target
selecting task using the supervised method. The averaged
hybrid accuracy across 10 subjects was 94.90 ±2.34% (PR:
79.90 ±4.11%, VEP: 89.17 ±4.20%), and the corresponding
ITR was 64.35 ±3.07 bpm (PR: 45.92 ±4.23 bpm, VEP:
57.55 ±5.04 bpm). Table 2 lists the results of the online
experiment using the unsupervised method. The averaged
hybrid accuracy across 10 subjects was 91.88 ±3.68% (PR:
84.06 ±4.52%, VEP: 61.56 ±7.93%), and the corresponding
ITR was 33.19 ±2.38 bpm (PR: 27.91 ±2.66 bpm, VEP:
16.90 ±4.00 bpm). One-way RMANOVA indicates that the
diﬀerences between the oﬄine and online experiment were not
signiﬁcant (for the supervised method, the accuracy of PR:
F(1,9) = 3.44, the accuracy of VEP: F(1,9) = 3.17, the accuracy
of hybrid: F(1,9) = 3.31, the ITR of PR: F(1,9) = 4.40, the ITR
of VEP: F(1,9) = 3.49, the ITR of hybrid: F(1,9) = 3.09; for
the unsupervised method, the accuracy of PR: F(1,9) = 1.21,
the accuracy of VEP: F(1,9) = 1.18, the accuracy of hybrid:
F(1,9) = 0.22, the ITR of PR: F(1,9) = 0.97, the ITR of VEP:
A B
FIGURE 11 | (A) Classiﬁcation accuracy and (B) ITR using the FBCCA method at different data lengths (from 2 s to 5 s with a step of 0.5 s). The error bars represent
standard errors across the subjects. The dot corresponds to the optimal classiﬁcation accuracy using one band-pass ﬁlter. The asterisk represents a signiﬁcant
difference between the single-modal method and the hybrid method (pairwise comparison, p<0.05).
FIGURE 12 | Classiﬁcation accuracy with different numbers of sub-bands using the unsupervised method at the data length of 4 s. The error bars represent
standard errors across the subjects.
Frontiers in Human Neuroscience | www.frontiersin.org 11 February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 12
Jiang et al. h-BCI With VEP and PR
F(1,9) = 1.70, the ITR of hybrid: F(1,9) = 0.11; p>0.05 for
all conditions).
Individual Differences
There were clear individual diﬀerences in the system
performance. Figure 13 shows the classiﬁcation accuracy
for each subject in the oﬄine experiment. For the supervised
method, some subjects had signiﬁcantly higher accuracy of VEP
than PR with short data lengths. The hybrid accuracy was higher
than that of VEP for Sub4, Sub5, Sub7, and Sub10. For the
unsupervised method, signiﬁcant diﬀerences appeared among
the subjects. For some subjects, EEG accuracy was always higher
than that of PR (Sub1, Sub8, and Sub9), while PR accuracy was
always higher than that of EEG in other subjects (Sub4, Sub5,
Sub7, and Sub10). Besides, several subjects had higher accuracy
of PR than EEG only when the data length was long (Sub2, Sub3,
and Sub6). Notably, the hybrid accuracy was higher than PR
and VEP for most subjects, especially when the data length was
short. Similar conclusions can be obtained from the results of the
online experiment, as shown in Tables 1,2. For the supervised
method, Sub4 and Sub7 had a higher accuracy of PR than
EEG, while the other 8 subjects showed the opposite condition.
However, the hybrid classiﬁcation performance was higher than
PR and EEG for all subjects, except for Sub6. In particular, the
accuracy of VEP was 66.67%, and the hybrid accuracy reached
91.67% after combining PR characteristics, with an increase of
25.00% for Sub7. For the unsupervised method, two subjects
(Sub1 and Sub8) had a lower accuracy of PR than VEP. It can
also be observed that the hybrid classiﬁcation performance
has been improved for all subjects compared to VEP, except
for Sub1. Especially for Sub7, the hybrid accuracy reached
95.83%, which was 54.16% higher than that of VEP. These results
indicate that the proposed hybrid BCI yielded a considerable
beneﬁt for some subjects and thereby alleviated the problem
of BCI illiteracy.
Behavioral Test
One-way RMANOVA shows that the behavioral scores of the
low-frequency band were signiﬁcantly higher than that of
the medium-frequency band (low vs. medium, comfort level:
3.42 ±0.31 vs. 1.75 ±0.22, F(1,11) = 34.38, p<0.05;
perception of ﬂicker: 2.50 ±0.23 vs. 1.75 ±0.18, F(1,11) = 9.00,
p<0.05; preference level: 3.17 ±0.34 vs. 1.92 ±0.23,
F(1,11) = 14.47, p<0.05), which meant that the low-frequency
TABLE 1 | Results of online cued-guided experiment (supervised method).
Subject Accuracy (%) ITR (bpm)
PR VEP Hybrid PR VEP Hybrid
Sub1 76.04 100.00 100.00 41.21 71.70 71.70
Sub2 94.79 97.92 100.00 62.92 68.19 71.70
Sub3 90.63 96.88 98.96 57.42 66.88 69.94
Sub4 89.58 77.08 91.67 55.92 41.00 58.55
Sub5 55.21 67.71 76.04 21.80 32.63 40.78
Sub6 66.67 95.83 94.79 31.48 64.68 63.37
Sub7 88.54 66.67 91.67 54.17 31.52 58.99
Sub8 87.50 98.96 100.00 53.04 69.94 71.70
Sub9 67.71 96.88 98.96 33.57 66.88 69.94
Sub10 82.29 93.75 96.88 47.62 62.06 66.88
Mean ±ste 79.90 ±4.11 89.17 ±4.20 94.90 ±2.34 45.92 ±4.23 57.55 ±5.04 64.35 ±3.07
TABLE 2 | Results of online cued-guided experiment (unsupervised method).
Subject Accuracy (%) ITR (bpm)
PR VEP Hybrid PR VEP Hybrid
Sub1 69.79 98.96 97.92 18.95 38.15 37.19
Sub2 93.75 73.96 100.00 33.61 21.18 39.11
Sub3 96.88 54.17 98.96 36.24 11.55 38.15
Sub4 84.38 33.33 83.33 27.20 4.54 26.88
Sub5 52.08 33.33 62.50 10.62 5.09 15.09
Sub6 88.54 71.88 97.92 29.89 19.42 37.19
Sub7 96.88 41.67 95.83 36.24 6.97 35.28
Sub8 84.38 97.92 97.92 27.65 37.19 37.44
Sub9 95.83 71.88 96.88 35.52 19.34 36.48
Sub10 78.13 38.54 87.50 23.15 5.53 29.11
Mean ±ste 84.06 ±4.52 61.56 ±7.93 91.88 ±3.68 27.91 ±2.66 16.90 ±4.00 33.19 ±2.38
Frontiers in Human Neuroscience | www.frontiersin.org 12 February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 13
Jiang et al. h-BCI With VEP and PR
A B
FIGURE 13 | Classiﬁcation accuracy using (A) supervised method and (B) unsupervised method for each subject in the ofﬂine experiment.
stimulus used in the proposed h-BCI was a more comfortable
and favorable stimulus than the alpha frequency band used in the
DISCUSSION
Compared with other traditional SSVEP-BCI in the alpha
frequency range, the h-BCI system based on the low-frequency
stimulations developed in this study is more comfortable
and applicable for all users. This study performed a pre-
experiment to compare the subjective perception of visual stimuli
(Chien et al., 2017) with diﬀerent frequencies (from 1 to 60 Hz,
with an interval of 1 Hz). The averaged results with 12 subjects
showed that the subjects’ comfortableness and preference level
at the low-frequency (e.g., 1 Hz) were better than the alpha
frequency range (e.g., 10 Hz). Compared with the high-speed
SSVEP-BCIs (Chen et al., 2015a,b;Nakanishi et al., 2018), the
relatively low classiﬁcation accuracy and ITR in the proposed
system can be explained in several aspects. First of all, the
poor performance of VEP for the unsupervised method may
be caused by the low response amplitude at the low-frequency
band. Herrmann compared the SSVEP response evoked by the
ﬂicker light at frequencies from 1 Hz to 100 Hz (in 1 Hz
step) and found that the amplitude reached the highest in the
medium frequency band (maximum at 15 Hz) (Herrmann,
2001). The response amplitude and SNR decreased signiﬁcantly
below 10 Hz (Wang et al., 2005), which was related to the
higher level of the spontaneous EEG oscillations (Gao et al.,
2003). Therefore, the frequency detection techniques used in
the unsupervised method were aﬀected by the background
noise, while the TRCA-based spatial ﬁltering techniques used
in the supervised method were eﬀective to remove background
EEG and enhance SNR (Nakanishi et al., 2018). Second, the
narrow frequency range and separable frequency interval of
PR limit the number of targets and aﬀect ITR. For the
frequency range, Naber veriﬁed that the PR only responded
to repetitive onsets and oﬀsets of stimuli at 0.3–2.3 Hz, not
at 3.4 Hz (Naber et al., 2013). Regarding to the frequency
interval, Muto classiﬁed two visual stimulus targets with a
luminance-modulation frequency between 0.75 and 2.5 Hz
and found that the classiﬁcation accuracy with the frequency
interval of 0.06 Hz was poor (Muto et al., 2020). Third, data
length also inﬂuences ITR. Compared with the alpha frequency
band, which was commonly used in the traditional SSVEP-
BCI, the low-frequency band took more time to complete
the same number of stimulus periods to obtain suﬃcient
eﬀective information.
Compared with other SSVEP-BCI systems with better user
experience than the system using stimulations in the alpha
frequency range, the proposed h-BCI system oﬀers higher BCI
performance. Chang generated an amplitude-modulated (AM)
stimulus (Chang et al., 2014) as the product of two sine waves,
including a carrier frequency higher than 40 Hz to reduce
eye fatigue and a modulating frequency ranged around the
alpha band (9–12 Hz) to utilize harmonic information. Online
experiments showed that the average accuracy of the six-target
classiﬁcation task was 91.2%, resulting in an ITR of 30 bpm.
Dreyer adapted a frequency-modulated (FM) stimulation from
the auditory domain (Dreyer and Herrmann, 2015), with a
Frontiers in Human Neuroscience | www.frontiersin.org 13 February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 14
Jiang et al. h-BCI With VEP and PR
modulation frequency of 10 Hz and a carrier frequency ranging
from 20 to 100 Hz. The subjective ﬂicker perception of FM-
SSVEPs with carrier frequencies above 30 Hz was slight or even
imperceptible, while the amplitude of FM-SSVEPs (e.g., carrier
frequency = 30 Hz: 0.36 µV) remained the same as that of SSVEPs
evoked rectangularly (0.44 µV) or sinusoidally (0.42 µV). BCI
performances between FM and sinusoidal (SIN) SSVEPs were
compared using a four-target classiﬁcation paradigm (Dreyer
et al., 2017). The highest classiﬁcation accuracy was 86% for
FM (11 s epochs) and 95% for SIN (11 s epochs), and the
highest ITR was 13.91 bpm for FM (2 s epochs) and 24.21 bpm
for SIN (1 s epochs). Jiang developed a four-class phase-
coded SSVEP-BCI by imperceptible ﬂickers at 60 Hz (Jiang
et al., 2019). The system achieved a classiﬁcation accuracy
of 92.71 ±7.56% in the online experiment and an ITR of
18.81 ±4.74 bpm.
This study proved that the combination of PR and VEP
can facilitate the implementation of visual BCIs with low-
frequency stimulations, and the proposed system achieved high
classiﬁcation accuracy and short stimulus duration compared
with the other existing h-BCIs based on PR and VEP. There
was an improvement for the hybrid method compared to PR
and VEP for some subjects, especially when the data length
was short, and the accuracy was not saturated. However, there
are ways to improve the system performance and practicability
of the proposed h-BCI. Firstly, other unsupervised algorithms
can be explored to improve the SNR at the low-frequency band
with low response amplitude and facilitate the classiﬁcation of
VEPs. The hybrid accuracy can be improved by customizing the
weighted coeﬃcient of the hybrid method for each participant
due to the clear individual diﬀerences in the system performance.
Secondly, the optimization of the grid stimulus paradigm may
be helpful for increasing ITR and reducing the ﬂickering
sensation of visual stimulation, including the spatial frequency,
the proportion of stimulation area, and illuminance (Ming et al.,
2021). More entrained VEPs can be generated and enhanced
by stimuli containing spatial contrast than the spatially uniform
stimuli (Williams et al., 2004), and the spatial frequency can be
optimized to maximize the ITR and reduce the visual irritation
(Waytowich et al., 2017). The proportion of stimulation area
and stimulation illuminance can be minished to reduce the
perception of ﬂicker and maintain the system performance.
Finally, in order to be convenient to use, the system should
be compact and portable (Gao et al., 2003). This study was
completed in the laboratory using high-precision eye tracker
and EEG ampliﬁer. A wearable hybrid BCI system that can
be used in daily life may be realized with a high-precision
consumer-grade eye tracker and a wearable EEG device in the
future. By addressing these issues, the h-BCI based on VEP
and PR can be further improved and be potential for more
practical applications.
CONCLUSION
This study designed a 12-target h-BCI system with low
stimulation frequencies of 0.8–2.12 Hz, which can simultaneously
induce signiﬁcant PR and VEP responses. The SNR of PR was
higher than VEP at the fundamental frequency but lower than
VEP at harmonics. PR and VEP data were recorded in the
oﬄine experiments to optimize the algorithm parameters, and
the system performance was veriﬁed through online experiments.
The averaged accuracy across 10 subjects was 94.90 ±2.34%
at the data length of 1.5 s for the supervised method and
91.88 ±3.68% at 4 s for the unsupervised method, corresponding
to the ITR of 64.35 ±3.07 bpm and 33.19 ±2.38 bpm in the
online experiment. The h-BCI performance was better than PR
or VEP for some subjects, especially for the short data length
and unsaturated accuracy. The proposed h-BCI provides a good
solution to a practical BCI with balanced system performance and
user experience.
DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be
made available by the authors, without undue reservation.
ETHICS STATEMENT
The studies involving human participants were reviewed and
approved by the Research Ethics Committee of Tsinghua
University. The patients/participants provided their written
informed consent to participate in this study.
AUTHOR CONTRIBUTIONS
LJ developed the experimental system, performed data collection
and data analysis, and wrote the manuscript. XL developed the
experimental system and performed data analysis. WP and XG
revised the manuscript. YW supervised the study. All authors
contributed to the article and approved the submitted version.
FUNDING
This work was supported by the National Key R&D Program of
China under grant 2017YFA0205903, National Natural Science
Foundation of China under grant 62071447, Beijing Science
and Technology Program under grant Z201100004420015, and
Strategic Priority Research Program of Chinese Academy of
Sciences under grant XDB32040200.
ACKNOWLEDGMENTS
The authors would like to thank the subjects for participating in
the experiments.
SUPPLEMENTARY MATERIAL
The Supplementary Material for this article can be found
online at: https://www.frontiersin.org/articles/10.3389/fnhum.
2022.834959/full#supplementary-material
Frontiers in Human Neuroscience | www.frontiersin.org 14 February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 15
Jiang et al. h-BCI With VEP and PR
REFERENCES
Allison, B., Luth, T., Valbuena, D., Teymourian, A., Volosyak, I., and Graser, A.
(2010). BCI demographics: how many (and What Kinds of) people can use
an SSVEP BCI? IEEE Trans. Neural Systems Rehabilitation Eng. 18, 107–116.
doi: 10.1109/TNSRE.2009.2039495
Bieger, J., and Molina, G. G. (2010). Light Stimulation Properties to Inﬂuence Brain
Activity: A Brain-CoMputer Interface application. Amsterdam: Philips Research.
Bouma, H. (1962). Size of the static pupil as a function of wave-length and
luminosity of the light incident on the human eye. Nature 193, 690–691. doi:
10.1038/193690a0
Brainard, D. (1997). The psychophysics toolbox. Spatial Vision 10, 433–436.
Brunner, C., Allison, B. Z., Krusienski, D. J., Kaiser, V., Gr, M. P., Pfurtscheller, G.,
et al. (2010). Improved signal processing approaches in an oﬄine simulation
of a hybrid brain–computer interface. J. Neurosci. Methods 188, 165–173. doi:
10.1016/j.jneumeth.2010.02.002
Chang, M. H., Baek, H. J., Lee, S. M., and Park, K. S. (2014). An amplitude-
modulated visual stimulation for reducing eye fatigue in SSVEP-based brain–
computer interfaces. Clin. Neurophysiol. 125, 1380–1391. doi: 10.1016/j.clinph.
2013.11.016
Chen, J., Maye, A., Engel, A. K., Wang, Y., and Zhang, D. (2019). Simultaneous
decoding of eccentricity and direction information for a single-ﬂicker SSVEP
BCI. Electronics 8:1554. doi: 10.3390/electronics8121554
Chen, X., Wang, Y., Gao, S., Jung, T. P., and Gao, X. (2015a). Filter bank
canonical correlation analysis for implementing a high-speed SSVEP-based
brain-computer interface. J. Neural Eng. 12:046008. doi: 10.1088/1741-2560/12/
4/046008
Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T. P., and Gao, S. (2015b). High-
speed spelling with a noninvasive brain-computer interface. Proc. Natl. Acad.
Sci. U S A. 112, E6058–E6067. doi: 10.1073/pnas.1508080112
Chien, Y. Y., Lin, F. C., Zao, J. K., Chou, C. C., Huang, Y. P., Kuo, H. Y.,
et al. (2017). Polychromatic SSVEP stimuli with subtle ﬂickering adapted to
brain-display interactions. J. Neural Eng. 14:016018. doi: 10.1088/1741-2552/
aa550d
Crawford, B. H. (1936). The dependence of pupil size upon external light stimulus
under static and variable conditions. Proc. R. Soc. B 121, 376–395. doi: 10.1098/
rspb.1936.0072
De’Sperati, C., Roatta, S., and Baroni, T. (2020). Decoding overt shifts of attention
in depth through pupillary and cortical frequency tagging. J. Neural Eng. doi:
10.1088/1741-2552/ab8e8f Online ahead of print.
Dreyer, A. M., and Herrmann, C. S. (2015). Frequency-modulated steady-state
visual evoked potentials: a new stimulation method for brain–computer
interfaces. J. Neurosci. Methods 241, 1–9. doi: 10.1016/j.jneumeth.2014.
12.004
Dreyer, A. M., Herrmann, C. S., and Rieger, J. W. (2017). Tradeoﬀ between user
experience and BCI classiﬁcation accuracy with frequency modulated steady-
state visual evoked potentials. Front. Hum. Neurosci. 11:391. doi: 10.3389/
fnhum.2017.00391
Fisher, R. S., Harding, G., Erba, G., Barkley, G. L., and Wilkins, A. (2005). Photic-
and pattern-induced seizures: a review for the Epilepsy Foundation of America
Working Group. Epilepsia 46, 1426–1441. doi: 10.1111/j.1528-1167.2005.
31405.x
Floriano, A., Carmona, V. L., Diez, P. F., and Bastos-Filho, T. F. (2019). A study
of SSVEP from below-the-hairline areas in low-, medium-, and high-frequency
ranges. Res. Biomed. Eng. 35, 71–76. doi: 10.1007/s42600-019-00005-2
Gandhi, V. (2007). Toward Brain-computer Interfacing. Cambridge, MA: MIT
Press. doi: 10.1111/j.1468-1331.2008.02463.x
Gao, S., Wang, Y., Gao, X., and Hong, B. (2014). Visual and auditory brain-
computer interfaces. IEEE Trans. Biomed. Eng. 61, 1436–1447. doi: 10.1109/
TBME.2014.2300164
Gao, X., Xu, D., Cheng, M., and Gao, S. (2003). A BCI-based environmental
controller for the motion-disabled. IEEE Trans. Neural Systems Rehabilitation
Eng. 11, 137–140. doi: 10.1109/TNSRE.2003.814449
Harding, G. F., and Harding, P. F. (2010). Televised material and photosensitive
epilepsy. Epilepsia 40, 65–69. doi: 10.1111/j.1528-1157.1999.tb00909.x
Hartmann, E., Lachenmayr, B., and Brettel, H. (1978). The peripheral critical ﬂicker
frequency. Vision Res. 19, 1019–1023. doi: 10.1016/0042-6989(79)90227-X
Herrmann, C. S., (2001). Human EEG responses to 1–100 Hz ﬂicker: resonance
phenomena in visual cortex and their potential correlation to cognitive
phenomena. Exp. Brain Res. 137, 346–353. doi: 10.1007/s002210100682
Hoﬀmann, U., Fimbel, E. J., and Keller, T. (2009). “Brain-computer interface based
on high frequency steady-state visual evoked potentials: a feasibility study,” in
Proceedings of the 2009 4th International IEEE/EMBS Conference on Neural
Engineering, (Piscataway, NJ: IEEE).
Hoﬀmann, U., Vesin, J. M., Ebrahimi, T., and Diserens, K. (2008). An eﬃcient
P300-based brain-computer interface for disabled subjects. J. Neurosci. Methods
167, 115–125. doi: 10.1016/j.jneumeth.2007.03.005
Hong, K. S., and Jawad, K. M. (2017). Hybrid brain–computer interface techniques
for improved classiﬁcation accuracy and increased number of commands: a
review. Front. Neurorobotics 11:35. doi: 10.3389/fnbot.2017.00035
Jiang, L., Li, X., Wang, Y., Pei, W., and Chen, H. (2020). “Comparison of pupil size
and visual evoked potentials under 1-6Hz visual stimulation,” in Proceedings of
the 42nd Annual International Conference of the IEEE Engineering in Medicine
and Biology Society (EMBC), (Piscataway, NJ: IEEE), doi: 10.1109/embc44109.
2020.9175893
Jiang, L., Wang, Y., Pei, W., and Chen, H. (2019). “A four-class phase-coded SSVEP
BCI at 60Hz using refresh rate,” in Proceedings of the 41st Annual International
Conference of the IEEE Engineering in Medicine and Biology Society (EMBC),
(Piscataway, NJ: IEEE).
Kret, M. E., and Sjak-Shie, E. E. (2019). Preprocessing pupil size data: guidelines
and code. Behav. Res. Methods 51, 1336–1342. doi: 10.3758/s13428-018-1075- y
Kuebier, A., Furdea, A., Halder, S., Hammer, E. M., Nijboer, F., and Kotchoubey, B.
(2009). A brain-computer interface controlled auditory event-related potential
(p300) spelling system for locked-in patients. Ann. N. Y. Acad. Sci. 1157,
90–100. doi: 10.1111/j.1749-6632.2008.04122.x
Lin, Z., Zhang, C., Wu, W., and Gao, X. (2006). Frequency recognition based on
canonical correlation analysis for SSVEP-Based BCIs. IEEE Trans. Biomed. Eng.
53, 2610–2614. doi: 10.1109/TBME.2006.889197
Luo, A., and Sullivan, T. J. (2010). A user-friendly SSVEP-based brain–computer
interface using a time-domain classiﬁer. J. Neural Eng. 7:026010. doi: 10.1088/
1741-2560/7/2/026010
Ma, X., Yao, Z., Wang, Y., Pei, W., and Chen, H. (2018). “Combining brain-
computer interface and eye tracking for high-speed text entry in virtual reality,
in Proceedings of the 23rd International Conference on Intelligent User Interfaces,
(New York, NY: ACM), 263–267. doi: 10.1145/3172944.3172988
Mao, X., Li, W., Hu, H., Jin, J., and Chen, G. (2020). Improve the classiﬁcation
eﬃciency of high-frequency phase-tagged SSVEP by a recursive bayesian-based
approach. IEEE Trans. Neural. Syst. Rehabil. Eng. 28, 561–572. doi: 10.1109/
TNSRE.2020.2968579
Mason, S. G., Bashashati, A., Fatourechi, M., and Navarrog, K. F. (2007). A
comprehensive survey of brain interface technology designs. Ann. Biomed. Eng.
35, 137–169. doi: 10.1007/s10439-006- 9170-0
Ming, G., Pei, W., Chen, H., Gao, X., and Wang, Y. (2021). Optimizing spatial
properties of a new checkerboard-like visual stimulus for user-friendly SSVEP-
based BCIs. J. Neural Eng. 18:056046. doi: 10.1088/1741-2552/ac284a
Muto, Y., Miyoshi, H., and Kaneko, H. (2020). Eye-gaze information input based
on pupillary response to visual stimulus with luminance modulation. PLoS One
15:e0226991. doi: 10.1371/journal.pone.0226991
Naber, M., Alvarez, G., and Nakayama, K. (2013). Tracking the allocation of
attention using human pupillary oscillations. Front. Psychol. 4:919. doi: 10.3389/
fpsyg.2013.00919
Nakanishi, M., Wang, Y., Chen, X., Wang, Y. T., Gao, X., and Jung, T. P. (2018).
Enhancing detection of SSVEPs for a high-speed brain speller using task-
related component analysis. IEEE Trans. Biomed. Eng. 65, 104–112. doi: 10.
1109/TBME.2017.2694818
Nakanishi, M., Wang, Y., Wang, Y. T., Mitsukura, Y., and Jung, T. P. (2014).
A high-speed brain speller using steady-state visual evoked potentials. Int. J.
Neural Syst. 24:1450019. doi: 10.1142/S0129065714500191
Odom, J. V., Bach, M., Barber, C., Brigell, M., Marmor, M. F., Tormene, A. P.,
et al. (2004). Visual evoked potentials standard (2004). Doc. Ophthalmol. 108,
115–123. doi: 10.1023/B:DOOP.0000036790.67234.22
Ponzio, F., Villalobos, A., Mesin, L., De’sperati, C., and Roatta, S. (2019). A human-
computer interface based on the "voluntary" pupil accommodative response.
Int. J. Hum. Comp. Stud. 126, 53–63. doi: 10.1016/j.ijhcs.2019.02.002
Frontiers in Human Neuroscience | www.frontiersin.org 15 February 2022 | Volume 16 | Article 834959
fnhum-16-834959 January 28, 2022 Time: 21:15 # 16
Jiang et al. h-BCI With VEP and PR
Sebastiaan, M., Jean-Baptiste, M., Lotje, V., Stefan, V., and Hedderik, V. R. (2016).
The mind-writing pupil: a human-computer interface based on decoding of
covert attention through pupillometry. PLoS One 11:e0148805. doi: 10.1371/
journal.pone.0148805
Vidal, J. J. (1977). Real-time detection of brain events in EEG. Proc. IEEE 65,
633–641. doi: 10.1109/PROC.1977.10542
Vidaurre, C., and Blankertz, B. (2009). Towards a cure for BCI illiteracy. Brain
Topogr. 23, 194–198. doi: 10.1007/s10548-009- 0121-6
Volosyak, I. (2011). SSVEP-based Bremen-BCI interface–boosting information
transfer rates. J. Neural Eng. 8:036020. doi: 10.1088/1741-2560/8/3/036020
Volosyak, I., Gembler, F., and Stawicki, P. (2017). Age-related diﬀerences in
SSVEP-based BCI performance. Neurocomputing 250, 57–64. doi: 10.1016/j.
neucom.2016.08.121
Wang, Y., Gao, X., Hong, B., Jia, C., and Gao, S. (2008). Brain-Computer interfaces
based on visual evoked potentials. IEEE Eng. Med. Biol. Mag. 27, 64–71. doi:
10.1109/MEMB.2008.923958
Wang, Y., Wang, R., Gao, X., and Gao, S. (2005). “Brain-computer interface based
on the high-frequency steady-state visual evoked potential,” in Proceedings of
the 1st International Conference on Neural Interface & Control, (Piscataway, NJ:
IEEE).
Waytowich, N. R., Yamani, Y., and Krusienski, D. J. (2017). Optimization of
checkerboard spatial frequencies for steady-state visual evoked potential brain–
computer interfaces. IEEE Trans. Neural. Syst. Rehabil. Eng. 25, 557–565. doi:
10.1109/TNSRE.2016.2601013
Williams, P. E., Mechler, F., Gordon, J., Shapley, R., and Hawken, M. J. (2004).
Entrainment to video displays in primary visual cortex of macaque and humans.
J. Neurosci. 24:8278. doi: 10.1523/JNEUROSCI.2716-04.2004
Woodhouse, J. M. (1975). The eﬀect of pupil size on grating detection at various
contrast levels. Vision Res. 15, 645–648. doi: 10.1016/0042-6989(75)90278-3
Woodhouse, J. M., and Campbell, F. W. (1975). The role of the pupil light reﬂex
in aiding adaptation to the dark. Vision Res. 15, 649–653. doi: 10.1016/0042-
6989(75)90279-5
Wu, Z., and Su, S. (2014). “Detection accuracy comparison between the high
frequency and low frequency SSVEP-based BCIs,” in Proceedings of the 2nd
International Conference on Communications, Signal Processing, and Systems,
Tianjin, 307–312.
Xu, M., Han, J., Wang, Y., Jung, T. P., and Ming, D. (2020). Implementing over
100 command codes for a high-speed hybrid brain-computer interface using
concurrent P300 and SSVEP features. IEEE Trans. Biomed. Eng. 67, 3073–3082.
doi: 10.1109/TBME.2020.2975614
Yao, Z., Ma, X., Wang, Y., Zhang, X., Liu, M., Pei, W., et al. (2018).
High-Speed spelling in virtual reality with sequential hybrid BCIs. IEICE
Trans. Inform. Systems 101, 2859–2862. doi: 10.1587/transinf.2018EDL
8122
Zhu, D., Bieger, J., Molina, G. G., and Aarts, R. M. (2010). A survey of stimulation
methods used in SSVEP-based BCIs. Comp. Intell. Neurosci. 2010:702357. doi:
10.1155/2010/702357
Conﬂict of Interest: The authors declare that the research was conducted in the
absence of any commercial or ﬁnancial relationships that could be construed as a
potential conﬂict of interest.
Publisher’s Note: All claims expressed in this article are solely those of the authors
and do not necessarily represent those of their aﬃliated organizations, or those of
the publisher, the editors and the reviewers. Any product that may be evaluated in
this article, or claim that may be made by its manufacturer, is not guaranteed or
endorsed by the publisher.
Copyright © 2022 Jiang, Li, Pei, Gao and Wang. This is an open-access article
The use, distribution or reproduction in other forums is permitted, provided the
original author(s) and the copyright owner(s) are credited and that the original
publication in this journal is cited, in accordance with accepted academicpractice. No
use, distribution or reproduction is permitted which does not comply with theseterms.
Frontiers in Human Neuroscience | www.frontiersin.org 16 February 2022 | Volume 16 | Article 834959
... Thus, a CCA-based spatial filter can be obtained and then the 8-channel data were processed by the CCA-based spatial filter. Lastly, the amplitude spectrum y(f) was calculated by fast Fourier Transform (FFT), and the SNR in decibels (dB) was defined as the ratio of y(f) to the mean value of the eight neighboring frequencies [40]. The flow chart of the calculation process is shown in Figure 3. One-way repeated-measures analysis of variance (ANOVA) was also used to test the difference of amplitude and SNR between the PreG electrode and the wet electrode at different frequencies. ...
Article
Full-text available
Electroencephalogram (EEG) electrodes are critical devices for brain-computer interface and neurofeedback. A pre-gelled (PreG) electrode was developed in this paper for EEG signal acquisition with a short installation time and good comfort. A hydrogel probe was placed in advance on the Ag/AgCl electrode before wearing the EEG headband instead of a time-consuming gel injection after wearing the headband. The impedance characteristics were compared between the PreG electrode and the wet electrode. The PreG electrode and the wet electrode performed the Brain-Computer Interface (BCI) application experiment to evaluate their performance. The average impedance of the PreG electrode can be decreased to 43 $\text{k}\Omega$ or even lower, which is higher than the wet electrode with an impedance of 8 $\text{k}\Omega$ . However, there is no significant difference in classification accuracy and information transmission rate (ITR) between the PreG electrode and the wet electrode in a 40 target BCI system based on Steady State Visually Evoked Potential (SSVEP). This study validated the efficiency of the proposed PreG electrode in the SSVEP-based BCI. The proposed PreG electrode will be an excellent substitute for wet electrodes in an actual application with convenience and good comfort.
... Such effects can be reduced with high frequency stimulation paradigms (Sakurada et al., 2015) but high frequency stimulation can in turn reduce BCI performance (Volosyak et al., 2011). Other methods like amplitude modulated signals (Chang et al., 2014) or specific low frequency signals (Jiang et al., 2022) have been reported to increase user experience as well. In addition, we have previously shown that FM stimulation signals with a high carrier frequency are less perceptible (Dreyer and Herrmann, 2015) and more comfortable (Dreyer et al., 2017) than traditionally used stimulation signals. ...
Article
Full-text available
Based on increased user experience during stimulation, frequency-modulated steady-state visual evoked potentials (FM-SSVEPs) have been suggested as an improved stimulation method for brain-computer interfaces. Adapting such a novel stimulation paradigm requires in-depth analyses of all different stimulation parameters and their influence on brain responses as well as the user experience during the stimulation. In the current manuscript, we assess the influence of different values for the modulation index, which determine the spectral distribution in the stimulation signal on FM-SSVEPs. We visually stimulated 14 participants at different target frequencies with four different values for the modulation index. Our results reveal that changing the modulation index in a way that elevates the stimulation power in the targeted sideband leads to increased FM-SSVEP responses. There is, however, a tradeoff with user experience as increased modulation indices also lead to increased perceived flicker intensity as well as decreased stimulation comfort in our participants. Our results can guide the choice of parameters in future FM-SSVEP implementations.
Article
Full-text available
Objective.Low-frequency steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems with high performance are prone to cause visual discomfort and fatigue. High-frequency SSVEP-based BCI systems can alleviate the discomfort, but always obtain lower performance. This study optimized the spatial properties of a proposed checkerboard-like visual stimulus toward high-performance and user-friendly SSVEP-based BCI systems.Approach.On the one hand, two checkerboard-like stimuli with distinct spatial contrasts (the black- and white-background) were designed to balance the tradeoff between BCI performance and user experience and compared with the traditional flickering stimulus. On the other hand, the impacts of the spatial frequency of the new checkerboard-like stimulus on the flicker perception and the intensity of the elicited SSVEP were clarified. The SSVEP-based BCI systems were implemented based on the checkerboard-like stimuli under low-frequency and high-frequency conditions. The user experience for each stimulation pattern was estimated by questionnaires for subjective evaluation.Main results.The comparison results indicate that the black-background checkerboard-like stimulus with an optimized spatial frequency achieved comparable performance and enhanced visual comfort compared with the flickering stimulus. Furthermore, the online nine-target BCI system using the black-background checkerboard-like stimuli achieved averaged information transfer rates (ITRs) of 124.0±2.3 bits/min and 109.0 ± 20.4 bits/min with low-frequency and high-frequency stimulation respectively.Significance.The new checkerboard-like stimuli with optimized properties show superiority of system performance and user experience in implementing SSVEP-based BCI, which will promote its practical applications in communication and control.
Article
Full-text available
Objective: Recently, electroencephalography (EEG)- based brain-computer interfaces (BCIs) have made tremendous progress in increasing communication speed. However, current BCI systems could only implement a small number of command codes, which hampers their applicability. Methods: This study developed a high-speed hybrid BCI system containing as many as 108 instructions, which were encoded by concurrent P300 and steady-state visual evoked potential (SSVEP) features and decoded by an ensemble task-related component analysis method. Notably, besides the frequency-phase-modulated SSVEP and time-modulated P300 features as contained in the traditional hybrid P300 and SSVEP features, this study found two new distinct EEG features for the concurrent P300 and SSVEP features, i.e. time-modulated SSVEP and frequency-phase- modulated P300. Ten subjects spelled in both offline and online cued-guided spelling experiments. Other ten subjects took part in online copy-spelling experiments. Results: Offline analyses demonstrate that the concurrent P300 and SSVEP features can provide adequate classification information to correctly select the target from 108 characters in 1.7 seconds. Online cued-guided spelling and copy-spelling tests further show that the proposed BCI system can reach an average information transfer rate (ITR) of 172.46±32.91 bits/min and 164.69±33.32 bits/min respectively, with a peak value of 238.41 bits/min (The demo video of online copy-spelling is enclosed and can be found at https://www.youtube.com/watch?v=EW2Q08oHSBo). Conclusion: We expand a BCI instruction set to over 100 command codes with high-speed in an efficient manner, which significantly improves the degree of freedom of BCIs. Significance: This study hold promise for broadening the applications of BCI systems.
Article
Full-text available
Among the Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs), the phase-tagged SSVEP (p-SSVEP) has been proved a reliable paradigm to extend the number of available targets, especially for high-frequency SSVEP-based BCIs. However, the recognition efficiency of the high-frequency p-SSVEP still remains relatively low. A longer data segment may achieve a higher classification accuracy, but the time consumption of computation leads to the decrease of information transfer rate. This paper presents a recursive Bayesian-based approach to improve the high-frequency p-SSVEP classification efficiency. In each signal processing period, the classification result is generated by the current scores, the condition probability and a recursive prior probability (dynamic prior probability). The experiment displays the SSVEP stimuli with 20 Hz and 30 Hz respectively, and each frequency contains six phases. This paper compared three classification approaches and the recursive Bayesian-based approach could obtain the highest classification accuracy and practical bit rate under the same data length. The mean accuracy and practical bit rate were 89.7% and 37.8 bits/min with 20Hz, and 89.0% and 36.5 bits/min with 30Hz, respectively Furthermore, the recursive Bayesian-based approach could reduce the individual differences among different subjects. Therefore, the recursive Bayesian-based approach can lead to high classification efficiency in high-frequency p-SSVEP.
Article
Full-text available
This study develops an information-input interface in which a visual stimulus targeted by a user’s eye gaze is identified based on the pupillary light reflex to periodic luminance modulations of the object. Experiment 1 examines how pupil size changes in response to periodic luminance modulation of visual stimuli, and the results are used to develop an algorithm for information input. Experiment 2a examines the effectiveness of interfaces with two objects. The results demonstrate that 98% accurate identification of the gaze targeted object is possible if the luminance modulation frequencies of two objects differ by at least 0.12 Hz. Experiment 2b examines the accuracy of a gaze directed information input method based on a keyboard configuration with twelve responses. The results reveal that keyboard input is possible with an average accuracy of 85% for luminance modulation frequencies from 0.75 to 2.75 Hz. The proposed pupillometry based information-input interface offers several advantages, such as low burden on users, minimal invasiveness, no need for training or experience, high theoretical validity, and no need for calibration. Thus, the pupillometry method presented herein has advantages for practical use without requiring the eye’s position to be calibrated. Additionally, this method has a potential for the design of interfaces that allow patients with severely limited motor function to communicate with others.
Article
Full-text available
The feasibility of a steady-state visual evoked potential (SSVEP) brain-computer interface (BCI) with a single-flicker stimulus for multiple-target decoding has been demonstrated in a number of recent studies. The single-flicker BCIs have mainly employed the direction information for encoding the targets, i.e., different targets are placed at different spatial directions relative to the flicker stimulus. The present study explored whether visual eccentricity information can also be used to encode targets for the purpose of increasing the number of targets in the single-flicker BCIs. A total number of 16 targets were encoded, placed at eight spatial directions, and two eccentricities (2.5 • and 5 •) relative to a 12 Hz flicker stimulus. Whereas distinct SSVEP topographies were elicited when participants gazed at targets of different directions, targets of different eccentricities were mainly represented by different signal-to-noise ratios (SNRs). Using a canonical correlation analysis-based classification algorithm, simultaneous decoding of both direction and eccentricity information was achieved, with an offline 16-class accuracy of 66.8 ± 16.4% averaged over 12 participants and a best individual accuracy of 90.0%. Our results demonstrate a single-flicker BCI with a substantially increased target number towards practical applications.
Article
Full-text available
Objectives: Changes in pupil size are governed by the autonomic nervous system but may be systematically driven by voluntary shifting the gaze in depth. Thus, the pupil accommodative response (PAR) that accompanies voluntary gaze shifts from a far to a near target might be exploited as a simple human-computer interface (HCI), bypassing the somato-motor system. Here we aim to characterize PAR in quasi-natural conditions with low-cost equipment and test the possibility to use PAR as a binary communication tool. Methods: Nineteen healthy subjects were instructed to voluntary switch the focus from a far to a near target upon presentation of an auditory cue. Three protocols addressed the effects of monocular/binocular vision, eye illuminance, duration of near vision, target texture and target brightness on PAR features. In a fourth protocol PAR was used to establish binary communication at different bit rates. Results: PAR amplitude decreased with increasing eye illuminance and was modulated by monocular/binocular vision, duration of near vision or target texture. PAR amplitude was larger with a bright near target and a dark far target than vice-versa. However, PAR was always clearly present, regardless of the experimental manipulations. PAR-based communication performance achieved an accuracy of 100% at 10 bits/min and 96% at 15 bits/min. Significance: Voluntary PAR is a robust signal, and can achieve a high communication speed when used as an HCI. This study provides a proof of concept for a PAR-based HCI, potentially useful to communicate with locked-in patients with preserved visual and autonomic functions.
Conference Paper
In order to explore the effect of low frequency stimulation on pupil size and electroencephalogram (EEG), we presented subjects with 1-6Hz black-and-white-alternating flickering stimulus, and compared the differences of signal-to-noise ratio (SNR) and classification performance between pupil size and visual evoked potentials (VEPs). The results showed that the SNR of the pupillary response reached the highest at 1Hz (17.19± 0.10dB) and 100% accuracy was obtained at 1s data length, while the performance was poor at the stimulation frequency above 3Hz. In contrast, the SNR of VEPs reached the highest at 6Hz (18.57± 0.37dB), and the accuracy of all stimulus frequencies could reach 100%, with the minimum data length of 1.5s. This study lays a theoretical foundation for further implementation of a hybrid brain-computer interface (BCI) that integrates pupillometry and EEG.
Article
Objective: We have recently developed a prototype of a novel human-computer interface for assistive communication based on voluntary shifts of attention (gaze) from a far target to a near target associated with a decrease of pupil size (Pupillary Accommodative Response, PAR), an automatic vegetative response that can be easily recorded. We report here an extension of that approach based on pupillary and cortical frequency tagging. Approach: In 18 healthy volunteers, we investigated the possibility of decoding attention shifts in depth by exploiting the evoked oscillatory responses of the pupil (Pupillary Oscillatory Response, POR, recorded through a low-cost device) and visual cortex (Steady-State Visual Evoked Potentials, SSVEP, recorded from 4 scalp electrodes). With a simple binary communication protocol (focusing on a far target meaning "No", focusing on the near target meaning "Yes"), we aimed at discriminating when observer's overt attention (gaze) shifted from the far to the near target, which were flickering at different frequencies. Main results: By applying a binary linear classifier (Support Vector Machine, SVM, with leave-one-out cross validation) to POR and SSVEP signals, we found that, with only twenty trials and no subjects' behavioural training, the offline median decoding accuracy was 75% and 80% with POR and SSVEP signals, respectively. When the two signals were combined together, accuracy reached 83%. The number of observers for whom accuracy was higher than 70% was 11/18, 12/18 and 14/18 with POR, SVVEP and combined features, respectively. A signal detection analysis confirmed these results. Significance: The present findings suggest that exploiting frequency tagging with pupillary or cortical responses during an attention shift in the depth plane, either separately or combined together, is a promising approach to realize a device for communicating with Complete Locked-In Syndrome (CLIS) patients when oculomotor control is unreliable and traditional assistive communication, even based on PAR, is unsuccessful.
Conference Paper
A four-class brain-computer interface (BCI) system based on steady-state visual evoked potentials (SSVEPs) was developed by presenting phase-coded 60Hz stimulations on a 240Hz LCD monitor. The task-related component analysis (TRCA) algorithm was used to detect SSVEPs with individual training data. In the BCI experiment with 10 subjects, the system achieved high classification accuracy of 94.50±6.70% and 92.71±7.56% in offline and online BCI experiments, resulting in information transfer rates (ITR) of 19.95±4.36 and 18.81±4.74 bpm, respectively. The behavioral tests on visual comfortableness and perception of flickering reveal that the proposed BCI system is very comfortable to use without any perception of flicker.
Article
Purpose Recent studies have analyzed steady-state visual evoked potentials (SSVEPs) measured on belowthe-hairline areas, such as behind-the-ears (temporal area) and face (frontal area) using different montages of channels and frequency bands. This study aims to investigate how both reference electrode and frequency ranges (low, medium, and high bands) affect the SSVEP measured on hairless areas of temporal and frontal. Methods The EEG signals were acquired from 12 individuals, and the elicited SSVEP was evaluated in terms of amplitude and signal-to-noise ratio (SNR). Results The best electrode combinations for measuring SSVEP on hairless areas are obtained with Fpz-Tp9 and Fpz-Tp10 (up to 40 Hz); however, for stimuli frequencies higher than 40 Hz, the best result is obtained with the temporal area and with the reference electrode located on the ear. Conclusion The SSVEP amplitude and the SNR depend on the combination of the electrode reference and the range of visual stimulus frequency. These findings can aid in the development of more practical and comfortable SSVEP-based BCIs.