Chapter

A Tri-Hybrid Brain-Computer Interface for Neuro-Information Systems

Authors:
Chapter

A Tri-Hybrid Brain-Computer Interface for Neuro-Information Systems

If you want to read the PDF, try requesting it from the authors.

Abstract

Brain-computer interfaces (BCIs) are computerized systems that convert brain activity into control commands to operate software or external devices. Though promising, BCIs currently have limited practicality and usership due to poor signal classification and large training data requirements. The present study aims to overcome both challenges by combining three brain signals. This paradigm could improve existing BCI technical efficacy, and extrapolate to applications where hands-free visual interfaces could equip users with communication and information resources that improve work processes.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... The thresholding pattern received the intended action and generalized fashion of the neural input, then to specify parameters related to the BCI technology used. After that, the process goes from neural data simulation to the user interface and interaction (acquisition) to display multimedia-based actions [5,50]. Further, an alteration between stimulation and data acquisition is possible shown in Fig. 2 [1,51]. ...
Article
Full-text available
A brain-computer interface (BCI) affords real-time communication, significantly improving the quality of lifecycle, brain-to-internet (B2I) connectivity, and communication between the brain and external digital devices. This assistive technology innovates information and communication development paradigms, such as directly connecting the brain and multimedia devices to the cyber world. The system converts brain information to understandable signals for multimedia devices without physical interference and replaces human-based languages with the external environment control protocols. This advancement challenges and limits security severely. For this reason, the rate of attacks, malware, ransomware, and other types of vulnerabilities is increasing drastically. Another reason is the need to improve traditional procedures to investigate cyberenvironment security aspects. Also, these malicious attackers' prime objective is to harm personal information, enable content security and privacy protocols and physical systems integrity, and create high risk between system and consumers. However, security's capital importance stems from the growing number of wearables (on-body) and in-body wireless devices. These limitations affect personal and healthcare wireless networks during the communication (such as on-chain and off-chain) between human and wearable sensors (sense and transmit) and actuators. This paper presents a novel, secure Blockchain Security Module (BSM) for BCI with Multimedia Life Cycle Framework (MLCF) (BSM-BCIMLCF) that safely connects wearables while investigating the present-day BCI life cycle (BCILC) protection. It homogenizes a Blockchain-based distributed permission network approach to overcome existing challenges. The Blockchain enables assistant cybersecurity for BCI distributed applications to identify brain operations in real-time.
Chapter
Full-text available
As the NeuroIS field expands its scope to address more complex research questions with electroencephalography (EEG), there is greater need for EEG analysis capabilities that are relatively easy to implement and adapt to different protocols, while at the same time providing an open and standardized approach. We present a series of open source tools, based on the Python programming language, which are designed to facilitate the development of open and collaborative EEG reserach. As supplementary material, we demonstrate the implementation of these tools in a NeuroIS case study and provide files that can be adapted by others for NeuroIS EEG research .
Chapter
Full-text available
In this research-in-progress paper, we propose an experiment to investigate the neurophysiological correlates of mind wandering using electroencephalography (EEG). Auditory oddball event related potentials have been observed to be sensitive to the mind wandering state and can be used as a real-time passive measure. This has advantages over standard survey techniques because it is an objective, non-disruptive real time measure. We describe an experiment to observe the neurophysiological correlates of mind wandering in online learning environments using an auditory oddball. In doing so, we introduce a new experimental paradigm to the IS literature which could be used to extend other attention-related research.
Article
Full-text available
A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.
Article
Full-text available
Background: Steady-state visual-evoked potential (SSVEP)-based brain-computer interfaces (BCIs) generate weak SSVEP with a monitor and cannot use harmonic frequencies, whereas P300-based BCIs need multiple stimulation sequences. These issues can decrease the information transfer rate (ITR). New method: In this paper, we introduce a novel hybrid SSVEP-P300 speller that generates dual-frequency SSVEP, allowing it to overcome the abovementioned limitations and improve the performance. The hybrid speller consists of nine panels flickering at different frequencies. Each panel contains four different characters that appear in a random sequence. The flickering panel and the periodically updating character evoke the dual-frequency SSVEP, while the oddball stimulus of the target character evokes the P300. A canonical correlation analysis (CCA) and a step-wise linear discriminant analysis (SWLDA) classified SSVEP and P300, respectively. Ten subjects participated in offline and online experiments, in which accuracy and ITR were compared with those of conventional SSVEP and P300 spellers. Results: The offline analysis revealed not only the P300 potential but also SSVEP with peaks at sub-harmonic frequencies, demonstrating that the proposed speller elicited dual-frequency SSVEP. This dual-frequency stimulation improved SSVEP recognition, increased the number of targets by employing harmonic frequencies, reduced the stimulation time for P300, and consequently improved ITR as compared to the conventional spellers. Comparison with existing methods: The new method reduces the stimulation time and allows harmonic frequencies to be employed for different stimuli. Conclusions: The results indicate that this study provides a promising approach to make the BCI speller more reliable and efficient.
Article
Full-text available
A recent study showed improved performance of the P300-speller when the flashing row or column was overlaid with translucent pictures of familiar faces (FF spelling paradigm). However, the performance of the P300-speller is not yet satisfactory due to its low classification accuracy and information transfer rate. To investigate whether P300-speller performance is further improved when the chromatic property and the FF spelling paradigm are combined. We proposed a new spelling paradigm in which the flashing row or column is overlaid with translucent green pictures of familiar faces (GFF spelling paradigm). We analyzed the ERP waveforms elicited by the FF and proposed GFF spelling paradigms and compared P300-speller performance between the two paradigms. Significant differences in the amplitudes of four ERP components (N170, VPP, P300, and P600f) were observed between both spelling paradigms. Compared to the FF spelling paradigm, the GFF spelling paradigm elicited ERP waveforms of higher amplitudes and resulted in improved P300-speller performance. Combining the chromatic property (green color) and the FF spelling paradigm led to better classification accuracy and an increased information transfer rate. These findings demonstrate a promising new approach for improving the performance of the P300-speller.
Article
Full-text available
Magnetoencephalography and electroencephalography (M/EEG) measure the weak electromagnetic signals generated by neuronal activity in the brain. Using these signals to characterize and locate neural activation in the brain is a challenge that requires expertise in physics, signal processing, statistics, and numerical methods. As part of the MNE software suite, MNE-Python is an open-source software package that addresses this challenge by providing state-of-the-art algorithms implemented in Python that cover multiple methods of data preprocessing, source localization, statistical analysis, and estimation of functional connectivity between distributed brain regions. All algorithms and utility functions are implemented in a consistent manner with well-documented interfaces, enabling users to create M/EEG data analysis pipelines by writing Python scripts. Moreover, MNE-Python is tightly integrated with the core Python libraries for scientific comptutation (NumPy, SciPy) and visualization (matplotlib and Mayavi), as well as the greater neuroimaging ecosystem in Python via the Nibabel package. The code is provided under the new BSD license allowing code reuse, even in commercial products. Although MNE-Python has only been under heavy development for a couple of years, it has rapidly evolved with expanded analysis capabilities and pedagogical tutorials because multiple labs have collaborated during code development to help share best practices. MNE-Python also gives easy access to preprocessed datasets, helping users to get started quickly and facilitating reproducibility of methods by other researchers. Full documentation, including dozens of examples, is available at http://martinos.org/mne.
Chapter
Full-text available
Brain-computer interface (BCI) systems can provide communication and control for many users, but not all users. This problem exists across different BCI approaches; a “universal” BCI that works for everyone has never been developed. Instead, about 20% of subjects are not proficient with a typical BCI system. Some groups have called this phenomenon “BCI illiteracy”. Some possible solutions have been explored, such as improved signal processing, training, and new tasks or instructions. These approaches have not resulted in a BCI that works for all users, probably because a small minority of users cannot produce detectable patterns of brain activity necessary to a particular BCI approach. We also discuss an underappreciated solution: switching to a different BCI approach. While the term “BCI illiteracy” elicits interesting comparisons between BCIs and natural languages, many issues are unclear. For example, comparisons across different studies have been problematic since different groups use different performance thresholds, and do not account for key factors such as the number of trials or size of the BCI’s alphabet. We also discuss challenges inherent in establishing widely used terms, definitions, and measurement approaches to facilitate discussions and comparisons among different groups.
Article
Full-text available
This study introduces a novel brain-computer interface (BCI) based on an oddball paradigm using stimuli of facial images with loss of configural face information (e.g., inversion of face). To the best of our knowledge, till now the configural processing of human faces has not been applied to BCI but widely studied in cognitive neuroscience research. Our experiments confirm that the face-sensitive event-related potential (ERP) components N170 and vertex positive potential (VPP) have reflected early structural encoding of faces and can be modulated by the configural processing of faces. With the proposed novel paradigm, we investigate the effects of ERP components N170, VPP and P300 on target detection for BCI. An eight-class BCI platform is developed to analyze ERPs and evaluate the target detection performance using linear discriminant analysis without complicated feature extraction processing. The online classification accuracy of 88.7% and information transfer rate of 38.7 bits min(-1) using stimuli of inverted faces with only single trial suggest that the proposed paradigm based on the configural processing of faces is very promising for visual stimuli-driven BCI applications.
Article
Full-text available
In this paper we review classification algorithms used to design brain–computer interface (BCI) systems based on electroencephalography (EEG). We briefly present the commonly employed algorithms and describe their critical properties. Based on the literature, we compare them in terms of performance and provide guidelines to choose the suitable classification algorithm(s) for a specific BCI.
Article
Full-text available
We propose an online learning algorithm for training a logistic regression model on nonstationary classification problems. The nonstationarity is captured by modelling the weights in a logistic regression classifier as evolving according to a first order Markov process. The weights are updated using the extended Kalman filter formalism and nonstationarities are tracked by inferring a time-varying state noise variance parameter. We describe an algorithm for doing this based on maximising the evidence of updated predictions. The algorithm is illustrated on a number of synthetic problems. I. Introduction This paper proposes an online learning algorithm for training a logistic regression model on nonstationary classification problems. By nonstationary 1 we mean that the statistics of each class may vary with time or, equivalently from the classification perspective, that the optimal decision boundary changes with time. The simplest online algorithm for training a logistic regression m...
Article
Electroencephalographic (EEG) signals have been widely used in medical applications, yet the use of EEG signals as user identification systems for healthcare and Internet of Things (IoT) systems has only gained interests in the last few years. The advantages of EEG-based user identification systems lie in its dynamic property and uniqueness among different individuals. However, it is for this reason that manually designed features are not always adapted to the needs. Therefore, a novel approach based on 1D Convolutional Long Short-term Memory Neural Network (1D-Convolutional LSTM) for EEG-based user identification system is proposed in this paper. The performance of the proposed approach was validated with a public database consists of EEG data of 109 subjects. The experimental results showed that the proposed network has a very high averaged accuracy of 99.58%, when using only 16 channels of EEG signals, which outperforms the state-of-the-art EEG-based user identification methods. The combined use of CNNs and LSTMs in the proposed 1D-Convolutional LSTM can greatly improve the accuracy of user identification systems by utilizing the spatiotemporal features of the EEG signals with LSTM, and lowering cost of the systems by reducing the number of EEG electrodes used in the systems.
Conference Paper
While many still consider interactive movies an unrealistic idea, current delivery platforms like Netflix, commercial VR, and the proliferation of wearable sensors mean that adaptive and responsive entertainment experiences are an immediate reality. Our prior work demonstrated a brain-responsive movie that showed different views of scenes depending on levels of attention and meditation produced by a commercialized home-entertainment brain sensor. Based on lessons learned, this demonstration exhibits the new interactions designed for our new brain-controlled movie, The MOMENT, being released in 2018.
Article
Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
Article
The support-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data. High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
Article
In recent years, various applications of brain computer interfaces (BCIs) have been studied. In this paper, we present a hybrid BCI combining P300 and motor imagery to operate an explorer. Our system is mainly composed of a BCI mouse, a BCI speller and an explorer. Through this system, the user can access his computer and manipulate (open, close, copy, paste, delete) files such as documents, pictures, music, movies and so on. The system has been tested with 5 subjects, and the experimental results show that the explorer can be successfully operated according to subjects’ intentions.
Article
Currently, the event-related potential (ERP)-based spelling device, often referred to as P300-Speller, is the most commonly used brain-computer interface (BCI) for enhancing communication of patients with impaired speech or motor function. Among numerous improvements, a most central feature has received little attention, namely optimizing the stimulus used for eliciting ERPs. Therefore we compared P300-Speller performance with the standard stimulus (flashing characters) against performance with stimuli known for eliciting particularly strong ERPs due to their psychological salience, i.e. flashing familiar faces transparently superimposed on characters. Our results not only indicate remarkably increased ERPs in response to familiar faces but also improved P300-Speller performance due to a significant reduction of stimulus sequences needed for correct character classification. These findings demonstrate a promising new approach for improving the speed and thus fluency of BCI-enhanced communication with the widely used P300-Speller.
Article
The individual properties of visual objects, like form or color, are represented in different areas in our visual cortex. In order to perceive one coherent object, its features have to be bound together. This was found to be achieved in cat and monkey brains by temporal correlation of the firing rates of neurons which code the same object. This firing rate is predominantly observed in the gamma frequency range (approx. 30–80 Hz, mainly around 40 Hz). In addition, it has been shown in humans that stimuli which flicker at gamma frequencies are processed faster by our brains than when they flicker at different frequencies. These effects could be due to neural oscillators, which preferably oscillate at certain frequencies, so-called resonance frequencies. It is also known that neurons in visual cortex respond to flickering stimuli at the frequency of the flickering light. If neural oscillators exist with resonance frequencies, they should respond more strongly to stimulation with their resonance frequency. We performed an experiment, where ten human subjects were presented flickering light at frequencies from 1 to 100 Hz in 1-Hz steps. The event-related potentials exhibited steady-state oscillations at all frequencies up to at least 90 Hz. Interestingly, the steady-state potentials exhibited clear resonance phenomena around 10, 20, 40 and 80 Hz. This could be a potential neural basis for gamma oscillations in binding experiments. The pattern of results resembles that of multiunit activity and local field potentials in cat visual cortex.
This paper describes the development and testing of a system whereby one can communicate through a computer by using the P300 component of the event-related brain potential (ERP). Such a system may be used as a communication aid by individuals who cannot use any motor system for communication (e.g., 'locked-in' patients). The 26 letters of the alphabet, together with several other symbols and commands, are displayed on a computer screen which serves as the keyboard or prosthetic device. The subject focuses attention successively on the characters he wishes to communicate. The computer detects the chosen character on-line and in real time. This detection is achieved by repeatedly flashing rows and columns of the matrix. When the elements containing the chosen character are flashed, a P300 is elicited, and it is this P300 that is detected by the computer. We report an analysis of the operating characteristics of the system when used with normal volunteers, who took part in 2 experimental sessions. In the first session (the pilot study/training session) subjects attempted to spell a word and convey it to a voice synthesizer for production. In the second session (the analysis of the operating characteristics of the system) subjects were required simply to attend to individual letters of a word for a specific number of trials while data were recorded for off-line analysis. The analyses suggest that this communication channel can be operated accurately at the rate of 0.20 bits/sec. In other words, under the conditions we used, subjects can communicate 12.0 bits, or 2.3 characters, per min.
Article
An event-related brain potential (ERP) study investigated how different processing stages involved in face identification are reflected by ERP modulations, and how stimulus repetitions and attentional set influence such effects. ERPs were recorded in response to photographs of familiar faces, unfamiliar faces, and houses. In Part I, participants had to detect infrequently presented targets (hands), in Part II, attention was either directed towards or away from the pictorial stimuli. The face-specific N170 component elicited maximally at lateral temporal electrodes was not affected by face familiarity. When compared with unfamiliar faces, familiar faces elicited an enhanced negativity between 300 and 500 ms ('N400f') which was followed by an enhanced positivity beyond 500 ms post-stimulus ('P600f'). In contrast to the 'classical' N400, these effects were parietocentrally distributed. They were attenuated, but still reliable, for repeated presentations of familiar faces. When attention was directed to another demanding task, no 'N400f' was elicited, but the 'P600f' effect remained to be present. While the N170 reflects the pre-categorical structural encoding of faces, the 'N400f' and 'P600f' are likely to indicate subsequent processes involved in face recognition. Impaired structural encoding can result in the disruption of face identification. This is illustrated by a neuropsychological case study, demonstrating the absence of the N170 and later ERP indicators of face recognition in a prosopagnosic patient.
Article
To test a training procedure designed to enable severely paralyzed patients to communicate by means of self-regulation of slow cortical potentials. Application of the Thought Translation Device to evaluate the procedure in patients with late-stage amyotrophic lateral sclerosis (ALS). Training sessions in the patients' homes. Two male patients with late-stage ALS. Patients learned voluntary control of their slow cortical potentials by means of an interface between the brain and a computer. Training was based on visual feedback of slow cortical potentials shifts and operant learning principles. The learning process was divided into small steps of increasing difficulty. Accuracy of self-control of slow cortical potentials (percentage of correct responses). Learning progress calculated as a function of training session. Within 3 to 8 weeks, both patients learned to self-regulate their slow cortical potentials and to use this skill to select letters or words in the Language Support Program. This training schedule is the first to enable severely paralyzed patients to communicate without any voluntary muscle control by using self-regulation of an electroencephalogram potential only. The protocol could be a model for training patients in other brain-computer interface techniques.
Article
Brain-computer interface (BCI) systems support communication through direct measures of neural activity without muscle activity. BCIs may provide the best and sometimes the only communication option for users disabled by the most severe neuromuscular disorders and may eventually become useful to less severely disabled and/or healthy individuals across a wide range of applications. This review discusses the structure and functions of BCI systems, clarifies terminology and addresses practical applications. Progress and opportunities in the field are also identified and explicated.
PyGame - Python Game Development (Version 1.9
  • P Shinners
Information Systems and Neuroscience. Lecture notes in Information Systems and Organization
  • A B Randolph
  • B Warren
  • F D Davis
  • R Riedl
  • Vom Brocke
A survey of the dummy face and human face stimuli used in BCI paradigm
  • L Chen
  • J Jin
  • Y Zhang
  • X Wang
  • A Cichocki