![User visualizing his brain activity using Teegi [Frey et al., 2014].](https://www.researchgate.net/profile/Lea-Pillette/publication/338964762/figure/fig5/AS:853596090417152@1580524575676/User-visualizing-his-brain-activity-using-Teegi-Frey-et-al-2014_Q320.jpg)
Content uploaded by Léa Pillette
Author content
All content in this area was uploaded by Léa Pillette on Feb 05, 2020
Content may be subject to copyright.
A preview of the PDF is not available
Redefining and Adapting Feedback for Mental-Imagery based Brain-Computer Interface User Training to the Learners’ Traits and States
Abstract and Figures
Mental-Imagery based Brain-Computer Interfaces (MI-BCIs) present new opportunities to interact with digital technologies, such as neuroprostheses or videogames, only by performing mental imagery tasks, such as imagining an object rotating. The recognition of the command for the system is based on the analysis of the brain activity of the user. The users must learn to produce brain activity patterns that are recognizable by the system in order to control BCIs. However, current training protocols do not enable 10 to 30% of persons to acquire the skills required to use BCIs. The lack of robustness of BCIs limit the development of the technology outside of research laboratories. This thesis aims at investigating how the feedback provided throughout the training can be improved and adapted to the traits and states of the users. First, we investigate the role that feedback is currently given in MI-BCI applications and training protocols. We also analyse the theories and experimental contributions discussing its role and usefulness. Then, we review the different feedback that have been used to train MI-BCI users. We focus on three main characteristics of feedback, i.e., its content, its modality of presentation and finally its timing. For each of these characteristics, we reviewed the literature to assess which types of feedback have been tested and what is their impact on the training. We also analysed which traits or states of the learners were shown to influence BCI training outcome. Based on these reviews of the literature, we hypothesised that different characteristics of feedback could be leveraged to improve the training of the learners depending on either traits or states. We reported the results of our experimental contributions for each of the characteristics of feedback. Finally, we presented different recommendations and challenges regarding each characteristic of feedback. Potential solutions were proposed to meet these recommendations in the future.
Figures - uploaded by Léa Pillette
Author content
All figure content in this area was uploaded by Léa Pillette
Content may be subject to copyright.
... Pourtant, c'est au cours des sessions d'entraînement [35] que les utilisateurs vont progressivement parvenir à la compréhension du fonctionnement des BCI et acquérir les compétences nécessaires pour produire des motifs EEG que la BCI pourra reconnaître. Trouvant leur inspiration dans d'autres domaines comme les sciences de l'éducation [19], plusieurs avancées ont été testées, notamment en ce qui concerne le feedback [27] ou les exercices d'entraînement. ...
... Le feedback correspond à l'information qui est fournie aux utilisateurs pour apprendre à exécuter une tâche déterminée, c'est un élément-clé des BCI [19]. Son efficacité varie toutefois en fonction du profil des utilisateurs (par exemple, leurs capacités attentionnelles) ( [16] [17] et [27]). Le feedback doit donc être soigneusement conçu [37]. ...
... Le feedback doit donc être soigneusement conçu [37]. Il est caractérisé par son contenu (l'information fournie), par la modalité de sa présentation (façon dont l'information est fournie), ainsi que par sa temporalité (le moment où l'information est fournie) [27]. ...
... One of the main characteristics of the feedback is its modality of presentation, which represents how the information that it conveys is presented [13]. Feedback is provided through external sources or displays, for example, visual, auditory or haptic displays. ...
... Somatosensory feedback, through the use of vibrotactile stimuli, functional electrical stimuli (FES), orthosis/exoskeleton or vibrations on the muscles and tendons, was used for BCI user training [13]. Initial research on somatosensory feedback focused on tactile feedback through the use of vibrotactile motors. ...
... Such influence could be, at least in part, due to a differential impact of feedback on learning outcomes, that is, BCI performances and user experience, depending on the participants' profile [13]. For instance, we found that a social presence and emotional feedback provided using a learning companion had a differential impact on the participants' performances and reported efficiency/effectiveness felt during BCI training, that is, a measure of the user experience, depending on their autonomy [32]. ...
By performing motor-imagery tasks, for example, imagining hand movements, Motor-Imagery based Brain-Computer Interfaces (MI-BCIs) users can control digital technologies, for example, neuroprosthesis, using their brain activity only. MI-BCI users need to train, usually using a unimodal visual feedback, to produce brain activity patterns that are recognizable by the system. The literature indicates that multimodal vibrotactile and visual feedback is more effective than unimodal visual feedback, at least for short term training. However, the multi-session influence of such multimodal feedback on MI-BCI user training remained unknown, so did the influence of the order of presentation of the feedback modalities. In our experiment, 16 participants trained to control a MI-BCI during five sessions with a realistic visual feedback and five others with both a realistic visual feedback and a vibrotactile one. training benefits from a multimodal feedback, in terms of performances and self-reported mindfulness. There is also a significant influence of the order presentation of the modality. Participants who started training with a visual feedback had higher performances than those who started training with a multimodal feedback. We recommend taking into account the order of presentation for future experiments assessing the influence of several modalities of feedback.
... It is a fundamental component of the MI-BCI training [25]. Several research were led in order to improve the feedback [37], for instance by using more realistic cues [34,47]. ...
... Users could need specific feedback characteristics depending on their profile [37]. For instance, previous results indicate that "tensed" and "non-autonomous" people (based on the dimensions of the 16PF5 psychometric questionnaire [7]) are disadvantaged when controlling BCIs [16]. ...
... As stated in the introduction, the personality and the cognitive profile of participants and experimenters can respectively influence BCI performances and the experimenter bias [37]. Therefore, we assessed the personality and the cognitive profile of both the participants and the experimenters. ...
Context: Motor Imagery based Brain-Computer Interfaces (MI-BCIs) enable their users to interact with digital technologies, e.g., neuroprosthesis, by performing motor imagery tasks only, e.g., imagining hand movements, while their brain activity is recorded. To control MI-BCIs, users must train to control their brain activity. During such training, experimenters have a fundamental role, e.g., they motivate participants. However, their influence had never been formally assessed for MI-BCI user training. In other fields, e.g., social psychology, experimenters gender was found to influence experimental outcomes, e.g., behavioural or neurophysiological measures. Objective: Our aim was to evaluate if the experimenters gender influenced MI-BCI user training outcomes, i.e., performances and user-experience. Methods: We performed an experiment involving 6 experimenters (3 women) each training 5 women and 5 men (60 participants) to perform right versus left hand MI-BCI tasks over one session. We then studied the training outcomes, i.e., MI-BCI performances and user-experience, according to the experimenters’ and subjects’ gender. Results: A significant interaction between experimenters and participants’ gender was found on the evolution of trial-wise performances. Another interaction was found between participants tension and experimenters gender on the average performances. Conclusion: Experimenters gender could influence MI-BCI performances depending on participants gender and tension. Significance: Experimenters influence on MI-BCI user training outcomes should be better controlled, assessed and reported to further benefit from it while preventing any bias.
... Feedback provides an information to the learners regarding aspects of their performances or their understanding of the task/skills to learn [57]. Depending on the theory on the underlying mechanisms of MT-BCI process which is considered, feedback aims at consciously or unconsciously enabling the learners to produce a brain activity which is recognizable by the computer [273]. It is a fundamental component of MT-BCI protocols [101,290]. ...
... While it is acknowledged that feedback can improve the learning outcome, its effects are variable and can even be detrimental [92]. These variations in the efficiency of feedback have notably been associated with its different characteristics [136,273]. Based on our study of the literature, we argue that feedback can be defined using three main characteristics: (1) its content, i.e. which information it conveys (2) its modality of presentation, i.e. how this information is provided and (3) its timing, when and how frequently this information is provided [273]. ...
... These variations in the efficiency of feedback have notably been associated with its different characteristics [136,273]. Based on our study of the literature, we argue that feedback can be defined using three main characteristics: (1) its content, i.e. which information it conveys (2) its modality of presentation, i.e. how this information is provided and (3) its timing, when and how frequently this information is provided [273]. ...
Mental-Tasks based Brain-Computer Interfaces (MT-BCIs) allow their users to interact with an external device solely by using brain signals produced through mental tasks. While MT-BCIs are promising for many applications, they are still barely used outside laboratories due to their lack of reliability. MT-BCIs require their users to develop the ability to self-regulate specific brain signals. However, the human learning process to control a BCI is still relatively poorly understood and how to optimally train this ability is currently under investigation. Despite their promises and achievements, traditional training programs have been shown to be sub-optimal and could be further improved. In order to optimize user training and improve BCI performance, human factors should be taken into account. An interdisciplinary approach should be adopted to provide learners with appropriate and/or adaptive training. In this article, we provide an overview of existing methods for MT-BCI user training - notably in terms of environment, instructions, feedback and exercises. We present a categorization and taxonomy of these training approaches, provide guidelines on how to choose the best methods and identify open challenges and perspectives to further improve MT-BCI user training.
... Most of all, these devices vary depending on the advice they provide. Based on the literature on feedback [1], we argue that such advice can be described using three main characteristics (see Figure 1). First, the modality of presentation of the advice represents which sensory perception it relies on, e.g., visual, auditory and tactile modalities. ...
... Based on recent reviews on wayfinding [97], [98], [99], many of the recommendations that we make can be generalized to the design of devices for the general population. Among the recommendations that we make several can be found in the literature on neurotypical people: (1) landmarks and wayfinding information should be salient and colorful [98], [99], (2) standardized spatial cues and specifically symbols, pictograms and photographs are recommended [97], [99] (3) the use of an egocentric map limits the risk of mistakes [97]. In the following paragraphs, we go through the different main recommendations for future research we made and elaborate on if they can be generalized for future research on the global population. ...
Technological developments provide solutions to alleviate the tremendous impact on the health and autonomy due to the impact of dementia on navigation abilities. We systematically reviewed the literature on devices tested to provide assistance to people with dementia during indoor, outdoor and virtual navigation (PROSPERO ID number: 215585). Medline and Scopus databases were searched from inception. Our aim was to summarize the results from the literature to guide future developments. Twenty-three articles were included in our study. Three types of information were extracted from these studies. First, the types of navigation advice the devices provided were assessed through: (i) the sensorial modality of presentation, e.g., visual and tactile stimuli, (ii) the navigation content, e.g., landmarks, and (iii) the timing of presentation, e.g., systematically at intersections. Second, we analyzed the technology that the devices were based on, e.g., smartphone. Third, the experimental methodology used to assess the devices and the navigation outcome was evaluated. We report and discuss the results from the literature based on these three main characteristics. Finally, based on these considerations, recommendations are drawn, challenges are identified and potential solutions are suggested. Augmented reality-based devices, intelligent tutoring systems and social support should particularly further be explored.
... Di erent types of feedback can be used to close the BCI loop One of the main goals of neurofeedback is to train users to adapt to the BCI task by providing specific cues to task-related brain activity. In addition to the content of the feedback, the way in which the feedback is presented also has a major impact on its effect (Pillette, 2019). The following feedback modalities have been explored (Table 2). ...
The brain-computer interface (BCI)-mediated rehabilitation is emerging as a solution to restore motor skills in paretic patients after stroke. In the human brain, cortical motor neurons not only fire when actions are carried out but are also activated in a wired manner through many cognitive processes related to movement such as imagining, perceiving, and observing the actions. Moreover, the recruitment of motor cortexes can usually be regulated by environmental conditions, forming a closed-loop through neurofeedback. However, this cognitive-motor control loop is often interrupted by the impairment of stroke. The requirement to bridge the stroke-induced gap in the motor control loop is promoting the evolution of the BCI-based motor rehabilitation system and, notably posing many challenges regarding the disease-specific process of post stroke motor function recovery. This review aimed to map the current literature surrounding the new progress in BCI-mediated post stroke motor function recovery involved with cognitive aspect, particularly in how it refired and rewired the neural circuit of motor control through motor learning along with the BCI-centric closed-loop.
... The fourth study aimed at developing a first comprehensive understanding of the different attentional states described in the model of van Zomeren and Brouwer using EEG data [70,71]. The term "Attention" encompasses several different attentional states. ...
Research on brain-computer interfaces (BCIs) has become more democratic in recent decades, and experiments using electroencephalography (EEG)-based BCIs has dramatically increased. The variety of protocol designs and the growing interest in physiological computing require parallel improvements in processing and classification of both EEG signals and bio signals, such as electrodermal activity (EDA), heart rate (HR) or breathing. If some EEG-based analysis tools are already available for online BCIs with a number of online BCI platforms (e.g., BCI2000 or OpenViBE), it remains crucial to perform offline analyses in order to design, select, tune, validate and test algorithms before using them online. Moreover, studying and comparing those algorithms usually requires expertise in programming, signal processing and machine learning, whereas numerous BCI researchers come from other backgrounds with limited or no training in such skills. Finally, existing BCI toolboxes are focused on EEG and other brain signals but usually do not include processing tools for other bio signals. Therefore, in this paper, we describe BioPyC, a free, open-source and easy-to-use Python platform for offline EEG and biosignal processing and classification. Based on an intuitive and well-guided graphical interface, four main modules allow the user to follow the standard steps of the BCI process without any programming skills: (1) reading different neurophysiological signal data formats, (2) filtering and representing EEG and bio signals, (3) classifying them, and (4) visualizing and performing statistical tests on the results. We illustrate BioPyC use on four studies, namely classifying mental tasks, the cognitive workload, emotions and attention states from EEG signals.
Mental-Imagery based Brain-Computer Interfaces (MI-BCI) present new opportunities to interact with digital technologies, such as wheelchairs or neuroprostheses, only by performing mental imagery tasks (e.g., imagining an object rotating or imagining hand movements). MI-BCIs can also be used for several applications such as communication or post-stroke rehabilitation. Though, their lack of reliability remains a barrier to a larger scale development of the technology. For example, one task between two is recognized on average 75% of the time. It has been shown that users are more likely to struggle using MI-BCIs if they are non-autonomous or tensed. This might, at least in part, result from a lack of social presence and emotional support, which have yet very little been tested in MI-BCI, despite recommendations from the educational literature. One way to provide such social and emotional context is by using a learning companion. Therefore, we designed, implemented and evaluated the first learning companion dedicated to the improvement of MI-BCI user training. We called this companion PEANUT for Personalized Emotional Agent for Neurotechnology User Training. PEANUT provided social presence and emotional support, depending on the performance and progress of the user, through interventions combining both pronounced sentences and facial expressions. It was designed based on the literature, data analyses and user-studies. We notably conducted various online user surveys to identify the desired characteristics of our learning companion in terms of appearance and supporting speech content. From the results of these surveys we notably deduced which should be the characteristics (personal/non-personal, exclamatory/declarative) of the sentences to be used depending on the performance and progression of a learner. We also found that eyebrows could increase expressiveness of cartoon-like faces. Then, once this companion was implemented, we evaluated it during real online MI-BCI use. We found that non-autonomous people, i.e., who are more inclined to work in a group, that are usually disadvantaged when using MI-BCI were advantaged compared to autonomous people when PEANUT was present with an increase of 3.9% of peak performances. Furthermore, in terms of user experience, PEANUT seems to have improved how people felt about their ability to learn and memorize how to use an MI-BCI by 7.4%, which is a dimension of the user experience we assessed.
The use of robotic devices to provide active motor support and sensory feedback of ongoing motor intention, by means of a Brain Computer Interface (BCI), has received growing support by recent literature, with particular focus on neurorehabilitation therapies. At the same time, performance in the use of the BCI has become a more critical factor, since it directly influences congruency and consistency of the provided sensory feedback. As motor imagery is the mental simulation of a given movement without depending on residual function, training of patients in the use of motor imagery BCI can be extended beyond each rehabilitation session, and practiced by using simpler devices than rehabilitation robots available in the hospital. In this work, we investigated the use of haptic stimulation provided by vibrating electromagnetic motors to enhance BCI system training. The BCI is based on motor imagery of hand grasping and designed to operate a hand exoskeleton. We investigated whether haptic stimulation at fingerpads proves to be more effective than stimulation at wrist, already experimented in literature, due to the higher density of mechano-receptors. Our results did not show significant differences between the two body locations in BCI performance, yet a wider and more stable event-relateddesynchronization appeared for the finger-located stimulation. Future investigations will put in relation training with haptic feedback at fingerpads with BCI performance using the handexoskeleton, in grasping tasks that naturally involve haptic feedback at fingerpads.
The sense of ownership, the feeling that our body belongs to ourselves, relies on multiple sources of sensory information. Among these sources, the contribution of visuomotor information is still debated. We tested the effect of active control in the sense of ownership in the moving Virtual Hand Illusion. Participants reported sense of ownership and sense of agency over a virtual arm in which we manipulated the morphological congruence of the hand and the visuomotor information. We found that congruent active control enhanced and maintained the reported sense of ownership over a hand that appeared detached from the body, but not in a morphological congruent limb. Also, incongruent active control, achieved by adding noise to the trajectory of the movement, decreased both reported sense of agency and ownership. Overall, our results are consistent with a framework in which active control acts as evidence for eliciting a sense of ownership.
Brain-computer interfaces (BCI) translate brain activity into control signals or commands for a device. Motor imagery of the limbs allows for modulating the sensorimotor rhythms (SMR), but there are up to 30% of the participants for whom electroencephalography (EEG) based SMR-BCI cannot detect any imagery-related changes. Individual variables, such as ability to concentrate on a task and error duration in a two-hand visuomotor coordination (VMC) task have been previously found to predict accuracy in an SMR-BCI. A first study attempted to substantiate those predictors by introducing a 30 min relaxation or VMC training period prior to an SMR-BCI session, but performance did not increase when compared to a control group. As the predictor training may have been too short, we applied 4 such training sessions on consecutive days in the current study. In a pre-post design, SMR-BCI accuracy of n = 39 participants increased from session 1 before to session 2 after the predictor training. While the manipulation of the predictor variables was successful, there was no effect on SMR-BCI performance. BCI accuracy correlated positively with the neurophysiological SMR predictor identified by Blankertz et al. [3], consolidating its predictive value, and with the state mindfulness scale. No other psychological predictor could be identified or replicated. Further studies should therefore focus more on delineating (partially) replicated or potential predictors such as VMC or mindfulness to help refining a sound model to predict SMR-BCI accuracy.
We have developed a brain-machine interface (BMI) rehabilitation system for patients with stroke and motor paralysis, which provides proprioceptive feedback upon successful generation of motor-imagery (MI)-induced event-related desynchronization (ERD) and a decrease in mu band (8–13 Hz) activity derived from hand motor imagery. This system consists of an electroencephalogram (EEG) amplifier operated using the MATLAB Simulink software; a pneumatic robotic exoskeleton to provide proprioceptive feedback to the paralyzed hand; and a tablet computer placed over the paralyzed hand to display a hand-action movie to facilitate ERD generation. The EEG amplifier was connected and synchronized via the exoskeleton and tablet computer with an Arduino microcomputer. Nine patients in the subacute stage of recovery after stroke participated in a neurofeedback training experiment, which employed the aforementioned system. During the 4 weeks of this study, the participants received 2 weeks of BMI-based or control interventions in a random and counterbalanced order, in addition to their daily conventional physiotherapy. The control intervention consisted of the same MI training as the BMI-based intervention, but the exoskeleton always provided proprioceptive feedback regardless of the ERD strength. The ERD strength in the affected hemisphere showed a desirable increase with a significant improvement of finger joint spasticity, only after the DMB-based intervention period, and not after the control intervention period. The proposed neurofeedback training can help patients with stroke and movement disorders, because increased ERD strength may lead to recovery of motor function.
Objective:
BCI decoding accuracy plays a crucial role in practical applications. With accurate feedback, BCI-based therapy determines beneficial neural plasticity in stroke patients. In this study, we aimed at improving sensorimotor rhythm (SMR)-based BCI performance by integrating motor tasks with tactile stimulation.
Methods:
Eleven stroke patients were recruited for three experimental conditions, i.e., motor attempt (MA) condition, tactile stimulation (TS) condition, and tactile stimulation-assisted motor attempt (TS-MA) condition. Tactile stimulation was delivered to the paretic hand wrist during both task and idle states using a DC vibrator.
Results:
We observed that the TS-MA condition achieved greater motor-related cortical activation (MRCA) in alpha-beta band when compared with both TS and MA conditions. Consequently, online BCI decoding accuracies between task and idle states were significantly improved from 74.5% in the MA condition to 85.1% in the TS-MA condition (p < 0.001), whereas the accuracy in the TS condition was 54.6% (approaching to the chance level of 50%).
Conclusion:
This finding demonstrates that sensory afferent from peripheral nerves benefits the neural process of sensorimotor cortex in stroke patients. With appropriate sensory stimulation, MRCA is enhanced and corresponding brain patterns are more discriminative.
Significance:
This novel SMR-BCI paradigm shows great promise to facilitate the practical application of BCI-based stroke rehabilitation.
Despite an increasing focus on the neural basis of human decision making in neuroscience, relatively little attention has been paid to decision making in social settings. Moreover, although human social decision making has been explored in a social psychology context, few neural explanations for the observed findings have been considered. To bridge this gap and improve models of human social decision making, we investigated whether acquiring a good reputation, which is an important incentive in human social behaviors, activates the same reward circuitry as monetary rewards. In total, 19 subjects participated in functional magnetic resonance imaging (fMRI) experiments involving monetary and social rewards. The acquisition of one's good reputation robustly activated reward-related brain areas, notably the striatum, and these overlapped with the areas activated by monetary rewards. Our findings support the idea of a "common neural currency" for rewards and represent an important first step toward a neural explanation for complex human social behaviors.
Real-time functional magnetic resonance imaging (fMRI)-based neurofeedback represents the latest applied behavioural neuroscience methodology developed to train participants in the self-regulation of brain regions or networks. However, as with previous biofeedback approaches which rely on electroencephalography (EEG) or related approaches such as brain-machine interface technology (BCI), individual success rates vary significantly, and some participants never learn to control their brain responses at all. Given that these approaches are often being developed for eventual use in a clinical setting (albeit there is also significant interest in using NF for neuro-enhancement in typical populations), this represents a significant hurdle which requires more research. Here we present the findings of a systematic review which focused on how psychological variables contribute to learning outcomes in fMRI-based neurofeedback. However, as this is a relatively new methodology, we also considered findings from EEG-based neurofeedback and BCI. 271 papers were found and screened through PsycINFO, psycARTICLES, Psychological and Behavioural Sciences Collection, ISI Web of Science and Medline and 21 were found to contribute towards the aim of this survey. Several main categories emerged: Attentional variables appear to be of importance to both performance and learning, motivational factors and mood have been implicated as moderate predictors of success, while personality factors have mixed findings. We conclude that future research will need to systematically manipulate psychological variables such as motivation or mood, and to define clear thresholds for a successful neurofeedback effect. Non-responders need to be targeted for interventions and tested with different neurofeedback setups to understand whether their non-response is specific or general. Also, there is a need for qualitative evidence to understand how psychological variables influence participants throughout their training. This will help us to understand the subtleties of psychological effects over time. This research will allow interventions to be developed for non-responders and better selection procedures in future to improve the efficacy of neurofeedback.