Rajesh P. N. Rao

• University of Washington

A key challenge in analyzing neuroscience datasets is the profound variability they exhibit across sessions, animals, and data modalities--i.e., heterogeneity. Several recent studies have demonstrated performance gains from pretraining neural foundation models on multi-session datasets, seemingly overcoming this challenge. However, these studies ty...

This study presents a systematic machine-learning approach for classifying acute pain from raw electrophysiological signals. We address binary and ternary classification tasks, leveraging Power-In-Band (PIB) and signal coherence as distinguishing features. Our method evaluates the effectiveness of traditional machine learning algorithms on a manual...

Humans frequently interact with agents whose intentions can fluctuate between competition and cooperation over time. It is unclear how the brain adapts to fluctuating intentions of others when the nature of the interactions (to cooperate or compete) is not explicitly and truthfully signaled. Here, we use model-based fMRI and a task in which partici...

We introduce dynamic predictive coding, a hierarchical model of spatiotemporal prediction and sequence learning in the neocortex. The model assumes that higher cortical levels modulate the temporal dynamics of lower levels, correcting their predictions of dynamics using prediction errors. As a result, lower levels form representations that encode s...

Touch sensation offers complex and innate biological feedback that underlies dexterous interactions with our environments, making the restoration of touch an important component of functional brain computer interface (BCI). To advance synthetic touch research, I developed a virtual reality (VR) platform for use with human patients receiving intracr...

There is growing interest in predictive coding as a model of how the brain learns through predictions and prediction errors. Predictive coding models have traditionally focused on sensory coding and perception. Here we introduce active predictive coding (APC) as a unifying model for perception, action, and cognition. The APC model addresses importa...

The Cerebral Cortex and Thalamus is guided by two central and related tenets, the thalamus plays an ongoing and essential role in cortical functioning, and the cortex is essential for thalamic functioning. Accordingly, neither the cortex nor the thalamus can be understood in any meaningful way in the absence of the other. With chapters written by m...

Objective. A major challenge in designing closed-loop brain-computer interfaces is finding optimal stimulation patterns as a function of ongoing neural activity for different subjects and different objectives. Traditional approaches, such as those currently used for deep brain stimulation, have largely followed a manual trial-and-error strategy to...

p> Virtual reality (VR) offers a robust platform for human behavioral neuroscience, granting unprecedented experimental control over every aspect of an immersive and interactive visual environment. VR experiments have already integrated non-invasive neural recording modalities such as EEG and functional MRI to explore the neural correlates of human...

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a compleme...

p> Virtual reality (VR) offers a robust platform for human behavioral neuroscience, granting unprecedented experimental control over every aspect of an immersive and interactive visual environment. VR experiments have already integrated non-invasive neural recording modalities such as EEG and functional MRI to explore the neural correlates of human...

Predictive coding is a unifying framework for understanding perception, action, and neocortical organization. In predictive coding, different areas of the neocortex implement a hierarchical generative model of the world that is learned from sensory inputs. Cortical circuits are hypothesized to perform Bayesian inference based on this generative mod...

Predictive coding has emerged as a prominent model of how the brain learns through predictions, anticipating the importance accorded to predictive learning in recent AI architectures such as transformers. Here we propose a new framework for predictive coding called active predictive coding which can learn hierarchical world models and solve two rad...

Objective: A major challenge in closed-loop brain-computer interfaces (BCIs) is finding optimal stimulation patterns as a function of ongoing neural activity for different subjects and objectives. Traditional approaches, such as those currently used for deep brain stimulation, have largely followed a trial- and-error strategy to search for effectiv...

Objective. Recent advances in neural decoding have accelerated the development of brain–computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible...

Virtual reality (VR) offers a robust platform for human behavioral neuroscience, granting unprecedented experimental control over every aspect of an immersive and interactive visual environment. VR experiments have already integrated non-invasive neural recording modalities such as EEG and functional MRI to explore the neural correlates of human be...

Inspired by Gibson's notion of object affordances in human vision, we ask the question: how can an agent learn to predict an entire action policy for a novel object or environment given only a single glimpse? To tackle this problem, we introduce the concept of Universal Policy Functions (UPFs) which are state-to-action mappings that generalize not...

We introduce dynamic predictive coding, a hierarchical model of spatiotemporal prediction and sequence learning in the neocortex. The model assumes that higher cortical levels modulate the temporal dynamics of lower levels, correcting their predictions of dynamics using prediction errors. As a result, lower levels form representations that encode s...

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available...

p> Virtual reality (VR) offers a robust platform for human behavioral neuroscience, granting unprecedented experimental control over every aspect of an immersive and interactive visual environment. VR experiments have already integrated non-invasive neural recording modalities such as EEG and functional MRI to explore the neural correlates of human...

p>Virtual reality tasks were well-tolerated in two inpatient intracranial recording subjects who completed over 94 minutes of cumulative time in VR. The HMD was found to introduce phase-locked line noise and intermittent broadband high frequency noise. Both seemed addressable by common average and common median referencing. This work represents the...

We introduce Active Predictive Coding Networks (APCNs), a new class of neural networks that solve a major problem posed by Hinton and others in the fields of artificial intelligence and brain modeling: how can neural networks learn intrinsic reference frames for objects and parse visual scenes into part-whole hierarchies by dynamically allocating n...

Predictive coding is a unifying framework for understanding perception, action and neocortical organization. In predictive coding, different areas of the neocortex implement a hierarchical generative model of the world that is learned from sensory inputs. Cortical circuits are hypothesized to perform Bayesian inference based on this generative mode...

Humans frequently interact with other agents whose intentions can fluctuate over time between competitive and cooperative strategies. How does the brain decide whether the others’ intentions are to cooperate or compete when the nature of the interactions is not explicitly signaled? We used model-based fMRI and a task in which participants thought t...

Research with human intracranial electrodes has traditionally been constrained by the limitations of the inpatient clinical setting. Immersive virtual reality (VR), however, can transcend setting and enable novel task design with precise control over visual and auditory stimuli. This control over visual and auditory feedback makes VR an exciting pl...

In perceptual decisions, subjects infer hidden states of the environment based on noisy sensory information. Here we show that both choice and its associated confidence are explained by a Bayesian framework based on partially observable Markov decision processes (POMDPs). We test our model on monkeys performing a direction-discrimination task with...

Tracking a turbulent plume to locate its source is a complex control problem because it requires multi-sensory integration and must be robust to intermittent odors, changing wind direction, and variable plume statistics. This task is routinely performed by flying insects, often over long distances, in pursuit of food or mates. Several aspects of th...

Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain...

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available...

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed "in the wild." Here, we leveraged recen...

Background: Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when...

In the ongoing COVID-19 pandemic, public health experts have produced guidelines to limit the spread of the coronavirus, but individuals do not always comply with experts’ recommendations. Here, we tested whether a specific psychological belief—identification with all humanity—predicts cooperation with public health guidelines as well as helpful be...

Objective. Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that ca...

The original predictive coding model of Rao & Ballard (1999) focused on spatial prediction to explain spatial receptive fields and contextual effects in the visual cortex. Here, we introduce a new dynamic predictive coding model that achieves spatiotemporal prediction of complex natural image sequences using time-varying transition matrices. We ove...

Direct cortical stimulation (DCS) has been used extensively as a tool in human neuroscience research and clinically in neurosurgery. Implanted electrocorticography electrodes can deliver electrical stimulation to the cortical surface, eliciting or inhibiting neural activity as an experimental manipulation, or to probe neural tissue function during...

Objective Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can...

Increasingly, neuroimaging researchers are exploring the use of real-time functional magnetic resonance imaging (rt-fMRI) as a way to access a participant’s ongoing brain function throughout a scan. This approach presents novel and exciting experimental applications ranging from monitoring data quality in real time, to delivering neurofeedback from...

We take a broader view of the purpose of writing and attempt to interpret the contents in a non-linguistic model to guide any ideas on what may be the content of writing in the Indus Script. We analyse the general characteristics of writing and then compare it with the insights provided by syntactic studies of the Indus script to get insights into...

Studying the neural correlates of sleep can lead to revelations in our understanding of sleep and its interplay with different neurological disorders. Sleep research relies on manual annotation of sleep stages based on rules developed for healthy adults. Automating sleep stage annotation can expedite sleep research and enable us to better understan...

A bstract Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed “in the wild.” Here, we lever...

Objective. Electrical stimulation of the human brain is commonly used for eliciting and inhibiting neural activity for clinical diagnostics, modifying abnormal neural circuit function for therapeutics, and interrogating cortical connectivity. However, recording electrical signals with concurrent stimulation results in dominant electrical artifacts...

Recent advances in brain recording technology and artificial intelligence are propelling a new paradigm in neuroscience beyond the traditional controlled experiment. Naturalistic neuroscience studies neural computations associated with spontaneous behaviors performed in unconstrained settings. Analyzing such unstructured data lacking a priori exper...

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Electrocorticographic brain computer interfaces (ECoG-BCIs) offer tremendous opportunities for restoring function in individuals suffering from neurological damage and for advancing basic neuroscience knowledge. ECoG electrodes are already commonly used clinically for monitoring epilepsy and have greater spatial specificity in recording neuronal ac...

Brain–computer interfaces (BCIs) benefit greatly from performance feedback, but current systems lack automatic, task-independent feedback. Cortical responses elicited from user error have the potential to serve as state-based feedback to BCI decoders. To gain a better understanding of local error potentials, we investigate responsive cortical power...

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three h...

Direct cortical stimulation (DCS) of primary somatosensory cortex (S1) could help restore sensation and provide task-relevant feedback in a neuroprosthesis. However, the psychophysics of S1 DCS is poorly studied, including any comparison to cutaneous haptic stimulation. We compare the response times to DCS of human hand somatosensory cortex through...

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three h...

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three h...

We introduce Graph-Structured Sum-Product Networks (GraphSPNs), a probabilistic approach to structured prediction for problems where dependencies between latent variables are expressed in terms of arbitrary, dynamic graphs. While many approaches to structured prediction place strict constraints on the interactions between inferred variables, many r...

We introduce Graph-Structured Sum-Product Networks (GraphSPNs), a probabilistic approach to structured prediction for problems where dependencies between latent variables are expressed in terms of arbitrary, dynamic graphs. While many approaches to structured prediction place strict constraints on the interactions between inferred variables, many r...

We present here a browser-based application for visualizing patterns of connectivity in 3D stacked data matrices with large numbers of pairwise relations. Visualizing a connectivity matrix, looking for trends and patterns, and dynamically manipulating these values is a challenge for scientists from diverse fields, including neuroscience and genomic...

Identification of intended movement type and movement phase of hand grasp shaping are critical features for the control of volitional neuroprosthetics. We demonstrate that neural dynamics during visually-guided imagined grasp shaping can encode intended movement. We apply Procrustes analysis and LASSO regression to achieve 72% accuracy (chance = 25...

The promise of robots assisting humans in everyday tasks has led to a variety of research questions and challenges in human-robot collaboration. Here, we address the question of whether and when a robot should take initiative during joint human-robot task execution. We designed a robotic system capable of autonomously performing table-top manipulat...

Can the human brain learn to interpret inputs from a virtual world delivered directly through brain stimulation? We answer this question by describing the first demonstration of humans playing a computer game utilizing only direct brain stimulation and no other sensory inputs. The demonstration also provides the first instance of artificial sensory...

We propose a new probabilistic framework that allows mobile robots to autonomously learn deep generative models of their environments that span multiple levels of abstraction. Unlike traditional approaches that attempt to integrate separately engineered components for low-level features, geometry, and semantic representations, our approach leverage...

A motor cortex-based brain-computer interface (BCI) creates a novel real world output directly from cortical activity. Use of a BCI has been demonstrated to be a learned skill that involves recruitment of neural populations that are directly linked to BCI control as well as those that are not. The nature of interactions between these populations, h...

Behavioral performance. (DOCX)

Individual behavioral results. (DOCX)

Response-locked STWC interactions. (DOCX)

Cue-locked STWC interactions. (DOCX)

Motivated by the growing evidence for Bayesian computation in the brain, we show how a two-layer recurrent network of Poisson neurons can perform both approximate Bayesian inference and learning for any hidden Markov model. The lower-layer sensory neurons receive noisy measurements of hidden world states. The higher-layer neurons infer a posterior...

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract dir...

Cortical stimulation through electrocorticographic (ECoG) electrodes is a potential method for providing sensory feedback in future prosthetic and rehabilitative applications. Here we evaluate human subjects' ability to continuously modulate their motor behavior based on feedback from direct surface stimulation of the somatosensory cortex. Subjects...

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract dir...

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsu...

Methods: This report builds upon the outcomes of a joint workshop between the US National Science Foundation (NSF) and the German Research Foundation (DFG) on New Perspectives in Neuroengineering and Neurotechnology convened in Arlington, VA, November 13-14, 2014. Results: The participants identified key technological challenges for recording an...

The link between object perception and neural activity in visual cortical areas is a problem of fundamental importance in neuroscience. Here we show that electrical potentials from the ventral temporal cortical surface in humans contain sufficient information for spontaneous and near-instantaneous identification of a subject's perceptual state. Ele...

Correct classifications, when the timing of events is pre-designated. Sorted by stimulus type (note that each number is out of a possible 150 correct). (PDF)

Power spectral analysis. (PDF)

Participant characteristics, and number of selected electrodes by r2<0.05 criteria (from first fold only). (PDF)

Choice of collision time does not inform classifier about timing of events. Number of false predictions as a function of the choice of maximum distance between predicted event times (Collision time), for classification using both ERP and ERBB. The monotonic decay form and lack of “dips” or “peaks” shows that the collision time chosen did not inform...

Decoupling the cortical spectrum. (PDF)

Errors for Spontaneous Predictions. (PDF)

Human visual processing is of such complexity that, despite decades of focused research, many basic questions remain unanswered. Although we know that the inferotemporal cortex is a key region in object recognition, we don’t fully understand its physiologic role in brain function, nor do we have the full set of tools to explore this question. Here...

An article published in Language (Sproat 2014a) questions our findings on the Indus script and Pictish symbols published in the journals Science (Rao et al. 2009a), PNAS (Rao et al. 2009b), IEEE Computer (Rao 2010), and the Proceedings of the Royal Society (Lee et al. 2010a,b). Sproat’s article does not accurately present our methods and findings,...

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategi...

A fundamental challenge in robotics today is building robots that can learn new skills by observing humans and imitating human actions. We propose a new Bayesian approach to robotic learning by imitation inspired by the developmental hypothesis that children use self-experience to bootstrap the process of intention recognition and goal-based imitat...

We present, to our knowledge, the first demonstration that a non-invasive brain-to-brain interface (BBI) can be used to allow one human to guess what is on the mind of another human through an interactive question-and-answering paradigm similar to the "20 Questions" game. As in previous non-invasive BBI studies in humans, our interface uses electro...