A framework for relating neural activity to freely moving behavior
Justin D. Foster, Student Member, IEEE, Paul Nuyujukian, Student Member, IEEE,
Oren Freifeld, Student Member, IEEE, Stephen I. Ryu, Michael J. Black*, Senior Member, IEEE,
Krishna V. Shenoy*, Senior Member, IEEE
Abstract—Two research communities, motor systems neu-
roscience and motor prosthetics, examine the relationship
between neural activity in the motor cortex and movement.
The former community aims to understand how the brain
controls and generates movement; the latter community focuses
on how to decode neural activity as control signals for a
prosthetic cursor or limb. Both have made progress toward
understanding the relationship between neural activity in the
motor cortex and behavior. However, these findings are tested
using animal models in an environment that constrains behavior
to simple, limited movements. These experiments show that, in
constrained settings, simple reaching motions can be decoded
from small populations of spiking neurons. It is unclear whether
these findings hold for more complex, full-body behaviors in
unconstrained settings. Here we present the results of freely-
moving behavioral experiments from a monkey with simultane-
ous intracortical recording. We investigated neural firing rates
while the monkey performed various tasks such as walking
on a treadmill, reaching for food, and sitting idly. We show
that even in such an unconstrained and varied context, neural
firing rates are well tuned to behavior, supporting findings of
basic neuroscience. Further, we demonstrate that the various
behavioral tasks can be reliably classified with over 95% ac-
curacy, illustrating the viability of decoding techniques despite
significant variation and environmental distractions associated
with unconstrained behavior. Such encouraging results hint at
potential utility of the freely-moving experimental paradigm.
A goal of motor systems neuroscience is to explain how
cortical areas involved in movement control behavior. Ex-
tensive studies over the past several decades in monkeys
*These authors contributed equally.
The work of J. D. Foster is supported by a Texas Instrument Stanford
Graduate Fellowship. The work of P. Nuyujukian is supported by a Stanford
NIH Medical Scientist Training Program grant. The work of M. J. Black is
supported by NIH-NINDS EUREKA (R01-NS066311). The work of K.
V. Shenoy is supported in part by a Burroughs Wellcome Fund Career
Award in the Biomedical Sciences, DARPA REPAIR (N66001-10-C-2010),
McKnight Foundation, Simbios, Weston Havens Foundation, NIH-NINDS
BRP (R01-NS064318), NIH-NINDS EUREKA (R01-NS066311), and an
NIH Director’s Pioneer Award (DP1-OD006409).
J. D. Foster is with the Department of Electrical Engineering, Stanford
University, Stanford, CA 94305 USA firstname.lastname@example.org
P. Nuyujukian is with Bioengineering and Stanford Medical School,
Stanford University, Stanford, CA 94305 USA email@example.com
O. Freifeld is with the Division of Applied Mathematics, Brown Univer-
sity, Providence, RI 02912 USA firstname.lastname@example.org
S. Ryu is with the Department of Neurosurgery, Palo Alto Medical
Foundation, Palo Alto, CA 94301 USA email@example.com
M. J. Black is with the Max Planck Institute for Intelligent Systems,
72076 T¨ ubingen, Germany and the Department of Computer Science, Brown
University, Providence, RI 02912 USA firstname.lastname@example.org
K. V. Shenoy is with the Departments of Electrical Engineering, Bio-
engineering, and Neurobiology, and the Neurosciences Program, Stanford
University, Stanford, CA, 94305 USA email@example.com
recorded synchronously with video streams while broadband neural activity
is recorded and transmitted wirelessly.
System overview. Unconstrained behavior of a monkey is
have developed many models of motor behavior , ,
, . These findings have fostered the development of
translational work in brain-machine interfaces (BMIs). Such
systems aim to decipher cortical activity into meaningful
control signals such as computer cursors or robotic limbs
, , , , , , . Both bodies of research
have led to many insights and show great promise, how-
ever a fundamental limitation is their applicability to less
constrained movements. It is unclear whether neuroscientific
findings and BMIs will generalize beyond the limited subset
of behaviors tested experimentally. Investigations into such
generalizations were hampered by the lack of experimental
tools and techniques, limiting research to the restrictive, but
highly controlled environment of neuroelectrophysiological
experimental rigs. Only in such setups could accurate mea-
surements of behavioral kinematics and neurophysiological
activity be taken. However, with the continued evolution
of wirelessly transmitting neural recording amplifiers and
computer vision technology, preliminary research with un-
constrained animal models may be possible , , ,
. In this study we aim to show that basic motor sys-
tems neuroscientific findings of neurally tuned behavior are
consistent in unconstrained behavior in one monkey. Further,
we show preliminary evidence that general types of behavior
can be differentiated and decoded quite accurately despite
the lack of rigid behavioral restrictions. Both findings are
important so that we may 1) verify the applicability of
in-rig results to broader domains of behavior and 2) have
confidence that BMIs may successfully translate to complex
use cases such as ambulatory patients.
34th Annual International Conference of the IEEE EMBS
San Diego, California USA, 28 August - 1 September, 2012
978-1-4577-1787-1/12/$26.00 ©2012 IEEE
of the wrist, elbow, and shoulder (contralateral to implant) are triangulated from video frames as the monkey moves through a the swing phase and b
the stance phase of walking, c reaches for food, d brings food to his mouth, and e drops his arm down. Simultaneously, broadband neural activity was
recorded from PMd. f Neural spiking from 32 channels is plotted with the behavior epochs highlighted.
Behavior and spike raster. Behavior was measured from 8 camera views as the monkey performed complex coordinated movements. Location
II. EXPERIMENTAL SETUP
A. Behavioral Task
All protocols were approved by the Stanford University
Institutional Animal Care and Use Committee. We trained
an adult male rhesus macaque (Monkey I) to walk on a
treadmill at speeds ranging from 2.0 kph to 3.5 kph as
shown in Fig. 1. Each session lasted approximately 10
minutes and was divided into blocks where the monkey
walked continuously for up to 2 minutes before a break.
During the break, the monkey reached for food at the front
of the environment. In some blocks, labeled ‘walk-reach’
blocks, food was presented at the front of the environment
while the animal was walking. After taking a step, the
monkey would reach out with his right arm to grab food,
put it in his mouth, and then continue walking. An example
trajectory is presented in Fig. 2a-e. This study comprises
one day’s session (I120130) where the monkey walked at
speeds ranging from 2.0-3.5 kph for 4 walking blocks and 2
B. Video Capture
Video was captured at 24 fps at a resolution of 1624×1224
pixels using eight Point Grey Grasshopper GRAS-20S4M/C
cameras. These cameras were placed around the workspace
of the monkey at various positions to capture multiple angles
of view. Image acquisition and export was performed using
a 4DViews 2DX Multi-Camera system.
C. Neural Recording
Monkey I was implanted with a 96-channel multielec-
trode array (Blackrock Microsystems, Salt Lake City, UT)
implanted in dorsal premotor cortex (PMd) as determined
by visual anatomical landmarks. Broadband neural activ-
ity on 32 electrodes was sampled at 30 kSamples/s and
transmitted wirelessly using the HermesD system . An
OrangeTree ZestET1 FPGA was programmed to package
the HermesD output datastream into a UDP Ethernet packet
stream, which was saved to disk. In addition, the ZestET1
was programmed to record times when video frames were
captured by listening to the video camera synchronization
line. We tested the synchronization by illuminating distinct
patterns on 4 LEDs visible in multiple camera views to
guarantee accuracy between neural recordings and video
frames. Thus, synchronization between the neural and video
data streams was accurate to within +/-5 ms.
D. Neural Data Processing
Each channel of neural recordings was filtered with a zero-
phase highpass filter to remove the local field potential (LFP),
since LFP is not the focus of the present study. Specifically,
a fourth order Butterworth filter with cutoff frequency of 250
Hz was used forward and reverse to ensure zero phase delay.
Spike timing was determined with a single threshold. Points
where the signal dropped below -4.0× the RMS value of the
channel were spike candidates. Occasional artifacts, likely
due to static discharge, were automatically rejected from the
candidate spike set based on the shape and magnitude of the
signal near the threshold crossing point.
rasters of approximately 50 trials for two channels during the swing phase
(green) and stance phase (orange). a Channel 7 b Channel 32
Modulation of neural activity during phases of walking. Spike
III. BEHAVIOR ANALYSIS
A. Hand-Tagged Kinematics
Kinematics were extracted from the recorded video man-
ually via frame-by-frame analysis. A custom-written Python
GUI was used to facilitate tagging of points and visualizing
their location across all cameras. Four points were tagged
on each frame of interest, as shown in Fig. 2. Ascending up
the arm, these were: wrist, elbow, shoulder, and a reference
point on the spine. These kinematics were linked at the spinal
reference point to form the final kinematic profile.
B. Behavioral Epoch Tagging
Freely-moving behavior, and in particular walking, has no
inherent trial structure. Therefore, to segment the neural data
to make it amenable for subsequent analysis, an artificial
trial structure was imposed on the behavioral data to label
epochs of time that were similar across the recorded datasets.
Eight distinct behavioral epochs were labeled in a frame-by-
frame manner using a custom written Matlab GUI. Two of
the epochs were related to the position of the right arm during
walking: the swing phase (Fig. 2a) and the stance phase (Fig.
2b). Four epochs were related to acquisition of food: reaching
for food while sitting, reaching for food while walking (Fig
2c), bringing food to mouth while sitting (similar to Fig.
2d), returning hand to floor while sitting (similar to Fig. 2e).
Two epochs were related to idle times–one sitting and one
standing. A total of 252 epochs were classified into one of the
aforementioned eight categories. The corresponding times in
the neural data were then pulled and assembled into their
behavioral category, forming the basis of the trial structure
(as shown in Fig. 2f) used for the subsequent analysis.
A. Behavioral Tuning
With the tagged epochs of behavior where the monkey
was walking, the neural data was aligned at the swing-
stance phase transition. Two channels of neural activity
Fig. 4. PCA plot of average firing rates. Plot of epoch average firing rate
categorized by behavior. Triangles represent walking epochs (swing phase
in green, stance phase in orange), X’s represent epochs reaching for food
(reaching to food while sitting in pink, reaching to food while walking in
red, bring food to mouth in gray, and returning his hand to the floor in
blue), and circles represent idle epochs (sitting idly in pink and standing
idly in purple).
at this transition time are shown in Fig. 3. Note that for
both depicted channels, neural firing rates increase during
the swing phase and are relatively less active during the
stance phase. The swing and stance phase across all channels
was compared using a two sample t-test, and in 25 of the
32 channels there was a statistically significant difference
(p < 0.001) in the firing rate of that channel between these
epochs. This finding suggests that many of the channels in
PMd are well tuned and modulated with walking activity.
B. Epoch Decoding
Having demonstrated neural tuning across epochs of walk-
ing, we next explored whether it was possible to differentiate
among these eight categories using decoding techniques. To
gain insight into the structure of the epoch firing rate, prin-
cipal component analysis (PCA) was performed on the data.
Fig. 4 plots the average firing rate of each epoch along its first
two principle components. Significant clustering by epoch
categories can be seen by visual inspection. This clustering
suggests that decoding epoch categories may be possible. To
decode, we used regularized discriminant analysis . The
neural data was regularized and fit to a multivariate Gaussian.
Decoding was performed using maximum likelihood and
leave-one-out cross-validation with a classification accuracy
of 96%. The success of classifying and differentiating these
categories suggests that real-time decoding of behavior may
The results shown in the previous section highlight a few
examples in which the freely-moving experimental model
is useful for verifying generalizability. The segmentation
of the walking trials along phases of movement revealed
strong motion tuning, demonstrating that despite the lack
of rigid constraints, such principles still appear to hold
true. This supports the findings of basic motor systems
neuroscience and suggests that despite the limitations of
task conditions, the constrained experimental environment
can uncover generalizable mechanisms.
Similarly, the success of decoding among the behavioral
epochs despite significant postural variability, environmental
distractions, or lack of controlled repeatable trial conditions,
strongly support the applicability of decoding techniques to
the freely-moving environment. This is rather surprising as
there were no controls or enforcements of posture or position.
Any of the aforementioned factors could have led to failure of
decoding epochs due to contamination of the neural activity
with aberrant and uncorrelated firing, yet the decoder was
robust to such variability. This success holds promise for
the translation of BMIs to the more generalized context
where they will have to perform well under more strenuous
conditions–where the neural signal may be masked by neural
noise stemming from environmental demands.
The ability to perform experiments similar to those con-
ducted in more traditional neuroelectrophysiological setups
in the freely-moving context is a step towards an animal
model that most closely resembles human behavior. These
experimental techniques are exciting as they may aid in
finding neuroscientific truths about the basis of generalized,
unconstrained movement as well as for developing and
testing BMIs in a strenuous fashion before translation to
A. Future Work
In the present study, hand-tagged images provided a
relatively good ground truth for interpolating the kinematic
position of the arm. However, hand-tagging is not feasible
for more complex studies of natural behavior for a number
of reasons: 1) it would be laborious to extend hand tagging
to a more complete kinematic model of body posture, 2) it
is somewhat qualitative and subject to user error, and 3) it
does not scale to large datasets.
It is promising that a relatively simple model for decoding
neural activity performed very well. At present these results
are from one monkey (I), and experiments are currently
under way with a second monkey (N) which will allow us to
determine if, and hopefully confirm that, these one monkey
results generalize. Subsequent work would incorporate more
complex models and aim to decode kinematic parameters,
ideally in real-time.
We thank M. Risch, J. Aguayo, E. Morgan, and C. Sher-
man for their expert surgical assistance, assistance in animal
training, and veterinary care; B. Oskosky for computing
assistance; and S. Eisensee, E. Castaneda, and B. Davis for
 E.V. Evarts, “Pyramidal tract activity associated with a conditioned
hand movement in the monkey.”, J. Neurophysiology, vol. 29, no. 6,
pp. 1011–1027, Nov. 1966.
 A.P. Georgopulous, J.F. Kalaska, R. Caminti, and T. Massey, “On the
relations between the direction of two-dimensional arm movements
and cell discharge in primate motor cortex.”, J. Neuroscience, vol. 2,
no. 11, pp. 1527–1537, Nov. 1982.
 E.E. Fetz,“Are movement parameters recognizably coded in the
activity of single neurons?”, Behavioral and Brain Sciences, vol. 15,
no. Special Issue 04, pp. 679–690, 1992.
 C.E. Vargas-Irwin, G. Shakhnarovich, P. Yadollahpour, J.M.K. Mislow,
M.J. Black, and J.P. Donoghue, “Decoding Complete Reach and Grasp
Actions from Local Primary Motor Cortex Populations”, Journal of
Neuroscience, vol. 30, no. 29, pp. 9659, 2010.
 J.M. Carmena, M.A. Lebedev, R.E. Crist, J.E. O’Doherty, D.M.
Santucci, D.F. Dimitrov, P.G. Patil, C.S. Henriquez, and M.A.L.
Nicolelis, “Learning to control a brainmachine interface for reaching
and grasping by primates”, PLoS Biol, vol. 1, no. 2, pp. e42, Oct.
 Leigh R. Hochberg, Mijail D. Serruya, Gerhard M. Friehs, Jon A.
Mukand, Maryam Saleh, Abraham H. Caplan, Almut Branner, David
Chen, Richard D. Penn, and John P. Donoghue, “Neuronal ensemble
control of prosthetic devices by a human with tetraplegia”, Nature,
vol. 442, pp. 164–171, 2006.
 G. Santhanam, S.I. Ryu, B.M. Yu, A. Afshar, and K.V. Shenoy, “A
high-performance braincomputer interface.”,
195–198, July 2006.
 M Velliste, S Perel, MC Spalding, AS Whitford, and AB Schwartz,
“Cortical control of a prosthetic arm for self-feeding.”, Nature, vol.
453, no. 7198, pp. 1098–1101, 2008.
 J.E. O’Doherty, M.A. Lebedev, P.J. Ifft, K.Z. Zhuang, S. Shokur,
H. Bleuler, and M.A.L. Nicolelis, “Active tactile exploration using a
brain-machine-brain interface.”, Nature, vol. 479, no. 7372, pp. 228–
231, Nov. 2011.
 C. Ethier, E.R. Oby, M.J. Bauman, and L.E. Miller,
of grasp following paralysis through brain-controlled stimulation of
muscles.”, Nature, vol. 485, no. 7398, pp. 368–371, May 2012.
 L.R. Hochberg, D. Bacher, B. Jarosiewicz, N.Y. Masse, J.D. Simeral,
J. Vogel, S. Haddadin, J. Liu, S.S. Cash, P. van der Smagt, and J.P.
Donoghue,“Reach and grasp by people with tetraplegia using a
neurally controlled robotic arm.”,
372–375, May 2012.
 C.A. Chestek, V. Gilja, P. Nuyujukian, R.J. Kier, F. Solzbacher,
S.I. Ryu, R.R. Harrison, and K.V. Shenoy, “HermesC: Low-Power
Wireless Neural Recording System for Freely Moving Primates”,
Neural Systems and Rehabilitation Engineering, IEEE Transactions
on, vol. 17, no. 4, pp. 330 –338, Aug. 2009.
 H. Miranda, V. Gilja, C.A. Chestek, K.V. Shenoy, and T.H. Meng,
“HermesD: A High-Rate Long-Range Wireless Transmission System
for Simultaneous Multichannel Neural Recording Applications”, IEEE
Transactions on Biomedical Circuits and Systems, vol. 40, no. 3, pp.
181–191, June 2010.
 H. Gao, R.M. Walker, P. Nuyujukian, K.A.A. Makinwa, K.V. Shenoy,
B. Murmann, and T.H. Meng, “HermesE: A 96-Channel Full Data
Rate Direct Neural Interface in 0.13 um CMOS”, IEEE Journal of
Solid-State Circuits, vol. PP, no. 99, pp. 1 –13, 2012.
 J.D. Foster, O. Freifeld, P. Nuyujukian, S.I. Ryu, M.J. Black, and K.V.
Shenoy, “Combining wireless neural recording and video capture for
the analysis of natural gait”, in Neural Engineering (NER), 2011 5th
International IEEE/EMBS Conference on, May 2011, pp. 613 –616.
 J.H. Friedman, “Regularized discriminant analysis”, Journal of the
American Statistical Association, vol. 84, no. 405, pp. 165–175, Mar.
Nature, vol. 442, pp.
Nature, vol. 485, no. 7398, pp.