Solving the brain synchrony eigenvalue problem: conservation of temporal dynamics (fMRI) over subjects doing the same task.

Solving the brain synchrony eigenvalue problem:
conservation of temporal dynamics (fMRI) over subjects
doing the same task
S. J. Hanson & A. D. Gagliardi & C. Hanson
Received: 1 April 2008 /Revised: 14 November 2008 /Accepted: 20 November 2008
# Springer Science + Business Media, LLC 2008
Abstract Brain measures often show highly structured
temporal dynamics that synchronize when observers are
doing the same task. The standard method for analysis of
brain imaging signals (e.g. fMRI) uses the GLM for each
voxel indexed against a specified experimental design but
does not explicitly involve temporal dynamics. Consequently,
the design variables that determine the functional brain areas
are those correlated with the design variation rather than the
common or conserved brain areas across subjects with the
same temporal dynamics given the same stimulus conditions.
This raises an important theoretical question: Are temporal
dynamics conserved across individuals experiencing the same
stimulus task? This general question can be framed in a
dynamical systems context and further be posed as an
eigenvalue problem about the conservation of synchrony
across all brains simultaneously. We show that solving the
problem results in a non-arbitrary measure of temporal
dynamics across brains that scales over any number of
subjects, stabilizes with increasing sample size, and varies
systematically across tasks and stimulus conditions.
Keywords Neuroimaging . Time series . Eigenvalue .
Event perception . Synchrony . Temporal dynamics .
Inter-subject-correlation
1 Introduction
Brain measures such as EEG/MEG or fMRI can be shown
to possess significant temporal structure in response to task
demands. These temporal dynamics have been studied at
different levels of neural organization in the brain and
behavior, on various time scales with various stimulus type
and duration in both human and animal systems (Kelso
1995; Friston et al. 2003; Buzsaki 2006; Hanson and
Timberlake 1983). In the present research we analyze the
temporal structure in fMRI and show that temporal dynamics
are conserved across subjects doing the same type of
behavioral task. Current practice in fMRI focuses on the
individual voxel level and treats it as a single variable in a
general linear model (GLM). If regressors (e.g. auto-
regressive) are constructed that have dependencies over time
or have some temporal structure, the GLM will, of course, be
able to identify tissue possessing temporal sensitivity.
However, the variation in a typical block or event related
design relies on categorical variations in the constructed
variables, usually without any temporal reference. In effect,
the GLM forces a decomposition of the sources contributing
to stimulus variation thereby accounting for the variation in
the response variables, in the case of fMRI, voxels in the
image space. Corrections to control for false discovery rate
are typically introduced as spatial procedures that either
explicitly re-estimate Type II errors or decrease the effective
number of variables (e.g., by smoothing), but are not
temporal in nature. Consequently, current practice in the
analysis of fMRI signals has required a model of the
presumed structure and effects of a given stimulus context
(as in a design specification) that might evoke a specific
brain response, rather than measuring the actual evoked
response that was experienced by subjects over time.
J Comput Neurosci
DOI 10.1007/s10827-008-0129-z
Action Editor: Peter Dayan
S. J. Hanson (*) : A. D. Gagliardi : C. Hanson
Psychology Department,
Rutgers Mind/Brain Analysis (RUMBA) Labs,
Rutgers University,
Newark, NJ, USA
e-mail: jose@tractatus.rutgers.edu

One possible strategy to directly study the evoked
response in the brain is to do a kind of system identification
that measures common or conserved responses between
what we expect to be nearly identical systems. To the extent
that a response pattern between sets of systems is
conserved, it is possible to identify structure that is
independent of any given stimulus properties (and unfortu-
nately, at the same time losing information about the
stimulus generation itself). Recently this approach has
become popular both in measuring so-called “resting state”
dynamics (Gusnard and Raichle 2001) and in a seminal
study that introduced a surprisingly simple method for
measuring conserved responses (Hasson et al. 2004) while
subjects passively watched a 1/2 hour segment of the same
commercial movie. Because of the continuous and uncon-
trolled nature of such a stimulus, standard analysis could
not be easily adapted to identify the conserved neural
response inferred from fMRI between pairs of subjects.
Instead, Hasson et al, correlated (Pearson Product Moment
Correlation—PPMC) fMRI timeseries (as phaselocked as
possible) between pairs of subjects producing a dynamical
signature of the “effect” of the time dependent waveform
(the movie). Interestingly, this can be seen as analogous to
creating a phase space between sets of variables in a
dynamical system. As the two variables vary in time, in this
case the fMRI time dependent signals, they can be
characterized by examining their dependency by parame-
terizing time, in effect, removing the temporal variable. For
a given dynamical system, these dependencies or trajecto-
ries in the phase plot are often organized by “attractors”, or
coordinate points (stable points) in the space toward which
the dynamical system tends in equilibrium, independent of
where the system starts (“initial conditions”) or how it
might be perturbed. Although noise in signals like fMRI
will tend to mask even low order dynamics, strong
correlations between variables can indicate that the under-
lying system has non-arbitrary properties related to stimulus
variation in time. For brain imaging signals, it thus seems
possible to examine these variations between voxels and to
explore such dynamics in the way that Hasson et al. (2004)
had suggested. In order to get a firmer theoretical grip on
this type of procedure and to put it in the proper framework,
we next consider the conventional generation process of the
BOLD signal from neural dynamics.
1.1 Brain dynamics
How might these brain imaging dynamics arise and be
synchronized? Consider a simple coupled linear dynamical
system that might govern two neurons (N and M) or for our
purposes here a cluster or populations of neurons that can
be approximated as mean field (e.g. <N>, <M>) strengths
where the change in activity of one population is inversely
proportional to its own activity (thus decaying exponen-
tially to its baseline without out further input where CN and
CM are fixed constants) and a function of its own strength
modulated by the strength of the other population:
dN=dt ¼ h N ;Mð Þ � N þ CN
dM=dt ¼ g N ;Mð Þ � M þ CM
ð1Þ
Convolving these neural populations with the BOLD
response function, B(x;t,s), which is a function of time (t)
and spatial (s) diffusion also makes the assumptions of time
invariance of the neural impulse (which itself is delayed by
milliseconds from stimulus onset) and of system linearity
(sums of stimulus events and therefore neural spiking
produce sums in BOLD responses—which is actually
unlikely to be true, Glover 1999; nonetheless for a first
order approximation for this analysis we will assume it to
be true; but see next section).
d B t; sð Þ � N tð Þ=dt½ ¼ h B t; sð Þ � N tð Þ;B t; sð Þ � M tð Þ½ �
�B t; sð Þ � N tð Þ þ bCN
d B t; sð Þ � M tð Þ=dt½ ¼ g B t; sð Þ � N tð Þ;B t; sð Þ � M tð Þ½ �
�B t; sð Þ � M tð Þ þ bCM
ð2Þ
Consequently, with these strong linearity assumptions
the convolution can preserve a spatio-temporal diffused and
instantaneous response (assuming the timing of the event
was recorded and labeled) of neural spiking and therefore
of the underlying stimulus waveform. If either the neural
response or the BOLD response to the neural response
includes a noise term the convolution of the two functions
will blend the noise terms as well (as shown in Eq. (2)),
making inversion of the BOLD signal to the stimulus or
neural waveform very difficult or impossible.
Assuming the neural spiking is entrained by the stimulus
and triggers neural resonance at various periods due to
coupling and feedback through functions such as h and g,
correlations could occur at multiple scales. These oscilla-
tions can be, of course, at multiple frequencies and
therefore be the basis for the type of time series associations
we observe in conserved fMRI responses across subjects.
Notwithstanding the noise level in fMRI and the probable
nonlinearity of the system, the synchronization would only
be modulated as long as the phase of both time series are
well within the period of the resonate frequency of the
system.
One common way to indicate the kind of dependency in
phase space of a homogenous linear dynamical system is to
find the eigenvalues associated with the matrix of constant
coefficients of the linear system. Depending on the nature
of the eigenvalues (distinct real or complex values) of the
J Comput Neurosci

linear system, the solution will contain stable points, saddle
points, or periodic solutions. Although the use of eigenvalues
with a given dynamical system can categorize the complexity
of that system, due to our ignorance concerning the actual
dynamics of the BOLD signal, they can only provide an
inspiration below for measuring the underlying complexity of
the temporal dynamics from the brains of subjects doing the
same task.
Many in the field have examined various measures of
voxel complexity over time (e.g. Hurst exponents, Bullmore
et al. 2001; Shimizu et al. 2004). Although such analyses
indicate the presence of high order dynamical structure in
voxel variation, they are neither sufficient to identify
anatomical function nor necessary to determine if such
dynamics are phase locked to any specific stimulus
variation. However, if such correlations arise independently
between voxels from a group of subjects experiencing the
same stimulus events, then it is unlikely that the implicated
anatomical areas are arbitrary relative to those stimulus
variations. This type of conservation is not a function of
simple spatial aggregation or averaging since it depends on
the dynamical time correlation. For neuroimaging data, it is
important, therefore, to discuss the BOLD signal’s potential
for the kind of temporal resolution which is known to
capture a given perceptual or cognitive function, since
conservation will be dependent on temporal resolution
perhaps even at multiple time scales.
1.2 Time resolution of the BOLD signal: is it just too slow?
In some ways time resolution in the BOLD signal has been
a paradox. On the one hand, the sluggish character of the
hemodynamic response (HDR) peaks anywhere from 4 to
6 s after a stimulus onset, depending on the tissue being
measured. The fastest onset of a measurable change from
the stimulus onset for the HDR is probably only about two
seconds. On the other hand, with a sufficient number of
trials, it may be possible to see measurable evoked HDRs at
sub-second resolution (e.g., see Menon et al. 1998) similar
in kind to that which can be obtained from phase-locked
averaging of EEG signals. Event related designs with
staggered event points are another way of obtaining sub-
second resolution as long as a sufficient number of sample
points are taken. The ability to obtain a temporal resolution
greater than that of HDR response may seem paradoxical if
the HDR signal was Gaussian and linear. However, the
BOLD response is highly nonlinear and its distribution
possesses a long tail (Hanson and Bly 2001; Chen et al.
2003). It is highly likely, therefore, that the fMRI signal has
a significant autocorrelation, a feature which is often
associated with a high temporal complexity, and one that
potentially can be used to obtain a higher temporal
resolution than that indicated by the sluggish HDR
diffusion process. In fact, an examination of the underlying
complexity of the BOLD time series using various
measures (e.g, Hurst exponents, AR(p) and other similar
estimates; also see Bullmore et al. 2001), even without a
task, one finds “long memory” temporal dependencies in
the time series (indicated by Hurst values >0.5, often near
0.7). These dependencies across various sets of regions may
reflect underlying anatomical or functional networks that
persist in recurrent or reciprocal neural activation. In any
case, it seems clear that there are significant dependencies
at various temporal scales (milliseconds, seconds, minutes)
in the BOLD signal during “rest” that could be differen-
tially modulated by stimulus contexts. Nonetheless, the
substantial noise level in fMRI, and its weak correlation
with the stimulus waveform, make it difficult to measure
such dynamics using stimulus regressors in any precise
way. Consequently, it becomes necessary to take a more
indirect approach to measuring temporal dynamics that can
provide signal sharpening while minimizing noise, based
on the conserved temporal dynamics across individuals
experiencing the same stimulus conditions.
1.3 Measuring the temporal dynamics across brains:
conserving synchrony
Assuming that temporal dynamics are more or less
stationary (once corrected for temporal drift and various
physiologic artifacts such as breathing, heart rate etc.), it
should be possible to measure temporal dynamics as a
function of stimulus conditions that are held constant over
some measurement interval. The challenge here is not
unlike that faced when noise in the fMRI measurement
must be controlled for, despite an unknown underlying
distribution. The GLM uses a “black box” contrast in which
it is assumed that signals that are not modulated by the
different stimulus conditions (e.g., noise) will be removed
during the linear contrast. Thus, the GLM assumes an
additive nature of the underlying factors (which include
noise) and estimates the mean effect between stimulus
conditions by treating the unknown noise structure as
though it were identical between conditions and brains.
Thus, to the extent that noise is correlated with a given
stimulus condition, the estimated mean effect will incorpo-
rate the correlated noise from the contrast and be smoothed
into the effects image. Unfortunately, the standard linear
model provides no way of completely removing correlated
noise without also directly modeling the noise itself in the
estimator. Thus, the GLM is effectively limited in revealing
parametric dependencies of different stimulus conditions by
its inability to accurately model the nonlinearity of the
HDR and the complexity of the noise in the BOLD signal.
However, if similar stimuli produce similar temporal
responses across different brains, it should be possible to
J Comput Neurosci

estimate the underlying temporal dynamics from the con-
served temporal patterns, and reduce the influence of noise, as
long as the signal and noise are marginally different. This type
of “signal sharpening” should increase with the number of
brains in the sample, improving asymptotically as the
underlying noise is incrementally removed (the exact size of
the sample is unknown, but see below).
Intuitively, temporal dynamics within a single brain
can be identified by considering the aligned synchrony
that is common between subjects as can be seen in the
time series plot of three brain/voxels shown in Fig. 1.
This phase space shows that common or conserved
temporal dynamics are equivalent to a projection of the
time points into a lower order manifold (plane) that is
shown here in 2-D as the first eigenvector of a
corresponding association matrix. As the three subjects’
projections cluster near the principle component in this
space, a high temporal synchrony is revealed. Mathemat-
ically, we can express potential synchrony across all
brains in a population (per voxel) as an eigenvalue
problem and ask whether, given b brains (per voxel)
across T (over t) time points, is there some non-zero
measure of synchrony, σ, that is conserved across all
brains such that voxel time patterns are a linear combi-
nation of one another? This eigenvalue problem can be
framed within a brain by brain covariance/correlation in
which σ represents the strength of the temporal synchrony
between subjects’ brains at each voxel and eigenvectors
indicate the agreement and directionality in subject space.
In this way, σ and the eigenvectors can be used as a
measure of overall subject synchrony and could also be
used as a measure of individual differences among
subjects (to the extent that coefficients contrast between
subjects). Although a brain by brain (b×b in the equation
below) covariance/correlation involves an assumption
about linearity concerning the temporal dynamics, the
actual pattern of these dynamics could be highly nonlin-
ear. Equation (3) states the eigenvalue problem including
the relevant matrices.
AbbSb ¼ sbSb
Abb ¼ COV V
t
b
� �
V tb ¼
Vb11Vb21Vb31
Vb12Vb22Vb32
Vb13Vb23Vb33
::::::::::
Vb1TVb2TVb3T
2
6
6
6
6
4
3
7
7
7
7
5
ð3Þ
The solution to Eq. (3) is equivalent to finding the
eigenvectors, S, that are associated with the largest
eigenvalues, σ per voxel over all brains and time points.
The largest eigenvector is directly related to Roy’s Largest
Root Statistic (Pillai 1965), and with appropriate degrees of
freedom, can be tested directly as an F statistic. Note
because the extraction is using ALL brains at once, there
are no order effects as there is in the PPMC/ISC method
(Hasson et al. 2004). Further note, that if N=2 this is
exactly the Hasson algorithm as we show below. By
calculating a family-wise error over the entire voxel
eigenvalue set a corrected statistical significance can be
achieved. It is also trivial to prove that the eigenvalue
between two brains must be monotonically related to a
product moment correlation. It is also obvious that the
synchrony between two brains (Hasson et al. 2004) is a
special case of the eigenvalue problem we have just
outlined. Furthermore all utilizations of the Hasson et al.
brain-PAIRWISE method (a recent example is Hejnar et al.
2007, but there are many others) are also subject the
limitations and interpretation problems we discuss below.
We will demonstrate this point by comparing our eigenval-
ue approach for ALL brains at the same time with a brain
pair-wise pearson product moment correlation (PPMC or
what they called Inter-subject correlation—ISC) for the
case of two brains exposed to the same continuous stimulus
Fig. 1 Constructed example
showing three subjects time se-
ries (right panel), and their
phase space showing (in green)
the times series values from the
right panel as they project to the
eigenvector 2D plane (red grid).
The closer the time series points
to the plane indicates high syn-
chronization between time series
J Comput Neurosci

(a short action sequence video). Before examining some of
the properties of the eigenvalue synchrony measure (EVS),
we will describe some types of stimuli and methods that
might be used with this novel approach to detecting
conservation of temporal dynamics in brain imaging
measures. Note that any group-wise analysis using other
temporal extraction algorithms (Maximum Likelihood
factor analysis or any other variance/covariance factoring
methods) will of course accomplish the same kind of
conservation given they are sensitive to the underlying
temporal dynamics. In contrast, factoring methods such as
ICA in the groupwise case (for a groupwise ICA method,
see for example, Beckman and Smith 2005), would instead
tend to extract hypothetical statistically independent pro-
cesses that exist in different areas of the brain across all
subjects. Consequently, ICA over brains (per voxel) would
tend to find a smaller set of conserved areas, if any, since
they are required to be both conserved and orthogonal.
Principal Components Analysis (PCA) or Independent
Components Analysis (ICA) can also be used to compress
the number of variables based on temporal variations, but
often in a more exploratory context, and do not need to be
related to the design variables and can be used for analysis
of resting state patterns and for separating signal from noise
(albeit in an unsupervised way, often requiring a keen eye
to determine which is which)
1.4 Summary of EV method
Because this is a novel method, its important to emphasize
the novel aspects of the approach. (1) This method works
across ALL subjects (brains) simultaneously. (2) The
largest eigenvalue is extracted over all brains for each
voxel (see Fig. 4 below for exact flow of computation and
pseudo-code). (3) This measure of dynamical conservation
is known as the Roy’s largest Root (or properly normed as
Wilkes Lambda—see below) for all brains, which is a
measure that scales over sample size (brains) and time
points simultaneously. (4) This is a novel method that finds
the synchronization of fMRI time series and generalizes
other approaches to this problem and provides a statistically
principled way to scale over subjects and time points unlike
other methods that possess sequential bias, are ad hoc, and
use less efficient permutation tests for statistical signifi-
cance (e.g. Hasson et al 2004). Note that the key aspect to
the new method is not the particular statistic (RLR) that we
have used to determine the significance of the temporal
synchrony, but rather the ability to exploit more brains in
the synchrony estimate (RLR or “r”) thus potentially
increasing the power of the approach dramatically.
In the next section we will apply the eigenvalue
synchrony (EVS) method to fMRI data sets acquired from
subjects watching short movies (minutes) and asked to push
a button when they detect a change point in the action
sequence. This will allow us to benchmark the method and
contrast it with the PPMC pairwise method.
1.5 Continuous stimuli paradigms: event cognition
Although any type of stimulus paradigm, including a block
design, could be used to find conservation of temporal
dynamics between subjects, a continuous stream of events
should create optimal conditions for observing the similar-
ity of temporal dynamics across subjects. We decided to
create our own video sequences, rather than use commer-
cially available movies, in order to systematically control
differences across sequences which would allow us to map
differences in response to specific changes in the stimuli.
Whereas commercially available movies can vary in the
number of camera views, actors, objects, as well as plot
complexity), we constructed three types of simple video
sequences with increasingly more complex goal structures.
One type of sequence (schema-rich) consisted of a single
actor performing a simple, familiar task (e.g., making a cup
of coffee) that took no more than a few minutes (four to
eight minutes) to complete. We characterized this type of
video as schema-rich because it involved a well-defined
task with a simple goal structure that was composed of
highly predictable actions. Previous work shows that to
comprehend the meaning of action streams, people natural-
ly and spontaneously parse activity into distinct units of
meaning (e.g., eating dinner, going to the movies, etc).
Moreover, observers who are asked to categorize action
sequences in real time produce consistent judgments about
action cluster start and end points (more details of video
construction can be found in Hanson and Hirst 1989;
Hanson and Hanson 1996; Heider and Simmel 1944; Zacks
and Tversky 2001; Bartels and Zeki 2005; Hanson et al.
2007). Thus, although parsing video action sequences into a
constituent structure is a relatively complex visual task, it is
one that is naturally and easily performed by most people
both inside and outside of the laboratory.
The second type of action sequence (we define as
“schema-poor”) we used was an animated set of geometric
shapes moving around each other. This stimulus was
modeled on one constructed originally by Hieder and
Simmel (1944) who showed in their seminal study that
subjects will anthropomorphize about moving geometric
shapes, creating a story involving goals and emotions, even
when the motion and nature of the stimuli are arbitrary. In
our animated video we produced a short sequence lasting
about 4 min in which a circle moved around a geometric
“house” consisting of a set shapes in fixed locations. This
sequence was less well-defined than the schema-rich
sequence, allowing for various plausible accounts rather
than one simple and obvious goal based structure.
J Comput Neurosci