A framework to support automated classification and labeling of brain electromagnetic patterns.

Abstract

This paper describes a framework for automated classification and labeling of patterns in electroencephalographic (EEG) and magnetoencephalographic (MEG) data. We describe recent progress on four goals: 1) specification of rules and concepts that capture expert knowledge of event-related potentials (ERP) patterns in visual word recognition; 2) implementation of rules in an automated data processing and labeling stream; 3) data mining techniques that lead to refinement of rules; and 4) iterative steps towards system evaluation and optimization. This process combines top-down, or knowledge-driven, methods with bottom-up, or data-driven, methods. As illustrated here, these methods are complementary and can lead to development of tools for pattern classification and labeling that are robust and conceptually transparent to researchers. The present application focuses on patterns in averaged EEG (ERP) data. We also describe efforts to extend our methods to represent patterns in MEG data, as well as EM patterns in source (anatomical) space. The broader aim of this work is to design an ontology-based system to support cross-laboratory, cross-paradigm, and cross-modal integration of brain functional data. Tools developed for this project are implemented in MATLAB and are freely available on request.

Full text

Page 1

Hindawi Publishing Corporation
Computational Intelligence and Neuroscience
Volume 2007, Article ID 14567, 13 pages
doi:10.1155/2007/14567
ResearchArticle
A Framework to Support Automated Classification and
Labeling of Brain Electromagnetic Patterns
Gwen A. Frishkoff,1Robert M. Frank,2Jiawei Rong,3Dejing Dou,3Joseph Dien,4and Laura K. Halderman1
1Learning Research and Development Center, University of Pittsburgh, Pittsburgh, PA 15260, USA
2NeuroInformatics Center, University of Oregon, 1600 Millrace Drive, Eugene, OR 97403, USA
3Computer and Information Sciences, University of Oregon, Eugene, OR 97403, USA
4Department of Psychology, University of Kansas, 1415 Jayhawk Boulevard, Lawrence, KS 66045, USA
Correspondence should be addressed to Gwen A. Frishkoff, gwenf@pitt.edu
Received 19 February 2007; Revised 28 July 2007; Accepted 7 October 2007
Recommended by Saied Sanei
This paper describes a framework for automated classification and labeling of patterns in electroencephalographic (EEG) and
magnetoencephalographic (MEG) data. We describe recent progress on four goals: 1) specification of rules and concepts that
capture expert knowledge of event-related potentials (ERP) patterns in visual word recognition; 2) implementation of rules in
an automated data processing and labeling stream; 3) data mining techniques that lead to refinement of rules; and 4) iterative
steps towards system evaluation and optimization. This process combines top-down, or knowledge-driven, methods with bottom-
up, or data-driven, methods. As illustrated here, these methods are complementary and can lead to development of tools for
pattern classification and labeling that are robust and conceptually transparent to researchers. The present application focuses on
patterns in averaged EEG (ERP) data. We also describe efforts to extend our methods to represent patterns in MEG data, as well as
EM patterns in source (anatomical) space. The broader aim of this work is to design an ontology-based system to support cross-
laboratory,cross-paradigm,andcross-modalintegrationofbrainfunctionaldata.Toolsdevelopedforthisprojectareimplemented
in MATLAB and are freely available on request.
Copyright © 2007 Gwen A. Frishkoff et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.
1.INTRODUCTION
The complexity of brain electromagnetic (EM) data has led
to a variety of processes for EM pattern classification and la-
beling over the past several decades. The absence of a com-
mon framework may account for the dearth of statistical
metaanalyses in this field. Such cross-lab, cross-paradigm re-
views are critical for establishing basic findings in science.
However, reviews in the EM literature tend to be infor-
mal, rather than statistical: it is difficult to generalize across
datasets that are classified and labeled in different ways.
To address this problem, we have designed a framework
to support automated classification and labeling of patterns
in electroencephalographic (EEG) and magnetoencephalo-
graphic (MEG) data. In the present paper, we describe the
framework architecture and present an application to aver-
aged EEG (event-related potentials, or ERP) data collected
in a visual word recognition paradigm. Results from this
study illustrate the importance of combining top-down and
bottom-up approaches. In addition, they suggest the need
for ongoing system evaluation to diagnose potential sources
of error in component analysis, classification, and labeling.
We conclude by discussing alternative analysis pathways and
ways to improve efficiency of implementation and testing of
alternative methods. It is our hope that this framework can
support increased collaboration and integration of ERP re-
sults across laboratories and across study paradigms.
1.1.ClassificationofERPs
A standard technique for analysis of EEG data involves aver-
aging across segments of data (trials), time-locking to stim-
ulus or response events. The resulting measures are charac-
terized by a sequence of positive and negative deflections dis-
tributed across time and space (scalp locations). In princi-
ple, activity that is not event-related will tend towards zero
as the number of averaged trials increases. In this way, ERPs

Page 2

2Computational Intelligence and Neuroscience
provide increased signal-to-noise, and thus increased sen-
sitivity, to functional (e.g., task) manipulations. Signal av-
eraging assumes that the brain signals of interest are time-
locked to (or “evoked by”) the events of interest. As illus-
trated in recent work on induced (nontime-locked) versus
evoked (time-locked) EEG activity, this assumption does not
always hold ([1, 2]).
In the past several decades, researchers have described
several dozen spatiotemporal ERP patterns (or components),
which are thought to index a variety of neuropsychologi-
cal processes. Some patterns are observed across a range of
experimental contexts, reflecting domain-general processes,
such as memory, decision-making, and attention. Other pat-
terns are observed in response to specific types of stimuli,
reflecting human expertise in domains such as mathematics,
facerecognition,andreadingcomprehension(forreviewssee
[3,4]).Previousinvestigationsofthesepatternshavedemon-
strated the effectiveness of ERP methods for addressing basic
questions in nearly every area of psychology.
Given the success of this methodology, ERPs are likely
to remain at the forefront of research in clinical and cog-
nitive neuroscience, even as newer methods for EEG and
MEG analyses are developed as alternatives to signal averag-
ing (e.g., [1, 2, 5–7]).
At the same time, ERP methods face some important
challenges. A key challenge is to identify standardized meth-
ods for measure generation, as well as objective and reli-
able methods for identification and labeling of ERP com-
ponents. Traditionally, researchers have characterized ERP
components in respect to both physiological (spatial, tem-
poral) and functional criteria [8, 9]. Physiological criteria in-
clude latency and scalp distribution, or topography. For ex-
ample, as illustrated in Figure 1, the visual “P100 compo-
nent” is characterized by a positive deflection that peaks at
∼100 milliseconds after onset of a visual stimulus (A) and is
maximal over occipital electrodes, reflecting activity in visual
cortex (B).
Despite general agreement on criteria for ERP compo-
nent identification [9], in practice such patterns can be hard
to identify, particularly in individual subjects. This difficulty
is due in part to the superposition of patterns generated by
multiple brain regions at each time point [10], leading to
complex spatial patterns that reflect the mixing of under-
lying patterns. Given this complexity, ERP researchers have
adopted a variety of solutions for scalp topographic analysis
(e.g., [11, 12]). It can therefore be difficult to compare re-
sults from different studies, even when the same experimen-
tal stimuli and task are used.
Similarly, researchers use a variety of methods for de-
scribing temporal patterns in ERP data [13]. For example,
early components, such as the P100, tend to be character-
ized by their peak latency, while the time course of later com-
ponents, such as the N400 or P300, is typically captured by
averaging over time “windows” (e.g., 300–500 milliseconds).
The latency of other components, such as the N400, has been
quantified in a variety of ways. Finally, there is variability
in how functional information (e.g., subject-, stimulus-, or
task-specific variables) is used in ERP pattern classification.
Some patterns, such as the P100, are easily observed as large
deflections in the raw ERP waveforms. Other patterns, such
as the mismatch negativity are more reliably seen in differ-
ence measures, calculated by subtracting ERP amplitude in
one condition from the ERP amplitude in a contrasting con-
dition.Thisinconsistencymayleadtoconfusion,particularly
whenthesamelabelisusedtorefertotwodifferentmeasures,
as is often the case.
1.2.Outlineofpaper
In summary, the complexity of ERP data has led to multi-
ple processes for measure generation and pattern classifica-
tion that can vary considerably across different experiment
paradigms and across research laboratories. Ultimately, this
limits the ability both to replicate prior results and to gener-
alize across findings to achieve high-level interpretations of
ERP patterns.
In light of these challenges, the goal of this paper is
to describe a framework for automated classification and
labeling of ERP patterns. The framework presented here
comprises both top-down (knowledge-driven) and bottom-
up (data-driven) methods for ERP pattern analysis, classi-
fication, and labeling. Following, we describe this frame-
work in detail (Section 2) and present an application to pat-
terns in ERP data from a visual word processing paradigm
(Section 3). Section 4 describes approaches to system eval-
uation. Section 5 describes data mining for refinement of
expert-driven (top-down) methods. In Section 6, we draw
some general conclusions and discuss extensions of our
framework for representation of patterns in source space,
and ontology development to support cross-paradigm,
cross-laboratory, and cross-modal integration of results in
EM research.
2.PATTERN CLASSIFICATION FRAMEWORK
AsillustratedinFigure 2,ourframeworkcomprisesfivemain
processes.
(i) Knowledge engineering. Known ERP patterns are cata-
loged (1). High-level rules and concepts are described
for each pattern (2).
(ii) Pattern analysis and measure generation. Analysis
methods are selected and applied to ERP data (3). The
goal is transformation of continuous spatiotemporal
data into discrete patterns for labeling. Statistics are
generated (4) to capture the rules and concepts identi-
fied in (2).
(iii) Data mining. Unsupervised clustering (7) and super-
vised learning (8) are used to explore how measures
cluster, and how these clusters may be used to identify
andlabelpatternsusingrulesderivedindependentlyof
expert knowledge.
(iv) Operationalization and application of rules. Rules are
operationalizedbycombiningmetricsin(4)withprior
knowledge (2). Data mining results (7-8) may be used
to validate and refine the rules. Rules are applied to
data, using an automated labeling process (6) detailed
below.

Page 3

Gwen A. Frishkoff et al.3
Time (ms)
−4
−3
−2
−1
0
1
2
3
4
Amplitude (μV)
“P100”
(∼ 120ms)
(a)
+ μV
− μV
(b)
Figure 1: (a) Time course of P100 pattern, plotted at left occipital electrode, O1. Time is plotted on the x-axis (0–700 milliseconds); each
vertical hash mark represents 100 milliseconds. Amplitude is plotted on the y-axis (scale, ±4μV). The dark vertical line marks the time of
peak amplitude (∼120 milliseconds). (b) Scalp topography of the P100 pattern, plotted at the time of peak amplitude. Red, positive. Blue,
negative.
Following, we describe how these processes have been im-
plemented in a series of MATLAB procedures. We then re-
port results from the application of this process to data from
a visual word processing experiment. Results are evaluated
against a “gold standard” that consists of expert judgments
regarding the presence or absence of patterns, and their pro-
totypicality, for each of 144 observations (36 subjects ×4 ex-
periment conditions).
2.1. Knowledgeengineering(process1,2)
The goal of knowledge engineering is to identify concepts
thathavebeendocumentedforaparticularresearchdomain.
Based on prior research on visual word processing we have
tentatively identified eight spatiotemporal patterns that are
commonly observed from ∼100 to ∼700 milliseconds after
presentation of a visual word stimulus, including the P100,
N100, late N1/N2b, N3, P1r, MFN, N400, and P300. Space
limitationsprecludeadetaileddiscussionofeachpattern(see
reviews in [3, 4]). The left temporal N3 and medial frontal
negativity (MFN) components are less well known, but have
been described in several high-density ERP studies of visual
word processing (e.g., [14–16]). The P1r [17] has also been
referred to as a posterior P2 [18]. The late N1/N2b has var-
iously been referred to as an N2, an N170, and a recogni-
tion potential (see [15] for discussion and references). It is
not clear that the late N1/N2 represents a component that is
functionally distinct from the N1 and N3, though it some-
times emerges in tPCA results as a distinct spatiotemporal
pattern (e.g., see Section 3). These eight patterns reflect a
working taxonomy of ERP in research on visual word pro-
cessing between ∼60–700 milliseconds. Application of the
present framework to large numbers of datasets collected
across a range of paradigms, and across different ERP re-
search labs, would contribute to the refinement of this tax-
onomy.
A note of caution is in order, concerning the labels for
scalp regions of interest (ROIs). By convention, areas of the
Table 1: Spatial and temporal concepts used to define the eight tar-
get patterns. Regions of interest (ROIs) are defined in Appendix A.
Pattern
P100
N100
N2
P1r
N3
MFN
N4
P300
Window
60–150
151–230
231–300
250–400
250–400
250–450
350–550
401–700
ROI
occipital
occipital
post-temporal
parietal
left anterior
frontal
parietal
parietal
scalp are associated with anatomical labels, such as “occipi-
tal,” “parietal,” “temporal,” and “frontal” (see Table 1). It is
well known, however, that a positive or negative deflection
over a particular scalp ROI is not necessarily generated in
cortex directly below the measured data. ERP patterns can
reflectsourcestangentialtothescalpsurface.Inthisinstance,
the positive and negative fields may be maximal over remote
regions of the scalp, reflecting a dipolar scalp distribution
(e.g., with a positive maximum over frontal scalp regions,
andanegativemaximumovertemporalscalpregions).Thus,
the ROI labels should not be interpreted as literal references
to brain regions. The ROI clusters used in the present study
are shown in Appendix A.
2.2.Datasummary
Prior to analysis, ERP data consist of complex waveforms
(time series), measured at multiple electrode sites. To sim-
plify analysis and interpretation of these data, a standard
practice is to transform the ERPs into discrete patterns. Tra-
ditional methods for data summary include identification of
peaklatencywithinaspecifiedtimewindow(“peakpicking”)
and computing the mean amplitude over a time window
for each electrode (“windowed analysis”), or averaged over

Page 4

4 Computational Intelligence and Neuroscience
3
4
Pattern analysis & measure generation
ERP component analysis
(PCA, peak picking, etc...)
Statistical measure generation
Function
Source-language
Time
Peak (ms)
Onset (ms)
Offset (ms)
Etc...
Stim modality
Subject group
Etc...
Space
ROI
Max ch.
Spatial r
Etc...
Concepts
Time
Space
Function
Etc...
P100:
∼ 120ms
Occipital
positivity
Stimulus - evoked
1
2
Knowledge engineering
Cataloging known
ERP patterns
High-level pattern description
Operationalization & rule application
Operationalization of rules
PTobs = P100 IFF
- 80ms <= TI-max < 150ms
- ROI = occipital
- IN-mean (ROI) > 0
- Modality = visual
Component labeling
5
6
Application of rules
(5) to measures (4)
Data mining
Unsupervised
clustering
78
Supervised learning
System evaluationRevise system
Figure 2: Pattern classification and labeling scheme. Knowledge engineering (processes 1, 2) includes “top-down” specification of ERP con-
cepts and rules, formulated by domain experts. Component analysis and measure generation (processes 3, 4) yield summary metrics that are
used for pattern classification and labeling. Implementation and operationalization of pattern rules (processes 5, 6) are detailed in Section 2.
Data mining (processes 7, 8) includes “bottom-up” or data-driven methods for clustering and discovery of pattern rules (Section 5). System
evaluation is detailed in Section 4.
electrode clusters (regions of interest—ROIs). An alternative
method is principal components analysis (PCA), which de-
composes the data into “latent” patterns, or factors. The fol-
lowing subsection describes this method in detail, and ex-
plains the utility of PCA for automated pattern classification.
2.2.1.TemporalPCAmethods(process3)
PCA belongs to a class of factor-analytic procedures, which
use eigenvalue decomposition to extract linear combinations
of variables (latent “factors”) in such a way as to account
for patterns of covariance in the data parsimoniously, that is,
with the fewest factors. Mathematically, the goal of PCA is to
take intercorrelated variables (x1,...,xn) and combine them
such that the tranformed data, the “principal components”
(PC),arelinearcombinationsofx,weightedtomaximizethe
amount of variance captured by each eigenvector (vi):
PC1= v11x1+v12x2+ ··· +v1nxn.
(1)
In this way, the original set of variables (x1,...,xn) is “pro-
jected” into a new data space, where the dimensions of this
new space are captured by a small number of latent factors
(the eigenvectors).
In ERP data, the variables (x1,...,xn) are the microvolt
readings either at consecutive time points (temporal PCA)
or at each electrode (spatial PCA). The major source of co-
variance isassumed tobe the ERPcomponents, characteristic
featuresofthewaveformthatarespreadacrossmultipletime
pointsandmultipleelectrodes.Ideally,eachlatentfactorcor-
responds to a separate ERP component, providing a statis-
tical decomposition of the brain electrical patterns that are
superposed in the scalp-recorded data. To achieve this ideal
factor-to-pattern mapping, the factors may be “rotated” so
that the variance associated with the original variables (time-
points) is redistributed across the factors in such a way that
maximizes “simple structure,” that is, that achieves a simple
and transparent mapping from variables to factors. (See [19]
for a review of PCA and related factor-analytic methods for
ERP data decomposition.)
Inthepresentapplication,weusedtemporalPCA(tPCA)
as implemented in the Dien PCA Toolbox [20]. In temporal
PCA, the data are organized with the variables correspond-
ingtotimepointsandobservationscorrespondingtothedif-
ferent waveforms in the dataset. The waveforms vary across
subjects,electrodes,andexperimentalconditions.Thus,sub-
ject, spatial, and task variance are collectively responsible for
covariance among the temporal variables. The data matrix

Page 5

Gwen A. Frishkoff et al.5
is then self-multiplied and mean-corrected to produce a co-
variance matrix. The covariance matrix is subjected to eigen-
value decomposition, and the resulting nonnoise factors are
rotated using Promax to obtain a more transparent relation-
ship between the PCA factors and the latent variables of in-
terest (i.e., ERP components).
After transformation of the ERP data into factor space,
the data are projected back into the original data space, by
multiplying factor scores by factor loadings and by the stan-
dard deviation at each timepoint (see the appendix in [21]).
In this way, it is possible to visualize and extract information
about the strength of the pattern at each electrode, to deter-
minethespatialdistributionofthepatternforagivensubject
and experiment condition. Visualizing the spatial projection
of each factor in this way is useful in interpreting tPCA re-
sults (e.g., see Figure 3(b)).
For our initial attempts to automate data description
and classification, tPCA offered several advantages over tra-
ditional methods. First, tPCA is able to separate overlap-
ping spatiotemporal patterns. Second, tPCA automatically
extracts a discrete set of temporal patterns. Third, when im-
plemented and graphed appropriately, tPCA results are eas-
ily interpreted with respect to previous findings, as illus-
trated below. tPCA is therefore easily incorporated in an
automated process for ERP pattern extraction and classifi-
cation. In the final section, we address some limitations of
tPCA as a method of ERP pattern analysis.
2.2.2.Measuregeneration(process4)
For each tPCA factor, we extracted 32 summary metrics that
characterize spatial, temporal, and functional dimensions of
the data. The full set of metrics, along with their definitions,
is listed in Appendix C. Note that our expert-defined rules,
which were used for the tPCA autolabeling process, mainly
involved two metrics (see Section 2.2.3 for details): In-mean
(ROI) and TI-max. In-mean (ROI) represents the amplitude
over a region-of-interest (ROI),averagedover electrode clus-
ters for each latent factor at the time of peak latency, after the
factor has been projected back into channel space. TI-max
is the peak latency and is measured on the factor loadings,
which are sign-invariant.
Although these two metrics intuitively capture the spa-
tial and temporal dimensions of the ERP data that are most
salient to ERP researchers, our prior data mining results sug-
gested that additional metrics might improve the tPCA au-
tolabeling results [22, 23]. In particular, some failures in the
autolabeling process (i.e., cases where the modal factor for
a given pattern did not show a match to the rule in a given
condition, for a given subject) were due to component over-
lap that remained even after tPCA. For example, in one of
our four pilot datasets [23], the P100 pattern was partially
captured by a factor corresponding to the N100. For some
subjects, most of the P100 was in fact captured by this “N100
factor.”Thefactorshowedaslownegativity,beginningbefore
the stimulus onset, and the P100 appeared as a positive going
deflection that was superposed on this sustained negativity.
However, because the rule specified that the mean amplitude
overtheoccipitalelectrodesshouldbepositive,thefactordid
not meet the P100 rule criteria.
To address this issue, we implemented onset and offset
metrics. Each onset latency was estimated as the midpoint of
four consecutive sliding windows in which corresponding t-
tests(threshold,P = .05)indicatedthatthemeansoftheirre-
spectivewindowedsignalsdivergedsignificantlyfromabase-
line value, typically zero. The subsequent offset was the tem-
poral midpoint at which the four consecutive t-tests showed
their windowed signal means returned to baseline. The pro-
cedure is implemented as described in [24].
Using the onset latency to determine a “baseline” (0-
point or onset) for each pattern, we then computed peak-to-
baseline and baseline-to-peak metrics to capture phasic de-
flections that could be confused with slow potentials. The
baseline intensity was computed as the signal mean within
an interval centered on component onset. We predicted that
data mining results would incorporate these measures to
yield improved accuracy in the labeling process.
In addition, we added metrics to capture variations in
amplitude due to experimental variables. Four measures
were computed: Pseudo-Known (difference in response to
nonwords versus words), RareMisses-RareHits (difference in
response to unknown rare words versus words that we cor-
rectly recognized), RareHits-Known (difference in response
to rare versus low-frequency words), and Pseudo-RareMisses
(difference in nonwords versus missed rare words). Because
prior research has shown that semantic processing can affect
the N2, N3, MFN, N4, and P3 patterns, we predicted that the
data mining procedures would identify one or more of these
metrics as important for pattern classification.
2.2.3.Ruleoperationalization(process5)
RulesforeachERPpatternwereformulatedinitiallybasedon
results from prior literature and were operationalized using
metricsdefinedinProcess4(Section 2.2.2).Afterapplication
oftheinitialrulestotestdata,weevaluatedtheresultsagainst
a “Gold Standard” (see Section 4 for details) and modified
the pattern rules to improve accuracy. For example, after ini-
tial testing, the visual “P100” pattern (P100v) was defined as
follows: for any n, FAn= P100v if and only if
(i) 80ms < TI-max (FAn) ≤ 150 milliseconds,
(ii) In-mean(ROI) > 0,
(iii) EVENT (FAn) = stimon,
(iv) MODALITY (EVENT) = visual,
where FAnis defined as the nth tPCA factor, and P100v is
the visual-evoked P100 (“v” stands for “visual”). TI-max is
the time of peak amplitude, In-mean(ROI) is the mean am-
plitude over the region-of-interest (ROI), and ROI for P100v
is specified as “occipital” (i.e., mean intensity over occipital
electrodes). “Stimon” refers to stimulus onset, which is the
event that is used for time-locking single trials to derive the
ERP. “MODALITY” refers to the stimulus modality (e.g., vi-
sual,auditory,somatosensory,etc.).SeeAppendix Bforafull
listing of rule formulae.