Content uploaded by Kurtulus Izzetoglu
Author content
All content in this area was uploaded by Kurtulus Izzetoglu on Mar 16, 2015
Content may be subject to copyright.
Content uploaded by Scott Bunce
Author content
All content in this area was uploaded by Scott Bunce on Jan 18, 2013
Content may be subject to copyright.
38
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
OPTICAL IMAGING
0739-5175/07/$25.00©2007IEEE
I
n the last decade, functional near-infrared spectroscopy
(fNIR) has been introduced as a new neuroimaging modali-
ty with which to conduct functional brain imaging studies
[1]–[24]. fNIR technology uses specific wavelengths of
light, irradiated through the scalp, to enable the noninvasive
measurement of changes in the relative ratios of deoxygenated
hemoglobin (deoxy-Hb) and oxygenated hemoglobin (oxy-Hb)
during brain activity. This technology allows the design of
portable, safe, affordable, noninvasive, and minimally intru-
sive monitoring systems. These qualities make fNIR suitable
for the study of hemodynamic changes due to cognitive and
emotional brain activity under many working and educational
conditions, as well as in the field.
Functional imaging is typically conducted in an effort to
understand the activity in a given brain region in terms of its rela-
tionship to a particular behavioral state or its interactions with
inputs from another region’s activity. The program of research in
cognitive neuroscience conducted by our optical brain imaging
group has utilized the current-generation fNIR system to investi-
gate brain activity, primarily in the dorsolateral and inferior
frontal cortex [20]–[24]. To date, the fNIR studies of cognition
and emotion have focused on functions associated with
Brodman’s areas BA9, BA10, BA46, BA45, BA47, and BA44.
Recent positron emission tomography (PET) and functional
magnetic resonance (fMRI) studies have shown that these areas
play a critical role in sustained attention, both the short-term stor-
age and the executive process components of working memory,
episodic memory, problem solving, response inhibition, and the
perception of smell (for a recent review, see [25] and [26]). In
addition, word recognition and the storage of verbal materials
activate Broca’s area and left hemisphere supplementary and
premotor areas [25], [27], [28]. To date, studies utilizing fNIR
have shown results consistent with fMRI and PET findings for
working memory and sustained attention [21]–[23].
In this article, we will describe the working principles of
fNIR and how the hemodynamic signals are extracted from
the raw fNIR measurements using the modified Beer-Lambert
Law. We will also introduce the fNIR system that we have
developed and used in our studies. Current results from the
augmented cognition research conducted in our laboratory are
also presented, and the merits of optical imaging in augmented
cognition are summarized.
Working Principles
Typically, an optical apparatus consists of a light source by
which the tissue is radiated and a light detector that receives
light after it has interacted with the tissue. Photons that enter
tissue undergo two different types of interaction, namely
absorption (loss of energy to the medium) and scattering [4],
[5], [19]. Most biological tissues are relatively transparent to
light in the near-infrared range between 700 to 900 nm, which
is usually called the “optical window.” This is mainly due to
the fact that within this optical window, the absorbance of the
main constituents in the human tissue (i.e., water, oxy-Hb, and
deoxy-Hb) is small, allowing the light to penetrate the tissue
(see Figure 1).
Among the main absorbers (chromophores) in the tissue,
oxy- and deoxy-Hb are strongly linked to tissue oxygenation
and metabolism. Fortunately, in the optical window, the
absorption spectra of oxy- and deoxy-Hb remain significantly
different than each other, allowing spectroscopic separation of
these compounds to be possible using only a few sample
wavelengths.
fNIR technology employs specified wavelengths of light
within the optical window. Once the photons are introduced
into the human head, they are either scattered by extra- and
intracellular boundaries of different layers of the head (skin,
skull, cerebrospinal fluid, brain, etc.) or absorbed mainly by
oxy- and deoxy-Hb. A photodetector placed a certain distance
away from the light source can collect the photons that are not
absorbed and those that traveled along the “banana shaped
path” between the source and detector due to scattering [9],
[29] as shown Figure 2.
In functional optical brain imaging studies, the attenuation
(reduction in the amount of photons detected by the photode-
tectors) due to scattering is assumed to be constant since the
amount of scatterers within different layers of the head does
not change due to cognitive activity. The change in the attenu-
ation measured as a result of cognitive activity is hence due to
the changes in absorption resulting from the variation in the
concentrations of oxy- and deoxy-Hb in the brain tissue. This
relationship is not surprising, since cerebral hemodynamic
changes are related to functional brain activity through a
mechanism that is called neurovascular coupling [8], [30]. In
fact, this physiological relationship and the ability of fNIR
Functional Brain Imaging
Using Near-Infrared
Technology
Assessing Cognitive Activity in Real-Life Situations
BY MELTEM IZZETOGLU, SCOTT C. BUNCE,
KURTULUS IZZETOGLU, BANU ONARAL,
AND KAMBIZ POURREZAEI
© BRAND X PICTURES
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
technology to measure the concentration changes in the oxy-
gen-related chromophores make the functional optical brain
imaging possible.
According to the modified Beer-Lambert Law [2]–[4], [19],
the light intensity after absorption and scattering by the bio-
logical tissue is expressed as:
I = GE
o
e
−
(
α
HB
C
HB
+α
HBO
2
C
HBO
2
)
L
(1)
where
G
is a factor that accounts for the measurement geome-
try and is assumed constant when concentration changes.
I
o
is
input light intensity;
α
HB
and
α
HBO
2
are the molar extinction
coefficients;
C
HB
and
C
HBO
2
are the concentrations of chro-
mophores deoxy-Hb and oxy-Hb, respectively; and
L
is the
photon path that is a function of absorption and scattering
coefficients
µ
a
and
µ
b
.
By measuring optical density (OD) changes at two wave-
lengths, the relative change of oxy- and deoxy-hemoglobin
versus time can be obtained. If the intensity measurement
at the initial time (baseline) is
I
b
, and at another time is
I
,
the OD change due to variation in
C
HB
and
C
HBO
2
during
that period is:
OD = log
I
b
I
= α
HB
C
HB
+ α
HBO
2
C
HBO
2
.(2)
Measurements performed at two different wavelengths allow
the calculation of
C
HB
and
C
HBO
2
. Oxygenation and blood
volume can then be deduced:
Oxygenation = C
HBO
2
− C
HB
(3)
BloodVolume = C
HBO
2
+ C
HB
. (4)
Using this technique and these measures, several types of
brain functions have been assessed, including motor [10], [11]
and visual activation [15]; auditory stimulation [17]; and
performance of various cognitive tasks [21]–[23]. In our stud-
ies described in this article, we used the oxygenation data for
the assessment of different cognitive functions.
fNIR System
Three distinct types of fNIR implementation have been devel-
oped: time-domain, frequency-domain, and continuous wave
(CW) spectroscopy systems [3]–[5]. In time-domain systems,
also referred to as time-resolved spectroscopy (TRS), extreme-
ly short incident pulses of light are applied to the tissue and
the temporal distribution of photons that carry the information
about tissue scattering and absorption is measured. In frequen-
cy-domain systems, the light source is amplitude-modulated to
the frequencies in the order of tens to hundreds of megahertz.
The amplitude decay and phase shift of the detected signal
with respect to the incident light are measured to characterize
the optical properties of tissue [31].
In CW systems, light is applied to tissue at constant ampli-
tude. The CW systems are limited to measuring the amplitude
attenuation of the incident light [31]. However, CW systems
possess a number of advantageous properties that have result-
ed in wide use among researchers interested in brain imaging
relative to other near-infrared systems: it is minimally intru-
sive and portable, affordable, and easy to engineer relative to
frequency- and time-domain systems [31], [32]. Our research
team has been developing a CW fNIR system that lends itself
to both portable and wireless designs to monitor brain func-
tion under both laboratory and field conditions.
In the studies described throughout this article, we used the
portable CW-fNIR system that was originally described by
Chance et al. [32]. The main components of the system are: 1)
the sensor that covers the entire forehead, 2) a control box for
data acquisition (current sampling rate is 1.6 Hz), and 3) a
computer for the data analysis software [23], [24]. The wire-
less CW-fNIR system operating with different sampling rates
is currently under investigation.
The current flexible sensor consists of four light sources that
contain three built-in LEDs having peak wavelengths at 730,
805, and 850 nm and ten detectors designed to image cortical
areas underlying the forehead (dorsolateral and inferior frontal
cortices). With a fixed source-detector separation of 2.5 cm,
this configuration results in a total of 16 signal channels (vox-
els). Communication between the data analysis computer and
the task presentation computer is established via a serial port
connection to time-lock fNIR measurement to the task events.
The flexible sensor is a modular design consisting of two
parts: a reusable, flexible circuit board that carries the necessary
Fig. 1. Absorption spectrum in near-infrared (NIR) window.
Fig. 2. Photon path inside the human head.
Optical Window
0.30
0.25
0.20
0.15
0.10
0.05
0
600 700 800 900 1,000 1,100
Wavelength (nm)
Aesorption Factor (cm
−1
)
Deoxy-Hb
Oxy-Hb
Water
Source
Detector
39
40
infrared sources and detectors, and a disposable, single-use
cushioning material that serves to attach the sensor to the partic-
ipant (see Figure 3) [21]. The flexible circuit provides a reliable,
integrated wiring solution, as well as consistent and repro-
ducible component spacing and alignment. Because the circuit
board and cushioning materials are flexible, the components
move and adapt to the contours of the participant’s head, allow-
ing the sensor elements to maintain an orthogonal orientation to
the skin surface, improving light coupling efficiency and signal
strength. We are currently working on a modular sensor design
that will be scalable to any forehead shape and size, including
adult and infants, allowing the adjustment of the sources and
detectors according to the international 10–20 system.
fNIR Studies on Augmented Cognition
In all our fNIR studies presented in this section, participants
have signed informed consent statements approved by the
Human Subjects Institutional Review Board at Drexel
University.
Cognitive Performance Measurement Study
In this study, we present the deployment and statistical analy-
sis of fNIR for the purpose of cognitive state assessment while
the user performs a complex task [21]. This work is based on
data collected during the Defense Advanced Research Projects
Agency (DARPA) Augmented Cognition-Technical
Integration Experiment session participated by a total of eight
healthy subjects (three females and five males), ranging in age
from 18 to 50.
The experimental protocol for this session used a complex
task resembling a videogame called the Warship Commander
Task (WCT). The WCT was designed and developed by
Pacific Science and Engineering Group under the direction of
the Space and Naval Warfare Systems Center to simulate
naval air warfare management [33]. A sample screen shot dur-
ing WCT is shown in Figure 4.
Task load and task difficulty were manipulated by changing
the number of airplanes that had to be managed at a given time
(six, 12, 18, and 24 plane “waves”), the number of unknown
versus known airplane identities (two levels of difficulty, low:
33% of the planes were unknown, and high: 67% of the planes
were unknown), and the presence or absence of a verbal mem-
ory task (a secondary task causing divided attention).
For this study, each participant completed four sets of
WCT. Each set was comprised of three repetitions of each of
the four wave sizes (in the order of six, 18, 12, and 24 planes),
where each wave lasted 75 s. The factors of four different
wave sizes, two different task difficulties (high versus low per-
centage of unknown airplanes), and full versus divided atten-
tion (secondary verbal memory task “on” or “off”) were
crossed to create a
4 × 2 × 2
repeated-measures design.
The fNIR measurements are first cleaned from motion arti-
facts [21], then for each wave of 75 s, the rate of change in
the oxygenation was calculated from the fNIR measurements
relative to the baseline collected during the rest period before
the protocol had started. Finally, blood oxygenation values
were averaged across eight voxels covering left and right
hemispheres.
The fNIR data analysis explored the relationships among
cognitive workload, the participant’s performance and
changes in blood oxygenation levels of the dorsolateral pre-
frontal cortex, and the effect of divided attention as elicited by
the secondary component of the WCT (the auditory task). Our
primary hypothesis was that blood oxygenation in the pre-
frontal cortex, as assessed by fNIR, would rise with increasing
task load and would exhibit a positive correlation with perfor-
mance measures. In support of our primary hypothesis, the
results indicated that the rate of change in blood oxygenation
was significantly sensitive across both hemispheres
(F = 16.24, p < 0.001)
to task load (wave size) changes [see
Figure 5(a) when secondary verbal was off].
When attention is divided by the secondary verbal task, the pri-
mary effect occurred in the 24-plane wave (the most difficult
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
Fig. 3. (a) Flexible sensor. (b) Participant wearing flexible
sensor.
Fig. 4. A snapshot during WCT where air warfare manage-
ment required the user to monitor “waves” of incoming air-
planes, to identify the identity of the unknown planes
(yellow) as friendly (blue) or hostile (red), and to warn and
then destroy hostile airplanes using rules of engagement.
(a)
(b)
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
condition) causing the mean oxygenation for this case to drop
below that of the 18-plane wave [see Figure 5(a)]. In line with the
stated hypothesis, a preliminary interpretation of this finding was
that a number of participants had reached their maximal level of
performance in this most difficult task level and lost their concen-
tration/effort, resulting in a drop in blood oxygenation.
The hypothesis also predicts that individuals who were able
to stay on task and continue to perform in this difficult condi-
tion should demonstrate increased oxygenation relative to both
their own oxygenation levels in the 18-plane wave and indi-
viduals who became overwhelmed and disengaged. Because
sustained concentration and engagement in the task should
result in increased performance, a positive correlation between
performance and blood oxygenation would provide support
for this interpretation. A Pearson’s product-moment correla-
tion indicated a very strong positive relationship between
blood oxygenation and performance in the 24-plane condition
[Pearson’s
r = 0.89, p = 0.003
; see Figure 5(b)].
A median split on the Percentage Game Score provided fur-
ther evidence of the hypothesized relationship between cogni-
tive effort and the blood oxygenation response. As can be seen
in Figure 5(c), the mean levels of oxygenation were higher for
both high and low performers in the 24-plane wave than the
18-plane wave when the secondary verbal task was off.
However, when the secondary verbal task was on, for the
more difficult condition, the individuals who performed well
on the 24-plane wave showed a higher mean level of oxygena-
tion for the 24-plane wave than for the 18-plane wave, where-
as those who performed poorly showed a decrease in
oxygenation relative to the 18-plane wave.
Working Memory Assessment Study
In order to assess the working memory, we used the n-back
task, which is a sequential letter task with varied workload
conditions that has frequently been used in working memory
studies by cognitive psychologists and neuroscientists [26],
[27]. The stimuli are single consonants presented centrally, in
pseudorandom sequences, on a computer monitor. Stimulus
duration is 500 ms, with a 2,500-ms interstimulus interval.
Four conditions were used to incrementally vary working
memory load from zero to three items. In the 0-back condi-
tion, subjects respond to a single prespecified target letter
Fig. 5. (a) Mean oxygenation change versus wavesize (n=8) by secondary verbal task. (b) Pearson’s correlation between
performance and oxygenation change. (c) Mean oxygenation change as a function of wavesize, secondary verbal task and
average Percentage of Game Score (from [21]).
Oxygenation Change
2nd Verbal On
2nd Verbal Off
12
10
8
6
4
2
0
−2
−4
−6
−8
Low Performance High
6121824 6
12
18 24
8
6
4
2
0
−2
−4
−6
−8
Oxygenation Change
Wavesize
6121824
2nd Verbal
High
2nd Verbal
Off
(a)
Oxygenation Change
20
10
0
−10
08 10 12 14 16 18 20 22
% of Total Possible Score
Rsq − 0.7899
(b)
(c)
These studies indicate that human performance
and cognitive activities such as attention,
working memory, problem solving, etc.,
can be assessed by fNIR technology.
41
42
(e.g., “X”) with their dominant hand (pressing a button to
identify the stimulus). In the 1-back condition, the target is
defined as any letter identical to the one immediately preced-
ing it (i.e., one trial back). In the 2-back and 3-back condi-
tions, the targets were defined as any letter that was identical
to the one presented two or three trials back, respectively.
Subjects pressed one button for targets (approximately 33% of
trials) and another for nontargets. This strategy incrementally
increased working memory load from the 0-back to the 3-back
condition. Each
n
-back block contained 20 letters, whether tar-
get or nontarget, and lasted for 60 s with 15 s of rest periods
between
n
-back blocks. The total test included seven trials of
each of the four
n
-back conditions (hence, a total of 28
n
-back
blocks) ordered in such a way that within one trial all four of
the
n
-back conditions are presented; however, their order is
changed randomly from trial to trial.
In the analysis performed on nine subjects (with age
between 18 and 25), filtering to eliminate physiologically
irrelevant data (such as respiration and heart pulsation effects)
and equipment noise, etc., is carried out on the raw fNIR
measurements. Then, for each
n
-back condition out of seven
trials, outliers are eliminated and the resulting trials are aver-
aged for each voxel. Once the oxygenation data is obtained
using the modified Beer-Lambert Law on these averaged raw
data, the overall mean for each
n
-back block is calculated and
used as a feature for comparison purposes.
Statistically significant differences between the n-back con-
ditions are obtained on the fourth-voxel fNIR measurements.
The location of the fourth-voxel measurement registered on
the brain surface is as shown in Figure 6(a). (Interested readers
can find a more detailed explanation of our fNIR data registra-
tion and visualization scheme in [34].) This result is in agree-
ment with the fMRI literature [28].
Statistical analysis revealed that the 0-back condition dif-
fered from 1-, and 2-back conditions; 1-back
>
0-back,
t = 3.21
,
p = 0.012
; 2-back
>
0-back,
t = 2.58
,
p = 0.032
.
The 1- to 2-back did not differ from each other at first.
However, we noticed that in the 2-back condition subject 5
had performed the worst compared to the others [as shown in
Figure 6(b)], which was more than 1.5 standard deviation
away from the overall mean. When the data of subject 5 are
treated as an outlier and eliminated from the analysis, 1- and
2-back differed from each other; 2-back
>
1-back,
t = 2.77
,
p = 0.0275
. No difference was found between 2- to 3-back
conditions.
Mean oxygenation data for the nine subjects’ individual
and averaged mean oxygenation data across nine subjects for
each workload condition are presented in Figure 6(c) and
Figure 6(d), respectively. A positive relationship between
increasing workload and the oxygenation is observed in dor-
solateral prefrontal cortex, again in agreement with fMRI
studies [28]. The drop in the oxygenation values in the most
difficult condition (3-back) can be interpreted using a hypoth-
esis similar to the human performance study discussed previ-
ously, where subjects get overwhelmed and lose their
concentration.
Problem Solving Study Using a Novel Single-Trial
Hemodynamic Response Extraction Method
Ongoing studies in problem solving of graded difficulties
(anagram solution) using both block and event-related (ER)
anagram protocols reveal that fNIR measurement of metabolic
activation and blood flow can be valuable as an educational
aid [35], [36]. In ER anagram study, subjects are presented an
anagram for 1 s and given 15 s to solve it until the next presen-
tation. This procedure allows the hemodynamic response to
fully evolve, which has been shown in the literature to take
10–12 s [36], [37]. ER studies provide insights to model the
hemodynamic response and are used widely in the assessment
of cognitive activation in different regions of the brain for
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
Fig. 6. N-back test results: (a) Imaging area of the brain. (b) Performance of all subjects on 2-back condition. (c) Mean oxy-
genation across subjects. (d) Averaged mean oxygenation for all subjects.
(a)
Performance on 2-Back Condition Across Subjects
1.200
1.000
0.800
0.600
0.400
0.200
0.000
1
23
45
6
7
8
9
(b)
Mean Oxygenation Across Subjects (n=9)
0.500
0.400
0.300
0.200
0.100
−0.100
−0.200
−0.300
−0.400
−0.500
−0.600
0.000
123456789
0-Back
1-Back
2-Back
3-Back
(c)
Averaged Mean Oxygenation for Nine Subjects
0.150
0.100
0.050
0.000
−0.050
−0.100
−0.150
−0.200
1
0-Back
1-Back
2-Back
3-Back
(d)
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
different task loads. However, in such studies, the protocol
time is long and they do not reflect real-world situations.
In block anagram study, subjects are shown as many ana-
grams as they can solve within 1-minute periods. Whenever
subjects solve the anagram, they press a certain button, which
results in immediate presentation of the next anagram. Since
most subjects solve the anagram within 2–5 s, the hemody-
namic responses overlap in time, which present challenges for
data analysis. Until now, in block anagram studies, it was not
possible to evaluate the subject’s response times or brain acti-
vation for single anagram presentation within a block for
graded difficulty analysis.
We developed a novel single-trial hemodynamic response esti-
mation algorithm and applied it to the block anagram solution
study to extract evoked responses to single anagram presentations
within each block [36]. Each ER hemodynamic response was
estimated on the basis of two postulates: 1) that each single-trial
hemodynamic response follows a gamma function,
hf
i
= A
i
t
α
i
i
e
β
i
t
i
as given in Figure 7(a), and 2) that the total oxy-
genation data can be modeled by the summation of individual
hemodynamic responses evoked by rapidly presented stimuli,
Oxy =
N
i =1
hf
i
. Each single trial was estimated by optimizing
the error between the total oxygenation data from fNIR measure-
ments and the linear model:
ε = min
A,α,β
(Oxy −
N
i =1
hf
i
)
2
.
The protocol we have used in this study involved presenta-
tion of anagram blocks on a computer screen that contains
sequences of three-letter (3L), four-letter (4L), and five-letter
(5L) anagrams starting from minimal (3L anagrams) proceed-
ing to the maximal level of difficulty (5L anagrams) and then
back down again to the starting point of 3L anagrams.
Between each anagram block session, there is a rest period of
30 s. Each anagram block is displayed for approximately one
minute, containing as many anagrams within it depending on
the number of processed anagrams by the subject. The deci-
sion of the subjects on each anagram processed and its timing
is recorded on a text file for further analysis.
All calculations are applied to the data gathered from the
left hemisphere of the prefrontal cortex on voxel 5 [location
shown in Figure 7(b)]. In a block anagram study based on 14
participants (age 18 to 23), the averaged recorded behavioral
response times, the extracted rise times or time to peak (min),
and the maximum amplitudes from the estimated evoked
hemodynamic responses with respect to the 3L, 4L, and 5L
anagram sets are presented in Figure 7(c).
Fig. 7. (a) A typical gamma function. (b) Imaging area of the brain for this study. (c) Subject averages of rise and response
times (min) and maximum amplitudes. (d) Scatter plot of rise time versus response time averages. (e) Scatter plot of maximum
amplitude versus response time averages for all anagram sets for all subjects (from [36]).
8
7
6
5
4
3
2
1
0
−1
−2
−3
01020304060
FWHM
Amplitude
Time to Peak
(a)
All Anagram Data, R=0.73
Maximum Amplitude
0.400
0.350
0.300
0.250
0.200
0.150
0.100
0.050
0.000
0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000
Response Time (s)
(e)
All Anagram Data, R=0.94
10.000
9.000
8.000
7.000
6.000
5.000
4.000
3.000
2.000
1.000
0.000
0.000
1.000
2.000 3.000
4.000 5.000
6.000
7.000
8.000
Rise Time (s)
Response Time (s)
(d)(c)
0.35
0.25
0.15
0.05
0.1
0.3
0.2
0
3L 1st Set 4L 1st Set 4L 2nd Set 3L 2nd Set5L
Average Response and Rise Times (min) and Maximum Amplitude of the
Single Trial Hemodynamic Response Estimates
Ave Resp Time
Ave Rise Time
Ave Max Time
(b)
43
44
It can be clearly seen that the estimated rise time, which is
the time required for the evoked hemodynamic response to
reach its maximum amplitude, follows the same pattern as the
behavioral (true) response time of the subjects having a corre-
lation of
R = 0.94
as presented in the scatter plot of the rise
time versus response time in Figure 7(d). Also, the estimated
maximum amplitudes are correlated with the true response
times
(R = 0.73)
as given in Figure 7(e).
The rise times and the maximum amplitude values increase
as the difficulty level of the anagram solution increases, mean-
ing that subjects need more time and more oxygen to solve
difficult anagrams. Estimation of the ER signals in a block
design allows more precise analysis of the brain’s function
during a cognitive/problem solving task.
Attention Measurement Study:
A Combined EEG and fNIR Study
In this last study we demonstrate the utility of the combined
EEG-fNIR system for studies of ER designs that tap into ubiq-
uitous cognitive functions such as attention [38]. The protocol
we have used to measure attention is a common visual oddball
paradigm modified for use with fMRI by McCarthy et al. [39].
The stimuli were two strings of white letters (XXXXX and
OOOOO) presented against the center of a dark background.
A total of 516 stimuli were presented, 480 context stimuli
(OOOOO) and 36 targets (XXXXX). Stimulus duration was
500 ms, with an interstimulus interval of 1,500 ms. Target
stimuli were presented randomly with respect to context stim-
uli with a minimum of 12 context stimuli between successive
targets to allow the hemodynamic response an opportunity to
return to baseline between target presentations.
Fifteen right-handed participants (four females and 11
males, age
20.8 ± 4.2
) were required to press one of two but-
tons on a response pad after each stimulus while both fNIR
and EEG were recorded simultaneously. One button was
pressed in response to targets (Xs), and another button was
pressed in response to context stimuli (Os).
The results for the event-related brain potentials (ERPs)
were consistent with the literature [40]; targets elicited a pro-
nounced P3 component with an average peak at 365 ms for
both electrodes Cz and Pz [see Figure 8(a) and Figure 8(b)].
The peak amplitude response to target stimuli was larger than
the response to context stimuli at both Cz (
t(14) = 7.58
;
p < 0.001
) and Pz (
t(14) = 7.81
;
p < 0.001
). These ERP
results confirm that the task parameters and participant
responses were comparable to other ERP studies.
Raw fNIR data was low-pass filtered at a frequency of
0.14 Hz in order to eliminate respiration and heart pulsation
effects. The continuous oxygenation data were then segmented
into 24-s-long epochs, with a prestimulus window of 9 s (six
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
Fig. 8. Averaged ERP and fNIR data for targets and contexts: (a) and (b) ERP’s from Cz and Pz electrode, respectively. (c) fNIR
data on voxel 11. (d) Location of the significant difference in fNIR measurements between target and context from ([38]).
25
20
15
10
5
0
−5
−10
−200
0 200 400 600 800 1,000
Time (ms)
Voltage (µV)
Target
Context
(a)
25
20
15
10
5
0
−5
−10
−200
0 200 400 600 800 1,000
Time (ms)
Voltage (µV)
Target
Context
(b)
0.015
0.01
0.005
0
−0.005
−0.01
Target
Context
Oxygenation (m)
−2
024681012
Time (s)
(c)
(d)
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
45
stimuli) and a poststimulus window of 15 s (nine stimuli).
These epochs were baseline corrected by subtracting the mean
of the baseline from the waveform and then outliers were
eliminated for each voxel. The remaining epochs were aver-
aged for the target and the context stimuli separately.
Repeated-measure ANOVA computed on the fNIR oxy-
genation data revealed that oxygenation values were greater
in response to targets than to controls in voxel 11, located
over the middle frontal gyrus of the right hemisphere [see
Figure 8(c)]. Differentiation occurred between 6 and 9 s post-
stimulus [see Figure 8(d)]. These results are consistent with
the fMRI literature for visual target categorization with
respect to increased oxygenation in response to targets, corti-
cal location, and time course [39], [41].
Conclusions
The use of fNIR technology has increased in recent years as a
means to measure hemodynamic changes in the cortex in
response to cognitive activity. Moreover, it is a noninvasive
and negligibly intrusive optical imaging modality. fNIR
instrumentation allows for safe, portable, and low-cost cortical
monitoring that can be applied in the laboratory as well as
field conditions. This article provided an overview of cogni-
tive studies carried out in our laboratory. These studies
indicate that human performance and cognitive activities such
as attention, working memory, problem solving, etc., can be
assessed by fNIR technology. Our findings are in agreement
with the results in current EEG and fMRI literature. We have
demonstrated that fNIR technology can be integrated with the
ERPs collected simultaneously with fNIR for better data clas-
sification. Such integration benefits from the high temporal
resolution characteristic of ERPs and better spatial resolution
characteristic of fNIR. We have automated signal processing
algorithms to solve problems that arise due to the time-scale
disparity between EEG and fNIR signals. Our studies suggest
that fNIR is a promising new technology for the study of cog-
nitive activity.
Acknowledgments
We would like to thank Dr. Shoko Nioka and Dr. Britton
Chance for their valuable research guidance in general and
for supplying in particular the anagram data. This work has
been sponsored in part by funds from the DARPA
Augmented Cognition Program, the Office of Naval
Research, and the Office of Homeland Security under agree-
ment numbers N00014-02-1-0524, N00014-01-1-0986, and
N00014-04-1-0119.
Meltem Izzetoglu was born in Samsun,
Turkey in 1971. She received the B.S.
and the M.S. both in electrical and elec-
tronics engineering from Middle East
Technical University, Ankara, Turkey, in
1992 and 1995, respectively. She re-
ceived the Ph.D. in electrical and comput-
er engineering from Drexel University,
Philadelphia, Pennsylvania, in 2002. She is currently a
research assistant professor in School of Biomedical
Engineering, Science, and Health Systems at Drexel
University. Her research interests include biomedical signal
analysis, adaptive and optimal signal processing, biomed-
ical optics, and scale-space processing tools.
Scott C. Bunce is an assistant professor of
psychiatry and director of the Clinical
Neuroscience Research Unit at the Drexel
University College of Medicine. He has
considerable experience in both clinical
and individual differences research. His
areas of expertise are in affective neuro-
science, theory of mind, and the effects of
psychological trauma on information processing. A major
focus in recent years has been on using neurophysiological
measures (EEG, ERPs, ERD, EMG) to assess information
and emotional processing that cannot be reported by the
patient/participant. He has also played an integral role in the
development of a safe, portable, near-infrared optical imag-
ing device for the assessment of hemodynamic changes dur-
ing cognitive and emotional tasks.
Kurtulus Izzetoglu gained his profession-
al software development and medical imag-
ing experience as a member of consulting
companies in the United States and the
Netherlands, respectively. In these posi-
tions, he worked as a senior analyst as well
as a software and analytical applications
developer. His experiences include the
development of professional medical imaging software pack-
ages, implementation of quantitative analysis, and imaging
techniques. Subsequent to five years of industrial experience,
he joined the functional optical imaging research team at
Drexel University where he currently serves as the project
engineer. His technical management responsibilities include
development of the Cognitive Workload Assessment Testing
and Analysis Platform and signal processing and experimental
protocol design and implementation. He received his M.S.E.E.
from Middle East Technical University in Ankara, Turkey.
Banu Onaral, H.H. Sun Professor of
Biomedical and Electrical Engineering,
received her Ph.D. from the University of
Pennsylvania in 1978 and her B.S.E.E. and
M.S.E.E. from Bogazici University in
Istanbul, Turkey. Her academic focus, both
in research and teaching, is centered on bio-
medical signals and systems engineering.
She has been a founding member of the Biomedical
Information Technology Laboratory, Scaling Signals and
Systems Laboratory, and the Bio-Electrode Research
Laboratory. She has led several curriculum development ini-
tiatives including the undergraduate telecommunication and
biomedical engineering programs. She has developed several
signals and systems engineering software products and was
recognized by the EDUCOM/NCRIPTAL Best Educational
Tool Award. She is the recipient of a number of faculty excel-
lence awards including the 1990 Lindback Distinguished
Teaching Award of Drexel University. Her professional ser-
vices include chair and membership on advisory boards and
strategic planning bodies of several universities and funding
agencies, including service on the NSF Engineering Advisory
Board (1997–1999) and on the proposal review panels and
study sections. Her editorial responsibilities have included ser-
vice on the editorial board of journals and the CRC
Biomedical Engineering Handbook as section editor for the
46
“Biomedical Signal Analysis” section. She has been active in
professional society leadership, in particular national and
international technical meeting organization; she served as
vice president for conferences and as president of the IEEE
Engineering in Medicine and Biology. She also served on the
inaugural board of the American Institute for Medical and
Biological Engineering. She is a Fellow of the IEEE; founding
Fellow of the American Institute for Medical and Biological
Engineering (AIMBE), Fellow of the American Association
for the Advancement of Science (AAAS), senior member of
the Society of Women Engineers (SWE), member of the
American Society for Engineering Education (ASEE), and the
Sigma Xi scientific research society.
Kambiz Pourrezaei received his B.S.
from Tehran University and M.S. from
Tufts University. He earned his Ph.D.
from Rensselaer Polytechnic Institute in
1982. He is currently a professor with the
School of Biomedical Engineering,
Science, and Health Systems at Drexel
University. He has active research pro-
grams in the areas of bio-nanotechnology and bio-optics.
Currently he is the co-director of the Nanotechnology
Institute in Philadelphia, Pennsylvania.
Address for Correspondence: M. Izzetoglu, Drexel
University, School of Biomedical Engineering, Science and
Health Systems, 3141 Chestnut Street, Philadelphia, PA
19104. E-mail: meltem@cbis.ece.drexel.edu.
References
[1] F.F. Jobsis, “Noninvasive infrared monitoring of cerebral and myocardial suffi-
ciency and circulatory parameters,” Science, vol. 198, no. 4323, pp. 1264–1267,
1977.
[2] M. Cope and D.T. Delpy, “System for long-term measurement of cerebral
blood flow and tissue oxygenation on newborn infants by infra-red transillumina-
tion,” Med. Biol. Eng. Comput., vol. 26, no. 3, pp. 289–294, 1988.
[3] Q. Luo, S. Zeng, B. Chance, and S. Nioka, “Monitoring of brain activation with
near infrared spectroscopy,” in Handbook of Optical Biomedical Diagnostics, V.V.
Tuchin, Ed. SPIE Press Monograph PM107.
[4] P. Rolfe, “In vivo near-infrared spectroscopy,” Annu. Rev. Biomed. Eng., vol. 2,
pp. 715–754, 2000.
[5] G. Strangman, D.A. Boas, and J.P. Sutton, “Non-invasive neuroimaging using
near-infrared light,” Biol. Psychiatry, vol. 52, no. 7, pp. 679–693, 2002.
[6] Y. Hoshi and M. Tamura, “Dynamic multichannel near-infrared optical imag-
ing of human brain activity,” J. Applied Physiol., vol. 75, no. 4, pp. 1842–1846,
1993.
[7] Y. Hoshi, “Functional near-infrared optical imaging: Utility and limitations in
human brain mapping,” Psychophysiology, vol. 40, no. 4, pp. 511–520, 2003.
[8] A. Villringer, J. Planck, C. Hock, L. Schleinkofer, and U. Dirnagl, “Near
infrared spectroscopy (NIRS): A new tool to study hemodynamic changes during
activation of brain function in human adults,” Neurosci. Lett., vol. 154, no. 1–2,
pp. 101–104, 1993.
[9] A. Villringer and B. Chance, “Non-invasive optical spectroscopy and imaging
of human brain function,” Trends Neurosci., vol. 20, no. 10, pp. 435–442, 1997.
[10] T. Suto, M. Ito, T. Uehara, I. Ida, M. Fukuda, and M. Mikuni, “Temporal
characteristics of cerebral blood volume change in motor and somatosensory cor-
tices revealed by multichannel near infrared spectroscopy,” Int. Congress Series,
vol. 1232, pp. 383–388, Apr. 2002.
[11] A. Maki, Y. Yamashita, Y. Ito, E. Watanabe, Y. Mayanagi, and H. Koizumi,
“Spatial and temporal analysis of human motor activity by using noninvasive NIR
topography,” J. Neurosci., vol. 11, no. 12, pp. 1458–1469, 1995.
[12] E. Gratton, V. Toronov, U. Wolf, M. Wolf, and A. Webb, “Measurement of
brain activity by near infrared light,” J. Biologic. Opt., vol. 10, no. 1, p. 011008-1-
13, 2005.
[13] M.A. Franceschini and D.A. Boas, “Noninvasive measurement of neuronal activ-
ity with near-infrared optical imaging,” Neuroimage, vol. 21, no. 1, pp. 372–386,
2004.
[14] G. Gratton, P.M. Corballis, E. Cho, M. Fabiani, and D.C. Hood, “Shades of
gray matter: Noninvasive optical images of human brain responses during visual
stimulation,” Psychophysiology, vol. 32, no. 5, pp. 505–509, 1995.
[15] H.R. Heekeren, H. Obrig, R. Wenzel, K. Eberle, J. Ruben, K. Villringer, R.
Kurth, and A. Villringer, “Cerebral haemoglobin oxygenation during sustained
visual stimulation - A near infrared spectroscopy study,” Philos. Trans. R. Soc.
Lond. B Biol. Sci., vol. 352, no. 1354, pp. 743–750, 1997.
[16] H. Sato, T. Takeuchi, and K. Sakai, “Temporal cortex activation during
speech recognition: An optical topography study,” Cognition, vol. 73, no. 3, pp.
B55–B66, 1999.
[17] P. Zaramella, F. Freato, A. Amigoni, S. Salvadori, P. Marangoni, and A.
Suppjei, “Brain auditory activation measured by near-infrared spectroscopy,”
Pediatric Res., vol. 49, no. 2, pp. 213–219, 2001.
[18] I.-Y. Son, M. Guhe, and B. Yazici, “Human performance assessment using
fNIR,” in Proc. SPIE, vol. 5797, pp. 158–169, 2005.
[19] M. Cope, “The development of a near-infrared spectroscopy system and its
application for noninvasive monitoring of cerebral blood and tissue oxygenation in
the newborn infant,” Ph.D. dissertation, Univ. College London, London, UK,
1991.
[20] S.C. Bunce, A. Devaraj, M. Izzetoglu, B. Onaral, and K. Pourrezaei,
“Detecting deception in the brain: A functional near-infrared spectroscopy study of
neural correlates of intentional deception,” in Proc. SPIE, vol. 5769, pp. 24–32,
2005.
[21] K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei, and B. Chance, “Functional
optical brain imaging using near-infrared during cognitive tasks,” Int. J. Human-
Comp. Interact., vol. 17, no. 2, pp. 211–227.
[22] K. Izzetoglu, G. Yurtsever, A. Bozkurt, B. Yazici, S. Bunce, K. Pourrezaei,
and B. Onaral, “NIR spectroscopy measurements of cognitive load elicited by
GKT and target categorization,” in Proc. 36th Hawaii Int. Conf. System Sciences,
Jan. 2003, pp. 129-130.
[23] M. Izzetoglu, K. Izzetoglu, S. Bunce, H. Ayaz, A. Devaraj, B. Onaral, and K.
Pourrezaei, “Functional near-infrared neuroimaging,” IEEE Trans. Neural Sys.
Rehab. Eng., vol. 13, no. 2, pp. 153–159, 2005.
[24] S.M. Platek, L.C.M. Fonteyn, M. Izzetoglu, T.E. Myers, H. Ayaz, C. Li, and
B. Chance, “Functional near infrared spectroscopy reveals differences in self-
other processing as a function of schizotypal personality traits,” Schizophrenia
Res., vol. 73, no. 1, pp. 125–127, 2005.
[25] R. Cabeza and L. Nyberg, “Imaging cognition II: An empirical review of 275
PET and fMRI studies,” J. Cogn. Neurosci., vol. 12, no. 1, pp. 1–47, 2000.
[26] E.E. Smith and J. Jonides, “Working memory: A view from neuroimaging,”
Cogn. Psychol., vol. 33, no. 1, pp. 5–42, 1997.
[27] E.E. Smith and J. Jonides, “Storage and executive processes in the frontal
lobes,” Science, vol. 283, Mar. 1999.
[28] T.S. Braver, J.D. Cohen, L.E. Nystrom, J. Jonides, E.E. Smith, and D.C. Noll,
“A parametric study of prefrontal cortex involvement in human working memory,”
NeuroImage, vol. 5, no. 1, pp. 49–62, 1997.
[29] G. Gratton, J.S. Maier, M. Fabiani, W.W. Mantulinm, and E. Gratton,
“Feasibility of intracranial near-infrared optical scanning,” Psychophysiology,
vol. 31, no. 2, pp. 211–215, 1994.
[30] H. Obrig and A. Villringer, “Beyond the visible-Imaging the human brain
with light,” J. Cereb. Blood Flow Metab., vol. 23, no. 1, pp. 1–18, 2003.
[31] D.A. Boas, M.A. Franceschini, A.K. Dunn, and G. Strangman, “Non-invasive
imaging of cerebral activation with diffuse optical tomography,” in In-Vivo Optical
Imaging of Brain Function, R. Frostig, Ed. Boca Raton, FL: CRC Press, pp.
193–221, 2002.
[32] B. Chance, E. Anday, S. Nioka, S. Zhou, L. Hong, K. Worden, C. Li, T.
Murray, Y. Ovetsky, D. Pidikiti, and R. Thomas, “A novel method for fast imaging
of brain function, non-invasively, with light,” Optics Exp., vol. 2, no. 10,
pp. 411–423, 1998.
[33] M. St. John, D.A. Kobus, and J.G. Morrison, “A multi-tasking environment
for manipulating and measuring neural correlates of cognitive workload,” in Proc.
IEEE 7th Human Factors and Power Plants Conf., 2002, pp. 7-10–7-14, 2002.
[34] H. Ayaz, M. Izzetoglu, S.M. Platek, S. Bunce, K. Izzetoglu, K. Pourrezaei,
and B. Onaral, “Registering fNIR data to brain surface image using MRI
templates,” in Proc. EMBS’06, Aug. 2006, pp. 2671-2674.
35] B. Chance, S. Nioka, S. Sadi, and C. Li, “Oxygenation and blood concentration
changes in human subject prefrontal activation by anagram solutions,” Adv. Exp.
Med. Biol., vol. 510, pp. 397–401, 2003.
[36] M. Izzetoglu, S. Nioka, B. Chance, and B. Onaral, “Single trial hemodynamic
response estimation in a block anagram solution study using fNIR spectroscopy,”
in Proc. ICASSP’05, 2005, pp. 633-636.
[37] F.M. Miezin, L. Maccotta, J.M. Ollinger, S.E. Petersen, and R.L. Buckner,
“Characterizing the hemodynamic response: Effects of presentation rate, sampling
procedure, and the possibility of ordering brain activity based on relative timing,”
NeuroImage, vol. 11, no. 6, pp. 735–759, 2000.
[38] S. Bunce, M. Izzetoglu, K. Izzetoglu, B. Onaral, and K. Pourrezaei,
“Functional near infrared spectroscopy: An emerging neuroimaging modality,”
IEEE Eng. Med. Biol. Mag, vol. 25, no. 4, pp. 54-62, 2006.
[39] G. McCarthy, M. Luby, J. Gore, and P. Goldman-Rakic, “Infrequent events
transiently activate human prefrontal and parietal cortex as measured by functional
MRI,” J. Neurophysiol., vol. 77, no. 3, pp. 1630–1634, 1997.
[40] J. Polich and A. Kok, “Cognitive and biological determinants of P300: An
integrative review,” Biol. Psychol., vol. 41, no. 2, pp. 103–146, 1995.
[41] B.A. Ardekani, S.J. Choi, G. Hossein-Zadeh, B. Porjesz, J.L. Tanabe, K.O.
Lim, R. Bilder, J.A. Helpern, and H. Begleiter, “Functional magnetic resonance
imaging of brain activity in the visual oddball task,” Cogn. Brain Res., vol. 14,
no. 3, pp. 347–356, 2002.
IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007
