ArticlePDF Available

Selective and Divided Attention Modulates Auditory-vocal Integration in the Processing of Pitch Feedback Errors

May 2015
European Journal of Neuroscience 42(3)

DOI:10.1111/ejn.12949

Source
PubMed

Project: Neural basis of auditory-motor integration for voice control

Authors:

Huijing Hu

Guang Dong Work Injury Rehabilitation Center

Jeffery A Jones

Wilfrid Laurier University

Show all 8 authorsHide

Speakers rapidly adjust their ongoing vocal productions to compensate for errors they hear in their auditory feedback. It is currently unclear what role attention plays in these vocal compensations. This event-related potential (ERP) study examined the influence of selective and divided attention on the vocal and cortical responses to pitch errors heard in auditory feedback regarding ongoing vocalizations. During the production of a sustained vowel, participants briefly heard their vocal pitch shifted up 2 semitones while they actively attended to auditory or visual events (selective attention), or both auditory and visual events (divided attention), or were not told to attend to either modality (control condition). The behavioral results showed that attending to the pitch perturbations elicited larger vocal compensations than attending to the visual stimuli. Moreover, ERPs were likewise sensitive to the attentional manipulations: P2 responses to pitch perturbations were larger when participants attended to the auditory stimuli compared to when they attended to the visual stimuli, and compared to when they were not explicitly told to attend to either the visual or auditory stimuli. By contrast, dividing attention between the auditory and visual modalities caused suppressed P2 responses relative to all the other conditions, and enhanced N1 responses relative to the control condition. These findings provide strong evidence for the influence of attention on the mechanisms underlying the auditory-vocal integration in the processing of pitch feedback errors. As well, selective attention and divided attention appear to modulate the neurobehavioral processing of pitch feedback errors in different ways. This article is protected by copyright. All rights reserved. This article is protected by copyright. All rights reserved.

Grand-averaged ERP waveforms in response to pitch perturbations of +200 cents across the four attention conditions. The red, blue, black and green solid lines denote the cortical responses in the Audition Attention, Visual Attention, Bimodal Attention and Bimodal Passive conditions, respectively. ERP responses are averages over electrodes FC1, FCz, FC2, C1, Cz and C2, across all subjects.

…

Topographical distributions of the N1 (top) and P2 (bottom) responses to pitch perturbations of +200 cents across the four attention conditions. From left to right are shown the responses in the Audition Attention, Visual Attention, Bimodal Attention and Bimodal Passive conditions, respectively.

…

Figures - uploaded by Hanjun Liu

Content may be subject to copyright.

Content uploaded by Hanjun Liu

Content may be subject to copyright.

Selective and divided attention modulates auditory–vocal

integration in the processing of pitch feedback errors

Ying Liu,

* Huijing Hu,

1,2,

* Jeffery A. Jones,

Zhiqiang Guo,

Weifeng Li,

Xi Chen,

Peng Liu

and Hanjun Liu

1,4

Department of Rehabilitation Medicine, The First Afﬁliated Hospital, Sun Yat-sen University, Guangzhou 510080, China

Guangdong Provincial Work Injury Rehabilitation Center, Guangzhou, China

Psychology Department and Laurier Centre for Cognitive Neuroscience, Wilfrid Laurier University, Waterloo, ON, Canada

Department of Biomedical Engineering, School of Engineering, Sun Yat-sen University, Guangzhou, China

Keywords: auditory feedback, auditory–motor integration, divided attention, selective attention, speech motor control

Abstract

Speakers rapidly adjust their ongoing vocal productions to compensate for errors they hear in their auditory feedback. It is cur-

rently unclear what role attention plays in these vocal compensations. This event-related potential (ERP) study examined the

inﬂuence of selective and divided attention on the vocal and cortical responses to pitch errors heard in auditory feedback regard-

ing ongoing vocalisations. During the production of a sustained vowel, participants brieﬂy heard their vocal pitch shifted up two

semitones while they actively attended to auditory or visual events (selective attention), or both auditory and visual events

(divided attention), or were not told to attend to either modality (control condition). The behavioral results showed that attending

to the pitch perturbations elicited larger vocal compensations than attending to the visual stimuli. Moreover, ERPs were likewise

sensitive to the attentional manipulations: P2 responses to pitch perturbations were larger when participants attended to the audi-

tory stimuli compared to when they attended to the visual stimuli, and compared to when they were not explicitly told to attend to

either the visual or auditory stimuli. By contrast, dividing attention between the auditory and visual modalities caused suppressed

P2 responses relative to all the other conditions and caused enhanced N1 responses relative to the control condition. These ﬁnd-

ings provide strong evidence for the inﬂuence of attention on the mechanisms underlying the auditory–vocal integration in the pro-

cessing of pitch feedback errors. In addition, selective attention and divided attention appear to modulate the neurobehavioral

processing of pitch feedback errors in different ways.

Introduction

Auditory feedback plays an important role in the acquisition and

maintenance of speech motor skills (Houde & Jordan, 1998; Hickok

et al., 2011). By exposing speakers to altered auditory feedback

(AAF) regarding their ongoing speech production, researchers have

demonstrated that speakers adjust their vocal output to compensate

for feedback perturbations in fundamental frequency (F

), vocal

intensity and formant frequency (Jones & Munhall, 2005; Bauer

et al., 2006; Liu et al., 2010; Cai et al., 2011; Scheerer et al.,

2013). Researchers have also shown that activity in the auditory cor-

tex is suppressed during active vocalisation relative to passive listen-

ing when the auditory feedback matches the actual vocal output

(Houde et al., 2002; Heinks-Maldonado et al., 2005; Flinker et al.,

2010; Behroozmand & Larson, 2011; Wang et al., 2014). By con-

trast, when there is a mismatch between auditory feedback and vocal

output, vocalisation elicits larger neural responses than during pas-

sive listening (Eliades & Wang, 2008; Behroozmand et al., 2009;

Chang et al., 2013; Chen et al., 2013). This enhanced cortical

activity is thought to reﬂect a mechanism that serves to compensate

for errors perceived during vocalisation (Chang et al., 2013).

Although recent research has greatly advanced our understanding

of the integration of sensory and motor information during speech,

we know little about the role, if any, that attention plays. Outside of

the laboratory environment, speakers are typically faced with the

task of simultaneously processing auditory feedback in conjunction

with other sensory information (e.g. visual, somatosensory, etc.). If

auditory feedback processing requires some degree of attention, and

given that attention is considered a limited resource (N€

a€

at€

anen,

1992), then the processing of perceived motor errors may be modu-

lated by the attentional load incurred by processing multimodal

stimuli.

The cognitive and neural processes that underlie attention have

been the subject of intense study for many years. For example, stud-

ies of unimodal selective attention have demonstrated enhanced

brain activity in the auditory cortex in response to attended vs. unat-

tended stimuli (Woldorff et al., 1993; Pugh et al., 1996; Ahveninen

et al., 2011). Studies of bimodal (e.g. auditory and visual) selective

attention have likewise shown more brain activity in the auditory

Correspondence: Hanjun Liu,

Department of Rehabilitation Medicine, as above.

E-mail: lhanjun@mail.sysu.edu.cn

These two authors contributed equally to this study.

Received 28 November 2014, revised 28 April 2015, accepted 11 May 2015

European Journal of Neuroscience, pp. 1–10, 2015 doi:10.1111/ejn.12949

cortex when participants attend to auditory stimuli while ignor-

ing visual stimuli, and more activity in the visual cortex when

participants attend to visual stimuli while ignoring auditory stimuli

(Woodruff et al., 1996; Johnson & Zatorre, 2005, 2006). In addition

to this enhancement in the attended modality, cross-modal inhibition

has been demonstrated in the form of decreases in activity observed

in the ‘ignored’sensory cortex (Shomstein & Yantis, 2004; Johnson

& Zatorre, 2005, 2006).

Although attention is typically manifest as a selective focus on

the processing of one aspect of the environment, attention can also

be divided when we must perform two (or more) tasks simulta-

neously. Not surprisingly, the complexity of dividing attention

across stimuli, or between modalities, involves activity in additional

regions of the brain. For example, in a study of divided visual atten-

tion, discrimination between shape, color and speed of a visual stim-

ulus activated the anterior cingulate and the prefrontal cortex (PFC)

in the right hemisphere (Corbetta et al., 1991). In addition, when

attention has to be divided between auditory and visual stimuli,

researchers have found signiﬁcant decreases in the level of activity

in both the auditory and visual cortices compared to when attention

is focused on either modality alone (Klingberg, 1998; Loose et al.,

2003; Johnson & Zatorre, 2006). Event-related potential (ERP) stud-

ies have similarly shown that divided attention causes processing

delays, which are reﬂected as increased latencies of the P300 com-

ponent (Hohnsbein et al., 1991).

The research on cross-modal attention thus indicates that attention

modulates activity in the sensory cortices. As speech motor control

relies on sensory feedback from audition, somatosensation and kin-

esthesis, whether or not the sensorimotor integration that is critical

to speech motor control can be shaped by attention is an important

question. Accurate estimation of the dynamic state of the vocal artic-

ulators relies on the detection of errors in voice auditory feedback.

In addition, considerable evidence has demonstrated that the central

auditory processing of speech sounds is highly dependent on atten-

tion. For example, larger ERPs (Hink & Hillyard, 1976; Stevens

et al., 2006) or enhanced brain activity (Ahveninen et al., 2006;

Sabri et al., 2008) in the auditory cortex is observed in response to

attended vs. unattended speech sounds. Moreover, disruption of the

lip representation induced by transcranial magnetic stimulation

results in increased left-hemisphere P50m response to attended

speech sounds (Mottonen et al., 2014), suggesting that the earlier

interactions between auditory and motor cortices depend on atten-

tion. Taken together, these studies suggest that attentional mecha-

nisms may inﬂuence the auditory–motor processing of vocal pitch

errors.

Evidence that attention modulates auditory–motor integration

comes from two recent studies. In an ERP study performed by Tum-

ber et al. (2014), participants were exposed to pitch perturbations

during vocalisation while they either passively viewed a rapid serial

visual presentation (RSVP) of letters, or actively identiﬁed target

stimuli in the RSVP of letters. The results showed that actively

attending to the RSVP elicited a decrease in the magnitude of vocal

compensation for pitch perturbations relative to passively viewing

the RSVP, whereas the P1-N1-P2 complex elicited by pitch pertur-

bations was not modulated by attention. This ﬁnding provides

behavioral evidence for the inﬂuence of attention on the auditory–

motor integration in voice control.

In another study conducted by Hu et al. (2015), participants were

exposed to pitch perturbations during vocalisation while they either

attended to a visual stimulus or attended to the pitch perturbations

in a low (counting the number of perturbations) or high (counting

the type of perturbations) attentional load condition. The results

revealed no systematic change in the vocal compensations for the

pitch perturbations, irrespective of whether the perturbations were

attended or not. However, larger P2 responses were observed when

participants attended to the pitch perturbations in the low-load atten-

tional condition relative to P2 responses observed in the high-load

attentional condition or those observed when the pitch perturbations

were unattended. The authors concluded that attentional load modu-

lates auditory cortical responses to perceived vocal pitch errors.

Although the above-cited studies provide support for the hypothe-

sis that there is a role for attention in auditory–motor integration

during speech production, certain shortcomings in these two studies

limit our understanding. For example, Tumber et al. (2014) investi-

gated whether taxing central attentional processes affects auditory–

motor integration by manipulating the visual attention load but did

not manipulate the attention their participants paid to the pitch per-

turbations. Thus, the question of whether dividing attention between

visual and auditory stimuli affects the auditory–motor processing of

pitch perturbations remains unanswered. Hu et al. (2015) found the

modulatory effect of attention on the cortical processing of vocal

pitch feedback errors as a function of load level, but ERP responses

for these attentional manipulations were not compared to a control

condition where participants would passively listen to the pitch per-

turbations and view the visual stimuli. Thus, Hu et al. (2015) were

unable to determine whether the observed differences in the cortical

responses between attentional conditions reﬂected enhancement in

the attended modality, inhibition in the unattended modality or both.

Therefore, whether and how attention inﬂuences the auditory–motor

integration in voice control remains poorly understood.

In the present study, we directly compared the inﬂuence of selec-

tive attention and divided attention on auditory–motor integration

during voice control. Participants sustained a vowel phonation while

they heard their voice pitch-shifted (i.e., auditory stimuli) and simul-

taneously saw ﬂashing lights (i.e., visual stimuli). The participants’

task was to attend to either the auditory or to the visual stimuli

(selective attention), to attend to both modalities (divided attention),

or to passively observe both the auditory and visual stimuli (control

condition). Vocal and cortical (N1-P2 complex) responses to the

pitch perturbations in these experimental conditions were measured

and compared. We hypothesised that larger vocal and cortical

responses would occur when participants attended to the pitch per-

turbations as compared to when participants were asked to ignore

them. Moreover, we predicted that divided attention would elicit

smaller vocal and ERP responses to the pitch perturbations than in

the selective attention condition.

Materials and methods

Al Subjects

Thirty-three students from Sun Yat-sen University of China partici-

pated in the experiment. Three participants were excluded from the

ﬁnal sample because of technical problems (N=2) and poor data

quality (N=1). Therefore, data from 30 subjects (seven males and

23 females) were analysed. All participants were right-handed,

native-Mandarin speakers with a mean age of 23 3 (SD) years. No

participant reported a history of speech, hearing, language or neuro-

logical disorders. All subjects passed a hearing screening at 25 dB

hearing level (HL) for octave intervals of 500–4000 Hz and had

normal or corrected-to-normal vision. Informed consent was

obtained from all participants. All the procedures, including subject

recruitment and data acquisition, were approved by the Institutional

Review Board of The First Afﬁliated Hospital at Sun Yat-sen Uni-

European Journal of Neuroscience,1–10

2 Y. Liu et al.

versity of China, and were in accordance with the Code of Ethics of

the World Medical Association (Declaration of Helsinki).

Apparatus

Participants were seated in a sound-attenuated booth throughout the

experiment. Prior to data recording, the experimental system was

calibrated to ensure that the intensity of voice feedback that partici-

pants heard was 10 dB (sound pressure level, SPL) higher than their

actual voice output to partially mask the airborne and bone-con-

ducted feedback (Behroozmand et al., 2009). The voice signals were

transduced by a dynamic microphone (model DM2200; Takstar Inc.,

Huizhou, China) and ampliﬁed by a MOTU Ultralite Mk3 FireWire

audio interface (Cambridge, MA, USA). The ampliﬁed signals were

then pitch-shifted by an Eventide Eclipse Harmonizer (Little Ferry,

NJ, USA), which was controlled by a custom program created with

Max/MSP (v.6.0 by Cycling 74, San Francisco, CA, USA). This

program also generated transistor–transistor logic (TTL) pulses that

marked the onset and offset of the pitch perturbations. The TTL

pulses were also sent to the electroencephalograph (EEG) recording

system via a FireWire cable. Finally, the pitch-shifted voices were

ampliﬁed by an ICON NeoAmp (Middleton, WI, USA) headphone

ampliﬁer and fed back to participants through insert earphones

(ER1-14A, Etymotic Research Inc., Elk Grove Village, IL, USA).

The original voice, the pitch-shifted feedback and the TTL pulses

were digitised at 10 kHz by a PowerLab A/D converter (model

ML880, AD Instruments, Castle Hill, Australia) and recorded using

LabChart software (v.7.0, AD Instruments).

Visual and auditory Stimuli

Visual stimuli

Two circles representing the blue and red indicator lights were gen-

erated by Max/MSP software and displayed on the computer screen.

The red indicator light began to ﬂash 500 ms after the blue indicator

light prompted participants to vocalise. Participants were instructed

to vocalise when the blue indicator light was turned on and termi-

nate their vocalisations when the blue indicator light was turned off,

during which they heard their voice auditory feedback unexpectedly

shifted upwards (see Auditory stimuli). During each vocalisation, the

red indicator light ﬂashed 1–7 times with variable inter-stimulus

intervals (ISIs) ranging from 400 to 1600 ms (400, 600, 800, 1000,

1200, 1400 and 1600 ms). The duration of the red light was ﬁxed at

200 ms.

Auditory stimuli

The pitch-shift stimulus (PSS) that participants heard was +200

cents (100 cents equals 1 semitone) with a ﬁxed duration of

200 ms. The number of PSS presented to subjects ranged from one

to four per vocalisation. The ﬁrst PSS was presented 500–1000 ms

after the vocal onset, and the succeeding stimuli occurred with an

ISI of 700–900 ms. The onsets of visual and auditory stimuli were

asynchronous.

Procedure

The present study included four conditions: (i) attend to the PSS but

ignore the red indicator light (Auditory Attention); (ii) attend to the

red indicator light but ignore the PSS (Visual Attention); (iii) attend

to both PSS and red indicator light (Bimodal Attention); and (iv)

passively listen to the PSS and view the lights (Bimodal Passive).

Across all conditions, subjects vocalised the vowel sound /u/ for ~5s

at a comfortable pitch when cued by the blue indicator light, and

saw the red indicator light ﬂashing. At the end of each vocalisation,

subjects were required to take a break of 2–3s prior to initiating the

next vocalisation. Production of ~40 consecutive vocalisations

constituted one block, which resulted in ~100 trials (i.e. PSS) per

condition. As a control condition, the Bimodal Passive condition

was always the ﬁrst block, while participants were unaware of the

attention-related tasks that were to follow. The order of the other

three conditions was counterbalanced across all subjects.

Across all but the Bimodal Passive condition, an immediate recall

test was performed after each vocalisation to ensure participants

attended to or ignored the stimuli as instructed. Subjects reported

the number of PSS that they heard in the Auditory Attention condi-

tion, the number of red indicator light ﬂashes that they saw in the

Visual Attention condition, or both in the Bimodal Attention condi-

tion. The percentage of correctly remembered stimuli across the

three conditions was calculated based on these data.

EEG data acquisition and analyses

Participants wore a 64-electrode Geodesic Sensor Net (Electrical

Geodesics Inc., Eugene, OR, USA) on their scalp, and EEG signals

were ampliﬁed by a Net Amps 300 ampliﬁer (Electrical Geodesics

Inc.) and recorded using NetStation software (v.4.5; Electrical Geo-

desics Inc.). EEG signals from all channels were referenced to the

vertex (Cz) during the recording and digitised at a sampling fre-

quency of 1 kHz. The impedances of individual sensors were main-

tained below 50 kΩthroughout the recording (Ferree et al., 2001).

After data acquisition, NetStation software was used for off-line

analyses of the EEG signals. Data from all channels were band-

pass-ﬁltered at 1–20 Hz and then segmented into epochs ranging

from 200 ms before to 500 ms after the onset of the PSS. Seg-

mented trials contaminated by excessive muscular activity, eye

blinks or eye movements were assessed using the Artifact Detec-

tion toolbox in NetStation and eliminated from further analyses.

Additional visual inspection on individual trials was performed to

ensure that artifacts were rejected appropriately. Individual elec-

trodes containing artifacts in >20% of the segments would be

excluded from further analyses. All channels were then re-refer-

enced to the average of electrodes on each mastoid, and artifact-

free epochs were averaged and baseline-corrected to generate an

overall response across each condition. The amplitude and latency

of both the N1 and P2 components across conditions were mea-

sured as the negative and positive peaks in the time windows of

80–180 and 160–280 ms after the onset of PSS, and submitted to

statistical analyses.

Vocal data analysis

Voice F

contours were extracted for each vowel production using

Praat (Boersma, 2001). F

values in Hz were converted to the cents

scale using the formula cents =100 9[12 9log

/reference)],

where the reference denotes the arbitrary reference note 195.997 Hz

(G4). Each trial was segmented into epochs ranging from 200 ms

before to 700 ms after the onset of the PSS. Each trial was visually

inspected to ensure that trials with vocal interruptions or signal pro-

cessing errors were excluded from further analyses. Artifact-free tri-

als for each of the four conditions were then averaged for each

participant to generate an overall vocal response. Vocal response

latencies in ms were measured as the time when the F

trajectory

European Journal of Neuroscience,1–10

Attention effects on voice feedback processing 3

exceeded 2 SDs above or below the pre-stimulus mean following

the perturbation onset. Vocal response magnitudes in cents were cal-

culated by subtracting the pre-stimulus mean from the peak value of

the voice F

contour following the response onset. The peak time of

vocal response in ms were determined at the point of greatest devia-

tion from the value at the onset of the stimulus.

Statistical analyses

The vocal and neurophysiological response data were subjected to

repeated-measures analyses of variance (RM-ANOVAs) in SPSS

(v.16.0). Speciﬁcally, the magnitudes, latencies and peak times of

vocal responses were analysed using one-way RM-ANOVAs with atten-

tion condition as the single factor (i.e. Auditory Attention, Visual

Attention, Bimodal Attention and Bimodal Passive). The amplitude

and latency of the N1-P2 complex, extracted from 10 electrodes

(FC1, FC2, FCz, FC3, FC4, C1, C2, Cz, C3, C4), were analysed

using three-way RM-ANOVAs (attention condition, anteriority and later-

ality). Frontal (FC1, FC2, FCz, FC3, FC4) and central (C1, C2, Cz,

C3, C4) electrodes were chosen as an anteriority factor, while lateral

left (FC3, C3), medial left (FC1, C1), midline (FCz, Cz), medial right

(FC2, C2) and lateral right (FC4, C4) were used as a laterality factor.

Responses from frontal and central electrodes were chosen for statis-

tical analyses because neurophysiological responses to PSS are

mostly pronounced in these two areas (Hawco et al., 2009; Chen

et al., 2012). Subsidiary RM-ANOVAs were calculated if higher-order

interactive effects reached signiﬁcance. Probability values were

adjusted using Greenhouse–Geisser correction when the assumption

of sphericity was violated.

Results

Behavioral performance

Participants’accuracy at counting and recalling the number of PSS

in the Auditory Attention condition, the number of red light ﬂashes

in the Visual Attention condition, and both of these auditory and

visual stimuli in the Bimodal Attention condition, was calculated as

a percentage of correct responses for each of these conditions. Par-

ticipants’accuracy at reporting the number of PSS in the Bimodal

Attention condition (79 2%) (mean SEM throughout unless

otherwise indicated) was signiﬁcantly lower than that in the Audi-

tory Attention condition (98 1%; t

=8.470, P<0.001). Simi-

larly, participants’accuracy at reporting the number of red light

ﬂashes in the Bimodal Attention condition (80 3%) was signiﬁ-

cantly lower than that in the Visual Attention condition (95 1%;

=6.293, P<0.001).

Vocal responses

Figure 1 shows the grand-averaged voice F

contours and the T-bar

plots of the absolute magnitudes of vocal response to the PSS across

the four attention conditions. A one-way RM-ANOVA of the response

magnitude revealed a signiﬁcant main effect of attention condition

3,87

=2.863, P=0.041). Post hoc Bonferroni comparison tests

revealed that the Auditory Attention condition (12.5 0.8 cents)

elicited signiﬁcantly larger vocal responses than the Visual Attention

condition (10.3 0.8 cents; P=0.026). Vocal responses during the

Bimodal Attention condition (12.2 1.1 cents) were larger than

during the Visual Attention condition (11.4 0.9 cents) but this

difference failed to reach signiﬁcance (P=0.203). Vocal response

magnitudes for the Bimodal Passive condition were not signiﬁcantly

different from the responses elicited in the other three conditions

(P>0.5). In addition, vocal response latencies did not differ as a

function of the attention condition (F

3,87

=0.486, P=0.693; Audi-

tory Attention, 86 7 ms; Visual Attention, 86 9 ms; Bimodal

Attention, 86 7 ms; Bimodal Passive, 91 8 ms). In addition,

no signiﬁcant main effect of attention condition was found for the

peak times of vocal responses (F

3,87

=0.697, P=0.556; Auditory

Attention, 352 34 ms; Visual Attention, 348 37 ms; Bimodal

Attention, 324 23 ms; Bimodal Passive, 326 25 ms).

Neurophysiological responses

Figure 2 shows the grand-averaged ERP waveforms in response to

the PSS for the Auditory Attention (red lines), Visual Attention

(blue lines), Bimodal Attention (black lines) and Bimodal Passive

conditions (green lines). Figure 3 shows the topographical distribu-

tions of the N1 and P2 components across the four attention condi-

tions. As can be seen in Figs 2 and 3, the Auditory Attention

condition elicited the largest P2 responses and the Bimodal Atten-

tion condition elicited the smallest P2 responses, while P2

responses were similar in the Visual Attention and Bimodal Pas-

sive conditions. In addition, N1 response also appeared to be

affected by attention, as reﬂected by the larger responses (i.e. more

negative) in the Bimodal Attention condition than in the Bimodal

Passive condition.

Fig. 1. (A) Grand-averaged voice F

contours and (B) T-bar plots of the magnitudes of vocal responses to pitch perturbations across the four attention condi-

tions. The thick solid line, the sparse dashed line, the dense dashed line and the thin solid line represent the vocal responses in the Auditory Attention, Visual

Attention, Bimodal Attention and Bimodal Passive conditions, respectively. *P<0.05 (P=0.026) for the size of the vocal responses between the Auditory

Attention and Visual Attention conditions. Error bars represent SEM.

European Journal of Neuroscience,1–10

4 Y. Liu et al.

One three-way RM-ANOVA revealed a signiﬁcant main effect of

attention condition (F

3,87

=5.205, P=0.006) on N1 amplitudes.

Post hoc Bonferroni comparison tests revealed that N1 amplitudes

were larger in the Bimodal Attention condition (-2.89 0.10 lV)

than in the Bimodal Passive condition (-2.13 0.08 lV; P<

0.001; see Fig. 4). Frontal electrodes (-2.63 0.07 lV) recorded

signiﬁcantly smaller N1 amplitudes than central electrodes (-2.35

0.06 lV; F

1,29

=16.695, P<0.001). The main effect of laterali-

ty also reached signiﬁcance (F

4,116

=8.607, P<0.001), which was

primarily driven by larger N1 amplitudes at the medial electrodes

(-2.74 0.11 lV) than at the lateral electrodes (left lateral,

-2.13 0.09 lV; right lateral, -2.32 0.10 lV). No signiﬁcant

difference, however, was found between left and right electrodes

(P>0.05).

Analysis of the N1 latencies revealed a signiﬁcant main effect of

laterality (F

4,116

=4.041, P<0.032). Midline electrodes (127 

1 ms) recorded signiﬁcantly shorter N1 latencies than both the right

lateral (131 1 ms; P<0.002) and the right medial electrodes

(130 1 ms; P<0.013). There was no systematic change in N1

latencies as a function of attention condition (F

3,87

=1.466,

P=0.229; Auditory Attention, 127 1 ms; Visual Attention,

128 1 ms; Bimodal Attention, 129 1 ms; Bimodal Passive,

131 1 ms) or anteriority (F

1,29

=2.148, P=0.154; frontal,

129 1 ms; central, 128 1 ms).

Fig. 2. Grand-averaged ERP waveforms in response to pitch perturbations of +200 cents across the four attention conditions. The red, blue, black and green

solid lines denote the cortical responses in the Audition Attention, Visual Attention, Bimodal Attention and Bimodal Passive conditions, respectively. ERP

responses are averages over electrodes FC1, FCz, FC2, C1, Cz and C2, across all subjects.

Fig. 3. Topographical distributions of the N1 (top) and P2 (bottom) responses to pitch perturbations of +200 cents across the four attention conditions. From

left to right are shown the responses in the Audition Attention, Visual Attention, Bimodal Attention and Bimodal Passive conditions, respectively.

European Journal of Neuroscience,1–10

Attention effects on voice feedback processing 5

For P2 amplitudes, there was a signiﬁcant main effect of attention

condition (F

3,87

=23.819, P<0.001). Post hoc Bonferroni compar-

ison tests showed that the Auditory Attention condition

(3.94 0.11 lV) elicited larger P2 amplitudes (i.e., more positive)

than the other three attention conditions (P<0.003), and the Bimo-

dal Attention condition (2.55 0.08 lV) elicited smaller P2 ampli-

tudes than the other three attention conditions (P<0.01; see

Fig. 4). No signiﬁcant difference was found between the Visual

Attention condition (3.11 0.08 lV) and the Bimodal Passive con-

dition (3.12 0.10 lV; P>0.05).

In addition to the main effect of attention, main effects of anteri-

ority (F

1,29

=68.498, P<0.001) and laterality (F

4,116

=49.634,

P<0.001) also reached signiﬁcance. The main effect of anteriority

was driven by P2 amplitudes that were larger at the frontal elec-

trodes (3.50 0.07 lV) than at the central electrodes (2.86

0.06 lV; P<0.001). The main effect of laterality was primarily

caused by larger P2 amplitudes at midline electrodes (3.92

0.11 lV) relative to lateral and medial electrodes (P<0.02), and

larger P2 amplitudes at medial electrodes (3.46 0.11 lV) than at

the lateral electrodes (2.53 0.09 lV; P<0.01). There was no

signiﬁcant difference between the left and right electrodes

(P>0.05).

A main effect of laterality (F

4,116

=5.363, P=0.006) was sig-

niﬁcant for P2 latencies. P2 latencies observed for right lateral

electrodes (227 1 ms) were longer than latencies observed for

left medial electrodes (222 1 ms; P=0.002), right medial elec-

trodes (224 1 ms; P=0.009) and midline electrodes

(223 1 ms; P=0.011). Main effects of attention

3,87

=1.191, P=0.318; Auditory Attention, 222 1 ms;

Visual Attention, 225 1 ms; Bimodal Attention, 222 1 ms;

Bimodal Passive, 226 1 ms) and anteriority (F

1,29

=0.019,

P=0.890; frontal, 224 1 ms; central, 224 1 ms) failed to

reach signiﬁcance.

Discussion

The present study examined the inﬂuence of selective and divided

attention on the auditory–motor processing of feedback errors dur-

ing vocal pitch regulation. The behavioral results revealed an

effect of selective attention as larger vocal compensations for

pitch perturbations were elicited when participants attended to the

auditory stimuli compared to when they attended to the visual

stimuli. This pattern extended to the cortical responses; larger P2

responses to pitch perturbations were elicited when participants

attended to the auditory stimuli as compared to when they

attended to the visual stimuli or passively observed the auditory

and visual stimuli. Moreover, when participants divided their

attention between auditory and visual stimuli, P2 responses were

signiﬁcantly smaller than when participants attended to the audi-

tory or visual stimuli, or passively observed both. In addition,

divided attention elicited more negative N1 responses than the

control condition. These ﬁndings support our hypothesis that audi-

tory–motor integration in voice control is modulated by attention.

Moreover, selective attention and divided attention appear to mod-

ulate the processing of auditory feedback during vocal pitch regu-

lation in different ways.

Attention-dependent modulation of the vocal response

It is generally thought that vocal compensation for feedback pertur-

bations is involuntary in nature and cannot be controlled con-

sciously. For example, participants produce compensatory vocal

responses to pitch perturbations even when told to ignore them

(Burnett et al., 1998; Hain et al., 2000; Zarate & Zatorre, 2008).

Keough et al. (2013) further reported that vocal responses observed

when vocally-trained and -untrained participants were instructed to

ignore the PSS did not differ from those observed when they were

instructed to compensate for the PSS. Although these studies sug-

gest that vocal compensation for feedback perturbations is indepen-

dent of attentional control, there were no explicit measures of the

degree of attentional engagement. In the present study, we employed

a post-presentation recall test providing conﬁrmation that visual and/

or auditory attention was engaged. Indeed, our results revealed lar-

ger vocal responses to the attended pitch perturbations than when

they were not attended. Similarly, Tumber et al. (2014) reported that

actively attending to an RSVP elicited smaller vocal compensation

for pitch perturbations than did passively viewing the RSVP. Note

that only when participants were asked to passively view the RSVP

before performing the vocal and RSVP task simultaneously did this

modulation of vocal compensation occur, which was because the

participants were unable to ignore the RSVP after previously per-

forming a visual task (Scheerer & Jones, 2014). Taken together,

these ﬁndings provide evidence that selective attention modulates

vocal compensation for pitch perturbations.

Our ﬁnding, however, is inconsistent with another study by Hu

et al. (2015) that showed no effect of attention on the level of vocal

compensations for pitch perturbations. It is noteworthy that only one

size of pitch perturbation (+200 cents) was used in the present

study, while pitch perturbations of +100, +200 and +500 cents were

randomly presented in the Hu et al. (2015) study. In addition, the

same auditory and visual stimuli were used across the four attention

conditions in the present study, while the visual stimuli (i.e., the red

Fig. 4. T-bar plots of the grand-averaged amplitudes of N1 (left) and P2 (right) response to pitch perturbations of +200 cents across 10 electrodes. *P<0.05

for differences in cortical responses between conditions. Error bars represent SEM.

European Journal of Neuroscience,1–10

6 Y. Liu et al.

indicator light) were not presented when participants were asked to

attend to the pitch perturbations in the Hu et al. (2015) study.

Therefore, the inconsistency in the effect of selective attention on

the vocal pitch monitoring between these two studies may be attrib-

uted to the differences in the presentation of the auditory and visual

stimuli across the attention conditions.

As expected, participants performed more poorly during the

Bimodal Attention condition than the Auditory Attention condition

in terms of their recall for the number of PSS they heard during

each utterance. This ﬁnding is in line with other cross-modal studies

that have shown behavioral deﬁcits when participants divided their

attention between modalities as compared to selectively attending to

one modality and ignoring the other (Bonnel & Hafter, 1998; Jolico-

eur, 1999). One might therefore hypothesise that the Bimodal Atten-

tion condition should elicit smaller vocal compensatory responses to

pitch perturbations than the Auditory Attention condition. However,

the vocal responses did not differ across these two conditions, which

suggests that divided attention does not affect the vocal responses.

Nonetheless, selectively attending to or ignoring the auditory modal-

ity did affect the vocal responses. Moreover, the vocal responses did

not differ across the Visual Attention, Bimodal Attention and Bimo-

dal Passive conditions. In addition, vocal responses were similar in

the Auditory Attention and Bimodal Passive conditions. Together,

this pattern of results suggests that the attention effect on vocal

compensation for feedback perturbations may be dependent on the

degree of attentional engagement. In other words, only when pitch

perturbations were fully attended vs. completely ignored did atten-

tion have an impact on the vocal compensation.

Selective attention effect on the neurophysiological responses

The present study found larger P2 responses were elicited when par-

ticipants attended to the pitch perturbations (i.e., Auditory Attention

condition) than when they ignored the pitch perturbations (i.e.,

Visual Attention condition) or were not asked to attend to the audi-

tory stimuli or the visual stimuli (i.e., Bimodal Passive condition),

indicating that selective attention modulates cortical processing of

auditory feedback during vocal pitch regulation. This aspect of our

ﬁnding is consistent with one recent study that showed larger P2

responses to attended vs. unattended pitch perturbations in voice

auditory feedback (Hu et al., 2015). Similarly, previous research on

auditory perception showed enhanced brain activity in the auditory

cortex (Grady et al., 1997; Petkov et al., 2004), larger P2 responses

(Picton & Hillyard, 1974; Woldorff & Hillyard, 1991; Neelon et al.,

2006) and mismatch negativity (N€

a€

at€

anen et al., 1993; Alain &

Woods, 1997; Woldorff et al., 1998; Sussman et al., 2007) elicited

by attended vs. unattended sounds. Our results are also consistent

with the effects of selective attention on bimodal sensory processing

(e.g., auditory and visual) reported in other neuroimaging studies

that showed more activity in the auditory cortex while participants

attended to an auditory stimulus than when the same stimulus was

ignored (Kawashima et al., 1999; Loose et al., 2003; Johnson &

Zatorre, 2005, 2006).

There are theories that account for the enhanced P2 responses to

attended vs. unattended pitch perturbations. One theory is the gain-

based theory of selective attention (Hillyard et al., 1998) that posits

that paying attention to one modality while ignoring another

increases the gain of neurons that are sensitive to the attended modal-

ity, which results in decreased perceptual thresholds for that modal-

ity. In the context of this theory then, attending to the pitch

perturbations while ignoring the ﬂashing lights in the Auditory Atten-

tion condition may lead to an increased gain for neurons involved in

auditory–motor integration that facilitates the detection/correction of

pitch errors in voice auditory feedback, resulting in enhanced P2

responses to attended vs. unattended pitch perturbations.

An alternative explanation for the enhanced P2 responses we

observed may be that cortical responses to the unattended pitch

perturbations decreased. There is evidence that suggests that

modality-speciﬁc selective attention results primarily from

decreased processing in the unattended modality rather than

increased processing in the attended modality (Mozolic et al.,

2008). Thus, the selective attention effect observed in the present

study may have resulted from inhibited activity in the sensory

cortices that subserve the processing of pitch perturbations, thereby

freeing central attentional resources for the processing of ﬂashing

lights in the Visual Attention condition.

There is also considerable evidence that both cross-modal facilita-

tion and inhibition underlie attentional modulation of the sensory

cortices, such that attention to one modality leads not only to

increased activity in the sensory cortical area involved in the pro-

cessing of that modality but also to suppressed activity in regions

associated with other modalities (Loose et al., 2003; Johnson & Za-

torre, 2005, 2006; Mozolic et al., 2008). This leaves open the possi-

bility that changes in cortical responses may result from not only

enhanced response to the attended pitch perturbations but also

decreased responses to the unattended pitch perturbations.

As the above interpretations involve the comparison of the audi-

tory and visual selective attention conditions with each other, rather

than to the control condition, it is unclear whether changes in the

cortical responses reﬂect only enhancement in the attended modality,

inhibition in the unattended modality, or both. However, our results

revealed that cortical responses to pitch perturbations in the Visual

Attention condition did not differ from those observed when partici-

pants passively observed the bimodal stimuli (i.e., Bimodal Passive

condition). Similarly, Tumber et al. (2014) reported that the P1-N1-

P2 complex to pitch perturbations did not change when participants

actively attended to the RSVP as compared to when they passively

viewed the RSVP. In other words, increased attention to the visual

stimuli, relative to the control condition, did not appear to suppress

the cortical processing of auditory stimuli during the online control

of vocal production. This ﬁnding does not support the hypothesis

that the processing of attended information leads to the inhibition of

processing of unattended information. Thus, the observed effect of

selective attention may be primarily due to increased gain in neurons

that are sensitive to the detection and/or correction of feedback per-

turbation during vocal production.

Divided attention effect on the neurophysiological response

The present study found smaller P2 responses in the Bimodal Atten-

tion condition than in the Auditory Attention condition. This ﬁnding

suggests that divided attention leads to decreased auditory cortical

processing of vocal pitch perturbations relative to selective attention.

These results are in line with previous bimodal studies that showed

decreased activity in the sensory cortex (i.e., auditory or visual) dur-

ing divided attention relative to the levels of activity observed dur-

ing selective attention (Loose et al., 2003; Johnson & Zatorre,

2006). This pattern of results ﬁts the classical theory of limited

attention resource (N€

a€

at€

anen, 1992), according to which fewer neu-

ral resources are allocated for the processing of auditory stimuli

when dividing attention to auditory and visual stimuli than when

selectively focusing attention on auditory stimuli. However, another

possible mechanism responsible for this divided attention effect is

that the auditory and visual systems reciprocally inhibit the process-

European Journal of Neuroscience,1–10

Attention effects on voice feedback processing 7