[Show abstract][Hide abstract] ABSTRACT: An essential task of our perceptual systems is to bind together the distinctive features of single objects and events into unitary percepts, even when those features are registered in different sensory modalities. In cases where auditory and visual inputs are spatially incongruent, they may still be perceived as belonging to a single event at the location of the visual stimulus -- a phenomenon known as the 'ventriloquist illusion'. The present study examined how audio-visual temporal congruence influences the ventriloquist illusion and characterized its neural underpinnings with functional magnetic resonance imaging (fMRI). Behaviorally, the ventriloquist illusion was reduced for asynchronous versus synchronous audio-visual stimuli, in accordance with previous reports. Neural activity patterns associated with the ventriloquist effect were consistently observed in the planum temporale (PT), with a reduction in illusion-related fMRI-signals ipsilateral to visual stimulation for central sounds perceived peripherally and a contralateral increase in illusion-related fMRI-signals for peripheral sounds perceived centrally. Moreover, it was found that separate but adjacent regions within the PT were preferentially activated for ventriloquist illusions produced by synchronous and asynchronous audio-visual stimulation. We conclude that the left-right balance of neural activity in the PT represents the neural code that underlies the ventriloquist illusion, with greater activity in the cerebral hemisphere contralateral to the direction of the perceived shift of sound location.
[Show abstract][Hide abstract] ABSTRACT: Multisensory integration does not only recruit higher-level association cortex, but also low-level and even primary sensory cortices. Here, we will describe and quantify two types of anatomical pathways, a thalamocortical and a corticocortical that possibly underlie short-latency multisensory integration processes in the primary auditory (A1), somatosensory (S1), and visual cortex (V1). Results were obtained from Mongolian gerbils, a common model-species in neuroscience, using simultaneous injections of different retrograde tracers into A1, S1, and V1. Several auditory, visual, and somatosensory thalamic nuclei project not only to the primary sensory area of their own (matched) but also to areas of other (non-matched) modalities. The crossmodal output ratios of these nuclei, belonging to both core and non-core sensory pathways, vary between 0.4 and 63.5 % of the labeled neurons. Approximately 0.3 % of the sensory thalamic input to A1, 5.0 % to S1, and 2.1 % to V1 arise from non-matched nuclei. V1 has most crossmodal corticocortical connections, projecting strongest to S1 and receiving a similar amount of moderate inputs from A1 and S1. S1 is mainly interconnected with V1. A1 has slightly more projections to V1 than S1, but gets just faint inputs from there. Concerning the layer-specific distribution of the retrogradely labeled somata in cortex, V1 provides the most pronounced feedforward-type outputs and receives (together with S1) most pronounced feedback-type inputs. In contrast, A1 has most pronounced feedback-type outputs and feedforward-type inputs in this network. Functionally, the different sets of thalamocortical and corticocortical connections could underlie distinctive types of integration mechanisms for different modality pairings.
Brain Structure and Function 01/2014; · 7.84 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Although multisensory integration has been an important area of recent research, most studies focused on audiovisual integration. Importantly, however, the combination of audition and touch can guide our behavior as effectively which we studied here using psychophysics and functional magnetic resonance imaging (fMRI). We tested whether task-irrelevant tactile stimuli would enhance auditory detection, and whether hemispheric asymmetries would modulate these audiotactile benefits using lateralized sounds. Spatially aligned task-irrelevant tactile stimuli could occur either synchronously or asynchronously with the sounds. Auditory detection was enhanced by non-informative synchronous and asynchronous tactile stimuli, if presented on the left side. Elevated fMRI-signals to left-sided synchronous bimodal stimulation were found in primary auditory cortex (A1). Adjacent regions (planum temporale, PT) expressed enhanced BOLD-responses for synchronous and asynchronous left-sided bimodal conditions. Additional connectivity analyses seeded in right-hemispheric A1 and PT for both bimodal conditions showed enhanced connectivity with right-hemispheric thalamic, somatosensory and multisensory areas that scaled with subjects' performance. Our results indicate that functional asymmetries interact with audiotactile interplay which can be observed for left-lateralized stimulation in the right hemisphere. There, audiotactile interplay recruits a functional network of unisensory cortices, and the strength of these functional network connections is directly related to subjects' perceptual sensitivity.
[Show abstract][Hide abstract] ABSTRACT: Recent electrophysiological studies have reported short latency modulations in cortical regions for multisensory stimuli, thereby suggesting a subcortical, possibly thalamic origin of these modulations. Concurrently, there is an ongoing debate, whether multisensory interplay reflects automatic, bottom-up driven processes or relies on top-down influences. Here, we dissociated the effects of task set and stimulus configurations on BOLD-signals in the human thalamus with event-related functional magnetic resonance imaging (fMRI). We orthogonally manipulated temporal and spatial congruency of audio-visual stimulus configurations, while subjects judged either their temporal or spatial congruency. Voxel-based fMRI-results revealed increased fMRI-signals for the temporal vs. spatial task in posterior and central thalamus, respectively. A more sensitive region of interest (ROI)-analysis confirmed that posterior thalamic nuclei showed a preference for the temporal task and central thalamic nuclei for the spatial task. Moreover, the ROI-analysis also revealed enhanced fMRI-signals for spatially incongruent stimuli in central thalamus. Together, our results demonstrate that both audio-visual stimulus configurations and task-related processing of spatial or temporal stimulus features selectively modulate thalamic processing and thus are in a position to influence cortical processing at an early stage.
[Show abstract][Hide abstract] ABSTRACT: Approaching or looming signals are often related to extremely relevant environmental events (e.g. threats or collisions) making these signals critical for survival. However, the neural network underlying multisensory looming processing is not yet fully understood. Using functional magnetic resonance imaging (fMRI) we identified the neural correlates of audiovisual looming processing in humans: audiovisual looming (vs. receding) signals enhance fMRI-responses in low-level visual and auditory areas plus multisensory cortex (superior temporal sulcus; plus parietal and frontal structures). When characterizing the fMRI-response profiles for multisensory looming stimuli, we found significant enhancements relative to the mean and maximum of unisensory responses in looming-sensitive visual and auditory cortex plus STS. Superadditive enhancements were observed in visual cortex. Subject-specific region-of-interest analyses further revealed superadditive response profiles within all sensory-specific looming-sensitive structures plus bilateral STS for audiovisual looming vs. summed unisensory looming conditions. Finally, we observed enhanced connectivity of bilateral STS with low-level visual areas in the context of looming processing. This enhanced coupling of STS with unisensory regions might potentially serve to enhance the salience of unisensory stimulus features and is accompanied by superadditive fMRI-responses. We suggest that this preference in neural signaling for looming stimuli effectively informs animals to avoid potential threats or collisions.
[Show abstract][Hide abstract] ABSTRACT: Perceptually ambiguous stimuli are useful for testing psychological and neuronal models of perceptual organization, e.g. for studying brain processes that underlie sequential segregation and integration. This is because the same stimuli may give rise to different subjective experiences. For humans, a tone sequence that alternates between a low-frequency and a high-frequency tone is perceptually bistable, and can be perceived as one or two streams. In the current study we present a new method based on response times (RTs) which allows identification ambiguous and unambiguous stimuli for subjects who cannot verbally report their subjective experience. We required two macaque monkeys (macaca fascicularis) to detect the termination of a sequence of light flashes which were either presented alone, or synchronized in different ways with a sequence of alternating low and high tones. We found that the monkeys responded faster to the termination of the flash sequence when the tone sequence terminated shortly before the flash sequence and thus predicted the termination of the flash sequence. This RT gain depended on the frequency separation of the tones. RT gains were largest when the frequency separation was small and the tones were presumably heard mainly as one stream. RT gains were smallest when the frequency separation was large and the tones were presumably mainly heard as two streams. RT gain was of intermediate size for intermediate frequency separations. Similar results were obtained from human subjects. We conclude that the observed RT gains reflect the perceptual organization of the tone sequence, and that tone sequences with an intermediate frequency separation, as for humans, are perceptually ambiguous for monkeys.
[Show abstract][Hide abstract] ABSTRACT: Philosophers, psychologists, and neuroscientists have long been interested in how the temporal aspects of perception are represented in the brain. In the present study, we investigated the neural basis of the temporal perception of synchrony/asynchrony for audiovisual speech stimuli using functional magnetic resonance imaging (fMRI). Subjects judged the temporal relation of (a)synchronous audiovisual speech streams, and indicated any changes in their perception of the stimuli over time. Differential hemodynamic responses for synchronous versus asynchronous stimuli were observed in the multisensory superior temporal sulcus complex (mSTS-c) and prefrontal cortex. Within mSTS-c we found adjacent regions expressing an enhanced BOLD-response to the different physical (a)synchrony conditions. These regions were further modulated by the subjects' perceptual state. By calculating the distances between the modulated regions within mSTS-c in single-subjects we demonstrate that the "auditory leading (A(L))" and "visual leading (V(L)) areas" lie closer to "synchrony areas" than to each other. Moreover, analysis of interregional connectivity indicates a stronger functional connection between multisensory prefrontal cortex and mSTS-c during the perception of asynchrony. Taken together, these results therefore suggest the presence of distinct sub-regions within the human STS-c for the maintenance of temporal relations for audiovisual speech stimuli plus differential functional connectivity with prefrontal regions. The respective local activity in mSTS-c is dependent both upon the physical properties of the stimuli presented and upon the subjects' perception of (a)synchrony.
Frontiers in Integrative Neuroscience 01/2012; 6:64.
[Show abstract][Hide abstract] ABSTRACT: In everyday life our brain often receives information about events and objects in the real world via several sensory modalities, because natural objects often stimulate more than one sense. These different types of information are processed in our brain along different sensory-specific pathways, but are finally integrated into a unified percept. During the last years, studies provided compelling evidence that the neural basis of multisensory integration is not restricted to higher association areas of the cortex, but can already occur at low-level stages of sensory cortical processing and even in subcortical structures. In this article we will review the potential role of several thalamic structures in multisensory interplay and discuss their extensive anatomical connections with sensory-specific and multisensory cortical structures. We conclude that sensory-specific thalamic structures may act as a crucial processing node of multisensory interplay in addition to their traditional role as sensory relaying structure.
[Show abstract][Hide abstract] ABSTRACT: Dopamine release in cortical and subcortical structures plays a central role in reward-related neural processes. Within this context, dopaminergic inputs are commonly assumed to play an activating role, facilitating behavioral and cognitive operations necessary to obtain a prospective reward. Here, we provide evidence from human fMRI that this activating role can also be mediated by task-demand-related processes and thus extends beyond situations that only entail extrinsic motivating factors. Using a visual discrimination task in which varying levels of task demands were precued, we found enhanced hemodynamic activity in the substantia nigra (SN) for high task demands in the absence of reward or similar extrinsic motivating factors. This observation thus indicates that the SN can also be activated in an endogenous fashion. In parallel to its role in reward-related processes, reward-independent activation likely serves to recruit the processing resources needed to meet enhanced task demands. Simultaneously, activity in a wide network of cortical and subcortical control regions was enhanced in response to high task demands, whereas areas of the default-mode network were deactivated more strongly. The present observations suggest that the SN represents a core node within a broader neural network that adjusts the amount of available neural and behavioral resources to changing situational opportunities and task requirements, which is often driven by extrinsic factors but can also be controlled endogenously.
Journal of Neuroscience 03/2011; 31(13):4955-61. · 6.91 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Attending to the spatial location or to nonspatial features of a stimulus modulates neural activity in cortical areas that process its perceptual attributes. The feature-based attentional selection of the direction of a moving stimulus is associated with increased firing of individual neurons tuned to the direction of the movement in area V5/MT, while responses of neurons tuned to opposite directions are suppressed. However, it is not known how these multiplicatively scaled responses of individual neurons tuned to different motion-directions are integrated at the population level, in order to facilitate the processing of stimuli that match the perceptual goals. Using functional magnetic resonance imaging (fMRI) the present study revealed that attending to the movement direction of a dot field enhances the response in a number of areas including the human MT region (hMT) as a function of the coherence of the stimulus. Attending the opposite direction, however, lead to a suppressed response in hMT that was inversely correlated with stimulus-coherence. These findings demonstrate that the multiplicative scaling of single-neuron responses by feature-based attention results in an enhanced direction-selective population response within those cortical modules that processes the physical attributes of the attended stimuli. Our results provide strong support for the validity of the "feature similarity gain model" on the integrated population response as quantified by parametric fMRI in humans.
Human Brain Mapping 02/2011; 32(12):2183-92. · 6.88 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Combining information across modalities can affect sensory performance. We studied how co-occurring sounds modulate behavioral visual detection sensitivity (d'), and neural responses, for visual stimuli of higher or lower intensity. Co-occurrence of a sound enhanced human detection sensitivity for lower- but not higher-intensity visual targets. Functional magnetic resonance imaging (fMRI) linked this to boosts in activity-levels for sensory-specific visual and auditory cortex, plus multisensory superior temporal sulcus (STS), specifically for a lower-intensity visual event when paired with a sound. Thalamic structures in visual and auditory pathways, the lateral and medial geniculate bodies, respectively (LGB, MGB), showed a similar pattern. Subject-by-subject psychophysical benefits correlated with corresponding fMRI signals in visual, auditory, and multisensory regions. We also analyzed differential "coupling" patterns of LGB and MGB with other regions in the different experimental conditions. Effective-connectivity analyses showed enhanced coupling of sensory-specific thalamic bodies with the affected cortical sites during enhanced detection of lower-intensity visual events paired with sounds. Coupling strength between visual and auditory thalamus with cortical regions, including STS, covaried parametrically with the psychophysical benefit for this specific multisensory context. Our results indicate that multisensory enhancement of detection sensitivity for low-contrast visual stimuli by co-occurring sounds reflects a brain network involving not only established multisensory STS and sensory-specific cortex but also visual and auditory thalamus.
Journal of Neuroscience 10/2010; 30(41):13609-23. · 6.91 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Effective adaptation to the demands of a changing environment requires flexible cognitive control. The medial and the lateral frontal cortices are involved in such control processes, putatively in close interplay with the BG. In particular, dopaminergic projections from the midbrain (i.e., from the substantia nigra [SN] and the ventral tegmental area) have been proposed to play a pivotal role in modulating the activity in these areas for cognitive control purposes. In that dopaminergic involvement has been strongly implicated in reinforcement learning, these ideas suggest functional links between reinforcement learning, where the outcome of actions shapes behavior over time, and cognitive control in a more general context, where no direct reward is involved. Here, we provide evidence from functional MRI in humans that activity in the SN predicts systematic subsequent trial-to-trial RT prolongations that are thought to reflect cognitive control in a stop-signal paradigm. In particular, variations in the activity level of the SN in one trial predicted the degree of RT prolongation on the subsequent trial, consistent with a modulating output signal from the SN being involved in enhancing cognitive control. This link between SN activity and subsequent behavioral adjustments lends support to theoretical accounts that propose dopaminergic control signals that shape behavior both in the presence and in the absence of direct reward. This SN-based modulatory mechanism is presumably mediated via a wider network that determines response speed in this task, including frontal and parietal control regions, along with the BG and the associated subthalamic nucleus.
Journal of Cognitive Neuroscience 05/2010; 23(2):362-73. · 4.49 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: The superior colliculus (SC) has been shown to play a crucial role in the initiation and coordination of eye- and head-movements. The knowledge about the function of this structure is mainly based on single-unit recordings in animals with relatively few neuroimaging studies investigating eye-movement related brain activity in humans.
The present study employed high-field (7 Tesla) functional magnetic resonance imaging (fMRI) to investigate SC responses during endogenously cued saccades in humans. In response to centrally presented instructional cues, subjects either performed saccades away from (centrifugal) or towards (centripetal) the center of straight gaze or maintained fixation at the center position. Compared to central fixation, the execution of saccades elicited hemodynamic activity within a network of cortical and subcortical areas that included the SC, lateral geniculate nucleus (LGN), occipital cortex, striatum, and the pulvinar.
Activity in the SC was enhanced contralateral to the direction of the saccade (i.e., greater activity in the right as compared to left SC during leftward saccades and vice versa) during both centrifugal and centripetal saccades, thereby demonstrating that the contralateral predominance for saccade execution that has been shown to exist in animals is also present in the human SC. In addition, centrifugal saccades elicited greater activity in the SC than did centripetal saccades, while also being accompanied by an enhanced deactivation within the prefrontal default-mode network. This pattern of brain activity might reflect the reduced processing effort required to move the eyes toward as compared to away from the center of straight gaze, a position that might serve as a spatial baseline in which the retinotopic and craniotopic reference frames are aligned.
PLoS ONE 01/2010; 5(1):e8691. · 3.73 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: The selection of one of two concurrent speech messages for comprehension was investigated in healthy young adults in two event-related potential experiments. The stories were presented from virtual locations located 30 degrees to the left and right azimuth by convolving the speech message by the appropriate head-related transfer function determined for each individual participant. In addition, task irrelevant probe stimuli were presented in rapid sequence from the same virtual locations. In experiment 1, phoneme probes (/da/ voiced by the same talkers as attended and unattended messages) and band-pass filtered noise probes were presented. Phoneme probes coinciding with the attended message gave rise to a fronto-central negativity similar to the Nd-attention effect relative to the phoneme probes coinciding with the unattended speech message, whereas noise probes from the attended message's location showed a more positive frontal ERP response compared to probes from the unattended location resembling the so-called rejection positivity. In experiment 2, phoneme probes (as in exp. 1) and frequency-shifted (+400 Hz) were compared. The latter were characterized by a succession of negative and positive components that were modulated by location. The results suggest that at least two different neural mechanisms contribute to stream segregation in a cocktail-party setting: enhanced neural processing of stimuli matching the attended message closely (indexed by the Nd-effect) and rejection of stimuli that do not match the attended message at the attended location only (indexed by the rejection positivity).
Brain research 10/2009; 1307:78-88. · 2.46 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Faces expressing fear may attract attention in an automatic bottom-up fashion. Here we address this issue with magneto-encephalographic (MEG) recordings in subjects performing a demanding visual search combined with the presentation of irrelevant neutral or fearful faces. The impact of the faces' emotional expression on attentional selection was assessed by analyzing the N2pc component--a modulation of the event-related magnetic field response known to reflect attentional focusing in visual search. We observed that lateralized fearful faces elicited an N2pc approximately between 240 and 400 msec in ventral extrastriate cortex that was independent of the N2pc reflecting target selection in visual search. Despite their clear influence on neural processing, fearful faces did not significantly influence behavioral performance. To clarify this discrepancy, we further performed an MEG experiment in which the demands of the search task were reduced. Under those conditions, lateralized fearful faces elicited an N2pc response that was again independent of the N2pc response to the search target. Behavioral performance was, however, influenced in a significant manner, suggesting that for behavioral effects to appear, sufficient attentional resources need to be left unoccupied by the search task--a notion put forward by the perceptual load theory. Our observations are taken to indicate that irrelevant fearful faces influence attentional processing in extrastriate visual cortex in an automatic fashion and independent of other task-relevant attentional operations. However, this may not necessarily be echoed at the behavioral level as long as task-relevant selection operations exhaust attentional resources.
Journal of Cognitive Neuroscience 09/2009; 22(12):2926-38. · 4.49 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: The present study investigated the neural basis of attention in the somato-sensory system. Subjects directed their attention towards their left or right hand while functional MRI data was collected during tactile stimulation of the fingers. Activations evoked by tactile stimuli when a stimulated hand was attended vs. unattended were contrasted. The tactile stimuli elicited hemodynamic responses in the contralateral primary and secondary somatosensory cortex. No attentional modulations of the BOLD-response could be observed in these regions. However, attention-related modulations were observed at more anterior locations in the ipsi- and contralateral primary motor cortex and in the supplementary motor area. This pattern of results suggests, that attention to somato-sensory events is directly linked to the motor system and the preparation for action. This mechanism appears to be in stark contrast to visual or auditory attention, which primarily serve to separate relevant from irrelevant information.
Journal of the neurological sciences 02/2009; 279(1-2):93-8. · 2.32 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: In our daily life we look at many scenes. Some are rapidly forgotten, but others we recognize later. We accurately predicted recognition success with natural scene photographs using single trial magnetoencephalography (MEG) measures of brain activation. Specifically, we demonstrate that MEG responses in the initial 600 ms following the onset of scene photographs allow for prediction accuracy rates up to 84.1% using linear Support-Vector-Machine classification (lSVM). A permutation test confirmed that all lSVM based prediction rates were significantly better than "guessing". More generally, we present four approaches to analyzing brain function using lSVMs. (1) We show that lSVMs can be used to extract spatio-temporal patterns of brain activation from MEG-data. (2) We show lSVM classification can demonstrate significant correlations between comparatively early and late processes predictive of scene recognition, indicating dependencies between these processes over time. (3) We use lSVM classification to compare the information content of oscillatory and event-related MEG-activations and show they contain a similar amount of and largely overlapping information. (4) A more detailed analysis of single-trial predictiveness of different frequency bands revealed that theta band activity around 5 Hz allowed for highest prediction rates, and these rates are indistinguishable from those obtained with a full dataset. In sum our results clearly demonstrate that lSVMs can reliably predict natural scene recognition from single trial MEG-activation measures and can be a useful tool for analyzing predictive brain function.
[Show abstract][Hide abstract] ABSTRACT: We show that concurrent auditory stimuli can enhance the visual system's ability to detect brief visual events. Participants indicated which of two visual stimuli was briefly blinked off. A spatially non-aligned auditory cue - simultaneous with the blink - significantly enhanced subjects' detection ability, while a visual cue decreased detection ability relative to a no-cue condition. Control experiments indicate that the auditory-driven enhancement was not attributable to a warning effect. Also, the enhancement did not depend on an exact temporal alignment of cue-target onsets or offsets. In combination, our results provide evidence that the sound-induced enhancement is not due to a sharpening of visual temporal responses or apparent prolongation of the visual event. Rather, this enhancement seems to reflect an increase in phenomenal visual saliency.
Brain Research 02/2008; 1220:157-63. · 2.88 Impact Factor