Immediate integration of prosodic information from speech and visual information from pictures in the absence of focused attention: A mismatch negativity study

ArticleinNeuroscience 161(1):59-66 · July 2009with38 Reads
DOI: 10.1016/j.neuroscience.2009.01.070 · Source: PubMed
Abstract
Language is often perceived together with visual information. Recent experimental evidences indicated that, during spoken language comprehension, the brain can immediately integrate visual information with semantic or syntactic information from speech. Here we used the mismatch negativity to further investigate whether prosodic information from speech could be immediately integrated into a visual scene context or not, and especially the time course and automaticity of this integration process. Sixteen Chinese native speakers participated in the study. The materials included Chinese spoken sentences and picture pairs. In the audiovisual situation, relative to the concomitant pictures, the spoken sentence was appropriately accented in the standard stimuli, but inappropriately accented in the two kinds of deviant stimuli. In the purely auditory situation, the speech sentences were presented without pictures. It was found that the deviants evoked mismatch responses in both audiovisual and purely auditory situations; the mismatch negativity in the purely auditory situation peaked at the same time as, but was weaker than that evoked by the same deviant speech sounds in the audiovisual situation. This pattern of results suggested immediate integration of prosodic information from speech and visual information from pictures in the absence of focused attention.
    • "Besides, in the V-only condition we observed a small visual negativity located in the occipital visual cortex indicating that the brain reacted to the visual prosodic changes. In the case of an enhancement of the visual reaction in the visuo-auditory context, we might expect this visual negativity to be enhanced in its amplitude as observed during AV oddball protocols (Li et al. 2009). No such effect is observed in the present data suggesting that the MMN-like response in the continuous AV speech does not correspond to a simple enhancement of the visual perception but rather reflects a separate cognitive process linked to the auditory stimulus. "
    [Show abstract] [Hide abstract] ABSTRACT: The visual cues involved in auditory speech processing are not restricted to information from lip movements but also include head or chin gestures and facial expressions such as eyebrow movements. The fact that visual gestures precede the auditory signal implicates that visual information may influence the auditory activity. As visual stimuli are very close in time to the auditory information for audiovisual syllables, the cortical response to them usually overlaps with that for the auditory stimulation; the neural dynamics underlying the visual facilitation for continuous speech therefore remain unclear. In this study, we used a three-word phrase to study continuous speech processing. We presented video clips with even (without emphasis) phrases as the frequent stimuli and with one word visually emphasized by the speaker as the non-frequent stimuli. Negativity in the resulting ERPs was detected after the start of the emphasizing articulatory movements but before the auditory stimulus, a finding that was confirmed by the statistical comparisons of the audiovisual and visual stimulation. No such negativity was present in the control visual-only condition. The propagation of this negativity was observed between the visual and fronto-temporal electrodes. Thus, in continuous speech, the visual modality evokes predictive coding for the auditory speech, which is analysed by the cerebral cortex in the context of the phrase even before the arrival of the corresponding auditory signal.
    Full-text · Article · Dec 2013
    • "On the other hand, differing from non-tonal languages, Chinese uses pitch contours to distinguish lexical meanings (e.g., hua1 [flower] vs. hua4 [picture]). To investigate prosodic processing in Chinese, Yang and colleagues carried out a series of studies with the ERP technique ( Li, Yang, & Ren, 2009). They found that Chinese listeners can rapidly identify the semantic consequence of accentuation. "
    [Show abstract] [Hide abstract] ABSTRACT: The Chinese language possesses linguistic properties that are distinct from those of the most widely studied European languages. Given such uniqueness, research on the neurocognitive processing of Chinese not only contributes to our understanding of language-specific cognitive processes but also sheds light on the universality of psycholinguistic models developed on the basis of these European languages. In this Introduction, we briefly review neurocognitive studies on the processing of Chinese in the past ten years, summarizing existing findings concerning lexical, sentential, and discourse processing in Chinese.
    Full-text · Article · Dec 2009
    • "On the other hand, differing from non-tonal languages, Chinese uses pitch contours to distinguish lexical meanings (e.g., hua1 [flower] vs. hua4 [picture]). To investigate prosodic processing in Chinese, Yang and colleagues carried out a series of studies with the ERP technique ( Li, Yang, & Ren, 2009). They found that Chinese listeners can rapidly identify the semantic consequence of accentuation. "
    [Show abstract] [Hide abstract] ABSTRACT: The Chinese language possesses linguistic properties that are distinct from those of the most widely studied European languages. Given such uniqueness, research on the neurocognitive processing of Chinese not only contributes to our understanding of language-specific cognitive processes but also sheds light on the universality of psycholinguistic models developed on the basis of these European languages. In this Introduction, we briefly review neurocognitive studies on the processing of Chinese in the past ten years, summarizing existing findings concerning lexical, sentential, and discourse processing in Chinese.
    Full-text · Article · Sep 2009
Show more