ArticlePDF Available

Predictive Processing of Syntactic Information: Evidence from Event-Related Brain Potentials

Authors:

Abstract and Figures

Several recent event-related potential (ERP) studies have observed a left (anterior) negativity (L(A)N) in (morpho)syntactically well-formed but semantically anomalous sentences, which are often referred to as semantic reversal anomalies (e.g., The window closes someone). Such a L(A)N elicitation for semantic reversal anomalies is not expected under the widely held assumption that LANs are associated with morphosyntactic processing loads and are insensitive to semantic anomalies. This raises the empirical question of why semantic reversal anomalies elicit a L(A)N effect. One possible explanation for this observation can be presented in relation the multi-stream processing model, according to which the independent semantic processing stream challenges an analysis outputted by the morphosyntactic processing stream, leading to a misperception of morphosyntactic ill-formedness. Alternatively, the L(A)N effect may reflect a mismatch between expected and actual syntactic structures. As an inanimate subject can trigger the expectation of an intransitive structure rather than an active transitive structure, the input of a transitive verb should violate such an expectation. The present study tests these two hypotheses by manipulating the temporal predictabilities of verb types to examine the underlying cognitive processes of L(A)N effects in semantic reversal anomalies. The results reveal a L(A)N effect for semantic reversal anomalies only when prediction of a verb type was possible, in favour of the latter prediction-based view. The implications of this finding are discussed with respect to predictive processing mechanisms and the architecture of processing systems.
Content may be subject to copyright.
A preview of the PDF is not available
... Analyses of trial orders have been incorporated in recent psychological and psychophysiological studies, including the studies of face and pain perception and cognitive controls, to investigate cognitive processes underlying what has been referred to as habituation and learning (Volpert-Esmond, Merkle, Levsen, Ito, & Bartholow, 2018;Von Gunten, Volpert-Esmond, & Bartholow, 2018;Vossen, Van Breukelen, Hermens, Van Os, & Lousberg, 2011). In contrast, such an investigation has not been conducted until recently in the literature of human language processing (see Delaney-Busch et al., 2019;Nieuwland et al., 2018;Smith & Kutas, 2015a, 2015bYano, 2018). In the present study, we use mixed-effects modeling to examine how the language-processing system dynamically changes the way it processes syntactic violations and discuss an implication for the representation-based and expectation-based accounts. ...
... The dependent variables included mean amplitudes calculated for each participant, trial, and electrode. 6 Although P600 effects were expected to appear in response to morphosyntactic violations in the time windows of 700-900 ms on the basis of previous studies in Japanese (Mueller, Hirotani, & Friederici, 2007;Nashiwa, Nakao, & Miyatani, 2007;Yano, 2018;Yano & Sakamoto, 2016;Yano, Suzuki, & Koizumi, 2018), the mean amplitudes of 300-500 ms and 500-700 ms were also analyzed to examine at which time-windows the ERPs started to change over the course of the experiment. The division of a P600 time-window into early and late time-windows (i.e., early P600 time-window of 500-700 ms and late P600 time-window of 700-900 ms) is motivated by previous studies showing that different types of sentences show a P600 effect with different peak latencies and topographical distributions (Barber & Carreiras, 2005;Carreiras, Salillas, & Barber, 2004;Hagoort & Brown, 2000;Kaan & Swaab, 2003b, 2003aMolinaro, Barber, & Carreiras, 2011, see also footenote 2). ...
... Since the peak latency of P600 is response-aligned (Sassenhagen & Bornkessel-Schlesewsky, 2015;Sassenhagen, Schlesewsky, & Bornkessel-Schlesewsky, 2014), it is plausible to expect a late P600 effect rather than an early P600 effect. Furthermore, previous experiments using similar materials and the same instruction have observed a P600 at the late time-window (Yano, 2018;Yano & Sakamoto, 2016;Yano et al., 2019. ...
Article
Full-text available
The present study conducted two event-related potential experiments to investigate whether readers adapt their expectations to morphosyntactically (Experiment 1) or semantically (Experiment 2) anomalous sentences when they are repeatedly exposed to them. To address this issue, we experimentally manipulated the probability of occurrence of grammatical sentences and syntactically and semantically anomalous sentences through experiments. For the low probability block, anomalous sentences were presented less frequently than grammatical sentences (with a ratio of 1 to 4), while they were presented as frequently as grammatical sentences in the equal probability block. Experiment 1 revealed a smaller P600 effect for morphosyntactic violations in the equal probability block than in the low probability block. Linear mixed-effect models were used to examine how the size of the P600 effect changed as the experiment went along. The results showed that the smaller P600 effect of the equal probability block resulted from an amplitude's decline in morphosyntactically violated sentences over the course of the experiment, suggesting an adaptation to morphosyntactic violations. In Experiment 2, semantically anomalous sentences elicited a larger N400 effect than their semantically natural counterparts regardless of probability manipulation. Little evidence was found in favor of adaptation to semantic violations in that the processing cost associated with the N400 did not decrease over the course of the experiment. Therefore, these results demonstrated a dynamic aspect of language-processing system. We will discuss why the language-processing system shows a selective adaptation to morphosyntactic violations. 2
... The present study used P600 data from Yano & Sakamoto (2016) and Yano (2018), and the direct replication of Yano & Sakamoto (2016), and P300 data from Yano et al. (2019). Although these studies tested native Japanese speakers to obtain a large dataset of P600, the present result of the sample size analysis can be generalised to other language speakers since the P600 effect of previous studies does not vary substantially across different languages. ...
... In the experiment, the participants were instructed to read a sentence silently and press either YES (acceptable) or NO (unacceptable) button when seeing a response cue presented 500 ms after the offset of the sentence. The only difference between Yano & Sakamoto (2016) and Yano (2018) is the manipulation of the SOA; the first phrase was presented for 800 ms followed by a 500 ms blank in Yano & Sakamoto (2016), whereas it was presented for 500 ms with a 100 ms blank in Yano (2018). The duration of the second phrase 2 Japanese marks the subject of intransitive and transitive sentences with the nominative case "-ga" (abbreviated as NOM in (1) and (2)) and an object with the accusative case "-o" (abbreviated as ACC) (i.e., nominative-accusative type language). ...
... In the experiment, the participants were instructed to read a sentence silently and press either YES (acceptable) or NO (unacceptable) button when seeing a response cue presented 500 ms after the offset of the sentence. The only difference between Yano & Sakamoto (2016) and Yano (2018) is the manipulation of the SOA; the first phrase was presented for 800 ms followed by a 500 ms blank in Yano & Sakamoto (2016), whereas it was presented for 500 ms with a 100 ms blank in Yano (2018). The duration of the second phrase 2 Japanese marks the subject of intransitive and transitive sentences with the nominative case "-ga" (abbreviated as NOM in (1) and (2)) and an object with the accusative case "-o" (abbreviated as ACC) (i.e., nominative-accusative type language). ...
Article
Full-text available
Reliably and efficiently detecting physiological differences between conditions of interest is of importance in psychophysiology. In particular, when it comes to the observation of relatively small differences, such as a P600 effect, a language-related brain potential elicited by ungrammatical sentences compared to grammatical sen- tences, inter-participant variability is a critical factor since a larger inter-participant variability decreases sta- tistical significance, and therefore increases the necessary sample size. The present study investigated how stable individual P600s are, at which sample sizes the P600 becomes stable, and how many participants are necessary to observe a P600 effect. P600s were recorded from 48 participants, as well as P300 (P3b) from 40 participants for comparison. Unlike the P3b effect, which had an approximately 10 μV difference between the target and standard stimuli, P600 increased in amplitude by only 1.4–1.7 μV at Pz during the processing of ungrammatical sentences relative to the grammatical counterparts. The sample size analysis suggests that 20 to 30 participants are needed to detect a P600 effect at Pz, and the distribution of variances does not change significantly with a larger sample size.
... In ERP research of visual narratives, anterior negativities (Fig. 3b), often with a leftward or bilateralized distribution, appear to index the cost of combinatorial processing of the narrative grammar Cohn et al., 2012Cohn et al., , 2014. Similar anterior negativities have been shown to violations of expectancies and combinatorial processing in language (Friederici, 2011;Neville, Nicol, Barss, Forster, & Garrett, 1991;Yano, 2018) and music (Koelsch, Gunter, Wittfoth, & Sammler, 2005;Patel, 2003). Anterior negativities in visual narratives appear to be insensitive to situational discontinuity, such as an incongruous change of characters, but they are sensitive to differences in narrative patterning . ...
Article
Full-text available
The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.
... If the incoming panel satisfies the bottom-up constraints, or allows for such assignment, the top-down schema can thereby determine the narrative category. In ERPs, violations of those forward-looking predictions results in an anterior negativity, which indexes the disconfirmation of structural predictions (Kluender & Kutas, 1993;Yano, 2018), or increased cost of combinatorial structure building (Hagoort, 2017). In visual narratives, such anterior negativities have been observed to comparisons between semantically incongruous sequences that varied in the presence of narrative structure , to violations of narrative constituent structure (Cohn, Jackendoff, Holcomb, & Kuperberg, 2014), and to narrative conjunction compared to non-conjunction sequences . ...
Chapter
Full-text available
The way we understand a narrative sequence of images may seem effortless, given the prevalence of comics and picture stories across contemporary society. Yet, visual narrative comprehension involves greater complexity than is often acknowledged, as suggested by an emerging field of psychological research. This work has contributed to a growing understanding of how visual narratives are processed, how such mechanisms overlap with those of other expressive modalities like language, and how such comprehension involves a developmental trajectory that requires exposure to visual narrative systems. Altogether, such work reinforces visual narratives as a basic human expressive capacity carrying great potential for exploring fundamental questions about the mind.
Article
Full-text available
We conducted an event-related potential experiment using aspectual coercion to examine whether aspectual information is predictively processed prior to verb input in Japanese. In the experiment, experimental sentences were presented with two presentation durations (500 ms vs. 800 ms) to manipulate the temporal predictability of a verb. Aspectually coerced sentences elicited an early anterior negativity (AN) only when participants had a sufficient time to generate an aspectual prediction for a verb. A late AN was observed in aspectual coercion irrespective of the manipulation of the duration. This result indicates that early AN is elicited by a prediction error pertaining to aspectual information, suggesting that aspectual information is computed predictively based on adverbials without waiting for verb information.
Article
Full-text available
Prediction in sentence comprehension is often investigated by measuring the amplitude of the N400 event-related potential (ERP) component to words that are more or less predictable from their preceding context. The N400 – linked to the activation of word-associated semantic information – is reduced for words that are predictable, indicating that preactivation can lead to facilitated processing. We addressed the question whether there is measurable neural activity related to the preactivation of linguistic information before input confirms or disconfirms this prediction. We therefore measured ERPs not only to moderately to highly predictable target-words, but also to preceding adverbs. Based on two separate cloze pretests we quantified the impact of the adverb upon the predictability of the subsequent target-word. Using linear mixed effects analyses we could show that the N400-amplitude at the target-word was inversely related to target cloze-value, thus replicating the finding that prediction has a facilitative effect on semantic processing. Crucially, the N400-amplitude at the pretarget adverb was modulated by adverb impact: When adverbs increased the predictability of the following word, the N400 was more negative-going. We argue that this effect is related to the preactivation of linguistic information. Our findings indicate that the specification of predictions can lead to additional processes before these predictions are confirmed or disconfirmed and that activation of word-associated information through prediction is highly comparable to activation through actual input.
Article
Full-text available
One of the frequent questions by users of the mixed model function lmer of the lme4 package has been: How can I get p values for the F and t tests for objects returned by lmer? The lmerTest package extends the 'lmerMod' class of the lme4 package, by overloading the anova and summary functions by providing p values for tests for fixed effects. We have implemented the Satterthwaite's method for approximating degrees of freedom for the t and F tests. We have also implemented the construction of Type I - III ANOVA tables. Furthermore, one may also obtain the summary as well as the anova table using the Kenward-Roger approximation for denominator degrees of freedom (based on the KRmodcomp function from the pbkrtest package). Some other convenient mixed model analysis tools such as a step method, that performs backward elimination of nonsignificant effects - both random and fixed, calculation of population means and multiple comparison tests together with plot facilities are provided by the package as well.
Article
Abstract Most neurocognitive models of language processing generally assume population-wide homogeneity in the neural mechanisms used during language comprehension, yet individual differences are known to influence these neural mechanisms. In this study, we focus on handedness as an individual difference hypothesized to affect language comprehension. Left-handers and right-handers with a left-handed blood relative, or familial sinistrals, are hypothesized to process language differently than right-handers with no left-handed relatives (Hancock and Bever, 2013 ; Ullman, 2004). Yet, left-handers are often excluded from neurocognitive language research, and familial sinistrality in right-handers is often not taken into account. In the current study we used event-related potentials to test morphosyntactic processing in three groups that differed in their handedness profiles: left-handers (LH), right-handers with a left-handed blood relative (RH FS+), and right-handers with no reported left-handed blood relative (RH FS-; both right-handed groups were previously tested by Tanner and Van Hell, 2014). Results indicated that the RH FS- group showed only P600 responses during morphosyntactic processing whereas the LH and RH FS+ groups showed biphasic N400-P600 patterns. N400s in LH and RH FS+ groups are consistent with theories that associate left-handedness (self or familial) with increased reliance on lexical/semantic mechanisms during language processing. Inspection of individual-level results illustrated that variability in RH FS- individuals’ morphosyntactic processing was remarkably low: most individuals were P600-dominant. In contrast, LH and RH FS+ individuals showed marked variability in brain responses, which was similar for both groups: half of individuals were N400-dominant and half were P600-dominant. Our findings have implications for neurocognitive models of language that have been largely formulated around data from only right-handers without accounting for familial sinistrality or including left-handers, and moreover highlight that there is systematic – and often ignored – variability in language processing outcomes in neurologically healthy populations.
Article
Event-related brain potentials (ERPs) have revealed multiple mechanisms by which contextual constraints impact language processing. At the same time, little work has examined the trial-to-trial dynamics of context use in the brain. In the current study, we probed intraindividual variability in behavioral and neural indices of context processing during reading. In a concurrent self-paced reading and ERP paradigm, participants read sentences that were either strongly or weakly constraining completed with an expected or unexpected target word. Our findings revealed substantial within-subject variability in behavioral and neural responses to contextual constraints. First, context-based amplitude reductions of the N400, a component linked to semantic memory access, were largest among trials eliciting the slowest RTs. Second, the RT distribution of unexpected words in strongly constraining contexts was positively skewed, reflecting an increased proportion of very slow RTs to trials that violated semantic predictions. Among those prediction-violating trials eliciting faster RTs, a late sustained anterior positivity was observed. However, among trials producing the differentially slowed RTs to prediction violations, we observed a markedly earlier effect of constraint in the form of an anterior N2, a component linked to conflict resolution and the cognitive control of behavior. The current study provides the first neurophysiological evidence for the direct role of cognitive control functions in the volitional control of reading. Collectively, our findings suggest that context use varies substantially within individual participants and that coregistering behavioral and neural indices of online sentence processing offers a window into these single-item dynamics.
Article
An important question in the reading literature regards the nature of the semantic information readers can extract from the parafovea (i.e., the next word in a sentence). Recent eye-tracking findings have found a semantic parafoveal preview benefit under many circumstances, and findings from event-related brain potentials (ERPs) also suggest that readers can at least detect semantic anomalies parafoveally (Barber, Van der Meij, & Kutas, Psychophysiology, 50(1), 48–59, 2013). We use ERPs to ask whether fine-grained aspects of semantic expectancy can affect the N400 elicited by a word appearing in the parafovea. In an RSVP-with-flankers paradigm, sentences were presented word by word, flanked 2° bilaterally by the previous and upcoming words. Stimuli consisted of high constraint sentences that were identical up to the target word, which could be expected, unexpected but plausible, or anomalous, as well as low constraint sentences that were always completed with the most expected ending. Findings revealed an N400 effect to the target word when it appeared in the parafovea, which was graded with respect to the target’s expectancy and congruency within the sentence context. Furthermore, when targets appeared at central fixation, this graded congruency effect was mitigated, suggesting that the semantic information gleaned from parafoveal vision functionally changes the semantic processing of those words when foveated.