Harm Brouwer

Tilburg University | UVT · Department of Cognitive Science and Artificial Intelligence

In traditional theories of language comprehension, syntactic and semantic processing are inextricably linked. This assumption has been challenged by the 'semantic illusion effect' found in studies using event related brain potentials. Semantically anomalous sentences did not produce the expected increase in N400 amplitude but rather one in P600 amp...

Ten years ago, researchers using event-related brain potentials (ERPs) to study language comprehension were puzzled by what looked like a Semantic Illusion: Semantically anomalous, but structurally well-formed sentences did not affect the N400 component---traditionally taken to reflect semantic integration---but instead produced a P600-effect, whic...

Event-Related Potentials (ERPs)—stimulus-locked, scalp-recorded voltage fluctuations caused by post-synaptic neural activity—have proven invaluable to the study of language comprehension. Of interest in the ERP signal are systematic, reoccurring voltage fluctuations called components, which are taken to reflect the neural activity underlying specif...

Event-related potentials (ERPs) provide a multidimensional and real-time window into neurocognitive processing. The typical Waveform-based Component Structure (WCS) approach to ERPs assesses the modulation pattern of components - systematic, reoccurring voltage fluctuations reflecting specific computational operations - by looking at mean amplitude...

Expectation-based theories of language comprehension, in particular Surprisal Theory, go a long way in accounting for the behavioral correlates of word-by-word processing difficulty, such as reading times. An open question, however, is in which component(s) of the Event-Related brain Potential (ERP) signal Surprisal is reflected, and how these elec...

Theories of the electrophysiology of language comprehension are mostly informed by event-related potential effects observed between condition averages. We here argue that a dissociation between competing effect-level explanations of event-related potentials can be achieved by turning to predictions and analyses at the single-trial level. Specifical...

The Cambridge Handbook of Computational Cognitive Sciences is a comprehensive reference for this rapidly developing and highly interdisciplinary field. Written with both newcomers and experts in mind, it provides an accessible introduction of paradigms, methodologies, approaches, and models, with ample detail and illustrated by examples. It should...

The integration of word meaning into an unfolding utterance representation is a core operation of incremental language comprehension. There is considerable debate, however, as to which component of the ERP signal—the N400 or the P600—directly reflects integrative processes, with far reaching consequences for the temporal organization and architectu...

To appear in: Ron Sun (ed.), Cambridge Handbook of Computational Cognitive Sciences. CUP: Cambridge, UK.

Expectation-based theories of language processing, such as Surprisal theory, are supported by evidence of anticipation effects in both behavioural and neurophysiological measures. Online measures of language processing, however, are known to be influenced by factors such as lexical association that are distinct from—but often confounded with—expect...

Decades of studies trying to define the extent to which artificial neural networks can exhibit systematicity suggest that systematicity can be achieved by connectionist models but not by default. Here we present a novel connectionist model of sentence production that employs rich situation model representations originally proposed for modeling syst...

The problem of spatiotemporal overlap between event-related potential (ERP) components is generally acknowledged in language research. However, its implications for the interpretation of experimental results are often overlooked. In a previous experiment on the functional interpretation of the N400 and P600, it was argued that a P600 effect to impl...

Natural language semantics has recently sought to combine the complementary strengths of formal and distributional approaches to meaning. More specifically, proposals have been put forward to augment formal semantic machinery with distributional meaning representations, thereby introducing the notion of semantic similarity into formal semantics, or...

Theories and models of spoken word recognition aim to explain the process of accessing lexical knowledge given an acoustic realization of a word form. There is consensus that phonological and semantic information is crucial for this process. However, there is accumulating evidence that orthographic information could also have an impact on auditory...

Natural language semantics has recently sought to combine the complementary strengths of formal and distributional approaches to meaning. More specifically, proposals have been put forward to augment formal semantic machinery with distributional meaning representations, thereby introducing the notion of semantic similarity into formal semantics, or...

Language is processed on a more or less word-byword basis, and the processing difficulty induced by each word is affected by our prior linguistic experience as well as our general knowledge about the world. Surprisal and entropy reduction have been independently proposed as linking theories between word processing difficulty and probabilistic langu...

The functional interpretation of two salient language-sensitive ERP components – the N400 and the P600 – remains a matter of debate. Prominent alternative accounts link the N400 to processes related to lexical retrieval, semantic integration, or both, while the P600 has been associated with syntactic reanalysis or, alternatively, to semantic integr...

Presentation of the paper "A Framework for Distributional Formal Semantics". https://www.researchgate.net/publication/333961011_A_Framework_for_Distributional_Formal_Semantics

Formal semantics and distributional semantics offer complementary strengths in capturing the meaning of natural language. As such, a considerable amount of research has sought to unify them, either by augmenting formal semantic systems with a distributional component, or by defining a formal system on top of distributed representations. Arriving at...

The processing difficulty of each word we encounter in a sentence is affected by both our prior linguistic experience and our general knowledge about the world. Computational models of incremental language processing have, however, been limited in accounting for the influence of world knowledge. We develop an incremental model of language comprehen...

The N400 and the P600 component are the most salient and well-studied Event-Related brain Potential (ERP) components for sentence processing, but their functional interpretation is subject to ongoing debate (Kutas et al., 2006; Brouwer et al., 2012). This has led to a proliferation of models, as illustrated for example by the diverse proposals that...

The study of language comprehension is ultimately about meaning: How can meaning be constructed from linguistic signal, and how can it be represented? The human language comprehension system is highly efficient and accurate at attributing meaning to linguistic input. Hence, in trying to identify computational principles and representations for mean...

The property of projection poses a challenge to formal semantic theories, due to its apparent non-compositional nature. Projected content is therefore typically analyzed as being different from and independent of asserted content. Recent evidence, however, suggests that these types of content in fact closely interact, thereby calling for a more int...

The effect of word association on sentence processing is still a matter of debate. Some studies observe no effect while others found a dependency on sentence congruity or an independent effect. In an attempt to separate the effects of sentence congruity and word association in the spatio-temporal domain, we jointly recorded scalp- and invasive-EEG...

The N400 and P600 are the two most salient language-sensitive components of the Event-Related Potential (ERP) signal. Yet, their functional interpretation is still a matter of debate. Traditionally, the N400 is taken to reflect processes of semantic integration while the P600 is linked to structural reanalysis [1,2]. These views have, however, been...

In online language comprehension, the N400 component of the Event-Related Potentials (ERP) signal is inversely proportional to semantic expectancy (Kutas & Federmeier, 2011). Among other factors, a word’s expectancy is influenced by both lexical-level (Bentin et al., 1985) as well as event-level (Metusalem et al., 2012) priming: the N400 amplitude...

Appendix S1. Simulation materials. Appendix S2. Derivation of word meaning representations. Appendix S3. Details of the training procedure. Appendix S4. Training on perfect word meaning representations.

In the ERP signal, referential ambiguities most commonly elicit Nref effects relative to controls. However, it is unclear whether Nref is sensitive to the degree of ambiguity (i.e. referential entropy). Previous work has also found important individual differences -- some participants instead show a late positive component (LPC). Finally, no ERP ma...

In the ERP signal, referential ambiguities most commonly elicit Nref effects relative to controls. However, it is unclear whether Nref is sensitive to the degree of ambiguity (i.e. referential entropy). Previous work has also found important individual differences -- some participants instead show a late positive component (LPC). Finally, no ERP ma...

Keynote Lecture

The study of language is ultimately about meaning: How can meaning be constructed from linguistic signal, and how can it be represented? The human language comprehension system is highly efficient and accurate at attributing meaning to linguistic input. Hence, in trying to identify computational principles and representations for meaning constructi...

A defining characteristic of human language is systematicity: “the ability to produce/understand some sentences is intrinsically connected to the ability to produce/understand certain others” (Fodor & Pylyshyn, 1988). Further, Fodor and Pylyshyn (1988) argue that connectionist models are not able to display systematicity without implementing a clas...

A novel connectionist model of sentence production is presented , which employs rich situation model representations originally proposed for modeling systematicity in comprehension (Frank, Haselager, & van Rooij, 2009). The high overall performance of our model demonstrates that such representations are not only suitable for comprehension, but also...

We present a neurocomputational model of the electrophysiology of language processing. Our model is explicit about its architecture and the computational principles and representations involved. It is effectively a recurrent neural network (of the ‘Elman’-type; [1]) that constructs a situation model of the state-of-the-affairs described by a senten...

Eye movements during linguistic-visual conflicts

We present a neurocomputational—recurrent artificial neural network—model of language processing that integrates linguistic knowledge and world/event knowledge, and that produces word surprisal estimates that take into account both. Our model constructs a cognitively motivated situation model of the state-of-the-affairs as described by a sentence....

We present a neurocomputational model of the electrophysiology of language processing. Our model is explicit about its architecture and the computational principles and representations involved. It is effectively a recurrent neural network (of the ‘Elman’-type; [1]) that directly instantiates a parsimonious functional-anatomic processing network li...

Visual attention can be directed by visual and linguistic information. It is not well understood how attention is directed when linguistic information conflicts with the visual scene. Knoeferle and Crocker (2006) established the coordinated interplay account model of sentence comprehension and linguistically mediated visual attention, but it did no...

One decade ago, researchers using Event-Related brain Potential (ERP) measurements stumbled upon what looked like a Semantic Illusion in language comprehension: Semantically anomalous, but otherwise well-formed sentences did not affect the meaning-related N400 component, but instead increased the amplitude of the structure-related P600 component. T...

A Neurocomputational Model of the N400 and the P600

We introduce PDRT-SANDBOX, a Haskell library that implements Projective Discourse Representation Theory (PDRT) (Venhuizen et al., 2013), an extension of Discourse Representation Theory (DRT) (Kamp, 1981; Kamp and Reyle, 1993). The implementation includes a translation from PDRT to DRT and first-order logic, composition via different types of merge,...

Implementing Projective Discourse Representation Theory

One decade ago, researchers using event-related brain potential (ERP) measurements stumbled upon what looked like a Semantic Illusion in language comprehension: Semantically anomalous, but otherwise well-formed sentences did not affect the meaning-related N400 component, but instead increased the amplitude of the structure-related P600 component. T...

Research into the electrophysiology of language comprehension has essentially been “speakerless.” This has left three vital aspects of communication—it is social, pragmatic, and dynamic—severely underresearched. This chapter makes a case for the investigation of language users involved in active conversation and describes the problems and possibili...

Conventional Implicatures (CIs; in the sense of Potts 2005) are part of a larger class of projection phenomena. These phenomena also include presuppositions and anaphora, and can be described as content that is not at-issue (see Simons, Tonhauser, Beaver & Roberts 2010). Despite the shared property of projection, CIs differ from other projection ph...

(See the abstract of the paper with the same title)

We propose a new functional-anatomical mapping of the N400 and the P600 to a minimal cortical network for language comprehension. Our work is an example of a recent research strategy in cognitive neuroscience, where researchers attempt to align data regarding the nature and time-course of cognitive processing (from ERPs) with data on the cortical o...

It sometimes happens that when someone asks a question, the addressee does not give an adequate answer, for instance by leaving out part of the required information. The person who posed the question may wonder why the information was omitted, and engage in extensive processing to find out what the partial answer actually means. The present study l...

People do not always behave as ideal interactants. When asked a question, they may bluntly say “No”, without apologizing or giving reasons. In an ERP experiment (measuring Event Related brain Potentials), we tested how language users process polite and less polite answers to different types of requests.

It sometimes happens that when someone asks a question, the addressee does not give an adequate answer, for instance by leaving out information. The person who posed the question may wonder why the information was omitted, and engage in extensive processing to find out what the partial answer means. In an ERP experiment (measuring Event Related bra...

In traditional theories of language comprehension, syntactic and semantic processing are inextricably linked. This assumption has been challenged by the ’semantic illusion effect’ found in studies using event related brain potentials. Semantically anomalous sentences did not produce the expected increase in N400 amplitude but rather one in P600 amp...

This paper investigates whether surprisal theory can account for differential processing difficulty in the NP-/S-coordination ambiguity in Dutch. Surprisal is es-timated using a Probabilistic Context-Free Grammar (PCFG), which is induced from an automatically annotated corpus. We find that our lexicalized surprisal model can account for the reading...