Stefan L. Frank's research while affiliated with Radboud University and other places

Publications (111)

Preprint
An eye-tracking experiment was conducted with speakers of Dutch (N=84, 36 male), a language that falls in between grammatical and natural gender languages. We tested whether a masculine generic pronoun causes a male bias when used in generic statements, that is, in the absence of a specific referent. We tested two different types of generic stateme...
Article
Full-text available
Many computational models of speech recognition assume that the set of target words is already given. This implies that these models learn to recognise speech in a biologically unrealistic manner, i.e. with prior lexical knowledge and explicit supervision. In contrast, visually grounded speech models learn to recognise speech without prior lexical...
Preprint
The ecology of communication is face-to-face. In these contexts, speakers dynamically modify their communication across vocal (e.g., speaking rate) and gestural (e.g., co-speech gestures related in meaning to the content of speech) channels while speaking. What is the function of these adjustments? Here we test the hypothesis that speakers produced...
Preprint
Background: Computational models of speech recognition often assume that the set of target words is already given. This implies that these models do not learn to recognise speech from scratch without prior knowledge and explicit supervision. Visually grounded speech models learn to recognise speech without prior knowledge by exploiting statistical...
Article
Full-text available
Words typically form the basis of psycholinguistic and computational linguistic studies about sentence processing. However, recent evidence shows the basic units during reading, i.e., the items in the mental lexicon, are not always words, but could also be sub-word and supra-word units. To recognize these units, human readers require a cognitive me...
Preprint
Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are purely text based, even though the human sensory experience is much richer. In this paper we create visually grounded word embeddin...
Article
Full-text available
To test whether error-driven implicit learning can explain cross-language structural priming, we implemented three different models of bilingual sentence production: Spanish-English, verb-final Dutch-English, and verb-medial Dutch-English. With these models, we conducted simulation experiments that all revealed clear and strong cross-language primi...
Preprint
Words typically form the basis of psycholinguistic and computational linguistic studies about sentence processing. However, recent evidence shows the basic units during reading, i.e., the items in the mental lexicon, are not always words, but could also be sub-word and supra-word units. To recognize these units, human readers require a cognitive me...
Preprint
This study addresses the question whether visually grounded speech recognition (VGS) models learn to capture sentence semantics without access to any prior linguistic knowledge. We produce synthetic and natural spoken versions of a well known semantic textual similarity database and show that our VGS model produces embeddings that correlate well wi...
Preprint
To test whether error-driven implicit learning can explain cross-language structural priming, we implemented three different models of bilingual sentence production: Spanish-English, verb-final Dutch-English, and verb-medial Dutch-English. With these models, we conducted simulation experiments that all revealed clear and strong cross-language primi...
Article
Full-text available
English sentences with double center-embedded clauses are read faster when they are made ungrammatical by removing one of the required verb phrases. This phenomenon is known as the missing-VP effect. German and Dutch speakers do not experience the missing-VP effect when reading their native language, but they do when reading English as a second lan...
Article
Full-text available
Two experiments tested whether the Dutch possessive pronoun zijn ‘his’ gives rise to a gender inference and thus causes a male bias when used generically in sentences such as Everyone was putting on his shoes . Experiment 1 ( N = 120, 48 male) was a conceptual replication of a previous eye-tracking study that had not found evidence of a male bias....
Conference Paper
Full-text available
Language users process utterances by segmenting them into many cognitive units, which vary in their sizes and linguistic levels. Although we can do such unitization/segmentation easily, its cognitive mechanism is still not clear. This paper proposes an unsupervised model, Less-is-Better (LiB), to simulate the human cognitive process with respect to...
Article
We trained a computational model (the Chunk‐Based Learner; CBL) on a longitudinal corpus of child–caregiver interactions in English to test whether one proposed statistical learning mechanism—backward transitional probability—is able to predict children's speech productions with stable accuracy throughout the first few years of development. We pred...
Preprint
A self-paced reading experiment tested if a generically-used masculine personal pronoun leads to a male bias in online processing. We presented Dutch native speakers (N=95, 47 male) with generic statements featuring the masculine pronoun hij ‘he’ (e.g., Someone who always promises that he will really be on time, such as Ms/Mr Knoop, will sometimes...
Preprint
Two experiments tested whether the Dutch possessive pronoun zijn ‘his’ gives rise to a gender inference and thus causes a male bias when used generically in sentences such as Everyone was putting on his shoes. Experiment 1 (N = 120, 48 male) was a conceptual replication of a previous eye-tracking study that had not found evidence of a male bias. Th...
Article
Full-text available
Spanish–English bilinguals rarely code-switch in the perfect structure between the Spanish auxiliary haber (“to have”) and the participle (e.g., “ Ella ha voted”; “She has voted”). However, they are somewhat likely to switch in the progressive structure between the Spanish auxiliary estar (“to be”) and the participle (“ Ella está voting”; “She is v...
Article
Full-text available
Code-switching is the alternation from one language to the other during bilingual speech. We present a novel method of researching this phenomenon using computational cognitive modeling. We trained a neural network of bilingual sentence production to simulate early balanced Spanish–English bilinguals, late speakers of English who have Spanish as a...
Preprint
A central question in the psycholinguistic study of multilingualism is how syntax is shared across languages. We implement a model to investigate whether error-based implicit learning can provide an account of cross-language structural priming. The model is based on the Dual-path model of sentence-production (Chang, 2002). We implement our model us...
Preprint
Recurrent neural networks (RNNs) have long been an architecture of interest for computational models of human sentence processing. The more recently introduced Transformer architecture has been shown to outperform recurrent neural networks on many natural language processing tasks but little is known about their ability to model human language proc...
Article
Full-text available
Although computational models can simulate aspects of human sentence processing, research on this topic has remained almost exclusively limited to the single language case. The current review presents an overview of the state of the art in computational cognitive models of sentence processing, and discusses how recent sentence‐processing models can...
Chapter
A unique overview of the human language faculty at all levels of organization. Language is not only one of the most complex cognitive functions that we command, it is also the aspect of the mind that makes us uniquely human. Research suggests that the human brain exhibits a language readiness not found in the brains of other species. This volume br...
Article
Full-text available
When the middle verb phrase is removed from an English double-embedded sentence, the remainder of the sentence is read faster in spite of the ungrammaticality. It has been shown that this “missing-VP effect” is reversed in German and Dutch. The current study demonstrates that the same cross-linguistic difference holds for sentences judgments: Nativ...
Preprint
Humans learn language by interaction with their environment and listening to other humans. It should also be possible for computational models to learn language directly from speech but so far most approaches require text. We improve on existing neural network approaches to create visually grounded embeddings for spoken utterances. Using a combinat...
Preprint
English sentences with double center-embedded clauses are read faster when they are made ungrammatical by removing one of the required verb phrases. This phenomenon is known as the missing-VP effect. German and Dutch speakers do not experience the missing-VP effect when reading their native language, but they do when reading English as a second lan...
Article
Full-text available
We review information-theoretic measures of cognitive load during sentence processing that have been used to quantify word prediction effort. Two such measures, surprisal and next-word entropy, suffer from shortcomings when employed for a predictive processing view. We propose a novel metric, lookahead information gain, that can overcome these shor...
Preprint
We review information-theoretic measures of cognitive load during sentence processing that have been used to quantify word prediction effort. Two such measures, surprisal and next-word entropy, suffer from shortcomings when employed for a predictive processing view. We propose a novel metric, lookahead information gain, that can overcome these shor...
Article
Full-text available
Current approaches to learning semantic representations of sentences often use prior word-level knowledge. The current study aims to leverage visual information in order to capture sentence level semantics without the need for word embeddings. We use a multimodal sentence encoder trained on a corpus of images with matching text captions to produce...
Preprint
Full-text available
Multilingual speakers are able to switch from one language to the other ("code-switch'') between or within sentences. Because the underlying cognitive mechanisms are not well understood, in this study we use computational cognitive modeling to shed light on the process of code-switching.We employed the Bilingual Dual-path model, a Recurrent Neural...
Preprint
Backward saccades during reading have been hypothesized to be involved in structural reanalysis, or to be related to the level of text difficulty. We test the hypothesis that backward saccades are involved in online syntactic analysis. If this is the case we expect that saccades will coincide, at least partially, with the edges of the relations com...
Preprint
Current approaches to learning semantic representations of sentences often use prior word-level knowledge. The current study aims to leverage visual information in order to capture sentence level semantics without the need for word embeddings. We use a multimodal sentence encoder trained on a corpus of images with matching text captions to produce...
Conference Paper
Full-text available
Multilingual speakers are able to switch from one language to the other (“code-switch”) between or within sentences. Because the underlying cognitive mechanisms are not well understood, in this study we use computational cognitive modeling to shed light on the process of code-switching. We employed the Bilingual Dual-path model, a Recurrent Neural...
Article
Full-text available
The contents and structure of semantic memory have been the focus of much recent research, with major advances in the development of distributional models, which use word co‐occurrence information as a window into the semantics of language. In parallel, connectionist modeling has extended our knowledge of the processes engaged in semantic activatio...
Preprint
The Simple Recurrent Network (SRN) has a long tradition in cognitive models of language processing. More recently, gated recurrent networks have been proposed that often outperform the SRN on natural language processing tasks. Here, we investigate whether two types of gated networks perform better as cognitive models of sentence reading than SRNs,...
Data
Representing words as syntactic categories. (PDF)
Data
Average euclidean length of vector representations at each point of 7- and 8-syllable Chinese [NP VP] sentences. Error bars indicate 95% confidence intervals. Plots are left aligned on the 3- or 4-syllable NP (labeled N1 to N4) as well as right aligned on the 4- or 5-syllable VP (labeled V−4 to V0). Sentence with a 3-syllable NP have no N4 and 4-sy...
Data
Power spectra resulting from processing English [NP VP] sentences with each word replaced by its most frequent syntactic category. (EPS)
Data
Localization of sentential- and phrasal-rate responses. (PDF)
Data
Vector lengths over the course of a sentence. (PDF)
Data
Setting model parameters. (PDF)
Data
Power spectra resulting from processing English [NP VP] sentences, for β = 50 and different values of μ and σ. (EPS)
Data
Power spectra resulting from processing English [NP VP] sentences, for different combinations of parameter values. (EPS)
Data
Model predictions on word salad conditions. Stimuli sequences were constructed by randomly drawing (with replacement) words from the English [NP VP] sentences. Left: all words randomly drawn. Right: adjectives and verbs keep their original position in the [NP VP] stimuli. (EPS)
Data
Matlab code for computing, plotting, and statistically analyzing power spectra. (ZIP)
Data
Matlab workspace with 7- and 8-syllable stimuli and their vector representations. (ZIP)
Conference Paper
Full-text available
We present the Narrative Brain Dataset, an fMRI dataset that was collected during spoken presentation of short excerpts of three stories in Dutch. Together with the brain imaging data, the dataset contains the written versions of the stimulation texts. The texts are accompanied with stochastic (perplexity and entropy) and semantic computational lin...
Preprint
The contents and structure of semantic memory have been the focus of much recent research, with major advances in the development of distributional models, which use word co-occurrence information as a window into the semantics of language. In parallel, connectionist modeling has extended our knowledge of the processes engaged in semantic activatio...
Article
Ding et al. (2017) contrast their view that language processing is based on hierarchical syntactic structures, to a view that relies on word-level input statistics. In this response to their paper, we clarify how, exactly, the two views differ (and how they do not), and make a case for the importance of sequential, as opposed to hierarchical, struc...
Article
Cognitive neuroscientists of language comprehension study how neural computations relate to cognitive computations during comprehension. On the cognitive part of the equation, it is important that the computations and processing complexity are explicitly defined. Probabilistic language models can be used to give a computationally explicit account o...
Preprint
Full-text available
Cognitive neuroscientists of language comprehension study how neural computations relate to cognitive computations during comprehension. On the cognitive part of the equation, it is important that the computations and processing complexity are explicitly defined. Probabilistic language models can be used to give a computationally explicit account o...
Conference Paper
It has been claimed that larger semantic distance between the words of a sentence, as quantified by a distributional semantics model, increases both N400 size and word-reading time. The current study shows that the reading-time effect disappears when word surprisal is factored out, suggesting that the earlier findings were caused by a confound betw...
Preprint
When the middle verb phrase is removed from an English double-embedded sentence, the remainder of the sentence is read faster in spite of the ungrammaticality. It has been shown that this "missing-VP effect" is reversed in German and Dutch. The current study demonstrates that the same cross-linguistic difference holds for sentences judgments: Nativ...
Article
Full-text available
Results from a recent MEG study on spoken sentence comprehension (Ding et al., 2016) have been interpreted as evidence for cortical entrainment to hierarchical syntactic structure. We present a simple computational model that predicts the power spectra from this study, even though the model's linguistic knowledge is restricted to the lexical level...
Article
Full-text available
Language comprehension involves the simultaneous processing of information at the pho-nological, syntactic, and lexical level. We track these three distinct streams of information in the brain by using stochastic measures derived from computational language models to detect neural correlates of phoneme, part-of-speech, and word processing in an fMR...
Article
We investigate the effects of two types of relationship between the words of a sentence or text – predictability and semantic similarity – by reanalysing electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data from studies in which participants comprehend naturalistic stimuli. Each content word's predictability given prev...
Poster
Full-text available
Probabilistic language models have been proven useful tools for studying how language is processed as a linear structure. Conditional probabilities between sequences of surface forms are at the basis of probabilistic measures such as surprisal and perplexity which have been successfully used as predictors of several behavioural and neural correlate...
Conference Paper
Statistical learning (SL) is increasingly invoked as a set of general-purpose mechanisms upon which language learning is built during infancy and childhood. Here we investigated the extent to which SL is related to adult language processing. In particular, we asked whether SL proclivities towards relations that are more informative of English are r...
Conference Paper
The contents and structure of semantic networks have been the focus of much recent research, with major advances in the development of distributional models. In parallel, connectionist modeling has extended our knowledge of the processes engaged in semantic activation. However, these two lines of investigation have rarely brought together. Here, st...
Article
Prior language input is not lost but integrated with the current input. This principle is demonstrated by “reservoir computing”: Untrained recurrent neural networks project input sequences onto a random point in high-dimensional state space. Earlier inputs can be retrieved from this projection, albeit less reliably so as more input is received. The...
Article
Full-text available
The notion of prediction is studied in cognitive neuroscience with increasing intensity. We investigated the neural basis of 2 distinct aspects of word prediction, derived from information theory, during story comprehension. We assessed the effect of entropy of next-word probability distributions as well as surprisal. A computational model determin...
Chapter
Inferencing is defined as 'the act of deriving logical conclusions from premises known or assumed to be true', and it is one of the most important processes necessary for successful comprehension during reading. This volume features contributions by distinguished researchers in cognitive psychology, educational psychology, and neuroscience on topic...
Article
Full-text available
Reading times on words in a sentence depend on the amount of information the words convey, which can be estimated by probabilistic language models. We investigate whether event-related potentials (ERPs), too, are predicted by information measures. Three types of language models estimated four different information measures on each word of a sample...
Chapter
Research on inferences in discourse processing has been characterized for a long time by the question of what kinds of inference are likely made online and what kinds of inference are not. Considering that in principle an unlimited number of inferences can be made while the human processing capacity is limited, the question arises how the inference...
Article
Over the past 15 years, there have been two increasingly popular approaches to the study of meaning in cognitive science. One, based on theories of embodied cognition, treats meaning as a simulation of perceptual and motor states. An alternative approach treats meaning as a consequence of the statistical distribution of words across spoken and writ...
Article
This chapter empirically investigates the issue of systematicity and connectionism under more realistic conditions than was the case in previous studies. A connectionist and a symbolic model of sentence processing are compared on their ability to perform systematically. Both models are trained on over 700,000 sentences, and tested on 361 sentences,...
Article
An English double-embedded relative clause from which the middle verb is omitted can often be processed more easily than its grammatical counterpart, a phenomenon known as the grammaticality illusion. This effect has been found to be reversed in German, suggesting that the illusion is language specific rather than a consequence of universal working...
Conference Paper
We investigated the effect of word sur-prisal on the EEG signal during sentence reading. On each word of 205 experimental sentences, surprisal was estimated by three types of language model: Markov models, probabilistic phrase-structure grammars, and recurrent neural networks. Four event-related potential components were extracted from the EEG of 2...
Article
The entropy-reduction hypothesis claims that the cognitive processing difficulty on a word in sentence context is determined by the word's effect on the uncertainty about the sentence. Here, this hypothesis is tested more thoroughly than has been done before, using a recurrent neural network for estimating entropy and self-paced reading for obtaini...
Article
Full-text available
We make available word-by-word self-paced reading times and eye-tracking data over a sample of English sentences from narrative sources. These data are intended to form a gold standard for the evaluation of computational psycholinguistic models of sentence comprehension in English. We describe stimuli selection and data collection and present descr...
Article
It is generally assumed that hierarchical phrase structure plays a central role in human language. However, considerations of simplicity and evolutionary continuity suggest that hierarchical structure should not be invoked too hastily. Indeed, recent neurophysiological, behavioural and computational studies show that sequential sentence structure h...
Conference Paper
Probabilistic accounts of language processing can be psychologically tested by comparing word-reading times (RT) to the conditional word probabilities estimated by language models. Using surprisal as a linking function, a significant correlation between unlexicalized surprisal and RT has been reported (e.g., Demberg and Keller, 2008), but success u...
Conference Paper
Full-text available
This study investigated the relation between word surprisal and pupil dilation during reading. Participants' eye movements and pupil size were recorded while they read single sentences. Surprisal values for each word in the sentence stimuli were estimated by both a recurrent neural network and a phrase-structure grammar. Higher surprisal correspond...