A Connectionist Model of Anticipation in Visual Worlds.

Full text

Page 1

A Connectionist Model of Anticipation in Visual Worlds
Marshall R. Mayberry, III, Matthew W. Crocker, and Pia Knoeferle
Department of Computational Linguistics,
Saarland University, Saarbr¨ ucken, Germany
{martym, crocker, knoferle}@coli.uni-sb.de
Abstract. Recent “visual worlds” studies, wherein researchers study language in
context by monitoring eye-movements in a visual scene during sentence process-
ing, have revealed much about the interaction of diverse information sources and
the time course of their influence on comprehension. In this study, five experi-
ments that trade off scene context with a variety of linguistic factors are modelled
with a Simple Recurrent Network modified to integrate a scene representation
withthe standard incremental input of asentence. The resultsshow that the model
captures the qualitative behavior observed during the experiments, while retain-
ing the ability to develop the correct interpretation in the absence of visual input.
1Introduction
There are two prevalent theories of language acquisition. One view emphasizes syntac-
tic and semantic bootstrappingduringlanguageacquisition that enable children to learn
abstract concepts from mappings between different kinds of information sources [1,2].
Another view emerges from connectionist literature and emphasizes the learning of lin-
guisticstructurefrompurelydistributionalpropertiesoflanguageusage[3,4].Whilethe
perspectives are often taken to be diametrically opposed, both can be seen as crucially
relying on correlations between words and their immediate context, be it the sentence
as a whole or extra-linguistic input, such as a scene.
We combine insights from both distributional and bootstrapping accounts in mod-
elling the on-line comprehensionof utterances in both the absence and presence of a vi-
sual scene. This is an importantachievementin at least two regards.First, it emphasizes
the complementarity between distributional and bootstrapping approaches–discovering
structure across linguistic and scene contexts [5]. Further, it is an important first step in
linking situated models of on-line utterance comprehension more tightly to accounts of
language acquisition, thus emphasizing the continuity of language processing.
We presentresultsfromtwosimulationsonaSimpleRecurrentNetwork(SRN; [3]).
Modification of the network to integrate input from a scene together with the charac-
teristic incremental processing of such networks allowed us to model people’s ability
to adaptively use the contextual information in order to more rapidly interpret and dis-
ambiguate a sentence. The model draws on recent studies that appeal to theories of lan-
guage acquisition to account for the comprehension of scene-related utterances [6,7].
Recent research within the visual worlds paradigm, wherein participants’ gazes to a
scenewhilelisteningtoanutterancearemonitored,providessupportforthisview.Find-
ingsfromthis paradigmsupportanaccountofscene-relatedutterancecomprehensionin
R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 849–861, 2005.
c ? Springer-Verlag Berlin Heidelberg 2005

Page 2

850M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle
which the rapid coordinated interaction of information from the immediate scene, and
linguisticknowledgeplays a majorrole in incrementaland anticipatorycomprehension.
2 Simulation 1
In Simulation 1, we simultaneously model four experiments that show the rapid influ-
ence of diverse informational sources–linguistic and world knowledge as well as scene
information–onutterancecomprehension.All experimentswereconductedin German,
a language that allows both subject-verb-object (SVO) and object-verb-subject (OVS)
sentence types. In the face of word order ambiguity, case marking indicates the subject
or object grammatical function, except in the case of feminine and neuter noun phrases
where the article does not distinguish the nominative and accusative cases.
2.1Anticipation Depending on Stereotypicality
The first two experiments that we modeled examined how linguistic and world knowl-
edge or stereotypicality enabled rapid thematic role assignment in unambiguous sen-
tences, thus determining who-does-what-to-whomin a scene.
Fig.1. Selectional Restrictions
Experiment 1: Morphosyntacticand lexical verb information. To examinethe influ-
ence of case-marking and verb plausibility on thematic role assignment, [8] presented
participants with utterances such as (1) or (2) that described a scene showing a hare, a
cabbage, a fox, and a distractor (see Figure 1) :
(1) Der Hase frisst gleich den Kohl.
The harenomeats shortly the cabbageacc.
(2) Den Hasen frisst gleich der Fuchs.
The hareacceats shortly the foxnom.
Afterhearing“Theharenomeats ...”and“Thehareacceats ...”,peoplemadeanticipatory
eye-movements to the cabbage and fox respectively. This reveals that people were able
to predict role fillers in a scene through linguistic/world knowledgethat identified who-
does-what-to-whom.

Page 3

A Connectionist Model of Anticipation in Visual Worlds851
Experiment 2: Verb type information. To further investigate the role of verb infor-
mation, the authors replaced the agent/patient verbs like frisst (“eats”) with experi-
encer/theme verbs like interessiert (“interests”). This manipulation interchanged agent
(experiencer)andpatient(theme)rolesfromExperiment1.ForFigure1andthesubject-
first (3) or object-first sentence (4), participants showed gaze fixations complementary
to those of Experiment 1, confirming that both case and semantic verb information are
used to predict relevant role fillers.
(3) Der Hase interessiert ganz besonders den Fuchs.
The harenominterests especially the foxacc.
(4) Den Hasen interessiert ganz besonders der Kohl.
The hareaccinterests especially the cabbagenom.
2.2 Anticipation Depending on Depicted Events
Thesecondset ofexperimentsinvestigatedwhetherdepictedeventsshowingwho-does-
what-to-whom can establish a scene character’s role as agent or patient when syntactic
and thematic role relations are temporarily ambiguous in the utterance.
Fig.2. Depicted Events
Experiment 3: Verb-mediated depicted role relations. [9] presented such initially
ambiguous spoken SVO (5) and OVS sentences (6) together with a scene in which a
princess both paints a fencer and is washed by a pirate (Figure 2):
(5) Die Princessin malt offensichtlich den Fechter.
The princessnompaints obviously the fenceracc.
(6) Die Princessin w¨ ascht offensichtlich der Pirat.
The princessaccwashes obviously the piratenom.
Linguistic disambiguation occurred on the second NP; disambiguation prior to the sec-
ond NP was only possible through use of the depicted events. When the verb identified
an action, the depicted role relations disambiguated towards either an agent-patient (5)
or patient-agent (6) role relation, as indicated by anticipatory eye-movementsto the pa-
tient (pirate) or agent (fencer)respectively for (5) and (6). This gaze-patternshowed the

Page 4

852 M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle
rapid influence of verb-mediateddepicted events on the assignment of thematic roles to
a temporarily ambiguous sentence-initial noun phrase.
Experiment 4:Weaktemporaladverbconstraint.[9] also investigatedGermanverb-
final active (7) and passive (8) constructions. In this type of sentence, the initial subject
noun phrase is role-ambiguous, and the auxiliary wird can have a passive or future
interpretation.
(7) Die Princessin wird sogleich den Pirat washen.
The princessnomwill right away wash the pirateacc.
(8) Die Princessin wird soeben von dem Fechter gemalt.
The princessaccis just now painted by the fencernom.
To evokeearly linguisticdisambiguation,temporaladverbsbiasedthe auxiliarywird to-
wardeitherthefuture(“will”)orpassive(“is-ed”)reading.Sincetheverbwassentence-
final,theinterplayofsceneandlinguisticcues(e.g.,temporaladverbs)wererathermore
subtle. When the listener heard a future-biased adverb such as sogleich, after the aux-
iliary wird, he interpreted the initial NP as agent of a future active construction, as evi-
denced by anticipatory eye-movementsto the patient in the scene. Conversely, listeners
interpreted the passive-biased construction soeben with these roles exchanged.
2.3 Architecture
The Simple Recurrent Network is a type of neural network typically used to process
temporal sequences of patterns such as words in a sentence. A common approach is
for the modeller to train the network on prespecified targets, such as verbs and their
arguments, that represent what the network is expected to produce upon completing a
sentence. Processing is incremental, with each new input word interpreted in the con-
text of the sentence processed so far, represented by a copy of the previous hidden
layer serving as additional input or context to the current hidden layer. Because these
types of associationist models automatically develop correlations among the data they
are trained on, they will generally develop expectations about the output even before
processing is completed because sufficient information occurs early in the sentence to
warrant such predictions. Moreover, during the course of processing a sentence these
expectations can be overridden with subsequent input, often abruptly revising an inter-
pretation in a manner reminiscent of how humans seem to process language. Indeed,
it is these characteristics of incremental processing, the automatic development of ex-
pectations, seamless integration of multiple sources of information, and nonmonotonic
revision that have endeared neural network models to cognitive researchers.
In Simulation 1, the four experiments described above have been modelled simul-
taneously using a single network. The goal of modelling all experimental results by a
single architecture required enhancements to the SRN, the development and presenta-
tion of the training data, as well as the training regime itself. We describe these next.
In two of the experiments, only three characters are depicted, the representation of
which can be propagated directly to the network’s hidden layer. In the other two exper-
iments, the scene featured three characters involvedin two events (e.g., pirate-washes-
princess and princess-paints-fencer, as shown in Figure 3). The middle character was

Page 5

A Connectionist Model of Anticipation in Visual Worlds853
hidden layer
context layer
event layers
waescht PrinzessinPirat
PAT
input layer
waescht
Fig.3. Scene Integration
involved in both events, either as an agent or a patient (e.g., princess). Only one of the
events, however, corresponded to the spoken linguistic input.
The representation of this scene information and its integration into the model’s
processing was the primary modification to the SRN. Connections between representa-
tions for the depicted characters and the hidden layer were provided. Encoding of the
depicted events, when present, required additional links from the characters and de-
picted actions to event layers, and links from these event layers to the SRN’s hidden
layer. Representations for the events were developed in the event layers by compress-
ing the scene representations of the involved characters and depicted actions through
weights corresponding to the action, its agent and its patient for each event. This event
representation was kept simple and only provided conceptual input to the hidden layer:
whodid what to whomwas encodedforbothevents,when depicted;richergrammatical
information (e.g., case and gender on articles) only came from the linguistic input.
Neural networks will usually encode any correlations in the data that help to min-
imize error. In order to prevent the network from encoding regularities in its weights
regarding the position of the characters and events given in the scene (such as, for ex-
ample,that the centralcharacterin the scene correspondsto the first NP in the presented
sentence)whicharenotrelevanttotherole-assignmenttask,onesetofweightswas used
for all characters, and another set of weights used for both events. This weight-sharing
ensured that the network had to access the information encoded in the event layers, or
determine the relevant characters itself, thus improving generalization. The representa-
tions for the characters and actions were the same for both input (scene and sentence)
and output.

Page 6

854M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle
The input assemblies were the scene representations and the current word from the
input sentence. The output assemblies were the verb, the first and second nouns, and an
assembly that indicated whether the first noun was the agent or patient of the sentence
(token PAT in Figure 3). Typically, agent and patient assemblies would be fixed in a
case-role representation without such a discriminator, and the model required to learn
to instantiate them correctly [10]. However, we found that the model performed much
better when the task was recast as having to learn to isolate the nouns in the order in
which they are introduced,and separately mark how those nouns relate to the verb. The
input and output assemblies had 100 units each, the event layers contained 200 units
each, and the hidden and context layers consisted of 400 units.
2.4Input Data, Training, and Experiments
We trainedthe networktocorrectlyhandlesentences involvingnon-stereotypicalevents
as well as stereotypical ones, both when visual context was present and when it was ab-
sent. As over half a billion sentence/scene combinations were possible for all of the
experiments, we adopted a grammar-based approach to randomly generate sentences
and scenes based on the materials from each experiment while holding out the actual
materials to be used for testing. Because of the complementary roles that stereotypical-
ity played in the two sets of experiments, there was virtually no lexical overlapbetween
them. In order to accurately model the first two experiments involving selectional re-
strictions on verbs, two additional words were added to the lexicon for each charac-
ter selected by a verb. For example, in the sentence Der Hase frisst gleich den Kohl,
the nouns Hase1, Hase2, Kohl1, and Kohl2 were used to develop training sentences.
These were meant to represent, for example,words such as “rabbit” and “jackrabbit” or
“carrot” and “lettuce” in the lexicon that have the same distributional properties as the
orignal words “hare” and “cabbage”. With these extra tokens the network could learn
that Hase, frisst, and Kohl were correlated without ever encountering all three words
in the same training sentence. The experiments involving non-stereotypicality did not
pose this constraint, so training sentences were simply generated to avoid presenting
experimental items.
Some standard simplifications to the words have been made to facilitate modelling.
For example, multi-word adverbs such as fast immer were treated as one word through
hyphenation so that sentence length within a given experimental set up is maintained.
Nominal case markings such as -n in Hasen were removedto avoid sparse data as these
markings are idiosyncratic, and the case markings on the determiners are more infor-
mative overall. More importantly, morphemes such as the infinitive marker -en and
past participle ge- were removed, because, for example, the verb forms malt, malen,
and gemalt, would all be treated as unrelated tokens, again contributing unnecessarily
to the problem with sparse data. The result is that one verb form is used, and to per-
form accurately, the network must rely on its position in the sentence (either second or
sentence-final), as well as whether the word von occurs to indicate a participial reading
rather than infinitival. All 326 words in the lexicon for the first four experiments were
given random representations.
We trained the network by repeatedly presenting the model with 1000 randomly
generated sentences from each experiment (constituting one epoch) and testing every

Page 7

A Connectionist Model of Anticipation in Visual Worlds855
100 epochs against the held-out test materials for each of the five experiments. Scenes
were providedhalf of the time to provide an unbiased approximationto linguistic expe-
rience. The network was initialized with weights between -0.01 and 0.01. The learning
rate was initially set to 0.05 and gradually reduced to 0.002 over the course of 15000
epochs. Four splits took a little less than two weeks to complete on 1.6Ghz PCs.
2.5Results
Figure 4 reports the percentage of targets at the network’s output layer that the model
correctly matches, both as measured at the adverb and at the end of the sentence. The
model clearly demonstrates the qualitative behavior observed in all four experiments in
that it is able to access the informationeither from the encodedscene or stereotypicality
and combine it with the incrementally presented sentence to anticipate forthcoming
arguments.
85
90
95
100
Exp 1Exp 2Exp 3 Exp 4
Percentage Correct
Adverb
NP2
Fig.4. Results
For the two studies using stereotypical information (experiments 1 and 2), the net-
work achieved just over 96% at sentence end, and anticipation accuracy was just over
95% at the adverb. Because these sentences are unambiguous, the model is able to
correctly identify the role of the upcoming argument, but makes errors in token iden-
tification, confusing words that are within the selectionally restricted set, such as, for
example, Kohl and Kohl2. Thus, the model has not quite mastered the stereotypical
knowledge, particularly as it relates to the presence of the scene.
In the other two experimentsusing non-stereotypicalcharacters and depicted events
(experiments 3 and 4), accuracy was 100% at the end of the sentence. More impor-
tantly, the model achieved over 98% early disambiguation on experiment 3, where the
sentences were simple, active SVO and OVS. Early disambiguation on experiment 4
was somewhat harder because the adverb is the disambiguating point in the sentence
as opposed to the verb in the other three experiments. As nonlinear dynamical sys-
tems, neural networks sometimes require an extra step to settle after a decision point is
reached due to the attractor dynamics of the weights. For both experiments,most errors
occurred on role-assignment due to the initially-ambiguous first noun phrase.

Page 8

856M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle
The difference in performance between the first two experiments and second two
experiments can be attributed to the event layer that was only available in experiments
3 and 4. Closer inspection of the model’s behavior during processing revealed that finer
discrimination was encoded in the links between the event layers and hidden layer than
that encoded in the weights between the characters and the hidden layer.
3 Simulation 2
The previous set of experiments demonstrated the rapid use of either linguistic knowl-
edge or depicted events to anticipate forthcoming arguments in a sentence. A further
important question is the relative importance of these two informational sources when
theyconflict. We first review an experimentalstudy by[6] designedto addressthis issue
and then report relevant modelling results.
Fig.5. Scene vs Stored Knowledge
Scene vs Stored Knowledge. One goal of the study by [6] was to verify that stored
knowledge about non-depicted events and information from depicted, but non-stereo-
typical, events each enable rapid thematic interpretation. Case-marking on the first NP
always identified the pilot as a patient. After hearing the verb in (9) more inspections to
the only food-serving agent (detective) than to the other agent showed the influence of
depicted events. In contrast, when people heard the verb in condition two (10), a higher
proportion of anticipatory eye-movementsto the only stereotypical agent (wizard) than
to the other agent revealed the influence of stereotypical knowledge (see Figure 5).
(9) Den Piloten verk¨ ostigt gleich der Detektiv.
The pilotaccserves-food-to shortly the detectivenom.
(10) Den Piloten verzaubert gleich der Zauberer.
The pilotaccjinxes shortly the wizardnom.
Second,the studydeterminedthe relativeimportanceofdepictedeventsandverb-based
thematic role knowledgewhen these informationsources competed.In conditions three
andfour((11)& (12))participantsheardan utterancein whichtheverbidentifiedbotha

Page 9

A Connectionist Model of Anticipation in Visual Worlds 857
stereotypical (detective) and a depicted agent (wizard). In this case, people preferred to
rely on the immediate event depictions over stereotypical knowledge, and looked more
often at the wizard, the agent of the depictedevent, than at the other,stereotypical agent
of the spying-action (the detective).
(11) Den Piloten bespitzelt gleich der Zauberer.
The pilotaccspies-on shortly the wizardnom.
(12) Den Piloten bespitzelt gleich der Detektiv.
The pilotaccspies-on shortly the detectivenom.
3.1Architecture, Data, Training, and Results
In simulation 1, we modelledexperimentsthat dependedon stereotypicalityor depicted
events, but not both. The experiment modelled in simulation 2, however, was specif-
ically designed to investigate how these two information sources interacted. Accord-
ingly,the network neededto learn to use either informationfrom the scene or stereotyp-
icality when available, and, moreover, favor the scene when the two sources conflicted,
as observed in the empirical results. Recall that the network is trained only on the final
interpretation of a sentence. Thus, capturing the observed behavior required manipula-
tion of the frequencies of the four conditions described above during training. In order
to train the network to develop stereotypical agents for verbs, the frequency that a verb
occurs with its stereotypical agent, such as Detektiv and bespitzelt from example (12)
above, had to be greater than for a non-stereotypical agent. However, the frequency
should not be so great as to override the influence from the scene.
The solution we adopted is motivated by theories of language acquisition that take
into account the importance of early linguistic experience in a visual environment (see
the General Discussion). We found a small range of frequencies that permitted the net-
work to develop an early reliance on the information from the scene while it gradually
learnedthe stereotypicalassociations. Figure6 shows the effect this trainingregimehad
over6000epochsonthe abilityofthe networkto accuratelyanticipatethe missingargu-
.
0.7
0.75
0.8
0.85
0.9
0.95
1
0 1000 2000 3000
Epochs
4000 5000 6000
Cond 1
Cond 2
Cond 3
Cond 4
Percentage
Fig.6. Acquisition of Stereotypicality

Page 10

858M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle
ment in each of the four conditions described above when the ratio of non-stereotypical
to stereotypical sentences was 8:1. The network quickly learns to use the scene for con-
ditions 2-4 (examples 10-12), where the action in the linguistic input stream is also
depicted, allowing the network to determine the relevant event and deduce the missing
argument. (Because the graph shows the accuracy of the network at anticipating the
upcoming argument at the adverb, the lines for conditions 3 and 4 are, in fact, identi-
cal.) But condition 1 (sentence 9) requires only stereotypical knowledge. The accuracy
of condition 1 remains close to 75% (correctly producing the verb, first NP, and role
discriminator, but not the second NP) until around epoch 1800 or so and then gradually
improves as the network learns the appropriate stereotypical associations.
Results from several separate runs with different training parameters (such as learn-
ing rate and stereotypicality ratio) show that the network does indeed model the ob-
served experimental behavior. The best results thus far exceed 99% accuracy in cor-
rectly anticipating the proper roles and 100% accuracy at the end of sentence.
As in simulation 1, the training corpus was generated by exhaustively combining
participants and actions for all experimental conditions while holding out all test sen-
tences. However,we foundthat we were able to use a larger learningrate, 0.1,than 0.05
as in the first simulation.
Analysis of the network after successful training suggests why this training pol-
icy works. Early in training, before stereotypicality has been encoded in the network’s
weights, patterns are developed in the hidden layer once the verb is read in from the in-
putstreamthatenablethenetworktoaccuratelydecodethatverbintheoutputlayer.Not
surprisingly,the network uses these same patterns to encode the stereotypical agent; the
only constraint for the network is to ensure that the scene can still override this stereo-
typicality when the depicted event so dictates.
4 General Discussion and Future Work
The model demonstrates that reliance on correlations from distributional information
in the linguistic input and the scene during training of the model enabled successful
modelling of on-line utterance comprehensionboth in the presence and absence of rich
visual contexts. The model that we present acquires stereotypical knowledge from dis-
tributionalpropertiesoflanguageduringtraining.Themappingfromwords to thescene
representations is established through cooccurrence of scene-related utterances and de-
picted events during training. The network that emerges from this training regime suc-
cessfully models five visual worlds eye-trackingexperimentsin two simulations. A first
simulation of four experiments models the influence of either thematic and syntactic
knowledgein the utterance[8], or of depictedevents showingwho-does-what-to-whom
on incremental thematic role assignment [9]. Crucially in modelling the fifth experi-
ment we are able to account for the greater relative priority of depicted events when
event depictions and event knowledge conflict with each other.
The simple accuracyresults belie the complexityofthe task in bothsimulations.For
experiments 3 and 4, the network has to demonstrate anticipation of upcoming roles
when the scene is present, showing that it can indeed access the proper role and filler
from the compressed representation of the event associated with the verb processed in

Page 11

A Connectionist Model of Anticipation in Visual Worlds859
the linguistic stream when available. This task is rendered more difficult because the
appropriate event must be extracted from the superimposition of the two events in the
scene, which is what is propagated into the model’s hidden layer. In addition, it must
also still be able to process all sentences correctly when the scene is not present.
Simulation 2 is more challengingstill. The experiment shows that informationfrom
the scene takes precedence when there is a conflict with stereotypical knowledge; oth-
erwise, each source of knowledge is used when it is available. In the training regime
used in this simulation, the dominance of the scene is established early because it is
much more frequent than the more particular stereotypical knowledge. As training pro-
gresses, stereotypical knowledge is gradually learned because it is sufficiently frequent
for the network to capture the relevant associations. As the network weights gradually
saturate, it becomes more difficult to retune them. But encoding stereotypical knowl-
edge requires far fewer weight adjustments, so the network is able to learn that task
later during training.
According to the “Coordinated Interplay” account in [7,6,11], the rapid integration
of scene and utterance information and the observed preferred reliance of the compre-
hension system on the visual context over stored knowledge might best be explained
by appealing to bootstrapping accounts of language acquisition. The development of
a child’s world knowledge occurs in a visual environment, which accordingly plays a
prominent role during language acquisition. The fact that the child can draw on two in-
formational sources (utterance and scene) enables it to infer information that it has not
yetacquiredfromwhatit alreadyknows.Bootstrappingaccountsforthefact thata child
can correlateeventstructure fromthe world aroundit with descriptionsof events.When
a child perceives an event, the structural information it extracts from it can determine
how the child interprets a sentence that describes the event in question.The incremental
interpretation of a sentence can in turn direct the child’s attention to relevant entities
and events in the environment. Events are only present for a limited time when utter-
ances refer to such events during child language acquisition. This time-limited pres-
ence might determine the tight coordination with which attention in the scene interacts
with utterance comprehension and information extracted from the scene during adult
language comprehension.This contextual development may have shaped both our cog-
nitive architecture(i.e., providingfor rapid, seamless integrationof scene and linguistic
information), and comprehension mechanisms (e.g., people rapidly avail themselves of
information from the immediate scene when the utterance identifies it).
The model presented in this paper extendscurrent models of on-line utterance com-
prehension when utterances relate to a scene [12] in several ways. Existing models ac-
count for processes of establishing reference in scene-sentence integrationwhen scenes
contain only objects. Our network accounts for processes of establishing reference, and
furthermoremodelstherapidassignmentofthematicrolesbasedonlinguisticandworld
knowledge, as well as scene events. In this way, it achieves rapid scene-utterance inte-
gration for increasingly rich visual contexts, includingthe construction of propositional
representations on the basis of scene events. It models the integration of utterances
and relatively rich scenes (that contain actions and events) in addition to objects. Fur-
thermore, the model–in line with experimental findings – successfully accounts for the
relative priority of depicted events in thematic interpretation. It importantly achieves

Page 12

860M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle
this through a modification of the training regime that prioritizes scene information.
This confirms suggestions from [7] that a rapid interplay between utterance compre-
hension and the immediate scene context during acquisition is one potential cause for
the relative priority of depicted events during on-line comprehension.
Connectionist models such as the SRN have been used to model aspects of cogni-
tive development, including the time-course of emergent behaviors [13], making them
highly suitable for simulating developmental stages in child language acquisition (e.g.,
first learningnamesofobjectsintheimmediatescene,andlater proceedingtotheacqui-
sition of stereotypical knowledge). The finding that modelling this aspect of develop-
mentprovidesanefficientwaytonaturallyreproducetheobservedadultcomprehension
behaviorpromisestoofferdeeperinsightinto howadultperformanceis at least partially
a consequence of the acquisition process.
Future research will focus on combining all of the experiments in one model, and
expand the range of sentence types and fillers to which the network is exposed. The
architecture itself is being redesignedto scale up to much more complex linguistic con-
structions and have greater coverage while retaining the cognitively plausible behavior
described in this study [14].
Acknowledgements
The first two authors were supported by SFB 378 (project “ALPHA”), and the third au-
thorby a PhD studentship(GRK 715),all awardedby the German Research Foundation
(DFG).
References
1. Steven Pinker. How could a child use verb syntax to learn verb semantics? In Lila Gleitman
and Barbara Landau, editors, The acquisition of the lexicon, pages 377–410. MIT Press,
Cambridge, MA, 1994.
2. Cynthia Fisher, D. G. Hall, S. Rakowitz, and Lila Gleitman. When it is better to receive
than to give: Syntactic and conceptual constraints on vocabulary growth. In Lila Gleitman
and Barbara Landau, editors, The acquisition of the lexicon, pages 333–375. MIT Press,
Cambridge, MA, 1994.
3. Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14:179–211, 1990.
4. Martin Redington, Nick Chater, and Steven Finch. Distributional information: A powerful
cue for acquiring syntactic categories. Cognitive Science, 22:425–469, 1998.
5. Deb Roy and Alex Pentland. Learning words from sights and sounds: A computational
model. Cognitive Science, 26(1):113–146, 2002.
6. Pia Knoeferle and Matthew W. Crocker. Stored knowledge versus depicted events: what
guides auditory sentence comprehension. In Proceedings of the 26th Annual Conference of
the Cognitive Science Society. Mahawah, NJ: Erlbaum, 2004. 714–719.
7. Pia Knoeferle and Matthew W. Crocker. The coordinated interplay of scene, utterance, and
world knowledge: evidence from eye-tracking. submitted.
8. Yuki Kamide, Christoph Scheepers, and Gerry T. M. Altmann. Integration of syntactic and
semantic information in predictive processing: Cross-linguistic evidence from German and
English. Journal of Psycholinguistic Research, 32(1):37–55, 2003.

Page 13

A Connectionist Model of Anticipation in Visual Worlds 861
9. Pia Knoeferle, Matthew W. Crocker, Christoph Scheepers, and Martin J. Pickering. The
influence of the immediate visual context on incremental thematic role-assignment: evidence
from eye-movements in depicted events. Cognition, 95:95–127, 2005.
10. Risto Miikkulainen. Natural language processing with subsymbolic neural networks. In
Antony Browne, editor, Neural Network Perspectives on Cognition and Adaptive Robotics,
pages 120–139. Institute of Physics Publishing, Bristol, UK; Philadelphia, PA, 1997.
11. Pia Knoeferle and Matthew W. Crocker. The coordinated processing of scene and utterance:
evidence from eye-tracking in depicted events. In Proceedings of International Conference
on Cognitive Science, Allahabad, India, 2004.
12. Deb Roy and Niloy Mukherjee. Towards situated speech understanding: Visual context prim-
ing of language models. Computer Speech and Language, 19(2):227–248, 2005.
13. JeffreyL.Elman, ElizabethA. Bates,Mark H.Johnson, AnnetteKarmiloff-Smith,Domenico
Parisi, and Kim Plunkett. Rethinking Innateness: A Connectionist Perspective on Develop-
ment. MIT Press, Cambridge, MA, 1996.
14. Marshall R. Mayberry and Matthew W. Crocker. Generating semantic graphs through self-
organization. In Proceedings of the AAAI Symposium on Compositional Connectionism in
Cognitive Science, pages 40–49, Washington, D.C., 2004.